“A desk is a dangerous place from which to view the world.” - John Le Carré
Technology debt, part one
In the late 90’s, after 96-97, I was a part of the demo scene in Slovenia. This was a somewhat informal gathering of like-minded hackers, programmers, artists, musicians who liked to have fun. There was a vibrant BBS scene at that time, where at least three or four BBS were active, one of them my own. I’m not even sure why we had BBS now, but before IRC and demo parties it seemed to be a good way to get a C++ compiler and read several tutorials and e-zines.
Amongst the digital underground, a few books and other materials were widely circulated. The memorable ones include the Phrack ezine, the Anarchist cookbook, and the Hacker Manifesto. Among the more comedic literature dealing with computers, you could read the adventures of the Bastard operator from Hell, and should absolutely take a look at the Tao of Programming (book).
To quote the latter:
A well-written program is its own heaven;
a poorly-written program is its own hell.
There is sometimes a very fine line between a new feature and technology debt. The difference between them, in my opinion, is that a feature consists of a requirement, which wasn’t expected or provisioned when an application was in it’s design stage. Technology debt is already a feature, which somehow slipped past some kind of quality assurance or just normal load or sanity checks, and presents a problem in it’s current state. This feature needs to be adapted, rewritten or completely reimplemented from the architecture standpoint.
"Coffee Machine, Right-hand Dispenser - The dispenser in all coffee machines at Valve that holds the decaffeinated coffee beans. To the best of our knowledge, these have never needed to be refilled. For all we know, the beans are decorative plastic."
— Valve: Handbook for new employees. My kind of people. (via baristacoffee)
"There are only two hard things in Computer Science: cache invalidation and naming things."
— Phil Karlton
MySQL optimization and diagnostics tactics
We have all been there. There’s that server, which has mysterious spikes in load, which come from nowhere and leave just as they came - quickly and unexplained. While sometimes these problems can be explained with slow queries, given enough time, development and optimization, you will run into scenarios that don’t register in the slow query log.
It is my opinion, that any kind of database load problem can be traced back to development. It’s a general extension of “You are writing unoptimized queries” to “You’re calling your optimized queries too often” or just maybe “The traffic you generate from the database is too much” or “The code you wrote does not scale to the number of users you have”. But picking up on those problems in production systems with high load and high availability is tricky.
This is why, my current best option is using tcpdump to save traffic in specific intervals, especially when the database is experiencing high load I wish it didn’t. But saving tcpdump data is only part of the solution, a first step. And here is where Percona’s Toolkit comes in. With it’s pt-query-digest, the tcpdump logs can be parsed and you can find out which operations to the database cause the load. The queries are sorted by time summary and also include the information about how many times they were called and how long the queries took to run, during this time when tcpdump was running.
This is all fine and well if you can afford to run tcpdump for long periods of time. Given the “production” and “high availability” part of your mysql deployments however, this might not be the ideal way. Running tcpdump for a few minutes might bog down your network traffic as well as raise cpu and disk utilization on the database server significantly, causing issues for your customers or users. You can run tcpdump remotely on one of your many worker nodes, and have much less impact on the database server. Or you can just run it on the database server, and pipe the output to a remote server, so you don’t increase the load on the database servers disks, while you’re saving the tcpdump data. And, especially for big clients, or clients that have the proper network equipment, the use of port mirroring is a good option.
Personally, I can’t run tcpdump for long periods of time on a specific database server. Not only I don’t get usable data for it, but like slow queries, the problematic queries in that specific system are a small percentage of the overall queries. Only certain tasks or actions from the applications trigger spikes on the database server, which occur in pretty much random intervals. My goal is to debug the load during theese spikes.
My tactics for this were simple. Judging by the graphs which display those spikes we don’t like, we define a threshold value of what cpu utilization is required to start our tcpdump process. With a little modification of this script that calculates CPU usage, we start tcpdump at times when we cross this threshold value.
Using this tactic, we already fixed two problems during the first hour this was tested out, regarding incorrectly set indexes for specific queries that were fast enough not to reach the slow query log, but were being run several thousand times at specific intervals and always ended up on the top of the list.
We have identified several other problems which indicate design issues in the applications using this database, not necessarily related to indexes but just poor normalization which ends up in overly complex queries. Some issues could be solved by smaller changes in design, while others would be better handled outside the database by using Redis or other data structure server, which supports intelligent sorting. Either way, the important thing to remember is, that this approach sheds light on what some problems are, so that resources can be directed towards fixes or new implementations.
Hopefully, this is a first step to analyzing load on your database server that takes you beyond what you find in the slow query log. For us, it has been just that.
- Tit Petric
It’s better to think about a problem for one hour, than trying to solve it in one day.
(Source: engineering-laughter)
Japanese cheat sheet. I think I’ve learned what I need to know :)
Maintaining broken code
Time moves on. Technology improves, sometimes taking care of bad decisions in the past, making it necessary to fix things related to that. PHP changed over the years in small and subtle ways, from register_globals, short_tags, autoglobals, and will hopefully continue to improve in the years to follow. But some things are not a design fault in PHP. Some things are a direct cause of little to no thought in the process of using PHP. PHP itself is a tool, and a powerful one. Unfortunately, it is as good of a tool, as the person using it. And the person who wrote this, was definitely a tool:
// unset unwanted variables created by register_globals
$var_list = get_defined_vars();
$safelist = array(‘_GET’, ‘_POST’, ‘_COOKIE’, ‘_SERVER’, ‘_ENV’, ‘_FILES’, ‘_REQUEST’, ‘_SESSION’);
foreach($var_list as $name => $value)
{
if(array_search($name, $safelist) === FALSE)
{
unset($$name);
}
}
unset($var_list, $name, $value, $safelist);
I died for an hour, figuring out what the hell caused $GLOBALS to be null, in a piece of code after the one above. Apparently, $GLOBALS did not show up in get_defined_vars() in older PHP versions, a thing which kicks you in the ass five years later.
As you see, the author asumed that:
- register_globals was turned on (it wasn’t!)
- that the auto globals list was fixed and would not change
When doing destructive things (ie, variable variables + unset), one should give some afterthought to what needs to be accomplished. Obviously the goal here was security, but the end result was a defunct product. Especially since it was using $GLOBALS originally.
Disclaimer: Code was written by third party and ordered by a client about 5 years ago. While being far from documented, and including some comments in finnish (dear god, perkele vittu), it is still in pretty good shape. It fails in what I observe to be a common pitfall of similar software today - it tries to do too much.
I myself use variable variables, but not as stupidly as the example above. The main problem with them is trying to explain them to anyone else than yourself. But if you know what you’re doing, they can be very powerful, like many other PHP features.
I need anger management classes after this.
- Tit Petric
The capacitor plague and wasted hardware
After being involved in computers for around 15 years now, every once in a while I experience one common occurrence and that is the failing of hardware. It’s been a few weeks since I had a failed disk. It’s been this week that I realized that three monitors died at our office in the last few years. Hardware seems to be dieing more recently than not. It’s a shame a lot of components seem to be usable, but are usually being discarded along with perfectly functioning hardware.
This is a power supply, out of a cheap desktop PC which was purchased for about $400 back in the day, today being practically worthless. The power supply is not functional, but the rest of the computer is functional and in good shape. It is almost surprising to me, that a common problem with inflated capacitors did not occur on this even-then budget machine.
The problem with this is, that as an almost worthless machine, it would cost $180 to replace the power supply with a new or refurbished one. At that cost it is more appropriate to put the same amount into a new hardware setup, which is more powerful and aesthetic and has media capabilities.
I find this sad. Obviously, investing in new hardware is needed, however creating waste out of perfectly functioning hardware seems … wasteful. The hardware has possibility for use in a non-critical environment (like your grandmothers apartment, if she isn’t filthy rich or is given iPads as birthday presents). It is more than capable to perform most tasks you and anyone do online and can even be used for gaming (Especially if you’re a fan of Quake Live).
So how where will this machine end up? It will be moved to our internal LAB workbench, where a Linux cluster will be set up, and will be used to test software as well as test hardware and network equipment. While it won’t be used for any real work, there is a lot of research that can be done on this aging equipment, and hopefully it will take a few more years, before we have to throw it out completely.
Unless I manage to get my hands on a 150W or 160W picoPSU for about $70 (or even less if possible), it will just have to run out off a standard size ATX PSU. Thankfully, I have a few of those in working condition just lying around collecting dust.
- Tit Petric
Working in silence …
Our friend blogged about an interesting subject. Of productivity and workplace noise. I would add one more opinion here. I have an office for myself and I know this enables me to work better, because I don’t have to consider whom I could disturb when my thinking process becomes more physical.
When I was in a shared office, I know I was (sometimes) bothering coworkers with the ocasional “hmm”, “wtf” or “aha” that come from doing research (or reading someone else’s code). More intense creative thinking could become even more interruptive.
