Darlenes company uses Siebel for managing their enterprise. Like most enterprise software packages, its complicated, incomprehensible, and any significant maintenance depends on very expensive consultants.

During an upgrade, one of those Highly Paid Consultants caught a new requirement: customers wanted to be able to change an order, replacing one product code with another, all the way up until the order went into fulfillment. Now, the logical thing would have been to cancel the changed order line and create a new one, but our HPC couldnt quite figure out how to cancel an individual line item, so he just decided to delete it instead.

This is the eScript (Siebels proprietary version of JavaScript) code he implemented:

var orderLineItemsBO = TheApplication().GetBusObject("Order Entry - Line Items");
var orderLineItemsBC = orderLineItemsBO.GetBusComp("Order Entry - Line Items");
orderLineItemsBC.ClearToQuery();
orderLineItemsBC.SetSearchSpec("Order Id", myOrderId);
orderLineItemsBC.SetSearchSpec("Product", "Garbage Product"); //"Garbage Product" is an example name of what a product code might be.
orderLineItemsBC.ExecuteQuery(ForwardOnly);
var record = orderLineItemsBC.FirstRecord();
while(record)
{
orderLineItemsBC.DeleteRecord();
record = orderLineItemsBC.FirstRecord();
}

Now, this code doesnt look terrible, aside from the lack of indenting, but immediately after this went into test, the QA team started complaining about performance. Editing an order takes an extremely long time. There were other, bigger requirements, editing an order wasnt a commonly used feature, and eventually the project managers just said, Its good enough, and shipped it into production.

And thats when all hell broke loose. People would place orders for product code, say, PQ1236, but before their order shipped, that line would be removed from their order without them doing anything.

The culprit was this specific line: orderLineItemsBC.SetSearchSpec("Order Id", myOrderId);.

Now, you might not know whats wrong with this, but youre not a highly paid Siebel consultant. The issue is that myOrderId isnt validated anywhere in the function, and if a null gets passed in, Siebel decides that you must want a wildcard search, and returns every order in the system.

Thats why performance was terrible, and it also highlighted a failure in their testing methodology: no one ever looked at old orders during the test process, otherwise they might have noticed the issue.

[Advertisement] Universal Package Manager - ProGet easily integrates with your favorite Continuous Integration and Build Tools, acting as the central hub to all your essential components. Learn more today!

http://thedailywtf.com/articles/lines-and-lines-and-lines-of-order-lines

It was a lazy, drowsy Saturday afternoon. The sun was shining, birds were singing. The kind of day when children should be playing outside, perhaps running bases in a sandlot someplace, carefree and smiling. Even indoors, thanks to the cost-saving measures at Big Online Retail Store™ HQ, it was warm enough to send tantalizing daydreams of comfortable naps in soft places to the employees working the weekend shift.

Production code pushes, of course, were anything but lazy. There were checks and balances, and the checks and balances had checks and balances. There was tension, and urgency, and the stakes were clear to all involved: don't you dare make a typo or you'll bring the whole company down. Most of the system was automated, and the rest of it was scripted by the developers who nurtured the system like a fussy toddler, willing to cater to its whims if it would just stop crying and let them get some sleep.

On the wall of the Ops War Room, a bright red digital counter ticked down the days: 13 days until Black Friday. Printed memes were tacked to the walls around it, promising dire consequences for screwing up the deployments now. The developers working on hardening the systems were haggard, bleary, frayed around the edges, and, at the time of this story, home sleeping off the previous night's deployment.

But there was a second system at Big Online Retail Store, one that had nowhere near the same oversight: the internal systems that monitored network traffic to the site and pulled out analyses from it. While the bean-counters were counting on the information it would provide during the Christmas season, the lead-up was so far more of a lazy river rafting ride, gently drifting toward the moment when the passengers would disembark and once more have to move under their own power. These systems didn't have teams of people dedicated to every aspect of their existence. Instead, a few developers maintained the dev and production environments, managing the servers themselves, the last vestiges of the maverick mentality that had gotten Big Online Retail Store this far.

On this particular Saturday, our hero Ashton was given the task of commissioning five new servers for the distributed network. They had been provisioned in a data center, shiny and new and ready for production; the system was architected as a series of small components that could be hosted on any machine, so he had about 20 machines to reconfigure in order to spread out the load evenly.

The building was nearly empty at this time of day, making it perfect for zoning out with some monotonous work and some nice, laid-back music. Ashton had definitely had worse Saturdays.

The routine was pretty straightforward:

Fifteen servers in, Ashton's mind was well and truly out the window. Is it warm enough that there'll be a line at the ice cream place? he pondered.

sudo root

Probably. Maybe the deli won't be too bad though.

Paste the new configuration lines.

Hmm, deli ... pickles ... would I rather have fried pickles than ice cream?

dd to remove the old lines.

Definitely fried pickles. All right, I'll finish this up, get some lunch, head home.

telinit 1

Maybe there's still time to wash my car before—hang on a minute.

Ashton had made a horrible mistake.

On a standard QWERTY keyboard, the q key is a hair's breadth away from the 1 key—and Ashton had hit the wrong key. While telinit q would simply instruct the init service to reload the configuration file, telinit 1 would drop the entire system down to runlevel 1: single-user mode, in which certain superfluous services are stopped and only one user at a time can run programs. Superfluous services like, for example, networking. You didn't need it to run, after all, as long as you were sitting in front of the box, typing away at an attached keyboard. As long as you weren't doing something stupid, like controlling the server from your desk via PuTTY while daydreaming about pickles.

Unable to reconnect, Ashton went straight to the Network On-Call person, Tori, messaging him via IM to explain the situation.

"OK, one sec, I'll bring it back up," Tori replied, and Ashton breathed a sigh of relief.

But not for very long. A moment later, Tori came back to the IM: "Hey, where's the bleedin' server? Did you restart it?"

Ashton felt his heart sink. Clearly he hadn't explained well enough. "Um ... I told ya ... I dropped it to run level 1."

"OK..... what was the machine name again?" Tori responded, a few excruciating moments later.

Ashton told him.

What followed was the stuff of legend. We at The Daily WTF are ever conscious of our public availability, and as such, have a moral duty not to repeat the precise text Tori sent in reply to Ashton, as expletives of that sort have been recently classified in some states as class-3 weapons of mass destruction. The string filled Ashton's screen, the verbal equivalent of a full ten-minute rant. It was the sort of rant that the Internet makes possible, wherein the aggrieved party, unhindered by a need to breathe, can go on and on, becoming more and more inventive with each suggestion.

Finally, the rant ceased, culminating in a final cry of dismay: "IT'S IN SALT LAKE CITY!"

Ashton didn't dare think about anything but his work as he finished the last four servers, keeping a nervous eye on Tori's "Do Not Disturb" icon the whole time. Finally, just as he was finishing up the last one, Tori messaged him again: "Called the datacenter. Took half an hour to find the box."

"Thank you!! I owe you!" Ashton replied, relieved, as he pinged the errant server and got a response.

"No. You owe Sam in Salt Lake. And you better believe he'll come collecting."

Ashton swallowed, closing out the IM window. Whatever the fallout might be, he knew he had no choice but to face it with grim determination.

[Advertisement] Incrementally adopt DevOps best practices with BuildMaster, ProGet and Otter, creating a robust, secure, scalable, and reliable DevOps toolchain.

http://thedailywtf.com/articles/a-costly-slip

Teppo works for a Finnish company that, among other things, develops a few mobile applications. This company is growing, and as growing companies do, it recently purchased another company.

One of the applications that came with this company had a mongrel past. It started as an in-house project, was shipped off to a vague bunch of contractors in Serbia with no known address, then back to an intern, before being left to grow wild with anyone who had a few minutes trying to fix it.

The resulting code logs in a mixture of Serbian and Finnish. Paths and IP addresses are hard-coded in, and mostly point to third party services that have long since stopped working. It has an internal ad-framework that doesnt work. The Git repository has dozens of branches, with no indication which one actually builds the production versions of the application. The back-end server runs a cron script containing lines like this:

* * * * * curl www.google.com  > ~/out.txt
* * * * * echo 'lalala' > ~/out1.txt

Its a terrible application that doesnt even barely work. The real test, of course, for an unsupportable mess of an application is this: how does it handle dates?

public static String getSratdate_time_date(String date) {
        String dtStart = date;
        try {
                SimpleDateFormat format = new SimpleDateFormat(
                        "yyyy-MM-dd HH:mm:ss");
                Date deals_date = format.parse(dtStart);

                String intMonth = (String) android.text.format.DateFormat.format(
                        "M", deals_date); // Jan
                String year = (String) android.text.format.DateFormat.format(
                        "yy", deals_date); // 2013
                String day = (String) android.text.format.DateFormat.format(
                        "dd", deals_date); // 20

                return (intMonth + " / " + day);

        } catch (Exception e) {
                e.printStackTrace();
        }
        return "";
}

This takes a string containing a date and converts it into a string containing M/dd. You may note, I used a date format string to describe what this code does, since the easiest way to write this might have been to do something like… DateFormat.format("M/dd", deals_date), which doesnt seem to be that much of a leap, since they used the DateFormat object.

Bonus points for using Hungarian notation, and triple that bonus for using it wrong.

[Advertisement] Infrastructure as Code built from the start with first-class Windows functionality and an intuitive, visual user interface. Download Otter today!

http://thedailywtf.com/articles/a-dated-inheritance

Most technical folks can recognize a developmestuction environment when they encounter one. The less fortunate among us have had one inflicted upon us. However, the one thing they all seem to have in common is that people simply make changes directly in production. Ive encountered a place that takes the concept to a Whole New Level O WTF™.

The company is a huge international conglomerate with regional offices on 5 continents, spread fairly evenly around the globe. The team for this particular project has several folks (developers, testers, QA, UAT and prod support) in each of the locations. Each region is mostly a self-contained installation of servers, databases and end users, but just to make it interesting, some of the data and messaging is shared across regions. Each region runs the normal business hours in its own time zone. As such, at any given time, one region is always doing intra-day processing, one is always in night time quiet-mode, and the other three are in various stages of ramp up, ramp down, or light traffic.

The application is a monstrously large suite of Java applications. Since each region has its own Java support contract, different regions run the same code base on different versions of Java. Accordingly, the automated build process builds the entire tree in each of several different versions of Java. This brings up all sorts of region-specific problems when theres an issue with a particular release of a JVM. Reproducing them back at development-central inevitably fails because were only allowed to install the version of the JVM for which we have a support contract.

For those who arent familiar with it, the default Java class loader traverses a path, looking for individually specified files, or jar files that it can search for a given class. It will take the first instance of a class that it finds. This means that if there are two or more different versions of a class on the path, only the first one will ever get loaded, yielding the Java version of dll-Hell.

Someone got the idea that to save money, we could abuse leverage the way the Java class loader works, and only have one set of machines in each region to provide dev, integration, QA, UAT and production environments. The basic premise is that you could introduce short-term-use classes earlier in the class path to patch (override) classes that were already there.

The class path search hierarchy was in this order:

    $Project/p-dev     individual developer class patches for testing
    $Project/p-int     team class patches for integration testing
    $Project/p-qa      QA class patches for interim testing between releases to QA
    $Project/p-uat     user acceptance patches between releases to UAT
    $Project/p-prod    patches to prod code between formal releases
    $Project/jars      formal releases

A patch consists of one or more class and/or configuration files (perhaps hundreds) that collectively fix/change/add/delete functionality. The rules for patching were as follows:

1. compile your class changes on your PC
2. grab all the patch files into a dedicated directory tree that mimics the class hierarchy
3. install your patch tree only on a region currently in night-mode
4. announce the (potentially large list of) classes youre patching to all other locations
5. manually ensure that nobody else is patching the same files
6. bounce the target production environment
7. do the testing or debugging
8. remove your patch files from the deployed tree
9. undo all the database changes you made
10. bounce the target production environment - again
11. announce that youre done
12. The underlying principle is that developer changes are very short term (install, test, remove). Changes to the other patch directories are to last until the next release at 1. that level.

Insane though it may have been, it sort-of worked& for small, localized changes. For changes that touched lots of files across many directories, folks routinely missed some files and needed to patch their patches.

Longer term patches for QA and UAT tended to stick around for a while and so were effectively already in production even though they werent formally released. This led to all sorts of interesting situations where users would notice some feature (that was only released to QA) and start using it, only to have it disappear when QA pulled it due to some bug.

However, changes that required structural changes to the database, application or regional interactions tended to have live ripple effects into other regions. As such, the end users were routinely told to expect different event handling, query results or mathematical formulas to change - in production - on an intermittent basis. Sometimes they were told that certain bits of functionality would be unavailable because someone was testing something.

Of course, if more than one person needed to test something, they needed to manually coordinate the patching/bouncing/un-patching/re-bouncing with the other folks.

Occasionally, two folks would need to change different methods of the same class. Since both were testing prior to checkin, neither could see the changes made by the other. This led to a lot of cut-paste-emailing of code in one direction or the other, with subsequent manual undo (source control revert would wipe out both sets of changes for whomever was doing the patching). Naturally, there was the inevitable missed undo that resulted in all sorts of confusion.

One side affect was that if you were changing communication code, the test messages you were sending were on the production network, and depending upon the nature of the message, might be processed in other (un-patched) regions.

An even better side affect was that if you attached a debugger to the process with patched code and hit a breakpoint, you have halted a production process, globally affecting processes that use it. Naturally, the team was advised to avoid using debuggers if at all possible, and if stopped, execution should be resumed as quickly as practical.

The best part was when I noticed test transaction records that were still in the production database and being included in regulatory reports (yeah, that one hit the fan - hard).

When I inquired as to how this could be allowed to happen, let alone flourish, I was told that they didnt want to make waves by insisting that the company buy enough servers to set up different environments.

Customers will only pay to fix a problem when it becomes more expensive to not fix it than to fix it.

I learned my lesson about trying to fix stupid at WTF Inc&

[Advertisement] Otter, ProGet, BuildMaster – robust, powerful, scalable, and reliable additions to your existing DevOps toolchain.

http://thedailywtf.com/articles/patchwork

"The bus sign in one of Tel Aviv's train station gives out some interesting info on where to catch your next bus," wrote Eran C.

"I was trying to apply for All Nippon Airways miles card when I got greeted with this nice error message," writes Francois P., "Maybe there's a trick with Japanese characters that I don't know about?"

Scott B. wrote, "Who'd have though Latin was the dominant language in the country town of Traralgon?"

"Looks like Sears doesn't like it when folks find out that their accounts exist," writes Thomas.

Peter A. wrote, "Usually, I'd jump to take advantage of such a good deal, but in this case, I'm in no rush."

James writes, "If it makes you feel better, yes, iMacros, yes. Whatever that means..."

"Yeah! I'd love a c# job! Pity I don't know any Java," wrote Mattias.

[Advertisement] Onsite, remote, bare-metal or cloud – create, configure and orchestrate 1,000s of servers, all from the same dashboard while continually monitoring for drift and allowing for instantaneous remediation. Download Otter today!

http://thedailywtf.com/articles/the-class-b-bus

In 800BC, if you had a difficult, thorny question, you might climb the slopes of Mount Parnassus with a baby goat, find the Castalian Spring, and approach the Pythia- the priestess of Apollo who served as the Oracle at Delphi. On the seventh day of the month, the Pythia would sanctify her body by bathing in the waters of the spring, drinking water from the Cassotis- a portion of the spring where a naiad was said to dwell- while the high priest would sprinkle holy water about the temple. Thus purified, theyd take your baby goat, lay it before the fires of Hestia, and cut it open to read your answer from its entrails. Unless it trembled the wrong way- that was a bad omen; theyd throw an exception and tell you to try again next month.

All that ritual and pomp and circumstance creates something I call The Oracle Effect. We supply some inputs, and then a fancy, complicated, and difficult to understand ritual is applied to them. The end result is an answer or prediction, and our brains have an interesting blind spot: we tend to weigh the answer more by the complexity and difficulty of the process than the actual value of the process. Ritual creates realism.

Which brings us to this study from Pro Publica, which discusses the use of risk assessment algorithms in the criminal justice system. Our goat in this case, is a collection of demographic and philosophical questions applied to a defendant. The ritual is a complicated statistical analysis (the mechanics of which are only known to the software vendor), and the end result is a simple number: on a scale of 110, how likely is this person to commit another crime in the future?

Spoiler Alert: the software isnt very good at this, and also may be racist. Very few people are looking at the actual accuracy of the results though, and instead, are focused on the process. The ritual looks pretty good, and thus, judges tend to use this risk assessment as a tool to help sentencing- high risk defendants get tougher sentences.

Or take the US Transportation Security Administrations use of behavioral threat analysis, which use a complicated analysis of microexpressions to identify terrorists in a quick glance. Its about as effective as using phrenology to detect potential criminals. It doesnt work, but theres a process! And speaking of phrenology, a startup hopes to bring that back, and just like these other methods, their process has to remain a secret. Gilboa said he… will never make his classifiers that predict negative traits available to the general public.

Its not just secret processes, though. This applies to any sufficiently complicated ritual. In a more prosaic example, Ive implemented a number of dashboards in my career. The goal is usually to take some complicated set of statistics about a business process or project and boil them down to a small collection of stoplight indicators: red, yellow or green lights. No matter how the dashboard starts out, before long, the process for manipulating the data gets complicated, arcane, and downright broken until everything always shows green, because thats what management wants to see. The fact that the output has no real meaning doesnt matter- theres a complicated, difficult to understand process, which puts a layer of perceived truth on that shiny little green light.

Even simple rituals can feed into this Oracle Effect. For example, PayPal doesnt want to handle transactions for ISIS, which isnt unreasonable, but how do you detect which transactions are made by honest citizens, and which by militants? What about just blocking transactions containing the letters isis? This seems like a pretty simple algorithm, but think about the amount of data flowing through it, and suddenly, it picks up the air of ritual- we have a magic incantation that keeps us from processing transactions for militants.

Using algorithms and decision-support systems isnt bad. Its not even bad if theyre complicated! Theyre solving a complicated problem, and wed expect the resulting system to reflect at least some of that complexity. A recent conference hosted at NYU Law spent time discussing how we could actually avoid biases in policing by using well-designed algorithms, despite also pointing out the risks and dangers to human rights. These sorts of decision-making tools can make things better- or worse. They're just a tool.

A few years ago, I stumbled across an ancient book on expert systems, because I find reading technology books from the 1980s enlightening. One of the points the book stressed was that an expert system/decision support system had two jobs: the first was to make a decision within its area of expertise, but the second was to be able to explain its reasoning. The output isnt useful if it cant be justified.

Imagine, if you will, a toddler tagging along at the hem of the Pythias ceremonial robes. As the priestess goes through the ritual, this toddler interrupts at each step, to ask, Why? The Oracle Effect is, at its core, an appeal to authority fallacy in a funny hat. The antidote, then, is to refuse to accept the statements of authorities, no matter how fancy their ritual is. Its not enough to say, oh, well, the software should be open source, then!, because thats frankly neither necessary or sufficient. Decision-support systems can feel free to keep their inner workings secret, so long as they can also provide a convincing argument to justify their conclusions. And we know all too well that merely reviewing the code of a complicated system is not enough to understand how the system actually operates on real-world data.

Many of our readers write un-sexy, line-of-business applications, below the waterline of the Software Development Iceberg (for all the Googles and Microsofts and Facebooks, most software development happens internal to companies that dont make or sell software, to solve their internal business problems). These applications generate data and reports that will be used to make decisions. And this brings us to the call-to-action portion of this soapbox. Work against the Oracle Effect by building software systems that do not provide conclusions, but arguments. Resist throwing some KPIs on a dashboard without working with your users to understand how this feeds into their decision-making. Keep in mind how your software is going to be used, and make sure its value, its reasoning is transparent, even if your code is proprietary. And make sure the software products you use can also answer the important question about every step of their process: Why?

[Advertisement] Otter, ProGet, BuildMaster – robust, powerful, scalable, and reliable additions to your existing DevOps toolchain.

http://thedailywtf.com/articles/the-oracle-effect

Theres a certain class of bad code weve all seen before:

boolean someFunction() {
    if (someBooleanExpression) {
        return true;
    } else {
        return false;
    }
}

Its dumb, but largely harmless. Weve posted variations on this idea in the past. Scott P, however, found a new variation that may be the pinnacle of this kind of simple-minded programming. Ill let him introduce it.

I inherited a project that was developed in isolation by a lone developer that didnt do code reviews. I find something new to be horrified about every day. The developer responsible has a knack for adding unnecessary complexity to what should have been a much simpler application. This method perfectly sums up what the architecture of this application is like on every level. Its layer upon layer of obfuscation or redundant code.

boolean hasFeatures() {
        if (license.getFeatures().size() > 0 ? true : false) {
                return true;
        }
        return false;
}

Honestly, I think the ternary operator doesnt get enough use. We should just start putting ternary expressions all over our code, even if it means writing things like: myvar = true ? valueReturningFunction() : valueReturningFunction();. Isnt that pretty great? I love it.

[Advertisement] Otter allows you to easily create and configure 1,000's of servers, all while maintaining ease-of-use, and granular visibility down to a single server. Find out more and download today!

http://thedailywtf.com/articles/returnary

George, an independent contractor, usually spent his first day home from a business trip plowing through the emails that'd piled up in his absence. In the midst of this grind, he received a call from Lucinda, a Microsoft contact from South Africa.

"We have a small banking client with a tricky performance problem after migrating to SQL Server," she explained. "Apparently, it's so bad that they're about to ditch us and revert to their original COBOL system."

"Oh, wow," George said, swiveling away from his keyboard. "Have you sent any techs over to look at it?"

"Yes. They couldn't find anything wrong." An odd little tone in Lucinda's voice indicated this wasn't the full story. "You're the best database performance-tuner we know. Could you call their IT manager and see what you can do?"

George was a little suspicious. Nevertheless, he told her, "Sure. I enjoy a challenge."

Lucinda gladly provided the contact details. From there, George phoned the client to see if he could indeed be of help.

"We spent a year implementing the system in SQL and C++." The client's IT manager pronounced spent as though he'd wanted to say wasted instead. "We were promised it'd be better, but it's even slower than our current system!"

"Was that implementation done in-house?" George asked.

"Yes—by Dawie, our top COBOL guru. He's the only one who deeply understands the business."

Well, a COBOL guy couldn't be expected to be an expert on SQL or relational databases. To George, this was starting to sound like a slam dunk. Swoop in like a comic book superhero, impelement some indexing, head home a champion.

"OK," George said, "I'd like to do an onsite visit to see the problem. From there, I'll see what recommendations I can come up with. Just keep in mind: with performance issues, it's hard to make any guarantees."

"I understand, really," the IT manager said. "Anything you do is appreciated! We're at our wits' end here."

Despite his standard CYA disclaimer, George felt pretty confident in the weeks leading up to his visit to the client site. At the appointed day and time, the IT manager and Dawie met him in the lobby, and from there it was only a short walk to Dawie's cube.

With apologies, the IT manager departed for a meeting.

"Have a seat," Dawie offered George. "I'll walk you through our system."

The newly implemented system allowed specially authorized users to access and maintain customer data. There was no doubt about it, the performance left something to be desired. Queries returning fewer than 100,000 records each took more than a minute to process, making the simplest of tasks an exercise in Zen patience.

"Could we look at the database schema?" George asked.

"Sure." Dawie opened the management studio, drilled down to the database, then expanded the table list.

Only one table appeared, named RECORDS.

George frowned in confusion. "Are you logged into SQL Server properly? Maybe we're viewing this as a user with limited permissions, and this user can't see all the tables."

The question confused Dawie in return. "'All the tables?' This is everything. Here, look."

Dawie brought up the query window and did a SELECT * on the table. This brought up 60 rows of 2 fields: an auto-numbered ID column, and a Data column containing what looked like massive lines of text.

"Can we look at your code?" George sputtered.

Upon switching to Visual Studio, the horror became all too clear. Dawie the COBOL guru had hit upon the idea to store each COBOL data file from the old system as text inside a SQL Server table. To retrieve a customer record, his code queried the RECORDS table, extracted the appropriate text value to memory, saved it to disk, then used file I/O to scan through and find the appropriate record. If customers were added or edited, the code wrote the updates into the text, then packed it back into the SQL record.

Seriously? George struggled not to gape. "Did you ever receive any SQL training or anything?"

"I got some documentation that I breezed through." Dawie glanced around as though to find said manuals, which were thoroughly lost amid the decades of clutter upon his desk. He shrugged. "Honestly, when you've been around as long as I have, these systems all start to look the same."

"Well." George tugged at his collar. "It certainly looks like you've done all you could. I'm sorry, but I don't think I'll be able to help out after all."

Upon returning to his hotel, George called Lucinda, explaining the situation. "I can't help them. I'm not sure who can. You may just want to accept the account as lost."

"Yeah. That's what our technical team said," Lucinda replied. "They didn't have the courage to tell the client, either."

[Advertisement] Otter, ProGet, BuildMaster – robust, powerful, scalable, and reliable additions to your existing DevOps toolchain.

http://thedailywtf.com/articles/the-mainframe-database

One time, out of boredom, I wrote a little utility called BitVerifier. It would loop over a folder and check every bit of each file. If the bit's value wasn't one or zero, it would prompt the user for the correct bit. At least in theory. I somehow never encountered a file with a "two" bit. But I got one key component right – an understanding of the valid range of values.

Rob K.'s colleague didn't even get that far.

private static final String[] HEX_STRING_VALUES = {"0", "1", "2", "3",
    "4", "5", "6", "7", "8", "9", "10", "A", "B", "C", "D", "E", "F"};

But wait, that's not the whole WTF! Consider this code later in the class:

StringBuilder sb = new StringBuilder();
for ( int h = 0; h < 6; h++ ) {
    int randomHex = randomGen.nextInt(HEX_STRING_VALUES.length);
    sb.append( HEX_STRING_VALUES[randomHex] );
}
//for some crazy reason, sometimes the loop above creates 7 chars.
//that is why the below usees substring to get exactly 6 Hex chars.
defaultVisualStyle.setColor( "0x" + sb.substring( 0, 6 ) );

"Some crazy reason" indeed. For "some crazy reason," the output could be up to twelve characters. It might have something to do with the fact that the "hex" array has 17 values, and that one of them is "10."

For some crazy reason, this individual won't be writing code in this project anymore.

[Advertisement] Otter allows you to easily create and configure 1,000's of servers, all while maintaining ease-of-use, and granular visibility down to a single server. Find out more and download today!

http://thedailywtf.com/articles/classic-wtf-some-crazy-reason

"Nice try ScreenConnect," writes J.R., "I'm wise to your supposed randomly generated access codes."

Herb L. wrote, "I'd like to track more packages but USPS has very strict limits."

"From an error link to status page that redirected to the landing page, finding the status page manually tells us status of the status is unavailable," wrote Jeff S.

"Steam sure is building quite a reputation for its pricing," writes Juan J.

"I already have enough trust issues, Excel," writes Tony.

"Yeah, I didn't expect to fail either," wrote Trevor G.

Jonathan writes, "Either there was something wrong with my feedback or there was something AWESOME about my feedback."

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

http://thedailywtf.com/articles/too-sexy-for-my-support-portal

Henry was a hotshot developer working on a team that specialized in performance tuning. When other teams in his company had performance problems they couldnt solve, Henrys team was called in. Through profiling, analysis of algorithms, and database tuning, his team excelled at turning inefficient, slow, bug-ridden software into applications that actually did real work in a timely manner.

Usually.

Henry was selected to help a customer with issues with a brand-new web application developed by another team in his company. Performance was terrible. Requests often timed out and the servers CPU typically sat at 100% usage for hours at a time. The server admins had to restart server processes multiple times per day to keep the application running.

He met with the lead developer of the other team to discuss their performance woes.

Weve combed through every line of code, the other developer, a young man named Colin, explained. I dont think its our codes fault. I think its the database, but we already tried switching from MySQL to PostgreSQL and it made no difference.

After looking at a few profile reports, Henry agreed that something was off with the database. Who wrote the database layer?

Bob did, Colin replied. Hes our manager. None of us know anything about SQL so he did all the database work. But hes on vacation for the next couple weeks.

Well, lets take a look at the customers database, see if anything stands out. After some back-and-forth with the customer, they were able to get a copy of the database in question. It came in at over thirty gigabytes.

Henry cringed when he opened the database up for inspection. The first thing he found was that primary keys and foreign keys did not always use the same datatype. Often, an ID would be a VARCHAR(36) while any foreign keys referencing it would be some other datatype such as NUMERIC, VARBINARY, or even BLOB, and queries were full of casts and other shims to make JOINing possible. For example,

    SELECT * FROM Product
    INNER JOIN Inventory ON CAST(Product.ID as CHAR(36)) = CAST(Inventory.ProductID as CHAR(36))
    WHERE Inventory.Quantity < 100;

Henry explained the concept of foreign keys and JOINs to Colin, then went on to describe how this was a performance problem. All these casts make MySQL ignore the keys indexes and do a much-slower linear search instead. It will help a lot if primary keys and foreign keys use the same datatype.

Colin nodded as he took notes to pass on to Bob.

Then he noticed that the vast majority of the databases contents were stored in a table called Audit_Table which was filled with VARCHAR(255) fields. The table was populated by a trigger and it appeared that every single possible user action was logged here. Even page refreshes and button clicks generated entries! The table had over 15 million rows in it, despite the application only being in production for a few days so far. And every single column was indexed.

No wonder it was so slow! Any time a user did anything, an audit record was generated. And since every column was indexed, the application server spent most of its time trying to keep all the indexes updated on a massive 30 GB table.

And then Henry noticed one of the columns was this:

    IsDeletable VARCHAR(255) DEFAULT 'Y'

A not-so-quick SELECT UNIQUE(IsDeletable) FROM Audit_Table revealed that every single row had a value of Y for this column. Indexing it was a pointless waste of CPU time.

Henry and Colin spent the week trying to fix the database design without breaking things: correcting mismatched datatypes, removing a massive number of unnecessary indexes, and a multitude of other problems that, all added together, easily explained their performance problems. Henry explained everything he did to Colin as he worked, and they committed their fixes to source control late that Friday.

Come Monday he got a panicked email from Colin. It contained a forward originally from Bob who had sent this while on vacation.

To: colin@initech.com
From: bob.the.phb@initech.com
Subject: unauthorized database changes
i noticed you removed indexes. i rolled it back. those are very important for performance, every column must be indexed!!

i locked the schema in source control so no one but me can edit it. you obviously dont know what youre doing. i suppose that is my fault for not training your team in database.

we will talk more about this infraction when i return from vacation.

Bob The PHB
Senior Development Manager, Development Teams C - H
bob.the.phb@initech.com
(985) 5552500

“If I really want to improve my situation, I can work on the one thing over which I have control - myself.” - Stephen Covey

Sent from my BlackBerry

Bob refused to budge on the database design, claiming that indexes always improved performance and the customer was forced to live with it. Several months later, Henry was pulled in again for a different reason. The customers server kept running out of disk space, due to the sheer size of the Audit_Table table. It was now measured in terabytes. After some investigation Henry realized that nothing actually used the table and nobody was reporting on this tracking data, so he showed Colin and the customer how to TRUNCATE it to free up disk space. As a happy side effect, the customer realized that the application performed much better after the audit table was truncated and agreed to make a nightly scheduled task out of it, finally defeating the performance-killing audit table indexes.

[Advertisement] Universal Package Manager – store all your Maven, NuGet, Chocolatey, npm, Bower, TFS, TeamCity, Jenkins packages in one central location. Learn more today!

http://thedailywtf.com/articles/say-y-to-indexes

Relational Database are a great way to structure data, but they have their warts. Certain kinds of structures dont model well relationally, some are difficult to tune for performance, and some queries are just expensive no matter what. Still, with some smart design choices, some indexes, and some tuning of the execution plan, you can make things work.

Hambai approached a tuning problem with that perspective. The database had a huge pile of financial information- stock transactions, commodity valuations, and currency exchange rates. When it was new, queries were fast, but now, years on, performance ground to a halt. One query that drew his attention was one for accessing the latest exchange rate for four different currencies. It was run frequently, and each access took up to thirty seconds.

Buried deep in a Postgres stored procedure, Hambai found this:

CREATE LOCAL TEMP TABLE cur_index_temp (
   rate_id INTEGER
);

INSERT INTO cur_index_temp(rate_id) VALUES(1);
INSERT INTO cur_index_temp(rate_id) VALUES(8);
INSERT INTO cur_index_temp(rate_id) VALUES(5);
INSERT INTO cur_index_temp(rate_id) VALUES(6);

FOR trec IN SELECT rate_id FROM cur_index_temp
LOOP
       SELECT INTO l_res.name c1.name || '/' || c2.name
       FROM currency c1 INNER JOIN rates ON (c1.id = rates.curr1_id) INNER JOIN currency c2 ON (c2.id = rates.curr2_id)
  WHERE rates.id = trec.rate_id;

    l_res.url = '/valuten/analysis.php?rid=30&s=&type=1&itemid=' || trec.rate_id::text;

  SELECT INTO l_res.last, l_res.change rates_history.last, rates_history.open
          FROM rates_history
            WHERE rates_history.rate_id = trec.rate_id
         ORDER BY rates_history.chk_time DESC
            LIMIT 1;

        l_res.change = round( CAST(l_res.last - l_res.change AS numeric), 2 );
     RETURN NEXT l_res;
END LOOP;

It wasnt that the original developer had used loops where a smartly-designed query could have sufficed that bother Hambai. It wasnt that this block of code was buried deep in a 5,000 line stored procedure that mostly contained dead branches that would never execute. It wasnt that with millions of records on the rates_history table, and no indexes, the ORDER BY clause was punishingly slow. It was that this particular algorithm for data-access was used for all of their time-based pricing data, including stock prices. With all the spaghetti code and general mess, it was easier to rewrite each stored procedure from scratch than it was to try and fix anything.

[Advertisement] Otter enables DevOps best practices by providing a visual, dynamic, and intuitive UI that shows, at-a-glance, the configuration state of all your servers. Find out more and download today!

http://thedailywtf.com/articles/the-latest-price

This is the story of Stacy and John.

Stacy was a secretary. It wasn't a glamorous job; nobody won an award for being the best secretary, or even got much recognition save for a paperweight or gift card once a year on "Administrative Assistant's Day" as a sort of grudging nod to the necessity of having someone paid to greet visitors, answer phones, and schedule appointments. But it was Stacy's job, and she was good at it. She was attentive. She was organized. She had neat handwriting, a pleasant voice, a good memory, and a professional manner. It wasn't demanding work, leaving her free to daydream of being a pop star—and it paid well enough to pay for singing lessons. Her instructor said she had talent, and maybe in a few years she could try out for American Idol.

John was IT support. It wasn't a glamorous job; your biggest perk was finding a fresh pot of coffee when you came in. But it was John's job, and he was good at it. John did his best to educate his users, and make sure that they understood how to use their computers, the network, and mission critical software. It might be a small network, without a lot of fancy administration tools and lots of manual steps, but it kept their business running.

As always in this new digital age, Stacy had a computer. She'd prefer a tablet, but her company was a little old-fashioned, and anyway she guessed that typing would be easier on a keyboard, but her cousin Mike got a tablet last year for his birthday and it looked so much easier just touching the icons she wanted than trying to figure out where everything was hidden in the Start Menu, right? Stacy and her computer had a bit of a love-hate relationship. She loved email, and she hated the impersonal touch Word gave her memos, preferring to hand-write them when she could.

Stacy spent a lot of time on Facebook. That was where she'd heard about Mike's new tablet, and about her nephew's first words, and about a really awesome singing audition at the mall that turned out to be a scam but wouldn't it have been cool if it was real? So she got a lot of emails from Facebook, and she dutifully opened every one to see what her friends were trying to tell her.

Today's email was no different. It said right in the subject line: "Urgent message from FACBOOK!"

That's weird, she thought for half a second, but I guess Facebook must be trying something new?

So she opened the email, ready to see what was so urgent.

"You have received a private message with an attachment!." the body said.

Odd—she wasn't expecting an attachment. Perhaps it was a video? Her friend Tracy said she'd recorded her performance at karaoke night. No, that wasn't Tracy's name; it was some stranger. Curious, Stacy scrolled up past the Facebook logo and blue header bar and right-clicked on the attachment. She selected Save As from the menu, just like John from IT showed her.

She chose to put it on her desktop for now, and watched as it downloaded ... only to vanish.

Had she done it wrong? No, that was exactly what she did for the memo her boss sent her last week to print and post around the building. Well, give it one more shot: right-click, Save As.

Nothing.

What followed was a protracted battle between the secretary and her definitely-not-as-cool-as-Mike's-tablet desktop PC. She tried saving it in another folder: no change. She tried renaming it in the Save As dialog: no dice. About to give up and call John, she finally noticed a piece of information she hadn't previously uncovered: whenever she tried to save it, a little bubble popped up by the clock, telling her the attachment was blocked.

Aha! thought Stacy. Whatever this Norton thing is, that's what's blocking my attachment!

She right-clicked on the icon, and there it was: a nice, handy Disable button.

Stupid Norton. This is from Facebook, let me open it!

Triumphant, Stacy downloaded the attachment.zip file to her desktop. Once she saw the icon, she knew what to do; John had taught her how to open those as well. She right-clicked on the icon (Computers are easy, just right-click on everything!) and clicked Extract Here.

But victory was snatched from her pearly white jaws: "This archive is not valid."

That stupid Norton thing broke my attachment! Stacy seethed, teeth clenched. I'll just have to write back.

From there, she clicked Reply. Her message was drafted hastily, but not impolitely:

Dear Mr Sanderson,

I have received your message with the attachment, but the antivirus program broke the attachment. Could you please send it again to my personal email?

Stacy.Johnson@gmail.com

Thank you,
Stacy Johnson

Several days passed. The incident slipped from Stacy's mind, filed away under "strange occurrence" and ignored in favor of more pressing needs, like why the catered lunches for the upstairs conference were instead re-routed to the basement where IT worked. Until, on the following Wednesday, after her computer had rebooted to install updates, it didn't show her her desktop. Instead, it gave her a terrifying message:

Pay $100 or you'll never see your files again.

A call was placed hastily, and John summoned up to her desk.

"Okay, no problem. Do you have a backup?" he asked Stacy, smiling, but she could see the worry in the creases at the corners of his eyes.

"Backup?" she asked, sheepishly.

John sighed. "Remember, I showed you, that's what this is for?" He tapped the black plastic box on her desk, and she remembered instantly: she was meant to run ... something or other, every day before she left, to copy her files to that external device.

"Sorry, I guess it slipped my mind. Well, not exactly, it's just, it took so long, and I had things to do ..."

"Okay, no problem. I think I set up a weekly backup when I was here last ... huh."

"Huh" was never a good thing with John. Stacy hunched her shoulders, embarrassed to have dropped the ball on what was apparently a super-important task after all. Well, everything her boss said was important! She didn't think John's stuff rated quite as highly as the company newsletter or answering the phone or making photocopies or ... well, just about everything, really.

"Yup. You don't have a backup in months," John reported. "Okay, okay, no problem."

Over the course of the next three days, Stacy learned that it was, in fact, a problem. Such a problem that 3 technicians moved her machine to their basement, leaving her with a "loaner laptop" which of course didn't have any of her files and could barely open a dozen Word documents at once. Practically Stone Age technology!

Thankfully, the brunt of her boss' wrath came down on the technicians for not working faster rather than on her for forgetting to run a backup.

Stacy promised herself she'd never forget again—she'd do it first thing in the morning, and before lunch, and before she left, every day until she made it big in her singing career and left this job. John even promised her she could have space on their network server to back things up to so she didn't have to remember to plug in the little drive he'd given her. This would never happen again!

And that's the story of how Stacy brought down the website by filling the content server with 6 terabytes of backups.

[Advertisement] Application Release Automation – build complex release pipelines all managed from one central dashboard, accessibility for the whole team. Download and learn more today!

http://thedailywtf.com/articles/indomitable-stupidity

Perhaps the greatest evil Microsoft ever perpetrated on the world was putting a full-featured IDE on every end users desktop: Microsoft Office. Its macro system is a stripped down version of Visual Basic, complete with a UI-building tool, and when used in conjunction with Access, allows anyone to build a database-driven application. Anyone thats spent enough time in an enterprise has probably inherited at least one Access application that was developed but somebody out at a manufacturing plant that magically became mission critical. Still, we cant blame the end users for that.

Theres a special subset of developer though, that when trying to come up with an application thats easy deploy, chooses Access as their development environment. Its already on all the users machines, they say. We can just put the MDB on a shared drive, they say. And thats how Ben gets handed an Access database and told, figure out why this is so slow?

The specific Access database Ben inherited was part of a home-brew customer-relationship-management package, which meant most of the queries needed to filter through a database of emails sent to customers. The SQL was already pretty bad, because it ran three times- once for all the emails with a certain timestamp, once for all the emails a minute earlier, and once for all the emails a minute later. Each query also filtered by the from-field, which really killed the performance since 25% of the emails were sent from the same email address- meaning the three queries did the same filtering three times, and Access isnt exactly all about the optimization of SQL performance.

What really caught Bens eye, though, was that the query didnt use the built in dateadd function to calculate the one minute earlier/later rule. It used a custom-defined timeAdjust function.

' This function takes a time and returns a rounded time.  If an adjustment
' has been given (+1 or -1 minute) then the time is increased or
' decreased by one minute.  Seconds are zeroed during the process and
' are only included in the returned time is the blSeconds flag is
' true.
Function timeAdjust(varTime, intAdjust As Integer, blSeconds As Boolean) As String
    Dim strHours
    Dim strMins
    Dim strSecs

    ' Get parts of the time
    strHours = Format(varTime, "hh")
    strMins = Format(varTime, "nn")
    strSecs = Format(varTime, "ss")

    ' Adjust time as required
    If intAdjust = 1 Then
        If strMins = "59" Then
            strMins = "00"
            If strHours = "23" Then
                strHours = "00"
            Else
                strHours = strHours + 1
                If Len(strHours) = 1 Then
                    strHours = "0" & strHours
                End If
            End If
        Else
            strMins = strMins + 1
            If Len(strMins) = 1 Then
                strMins = "0" & strMins
            End If
        End If
    End If

    If intAdjust = -1 Then
        If strMins = "00" Then
            strMins = "59"
            If strHours = "00" Then
                strHours = "23"
            Else
                strHours = strHours - 1
                If Len(strHours) = 1 Then
                    strHours = "0" & strHours
                End If
            End If
        Else
            strMins = strMins - 1
            If Len(strMins) = 1 Then
                strMins = "0" & strMins
            End If
        End If
    End If

    ' Rebuild time
    If blSeconds Then
        timeAdjust = strHours & ":" & strMins & ":00"
    Else
        timeAdjust = strHours & ":" & strMins
    End If

    Exit Function
ErrHandler:
    GenericADOErrHandler "clsADOTeamTallies - timeAdjust"
End Function

After a lengthy campaign, Ben received permission to re-implement the application in C# with a real database on the backend. That project was completed and left behind many happy customers.

[Advertisement] Otter allows you to easily create and configure 1,000's of servers, all while maintaining ease-of-use, and granular visibility down to a single server. Find out more and download today!

http://thedailywtf.com/articles/data-date-access

Anonymous went to configure some settings and found his options were a little constrained.

Fortunately, real numbers are a contniuum, so Anonymous has an infinite number of possible values to choose from.

Kaylee was poking through a codebase and found this DLL, which leaves us with so many questions.

Were left feeling inside out, and perhaps a bit unmoored in time…

Were losing track of time… how old are we? Benji found that question far more difficult to answer than normal…

Then again, its hard not to lose track of time. Time and space are a continuum, and movements through space are also movements through time, and vice versa. Ricardo sends us this Italain train schedule:

If you dont read Italian, pay close attention to the km field- which is measured in dates, apparently.

As we all learned in Back to the Future III, trains are incredibly useful devices for time travel. As any time traveler knows, representing dates and times when youre crossing history is a challenge. Nick stepped aboard the wrong train, and who knows when he might be now?

This all gets so confusing. Is there some sort of error code we could report about all of this? Larry, do you have anything? Oh, you do?

[Advertisement] BuildMaster integrates with an ever-growing list of tools to automate and facilitate everything from continuous integration to database change scripts to production deployments. Interested? Learn more about BuildMaster!

http://thedailywtf.com/articles/errord-in-time-and-space

Hold your souls tightly, for today, we pierce the veil into the great beyond. We shall examine existential questions and commune with spirits. We shall learn what eternity holds for us all.

First, we must bring ourselves to the edge of death, into a liminal state where time does not pass, where the conscious mind takes a back-seat to the spiritual realm. Mark found this C# code to do the job:

    if (Thread.CurrentThread.ThreadState != ThreadState.Running)
        return false;

Are we alive? Are we dead? What is this darkness? Where are we? Surely, we must now find our path. Many languages, like Python, have useful libraries to help us parse out and understand the structure of our path. Chris sent us this code, which eschews the well traveled os.path library, and reinvents that wheel:

    samp = file[file.find('intFiles')+9:].split("/")[0]
    fName = file.split("/")[-1]

This code pulls off the portion of the path that occurs after intFiles, and splits off the last portion, which we hope is a filename. Now, with that filename, the collective sin on our souls must be weighed, and our eternal reward- or punishment- will be delivered.

Brandon knows what the afterlife holds for us all… welcome to hell:

If Error=0 Then
    Error = DeleteSummaryTables(Conn, sJobId, "Summary_Grips")
    If Error = 0 Then
        Error = DeleteSummaryTables(Conn, sJobId, "Summary_GroundType")
        If Error = 0 Then
            Error = DeleteSummaryTables(Conn, sJobId, "Summary_Enviroment")
            If Error = 0 Then
                Error = DeleteSummaryTables(Conn, sJobId, "Summary_FloorType")
                If Error = 0 Then
                    Error = DeleteSummaryTables(Conn, sJobId, "Summary_Control")
                    If Error = 0 Then
                        Error = DeleteSummaryTables(Conn, sJobId, "Summary_HandToolType")
                        If Error = 0 Then
                            Error = DeleteSummaryTables(Conn, sJobId, "Summary_FingerMovement")
                            If Error = 0 Then
                                Error = DeleteSummaryTables(Conn, sJobID, "Summary_LiftHeights")
                                If Error = 0 Then
                                    Conn.CommitTrans
                                Else
                                    DisplayError "Error deleting record from the Summary_LiftHeights tables.", err.number, err.Description
                                    Conn.RollbackTrans
                                End If
                            Else
                                DisplayError "Error deleting record from the Summary_FingerMovement tables.", err.number, err.Description
                                Conn.RollbackTrans
                            End If
                        Else
                            DisplayError "Error deleting record from the Summary_HandToolType tables.", err.number, err.Description
                            Conn.RollbackTrans
                        End If
                    Else
                        DisplayError "Error deleting record from the Summary_Control tables.", err.number, err.Description
                        Conn.RollbackTrans
                    End If
                Else
                    DisplayError "Error deleting record from the Summary_FloorType tables.", err.number, err.Description
                    Conn.RollbackTrans
                End If
            Else
                DisplayError "Error deleting record from the Summary_Enviroment tables.", err.number, err.Description
                Conn.RollbackTrans
            End If
        Else
            DisplayError "Error deleting record from the Summary_GroundType tables.", err.number, err.Description
            Conn.RollbackTrans
        End If
    Else
        DisplayError "Error deleting record from the Summary_Grips tables.", err.number, err.Description
        Conn.RollbackTrans
    End If
Else
    DisplayError "Error deleting record from the Summary_Grips tables.", err.number, err.Description
    Conn.RollbackTrans
End If
    Else
    DisplayError "Error deleting record from the JobHeader_WorkAReas tables.", err.number, err.Description
    Conn.RollbackTrans
    End If
    Else
        DisplayError "Error deleting record from the Tasks tables.", err.number, err.Description
        Conn.RollbackTrans
    End If
    Else
        DisplayError "Error deleting record from the subtask tables.", err.number, err.Description
        Conn.RollbackTrans
    End If
End If

[Advertisement] Manage IT infrastructure as code across all environments with Puppet. Puppet Enterprise now offers more control and insight, with role-based access control, activity logging and all-new Puppet Apps. Start your free trial today!

http://thedailywtf.com/articles/a-spiritual-journey

Ivan worked for a mobile games company that mass-produced freemium games. These are the kinds of games you download and play for free, but you can pay for in-game items or perks to make the game easieror in some cases, possible to beat once you get about halfway through the game.

Since that entire genre is dependent upon microtransactions, youd think developers would have rock-solid payment code that rarely failed. Code that worked almost all the time, that was nearly unhackable, and would provide a steady stream of microtransactions to pay everybodys salary. But who am I kidding? This is The Daily WTF! Of course reliable in-app payment code for a company completely dependent on microtransactions isnt going to happen!

Ivan found himself debugging the companys core payment library, originally written by a developer named Hanlon who happened to have a PhD in Astrophysics. Hanlon still worked for the company but had moved on to other roles. However, he reported a bug that Ivan had to fix. The payment library would never report PaymentValidationResult.Success, only PaymentValidationResult.Error. In-app purchases were completely broken- across their entire library of released freemium games- and this was obviously unacceptable.

Ivan went spelunking but quickly found that the web service payment did indeed return PaymentValidationResult.Success when a microtransaction was successful. So what was the client-side library doing wrong? He opened that project up and found this:

    PaymentResponse response = _networking.SendRequest(request);
    PaymentValidationResult result = PaymentValidationResult.Success;
    if (response.StatusCode.Is(400))
    {
        result = PaymentValidationResult.Failure;
    }
    else if (response.HasError())
    {
        result = PaymentValidationResult.Error;
    }

    return result;

It assumes from the start that the payment was successful, and only returns an error if the server response explicitely says something bad happened. In the case of timeouts, corruption, or other network errors when a server response is not received, this library would actually report to the application that the microtransaction succeeded, and the player could get free stuff!

But this doesnt explain why it always failed.

He decided to peek at the response.HasError() function.

    public bool HasError()
    {
        bool hasError = true;

        if (BodyString != null)
        {
            var body = ParseBody();
            if (body != null)
            {
                hasError = body["error_message"] != null;
            }
        }

        return hasError;
    }

This function kind of does the opposite. It assumes an error has always occurred, and then validates and changes its mind if it finds proof that no error occurred. For this to happen, the response it receives from the server must contain a JSON document that does not have an error_message key.

Upon re-examining the server-side code, Ivan found that the server simply returns an HTTP status code to indicate payment status, with values of 400 or more indicating error (e.g. 401 = validation info missing from request, 402 = no information retrieved from app store, 200 = all good!). It doesnt issue any kind of JSON document. Ever. The client code always failed because BodyString was always null.

Thats right, the core business-critical microtransaction library used in all of the companys freemium games was incapable of actually handling a microtransaction!

Ivan rewrote the PaymentResponse.HasError() function to look like this:

    public bool HasError()
    {
        int responseCode = (int)StatusCode;
        return responseCode >= 400;
    }

They pushed this fix out to the entire company and all its products, issued rounds of updates to the various mobile app stores, and the company started making money, presumably for the first time ever.

[Advertisement] Application Release Automation – build complex release pipelines all managed from one central dashboard, accessibility for the whole team. Download and learn more today!

http://thedailywtf.com/articles/unprocessed-payments

One of the hallmarks of bad code is when someone reinvents the wheel. In many Code SODs, we show code that could be replaced with a one-line call to a built in, standard library.

Thats one of the the advantages to a high-level language operating on modern hardware. Andrew doesnt live in high-level land. He does embedded systems programming, often on platforms that dont have conveniences like standard libraries, and so they end up reinventing the wheel from time to time.

For example, a third party wrote them some software. Among other things, they needed to implement their own versions of things like trigonometric functions, random functions, square root, and so on- including atoi- a function for converting arrays of characters to integers. Andrew was called upon to port this software to slightly different hardware.

In the header file, a comment promised that their atoi implementation was compatible with ANSI atoi. The implementation, however…

long atoi(char *s)
{
        char *p;
        int l;
        long m=1, t=0;  /* init some vars */
        bool negative = FALSE;

        /* check to see if its negative */
        if(*s == '-') {
                negative = TRUE;
                s++;
        }
        /* not negative is it a number */
        else if(s[0] < '0' || s[0] > '9')
        {
            return 0;
        }

        l = strlen(s);

        p = s + l;                           /* start from end of string */
        /* for each character in the string */
        do {
                p--;                         /* work backwards */
                t += m * (*p - '0');         /* add new value to total, multiplaying by current multiplier (1, 10, 100 etc.) */
                m = (m << 3) + (m << 1);     /* multiply multiplier by 10 (fast m * 10) */
                l--;                         /* decrement the count */
        } while(l);
        if(negative)
                t = -t;                      /* negate if we had a - at the beginning */
        return t;                            /* return the total */
}

The comments came from the original source. Now, this ANSI compatible function has a rather nice boundary check to make sure the first character in either a - or a digit. Of course, thats only the first character. Andrew helpfully provided some example calls that might have unexpected behaviors:

    atoi("2z") == 94
    atoi("-z") == -74 // at least it got the sign right
    atoi("") == [memory violation] // usually triggering a hardware interrupt

For bonus points, their versions of sin and cos can have drift of up to two degrees, and their tan implementation is just a macro of sin/cos, so it has ever larger errors. Their rand function is biased towards zero, and their sqrt implementation uses convergence- it successively approximates the correct result. That much is fine, and pretty normal, but a good sqrt function uses a target epsilon to know when to stop- keep approximating until the error margin shrinks below the point where you care. This particular sqrt function just used a fixed number of iterations, so while sqrt(10) might give you 3.1623, sqrt(10000) gave you 1215.5.

[Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

http://thedailywtf.com/articles/unstandard-lib

Contracting seemed the best way for Ann-Marie to gain a foothold in the IT industry. She wasn't the best or the brightest, and her CV was heavy on toy languages and light on experience. But if she got a good solid chance, she knew her no-nonsense attitude and general intelligence would endear her to her boss.

Or maybe it would have, if it weren't for Cindy the Project Manager.

Cindy liked to chew gum at work. No, not chew—pop. It was all Ann-Marie could do to keep her face neutral during her onboarding, when Cindy handed over the employee handbook the team lead had asked her to deliver.

"So, like, these are our standards I guess? Whatever, you'll do fine, I'm not even worried."

Once she was gone, however, things started looking up. Unlike the last few places Ann-Marie had contracted, this one had JIRA and used it heavily, requiring all commits to include a JIRA number for tracking purposes. In addition, she was pleasantly surprised to see a QA department, and reasonable handoff practices: to be handed over to QA, the build had to be passing on the CI server, and earmarked as a release candidate. Commit messages were vital for keeping track of items in progress. They had to be descriptive, yet concise, and no more than one commit per JIRA item so that a relationship between tickets and commits could be established and tracked.

Yeah ... I could stay here, Ann-Marie decided. What's a little gum-popping in exchange for a Joel Number this high?

And so, eager to please, she threw herself into her work. On the first day, she closed five JIRA tickets—all small fixes in disparate portions of the application—as she tried to get a feel for the codebase.

"Wow, you like, really know your stuff," Cindy said, stopping by at the end of the day. "Our last guy didn't do anything for four days. This is going to be the best Bug Blitz ever. Well done!"

Ann-Marie stopped for a smoothie on her way home, confident she'd fit right in at this new job.

It was day three before the first incident happened. She was off to another great start, closing two tickets before 10:00 AM. At first, she assumed that was why Cindy was walking over, until she saw the frown on her manager's face.

"So, like, I don't know if you read the handbook, but we have a policy of not closing more than one ticket with the same JIRA number," Cindy said.

"Yeah, right, so you don't have to pick apart commits if you send one feature but not the other. Totally cool, definitely a best practice." Ann-Marie wondered why this was coming up. Had she accidentally broken policy? No, she'd been careful. One change, one commit.

"Right, right. So what's up with 44 and 89?" Cindy asked, rattling off the numbers with a punctuative pop.

"Uh ..." Ann-Marie pulled up the JIRA tab on her machine, rapidly navigating to 89 and skimming over the comments. "Oh, right, 89! That was fixed by the commit I did yesterday for 45. It had the same root cause."

"Riiight ... so they used the same commit," Cindy said.

"No, see?" Ann-Marie said. "Here, 45 was an error when downloading a document, and 89 was a blank document ID in the details for the document. The user couldn't download the document because it was sending them to a URL with a blank ID in it. The reason for that was because the file ID was omitted from the select columns in the database query that was shared between the two. By changing it so the document's file ID was included in that query, I killed two birds with one stone."

"Right, yeah, see, two birds, one commit. We have a strict policy against that. This can't happen again."

Ann-Marie blinked at Cindy. Was she just not getting through? "Surely something like this has happened before?"

Cindy gave her a flat, vacant look. "Not really." Pop. "We cool?" Pop.

"I ... I really don't know what to say here," Ann-Marie replied, flustered. "I mean, I agree with the policy, it makes it easier to roll back if one fix is broken, but this? This is a different situation entirely."

"You'll figure something out." Without waiting for a reply, Cindy turned and walked away.

Wonderful, Ann-Marie groaned inwardly. What do I do now?

Hoping against hope that she was just misunderstanding something, that someone else could enlighten her as to what Cindy's problems were and how to give her what she wanted, Ann-Marie took a walk to Garrett's cube. Garrett was a senior developer, one who'd been useful over the past few days as she'd rooted out a few harder-to-find bugs. Surely he'd run into this issue before. Maybe he'd even have an answer?

"Ooooh, yeah. THAT policy," Garrett began, and Ann-Marie's heart sank. "Cindy gets a little OCD about this stuff. We know exactly what you're talking about. Different developers go about resolving it in different ways. I, personally, simply commit one thing in such a way that 'breaks' the other issue again. Like, in your case, I would have both included the ID column and then intentionally blanked out the ID in the details view for my first commit. That commit gets mapped to the download error JIRA ticket. Then, I commit a second change that undoes the blanked-out ID in the view, that goes to the other ticket. It leaves a sour taste in your mouth, but you get used to it."

Ann-Marie sighed, feeling a headache creeping up on the edges of her temples. "I should have just marked it a duplicate," she mumbled to herself.

"Oh, no, Cindy doesn't like that either." Garrett assured her with a sympathetic look.

"Committed a whitespace change for the second ticket?" Ann-Marie fired back, not even bothering to get her hopes up.

"She reviews every single commit. Not that she's ever been a programmer, but she's taking night classes and has a keen eye for BS. Thankfully not keen enough to catch on to what we're doing with the intentional-breakage strategy, though."

Ann-Marie sighed. "Well, that's me out of ideas."

"Sorry," Garrett replied, a gentle smile on his face. "Sometimes you just have to throw in the towel and find a loophole."

For the duration of the Bug Blitz, Ann-Marie did just that, committing 'accidental' breakages for each duplicate bug—and even, once, for a triplicate ticket.

When the blitz finished and she was offered a permanent position, she politely declined. Stability was overrated.

[Advertisement] Release! is a light card game about software and the people who make it. Play with 2-5 people, or up to 10 with two copies - only $9.95 shipped!

http://thedailywtf.com/articles/double-play

"I have no idea what that giant glowing ball in the sky is. Space aliens, I suppose?" wrote Ian S.

Andreas writes, "LAN (the airline) and I seem to have differing definitions of optional."

"Upgrade? More looks more like a mutation to me," wrote Samuel G.

"I received this email with my ticket for a concert in the Netherlands," Martin B. writes, "Apparently learning Dutch is going to be harder than I was anticipating."

Roger writes, "Um...is Pidgin trying to tell me something?"

"Looks like the Carbon Capture and Storage Association should really capture and store some links," wrote M. W.

James D. writes, "I kid you not, this is what management wants us to use instead of our own in-house bug tracker!"

[Advertisement] Onsite, remote, bare-metal or cloud – create, configure and orchestrate 1,000s of servers, all from the same dashboard while continually monitoring for drift and allowing for instantaneous remediation. Download Otter today!

http://thedailywtf.com/articles/sorry-about-the-moon