Andrei had just moved to a new country, and was looking for work. With the ideal job not immediately forthcoming, he jumped when an interesting opportunity opened at a small tech company: OldIsNewCo.

OldIsNewCo was one of the pioneers bridging the gap between old methods of communication (e.g. paper) and new methods of communication (i.e. the Internet), and was looking to overhaul their entire infrastructure. Their existing backend was built in C++, and according to The Big Boss: “C++ developers are expensive and hard to find. PHP developers are plentiful and cheap. Therefore, we need to re-implement everything in PHP.”

This seemed fishy to Andrei right from the start, but he figured it would be a good opportunity to brush up on his C++ and PHP skills and experience, and “a job’s a job”. So he accepted the position and met with his development tour guide to start diving into the sources and seeing how the backend applications worked.

Or, he tried - only to quickly discover Problem #1: There wasn’t a complete version of the source code in any single location. The sources for the backend were spread out over different production machines (i.e. the machines running the compiled code). Because these machines had variations, building was done on each machine locally, since the code might not build on a different machine with different software or configuration. This contributed to Problem #2: Making modifications to some of the backend was done by editing code on the same machine where that component ran.

As Andrei wrapped his head around those concerns, he began to wonder how the developers tested their changes. His development tour guide responded: “Oh, we don’t have a testing environment. We just deploy the new version to the live servers.” Deploying to production consisted of: waiting until the night when the system was mostly free of clients, compiling the code on the same machine it would then run, and hoping for the best. If the new build crashed the next morning when users returned, they would just revert to the old build and repeat the attempt the next evening.

But project organization, testing and deployment were far from the only concerns. As he finally began to dig into the code and architecture itself, Andrei discovered that the existing C++ backend had a few problems of its own:

• The core backend app simply operated on files - it read them in, processed the contents, and output data to a database and other machines via the network. But this was implemented in the most complicated way possible, with numerous multithreading issues that resulted in the deployed application crashing frequently
• It crashed so frequently, in fact, that it was one employee’s main job to watch the servers, wait for the app to crash, and manually restart it.

http://thedailywtf.com/articles/the-backend