Joe T recently decided it was time to go job hunting. This mostly meant deflecting emails from head-hunters, doing phone interviews with ignorant HR departments, and the occasional on-site interview with a possible employer. One of those on-site interviews brought him to an IT services company which handled a few large US government contracts.

The building itself was buried way in the back of a tech park, behind a small pond inhabited by ducks and migrating geese. It was modern, in a circa 2002 way, gleaming white, with a large and mostly empty parking lot, and a glossy lobby and a pair of cheerful receptionists.

One of the receptionists made a call. A few moments later, Ricky, Joes contact, stumbled out of the elevator. He had the look of a man who hadnt slept in a month. As they piled into the elevator, he said, We are really eager to fill this position. Just between you and me, I think youve got it sewn up, but you need to meet the team.

Ricky led him into a conference room, where a few other sleep-deprived looking techs nursed their coffees. After a round of handshakes, they dove into a series of sysadmin/operations questions. Some of them were just dumb trivia: Whats the purpose of the -i flag on the ls command? Some were dumb soft-balls: What time management techniques do you use?

After the warmup, Ricky leaned forward, and got right to the meat of it. Youve got three data centers, he said, how would you monitor all three of them?

That was a vague question, so Joe could only give a vague reply. Well, Id imagine all three are connected to the same management network. So Id run some sort of monitoring package on that network.

You dont have that, Ricky said. Joe couldnt be sure, but he thought he might have seen a grimace on the face of one of the techs.

Well, Joe said, then wed just have to set up monitoring in each DC independently, and then aggregate somehow, but thats really not the best option. Setting up a management network is-

Ah, Ricky interrupted, but what if you dont have the budget for any monitoring software?

Joe, sitting in the middle of the interview, jumped to a conclusion about Rickys line of questioning. This was one of those old interview tactics: no matter how the candidate replies, throw up an obstacle, and see how they can think outside the box. This was just a bizarre hypothetical situation. He played along. I guess wed need to install Nagios or some other open source monitoring tool.

The customer doesnt allow any open source software on the network, Ricky countered.

None? Joe asked. But dont you have some Linux servers?

Ricky nodded. We do. But no open source software on the network.

Well, I mean, if you cant buy software, and you cant use free software, you cant really do any meaningful monitoring.

Now Joe was certain: the techs were all grimacing, and oh so very slightly nodding along with him. This was what they expected him to say, but Ricky wasnt going to give in yet. Now, you cant just throw your hands up and say, It cant be done. You have to monitor these datacenters, so what do you do?

Well… I mean, at that point, were just writing our own scripts and running cron jobs. But thats a totally ridiculous scenario. No one would actually run a datacenter that way! Joe punctuated the statement with a laugh, because obviously this was all just silly.

No one else laughed. Rickys line of questioning wasnt a hypothetical, it was an accurate description of their infrastructure in their datacenters. After a solid minute of awkward silence, Ricky started asking questions about the previous jobs on Joes resume.

Somehow, Joe left the interview with a job offer, but he politely declined.

http://thedailywtf.com/articles/monitoring-the-situation