YouTube, like any reasonable video service, offers closed captioning. They'll even throw machine learning at the problem, and autogenerate captions, though that is usually only good for comedy, rather than actual accesibility.

Any closed captioning system will generally let you specify the colors of the captions as well as the actual text. YouTube is no exception to that. YouTube offers an online editor, but anyone serious about producing content is going to upload their own subtitle files, and up until recently, this could be done in an XML file which would allowed a lot of control over the styling of the captions.

But XML isn't cool, so YouTube rolled out a new JSON format. Which broke everything. Specifically, instead of being able to set any hex triplet as your color, you could only set a relative handful, and any "invalid" triple would just turn to white.

For some reason, whatever the user inputs isn't propogated to this as a hex triplet, like #FF0000, but instead as a hex value- 0xFF0000, or 16711680, in decimal. I'm not sure if that's on the "filling out the JSON" side, or if it's a conversion which happens elsewhere, but regardless: the front end needs to map a value like 16711680 to #FF0000. Well, a value like that, but not exactly that value, as we'll see.

An anonymous submitter already trawled through the minified code and found the code block responsible, which they've helpfully de-minified for us:

var colorMap = new Map([
    [0xFFF,    "#fff"],
    [0xFF0,    "#ff0"],
    [0x0F0,    "#0f0"],
    [0x0FF,    "#0ff"],
    [0x00F,    "#00f"],
    [0xF0F,    "#f0f"],
    [0xF00,    "#f00"],
    [0x080808, "#080808"]
]);

if (jsonPen.fcForeColor)
    pen.color = colorMap.get(jsonPen.fcForeColor) || "#fff";

Original, unminifed version. In that version, I want to note that the hex values were expanded into decimal: new Map([[4095,"#fff"],[4080,"#ff0"],[240,"#0f0"],[255,"#0ff"],[15,"#00f"],[3855,"#f0f"],[3840,"#f00"],[526344,"#080808"]].

Setting aside the "I know, let's break a feature that worked fine," this… is wrong. You can set red, for example, but not by doing 0xFF0000- you must instead do 0xF00. If you send 0x000F00, expecting a dark green color, you'll get red.

And if the rule were "hey, we only support a single hex digit per color channel," you could document that and move on. But it doesn't: we can see that 0x080808 is valid. That's nearly black, which it's worth noting that actually black isn't in the whitelist of colors, and worse: it wouldn't even check to see if black (0) is in there, because if (jsonPen.fcForeColor) is falsy. We won't even change the color if you try and set the color to black! Don't worry, elsewhere in the code it still defaults back to being white.

This code is, at the time of this writing, disabled. It went live, briefly, last week, breaking a bunch of closed captioned videos in ugly ways. I have no idea what the final version of this will look like- maybe exactly like this, but with clearer documentation?- but it is an ugly way to solve a problem which is already well solved.

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

https://thedailywtf.com/articles/a-splash-of-color

"Hey Alexa, can you help Amazon with their math?" Timothy W. wrote.

"Name my %1$s so I can identify it later? How about 'Placeholder Text Goes Here'?" writes Vladim'ir C.

Adam L. wrote, "Now I haven't seen the code behind the scenes here, but I imagine someone might have gotten a greater / less than operator backwards."

Bob writes, "When quiz randomization generates recursive options, the choice is absolutely clear."

"There's normal tests, and then there are the sponsored tests on my Samsung 'Smart' TV," Malhar S. writes.

Jerry wrote, "Somehow it feels like this form does not trust people easily."

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

https://thedailywtf.com/articles/amazon-deal-or-no-deal

Are we sick of of year rollover bugs yet? Well, let’s just agree that people aren’t sick of making these kinds of bugs.

A long time ago, someone at Oleksandr’s company needed to write a Python script that shipped a 4-digit year to a system that only accepted 2-digit years. So 2010 needed to turn into 10.

They could have used date formatting to format the date into a two digit year, but that involves understanding how to format dates. That just seems like too much overhead, when there’s already a perfectly good way to mangle a string.

year = str(timezone.now().year)
year = year.replace('20', '')

This particular method worked just fine for the vast majority of the 21st century, to date, but mysteriously stopped working on January 1st, 2020. Or, as this script might write the date, “January 1st, .”

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

https://thedailywtf.com/articles/a-short-year

What, exactly set of features divide a "text editor" from an IDE is a bit of a blurry line. Developers are not the sort to use static, unchangeable tools. They want configuration, they want plugins, they want quick access to a terminal, they want debugging support, and they'll bolt those features into just about anything.

There's a fine line between an IDE that provides nice utility to the developer, and an IDE which is opinionated. I had the misfortune long ago to use the WebSphere IDE, which was essentially a repackaged version of early Eclipse bundled with highly opinionated plugins about how you were supposed to build a Java application. As Matt puts it: "An IDE is, to some, a high-functionality tool for developing applications, and to others a classic example of an Inner Platform."

Matt is saddled with a vendor-specific IDE which falls solidly on the Inner Platform side of things. Like many developers, Matt has a laptop which he can lug to meetings, and a docking station with a large external monitor he can use for actual work.

There's just one problem. Any time this vendor-specific IDE wants to pop up a dialog box, the dialog box always appears on the primary monitor. Now, Matt leaves his laptop screen set as the primary monitor, but he's doing his actual work on the large monitor, so this is a nuisance, to be certain. Ideally, dialogs should appear on whatever monitor the active window which triggered the event is on.

This is a minor bug. Almost trivial. But it is a bug, and enough of a nuisance to Matt that he entered a bug ticket. He fired it off, and largely forgot about it over the next two weeks, until he got a reply.

Closed: Cannot Reproduce

That raised Matt's eyebrow. He took a quick survey around the office, and sure enough, he could reproduce this bug 100% of the time on any docked laptop. He re-opened the bug and started fighting with the escalation pathway.

After a few more rounds of:

Closed: Cannot Reproduce

complete with Matt sending screenshots, screen recordings, and every log and configuration file stored by the IDE on his machine, the vendor finally scheduled a screen sharing session with a QA tester.

The QA tester shared their screen, and asked Matt to walk through the steps to reproduce.

"Okay, so, you'll need to move the IDE window over to your secondary screen," Matt said.

"I don't understand what you mean," the tester replied. "What is a 'secondary screen'?"

"It's a second monitor? Are you on a laptop or desktop? It'd be a second screen plugged into the device."

"I am on a laptop."

"With, like, a docking station?"

"I have no idea what a 'docking station' is. My laptop has a monitor. There is no second monitor," the tester said. Their voice had started to drift away from "neutral tech support voice" and into "I think I'm talking to a madman" territory.

"Right, but you can plug a second monitor into the laptop," Matt said.

"I have never heard of such a thing."

"So you never test the IDE in a multimonitor setup?"

"What is a 'multimonitor setup'?"

"It's when you plug in a second- or sometimes even a THIRD monitor into your computer?"

"We know nothing about these setups. Everyone here has a laptop which also already has a monitor. I am sorry, but your issue cannot be reproduced, so I am going to close it as 'Cannot Reproduce: won't fix.' Is there anything else I can help you with today?"

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

https://thedailywtf.com/articles/two-heads-are-better-than-one

There’s a simple rule about “enterprise” software: if the word “enterprise” is used in any way to describe the product, it’s a terrible product. Enterprise products are usually pretty special purpose and serve a well-capitalized but usually relatively small market. You aren’t going to sell licenses to millions of companies, but maybe tens of thousands. Often hundreds. There are some extremely niche enterprise products that have only two or three customers.

Lots of money, combined with an actually small market in terms of customers, guarantees no real opportunity for competition. Couple that with the fact that each of your customers wants the off-the-shelf product you’re selling them to have every feature they need for their business case, you’re on a fast track to bloated software, inner platforms, and just general awfulness.

Today, we’re not talking about Oracle, though. Jacek is in the healthcare industry. Years ago, his company decided to shove off their patient scheduling and billing to a Software-as-a-Service enterprise product. Integrating that into all of their other systems has kept the development teams busy ever since, and each successive revision of the API has created a boatload of new work.

Currently, the API is on version six. V6 is notable because this is the first version which offers a REST/JSON API. V5 was a SOAP-based API.

Here’s an example message you might send to their scheduling services:

{
    "Id": 12,               
    "StartDateTime": "2018-06-16T11:00:00",
    "EndDateTime": "2018-06-16T11:30:00",             
    "Description": "Lunch"
}

That schedules a blackout window during which a service provider might be unavailable. Nothing about that seems too bad, does it?

What if, on the other hand, we wanted to schedule a wellness class, on say proper nutrition?

{
  "siteId": 7,
  "locationId": 2,
  "classScheduleId": 9,
  "classDescriptionId": 15,
  "resources": [
    {
      "id": 2,
      "name": "Conference Room 3"
    }
  ],
  "maxCapacity": 24,
  "webCapacity": 20,
  "staffId": 5,
  "staffName": "Sallybob Bobsen",
  "isActive": true,
  "startDate": "2018-07-22",
  "endDate": "2020-07-22",
  "startTime": "10:30:00",
  "endTime": "11:30:00",
  "daysOfWeek": [
    "Tuesday",
    "Thursday"
  ]
}

Well, that message on its own doesn’t look terrible either. But when you combine the two, you’ll notice that whether you use PascalCase or camelCase varies based on which endpoint you’re invoking.

If you want to create a client, you’ll post a document like:

{"FirstName": "Suebob", "LastName": "Bobsen"…}

I’ve skipped over a giant pile of fields- address, relationship to other clients, emergency contacts, etc. They’re not relevant, because I simply want to compare creating a client to updating a client:

{
  "Client": {
    "Id": "123456789",
    "LastName": "Bobsdottir",
    "CrossRegionalUpdate": false
  }
}

So we send different shaped documents to perform different operations on the same resource.

And it’s important to note, none of these resources have their own endpoints. In a RESTful webservice, I’d want to send my updates as a post to https://theservicehost/clients/123456789, while a create would just be a post to https://theservicehost/clients.

Here, we have different routes for addclient and updateclient. So this isn’t a RESTful API at all- it’s a shoddily assembled RPC API, probably just a JSON wrapper around their SOAP API. It has no concept of resources. Fine, we can accept that and move on, and hey, the documentation at least lays out the correct shape of the documents, and what the fields mean, right?

Well, look back at that update message. See the CrossRegionalUpdate field? Well, per the documentation:

When CrossRegionalUpdate is omitted or set to true, the client’s updated information is propagated to all of the region’s sites.

So, it sounds like if you have a region that contains multiple sites, you want to set that to true (or omit it), right? Wrong. Most customers should set the flag explicitly to false. When Jacek talked to a technical support rep from the company, they said: “You would know if this should be set to true. Just set it to false.”

With all that, you can imagine how their error handling works, which is to say, it doesn’t. There are two things which happen when an error occurs: it either returns a 400 status code with no body, or it returns a 302 status code which redirects to another endpoint which serves up a plain-text version of your request. Your request, not the response to the request.

This nearly created a serious problem. The requests contain personally identifiable information about patients. Their initial naive logging approach was just to dump error responses to a log file, assuming that the API would never return an error with PII in it. At first, it looked like they should just treat a 302 as an error, and log the response, but they discovered in testing that would basically be dumping PII into an insecure store reviewed by IT support personnel.

Jacek is “eagerly” looking forward to v7 of this API, which is certain to contain more “improvements”. The real question is: if this is their sixth attempt to get it right, what did v1 look like, and did anyone survive attempting to use it?

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

https://thedailywtf.com/articles/an-enterprise-api

Daniel didn't believe the rumor at first. Whenever his company chased after the hottest new business trends, they usually pursued the worst trends imaginable. But word was that this time, they'd seen fit to mimic Google’s fabled "20% Time."

The invitation for a company-wide meeting soon landed in everyone's inbox, turning rumor into reality. For a moment, Daniel dared to dream about the various time-saving scripts he'd always wanted to write, and the applications he'd always wanted to re-implement from the ground up—all those things he'd been prevented from even thinking about as he'd been flung from one urgent project to the next. With 20% of his work time dedicated toward personal pursuits, all of that was set to change ...

He forced himself not to get excited. As a hardened corporate veteran, Daniel had been down this road before. It wasn't a question of would this go wrong, but how would it go wrong.

At the company meeting, which Daniel attended via conference call, a parade of high-level executive voices gushed about the new corporate policy, dubbed "Take-12." "From now on, everyone gets one hour each month to put toward personal work and development!"

Twelve hours a year, out of the 2,000 that most people were expected to work. Daniel supposed "0.6% Time" didn't have quite the same ring to it.

And how would the new policy be reconciled with the company's already Byzantine timesheet and internal billing system? Well, the company didn't even even attempt such a thing; there was simply no room in anyone's budget. Instead, someone had the bright idea of co-opting a different internal system for curating video playlists for online training courses. The idea was that each employee would create a new playlist of 12 hour-long videos, one for each month. The videos wouldn't actually be videos, just placeholders. By "watching" a "video," an employee would be logging that they'd spent one hour of time on personal development.

The premise started off sketchy, and only got sketchier as difficulties were encountered during implementation. Daniel could only shake his head in astonishment as he read the User Guide detailing what he had to do:

Set up your playlist each year:
1. Click “Technical Academy” on the home screen
2. The “Take 12: Development Time” modal will open
3. Click “Add to Playlist” in the title bar
4. The “Add to Playlist” menu will appear
5. Click to expand the menu options
6. Click on “Create a Playlist” option
7. Text entry boxes will appear for your new playlist
8. Type in a playlist title. E.g. “Take 12 Development Time 2019”
9. Type in a description
10. Click the “Add Playlist” button located under the text fields
11. Select your new playlist from the playlist dropdown
12. Click the “Add to Playlist” button
13. The word “Added” will appear
14. Click the home button in the screen
15. Your newly created “Take 12: Development Time” playlist will show in the “My Playlists” section of the home page (You may need to refresh the screen for it to show first time)

Using your Take 12 Development Time each month:
1. Click on your Take 12 playlist to open it
2. Expand the “Outstanding Content” section of your playlist
3. Click on the month you wish to register time for
4. A page will open which has some text on it, there’s nothing you need to do here.
5. Exit this window by using the ‘home’ button
6. When you’ve finished your hour of learning, click on your Take 12 playlist again to open it
7. Expand the “Outstanding Content” section of your playlist
8. The current month has a “Completed?” button next to it.
9. Click the “Completed?” button
10. The selected month will move from the “Outstanding Content” section to the “Completed Content” section
11. Close your playlist until next month

People ended up spending more time troubleshooting their buggy playlists than on personal development. With little by way of profit or benefit to justify its continued existence, "Take-12" was eventually done away with altogether. Daniel congratulated himself for having lowered his expectations.

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

https://thedailywtf.com/articles/painful-self-development

"The fact that Microsoft values my PC's health over copyediting is why I thumbed up this window," Eric wrote.

"Great! Thanks Steam! How am I going to contact Paypal now?"

"Now serving order number Not a Number. A totally normal order number, don't question it," Pierre-Luc wrote.

"Sure, I had a pretty rough start getting my cheapo smart power outlet up and running, but hey, on the plus side it does look like 2 indeed got changed to 'two'" writes Bob.

Andy writes, "I'm not sure how much processing Dreamhost needs to do when making a password with the complexity of YXmztnS5vxA6, but this screenshot was taken after 45 seconds of loading."

"I depend heavily on Microsoft Null, but I can't really imagine what kind of updates it might require," Matthew F. wrote.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

https://thedailywtf.com/articles/text-should-go-here

There are certain things which you see in code that, at first glance, if you haven’t already learned better, look like they might almost be clever. One of those in any construct that starts with:

switch(true) {…}

It seems tempting at various points. Your cases can be boolean conditions now, but you can also collapse cases together, getting tricky with breaks to build complex logic. It’s more compact than a chain of ifs. It’s also almost always the wrong thing to do.

Kasha stumbled across this while tracking down a bug:

    // The variable names in this code have been anonymized to protect the guilty. In the original they were actually ok.
    private function foo($a, $b)
    {
        switch (true){
            case ($a&&$b): return 'b';
                break;
            case (!$a&&!$b): return 'c';
                break;
            case $a: return 'a';
                break;
            casedefault: return 'unknown';
                break;
        }
    }

As Kasha’s comment tells us, we won’t judge by the variable names in use here. Even so, the awkward switch also contains awkward logic, and seems designed for unreadability. It’s not the most challenging logic to trace. Even Kasha writes “I comprehended the piece just fine at first while looking at it, and only later it hit me how awkward it is.” And that’s the real issue: it’s awkward. It’s not eye-bleedingly bad. It’s not cringe-worthy. It’s just the sort of thing that you see in your codebase and grumble about each time you see it.

And that’s the real WTF in this case. The developer responsible for the code produces a lot of this kind of code. They never use an if if they can compact it into a switch. They favor while(true) with breaks over sensible while loops.

And in this case, they left a crunch-induced typo which created the bug: casedefault is not part of the PHP switch syntax. Like most languages with a switch, the last condition should simply be default:.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

https://thedailywtf.com/articles/switch-off

We’re still in the early part of the year, and as little glitches show up from “sliding window” fixes to the Y2K bug, we’re seeing more and more little stories of other date rollover weirdness in our inbox.

Like, for example, the Y2K15 bug, which Encore got to get surprised with. It feels like date issues are turning into a sports game franchise: new releases of the same thing every year.

A long, long time ago, Encore’s company released a piece of industrial machinery with an embedded controller. It was so long ago and so embedded that things like floating point operations were a little to newfangled and expensive to execute, and memory was at an extreme premium.

The engineer who originally designed the device had a clever solution to storing dates. One byte of EEPROM could be dedicated to storing the last two digits of the year. In RAM, a nibble- 4 bits- would then store an offset relative to that base year.

Yes, this had Y2K issues, but that wasn’t really a concern at the time. It also had a rollover issue every 16 years. That also wasn’t really a concern, because it was attached to a giant machine which needed annual service to keep functioning properly. Every few years, the service tech could bring an EEPROM progammer device and flash the base year value in the EEPROM. And if someone missed 16 years worth of service calls, they probably had other problems.

Time passed. Some customers did miss 16 years of service calls. Over time, new features got added. The control interface got an improved LCD. Bluetooth got attached. The networking stack changed. A reporting database got bundled with the product, so all the data being produced by the device could get aggregated and reported on. The way the software interacted with the hardware changed, and it meant that the hardware ran at a lower temperature and could go longer between service calls. But at its core, the chip and the software didn’t change all that much.

In that time, there were also changeovers in the engineering team. People left the company, new engineers joined, documentation languished, never getting updated. Years might pass without anybody touching the software, then suddenly a flurry of customer requests that needed patched RIGHT NOW would come through, and anybody who vaguely understood the software got roped in to do the work, then shunted back off to other projects.

On New Year’s Day, 2016, a deluge of tickets started coming in. Encore, as the last person to have touched the software, started picking them up. They all expressed the same problem: the date had rolled over to 2000. The reporting database was confused, the users were confused, and even if they tried to set the clock to 2016 manually, it would roll back from 2015 to 2000.

Now, no one at the company, including Encore, actually knew about the date system in use at this point. The support manual did say that rollovers meant the device had gone 16 years without being properly serviced, but some of these customers had brand new devices, less than a year old. And customers with devices older than 16 years weren’t seeing this problem.

Encore investigated, and picked apart how the date handling worked. That, itself, wasn’t the problem. It took a lot more investigation to track down the problem, including going back to the board schematics to trace how various hardware components were connected. After a few hair-on-fire weeks of crisis management, Encore pieced together the series of events as they were best able.

Sometime after the year 2000, Bluetooth was added to the device. Something about how the Bluetooth module connected to the other components had broken the flasher-software that could update the base year. This meant that the devices had never had their base year set, and simply had a 0 value- 0x00, or the year 2000.

Which meant, for the next 16 years, everything was fine. Techs went out, tried to flash the EEPROM, reset the clock to the correct date, and went about their business, never aware that they hadn’t actually done anything. But come 2016, all of these devices rolled back over to the year 2000.

Encore was able to figure out a script to trick the system into adjusting the output to correct the base year issue, but it also meant many customers had database crammed with bad data that needed to be adjusted to correct the erroneous year.

After this, Encore’s company released upgraded version of the system which contained a GPS receiver, so that it could set its date based on that, but a large number of their customers weren’t interested in the upgrade. Encore has already blocked off the first few weeks of 2032 in preparation.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

https://thedailywtf.com/articles/y2k15

There’s always a hope that in the future, our code will be better. Eventually, we won’t be dealing with piles of krufty legacy code and unprepared programmers and business users who don’t understand how clicking works. It’s 2020: we officially live in the future. Things aren’t better.

Duane works in Go, and has a piping hot “Representative Line” written in 2020. If, like me, you don’t quite know Go, it still looks pretty terrible at first glance:

db.Raw("update backlog set id = ? where id = ?;", job_id, job_id).Row()

The SQL query is… a head scratcher, for sure. id is the primary key in this case, and as a general rule, updating the primary key in a relational database is not a good idea: it is the identity of the record, and if it’s used in relationships you can have weird cascading failures. But it’s okay in this case, since we’re setting the id equal to job_id where id already equals job_id, which gives us a nice NOP.

Theoretically, this might cause some triggers to fire, but that’s it’s own WTF.

There are other problems here, if you know a little bit about Go. First, db is a “GORM” object- Go’s ORM layer. If you just want to update a single object, using the ORM layer directly is probably cleaner and more readable. But if you do want to execute raw SQL that returns no results, like an update, the correct method to use is Exec, not Raw. Raw + Row is used when you intend to capture the results.

Duane adds: “The return result from Row() isn’t assigned to a variable. So this line ignores any output that Row() might have had, including any errors.”

Duane also adds: “This particular programmer is no longer working for us for some reason.”

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

https://thedailywtf.com/articles/gormless-and-gone

As we've discussed in the past, video game code probably shouldn't be held to the standards of your average WTF: they're operating under wildly different constraints. So, for example, when a popular indie game open sources itself, and people find all sorts of horrors in the codebase: hey, the game shipped and made money. This isn't life or death stuff.

It's a little different when you're building the engine. You're not just hacking together whatever you need to make your product work, but putting together a reusable platform to make other people's products work.

Rich D, who previously shared some horrors he found in the Unreal engine, recently discovered that UnrealScript has a useful sounding JsonObject. Since Rich is thinking in terms of mods, being able to read/write JSON to handle mod configuration is useful, but anyone designing a game might have many good reasons to want JSON documents.

The file starts promisingly with:

class JsonObject extends Object
	native;


/// COMMENT!!


…

It's good that someone put that comment there, because I assume it was meant as a reminder: comment this code. And the comments start giving us hints of some weird things:

/**
 * Looks up a value with the given key in the ObjectMap. If it was a number
 * in the Json string, this will be prepended with \# (see below helpers)
 *
 * @param Key The key to search for
 *
 * @return A string value
 */
native function string GetStringValue(const string Key);

The method GetStringValue returns a string from JSON, but if the string is a number, it… puts a \# in front of it? Why?

function int GetIntValue(const string Key)
{
	local string Value;
	
	// look up the key, and skip the \#
	Value = Mid(GetStringValue(Key), 2);
	return int(Value);
}

Oh… that's why. So that we can ignore it. There's a similar version of this method for GetFloatValue, and GetBoolValue.

So, how do those \#s get prepended? Well, as it turns out, there are also set methods:

function SetIntValue(const string Key, int Value)
{
	SetStringValue(Key, "\\#" $ Value);
}

In addition to these methods, there are also native methods (e.g., methods which bind to native code, and thus don't have an UnrealScript body) to encode/decode JSON:

/**
 * Encodes an object hierarchy to a string suitable for sending over the web
 *
 * @param Root The toplevel object in the hierarchy
 *
 * @return A well-formatted Json string
 */
static native function string EncodeJson(JsonObject Root);

/**
 * Decodes a Json string into an object hierarchy (all needed objects will be created)
 *
 * @param Str A Json string (probably received from the web)
 *
 * @return The root object of the resulting hierarchy
 */
static native function JsonObject DecodeJson(const string Str);

Guided by this code, Rich went on to do a few tests:

• Calling SetStringValue with a string that happens to start with \# causes EncodeJson to produce malformed output.
• Calling SetStringValue with any string that might require escape characters (newlines, backslashes, etc.) will not escape those characters, producing malformed output.
• Which means that the output of EncodeJson cannot reliably be parsed by DecodeJson, as sometimes the output is invalid
• Sometimes, when DecodeJson receives an invalid document, instead of throwing an error, it just crashes the entire game

Rich has wisely decided not to leverage this object, for now.

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

https://thedailywtf.com/articles/an-unreal-json-parser

"I guess my kid cousins won't be putting these together for a while," Travis writes.

Noah writes, "Who doesn't want a little error in their coffee first thing in the morning?"

"It would appear that Walmart and I have very different ideas about what constitutes 'Hard Candy'," wrote Todd P.

"Looks like the push to build roads in Poland paid off! Like, really REALLY paid off," Krzysztof wrote.

"Google nailed it - it caught all the cities that I visited in the um...Pacific region?" Francois wrote.

Darnell S. writes, "I beg to differ with WizzAir Cars' opinion of what is a 'good choice'...I want that free one!"

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

https://thedailywtf.com/articles/are-you-old-enough-to-know-better

"So, let me get this straight," Derrick said. He closed his eyes and took a deep breath while massaging his temples before letting out an exasperated sigh. "Not a single person... in this entire organization... is taking ANY responsibility for Ropeway? No one is even willing to admit that they know anything about this application...?"

The Operations team had grown accustomed to their new director's mannerisms and learned it's just better to stay silent and let Derrick think out loud. Afterall, no one envied his job or his idealistic quest for actual compliance. If had he been at the bank as long as his team had, Derrick would have learned that there's compliance... and then there's "compliance."

"But we figured out that Ropeway somehow automatically transfers underwriting overrides from ISAC to AppPortal?" Derrick paused to collect his thoughts before a lightbulb went off. "Wait, wait. Those systems are both covered under our IBM Master Service Agreement, right? What did they say? Chris... did you reach out to our IBM liaison?"

"Well," Chris silently thanked everything good that Ropeway wasn't his problem. "IBM says that they have no idea. They said it's not in the scope of the MSA or any SOW, but they'd be happy to come out and—"

"Ab-so-lute-ly not," Derrick interrupted. He wasn't IBM's biggest fan, to put it mildly. "I've already eaten into next year's budget on this SSL initiative, and there's no way I'm gonna pay them just to tell me I have to pay them even more to fix what shouldn't even by my problem!"

Derrick let out another sigh, rubbing his temples again. "All I want," he grimaced, "is for Ropeway to use HTTPS instead of HTTP. That's all! Fine... fine! Chris, let's just move the whole damn Ropeway server behind the proxy."

"Roger that," Chris nodded, "We'll start prepping things for next week's maintenance window."

There was a lot of risk to moving Ropeway. The Operations team knew how to keep it running – it was just a Windows Service application – but they had no way of knowing if the slightest change in the environment would break things. Moving the server behind the http-to-https proxy meant a new IP and a new subnet, and they had seen far too may "if (IP==10.10.22.30) production_env = true" traps to know they can't just move things without a test plan.

But since no one on the business or Development side was willing to help, they were on their own and it'd be Derrick's head if ISAC or AppPortal stopped working once that maintenance window was over. But for the sake of actual compliance – not "compliance" – these were the risks Derrick was willing to take: SSL was simply non-negotiable.

##

"You'll never believe what I found on that Ropeway server," Chris said while popping into to Derrick's office. Actually, he knew that wasn't true; Derrick had come to expect the unbelievable, but Chris liked to prep Derrick nonetheless. Derrick took a deep breath and moved his hand towards his forehead.

"I found this." Chris plopped down a thick, tattered manila envelope that was covered in yellowed tape. "It was... um... taped to the server's chassis."

Derrick was legitimately surprised and started riffling through the contents as Chris explained things. "So apparently Ropeway was built by this guy, Jody Dorchester, at Roman, uh, wait. Ronin Software or something."

"And yeah," Chris continued as Derrick's eyes widened while he flipped through page-after-page-after page of documentation, "Jody apparently wrote all sorts of documentation... installation instructions, configuration instructions – and all the source code is on that enclosed CD-ROM."

Derrick was speechless. "This," he stuttered, "is dated... March ...of 2000."

"Yup," Chris jumped in nonchalantly, "but I took a shot in the dark here and sent Jody an email."

"And...," Chris said, smiling. He handed Derrick another document and said, "here's his reply."

Ropeway! Wow... that takes me back. I can't believe that's still around, and if you're contacting me about it... you must be desperate ;)

I built that app in a past life... I don't know how much this will help, but I'll tell you what I remember.

There was a fellow at the bank, Eric (or maybe Edward?), and he was a Director of Technology and a VP of something. I met him at a local networking event, and he told me about some project he was working on that was going to reduce all sorts of mindless paperwork.

One department over there was apparently printing out a bunch of records from their system and then faxing all that paper to a different department, who would then manually re-enter all those records into a different system. They had a bunch of data entry people, but the bigger problem was that it was slow, and there were a lot of mistakes, and it was costing a lot of money.

It sounded like a huge, complicated automation project – but actually, your guy had it mostly figured out. The one system could spit-out CSV files to some network share, and the other could import XML files via some web interface. He asked if I could help with that, and I said sure... why not? It just seemed like a really simple Windows service application.

It took a bit longer to wrangle those two formats together than I hoped, but I showed it off and he seemed absolutely thrilled. However, I'll never forget the look on his face when I told him the cost. It was something like 16 hours at $50/hr (my rate back then). I thought he was upset that I took so long, and billed $800 for something that seemed so simple.

Not even close. He said that IBM quoted over $100k to do the exact same thing – but there was just no way he could sell that he got this massive integration project accomplished for only $800. He said I needed to bill more... a lot more.

So, I made that little importer app as robust and configurable as what, VB4 would allow? Then I spent, like, a week writing that documentation you guys found, taped to the server. You know, I actually bought that server and did all the installation myself? Well after all that, he was satisfied with my new bill.

Anyways, I can't remember anything about the application, but there's probably a whole section of the docs dedicated to configuring it. Hopefully you guys can figure it out... good luck!

Jody was right. On page 16, there was, in fact, a section dedicated to the Web API configuration. To change use SSL, one would just have to open the service control tool, check off the "use secure connection," and then Ropeway would construct the Service URL using HTTPS instead of HTTP. That was it, apparently.

"I just... I can't believe it," Derrick paused, and shook his head before massaging his forehead – this time more in surprise that his usual temple massage. "No way. It can't be that simple. Nothing here is that simple!"

Chris shrugged his shoulders and said, "Checking that box does seem a bit simpler than moving—"

"Look," Derrick interrupted, "even if we change the setting, and that setting works, someone needs to own Ropeway, and take responsibility for it going forward. This is like rule number one of actual compliance!"

Chris nodded, and just let Derrick build up to what would certainly be yet another compliance rant.

"Come on," Derrick said enthusiastically, "Ropeway is so absurdly documented... look at this! Surely someone in Core Dev, Biz Apps, Vendor Apps, or heck, even Apps Integration will adopt this thing!? I mean... you realize what will happen if AppPortal craps out because the Ropeway integration breaks some day?"

Obviously, Chris knew exactly how bad it would be, but he let Derrick tell him anyways.

"I can't even imaginae it," Derrick took a breath. "The crap storm that all of those groups would face... this might even make the shareholder meeting! We gotta do something. How about we go to the ISAC team lead and say... here's Ropeway. It's your baby now. Congrats! Here's the docs, it's really easy to use, just--"

"Look," Chris cut in, soberly. "If you want me to go deliver that message -- in the middle of their whole Agile ScrumOps whatever war with Biz Apps – I will. But I'm pretty sure they're gonna go straight to the execs and push to expand the MSA with IBM to cover the App Portal integration..."

Chris paused for a few moments to let Derrick realize exactly who's budget an MSA expansion might come from. Derrick's eyes widened as he took another deep breath.

"Oooor", Chris continued, "this envelope still, umm, has a quite a bit of tape attached to it. Maybe... just maaaaybe...we never found the envelope in the first place? And perhaps, I don't know, Ropeway just... uhhh... happens to check that box itself one day? I don't know? It's an old app... old apps do weird things. Who knows? I don't... do you? Wait... I've already forgotten, what's Ropeway?"

Derrick slowly shook his head and started massaging the bridge of his nose with his index fingers. He let out a completely different sigh, defeated sigh, then uncharacteristically mumbled, "...better get some more tape..."

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

https://thedailywtf.com/articles/the-compliance-ropeway

"For my sins," John writes, "I'm working on a SharePoint 2010 migration."

This tells us that John has committed a lot of sins. But not as many as one of his coworkers.

Since they were running a farm of SharePoint servers, they needed to know what was actually running, which was quite different from the documentation which told them what was supposed to be running. John's coworker did some googling, some copy-and-pasting, some minor revisions of their own, and produced this wad of PowerShell scripting which does produce the correct output.

# there could be more than one search service, so this code is wrapped in a for loop with $tempCnt as the loop variable
$cmdstr = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $searchServiceAppIds[$tempCnt] | Select Id | ft -HideTableHeaders | Out-String -Width 1000
$cmdstr = $cmdstr.Trim().Split("`n")

for($i = 0; $i -lt $cmdstr.Length ; $i++)
{
    $cmdstr2 = $cmdstr[$i].Trim() 
    $searchServiceAppID = $searchServiceAppIds[$tempCnt]
    $tempXML = [xml] (Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $searchServiceAppIds[$tempCnt] | select Name, Type, DeleteCount, ErrorCount, SuccessCount, WarningCount, StartAddresses, Id, CrawlStatus, CrawlStarted, CrawlCompleted, CrawlState | where {$_.Id -eq $cmdstr2 } | ConvertTo-Xml -NoTypeInformation)
    $tempstr = [System.String] $tempXML.Objects.Object.InnerXML
    $searchServiceAppID = $searchServiceAppID + "|" + $cmdstr2				 
    $global:SearchConfigContentSources.Add($searchServiceAppID, $tempstr)
}	

This is particularly ugly, even ignoring the SharePoint-ness that's going on. This block's final output is a hashtable where the keys are the IDs of various SharePoint "content sources", and the value is a description in XML.

If we trace through it, we can see the basic flow of logic:

Use Get-SPEnterpriseSearchCrawContentSource to get a list of all of the content source IDs, using Select and ft to filter the output. Format the output with Out-String -Width 1000, making each row in the result set 1000 characters long.

We can then split on newlines, and start iterating across the list. Since everything is 1000 characters long, let's trim them, and then let's… invoke Get-SPEnterpriseSearchCrawlContentSource to fetch the full set of content sources again, but this time we'll grab different fields and use where to filter down to the one we're currently looking for. This has the nice effect of forcing us to fetch all the records once for every content source.

Then, of course, we have $cmdstr and $cmdstr2, neither of which are command strings, but instead store content source IDs.

This script will be invoked many times during their migration, and with a large number of content sources, it's not exactly going to perform terribly well. This tortured logic is pretty typical of John's coworker's approach to problems, but as this script is only needed during the migration and will eventually be thrown away, John adds:

The output's basically fine, so I'm not even gonna fix it.

Which is likely the right answer. The real problem to fix is how this code got written in the first place.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

https://thedailywtf.com/articles/sharing-the-power

Twenty years out, people have a hard time remembering that Y2K was an actual thing, an actual problem, and it was only solved because people recognized the danger well ahead of time, and invested time and effort into mitigating the worst of it. Disaster didn’t come to pass because people worked their butts off to avoid it.

Gerald E was one of those people. He worked for a cellular provider as a customer service rep, providing technical support and designing the call-center scripts for providing that support. As 1999 cranked on, Gerald was pulled in to the Y2K team to start making support plans for the worst case scenarios.

The first scenario? Handling calls when “all phone communication stopped working”. Gerald didn’t see much point in building a script for that scenario, but he gamely did his best to pad “we can’t answer the phones if they don’t ring” into a “script”.

There were many other scenarios, though, and Gerald was plenty busy. Since he was in every meeting with the rest of the Y2K team, he got to watch their preparedness increase in real time, as different teams did their tests and went from red-to-green in the test results. A few weeks before the New Year, most everything was green.

Y2K fell on a Saturday. As a final preparation, the Y2K team decided to do a final dry-run test, end-to-end, on Wednesday night. They already ran their own internal NTP server which every device on the network pulled from in one way or another, so it was easy to set the clock forward. They planned to set the clock so that at December 29th, 22:30 wall-clock time the time server would report January 1st, 00:00.

The Y2K team gathered to watch their clock count down, and had plans to watch the changeover happen and then go party like it was 1999 while they still had time.

At 22:29, all systems were green. At 22:30- when the time server triggered Y2K- the entire building went dark. There was no power. The backup generator didn’t kick on. The UPSes didn’t kick over. Elevator, Phones, HVAC, everything was down.

No one had expected this catastrophic a failure. The incident room was on the 7th floor of the building. The server room was in the basement. Gerald, as the young and spry CSR was handed a flashlight and ended up spending the next few hours as the runner, relaying information between the incident room and the server room.

In the wee hours of the morning, and after Gerald got his cardio for the next year, the underlying problem became clear. The IT team had a list of IT assets. They had triaged them all, prioritized their testing, and tested everything.

What no one had thought to do was inventory the assets managed by the building services team. Those assets included a bunch of industrial control systems which managed little things, like the building’s power system. Nothing from building services had ended up in their test plan. The backup generator detected the absence of power and kicked on- but the building’s failure meant that the breakers tripped and refused to let that power get where it was needed. Similar issues foiled their large-scale UPS- they could only get the servers powered up by plugging them directly into battery backups.

It was well into the morning on December 30th when they started scrambling to solve the problem. Folks were called back from vacation, electricians were called in and paid exorbitant overtime. It was an all-hands push to get the building wired up in such a way that it wouldn’t just shut down.

It was a straight crunch all the way until New Year’s Eve, but when the clock hit midnight, nothing happened.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

https://thedailywtf.com/articles/y-ok

As a general rule, if you find yourself writing an extension system for your application, stop and do something else. It's almost always in the case of YAGNI: you ain't gonna need it.

George is a "highly paid consultant", and considers himself one of the "good ones": he delivers well tested, well documented, and clean code to his clients. His peer, Gracie on the other hand… is a more typical representative of the HPC class.

George and Gracie found themselves with a problem: based on the contents of a configuration file, they needed to decide what code to execute. Now, you might be thinking that some kind of conditional statement or maybe some sort of object-oriented inheritance thing would do the job.

There were five different code paths, and no one really expected to see those code paths change significantly. Gracie, who was identified as "the architect" on the responsibility matrix for the project, didn't want to write five different ways to do a similar task, so instead, she wrote one way to do all those tasks.

Here's the YAML configuration file that her efforts produced:

use.cases:
  0:
    description: Default product
    user_file_name_cond: ('product_start' in file_name) and ('pilot' not in file_name)
    user_file_name_pf_truncate: '.sig'
    data_files: False
    downstream_file_name_templates:
      signal: _product_start_.sig
    downstream_script: another_script.sh
  1:
    description: Full forced product
    user_file_name_cond: ('force_all_products' in file_name) and ('pilot' not in file_name)
    user_file_name_pf_truncate: '.sig'
    data_files: False
    downstream_file_name_templates:
      signal: _product_start__pilot.sig
    conf_override: ['FORCE_PRODUCT=TRUE']
    downstream_script: another_script.sh
  2:
    description: Pilot forced product
    user_file_name_cond: ('force_all_offers' in file_name) and ('pilot' in file_name)
    user_file_name_pf_truncate: '_pilot_user.sig'
    data_files: True
    downstream_file_name_templates:
      signal: _product_start__pilot_user.sig
      pilot_users: _pilot_users_.txt
    conf_override: ['FORCE_PRODUCT=TRUE']
    downstream_script: another_script.sh
  3:
    description: Pilot product
    user_file_name_cond: ('product' in file_name) and ('pilot_user' in file_name)
    user_file_name_pf_truncate: '_pilot.sig'
    data_files: True
    downstream_file_name_templates:
      signal: _product_start__pilot_user.sig
      pilot_users: _pilots_.txt
    conf_override: ['FORCE_OFFERMATCH=FALSE']
    downstream_script: another_script.sh
  4:
    description: Forced assignment
    user_file_name_cond: ('user_offer_mapping' in file_name) or ('budget' in file_name) and ('csv' in file_name)
    user_file_name_pf_truncate: '_user_offer_mapping.sig'
    data_files: True
    downstream_file_name_templates:
      signal: _force_offer_assignment_start__user_product_mapping.sig
      user_offer_mapping: _user_products_.txt
      budget: budget_.csv
    downstream_script: product/python/forced_assignment/src/main/forced_assignment.py
  5:
    description: Test
    user_file_name_cond: ('dummy' in file_name)
    user_file_name_pf_truncate: '.sig'
    data_files: False
    downstream_file_name_templates:
      signal: _dummy_.sig
    downstream_script: python/forced_offers/semaphore/bin/dummy_downstream_script.sh
    wait_strs: ['dummy_downstream_script.sh']

Take a look at the user_file_name_cond field in each of those entries. Are you thinking to yourself, "Boy, that looks like some Python syntax in there?"

def get_file_use_case(file_name):
    for use_case, use_case_conf in Configuration.get_conf('use.cases').items():
        user_file_name_cond = use_case_conf['user_file_name_cond'].replace('\', '')
        if eval(user_file_name_cond):
            return use_case

You know that's an eval. It's at least not an exec, which is allowed to use the assignment operator, but there's nothing in eval that prevents it from having side effects. While a malicious config file would fail to do user_case_conf['downstrem_script'] = 'my_hostile_script', it could easily do user_case_conf.__setitem__('downstream_script', 'my_hostile_script').

Of course, hostile attacks aren't really the concern here, at least intentionally hostile. This code was going to run in production, and clients were going to be free to edit and modify those files however they liked. Malicious attacks aside, idiotic mistakes, or worse: clever hacks were entirely possible.

George did his best to illustrate why this was a terrible idea. Gracie, who was the architect, gleefully ignored him. The code got shipped. The client signed off on it. It runs in production.

There is a silver-lining here. George aggressively documented their solution, and his reasons for opposing it, and followed the company policy for involving management in conflicts. Management saw his point, and there was a reshuffling in the teams: George was made the architect, Gracie was "pivoted" to doing more development. While it was too late to fix the code, George used his position to make some policy changes: he worked with the team to democratically build a set of official code standards, made sure that they were applied to all code equally, and revised the team's approach to retros to improve their code in the future.

Of course, there's one person on the team who isn't interested in these changes. Their PRs end up being lengthy flamewars over whether or not they should be following policies. You can guess who that is…

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

https://thedailywtf.com/articles/yet-another-master-of-evil

Brian writes, "Of course server %1 is trustworthy, I couldn't do my work without it!"

"Rockefeller lived on Millionaires' Row, but I think it was a little bit later than the 18th century," Bryan writes.

"I've heard about 64-bit builds being inherently less efficient than their 32-bit cousins, but I didn't know Notepad++ would need to download this much extra data!" wrote Lee M.

Shawn M. writes, "I wonder if this is how new hackers get their start at pwning web sites?"

"This trip left me feeling a bit empty, regardless, I still gave 5 stars," Martin C. wrote.

"Let me guess Outlook 2016, you aren't 'old enough' to understand umlauts?" Justus B. wrote.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

https://thedailywtf.com/articles/variable-trust

Kaylee E made an "oops" and checked in a unit test with a bug in it which caused the test to fail. She didn't notice right away, and thus the commit hit their CI pipeline and was automatically pulled by the build server. She assumed that when she checked the logs she'd see the error, but she didn't. The build completed, and Tests (0/0) ran successfully.

Now, Kaylee was new to the codebase, and since she'd been doing small changes, she'd simply written and run tests around explicitly the functionality she was testing. She hadn't yet done a full test run locally, so that was her next step. From there, it was easy to see why the build server didn't automatically run tests.

[Test]
public void TestDateNew() {
    String date = DateTime.Now.ToShortDateString();
    Assert.IsTrue(date = "26/08/2016");
}

Visual Studio's Git integration also displayed a little tip above the method, announcing: 0 References | Murray Linwood, 646 days ago | 1 author, 1 change.

Now, this test could work if you mocked out the DateTime object, which is what you should do if you're testing dates. Of course, that doesn't make the test any more useful, as it's just testing basic, built-in .NET functionality. Besides, I suspect that "Murray" is already familiar with mocking.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

https://thedailywtf.com/articles/untested-builds

We close out our recap of 2019 and head into the new year with one last flashback: when vendors forget what the definition of "unique" is. Original -- Remy

Gather 'round, young'uns, for a tale from the Dark Ages of mobile programming: the days before the iPhone launched. Despite what Apple might have you believe, the iPhone wasn't the first portable computing device. Today's submitter, Jack, was working for a company that streamed music to these non-iPhone devices, such as the Palm Treo or the Samsung Blackjack. As launch day approached for the new client for Windows Mobile 6, our submitter realized that he'd yet to try the client on a non-phone device (called a PDA, for those of you too young to recall). So he tracked down an HP iPaq on eBay just so he could verify that it worked on a device without the phone API.

The device arrived a few days out from launch, after QA had already approved the build on other devices. It should've been a quick test: sideload the app, stream a few tracks, log in, log out. But when Jack opened the app for the first time on the new device, it was already logged into someone's account! He closed it and relaunched, only to find himself in a different, also inappropriate account. What on earth?!

The only thing Jack could find in common between the users he was logged in as was that they were running the same model of PDA. That was the crucial key to resolving the issue. To distinguish which device was making the calls to the streaming service, Jack used a call in Windows Mobile that would return a unique ID for each mobile device. In most devices, it would base this identifier on the IMEI, ensuring uniqueness—but not on the HP iPaq. All HP devices could automatically log into the account of the most recently used iPaq, providing the user logged out and back in, as it would generate a recent-user record with the device ID.

Jack had read the documentation many times, and it always stated that the ID was guaranteed to be unique. Either HP had a different definition of "unique" than anyone else, or they had a major security bug!

Jack emailed HP, but they had no plans to fix the issue, so he had to whip up an alternate method of generating a UUID in the case that the user was on this device. The launch had to be pushed back to accommodate it, but the hole was plugged, and life went on as usual.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

https://thedailywtf.com/articles/best-of-2019-when-unique-isn-t-unique

Did you get some nice shiny new IoT devices for the holidays this year? Hope they weren't the Initech brand. Original --Remy

Mindy was pretty excited to start her internship with Initech's Internet-of-Things division. She'd been hearing at every job fair how IoT was still going to be blowing up in a few years, and how important it would be for her career to have some background in it.

It was a pretty standard internship. Mindy went to meetings, shadowed developers, did some light-but-heavily-supervised changes to the website for controlling your thermostat/camera/refrigerator all in one device.

As part of testing, Mindy created a customer account on the QA environment for the site. She chucked a junk password at it, only to get a message: "Your password must be at least 8 characters long, contain at least three digits, not in sequence, four symbols, at least one space, and end with a letter, and not be more than 10 characters."

"Um, that's quite the password rule," Mindy said to her mentor, Bob.

"Well, you know how it is, most people use one password for every site, and we don't want them to do that here. That way, when our database leaks again, it minimizes the harm."

"Right, but it's not like you're storing the passwords anyway, right?" Mindy said. She knew that even leaked hashes could be dangerous, but good salting/hashing would go a long way.

"Of course we are," Bob said. "We're selling web connected thermostats to what can be charitably called 'twelve-o-clock flashers'. You know what those are, right? Every clock in their house is flashing twelve?" Bob sneered. "They can't figure out the site, so we often have to log into their account to fix the things they break."

A few days later, Initech was ready to push a firmware update to all of the Model Q baby monitor cameras. Mindy was invited to watch the process so she could understand their workflow. It started off pretty reasonable: their CI/CD system had a verified build, signed off, ready to deploy.

"So, we've got a deployment farm running in the cloud," Bob explained. "There are thousands of these devices, right? So we start by putting the binary up in an S3 bucket." Bob typed a few commands to upload the binary. "What's really important for our process is that it follows this naming convention. Because the next thing we're going to do is spin up a half dozen EC2 instances- virtual servers in the cloud."

A few more commands later, and then Bob had six sessions open to cloud servers in tmux. "Now, these servers are 'clean instances', so the very first thing I have to do is upload our SSH keys." Bob ran an ssh-copy-id command to copy the SSH key from his computer up to the six cloud VMs.

"Wait, you're using your personal SSH keys?"

"No, that'd be crazy!" Bob said. "There's one global key for every one of our Model Q cameras. We've all got a copy of it on our laptops."

"All… the developers?"

"Everybody on the team," Bob said. "Developers to management."

"On their laptops?"

"Well, we were worried about storing something so sensitive on the network."

Bob continued the process, which involved launching a script that would query a webservice to see which Model Q cameras were online, then sshing into them, having them curl down the latest firmware, and then self-update. "For the first few days, we leave all six VMs running, but once most of them have gotten the update, we'll just leave one cloud service running," Bob explained. "Helps us manage costs."

It's safe to say Mindy learned a lot during her internship. Mostly, she learned, "don't buy anything from Initech."

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

https://thedailywtf.com/articles/best-of-2019-the-internship-of-things