Once a month, web developers from across the Mozilla Project get together to try and transmute a fresh Stanford graduate into a 10x engineer. Meanwhile, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Osmose: SpeedKills and Advanced Open File

First up was Osmose (that’s me!) presenting two packages for Atom, a text editor. SpeedKills is a playful package that plays a guitar solo and sets text on fire when the user types fast enough. Advanced Open File is a more useful package that adds a convenient dialog for browsing the file system and opening files by path rather than using the fuzzy finder. Both are available for install through the Atom package repository.

new_one: Tab Origin

Next was new_one, who shared Tab Origin, a Firefox add-on that lets you return to the webpage that launched the current tab, even if the parent tab has since been closed. It’s activated via a keyboard shortcut that can be customized.

Potch: WONTFIX and Presentation Mode

Continuing a fine tradition of batching projects, Potch stopped by to show off two Firefox add-ons. The first was WONTFIX, which adds a large red WONTFIX stamp to any Bugzilla bug that has been marked as WONTFIX. The second was Presentation Mode, which allows you to full-screen any content in a web page while hiding the browser chrome. This is especially useful when giving web-based presentations.

Peterbe: premailer.io

Peterbe shared premailer.io, which is a service wrapping premailer. Premailer takes a block of HTML with a style tag and applies the styles within as style attributes on each matching tag. This is mainly useful for HTML emails, which generally don’t support style tags that apply to the entire email.

ErikRose: Spam-fighting Tips

ErikRose learned a lot about the current state of spam-fighting while redoing his mail server:

• Telling Postfix to be picky about RFCs is a good first pass. It eliminates some spam without having to do much computation.
• spamassassin beats out dspam, which hasn’t seen an update since 2012.
• Shared-digest detectors like Razor help a bit but aren’t sufficient on their own without also greylisting to give the DBs a chance to catch up.
• DNS blocklists are a great aid: they reject 3 out of 4 spams without taking much CPU.
• Bayes is still the most reliable (though the most CPU-intense) filtration method. Bayes poisoning is infeasible, because poisoners don’t know what your ham looks like, so don’t worry about hand-picking spam to train on. Train on an equal number of spams and hams: 400 of each works well. Once your bayes is performing well, crank up your BAYES_nn settings so spamassassin believes it.
• Crank up spamc’s –max-size to 1024000, because spammers are now attaching images > 512K to mails to bypass spamc’s stock 512K threshold. This will cost extra CPU.

With this, he gets perhaps a spam a week, with over 400 attempts per day.

---

We were only able to get a 3x engineer this month, but at least they were able to get a decent job working on enterprise software.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

https://blog.mozilla.org/webdev/2015/06/29/beer-and-tell-june-2015/

As soon as a developer at Mozilla starts integrating a new WebAPI feature, the Mozilla Security team begins working to help secure that API. Subtle programming mistakes in new code can introduce annoying crashes and even serious security vulnerabilities that can be triggered by malformed input which can lead to headaches for the user and security exposure.

WebAPIs start life as a specification in the form of an Interface Description Language, or IDL. Since this is essentially a grammar, a grammar-based fuzzer becomes a valuable tool in finding security issues in new WebAPIs because it ensures that expected semantics are followed most of the time, while still exploring enough undefined behavior to produce interesting results.

We came across a grammar fuzzer Ben Hawkes released in 2011 called “Dharma.” Sadly, only one version was ever made public. We liked Ben’s approach, but Dharma was missing some features which were important for us and its wider use for API fuzzing. We decided to sit down with our fuzzing mates at BlackBerry and rebuild Dharma, giving the results back to the public, open source and licensed as MPL v2.

We redesigned how Dharma parses grammars and optimized the speed of parsing and the generating of fuzzed output, added new grammar features to the grammar specification, added support for serving testcases over a WebSocket server, and made it Python 3 ready. It comes with no dependencies and runs out of the box.

In theory Dharma can be used with any data that can be represented as a grammar. At Mozilla we typically use it for APIs like WebRTC, WebAudio, or WebCrypto.

Dharma has no integrated harness. Feel free to check out the Quokka project which provides an easy way for launching a target with Dharma, monitoring the process and bucketing any faults.

Dharma is actively in use and maintained at Mozilla and more features are planned for the future. Ideas for improvements are always greatly welcomed.

Dharma is available via GitHub (preferred and always up-to-date) or via PyPi by running “pip install dharma”.

References
– https://github.com/mozillasecurity/dharma
– https://github.com/mozillasecurity/quokka
– https://code.google.com/p/dharma/

https://blog.mozilla.org/security/2015/06/29/dharma/

A brief retrospective

I just returned from Whistler, BC, along with 1300 other Mozillians.

Because we’re a global organization, it’s important to get everyone together every now and then. We matched faces with IRC channels, talked shop over finger-food, and went hiking with random colleagues. Important people gave speeches. Teams aligned. There was a massive party at the top of a mountain.

+++++

Typically, I learn the most from experiences like these only after they have ended. Now that I’ve had 48 hours to process (read: recover), some themes have emerged…

1. Organizing and mobilizing a bunch of smart, talented, opinionated people is hard work.
2. It’s easy to say “you’re doing it wrong.” It takes courage to ask “how could we do it better?”
3. Anyone can find short-term gains. The best leaders define a long-term vision, then enable individuals to define their own contribution.
4. Success is always relative, and only for a moment in time.
5. Accelerated change requires rapid adaptation. Being small is our advantage.
6. The market is what you make it. So make it matter.
7. It’s been said that when technology works, it’s magic. I disagree. Magic is watching passionate people – from all over the world – create technology together.

Also, it has now been confirmed by repeatable experiment: I’m not a people person – especially when lots of them are gathered in small spaces. I’m fucking exhausted.

Officially signing back on,

-DCROBOT

http://autodidacticdesigner.com/2015/06/29/not-working-in-whistler/

http://blog.mozilla.org/community/2015/06/29/storify-recap-of-participation-at-whistler/

A few days ago I’ve attended the launch of SohoXI – Hook and Loop’s latest product that aims to change the look and feel of the Enterprise applications forever. I am close to believing that and not only because I work for the same company as they do. They have delivered extremely revolutionary UI in … Continue reading Voice-controlled UI plus a TTS engine.

http://talkweb.eu/voice-controlled-ui-plus-a-tts-engine/

Last week was Mozilla’s first work week of 2015 in Whistler, BC. It was my first visit to Whistler having joined Mozilla shortly after their last summit there in 2010, and it was everything I needed it to be. Despite currently feeling jetlagged, I have been recharged and I have renewed enthusiasm for the mission and I’m even more than a little excited about Firefox OS again! I’d like to share a few of my highlights from last week…

• S’mores – The dinner on Wednesday evening was followed by a street party, where I had my first S’more. It was awesome!
• Firefox OS – Refreshing honesty over past mistakes and a coherant vision for the future has actually made me enthusiastic about this project again. I’m no longer working directly on Firefox OS, but I’ve signed up for the dogfoxfooding program and I’m excited about making a difference again.
• LEGO – I got to build a LEGO duck, and we heard from David Robertson about the lessons LEGO learned from near bankruptcy.
• Milkshake – A Firefox Q&A was made infinitely better by taking a spontaneous walk to Cow’s for milkshakes and ice cream with my new team!
• Running – I got to run with #running friends old and new on Tuesday morning around Lost Lake. Then on Thursday morning I headed back and took on the trails with Matt. These were my first runs since my marathon, and running through the beautiful scenary was exactly what I needed to get me back into it.
• Istanbul – After dinner on Tuesday night, Stephen and I sat down with Bob to play the board game Istanbul.
• Hacking – It’s always hard to get actual code written during these team events, but I’m pleased to say we thought through some challenging problems, and actually even managed to land some code.
• Hike – On Friday morning I joined Justin and Matt on a short hike up Whistler mountain. We didn’t have long before breakfast, but it was great to spend more time with these guys.
• Whistler Mountain – The final party was at the top of Whistler Mountain, which was just breathtaking. I can’t possibly do the experience justice – so I’m not even going to try.

Thank you Whistler for putting up with a thousand Mozillians, and thank you Mozilla for organising such a perfect week. We’re going to keep rocking the free web!

http://blargon7.com/2015/06/whistler-work-week-2015/

Since my last post on “remoties”, I’ve done several more presentations, some more consulting work for private companies, and even started writing this down more explicitly (exciting news coming here soon!). While I am always refining these slides, this latest version is the first major “refactor” of this presentation in a long time. I think this restructuring makes the slides even easier to follow – there’s a lot of material to cover here, so this is always high on my mind.

Without further ado – you can get the latest version of these slides, in handout PDF format, by clicking on the thumbnail image.

Certainly, the great responses and enthusiastic discussions every time I go through this encourages me to keep working on this. As always, if you have any questions, suggestions or good/bad stories about working remotely or as part of a geo-distributed teams, please let me know (either by email or in the comments below) – I’d love to hear them.

Thanks
John.

http://oduinn.com/blog/2015/06/28/we-are-all-remoties-jun2015-edition/

And at long last the 38 beta is very happy to meet you too (release notes, downloads, hashes). Over the next few weeks I hope you and the new TenFourFox 38 will become very fond of each other, and if you don't, then phbbbbt.

There are many internal improvements to 38. The biggest one specific to us, of course, is the new IonPower JavaScript JIT backend. I've invested literally months in making TenFourFox's JavaScript the fastest available for any PowerPC-based computer on any platform, not just because every day websites lard up on more and more crap we have to swim through (viva Gopherspace) but also because a substantial part of the browser is written in JavaScript: the chrome, much of the mid-level plumbing and just about all those addons you love to download and stuff on in there. You speed up JavaScript, you speed up all those things. So now we've sped up many browser operations by about 11 times over 31.x -- obviously the speed of JavaScript is not the only determinant of browser speed, but it's a big part of it, and I think you'll agree that responsiveness is much improved.

JavaScript also benefits in 38 from a compacting, generational garbage collector (generational garbage collection was supposed to make 31 but was turned off at the last minute). This means recently spawned objects will typically be helplessly slaughtered in their tender youth in a spasm of murderous efficiency based on the empiric observation that many objects are created for brief usage and then never used again, reducing the work that the next-stage incremental garbage collector (which we spent a substantial amount of time tuning in 31 as you'll recall, including backing out background finalization and tweaking the timeslice for our slower systems) has to do for objects that survive this pediatric genocide. The garbage collector in 38 goes one step further and compacts the heap as well, which is to say, it moves surviving objects together contiguously in memory instead of leaving gaps that cannot be effectively filled. This makes both object cleanup and creation much quicker in JavaScript, which relies heavily on the garbage collector (the rest of the browser uses more simplistic reference counting to determine object lifetime), to say nothing of a substantial savings in memory usage: on my Quad G5 I'm seeing about 200MB less overhead with 48 tabs open.

I also spent some time working on font enumeration performance because of an early showstopper where sites that loaded WOFF fonts spun and spun and spun. After several days of tearing my hair out in clumps the problem turned out to be a glitch in reference counting caused by the unusual way we load platform fonts: since Firefox went 10.6+ it uses CoreText exclusively, but we use almost completely different font code based on the old Apple Type Services which is the only workable choice on 10.4 and the most stable choice on 10.5. ATS is not very fast at instantiating lots of fonts, to say the least, so I made the user font cache stickier (please don't read that as "leaky" -- it's sticky because things do get cleaned up, but less aggressively to improve cache hit percentage) and also made a global font cache where the font's attribute tag directory is cached browser-wide to speed up loading font tables from local fonts on your hard disk. Previously this directory was cached per font entry, meaning if the font entry was purged for re-enumeration it had to be loaded all over again, which usually happened when the browser was hunting for a font with a particular character. This process used to take about fifteen to twenty seconds for the 700+ font faces on my G5. With the global font cache it now takes less than two.

Speaking of showstoppers, here's an interesting one which I'll note here for posterity. nsChildView, the underlying system view which connects Cocoa/Carbon to Gecko, implements the NSTextInput protocol which allows it to accept Unicode input without (as much) mucking about with the Carbon Text Services Manager (Firefox also implements NSTextInputClient, which is the new superset protocol, but this doesn't exist in 10.4). To accept Unicode input, under the hood the operating system actually manipulates a special undocumented TSM input context called, surprisingly, NSTSMInputContext (both this and the undocumented NSInputContext became the documented NSTextInputContext in 10.6), and it gets this object from a previously undocumented method on NSView called (surprise again) inputContext. Well, turns out if you override this method you can potentially cause all sorts of problems, and Mozilla had done just that to handle complex text input for plugins. Under the 10.4 SDK, however, their code ended up returning a null input context and Unicode input didn't work, so since we don't support plugins anyhow the solution was just to remove it completely ... which took several days more to figure out. The moral of the story is, if you have an NSView that is not responding to setMarkedText or other text input protocol methods, make sure you haven't overridden inputContext or screwed it up somehow.

I also did some trivial tuning to the libffi glue library to improve the speed of its calls and force it to obey our compiler settings (there was a moment of panic when the 7450 build did not start on the test machines because dyld said XUL was a 970 binary -- libffi had seen it was being built on a G5 and "helpfully" compiled it for that target), backed out some portions of browser chrome that were converted to CoreUI (not supported on 10.4), and patched out the new tab tile page entirely; all new tabs are now blank, like they used to be in previous versions of Firefox and as intended by God Himself. There are also the usual cross-platform HTML5 and CSS improvements you get when we leap from ESR to ESR like this, and graphics are now composited off-main-thread to improve display performance on multiprocessor systems.

That concludes most of the back office stuff. What about user facing improvements? Well, besides the new blank tabs "feature," we have built-in PDF viewing as promised (I think you'll find this more useful to preview documents and load them into a quicker viewer to actually read them, but it's still very convenient) and Reader View as the biggest changes. Reader View, when the browser believes it can attempt it, appears in the address bar as a little book icon. Click on it and the page will render in a simplified view like you would get from a tool such as Readability, cutting out much of the extraneous formatting. This is a real godsend on slower computers, lemme tell ya! Click the icon again to go back. Certain pages don't work with this, but many will. I have also dragged forward my MP3 decoder support, but see below first, and we have prospectively landed Mozilla bug 1151345 to fix an issue with the application menu (modified for the 10.4 SDK).

You will also note the new, in-content preferences (i.e., preferences appears in a browser tab now instead of a window, a la, natch, Chrome), and that the default search engine is now Yahoo!. I have not made this default to anything else since we can still do our part this way to support MoCo (but you can change it from the preferences, of course).

I am not aware of any remaining showstopper bugs, so therefore I'm going ahead with the beta. However, there are some known issues ("bugs" or "features" mayhaps?) which are not critical. None of these will hold up final release currently, but for your information, here they are:

• If you turn on the title bar, private browsing windows have the traffic light buttons in the wrong position. They work; they just look weird. This is somewhat different than issue 247 and probably has a separate, though possibly related, underlying cause. Since this is purely cosmetic and does not occur in the browser's default configuration, we can ship with this bug present but I'll still try to fix it since it's fugly (plus, I personally usually have the title bar on).

• MP3 support is still not enabled by default because seeking within a track (except to the beginning) does not work yet. This is the last thing to do to get this support off the ground. If you want to play with it in its current state, however, set tenfourfox.mp3.enabled to true (you will need to create this pref). If I don't get this done by 38.0.2, the pref will stay off until I do, but the rest of it is working already and I have a good idea how to get this last piece functional.

• I'm not sure whether to call this a bug or a feature, but scaling now uses a quick and dirty algorithm for many images and some non-.ico favicons apparently because we don't have Skia support. It's definitely lower quality, but it has a lot less latency. Images displayed by themselves still use the high-quality built-in scaler which is not really amenable to the other uses that I can tell. Your call on which is better, though I'm not sure I know how to go back the old method or if it's even possible anymore.

• To reduce memory pressure, 31 had closed tab and window undos substantially reduced. I have not done that yet for 38 -- near as I can determine, the more efficient memory management means it is no longer worth it, so we're back to the default 10 and 3. See what you think.

Builders: take note that you will need to install a modified strip ("strip7") if you intend to make release binaries due to what is apparently a code generation bug in gcc 4.6. If you want to use a different (later) compiler, you should remove the single changeset with the gcc 4.6 compatibility shims -- in the current changeset pack it's numbered 260681, but this number increments in later versions. See our new HowToBuildRightNow38 for the gory details and where to get strip7.

Localizers: strings are frozen, so start your language pack engines one more time in issue 42. We'd like to get the same language set for 38 that we had for 31, and your help makes it possible. Thank you!

As I mentioned before, it's probably 70-30 against there being a source parity version after 38ESR because of the looming threat of Electrolysis, which will not work as-is on 10.4 and is not likely to perform well or even correctly on our older systems. (If Firefox 45, the next scheduled ESR, still allows single process operation then there's a better chance. We still need to get a new toolchain up and a few other things, though, so it won't be a trivial undertaking.) But I'm pleased with 38 so far and if we must go it means we go out on a high note, and nothing says we can't keep improving the browser ourselves separate from Mozilla after we split apart (feature parity). Remember, that's exactly what Classilla does, except that we're much more advanced than Classilla will ever be, and in fact Pale Moon recently announced they're doing the same thing. So if 38 turns out to be our swan song as a full-blooded Mozilla tier 3 port, that doesn't mean it's the end of TenFourFox as a browser. I promise! Meanwhile, let's celebrate another year of updates! PowerPC forever!

Finally, looking around the Power Mac enthusiast world, it appears that SeaMonkeyPPC has breathed its last -- there have been no updates in over a year. We will pour one out for them. On the other hand, Leopard Webkit continues with regular updates from Tobias, and our friendly builder in the land of the Rising Sun has been keeping up with Tenfourbird. We have the utmost confidence that there will be a Tenfourbird 38 in your hands soon as well.

Some new toys to play with are next up in a couple days.

http://tenfourfox.blogspot.com/2015/06/hello-38-beta-its-nice-to-meet-you.html

http://blog.bonardo.net/2013/01/29/add-ons-devs-heads-up-history-api-removals-in-places

Pillow 2.9.0 will be released on July 1, 2015.

http://blog.aclark.net/2015/06/28/pillow-2-9-0-is-almost-out/

Long time readers of this blog will remember when The Architecture of Open Source Applications (vol2) was published, containing a chapter describing the tools and mindsets used when re-building Mozilla’s Release Engineering infrastructure. (More details about the book, about the kindle and nook versions, and about the Russian version(!).

Dr Dobbs recently posted an article here which is an edited version of the Mozilla Release Engineering chapter. As a long time fan of Dr Dobbs, seeing this was quite an honor, even with the sad news here.

Obviously, Mozilla’s release automation continues to evolve, as new product requirements arise, or new tools help further streamline things. There is still lots of interesting work being done here – for me, top of mind is Task Cluster, and ScriptHarness (v0.1.0 and v0.2.0). Release Engineering at scale is both complex, and yet very interesting – so you should keep watching these sites for more details, and consider if they would also help in your current environment. As they are all open source, you can of course join in and help!

For today, I just re-read the Dr. Dobbs article with a fresh cup of coffee, and remembered the various different struggles we went through as we scaled Mozilla’s infrastructure up so we could quickly grow the company, and the community. And then in the middle of it all, found time with armenzg, catlee and lsblakk to write about it all. While some of the technical tools have changed since the chapter was written, and some will doubtless change again in the future, the needs of the business, the company and the community still resonate.

For anyone doing Release Engineering at scale, the article is well worth a quiet read.

http://oduinn.com/blog/2015/06/27/mozillas-release-engineering-now-on-dr-dobbs/

Lots is changing in Telemetry land. If you do occasionally run data analyses with our Spark infrastructure you might want to keep reading.

Background

The Telemetry and FHR collection systems on desktop are in the process of being unified. Both systems will be sending their data through a common data pipeline which has some features of both the current Telemetry pipeline as well the Cloud Services one that we use to ingest server logs.

The goals of the unification are to:

• avoid measuring the same metric in multiple systems on the client side;
• reduce the latency from the time a measurement occurs until it can be analyzed on the server;
• increase the accuracy of measurements so that they can be better correlated with factors in the user environment such as the specific build, enabled add-ons, and other hardware or software characteristics;
• use a common data pipeline for client telemetry and service log data.

The unified pipeline is currently sending data for Nightly, Aurora and Beta. Classic FHR and Telemetry pipelines are going to keep sending data to the very least until the new unified pipeline has not been fully validated. The plan is to land this feature in 40 Release. We’ll also continue to respect existing user preferences. If the user has opted out of FHR or Telemetry, we’ll continue to respect that for the equivalent data sets. Similarly, the opt-out and opt-in defaults will remain the same for equivalent data sets.

Data format

A Telemetry ping, stored as JSON object on the client, encapsulates the data sent to our backend. The main differences between the new unified Telemetry ping format (v4) and the classic Telemetry one (v2) are that:

• multiple ping types are supported beyond the classic saved-session ping, like the main ping;
• pings have a common top-level which contains basic information shared between types, like build-id and channel;
• pings have an optional environment field which consists of data that is expected to be characteristic for performance and other behavior.

From an analysis point of view, the most important addition is the main ping which includes the very same histograms and other performance and diagnostic data as the v2 saved-session pings. Unlike in “classic” Telemetry though, there can be multiple main pings during a single session. A main ping is triggered by different scenarios, which are documented by the reason field:

• aborted-session: periodically saved to disk and deleted at shutdown – if a previous aborted session ping is found at startup it gets sent to our backend;
• environment-change: generated when the environment changes;
• shutdown: triggered when the browser session ends;
• daily: a session split triggered in 24h hour intervals at local midnight; this is needed to make sure we keep receiving data also from clients that have very long sessions.

Data access through Spark

Once you connect to a Spark enabled IPython notebook launched from our self-service dashboard, you will be prompted with a new tutorial based on the v4 dataset. The v4 data is fetched through the get_pings function by passing “v4'' as the schema parameter. The following parameters are valid for the new data format:

• app: an application name, e.g.: “Firefox”;
• channel: a channel name, e.g.: “nightly”;
• version: the application version, e.g.: “40.0a1'';
• build_id: a build id or a range of build ids, e.g.:”20150601000000'' or (“20150601000000'', “20150610999999”)
• submission_date: a submission date or a range of submission dates, e.g: “20150601” or (“20150601'', “20150610”)
• doc_type: ping type, set to “main” by default
• fraction: the fraction of pings to return, set to 1.0 by default

Once you have a RDD, you can further filter the pings down by reason. There is also a new experimental API that returns the history of submissions for a subset of profiles, which can be used for longitudinal analyses.

http://robertovitillo.com/2015/06/27/a-glance-at-unified-fhrtelemetry/

Friday, foxyeah!

It’s been a very busy and successful work week here in beautiful Whistler, BC. People are taking advantage of being in the same location to meet, plan, hack, and socialize. A special thanks to Jordan for inviting us to his place in beautiful Squamish for a BBQ!

(Note: No release engineering folks were harmed by bears in the making of this work week.)

tl;dr

Whistler: Keynotes were given by our exec team and we learned we’re focusing on quality, dating our users to get to know them better, and that WE’RE GOING TO SPACE!! We also discovered that at LEGO, Everything is Awesome now that they’re thinking around the box instead of inside or outside of it. Laura’s GoFaster project sounds really exciting, and we got a shoutout from her on the way we manage the complexity of our systems. There should be internal videos of the keynotes up next week if you missed them.

Internally, we talked about Q3 planning and goals, met with our new VP, David, met with our CEO, Chris, presented some lightning talks, and did a bunch of cross-group planning/hacking. Dustin, Kim, and Morgan talked to folks at our booth at the Science Fair. We had a cool banner and some cards (printed by Dustin) that we could hand out to tell people about try. SHIP IT!

Taskcluster: Great news; the TaskCluster team is joining us in Platform! There was lots of evangelism about TaskCluster and interest from a number of groups. There were some good discussions about operationalizing taskcluster as we move towards using it for Firefox automation in production. Pete also demoed the Generic Worker!

Puppetized Windows in AWS: Rob got the nxlog puppet module done. Mark is working on hg and NSIS puppet modules in lieu of upgrading to MozillaBuild 2.0. Jake is working on the metric-collective module. The windows folks met to discuss the future of windows package management. Q is finishing up the performance comparison testing in AWS. Morgan, Mark, and Q deployed runner to all of the try Windows hosts and one of the build hosts.

Operational: Amy has been working on some additional nagios checks. Ben, Rail, and Nick met and came up with a solid plan for release promotion. Rail and Nick worked on releasing Firefox 39 and two versions of Firefox ESR. Hal spent much of the week working with IT. Dustin and catlee got some work on on migrating treestatus to relengapi. Hal, Nick, Chris, and folks from IT, sheriffs, dev-services debugged problems with b2g jobs. Callek deployed a new version of slaveapi. Kim, Jordan, Chris, and Ryan worked on a plan for addons. Kim worked with some new buildduty folks to bring them up to speed on operational procedures.

Thank you all, and have a safe trip home!

…

And here are all the details:

Taskcluster

• We got to spend some quality time with the our new TaskCluster teammates, Greg, Jonas, Wander, Pete, and John. We’re all looking forward to working together more closely.
• Morgan convinced lots of folks that Taskcluster is super amazing, and now we have a lot of people excited to start hacking on it and moving their workloads to it.
• We put together a roadmap for TaskCluster in Trello and identified the blockers to turning Buildbot Scheduling off.

Puppetized Windows in AWS

• Rob has pushed out the nxlog puppet module to get nxlog working in scl3 (bug 1146324). He has a follow-on bug to modify the ec2config file for AWS to reset the log-aggregator host so that we’re aggregating to the local region instead of where we instantiate the instance (like we do with linux). This will ensure we have Windows system logs in AWS (bug 1177577).
• The new version of MozillaBuild was released, and our plan was to upgrade to that on Windows (bug 1176111). An attempt at that showed that the way hg was compiled requires an external dll (likely something from cygwin), and needs to be run from bash. Since this would require significant changes, we’re going to install the old version of MozillaBuild and put upgrades of hg (bug 1177740) and NSIS on top of that (like we’re doing with GPO now). Future work will include splitting out all the packages and not using MozillaBuild. Jake is working on the puppet module for metric-collective, our host-level stats gathering software for windows (similar to collectd on windows/OS X). This will give use Windows system metrics in graphite in AWS (bug 1097356).
• We met to talk about Windows packaging and how to best integrate with puppet. Rob is starting to investigate using NuGet and Chocolatey to handle this (bugs 1175133 and 1175107).
• Q spun up some additional instance types in AWS and is in the process of getting some more data for Windows performance after the network modifications we made earlier (bug 1159384).
• Jordan added a new puppetized path for all windows jobs, fixing a problem we were seeing with failing sendchanges on puppetized machines (bug 1175701).
• Morgan, Mark, and Q deployed runner to all of the try Windows hosts (bug 1055794).

Operational

• The relops team met to perform a triage of their two bugzilla queues and closed almost 20% of the open bugs as either already done or wontfix based on changes in direction.
• Amy has been working on some additional nagios checks for some Windows services and for AWS subnets filling up (bugs 1164441 and 793293).
• Ben, Rail, and Nick met and came up with a solid plan for the future of release promotion.
• Rail and Nick worked on getting Firefox 39 (and the related ESR releases) out to our end users.
• Hal spent lots of time working with IT and the MOC, improving our relationships and workflow.
• Dustin and catlee did some hacking to start the porting of treestatus to relengapi (one of the blockers to moving us out of PHX1).
• Hal, Nick, Chris, and folks from IT, sheriffs, dev-services tracked down an intermittent problem with the repo-tool impacting only b2g jobs (bug 1177190).
• Callek deployed the new version of slaveapi to support slave loans using the AWS API (bug 1177932).
• Kim, Jordan, Chris, and Ryan discussed the initial steps for future addon support.
• Coop (hey, that’s me) held down the buildduty fort while everyone else was in Whistler

See you next week!

http://coopcoopbware.tumblr.com/post/122595134980

31.8.0 is available, the last release for the 31 series (release notes, downloads, hashes). Download it and give it one last spin. 31 wasn't a high water mark for us in terms of features or performance, but it was pretty stable and did the job, so give it a salute as it rides into the sunset. It finalizes Monday PM Pacific time as usual.

I'm trying very hard to get you the 38.0.1 beta by sometime next week, probably over the July 4th weekend assuming the local pyros don't burn my house down with errant illegal fireworks, but I keep hitting showstoppers while trying to dogfood it. First it was fonts and then it was Unicode input, and then the newtab crap got unstuck again, and then the G5 build worked but the 7450 build didn't, and then, and then, and then. I'm still working on the last couple of these major bugs and then I've got some additional systems to test on before I introduce them to you. There are a couple minor bugs that I won't fix before the beta because we need enough time for the localizers to do their jobs, and MP3 support is present but is still not finished, but there will be a second beta that should address most of these problems prior to our launch with 38.0.2. Be warned of two changes right away: no more tiles in the new tab page (I never liked them anyway, but they require Electrolysis now, so that's a no-no), and Check for Updates is now moved to the Help menu, congruent with regular Firefox, since keeping it in its old location now requires substantial extra code that is no longer worth it. If you can't deal with these changes, I will hurt you very slowly.

Features that did not make the cut: Firefox Hello and Pocket, and the Cisco H.264 integration. Hello and Pocket are not in the ESR, and I wouldn't support them anyway; Hello needs WebRTC, which we still don't really support, and you can count me in with the people who don't like a major built-in browser component depending exclusively on a third-party service (Pocket). As for the Cisco integration, there will never be a build of those components for Tiger PowerPC, so there. Features that did make the cut, though, are pdf.js and Reader View. Although PDF viewing is obviously pokier compared to Preview.app, it's still very convenient, generally works well enough now that we have IonPower backing it, and is much safer. However, Reader View on the other hand works very well on our old systems. You'll really like it especially on a G3 because it cuts out a lot of junk.

After that there are two toys you'll get to play with before 38.0.2 since I hope to introduce them widely with the 38 launch. More on that after the beta, but I'll whet your appetite a little: although the MacTubes Enabler is now officially retired, since as expected the MacTubes maintainer has thrown in the towel, thanks to these projects the MTE has not one but two potential successors, and one of them has other potential applications. (The QuickTime Enabler soldiers on, of course.)

Last but not least, I have decided to move the issues list and the wiki from Google Code to Github, and leave downloads with SourceForge. That transition will occur sometime late July before Google Code goes read-only on August 24th. (Classilla has already done this invisibly but I need to work on a stele so that 9.3.4 will be able to use Github effectively.) In the meantime, I have already publicly called Google a bunch of meaniepants and poopieheads for their shameful handling of what used to be a great service, so my work here is done.

http://tenfourfox.blogspot.com/2015/06/3180-available-say-goodbye.html

A software organization wants to make a promise, for example about its data practices. For example, “We don’t store information on your location”. They can keep that promise in two ways: code or policy.

If they were keeping it in code, they would need to be open source, and would simply make sure the code didn’t transmit location information to the server. Anyone can review the code and confirm that the promise is being kept. (It’s sometimes technically possible for the company to publish source code that does one thing, and binaries which do another, but if that was spotted, there would be major reputational damage.)

If they were keeping it in policy, they would add “We don’t store information on your location” to their privacy policy or Terms of Service. The documents can be reviewed, but in general you have to trust the company that they are sticking to their word. This is particularly so if the policy states that it does not create a binding obligation on the company. So this is a function of your view of the company’s reputation.

Geeks like promises kept in code. They can’t be worked around using ambiguities in English, and they can’t be changed without the user’s consent (to a software upgrade). I suspect many geeks think of them as superior to promises kept in policy – “that’s what they _say_, but who knows?”. This impression is reinforced when companies are caught sticking to the letter but not the spirit of their policies.

But some promises can’t be kept in code. For example, you can’t simply not send the user’s IP address, which normally gives coarse location information, when making a web request. More complex or time-bound promises (“we will only store your information for two weeks”) also require policy by their nature. Policy is also more flexible, and using a policy promise rather than a code promise can speed time-to-market due to reduced software complexity and increased ability to iterate.

Question: is this distinction, about where to keep your promises, useful when designing new features?

Question: is it reasonable or misguided for geeks to prefer promises kept in code?

Question: if Mozilla or its partners are using promises kept in policy for e.g. a web service, how can we increase user confidence that such a policy is being followed?

http://feedproxy.google.com/~r/HackingForChrist/~3/3moEEvP5TEg/

Introduction

Aaron Klotz, Avi Halachmi and I have been studying Firefox’s performance on Android & Windows over the last few weeks as part of an effort to evaluate Firefox “content performance” and find actionable issues. We’re analyzing and measuring how well Firefox scrolls pages, loads sites, and navigates between pages. At first, we’re focusing on 3 reference sites: Twitter, Facebook, and Yahoo Search.

We’re trying to find reproducible, meaningful, and common use cases on popular sites which result in noticeable performance problems or where Firefox performs significantly worse than competitors. These use cases will be broken down into tests or profiles, and shared with platform teams for optimization. This “Content Performance” project is part of a larger organizational effort to improve Firefox quality.

I’ll be regularly posting blog posts with our progress here, but you can can also track our efforts on our mailing list and IRC channel:

Mailing list: https://mail.mozilla.org/listinfo/contentperf
IRC channel: #contentperf
Project wiki page: Content_Performance_Program

Summary of Current Findings (June 18)

Generally speaking, desktop and mobile Firefox scroll as well as other browsers on reference sites when there is only a single tab loaded in a single window.

• We compared Firefox vs Chrome and IE:
  • Desktop Firefox scrolling can badly deteriorate when the machine is in power-saver mode¹ (Firefox performance relative to other browsers depends on the site)
  • Heavy activity in background tabs badly affects desktop Firefox’s scrolling performance¹ (much worse than other browsers — we need E10S)
  • Scrolling on infinitely-scrolling pages only appears janky when the page is waiting on additional data to be fetched
• Inter-page navigation in Firefox can exhibit flicker, similar to other browsers
• The Firefox UI locks up during page loading, unlike other browsers (need E10S)
• Scrolling in desktop E10S (with heavy background tab activity) is only as good as the other browsersn¹ when Firefox is in the process-per-tab configuration (dom.ipc.processCount >> 1)

¹ You can see Aaron’s scrolling measurements here: http://bit.ly/1K1ktf2

Potential scenarios to test next:

• Check impact of different Firefox configurations on scrolling smoothness:
  • Hardware acceleration disabled
  • Accessibility enabled & disabled
  • Maybe: Multiple monitors with different refresh rate (test separately on Win 8 and Win 10)
  • Maybe: OMTC, D2D, DWrite, display & font scaling enabled vs disabled
    • If we had a Telemetry measurement of scroll performance, it would be easier to determine relevant characteristics
• Compare Firefox scrolling & page performance on Windows 8 vs Windows 10
  • Compare Firefox vs Edge on Win 10
• Test other sites in Alexa top 20 and during random browsing
• Test the various scroll methods on reference sites (Avi has done some of this already): mouse wheel, mouse drag, arrow key, page down, touch screen swipe and drag, touchpad drag, touchpad two finger swipe, trackpoints (special casing for ThinkPads should be re-evaluated).
  • Check impact of pointing device drivers
• Check performance inside Google web apps (Search, Maps, Docs, Sheets)
  • Examine benefits of Chrome’s network pre-fetcher on Google properties (e.g. Google search)
  • Browse and scroll simple pages when top Google apps are loaded in pinned tabs
• Compare Firefox page-load & page navigation performance on HTTP/2 sites (Facebook & Twitter, others?)
• Check whether our cache and pre-connector benefit perceived performance, compare vs competition

Issues to report to Platform teams

• Worse Firefox scrolling performance with laptop in power-save mode
• Scrolling Twitter feed with YouTube HTML5 videos is jankier in Firefox
• bug 1174899: Scrolling on Facebook profile with many HTML5 videos eventually causes 100% CPU usage on a Necko thread + heavy CPU usage on main thread + the page stops loading additional posts (videos)

Tooling questions:

• Find a way to to measure when the page is “settled down” after loading, i.e. time until last page-loading event. This could be measured by the page itself (similar to Octane), which would allow us to compare different browsers
• How to reproduce dynamic websites offline?
• Easiest way to record demos of bad Firefox & Fennec performance vs other browsers?

Decisions made so far:

• Exclusively focus on Android 5.0+ and Windows 7, 8.1 & 10
• Devote the most attention to single-process Nightly on desktop, but do some checks of E10S performance as well
• Desktop APZC and network pre-fetcher are a long time away, don’t wait

https://blog.mozilla.org/vdjeric/2015/06/26/announcing-the-content-performance-program/

I’m delighted to see that development of Mozilla’s Web Literacy Map is still continuing after my departure a few months ago.

Mark Surman, Executive Director of the Mozilla Foundation, wrote a blog post outlining the way forward and a working group has been put together to drive forward further activity. It’s great to see Mark Lesser being used as a bridge to previous iterations.

Another thing I’m excited to see is the commitment to use Open Badges to credential Web Literacy skills. We tinkered with badges a little last year, but hopefully there’ll be a new impetus around this.

The approach to take the Web Literacy Map from version 1.5 to version 2.0 is going to be different from the past few years. It’s going to be a ‘task force’ approach with people brought in to lend their expertise rather than a fully open community approach. That’s probably what’s needed at this point.

I’m going to give myself some space to, as my friend and former colleague Laura Hilliger said, 'disentangle myself’ from the Web Literacy Map and wider Mozilla work. However, I wish them all the best. It’s important work.

---

Comments? Questions? I’m @dajbelshaw on Twitter or you can email: mail@dougbelshaw.com

http://literaci.es/web-literacy-map-v20

Dineapple is an online food delivery gig that I have been working on recently. In essence, a new food item is introduced periodically, and interested customers place orders online to have their food delivered the next day.

Getting down to the initial build of the online ordering site, I started to think about the technical whats and hows. For this food delivery service, a customer places an order by making an online payment. The business then needs to know of this transaction, and have it linked to the contact information of the customer.

Oh okay, easy. Of course I’ll set up a database. I’ll store the order details inside a few tables. Then I’ll build a mini application to extract this information and generate a daily report for the cooks and delivery people to operate on. Then I started to build these things in my head. But wait, there is a simpler way to get the operations people aware of orders. We could just send an email to the people on every successful transaction to notify them of a new incoming order. But this means the business loses visibility and data portability. Scraping for relational data from a bunch of automated emails, although possible, will be a nightmare. The business needs to prepare to scale, and that means analytics.

Then I saw something that now looks so obvious I feel pretty embarrassed. Payments on the ordering service are processed using Stripe. When the HTTP request to process a payment is made, Stripe provides an option to submit additional metadata that will be tagged to the payment. There is a nice interface on the Stripe site that allows business owners to do some simple analytics on the payment data. There is also the option to export all of that data (and metadata) to CSV for more studying.

Forget about ER diagrams, forget about writing custom applications, forget about using automated emails to generate reports. Stripe is capable of doing the reporting for Dineapple, we just had to see a way to adapt the offering to fit the business’s use case.

Beyond operations reporting through Stripe, there are so many existing web services out there that can be integrated into Dineapple. Just to name a few, an obvious one would be to use Google Analytics to study site traffic. Customers’ reviews on food and services could (and probably should) be somehow integrated to work using Yelp. Note that none of these outsourced alternatives, although significantly easier to implement, compromise on the quality of the solution for the business. Because at the end of the day, all that really matters is that the business gets what it needs.

So here’s a reminder to my future self. Spend a little more time looking around for simpler alternatives that you can take advantage of before jumping into development for a custom solution.

Engineers are builders by instinct, but that isn’t always a good thing.

http://blog.williecheong.com/maximum-value-minimum-effort/

I truly believe, that to make Mozilla a place worth ‘hanging your hat‘, we need to get better at being ‘forces of good for each other’.  I like to think this idea is catching on, but only time will tell.

This month’s #mozlove post is for Tom Farrow AKA ‘Tad’,  a long time contributor, in numerous initiatives across Mozilla.  Although Tad’s contribution focus is in Community Dev Ops, it’s his interest in teaching youth digital literacy that first led to a crossing of our paths. You’ll probably find it interesting to know that despite being in his sixth(!!) year of contribution to Mozilla –  Tad is still a High School in Solihull Birmingham, UK.

Tad starting contribution to Mozilla after helping a friend install Firefox on their government-issued laptop, which presented some problems. He found help on SUMO, and through being helped was inspired to become a helper and contributor himself.  Tad speaks fondly of starting with SUMO, of finding friends, training and mentorship.

Originally drawn to IT and DevOps contribution for the opportunity of ‘belonging to something’, Tad has become a fixture in this space helping design hosting platforms, and the evolution of a multi-tenant WordPress hosting. When I asked what was most rewarding about contributing to Community Dev Ops, he shared that pride in innovating a quality solution.

I’m also increasingly curious about the challenges of participation and asked about this as well.  Tad expressed some frustration around ‘access and finding the right people to unlock resources’.  I think that’s probably something that speaks to the greater challenges for the Mozilla community in understanding pathways for support.

Finally my favorite question:  “How do your friends and family relate to your volunteer efforts? Is it easy or hard to explain volunteering at Mozilla?”.

I don’t really try to explain it – my parents get the general idea, and are happy I’m gaining skills in web technology.

I think it’s very cool that in a world of ‘learn to code’ merchandizing, that Tad found his opportunity to learn and grow technical skills in participation at Mozilla :)

I want to thank Tad for taking the time to chat with me, for being such an amazing contributor, and inspiration to others around the project.

* I set a reminder in my calendar every month, which this month happens to be during Mozilla’s Work Week in Whistler.  Tad is also in Whistler, make sure you look out for him – and say hello!

 

 

 

 

 

http://tiptoes.ca/mozlove-for-tad/

A few people have suggested I look at other packages for config solutions. I thought I'd record some of my thoughts on the matter. Let's look at requirements first.

Requirements

1. Commandline argument support. When running scripts, it's much faster to specify some config via the commandline than always requiring a new config file for each config change.
   
   
2. Default config value support. If a script assumes a value works for most cases, let's make it default, and allow for overriding those values in some way.
   
   
3. Config file support. We need to be able to read in config from a file, and in some cases, several files. Some config values are either too long and unwieldy to pass via the commandline, and some config values contain characters that would be interpreted by the shell. Plus, the ability to use diff and version control on these files is invaluable.
   
   
4. Multiple config file type support. json, yaml, etc.
   
   
5. Adding the above three solutions together. The order should be: default config value -> config file -> commandline arguments. (The rightmost value of a configuration item wins.)
   
   
6. Config definition and validation. Commandline options are constrained by the options that are defined, but config files can contain any number of arbitrary key/value pairs.
   
   
7. The ability to add groups of commandline arguments together. Sometimes familes of scripts need a common set of commandline options, but also need the ability to add script-specific options. Sharing the common set allows for consistency.
   
   
8. The ability to add config definitions together. Sometimes families of scripts need a common set of config items, but also need the ability to add script-specific config items.
   
   
9. Locking and/or logging any changes to the config. Changing config during runtime can wreak havoc on the debugability of a script; locking or logging the config helps avoid or mitigate this.
   
   
10. Python 3 support, and python 2.7 unicode support, preferably unicode-by-default.
    
    
11. Standardized solution, preferably non-company and non-language specific.
    
    
12. All-in-one solution, rather than having to use multiple solutions.
    
    

Packages and standards

argparse

Argparse is the standardized python commandline argument parser, which is why configman and scriptharness have wrapped it to add further functionality. Its main drawbacks are lack of config file support and limited validation.

1. Commandline argument support: yes. That's what it's written for.
   
   
2. Default config value support: yes, for commandline options.
   
   
3. Config file support: no.
   
   
4. multiple config file type support: no.
   
   
5. Adding the above three solutions together: no. The default config value and the commandline arguments are placed in the same Namespace, and you have to use the parser.get_default() method to determine whether it's a default value or an explicitly set commandline option.
   
   
6. Config definition and validation: limited. It only covers commandline option definition+validation, and there's the required flag but not a if foo is set, bar is required type validation. It's possible to roll your own, but that would be script-specific rather than part of the standard.
   
   
7. Adding groups of commandline arguments together: yes. You can take multiple parsers and make them parent parsers of a child parser, if the parent parsers have specified add_help=False
   
   
8. Adding config definitions together: limited, as above.
   
   
9. The ability to lock/log changes to the config: no. argparse.Namespace will take changes silently.
   
   
10. Python 3 + python 2.7 unicode support: yes.
    
    
11. Standardized solution: yes, for python. No for other languages.
    
    
12. All-in-one solution: no, for the above limitations.
    
    

configman

Configman is a tool written to deal with configuration in various forms, and adds the ability to transform configs from one type to another (e.g., commandline to ini file). It also adds the ability to block certain keys from being saved or output. Its argparse implementation is deeper than scriptharness' ConfigTemplate argparse abstraction.

Its main drawbacks for scriptharness usage appear to be lack of python 3 + py2-unicode-by-default support, and for being another non-standardized solution. I've given python3 porting two serious attempts, so far, and I've hit a wall on the dotdict __getattr__ hack working differently on python 3. My wip is here if someone else wants a stab at it.

1. Commandline argument support: yes.
   
   
2. Default config value support: yes.
   
   
3. Config file support: yes.
   
   
4. Multiple config file type support: yes.
   
   
5. Adding the above three solutions together: not as far as I can tell, but since you're left with the ArgumentParser object, I imagine it'll be the same solution to wrap configman as argparse.
   
   
6. Config definition and validation: yes.
   
   
7. Adding groups of commandline arguments together: yes.
   
   
8. Adding config definitions together: not sure, but seems plausible.
   
   
9. The ability to lock/log changes to the config: no. configman.namespace.Namespace will take changes silently.
   
   
10. Python 3 support: no. Python 2.7 unicode support: there are enough str() calls that it looks like unicode is a second class citizen at best.
    
    
11. Standardized solution: no.
    
    
12. All-in-one solution: no, for the above limitations.
    
    

docopt

Docopt simplifies the commandline argument definition and prettifies the help output. However, it's purely a commandline solution, and doesn't support adding groups of commandline options together, so it appears to be oriented towards relatively simple script configuration. It could potentially be added to json-schema definition and validation, as could the argparse-based commandline solutions, for an all-in-two solution. More on that below.

json-schema

This looks very promising for an overall config definition + validation schema. The main drawback, as far as I can see so far, is the lack of commandline argument support.

A commandline parser could generate a config object to validate against the schema. (Bonus points for writing a function to validate a parser against the schema before runtime.) However, this would require at least two definitions: one for the schema, one for the hopefully-compliant parser. Alternately, the schema could potentially be extended to support argparse settings for various items, at the expense of full standards compatiblity.

There's already a python jsonschema package.

1. Commandline argument support: no.
   
   
2. Default config value support: yes.
   
   
3. Config file support: I don't think directly, but anything that can be converted to a dict can be validated.
   
   
4. Multiple config file type support: no.
   
   
5. Adding the above three solutions together: no.
   
   
6. Config definition and validation: yes.
   
   
7. Adding groups of commandline arguments together: no.
   
   
8. Adding config definitions together: sure, you can add dicts together via update().
   
   
9. The ability to lock/log changes to the config: no.
   
   
10. Python 3 support: yes. Python 2.7 unicode support: I'd guess yes since it has python3 support.
    
    
11. Standardized solution: yes, even cross-language.
    
    
12. All-in-one solution: no, for the above limitations.
    
    

scriptharness 0.2.0 ConfigTemplate + LoggingDict or ReadOnlyDict

Scriptharness currently extends argparse and dict for its config. It checks off the most boxes in the requirements list currently. My biggest worry with the ConfigTemplate is that it isn't fully standardized, so people may be hesitant to port all of their configs to it.

An argparse/json-schema solution with enough glue code in between might be a good solution. I think ConfigTemplate is sufficiently close to that that adding jsonschema support shouldn't be too difficult, so I'm leaning in that direction right now. Configman has some nice behind the scenes and cross-file-type support, but the python3 and __getattr__ issues are currently blockers, and it seems like a lateral move in terms of standards.

An alternate solution may be BYOC. If the scriptharness Script takes a config object that you built from somewhere, and gives you tools that you can choose to use to build that config, that may allow for enough flexibility that people can use their preferred style of configuration in their scripts. The cost of that flexibility is familiarity between scriptharness scripts.

1. Commandline argument support: yes.
   
   
2. Default config value support: yes, both through argparse parsers and script initial_config.
   
   
3. Config file support: yes. You can define multiple required config files, and multiple optional config files.
   
   
4. Multiple config file type support: no. Mozharness had .py and .json. Scriptharness currently only supports json because I was a bit iffy about execfileing python again, and PyYAML doesn't always install cleanly everywhere. It's on the list to add more formats, though. We probably need at least one dynamic type of config file (e.g. python or yaml) or a config-file builder tool.
   
   
5. Adding the above three solutions together: yes.
   
   
6. Config definition and validation: yes.
   
   
7. Adding groups of commandline arguments together: yes.
   
   
8. Adding config definitions together: yes.
   
   
9. The ability to lock/log changes to the config: yes. By default Scripts use LoggingDict that logs runtime changes; StrictScript uses a ReadOnlyDict (sams as mozharness) that prevents any changes after locking.
   
   
10. Python 3 and python 2.7 unicode support: yes.
    
    
11. Standardized solution: no. Extended/abstracted argparse + extended python dict.
    
    
12. All-in-one solution: yes.
    
    

Corrections, additions, feedback?

As far as I can tell there is no perfect solution here. Thoughts?

comments

http://escapewindow.dreamwidth.org/246718.html