Once a month, web developers from across the Mozilla Project get together to forge Bitcoins. While we magnetize needles for carving out our counterfeit bits directly on hard drive platters, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

ErikRose: Rubik’s Magic Cube

First up was ErikRose, who talked about the process of learning how to solve a Rubik’s Cube and how it affected his thinking. The learning process mirrored that of learning a language: first, you only see the cube as a block of individual colors, but as you progress you start to recognize specific cubes and arrangements of cubes as “words”, and eventually you become able to recognize abstract patterns of cubes relative to each other.

Check out the recording for a detailed walk through part of the cube-solving process!

Mythmon: N-Bodies Simulation in Rust

Next was Mythmon, who shared a physics simulation of bodies floating in space being affected by gravity. The simulation was written in Rust and relies on Piston for drawing graphics to the screen.

Potch: Canvas Blur

Potch was next with a demo of performing a Gaussian blur on a canvas in JavaScript. Branching off some experiments around auto-cropping algorithms, Potch’s demo processes the blur in chunks and produces results that are comparable to blurs produced in Photoshop, albeit much slower and with less sampling for the blur. While not intended to actually be used for anything, future improvements include using Web Workers to process the blur asynchronously and in parallel, as well as performing other convolutions besides blurring.

BWalker: ASCII Art Dashboard

Last up was bwalker, who demoed an ASCII art dashboard powered by blessed-contrib. The library provides the ability to make console-based dashboards with widgets for line graphs, bar charts, tables, and even a world map. This particular demo pulled some statistics from Github as well as graphing randomly generated numbers.

The highlight of this week’s meetup was lonnen’s impressive feat of creating 5 bitcoins on a flash drive using only a lighter and a chiropractic activator.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

https://blog.mozilla.org/webdev/2015/10/23/beer-and-tell-october-2015/

http://blog.lizardwrangler.com/2015/10/23/mozilla-launches-open-source-support-program/

Initial Allocation of One Million US Dollars

Today Mozilla is launching an award program specifically focused on supporting open source and free software. Our initial allocation for this program is $1,000,000. We are inviting people already deeply connected to Mozilla to participate in our first set of awards.

Mozilla is a part of the open source and free software movement. We were born out of this movement. We prosper because of the technology and activism which comes from this movement. And we know that open source and free software remains a key part of the Internet and the online life we seek to build. We have had a grant program for many years. Now it is time to formalize a systematic way to provide a new level of support to this community.

The Mozilla Open Source Support program is designed to recognize and celebrate communities who are leading the way with open source projects that contribute to our work and the health of the Web. It encompasses both: a) a “give back” element for open source and free software projects that Mozilla relies on; and b) a “give forward” component for supporting other projects where financial resources from Mozilla can make our entire community more successful. We’ll give more specific names to these components as we go forward. The Mozilla Open Source Support program will also encompass a component supporting increased attention to the security of open source and free software programs. Our initial allocation for Mozilla Open Source Support is one million US dollars. As we develop this program we will determine future allocations.

We will start immediately implementing the “give back” component for open source and free software projects on which Mozilla relies. Our goal is to identify up to 10 projects we rely on and can fund in a thoughtful, meaningful way by December 12th.

I invite Mozillians and our community to participate in the further refinement of this program by suggesting improvements to its terms, which you will find at https://wiki.mozilla.org/MOSS I’ll take input on the design of the program for a week and then will finalize the terms for the first set of awards. Please send suggestions to the Mozilla Open Source Support mailing list; you can subscribe here. The wiki contains a FAQ with more information about the program.

I also invite Mozillians and friends of Mozilla to begin identifying projects we rely on and are good candidates for this program. We are compiling a basic list of projects we rely on here.

I am reminded regularly of how deeply Mozillians identify open source and free software as a critical element of an open Internet and healthy, trustworthy online experiences. I am excited to build a program that helps us bring concrete support to this worldview. You are the key to making this program great – to identifying great projects, to helping figure out what engagement from Mozilla would make a meaningful difference and to deepening Mozilla’s connections with our open source and free software compatriots.

I’m here to support this effort, and to support you in making it awesome.

(Originally posted on my blog, Lizard Wrangling)

https://blog.mozilla.org/blog/2015/10/23/mozilla-launches-open-source-support-program/

After seeing a post from our webdev colleagues a while back, we in IT are jumping on the battlestation post bandwagon. IT is scattered around the globe, with a good mix of office employees and remoties.

With all of the responses I got to for such a post, I will break this up into a couple of posts for the sake of those reading through a feed.

Melissa O’Conner
Location: New Hampshire, USA
Works on: EUS Management, IT Project Management

(she keeps a smartsheet on her wall)

Richard Weiss
Location: San Francisco Office
Works on: IT AWS architecture

Richard has opted for one of the new fancy adjustable standing desks.

Ludovic Hirlimann
Location: France
Works on: Mozilla Operations Center (MOC)

“I don’t drink coffee – my coffee used to be “Tales of the arabian nights”. It needed servicing first, hence it being open.”

Greg Cox
Location: Oregon
Works on: Storage & Virtualization

“one-of-a-kind Mk1 lap, built it myself” …  “work is anywhere the VPN isn’t blocked”

Ashish Vijayaram
Location: San Francisco
Works on: Mozilla Operations Center (MOC)

(Ashish had just returned from maternity leave)  “Thanks to *some* enterprising colleagues, my desk overnight turned into a hybrid working desk + changing table for infants. Designed for oncall ops and dadops – deal with multiple types of incidents – all from the same desk!”

Linda Ypulong
Location: San Francisco
Works on: Mozilla Operations Center (MOC)

“Going for the true minimalist approach – I tend to either be in meetings or sit in the center of the MOC …”

Ryan Watson
Location: UK
Works on: Web Operations

“Corey for scale”
(editors note: sigh..)

Julien Vehent
Location: Florida, USA
Works on: IT Security

“The Gopher”, in its natural state

Philippe Chiasson
Location: Quebec, CA
Works on: AWS / Web Operations

“Believe it or not, but I happen to like the colour green! What’s missing here is the 4 children, cat and dog roaming around, obviously!”

https://blog.mozilla.org/it/2015/10/23/desks-of-it-pt1/

Hello, SUMO Nation!

Here we are, another Friday, another bunch of updates for you. Autumn is fully upon us, and the time change coming this weekend is letting some of us sleep more – hooray! Anyway, sleep aside, here are the news in brief:

Welcome, New Contributors!

• tsakshi
• Jameson Green
• priyansuganguly
• Abbackar DIOMANDE
• echo15
• yefuu

Community

• Philipp has suggested some updates regarding plugin information in Firefox and in issue reporting – support his suggestions!
• The most important update in the last few weeks: Etherpad = Etherpad Lite – read all about it here.
• Reminder: let us know what you think about contributor information at SUMO.
• If you think you can benefit from getting a second-hand device to help you with contributing to SUMO, you know where to find us.

Support Forum

• Thanks to everyone who rocked the forums during yesterday’s SUMO Questions Day!
• We’re getting ready for version 42 launch, stay tuned!

https://blog.mozilla.org/sumo/2015/10/23/whats-up-with-sumo-23rd-october/

We’re just wrapping up another MozReview work week, this time in Toronto. Our main goal was to indoctrinate Glob into MozReview development as he is joining us for at least a few quarters. Since we reserve our fortnightly “Engineering Productivity Updates” for significant contributions, here is a list of my insignificant contributions from this week instead:

• Bug 1177544 This enabled autoland to version-control-tools, which lets us dogfood autolanding to non-Try destinations prior to turning it on for mozilla-inbound. The biggest blocker to landing to mozilla-inbound is Bug 1160479 which will allow Autoland to rewrite commit messages to reflect who actually granted review on a commit in MozReview. As it stands, if you have an r? in your commit message, you’ll have to amend your commit to make it an r= and then push it to MozReview prior to autolanding, which isn’t much better than just pushing it to inbound yourself.

• Bug 1211564 is a workflow fix to allow publishing from the command line even if reviewers have not been specified.

• Bug 1216297 takes advantage of Mercurial 3.5 allowing empty commits to remove the use of mercurial queues in Autoland when creating commits to contain try syntax.

• Bug 1216947 is a small Autoland fix to save the MozReview request json directly rather than breaking it up into separate fields in the database, which will help with Bug 1160479.

• Bug 1217108 adds the capability for Autoland to deal with push races by automatically rebasing if the attempt to push to the destination tree would create a new head.

• Bug 1216078 reduces the amount of information mirrored back to Bugzilla by only posting the full commit message when a commit is first added to MozReview. Each subsequent time that a commit is pushed a link to the interdiff will be posted to Bugzilla instead.

• Bug 1214545 fixed BiquadFilterNode.getFrequencyResponse to return NaN when passed invalid frequencies. Despite being at Mozilla for two and a half years, I had never written any code for the platform, so I was pretty excited to see this land, even though it’s a trivial fix.

http://www.lowleveldrone.com/mozilla/mozreview/2015/10/23/mozreview-workweek.html

Hello Mozillians,

We are happy to announce that Friday, October 30th, we are organizing Firefox 43.0 Aurora Testday. We will be focusing our testing on the new Add-ons Signing feature, and last but not least, Search Suggestions and Unified Autocomplete. Check out the detailed instructions via this etherpad.

No previous testing experience is required, so feel free to join us on #qa IRC channel where our moderators will offer you guidance and answer your questions.

Join us and help us make Firefox better! See you on Friday!

https://quality.mozilla.org/2015/10/firefox-43-0-aurora-testday-october-30th/

Due to a bug with the new ftp server we’ve had to disable updates for

• Firefox for Android Nightly
• Firefox for Android Aurora

They’ll resume just as soon as we can get the fix landed.

Update (Oct 25th): Updates are re-enabled, thanks to Mike Shal for the fix.

https://ftangftang.wordpress.com/2015/10/23/updates-disabled-for-android-nightly-and-aurora/

As part of the Content-Performance program, we recently completed an extensive test of scroll and navigation performance comparison using 20 top sites between 3 browsers on Android (5.0.1, Galaxy S4): Fennec 43 Aurora, Chrome and “Internet” (the default browser on the S4).

• In general, Fennec scrolls worse than Chrome, especially on non trivial pages. But there are other issues too. We filed the following bugs:
  • Bug 1217415 - Fennec page navigation is slower than Chrome on some sites (e.g. Wikipedia, ebay)
  • Bug 1217372 - Fennec has text input lag in autocomplete boxes (google, bing) which Chrome doesn’t.
  • Bug 1217370 - On fast scroll swipes, sometimes the momentum is less than expected (and less than Chrome).
  • Bug 1217364 - Inconsistent scroll progression (momentum) without user inputs.
  • Bug 1217366 - Visible low resolution rendering, especially for fast swipes.

Hopefully, at least some of the scroll issues would be resolved or at the very least improved once Fennec gets async-pan-and-zoom (APZ). APZ is expected to land soon on Fennec nightly 44 before it becomes Aurora (on Firefox-desktop - APZ is already enabled by default on nightly builds).

Content-perf observations

Also on the subject of content-perf, we’ve recently filed quite a few bugs for desktop Firefox which were observed throughout our experiments. Vladan also blogged about it here and more recently here.

However, the bugs which we filed usually relate to specific cases or issues, but don’t expose non-issues, i.e. cases where Firefox is similar or better than other browsers, and they also don’t expose the scope of the experiments and the big picture in order to better assess the weight of the existing issues, nor do they expose general observations which were made.

To address this, we created a content-performance observations page. This page summarizes all the experiments which we performed, including their scope, procedures and observations, for both Desktop Firefox and fennec.

Let us know if you have any feedback on existing experiments and results, or suggestions for more experiments on either Desktop or Android, especially if there are no existing bugs where such discussion could take place.

http://avih.github.com/blog/2015/10/23/fennec-vs-chrome-scrolling-and-navigation-contentperf/

Design patterns are "general reusable solutions to a commonly occurring problem within a given context in software design". Design patterns are a great way to describe some of the culture and 'tribal knowledge' of programming in a language. Design patterns are very language-specific - what is a pattern in one language may be unnecessary in another due to a language feature, or impossible to express due to a missing feature.

If overused, design patterns can add unnecessary complexity to programs (e.g., see AbstractSingletonProxyFactoryBean). However, I think they are a great way to share intermediate and advanced level knowledge about a programming language.

So for these reasons, I have been thinking about design patterns in Rust, and in particular those that are unique to Rust or more common in Rust than other languages. I have started a pattern catalogue for Rust. It covers idioms (small, simpler design patterns), design patterns, and anti-patterns (patterns which you should strive to avoid). There's not a lot there at the moment. I hope to improve it over the next few months. I would love some help with that - if you like writing documentation and fancy describing some design patterns, please send a PR! There is a list of design patterns which need a description in the contents. If you know of other design patterns in Rust and want to add them, or want to improve some of the existing descriptions, that would be awesome.

I'll also be giving a talk about design patterns in Rust at the NOOL workshop at SPLASH/OOPSLA next week in Pittsburgh. If you're at OOPSLA, please come along!

http://www.ncameron.org/blog/design-patterns-in-rust/

http://danielpocock.com/new-planet-sites-for-sip

for the next couple of quarters (at least) i’ll be shifting my attention full time from bugzilla to mozreview. this switch involves a change of language, frameworks, and of course teams. i’m looking forward to new challenges.

one of the first things i’ve done is sketch out a high level architectural diagram of mozreview and its prime dependencies:

mozreview exists as an extension to reviewboard, using bugzilla for user authentication, ldap to check commit levels, with autoland pushing commits automatically to try (and to mozilla-central soon).  there’s mecurial extensions on both the client and server to make pushing things easer, and there are plans to perform static analysis with bots.

https://globau.wordpress.com/2015/10/23/moving-from-bugzilla-to-mozreview/

Zweiw"ochentliches Meeting der deutschsprachigen Community / German speaking community bi-weekly meeting

https://air.mozilla.org/german-speaking-community-bi-weekly-meeting-20151022/

As mentioned in my previous post on MozReview, one of the biggest sources of confusion is the way we present the “squashed” diffs, that is, the diff that show all of the changes in a commit series, the sum of all the proposed changes. We also refer to these as “parent” review requests, since they function as something to hold all the commits together. They are stored in MozReview as separate review requests, similar to the individual commits.

The confusion results from several things:

• The links to the parent are confusing: they are currently labelled “Complete diff” and “Review summary”. “Complete diff” doesn’t clearly indicate that it is the complete diff of all commits together, and “Review summary” is almost totally meaningless, since it doesn’t include all the reviews left on the commits themselves—only reviews left on the overview diff.

• There is nothing in the UI that clearly indicates that you are viewing the squashed diff. The only indication is that none of the rows in the commit table are highlighted. This is particularly confusing when there is only one commit, since the squashed diff is identical to the commit diff.

• You can leave reviews on the squashed diff, but you can’t leave a “ship it”. This is because we are enforcing reviewers to review individual commits. However, because there isn’t much to distinguish parent review requests from commit review requests, it can look like the review dialog is just broken.

There are a few simple things we can do to fix these problems: use better link names, put a big “This is an overview of the commit series” message, and/or put a warning “You must review individual commits” on the review dialog. But really, we need to step back and think about the way we present the squashed diffs, and if they even make sense as a concept in MozReview.

To reiterate, squashed diffs provide a complete view of a whole commit series. The concept of a commit series doesn’t exist in core Review Board (nor does it exist in many other code-review tools), but it’s central to the idea of the repository-centric approach (like in GitHub pull requests). We added this concept by storing metadata resulting from pushes to tie commit series together with a parent, and we added UI elements like the commits table.

There are three broad ways we can deal with squashed diffs going forward. We need to settle on one and make the associated UI changes to make our model clear to users.

1. Remove squashed diffs altogether.

   This is the simplest option. Squashed diffs aren’t actually technically necessary, and they can distract reviewers from the individual commits, which is where they should be spending most of their time, since, in most cases, this is how the code will be landing in the main repository. Some other repository-centric review tools, like Critic, don’t have the concept of an overview diff, so there are precedents. However, it might be a bit heavy handed to tell reviewers that they can’t view all the commits as a single diff (at least, without pulling them down locally).

2. Continue to allow reviews, of some sort, on squashed diffs.

   This is what we have now: reviewers can leave reviews (at the moment, comments only) on squashed diffs. If we decide we want to continue to allow users to leave reviews on squashed diffs, we’ll need to both figure out a better UI to distinguish them from the individual commits and also settle several open questions:

   • Should reviewers be able to grant ship its (i.e. r+s) on squashed diffs? This would imply that the commits probably haven’t been reviewed individually, which would defeat the purpose of a commit-centric system. That said, reviewer time is very important, so we could have a trade off to support more work flows.

   • Conversely, should reviewers be able to leave comments on the parent diff? For simplicity, we could allow reviewers to leave a “ship it” review on a squashed diff that would apply to all commits but force them to leave any comments on diffs on the commits themselves. This would essentially remove the ability to review squashed diffs themselves but would leave the convenience of saying “this is all good”.

   • If we do want to allow review comments on squashed diffs, how should they be consolidated with the reviews on individual commits? Right now, reviews (general comments and comments on diffs) for the squashed diff and all commits are all on separate pages/views. Giving one view into all activity on a commit series would be ideal if we want to support squashed-diff reviews. Arguably, this would be valuable even if we didn’t have reviews on squashed diffs.

   For comparison, GitHub pull requests support this model. There are three tabs in a pull request: “Files changed”, which is the squashed diff; “Commits”, which is a list of commits with links to the individual commit diffs; and “Conversation”, which shows comments on the commits and on the squashed diff (along with other events like updates to the commits). The way they are presented is a little confusing (comments on the squashed diff are just labelled “ commented on the diff”, whereas comments on the diffs are of the form “ commented on in ”), but it is a useful single view. However, note that pull requests do not have the concept of a “ship it” or “r+”, which makes the GitHub interface simpler.

   This approach would support multiple reviewer work flows, but it is also the most complicated, both in terms of UX and technical implementation, and it waters down the philosophy behind MozReview.

3. Provide read-only overview diffs.

   The third approach is to keep squashed diffs but make them read only. They could be used as reference, to get a big picture of the whole series, but since they are read only, they would be easily distinguishable from commits and would force reviewers to look at the individual commits. This is really just option 1 above, with a reference view of the whole series. It would be more work than option 1 but less than option 2, and would preserve the philosophy.

The MozReview team has been leaning towards option 3. We have a mock-up that strips away a lot of the UI that would be useless in this scenario and makes the intention clear. It’s not the prettiest, but it wouldn’t take too much work to get here:

However, we’d like to hear user feedback before making any decisions. Whichever option we go with, we’ll come up with a plan to get there that ideally will have incremental improvements, depending on the complexity of the full solution, so that we can start to fix things right away.

https://mrcote.info/blog/2015/10/22/parental-issues/

This is our weekly gathering of Mozilla'a Web QA team filled with discussion on our current and future projects, ideas, demos, and fun facts.

https://air.mozilla.org/web-qa-weekly-meeting-20151022/

When building products used by millions, how can we go beyond the anecdotal to extrapolate to the macro? What ways can we better understand the...

https://air.mozilla.org/october-2015-brantina-data-as-emapthy-with-frances-haugen/

This is a weekly call with some of the Reps council members to discuss all matters Reps, share best practices and invite Reps to share...

https://air.mozilla.org/reps-weekly-20151022/

I finally got around to releasing rr 4.0!

rr is Mozilla's low overhead record-and-replay debugging tool for large C++ applications like Firefox. For more information see the rr project page, or watch my Technion talk:

rr 4.0 is the first stable release with reverse-execution enabled. This means gdb's reverse-continue, reverse-step, reverse-next and reverse-finish commands work under rr. Not only do they work, but under rr they're quite efficient and have unlimited scope. It's completely reasonable to reverse-continue from the end of a Firefox run all the way back to the beginning. Furthermore, breakpoints and hardware data watchpoints work with reverse execution. Suppose we're debugging Firefox layout:

Breakpoint 1, nsCanvasFrame::BuildDisplayList (this=0x2aaadd7dbeb0, aBuilder=0x7fffffffaaa0, aDirtyRect=..., aLists=...)
    at /home/roc/mozilla-inbound/layout/generic/nsCanvasFrame.cpp:460
460   if (GetPrevInFlow()) {
(gdp) p mRect.width
12000

(gdb) watch -l mRect.width
(gdb) reverse-cont
Continuing.
Hardware watchpoint 2: -location mRect.width
Old value = 12000
New value = 11220
0x00002aaab100c0fd in nsIFrame::SetRect (this=0x2aaadd7dbeb0, aRect=...)
    at /home/roc/mozilla-inbound/layout/base/../generic/nsIFrame.h:718
718       mRect = aRect;

This release is about as stable as rr has ever been --- meaning it seems to be generally working for people and very few new issues have been coming in. However, rr depends on gnarly details of the kernel, system libraries and CPU so I expect as the number of rr users grows we'll need to keep fixing bugs and updating rr regularly.

By the way, Mozilla is interested in hiring people to work on rr-related technology, more closely integrated with the browser. I'd like to hear from people who're interested!

http://robert.ocallahan.org/2015/10/rr-40-released-with-reverse-execution.html

Today is a great day , I’ve finally got my onboarding swag. Been with the company since 2011 (Before that it was momo).

Whomever did this : THANKS

http://sietch-tabr.tumblr.com/post/131680656682

Mercurial 3.6 (scheduled for release on or shortly after November 1) contains a number of improvements to cloning. In this post, I will describe a new feature to help server operators reduce load (while enabling clients to clone faster) and some performance work to make clone operations faster on the client.

Cloning repositories can incur a lot of load on servers. For mozilla-central (the main Firefox repository), clones require the server to spend 4+ minutes CPU time and send ~1,230 MB over the network. Multiply by thousands of clients from build and test automation and developers, and you could quickly finding yourself running out of CPU cores or network bandwidth. Scaling Mercurial servers (like many services) can therefore be challenging. (It's worth noting that Git is in the same boat for reasons technically similar to Mercurial's.)

Mozilla previously implemented a Mercurial extension to seed clones from pre-generated bundle files so the Mercurial servers themselves don't have to work very hard for an individual clone. (That linked post goes into the technical reasons why cloning is expensive). We now offload cloning of frequently cloned repositories on hg.mozilla.org to Amazon S3 and a CDN and are diverting 1+ TB/day and countless hours of CPU work away from the hg.mozilla.org servers themselves.

The positive impact from seeding clones from pre-generated, externally-hosted bundles has been immense. Load on hg.mozilla.org dropped off a cliff. Clone times on clients became a lot faster (mainly because they aren't waiting for a server to dynamically generate/stream bits). But there was a problem with this approach: it required the cooperation of clients to install an extension in order for clone load to be offloaded. It didn't just work.

I'm pleased to announce that the ability to seed clones from server-advertised pre-generated bundles is now a core feature in Mercurial 3.6! Server operators can install the clonebundles extension (it is distributed with Mercurial) to advertise the location of pre-generated, externally-hosted bundle files. Compatible clients will automatically clone from the server-advertised URLs instead of creating potentially excessive load on the Mercurial server. The implementation is almost identical to what Mozilla has deployed with great success. If you operate a Mercurial server that needs to serve larger repositories (100+ MB) and/or is under high load, you should be jumping with joy at the existence of this feature, as it should make scaling problems attached to cloning mostly go away.

Documentation for server operators is currently in the extension and can be accessed at the aforementioned URL or with hg help -e clonebundles. It does require a bit of setup work. But if you are at the scale where you could benefit from the feature, the results will almost certainly be worth it.

One caveat is that the feature is currently behind an experimental flag on the client. This means that it doesn't just work yet. This is because we want to reserve the right to change some behaviors without worrying about backwards compatibility. However, I'm pretty confident the server parts won't change significantly. Or if they do, I'm pretty committed to providing an easy transition path since I'll need one for hg.mozilla.org. So, I'm giving server operators a tentative green light to deploy this extension. I can't guarantee there won't be a few bumps transitioning to a future release. But it shouldn't be a break-the-world type of problem. It is my intent to remove the experimental flag and have the feature enabled by default in Mercurial 3.7. At that point, server operators just need clients to run a modern Mercurial release and they can count on drastically reduced load from cloning.

To help with adoption and testing of the clone bundles feature, servers advertising bundles will inform compatible clients of the existence of the feature when they clone:

$ hg clone https://hg.mozilla.org/mozilla-central
requesting all changes
remote: this server supports the experimental "clone bundles" feature that should enable faster and more reliable cloning
remote: help test it by setting the "experimental.clonebundles" config flag to "true"
adding changesets
adding manifests
adding file changes
...

And if you have the feature enabled, you'll see something like:

$ hg clone https://hg.mozilla.org/mozilla-central
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-central/daa7d98525e859d32a3b3e97101e129a897192a1.gzip.hg
adding changesets
adding manifests
adding file changes
added 265986 changesets with 1501210 changes to 223996 files
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

This new clone bundles feature is deployed on hg.mozilla.org. Users of Mercurial 3.6 can start using it today by cloning from one of the repositories with bundles enabled. (If you have previously installed the bundleclone extension, please be sure your version-control-tools repository is up to date, as the extension was recently changed to better interact with the official feature.)

And that's the clone bundles feature. I hope you are as excited about it as I am!

Mercurial 3.6 also contains numerous performance improvements that make cloning faster, regardless of whether you are using clone bundles! For starters:

• Caching just-added entries made changelog writing 25% faster.
• Reusing file handles when adding revlog entries drastically reduced the number of file opens, closes, and writes.
• Avoiding excessive file flushing when adding revlog entries drastically reduced system call count.

These performance enhancements will make all operations that write new repository data faster. But it will be felt most on clone and pull operations on the client and push operations on the server.

One of the most impressive performance optimizations was to a Python class that converts a generator of raw data chunks to something that resembles a file object so it can be read() from. Refactoring read() to avoid collections.deque operations and an extra string slice and allocation made unbundle operations 15-20% faster. Since this function can handle hundreds of megabytes or even gigabytes of data across hundreds of thousands of calls, small improvements like this can make a huge difference! This patch was a stark reminder that function calls, collection mutations, string slicing, and object allocation all can have a significant cost in a higher-level, garbage collected language like Python.

The end result of all this performance optimization on applying a mozilla-central gzip bundle on Linux on an i7-6700K:

• 35-40s wall time faster (~245s to ~205s) (~84% of original)
• write(2) calls reduced from 1,372,411 to 679,045 (~49% of original)
• close(2) calls reduced from 405,147 to 235,039 (~58% of original)
• total system calls reduced from 5,120,893 to 2,938,479 (~57% of original)

And the same operation on Windows 10 on the same machine:

• ~300s wall time faster (933s to 633s) (~68% of original)

You may have noticed the discrepancy between Linux and Windows wall times, where Windows is 2-4x slower than Linux. What gives? The reason is closing file handles that have been appended to is slow on Windows. For more, read my recent blog post.

Mercurial writes ~226,000 files during a clone of mozilla-central (excluding the working copy). Assuming 2ms per file close operation, that comes out to ~450s just for file close operations! (All operations are on the same thread.) The current wall time difference between clone times on Windows and Linux is ~428s. So it's fair to say that waiting on file closes accounts for most of this.

Along the same vein, the aforementioned performance work reduced total number of file close operations during a mozilla-central clone by ~165,000. Again assuming 2ms per file close, that comes to ~330s, which is in the same ballpark as the ~300s wall time decrease we see on Windows in Mercurial 3.6. Writing - and therefore closing - hundreds of thousands of files handles is slower on Windows and accounts for most of the performance difference on that platform.

Empowered by this knowledge, I wrote some patches to move file closing to a background thread on Windows. The results were promising (minutes saved when writing 100,000+ files). Unfortunately, I didn't have time to finish these patches for Mercurial 3.6. Hopefully they'll make it into 3.7. I also have some mad scientist ideas for alternate storage mechanisms that don't rely on hundreds of thousands of files. This should enable clones to run at 100+ MB/s on all platforms - basically as fast as your network and system I/O can keep up (yes, Python and Windows are capable of this throughput). Stay tuned.

And that's a summary of the cloning improvements in Mercurial 3.6!

Mercurial 3.6 is currently in release candidate. Please help test it by downloading the RC at https://www.mercurial-scm.org/. Mercurial 3.6 final is due for release on or shortly after November 1. There is a large gathering of Mercurial contributors in London this weekend. So if a bug is reported, I can pretty much guarantee a lot of eyeballs will see it and there's a good chance it will be acted upon.

http://gregoryszorc.com/blog/2015/10/22/cloning-improvements-in-mercurial-3.6