As tradition dictates, I do a “the state of curl” presentation every year. This year, as there’s no physical curl up conference happening, I have recorded the full presentation on my own in my solitude in my home.

This is an in-depth look into the curl project and where it’s at right now. The presentation is 1 hour 53 minutes.

The slides: https://www.slideshare.net/bagder/the-state-of-curl-2020

https://daniel.haxx.se/blog/2020/04/30/the-state-of-curl-2020/

Digital contact tracing apps have emerged in recent weeks as one potential tool in a suite of solutions that would allow countries around the world to respond to the COVID-19 pandemic and get people back to their daily lives. These apps raise a number of challenging privacy issues and have been subject to extensive technical analysis and argument. One important question that policymakers are grappling with is whether they should pursue more centralized designs that share contact information with a central authority, or decentralized ones that leave contact information on people’s devices and out of the reach of governments and companies.

Firefox Chief Technology Officer Eric Rescorla has an excellent overview of these competing design approaches, with their different potential risks and benefits. One critical insight he provides is that there is no Silicon Valley wizardry that will easily solve our problems. These different designs present us with different trade-offs and policy choices.

In this post, we want to provide a direct answer to one policy choice: Our view is that centralized designs present serious risk and should be disfavored. While decentralized systems present concerns of their own, their privacy properties are generally superior in situations where governments have chosen to deploy contact tracing apps.

Should your government have the social graph?

Centralized designs share data directly with public health professionals that may aid in their manual contact tracing efforts, providing a tool to identify and reach out to other potentially infected people. That is a key benefit identified by the designers of the BlueTrace system in use in Singapore. The biggest problem with this approach, as described recently by a number of leading technologists, is that it would expand government access to the “social graph” — data about you, your relationships, and your links with others.

The scope of this risk will depend on the details of specific proposals. Does the data include your location? Is it linked to phone numbers or emails? Is app usage voluntary or compulsory? A number of proposals only share your contact list when you are infected, and, if the infection rate is low, then access to the social graph will be more limited. But regardless of the particulars, we know this social graph data is near impossible to truly anonymize. It will provide information about you that is highly sensitive, and can easily be abused for a host of unintended purposes.

Social graph data could be used to see the contacts of political dissidents, for criminal investigations, or for immigration enforcement, to give just a few examples. This isn’t just about risk to personal privacy. Governments, in partnership with the private sector, could use this data to target or discriminate against particular segments of society.

Recently, many have pointed to well-established privacy principles as important tools that can mitigate privacy risk created by contact tracing apps. These include data minimization, rules governing data access and use, strict retention limits, and sunsetting of technical solutions when they are no longer needed. These are principles that Mozilla has long advocated for, and they may have important applications to contact tracing systems.

These protections are not strong enough, however, to prevent the potential abuse of data in centralized systems. Even minimized data is inherently sensitive because the government needs to know who tested positive, and who their contacts are. Recent history has shown that this kind of data, once collected, creates a tempting target for new uses — and for attackers if not kept securely. Neither governments nor the private sector have shown themselves up to the task of policing these new uses. The incentives to put data to unintended uses are simply too strong, so privacy principles don’t provide enough protection.

Moreover, as Mozilla Executive Director Mark Surman observes, the norms we establish today will live far beyond any particular app. This is an opportunity to establish the precedent that privacy is not optional. Centralized contact tracing apps threaten to do the opposite, normalizing systems to track citizens at scale. The technology we build today will likely live on. But even if it doesn’t, the decisions we make today will have repercussions beyond our current crisis and after we’ve sunset any particular app.

At Mozilla, we know about the pitfalls of expansive data collection. We are not experts in public health. In this moment of crisis, we need to take our cue from public health professionals about the problems they need to solve. But we also want policymakers, and the developers building these tools, to be mindful of the full costs of the solutions before them.

Trust is an essential part of helping people to take the steps needed to combat the pandemic. Centralized designs that provide contact information to central authorities are more likely to create privacy and security issues over time, and more likely to erode that trust. On balance we believe decentralized contact tracing apps, designed with privacy in mind, offer a better tool to solve real public health problems and establish a trusted relationship with the technology our lives may depend on.

The post Contact Tracing, Governments, and Data appeared first on The Mozilla Blog.

https://blog.mozilla.org/blog/2020/04/29/contact-tracing-governments-and-data/

https://blog.mozilla.org/blog/2020/04/29/designs-contact-tracing-apps/

Seven years ago, on April 29th 2013, I walked into the old Castro Street Mozilla headquarters in Mountain View for my week of onboarding and orientation. Jubilant and full of imposter syndrom, that day marked the start of a whole new era in my professional career.

I'm not going to spend an entire post reminiscing about the good ol' days (though those days were good indeed). Instead, I thought it might be useful to share a few things that I've learned over the last seven years, as I went from senior engineer to senior manager.

Be brief

Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte.

- Blaise Pascal

Pascal's famous quote - If I had more time, I would have written a shorter letter - is strong advice. One of the best way to disrupt any discussion or debate is indeed to be overly verbose, to extend your commentary into infinity, and to bore people to death.

Here's the thing: nobody cares about the history of the universe. Be brief. If someone asks for someone specific, give them that information, and perhaps mention that there's more to be said about it. They'll let you know if they are interested in hearing the backstory.

People like to hear themselves talk. I certainly do. But over time I realized that, by being brief, I get a lot more attention from people, so they ask more questions, reach out more often, because they know I won't drag them into a lengthy debate.

But don't fall into the extreme opposite. Being overly brief can be detrimental to a conversation, or make you appear like you don't care to participate. The is a right balance to be found between too short and too long, depending on the context and your audience.

Tell a good story

Or more accurately, tell a story that touches your audience directly. If you're working on a security review of a nodejs application hosted in heroku, tell them stories of other nodejs applications hosted in heroku, ideally from a nearby team or organization. Don't go lecture them on the need for network security monitoring in datacenters, they simply won't care, as it doesn't touch them.

If you want to get people interested in what you're doing, first you need to take interest into what they are doing, then you need to tell them a good story they will care about. Finding out what that is will increase your chances of success.

That story will also change depending on your audience. Engineers will care about one thing, their managers something else, and the executives another thing entirely. Emphasize the parts of your project or idea that your audience will be most interested in to catch their attention, without losing the nature of your work.

This isn't rocket science. In fact, it's old school business playbook. Dale Carnegie's 1936 "How to Win Friends & Influence People" covers this at length. And while I certainly wouldn't take that book to the letter, it raises a number of points which I think are relevant to security professionals.

Be technical

I started out at Mozilla as a senior security engineer focused entirely on operations and infrastructure. I spent my days doing security reviews, making guides and writing code. 100% technical work. When I got promoted to staff, then to manager, then to senior manager, the proportion of technical work gradually reduced to make room for managerial work.

Managing is important. With 9-or-so people on the team, being able to accurately focus attention on the right set of problems is critical to the security of the perimeter. In fact, there's an entire school of thought that advocates that managers should be entirely focused on management tasks, and stay away from technical work.

For better or worse, I don't buy into that. I believe, for myself and for my team, that I'm a better team manager and security strategist when I have a deep technical understanding of the issues at hand.

That doesn't mean non-technical managers are bad. In fact, I think there are many situations where a non-technical manager is a better choice than a technical one. But for the field of operations security, at Mozilla, managing the people I manage, I think being technical is a strength.

How do you remain technical while being a manager? There are certainly areas in which I don't have a deep technical understanding and struggle to acquire one. But in general, I find that experimenting outside core projects, and picking up tasks outside the critical path, helps remain current and relevant.

For example, if the organization decide to switch to writing web applications in Rust with Actix, I'll write one myself. I won't get to the level of expertise I have in other areas, but I'll know enough to be relevant during security reviews and threat modeling sessions. And I continue to acquire knowledge in my areas of specialty: cloud infrastructure, cryptographic services, etc.

I don't expect to leave the management track any time soon. In fact, I expect to continue to grow in it. But I find it important that I could go back to a senior staff engineer role if I wanted to. Perhaps it is hubris, time will tell.

Never make assumptions

A year ago, I spent a night seated on a small table outside the reception of the Wahweap campground in Lake Powell, Arizona, as I was helping my team re-issue an intermediate certificate used to sign Firefox add-ons. It was freezing outside. I caught a cold, and a strong lesson, as dozens of my peers where untangling a mess I had helped create.

We called this incident "Armagaddon" internally, and it all started because we made a few assumptions we never took time to verify. We assumed that certificate expiration checking was disabled when verifying add-ons signatures, when in fact it was only disabled for end-entities. When the intermediate expired, everything blew up.

I learned that lesson. I also learned to identify and question every assumptions that we, engineers, make. The more complex a system becomes - and the Firefox ecosystem certainly is a complex one - the more assumptions people make. Learning to identify and consistently call out those assumption, forcing myself and others to verify them, and basing decisions on hard data and tests is critically important.

There is a place and a time where assumptions can be made and risks be taken, but not always, and certainly not on mission critical components. As an industry, we've bought into the "go fast and break things" mindset that is plain wrong for a lot of environments. Learning to slow down and taking the time to verify assumptions is, perhaps, the biggest cultural change we need.

Don't ignore backward compatibility

You think differently about engineering when you have to maintain compatibility with devices and software that haven't been updated in one, and sometime two, decades. Firefox falls into that category. Every time we try to change something that's been around a while, we run into backward compatibility issues.

Up until recently, we maintained a separate set of HTTPS endpoints that supported SSL3 and SHA-1 certificates, issued by decomissioned roots, to allow XP SP2 users to download Firefox installers. And when I say recently, I mean one or two years ago. Long after Microsoft had stopped supporting those users.

I have tons of examples of having to maintain weird configurations and infrastructure for deprecated users no one wants to think about. Yet, they exists, and often represent a sizeable portion of our users that cannot simply be ignored.

As a system designer, learning to account for backward compatibility is a learning curve. It's certainly much easier to greenfield a process while ignoring its history than to design a monster that needs to adopt modern techniques while serving old clients.

Some folks are better at this than others, and this is where you really feel the importance of experience and the value of tenured employees. Those people who jump ship every 18 months? They can't tell you a thing about backward compatibility. But that engineer who's been maintaining a critical system for the past 5 or 10 years absolutely can. Seek them, ask for the history of things, it's always interesting to hear!

Use boring tech

Nobody ever got fired for building a site in Python with Django and Postgresql. Or perhaps you'd like to keep using PHP? Maybe even Perl? The cool kids at the local hackathon will make fun of you for not using the latest javascript framework or Rust nightly, but your security team will probably love you for it.

The thing is, in 99% of cases, you'll be an order of magnitude more productive and secure with boring tech. There are very few cases where the bleeding edge will actually give you an edge.

For example, I'm a big fan of Rust. And not because not being a big fan of Rust at Mozilla is a severe faux-pas. But because I think the language team is doing a great job of distilling programming best practices into a reasonable set of engineering principles. Yet, I absolutely do not recommend anyone to write backend services in Rust, unless they are ready to deal with a lot of pain. Things like web frameworks, ORMs, cloud SDKs, migration frameworks, unit testing, and so on all exist in Rust, but are nowhere as mature or tested as their Python equivalents.

My favorite stack to build a quick prototype of a web service is Python, Flask, Postgresql and Heroku. That's it. All stuff that's been around for over a decade and that no one considers new or cool.

Bugzilla is written in Perl. Phabricator or Pocket are PHP. ZAP is Java. etc. There are tons of examples of software that is widely successful by using boring tech, because their developers are so much more productive on those stacks than they would be on anything bleeding edge.

And from a security perspective, those boring techs have acquired a level of maturity that can only be attained by walking the walk. Sure, programming languages can and do prevent entire classes of vulnerabilities, but not all of them, and using Rust won't stop SQL injections or SSRF.

So when should you not use boring tech? Mostly when you have time and money. If you're flush on cash and you unique problems, taking six months or a year to ramp up a new tech is worthwhile. The ideal time to do it is when you're rewriting a well-established service, and have a solid test suite to verify the rewrite is equivalent to the original. That's what the Durable Sync team did at Mozilla, when they rewrote the Firefox Sync backend from Python/Mysql to Rust/Spanner.

Delete your code

My first two years at Mozilla were focused on an endpoint security project called MIG, for Mozilla Investigator. It was an exciting greenfield project and I got to use all the cool stuff: Go, MongoDB (yurk), RabbitMQ (meh), etc. I wrote tons and tons of code, shipped a fully functional architecture, got a logo from a designer friend, gave a dozen conference talks, even started a small open source community around it.

And then I switched gears.

In the space of maybe 6 months, I completely stopped working on MIG to focus on cloud services security. At first, it was hard to let go of a project I had invested so much into. Then, gradually, it faded, until eventually the project died off and got archived. My code was effectively deleted. Two years of work out the window. This isn't something you're trained to deal with. And in fact, most engineers, like I was, are overly attached to their code, to the point of aggressively fighting any change they disagree with.

If this is you, stop, right now, you're not doing anyone any favors.

Your code will be deleted. Your projects will be cancelled. People are going to take over your work and transform it entirely, and you won't even have a say in it. This is OK. That's the way we move forward and progress. Learning to cope with that feeling early on will help you later in your career.

Nowadays, when someone is about to touch code I have previously written, I explicitely welcome them to rip out anything they think is worth removing. I give them a clear signal that I'm not attached to my code. I'd much rather see the project thrive than keep around old lines of code. I encourage you to do the same.

Be passionate, and respectful

Security folks generally don't need to be told to be passionate. In fact, they are often told the opposite, that they should tone it down a notch, that they are making too many waves. I disagree with this. I think it's good and useful to an organization to have a passionate security team that truly cares about doing good work. If we're not passionate about security, who else is going to be? That's literally what we're paid to do.

When a team comes ask for a security review about a new project, they expect to receive the full-on adversarial doomsday threat model experience from us. They want to talk to someone who's passionate about breaking and hardening services to the extreme. So this is what we do in our risks assessments meetings. We don't tone it down or play it safe, we push things to the extreme, we're unreasonable and it's fun as hell!

But when folks respectfully disagree with my recommendation to encrypt all databases with HSM-backed AES keys split with a Shamir threshold of 4 and store each fragment in underground bunkers around the world to prevent the NSA from compromising several of our employees to access the data, I remain respectful of their opinion. We have a productive and polite debate that leads to a reasonable middle-ground which sufficiently addresses the risks, and can be implemented within timeline and budget.

Both passion and respect are important quality of a successful engineer (or a person, really). I like to end a day knowing that I've done the right thing and that, even if everything didn't go my way, I have made a good case and I'm satisfied with the outcome.

Onward.

And so this is seven years at Mozilla. A lot more could be said about the work accomplished or the lessons learned, but then this would turn into a book, and I swore to myself I wouldn't write another one of those just yet.

Working at Mozilla is a heck of a job, even in those trying times. The team is fantastic, the work is fascinating, and, as they said during my first week of onboarding back in the old Castro street office, we get to work for Mankind, not for The Man. That's gotta be worth something!

https://j.vehent.org/blog/index.php?post/2020/04/28/7-years-at-Mozilla

We’ve done many curl releases over the years and this 191st one happens to be the 20th release ever done in the month of April, making it the leading release month in the project. (February is the month with the least number of releases with only 11 so far.)

Numbers

the 191st release
4 changes
49 days (total: 8,076)
135 bug fixes (total: 6,073)
262 commits (total: 25,667)
0 new public libcurl function (total: 82)
0 new curl_easy_setopt() option (total: 270)
1 new curl command line option (total: 231)
65 contributors, 36 new (total: 2,169)
40 authors, 19 new (total: 788)
0 security fixes (total: 92)
0 USD paid in Bug Bounties

Security

There’s no security advisory released this time. The release of curl 7.70.0 marks 231 days since the previous CVE regarding curl was announced. The longest CVE-free period in seven years in the project.

Changes

The curl tool got the new command line option --ssl-revoke-best-effort which is powered by the new libcurl bit CURLSSLOPT_REVOKE_BEST_EFFORT you can set in the CURLOPT_SSL_OPTIONS. They tell curl to ignore certificate revocation checks in case of missing or offline distribution points for those SSL backends where such behavior is present (read: Schannel).

curl’s --write-out command line option got support for outputting the meta data as a JSON object.

We’ve introduced the first take on MQTT support. It is marked as experimental and needs to be explicitly enabled at build-time.

Bug-fixes to write home about

This is just an ordinary release cycle worth of fixes. Nothing particularly major but here’s a few I could add some extra blurb about…

gnutls: bump lowest supported version to 3.1.10

GnuTLS has been a supported TLS backend in curl since 2005 and we’ve supported a range of versions over the years. Starting now, we bumped the lowest supported GnuTLS version to 3.1.10 (released in March 2013). The reason we picked this particular version this time is that we landed a bug-fix for GnuTLS that wanted to use a function that was added to GnuTLS in that version. Then instead of making more conditional code, we cleaned up a lot of legacy and simplified the code significantly by simply removing support for everything older than this. I would presume that this shouldn’t hurt many users as I suspect it is a very bad idea to use older versions anyway, for security reasons if nothing else.

libssh: Use new ECDSA key types to check known hosts

curl supports three different SSH backends, and one them is libssh. It turned out that the glue layer we have in curl between the core libcurl and the SSH library lacked proper mappings for some recent key types that have been added to the SSH known_hosts file. This file has been getting new key types added over time that OpenSSH is using by default these days and we need to make sure to keep up…

multi-ssl: reset the SSL backend on Curl_global_cleanup()

curl can get built to support multiple different TLS backends – which lets the application to select which backend to use at startup. Due to an oversight we didn’t properly support that the application can go back and cleanup everything and select a different TLS backend – without having to restart the application. Starting now, it is possible!

Revert “file: on Windows, refuse paths that start with \\”

Back in January 2020 when we released 7.68.0 we announced what we then perceived was a security problem: CVE-2019-15601.

Later, we found out more details and backpedaled on that issue. “It’s not a bug, it’s a feature” as the saying goes. Since it isn’t a bug (anymore) we’ve now also subsequently removed the “fix” that we introduced back then…

tests: introduce preprocessed test cases

This is actually just one out of several changes in the curl test suite that has happened as steps in a larger sub-project: move all test servers away from using fixed port numbers over to using dynamically assigned ones. Using dynamic port numbers makes it easier to run the tests on random users’ machines as the risk for port collisions go away.

Previously, users had the ability to ask the tests to run on different ports by using a command line option but since it was rarely used, new test were often written assuming the default port number hard-coded. With this new concept, such mistakes can’t slip through.

In order to correctly support all test servers running on any port, we’ve enhanced the main test “runner” (runtests) to preprocess the test case files correctly which allows all our test servers to work with such port numbers appearing anywhere in protocol details, headers or response bodies.

The work on switching to dynamic port numbers isn’t quite completed yet but there are still a few servers using fixed ports. I hope those will be addressed within shortly.

tool_operate: fix add_parallel_transfers when more are in queue

Parallel transfers in the curl tool is still a fairly new thing, clearly, as we can get a report on this kind of basic functionality flaw. In this case, you could have curl generate zero byte output files when using --parallel-max to limit the parallelism, instead of getting them all downloaded fine.

version: add ‘cainfo’ and ‘capath’ to version info struct

curl_version_info() in libcurl returns lots of build information from the libcurl that’s running right now. It includes version number of libcurl, enabled features and version info from used 3rd party dependencies. Starting now, assuming you run a new enough libcurl of course, the returned struct also contains information about the built-in CA store default paths that the TLS backends use.

The idea being that your application can easily extract and use this information either in information/debugging purposes but also in cases where other components are used that also want a CA store and the application author wants to make sure both/all use the same paths!

windows: enable UnixSockets with all build toolchains

Due to oversights, several Windows build didn’t enable support for unix domain sockets even when built for such Windows 10 versions where there’s support provided for it in the OS.

scripts: release-notes and copyright

During the release cycle, I regularly update the RELEASE-NOTES file to include recent changes and bug-fixes scheduled to be included in the coming release. I do this so that users can easily see what’s coming; in git, on the web site and in the daily snapshots. This used to be a fairly manual process but the repetitive process finally made me create a perl script for it that removes a lot of the manual work: release-notes.pl. Yeah, I realize I’m probably the only one who’s going to use this script…

Already back in December 2018, our code style tool checksrc got the powers to also verify the copyright year range in the top header (written by Daniel Gustafsson). This makes sure that we don’t forget to bump the copyright years when we update files. But because this was a bit annoying and surprising to pull-request authors on GitHub we disabled it by default – which only lead to lots of mistakes still being landed on the poor suckers (like me) who enabled it would get the errors instead. Additionally, the check is a bit slow. This finally drove me into disabling the check as well.

To combat the net effect of that, I’ve introduced the copyright.pl script which is similar in spirit but instead scans all files in the git repository and verifies that they A) have a header and B) that the copyright range end year seems right. It also has a whitelist for files that don’t need to fulfill these requirements for whatever reason. Now we can run this script one every release cycle instead and get the same end results. Without being annoying to users and without slowing down anyone’s everyday builds! Win-win!

The release presentation video

Credits

The top image was painted by Dirck van Delen 1631. Found in the Swedish National Museum’s collection.

https://daniel.haxx.se/blog/2020/04/29/curl-7-77-0-with-json-and-mqtt/

https://this-week-in-rust.org/blog/2020/04/29/this-week-in-rust-336/

[ 3, 6516407 ],
[ 3, 12856365 ],
[ 3, 12899916 ],
[ 3, 13034426 ],
[ 3, 13581214 ],
[ 3, 13646510 ],
...

[ "firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1" ],
[ "mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1" ],
[ "Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71" ],
[ "xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1" ],
...

0  xul.pdb  mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction)
1  xul.pdb  mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()",
2  xul.pdb  mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray const&)
3  xul.pdb  std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call()
...

pip install symbolic

import symbolic

# This is a Breakpad symbols file I have on disk.
archive = symbolic.Archive.open("XUL/75A79CFA0E783A35810F8ADF2931659A0/XUL.sym")

# We do debug ids as all-uppercase with no hyphens. However, symbolic
# requires that get normalized into the form it likes.
debug_id = symbolic.normalize_debug_id("75A79CFA0E783A35810F8ADF2931659A0")

# This parses the Breakpad symbols file and returns a symcache that we can
# look up addresses in.
obj = archive.get_object(debug_id=ndebug_id)
symcache = obj.make_symcache()

# Symbol lookup returns a list of LineInfo objects.
lineinfos = symcache.lookup(0xf5aa0)

print("line: %s symbol: %s" % (lineinfos[0].line, lineinfos[0].symbol))

https://bluesock.org/~willkg/blog/mozilla/experimenting_with_symbolic.html

On May 7, 2020 I will present common mistakes when using libcurl (and how to fix them) as a webinar over Zoom. The presentation starts at 19:00 Swedish time, meaning 17:00 UTC and 10:00 PDT (US West coast).

[sign up to attend here]

Abstract

libcurl is used in thousands of different applications and devices for client-side Internet transfer and powers a significant part of what flies across the wires of the world a normal day.

Over the years as the lead curl and libcurl developer I’ve answered many questions and I’ve seen every imaginable mistake done. Some of the mistakes seem to happen more frequently and some of the mistake seem easier than others to avoid.

I’m going to go over a list of things that users often get wrong with libcurl, perhaps why they do and of course I will talk about how to fix those errors.

Length

It should be done within 30-40 minutes, plus some additional time for questions at the end.

Audience

You’re interested in Internet transfer, preferably you already know what libcurl is and perhaps you have even written code that uses libcurl. Directly in C or using a binding in another language.

Material

The video and slides will of course be made available as well in case you can’t tune in live.

Sign up

If you sign up to attend, you can join, enjoy the talk and of course ask me whatever is unclear or you think needs clarification around this topic. See you next week!

https://daniel.haxx.se/blog/2020/04/28/webinar-common-libcurl-mistakes/

by Astrid Rivera For the past month or so I’ve been living my life exclusively indoors and online. This new normal means that everything has shifted to a virtual realm … Read more

The post Now that my entire life is online, how do I protect my personal data? appeared first on The Firefox Frontier.

https://blog.mozilla.org/firefox/data-request-ccpa/

Hubs Cloud is a new product offering from the Mozilla Mixed Reality group that allows companies and organizations to create their own private, social spaces that work with desktop, mobile, and VR headsets. Hubs Cloud contains the underlying architecture that runs hubs.mozilla.com, and is being offered in Early Access on AWS. With Hubs Cloud, it is now possible for external organizations to deploy, customize and configure their own unique instances of the Hubs platform.

When we began building Hubs, the plan was not to try and create a singular, shared online experience owned entirely by Mozilla. Instead, we wanted to create a platform that could be adopted and deployed by organizations to suit their own needs and build their own social applications that worked on both 2D and on VR devices. With that in mind, the team set out to build a set of collaboration tools with versatile web frameworks that could lay the foundation of a platform that provides private rooms, customizable avatars, and the power of virtual reality to connect people regardless of whether or not they were in a shared physical location.

Over the past several months, we’ve done a lot of work to provide a way for external development teams to stand up private instances of our Hubs infrastructure on an organizational AWS account. Hubs Cloud instances are compatible with the same avatars and scenes that are published on the main hubs.mozilla.com site, or you can choose to create your own content, unique to your application. Hubs Cloud offers access to an administrator panel that allows you to customize the branding for your site, approve scenes and avatars that are submitted, import default environments and set platform-wide room settings, and use your own domain names. And, since the deployment is done through your organization’s AWS account, you control the account access and data for your individual instances.

We’ve been incredibly excited by the use cases that we’ve seen from our early partners who have deployed Hubs Cloud. Last month, IEEE deployed a custom Hubs Cloud instance to host an online experience for their VR conference, and brought viewing parties, poster sessions, and breakout sessions into shared virtual spaces. Companies have also begun deploying custom instances of Hubs for industry-specific verticals, bringing the power of collaborative 3D computing to their existing workflows in areas such as accident visualization and reconstruction. We’ve also seen explorations in educational initiatives at the K-12 and university levels, and look forward to sharing more about what our partners are working on in the coming months.

Hubs Cloud is available in Personal and Enterprise editions. For both editions, billing is based on hourly metering and the instance sizes used. On the AWS Marketplace page, there is a cost estimation calculator to help estimate these ahead of time, and are dependent on the concurrency expected, uptime for the system, data, and storage costs. Both Personal and Enterprise offer the same platform features, but Personal is configured to use a smaller instance size at lower costs and has limits on system-wide scalability.

In the coming months, we’ll be working on bringing Hubs Cloud to additional providers, with Digital Ocean as our next target platform.

Get started with deploying Hubs Cloud today at hubs.mozilla.com/cloud, check out the documentation, or send us a message at hubs@mozilla.com to learn more.

https://blog.mozvr.com/announcing-hubs-cloud/

The post Which Video Call Apps Can You Trust? appeared first on The Mozilla Blog.

https://blog.mozilla.org/blog/2020/04/28/which-video-call-apps-can-you-trust/

This option only has a long version and it is --remote-name-all.

Shipped curl 7.19.0 for the first time – September 1 2008.

History of curl output options

I’m a great fan of the Unix philosophy for command line tools so for me there was never any deeper thoughts on what curl should do with the contents of the URL it gets already from the beginning: it should send it to stdout by default. Exactly like the command line tool cat does for files.

Of course I also realized that not everyone likes that so we provided the option to save the contents to a given file. Output to a named file. We selected -o for that option – if I remember correctly I think I picked it up from some other tools that used this letter for the same purpose: instead of sending the response body to stdout, save it to this file name.

Okay but when you selected “save as” in a browser, you don’t actually have to select the full name yourself. It’ll propose the default name to use based on the URL you’re viewing, probably because in many cases that makes sense for the user and is a convenient and quick way to get a sensible file name to save the content as.

It wasn’t hard to come with the idea that curl could offer something similar. Since the URL you give to curl has a file name part when you want to get a file name, having a dedicated option pick the name from the rightmost part of the URL for the local file name was easy. As different output option that -o,it felt natural to pick the uppercase O option for this slightly different save-the-output option: -O.

Enter more than URL

curl sends everything to stdout, unless to tell it to direct it somewhere else. Then (this is still before the year 2000, so very early days) we added support for multiple URLs on the command line and what would the command line options mean then?

The default would still be to send data to stdout and since the -o and -O options were about how to save a single URL we simply decided that they do exactly that: they instruct curl how to send a single URL. If you provide multiple URLs to curl, you subsequently need to provide multiple output flags. Easy!

It has the interesting effect that if you download three files from example.com and you want them all named according to their rightmost part from the URL, you need to provide multiple -O options:

curl https://example.com/1 https://example.com/2 https://example.com/3 -O -O -O

Maybe I was a bit sensitive

Back in 2008 at some point, I think I took some critique about this maybe a little too hard and decided that if certain users really wanted to download multiple URLs to local file names in an easier manner, that perhaps other command line internet download tools do, I would provide an option that lets them to this!

--remote-name-all was born.

Specifying this option will make -O the default behavior for URLs on the command line! Now you can provide as many URLs as you like and you don’t need to provide an extra flag for each URL.

Get five different URLs on the command line and save them all locally using the file part form the URLs:

curl --remote-name-all https://example.com/a.html https://example.com/b.html https://example.com/c.html https://example.com/d.html https://example.com/e.html

Then if you don’t want that behavior you need to provide additional -o flags…

.curlrc perhaps?

I think the primary idea was that users who really want -O by default like this would put --remote-name-all in their .curlrc files. I don’t this ever really materialized. I believe this remote name all option is one of the more obscure and least used options in curl’s huge selection of options.

https://daniel.haxx.se/blog/2020/04/27/curl-ootw-remote-name-all/

The Picture-in-Picture feature in the Firefox browser makes multitasking with video content easy, no window shuffling necessary. With Picture-in-Picture, you can play a video in a separate, scalable window that … Read more

The post Try Firefox Picture-in-Picture for multi-tasking with videos appeared first on The Firefox Frontier.

https://blog.mozilla.org/firefox/firefox-picture-in-picture-for-videos/

Again, I say "semi-review" because if I were going to do this right, I'd have set up both the dual-4 and the dual-8 identically, had them do the same tasks and gone back if the results were weird. However, when you're buying a $7000+ workstation you economize where you can, which means I didn't buy any new NVMe cards, bought additional rather than spare RAM, and didn't buy another GPU; the plan was always to consolidate those into the new machine and keep the old chassis, board and CPUs/HSFs as spares. Plus, I moved over the case stickers and those totally change the entire performance characteristics of the system, you dig? We'll let the Phoronix guy(s) do that kind of exacting head-to-head because I pay for this out of pocket and we've all gotta tighten our belts in these days of plague. Here's the new beast sitting beside me under my work area:

(By the way, I'm still taking candidates for #ShowUsYourTalos. If you have pictures uploaded somewhere, I'll rebroadcast them here with your permission and your system's specs. Blackbirds, POWER8s and of course any other OpenPOWER systems welcome. Post in the comments.)

This new "consolidated" system has 64GB of RAM, Raptor's BTO option Radeon WX7100 workstation GPU and two NVMe main drives on the current Talos II 1.01 board, running Fedora 31 as before. In normal usage the dual-8 runs a bit hotter than the dual-4, but this is absolutely par for the course when you've just doubled the number of processors onboard. In my low-Earth-orbit Southern California office the dual-4's fastest fan rarely got above 2100rpm while the dual-8 occasionally spins up to 2300 or 2600rpm. Similarly, give or take system load, the infrared thermometer pegged the dual-4's "exhaust" at around 95 degrees Fahrenheit; the dual-8 puts out about 110 F. However, idle power usage is only about 20W more when sitting around in Firefox and gnome-terminal (130W vs 110W), and the idle fan speeds are about the same such that overall the dual-8 isn't appreciably louder than the very quiet dual-4 was with the most current firmware (with the standard Supermicro fan assemblies, though I replaced the dual-4's PSUs with "super-quiets" a while back and those are in the dual-8 now too).

Naturally the CPUs are the most notable change. Recall that the Sforza "scale out" POWER9 CPUs in Raptor family workstations are SMT-4, i.e., each core offers four hardware threads, which appear as discrete CPUs to the operating system. My dual-4 appeared to be a 32 CPU system to Fedora; this dual-8 appears to have 64. These threads come from "slices," and SMT-4 cores have four which are paired into two "super-slices." They look like this:

Each slice has a vector-scalar unit and an address generator feeding a load-store unit. The VSU has 64-bit integer, floating point and vector ALU components; two slices are needed to get the full 128-bit width of a VMX vector, hence their pairing as super-slices. The super-slices do not have L1 cache of their own, nor do they handle branch instructions or branch prediction; all of that is per-core, which also does instruction fetch and dispatch to the slices. (This has some similar strengths and pitfalls to AMD Ryzen, for example, which also has a single branch unit and caches per core, but the Ryzen execution units are not organized in the same fashion.) The upshot of all this is that certain parallel jobs, especially those that may be competing for scarcer per-core resources like L1 cache or the branch unit, may benefit more from full cores than from threads and this is true of pretty much any SMT implementation. In a like fashion, since each POWER9 slice is not a full vector unit (only the super-slices are), heavy use of VMX would soak up to twice the execution resources though amortized over the greater efficiency vector code would offer over scalar.

The biggest task I routinely do on my T2 are frequent smoke-test builds of Firefox to make sure OpenPOWER-specific bugs are found before they get to release. This was, in fact, where I hoped I would see the most improvement. Indeed, a fair bit of it can be run parallel, so if any of my typical workloads would show benefit, I felt it would likely be this one. Before I tore down the dual-4 I put all 64GB of RAM in it for a final time run to eliminate memory pressure as a variable (and the same sticks are in the dual-8, so it's exactly the same RAM in exactly the same slot layout). These snapshots were done building the Firefox 75 source code from mozilla-release (current as of this writing) with my standard optimized .mozconfig, varying only in the number of jobs specified. I'm only reporting wall time here because frankly that's the only thing I personally cared about. All build runs were done at the text console before booting X and GNOME to further eliminate variability, and I did three runs of each configuration back to back (./mach clobber && ./mach build) to account for any caching that might have occurred. Power was measured at the UPS. Default Spectre and Meltdown mitigations were in effect.

Dual-4 (-j24)

32:22.65
31:19.17
30:49.66
average draw 170W

Dual-8 (-j48)

19:16.28
19:09.18
19:08.32
average draw 230W

Dual-8 (-j24)

19:16.46
19:13.78
19:10.14
average draw 230W

The dual-8 is approximately 40% faster than the dual-4 on this task (or, said another way, the dual-4 was about 1.6x slower), but doubling the number of make processes from my prior configuration didn't seem to yield any improvement despite being well within the 64 threads available. This surprised me, so given that the dual-8 has 16 cores, I tried 16 processes directly:

Dual-8 (-j16)

21:49.72
21:40.18
21:41.33
average draw 215W

This proves, at least for this workload, that SMT does make some difference, just not as much as I would have thought. It also argues that the sweet spot for the dual-4 might have been around -j12, but I'm not willing to tear this box back down to try it. Still, cutting down my build times by over 10 minutes is nothing to sneeze at.

For other kinds of uses, though, I didn't see a lot different in terms of performance between DD2.2 and DD2.3 and to be honest you wouldn't expect to. DD2.3 does have improved Spectre mitigations and this would help the kind of branch-heavy code that would benefit least from additional slices, but the change is relatively minor and the difference in practise indeed seemed to be minimal. On my JIT-accelerated DOSBox build the benchmarks came in nearly exactly the same, as did QEMU running Mac OS 9. Booted into GNOME as I am right now, the extra CPU resources certainly do smooth out doing more things at once, but again, that's of course more a factor of the number of cores and slices than the processor stepping.

Overall I'm pretty pleased with the upgrade, and it's a nice, welcome boost that improves my efficiency further. Here are my present observations if you're thinking about a CPU upgrade too (or are a first time buyer considering how much you should get):

• Upgrading is pretty easy with these machines: if you bought too little today, you can always drop in a beefier CPU or two tomorrow (assuming you have the dough and the sockets), and the Self-Boot Engine code is generic such that any Sforza POWER9 chip will work on any Raptor system that can accommodate it. I have been repeatedly assured no FPGA update is needed to use DD2.3 chips, even for "O.G." T2 systems. However, if you're running a Blackbird, you should think about case and cooling as well because the 8-core will run noticeably hotter than a 4-core. A lot of the more attractive slimline mATX cases are very thermally constrained, and the 8-core CPU really should be paired with the (3U) high speed fan assembly than the (2U) passive heatsink. This is a big reason why my personal HTPC Blackbird is still a little 4-core system.

• The 4 and 8-core chips are familiar to most OpenPOWER denizens but the 18 and 22-core monsters have a complicated value proposition. People have certainly run them in Blackbirds; my personal thought is this is courting an early system demise, though I am impressed by how heroic some of those efforts are. I wouldn't myself, however: between the thermal and power demands you're gonna need a bigger boat for these sharks.

  The T2 is designed for them and will work fine, but after my experience here one wonders how loud and hot they would get in warmer environments. Plus, you really need to fill both of those sockets or you'll lose three slots (those serviced by the second CPU's PCIe lanes), which would make them even louder and hotter. The dual-8 setup here gets you 16 cores and all of the slots turned on, so I think it's the better workstation configuration even though it costs a little more than a single-18 and isn't nearly as performant. The dual-18 and dual-22 configurations are really meant for big servers and crazy people.

  With the T2 Lite, though, these CPUs make absolute sense and it would be almost a waste to run one with anything less. The T2 Lite is just a cut-down single-socket T2 board in the same form factor, so it will also easily accommodate any CPU but more cheaply. If you need the massive thread resources of a single-18 (72 thread) or single-22 (88 thread) workstation, and you can make do with an x16 and an x8 slot, it's really the overall best option for those configurations and it's not that much more than a Blackbird board. Plus, being a single CPU configuration it's probably a lot more liveable under one's desk.

• Simply buying a DD2.3 processor to replace a DD2.2 processor of the same core count probably doesn't pay off for most typical tasks. Unless you need the features (and there are some: besides the Spectre mitigations, it also has Ultravisor support and proper hardware watchpoints), you'll just end up spending additional money for little or no observable benefit. However, if you're going to buy more cores at the same time, then you might as well get a DD2.3 chip and have those extra features just in case you'll need them later. The price difference is almost certainly worth a little futureproofing.

https://www.talospace.com/2020/04/eight-four-two-one-twice-cores-is.html

Just a quick note that, as a side-effect of the work I mentioned a while ago to add telemetry to mozregression, mozregression now has a graphical Mac client! It’s a bit of a pain to install (since it’s unsigned), but likely worlds easier for the average person to get going than the command-line version. Please feel free to point people to it if you’re looking to get a regression range for a MacOS-specific problem with Firefox.

More details: The Glean Python SDK, which mozregression now uses for telemetry, requires Python 3. This provided the impetus to port the GUI itself to Python 3 and PySide2 (the modern incarnation of PyQt), which brought with it a much easier installation/development experience for the GUI on platforms like Mac and Linux.

I haven’t gotten around to producing GUI binaries for the Linux yet, but it should not be much work.

Speaking of Glean, mozregression, and Telemetry, stay tuned for more updates on that soon. It’s been an adventure!

https://wlach.github.io/blog/2020/04/mozregression-for-macos/?utm_source=Mozilla&utm_medium=RSS

I last wrote in December 2019 about my work on speeding up the Rust compiler. Time for another update.

Incremental compilation

I started the year by profiling incremental compilation and making several improvements there.

#68914: Incremental compilation pushes a great deal of data through a hash function, called SipHasher128, to determine what code has changed since the last compiler invocation. This PR greatly improved the extraction of bytes from the input byte stream (with a lot of back and forth to ensure it worked on both big-endian and little-endian platforms), giving incremental compilation speed-ups of up to 13% across many benchmarks. It also added a lot more comments to explain what is going on in that code, and removed multiple uses of unsafe.

#69332: This PR reverted the part of #68914 that changed the u8to64_le function in a way that made it simpler but slower. This didn’t have much impact on performance because it’s not a hot function, but I’m glad I caught it in case it gets used more in the future. I also added some explanatory comments so nobody else will make the same mistake I did!

#69050: LEB128 encoding is used extensively within Rust crate metadata. Michael Woerister had previously sped up encoding and decoding in #46919, but there was some fat left. This PR carefully minimized the number of operations in the encoding and decoding loops, almost doubling their speed, and giving wins on many benchmarks of up to 5%. It also removed one use of unsafe. In the PR I wrote a detailed description of the approach I took, covering how I found the potential improvement via profiling, the 18 different things I tried (10 of which improved speed), and the final performance results.

LLVM bitcode

Last year I noticed from profiles that rustc spends some time compressing the LLVM bitcode it produces, especially for debug builds. I tried changing it to not compress the bitcode, and that gave some small speed-ups, but also increased the size of compiled artifacts on disk significantly.

Then Alex Crichton told me something important: the compiler always produces both object code and bitcode for crates. The object code is used when compiling normally, and the bitcode is used when compiling with link-time optimization (LTO), which is rare. A user is only ever doing one or the other, so producing both kinds of code is typically a waste of time and disk space.

In #66598 I tried a simple fix for this: add a new flag to rustc that tells it to omit the LLVM bitcode. Cargo could then use this flag whenever LTO wasn’t being used. After some discussion we decided it was too simplistic, and filed issue #66961 for a more extensive change. That involved getting rid of the use of compressed bitcode by instead storing uncompressed bitcode in a section in the object code (a standard format used by clang), and introducing the flag for Cargo to use to disable the production of bitcode.

The part of rustc that deals with all this was messy. The compiler can produce many different kinds of output: assembly code, object code, LLVM IR, and LLVM bitcode in a couple of possible formats. Some of these outputs are dependent on other outputs, and the choices on what to produce depend on various command line options, as well as details of the particular target platform. The internal state used to track output production relied on many boolean values, and various nonsensical combinations of these boolean values were possible.

When faced with messy code that I need to understand, my standard approach is to start refactoring. I wrote #70289, #70345, and #70384 to clean up code generation, #70297, #70729 , and #71374 to clean up command-line option handling, and #70644 to clean up module configuration. Those changes gave me some familiarity with the code, simplifed it, and I was then able to write #70458 which did the main change.

Meanwhile, Alex Crichton wrote the Cargo support for the new -Cembed-bitcode=no option (and also answered a lot of my questions). Then I fixed rustc-perf so it would use the correct revisions of rustc and Cargo together, without which the the change would erroneously look like a performance regression on CI. Then we went through a full compiler-team approval and final comment period for the new command-line option, and it was ready to land.

Unfortunately, while running the pre-landing tests we discovered that some linkers can’t handle having bitcode in the special section. This problem was only discovered at the last minute because only then are all tests run on all platforms. Oh dear, time for plan B. I ended up writing #71323 which went back to the original, simple approach, with a flag called -Cbitcode-in-rlib=no. (Edit: note that libstd is still compiled with -Cbitcode-in-rlib=yes, which means that libstd rlibs will still work with both LTO and non-LTO builds.)

The end result was one of the bigger performance improvements I have worked on. For debug builds we saw wins on a wide range of benchmarks of up to 18%, and for opt builds we saw wins of up to 4%. The size of rlibs on disk has also shrunk by roughly 15-20%. Thanks to Alex for all the help he gave me on this!

Anybody who invokes rustc directly instead of using Cargo might want to use -Cbitcode-in-rlib=no to get the improvements.

Miscellaneous improvements

#67079: Last year in #64545 I introduced a variant of the shallow_resolved function that was specialized for a hot calling pattern. This PR specialized that function some more, winning up to 2% on a couple of benchmarks.

#67340: This PR shrunk the size of the Nonterminal type from 240 bytes to 40 bytes, reducing the number of memcpy calls (because memcpy is used to copy values larger than 128 bytes), giving wins on a few benchmarks of up to 2%.

#68694: InferCtxt is a type that contained seven different data structures within RefCells. Several hot operations would borrow most or all of the RefCells, one after the other. This PR grouped the seven data structures together under a single RefCell in order to reduce the number of borrows performed, for wins of up to 5%.

#68790: This PR made a couple of small improvements to the merge_from_succ function, giving 1% wins on a couple of benchmarks.

#68848: The compiler’s macro parsing code had a loop that instantiated a large, complex value (of type Parser) on each iteration, but most of those iterations did not modify the value. This PR changed the code so it initializes a single Parser value outside the loop and then uses Cow to avoid cloning it except for the modifying iterations, speeding up the html5ever benchmark by up to 15%. (An aside: I have used Cow several times, and while the concept is straightforward I find the details hard to remember. I have to re-read the documentation each time. Getting the code to work is always fiddly, and I’m never confident I will get it to compile successfully… but once I do it works flawlessly.)

#69256: This PR marked with #[inline] some small hot functions relating to metadata reading and writing, for 1-5% improvements across a number of benchmarks.

#70837: There is a function called find_library_crate that does exactly what its name suggests. It did a lot of repetitive prefix and suffix matching on file names stored as PathBufs. The matching was slow, involving lots of re-parsing of paths within PathBuf methods, because PathBuf isn’t really designed for this kind of thing. This PR pre-emptively extracted the names of the relevant files as strings and stored them alongside the PathBufs, and changed the matching to use those strings instead, giving wins on various benchmarks of up to 3%.

#70876: Cache::predecessors is an oft-called function that produces a vector of vectors, and the inner vectors are usually small. This PR changed the inner vector to a SmallVec for some very small wins of up to 0.5% on various benchmarks.

Other stuff

I added support to rustc-perf for the compiler’s self-profiler. This gives us one more profiling tool to use on the benchmark suite on local machines.

I found that using LLD as the linker when building rustc itself reduced the time taken for linking from about 93 seconds to about 41 seconds. (On my Linux machine I do this by preceding the build command with RUSTFLAGS="-C link-arg=-fuse-ld=lld".) LLD is a really fast linker! #39915 is the three-year old issue open for making LLD the default linker for rustc, but unfortunately it has stalled. Alexis Beingessner wrote a nice summary of the current situation. If anyone with knowledge of linkers wants to work on that issue, it could be a huge win for many Rust users.

Failures

Not everything I tried worked. Here are some notable failures.

#69152: As mentioned above, #68914 greatly improved SipHasher128, the hash function used by incremental compilation. That hash function is a 128-bit version of the default 64-bit hash function used by Rust hash tables. I tried porting those same improvements to the default hasher. The goal was not to improve rustc’s speed, because it uses FxHasher instead of default hashing, but to improve the speed of all Rust programs that do use default hashing. Unfortunately, this caused some compile-time regressions for complex reasons discussed in detail in the PR, and so I abandoned it. I did manage to remove some dead code in the default hasher in #69471, though.

#69153: While working on #69152, I tried switching from FxHasher back to the improved default hasher (i.e. the one that ended up not landing) for all hash tables within rustc. The results were terrible; every single benchmark regressed! The smallest regression was 4%, the largest was 85%. This demonstrates (a) how heavily rustc uses hash tables, and (b) how much faster FxHasher is than the default hasher when working with small keys.

I tried using ahash for all hash tables within rustc. It is advertised as being as fast as FxHasher but higher quality. I found it made rustc a tiny bit slower. Also, ahash is also not deterministic across different builds, because it uses const_random! when initializing hasher state. This could cause extra noise in perf runs, which would be bad. (Edit: It would also prevent reproducible builds, which would also be bad.)

I tried changing the SipHasher128 function used for incremental compilation from the Sip24 algorithm to the faster but lower-quality Sip13 algorithm. I got wins of up to 3%, but wasn’t confident about the safety of the change and so didn’t pursue it further.

#69157: Some follow-up measurements after #69050 suggested that its changes to LEB128 decoding were not as clear a win as they first appeared. (The improvements to encoding were still definitive.) The performance of decoding appears to be sensitive to non-local changes, perhaps due to differences in how the decoding functions are inlined throughout the compiler. This PR reverted some of the changes from #69050 because my initial follow-up measurements suggested they might have been pessimizations. But then several sets of additional follow-up measurements taken after rebasing multiple times suggested that the reversions sometimes regressed performance. The reversions also made the code uglier, so I abandoned this PR.

#66405: Each obligation held by ObligationForest can be in one of several
states, and transitions between those states occur at various points. This
PR reduced the number of states from five to three, and greatly reduced the
number of state transitions, which won up to 4% on a few benchmarks. However, it ended up causing some drastic regressions for some users, so in #67471 I reverted those changes.

#60608: This issue suggests using FxIndexSet in some places where currently an FxHashMap plus a Vec are used. I tried it for the symbol table and it was a significant regression for a few benchmarks.

Progress

Since my last blog post, compile times have seen some more good improvements. The following screenshot shows wall-time changes on the benchmark suite since then (2019-12-08 to 2020-04-22).

The biggest changes are in the synthetic stress tests await-call-tree-debug, wf-projection-stress-65510, and ctfe-stress-4, which aren’t representative of typical code and aren’t that important.

Overall it’s good news, with many improvements (green), some in the double digits, and relatively few regressions (red). Many thanks to everybody who helped with all the performance improvements that landed during this period.

https://blog.mozilla.org/nnethercote/2020/04/24/how-to-speed-up-the-rust-compiler-in-2020/

WebGPU is an emerging API that provides access to the graphics and computing capabilities of hardware on the web. It’s designed from the ground up within the W3C GPU for the Web group by all major browser vendors, as well as Intel and a few others, guided by the following principles:

We are excited to bring WebGPU support to Firefox because it will allow richer and more complex graphics applications to run portably in the Web. It will also make the web platform more accessible to teams who mostly target modern native platforms today, thanks to the use of modern concepts and first-class WASM (WebAssembly) support.

API concepts

WebGPU aims to work on top of modern graphics APIs: Vulkan, D3D12, and Metal. The constructs exposed to the users reflect the basic primitives of these low-level APIs. Let’s walk through the main constructs of WebGPU and explain them in the context of WebGL – the only baseline we have today on the Web.

Separation of concerns

The first important difference between WebGPU and WebGL is that WebGPU separates resource management, work preparation, and submission to the GPU (graphics processing unit). In WebGL, a single context object is responsible for everything, and it contains a lot of associated state. In contrast, WebGPU separates these into multiple different contexts:

• GPUDevice creates resources, such as textures and buffers.
• GPUCommandEncoder allows encoding individual commands, including render and compute passes.
• Once done, it turns into GPUCommandBuffer object, which can be submitted to a GPUQueue for execution on the GPU.
• We can present the result of rendering to the HTML canvas. Or multiple canvases. Or no canvas at all – using a purely computational workflow.

Overall, this separation will allow for complex applications on the web to stream data in one or more workers and create any associated GPU resources for it on the fly. Meanwhile, the same application could be recording work on multiple workers, and eventually submit it all together to GPUQueue. This matches multi-threading scenarios of native graphics-intensive applications and allows for high utilization of multi-core processors.

Pipeline state

The second important change is how WebGPU encapsulates pipeline state.

In WebGL, the user would create a shader program at program initialization. Later when the user attempts to use this shader program, the driver takes into consideration all the other states currently set, and may need to internally recompile the shader program. If the driver does recompile the shader program, this could introduce CPU stalls.

In contrast, WebGPU has the concept of a pipeline state object (namely, GPURenderPipeline and GPUComputePipeline). A pipeline state object is a combination of various states that the user creates in advance on the device – just like in native APIs. The user provides all this state upfront, which allows the browsers and hardware drivers to avoid extra work (such as shader recompilation) when it’s used later in GPU operations.

From the developer perspective, it’s easier to manage these coarse state objects as well. They don’t have to think as much about which of the fine-grained states to change, and which ones to preserve.

The pipeline state includes:

• Shaders
• Layouts of vertex buffers and attributes
• Layouts of bind groups
• Blending, depth, and stencil states
• Formats of the output render targets

Binding model

A third difference between WebGPU and WebGL is the binding model. The WebGPU binding model is largely inspired by Vulkan (as an intersection of capabilities of the target native APIs) and allows resources to be grouped together into a GPUBindGroup object. Then we bind GPUBindGroups during command recording in order to use the resources within shaders.

By creating these bind groups upfront, the graphics driver can perform any necessary preparations in advance. This allows the browser to change resource bindings much faster between draw calls.

Most importantly, the user has to describe the layout of resource bindings ahead of time, baking it into a GPUBindGroupLayout object. Both pipeline states and concrete bind groups know about the bind group layout as well. This knowledge serves as a contract between the shader and the API. It allows the browser or the driver to lay out resources in a way that allows faster binding.

Complementary talks

There are public talks by the active members of the standards group, which may help better understanding of what the API is, how it evolved, and how we expect it to be used. Dzmitry Malyshau from Mozilla talked about the Rust aspect of implementing WebGPU for native platforms at Fosdem 2020. Earlier, Corentin Wallez from Google gave an overview of WebGPU API at DevFest Toulouse 2019. Last but not the least, the Google I/O 2019 presentation was full of shiny demos and code samples.

Firefox story

Tech stack

In Firefox, we are working on a complete ground-up implementation of the WebGPU specification. The core logic is provided by the wgpu-core project that is written by the Rust community with Mozilla’s help. It’s based on the gfx-rs project which is able to translate Vulkan-like GPU workloads to D3D12, D3D11, Metal, and even OpenGL (to some degree).

We are also working on the shader infrastructure that would allow us to work with WebGPU Shading Language, validate it against the API expectations, and convert it to the backend shading language that is expected by the driver.

The latest and greatest of our work can be seen in Nightly with the “dom.webgpu.enabled = true” preference set, which also requires “gfx.webrender.all = true”. It should work on Windows 7 and Linux with Vulkan drivers, Windows 10, macOS, and even on qualified Android devices. Be prepared for a bumpy ride because everything is still a work in progress!

Examples

At the time of writing, Firefox Nightly can run all of the Google SPIR-V-based WebGPU samples with an exception of “animometer” (which relies on GPURenderBundle that we haven’t yet implemented). Here is Nightly rendering the “fractal cube” on Linux/Vulkan:

We can also execute computational workloads. For example, here is Nightly rendering the “boids” example on Windows 10/Vulkan:

The Rust community has also been working on targeting the WebGPU directly in wgpu-rs (which provides a Rust API and uses the same wgpu project for implementing it). This exciting work opens the doors for us to have many existing applications in the Rust ecosystem running in the browser. The first batch of these applications is wgpu-rs own examples. Here is Nightly rendering the “shadow” example on macOS/Metal:

Future work

There are many things missing in Firefox Nightly to make this WebGPU implementation truly usable. It’s still early days, and we just got the first examples working.

Error handling

One major area that is missing is the error model. WebGPU errors work on the concept of “contagious internal nullability” (also called “maybe monad”): at some point, an object that’s incorrectly used may become “Invalid”. The actual error will be returning back to the content side asynchronously. If any other object becomes dependent on it (for example, a texture as a part of a bind group object), that parent object also becomes “Invalid”  – so that state is contagious. Implementing this error model will allow developers to iterate on code without crashing the GPU process (which is typically a safe Rust panic), or causing any other side effects.

Accelerated presentation

Another important bit to implement is hardware support for presentation. Currently, to display the rendered image to an HTML canvas, the image is first rendered on the GPU. It is then read back to a CPU-side buffer, which we provided to WebRender as an “external image”. WebRender uploads the image contents to the GPU again and finally displays it on the HTML canvas. This round-trip is unnecessary in general. Instead, we want to use the platform-specific mechanisms of surface sharing between the WebGPU backend APIs and WebRender. We’ll need to approach each platform independently to implement this, while keeping the current path as fall-back.

API coverage

Finally, there are bits of the specification we have not implemented yet. This will be a constant process of catching up as the spec evolves, until it eventually stabilizes. One of the missing pieces is support for GPURenderBundle objects. Each bundle contains a small sequence of rendering commands that it can use and reuse multiple times in recording of passes. It’s currently the only mechanism of re-using the commands in the API. It will be important for the more complex kinds of content on the Web, such as open-world games.

State of the spec

The WebGPU specification is developed on GitHub in the form of Bikeshed documents (separate for the main WebGPU spec and the WebGPU shading language). It’s open for participation!

The group has mostly resolved the major architecture issues of the API. Recently we agreed on the WebGPU Shading Language direction based on the Tint prototype. We still need to solve a lot of design riddles before we make it available to end users to write shaders in.

One of the unresolved issues is the API for data transfers between CPU and GPU. Working with memory directly is where the web platform differs greatly from native platforms. We’ve discussed a dozen different proposals but have not yet found a design solution that fulfills our principles.

Overall, the spec is still heavily a work in progress. It’s available for early hackers but not recommended for any use in production yet. We are hoping to get a minimum-viable-product version of the spec and implementations by the end of 2020. The current state of implementations can be checked on webgpu.io.

The post A Taste of WebGPU in Firefox appeared first on Mozilla Hacks - the Web developer blog.

https://hacks.mozilla.org/2020/04/experimental-webgpu-in-firefox/

Firefox has one of the oldest security bug bounties on the internet, dating back to 2004. From 2017-2019, we paid out $965,750 to researchers across 348 bugs, making the average payout $2,775 – but as you can see in the graph below, our most common payout was actually $4,000!

After adding a new static analysis bounty late last year, we’re excited to further expand our bounty program in the coming year, as well as provide an on-ramp for more participants. We’re updating our bug bounty policy and payouts to make it more appealing to researchers and reflect the more hardened security stance we adopted after moving to a multi-process, sandboxed architecture. Additionally, we’ll be publishing more posts about how to get started testing Firefox – which is something we began by talking about the HTML Sanitization we rely on to prevent UXSS . By following the instructions there you can immediately start trying to bypass our sanitizer using your existing Firefox installation in less than a minute.

Firstly, we’re amending our current policy to be more friendly and allowing duplicate submissions. Presently, we have a ‘first reporter wins’ policy, which can be very frustrating if you are fuzzing our Nightly builds (which we encourage you to do!) and you find and report a bug mere hours after another reporter. From now on, we will split the bounty between all duplicates submitted within 72 hours of the first report; with prorated amounts for higher quality reports. We hope this will encourage more people to fuzz our Nightly ASAN builds.

Besides rewarding duplicate submissions, we’re clarifying our payout criteria and raising the payouts for higher impact bugs. Now, sandbox escapes and related bugs will be eligible for a baseline $8,000, with a high quality report up to $10,000.  Additionally, proxy bypass bugs are eligible for a baseline of $3,000, with a high quality report up to $5,000. And most submissions are above baseline: previously the typical range for high impact bugs was $3,000 – $5,000 but as you can see in the graphic above, the most common payout for those bugs was actually $4,000!  You can find full details of the criteria and amounts on our Client Bug Bounty page.

Again, we want to emphasize – if it wasn’t already – that a bounty amount is not determined based on your initial submission, but rather on the outcome of the discussion with developers. So improving test cases post-submission, figuring out if an engineer’s speculation is founded or not, or other assistance that helps resolve the issue will increase your bounty payout.

And if you’re inclined, you are able to donate your bounty to charity – if you wish to do so and have payment info on file, please indicate this in new bugs you file and we will contact you.

Lastly, we would like to let you know that we have cross-posted this to our new Attack & Defense blog. This new blog is a vehicle for tailored content specifically for engineers, security researchers, and Firefox bug bounty participants. We’ll be highlighting improvements to the bug bounty program (which will often be posted to this Security blog also) – but also posting guides on how to test different parts of Firefox. You can follow us via RSS or Twitter, where we’ll be boosting tweets relevant to browser security: exploitation walkthroughs, conference talks, and the like.

The post Firefox’s Bug Bounty in 2019 and into the Future appeared first on Mozilla Security Blog.

https://blog.mozilla.org/security/2020/04/23/bug-bounty-2019-and-future/

On April 22nd 2019, we announced our current, this, incarnation of the curl bug bounty. In association with Hackerone we now run the program ourselves, primarily funded by gracious sponsors. Time to take a closer look at how the first year of bug bounty has been!

Number of reports

We’ve received a total of 112 reports during this period.

On average, we respond with a first comment to reports within the first hour and we triage them on average within the first day.

Out of the 112 reports, 6 were found actual security problems.

Bounties

All confirmed security problems were rewarded a bounty. We started out a bit careful with the amounts but we are determined to raise them as we go along and we’ve seen that there’s not really a tsunami coming.

We’ve handed out 1,400 USD so far, which makes it an average of 233 USD per confirmed report. The top earner got two reports rewarded and received 450 USD from us. So far…

But again: our ambition is to significantly raise these amounts going forward.

Trends

The graph above speaks clearly: lots of people submitted reports when we opened up and the submission frequency has dropped significantly over the year.

A vast majority of the 112 reports we’ve received have were more or less rubbish and/or more or less automated reports. A large amount of users have reported that our wiki can be edited by anyone (which I consider to be a fundamental feature of a wiki) or other things that we’ve expressly said is not covered by the program: specific details about our web hosting, email setup or DNS config.

A rough estimate says that around 80% of the reports were quickly dismissed as “out of policy” – ie they reported stuff that we documented is not covered by the bug bounty (“Sirs, we can figure out what http server that’s running” etc). The curl bug bounty covers the products curl and libcurl, thus their source code and related specifics.

Bounty funds

curl has no ties to any organization. curl is not owned by any corporation. curl is developed by individuals. All the funds we have in the project are graciously provided to us by sponsors and donors. The curl funds are handled by the awesome Open Collective.

Security is of utmost importance to us. It trumps all other areas, goals and tasks. We aim to produce solid and secure products for the world and we act as swiftly and firmly as we can on all reported security problems.

Security vulnerability trends

We have not published a single CVE for curl yet this year (there was one announced, CVE-2019-15601 but after careful considerations we have backpedaled on that, we don’t consider it a flaw anymore and the CVE has been rejected in the records.)

As I write this, there’s been exactly 225 days since the latest curl CVE was published and we’re aiming at shipping curl 7.70.0 next week as the 6th release in a row without a security vulnerability to accompany it. We haven’t done 6 “clean” consecutive release like this since early 2013!

Looking at the number of CVEs reported in the curl project per year, we can of course see that 2016 stands out. That was the year of the security audit that ended up the release of curl 7.51.0 with no less than eleven security vulnerabilities announced and fixed. Better is of course the rightmost bar over the year 2020 label. It is still non-existent!

As you can see in the graph below, the “plateau” in the top right is at 92 published CVEs. The previous record holder for longest period in the project without a CVE ended in February 2013 (with CVE-2013-0249) at 379 days.

2013 was however quite a different era for curl. Less code, much less scrutinizing, no bug bounty, lesser tools, no CI jobs etc.

Are we improving?

Is curl getting more secure?

We have more code and support more protocols than ever. We have a constant influx of new authors and contributors. We probably have more users than ever before in history.

At the same time we offer better incentives than ever before for people to report security bugs. We run more CI jobs than ever that run more and more test cases while code analyzers and memory debugging are making it easier to detect problems earlier. There are also more people looking for security bugs in curl than ever before.

Jinx?

I’m under no illusion that there aren’t more flaws to find, report and fix. We’re all humans and curl is still being developed at a fairly high pace.

Please report more security bugs!

Credits

Top image by Luisella Planeta Leoni from Pixabay

https://daniel.haxx.se/blog/2020/04/23/report-curls-bug-bounty-one-year-in/

Please note some of the information provided in this report may be subject to change as we are sometimes sharing information about projects that are still in early stages and are not final yet. 

Before we get into the report, we must share that Youghourta Benali, one of the Arabic l10n community’s managers, has passed away due to prior health issues. He was a passionate activist for the open Web and Arabic’s presence on the Web, localizing Mozilla projects for over 7 years. We’ll all miss him and wish his family and friends peace at this time. The surviving Arabic managers are currently writing a guest post that we’ll post here when ready.

Welcome!

Are you a locale leader and want us to include new members in our upcoming reports? Contact us!

New community/locales added

• Central Kurdish (ckb)

Other updates

Enforcing 2 factors authentication in GitHub for mozilla-l10n

As recently communicated in our mailing lists, on April 27 we will enforce the 2 factor authentication (2FA) in GitHub for the mozilla-l10n organization.

If you are part of the mozilla-l10n organization, but don’t have 2FA enabled, you will be automatically removed. This won’t prevent you from opening pull requests or filing issues, for example, and you will be able to rejoin (if you want) after enabling 2FA.

As explained in the original announcement, a lot has changed since we started using GitHub for Mozilla Localization, almost 5 years ago. We have consolidated contribution paths to l10n to Pontoon, and having direct access to repositories is not necessary anymore. Write access is already not allowed for some repositories, since formats are not user-friendly and can be properly managed only through localization tools.

For more information about enabling 2FA in GitHub, refer to this document.

New content and projects

What’s new or coming up in Firefox desktop

Upcoming deadlines:

• Firefox 76 is currently in beta and will be released on May 5. The deadline to update localization has just passed (April 21).
• The deadline to update localizations for Firefox 77, currently in Nightly, will be May 19 (4 weeks after the previous deadline).

As explained in previous editions of the l10n report, In terms of localization priority and deadlines, note that the content of the What’s new panel, available at the bottom of the hamburger menu, doesn’t follow the release train. For example, content for 76 has been exposed on April 15, but it will be possible to update translations until the very end of the cycle, beyond the normal deadline for that version. You can find these strings in the asrouter.ftl file in Pontoon.

To test on Nightly:

• Type about:config in the address bar and press enter. Click the button to accept the risk and continue.
• Search for browser.newtabpage.activity-stream.asrouter.devtoolsEnabled and make sure that it’s set to true (double click it if it’s false). This will enable a new wrench icon in the New Tab page.
• Search for browser.newtabpage.activity-stream.asrouter.useRemoteL10n and set it to false. This will force the browser to use translations available in the build, instead of translations hosted remotely (those are updated less frequently). It’s important to use Nightly and not Beta for this.
• Open a New Tab, click on the wrench icon in the top right corner. This should open the about:newtab#devtools page. On the left sidebar, click on What’s New Panel, then click the button Open What’s New Panel.

There is also a new warning for vulnerable passwords in about:logins. You can find information on how to trigger this warning in this page.

What’s new or coming up in mobile

Firefox iOS

• The l10n cycle for Firefox for iOS v25 ends 27 April.
• New screenshots have been added to the project landing page in Pontoon.

Fenix

The Fenix launch continues to be a high priority for Mozilla. The date of that launch will depend on the outcome of a handful of experiments. In the meantime, string updates should be minimal, and we continue to aim for shipping all locales that are present on Fennec.

Milestones

• Feature complete date: to be announced when set
• Source string freeze date: to be announced when set
• Localized string freeze date: estimated to be July
• Fenix launch date: no sooner than July

Current progress

• Current average Fenix completion percentage across all locales: 74% (1% increase)
• Target average completion percentage across all locales: 90%

Just a reminder, you can find the Fenix Nightly builds in the Google Play Store for testing your localization. Please take the time to test! Your language will appear in the app settings once you begin localizing the project in Pontoon and the Fenix team imports your locale into the build process. If you don’t localize the project, your language will not appear in the builds. Please note that at least Android 5 is necessary to use and test Fenix on an Android device.

What’s new or coming up in web projects

Mozilla.org

There have been a few updates since the last report. The following two are new:

• firefox/whatsnew_76.lang: the page is due on 26 April for these locales: de, en-CA, en-GB, and fr. Other locales are nice to have, with a fallback to WNP.
• firefox/enterprise/index.lang.

Legal Documentation

The following languages for the Common Voice project have reached the milestones that require legal documents for Privacy Notice and Terms of Use. They are still in the work as we speak. If you are interested in reviewing them, the documents will be ready by early next week. Please note, any review feedback requires a peer review before being pushed to production.

Newly added languages: Abkhaz, Arabic, Assamese, Greek, Czech, Frisian, Indonesian, Japanese, Latvian, Maltese, Punjabi (India), Polish, Portuguese (pt-BR), Romansh (for Romansh Sursilvan & Romansh Vallader), Romanian, Slovenian, and Ukrainian

Events

We planned to organize multilocale workshops again for this year. With country closures, COVID-19 outbreaks, and the long road to economic recovery for industries we rely on for the logistics of these events, we have decided to cancel these workshops for 2020. We’re working to figure out a suitable replacement for these in-person events in 2020, stay tuned. We will revisit in October to determine what we can do for 2021.

As you’ve seen from the Mozilla Reps, they’ve made some changes to the event funding process to respect local effects of COVID-19. Effectively, if your local government has issued a quarantine or shelter-in-place order, please respect those orders and cancel your Fenix l10n hackathon. If that does not apply to you, you can still organize a local hackathon. I have created an event in the new community portal for Fenix localization. If you would like to organize a local l10n hackathon in the coming weeks to bring your locale to 100% complete, please reach out to a Resource Rep. Budget requests from Resource Reps for under USD$100 will be fast-tracked for approval.

Friends of the Lion

Image by Elio Qoshi

Know someone in your l10n community who’s been doing a great job and should appear here? Contact on of the l10n-drivers and we’ll make sure they get a shout-out (see list at the bottom)!

Useful Links

• #l10n-community channel on Matrix
• Dev.l10n mailing list and Dev.l10n.web mailing list – where project updates happen. If you are a localizer, then you should be following this
• Twitter
• Telegram (contact one of the l10n-drivers below so we will add you)
• L10n blog

Questions? Want to get involved?

• If you want to get involved, or have any question about l10n, reach out to:
  • Delphine – l10n Project Manager for mobile
  • Peiying (CocoMo) – l10n Project Manager for mozilla.org, marketing, and legal
  • Francesco Lodolo (flod) – l10n Project Manager for desktop
  • Th'eo Chevalier – l10n Project Manager for Mozilla Foundation
  • Axel (Pike) – l10n Tech Team Lead
  • Sta's – l20n/FTL tamer
  • Matjaz – Pontoon dev
  • Jeff Beatty (gueroJeff) – l10n-drivers manager
  • Giulia Guizzardi – Community manager for Support

Did you enjoy reading this report? Let us know how we can improve by reaching out to any one of the l10n-drivers listed above.

https://blog.mozilla.org/l10n/2020/04/23/l10n-report-april-2020-edition/