Bill Walker: Birdsongs, Musique Concr`ete, and the Web Audio API |
In January 2015, my friend and collaborator Brian Belet and I presented Oiseaux de M^eme — an audio soundscape app created from recordings of birds — at the first Web Audio Conference. In this post I’d like to describe my experience of implementing this app using the Web Audio API, Twitter Bootstrap, Node.js, and REST APIs.
Screenshot showing Birds of a Feather, a soundscape created with field recordings of birds that are being seen in your vicinity.
We wanted to create a web-based Musique Concr`ete, building an artistic sound experience by processing field recordings. We decided to use xeno-canto — a library of over 200,000 recordings of 9,000 different bird species — as our source of recordings. Almost all the recordings are licensed under Creative Commons by their generous recordists. We select recordings from this library based on data from eBird, a database of tens of millions of bird sightings contributed by bird watchers everywhere. By using the Geolocation API to retrieve eBird sightings near to the listeners’ location, our soundscape can consist of recordings of bird species that bird watchers have reported recently near the listener — each user gets a personalized soundscape that changes daily.
We use the browser’s Web Audio API to play back the sounds from xeno-canto. The Web Audio API allows developers to play back, record, analyze, and process sound by creating AudioNodes that are connected together, like an old modular synthesizer.
Our soundscape is implemented using four AudioBuffer nodes, each of which plays a field recording in a loop. These loops are placed in a stereo field using Panner nodes, and mixed together before being sent to the listener’s speakers or headphones.
After all the sounds have loaded and begin playing, we offer users several controls for manipulating the sounds as they play:
The controls described above are implemented as typical JavaScript event handlers. When the Pan button is pressed, for example, we run this handler:
// sets the X,Y,Z position of the Panner to random values between -1 and +1 BirdSongPlayer.prototype.randomizePanner = function() { this.resetLastActionTime(); // NOTE: x = -1 is LEFT this.panPosition = { x: 2 * Math.random() - 1, y: 2 * Math.random() - 1, z: 2 * Math.random() - 1} this.panner.setPosition( this.panPosition.x, this.panPosition.y, this.panPosition.z); }
I had a few minor issues where I had to work around shortcomings in the Web Audio API. Other authors have already documented similar experiences; I’ll summarize mine briefly here:
setPosition()
.
This has had a minimal impact on my code so far. My longer-term concern is that I’d rather store the position in the Panner and retrieve it from there, instead of storing a copy elsewhere. In my experience, multiple copies of the same information becomes a readability and maintainability problem as code grows bigger and more complex.
linearRampToValueAtTime()
on the playbackRate AudioParam
. As far as I can tell, AudioParams don’t let me retrieve their values after calling linearRampToValueAtTime()
, so I’m obliged to keep a duplicate copy of this value in my JS object.playbackRate
between random values, it is very difficult to compute the current playback position within the buffer. Unlike some API users, I don’t need a highly accurate position, I just want to visualize for my users when the current sound loop restarts.In the screenshot above, you can see the four AudioBufferSource
s, each feeding through a GainNode
and PannerNode
before being summed by an AudioDestination
. Note that each recording is also connected to an AnalyzerNode
; the Analyzers are used to create the scrolling amplitude graphs for each loop.
As the soundscape evolves, users often want to know which bird species is responsible for a particular sound they hear in the mix. We use a scrolling visualization for each loop that shows instantaneous amplitude, creating distinctive shapes you can correlate with what you’re hearing. The visualization uses the Analyzer node to perform a fast Fourier transform (FFT) on the sound, which yields the amplitude of the sound at every frequency. We compute the average of all those amplitudes, and then draw that amplitude at the right edge of a Canvas. As the contents of the Canvas shift sideways on every animation frame, the result is a horizontally scrolling amplitude graph.
BirdSongPlayer.prototype.initializeVUMeter = function() { // set up VU meter var myAnalyser = this.analyser; var volumeMeterCanvas = $(this.playerSelector).find('canvas')[0]; var graphicsContext = volumeMeterCanvas.getContext('2d'); var previousVolume = 0; requestAnimationFrame(function vuMeter() { // get the average, bincount is fftsize / 2 var array = new Uint8Array(myAnalyser.frequencyBinCount); myAnalyser.getByteFrequencyData(array); var average = getAverageVolume(array); average = Math.max(Math.min(average, 128), 0); // draw the rightmost line in black right before shifting graphicsContext.fillStyle = 'rgb(0,0,0)' graphicsContext.fillRect(258, 128 - previousVolume, 2, previousVolume); // shift the drawing over one pixel graphicsContext.drawImage(volumeMeterCanvas, -1, 0); // clear the rightmost column state graphicsContext.fillStyle = 'rgb(245,245,245)' graphicsContext.fillRect(259, 0, 1, 130); // set the fill style for the last line (matches bootstrap button) graphicsContext.fillStyle = '#5BC0DE' graphicsContext.fillRect(258, 128 - average, 2, average); requestAnimationFrame(vuMeter); previousVolume = average; }); }
I’m continuing to work on cleaning up my JavaScript code for this project. I have several user interface improvements suggested by my Mozillia colleagues that I’d like to try. And Prof. Belet and I are considering what other sources of geotagged sounds we can use to make more soundscapes with. In the meantime, please try Oiseaux de M^eme for yourself and let us know what you think!
http://softwarewalker.com/2015/07/08/birdsongs-musique-concrete-and-the-web-audio-api/
|
Air Mozilla: Product Coordination Meeting |
Duration: 10 minutes This is a weekly status meeting, every Wednesday, that helps coordinate the shipping of our products (across 4 release channels) in order...
https://air.mozilla.org/product-coordination-meeting-20150708/
|
Air Mozilla: The Joy of Coding (mconley livehacks on Firefox) - Episode 20 |
Watch mconley livehack on Firefox Desktop bugs!
https://air.mozilla.org/the-joy-of-coding-mconley-livehacks-on-firefox-episode-20/
|
Benjamin Kerensa: The Glucosio Project |
I have come up with a new phrase and I am going to keep saying it and it is “The most important open source software has not yet been made.” But why is this phrase true? Simply put we have a lot of great open source software out there but the most important open source software is the one that’s not been written because of some barrier or challenge.
For every person, different software has different levels of importance right? So what is the most important unwritten open source software for me? Well it is health tech software that enables people to better understand how their health is and how their choices can impact it positively and negatively.
I was recently diagnosed with Type 2 Diabetes and I have from the get-go tried to use technology and software to help me manage it. From graphing my glucose levels so I know how different foods impact me to tracking medication and other important metrics. But one thing stuck out when I was looking at available tools is that there are not many open source health tech applications and tools available and those that did exist were inferior to the proprietary ones.
So why is it important to have these tools be open source if the proprietary ones work well? Simply put, if you have the source code you can trust your data is kept private and safe but also you can build off the tools and integrate them with other services and tools that work specifically for you.
That being said, I came up with the idea of launching a Open Source Project and have formed a team of amazing individuals who share my vision of creating tools to help the millions of people worldwide suffering from both Type 1 and Type 2 Diabetes. We are moving forward with that and are right now in the planning stage of launching the Glucosio Project (Italian for Glucose). The project will initially launch an Android app, then iOS and finally a web app (Think Tizen, Ubuntu Phone, Firefox OS) to allow diabetics to track their glucose and connect with third party services (IFTT, Phillips Hue, Pushbullet, Pushover etc), share the data and better understand the impact of their choices. We very well may expand as the project and contributor base but this is what we have envisioned so far.
I would like to thank Elio Qoshi, Satyajit Sahoo, Paolo Rotolo, Georgi Karavasilev, Priyanka Nag, Joshua Fogg, Viking Karwur, Stefan Angelov, Rahul Kondi and Ahmar Siddiqui for sharing this vision with me and joining as initial project contributors and the core team that will be behind the Glucosio Project.
We still have room for more (Dev, Doc, Creative, l10n etc) and if you are interested in contributing to this project please get in touch with us at hello [at] glucosio.org or follow the project on Github. We hope to make a big difference in many people’s lives with the apps we are working on and hope you will join us!
http://feedproxy.google.com/~r/BenjaminKerensaDotComMozilla/~3/U7DMxzKIrpA/the-glucosio-project
|
Christian Heilmann: Slimming down the web: Remove code to fix things, don’t add the “clever” thing |
Today we saw a new, interesting service called Does it work on Edge? which allows you to enter a URL, and get that URL rendered in Microsoft Edge. It also gives you a report in case there are issues with your HTML or CSS that are troublesome for Edge (much like Microsoft’s own service does). In most cases, this will be browser-specific code like prefixed CSS. All in all this is a great service, one of many that make our lives as developers very easy.
If you release something on the web, you get feedback. When I tweeted enthusiastically about the service, one of the answers was by @jlbruno, who was concerned about the form not being keyboard accessible.
The reason for this is simple: the form on the site itself is none insofar there is no submit
button of any kind. The button in the page is a anchor pointing nowhere and the input element itself has a keypress
event attached to it (even inline):
There’s also another anchor that points nowhere that is a loading message with a display of none. Once you click the first one, this one gets a display of block and replaces the original link visually. This is great UX - telling you something is going on – but it only really works when I can see it. It also gives me a link element that does nothing.
Once the complaint got heard, the developers of the site took action and added an autofocus attribute to the input field, and proudly announcing that now the form is keyboard accessible.
Now, I am not having a go here at the developers of the site. I am more concerned that this is pretty much the state of web development we have right now:
In this case, the whole construct is far too complex for the job at hand. If you want to create something like this and make it accessible to keyboard and mouse users alike, the course of action is simple:
Use the REST URL of your service (which I very much assume this product has) as the action and re-render the page when it is done.
If you want to get fancy and not reload the page, but keep all in place assign a submit handler to the form element, call preventDefault() and do all the JS magic you want to do:
Visually and functionally on a browser that was lucky enough to not encounter a JavaScript error until now, the autofocus solution does look like it does the job. However, what it does is shift the focus of the document to the input field once the page has loaded. A screenreader user thusly would never ever learn what the site is about as you skip the header and all the information. As the input element also lacks a label, there isn’t even any information as to what the user is supposed to enter here. You sent that user into a corner without any means of knowing what’s going on. Furthermore, keyboard users are primed and ready to start navigating around the page as soon as it loads. By hijacking the keyboard navigation and automatically sending it to your field you confuse people. Imagine pressing the TV listings button on a TV and instead it just sends you to the poodle grooming channel every time you do it.
So here’s my plea in this: let’s break that pattern of working on the web. Our products don’t get better when we use fancier code. They get better when they are easier to use for everybody. The fascinating bit here is that by understanding how HTML works and what it does in browsers, we can avoid writing a lot of code that looks great but breaks very easily.
There is no shortage of articles lamenting how the web is too slow, too complex and too big on the wire compared to native apps. We can blame tools for that or we could do something about it. And maybe not looking for a readymade solution or the first result of Stackoverflow is the right way to do that.
Trust me, writing code for the web is much more rewarding when it is your code and you learned something while you implemented it.
Let’s stop adding more when doing the right thing is enough.
|
Zack Weinberg: Operating system selection for $PROJECT, mid-2015 |
Presented without context, for amusement purposes only, a page from my notes:
FreeBSD | NetBSD | Linux | |
---|---|---|---|
Per-process default route |
Poorly documented, possibly incomplete |
Probably not |
Poorly documented, buggy |
Can compile PhantomJS | Probably | Probably | Yes |
Jails | Yes | No | Not really |
Xen paravirtual guest | Incomplete | Yes | Yes |
System call tracing | truss | None? | strace |
pipe2 | Yes | Yes | Yes |
closefrom | Yes | Yes | No |
sysctl | Yes | Yes | No |
getauxval | No | No | Yes |
signalfd | No | No | Yes |
execvpe | No | Yes | Yes |
Reference documentation | Acceptable (YMMV1) | Acceptable (YMMV) | Major gaps |
Tutorial documentation | Terrible | Terrible | Terrible |
Package management | Broken as designed | Broken as designed | Good |
System maintenance automation | I can’t find any | I can’t find any | Acceptable |
QA reputation | Excellent | Good | Good |
Security reputation | Good | Good | Debatable |
Development community | Unknown to me | Unknown to me | Full of assholes |
1 It makes sense to me, but then, I taught myself Unix system programming and administration by reading the SunOS 4 manpages.
https://www.owlfolio.org/possibly-useful/os-for-project-x-2015/
|
Air Mozilla: Webdev Extravaganza: July 2015 |
Once a month web developers across the Mozilla community get together (in person and virtually) to share what cool stuff we've been working on.
|
Air Mozilla: Martes mozilleros |
Reuni'on bi-semanal para hablar sobre el estado de Mozilla, la comunidad y sus proyectos.
|
Gregory Szorc: Publish When Pushing to MozReview |
A lot of people contributed some really great feedback about MozReview at Whistler. One of the most frequent requests was for the ability to publish submitted review requests without having to open a browser. I'm pleased to report that as of yesterday, this feature is implemented! If reviewers have been assigned to all your review requests, Mercurial will now prompt you to publish the review requests during hg push. It should just work.
As part of this change, we also introduced more advanced feature negotiation into the handshake between client and server. This means we now have a mechanism for detecting out-of-date client installations. This will enable us to more aggressively drop backwards compatibility (making server-side development easier) while simultaneously ensuring that more people are running modern and hopefully better versions of the client code. This should translate to moving faster and a better experience for everyone.
http://gregoryszorc.com/blog/2015/07/07/publish-when-pushing-to-mozreview
|
Rail Aliiev: Funsize is ready for testing! |
Funsize is very close to be enabled in production! It has undergone a Rapid Risk Assessment procedure, which found a couple of potential issues. Most of them are either resolved or waiting for deployment.
To make sure everything works as expected and to catch some last-minute bugs, I added new update channels for Firefox Nightly and Developer Edition. If you are brave enough, you can use the following instructions and change your update channels to either nightly-funsize (for Firefox Nightly) or aurora-funsize (for Firefox Developer Edition).
TL;DR instruction look like this:
pref("app.update.channel", "aurora-funsize"); // Firefox Developer Edition
or
pref("app.update.channel", "nightly-funsize"); // Firefox Nightly
You can check your channel in the About dialog and it should contain the channel name:
If you see any issues, please report them to Bugzilla.
http://rail.merail.ca/posts/funsize-is-ready-for-testing.html
|
Robert O'Callahan: rr Talk Video From TCE 2015 |
The good people at the Technion have uploaded video of the TCE 2015 talks, in particular video of my rr talk. My slides are also available. This is a fairly high-level talk describing the motivation and design of rr, delving into a few interesting details and concluding with a discussion of some of the exciting possibilities that could be enabled by rr and similar technology.
http://robert.ocallahan.org/2015/07/rr-talk-video-from-tce-2015.html
|
Byron Jones: happy bmo push day! |
the following changes have been pushed to bugzilla.mozilla.org:
discuss these changes on mozilla.tools.bmo.
https://globau.wordpress.com/2015/07/07/happy-bmo-push-day-149/
|
Ehsan Akhgari: Local Autoland |
It has been a while since I’ve asked myself: “Is the tree open?”
These days, when I want to land something on mozilla-inound, I switch to my git-workdir[1], I cherry-pick the commit that I want to land, and I type the following in my terminal:
$ land
Land is a very sophisticated bot that tries to land your commits for you! It assumes you use git-cinnabar, which you should if you use git. Porting it to Mercurial is left as an exercise.
|
Cameron Kaiser: Beta 1 aftermath |
Confirmed bugs: Facesuck seems to be totally whacked in Ion mode (it works fine in Baseline only mode). IonPower passes all the JIT tests, though, so this must be something that the test suite does not cover. I'm investigating some other conformance test suites and corrected a couple other variances so far between Baseline and Ion but none of them appear to be what's ailing Faceblech yet.
Also, we have another web font that makes ATSUI puke, except it has an inconveniently null PostScript name so we can't filter it with the existing method. Fortunately Tobias had come up with an alternative font filter system some time ago that should work with 10.4.
Not confirmed (yet?): a few people have reported that memory usage skyrockets upon quit and the browser crashes (inevitably after exceeding its addressing space), on a variety of systems, both 10.4 and 10.5. I can't reproduce this on any of the test machines.
I need to do more looking into the stored passwords question.
Since we're out of runway, i.e., ESR31, and we need one more beta before release, I'm going to keep working on the Facebork problem (or at least try to fix it by fixing something else) until July 24. If we can't do it by then, I guess we launch without IonPower, which is unfortunate and will regress JavaScript performance, but we will still at least have Baseline. Faceburp is just too big to debug in place, so I need you folks to find another site that has similar problems. I haven't been able to yet myself.
http://tenfourfox.blogspot.com/2015/07/beta-1-aftermath.html
|
Nicholas Nethercote: Compacting GC |
Go read Jon Coppeard’s description of the compacting GC algorithm now used by SpiderMonkey!
https://blog.mozilla.org/nnethercote/2015/07/07/compacting-gc/
|
John O'Duinn: “Hot Seat: The CEO Guidebook” by Dan Shapiro |
This book just came out and I loved it. If you are starting a company, or thinking of it, you need to read this book. Period.
Dan covered a whole range of topics very succinctly, and in easy-to-follow language. When and how to raise funds. What all those terms mean. Who should (and should not!) be on your board, and why. How to allocate shares and ownership between co-founders. Where to incorporate your company (Dan has strong opinions on this!). How to create (and then also maintain) company culture. A great section on decision making. A section on “Hiring” in the context of the Manhattan project vs the moon shot Apollo project that I think every engineering hiring manager should read before building a team. Several true stories about startups where co-founders mismatches caused company threatening problems (trivia: 6 of 10 startups lose a co-founder in early days). And some good (and bad!) stories of how important trust was.
Some great quotes that resonated with me:
“You have limited resources of time and money. When they run out, you go bankrupt. The important thing is not cost/benefit: it’s opportunity cost.”
(in the context of how much travel was needed for all the in-person meetings with investors when raising funding) “…Alaska Airlines gave me MVP status for my efforts. In January.”
“Entrepreneurship is the pursuit of opportunity without regard to the resources currently controlled”. Prof Stevenson, Harvard.
In a variation of the “fail fast” mantra in developer circles, Dan notes that “…while it might seem like cold comfort now, the sooner you fail, the sooner you can try again.” Oh, and he’s not just saying it – that was the ending of a chapter where he detailed the failure of one of his startups.
His tolerance for large volumes of coffee and pointer to suggested reading “Coffee, CYP1A2 Genotype, and Risk of Myocardial Infarction” was a great and unexpected tangent for me personally. (More info here Journal of American Medical Association)
“Startups don’t out think their competitors; they out-execute them.”
“If leadership is the forest, then management is the trees. Day to day, it’s what consumes your time, and its imperative that you get it right.”
It takes skill and seasoned-experience-in-the-field to have one person cover all these different topics. Even more skill to do so clearly, and concisely. Putting them all together in a way that makes sense was great. Just great. If you are starting a company, or thinking of it, you need to read this book. Period.
Aside: Having this on my kindle app, on my trusty nexus5 phone was quite a good reading experience. The book was written in short, digestible chapters, which I could quickly complete standing a store line, or in the back of a taxi between meetings. It also encouraged me to think more about the chapter I just finished in the time before I got to stop and read some more. A nice way to digest the many lessons in here. I’m still experimenting with what books I find do work best on phone+kindle vs ink-on-paper, but at least for this book, reading on kindle worked for me.
(Disclaimer: I bought this book because I’m starting my own company, and that is the basis of the above review. As this book is published by O’Reilly Press, it feels important to disclose that I am also currently doing some work with O’Reilly… which did not influence anything I wrote here.)
http://oduinn.com/blog/2015/07/06/hot-seat-the-ceo-guidebook-by-dan-shapiro/
|
Chris Cooper: Releng & Relops weekly highlights - July 3, 2015 |
Welcome to the weekly releng Friday update, Whistler hangover edition.
Half the team took time off after Whistler. With a few national holidays sprinkled in, things were pretty slow last week. Still, those of us who were still around took advantage of the lull to get stuff done.
Taskcluster: Our new intern, Anthony Miyaguchi, started in San Francisco and will working on crash symbols uploads in TaskCluster. Our other intern, Anhad, has almost finished his work migrating spidermonkey to taskcluster. Morgan and Jonas are investigating task graph creation directly from github. Dustin continues to make efficiency improvements in the Fennec Taskcluster builds.
Modernize infrastructure: Mark, Q, and Rob continue to work on standing up our new Windows build platform in AWS. This includes measuring some unexpected performance improvements.
Improve release pipeline: We’re standing up a staging version of Ship-It to make it easier to iterate. Ben’s working on a new-and-improved checksum builder for S3, and Mike fixed a problem with l10n updates.
Improve CI pipeline: Jordan pushed the archiver relengapi endpoint and client live. They are now being actively used for mozharness on the ash project branch. catlee deployed the hg bundleclone extension to our Mac and Linux platforms, and Rail deployed a new version of fun size with many integrity improvements.
Release: Firefox 39.0 is in the wild!
Tune in again next week!
…
And here are all the details:
See you next week!
|
Air Mozilla: Tech Talk: The Power of Emotion and Delight |
Ricardo Vazquez will be speaking on, "The Power of Emotion and Delight: Microinteractions."
|
Air Mozilla: Mozilla Weekly Project Meeting |
The Monday Project Meeting
https://air.mozilla.org/mozilla-weekly-project-meeting-20150706/
|
About:Community: MDN Fellows Successfully Oriented |
The first-ever cohort of MDN Fellows convened at the Mozilla Vancouver space the weekend of June 20. This MDN Fellowship Pilot is an experiment for Mozilla to engage advanced web developers in strategic projects at Mozilla to advance our teaching and learning objectives.
Day 1: Learning About Learning
One of the first things we did was to collectively identify shared goals for the weekend:
And Day 1 was an exercise in integrity: because one of the least effective ways people learn is by lecture – and since we wanted our fellows to learn about learning – we all jumped in and engaged with learning content. Bill Mills, Community Manager for Mozilla’s Science Lab, conveyed several principles. A few nuggets that our teams have already started to apply to their projects:
Day 2: Getting to Brass Tacks
Day 2 had our Fellows applying their new knowledge to their own projects. They developed a plan of attack for their respective work for the remainder of the Fellowship. Some highlights:
The Curriculum team was well-served by referencing the Dunning-Kruger effect in designing its pre-requisites list. Specifically, they decided to parse this out using a “get information as you need it” approach for the pre-reqs rather than present their potential instructors with one long daunting list.
Both the Service Workers team and the WebGL team are embracing the above-mentioned concept of modularizing their content to make it more manageable. Specifically, Service Workers will create different approaches for different use cases to accommodate the evolving nature of its nascent technology; and WebGL will parse out different components so instructors and students can create reusable hackable code samples.
The Test The Web Forward team is employing “reverse instructional design” so its instructors can help others understand how problems are solved a step-by-step basis that students can dissect rather than simply see the final ‘answers.’ If you’ve heard of “reverse engineering” then “reverse instructional design” should make sense.
The Web App Performance Team, taking into consideration the complexity of performance and the difference of optimizing the network vs the front-end, will compartmentalize the courses. To keep the introductory course short & crisp, and to further help trainers & students to master performance, each module will have an advanced follow-up. Examples of bad and good performance are linked throughout the course along with practical code to best showcase these performance tactics.
How MDN Fellows Support the Mozilla Mission
Last year MDN began working with our colleagues at the Mozilla Foundation to see how we might partner to advance our common goals of growing web literacy. The work MDN is doing to expand beyond documentation and into teaching and learning dovetails nicely with the Foundation’s efforts to harmonize Mozilla’s learning and fellowship programs. This is a work in progress and we expect our MDN Fellows to play a key role in informing this.
http://blog.mozilla.org/community/2015/07/06/mdn-fellows-oriented/
|