Steinar H. Gunderson

Sun, 02 Feb 2014 - FOSDEM video streaming, post-mortem

Wow, what a ride that was. :-)

I'm not sure if people generally are aware of it, but the video streaming at FOSDEM this year came together on extremely short notice. I got word late Wednesday that the video team was overworked and would not have the manpower to worry about streaming, and consequently, that there would be none (probably not even of the main talks, like last year).

I quickly conferred with Berge on IRC; we both agreed that something as big as FOSDEM shouldn't be without at least rudimentary streams. Could we do something about it? After all, all devrooms (save for some that would not due to licensing issues) would be recorded using DVswitch anyway, where it's trivial to just connect another sink to the master, and we both had extensive experience doing streaming work from The Gathering.

So, we agreed to do a stunt project; either it would work or it would crash and burn, but at least it would be within the playful spirit of free software. The world outside does not stand still, and neither should we.

The FOSDEM team agreed to give us access to the streams, and let us use the otherwise unused cycles on the “slave” laptops (the ones that just take in a DV switch from the camera and send it to the master for mixing). Since I work at Google, I was able to talk to the Google Compute Engine people, who were able to turn around on extremely short notice and sponsor GCE resources for the actual video distribution. This took a huge unknown out of the equation for us; since GCE is worldwide and scalable, we'd be sure to have adequate bandwidth for serving our viewers almost no matter how much load we got.

The rest was mainly piecing together existing components in new ways. I dealt with the encoding (on VLC, using WebM, since that's what FOSDEM wanted), hitting one or two really obscure bugs in the process, and Berge dealt with all the setup of distribution (we used cubemap, which had already been tuned for the rather unique needs of WebM during last Debconf), parsing the FOSDEM schedule to provide live program information, and so on. Being a team of two was near-ideal here; we already know each other extremely well from previous work, and despite the frantic pace, everything felt really relaxed and calm.

So, less than 72 hours after the initial “go”, the streaming laptops started coming up in the various devrooms, and I rsynced over my encoding chroot to each of them and fired up VLC, which then cubemap would pick up and send on. And amazingly enough, it worked! We had a peak of about 380 viewers, which is about 80% more than the peak of 212 last year (and this was with almost no announcement before the conference). Amusingly, the most popular stream by far was not a main track, but that of the Go devroom; at times, they had over half the total viewers. (I never got to visit it myself, because it was super-packed every time I went there.)

I won't pretend everything went perfect—we found a cubemap segfault on the way, and also some other issues (such as initially not properly restarting the encoding when the DVswitch master went down and up again). But I'm extremely happy that the video team believed in us and gave us the chance; it was fun, it was the perfect icebreaker when meeting new people at FOSDEM, and hopefully, we let quite a few people sitting at home learn something new or interesting.

Oh, and not the least, it made my own talk get streamed. :-)

[22:06] | | FOSDEM video streaming, post-mortem

Steinar H. Gunderson <sgunderson@bigfoot.com>