Steinar H. Gunderson

Sat, 14 Jan 2012 - The art of streaming from a demoparty

As 2011 just has come to a close, it's pretty safe to say that video distribution over the Internet is not the unreliable, exotic mess it was ten years ago. Correspondingly, more and more demoparties (and other events, which I won't cover) want to stream their democompo showings to the outside world. Since I've seen some questions, notably on Pouët's BBS, I thought I could share some of my own experiences with the subject.

Note that almost all of my own experience comes from The Gathering (TG), which have been streaming video internally since at least 1999 and externally since 2003; information about other parties is based on hearsay and/or guesswork from my own observations, so it might be wrong. Let me know if there are any corrections.

With that in mind, we jump onto the different pieces that need to fit together!

Your ambition level

First of all, you need to determine how important video streaming really is for you. Unless you have a huge party, any streaming is going to be irrelevant for the people inside the hall (who are, after all, your paying visitors); it might have a PR effect externally, but all in all you probably want to make sure you have e.g. adequate sleeping space before you start worrying about streaming.

This is the single most important point of all that I have to say: Make sure that your ambitions and your expected resource usage are in line. TG has sky-high ambitions (going much broader than just streaming compos), but backs that up with a TV production crew of 36 people, a compo crew of thirteen, and two dedicated encoding people rotating through the day and night to make sure the streaming itself works. Obviously, for a party with 120 visitors, this would make no sense at all.

Relevant data point: Most parties don't stream their compos.

The input signal

Getting the signal in digital form, and getting a good one, is not necessarily easy.

First of all, think of what platforms you are going to stream from. Older ones might be very easy or very hard; e.g. the C64 has a not-quite-50Hz signal (it's timed to power) with not-quite-576i (it inserts a fake extra line in every other field to get a sort-of 288p50 display), which tends to confuse a lot of video equipment. Modern PC is somewhat better; most demos, however, will want to change video resolution and/or refresh rate on startup, which can give you problems to no end.

The ultra-lowtech solution is simply to put a video camera on a tripod and point it at the big screen. This will give a poor picture, and probably also poor sound (unless you can get a tap from your sound system), but remember what I said about ambition. Maybe it's good enough.

Larger parties will tend to already have some kind of video production pipeline, in which case you can just tap that. E.g., TG uses an HD-SDI setup with a production bus where most of the pipeline is native, uncompressed 720p50; Revision uses something similar in 1080p based on DVI. Also, a video mixer will allow you to fade between the compo machine and one showing the information slide without ugly blinking and stuff.

Such pipelines tend to include scan converters in one form or the other; if you can get your hands on one, it will make your life much easier. They tend to take in whatever (VGA/HDMI/composite) and convert it nicely to one resolution in one connector (although refresh rates could still be a problem).

At TG, we use the Blackmagic cards; although their drivers are proprietary, they're pretty cheap, easy to get to work under Linux (as we used), and the API is a joy to use, although it descends from COM. The documentation is great, too, so it was easy to cobble together a VLC driver in a few nights (now part of VLC mainline).

The cheaper alternative is probably DV. Many video cameras will output DV, and finding a machine with a Firewire input is usually not hard. The quality is acceptable, typically 576i50 with 4:2:0 subsampling (we did this at TG for many years). The largest problem with DV is probably that it is interlaced, which means you'll need to deinterlace either on the server or on the client (the latter is only really feasible if you use VLC). For most people, this means cutting the framerate in half to 25fps, which is suboptimal for demos. (All of this also goes for analogue inputs like S-video or composite, except those also have more noise. Don't confuse composite with component, though; the former is the worst you can get, and the latter is quite good.)

The software

The base rule when it comes to software is: There is more than one way to do it.

Revision uses Adobe's Flash suite; it's expensive, but maybe you can get a sponsored deal if you know someone. The cheaper version of this is Wowza; it gives you sort-of the same thing (streaming to Flash), just a lot cheaper. At least Assembly uses Wowza (and my guess is also TUM and several other parties); TG used to, but we don't anymore.

TG has been through a lot of solutions: Cisco's MPEG-1-based stuff in 1999, Windows Media streaming from 2003, Wowza some years, and VLC exclusively the last few ones. VLC has served us well; it allows us to run without any Windows in the mix (a win or a loss, depending on who you ask, I guess), very flexible, totally free, and it allows streaming to Flash.

Generally, you probably just want to find whatever you have the most experience with, and run with that.

The encoder

The only codec you realistically want for video distribution these days is H.264; however, there's huge variability between the different codecs out there. Unfortunately, you are pretty much bound by whatever software you chose in the previous step.

Encoding demos is not like encoding movies. Demos has huge amounts of hard edges and other high-frequency content, lots of rapid and unpredictable movement, noise, and frequent flashing and scene changes. (Also remember that there is extra noise if your input signal is analogue; even a quite high-end DV cam will make tons of noise in the dark!) This puts strains on the video encoder in a way quite unlike most other content. Add to ths the fact that you probably want to stream in 50p if you can (as opposed to 24p for most movies), and it gets really hard.

In my quite unscientific survey (read: I've looked at streams from various parties), Wowza's encoder is really poor; you don't really need a lot of action on the screen before the entire picture turns to blocky mush. (I'm not usually very picky about these things, FWIW.) The Adobe suite is OK; x264 (which VLC uses) is really quite clearly the best, although it wants your CPU for lunch if you're willing to give it that. Hardware H.264 encoders are typically about in the middle of the pack—“hardware” and “expensive” does not automatically mean “good”, even though it's easy to be wooed by the fact that it's a shiny box that can't do anything else. It's usually a pretty stable bet, though, if you can get it working with your pipeline and clients in the first place.

There are usually some encoder settings—first of all, bitrate. 500 kbit/sec seems to be the universal lowest common denominator for video streams, but you'll want a lot more for full 576p or 720p. (More on this below.) You can also usually select some sort of quality preset—just use the most expensive you can go without skipping frames. At TG, this typically means “fast” or “veryfast” in VLC, even though we have had quite beefy quad- or octocores. Also, remember that you don't really care much about latency, so you can usually crank up the lookahead to ten seconds or so if it's set lower.

However: No encoder will save you if some Danes throw a million particles into your compo, unless you have cranked your bitrate obscenely high. Sorry. The best you can hope for is mush.

The distribution

Mantra: The most likely bottleneck is the link out of your hall. The second most likely bottleneck you will have no control over. (It is the public Internet.)

Most parties, large and small, have completely clogged outgoing Internet lines. You will need to plan accordingly. (And even if MRTG shows your line is not clogged, TCP works in mysterious ways when traversing congested links.) By far the easiest tactic is to get one good stream out of the hall, to some external reflector; ask a bit around, and you can probably get something done at some local university or get some sponsorship from an ISP.

Getting that single copy out of the hall can also require some thought. If you have reasonably advanced networking gear, you can probably QoS away that problem; just prioritize the stream traffic (or shape all the others). If you can't do that, it's more tricky—if your reflector is really close to you network-wise, it might “win”, since TCP often fares better with low-latency streams, but I wouldn't rely on that. You could let them traverse entirely different links (3G, anyone?), or simply turn off the public Internet access for your visitors altogether. It's your choice, but don't be unprepared, because it will happen.

Note that even if you would have spare bandwidth out of your site (or if you can get it safely to some other place where the tubes are less crowded), this doesn't mean you can just crank up the bitrate as you wish. The public Internet is a wonderful but weird place, and even if most people have multi-deca-megabit Internet lines at home these days, that doesn't mean you can necessarily haul a 2 Mbit/sec TCP stream stably across the continent. (I once got a request for a version of the TG stream that would be stable to a small Internet café in Cambodia. Ideally in quarter-quarter-screen, please.)

There are two solutions for this: Either use a lowest common denominator, or support a variety of bandwidths. As mentioned before, the standard seems to be 360p or so in about 500 kbit/sec, where 64 kbit/sec of those are used for AAC audio. (Breakpoint used to change the bitrate allocation during music compos to get high-quality audio, but I don't know if Revision has carried this forward.) For high-quality 720p50, you probably want something like 8 Mbit/sec. And then you probably want something in-between those two, e.g. 576p50 (if you can get that) at 2 Mbit/sec. (I can't offhand recall anyone who streams in 1080p, and probably for good reason; it's twice the bandwidth and encoding/decoder power without an obvious gain in quality, as few demos run in 1080p to begin with.)

Ideally you'd want some sort of CDN for this. The “full-scale” solution would be something like Akamai, but that's probably too expensive for a typical demoparty. (Also, I have no idea how it works technically.) Gaming streams, typically with a lot more visitors (tens of thousands for the big competitions), are typically done using uStream or Justin.tv, but I've never seen a demoparty use those (except for when we used a laptop's webcam during Solskogen once)—the picture quality is pretty poor stuff, and even though they will work about equally for 10 and 10000 viewers, that doesn't really mean they work all that well for 10 always.

In short, the common solution is to not use a CDN, and just have a single reflection point, but usually outside the hall.

The player

The standard player these days is Flash.

I'm personally not a big fan of Flash; the plug-in is annoying, it necessitates Flashblock or Adblock if you want to surf the web without killing your battery, and it's generally unstable. (Also, it can't deinterlace, so you'd need to do that on the server if your stream is interlaced, so worse quality.) But for video distribution on the web, having a Flash client will give you excellent coverage, and these days, the H.264 decoder isn't all that slow either.

Adobe's own media suite and Wowza will stream to Flash natively; VLC will with some light coaxing. I don't think Windows Media will do, though; you'd need Silverlight, which would make a lot more people pretty mad at you (even though it's getting more and more popular for commercial live TV distribution, maybe for DRM reasons).

There's nothing saying you can't have multiple players if you want to make people happy, though. In some forms (HTTP streaming), the VLC client can play Flash streams; in others (RTMP), it's a lot harder. I doubt many play demoparty streams from set-top boxes or consoles, but a separate player is generally more comfortable to work with once you need to do something special, e.g. stream to your ARM-based HTPC, do special routing with the audio, or whatnot.

One thing we've noticed is that bigger is not always better; some people would just like a smaller-resolution stream so they can do other things (code, surf, IRC, whatever) while they keep the stream in a corner. One thing you can do to accomodate this is to make a separate pop-up player this has very low cost but huge gains, since people are free to move the player around as they wish, unencumbered by whatever margins you thought would be aesthetically pleasing on the main page. Please do this; you'll make me happy. :-)

If you have other formats (e.g. VLC-friendly streaming) or other streams (e.g. HD), please link to them in a visible place from the stream player. It's no fun having to navigate several levels of well-hidden links to try to find the RTSP streams. (I'm looking at you, Assembly.)

Final words

This section contains the really important stuff, which is why I put it at the end so you've gotten bored and stopped reading a while ago.

I already said that the most important advice I have to give is to align your ambition level and resources you put into the stream, but the second most is to test things beforehand. My general rule of thumb is that you should test early enough that you have time to implement a different solution if the test fails; otherwise, what good would testing be? (Of course, if “no stream” would be an acceptable plan B, you can test five minutes before, but remember that you'll probably make people more unhappy than if you never announced a stream in the first place, then.)

There's always bound to be something you've forgotten; if you test the day before, you'll be able to get someone to get that single €5 cable that would save the day, but you probably won't be able to fix “the video mixer doesn't like the signal from the C64, we need to get a new one”. In general, you want to test things as realistically as possible, though; ideally with real viewers as well.

Finally, have contact information on your page. There are so many times I've seen things that are really easy to fix, but nowhere to report them. Unless you actually monitor your own stream, it's really easy to miss clipped audio, a camera out of focus, or whatnot. At the very least, include an email address; you probably also want something slightly more real-time if you can (e.g. IRC). Also, if there is a Pouët thread about your party, you should probably monitor it, since people are likely to talk about your stream there.

And now, for the final trick: No matter if you have or don't have a video stream, you should consider inviting SceneSat. You'll need to ensure stable delivery of their stream out of the hall (see the section on QoS :-) ), but it gives you a nice radio stream capturing the spirit of your party in an excellent way, with an existing listener base and minimal costs for yourself. If you don't have a video stream, you might even get Ziphoid to narrate your democompo ;-)

[11:28] | | The art of streaming from a demoparty

Steinar H. Gunderson <sgunderson@bigfoot.com>