Yesterday, I introduced my (ongoing) project to set up a low-latency video stream with VLC. Today it's time to talk a bit about the overall architecture, and the signal acquisition.
First of all, some notes on why we're using VLC. As far as I can remember, we've been using it at TG, either alone or together with other software, since 2003. It's been serving us well; generally better than the other software we've used (Windows Media Encoder, Wowza), although not without problems of its own. (I remember we discovered pretty late one year that while VLC could encode, play and serve H.264 video correctly at the time, it couldn't actually reflect it reliably. That caused interesting headaches when we needed a separate distribution point outside the hall.)
One could argue that writing something ourselves would give more control (including over latency), but there are a few reasons why I'm reluctant:
- Anything related to audio or video tends to have lots of subtle little issues that are easy to get wrong, even with a DSP education. Not to mention that even though the overarching architecture looks simple, there's heck of a lot of small details (when did you last write a TS muxer?).
- VLC is excellent as generalist software; I'm wary of removing all the tiny little options, even though we use few of them. If I suddenly need to scale the picture down by 80%, or use a different input, I'm hosed if I've written my own, since that code just isn't there when I need it. We're going to need VLC for the non-low-latency streams anyhow, so if we can do with one tool, it's a plus.
- Contributing upstream is, generally, better than making yet another project.
So, we're going to try to run it with VLC first, and then have “write our own” as not plan B or C, but probably plan F somewhere. With that out of the way, let's talk about the general latency budget (remember that our overall goal is 80ms, or four frames at 50fps):
- Inputting the frame: 20ms. While SDI is a cut-through system, there's no way we can get VLC to operate on less than a whole frame at a time, and the SDI card doesn't support it anyway as far as I know. (SDI sends the frame at the signal rate, so it really takes one frame to input one frame.)
- H.264 encoding: 10ms. We assume we can encode the frame at twice the realtime speed in acceptable quality. This probably requires a relatively fast quad- or octocore.
- Network latency: 5ms. Since we're only distributing within the local network, bandwidth is essentially free, but there's always going to be some delay in various networking stacks, routers, etc. anyway, and of course, outputting a megabit of data at gigabit speeds will take you a millisecond. (A megabit per frame would of course mean 50Mbit/sec at 50 fps, and we're not going that high, but still.)
- H.264 decoding: 10ms. We assume decoding can happen in twice the realtime speed; of course, decoding is usually a lot faster than encoding (even though VLC cannot, as far as I know, use H.264 hardware acceleration at this point); this should give us some leeway for slower computers.
- Client buffering: 20ms. We don't really have all that much control over the client, and this is sort of a grey area, so it's better to be prepared for this.
- Screen display: 20ms. Again, there might various forms of buffering in place here, not to mention that you'll need to wait for vertical sync.
As you can see, a lot of these numbers are based on guesswork. Also, they sum up to 85ms — slightly more than our 80ms goal, but okay, at least it's in the right ballpark. We'll see how things pan out later.
This also explains why we need to start so early — not only are there lots of unknowns, but some of the latency is pretty much bound to be lurking in the client somewhere. We have full control over the server, but we're not going to be distributing our own VLC version to the 5000+ potential viewers (maintaining custom VLC builds for Windows, OS X and various Linux distributions is not very high on my list), so it means we need to get things to upstream, and then wait for the code to find its way to a release and then down onto people's computers in various forms. (To a degree, we can ask people to upgrade, but it's better if the right version is just alreayd on their system.)
So, time to get the signal into the system. Earlier we've been converting the SDI signal to DV, plugged it into a laptop over Firewire, and then sent it over the network to VLC using parts of the DVSwitch suite. (The video mixing rig and the server park are physically quite far apart.) This has worked fine for our purposes, and we'll continue to use DV for the other streams (and also as a backup for this one), but it's an unneccessary part that probably adds a frame or three of latency on its own, so it has to go.
Instead, we're going to send SDI directly into the encoder machine, and multicast straight out from there. As an extra bonus, we'll have access to the SDI clock that way; more on that in a future post.
Anyhow, you probably guessed it: VLC has no support for the Blackmagic SDI cards we plan to use, or really any SDI cards. Thus, the first thing to do was to write an SDI driver for VLC. Blackmagic has recently released a Linux SDK, and it's pretty easy to use (kudos for thorough documentation, BM), so it's really only a matter of writing the right glue; no reverse engineering or kernel hacking needed.
I don't have any of these cards myself (or really, a desktop machine that could hold them; I seem to be mostly laptop-only at home these days), but I've been given access to a beefy machine with an SDI card and some test inputs by Frikanalen, a non-commercial Norwegian TV station. There's a lot of weird things being sent, but as long as it has audio and is moving, it works fine for me.
So, I wrote a simple (input-only) driver for these cards, tested it a bit, and sent it upstream. It actually generated quite a bit of an email thread, but it's been mostly very constructive, so I'm confident it will go in. Support for these cards seems to be a pretty much sought-after feature (only VLC support for them earlier was on Windows, with DirectShow), so a lot of people want their own “pony feature” in, but OK. In any case, should there be some fatal issue and it would not go in, it's actually not so bad; we can maintain a delta at the encoder side if we really need to, and it's a pretty much separate module.
Anyhow, this post became too long too, so it's time to wrap it up. Tomorrow, I'm going to talk about perhaps the most obvious sort of latency (and the one that inspired the project to begin with — you'll understand what I mean tomorrow), namely codec latency.