What is so difficult about broadcasting MLB.TV?

As a member of the development team for the Major League Baseball MLB.TV project, I’ve had a somewhat unique perspective on the successes and failures of the project. There is a bit of backlash in the blogosphere about the technologies surrounding the players, and the inherent superiority of one technology over the other. This post examines only the technologies involved in the situation, and not any of the underlying business decisions surrounding it.

So, what is so difficult about broadcasting MLB.TV? Funny you should ask, regardless of all the Flash vs Silverlight debate that has surrounded the MLB.TV launch this year, the player itself represents only about 10% of the complexity. So, how does it all work? There are a series of inter-related processes involved.

  1. Acquisition of feeds – Home and away video feeds from the ball park are provided by the networks covering the games. If there are problems with those feeds, the video/audio never makes it to MLB. As an example of this, in one of the opening games, an unplugged cable in the feed truck was responsible for lack of an audio feed in the MLB.TV player. There are any number of things which could go wrong at the ball park or in the broadcast trucks to provide the signal. Assuming all is acquired properly, the next step is…
  2. Encoding of feeds – The video and audio content streams from the broadcasters at the ballpark across the network to the MLB Advanced Media offices in New York City. There, the data is encoded into various formats for the various players, including MLB.TV, Gameday Audio, iPhone, etc. For MLB.TV, the stream is encoded at 7 different bit-rates, so the proper quality of video can be delivered, based on the end users connection speed to the internet.
  3. Provisioning – As a game starts, all of the data about the game and the feeds available for it needs to be provisioned into internal systems at MLB. This data is used by all the various applications to determine what game’s are available, what feeds are available for each game, and which qualities are available for each feed. If one of the encoders fails, or one of the feeds drops, or anything else goes wrong in steps 1 and 2, the data needs to be re-provisioned to reflect the current accurate data for a game.
  4. Services – There are a series of Java based services which consume the provisioned data, and make it available to the player and other applications. Amongst the jobs the services provide are login, authentication, geo detection (to help enforce blackout restrictions), managing stream security and more. Of course, these services are dependent on the data about each game being available and accurate.
  5. Streaming Servers – There are dozens of Flash Media Servers at the Content Delivery Network (CDN) partner of MLB, from which all the video and audio (and lots of other things too) are streamed to the end user. The CDN has the capacity to stream hundreds of thousands of games at once.
  6. MLB.TV player – While this is the only part of the process most end users see, this is merely a front end to all the systems described above. While we are constantly finding problems, fixing them, making improvements, testing to ensure we haven’t caused other problems, 99% of all bugs reported in the Forums and MLBlog actually relate to one of the “upstream” elements of this system. As I mentioned earlier, an unplugged cable caused hundreds to complain that our flash version of the media player couldn’t properly play sounds. Obviously, that wasn’t the problem, as all the other games, and even the other feed for that game properly had the audio channels playing, it’s simply a matter of the player playing the content which was provided to it.

So, let’s stop all the Silverlight vs Flash bickering shall we? Both are capable of doing the job. This year, MLB chose flash, last year they chose Silverlight. Neither decision should be used to bludgeon the other over the capabilities or lack thereof that the technology represents. Both would have a similar set of problems displaying streams that aren’t properly acquired or encoded. Both would similarly display correct or incorrect information about blackouts, based on what the services layer provides.
All told, I’m honored to be a part of the team which made this all possible, am proud that the vast majority of users are able to view high quality content of the games as they are played, and will work tirelessly to help fix the problems reported by the others.

19 Comments

    So what excuse will you use for the quality of MLB.TV now that the season has started? 2 games in and MLB.TV with Nexdef is no more than Alpha software. Fans pay a premium for a premium service but are unable to use any of it.
    Like I said… 2 games into the season and I haven’t been able to watch a full inning without failures.
    Enjoy this article and know that most of the MLB.TV subscribers experienced the same and feel the same way as the author.
    http://bleacherreport.com/articles/152306-mlbtv-strikes-out-on-opening-day.

    I won’t be offended in you not posting this response. I just had to inform you about the truth.

    • by Anthony
    • 9:04 pm, April 8, 2009
    • Reply

    Just a quick question… Where does nexdef fall into all of this then? Everything has run fine with nexdef feeding the highest quality until the servers get hammered and more than 3 games are being streamed at a time. During those busy times, the service with nexdef enabled falls apart and is unwatchable.

    Thanks for posting this, it really goes to show people how much really has to go right to get a feed from the field onto our PC’s. Thanks for the hard work and I’m confident things will work out soon.

    • by Steind
    • 5:04 pm, April 8, 2009
    • Reply

    To quote Dagny Taggart, "I asked you to do a job, not to do your best. Whatever that is."

    Also, there are infinite complaints about the service from the MLB end, including the whole idea of charging people extra for the premium service that is not delivered. Both sides are pretty terrible here.

    • by William Brinkman
    • 5:04 pm, April 8, 2009
    • Reply

    So what is your explanation for the widespread NextDef problems experienced by
    thousands of customers since the regular season began and MLB Support’s complete
    refusal to address the issue? Thousands have reported choppy video when using
    NextDef and nothing is being told to any of us despite the fact that we paid extra
    money for the premium service you advertised. The support’s laughable response has
    been to reboot your computer and clear cache. Please start taking the problems of
    taking paying customers seriously and then we’ll stop "bludgeoning" MLB.tv. All we want
    is it to work!

    Great post. This is also what i had heard but it’s good to have someone from the team confirm it. It’s sad that some websites that are supposed to be impartial (CNET amongst others) have turned into a mere gathering of gossips and trying to attract views by reporting unverified claims. Thankfully we have moderate informed people to correct the idiotic assumptions.

    Thanks,
    Laurent

    Nice try buddy… the insecurities are more fun to sequester if you bash another’s technology with like-minded invidiuals, but hey, A for effort in trying to set the record straight!

    Interesting insight there. For me, MLB.TV is a life savor, I can’t get the Extra innings package and my favorite team is out of market (Braves). There have been a few problems early on like I expected but today it has worked great. I am very impressed with all the cool features and the great quality of live streaming video. Keep up the good work! :)

    • by John
    • 7:04 pm, April 9, 2009
    • Reply

    Great insight to the process Jeff.
    So, tell us what business decisions made you go from Silverlight to Flash 😉

    • by Hilary
    • 6:04 am, April 10, 2009
    • Reply

    Steind –

    NexDef actually intermingles at a few different places here, including a separate set of encoders, a different set of streams from the CDN, and a different process running on the individual users machines. For NexDef, effectivly, the flash player asks the local process for the content, and that local process manages requesting the appropriate streams from the CDN, allowing for it to pause, rewind, etc.

    James –

    If you see my comment to Steind, I give a bit of insight into what the NexDef process is. That too has a number of places where things can go wrong. But believe me that all the various teams are continuing to work to make the experience better.

    Anthony –

    I dont make any excuse for anything that has happened. It may come as news to you, but software always has bugs, and sometimes doesnt work right. Bugs in software is not unique to MLB.TV, Flash, SilverLight, Windows, MacOS, Linux, or any other piece of software you may have ever used. We, as software developers continue to try and fix the bugs that exist in our software, while trying not to introduce new ones.

    However, even if there arent any bugs whatsoever (as unlikely as that may be), I’m trying to point out in this article all the various things that can still go wrong in the process. So, specifically, poor quality could be a result of:
    1) Bad Feeds From the Ball Park
    2) Excessive Internet Traffic Between Ballpark and Encoders
    3) Bad Encoding
    4) Excessive Internet Traffic between encoders and CDN
    5) Problems at CDN
    6) Excessive Internet Traffic between CDN and your location
    7) Slow internet connection on your machine
    8) Other software on your machine running slowly, so the player doesn’t have adequate CPU to run
    probably a bunch of other things i cant think of too.

    Anyhow, no excuses, but from your post I’m not sure that you realize that the vast majority of users are watching games daily with very few issues? I’m personally sorry ( I’m not in a position to apologize on MLB’s behalf ) that you have issues with the player.

    As frustrating as it is to hear though, you should publish any specifics about these issues on the Official MLB Support Forums, so they can be reviewed and added to our official bug queue. My blog is not an official support channel, and while I take things posted here seriously, I doubt "the powers that be" are looking here for bug reports.

    Jester –

    im as much for poking at others insecurities as the next guy, but I prefer to keep it on a personal level. So whats up with that shirt?

    No seriously, it wouldnt take any one reading this more than a few clicks to figure out where i sit in the whole flash v silverlight thing, but was trying to get above the fray on that one for once.

    John –

    It is great to hear from folks who are having a good experience. I’m lead to believe that this is the case for the majority of users, however, most for whom it just works arent posting much about it online, resulting in lots of posts about the problems and few about what does work. Not surprising, thats just the way the world works.

    Anyhow, thanks for posting, and glad you are enjoying it

    Hilary –

    Even if I knew that answer, I couldnt tell you. All I really know though is my company (Digital Primates) was hired to help with the Flash front end on this project. By the time we were hired, MLB had already made the decision to switch from Silverlight.

    Honestly, we’ve been involved in a number of projects like this where a client was switching from SilverLight (or AJAX, or JavaFX, etc.) to Flash, just like many former clients of ours have switched from Flash to one of the other technologies. Each technology has a different place where they excel, so there is no one right answer for every project. (Jesse, am i being too technologically politically correct again?)

    Good on ya, Jeff. I just know while you’ll be praised for trying to set the record straight, a lot of people have vested interest in bashing technologies; whether it’s to bash a competitors, or to simply bash another to make oneself feel better. Hopefully this’ll get higher in the search engine responses.

    I am a customer of the Gameday Audio instead of the Video product. I have listened to streaming audio of baseball games for many years (from the time when the games were broadcast by the stations themselves and not by MLB). This year, the quality took a severe nosedive. I have never had problems before but this year I rarely have success listening to a whole game. The stream constantly stops or the quality is so bad that I have to restart the player. I’m sure that it is possible to deliver a good audio stream with Flash, but MLB has not done it. Yes I’m sure it is complicated, but that is why MLB GETS PAID TO DO IT! The fact that they delivered the product in past years but can’t do it this year is a sign that something is terribly wrong. I am talking afterall about a simple audio stream — something that has been possible for many years. This is not cutting edge technology.

    • by qbfine
    • 8:06 pm, June 17, 2009
    • Reply

    qbfine – you should contact customer support. i am not an official support channel, but i can tell you that the vast majority is able to play gameday audio without issue. I suspect that a quick call to MLB.com customer support can most likely help you through whatever issues you may be having.

    Good luck!

    jeff

    Hi. I am a fan of MLB. I really like MLB.TV project. My opinion is such that this project is very successful.

    This is also what i had heard but it’s good to have someone from the team confirm it. It’s sad that some websites that are supposed to be impartial (CNET amongst others) have turned into a mere gathering of gossips and trying to attract views by reporting unverified claims.
    http://www.realtorrentz.com

    • by cavla
    • 3:06 am, June 29, 2010
    • Reply

Leave a Reply to Steind Cancel reply