Multicast IP Video
Transcript
we go hi it's keith powder from the CTO advisor with a you know what hybrid this is something that I used to do all the time overall in virtualized geek.com have these tech talks if you're watching this you you might have found this via the post i just did on the CTO advisor calm talking about architects and troubleshooting high level systems or distributed systems the distributed system that i gave an example of was an IP webcast system very very complex set of problems a very high
risk from a business perspective a lot of attention the project received that was I think a great insight into what it functionally means to be an architect I think this is more of a for of the geeks who were curious as to what the exact technical problem was what was the solution and how do we get there let's have that discussion so let's draw out what we had going on from a technology perspective so if you read the blog post you read that basically I had
two different video sources one was a studio and the oven was a HD teleconference videos solution in this studio had a connection to a networking holder a video encoder basically a codec that converted the analog signal went to a digital signal that our system could interface with that gap connected to via a network a media server that media server connected over a one gigabit connection to our enterprise data center and then at our enterprise data center we had our network which was basically the network core
will just give to the network or the network core was a pair of Cisco 7 k's event from there we went to a web server and then from the web server we went out to the win which was a by the way your frame i didn't put in the notes this is frame relay didn't remember a fee wasn't a tent remember was frame relay or ATM i'm getting old and then from there we went out to various sites including back to the studio which had a
bunch of participants watching in their data centers or in there are different offices and then we had a roughly 85 win sites and then these websites the configuration was pretty typical we had a cisco 6509 class closet switch for terminating the connection obviously not turning in connection obviously we had different answer otters there but the symptoms were pretty much all the same so it really doesn't matter and then this HD connection might have been a remote site with a hanging off a 65 load just simplified
at 6509 and a bunch of end users there as well so relatively simple concept from the two video sources we combine those that the wind fed it back to the DC the DC at the DC we had the media encoder which encoded the media center to the web server the web server feed it back out to the wind and then to the locations video at this age is isn't complex in 2012 this is circa 2012 you know we knew how to do video uh well then
prioritize video there you if you remember if your networking guy you remember there was a rush to do qos everywhere this was probably circuit 2009 when everyone was saying you know what qos all the way to the edge that i did had just started dying down and we've started to apply qos where it made sense versus seek us at you know at the 6509 qos in the data center that it just became a nightmare to manage from a configuration perspective it also was no subjective what
was called qos on a switch versus what's qsr on router what what techniques are you using to implement qos that wasn't you know it wasn't qos was more of a than it was a defined science the problem that we were in countering was that video if I met a remote site and I connect to this web server via IP just pure tcp/ip unicast I had a great viewing experience however if I'm going to serve video to 12,000 workstations and an enterprise over t1 anything ranging from
a t1 all the way up to a ds3 I cannot I don't have enough bandwidth to to send all that video traffic via IP I'm going to use a multicast technology in an enterprise this is the advantage of enterprise-class network I control the network for the most part and if I want to use multicast to have a simultaneous viewing experience multicast is most definitely a great option it limits the amount of IP traffic that traverses the network and more or less everyone should have a pretty
good viewing experience so what we experienced is that in workstations in the DC video more or less was fine when it hit the DC and it went out to the closet switches that were in the same physical network as the DC everything seemed to be fine at headquarters where we would have to traverse back over the wind the experience wasn't good but however it was extremely again if I this one could get big connection if I did I PCAST over that network everything was fine so
what I told us that the problem was most definitely proudly in the network however if you read the blog post you knew we got an awful lot of awful lot of attention when it came came to this problem so we needed to attack it from a bunch of different areas we contacted several people from our media encoder folks to Cisco to the of when provider we got the desktop team involved we got the company that made our interface for our web application involved we brought all
these folks to kind of a troubleshooting the war room session over this is over a couple of months this is not a one-time event start to dig in it the cisco we paid a good amount of money for a cisco consultant to come in and tell us that our 6509 are configured correctly our edge devices are configured directly for a multicast they looked at there was no packet overruns at the on the routers there was no packet overruns on the closet if switches then colder guys
would come in say you know what their encoders are not breaking a sweat you guys are really not doing anything special this is HD video you're doing standards for 20 p video so these problems you're seeing we shouldn't be seeing them it's probably in the network again we kept drilling down the team you know in our troubleshooting sessions basically you know we gear up for these big town hall events these huge multicast sessions one of our biggest challenges was reproducing the problem in a test environment
since we're using windows media server to serve out the content and we were also using windows media player to consume the content only it was an embedded player how do you generate that much traffic kind of ad hoc we could have you know we tried a ton of things we had a couple of people at each site run video so you know from a moat form of pure multicast if the video is making it over the wind and trying to eliminate the win as a bottleneck
we had just folks and as many sites as we could start up the video and you know what this was distress the system at the most to about 50 sites so a 50 sites could simultaneously poor video over multicast and experience was good you know we that's when we kind of started to shake our stretch scratch our heads we couldn't see packet loss we couldn't see drops on the edge routers on the cisco closet switches we just couldn't replicate the problem until one of the engineers
came up for bright idea he used a net is a networking network agent to actually spin up media player windows server on a on all the workstations and enterprise so we actually had as many as 12,000 machines as many of the machines that that were left kick up web um the web interface to this application starts streaming and start seeing where it broke so we could successfully recreate the problem which is a critical portion of troubleshooting the distributed system is to be or any system is
to being to recreate the problem we're finally able to recreate the problem but the secondary problem that we had is that we still didn't know why the data that our systems were given us didn't tell us what the problem was we looked at our network equipment when we looked at the workstations the systems seemed in it kind of in range of the performance range that you would expect for multicast there was no packet overruns CPU utilization on the edge routers and the 6509 we're fine everything
looked great until one of the smarter engineers and these are all smart guys notice that the notice a pattern that the stuttering and the poor video quality only happened during rapid movement on the video so if you know anything about video encoding if you do what's called constant bit rate versus variable bitrate there's a trade-off between the two bitrate broadcasting I can create a constant kind of size of my package size and shape of my packets and even out the traffic in variable big bit the
disadvantage to that is that traffic use is inefficient so if I'm sinning this whiteboard picture and this whiteboard pitcher doesn't change then that's a waste of bandwidth I should never resend this image if this image this portion of the image doesn't change constant bit rate would constantly send this whiteboard image over and over again inefficient user space but consistent variable bit rate would only change when we send the changes to the video so as my arm is moving in this video variable bit rate with change
would send that traffic over so when we saw either camera camera movement rapid change in the presenters that's when we noticed that there was a problem with video performance so that would lend us to say that somewhere either the network then cold or something isn't handling that that rate of change ultimately what we've discovered what it was is that the 6509 we're getting overrun if you think about what happens when you do multi casting you create a packet that single packet is copied across all the
ports one they on the destination switch for the for the users who were consuming what we discovered that when we enter one of these variable bit rate spikes the 6509 s would not be able to copy the all the multicast packets at one time solution replace all the 6509 not realistic 6509 s were already ultra expensive closet switches and replacing them just what weren't an option over 85 sites and then this problem also experienced itself in lower class which is where we had like.we you know
3500 series switches it was a consistent problem throughout the cisco universe what we ended up doing was we put a 65 935 90 I forget the motto of switch but basically what the switch did was it could take the that variable bit rate knock it down into fragments that were basically buffer the transmission of though of those high utilization or that those spikes Bakley smooth out those pipes sitting over the wind and when the edge switches receive those multicast packets they were brought in a bit
rate that the switches could actually handle to reproduce problems all we won the day we save the contract blase blase blase that's not the important part the important part is that when you troubleshoot a distributed system like this you have to understand all the different bottlenecks the points of pain are the attributes of the application so in this case we're talking about IP video this could happen to any application not just IP video it could be SI p telephony anything that has a certain characteristic we
need to understand those characteristics what could impact the performance of the application and how do we measure in other pieces how do we measure a kind of success so throughout this whole process we develop all those processes for how to troubleshoot IP multicast video extremely difficult from a technical perspective extremely difficult from a business and political perspective but one of the more rewarding troubleshooting events of my career I'll forever remember this design out forever remember the experience in the guys on the team that helped made
it happen that's it for this pseudo tech talk I actually enjoyed doing the tech talk for the first time in a long time follow me on the twitters i'm at the CTO I'm sorry at sitio advisor on the web I'm the CTO advisor com thanks
