Internet Bandwidth Growth: Dealing with Reality Held during IETF 76, Hiroshima, Japan 11:45am - 12:45pm (UTC+9), Tuesday 10 November, during the IETF lunch break Orchid West Room, ANA Crowne Plaza Hotel For links to slides and audiocast archive, see: http://www.isoc.org/bandwidth LESLIE DAIGLE: All right. Is this on? All right. I think we're going to try to get started, and then, get settled. All right. So, thank you for coming to today's bandwagon briefing panel. Certainly very pleased with the interest in today's session. I did want to note that we are being live audiocast and the audiocast file and a transcript will be available for later download. I do also want to emphasize the session here today, although you may have stepped out of an IETF meeting, and this looks familiar to an IETF meeting room, this is not the IETF for the next hour. We have been transported elsewhere. This is just an informal ISOC organized discussion of the topic of bandwidth. And to emphasize that, at quarter to 1, we are going to retransport ourselves back to the IETF and clear this room in plenty of time for the next working group session that is supposed to be here and come in and set up. In the meantime, again, this is not a technical debate. It is an opportunity for information and discussion about the topic, and trying to take some of the discussions that we're having in working group sessions this week, and take them up to another level, and so that we can take it to the real world, so to speak. So, as you all know, hopefully you read the part of the message, the announcement of this, about the topic, and not just the free lunch. Today's topic is bandwidth in the Internet and the key question is: Is the Internet on the verge of collapse, again? Decision makers want to know if this is the network neutrality debate. Business decisions are being made based on what actually is happening with the Internet. By the way, what is the Internet anyway? So, questions include, where are the bottlenecks? What causes congestion? Is congestion in fact bad? What is the impact of traffic? What is being done about it? So we put together a panel here, people with real data and real network experience, and people working on technologies to make bandwidth use more effective and efficient. So, with that, just briefly introducing the panel, Kenjiro Cho will talk about research, his research on consumption of bandwidth in residential broadband. And Rich Woundy will talk about residential service providers and issues in managing broadband bandwidth. And Danny M cPherson has some observations to share. And Lars Eggert will talk about standards activities. Without further ado, I'll ask Kenjiro Cho to step up and tell us about it. KENJIRO CHO: So, I want to talk about trends in Japanese residential traffic. I'm a senior researcher can at IIJ and I'm also a board member of the WIDE project, the host of this IETF. And I have been involved in traffic measurement since 2004, using IIJ's data and also, we have collected data from other ISPs in Japan. And so, basically, background here is, this is a number of broadband subscribers in Japan. It reached 63 percent of households, and increased only three percent in 2008. So, as you can see, the, this is, green one is fiber to the home. And DSL red line is decreasing in, here in Japan. And 60 percent of the Internet traffic, backbone traffic in Japan is basically residential related traffic. So, our data collection experience is basically with six ISP. It started in 2004, covering 42 percent of Japanese traffic and it's a (inaudible) for ISPs. The goal here is to answer concerns about rapid growth of residential traffic. So, ISP's concerns often are not shared by other parties, because no data is available. So, concerns on technologies, fairness or net neutrality or profitability. So, although most ISPs collect data, data is seldom made available to others. And measurement methods and policies differ from ISP to ISP. So it's very difficult to compare. And, what is specific to Japan here is, we have higher penetration of fiber access, so that leads to a larger skew, so more traffic and bandwidth usage here. So, this graph shows the traffic growth. The left one is called backbone and the right one is for residential traffic. So, the red one is total growth, and the blue line is more interesting. It's annual growth rate. So for the last five years, we have very stable growth rate, around 40 percent. And this is the corresponding graph for residential traffic since 2004. And as you can see, the growth rate is very stable for the last five years. It's around 30 percent. So, changes in residential traffic has daily traffic and peak is at 9 p.m. to 11 p.m. And back in 2005, in and out were similar. Back then, we had peer to peer file sharing dominating the traffic. But now, difference became larger, and which suggests a shift from peer to peer file sharing to content services. So, this graph shows increase in daily traffic volume per user. X axis is daily traffic. Y axis is probability density. So this green line is for download and red one is for upload. So, download, the mode peak of the distribution moved from three megabytes per day, to 114 megabytes per day in 2009. And the average is much much higher. The average is about one gig, one gigabyte per day, because much of those is heavy hitters. Traffic is very skewed among users. And, this graph shows inbound and outbound traffic per user. And as you can see, there are roughly two classes. This one is the user downloading about ten times larger than uploading. And this one is symmetry, hybrid users. But there's no clear boundary between heavy hitters and others, and client type here. Client download, client type users and peer to peer type users. So, the reason is, most users use both client server type of applications and peer to peer style applications. They have different means. So, there are large diversity here. And, this one basically shows, if we look at the total volume, the traffic is dominated by TCP. But if we look at client type here, we just extract client type users, using less than 100 hundred megs, uploading less than 100, and here, UDP is, the whole data is increasing, and majority is peer to peer data. So, key observations here, is growth of Japanese residential traffic has been very stable at around 30 percent per year, for the last five years. And there is a shift in traffic patterns. Peer to peer file sharing is still dominant in volume. But a shift to content services is clear. And individual users have diverse traffic mix. And a few other observations: So high penetration of fiber access in Japan leads to larger skew in bandwidth usage among users, and we do have congestion issues in increasing mobile wireless access. And also, we have observed higher growth in international traffic for the last five years, but it is difficult to predict the future traffic, because it is significantly impacted by the behavior of the heavy hitters. And also, there are lots of factors that will affect the behavior of the users. Okay. That's it for me. LESLIE DAIGLE: Thank you. RICH WOUNDY: Hi, Rich Woundy. I'm going to give some perspectives and I think some of these slides, some folks might have seen them before. I stole at least one of these slides from the peer to peer infrastructure talk that Jason gave back in Cambridge back in 2008, and some of this you might have seen outside of the IETF. But this is just going to be a really high level overview. So, why does an ISP worry about congestion management? Why does it worry about bandwidth? Ultimately what it comes down to is, we have to worry about customer experience. We have to worry about where the Internet is going, and applications, and application diversity, which is a great thing, but then we actually have to make it work. And we have to make sure that we don't piss off our customers in the entire Internet community. Comcast now absolutely wants to not piss off the IETF community. So, why do we worry about this? We're always worried about what our customers think about our service, because if they don't like our service, they stop paying us, and that doesn't help anybody. We also worry about, you know, calling and complaining and saying, how come my service doesn't work? Or how come I was able to download at 12 megabytes per second last night and today I can only download 12 kilobits per second. So those things are happening and there's a whole diversity of applications. Sometimes they're more latency sensitive or more latency tolerant and it's more about throughput. That's what's more of interest. And the way that that manifests itself in our networks, particularly for parts of our infrastructure that didn't do queueing properly, would be that certain applications are filling up queues and driving down delay, and then that has a negative impact on the delay time, delay sensitive applications. That would be voice over IP, and that would be online gaming. There's a bunch of applications like that. And the ironic part is, we go and try to tune our network to make sure that Vonage and competing voice provider works, and then we have to worry about whether we've unfairly disadvantaged some competing video provider. That's the balancing act we have as an ISP. And so why do we have to worry about? The Internet community, government regulators, there's a whole FCC open Internet MPRN going on right now. And of course, if we're not selling Internet service, people aren't going to buy Internet service. So it's got to be Internet service. In terms of capacity brown-outs, we need to increase the capacity of our network. Congestion management is not about, you stop doing upgrades, stop adding capacity. It's about, how do you make sure that as do a reasonable upgrades schedule, that when Flash crowd incidents happen, or I don't know, a whole bunch of students come back after summer break and all of a sudden, they've all learned some new streaming applications, that just chews up all kinds of bandwidth and can take down our network in a matter of days. And it takes weeks to put out more capacity so that we can handle all those situations. So, with help of many folks in this room -- and by the way, this is now documented in an Internet draft that Jason and I had put out there, and I think some other folks here. This is the congestion management technique that we put out a year ago, and apparently is even being slashed this past week, which is kind of strange. I'll talk about that. So the goal is being able to provide consistent performance on our network, without looking at applications, without picking on peer to peer, without picking on any kind of traffic type, without trying to have deep packet inspection devices that are looking to see whether, whether we, the ISP, think it's real time or not real time, and maybe we get it right or maybe we get it wrong, and to step back and just look at it from a total consumption point of view, and to see from a capacity point of view what's going on. So to summarize this, because we don't have a lot of time to talk, if during a certain period of time, there's a lot of traffic happening in one part of the network, and there are small number of users that are contributing to that particular capacity event, then the idea is that we don't rate shake them, but we set a different priority in our cable network, so that the lighter users get a higher priority than the other users. And if those high users, if there is sufficient bandwidth, they go ahead and do what they want to do, which is the right thing to do. If there is the capacity, they should be able to use it. That's why we have a network. But at the same time, at the packet folding level, we have to make a decision about, do I let this packet through or that packet through, and I just can't satisfy the demand simultaneously, that there is a way to arbitrate it, so that we maximize the experience for all users. So, yes -- well, I helped make this slide. (Laughter.) RICH WOUNDY: So let me go into my analogy here. This is, the cable downstream is very simple. There's a router called CMTS and it's a single transmitter. You queue things up there and everything works. On the upstream, it's complicated, because everybody can't transmit at once, because they transmit on top of each other and it doesn't work. So the way things really happen is, to go out of IETF speak and into ISOC speak, think of it as a train schedule and there's trains that are leaving every ten minutes here. If a train is full, you can't get on the train, or somebody cannot get on the train. So too many people here, too many people here, they're not getting on the train. So, if we're going to do things right and manage congestion at the right level, first of all, with our prioritization scheme, we want the higher priority people to get in here, so the people that have less network activity, shove them on the train, and get their stuff out of the way, and minimize the delay for all the overflow traffic that we have. And so, now, you know, if I use a ten minute train analogy, that's a long time. And in cable time, this is like two milliseconds. And so these packets are only being delayed four milliseconds. And maybe that's not quite so bad. The experience that we've had is that the quantity folks that are, that have been sending a lot of traffic at a time when we are having capacity constraints is less than one percent. And, looking at the engineer who is responsible for calculating that, he's nodding. So, that's what we're doing doing here. This is the -- we're trying to do the lightest touch that we possibly can in order to be able to make sure we have consistent network experience, but at the same time, what we are allowing, or we're, we're giving a slight edge to a set of customers that maybe are using less bandwidth. What's interesting is that, my experience is that, because this is all transparent, because this is all documented, it's an Internet draft, it's on our web page and all that stuff, that outside people actually have put out recommendations about how to work around this scheme, which is interesting. So, I'll finish up. This is my last slide. We, our belief is that this is application neutral. It supports Internet innovation. Don't have to worry about whether Comcast is accidentally labeling something as peer to peer traffic when it shouldn't be, or vice versa. We don't care. Send the traffic to our network and see what happens. We're using this experience to hammer out some other efforts like Conex, where we can compare and contrast things. Conex is doing things that we're trying to do. And there are lessons and we're very involved with Conex, Alto, Ledbat, and other people that experience congestion. So, thank you very much. DANNY McPHERSON: All right. So, I'm Danny McPherson. I'm going to talk for a couple of minutes about something Arbor just did, my employer on the research side, not the product side, called the Internet observatory report. And we actually, we actually published some slides on the Nanog site and the network group, and there's a tech report coming out, and it has more methodology and things, and more technical things. Long and short of it, Arbor, on the product side, builds stuff that helps people understand routing and security and other stuff like that on the network. We have about 450 service provider customers, and so what we decided to do is put a program on those systems that they can enable and opt into and share information with us, mostly, well, completely annonymous, in the sense of no IP address and that sort of thing, but a lot of information on AS numbers, protocols and distributions and path links. So that's what you're looking at. It's all based on flow data, with the exception of one peer to peer analysis study we did based on a little bit of DPI data, pretty global distribution. It's about 20 percent or so of our customers on the service provider side that have enabled this. Here's just a, you know, there's some numbers there on some of the statistics associated with this. We only share interdomain traffic information, so we don't have information on the customer side, only information exchanged with other AS's on the Internet, to the tune of about fifteen terabits per second, which we reckon is about a third or fourth of Internet traffic. Anyway, some of my co-collaborators at the bottom there, if you can read that. Craig Labovitz actually presented this and was the lead author. So anyway, some of the major findings -- I have quite a few slides, but I'm going through really quick, a huge consolidation of content. So we've been doing this for over two years now. When we started measuring, about 5,000 AS's made up about 50 percent of the traffic we saw. And today about 150 AS; s make up that. So there's huge consolidation. You can see a couple of orders of magnitude there. As well, something we sort of termed hyper giants, so 30 percent of everything we see on the interent comes from 30 AS's or 30 organizations that may be represented by multiple AS's in the Internet system. Some other things we've seen is certainly, as Kenjiro Cho mentioned, we see a huge migration to TCP, you know. In the end, look at the hourglass with the thin waist, and it's all over TCP, to get over, to get over firewalls, and you know, other things, like host end systems, or other things that are blocking the end to end transactions. And then, a lot of changes in sort of the Internet, inter connection ecosystem, in particular, something, to borrow a financial termd, disintermediation. If you look at the left side, there's a strict hierarchy, tier one, tier two, and so forth. And today you see a lot more inter connection, and direct exchanges of information between them, and a lot of different economic models associated with those. That has lots of IP packet. It has impact on, for example, on the routing system, routing security, scaleability, certainly on the economics of being somewhere higher or lower in the hierarchy, and what the money changing hands looks like. So, then, and now, this chart illustrates the top sources or transit networks for traffic in 2007, as opposed to now, and you can make a lot of assumptions or share what we believe our observations are. The two that stand out is Google as a new entry in 2009 and as well as Comcast. There are lots of reasons we believe contributed to that. Certainly Google's YouTube acquisition added a lot of traffic -- and we decouple this in other slides -- and Comcast doing wholesale transit, and other things. We omitted some people from the right column because we didn't want to clearly express who dropped out at this point, and because we have a lot of service provider customers. So some of the other data we publish, most of the people are pretty okay with sharing this information. So it's been, anyway, so, here's a CDF of what I think clearly illustrates the point I was making earlier. In 2007, a much larger number of organizations accounted for a larger, you know, about 50 percent, say, 30 percent of the content on the Internet. Today that's much much smaller. And, so on one hand, from, be it a national security perspective, or whatever way you want to look at, that's a good thing this content is consolidated. The distribution of inter connections and the previous observations we made is much wider, you know, means there's more resiliant Internet and more densely connected and so forth. But at the same time,you know, so on one hand, inter connection is getting more dense but content -- I mean, getting more sparse. Inter connection is getting more sparse, but content consolidation is getting more dense. So, some of the other things we believe is, you know, we've seen this growth in wholesale transit prices, it's a huge commodity now, and it's a market share game. We don't believe it's something that's going to be sustainable. It's a losing model today on the call front. If you look at why some of the people provided services today, it's challenging economically. It's a cut throat game. Drop in video -- and I'm running out of time. One minute, so pass that slide. Applications, here's sort of the change in applications. A couple of things to really highlight here are, you know, web TCP port 80 is what we're looking at. It's growing and is still larger. The drop in peer to peer is a significant one, and we believe a lot of that is attributed to on demand content. Video, people can get things right here, right now, instead of waiting on chunks. Here's actually the global peer to peer trend, showing it dropped slightly. You should note this is not an absolute volume drop. It's a drop related to aggregate Internet traffic ratios. So the numbers are going up. It's the representation of the aggregate traffic we see in percentage wise, not as high as it used to be. Our results that were published before I saw his slides, you know, do actually jibe with what Kenjiro Cho was illustrating as well. Direct download, this is one of the things, this little Carpathia, you know, all of a sudden showed up on our radar. We're almost one percent of it. It picked up a lot of on demand stuff, and big companies that do on demand Internet distribution. And it's pretty bandwidth heavy. Internet growth size, we made some predictions, and you know, and talked to providers. We coupled in the actual report of a survey of providers. We did a detail survey with ten of our participants. We see the Internet as growing at about 44 and a half percent per year. And then there's some other stats, and you can look at that, Cisco, and some other people there. And then, finally on this, we do see this as a critical inflection point, consolidation of content but more sparseness of connectivity. The economics of scale and financial attributes and other things that are of benefit to community but there are off shots with that of course. What else do I want to highlight? Certainly video. All these other assumptions you can make from this. Then I get to throw out all these other random things about network neutrality, may or may not be derived from the data in the report. So I want to share this. So, some things I threw up here, illustrate here, one of the things that's really interesting, if you're a transit provider and people directly interconnect more regionally or more sparsely around you, the more traffic you see, is going to have to be carried further distance, so it's going to cost more. Transit is becoming a commodity. And that's a huge issue. Disintermediation, there's no regulation. If you have 30 of these hyper giants and they tell the broadband provider that you have to inter connect to me, the new entrants might be challenged. They can't get the same terms as the hyper giant. One final key thing is that a lot of times you hear people refer to the Internet as utility, and there's class and wireless and utility like, I guess, but if think of it like your refrigerator, that pulls power from the local substation, but the PC in your home may pull ten percent of the Internet bandwidth or capacity from a local content distribution network, but 90 percent may come from Asia or maybe variants among subscribers, and in the mobile model, you recoup that, weekends and nights. On the Internet side, you don't have the capability to recoup those costs. So that cost model is spread out among all the subscribers and it becomes challenging for providers to provide services with providers who use a lot of services. I'm out of time. LESLIE DAIGLE: Thank you. LARS EGGERT: Hi, yes, I'm Lars Eggert, from Nokia Research Center. And I'm one of the transport area directors and sort of congestion is what we try to control and avoid. So, stepping back for a moment, the Internet has always been about capacity sharing, from the very, very beginning, right? It was connectionless. There was no isolation between the flows. Your flow and my flow, we meet at the router and get through. There was no delivery guarantees. We tried in the past to make the Internet more controlled environment. That hasn't really happened end to end. It has happened in some sort of more controlled deployments, but not really for the average end users. And so the principle that has made the Internet scale is really the edges are smarter and the core is dumb. So that really has enabled several industries to grow, which, you know, I also have a hard time imagining any other technology to do that in the recent past. It's all about sharing. So sharing means caring. So, as I said, packets go to different flows. They share the path through the network, between the sources and the destinations, and that means that the behavior of one flow affects what the other gets out of the Internet. In terms of congestion, you know, if one flow sends a whole lot of packets, it's going to take capacity away from another flow. And also, with queueing delay, depending on what queue you use on the path, one flow, it's very aggressive and can build up the queue. And even though the other flow doesn't want a lot of capacity, it still sees the extra delay. So effects that come from sharing a global resource, so, the applications and protocols, sort of needed some sort of social behavior mechanisms. And that's what TCP style congestion control has always provided to some degree. And if we didn't have that, the Internet would have stopped being a useful resource a long time ago. And it actually has stopped while we had the congestion collapse episode in the '80s. So TCP has really helped here. It's one of the tools in the IETF toolbox that, you know, helps applications and users share this sort of shared capacity in the network, intelligently. So principles that I already mentioned, smart edges, dumb core. That's a split of responsibilities between the end systems and the network. It's not that, you know, it's all up to the end network. In the telephone network it is all up to the network, because the edges were dumb and the core was smart. But that has proven not to scale very well. And reverse it is much better. But that doesn't mean that the network doesn't get to do anything or doesn't need to do anything. And we saw in Comcast, there's valid things the network needs to do. It's not all about the edges. At a very high level, it's split between the core and the edges that the network is responsible for providing applications and transport protocols, to serve as agnostic conditions or driving conditions. Loss rate, they can be measured at ITT. Maybe in the future there will be other information in the path. And this path is really congested, and that's basically what it means. It's not a catastrophic event. It's normal on the Internet that there's some level of congestion, because it's really a best effort then. And then, applications for transport protocols, they are responsible for taking the network provided information and doing something smart with it. You know, if the network says, hey, there's congestion, it's there, and then we have to reduce the rate and say, okay, I'm reacting to it. And exactly how react to it can be application specific. TCP congestion control is one way, but there's other mechanisms that are appropriate for traffic flows that cannot work with TCP. For example, some of the media streaming flows, we would like them to be congestion aware, but it's clear they need a different response. It's fine. The Internet can deal with it. Coming back to the toolbox, I mentioned TCP congestion control. And basically the host determines the rate to use, based on the RTT and losses. That's only what it is. And we've known this for a long time, and we continue to tweak it and make it work better and better. For wireless environments, there's optimizations and extensions, like explicit ECN doesn't drop your packets. It starts to mark them as it approaches congestion. And if you get to the marks, you can prevent losses. Active queue management, which is something smarter than drop tail, where you build a long queue. Those are techniques that, during the home gate BOF, came up. But these are the designs in the '90, when the core speeds were 45 megabits. And I think it was Rich or Jason that had the observation that we have those speeds in the access network and the stuff that we did back then for the core probably should be revisited and see what we can use in the access column. So, new stuff -- this is old stuff, old stuff. We have a new working group that started about a year ago, which tries to provide a transport protocol for applications that want to move a lot of data around the network. But they want to be nice about it. If you're a UNIX user, you remember the nice protocol, only schedule me on the CPU if nobody else wantss it. This is sort of the equivalent of that for the network. It says, only give me capacity if it's available, and I don't care if I'm dropped when you're congested. That together, especially together with ISP incentives, if the end users are nice, maybe the ISP says, I'm not going to charge for it, or charge way less for it, and it could be a win/win here. And that's the sort of mechanisms that we try to build. Another thing we're doing is multipath TCP. If you have end systems that have multiple connections to the network, they're actually able to shift traffic between paths on time scales that are very fast, much faster than the time scales that ISP would use for the traffic engineering. That's another way in which an end system can sort of, when it detects congestion somewhere, it has another way to go to the destination and it can shift traffic there. It's going to help balance congestion and load across the network. It happened yesterday. Another thing we're doing is more directly targeted at peer to peer layer traffic optimization. The idea here is that if you can improve peer to peer application performance in a way that actually also is good for the operator, everybody wins. And the idea is that, give a peer to peer application information about which might be good for peers before you start downloading from it. So you don't have to randomly search around the network until you find a good set of peers. You, for the operator, it's nice, because they can maybe give some hints to the application of which peers they would prefer them to use. And again, this is sort of a win/win that's really, I think needed, if any of these mechanisms want to be deployed. Finally, congestion exposure. Rich mentioned it already. This is the idea that we want to provide additional information from the network to the end system, so that you can react better to traffic conditions, and also, gives you more flexibility as an end system. This is the BOF that's actually later on this afternoon, so you can still go to it. It's going to be an interesting one. It's a mechanism that's very powerful. It has lots of different applications because it's about providing information exchange between the core and edges that wasn't there before. And so it has lots of potential uses. And it's going to be interesting to see which ones we're going to do. One final thing, recommendation for home gateways is not sort of a tool or toolbox. It's more like a toolbox. We're saying, if you want to connect a home network to the Internet, we should put a device there that implements a certain number of IETF core protocols correctly. Because otherwise, the user experience, or experience you provide to your user will not be very great. This is not yet a working group. We're still trying to exactly scope what we should be doing here, in collaboration with other partners in the industry, other forums that deal with these aspects already. And that's going to be interesting. Finally, we have many tools to share capacity in a good way, and we're busy designing new ones as we learn more about the needs of the operator and changing applications in the network. But also, the bottom point here, a lot could already be gained by using some of the tools we already have more consistently and more appropriately. I think that's sort of something we need to talk more to the operator community. So I'm very glad this discussion has picked up over the last few years. Thanks. LESLIE DAIGLE: Thank you all very much. So we now have about fifteen minutes for questions, comments, discussion. So if you'd like to ask something, please make your way to the microphone, and we would like to know who you are. MATT MATHIS: I made a career of trying to unsuck TCP. And what I mean by that is, a decade ago, if you took a TCP, you could very easily see things that they did wrong. As of the recent Microsoft releases, all major stacks are no longer the first bottleneck. And what this means is, previously, for moderate scale networks, actually, I would say fairly low scale networks, the end systems were always the bottleneck. The end systems are no longer the bottleneck. If you fire up a connection between your laptop here and a server across the globe, it's got a fair amount of data to transmit. Barring a few caveats, it will either raise the round trip time or the loss rate on the path, on the global scale path today. This was not true two years ago. So, and the technical details, the maximum window size is now a couple of megabytes for all stacks, by default, out of the box. It was previously 64K, or something like that. This, I'll use an automobile analogy. Until fairly recently, it's like having all cars shift in first gear, and nobody knows how to fix anything. Now we have cars with automatic transmissions and turbo chargers. What the car industry discovered, when turbo chargers were invented, is in fact the car doesn't have good enough brakes. And the first approximation of the Internet, it doesn't have any brakes. That's the problem. LESLIE DAIGLE: Thanks. Any comments from the panel? LARS EGGERT: He's correct. RICH WOUNDY: I think we're also at the point where the broadband speeds would actually allow you to also be able to send and receive that much data too. I mean, you know, it wasn't that many years ago that, you know, broadband meant you had a one and a half megabit per second pipe. And at that point, restrictions didn't matter as much. LARS EGGERT: The analogy, the Internet is not that we don't have brakes, mechanisms to slow things down. It's like we need mechanisms to handle the speeds, safely. >> Bob Brisco. I want to take on, continue on this analogy of the Internet not having any brakes. And I think what we've seen over the last three decades is, essentially, the, most of the capacity is shared by TCP. It's still something like 90 percent of the bytes on the Internet, TCP controlled. And if you think of this like a vector diagram, TCP is trying to send us off in that direction, or along that road, to use Lars's analogy. And if you look at what the traffic management techniques are trying to do, they're trying to pull things over that way. And so, we end up going somewhere forward, but less than we would do if we swung TCP around to go forward, and we could go a lot faster, rather than both these techniques trying to fight each other. If we actually worked out what the problem is, and I think the problem is that TCP only allocates capacity instantaneously. It doesn't have a concept of time. And so when some users are using a lot at a time and others aren't, TCP doesn't have any memory that they were just using it. LARS EGGERT: You're right. Thank you. So, Bob is one of the pushers behind the Conex BOF, which is about, how do you make TCP and Internet traffic work together, rather than fight. LESLIE DAIGLE: I'd like to add to that as well. I think another piece of the puzzle is, we talked about trying to solve the problem, and trying to identify what plane the problem is in, is the first challenge, because we could easily look at the Internet observatory report and say, well, clearly, what we have to do is make this work for video, video over, and everything will be grand. But that's probably not engineering in the future in a useful way. ALAIN DURAND: I think, as a community, we are extremely lucky, because when I was looking at the shift from peer to peer dominance to download real time being more important, my initial reaction was to think, oh, this is going to all move to UDP. If it was all moving to UDP, we would have lost all the fairness, or partial fairness of TCP. This did not happen. So I think we have been very, very lucky. So I would like to get a sense of the panel, as of why did this happen? LARS EGGERT: I'm speaking too much, but I'll give you my sense. People always want something light weight and they want TCP. Then they discover they need to add some extra mechanisms. Then you have half of TCP already. You want flow control, for example, so don't overshoot the receiver. And maybe also, so it's -- TCP is a very, very, very, very highly engineered protocol. And it's very good. And it's very hard to do better, even, you know, if you want something more light weight. Because it is, you know, it's not just, you know, swaying packets there. It's not going to get you anything. You need to have some mechanisms if you want to have reliable and predictable conditions for your application sessions. DANNY McPHERSON: I have a comment as well. I think one of the things you spend a lot of time on, on the security side, five or eight years ago, everybody was concerned about network based worms, and so they applied nats and proxies and all these things. And the only thing that worked was TCP port 80. So if you you look, for example, six months ago, X box used port 80, and instead of using -- and the reason why is because it works more deterministically. Maybe the only good thing of compromising in such a way with the middle boxes is that we kind of scoped it to at least congestion or friendly protocol. LESLIE DAIGLE: Just a logistical point, in the interest of getting ourselves back to the IETF planet on time, I'm going to close the mic lines as they are now. MURARI SRIDHARAN: Two questions, one is about, how do you, as operators, how do you think about congestion challenge? Because a few years back, telecons used to upgrade capacity based on peak. Whatever capacity there was, most end hosts can completely fill it up, forgetting users for a second, if you just went by peak utilization? So, how do you think about capacity? And one comment on the fact that Internet doesn't have brakes, that's actually incorrect. There are huge walls. And so, let's not forget those. RICH WOUNDY: I tried to move some of those walls. Seriously, the way that a lot of operators look at capacity management is, they are still worried about peak, because peak demand over a particular interval really ends up driving your capital spending. Capital spending is what the investors focus on, and Wall Street focuses on, and they say, you're spending too much or too little, whatever. It's going to be interesting as things evolve, because if we look at Conex, it's not clear that just merely measuring like what percentage of a particular link is being used for traffic, is necessarily the right measure. So that's certainly one of the things that we need to look at in the future. LESLIE DAIGLE: Okay. Any other comments from the panel? Bob? BOB HINDEN: Bob Hinden from Checkpoint. So, I'm trying to think, when I listen to all the talks, and summarize what my take aways were, I think I heard, in Japan, you can get way more bandwidth with residential customers than I can ever get where I live, and traffic seems to be growing at 40 percent a year, and that sounds okay. I didn't hear any discussion about problems doing that, or delivering that. I heard in, at least from someone in the U.S., I heard a lot of discussion about ways of controlling traffic in the network, but not sort of how to get more traffic in the network. And then I heard that half the traffic is TCP traffic, and it's all the fault of 30 companies, something like that. And then I heard that IETF is doing lots of cool transport congestion mechanisms, but very little of it actually gets deployed, or is allowed to be used because of intermediate boxes. So, I'm not really sure what to think about all this. LESLIE DAIGLE: I think you probably got the right take aways in large part, and I think Lars outlined all the places where you could go and scratch heads with other people on that. BOB HINDEN: If there's any conclusion from the speakers, that would be interesting. RICH WOUNDY: Let me comment about my part. I think the wrong take away is that, you know, ISPs need to stop spending money and let the networks congest, and turn into these cash cows with terrible (inaudible), that is not, is not what we want to do. What we want to do is, we want to invest our networks and keep adding capacity. Interestingly enough, we, at Comcast are deploying the same technology that Jcom here deploys. And I don't know how they do it. That's an off line topic. But, essentially, at the core, we're using the same technology. And our biggest concern is not so much, you know, how can we stop spending 5 or $6 million a year. It's more, how can we make it so that, when traffic increases in an unexpected way, not the 40 or 50 percent that we're all talking about, but you know, when it doubles overnight, when something happens that we cannot plan for, and you have to order equipment in advance and put people on hold, that's the kind of time where we think that congestion management makes sense. But that needs to be followed up with capacity upgrades. Don't do one without the other, because otherwise, you're just letting your service fall apart. KENJIRO CHO: My comment here is, you should worry about the business between supply and demand. For the last five years, we're on the over supply side. And we might need to worry about over supplying in the future. But at the moment, I think we are doing okay right now. LARS EGGERT: In Japan. KENJIRO CHO: Maybe similarly in the States and Europe, you know. Traffic growth is very important, around 50 percent, so, probably, you know, the (inaudible) type access increase, probably 50 percent per year. LARS EGGERT: I thought I heard you want more bandwidth. It doesn't seem like it's a problem. LESLIE DAIGLE: Okay. Well, I had to close the mic, so I'm afraid, we're not going to get to you. We need to release the room for the next working group session. So at this point, I would like to remind you all to take your boxes with you. And I'd like to thank the panel very much for coming here today. (Applause.)