Yongguang Zhang <ygz@isl.hrl.hac.com>
Dante De Lucia <dante@isl.hrl.hac.com>
Bo Ryu <ryu@isl.hrl.hac.com>
Son K. Dao <son@isl.hrl.hac.com>
Hughes Research Laboratories
USA
Recent deployment of commercial products in geosynchronous earth orbit (GEO) satellite communication networks demonstrates the promise of ubiquitous access to the Internet. Delivering this promise to end users requires integrating satellite communications into existing Internet transmission links. This effort is the topic of heated debate over whether common Internet applications can perform satisfactorily over networks with high-latency links, such as those involving GEO satellite transmissions. Through simulations and actual satellite experiments, we contend that most applications can work effectively. Certain TCP-based applications have technical shortcomings that surface in both terrestrial and satellite high-speed networks, but feasible solutions exist. Furthermore, multicast applications and the network as a whole can benefit from the broadcast nature of satellite transmission and its simple network topology, as demonstrated by our experiments using NASA's Advanced Communications Technology Satellite (ACTS).
Most current Internet backbones and subnets are wired terrestrial networks (e.g., cable, telephone, and fiber optics) with bandwidth ranging from T1 (1.5 megabits per second [Mbps]) to OC-12 (622 Mbps). Currently, researchers are working on the next-generation Internet that will support high-bandwidth applications (> 45 Mbps), and ubiquitous computing with mobile/wireless networks (wireless local area network [LAN] and ATM, wireless metropolitan networks, and satellite networks). Among these mobile/wireless networks, GEO satellite networks offer great potential for multimedia applications with their ability to broadcast and multicast large amounts of data over a very large area, thus achieving global connectivity [12]. Internet distribution via satellites, particularly GEO satellites, has the following merits:
On the other hand, satellite communications present one major challenge with respect to the performance of Internet applications -- the communication latency between two earth stations connected by a satellite. For GEO satellite communications systems, the latency is at least 250 milliseconds (sometimes framing, queuing, and on-board switching can add extra delays, making the end-to-end latency as high as 400 milliseconds). This is approximately 10 times higher than a point-to-point fiber optics connection across the continental United States. The latency might not affect bulk data transfer and broadcast-type applications, but it will hurt highly interactive applications that require extensive handshaking between two sites. Unfortunately, one of the major Internet transport protocols, TCP, requires such interaction.
LEO (low earth orbit) and MEO (medium earth orbit) satellites can also provide global and broadband communication capacities. Latency in a LEO system is comparable with terrestrial fiber optics, usually only about twice as large. Because neither a LEO nor a MEO satellite can stay in a fixed position relative to the surface of the earth, a constellation of many satellites is required to provide complete coverage, increasing the complexity of the system relative to GEO systems. Network management is a much harder problem because handoff, tracking, and routing need to be done properly. The advantage of simple topology is lost, and so is single-source broadcast/multicast capability.
Common Internet applications include Web browsing, file transfer protocol (FTP), remote login (Telnet), video teleconferencing, e-mail, and broadcast. They all use IP (Internet Protocol) as the transmission mechanism, so they can seamlessly run over satellite networks. However, performance varies among applications because their requirements on network bandwidth and responsiveness, their tolerance to communication noise, and their implementation techniques are very different.
The following figure summarizes these applications. They vary substantially in demands on bandwidth and responsiveness. The implementation techniques are also very different, as are the impacts of network delays. Some applications require guaranteed delivery; they use TCP and are sensitive to latency. Others can use UDP or other real-time protocols that can tolerate delays. Multicast/broadcast applications use reliable multicast such as SRM, which is less sensitive to delays than TCP and thus works fine over satellite links. It is therefore inappropriate to make simple statements on whether GEO satellites make better Internet access. The purpose of this paper is to show the prospects for Internet distribution over satellites and let users decide by their application priorities.
On today's Internet, TCP is the protocol used by the vast majority of applications. The performance of TCP over long-delay networks will have a direct impact on the performance of Internet access using GEO satellites. In this section, we will analyze the performance issues in TCP control mechanisms and current approaches to adopt and improve TCP for long-delay networks.
TCP flow control starts from the "window size" concept of a TCP connection. It determines how much data can be outstanding (i.e., unacknowledged) in the network. In long-delay networks, there can be many unacknowledged segments. Theoretically, the amount of data that can be in a transit is given by the bandwidth-delay product. In practice, memory and operating system resources limit the window size. In the current TCP standard, the maximum window size in TCP is 64 Kb. (Because of historical implementation issues concerned with signed and unsigned arithmetic, the practical window size is often limited to 32 Kb.)
To maximize bandwidth utilization in a satellite network, TCP needs a much larger window size. For example, on a satellite link with a round-trip delay of 0.8 seconds and bandwidth of 1.54 Mbps, the theoretical optimal window size is 154 Kb, or considerably more than a maximum window size of 32K or 64K.
A new TCP extension, or TCP-LW for "large-window" [6], has been defined to increase the maximum window size from 216 to 232, allowing better utilization of links with large bandwidth delay products. To obtain good TCP performance over satellite links, both sender and receiver use a version of TCP that implements TCP-LW. Applications should also set the size of the send and receiver buffers to be bandwidth times delay.
TCP adapts to the available bandwidth of the network by increasing its window size as congestion decreases and reducing the window size as it increases. The speed of the adaptation is proportional to the latency, or the round trip time of the acknowledgment. In a satellite network with longer latency, bandwidth adaptation takes longer and, as a result, TCP congestion control is not as effective. Furthermore, it will take much longer for TCP's linear increase to recover the window size after a packet loss if a TCP "large-window" extension is used.
The standard TCP acknowledgment scheme is cumulative. If a segment is lost, TCP senders will retransmit all data sent starting from the lost segment without regard to the successful transmission of later segments. TCP considers this lost segment as an indication of congestion and reduce its window size by half. Recently the newly defined standard TCP-SACK (Selective ACKnowledge) allows the receiver to explicitly inform the sender of the loss. Consequently, a sender can retransmit the lost segments immediately rather than waiting for a timeout, reacting to supposed congestion, and multiplicatively decreasing its window. If lost segments are not caused by congestion, or the congestion is transient, throughput in TCP-SACK should be much better. This will be helpful in satellite networks because anything that triggers timeouts and window size reduction will force a lengthy recovery in TCP.
When a TCP connection first starts up or is idle for a long time, it needs to quickly determine the available bandwidth on the network. It does so by starting with an initial window size of one segment (usually 512 bytes), then increasing the window size as packets are delivered successfully and acknowledgments arrive, until reaching the network saturation state (indicated by a packet drop). On the one hand, slow start avoids congesting the network before it has a good assessment of the available bandwidth; on the other hand, TCP bandwidth utilization is suboptimal during the procedure. Therefore, the shorter TCP slow start lasts, the better performance it can achieve. The total time of a TCP slow start period is approximately RTT*log2(B/MSS), where RTT is the round trip time (twice the latency), B the available bandwidth, and MSS the TCP segment size. Although the growth is exponential, for high-bandwidth and long-delay networks (e.g., satellite links and terrestrial gigabit network), this can take a significant amount of time.
Recently, new techniques have been introduced in TCP to avoid congestion before it happens. The first approach, called RED (random early detection) gateways [5], requires each gateway to monitor its own queue length. When imminent congestion is detected the TCP sender is notified. By dropping a packet earlier than it would normally, RED sends an implicit notification of congestion. The sender is effectively notified by the timeout of this packet. The principle behind the RED approach is that a few earlier-than-usual drops may help avoid more packet drops later on. The TCP sender can then reduce its window before serious congestion occurs.
Another approach is to have the TCP sender predict when congestion is about to occur and reduce its transmission window before intermediate routers drop packets (TCP Vegas [3]). TCP can keep track of the minimum round trip time seen during a transfer and use the most recently observed round trip time to compute the data queued in the network. TCP can also keep track of the throughput before and after the congestion window changes to estimate the network congestion level. If estimates indicate that the number of packets queued in the network is rising, it reduces the congestion window. As it observes the number decreasing it increases the congestion window.
Although neither approach has been widely adopted, both hold promise for satellite networks. As we mentioned earlier, TCP congestion control responds to congestion slowly because of latency. If such congestion can be avoided before it happens, it is a big win for high-speed and long-delay networks.
Many TCP applications involve only simple communications between the client and the server. The interaction is called a transaction: a client sends a request to a server and the server replies. The Hypertext Transfer protocol (HTTP) for WWW browsing applications is a typical example of TCP with transactional behavior. Under standard TCP, even a small transaction involving a single request segment and a reply must undergo TCP's three-way handshake in preparation for bidirectional data transfer. If the request is bigger than a segment, TCP must also undergo the slow start procedure. It is very inefficient to establish such a TCP connection, send and receive an insignificant amount of data, and then tear it down.
Transaction TCP, or T/TCP [2,10], is an extension to TCP designed to make such behavior more efficient. T/TCP does this by bypassing the three-way handshake and slow start, using the cached state information from previous connections. Although T/TCP is designed mainly for short client-server interaction applications, it can be used to reduce the impact of latency on the beginning of a TCP connection. If slow start can be avoided, significant performance improvement can be achieved in a satellite-based network.
We have conducted a simple set of simulation experiments to explore the correlation among TCP window size, network bandwidth, latency, and end-to-end performance (response time and throughput). The network simulation tool we used, NS [8], allows arbitrary network configuration and different TCP versions. The figure below describes the network topology of our experiment. We varied the interconnection link bandwidth from 128 Kbps (kilobits per second) to 6.176 Mbps, the latency from 10 ms to 400 ms (which covers local network, LEO, MEO, and GEO satellite links), and the workload (the amount of data sent in one TCP connection) from 1 KB to 100 MB. We also varied the TCP window size from 2 KB to 512 KB.
We recorded the throughput and response time of each TCP connection. Throughput determines the bandwidth utilization of the link from the system manager's point of view, while the response time reflects performance as perceived by the user. A small sample of the results is compiled in the table below:
| Bandwidth | 6.176 Mbps | 1.544 Mbps | 0.384 Mbps | 0.384 Mbps |
| Amount Transfer | 100M | 10K | 1M | 10K |
| Measurement | throughput | throughput | response time | response time |
| Result in figure (click for detail and explanation) |  |
 |
 |
 |
From the above results we can see that latency is a decisive factor in a satellite network only for the response times of small transfers. If an interactive application often opens a TCP connection only to send a small amount of data, it will perform very inefficiently. The next section will suggest better alternatives under this condition.
Over the years, people have developed various techniques to mitigate the impact of latency. The first alternative is to adopt a version of TCP that performs better over the satellite and does not hinder performance over terrestrial networks. The second approach relies on Internet gateways at the satellite network boundaries to perform special functions to speed up TCP sessions. The third approach is to develop better implementations of common applications that use TCP more efficiently and more sensitively.
Some of the TCP problems experienced on GEO satellite links today will arise in future high-speed terrestrial networks because of the similar bandwidth-times-delay property. Problems like large window size, prolonged slow start period, and ineffective bandwidth adaptation affect both networks. They place satellite networks and gigabit terrestrial networks in the same class of extensions for better performance. Some of the techniques discussed earlier, including TCP-LW, TCP-SACK, TCP-Vegas, T/TCP, and RED gateways, can alleviate the problems for both networks. For example, TCP-Vegas can reduce the number of packets lost to congestion, hence reducing long delays after packet drops. In a similar way, TCP-SACK can reduce the need for retransmission of unnecessary data. Other techniques like rate control and selective negative acknowledgments may further improve efficiency. The benefits of applying these extensions/improvements have been demonstrated in MITRE's attempt to develop TCP modifications specifically tailored for space communications [4].
Great performance improvements can be made in many cases by working at the Internet infrastructure level without the need to modify TCP. This layer is sometimes referred to as the middleware layer. While enhancements at the transport-layer require changes to the operating system of each end host, enhancements at the middleware layer usually require few or no such modifications.
TCP has its shortcomings when used over long-delay networks. These can be avoided if Internet applications use TCP more effectively, for example, by avoiding small and short transfers, as suggested in the previous section.
GEO satellites can be a natural media for MBone (the virtual network over Internet for multicast applications [7]) because of their large coverage area and broadcast capability. Currently, to multicast over the terrestrial MBone, the data must travel over numerous links and duplicate itself at numerous intermediate routers (see figure below for a recent map of MBone [1]). This takes up significant bandwidth and raises the possibility of transient congestion from TCP cross traffic at each router along the path.
On the contrary, multicast over satellite can deliver data directly to end-user networks or hosts with minimal cost. The following figure envisions the future MBone with multicast satellite connecting strategic points on the Internet.
To demonstrate the potential role of satellites in Internet multicast, we have set up an experimental MBone link over the NASA ACTS satellite between the NASA Lewis Research Center (LeRC) in Ohio and Hughes Research Laboratories (HRL) in California. The link allows two sites to connect directly instead of going through tens of hops over the terrestrial MBone. (See above figure for location of LeRC and HRL.)
Internet applications that use multicast can gain substantial benefit from a satellite network. First, multicast applications do not use TCP and are not sensitive to long bandwidth delay products in the same way TCP is. Moreover, satellite networks can bypass potentially tens of intermediate nodes, thus reducing the chances of packet drop and large delay jitter attributable to congestion. To illustrate this, we have conducted an MBone videoconferencing experiment between LeRC and HRL. LeRC transmits two identical video streams with video tool vic [9] to HRL simultaneously, one over terrestrial MBone and the other over MBone over ACTS. At HRL, two workstations receive the streams respectively, and record the following quality of service (QoS) parameters at the receiver: packet loss, delay jitter, bandwidth, and frame rate. We set the transmission rate at the source at 128 Kbps and 8 frames per second. (Because of the current MBone bandwidth restriction on the terrestrial Internet, we limited the bandwidth to 128 Kbps, otherwise the performance advantage of MBone over ACTS would be even bigger.) The following table shows the performance comparison for each QoS parameter.
| Measurement | bandwidth | frame rate | transient packet loss rate | packet delay variation |
| Results in figure (click for details) |  |
 |
 |
 |
The first and second charts compare packet loss and delay jitter. Along the terrestrial MBone path, the packet loss rate is well above 70 (first chart) and jitter occasionally hits 200 milliseconds or more (second chart). These results translate into poor performance through low frame rate and low bandwidth. The third chart shows that the video bandwidth received over terrestrial MBone is only one-fourth of that received over ACTS, and the fourth chart shows that the frame rate over terrestrial MBone is only one-sixth. However, we observe no packet loss over ACTS from the first chart, and from the second chart jitter was kept under 100 milliseconds. Consequently, the receiver gets the original bandwidth and frame rate transmitted by the sender. These results imply that the use of GEO satellites as part of MBone can deliver high-quality services to end users in a bandwidth- and cost-effective manner.
Satellite networks promise a new era of global connectivity, but also present new challenges to common Internet applications. In this work, we have shown that many popular Internet applications perform to user expectation over satellite networks, such as videoteleconferencing, MBone multicast, bulk data transfer, background electronic mail, and non-real-time information dissemination. Some other applications, especially highly interactive applications such as Web browsing, do suffer from the inefficiencies of the current TCP standard over high-bandwidth long-latency links. However, performance of these applications can be improved by utilizing many of the techniques discussed in this paper. In the long term, further improvements can be made at the protocol level by extending the current TCP standard, although much work needs to be done on possible extensions to ensure that they do not negatively affect the Internet as a whole.
We would like to extend our thanks to NASA Lewis Research Center for its assistance in some of the satellite experiments. We would also like to thank the Hughes Spaceway Group for valuable feedback and technical assistance.
ns - lbnl network
simulator.
http://www-nrg.ee.lbl.gov/ns/.
vic:
A flexible framework for packet video. ACM Multimedia,
November 1995, San Francisco, California, pages 511-522, November 1995.