![[INET'99]](/web/20100311063500im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/inet99sm.gif)
![[ Up ]](/web/20100311063500im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/up.gif)
![[Prev]](/web/20100311063500im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/prev.gif)
![[Next]](/web/20100311063500im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/next.gif)
Benjamin TEITELBAUM <ben@internet2.edu>
Internet2 (University Corporation for Advanced Internet Development) / Advanced Network & Services
USA
Rüdiger GEIB <geib@advanced.org>
Deutsche Telekom / Advanced Network & Services
Germany
The Internet2 project is a partnership of over 130 U.S. universities, 40 corporations, and 30 other organizations. Since its inception, one of the primary technical objectives of Internet2 has been to engineer scalable, interoperable, and administrable interdomain quality of service (QoS) to support an evolving set of new advanced networked applications. Applications such as distance learning, remote instrument access and control, advanced scientific visualization, and networked collaboratories will allow universities to fulfill their research and education missions into the future, but only if the network QoS that these applications require can be assured. To meet this challenge, the Internet2 QBone initiative has brought together a dedicated group of U.S. university and federal agency networks, international research networks, engineers, researchers, and applications developers to build a test bed for interdomain IP (Internet protocol) differentiated services.
Although the QBone initiative is still quite young, it rests on more than a year of collective head-scratching by members of the Internet2 quality of service (QoS) working group. Starting in fall 1997, this working group (including more than 30 distinguished networking experts from academia, industry, and government) has struggled to understand the QoS requirements of next-generation Internet (NGI)/Internet2 networks and how Internet2 could make progress toward meeting them. At a series of workshops in fall 1997 and winter 1998, the working group heard from advanced applications developers, campus network planners, and gigaPoP operators, and identified a demanding set of requirements for Internet2 QoS. Chief among these are:
In parallel with the working group's efforts, the differentiated services (DiffServ) approach to QoS began to attract significant interest from the Internet Engineering Task Force (IETF). Although much of the push for DiffServ at this time was from commercial Internet service providers who saw a sizable and immediate market for differentiated classes of best-effort Internet protocol (IP) service, architectural engineering concerns played a significant role as well. From a technical perspective, DiffServ is a reaction against the perceived scalability problems of RSVP (resource reservation protocol)/IntServ. DiffServ is an attempt to find simple, scalable forms of QoS that can provide a variety of end-to-end services across multiple, separately administered domains, without necessitating complex interprovider business arrangements or complex behaviors in the forwarding equipment.
The DiffServ architectural framework achieves its scaling properties by indicating in each packet's header [RFC2474] one of a few standardized, simple differentiated forwarding treatments. These simple aggregate packet treatments, also known as per-hop behaviors (PHBs), are combined with a much larger number of policing policies enforced at the network edge to provide a broad and flexible range of services, without requiring state or complex forwarding decisions in core routers.
Each DiffServ micro-flow is policed and marked at the first trusted downstream router according to a contracted service level agreement (SLA), usually a token bucket filter. The QBone does not interfere with the strictly bilateral negotiations of SLAs. For this reason, the term "service level specification" is often used to refer to what the QBone itself defines. However, in order to emphasize the requirement that this kind of specification be part of all SLAs between QBone participants, we continue to use the term SLA in what follows.
When viewed from the perspective of a network administrator, the first trusted downstream router is a leaf router at the periphery of the trusted network. Downstream from the nearest leaf router, a DiffServ flow is mingled with similar DiffServ traffic into a behavioral aggregate; all subsequent forwarding and policing is performed on aggregates. At interprovider boundaries, SLAs specify the transit service to be given to each aggregate. Aggregate SLAs are also characterized by traffic profiles (again, often based on token bucket filters). By carefully enforcing the aggregate traffic contracts between clouds and ensuring that new reservations do not exceed aggregate traffic capacity, the DiffServ architecture provides well-defined end-to-end services over concatenated chains of separately administered clouds. Furthermore, since SLAs exist only at the boundaries between clouds, the result is a set of simple bilateral SLAs that mimics current interprovider exchange agreements.
In addition to packet forwarders capable of implementing the emerging PHB standards, the DiffServ architecture [RFC2475] requires edge devices that implement classifying, metering, marking, shaping, and dropping (see figure 1). Although not currently part of the DiffServ architecture, a new kind of network component known as a bandwidth broker (BB) is expected to play an important role in automating admissions control for DiffServ networks.
Figure 1. The Differentiated Services Architectural Framework
Per-flow policing and marking is performed by the first trusted leaf router downstream from the sending host. When a local admissions control decision has been made by the sender's cloud, the leaf router is configured with the contracted per-flow service profile. Downstream from the first leaf router, all traffic is handled as aggregates. At cloud ingresses, incoming traffic is classified according to a traffic conditioning agreement (TCA) into behavior aggregates, which are policed according to the SLA in place. Depending on the particular DiffServ service model in question, out-of-profile packets are either dropped at the edge or marked with a different PHB.
In order to reduce burstiness, it is very important that each cloud that initiates a QoS flow have the ability to do its own traffic shaping. In addition, as in-profile traffic traverses a cloud, it may experience induced burstiness caused by queuing effects or increased aggregation. Consequently, clouds may need to shape on egress to prevent otherwise conforming traffic from being unfairly policed at the next downstream cloud.
Finally, to make appropriate internal and external admissions control decisions, and to configure leaf and edge device policers correctly, each cloud may be outfitted with a BB. When a sender signals its local BB to initiate a reservation, the requesting user is authenticated and subject to a local admissions control decision. On behalf of the sender, the BB may then initiate an end-to-end reservation request along the chain of BBs representing the clouds to be traversed by the flow. The BB abstraction is critically important because it allows separately administered network clouds (possibly implemented with very different underlying Layer-2 technologies and subject to very different policies) to manage their network resources as they see fit.
Recognizing the startling similarity between the root engineering motivations behind DiffServ and the primary Internet2 QoS requirements, the Internet2 QoS working group began to develop an Internet2 QoS architecture based on the evolving IETF DiffServ architecture. In May 1998, at the First Internet2 Joint Applications/Engineering QoS Workshop [I2QoS98], the working group presented the outlines of this architecture and began a dialogue within the greater Internet2 community. This dialogue culminated in consensus around the need to build an interdomain test bed to explore and advance DiffServ technologies and to iteratively provide an increasingly robust infrastructure for experimentation with new advanced collaborative applications.
Although the evolving IETF DiffServ architectural framework offers a promising approach to overcoming the scalability, interoperability, and administrability problems that have plagued previous QoS efforts, the strength of the architecture and the mindshare momentum currently behind it do not alone guarantee success. DiffServ has not yet been evaluated in the wide area, and the architectural framework begs many questions and leaves many difficult research, engineering, and policy problems unaddressed. For example, it is far from clear how to perform efficient admissions control for connectionless networks, what implications DiffServ will have for traffic engineering, how to design protocols for interdomain DiffServ reservation setup, how to provide for advanced reservations (e.g., to support scheduled distance-learning courses), or what protocols and admissions control algorithms are needed to support multicast DiffServ.
Because of the research and higher education community's openness and need to find common solutions to enable new advanced applications, and because of the tolerance of its applications developers and users for pre-production Internet services, Internet2 is uniquely situated to build the first interdomain test bed for differentiated services and to begin to tackle the problems mentioned above. In September 1998, Internet2 announced the QBone initiative with a call for participation (CFP). The primary goal of the CFP was to identify a small and focused initial group of participants who would cooperate to build an open and heavily instrumented test bed. In this test bed, experimental interdomain differentiated services could be deployed, debugged, analyzed, and refined by networking engineers and researchers working in close collaboration with the users and developers of new advanced networked applications.
The response to the CFP was overwhelming -- 37 proposals were submitted from more than 73 organizations. Most proposals were of extremely high quality, and many came from teams already representing collaborations between multiple organizations. A subcommittee of the Internet2 QoS working group reviewed the submitted proposals carefully with the primary goal of identifying a small initial group that was topologically contiguous and able to participate in building the initial interdomain test bed. This group was dubbed the QBone Interoperability Group (QIG).
The working group also announced the formation of the QBone Solutions Group (QSG), a second prong of the QBone initiative. The focus of the QSG is on supporting research and engineering relating to the deployment of intradomain differentiated services. This group will participate in a broad range of discussions on engineering and deployment issues, and will include both teams that plan to join the QIG and teams that do not anticipate joining this core group but are interested in working together to share DiffServ implementation experiences and find common solutions. Participation in this group is open to the entire NGI/Internet2 community. The major initial activity of this group will be planning a large information-sharing and problem-solving workshop to be held in the spring of 1999.
Current participants in the QIG include vBNS, Abilene, ESNet, NREN, CA*Net2, SURFnet, TransPac, MREN, NYSERNET, NCNI, and the Texas GigaPoP, as well as numerous universities and labs. A map showing the set of initial participants and their connectivity is shown in figure 2. A full listing of participants with links to individual project pages can be found on the QBone home page [QBone].
Figure 2. Initial QBone Participants and Connectivity. Actual
connectivity and participant groups will change as deployment
progresses.
Recognizing the importance of international academic research collaboration and the central role of QoS in enabling advanced collaborative applications, the Internet2 QBone project has from its beginning sought active participation from non-U.S. research and higher education networks. It is anticipated that the initial QBone will include connectivity to Canada's CAnet*2 (and future CAnet*3) advanced networks; to the Asia-Pacific Advanced Network (APAN) and SingAREN in Asia; and to SURFnet in the Netherlands. SURFnet has even offered to share its QBone connectivity with other interested European research institutions. The QBone CFP also yielded numerous proposals from international research and engineering networks that were unable to participate in the initial QIG because of deployment timeline or connectivity considerations. These networks have been invited to coordinate their efforts with the evolving QBone architecture through participation in the QSG.
Beyond physical network participation in the QBone, a number of groups from around the world have BB prototypes and are participating in the Internet2 QBone Bandwidth Broker Advisory Council. These include CTIT from the University of Twente (Netherlands), BCIT from British Columbia (Canada), and Telia Research in cooperation with the University of Lulea (Sweden).
Finally, through the Coordinating Committee for Intercontinental Research Networking, Internet2 is assuring architectural compatibility between the QBone and related QoS test bed efforts, such as the European TEN155-Quantum project [Quantum].
Even as the QBone network planners put together the initial QBone network architecture, several applications groups are already preparing to experiment with and stress-test new QBone services. We briefly summarize three QBone applications projects: the nanoManipulator project; a research project to understand and design mechanisms for intelligent service management for multistream client-server multimedia applications; and the iCAIR advanced video services project.
The nanoManipulator (nM) system [NANO] provides a virtual environment interface to scanned-probe microscopes, giving the scientist telepresence on the sample surface scaled up by a factor of about a million. The nM system is a continually evolving tool, developed in close collaboration with real users, that provides new ways of interacting with materials and objects at the nanometer scale. This has led to new results in biology, materials science, electrical engineering, and the study of carbon nanotubes. The nM project is a collaboration between the departments of computer science, physics and astronomy, and chemistry; the school of library science; and the school of education at the University of North Carolina (UNC) at Chapel Hill. The project is developing an improved, natural interface to scanning probe microscopes, including scanning tunneling microscopes and atomic force microscopes. The North Carolina Networking Initiative (NCNI) will first attempt to push the nM project from one isolated local area network (LAN) and campus at UNC Chapel Hill, across the NCNI gigaPoP backbone network, to LAN access on the North Carolina State University and Duke campuses. Then attempts will be made to accomplish the same access across the Internet2 QBone infrastructure through STARTAP to Nortel facilities in Ottawa, Canada.
This project is a joint effort between researchers at the University of Pennsylvania and the University of Massachusetts, Amherst. The project seeks to understand what engineering support is most useful to applications when accessing the DiffServ capabilities of the QBone. The investigation of these issues will be carried out using a rich multimedia client-server type application. The application will combine multiple types of streams -- audio, video, and data -- which not only have different QoS requirements, but also exhibit dependencies between streams in terms of the relative QoS they should receive. This application will demand a QoS mechanism that knows which streams to degrade first, which streams to degrade jointly, and so on. The goals of the experiments will be to:
The International Center for Advanced Internet Research (iCAIR), at Northwestern University near Chicago, has organized an international consortium that has established a differentiated services research project as part of the Internet2 QBone initiative. The consortium is developing a worldwide differentiated services test bed to address a wide range of key QoS issues. A key aspect of this test bed is that it will allow for performance testing across the globe (e.g., to and from Asia and the Netherlands), which allows for investigations that are not possible within national test beds.
A principal goal is to conduct performance testing with key real-time latency-intolerant applications, such as digital video, IP telephony, and Teleimmersion. The Center is the lead institution on the I2 Digital Video Network (I2-DVN) project, which is creating a national digital video network. These applications must be able to signal their requirements and have networks understand and interpret these requirements and translate them into consistent resources. They must also allow for a certain amount of dynamic adjustment in resource allocation depending on variations in requirements/resources ratios over time. This second requirement applies particularly to non-premium-service applications, for which it is important to develop a secondary control channel to continually monitor and adjust performance.
Traffic contracts must be established between a specific application and a service category that must relate to reference configurations, baseline standards, and dependability parameters. This will make it possible to measure and audit performance against that specific contract, provide for pre-fault diagnostic analysis, and allow conflicts to be resolved through defined negotiation rules. Optimization, managing delay intolerance/latency, and jitter make sense only within the context of such definitions, together with other information such as application and point-of-origin identifiers. This project is attempting to develop common notions of appropriate responses to noncompliance and priority conflicts, especially in instances of packet discard or severe reduction in packet processing. For example, BB models are being investigated, built on the system model of computer resource brokers for shared components.
The QBone architecture seeks to remain consistent with the emerging IETF standards for DiffServ. In addition to specifying which subset of the IETF DiffServ architecture [RFC2475] must be implemented, the QBone Architecture Draft [QBoneArch] specifies a QBone Premium Service (QPS), consequent minimum requirements for an interdomain SLA, requirements for an integrated measurement infrastructure, and a set of common operational practices for establishing interdomain reservations.
QPS will make interdomain, peak-limited bandwidth assurances with virtually no loss and virtually no delay or jitter due to queuing effects. QPS exploits the expedited forwarding (EF) per-hop forwarding behavior, which is specified in [EF]. As QBone deployment progresses, QPS will increasingly come to resemble the virtual leased-line premium service proposed by Van Jacobson and initially demonstrated across ESNet to the show floor of SuperComputing '97 [SC97].
A QPS reservation is for a specified peak rate of EF traffic and a specified "service maximum transmission unit (MTU)." It offers the following transmission assurances:
Consistent with the DiffServ architectural model, all SLAs are determined bilaterally between adjacent QBone networks (dubbed "DS Domains" in [RFC2475]). However, to implement QPS, certain minimum requirements for any QBone SLA must be met. The following is a list of recommendations ("shoulds") and requirements ("musts") for any QBone SLA supporting QPS. The list assumes a bilateral SLA between an upstream QBone DS domain U and a downstream QBone DS domain D.
An integrated measurement infrastructure is key to understanding and debugging end-to-end QoS performance. The QBone architecture requires that a set of performance parameters be collected at the ingresses and egresses of each participating QBone DS domain. These data are to be collected through both active and passive monitoring and are to include such parameters as:
To allow QBone deployment and experimentation to begin as soon as possible, reservations will initially be long-lived and will be established manually, relying on human operators to make admissions control decisions, provision appropriately, and configure edge devices. This will all take place according to a set of common operational practices agreed upon by QIG participants. The complexities of manual resource allocation, device configuration, and policy management are expected to soon overwhelm the capabilities of the human operator.
To address this issue, it has been suggested that integral to the DiffServ architecture should be a BB -- an automated admissions control agent that makes resource management and policy decisions in response to requests for interdomain bandwidth reservations. Within the QBone initiative, a QBone Bandwidth Broker Advisory Council [BBAC] has been formed to recommend intradomain BB solutions and to develop a prestandards interdomain BB signaling protocol for experimental deployment in the QBone. This group is being led by Sue Hares of Merit Network.
The QBone will be the first wide-area test of the evolving differentiated services architecture and the first experimental deployment of interdomain differentiated services. It is envisioned that the QBone will grow incrementally as new QoS services mature. By building a highly instrumented test bed that is open and accessible to researchers and advanced development efforts, the QBone initiative seeks to advance the state of DiffServ technology. Further, by working together with the broader Internet2 community to come to terms with the profound administrative, economic, and policy implications of QoS, the QBone aims to start a process that will open the horizon for new advanced networked applications to flourish.
The authors gratefully acknowledge Guy Almes for clear-sighted guidance and moral encouragement. In addition, we would like to recognize the invaluable editorial assistance of Jeff Ubois and Ben Chinowsky on this paper.