STAR TAP, International Exchange Point for High-Performance Applications
Thomas A. DEFANTI <email@example.com>
Steven N. GOLDSTEIN <firstname.lastname@example.org>
Bill ST. ARNAUD <email@example.com>
Rick WILDER <firstname.lastname@example.org>
Paul ZAWADA <email@example.com>
Bruce DAVIE <firstname.lastname@example.org>
Maxine D. BROWN <email@example.com>
The US National Science Foundation (NSF)-sponsored Science, Technology, and Research Transit Access Point (STAR TAP) initiative provides an international exchange point for high-performance networked applications. STAR TAP will be one of the exchange points for the Next-Generation Internet (NGI). Also, STAR TAP is housed in the same facility as one of the emerging Internet 2 gigapops. Unlike most other Internet exchange points (IXs), STAR TAP will have to cope with sorting of traffic to route acceptable use policy (AUP)-compliant flows to connecting high-performance networks, such as NSF's vBNS and CANARIE's CA*net II. The US Department of Energy's ESnet, the US National Aeronautics and Space Administration's NREN, and Singapore's SINGAREN also connect to STAR TAP; Taiwan's TANet, the Asia-Pacific Advanced Network (APAN), and several other high-performance networks from Asia-Pacific, Europe, and perhaps Latin America will connect soon. This paper will discuss the policy implications of restricting traffic to institutions carrying out high-performance meritorious applications as opposed to general research and educational institutions. It will also discuss the technological approaches (e.g., Multi-Protocol Label Switching) being considered or tested for implementing the policies over a multinetwork switching and routing fabric.
In February 1995, the G7 nations--the United States of America, Canada, France, Germany, Italy, Japan, and the United Kingdom--as part of the Global Information Infrastructure (GII) initiative, commissioned the Global Interoperability of Broadband Networking (GIBN) project, designed to encourage the establishment of international links among existing high-speed networks of the G7 and other industrialized countries. These networks would serve as testbeds for a wide variety of applications, thus providing an opportunity to experiment on interconnectivity and interoperability and to promote the rapid establishment of standards. 
GIBN continues to seek ways to interconnect high bandwidth networks to ensure interoperability. One approach to facilitate the long-term interconnection and interoperability of advanced international networking is the Science, Technology and Research Transit Access Point (STAR TAP), a program supported by the U.S. National Science Foundation (NSF). 
Two major U.S. initiatives in high-performance networking are the Next Generation Internet (NGI),  a government initiative that President Clinton proposed in October, 1996, and Internet 2 (I2),  a cooperative activity of over 100 American universities that seek to develop ways to improve Internet services. The NSF Very High-Speed Backbone Network Service (vBNS)  plays a key role in both the NGI and I2. The goals and strategies of NGI and I2 have striking similarities:
Several other countries have commissioned similar initiatives with similar objectives. Canada has an initiative called the Canadian Network for the Advancement of Research, Industry, and Education (CANARIE).  CANARIE is also an industry-government-academia partnership. CANARIE commissioned CA*net II to develop advanced Internet capabilities and to transfer them into the private sector. Singapore carved out of its commercial backbone the Singapore Internet Next Generation Advanced Research and Education Network (SINGAREN).  Taiwan has TANet.  France is readying its Renater-2. NORDUnet, the academic backbone network for the Nordic countries, is approaching a NORDUnet-2 initiative. Germany is commissioning an advanced high-performance network. The Asia-Pacific Advanced Network forum (APAN) is a cooperation of several countries, including Japan, Korea, Singapore, and Australia (founding members), joined by Hong Kong, Indonesia, and Thailand.
The vBNS, CA*net II, and SINGAREN are already connected to STAR TAP, TANet will soon connect, and APAN is seeking a way to connect its members' advanced networks. Several European advanced networks, mentioned above, are pursuing connections. In addition to NSF, other U.S. government agencies that participate in NGI are connected at the STAR TAP; these include the Department of Energy's Energy Sciences Network (ESnet)  and the National Aeronautics and Space Administration's NASA Research and Education Network (NREN) .
STAR TAP, the NSF-supported persistent infrastructure (NSF Award NCR-9712283, for the period 1997-2000), is managed by the University of Illinois at Chicago, the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign, and the Mathematics and Computer Science Division of Argonne National Laboratory, and is operated by Ameritech in Chicago, Illinois, U.S.A. STAR TAP anchors the NSF vBNS International Connections program.
STAR TAP is essentially a large switch with enough capacity to provide stable configurations of emerging networking technology. The principal contribution anticipated is the design and enabling of a truly integrated approach to the management, performance measuring, scheduling, and consumption of geographically distributed network, computing, storage, and display resources, with focus on advanced computational science and engineering applications. Specifically, this connectivity furthers a decade-long effort of the University of Illinois at Chicago, the National Center for Supercomputing Applications, and Argonne National Laboratory to develop highly leveraged collaborations over broadband networks using high-performance computing and communications technologies, virtual reality, and scientific visualization. The goal is to encourage the development of teams, tools, system software, online documentation, and human interface models to enable international-scale, multi-site collaborations.
Following previous successful models, such as the I-WAY [3, 4, 9], this project will supply the necessary network engineering, application programming assistance, and Web documentation, creation, and maintenance to U.S. scientific researchers and their approved international partners. This support is key to the formation of long-lasting, productive, international research relationships.
Instead of a new infrastructure, STAR TAP was built on the commercially available ATM service provided by Ameritech Advanced Data Services (AADS) in Chicago. By partnering with Ameritech, STAR TAP could provide a high level of service without investing in facilities and related support services. The STAR TAP engineer works closely with Ameritech personnel to tailor services to the needs of the international high-performance networking community.
As shown in Figure 2, the STAR TAP ATM service is one of three primary public ATM services provided by Ameritech. The ATM network that supports STAR TAP also supports the Metropolitan Research and Education Network (MREN)  as well as the Chicago Network Access Point (NAP). MREN is a collaborative effort between Ameritech and several universities and research institutions located across the Midwestern USA. The Chicago NAP is one of many interconnection points of Internet Service Providers (ISP) in the USA. By having the STAR TAP, MREN, and Chicago NAP all interconnected through the same ATM fabric, members of one logical entity can exchange traffic with an entity outside of their "core" group. For example, an MREN institution can exchange traffic with an international network connected to the STAR TAP and buy commercial Internet connectivity from an ISP connected to the Chicago NAP. This means that building STAR TAP on top of Ameritech's commercially available service eliminates duplication of effort and facilitates interconnection with other existing networks.
The Ameritech STAR TAP ATM service currently provides Permanent Virtual Circuits (PVCs) and Permanent Virtual Paths (PVPs) to other STAR TAP participants. The ATM service (currently delivered by a Lucent Technologies' Globeview 2000) is an Unspecified Bit Rate (UBR) service with the Sustained Cell Rate (SCR) and Peak Cell Rate (PCR) both set to the line rate of the incoming connections. All STAR TAP connections are fully meshed with PVCs and PVPs so that any STAR TAP connection can send traffic to any other STAR TAP connection. If a STAR TAP participant wishes to connect to an MREN or NAP participant, a PVP or PVC will be built on request.
Once an ATM connection to STAR TAP is completed, connections at the IP layer are made on a bilateral basis. When two STAR TAP participants wish to exchange traffic, they configure routers in their networks to use the preconfigured PVC or PVP through STAR TAP. Once IP connectivity is established between the two networks, BGP4 (Border Gateway Protocol version 4) is used to route the networks between the two clouds, just as in the commercial Internet.
Since each STAR TAP participant has a unique PVC, traffic between different pairs of peers can be handled by participants' routers in different ways. In other words, STAR TAP has no single AUP that restricts the type of IP traffic that is passed among participants. Moreover, a STAR TAP participant can use different AUPs when handling traffic from multiple peers.
While most STAR TAP connections will be put in place for specialized, high-performance connections between research and education networks, no policy prohibits participants from requesting PVCs to commercial Internet providers. Since the STAR TAP and Chicago NAP are implemented on a shared ATM infrastructure, it is possible to interconnect STAR TAP participants and commercial ISPs. A business relationship needs to be established between the two networks, but the shared ATM infrastructure and the lack of STAR TAP peering requirements makes connectivity to the commercial Internet possible without the need for additional physical connectivity.
A test platform is being added to the basic Ameritech ATM service to provide a means for STAR TAP to test new services in a non-production environment. This will allow STAR TAP users to be early adopters of such technology as public Switched Virtual Circuits (SVCs), IP routing, and Multi-Protocol Label Switching (MPLS) before the Ameritech deployment schedule would normally allow. Early adoption of these services also provides Ameritech with valuable experience that can be used when making decisions on offering these services to the public at large. The test platform will consist of an ATM switch connected via an OC-3c line to the Ameritech production network. A high-performance router will be connected to the test platform ATM switch. Candidate platforms for the ATM switch are the Cisco LS1010 and the Cascade 500; candidate platforms for the router are the Cisco 7500 and the Cisco 12000. The actual hardware used in the test platform is expected to change over time. Since the test platform is not a production entity, different platforms can be swapped on short notice for comparison testing.
One of the first experiments will be the use of SVCs. SVCs will initially be provided by building PVPs through Ameritech's production ATM service into the test platform. If SVCs prove useful, their functionality will be introduced into the production ATM service.
To meet the policy requirements of STAR TAP, routing capabilities beyond those in widespread use today are needed. Virtually all routing in IP networks is destination based; i.e., the decision on where to forward a packet is based only on its destination address. In principle, other fields in the IP header (e.g., source address, type of service bits) can be used when deciding where to forward a packet, but various design and performance considerations have led to router designs that almost always forward packets based on destination address alone. This presents a difficulty for STAR TAP, as the following example illustrates.
The arrangement of routers in Figure 3 is referred to as "the fish picture" because of its fish-like shape (the fish's tail is to the left). It is the canonical example of the limitations of destination-based routing. Consider the router at point C. Using normal destination-based routing, there is only one "shortest path" to point E, which goes though D. That is, any packet originating at A or B that needs to be forwarded to E will follow that shortest path. Thus, a policy along the lines of "packets going from A to E should go via D, and packets from B to E should go via F" cannot be realized using destination-based routing.
Assume that point C is inside the STAR TAP network, A and B are located inside two different networks that connect to STAR TAP, and the paths C-D-E and C-F-G-E are paths to a destination that has different restrictions on acceptable use. The AUP that applies to one path might make it suitable for traffic from A, while the AUP of the other path might make it suitable for traffic from B.
One solution to the above problem hinges on the use of MPLS  and some of the explicit routing capabilities developed as part of Cisco's proprietary Tag Switching architecture implementation [12, 13]. If the device at point C were a layer 2 switch (e.g., an ATM or Frame Relay switch), there would be no difficulty setting up PVCs to provide the appropriate connectivity. However, there are several disadvantages to offering only layer 2 connectivity in an interconnect point. MPLS, a technology that blends layer 3 routing with layer 2 (label switching) forwarding, is able to provide the sort of policies described above.
Figure 4 illustrates how MPLS supports the policies required by STAR TAP. Using MPLS, two Label Switched Paths (LSPs) have been constructed, one from A to C to D, and another from B to C to F. This allows packets from A that are destined to E to follow one path while those from B to E to follow another.
Operationally, LSPs are configured from their "heads," i.e., from routers A and B. The operator configures the LSPs as a series of hops (e.g., A-C-D) and then specifies which traffic is to follow the LSP. The configuration of the LSP causes a path establishment message to travel along the specified path, establishing label switching table entries at the appropriate hops. In the present implementation, the selection of traffic that is to follow a certain LSP is based on BGP next hop (although more complex selection criteria are possible and may be implemented in the future). It is possible to configure backup LSPs. All traffic that does not follow an LSP (either because it was not selected for an LSP or because link failure has ruled the LSP out of consideration) is forwarded using normal IP routing.
When an LSP is established, then packets arriving at the head of an LSP (e.g., router A) that match the criteria for that LSP are labeled, and the label effectively identifies the LSP. They are then forwarded to the next hop of the LSP (e.g., router C). At subsequent hops, they are forwarded using label switching, until the end of the LSP is reached. At that point, the label is stripped and normal IP forwarding resumes.
The use of MPLS rather than layer 2 switching offers several advantages. The above arrangement does not require any peering from a routing perspective between routers A and D or between routers B and F, as would be the case if C were a layer 2 switch. Also, there is no necessity to configure LSPs for all traffic--only for the set of traffic that requires some treatment different than normal IP routing would provide.
We have not shown administrative boundaries in this example, but there are several options. For example, it might be the case that the routers A, B, C, D and F are all inside the STAR TAP network. In this case, none of the networks that connect to STAR TAP would need to participate in MPLS--they could use standard IP routing. However, if STAR TAP customers also participate in MPLS, then the administrative boundaries might be almost anywhere in this picture. For example, routers A and B might be outside STAR TAP while router C is inside.
A number of advanced research networks are being established around the world to help develop the next-generation Internet services and applications. These networks support trial activities that are not practical using the regular Internet. These activities include QoS, IPv6, some forms of multicasting, etc. More importantly, these advanced networks support high-performance applications that will not work properly, if at all, on the big "I" Internet--applications such as medical imaging, tele-immersion, collaboratoriums, distributed genome sequencing, etc.
It has been long recognized that most high-performance applications development occurs at leading university and research institutes. As in the early days of the Internet, the research community pushes the envelope in terms of high-performance applications. Funding and infrastructure support enables not only these applications, but basic research as well. A "tertiary" benefit of applications development is that network requirements and design are driven by user "needs" rather than network designers' perceptions of what is required. Satisfying the real needs of researchers will allow the development of applications and services that can easily migrate into the commercial sector.
A classic example of this "tertiary" benefit is the development of the World Wide Web. It was originally developed to serve the needs of high energy researchers at CERN. It is expected that today's next-generation networks will lead to similar application developments and spin-offs.
The next-generation of Internet networks being deployed worldwide generally have restrictive AUPs that limit connectivity to a subset of institutions carrying out high-performance meritorious applications. In most cases, this institutional AUP is considered an interim AUP until routing technology allows the implementation of an "applications based" AUP.
Currently, in order to interconnect "cooperating" next-generation Internet networks, network operators enter into peering agreements where they identify specific institutions that are permitted to communicate with each other across a common interconnect point, like STAR TAP. The routes for these institutions are then advertised by each respective network to their member institutions.
CA*net II and vBNS use the BGP's Community Attributes feature to tag "foreign" network routes. These routes are not propagated outside the home network to any other attached network. The use of community attributes is, however, only a partial solution and only works properly if the foreign network has implemented an "explicit" routing architecture.
Today's high-performance applications have increasing requirements to establish network topologies that follow research collaboration linkages rather than network topologies. For example, there is a joint project among Canada, the U.S., United Kingdom, Chile, Argentina, and Brazil called Gemini, for two very powerful next-generation "twin" telescopes to be built in Chile and Hawaii.  The telescopes will use atmosphere-correcting lenses, which should result in images almost as good as those produced from the Hubble space telescope. Also, these telescopes will use electronic imaging systems rather than film. Images could be transmitted over high-performance networks to a researcher's desk. In time, it should be possible for researchers to operate the telescope remotely in real time.
These images cannot use any lossy compression technique, as very faint star images may be lost in the process; hence, image files will be several megabytes, if not gigabytes, in size. These images will be exchanged back and forth among a small set of researchers located primarily in the participating countries. This application requires high-performance connectivity among sites, yet currently each site is connected to separate research networks.
With the existing interconnection agreement, collaboration linkages among researchers are driven by the somewhat arbitrary designation of approved institutions in respective networks. In an ideal world, it would be advantageous to set up "wide-area virtual high-bandwidth networks" among collaborating institutions. These virtual high-performance networks would span a number of underlying research networks. The analogy that is commonly used is a set of overlapping Venn diagrams of virtual private networks (VPNs). These are somewhat different in concept from today's commercial I-VPNs in that an institution can be a member of a number of different communities of interest. More important, users are not isolated from the Internet as they would be with I-VPNs. In fact, the Internet could be considered the global "community of interest" of which every Internet user is a member, as illustrated in Figure 5.
Using the Gemini project as an example, Canada's University of Toronto, the U.S.'s NASA Goddard Space Center, Chile's National University of Chile, and select institutions in the United Kingdom, Argentina, and Brazil could be part of one virtual private network that is dedicated to one specific collaborative application.
Using what are commonly referred to as "explicit routing" technologies, such as MPLS, these communities of interest could be assigned their own QoS mechanisms where not only the advertised routes but other parameters, such as bandwidth and delay, could also be configured dynamically.
Another important feature of the "communities of interest" concept is that they are layer 2 independent. There is strong likelihood that next-generation networks, including SONET, Giga-ethernet, Optical IP, and ATM, will use a number of layer 2 transport services. A community-of-interest or VPN strategy that is layer 2 independent is very attractive to many network operators.
The need for virtual high-performance networks will be increasingly important for funding agencies as well. In the next few years, it will be increasingly difficult for funding agencies to continue to fund "generic" backbone networks for the research community, even if those networks have restrictive AUPs. On the other hand, it is recognized that many advanced institutions and applications need high-performance networks to carry out their fundamental missions. Virtual networks that are clearly identified with a specific research objective or even a specific application will be much easier to support from public funds.
Several high-performance, collaborative applications have used the STAR TAP connection. These applications are briefly mentioned here; for further information, see the STAR TAP Web pages. 
Intercontinental online visualization and control of T3E simulations of black hole interactions and gravitational waves
For the SC'97 conference in November 1997 in San Jose, California, the following institutions collaborated on a transcontinental astrophysics project:
Researchers are collaborating to solve the 3-D Einstein equations to simulate interactions between black holes and gravitational waves in support of design and interpretation of detection experiments. As illustrated in Figure 6, gravitational wave iso-surfaces were selected and displayed near-to-real-time using the ImmersaDesk virtual reality display.
For the SC'97 conference in November 1997 in San Jose, California, the following institutions collaborated on a transcontinental astronomical/fluid dynamics project:
The Pittsburgh Supercomputing Center (PSC) and Sandia National Laboratories (SNL) demonstrated an international metacomputing environment with the Rechenzentrum Universitaet Stuttgart (RUS). The three sites utilized a connection spanning the vBNS, CA*net II, ESnet, and a transatlantic connection from Teleglobe. Intended as a prototype for international high-performance networking, the project coupled Pittsburgh's 512-processor Cray T3E with another 512-processor T3E at the High Performance Computing Center in Stuttgart and a Paragon at SNL. As shown in Figure 8, they visualized fluid dynamics simulations of the re-entry of a space vehicle, the entry of meteorites into the Earth's atmosphere as well as their impact on the Earth, and ab-initio calculations of magnetic properties of alloys.
This distributed virtual reality demonstration took place in the Spring of 1997. Participants included:
The groups established an interactive, real-time network connection between two virtual reality visualization systems across the North Atlantic and evaluated the capabilities, practicality, performance, and cost of distributed virtual reality technology for collaborative product or process design review. The vehicle being prototyped is shown in Figure 9.