INET Conferences

Conferences

INET


NDSS

Other Conferences

[INET'98] [ Up ][Prev][Next]

A High-Performance Network Connection for Research and Education Between the vBNS and the Asia-Pacific Advanced Network (APAN)

Michael A. MCROBBIE <vpit@indiana.edu>
Karen H. ADAMS
Dennis B. GANNON
Donald F. MCMULLEN
Douglas D. PEARSON
R. Allen ROBEL
Steven S. WALLACE
James G. WILLIAMS
Indiana University
USA

Abstract

The development of high-speed research networking in the US, and the rise of the vBNS in particular, has been paralleled in Asia by the development of the APAN consortium and high-performance networks in the countries of APAN's constituent members. These rapid and recent developments on both sides of the Pacific have been driven by the emergence of high-speed networking as a central requirement for the support of basic science, engineering, and other disciplines. The growth of international scientific collaborations in, for example, astronomy, high energy physics, medicine, and computational science leads to a requirement for a global data communications infrastructure on par with what is available domestically.

This paper will describe the establishment of a high bandwidth international Internet connection from the vBNS to the Asia Pacific Advanced Network (APAN). Indiana University will develop a US-Asia Pacific network, TransPAC, through a joint effort with the APAN consortium, AT&T as the US domestic and international service provider, and Kokusai Denshin Denwa Co., Ltd. (KDD) of Japan and Korea Telecom (KT) of Korea as APAN's main network service providers. The TransPAC network will initially provide 35Mbps VBR-nrt ATM service from the vBNS connection at the STAR TAP switch in Chicago to the APAN eXchange point (XP) switch in Tokyo. This network will require support from the National Science Foundation's High Performance International Internet Services (HPIIS).

The paper will provide an overview of the research associated with constructing and characterizing the Layer 3 IP services that will be available end-to-end for HPIIS institutions. The paper will also discuss issues related to the dual nature of TransPAC as a production network for international scientific collaborations and an R&D testbed for new networking technologies, including multicast routing (for Mbone), IPv6, QoS using RSVP and other methods, video conferencing, shared virtual environments, and tele-immersion.

The use of global-scale networks in support of very widely distributed real-time, high-performance applications is a virtually unplowed field, one which will clearly become more important as the necessity of international cooperation in scientific research grows. The establishment of TransPAC is, therefore, expected to provide tremendous dividends in the development and refinement of communications equipment and protocols, and in the development of support software and systems for applications developers.

Contents

Introduction

The development of high-speed research networking in the US, and the rise of the vBNS in particular, has been paralleled in Asia by the development of the Asia Pacific Advanced Network (APAN) consortium and high-performance networks in the countries of APAN's constituent members. These rapid and recent developments on both sides of the Pacific have been driven by the emergence of high-speed networking as a central requirement for support of basic science, engineering, and other disciplines. The growth of international scientific collaborations in, for example, astronomy, high-energy physics, medicine, and computational science leads to a requirement for a global data communications infrastructure on par with what is available domestically.

This paper provides a description of the establishment of a high-bandwidth international Internet connection from the vBNS to the Asia Pacific Advanced Network (APAN). Indiana University (IU) is leading the development of a US-Asia Pacific (AP) network, TransPAC, through a joint effort of the United States National Science Foundation (NSF) and the APAN consortium, with AT&T as the US domestic and international service provider, and Kokusai Denshin Denwa Co., Ltd. (KDD) of Japan and Korea Telecom (KT) of Korea as APAN's main network service providers.

The use of global scale networks in support of very widely distributed real-time, high-performance applications is a virtually unplowed field, one which will clearly become more important as the necessity of international cooperation in scientific research grows. TransPAC will create a network infrastructure that will make possible these critical scientific collaborations. In addition, TransPAC is expected to provide opportunities for the development and refinement of communications equipment and protocols and the development of support software and systems for high-performance computing applications developers.

The TransPAC network will initially provide 35Mbps VBR-nrt ATM service from the vBNS connection at the STAR TAP (Science, Technology And Research Transit Access Point) switch in Chicago to the APAN eXchange point (XP) switch in Tokyo. In additional to the use of TransPAC as a production network for international scientific collaborations, it will be an R&D testbed for new networking technologies (e.g., multicast routing for Mbone, IPv6, QoS [quality of service] using RSVP and other methods, video conferencing, shared virtual environments, and tele-immersion).

HPIIS program overview

The TransPAC network is partially funded through the United States NSF's High-Performance International Internet Services (HPIIS) and the Japanese Science and Technology Agency (STA). The HPIIS program assists the US research and education community in meeting its needs for next-generation international Internet services by encouraging high-performance connectivity between the NSF's very High Speed Backbone Network Service (vBNS) and high-performance networks of major international research partners, such as APAN. The goal of the HPIIS program is to advance the development of next-generation applications that support international collaborations. High-performance Internet services, based on high-bandwidth communication links, enable interaction with global information and experimental resources. These services are also assumed to provide new Internet features such as the ability to dynamically reserve network resources and to guarantee various qualities of service to applications.

This NSF solicitation was posted in June of 1997 and the TransPAC HPIIS grant was awarded in March 1998.

APAN overview

The Asia-Pacific Advanced Network (APAN) Consortium was formed under a Memorandum of Understanding in June 1997 to promote advanced research in networking technologies and the development of high-performance broadband applications and infrastructure in the AP region. APAN is supported by four primary member countries: Australia, Japan, Korea, and Singapore. There are also several associate members, including Indonesia, Hong Kong, and Thailand. The United States and Canada are Liaison Members. APAN is a consortium of application developers at research institutions, telecommunications providers, and high-performance research network organizations in the member countries. The primary network operation centers (NOCs) in Japan and Korea are controlled by the APAN Secretariat.

APAN is controlled by a committee consisting of representatives from the four founding members. There are Application Working Group, Technology Working Group, NOC Working Group, and Bandwidth Allocation subcommittees. Membership in APAN is open to any individuals or organizations. Members pay fees to support operations based on a scale established by the APAN Committee.

The working groups listed above drive the development of network infrastructure and applications and promote cooperation between the member institutions. The Application Working Group (WG) develops advanced research applications over high-performance Internet services and clarifies the requirements for underlying network services. The Technology WG specifies network components, topology, and protocols, and has several sub-working groups investigating emerging protocols and services such as IPv6, Web cache, streaming media support, quality of service, and network security. Network operations are provided by the NOC WG in collaboration with the Technology WG.

Applications under development include medical and biomedical engineering, high-energy physics, astrophysics, environmental engineering, high-performance computing, and next-generation network services. APAN provides testbeds for developing advanced services and facilitates their commercialization. A number of these projects involve US partners at vBNS institutions and these relationships are the basis for the development of TransPAC and high-speed connectivity to the US through STAR TAP.


Figure 1. APAN topology.

TransPAC organization

TransPAC is a joint effort by several groups and commercial carriers in the US and AP to join the APAN research network testbed with high-speed networks in the US through the STAR TAP switch in Chicago. The TransPAC link is a high-speed ATM service running from the STAR TAP to APAN, connecting at the Tokyo XP. The result is a global high-speed network for research that supports approximately $276M in US research investments and $105M in investments made by APAN member countries. End to end, vBNS-TransPAC-APAN is the largest high-speed research network in the world and constitutes the premier global testbed for developing next-generation network protocols and services.

The TransPAC consortium includes the following organizations:

TransPAC is organized as an interlocking set of committees and service groups as illustrated below:


Figure 2. TransPAC organization

Overall governance is provided by the TransPAC Management Committee (TMC), chaired by the Principal Investigators. This committee manages contracts with the funding agencies (NSF in the US, STA in Japan) and provides overall management for the rest of the organization.

Reporting to the Management Committee is the Technical Coordinating Committee (TCC), composed of representatives from APAN, AT&T, STARTAP, KDD, and Indiana University. The TCC is responsible for overall technical design and engineering; policy formation in conjunction with the Management Committee; and ongoing technical assessment.

Using input from the reports of performance, utilization, security, and effectiveness as well as comments from User Services, provider NOCs, and the member institutions, the TCC makes recommendations for growth and engineering changes. The TCC meets virtually (via teleconference or other network-based collaboration facilities) on an ad hoc basis to coordinate ongoing engineering activities, to assess the feasibility of special requests for TransPAC utilization, and to address security and policy concerns.

TransPAC applications

A number of institutions within the eight-nation APAN consortium have developed a rich set of applications in a broad range of disciplines, which routinely depend on high-speed networking within APAN. Many of those institutions have extended collaborations with vBNS-authorized institutions (vAIs) in the US and the applications under development by these US-APAN partnerships require considerably more than the 10Mbps currently available to achieve their full potential. These applications fall into several main categories: astronomy, space exploration, earth observation for resource management, meteorology and climatology, medicine, molecular biology, genome databases, epidemiology databases, experimental high-energy physics, computer science, network protocols, virtual reality/telepresence, and distributed high-performance computing, video conferencing, and distance education. All require bandwidth in excess of what is currently available, and many rely on next-generation services. A complete list of current applications can be found at http://www.transpac.org.

TransPAC applications are characterized by diverse quality of service (QoS) requirements. Many of the computer-intensive applications will greatly benefit simply from the increase in interinstitutional bandwidth provided by the TransPAC link. Bounds on bandwidth and latency may offer performance advantages for some applications, particularly those that involve time-critical media transport or in which distributed processes frequently exchange partial results.

To assist with the development of applications and to gather requirements for further network R&D, TransPAC provides a High Performance Computing and Communications (HPCC) Application Engineer and an HPCC Support Specialist in the US and AP User Services groups. These network application software specialists participate with users in all phases of application development, including design, initial testing and performance characterization, and problem resolution after the application is in production.

Several testbed projects are also underway to explore emerging services in a global network context. These include IPv6, RSVP, multicast (Mbone) and streaming media, Web caching, geographically distributed high-performance computing, and tele-immersion. Each has an associated international working group and a research agenda.

TransPAC user services (user support groups)

User support for TransPAC is provided through a distributed network operations center (NOC), network information center (NIC), and user support group (USG). These distributed organizations are based at Indiana University in the US and Tokyo in Japan. They work as one but support their local (i.e., US or AP) communities, to some extent reducing problems associated with language and time differences across the network's geographic span. STAR TAP provides a network engineer to TransPAC located at the STAR TAP to monitor performance and resolve problems.

Typical questions handled include access, resource availability, service problems, experimental opportunities, and end-system tuning. The USG also markets TransPAC network resources to qualified institutions and researchers and provides education and training opportunities to the TransPAC user community. USG may also facilitate interactions between TransPAC AP users and networks other than the vBNS which may be connected to the STAR TAP.

The USG is staffed at Indiana University by an HPCC applications engineer and support specialist and by peers in APAN at KDD and KT. The US HPCC Support Specialist is the primary point of contact between the vBNS-TransPAC user community and the National Computational Science Alliance (NCSA) and coordinates large-scale multisite experiments involving NCSA Grid sites and participating AP institutions. An example of such a large-scale experiment would be the inclusion of computing resources in APAN in the Gusto computational testbed for an extremely large-scale globally distributed computation of a weather simulation model.

Language and time differences across the TransPAC represent significant barriers to effective end-user support, but are also areas for exploration and problem solving.

TransPAC technical overview

TransPAC provides ATM service (35Mbps non-real time VBR VP) across the KDD and AT&T ATM networks terminating at the Ameritech Lucent switch in Chicago (STAR TAP) and at the APAN Tokyo eXchange Point (XP) switch. The VBR-nrt service, based on AT&T and KDD commercial offerings, is provisioned as a single PVP. TransPAC can configure UBR PVCs within the PVP as needed by HPIIS institutions in the US, the Asia-Pacific, and other STAR TAP-connected networks.

Protocol support for IP Internet services are provided by routers located at the Tokyo XP on the APAN end and by routers associated with networks connected to the STAR TAP at the US end. IP service to the vBNS is provided by MCI through its connection between the vBNS NAP at Downers Grove and the STAR TAP.

The AT&T/KDD ATM VBR-nrt service offering is designed to accommodate bursty data traffic which can tolerate some delay variation. The engineering of the AT&T/KDD network assumes some VBR-nrt connections in the network to be inactive at any given time. This allows flexibility of provisioning VBR-nrt traffic on network trunks, increasing overall utilization and cost efficiency of the trunk network. A trade-off exists between the extent of this provisioning and the cell loss ratio performance. The TransPAC NOC in cooperation with AT&T/KDD monitor the VBR-nrt performance-to-QoS objectives and use this information to determine appropriate provisioning of the service.

Use of the TransPAC network is permitted on a per-institution basis. Routing permissions are established between paired institutions engaged in meritorious research and education application partnership. Routing policy is exercised by means of two methods:

  1. TransPAC-authorized APAN institutions which have capability of ATM VC connection from the border of the institution to the APAN Tokyo XP peer on the TransPAC inter-network. Standard BGP routing at the institution border routes traffic to TransPAC or to commodity VCs appropriately. Figure 3 conceptually depicts this architecture.
  2. APAN institutions which don't have the capability of ATM VC connection to the Tokyo XP present a mix of TransPAC-authorized and unauthorized traffic at the inter-network exchange points. Segregation of the traffic is accomplished by means of policy-based routing using Cisco source/destination route map and route filtering. Depending on the varied requirements for commodity service of APAN constituent inter-networks, the segregation occurs at either the local inter-network exchange point border or at the Tokyo XP. Figure 4 conceptually depicts an inter-network border policy router architecture and Figure 5 depicts policy routing the Tokyo XP.

Currently, the majority of APAN institutional connections are not ATM-based and require traffic segregation by policy routing. Policy routing for Japan's traffic occurs at the Tokyo XP (Figure 5). Other APAN member countries segregate traffic by policy routing at the border of the national inter-network (Figure 4). The APAN consortium is recommending that members migrate to ATM VC connection.


Figure 3. ATM-capable institutions


Figure 4. Non-ATM-capable institutions, inter-network border policy routing


Figure 5. Non-ATM-capable institution, Tokyo XP policy routing

Performance of policy routing based upon source/destination pairs has historically been poor. In the 11.3 IOS, Cisco implemented support for policy-based routing in the fast switching path. Although fast-switched policy routing provides an order of magnitude performance improvement over process-switched implementations, performance is still not adequate to support a full mesh routing matrix of all TransPAC-authorized institutions. CPU utilization increases linearly and rapidly with the position matched within the access lists. Large meshes are not feasible. Cisco intends to marry NetFlow and policy routing technologies in the midterm future, which should generate significant additional performance capability.

In order to accommodate the existing policy routing performance limitations, TransPAC routing is currently limited to paired institutions engaged in collaborative research and education projects. Full mesh routing of all TransPAC-authorized institutions is not supported. As improvements in technology are realized, i.e., NetFlow-based policy routing, full mesh routing will be supported.

General issues

Many of the specific issues encountered in the development of the TransPAC proposal are applicable to all international networking projects, regardless of network endpoints. In this section we discuss four general categories of issues to which developers and implementers of international networks need to pay particular attention: technical issues associated with long distance international networks, support issues associated with networks involving partners from different time zones, application issues, and finally general communication and partnership issues that must be addressed to make any international effort a success. We offer our advice and point out areas where we expect to gain additional experience over the next year with TransPAC.

Technical issues

The most difficult technical issue addressed in TransPAC was the requirement to separate or segment high-performance traffic, which could use the TransPAC link, from commodity traffic that was restricted to standard commercial Internet services. This problem would seem to be common to most international efforts concerned with high-performance computing and communications. The TransPAC technical overview, in the section above, outlines our solution to this problem. Although the technical solution to this problem is not perfect, we are confident it will serve as a solution until a better technological alternative such as Multi-protocol Label Switching (MPLS, sometimes called Tag Switching) is available.

It should be noted that the technical solution is based on a routing policy agreed to by all the TransPAC members. The routing policy and its accompanying AUPs were among the most complex issues addressed by the TransPAC team. The AUPs had to consider the current requirements for participation in the vBNS, based on NSF funding, and current requirements for participation in APAN, based on STA funding. Finally, developing the APAN AUP was complicated by the fact that it was not simply an AUP for national network. It had to be an AUP for an existing set of national networks, with existing rules and understandings about routing and traffic policing. The complete TransPAC agreement, containing the APAN AUP, is available on the TransPAC Web page http://www.transpac.org

Another critical technical issue that will be explored as a part of the TransPAC project is the performance and monitoring of the AT&T service (ATM variable bit rate, non-real time). At present the exact interaction between the ATM VBR-nrt service, the IP transport, and applications is unclear. Tuning of applications is done on an individual basis. We will be working closely with various national initiatives in the US, such as the NLANR performance and measurement efforts, both to develop additional expertise and to offer the insight we gain in TransPAC. We hope to install various monitoring tools, such as OC-3mon at STAR TAP, the Tokyo XP, and various sites within APAN.

Support issues and resource management

An effective support model for both applications and network problems requires that differences in time and language be taken into account. Initial support for US investigators will originate primarily from the TransPAC-US NOC located at Indiana University. Support for APAN investigators will originate from the TransPAC-JP NOC in Tokyo. This will minimize the time and language problems. Network support may prove more difficult, as it must be delivered in real-time between Japan and the United States. This issue is discussed in the Communications Issues section below.

In the TransPAC project, the connection between Japan and the STAR TAP is limited to 35Mbps. Given the number of institutions and projects that wish to access this link, resource management and scheduling are a primary concern. These issues extended beyond the TransPAC infrastructure and involve computing and network resources in US and AP institutions.

At the coarsest level, the TransPAC policy router allows only TransPAC-approved institutional interactions across the international link. Decisions about resource allocations (how much of the available bandwidth can a project use for how long) are made by the TMC during the project approval process. Actual scheduling of link resources is the responsibility of the TransPAC-US NOC at Indiana University. As the TransPAC-JP NOC in Tokyo develops, this responsibility will be shared between each constituent NOC of the overall TransPAC NOC. The scheduling of site-specific resources remains the responsibility of the sites engaged in the HPCC research. The TransPAC HPCC Support Specialist interacts with the sites if problems arise.

Application issues

A number of specific issues related to application development and execution in the TransPAC environment have been identified. Some of these problems are being addressed through staffing. For instance, HPCC consultants are provided to assist the US and AP communities and to coordinate large-scale network computing experiments. Issues related to training and information exchange are covered through TransPAC workshops and meetings. Other issues include

  • Documentation concerning application development and execution in languages appropriate to the TransPAC user community
  • Appropriate parallel computing and distributed resource management tool kits, such as Nexus and Globus from Argonne National Laboratories
  • Long-term support and coordination for testbed projects (IPv6, multicast, streaming media and QoS, Web caching, tele-immersion, and globally distributed high-performance computing)
  • Software instrumentation for performance monitoring
  • Resource scheduling for large bandwidth applications
  • Problem resolution in real time during application start-up and production runs

In such a diverse user community, other problems and issues are likely to arise, but with good communication between users and TransPAC support groups, it is expected that these can be resolved quickly.

Communications issues

The construction of an international network is a very complex undertaking. Even with the best of intentions, communication problems arise. This is particularly serious when network problems arise that need immediate attention. There is much common technical language in computing and networking. However, the almost 12-hour time differential between Japan and the US complicates the issue. Most of the APAN technical staff working on TransPAC have a working grasp of English and a clear US representatives as it relates to Japanese or Korean. Consequently, we are forced to use English as a communication medium for TransPAC.

In an attempt to address these problems, each constituent NOC of the TransPAC NOC has designated an on-call person. This person is available outside the 8 a.m.-5 p.m. typical workday. If problems arise, the on-call person is paged and responds within 30 minutes. In the United States, the on-call person is learning the technical details of the TransPAC network and developing an ability to natively communicate with the APAN technical peers, at least at the most rudimentary level.

The cost of communication, other than e-mail, is also an issue. A phone call to Japan for an extended technical discussion is quite expensive. One of the initial technical projects TransPAC is undertaking is to test the quality and reliability of voice over IP technology to reduce these costs. Experimentation will begin with IP-based trans-Pacific video conferencing to reduce travel costs, although the time difference again complicates this problem.

The final point to be made regarding communications seems obvious now, but has often been overlooked. Delicate international policy issues require more time for consideration and must be considered from a variety of perspectives. Initially, the US participants viewed TransPAC as a project of the US NSF. However, as was often pointed out by the APAN representatives, the NSF contribution to the total TransPAC budget is only about half. This "one-sided" view of TransPAC by the US participants leads to occasional difficulties, particularly in negotiating policy-related issues such as network access and AUPs. When the symmetry of funding was translated into a symmetry of responsibility, the development of fair and equitable policies became a much simpler matter.

Conclusions

TransPAC is a work in progress, but at this early point some things seem clear. There were significant technical problems in constructing the TransPAC network, particularly with the policy routing (traffic segregation) concerns. These problems have not been resolved in a completely satisfactory manner. But adequate temporary solutions exist, and more robust, longer term solutions are anticipated. The technical concerns we had during TransPAC development have been resolved for the present time.

It is felt that the support model is more than adequate. However, it remains largely untested. It is believed that it effectively addresses issues that other international efforts have raised concerning application and network support. But the effectiveness of these support mechanisms, as measured by the quality of service facilitated by TransPAC, is still unmeasured.

Communications between the TransPAC partners remains the most difficult issue to overcome on a daily basis. Simple time differences sometimes directly interfere with timely communication. Delicate policy matters continue to take a long time to resolve to the satisfaction of all partners. This is an issue that all the TransPAC staff are working on.

In summary, it is felt that these technical solutions will be successful and the support mechanisms will facilitate the scientific interactions that are the goal of TransPAC. Communications among the various partners and more timely and effective ways to facilitate interactions will continue to be areas of concern.

Current and historical information about TransPAC, including various working papers and current committee assignments, can be found at http://www.transpac.org

[INET'98] [ Up ][Prev][Next]