![[INET'99]](/web/20100311064333im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/inet99sm.gif)
![[ Up ]](/web/20100311064333im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/up.gif)
![[Prev]](/web/20100311064333im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/prev.gif)
![[Next]](/web/20100311064333im_/http://www.isoc.org/isoc/conferences/inet/99/proceedings/next.gif)
Roberto SABATINO <roberto.sabatino@dante.org.uk>
Jan NOVAK <jan.novak@dante.org.uk>
DANTE
United Kingdom
This paper addresses the implementation of a pan-European network to support cooperative research amongst European researchers -- TEN-155. Besides providing a high speed Internet protocol (IP) service, the purpose of TEN-155 is also to support research in networking by providing an international test bed for advanced networking technologies (the Quantum Test Programme) and by providing virtual private networks, with dedicated and guaranteed bandwidth, for specific research projects in countries connected to TEN-155.
Predecessors to TEN-155 were TEN-34 and EuropaNet. TEN-155 supersedes these two networks not only in terms of capacity offered but also in terms of the managed bandwidth service (MBS) offered.
This paper will illustrate the use of asynchronous transfer mode (ATM) technology to support the MBS and the Quantum Test Programme in coexistence with the standard best-efforts IP service.
The success of EuropaNet [1] and TEN-34 [2] has demonstrated that a dedicated networking infrastructure to the European research community is essential for successful cooperation among European researchers. Both these networks have had a short lifetime, mainly a result of the excessive cost of international capacity. Nevertheless, with the help of funding from the European Commission (EC), it has been possible to deploy them and demonstrate their vital importance to the research community. EC funding for TEN-34 ended in July 1998, and as a consequence, effort has been put into the Quantum project, which led to the deployment of a replacement network for TEN-34: TEN-155.
This paper will outline the Quantum project and describe in detail the resulting TEN-155 network which supersedes TEN-34 in terms of available bandwidth and above all in the ability to offer a managed bandwidth service (MBS) to guarantee end-to-end quality of service. TEN-155 makes use of asynchronous transfer mode (ATM), considered the most effective technology to offer guaranteed capacity end-to-end.
A section is dedicated to the Quantum Test Programme (QTP), the purpose of which is to evaluate new and emerging technologies and investigate the possibility of their deployment on the production network.
Finally, a description of MBS in terms of its organization and development phase will be provided.
The Quantum project (Quality Network Technology for User-Oriented Multimedia, http://www.dante.net/quantum) foresees the exploration and implementation of providing quality of service across a pan-European network of very high speed. The Quantum project also calls for experimentation of new IP (Internet protocol) and ATM technology using a wide-area and international test network.
TEN-155 is the operational network built as a result of the Quantum project.
A group of 16 national research networks (NRNs) and one regional network, coordinated by DANTE, are responsible for the Quantum project which is cofunded under a joint initiative by DGXIII (Telematics for Applications, Esprit, and ACTS) of the EC.
DANTE is a not-for-profit company set up in 1993 by European NRN organizations. DANTE plans, builds, and manages advanced networking services for the European research community.
The physical topology of TEN-155 was dictated by a combination of the following factors:
Following the issue of an open tender, several offers from the tenderers were evaluated, and in August 1998 a contract was awarded to Unisource Belgium for the provision of a number of 155 Mbps synchronous digital hierarchy (SDH) circuits, the supply and management of an ATM service in all TEN-155 countries, and facilities management in most of them. Unisource Belgium relies on KPN (The Netherlands) for the implementation and technical support of these services. Figure1 illustrates the physical topology of TEN-155.
Figure 1. TEN-155 physical topology
The figure outlines the existence of transit nodes (AT CH DE FR IT NL SE UK), interconnected via nonprotected SDH STM-1 (synchronous transport module 1) circuits, to which NRNs are directly connected in addition to international circuits to peripheral sites (HU SI GR ES CZ LU BE PT PL IE) or other transit sites.
When comparing TEN-155 to TEN-34, a huge increase in available capacity is immediately noticeable. On TEN-34 the highest bandwidth available was a 34 Mbps leased line between Germany and Switzerland; now it is 155 Mbps on many circuits. In addition, even in countries where capacity still has relatively high prices (Greece, Portugal, Slovenia) it has been possible to deploy 34 Mbps circuits. As a result of the 1998 liberalization of European telecom services, the overall cost of TEN-155 is similar to that of TEN-34, but the amount of available bandwidth is much higher.
The engineering of the TEN-155 network took into account that:
These requirements and conditions led to the following design decisions:
Figure 2 illustrates an example PoP setup. The example outlines the benefits of a full mesh of UBR-like PVCs to carry the IP traffic: The capacity (hence, the number of interfaces) required between switch and router is equivalent to the capacity of the connected NRNs, in this case DFN (155Mbps) and Cesnet (34Mbps). Therefore three STM-1 connections between switch and router are required. In the event of hop-by-hop PVCs, the capacity required between switch and router will have to accommodate all the transit traffic; that is, nine STM-1 interfaces will be necessary.
Figure 2. TEN-155 DE PoP setup
The figure also outlines that two PoP workstations are available: one is to host operational services such as Mbone and advanced monitoring, the other for experimentation within the QTP.
The UBR-like PVCs are implemented by SBR3 (VBR, with SCR=10cps, PCR=line rate). Cell tagging is enabled on these PVCs, so all cells that exceed the contract (virtually all cells, given SCR=10cps) are tagged with CLP=1. Cell tagging is not enabled on the CBR PVCs; hence the cells are transmitted with CLP=0. In cases of congestion, the switch will discard CLP=1 cells first, and early packet discard (EPD) is implemented. This ensures that complete AAL-5 frames (i.e., whole IP packets) are discarded, which has the beneficial effect of removing entirely from the network cells that no longer serve any useful purpose. In fact, IP packets are mapped onto AAL-5 frames, and if a single cell is dropped, the whole corresponding IP packet is invalid and will result in a CRC (cyclic redundancy check) error detected on the router which in turn represents a serious waste of bandwidth. By removing a complete AAL-5 frame, the network is able to entirely accommodate the next incoming packet.
This mechanism has been thoroughly tested in a laboratory environment, and in cases of 200 percent congestion, the result was to have 100 percent utilization on the congested link and more than 95 percent goodput. Without the EPD mechanism, situations of congestion as low as 110 percent may result in approximately 15 percent goodput on some switches. Several switches have been tested, and figure 3 illustrates a simple setup that was used for testing the EPD mechanism.
Figure 3. EPD test setup
TTCP (test transmission control protocol), with UDP (user datagram protocol), sessions between WS-1 and WS-3 and between WS-2 and WS-3 were run at the same time; therefore the bottleneck was the physical STM-1 connection between the switch and WS-3. The goodput was measured on WS-3 and was calculated by adding the results reported by WS-3 for the two TTCP sessions. The ATM software on the workstation was also able to report the number of CRC errors occurring when EPD was not enabled.
The detailed results cannot be published because of nondisclosure agreements with the switch suppliers.
On IP level, for the implementation of the best-effort IP service, TEN-155 is configured as one autonomous system with the necessary full mesh of iBGP peerings using loop-backs. This avoids loss of connectivity in the event of an ATM PVC outage. TEN-155 uses OSPF (open shortest path first) as internal routing protocol with OSPF cost of ATM PVCs configured to reflect the underlying physical and ATM topology. As ATM level re-routing of PVCs is not configured; this setup ensures:
The rule applied for calculating the OSPF cost between two nodes (see figure 4) is:
100 for the first physical line, 20 for each subsequent ATM level hop. Or
100 + 20 * (number_of_hops - 1)
In other words, two IP level hops always have a longer path than one direct ATM VC between any two nodes, as the highest ATM hop count in the backbone does not exceed three. Therefore two IP level hops between two nodes would have cost 200, while the maximum cost for a direct ATM PVC is 140.
Figure 4. TEN-155 OSPF setup
One of the reasons for deploying TEN-155 was to provide more bandwidth for European research traffic; therefore, an analysis of the immediate impact of providing more capacity was carried out.
Table 1 shows an extract (a full matrix would not be easily readable) of inter-NRN traffic measurements on TEN-34 in November 1998 (the last month of full operation of TEN-34) and on TEN-155 in January 1999. The values shown are monthly daytime averages of traffic between NRNs expressed in Mbps and are derived from DANTE's inter-NRN traffic statistics package [3], initially deployed on TEN-34.
From the table it is clear that the total traffic on TEN-34 was much higher, mainly for two reasons:
On the other hand, a significant increase of traffic between the NRNs that did migrate to TEN-155 is noticeable. This increase is partly due to the natural increase of bandwidth usage on the Internet, but also to the availability of more bandwidth, which removes bottlenecks from the network. On TEN-34, traffic between Nordunet(SE), SURFnet(NL), DFN(DE), and Janet(UK) was significantly affected by these bottlenecks.
| TEN-34, November 1998 | - | TEN-155, January 1999 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| src\dst | DE | NL | SE | UK | - | DE | NL | SE | UK |
| DE | - | 1.04 | 1.89 | 2.51 | - | - | 2.38 | 3.06 | 2.50 |
| NL | 0.96 | - | 1.18 | 0.96 | - | 2.74 | - | 1.87 | 1.34 |
| SE | 3.04 | 2.50 | - | 2.09 | - | 5.74 | 4.10 | - | 2.77 |
| UK | 4.81 | 1.87 | 2.24 | - | - | 5.04 | 2.40 | 2.50 | - |
| total of extract: 25.09 | - | total of extract: 36.44 | |||||||
| total of TEN-34: 192.27 | - | total of TEN-155: 114.19 | |||||||
The table emphasizes that inter-NRN traffic has increased by almost 50 percent, outlining both the benefits and need for more bandwidth to support the European research community. It is expected that once all NRNs have migrated to TEN-155 the increase of inter-NRN traffic will be even more significant.
Prices of intercontinental circuits are falling, making it more feasible to obtain high-capacity connections from European cities to the United States. However, it is still more economic for some countries to organize the setup of shared transatlantic capacity. To this end, DANTE has organized the procurement of a SDH/STM-1 circuit from Frankfurt to New York. The transatlantic SDH/STM-1 terminates on ATM switches (the TEN-155 switch in Frankfurt, and another switch managed by DANTE in New York). The service offered to the subscribing NRNs is ATM directly into the NRN, so it is as if the NRN had its own transatlantic link on IP level. The advantages of this are both economic and technical. From the economic point of view it is cheaper to purchase SDH/STM-1 than to separately purchase capacities in the order of 15-40 Mbps, while from the technical point of view the setup allows guaranteed capacity to the subscribing NRNs and simplifies the IP level setup for TEN-155.
Another add-on to the TEN-155 network is the connection of Israel and Cyprus as a result of the EC-approved Q-MED project. DANTE is coordinating the connection of these countries to TEN-155 which in the case of Israel will be implemented by an E3 circuit from London to Tel Aviv. The connection is engineered so as to offer to Israel the same services available to the NRNs taking part in the Quantum project. Planning of the connection to Cyprus is still in progress.
As previously mentioned, the QTP is a substantial component of the Quantum project. The main objectives of QTP are to carry out evaluation of advanced networking technology and network-related technology on a dedicated test bed and migrate where possible the technology to the production service. The dedicated test bed is obtained by setting up a set of PVCs to create a VPN. QTP activities are carried out by a joint DANTE-TERENA task force, TF-TANT, which carries out evaluation of technologies that are also relevant to the TERENA working group on lower layer technologies (WG-LLT). The activities relevant to QTP are as follows:
At the time of writing, the QTP activity has just started so no results are published in this paper. However, these will be available at the DANTE website (http://www.dante.net/quantum/qtp)
Particular priority has been assigned to native IP multicast, in order to gain experience with PIM (protocol-independent multicast) [4, 5] and MBGP (multicast border gateway protocol) [6] and transfer this technology to the production network. Some NRNs already deploy PIM and MBGP in their own domain, but in a multidomain environment experimentation is needed before it can be deployed without the risk of disrupting production service. Therefore DANTE and QTP will commit efforts and resources to the experimentation of PIM and MBGP using workstations and GateD initially to gain experience with the concepts. The scheme of experimental setup is shown in figure 5 below. In parallel, the procurement of test routers, of the same type as the production routers, to deploy in the dedicated VPN will be organized.
Testing of PIM and MBGP will then resume on this infrastructure. It is estimated that by the end of 1999, TEN-155 will support a native IP Mbone.
Currently multicast is supported on TEN-155 by deployment of a European Mbone, described in detail in Nobak and Sabatino (1998) [7] for TEN-34 and migrated to TEN-155 (http://www.dante.net/mbone). This Mbone is implemented by setting up DVMRP (distance vector multicast routing protocol) tunnels between TEN-155 PoP workstations and multicast capable routers in the NRNs.
Figure 5. Experimental IP multicast setup
IP over ATM furthermore is viewed as high priority because the intention is to make use of the most suitable ATCs for carrying IP traffic and guaranteed traffic. These have been initially identified as SBR-3 with SCR=~0 and DBR (deterministic bit rate), but also the use of SBR-3 with SCR0, SBR-2, and ABR (available bit rate) needs to be thoroughly evaluated, in cooperation with the supplier of the ATM service (KPN).
In July 1999 there is a contractual obligation of the supplier of the service (KPN) to provide ATM signaling -- but it is not yet defined how it is possible to use it in TEN-155, especially for the MBS service which would require end-to-end signaling, hence across management domains. Therefore the QTP will dedicate resources to the investigation of the deployment of SVCs.
ATM is not viewed as the only way to provide guaranteed capacity to users. In fact, the networking community is considering both ATM and IP mechanisms. The performance and behavior of ATM QoS is well known (if the most simple ATM technology is used), while the IP QoS/class-of-service mechanisms are still under development. However, there is strong belief in their potential. Consequently QTP will investigate the developments of diff-serv, RSVP, and RSVP to ATM signaling mapping. One issue that will need immediate investigation is the coexistence of a fully meshed ATM backbone and the IP QoS mechanisms and techniques for their interoperability.
The MBS (http://www.dante.net/mbs) is one of the major components of the Quantum Project. It aims at enabling end-to-end guaranteed capacity between sets of hosts or networks across Europe in order to create VPNs with dedicated bandwidth in the support of pan-European research projects.
The capability for establishing VCs/VPNs and closing them down at short notice or according to a predefined timetable is an essential component of the MBS. Setting up operational and management procedures and tools is the main challenge of this activity, as from the technical point of view it is possible to create VPNs using a set of ATM VCs between the participating nodes. Before the MBS becomes an operational service, it will undergo a pilot or alpha test phase with selected users. These users have been identified within ERCIM [8], while the project to make use of the MBS in the alpha phase is the MECCANO [9] project. The aim of the alpha test phase is to identify the exact operational procedure requirements and necessary support tools. This phase is expected to last from January through March 1999. Once the alpha phase has been considered a success, a second phase (beta) will follow for two months, expanding the number of projects and NRNs involved.
The purpose of the beta phase is to prepare the service for production by verifying procedures and reducing time consumed in critical tasks. Production service is expected before June 1999. The QTP will make use of the MBS to set up the VPNs necessary to carry out evaluation of networking technologies.
The Quantum project has been successful in deploying a pan-European network for research, providing a huge increase in capacity when compared with its predecessors and at a similar price, mainly because of developments in the European telecommunications market. From the technical point of view, TEN-155 has been designed to make efficient use of the available bandwidth and fair sharing of the bandwidth in situations of congestion. These targets have been met by combining different ATCs, and the results obtained in test laboratories confirmed the theoretical expectations.
Rollout of the MBS and the work carried out within the Quantum Test Programme is expected to enhance even further the capabilities of the network, with the deployment of native multicast and developments in the area of the coexistence of differentiated and integrated services.
The authors would like to thank the European Commission for cofunding the Quantum Project. Numerous technical experts from the European National Research Networks have provided advice on the engineering of the TEN-155 network, and also carried out physical work beyond any expectation for the deployment of the network. Members of TF-TANT, the task force that carries out the technical evaluation of networking technology in the Quantum Test Programme, provided continuous input to the introduction of new services on the operational network.