Che-Hoo Cheng <chcheng@cuhk.edu.hk>
Head, Data Communications and Networking Section
Information Technology Service Unit
Chinese University of Hong Kong
Shatin, N.T., Hong Kong
Tel: +852 2609-8848
Fax: +852 2603-5001
http://www.cuhk.edu.hk/hkix/
Most people think that the United States is the "backbone" of the Internet. Many ISPs in other countries choose to connect to the Internet by leasing circuits to the United States. With this "star topology," local traffic within a country or a city may have to be routed through the United States if there are no local connections among local ISPs. This is highly undesirable because the long-distance circuits are very expensive and are often of relatively slow speed.
The situation in Hong Kong may not be unique. There are more than 40 ISPs in this small city, more than 10 of which have their own links to overseas, mainly the United States. On the other hand, local circuits are quite affordable because of the short distances involved, and this can help the development of local high-speed connections. In view of this, The Chinese University of Hong Kong (CUHK) made use of its own resources to set up a neutral interconnection point called Hong Kong Internet eXchange (HKIX), mainly for the routing of intra-Hong Kong traffic. This has proved to be very successful. As of late March 1996, 26 commercial ISPs are connected. Among the 26, there are even regional or global ISPs such as AT&T, IBM Global Network and Global SprintLink.
In this paper, the key reasons for the success of HKIX are presented. Other issues, including the history, the technical aspects, the problems, the funding issue and the future of HKIX, are also addressed.
There is no doubt that the Internet is blooming in a very fast pace around the world. Many organizations and individuals are rushing to the Internet. It is now very difficult to find a single well-known multinational company that does not have a presence on the Internet.
Although the Internet is a worldwide computer network, many people still think that the United States is the "backbone" of the Internet. In fact, more than 60 percent of the users and nodes of the Internet are located inside the United States. And the United States carries most of the intercontinental traffic. Because the Internet is a loosely controlled network, the connections among different networks are often arbitrary, at least from an outsiders' point of view. In fact, nobody in the world can have the exact configuration of who is connecting to whom. Many ISPs in other countries choose to connect to the Internet by leasing circuits to the United States. Although intraregional network connections are gradually being set up, this has not occurred to much extent in the Asia Pacific region. Of course, the main reason for this is because of the high cost of intraregional leased circuits and the low volume of traffic among the countries within the region. But with this "star topology," local traffic within a country or a city may have to be routed through the United States if there are no local connections among local ISPs. This is highly undesirable because the long-distance circuits are very expensive. Therefore, setting up Internet eXchange (IX) for the routing of local traffic surely can benefit all parties involved. But how can arrangements for interconnecting local ISPs be made, especially when there are many ISPs involved? Many unsuccessful stories have been heard in some regions and countries. The reasons are varied, but it is believed that a lot of politics and competition are involved in most of the cases.
The first 64kbps Internet link in Hong Kong was set up in September 1991 by The Chinese University of Hong Kong (CUHK). The link started to be shared by all higher educational institutions in Hong Kong in 1992. The network linking up all institutions is called HARNET (Hongkong Academic and Research NETwork). In September 1993, the link was upgraded to 128kbps after some delay. Before then, there were very few commercial elements within the Internet community in Hong Kong. In late 1993, two small commercial Internet service providers (ISPs) were set up with their own 64kbps links to the United States, and they started to change the status quo. As one of them (HK Supernet) was a spin-off of a university (The Hong Kong University of Science and Technology), it had a direct connection to HARNET from the very beginning. The other one (HKIGS) was a small independent ISP, so it had no connections to the other two networks. At that time, there was very little need to have full interconnections because the gold mine was at the other side of the ocean and there was little value to connect to local sites. So people could live with that situation then.
In 1994, Internet on the nonacademic side continued to grow. A number of small ISPs were set up to serve mostly individual subscribers. The new ones did not have their own links to the United States. Instead, they were just piggybacked to HKIGS. But the growth rate was still not very high. The blooming of Internet development in Hong Kong in fact happened in 1995. In February and March of that year, two incidences (mostly related to the licensing requirements of operating an ISP in Hong Kong) caused the close-down of quite a number of ISPs, but at the same time awakened the general public to the Internet. After the licensing issue was clarified, most ISPs were back in operation. At the same time, many ISPs were entering the market. Some were even established by listed companies or large multinational companies. Many companies in Hong Kong, no matter how large or how small, started to consider having a presence on the Internet. On the other hand, students, professionals and computer hobbyists are rushing to join the Internet. This started to create more and more local content and importance, and intra-Hong Kong communications became more and more needed.
At the end of 1995, more than 40 ISPs were doing business. Quite a number of them, especially the larger ones, have direct links to overseas. Others were just piggyback ISPs relying on other ISPs that had local presence to do transit for them.
As mentioned above, there was only one interconnection initially. In September 1994, HKIGS set up a local T1 circuit to CUHK allowing their customers including those of their downstream ISPs to have more direct and faster communications with HARNET. HKIGS was in charge of the rental of the local T1 circuit (around the United States, $1,200/month then), and CUHK provided the router port for the connection. As can be seen, all parties involved gained benefits. Still, HK Supernet and HKIGS were not connected locally in any way because HARNET could not do transit for them.
Things started to change dramatically in 1995. With many more ISPs entering the business, they needed efficient network infrastructure very much in order to lower the cost of operations for better competitiveness. They could not afford to route intra-Hong Kong traffic overseas because their overseas links were expensive and relatively slow. Although all of them wanted to set up interconnections, they could not do it easily, mostly because they are all competitors to one another and it was impossible to have most of them come together and discuss interconnection. Having full-mesh interconnections among them was out of the question at that time. In view of this, and having the precedence of connecting to HKIGS, CUHK saw the need to do something for Internet development in Hong Kong again and set up the framework of HKIX (see Figure 1). CUHK started to negotiate with newly established local ISPs that had direct links to overseas. Most of them agreed with the idea very much and committed to order circuits to CUHK immediately.
After all the hard work, the first two HKIX connections (LinkAGE Online and Global Link) were set up in April 1995. The new arrangement was that the leased circuits to CUHK and the routers placed here were provided by the participating ISPs. And CUHK was responsible for providing space, electricity, air conditioning and a simple Ethernet network to connecting all routers of the participants. HKIGS was migrated to this new HKIX connection a little bit later after its router was delivered to CUHK. HK Supernet was connected to HKIX in October 1995. (For the current list of participants, please refer to http://www.cuhk.edu.hk/hkix/connected.html.) As of late March 1996, 26 commercial ISPs were connected, including some regional and global ISPs such as Global SprintLink, IBM Global Network, AT&T and Cable & Wireless (represented by Hongkong Telecom).
HKIX is a layer 2 interconnection point. The physical part of the HKIX is very simple. Each participant leases a dedicated circuit to CUHK and places a router here. CUHK provides the Ethernet connections among the routers. The minimum speed of the leased circuit is 512kbps. By July 1996, the minimum speed will be T1 (1.536Mbps). Initially, the HKIX Ethernet was just a coaxial cable segment. After more than 10 ISPs were connected, the shared Ethernet became quite congested, so the coaxial cable was replaced by a Cisco/Kalpana EtherSwitch PRO16 in December 1995 (see Figure 2).
As of late March 1996, the highest speed to HKIX is dual T1. One T3 connection is on order. In order to cater for better support of direct ATM connections, Cisco Catalyst 5000 will be used to replace the PRO16 in April 1996. Initially, the first T3 connection will be supported by the 100Mbps Ethernet port of the switch. Later when ATM switch is installed, this T3 connection and other new T3 connections will be migrated to ATM (see Figure 3).
The initial routing setup is also simple if the participants have enough knowledge of Internet routing, especially BGP4. As it is desirable to have mandatory multilateral peering agreement to ensure greatest possible benefits to all, a router server (Cisco 2501) is used to provide a single view of routing for all participants. Each router on HKIX belongs to the same autonomous system (AS) as the corresponding ISP. It must peer with the route server using BGP4 via the HKIX Ethernet and announce all routes of the internal networks and downstreams. It must also accept all routes distributed by the route server.
This arrangement aligns with the philosophy that everyone on HKIX is equal and will be treated fairly. It also makes sure that intra-Hong Kong traffic is routed through the fastest possible path and all participants can gain the greatest possible benefits.
The current way to control routing is by the IP network access lists on the route server, which filter incoming BGP routes. As more participants join HKIX, this method does not scale. Materials of Routing Arbiter Project are being studied to see if RA database and programs suit the purpose and are easily implemented and managed.
As for fully utilizing the connections to HKIX, all participants are encouraged to have more cooperation through the HKIX, e.g., news exchange and Domain Name System (DNS) backup. Although not much has been done so far except news exchange, it is seen that cooperation among ISPs is gradually increasing, although slowly.
Another important philosophy is that all participants must have global Internet connectivity independent of HKIX facilities. As the initial setup was very simple and the lowest possible investment was made for earliest possible establishment, this philosophy was thought out to make sure that HKIX was not used as the sole connection to the outside world for better reliability. Although the philosophy is still valid, it is encouraged that participants seek backup global Internet connectivity via HKIX.
There is no doubt that the HKIX project is extremely successful. As a matter of fact, nobody could imagine this at the very beginning. To look back, the reasons may include the following:
The requirements for ISPs to join HKIX are as follows:
ISPs must fulfill all these requirements in order to join HKIX.
Of course, HKIX is not without problems. When a customer of ISP A switches to ISP B, we must be very careful about the modification of access lists because this will affect the reachability of that customer on HKIX. We sometimes need to act as a mediator when the customer holds an IP network address sub-block of ISP A and wants to use it from ISP B. Fortunately, all such incidences so far have been handled without many difficulties.
Another problem is that many network engineers of local ISPs do not have experience with Internet routing. They often use RIP as their interior gateway protocol (IGP) and do not know CIDR. We have spent significant amounts of time to deal with them when setting up the HKIX routing using BGP4. But with more books and documents available to help them to understand Internet routing, CIDR and BGP4, they seem to have caught up quite a bit.
The most critical problem is that HKIX is still a project of CUHK starting from the very beginning. The project was initiated by CUHK purely as a community service to Hong Kong. Everything is done with the goodwill of CUHK. The policy is set up by CUHK with reference to the policies of other exchanges and after consultation with participants and other experts in this area. Anyway, the final say is still in the hands of CUHK. Although we have tried our best to act fairly and make decisions largely based on the amount of benefits to the whole community, not just to individual parties, complaints and challenges from some ISPs are received from time to time. It seems that whenever they see a potential threat to their business, they will stand up and fight against it. They consider HKIX as a monopoly and try to hold control of it because they fear that HKIX will turn into another major competitor of theirs and grab significant market share very easily because of the name of HKIX and CUHK. But at the same time, some other ISPs want us to maintain the control because they think we are more neutral than anyone else. So we are somehow facing pressure from both sides.
With the issue of whether to support the piggyback ISPs, we faced a lot of challenges from some first-tier ISPs that have they own links to overseas. (Initially, HKIX served first-tier ISPs only.) They feared that if HKIX supports those piggyback ISPs, HKIX would expand to take a too important role in Hong Kong and could not "resist the temptation" to turn into a major transit provider that would immediately have those piggyback ISPs as customers. On the other hand, we were facing pressure from some newly established piggyback ISPs and some large transit providers to open up the HKIX for those piggyback ISPs. Finally, after an unofficial opinion poll and consultation with some neutral experts and individuals, it was decided to "open up" the HKIX. Still, other requirements for ISPs to join HKIX were not changed.
In the long run, a more formal organization has to be set up to oversee the management of HKIX. Of course, the membership should include all the HKIX participants so as to ensure that the interests of all parties can be taken care of. Besides being in charge of the operations of HKIX, CUHK probably will still play a major role on the management side, especially when there are major arguments and issues that cannot be resolved among the members. CUHK will still care more about the benefits of the whole community.
Another problem of HKIX is funding. HKIX is totally "funded" by CUHK so far. The resources used are all provided by CUHK, including staffing (part-time work only), network equipment, Ethernet cables, space, electricity and air conditioning. Although the participants need to provide their own leased circuits and router equipment, they do not need to pay any charges to CUHK for the service. In return, they receive only best-effort service from us. As the scale becomes larger and larger and HKIX becomes more and more important, we need dedicated staff to handle all the work. If we cannot have dedicated staff, the reliability of HKIX will become questionable. On the other hand, as HKIX migrates to more sophisticated infrastructure being built up with ATM switch and accompanying Ethernet switch, more resources are needed for network equipment, the expenses of which cannot easily be absorbed by CUHK.
The most logical way to obtain funding is to implement charging for services to cover the cost of operations. The other way is to request support from the government. As of late March 1996, the chances of getting funding from the government are quite high. If the funding is approved, the cost of operations of HKIX can be covered for two more years. After these two years, HKIX probably will need to implement charging in order to sustain operations.
When Internet grows to a state that is out of control of everybody, it may be time to do something to make it more controllable so that the growth can be sustained. Setting up one and only one local IX for the routing of local traffic is one thing that should be done in every major country and city that has a significant amount of Internet users and information content. If local IX is set up, the growth of long-distance bandwidth can be slowed down a little bit. It surely can help the Internet to develop more healthily.
But in a highly competitive community such as Hong Kong, setting up such important infrastructure as HKIX can hardly be achieved by getting all parties involved together and having everything agreed upon by all before setting it up. Everyone is everyone's competitor, so total agreement can hardly be reached. Doing it with goodwill by a relative neutral party may be the most effective way to implement it. After everything is built up and running smoothly, it may be time to hand the management over to the participants. But still, the local IX should be operated at a neutral point in order for it to survive.