February 1997

The Namibian Internet Development Foundation has been operating the first IP link from Namibia to the Internet since September 1995 on a non-profit basis, as asynchoneous land line to the South African aca- demic network. No outside funding was used and the effort is self sus- tainable. Overloading of the leased line was initially caused by faulty DNS service, later by routing problems beyond NAMIDEF's con- trol. Upgrading of the leased line modems and router-to-router com- pression provided some relief but caused other problems. No serious hardware failure occured. 4 commercial Internet Service Providers have entered the market with a combined bandwidth of 512 Kbps. A DNS registry is currently operated free of charge for all Internet Service Providers in Namibia. Maintenance of an ISP operation such as NAMIDEF's can be done on a non profit basis but highly motivated vol- unteers have to be on site or salaried staff. The Internet Service operation is being outsourced to a commercial provider in order to allow NAMIDEF staff to concentrate on Internet development in Namibia. This article describes the development of the Internet in Namibia, gives an overview the problems encountered and describes lessons learned from the project.

1. Introduction

1.1. Electronic Networking

Electronic Networking has grown exponentially over the last 10 years, originating from the ARPANet in the United States which has become the Internet, a world wide network of computer systems using common protocols.
This article is to give a short overview of the history and the present state of Electronic Networking in Namibia.

1.2. Namibia

Namibia is located on the south western coast of Africa sharing borders with Angola in the north, Zambia in the northeast, Botswana in the east and South Africa in the south. It has a territory of about 500000 square kilometers and in 1990 had about 1.4 Million inhabitants. 10% of the population live in the capital Windhoek and 50% in the rural north.
The infrastructure in Namibia is much better than in the other sub- saharan countries. The major rural exchanges have been replaced by digital exchanges by the end of 1995 and fibre optic cables are being laid everywhere.

2. Early Developments

Electronic Networking in Namibia began in ``1990''. The computer center of the Rhodes University in Grahamstown, South Africa, had established a dialup connection to the US at about that time, using a store and forward system, FIDO at first, switching to ``UUCP'' later.
The main practical difference as shown for example by ``Korver'' between FIDO and ``UUCP'' is that UUCP can be used as a pure transport layer without concern to the actual addressing mechanism. That means one can use any ``RFC822/1023/1036'' compliant Mail Transfer Agent already in use on the Internet making the transition to full Internet much easier. FIDO uses its own addressing standard which is not compatible with the one used on the Internet (``RFC822/1023/1036''). (-- There are gateway programs between FIDO and ``RFC822/1023/1036'' but they add additional layers of complications--)
The director of the computer center, who is now the national manager of the South African academic network, (UNINET-ZA) first offered the author a direct dial in account on one of their Unix systems. However it incurred high telephone costs due to the time spent on line.
This led 1991 to the installation of UUPC/extended, the MS-DOS implementation of ``UUCP'', written by Drew Derbyshire (-- UUPC/extended conforms to the ``RFC822/1023/1036'' standard in addition to being a pure transport mechanism.--)
In 1992 the University of Namibia's Department of Computer Science expressed interest. In early 1993 the department installed a commercial Unix system which formed the backbone of a TCP/IP based ethernet in the student laboratory. A ``UUCP'' dial in account was created in Grahamstown for this computer and the domain NA was registered with the Internet Network Information Center. In the meantime a leased line from Grahamstown to the United States had come into operation allowing direct internet access.
Sophisticated (``RFC822/1023/1036'' compliant) routing software (smail) was installed on the computer system at the University which allowed for transparent compression of the traffic between the two systems connected via long distance telephone calls.
That system also used a new implementation of the ``UUCP'' software written by Ian Taylor. It introduces a protocol variant, which under certain circumstances can double the throughput. It can also pick up an interrupted transfer which did not play a major role, but if the line quality is poor it can save significantly on costs.
Entries were made in the Domain Name Service (DNS) databases so that all Internet traffic destined for NA was routed through this gateway with a view to future expansion.
Some individuals expressed interest in 1992. To facilitate UUPC's complex installation procedure an installation package was developed by the author based on a similar package that had been written by Mike Lawrie.
In the middle of 1993 an early version of the linux operating system was installed on the author's 80386 system at home. Linux is a complete rewrite of the Unix kernel and is available free of charge as source code. It spread quickly over the Internet and numerous packages were ported to make it a complete Unix compatible operating system at a very low price (typically between 20 and 50 US Dollars)

3. Namibian Internet Development Foundation

3.1. NAMIDEF

In the middle of 1994 so many individuals and organizations were connected that the Namibian Internet Development Foundation (NAMIDEF) was founded as an Association not for Gain.
The idea to get all major players but also small organizations and individuals together under one neutral umbrella was well received. UNICEF and the two largest computer dealers in the country immediately expressed their support financially and in kind resulting in a slow but steady rise in subscriptions. The startup and administrative costs were low and with the step wise, incremental growth approach only a small financial risk was incurred.
Members of NAMIDEF pay a monthly fee of currently 60 Namibian Dollars per month, which is approximately 14 US Dollars. There is no volume limit but members are expected to be reasonable in their use. Corporate members negotiate their fees with NAMIDEF's management.

3.2. NamNet

3.2.1. Domain Structure

Subdomains were established with a view to decentralize and simplify the DNS administration.

3.2.2. Network Size

In February 1997 approximately 400 subdomains (or individual accounts) were known to the Foundation, most of them single user systems, but also several large Local Area Networks, some 700 computer systems appear to be connected and about 1000 individuals are estimated to make use of electronic networking through NAMIDEF.

3.2.3. Internet Connectivity

As the membership base had grown in 1995 to the extent that a leased line was financially feasible, a 9600 bps land line was leased from TeleCom Namibia which connects to UNINET-ZA at the Witwatersrand University in Johannesburg. This is realized as a national microwave circuit from the capital Windhoek to the border with South Africa (TeleCom Namibia) and as a national circuit from the border to Johannesburg (TelCom South Africa). These two national circuits are much cheaper than an international circuit of the same speed.
10 telephone lines were rented and installed on a hunting sequence. UNINET subsidized the South Arican part of the leased line to a small extent and provided configuration services to the Foundation's router as part of the agreement which allowed the Foundation's technical staff to learn from UNINET's expertise.
The transition from dialup to TCP/IP went very smoothly and totally transparent to the users. Outgoing mail was first sent via ``UUCP'' but then the mail routing software smail was configured to establish an SMTP connection to the Mail Exchanger in Grahamstown. After that stabilized it was reconfigured to use the DNS system and deliver the messages directly to the recipient system.
On September 13, 1995 the first message over the IP link was sent by the author.
A WWW page was installed as http://www.net.na/namidef/.
Several volunteers with technical background were available on a roster basis to have a look when a system failure became apparent.

3.3. Internet Africa

In November 1996 the Foundation contracted with UUNET Internet Africa Namibia (Pty) Ltd (UAIN), a member of the Internet Africa group which is the largest Internet Service Provider in Africa and is in turn a member of the UUNET group which is one of the largest Internet Service Providers in the world.
This is realized by ethernet access to UIAN's satellite circuit to Cape Town currently at 128 Kilobits per second and about to be upgraded to 256 Kbps.
Dial-in access is realized by advanced Portmaster modem servers operated by UIAN staff.

4. Experiences

4.1. Equipment

A Chase IOLAN server was used initially for dialup but did not seem stable enough to our staff at the time so it was abandoned for a multi port serial card solution installed in a linux system, working very well.
The CISCO 2501 router operated well. UNINET configured the it to their standard profiles which uses a routing discovery protocol (RIP). This is a very elegant solution but this dynamic protocol is prone to a phenomenon called flapping which basically means due to short interruptions of the routing protocol the route to the Internet is lost causing a ripple effect where all connected computers and routers loose their routes as well.
Unfortunately UNINET continuously experienced difficulties with their routes. The fact that they have only one technician for the whole South African academic network didn't help either. This had a serious impact on the performance in particular when coupled with the phenomenon that the routers would seem to be operating normally but almost no productive data could be sent over the link.
In the end it turned out to be the compression used between the routers causing the two routers' timers to go out of sync. The fix worked until the line was regularly loaded to capacity. The CISCO documentation advises not to use compression if the line is being used to capacity, a fact we only found out very late.
We were advised by all consultants we discussed these routing issues with to use static routing but we were unable to convince UNINET to do this. UIAN has a slightly different approach allowing our small networks to do static routing wheres they themselves do route dynamically. Since the change this was no problem any more.
A CISCO 4000 router donated by ``RINAF'' was configured by the vendor's local office and a local consultancy resulting in a serious misconfiguration.
On rare occasions the link between the router in Windhoek and Johannesburg failed. The ``UUCP'' store and forward system was fortunately kept in place. It connected over the IP link hourly to Grahamstown and if it failed repeatedly it would fall back four times a day into the telephone dialup mode as redundant backup. Both South African and Namibian TeleCom have been quite fast in clearing these faults. A 24 hour fault line is in operation where the fault on the circuit is articleed and a reference number is obtained. Escalation procedures are initated if a fault is not cleared within a predetermined timespan.

4.2. Software

4.2.1. Domain Name Service

The (hierarchical) concept of the DNS was not understood by the

individual doing the initial setup, resulting in one huge file having all information in it. The master file was initially kept on a server in Germany, then that server loaded from our server (hidden primary), and because of a flaw in the commercial version of the software used in Germany, errors occured that could not be fixed. The poor design of the system contributed to this problem causing what is called DNS storms. This is unproductive traffic generated by unresolvable DNS queries which caused overload of the leased line. After we moved away from the server in Germany (which in the meantime has upgraded its software) and we rewrote the DNS system from top to bottom in a hierarchical, organized way, the throughput improved significantly.
In early October 1995 the system in Grahamstown traditionally having been the MX mail exchange host stopped providing this service. It should not have affected operations but the DNS system all but failed. Rectifying that problem made us aware of the fact that the routing software smail did not make use of the DNS service. Fortunately it is available as source code so the relevant library was linked into the binaries. When the new binaries were installed, a permission requirement was overlooked and two days worth of messages were lost until the problem was noticed and rectified.
Three of the four commercial ISPs operating in Namibia register their customers' networks within the COM.NA and ORG.NA domains administered by the author. The service is currently free of charge and NAMIDEF networks will always be registered free of charge. However one Namibian and one South African Internet Service Provider seem to be unable to submit correct data making the registration unnecessarily work intensive. As the DNS service is the backbone of the Internet and any fault can have serious repercussions strict adherence to the Internet regulations is being enforced.
The University of Namibia (connecting through IWWN) misconfigured their DNS so severely that it had adverse impact on our DNS server and the leased line. Encouragements to sort this out failed until we offered to remove their entries from the DNS data base. Their supplier flew in a system analyst from South Africa who sorted this out within a short time.

4.3. Mail Services

In January 1992 five messages traversed the mail server per week, in August 1996 approximately 45 MB per month and in January 1997 the volume had risen to 20 MB per day.
We follow up each and every of the few complaints about mail messages allegedly having gotten lost. Misaddressing on the part of the sender was the predominant cause of the messages not reaching their recipients. Some messages were in fact delivered to the recipients' mail servers overseas but were lost there.
During the change from UNINET to UIAN the backup Mail gateway (MX) was kept by mistake but first contacted via telephonic UUCP dialup (through a misconfiguration) and then it was not noticed that the UUCP connection attempts via TCP/IP were failing. This resulted in several hundred messages waiting in a wrong queue. When this was noticed the messages were requeued, the senders and recipients informed of the problem and steps were taken to avoid this from happening again.

4.3.1. Routing

We encountered several routing problems.
Our computers would periodically loose the routing information, on how to find the path to the Internet. When we started using dynamic routing the program concerned (routed) was misconfigured causing each of our three computers and the router to advertise a connection to the Internet, which only the router had. It was then left to chance whether the router's information was seen by one of the computers on the ethernet.
NAMIDEF's networks needed to be entered into routing arbiter databases, a fact we became aware off when we found that we could not reach certain networks in the US.

4.3.2. Miscellaneous

Linux and the software running thereon (PPP, UUCP and the Apache HTTPD) have proven themselves. With the exception of some minor problems, which however caused significant headaches at the time, they only required little attendance, such as trimming of log files which has now been automated and the occasional upgrading of the operating system itself (kernel).

4.3.3. Some Subscribers

A standard survey performed by the Internet Society last year showed approximately 80 IP addresses in use, currently I would estimate more then 300 IP addresses being used within Namibia, some within NAMIDEF's name space some outside.
Some local consultancies, the Parliament, the Ministry of Basic Education and Culture, the National Museum, and the UN offices in Windhoek all operate their own networks connecting via leased lines to NAMIDEF.
The UN Field Offices in Windhoek with UNICEF and UNDP being the lead agencies, operate a linux gateway to its DOS based LAN. This has since been upgraded to a full Internet connection with thelinux server continuing as gateway to the LAN but also serving web pages for the participating field offices. UNICEF standarises on a commercial package for LAN mail so gateway software was obtained which works very well with the linux gateway.
The Media Institute of Southern Africa deals with human rights violations, press freedom related issues and maintains a database of stories and pictures from its member publications. It operates a Macintosh based ethernet with a linux gateway. The Namibian, a Windhoek daily newspaper and several newspapers within the region contribute to the database by electronic mail.
The author receives medical abstracts by electronic mail, participates in several bulletin boards (newsgroups or lists) dealing with essential drugs, Maternal & Fetal Medicine, development, communicates with medical practitioners all over the world.

5. The Future

The Ministry of Education is planning its own data circuit to the northern rural area and is likeley to share the circuit with the Foundation which has this particular area as a main target for school related activities. Attempts to support local schools have been not very successful to date, mainly because of lack of interest on the part of the principals.
NAMIDEF is in contact with the Namibian NGO Forum in order to support the NGOs' activities.
On the coast the Sea Fisheries Institute is in the process of setting up a Fisheries' Information and Management System and is about to connect it through a firewall into NAMIDEF's network.
Two local Internet Service Providers have emerged, the largest South African ISP concentrates on corporate clients and a small South African ISP has a Point of Presence in Windhoek. Efforts are underway of establishing peering between all ISPs and NAMIDEF in order to lower costs for common services (such as Usenet News) which now have to be duplicated. Competition in the commercial ISP sector is fierce, though.
The author currently works on connecting the four district hospitals in the (coastal) Erongo Region (province) to the Regional Health Office through PPP dialup to a linux server, which in turn will support demand dialling to the local Point of Presence to give the medical and nursing staff Internet access but also to share the server disk space (using SAMBA for administrative purposes, for example having the budget spreadsheets on the central server will allow the Region to update their budget spreadsheet on an ongoing basis.

6. Conclusions

It was a difficult startup phase, but an enormous amount of experience was acquired. The network grew, services were expanded but we still managed to maintain a high reliability in particular for the mail system. Much of the line speed problems were beyond our control, it is important to note that majority of local faults were caused by a ``single individual'' and that all faults were discovered by local staff which helped with building confidence towards the system and in their abilities. Without having to rely on expensive outside consultants flying into Namibia to troubleshoot and leave again we managed to keep the network operational. The most common advice we received via email were pointers towards documentation. The author has been invited via electronic mail to teach Network Administrators' courses on ``UUCP'' & TCP/IP in Senegal, December 1995, Swaziland, December 1996 and Kenya, February 1997 as part of the ``RINAF'' (Regional Informatics Network for Africa) project sponsored by UNESCO and the Italian government.
The careful step wise approach (incremental growth) so far has been very well received. NAMIDEF is so confident that it has written Sustainable Development into its constitution and has declined loans. NAMIDEF is of course looking at grants from donors but only for infrastructure, training and expansion.
Keeping up with the demand has proven to be the biggest challenge. Inspite of not having posted a single advertisement NAMIDEF has 400 members requiring the use of a high speed digital link by way of outsourcing to UIAN.
In 1995 only 9 countries in Africa had IP connections with traceable routes, namely, Tunesia, Algeria, Zimbabwe, Zambia, Mozambique, Egypt, South Africa, Namibia, and Kenya. In 1997 approximately 30 countries in Africa have some form of Internet connectivity.
The Namibian Internet Development Foundation however has brought up the NamNet without outside resources. No other developing country (in Africa) has achieved this to the author's knowledge.

7. Lessons Learned

Not surprisingly many lessons learned in Namibia have been learned ``elsewhere'':
o ``Email is going to change your life''
o If Fax works, E-Mail works! (``Lawrie's Law'')
o ``Use appropriate technology''
o ``Don't buy software''. Get free packages off the net, they are usually better and invariably better supported
o A neutral body can resolve conflicts
o ``Seed money is a great help''
o ``Don't charge by volume, divide the running costs''
o Spend what you have raised, don't try and raise what you want to spend.
o Satisfied subscribers spread the word.
o Only connect sites that are willing to put in some work
o Plan ahead. Plan far ahead! Plan ``very far'' ahead!
o Use Internet ``protocols'' and ``standards''
o If it works on the first attempt, something is ``seriously wrong''
o You don't need outside experts, a ``friend'' on the other end of the link is enough
o RTFM!

8. Bibliography

1. L. Abba, A. Gebrehiwot, A. Lazzaroni, and S. Trumpy Status and Objectives of the RINAF project. FID News Bulletin, 45(7/8):238--241, July/August 1995.

2. David H. Crocker Standard for the Format of Arpa Internet Text Messages. Technical article, Dept. of Electrical Engineering, University of Delaware, University of Delaware, Newark, DE 19711, 1982.

3. David A. Korver Email in Lusophone Countries in Africa Masters Thesis, Catholic University Nijmegen, 1997

4. M. Lawrie UNINET-ZA Networking Experience from the Beginning until December 1993. In Electronic Networking for West African Universities, AAAS, Washington, DC, December 1993 Workshop Article

5. T. O'Reilly and G. Todino Managing UUCP and Usenet, O' Reilly & Associates, Sebastopol, 1992