Design and Implementation of a Network Performance Evaluation System Through Client Observation

Hiroki SAITO <hiro-s@cs.meiji.ac.jp>
Takeshi CHUSHO <chusho@cs.meiji.ac.jp>
Meiji University
Japan

Abstract

One of the major issues in the Internet is the performance management for network system usability. To improve network system quality of service, system administrators should evaluate how their systems are working, and should operate their systems to perform users' requests and optimize performance. To manage network system performance, it is important for administrators to be aware of system usability factors such as access delay, processing time, data transfer throughput, and so on.

Several ways to measure network performance have been developed so far. However, performance management is still difficult because of the lack of effective tools to evaluate network system usability.

The goal of our work is to develop a new performance evaluation tool for network system usability. In our approach, we measure system usability and performance through mechanisms installed in the clients. The purpose of our system is to make clear the behavior of the client applications, and allows us to measure system performance for client usability.

In this paper, we discuss the fundamental principles for client observation and its performance indices, and the design and implementation of the performance evaluation system through observation. Also, we confirm the effectiveness of our system through experiments.

1. Introduction

Currently, various applications provide useful services in the Internet. As these applications are actively used, the network management usability becomes a major issue in the Internet. To improve network system quality of service, system administrators should evaluate how their systems are working, and should operate their systems to perform users' requests and optimize performance. To manage network system performance, it is important for administrators to be aware of system usability factors such as access delay, processing time, data transfer throughput, and so on.

Several ways to evaluate the performance of network systems have been developed so far.

Statistical analysis is applied to the activity logging by the servers. This is a popular way to evaluate server performance. However, this analysis cannot provide any hints on network links and the clients.

The benchmark is another performance measurement method. It can provide various indices of server performance. However, the benchmark requires a special environment, and the results are valid only for that environment.

Network monitoring allows us to evaluate network usage at the datalink level. However, performance indices in the datalink layer are not always related the application performance. This is because the application level performance includes not only the characteristics of the datalink, but also many other performance factors.

The goal of our work is to develop a new usability performance evaluation tool. In our approach, the performance through measurement mechanisms installed in the clients is the indices the systems' usability. In our approach, we measure system usability and performance through mechanisms installed in the clients.

2. Performance evaluation for network systems

2.1 Necessary functions for evaluation tools

To evaluate network system performance from the point of view of usability, the system administrators must know how their services are working and must improve them to satisfy user requests. Network system performance with regard to usability is determined by how the client provides performance to the user, that is, the system administrator should be aware of client system performance factors such as:

• How long the client takes to access the server.
• How long the client takes to process the transaction.
• How much data is throughput in the client?

Let's consider a common framework to evaluate the performance of the end-point application. The evaluation tool should have the following functions:

1. The tool is able to measure the throughput and response speed at the end-point applications, which has an impact on the user. The performance evaluation tool aims to improve the performance of the end-point application. The total system performance doesn't always interrelate with the performance of the network path. Therefore, the network system performance should be evaluated in the client applications.
2. The tool should handle various kinds of datalinks. The Internet has been used on various datalinks such as Ethernet, ATM, FDDI, Token Ring, X.25, Integrated Services Digital Network (ISDN), and so on. Transmission Control Protocol/Internet Protocol (TCP/IP) technology is a set of protocols of the upper layer of these datalinks. Therefore, the measurement method should be independent of the datalinks.
3. The measurement tool should be independent of applications. There are various applications and application protocols utilized in the Internet. The performance measurement should be a standard framework, and it should not be dependent on a single application and a single application protocol.
4. The measurement tool should be able to be applied to existing applications without any modification. It is costly to modify application software to operate the measurement tool. Also, there are number of applications that would be difficult to modify for use with the measurement tool.
5. The measurement tool should be able to be applied to running systems. Using computer simulation, it is difficult to calculate all performance factors, and the benchmark is only valid under specified conditions. It is more effective to evaluate running systems by measuring the performance in the actual situation.
6. The measurement tool should be able to be applied not only to the network links, but also to the server and client systems. The total performance of the network systems is affected by the servers, the clients, and the network links.

2.2 Existing performance measurement tools

Several tools have been developed to evaluate network performance.

1. Statistical analysis of servers logs. Statistical analysis from server access logs allows us to determine the operation status of the servers, such as the number of accesses, number of data transfers, and processing time. However, the results of the analysis of server logs determine only the performance of the servers themselves. The performance of the clients and the network links is not included in the results.
2. Measurement of network usage and Round Trip Time (RTT). Simple Network Management Protocol (SNMP) is widely used to measure network usage [1][2]. System administrators can use simple tools such as ping and traceroute to measure system usage. However, TCP performance degrades as network usage increases, and it is also affected by the characteristics of the network links [3][4]. The performance of end-point application is affected by not only the network usage but also by the characteristics, end-to-end throughput, and capacity of the servers and clients.
3. Benchmark. We can find some benchmark tools such as SPECweb [5], WebStone, ttcp, DBS, [6] and so on. These tools provide many indices of exact application performance. However, it is difficult to set up suitable benchmark conditions to reproduce those of servers under actual operating conditions.
4. Packet dumping. The analysis of packet dumping provides many indices on the datalink level. Furthermore, some tools such as RMONv2 and ENMA [7] can calculate the indices in the TCP layer. However, packet dumping can only apply to specific datalinks and the traffic shared network. Even in the Ethernet environment, the packets could not be observed in the switched network.

Accordingly, we need a new performance evaluation tool for network systems that can be applied to actual systems operation under various configurations.

3. Proposal of evaluation tool through client observation

3.1 Fundamental principle of client observation

The goal of our performance evaluation method is to provide a suitable, common framework to measure network system performance usability. We do this by measuring the actual network system performance from the users point of view.

Figure 1: Fundamental concept of performance measurement

The proposed method is to observe the behavior of the client applications, as shown in figure 1. To access the network facility and communicate from a client application to a server, the applications always use the system call in the Operating System (OS). A layer attached to observe the applications, called "Observation layer", reveals all system call-level behavior in the client applications.

Because the observation layer can observe the client applications directly, the observation layer measures the nearest result that affects the users.

Moreover, the observation layer interfaces are the same as the standard Application Programming Interface (API) such as BSD Socket, Transport Layer Interface (TLI), and Winsock. In other words, the applications can access the network via the observation layer through the standard system calls without modification.

3.2 Communication flow

In this section, we describe the target communication model for measurement through the observation layer.

In popular Internet application protocols such as Hypertext Transfer Protocol (HTTP) [8], Simple Mail Transfer Protocol (SMTP) [9], Post Office Protocol 3 (POP3) [10], and so on, the clients act by following procedures that are called the "Request-Response Communication Model."

• Connection establishment - connect() system call
• Sending of request 1 - write() system call
• Receiving of result 1 - read() system call
• Sending of request 2 - write() system call
• Receiving of result 2 - read() system call
• ...
• Sending of request n - write() system call
• Receiving of result n - read() system call
• Disconnect - close() system call

Figure 2: Communication flow in HTTP

For example, figure 2 shows the communication flow diagram in the HTTP. The client sets up the connection, and sends the server the request as GET, PUT and POST, then receives the results that the server processed.

3.3 Performance index

The observation layer can measure various performance indices by observing the system calls. Figure 3 shows the performance indices based on the Request-Response Communication Model. The contents of the measurement are the time period of T1-T7 and the amount of data transferred.

Figure 3: Performance indices

Our system is able to measure following performance indices:

• Transaction time (Tt) (T7-T1). Transaction time (Tt) is the time elapsing for the TCP connection. Accurately, it is defined as the period from the time when the open() system call is invoked (T1 in figure 3) to the time when the close() system call is returned (T7 in figure 3). This value indicates the processing time for a transaction.
• Connection establishment time (Te) (T2-T1). Connection establishment time (Te) is the time elapsed for the connection setup. It is defined as the time from when the connect() system call is invoked (T1 in figure 3) to when it is returned (T2 in figure 3). This value indicates the RTT of the network and TCP/IP performance of the clients and servers.
• Request data transfer time (Ts) (T4-T3). Request data transfer time (Ts) is the time elapsed for the data transfer to send the servers to the request data in the Request-Response Communication Model. It is defined as the time from when the write() system call is invoked (T3 in figure 3) to when it is returned (T4 in figure 3).
• Response data transfer time (Tr) (T6-T5). Response data transfer time (Tr) is the time elapsed for the data transfer to receive the processed data in the servers, defined as the time from when the read() system call is invoked (T5 in figure 3) to when it is returned (T6 in figure 3). It includes not only the time elapsed for the data transfer, but also the time to process the request in the servers. Therefore, this value indicates the performance of the server systems and the network.
• Request data size (Ds). Request data size (Ds) is the amount of data transferred to the servers as the request. It is defined as transmitted bytes by write() system call.
• Response data size (Dr). Response data size (Dr) is the amount of data transferred from the servers as the response. It is defined as received bytes by read() system call.
• Request data transfer rate (Rs). Request data transfer rate (Rs) is the transfer rate of the request data to the servers (KB/sec). It is defined as Ds (request data size) / Ts (request data transfer time).
• Response data transfer rate (Rr). Response data transfer rate (Rr) is the transfer rate of the response data from the servers (KB/sec). It is defined as Dr (response data size) / Tr (response data transfer time).
• Transaction transfer rate (Ra). Transaction transfer rate (Ra) is the rate of all data transferred during the time elapsed for all process, that is (Ds + Dr) / Tt.
  It is the performance index according to the data size of the transaction.

4. Design and implementation of the performance evaluation system

4.1 System model

Figure 4: System model

Figure 4 shows the system model of the performance evaluation system through client observation. The system consists of four components: an observation layer, a data slicer, a database, and a manager.

Observation layer

The observation layer monitors the system calls between the applications and the OS.

The observation layer has proxy system call functions. The proxy system call functions stands for the application instead of the actual system call functions in the OS. When the application calls the system calls, the proxy system call procedures in the observation layer are invoked. Then the observation layer records time of each state (T1-T7), counts the amount of data transferred in each TCP connection, and simultaneously, invokes the actual system call in the OS.

Data slicer

The data slicer processes the measured raw data in the observation layer with the conditions such as IP address and the port number, and performs the basic analysis.

Database

The data slicer writes the processed data to the database. This database is used for further analysis by the manager.

The database consists of a matrix into which are put the server address, port number, time elapsing in each state, data size, and transfer rate in order time stamp. The structure of the database is quite simple because a data entry for each TCP connection forms a single row of data. Therefore, it reduces the amount of overhead for measurement in the clients.

Furthermore, the database contains an improved multidimensional matrix to comply with future analyses.

Manager

The manager collects the measurement data from the target of the remote clients. It analyzes the data and provides a graphical intrepretation of the results. Currently, the manager is implemented as the interface to GNU Plot to view the analysis results.

4.2 Implementation

We are implementing the performance evaluation system for Unix platforms. The observation layer, data slicer, and database are implemented in ANSI C on Sun Solaris 2.6. The manager is implemented in shell programs and C programs.

The observation layer has the proxy BSD Socket interface for invoking from the target application.

The proxy interface was implemented as a shared library that uses the dynamic linking. It enables the replacement of the actual Socket interface in the OS with the proxy Socket interface in the observation layer when the application is executed dynamically. Therefore, no modification of the application programs is needed for the measurement. Even if the target system does not have the dynamic linking facilities, we do not have to modify the applications; we recompile/link to the observation layer module as a static library.

The proxy Socket interface is invoked when the application calls the Socket system call to access the network. The observation layer monitors the connection open, close, write, and read, records the time elapsed, and counts the data transfers. Continuously, it calls the real Socket system call in the OS.

In our current implementation, our own unique protocol collects measurement data between the manager and the target of the remote clients. In future work, we plan develop a protocol that corresponds to standard protocols such as SNMP.

5. Evaluation

5.1 Overhead for the measurement

Our system records the time and counts the data when the application transfers data to/from the network. This procedure affects the performance of the system itself. To evaluate the capability of our system, we compared the throughput with the observation layer and one without it.

Figure 5: Configuration of experiment

The target system is configured on an Apache HTTP server [11], a Sun Ultra 60 workstation and an ApacheBench client program on a Sun Ultra 10 workstation. These devices are connected on 100Base-T Ethernet as shown in table.1 and figure 5.

Figure 6: Throughput with/without observation layer

Figure 6 shows the average throughput (KB/sec) per connection when the data were transferred concurrently. The result shows that the overhead gets to 5 percent or less on average. The overhead could be consider to ignore in the 100Mbps network.

Almost all of the workgroup-scale networks have been configured to the 100Base-T or 10Base-T network. Therefore, our system can apply to workgroup environments.

5.2 Server access log versus our system

The statistical analysis of the server access log was used to evaluate performance thus far. We compared the differences between the analysis results of the server access log and the measurement of our system.

Figure 7: Transaction time by observation layer and server log

The configurations of the target system are also shown in Section 5.1. Figure 7 shows the process time elapsed (milliseconds) from the server log and the Transaction Time (Tt) (ms) in our system.

The result of the server log is less than one of our system, and the maximum difference reaches 40 percent.

The Tt in our system is the entire time elapsed in the transaction, including the time for the connection setup, data transfer, and connection close procedures in the Socket layer. Comparatively, the server log is recorded only the server side application activity. Therefore, the server log cannot measure the performance regarding connection setup/closing. Moreover, the server log is the performance record in the application level. Once the data are put into the buffer in the OS, before the actual transmission, it appears that the data transfers are done early in the server programs.

6. Summary

In this paper, we described why a new method for network system performance evaluation is needed. Our proposal is to observe the behavior of the client application and exactly measure its performance. Our system can effectively evaluate the network system performance because the client application stands on the end-user side directly.

Also, we applied our system to the actual WWW system and confirmed its effectiveness through the experiments.

7. References

• [1] Leinwand, A.: Accomplishing Performance Management with SNMP, Proc. of INET'93 (1993).
• [2] D. Harrington, R. Presuhn and B. Wijnen: An Architecture for Describing SNMP Management Frameworks, RFC 2271 (1998).
• [3] Mark Allman, Chris Hayes, Hans Kruse and Shaen Osterman: TCP Performance over Satellite Links, Proc. of the 5th International Conference on Telecommunications Systems (1997).
• [4] V. Jacobson, R. Braden and D. Borman: TCP Extentions for High Performance, RFC 1323 (1992).
• [5] SPECweb99 Benchmark, http://www.spec.org/osg/web99/ (1999).
• [6] Yukio Murayama and Suguru Yamaguchi: DBS: Powerful Tool for TCP Performance Evaluation, Proc. of SPIE, Vol.3231 (1997).
• [7] Yutaka Nakamura, Ken-ichi Chinen, Hideki Sunahara, Suguru Yamaguchi and Yuji Oie: ENMA: The WWW Server Performance Measurement System via Packet Monitoring, Proc. of INET'99 (1999).
• [8] R. Fielding, J. Getty, J.Mogul, H. Frystyk and T. Berners-Lee: Hyper Text Transfer Protocol - HTTP/1.1, RFC 2068 (1997).
• [9] J. Postel: Simple Mail Transfer Protocol, RFC 821 (1982).
• [10] J. Myers and M. Rose: Post Office Protocol - Version 3, RFC 1939 (1996).
• [11] Apache HTTP Server Project, http://www.apache.org/ (2000).