Abstract
In this paper, we define a framework that can be used to evaluate the quality and completeness of the monitoring and management of a web site. This approach is scalable and extensible as it can be used for a single web site, a collection of sites, or an entire hosting center. This framework can also address the support of the launch of a new site, an upgrade to an existing site, or a one-time quality and completeness assessment. The approach is supported by a methodology that is based on a series of "perspectives" which incorporate a comprehensive view of tools, processes, organizational structure, and staff skills. The perspectives discussed include system, support, and end-user. The system perspective has as its focus the monitoring of essential infrastructure, application, and business-system components. The support perspective focuses on team processes and related tools. Key processes include change, problem, performance, and security. The end-user perspective is cente! red on measurement and improvement of the end user experience. The framework, and its supporting perspectives, provides the opportunity to take a comprehensive view of the management of a site. In support of the framework, we have developed a systematic methodology that uses a series of data tables to drive and support the analysis. These tables are used to clearly identify and document the monitoring and management components, processes, and tools that are the focus of the activity.

3.1 Creating the infrastructure table 6
3.2 Creating the application table 8
3.3 Creating the business table 8

5.1 Change management 13
5.2 Problem management 17
5.3 Performance management 22
5.4 Security management 26

1 Introduction
In many enterprises, web sites have become a mission critical component of the organization as more and more businesses have come to rely on the Web for commerce as well as internal and external communication. In 1997, the global impact of the Internet trade in services accounted for well over $40 billion of U.S. exports (A Framework, 1997). By 2003, the U.S. Commerce Department estimates that business to consumer e-commerce will likely be in the range of $75 to $144 billion. Business to business e-commerce could reach between $634 billion and $3.9 trillion. (Leadership for the new millennium, 2001). Because of this, many IT organizations are putting new focus on establishing and maintaining 7x24 site availability along with high quality performance and response time. The consequences of Web site failures can be costly. E*TRADE has felt the pain of costly failures. From February 3, 1999 through March 3, 1999 E*TRADE experienced four outages of at least five hours. The dir! ect cost of these failures is not known but the company's stock price declined twenty two percent on February 5 -- just two days after the initial failure (Frick, 2000). Web management tools are plentiful but alone they are not enough to do the job.

Many organizations use multiple, ad hoc methods, tools, and services in the quest for five 9s reliability and sub 3 second response time. This collection of resources makes determining the effectiveness of Web site management tools is a challenge for many IT organizations. This is true whether you use a framework-based management solution or a collection of point products. Many organizations spend a great deal of time and money evaluating the capabilities of a given tool but do very little to understand how the full range of tools will perform in the context of the organization's staff, skills, and processes. The main challenge of web site monitoring and management is to be able to detect (and correct) a variety of problems quickly. Often these problems go undetected because the monitoring and management implementation is ineffective and does not fully leverage the capabilities of the toolset Many tools are powerful and specific to the problems that occur with the! se sites yet fail to perform at the expected level. Tivoli has dozens of products that plug into its framework that can be used to manage sites and their application components (Tivoli product index, 2000). BMC Patrol has a wide variety of Knowledge Modules (Patrol, 2000) and detailed white papers to guide the technologist. Event management is a good example of this phenomenon (Event management and notification, 2000). Considering the complexity of this implementation challenge, what analysis and planning is required to implement effective Web site-specific monitoring and management? How do you go about anticipating problems before they happen, and when they happen, how do you correct them in a timely manner? This paper provides a framework for the analysis and planning that is necessary to identify and evaluate the important monitoring and system management components for a Web site. This approach can be used by the enterprise to plan monitoring and management if it is be! ing hosted in-house or to evaluate the quality of monitoring and management if it is being outsourced to a web hosting organization.

Our approach is based on perspectives that incorporate tools, processes, organizational structure, and staff skills. The notion of perspectives is used by Strum & Bumpus (1999) to give a simple name to what others called disciplines, domains, functions, processes, and services. In our use, a perspective is a point-of-view or focus. Our perspectives include the system, support, and end user. The system perspective has a focus on hardware and software grouped by infrastructure, application, and business components. The support perspective is centered on four processes -- change, problem, performance, and security with a concentration on team, process and tools. The end user perspective is focused on measurement and improvement of availability, performance, consistency, and reliability. The approach is implemented using a three-part method.

2 Method
We have developed a systematic methodology supported by a series of data tables. This approach is used by IT consultants delivering services to customers like those supporting Information Technology Infrastructure Library (ITIL) Services (Introduction to ITIL, 2001) and Information Systems Management Architecture processes (Harikian, Blust, Campbell, Cooke, Foley, Gulla, Gayo, Howlette, Mosher, & O'Mara, 1996). These tables clearly identify and document the monitoring and management components, processes, and tools that are the focus of the framework. The method consists of three steps. The steps are preplanning, analysis, and review. The focus of the preplanning step is to gather materials and to perform early planning activities. The materials collected include web site design diagrams and configurations. Planning activities include listing site components and examining items used in production like restart scripts, monitors, and other tools. Preliminary system, suppo! rt, and end-user perspective tables are built during this step.

An important activity of the analysis step is the completion of the system, support, and end-user perspective tables. These completed tables are used as the basis for a presentation or report. With partially completed tables in hand, a meeting should be held to review what has been found with the owners/developers of the web site. This meeting will offer the opportunity to validate the requirements that have been discovered, as well as gather any additional information that will have an impact on the assessment. During the review step, a presentation or report that contains finding and recommendations is presented. Final versions of the system, support, and end-user perspective tables are also provided and explained. The report is a major tool that should be used to drive the activities to put the systems management solution in place to support the production web site. An appendix to the presentation or report contains a high-level implementation plan, whic! h should be used as a guide to implement the action items from the study. The system, support, and end user perspectives are now discussed in detail.

3 System perspective
The system perspective has three areas of focus -- infrastructure, applications, and business functions. Each area is different and taken as a whole, they cover the system aspects of the Web site. The infrastructure focus concentrates on the operating system, server and network hardware, and other devices such as firewalls. The practical aspects of infrastructure support have been discussed in detail elsewhere. A good example is Welter's work (1999). The application focus places specific attention on the database, middleware, and the application itself. Business functions focus on the comprehensive management of a collection of applications. Comprehensive means business views, monitors, and command and control.

3.1 Creating the infrastructure table
For the site's infrastructure focus, create a table with as many specific components as you determine to be key to the health of the infrastructure. A good starting point should include the operating system, server hardware, network hardware, and other devices like firewalls and load-balancing servers. Derive the list of infrastructure components from the documentation for the web site. For each specific component, identify a set of detailed components. For the operating system, this should include detailed components like CPU utilization, file systems, paging space, memory utilization, etc. These detailed components will become the focus of the monitors that will be used for ensure the availability of the infrastructure. Table 1 contains examples of infrastructure specific components and component details.¹

3.2 Creating the application table
For the Web site's application focus, the concentration is on the database, middleware, and the application itself. Table 2 contains examples of application specific components and component details.

3.3 Creating the business table
For the business aspect of the system perspective, the focus is on relating the applications as business systems. We first observed this idea from Tivoli's Global Enterprise Manager product (Tivoli global enterprise manager instrumentation guide, 1998). However, our approach in this paper is product independent. To relate applications as business systems, components are grouped and taken as a whole. Views are used to visually manage the business systems. Business system monitoring is more inclusive than regular monitors are. For example, a business system monitor could have a monitor called all business system interfaces. This monitor could check end-points of all application interfaces, check availability of queues between applications, and run a test transaction --all part of the same monitor -- all related to one business system. Business system command and control has function that is more inclusive than a regular command. For example, the command could have a f! unction to shutdown and restart selected or all daemons (services) of an application for all servers in the business system. Table 3 contains examples of business specific components and component details.

After the infrastructure, application, and business components are identified and documented important data should be transferred to the monitoring evaluation table.

4 Creating the monitoring evaluation table
The monitoring evaluation table serves as a tool to identify monitors to be used (or developed) to address the management needs of the web site. The table also contains information about scripts, commands, and views that are needed for the management of the Web site. Table 4 is an example of a completed monitoring evaluation table.

5 Support perspective
The key components of the support perspective are team, process and tools. These components are explored in the context of four disciplines -- change, problem, performance, and security. Change, problem, performance, and security management are widely practiced disciplines in the industry. They are basic to IBM systems management (Mangold & Brandner, 1993) and many other organizations focused on the management of systems like ISIL (Introduction to ITIL, 2001). The scope of the support perspective is both broad and narrow. The broad scope has to do with the readiness of change, problem, performance, and security teams to handle the Web site's needs. The narrow scope is how monitoring, command, and control interact with the specific functional perspectives. For example, during a change window, how is monitoring handled to avoid a flood of false alerts?

5.1 Change management
Change management is a process whose goal is to provide defect-free implementation of changes to the system environment. This process includes planning and documentation of the change, real time management of the change, verification of completion or, in the case of failure, verification of restoration back to the original state, and follow-up analysis and reporting. Changes, including both installation and modifications, should be accomplished in a logical and orderly fashion, which achieve the expected result without undesired service disruption. From a system-monitoring viewpoint, we see three important issues:

1. How to shut down the monitoring system during a change activity in order to avoid a flood of false positive alerts
   
   
2. How to reactivate the monitoring system follows a change activity to verify that all systems and services are functioning normally.
   
   
3. How to convert problem management activities into change activities

From an overall management viewpoint, the change evaluation table has a series of questions that are designed to discover how prepared the change management team, tools, and processes are to handle the Web site being evaluated. Table 5 is an example of a completed change evaluation table.

5.2 Problem management
Problem management is the successful awareness of and response to all monitoring tool alerts and other manually reported or detected problems and the resolution of any events, conditions, failures, etc indicated by this information. The entire set of activities is focused on ensuring that the site is available and functioning in the manner in which it was designed. The essential issues include:

1. Configuring automated alert tools with the appropriate level of sensitivity - A tool that queries too often can impact the functioning of the site, if it queries too infrequently it will let problems remain undetected for an unacceptable length of time. Tools that are too sensitive can also generate too many alerts or tickets, creating a flood of false positives that obscure the actual functional state of the system. Tools that are too insensitive miss problems and have little value. Getting the balance correct can be a challenge.
   
   
2. The 7x24 nature of e-business web sites and the number of systems used in the typical web site indicate that in order to scale the problem management system and control costs some degree of automation is required. Automation can insure a rapid response to simple problems regardless of when they occur.
   
   
3. How to achieve timely resolution of problems? This is possible but you need the right people, tools, and processes.

From an overall management viewpoint, the problem evaluation table has a series of questions that are designed to discover how prepared the problem management team, tools, and processes are to handle the Web site being evaluated. Table 6 is an example of a completed problem evaluation table.

5.3 Performance management
Performance management is focused on the measurement and reporting of system resources by the application and its users. Performance management can be used to report problems in real time but is generally used to determine performance trends and to plan for necessary resources upgrades or modifications. From an overall management viewpoint, the performance evaluation table has a series of questions that are designed to discover how prepared the performance management team, tools, and processes are to handle the Web site being evaluated. Table 7 is an example of a completed performance evaluation table.

5.4 Security management
Security management has the goal of maintaining the integrity of controls regarding who has access to what areas of the system and what is viewable or changeable. Security management takes a variety of forms including perimeter security (firewalls and site hardening), authentication/authorization (passwords and associated permissions), intrusion detection, and policy oversight. From an overall management viewpoint, the security management evaluation table has a series of questions that are designed to discover how prepared the security management team, tools, and processes are to handle the Web site being evaluated. Table 8 is an example of a completed security evaluation table.

6 End user perspective
The focus of the end user perspective is on measurement, evaluation, and improvement. From an overall management viewpoint, the end user evaluation table has a series of questions that are designed to discover how well availability, performance, consistency, and reliability are being addressed. This perspective is focused on the measurement and improvement in the quality of the end user experience. Table 9 is an example of a completed end user evaluation table.

7 Review report or presentation to complete the review
When the analysis is complete, a report or presentation should be created to share the finding and recommendations. We have found that some teams respond to a detailed written report whereas others require a presentation. The report or presentation should contain the following parts:

1. Executive summary
   
   
2. Detailed discussion and findings including a discussion of exposures and opportunities
   
   
3. Recommendations

The report should also include two appendixes. Appendix A should contain the completed tables. This supporting detail will lend support to the findings and recommendations in the body of the report or presentation. Appendix B should contain a high-level implementation plan. This plan will provide linkage to the next phase of the project.

8 Summary
This paper described a framework and supporting method to evaluate management tools and methods for new or existing web site. Three perspectives are the basis of the framework -- system, support, and end-user. The method is used for proactive planning or because of problem solving activities. The method involves three steps -- preplanning, analysis, and review. A number of tables were used to support the analysis associated with the method. Findings and recommendations were discussed in a report or presentation. Appendix B of the report contains a high-level implementation plan.

About the authors
Joseph Gulla is a Senior Consulting Information Technology Manager for IBM Global Services. Mr. Gulla works with customers of the IBM Web Hosting facility in Research Triangle Park, NC. Mr. Gulla has written a number of IBM internal publications and co-authored the IBM Redbook Distributed systems management design guidelines: The smart way to design (1996). Mr. Gulla has spoken at a number of technology forums including Planet Tivoli, ESM Sharenet, IBM Security Seminars, and the NorthEast Information Systems User Group where he served on the Advisory Board. He has held a number of technical and management positions over the last 22 years.

John Hankins is a Senior Architecture and Delivery Specialist on the IBM Global Services Web Offerings team. He has worked in the field of Internet services development and management for the past 10 years. The past 5 years have focused on the specific area of web hosting services. Mr. Hankins has presented at numerous conferences on Internet services and has served as a co principal investigator on three National Science Foundation grants. He has held management positions in a variety of technology organizations over the past 15 years.

Footnotes
1. About the example tables: These tables do not contain information about a specific IBM web-hosting customer. The tables are also not representative of IBM's change, problem, performance, and security management policies. The data in the tables has been made up simply to illustrate the main points of the paper and the power of the methodology.

2. Copyright information: Tivoli and NetView are trademarks of Tivoli Systems, Inc; ESM is an abbreviation for Enterprise Security Manager which is a trademark of AXENT Technologies, Inc; Trend is a trademark of Desktalk, Inc; Silk is a trademark of Trinagy, Inc; Keynote Perspective is a product of Keynote Systems; All other products and company names are either trademarks or registered trademarks of their respective companies.