INET Conferences

Proceedings

Can Significant Improvements in Individuals' WWW Search Skills Cause Fundamental Changes in Research Practices? A Developing Nations Perspective

Stephen RUTH <ruth@gmu.edu>
Bryan HARTER <bharter1@osf1.gmu.edu>
George Mason University
USA

Dan TUFIS <tufis@valhalla.racai.ro>
Romanian Academy of Science
Romania

Abstract

The diffusion of information technology depends on good training, but the results of training are difficult to measure. Whether the focus is the student, the businessperson, mechanic, manager, or researcher, the simple act of delivering training in a specific technology has little effect; it must be leveraged through careful practice and repeated use. Most of the research on Internet training indicates that success measurements for such courses are primarily based on measures of like or dislike of trainers, facilities, class notes, bandwidth, etc. -- subjective satisfaction. Subjective satisfaction measurements are poor predictors of achievement. They usually show that persons like to learn how to use the Internet but disclose nothing about the students' use of the learned skills. Being satisfied with an experience does not predict that the results will be beneficial in long-term use. IBM, the World Bank, and many other organizations have concluded that it is far more important to evaluate what really happened after training (outcomes) rather than simply measure the immediate perceptions of course participants (outputs). This study, a sample of a larger work to be published in late 1998, attempts to integrate the measurement of results (outcomes) into the problem of assessing Internet training for professors in the Romanian Academy of Sciences. What sets it apart is the concentration on actual results in terms of careers, confidence, and earning power, instead of the more simplistic measures normally employed.

Contents

• Overview
• Romanian academy: Increasing its I.T. capacity
• The Internet laboratory
• Plan for the internet academy
• Internet school curriculum
• Results: Emphasis on long-term value of the course
• Conclusions
• References
• Acknowledgments

Overview

The setting for this study is the Romanian Academy of Sciences, Bucharest, Romania. We feel that the course we describe is one of the more rigorous of its type in the region, but it is certainly not the only one. For instance, the Kiev Regional Office of the Eurasia Foundation in 1995 offered grant opportunities for facilitating Internet Communications in Ukraine, Belorussia, and Moldova. The University of Moldova proposed the development of an Internet site to train users in navigation on the Internet. The successful 1996 program trained over 150 lecturers, scientists, postgraduates, and students over a 3-month period in electronic mail, tools for remote access and searching, and general Internet resources. (R. Dumbraveanu, 1997). A study of researchers in Chile (Ruth and Goet, 1993) also found good training to be essential to Internet technology diffusion, but the focus was countrywide, not specific to a specific location.

The Romanian Academy of Sciences is a large, multifaceted alliance of academics representing all fields of study, from drama to engineering. It was founded in 1866 in Bucharest and included members from all the Romanian provinces: Muntenia, Moldavia, Transylvania, Bassarabia, Banat, Maramures, Bukovina and Macedonia. It functioned much like other European science academies until the Communist takeover. In 1948, the traditional Academy was dismantled, its constitution and bylaws were modified, and a state Academy was installed by directly nominating a number of full and corresponding members. It functioned successively under the name of the Academy of the Romanian People's Republic and the Academy of the Socialist Republic of Romania. Over 100 former members were not included into this new institution, especially those in the fields of philosophy and history. A number of leading communist names were included, many being individuals with no academic credentials. The Academy was transformed into an ideologically dominated institution .

December 1989 has marked the rebirth of the Romanian Academy. In addition to welcoming back many members of the Academy expelled in 1948, the Academy elected some 130 new corresponding and full members as well as 31 honorary Romanian members. Since its foundation in 1866 until now, the Romanian Academy has elected over 1,000 members from representatives of the scientific, cultural, artistic, ecclesiastical, political and military life of the country. Outstanding personalities of science and culture from different countries of Europe, Asia and America have been included among its members. Since 1869, the year of the first elections of foreign members, 434 persons from 33 countries have been elected by the highest forum in Romania, 21 of them being Nobel Prize winners

Romanian academy: Increasing its I.T. capacity

The Romanian Academy has three regional branches (Cluj, Timisoara and Iasi). It is organized into 14 sections: Philology and Literature, Historical and Archaeological Sciences, Mathematical Sciences, Physical Sciences, Chemical Sciences, Biological Sciences, Economical Sciences, Engineering Sciences, Agricultural and Forestry Sciences, Medical Sciences, Economic and Law and Sociological Sciences, Philosophical and Psychological and Pedagogical Sciences, Arts and Architecture and Audio-Visual and Information Science and Technology (the newest section) and has 66 institutes and centers, with more than 2,600 researchers.

The Internet laboratory

Being mostly oriented towards the humanities and theoretical sciences, the Romanian Academy had little contact with Internet technology until recently. In August 1994, after three years of discussion, the Section for Information Science and Technology, with the support of Academician Mihai Draganescu -at that time the President of the Romanian Academy- managed to establish a small research center focused on basic research in Artificial Intelligence: Romanian Academy Center for Advanced Research in Machine Learning, Natural Language Processing and Conceptual Modeling -- RACAI for short. With a modest endowment, consisting of three SUN3/50 and a MAC II donated by the IDSIA-Lugano, plus a PC486, the new Center established an Internet node in the "Casa Academiei," a site that hosts 22 research institutes and centers. Although the initial speed was slow (14.4 kbs on a half duplex leased line), it was a beginning. With funds from several European projects, the center partnered with Ilp-Net, Multext-East, Elsnet-Goes-East; the infrastructure gradually improved. By early 1995 a powerfully configured SUN4/85, a Pentium, a Cisco router, and 2 fast modems were added.

Plan for the internet academy

Based on discussions with George Mason University and the Andrew W. Mellon Foundation New York, RACAI established the Open Internet Laboratory based on its own network. The Mellon grant added five new SunSparc stations (1 Sun Sparc-20, 4 Sun Sparc-4/85) and a Macintosh. Sun Microsystems provided additional support (in the form of 5 Sun Sparc Classic). The Romanian Academy provided the lab space, bought the furniture, and hired a lab and network administrator. As noted before, improving the hardware infrastructure was necessary in order to give Internet access to more scientists, but a prerequisite was to build up a continuous education program on Internet technology, specifically designed for non-computational academic community.

In December 1995, RACAI started a series of online free introductory courses on network communication aimed at a 200-person segment of researchers from the institutes of the Romanian Academy. Special emphasis was placed on researchers from the humanities. This experiment, known as the RACAI Internet School (Tufis, 1997), received an overwhelmingly positive response, and was extended for another two years (1997-1998) with a horizon of 400 more scientists. Recently the Soros Foundation also added funds to this effort.

At present, the Open Internet Laboratory, which serves as a classroom, a research laboratory and a communication node for Romanian Academy (to which several new local networks have been or are in process of being connected), is equipped with 3 Cisco routers, 10 modems for dial-up services, 11 Sun Sparc stations (plus 3 rarely used Sun 3 machines), 1 PowerDuo Macintosh, 3 Pentiums, 2 network laser printers and 2 back-up units. The Internet access is ensured by RACAI through two full duplex leased-lines connected to the national academic Internet service provider, being integrated in the National Computer Network (RNC): a 128 kbps connection to CNSCC, and a 33 kbps connection to ICI (RNC operator). The latter connection will be soon replaced by a 512 kbps line. Due to the gateway connections between RNC and all other Internet service providers, any user connected to a national or regional Internet node (regardless of the service provider's identity) will reach RACAI server(s) through local lines only. The RACAI node is directly connected to the main node of RNC, which in turn has direct connections to the main nodes of the high-schools network (Soros Foundation and Mellon-funded), and of KAPPA (network service provider for the Romanian government and for many domestic users), which is widely accessible for certain categories of Internet users. The RACAI server may also be accessed by switched line, and up to 10 users can have simultaneous accesses (hunting system).

The lab, maintained and supported by RACAI, functions as a shared resource to which the researchers from the Romanian Academy have free access. A significant number of those taking the Internet courses become regular users of the Internet lab.

Internet school curriculum

The Center organizes courses every two weeks in the Internet Lab to introduce the main network communication tools to the researchers from the Romanian Academy, many of which are from humanistic domains and have not used computers before.

Each course lasts 3 days (8 hours/day) and is practice-oriented, the students learning to effectively use the Internet tools. The first half of each day is an interactive presentation of the day's topic by a lecturer assisted by a supervisor. The students are guided through several lessons by their assigned computer. The second part of each day is for individual exercises, with the students being assisted by the lecturer and the supervisor.

The instructors (lecturers and supervisors) are researchers from the Center, most of them holding a joint appointment with the Polytechnic University of Bucharest and having extensive teaching experience. Comprehensive lecture notes (of about 180 pages) have been prepared, specially targeted to first-time users of computers and Internet. The contents of the lecture notes are:

1. Introduction
2. Basic notions for understanding and use of computer networks
3. Electronic mail
4. FTP - electronic transfer of documents
5. Searching for information in the Internet
6. Netscape
7. Harvest
8. Hot Java
9. Glossary
10. Bibliography

Given the specific profile of the students, who had had no (3%) or very little (85%) previous experience in using a computer but had very specific interests in their own domains of activity, the courses were dynamically adapted, so as to diminish the risk of frustration and to stimulate scientific curiosity. There were no enrollment restrictions. The course, all taught by RACAI staff, had a public announcement from the management of the institutes and has been advertised in the mass media (radio, press: the "Academica," a monthly review). The courses are free and take place in the Romanian Academy House, where the majority of the institutes in Bucharest have their sites; consequently, for the researchers living in Bucharest there were no impediments to participation. The classroom, equipped with 10 Sun workstations, accommodates 20 students at a time. A WWW site (http://www.icasit.org/romania/index.html) has been established to show the center and the labs. (Harter, 1997) Besides the lecturer, a supervisor offers technical assistance and additional explanations on practical assignments. The lecture notes are distributed as an auxiliary teaching material. The text is rendered in a reader- friendly form and does not require previous knowledge in the field. More than 400 participants attended the courses organized in 1996 and 1997.At least 200 more students will take the courses in 1998. After graduation all the students are granted unconditional access to the Internet laboratory facilities with follow-up assistance and consequently, most of them became regular users of the Internet laboratory. The majority of the attendees were from the humanistic institutes (Linguistics, Psychology, Philosophy, Art History, Literary History and Theory, Geography, Sociology, Legal Research); the others were from positivist sciences (Physical Chemistry, Applied Mathematics, and Mathematical Statistics)

Table 1. Subjective Satisfaction Scores for Internet Courses Taught at the Romanian Academy, 1995-1997

Marks:	Questions:
1 = unsatisfactory	Q1. Was the instructor well prepared for the course?
3 = adequate	Q2. Was the course material presented in an organized way?
5 = remarkable	Q3. Have you been intellectually inspired during the lesson?

	Fundamental concepts	E-mail	FTP	Xarchie	Netscape	Harvest
Q1	4.58	4.69	4.61	4.67	4.71	4.68
Q2	4.42	4.52	4.26	4.42	4.44	4.40
Q3	4.15	4.17	4.08	4.23	4.20	4.18

Table 2. Age Distribution of Respondents

	Frequency	Percent	Cumulative Percent
20-25	24	18.2	18.2
26-30	27	20.5	38.6
31-40	29	22.0	60.6
41-50	32	24.2	84.8
51-60	16	12.1	97.0
61+	4	3.0	100
Total	132	100

Results: Emphasis on long-term value of the course

Subjective satisfaction scores immediately after course completion were high, as they often are in well-managed training. Table 1 gives a summary of course evaluations across six major course elements. Our study is aimed at results after this training occurs so the remaining tables describe a new set of questions presented to respondents up to two years after they had taken the training. We concentrated exclusively on reported results in the workplace, after the training had time to be used on the job. Since most of the results will be published in a more detailed report later in 1998, we include here some sample outcomes. Table 2 describes the age distribution of the students. Only 15 percent were over 50 years old, about the expected distribution in a community of teachers and researchers. Table 3 shows the distribution of the sample population with respect to gender and research productivity. About two-thirds of the students were women, because one of the basic requirements of the grant that established the training was that women be strongly represented. A proxy for research productivity was established based on the reported output across traditional measurements of academic productivity: journal articles (in and outside Romania) published, grants received, papers presented, and the like. The population is generally quite mature in its reported research profile.

Table 3. Reported Research Productivity of Respondents by Gender and Academic Discipline

	Low	Moderate			High
Male (48)	10	12			26
Female (82)	31	19			32
Engineering/Technical (27)	11	6			10
Mathematics (36)	9	6			21
Arts/Humanities/Law (45)	12		18	28

Tables 4 and 5 give detailed examples of the more comprehensive analysis that can be obtained in concentrating on on-the-job results. It describes the effect of very practical issues like whether Internet training was responsible for changes in individual's approach to his or her job, the ability to earn a living or help students, etc. Two types of insights are possible from the data shown in Tables 4 and 5 -- analysis of mean scores and analysis of differences between group responses (ANOVA). The mean scores convey a clear sense of the respondents' reactions to questions that are based on subsequent use of the technology. The Likert scale employed uses 1 as the lowest score and 6 as the highest. Mean scores of 4 and above can be considered progressively more positive and 4 and below more negative. Many clusters of respondent categories are possible but the simplest and most direct, shown here, groups "hard sciences" (engineering, mathematics, etc.) and "soft sciences" (arts, law, humanities, social sciences) . The larger report will assess the effect of other groupings, e.g. by gender, individual research specialty, research productivity, etc.

Table 4. Sample Results of Respondents Based on Questions Related to Outcomes of Internet Training Course, Grouped by Arts, Humanities, and Law (Soft Sciences) and by Mathematics And Engineering (Hard Sciences)

TOPIC	GROUP	N	MEAN	Std. Dev.	Std. Error	Sig
Changed my approach to work?	arts, humanities, law, social sciences	53	4.49	3.89	0.53	0.598
	hard sciences, mathematics	62	4.21	1.43	0.18
improved comm w/Romanian colleagues?	arts, humanities, law, social sciences	54	3.35	1.80	0.25	0.54
	hard sciences, mathematics	62	3.55	1.64	0.21
	TOTAL	116	3.46	1.71	0.16
improved ability to earn living?	arts, humanities, law, social sciences	52	2.81	1.69	0.23	0.149
	hard sciences, mathematics	60	3.27	1.65	0.21
	TOTAL	112	3.05	1.68	0.16
supervisors/superiors encourage?	arts, humanities, law, social sciences	54	4.02	1.72	0.23	0.007
	hard sciences, mathematics	63	4.78	1.25	0.16
	TOTAL	117	4.43	1.53	0.14
improved opps to stay current?	arts, humanities, law, social sciences	57	4.81	1.33	0.18	0.446
	hard sciences, mathematics	62	4.98	1.19	0.15
	TOTAL	119	4.90	1.26	0.12
helpful for my students?	arts, humanities, law, social sciences	19	5.00	1.41	0.32	0.414
	hard sciences, mathematics	27	4.67	1.30	0.25
	TOTAL	46	4.80	1.34	0.20
do better on my job?	arts, humanities, law, social sciences	55	4.25	1.25	0.17	0.019
	hard sciences, mathematics	61	4.82	1.30	0.17
	TOTAL	116	4.55	1.30	0.12

Table 5. Additional Sample Results of Respondents Based on Questions Related to Outcomes of Internet Training Course, Grouped by Arts, Humanities, and Law (Soft Sciences) and by Mathematics and Engineering (Hard Sciences)

improved ability required research?	arts, humanities, law, social sciences	55	4.36	1.37	0.18	0.684
	hard sciences, mathematics	62	4.47	1.39	0.18
	TOTAL	117	4.42	1.37	0.13
more open to other opinions?	arts, humanities, law, social sciences	56	4.13	1.49	0.20	0.394
	hard sciences, mathematics	62	4.35	1.43	0.18
	TOTAL	118	4.25	1.46	0.13
regularly use when helping students?	arts, humanities, law, social sciences	18	2.39	1.38	0.32	0.011
	hard sciences, mathematics	26	3.65	1.65	0.32
	TOTAL	44	3.14	1.65	0.25
better team projects?	arts, humanities, law, social sciences	51	3.75	1.57	0.22	0
	hard sciences, mathematics	60	4.82	1.27	0.16
	TOTAL	111	4.32	1.51	0.14
manager actively supports?	arts, humanities, law, social sciences	52	3.31	1.73	0.24	0
	hard sciences, mathematics	61	4.59	1.56	0.20
	TOTAL	113	4.00	1.76	0.17
more open to new ideas?	arts, humanities, law, social sciences	54	4.35	1.48	0.20	0.14
	hard sciences, mathematics	62	4.74	1.34	0.17
	TOTAL	116	4.56	1.42	0.13
no constraints from managers/colleagues?	arts, humanities, law, social sciences	56	5.07	1.39	0.19	0.306
	hard sciences, mathematics	61	5.31	1.13	0.15
	TOTAL	117	5.20	1.26	0.12

Analysis of the mean scores shows that the respondents were definitely willing to attribute changes in their work habits to the training received. Positive means for both groups were found in "changed my approach to work," "improved communication with non-Romanian colleagues" ("communications with Romanian colleagues" received negative responses), "improved opportunities to stay current in my field," "helpful for my students," "do better on my job," "improved ability to do research" and "more open to new ideas". In nearly all of these results, the hard science group was noticeably more positive than the soft science group, even though both were positive. This is probably due to the lack of familiarity of the latter group with the possibilities that Internet use can bring. There were a few variables that elicited negative results from soft science respondents and positive from the hard sciences group. The differences on the "better on team projects" question indicate that there is so far little interest in changing from traditional methods to cyberspace-oriented team projects for the soft science group. The "manager actively supports" differences are difficult to explain but probably stem from the fact that hard sciences managers, like the researchers, already understand the potential benefits of Internet better than managers in the soft sciences.

Conclusions

It is clear that these researchers achieved many long-term results from their three days of training in Internet use. We feel that this type of questioning is very helpful in order to determine the degree to which such training can be an important part of a researcher's outlook toward his or her work, not simply as a course to be taken and forgotten. Since there are significant differences between groups, the nature of these differences is itself an interesting research issue. And many policy and strategy questions can be considered, based on these preliminary results. Should donors be more willing to invest perhaps 20-30 percent of an IT budget in training instead of the current 3-5% or less? Should soft science disciplines be given greater portions of the IT budgets to help them achieve parity? It may be possible that the greatest benefit of this training is in the openness and confidence that it instills in researchers so that they can go beyond their boundaries--social and technical. A recent Atlantic Monthly article sums up what may be the greatest advantage:

In the end the primary payoff from advances in Information Technology may be not in new and better goods and services but in new and better democracies. (Blinder and Quandt, 1997)

Or, to quote one of the graduates of this Internet Academy:

Nowadays the Internet access is vital for the true, professional condition of a researcher in Romania. Without it, the iron curtain falls back again, this time for other reasons. For the time being, the Internet is the only constant connection we have with the scientific community outside the country, the only opportunity we have to get new, up-to-date knowledge, to be informed and finally, to be "contemporary" in the real sense of this word." (G. Nagât from the Institute of Philosophy)

References

Blinder and R. Quandt, "The Computer and the Economy," Atlantic Monthly. December 1997, pp 26-32

R. Dumbraveanu, "Teaching The Internet To University Staff," http://www.sfos. ro/events/ rilw/Papers/Roza.html)

Harter, Bryan. http://www.icasit.org/romania/index.html

S. Ruth and R. Gouet, "Must Invisible Colleges be Invisible? An Approach to Examining Large Communities of Network Users," Electronic Networking Research and Practice, 3 (1), Spring 1993 pp 34-60

D. Tufis "INTERNET-Lab: Exposing Romanian Academics from the Humanities to the Internet Technology," International Seminar on Internet Training, Rodes, September, 1997

World Bank, Internet Toolbook for Task Managers, Washington, DC, World Bank, 1996 (also published at http://www.worldbank.org/html/fpd/ toolkit)

Acknowledgments

The course described in this paper has been supported by grants from the Andrew W. Mellon Foundation, the Soros Foundation and Sun Microsystems -- and it has been functioning continuously for almost three years.