Code Number: 107-124-E
Division Number: VII
Professional Group: User Education
Joint Meeting with: -
Meeting Number: 124
Simultaneous Interpretation:   No

A study of undergraduate information literacy and skills: the inclusion of information literacy and skills in the undergraduate curriculum

Abstract

Paper

Background

Increasingly there is a consensus that "human capital is worth more than either produced assets or natural resources" (World Bank 1998).

Concepts such as the 'information society' or the 'knowledge based' society also reflect this view, as does at a more micro level the 'knowledge based enterprise'. Within many organisations the ability to initiate and process change and the capacity to turn new ideas quickly into marketable products or services has acquired a premium, (Ashton and Sung, 1994). Staff are increasingly required who are multi skilled, have problem solving skills, can deal with less routine tasks, are IT literate and capable of independent learning.

These demands have significant implications for the education of primary, secondary and tertiary students.

In Singapore the government has realized that these skills are important for the longevity of the Singapore economy. One sector where it has pursued its 'vision' is in the co-ordination of change in education and training. Similar initiatives are evident in the United States, Europe, Australia, and South Africa where information literacy as well as a more collaborative and constructivist approach to learning is considered an economic necessity as well as helping to achieve personal fulfillment. (Bruce, 1995; Doyle, 1992, 1994; 3Com calls, 1997; Technological change, 1998). Curriculum changes have taken place at the primary and secondary school level to encourage these skills. Changes are also starting to take place at the tertiary level.

The meaning of information literacy and skills

Information skills and literacy have been defined as,

"the process of acquiring knowledge of attitudes towards and skills in information, as a major determinant of the way in which people exploit reality, develop, live, work and communicate in an information society"

(Marais, 1992, p75)

The concept of information literacy and skills has been interpreted in various ways since the early seventies. Terms such as 'study skills', 'research skills' and 'library skills' tend to be used in the educational context. 'Lifelong learning' and 'creative thinking', imply preparation for the work environment and continued productivity. Lifelong learning, however, also has the nuance of self empowerment and is not necessarily only associated with the world of work. 'IT literate' on the other hand in the information literacy context relates to those skills associated with the use of the electronic medium of information access and delivery. The following definition encompasses the concepts above and leads to the goal of achieving a 'lifelong learner' i.e. someone who can educate themselves, (Kuhlhau, 1991; Doyle, 1992, 1994; Bruce, 1995; Eisenberg & Brown, 1992). It also extends previous definitions to encompass the gathering and use of primary data. Different contexts and situations as well as individual style will result in the choice of certain strategies. Generally, however, a student who is information literate and has information skills is expected to:

• recognize that accurate and complete information (textual, numeric or graphical) is the basis for intelligent decision making. In other words they have values which promote information use and can implement information processes. This would include the confidence to tackle relatively undefined problems through the knowledge that information could be found.

• determine exactly what problem or aspect of the problem they are trying to resolve. This would include skills such as brainstorming or concept mapping which would enable the individual to place the problem within a particular knowledge context.

• define and determine what information is required for the task in terms of type of material, media and comprehensiveness. This requires the individual to understand what may be expected in terms of the final 'product' whether it is a final year project report or an executive summary.

• formulate questions based on information needs that help to define what content is required. For example would case studies, past experiments, information on methods, regulatory information etc. be useful. What exactly do they need to know?

• identify potentially relevant and valuable primary, secondary and tertiary sources of information. This implies a knowledge of the 'world of information', including journals, books, people, organisations, electronic newsgroups etc. In terms of primary data this would also imply knowledge of appropriate research techniques.

• develop successful search strategies in both the paper based and electronic domain for identifying secondary information. This requires knowledge of relevant paper and electronic full text and surrogate indexes as well as the principles along which such systems are organized. In the area of primary data knowledge of qualitative and quantitative research techniques is required. It also implies the use of effective scanning techniques such as the use of contents pages, headings, conclusions, summaries etc.

• gather information and data through experiment or through secondary sources. This may involve the use of appropriate research methodologies such as sampling and the application of good experimental practice such as taking laboratory notes. It may involve the use of information technology such as the Internet and World Wide Web search engines or publicly available online catalogues. It would also involve effective adaptation of search strategies, browsing and recording of useful data.

• Organize and store information. This requires the use of information technology to develop databases, containing for example, bibliographic citations, quantitative or qualitative data. It would also include the basic skills of note taking and organization of data and information.

• interpret, analyse, synthesise, evaluate and also to critically challenge the validity of collected information. This would include recognition of the need to evaluate and corroborate before adopting information.

• develop insights, judgements, and predictions. This would involve extrapolation and the application of findings to different and possible future situations.

• use effective and appropriate tools and methods for the presentation and visualization of data and findings. This involves knowledge of effective presentation and communication style and format as well as an understanding of what is appropriate to the situation. It increasingly requires knowledge of presentation and data visualization software.

• develop strategies and techniques for the publishing of results and reports. This requires knowledge of the avenues for publishing in that domain and increasingly the ability to distribute information using electronic formats such as hypertext mark up language (HTML) or portable document format (PDF).

• adapt these cognitive and behavioural information 'strategies' to different situations and contexts.

Information Literacy and Skills in this context is therefore far more than what is generally termed 'library skills' which tends to focus on the location and access of information. It is also broader than the mechanistic skills associated with the concept 'IT literate'. It should be evident that the library can play a key role in helping to develop this knowledge and that the librarian, particularly the reference librarian, is generally well versed in the majority of these areas.

Information literacy and skills of tertiary students at Nanyang Technological University

Due to the increasing importance of the knowledge and skills outlined above as well as anecdotal evidence to suggest that students may be weak in a number of these areas a study of students at Nanyang Technological University took place.

To investigate the student's information literacy and skills a qualitative approach was taken and applied while students conducted their research. This was felt necessary to get an insight into the student's thinking and behaviour while the seeking and gathering of information took place. This contrasts with previous research that has tended to concentrate on faculty perceptions of student's information literacy and skill. It was also felt that this approach would give a richer understanding of the problem situation than the use of questionnaire or interview techniques. The study had two primary objectives concerning information literacy and skills. Firstly to understand the process students went through when undertaking a piece of independent study. Secondly to understand the difficulties these students experienced.

Students were given a choice of research questions to research. A combination of the following methods were used to study the students:

• Task analysis: Task analysis involves asking the respondent questions about what they have to do to complete a specific task and how they intend to do it. This method was used to capture the student's perception of the research process and how they intended to undertake the various tasks. This was expected to give an insight into their knowledge of the information environment and also an insight into what they thought they should do.

• Talk-through: Talk-through involves getting the respondent to verbalize their thoughts while undertaking a task. This helps to understand the cognitive process the student goes through and also the problems they may have such as a lack of understanding or confusion.

• Observation: Observation was used to capture data on what the respondents did, for example, which systems did the students use and what commands did they enter and also how did they find material in the library.

There were limitations associated with the research. For example in the laboratory where the students accessed the OPAC, networked CDROMs and the Internet there were some sources that the student could not access. In addition more use of the paper-based domain, for example the browsing of journals and reference material may have taken place if the search had started in the library rather than the laboratory. More collaboration between students may also have taken place if they had not been aware of being observed (Twidale et al. 1997).

Another factor that is significant in terms of the difficulty students experienced and the behaviour witnessed is that the students were tackling topics that they were unfamiliar with. This is very different from either working in a very well known domain or looking for a specific known item.

Lastly in this situation emphasis was placed on the use of secondary sources i.e. library and Web based material. There was no opportunity to conduct primary research. In the work environment a survey or experiment of some kind would have been expected.

Despite these constraints the study was able to elicit data on students information seeking behaviour and areas of student difficulty were identified.

Findings

The following highlights students' major areas of difficulty when conducting the project.

Defining the problem. Students found it difficult to place the problem (question) in the broader context. Little time was spent determining what branches of knowledge encompass or are related to the question. Instead students tended to make a very literal interpretation of the question and started looking for material that mentioned those words listed in the question. This led to great frustration when searching for information. This was because without having gone through the process of trying to understand the subject domain using for example brainstorming or concept mapping techniques students found it difficult to derive related concepts, synonyms etc. that would have helped to find relevant records.

This may be partly because students are more familiar with dealing with situations where the nature of the problem is carefully defined by the teacher or lecturer, and also where clear guidelines are given for the answer. This has generally been the case in both secondary and tertiary education in Singapore. This is however now changing (Information Literacy 1997a, 1997b, The extensive reading, 1997). It may also because of the 'academic' nature of the problem. In their personal or work life they may be more motivated to define the nature of the problem.

Defining where to go for information. Generally students were unaware of the range of sources of information that could be used to identify relevant information; they had a poor understanding of the 'information landscape'. For example many students were unaware of the role of the Online Public Access Catalogue, (OPAC), assuming incorrectly that articles as well as books could be found via the OPAC. The majority were unaware that indexes to articles could be found on CD ROMs that related to their discipline. They generally had little idea of the production of knowledge or the relative merits of different sources of information. For example it was not obvious to students that different value would be placed on a refereed journal article to that retrieved from a newsgroup.

Presumably this indicates that the students had had little contact with such tools before. Nor had they been expected to critically evaluate sources of information in the past.

Developing search strategies. Although aware of boolean logic students had difficulty creating search strategies. They did not appreciate how information is structured in systems nor did they appreciate that systems work in very different ways. Little use was made of help systems. Truncation, synonyms, the use of fields (other than author or title) were rarely used. Systematic approaches for constructing searches, narrowing or broadening searches were not evident. In fact students experienced a great deal of frustration when they searched for information. The success rate of the students in identifying relevant information was low.

Finding material in the library proved difficult. Students were unfamiliar with the organisation of the library environment and found it difficult to locate material.

Developing insights and extrapolating also proved very challenging for the students. This seemed to be partly due to a lack of confidence, the fear of 'getting it wrong' and also a lack of experience in using information to creatively derive their own ideas.

Students were however found to be capable in the areas of extracting information, note taking, synthesising the information and then developing impressive presentations. No opportunity was given to the students in this study to 'publish' their material.

Overall their self-directed learner skills seemed a little weak. These findings to a great extent are consistent with those reported by Leckie and Fullerton (1999) in the United States. Attitudes, knowledge and skills in the area of information literacy therefore need to be enhanced and developed.

Incorporating information literacy and skills into the undergraduate curriculum

As a result of these findings initial steps were taken to see how information literacy and skills could be incorporated into the undergraduate curriculum. To aid implementation the following steps, including getting backing from the administration as well as 'prototyping' initial strategies, were taken.

Stage 1: A report was presented to the Dean of the School of Applied Science, highlighting problems and possible solutions, who in turn gained acceptance of these ideas amongst the other Deans. As a result lecturers were encouraged in the Schools to address this area. The Communication Skills staff in Civil and Structural Engineering (CSE) were the first to approach the Division of Information Studies to elicit help to incorporate these skills into their curriculum.

Stage 2: It was decided that the preparation of students for their In-house Practical Training (IHPT) project would provide a good opportunity to incorporate information literacy and skills. In IHPT students explore a given topic, conduct experiments, write a report and present findings. The decision to incorporate information literacy training in IHPT was based on the assumption that these attitudes, knowledge and skills cannot be taught in isolation. Students need to apply the skills while conducting coursework. The students need to be able to see how these skills can help them in the completion of specific tasks.

An initial lecture was given by the Division of Information Studies on Information Literacy and Skill and its importance in both the educational, work and domestic environment. This was followed by hands-on training of students to use the OPAC, CD ROMs as well as the Internet to search for material that would help them with their IHPT project. Staff from the Division of Information Studies in conjunction with Library staff developed and ran hands-on training sessions in tutorials. The students were able therefore to identify similar projects, develop their own ideas and place their project in the wider context. It was felt that students would be motivated to absorb this knowledge by providing the students with concrete evidence of how these skills could help them complete their assignments. The CSE lecturers stated that the reports and presentations that were eventually generated by the students did show increased evidence of having benefited from the training. This was demonstrated by their use of a wider range of sources than usually evident; by placing their topic in the wider Singaporean and international context; the inclusion of Web based materials such as graphics, and the use of bibliographic standards.

It should be emphasised that this provided students with an introduction only and their information literacy and skills were still relatively weak. Using IHPT, on its own, as a vehicle to convey these skills was therefore seen to be only partially successful.

Stage 3: On the basis of stages 1 and 2 a further proposal was put to the Dean of the School of Applied Science (SAS), which resulted in the Division of Information Studies taking on the task of incorporating Information and Literacy Skills into the SAS curriculum. It also led to a decision to develop a free elective course called 'Information Skills for Engineers'. The latter is expected to enable students who take this course to develop a higher order of information literacy and skills that relate to the materials, computer software and engineering students. It is hoped that both initiatives would lead to the instigation of similar initiatives in other Schools.

Stage 3 led to the incorporation of these skills in the first year Communcation Skills course (CE105/ME105, Effective Communication). During the undergraduates' first semester communication skills lecturers introduce students to a range of skills including report writing. Part of this is to conduct a small piece of independent research and to produce a report. It was felt that this would provide a good opportunity to introduce students to information literacy and skills.

A survey of lecturers in the school was conducted to get their views on how information literacy could be incorporated in the undergraduate curriculum. Unfortunately there was little response to this survey. However subsequent meetings with the heads of Division did help to identify additional opportunities for incorporating information literacy and creative thinking into the curriculum. These meetings helped to ensure that the information literacy and skills programme related to the teaching and learning context of the School of Applied Science. As a result the following recommendations were made:

Year of Study	Course	Student Activity and Information Literacy and Skills instruction	Information Literacy and Skills outcome and evaluation
1st Year Semester 1 Semester 2	CE105 (Communication Skills)	Report writing and presentation (students are asked to choose a topic, research it and produce a presentation). Knowledge conveyed via one lecture followed by a hands-on tutorial exercise. Students searched for information that related to their presentation topics.	Basic Skills: Evidence of defining the problem and related areas of knowledge. Show awareness of relevant tools for finding information (such as the OPAC, Internet, CDROMs). Ability to formulate elementary search strategies (deriving relevant terms and concepts, constructing boolean searches, utilizing the most commonly used functionality of systems). Basic library skills of being able to locate relevant material. Ability to extract and incorporate information. Ability to present information (including evidence of using secondary sources through correct referencing).
End of Year 1. In-house Practical Training (IHPT)	As in CSE lecturers felt that IHPT provided an opportunity to expand and reinforce information literacy and skills. However it was noted that in Materials Engineering, in particular, the primary benefit of IHPT was the practical 'hands on' nature of the exercise. As a result in this case less emphasis was placed on IHPT as a suitable point for IL & IS in ME.	Sourcing information that would help students identify relevant methods, techniques and case studies relevant to their IHPT topic. Reinforce the previous teaching through brief lecture and practical 'hands-on' session where students look for information that relates to their topic. As indicated in the next column students would be expected to demonstrate more sophisticated search techniques and a greater awareness of the various sources of information.	Marking would need to reward students who, place the topic in the related area of study (describe the area of study and those areas that are related to it as well as giving examples of similar projects). show that they have been able to identify relevant terms, concepts that relate to the topic and can be used to search for material. identify relevant sources of information (books, journals, organisations with relevant library collections in Singapore etc.) and systems (CDROM indexes, WWW, OPAC etc.) construct efficient searches that demonstrate an understanding of the structure of search systems. effectively utilise information sources to identify useful methods, techniques, similar or comparative material. extract, synthesize and incorporate information and provide correct citations to the original sources.
2nd & 3rd Year	Courses involving experiments	Students would be expected to independently derive appropriate experimental methods to solve a problem and to identify and corroborate the results. In most experimental situations methods and outcome are predefined. This is due to a) the need to learn basic skills and b) lack of time in the curriculum. Sourcing information to help identify relevant experimental techniques. To find information to enable students to predict the results of experiments and then compare their findings with the results of previous similar experiments.	It is recognised that this could only take place with a few experiments that are more 'open-ended' (black box scenarios). Time does not permit this happening often. Again this would build on and reinforce skills identified above. Critical and creative thinking skills would also be encouraged. On certain courses this kind of activity would take place over a period of time and in groups. In these situations collaborative skills would be encouraged, for example, the sharing and co-ordination of information between members of the group. Information management skills specific to experimental situations would also be monitored and reinforced, i.e. 'good' laboratory practice, such as the recording of data and findings in laboratory note books.
Final Year	Final Year Project	Sourcing of information about the project area, the topic, the methodology, and previous work. Increased emphasis is placed on the thoroughness of the exercise. Students are expected to extrapolate. This requires great emphasis since students are reluctant to infer or make subjective statements.	High Level Skills: At this point in their career the student should be able to demonstrate a range of information literacy and skills competency. There would therefore be high expectations from staff in terms of the students ability to define the problem, place it in the context by identifying related areas of knowledge, relate their work to work done previously; make comparisons; provide evidence of having searched the full range of possible information sources and services, repositories of information and knowledge that could be of use (including people and less formalised avenues); have synthesised this data and shown how it is related to their own study; conducted their experiments and documented this process effectively; used appropriate and effective methods of visualisation to present data and findings; developed insights and placed their conclusions in the broader context; provided full references to any material incorporated in their report (numeric, graphical or textual) according to a recognised notation.

Implications for staff and the tertiary institution.

These recommendations for incorporating information literacy and skills into the undergraduate computer/software and materials engineering curriculum raise a number of issues concerning staff and teaching in the tertiary institution.

Although individual Divisions can introduce such changes it would be more effective on a university wide basis. This is because the resources available in individual Divisions or Schools may not be adequate and they will need to draw on centralised expertise and resources, for example, the library and technology. Ideally such an initiative would be a coordinated University wide programme. The initiative would encompass the following.

1. Training lecturers to be competent in this area particularly in the use of relevant sources and technology. It is incorrect to assume that lecturers, through their own studies, are necessarily competent in all aspects of contemporary information literacy and skill. Subsequent to this research staff training courses run by the NTU Library staff in the use of the Internet proved to be popular.

2. Lecturers would need support to help them incorporate such approaches into their curriculum. Examples of what has been done elsewhere can be useful, (Mosley, 1998).

3. Lecturers need to perceive the importance of this initiative. This implies 'sponsorship' of such an initiative by the senior staff in the University. This is particularly the case in universities with a rigid hierarchial decision-making structure.

4. Resources need to be made available to implement such changes. These would include paper-based resources and increasingly electronic resources. For example, Internet and CDROM access from lecture theatres, laboratories and sufficient telecommunications infrastructure to support these activities. Appropriate licenses with electronic data providers need to be negotiated to enable distributed access. Independent learning would be encouraged by access to such resources from student hostels or their homes.

5. Library staff will be expected to play a significant role in this exercise, particularly in the identification of relevant sources. In addition they would probably be involved in the hands-on training of students. Alternatively their efforts could be concentrated on the training of lecturers and teaching assistants. In addition increased effort needs to go into the provision of computerized and paper based training materials as well as subject specific guides. These are labour intensive activities and additional budget would need to be found for staff and resources.

6. Library staff would have to undergo training. This would include 'training for trainers' courses as well as courses that upgrade their knowledge of subject specific information sources and access tools. To enable appropriate support to be offered librarians need to get a better understanding of the specific needs associated with different disciplines, (Leckie et al 1999). This can be achieved by talking to and collaborating with faculty staff and also by studying the information behaviour of both academics and students. It could also be argued that librarians need to broaden their perception of information management to include the management of data. This is becoming increasingly apparent in the digital library environment, (Atkinson, 1999) and implies that they would benefit from courses in research methodology and experimental practice as well as methods for presenting and visualising information.

7. Technological infrastructure would need to be implemented to enable access to information, collaborative work, the sharing and presentation of information.

8. Methods of assessment have to be defined to ensure that information literacy and skills are assessed and hence taken seriously by the students.

9. Lecturers may need to change their style of teaching particularly where there is a tendency for lecturers to provide all the resources required by the students rather than encouraging students to explore the knowledge base of the subject independently. In these situations students would need to be clear that the process is important as well as the outcome. Students would therefore expect less well defined assignments. At present students can be critical of staff who do not define the tasks precisely. Lecturers are therefore wary of introducing assignments that depend on independent learning since it may lead to negative appraisals of the staff when students provide formal feedback.

10. Emphasis on the use of lectures for the transmission of information would need to be reduced, allowing more scope for project based independent work. However this is not easily done due to the quantity of information that has to be conveyed at the undergraduate level.

11. Information skills need to be fostered and supported throughout the student's university career and not through 'one off' exercises. Traditional approaches to bibliographic instruction and the library 'tour' provide only a limited contribution to this process. There is therefore a need for a staged approach to imparting information literacy and skills. Early on in the student's career basic information skills are required such as use of the OPAC, CD ROMs and the Internet. Students' would need to gradually gain confidence in brainstorming ideas and generating topic definitions. Later on a sophisticated approach to information, its access and use would be expected, particularly by the time Final Year Projects were conducted. Early coursework needs to incorporate the use of the information resources. Initial orientation of students to the library and information services within the University could be achieved through the development of an information literacy and skills package available via the network in addition to traditional library orientation and searching courses. Information literacy and skills evaluation criteria would become more stringent during the student's university career.

12. University wide policies for the publication of student's work (as well pedagogical material) via paper based and electronic means would need to be developed as well as a technological infrastructure to support it. Students would then be exposed to the complete cycle of knowledge generation.

Conclusion

This research derived empirical data that showed students faced considerable challenges in the area of information literacy and skills. A pragmatic framework was proposed for the incorporation of information literacy and skills into the undergraduate curriculum at Nanyang Technological University in Singapore. The implications of these changes are defined and a general framework for change given. These changes need to take place, it is argued, to ensure that future students leave tertiary education with information literacy and skills appropriate to the information age.

It is however not the sole responsibility of the tertiary institutions to foster this area of knowledge and expertise. The tertiary institutions depend on the educational experience of students in primary and secondary schools. It is fundamental that these schools encourage the independent learner and incorporate essential information literacy and skills in the curriculum. Singapore has made progress in implementing these changes. These changes are however not easy and take time and resources. One major challenge is the amount of information teachers in schools and universities have to convey. This is perhaps one of the strongest arguments for incorporating information literacy and skills in the curriculum and not treating it as a separate subject. The other is that unless students learn this knowledge in context and are shown how it can help them achieve better results then they are unlikely to absorb these relatively abstract ideas.

References

3Com calls, (1997). 3Com calls on schools to help Europe compete in the information age. M2 Presswire. October 27, 1997. Retrieved from Dow Jones News Retrieval.

Ashton, D. N. and Sung, J. (1994). The State, Economic Development and Skill Formation: A New East Asian Model, Working paper no.3, Leicester, UK:University of Leicester, Centre for Labour Market Studies.

Atkinson, R. (1999). Towards a redefinition of library services. In Johnson, P. MacEwan, B. (eds.) Virtually yours: Models for managing electronic resources and services. Chicago: American Library Association.

Bruce, C. S. (1995). Information literacy: a framework for higher education. The Australian Library Journal. 44(3), 158-70

Doyle, C. S. (1994). Information literacy in an Information Society: A concept for the information age. Syracuse, NY: ERIC Clearinghouse on Information & Technology. (IR - 97)

Doyle, C. S. (1992). Final report to National Forum on Information Literacy. Syracuse, NY: ERIC Clearinghouse on Information Resources. (ED 351 033)

Eisenberg, M. B. & Brown, M. K. (1992). Current themes regarding library and information skills: research supporting and research lacking. School Library Media Quarterly, 20(2), 103-110.

Information Literacy, (1997a). Information Literacy Guidelines. Singapore: School Libraries Unit, Language and Libraries Branch, Ministry of Education.

Information Literacy, (1997b). Information Literacy Supplementary Materials. Singapore: School Libraries Unit, Language and Libraries Branch, Ministry of Education.

Kuhlthau, C. C. (1991). Inside the search process: Information Seeking from the User's Perspective. Journal of American Society of Information Science, 42(5), 361-371

Leckie, G. J. Fullerton, A. (1999). Information literacy in Science and Engineering undergraduate education: faculty attitudes and pedagogical practices. College & Research Libraries, 60(1), 9-29

Marais, J. J. (1992). Evolution of information literacy as product of information education. South African Journal of Library and Information Science. 60(2), 75-79

Mosley, P. A. (1998). Creating a library assignment workshop for university faculty. Journal of Academic Librarianship, 24(1), 33-41

Technological change (1998). Technological change, feeding on itself, seems to be increasing at a rate where the only certainty is ever-more-rapid change. The Independent. May 5, 1998. p3. Retrieved from Dow Jones News Retrieval.

The extensive reading, (1997). The extensive reading and information literacy (ERIL) programme: implementation guidelines for secondary schools. Singapore: The English and School Libraries Units, Language and Libraries Branch, Ministry of Education.

Twidale, M. B. Nichols, D. M. Paice, C.D. (1997). Browsing is a collaborative process. Information Processing & Management, 33(6), 761-83

World Bank. (1998). Monitoring Environmental Progress, found at http://www-esd.worldbank.org/envmat/publicat/mep/mep04.htm. September 1998