65th IFLA Council and General
August 20 - August 28, 1999
Code Number: 110-84(WS)-E
Division Number: VII
Professional Group: Library Theory and Research: Workshop
Joint Meeting with: -
Meeting Number: 84
Simultaneous Interpretation: No
Value and impact studies
Department of Information Studies.
The University of Sheffield
At a time when emphasis is being placed on quality, standards, and accountability the research world is spending more time assessing the value and impact of library organizations. The paper argues that research in this area requires sociological and psychological research skills. Qualitative data are as important as statistics. Different research methodologies are identified in a review of some recent studies. The result of such work must be made known, and new research is required to explore the relationship between research and the decisions taken by professionals and policy makers. We must ensure that value and impact studies have value and impact.
At a time when so much emphasis is being placed on quality, standards, and accountability, it should not come as a surprise that the research world is spending more time assessing the value and impact of organizations. Serious difficulties arise however when researchers try to deal with these issues using inappropriate tools. Tools that can be used to assess outputs, may not be the same as those we use to evaluate outcomes. Value and impact studies are much more than mere measurement. Statistics are just a small part of the reality of the library and any meaningful evaluation has to go beyond simplistic quantification. Even the quality guru Deming, who is often associated with the use, if not over use of statistics, has admitted that the most important things in life can not be measured. How does one measure love, kindness, and generosity? Many of the measures that are set out in the managerialist literature are often substitutes for qualities that are intangible or indirect..
Academic librarians have observed that, "relatively little attention has been paid to qualitative measures, or to output measures, but indicators which fail to take such factors into account will be inadequate and misleading." (Joint Funding Councils' 1993) In the public sector it has been argued that, "performance can seldom be expressed in a meaningful way by quantitative data only. To a great extent, analysis of performance has to be based on qualitative descriptions and statements." (Arvidsson 1986) If library services are to be evaluated in terms of their impact on individuals, and society as a whole, then we require sociological and psychological research skills. It means that we must be concerned with the "soft" as well as "hard" areas. As Stewart and Ranson(1988) have observed: "Performance monitoring in the public domain is not merely concerned with effectiveness in achieving stated values, but with unexpected impact, and of values denied."
In this brief review I am therefore going to concentrate on qualitative methodologies as used in some recent studies. We can also borrow research methodologies from other fields. For example, although a great amount of work has been carried out on "the library user", we still know relatively little about the functions of library materials in terms of their impact on the individual or the community. In a Sheffield study, on the value and impact of imaginative literature, we plan to combine the methodology, developed in our previous social audit work, with the "uses and gratification's" techniques used by researchers in the mass media (e.g. Brown 1976, McQuail 1969), and the reader response approach which is seen as integral to the understanding of literature. Similarly, in evaluating and planning services for communities, there are techniques such as cognitive mapping, which may help library managers gain a clearer impression of people's perception of an area or locality.
Perceptions play a significant part in value and impact studies, and research from other areas points to the dangers of making management, or indeed political, decisions on the basis of users' perceptions alone. Simply asking users is often not enough. Users know what they like, they sometimes know what information they need, but they often do not know what is possible for the library to provide. Some explanation of the different perceptions between librarians and users can be found in the literature on the psychology of prediction. For instance Slovic et al (1982) Tversky and Kahneman (1982) suggest that people judge events as likely if instances of it are easy to imagine or recall. What people can recall will depend on their knowledge and experience of the area. Library users by definition will have personal experience of using libraries. They will therefore have some relevant knowledge but as Stewart and Walsh (1989) suggest, from a public service perspective, in a number of areas this knowledge will be incomplete.
The public library service is above all about providing equality of access to information, ideas, and works of imagination, and it is important that this is reflected in the way services are evaluated. With equity, as with anything else, performance has to be evaluated against objectives. In the UK there are national standards, such as those set by the Audit Commission, but the evaluation of equality is still in its infancy. There is a tendency to rely to a great extent on numbers. For example counting the number of books in a foreign language, or the number of people from a particular community, or background, who use, or do not use a service. These kinds of data are, of course, important, if nothing else they are an indicator of an authorities' commitment. However such figures tend only to deal with inputs and outputs. In evaluating equality, people's experience of using the service, and their perceptions of it, can be used to evaluate outcomes.
Such qualitative outcome indicators are often a more meaningful way of assessing the value and impact of a service and its achievements. This, more qualitative approach, has been used in a number of recent studies of social impact. In Beyond book issues Matarasso (1998) reviewed library based projects entered for the Library Association Community Initiative Award, and assessed the extent to which they have produced social benefits and seeks to identify factors which lead to success. Another piece of work, undertaken by the Community Development Foundation, looked at working with local people to develop indicators of social benefit. (Harris 1998)
The Sheffield Social Audit study was the subject of a paper at last year's IFLA (Usherwood & Linley 1999) so suffice it to say that the technique has much in common with quality audits, as defined by Percy-Smith (1992), and implemented by a number of local authorities with regard to recreation, transport and information technology services. It is has also been used by a variety of voluntary organizations, and companies such as Traidcraft and The Body Shop. To quote Gray (1995) social accounting has experienced "a (long overdue) resurgence as academics ...look for new ways of providing accounts of organisational life." The technique can be used to examine the success or failure of a portfolio of activities and services offered by a particular kind of library. For example inner city, urban or rural branch libraries or those serving designated areas of poverty.
Another Sheffield project, on the Impact of library closures and reductions in opening hours (Proctor, Lee & Reilly 1999), involved before and after studies of public library users in communities where closures and reductions were taking place. The data show that when a local library closes up to a third of adults may be deprived of access to the public library. Worst affected are young children and elderly people. Parents and teachers told the researchers that the impact of library closure was devastating. Although some children were reading as much as before, their choice and quality of reading suffered dramatically. People felt that only a local library helped them to feel a part of their own community and play an active part in it. It helped them to be part of a local information "network", reduced loneliness and, encouraged friendships.
The Sheffield strike research is an example of opportunistic research. In the Summer of 1995 Sheffield Libraries closed down for eight weeks. This gave researchers the chance to find out how people responded to the lack of a library service. Over 500 library users were interviewed after the libraries re-opened. Other local libraries and bookshops were surveyed to see whether people had transferred their use there. A survey of telephone callers using the information services was also carried out. The research showed that the social value of the local public library has been underestimated. People enjoy the experience of going to the public library, whether or not they need to borrow books or seek information. It seems to make a significant contribution to their quality of life. This seems especially true in communities with a high incidence of economic and educational deprivation. The local library can be an important resource for personal development, particularly when users have had a poor experience of formal education. The data also demonstrated the extent to which reading is an essential and critical factor in the lives of library users. Most people cannot afford to buy enough books to replace those provided through the library service. Turning from the established to the electronic technology, colleagues at Manchester Metropolitan University are currently looking at the value and impact of End-User IT services in public libraries. They are using what they describe as a "mix of methodologies, including desk analyses, structured interviews, and observation. Another Sheffield project seeks to demonstrate the value and impact of public libraries on educational attainment and the reduction of educational disadvantage in British Society. It will investigate the role of the public library in supporting and encouraging learning, in its widest sense, among the educationally disadvantaged. Learning will be taken to include both the absorption of new knowledge and information, and the acquisition of information seeking and retrieval skills.
The value and impact of library research.
Crucially the results of research must be made known. For a number of years we have strongly argued the need to establish a centre for the effective dissemination of public library research. We emphasised that this should not be confined to work carried out in academic institutions and other research centres but also include the "hidden research" that is undertaken as part of the day to day management of libraries. We have made some suggestions regarding the dissemination of research of interest to the library community. These include : Establishing a clearing house for the dissemination of reports and surveys prepared by different library organizations; developing a data base of these activities and; establishing a web site which would keep practitioners, academics, policy makers and others up to date with research being carried in library authorities, in universities, and other organizations with an interest in the field.
New research is required to explore the relationship between the decisions taken by library managers and policy makers, and the research that is undertaken by universities and other research organizations. In suggesting this approach I do of course recognise that decisions are constantly made in libraries which are not always based on systematic research. Such has been the pace of change that there will be times when, "Life cannot wait for social research to catch up." (Shipman 1988). Equally the quality of decisions may be improved if more people within library organizations catch up with research that has been undertaken.
Particular attention should paid to the ways in which research results are communicated within library organizations and how far the research focus of the organization is at risk through dependency on individuals rather than systems. Key professional staff and policy makers need to recognize the value of research as a contributor to performance. As researchers we need to ask, how far library services can be described as having a research ethos, through which the value of research is conveyed to professionals and policy makers. In the early 1970s a number of research officer posts were established in library directorates but most seem to have disappeared, although I did notice recently the appointment of a Research & Support Officer in a British public library.
The 'implementation' of research findings involves more than simply applying a recipe. Research is often about innovation and all innovations require organizational change to some degree if they are to be implemented effectively. For research to be 'implemented', someone needs to know about it, they must influence the organization and persuade policy makers that change along the indicated lines will be valuable. "Thus whilst knowledge of the results of research may be important, it is rarely by itself sufficient to change practice, and other factors need to be taken into account" (Watt 1996). Senior library managers need to persuade and cajole others in their organizations to try out new things, and to invest in people and other resources to provide the training that changes will require. In other words, implementing research findings is a major organizational task and not something simply to be bolted on.
Much of the recent work on the dissemination and implementation of research is to be found in the clinical and medical literature. Some of the approaches used in the Health Services GRIPP (Getting Research into practising and purchasing) initiative and by Canadian clinicians (Clark 1995), could be adopted for use in the library sector. We can also note that many of the models used by health researchers derive from the social sciences. For example in their use of Rogers (1995) work on the diffusion of innovations. Whatever model we adopt we must ensure that our research; illuminates the activities of library users; helps professionals and policy makers assess the impact of the services they provide, and enables them to identify the reasons for their success or failure. The research should help managers guide and monitor the service, and improve the way the value of the service is reported to policy makers. It should enable stakeholders to make better judgements about the service, and affect organizational behaviour. In short we must make sure that value and impact studies have value and impact.
1. Arvidsson, R. (1986) Performance evaluation In Kaufman, F.X, et al eds. Guidance , control and evaluation in the public domain. Berlin. de Gruyter
2. Brown, R (1976) Children and television Collier Macmillan
3. Clark, H.H. (1995) The Nightingale legacy Paper given at Child Health 2000. International Pediatric (sic) Nursing Conference. June 2 http://caliban.ido.gmu.edu/
4. Gray, R. (1995) Social and environmental accounting research. Briefings from the GEC Programme. GEC Programme Office, University of Sussex.
5. Harris, K. (1998) Open to interpretation: Community perceptions of the social benefits of public libraries. Community development Foundation. (BLRIC report 88.)
6. Joint Funding Councils' Libraries Review Group Report (1993) Libraries review group, Education Funding Council Bristol, HFCE. [Group chaired by Sir Brian Follett]
7. Matarasso, F. (1998) Beyond book issues: The social potential of library projects. Comedia (BLRIC Report 87)
8. McQuail, D (1969) Mass communications theory an introduction Sage
9. Percy-Smith, J .(1992). 'Auditing Social Needs.' Policy and Politics. 20 (1) p.29-34. (British Library research & Innovation Centre Report 90)
10. Proctor, R, Lee, H. & Reilly. Access to public libraries. The impact of opening hours reductions and closures 1986 -1997 Centre for the Public Library in the Information Society (British Library research & Innovation centre Report 90)
11. Rogers, E. M.(1995) The Diffusion of innovations New York. The Free Press
12. Shipman, M (1988) The limitations of social research. 3rd ed. London, Longman
13. Slovic, P et al (1982) Facts and fears: understanding perceived risk In: Schwing, R and Albers, W, A, eds. Societal risk assessment. How safe is safe enough. New York Plenum.
14. Sobczyk,G & Proctor,R. & Usherwood, B. (1996) What do people do when their public library service closes down? an investigation into the impact of the Sheffield Libraries strike. British Library Research and Development Department. 1996. (British Library R&D Report 6224)
15. Stewart, J & Ranson. (1988) Management in the public domain. Public Money and Management Spring Summer 13-19
16. Stewart, J and Walsh, K (1989) The search for quality. Luton, Local Government Training Board.
17. Tversky, A and Kahneman, B (1982) Judgement under uncertainty: heuristic and biases. Science, 185.
18. Usherwood, B. & Linley, R New library - new measures; A social audit of public libraries . IFLA Journal 25 (2) 90-96
19. Watt, I (1996) The dissemination of R & D Information IFMH INform 7 (1) Spring 1 -4