As of 22 April 2009 this website is 'frozen' in time — see the current IFLA websites
This old website and all of its content will stay on as archive –
A methodology for evaluating outcomes has been developed as an integral part of the Sheffield University, social audit of libraries in Newcastle and Somerset. This research, which was funded by the British Library, sought to make sense of complex public sector outcomes. Some writers, (eg.Percy-Smith 1992) have made a distinction between social audits in the private sector - which they describe as social accounting - and those which address the effect of public policy on people. Perhaps the distinctive feature of the public sector, compared to the private sector, is that it exists to meet needs. We should, of course, acknowledge here the division of needs into existing, latent, or perceived.
In setting about a social audit of library services our objectives were:-
Given the primary interest of this section, this paper concentrates on the first of these. That is the methodological approach we used.
The basic research premise was to use a 'social process audit' (as described by Blake et al, 1976) to evaluate the social impact of libraries. The project sought to analyse the goals (aims), inputs (resources), outputs (the programme or service) and outcomes (actual experience) of the public library and information service. The research was undertaken through two separate but related studies in the City of Newcastle upon Tyne and the County of Somerset. These two local authorities offer considerable contrasts in terms of geography, demography, politics and the local economy. The methodology has therefore been tested in both rural and inner city locations. The overall research design is shown in Figure 1.
The 'starting point for any social audit process is to identify and clarify the values against which the activities of the business or other type of organisation are to be judged.' (Zadek and Evans, 1993) In this project, our starting point was the stated social objectives of the two authorities.
The literature search examined the library and information science field and the broader social science literature dealing with outcomes in the not-for-profit sector. We were also aware of several other projects related to the social impact of libraries; the findings from these, where appropriate, have been integrated into the thematic discussion that forms our final report. Policy statements and local authority information about the case study areas were also examined to inform the design of research instruments.
The geographical areas to be examined were agreed with the staff in Newcastle and Somerset. Before undertaking the fieldwork the project worker visited the two authorities, and undertook accompanied visits to the case study locations. This offered the opportunity to profile the areas using more informal observational techniques. Following this, profiles describing the nature of the areas, were written, with reference to the observations and material supplied by the two authorities. (Newcastle City Council, 1993; Somerset County Council 1995)
In Newcastle, interviewees included the Vice-Chair of the Leisure Services Committee, all but one of the elected members in the selected wards, and an opposition member of the Leisure Services Committee. Gaining the viewpoint of an opposition politician was regarded as important, in terms of balance, because the case study wards were all represented by the same political party, whereas the City as a whole had a quite substantial opposition group.
A cross-section of library staff was also interviewed. This included staff based in the four branch libraries in the selected wards, professional staff whose responsibilities covered these wards, and staff based in the City Library. Additional interviews were also carried out with the Community Co-ordinators for the wards in the study. This added dimension was suggested during an interview by one of the elected members, and proved valuable in complementing the data gathered from politicians and library staff. The interviews were semi-structured in nature, although some flexibility had to be built in to reflect individual factors such as library staff having responsibilities covering more than one ward. A 'City-wide' adaptation of the interview schedule was used for the two Leisure Services Committee members who represented wards from outside the case study area.
A similar approach was subsequently taken in Somerset. There, all but two of the elected members for County Council divisions in the case study districts were interviewed. Interviews were also carried out with the Leader of the Council and the Chair of the Libraries, Museums and Records Committee. Eighteen Somerset staff were interviewed, including three mobile library staff. The relatively large number of interviews reflected the size of the case study area. All the interviews were recorded and transcribed in full.
In Newcastle, the recruitment of the focus groups was undertaken by the research section of the Chief Executive's Department. These focus groups included both library users and non-users. Names of potential participants were collected from outside local housing offices in each ward. Whether individuals used libraries or not determined which focus group they were invited to, with groups held for both users and non users in each ward. In addition to these 'general' focus groups, a specific group was identified in each ward, with reference to Newcastle City Profiles. The groups chosen were in line with Newcastle Library and Information Service's vision statement, and therefore involved ethnic minorities, elderly people, young people, and lone parents.
In Somerset, there were not the same kind of research and community infrastructures and the groups were therefore recruited in different ways. Three were recruited through staff in Somerset County Council Information Points and community schools in West Somerset. An attempt was made to make these groups reflect the age and employment profiles of the local community. A further group was recruited through a Community Education Office. Unlike the participants recruited through community schools, most of this group were undertaking some form of continuing education and were all library users. The research in Somerset also involved an older people's discussion group, a parents' support group, a group drawn from a detached youth project, and a group of disabled people and carers, recruited with the help of a voluntary worker. The focus group discussions like the interviews were recorded, and subsequently transcribed.
All participants, other than the discussion club in Somerset, were given expenses for taking part In Newcastle, 90 people attended 14 groups, with library users being keener to take part than non-users. In Somerset, a similar number attended just eight groups. The relatively poor attendance in Newcastle may reflect a degree of cynicism or apathy in communities which may feel they have been 'over-researched.'
An extension to the project was designed to address these issues. The additional time was used to evaluate specific services across a wider geographical area. The choice of the two services investigated; business information service and local studies, was largely informed by the workshop discussion.
In Newcastle, focus group discussions were held with three local history groups. For the business information strand, it proved difficult to organise focus groups with business organisations, but a discussion was held with an individual member of Business Link staff. Focus groups were held at a Job Club and with a training group for people seeking to improve their IT skills. Considering the viewpoints of those seeking work and receiving training was considered particularly relevant in Newcastle given the authority's emphasis on skills and employment.
In Somerset, one local history focus group was held. A written response was provided by another local history society, with whom it was not possible to arrange an interview. One business information focus group was held, with a defence contractors' group. This was supplemented by interviews with business link staff.
Reports on the Newcastle focus group data were prepared by the City Council's research services department, and the themes described therein were used to inform the data analysis. (Research Services, 1997) Initially, the Newcastle and Somerset material was written as two separate case studies. This was reflected in the interim report which reported the two sets of finding and then drew out some common themes. The workshop discussed both the interim report and four short papers prepared to stimulate discussion of particular issues. Notes taken at the four group discussions and the following plenary session were used to further inform the data analysis. The raw data were coded into key categories which were then developed into an outline. The outline was used as a guide to arrange and present the data. (Mellon, 1992) The final steering group meeting was used to discuss the themes emerging, before detailed conclusions were reached.
Generalising from qualitative findings has been described as an 'impossibility.' (Patton, 1987) Our findings reflect what was found in two specific locations. However, we do suggest that reasonable 'extrapolation' is possible. (Cronbach, in Patton, 1987) and that they can be applied to other situations under similar, but not identical, conditions.' (Patton, 1987)
As with a financial audit, so with a social audit one has to trust the auditors, and the use they make of the data. In conducting this social audit we have made use of mainly qualitative data. We have used it in a way that honestly reflects the situations we found. Illustrative quotations have been chosen on that basis. They have been widely used to exemplify our key findings. Although it would not have been appropriate to reproduce them all in the final report the transcripts of our focus groups can be made available to interested colleagues. However, the individual interview transcript interviews will not be released so as to guarantee the confidentiality of individual respondents. The methods used in this study can be used by staff in public libraries to help assess the impact of the services they provide. However, it should be noted that social auditing, as developed by the NEF, does include an external verification process. (Zadek, 1994), In this project the researchers were from outside the two services being audited, and, the research was overseen by the independent steering group. If public library services were to use this technique to evaluate the impact of their own services, some means of external verification might be thought desirable..
It was tempting to provide a model for measuring the social impact of the public library. It is a temptation we have resisted because as Smith (1996) suggests there is a:-
Moreover in our original proposal we stated that "while an objective quantification of concern or social need is not possible, it is practical to develop a framework for an informed value judgement.".. We believe that we have made some progress towards that framework, and developed a practical tool that can be used by practising librarians. The success or failure of this will be influenced by the research culture and infrastructure of the organisation concerned. Figure 2 represents our current thoughts on what we term a framework for an informed value judgement.
Such a framework enables us to identify some of the matches, mismatches, and differences between the local authority's social objectives and the intermediate and final outcomes of the service. These will be discussed with the politicians and professionals responsible for the service, and the reasons for the matches, mismatches, and differences analysed in terms of the various components of the audit. Managers, armed with this information, can then take appropriate action to maximise the matches and minimise the mismatches.
The process has also enabled us to identify differences between objectives and outcomes. That is to say there may be beneficial outcomes which may not have been identified as library or even local authority objectives. The role of the public library in developing social cohesion and building individual and community confidence may be examples of such benefits. At the same time the technique will enable us to identify unintended disbenefits that may result from social objectives.
Our work has provided us with some fascinating data on the social impact of the library. We are able to say some very positive things about:
The sense of ownership that communities feel for their library service
The educational role of the library
Its economic impact
Its impact on reading and literacy
The part played by the public library in developing community identity and confidence.
Managing public library services for equity
On the basis of our data it is reasonable to claim that public libraries help individuals and communities 'get started' and 'keep going' on a wide range of activities. In addition, sometimes with the help of other agencies, libraries help advance and maintain individual and community development. The recognised and established functions of the public library in terms of education, information, culture, and leisure, remain important. In particular, respondents suggest that the public library is a significant resource for school children and adult learners, and an important source of information on careers and training opportunities. The library is perceived as providing equity for older people, those with disabilities and people from ethnic minorities. The equity impact is less strongly felt by lone parents and unemployed young people. In addition, some groups, especially lone parents, tended not to welcome being identified as a priority group. That being said, the data do suggest that library use improves the life chances of individuals, in terms of education and job opportunities. Moreover the provision of public library services helps promote social cohesion and community confidence. In both authorities libraries were perceived as, 'part of the cement in the social fabric.'
They are also seen as community landmarks that reinforce community identity. In particular it was found that local studies services play a significant part in sustaining community identity. For example in Newcastle, this record of community life was especially important given the decline in the industries that have traditionally helped shape the local regional identity. For many people public libraries are, 'the heart of the community'. Respondents, including local politicians saw the local library as having a symbolic value that was well regarded by non-users as well as users.
Researchers often talk of the importance of looking at data and telling a story. Our respondents have provided us with many stories, and many of these will be included in the final report of the project which is to be published shortly. (Linley & Usherwood forthcoming.) They show that the public library helps individuals and communities "get started" on a range of activities. It enriches the lives of many people and we believe that the social audit technique makes that enriching process visible. Many of our conclusions are based on qualitative, if you like, anecdotal evidence. This is real world data that has been obtained in a rigorous way. We make no apology for our approach, and if there is one message that we want you take away from today's presentation it is that, qualitative data, properly gathered, are valid evidence and should be treated as such by politicians and professionals alike.
The social audit technique makes known individual and community experiences of using the library service. In this contribution I have tried to explain our methodology in some detail and would welcome comments on it. The use of participants' accounts to explain causes and consequences of actions has a long history in the social sciences. As long ago as 1928 Thomas (1928 p572) wrote "if men define situations as real they are real in their consequences." Analysing the way 'men' define situations can help us understand the situation better. We believe that maxim can also be applied to LIS research.
Moreover we hope that the social audit approach will be used by staff in public libraries, and other public and voluntary sector information organisations, to assess the impact of the services they provide, and to enable them to identify the reasons for their success or failure. It is a practical tool, which can, help library managers monitor and guide the service, improve the way the value of the service is reported to policy makers, enable stakeholders to make a judgement on the service, and affect organizational behaviour. In short, a social audit can help improve the social performance of the public library
Working within the constraints of the present project we have only been able to involve a limited number of stakeholders. The perceptions of a greater variety of people may well have identified additional indicators. That said, we feel justified in suggesting that the process we have described is one that can be used and developed by professionals and policy makers. It is not yet perfect but to quote Blake et al (1976):-
Borrowed time? The future of public libraries in the United Kingdom .(1993) Bournes Green Comedia.
Linley, R & Usherwood, B. (forthcoming) New Measures for the New Library. A social audit of public libraries .
Matarasso, F. (1997) Beyond book issues: the social potential of library projects . Comedia: Bournes Green, Stroud.
Mc Cracken, G (1988) The long interview. (Qualitative research methods series no.13) Newbury Park, CA. Sage.
Mellon, C. A. (1990) Naturalistic inquiry for library science . New York Greenwood Press
Newcastle upon Tyne City Council (1993) Newcastle upon Tyne 1991 City Profiles; Results from the 1991 Census . Newcastle upon Tyne: Chief Executive's Department, Newcastle upon Tyne CC.
Patton,M. (1987) How to use qualitative methods in evaluation London:Sage,
Patton, M.Q.(1990) Qualitative evaluation and research methods . Newbury Park, CA Sage.
Percy Smith, J (1992) Auditing social needs Policy and Politics 20(1) 29 -34
Smith, P.ed. (1996) Measuring outcome in the public sector London:Taylor & Francis
Somerset County Council(1995) Somerset rural needs analysis project. Taunton: Somerset CC, Department for the Environment.
Thomas, W. (1928) The child in America . New York Alfred A Knopt.
Usherwood, B. (1996) Rediscovering public library management . London Library Association Publishing.
Zadek, S. (1994) Ethical accountability and strategic planning: the social audit approach. Website at:
Zadek, S. & Evans, R. (1993) Auditing the market: a practical approach to social auditing . Gateshead: Traidcraft /New Economics Foundation.)
Bob Usherwood is Reader in Librarianship & Information Studies at the University of Sheffield and President of The Library Association. Rebecca Linley, a graduate of Sheffield University, was the research worker on the Social Audit project.