As of 22 April 2009 this website is 'frozen' in time — see the current IFLA websites

This old website and all of its content will stay on as archive – http://archive.ifla.org

64th IFLA Conference Logo

64th IFLA General Conference
August 16 - August 21, 1998

Code Number: 054-94-E
Division Number: VII.
Professional Group: Library Theory and Research
Joint Meeting with: -
Meeting Number: 94.
Simultaneous Interpretation: No

New library - new measures: a social audit of public libraries

Bob Usherwood

and

Rebecca Linley

Department of Information Studies
The University of Sheffield
Sheffield, UK

Abstract:

The full impact of public library services can not be demonstrated by statistics alone. Library managers must be prepared to use indicators that use "soft" as well as "hard" data. The paper describes the social audit approach to evaluate the impact of public library services in Newcastle and Somerset. The various components of the audit are described and the results of the study briefly discussed. The author concludes that qualitative data, properly gathered are valid evidence and argues that they should be treated as such by politicians and professionals alike. The social audit is a practical approach which can help library managers monitor and guide the service, improve the way the value of the service is reported to policy makers, enable stakeholders to make a judgement on the service, and affect organizational behaviour

Paper

The full impact of public library services on individuals and communities can not be demonstrated by statistics alone. Library managers must be prepared, in all senses of that word, to use indicators that use "soft" as well as "hard" data. In the words of a County Librarian, they "need to focus on performance measurement which is meaningful to the user (i.e. qualitative) rather than convenient in Audit Commission terms." (quoted in Usherwood 1996.) Moreover, as the Comedia report (Borrowed Time 1993) stated, traditional performance indicators, "in no way can measure the quality of the relationship between a library, its...users and the geographical area it serves". They have revisited this argument in their latest publication, Beyond book issues . This, as its title suggests, seeks to develop performance indicators other than the simple number of books issued. Their suggested "measures" include such things as the "proportion of staff time dedicated to contact with the public (excluding book issues)" (Matarasso 1997). However in some ways these are only "milestone indicators", in that, although they go part of the way, they still tend to tell us more about outputs than outcomes.

A methodology for evaluating outcomes has been developed as an integral part of the Sheffield University, social audit of libraries in Newcastle and Somerset. This research, which was funded by the British Library, sought to make sense of complex public sector outcomes. Some writers, (eg.Percy-Smith 1992) have made a distinction between social audits in the private sector - which they describe as social accounting - and those which address the effect of public policy on people. Perhaps the distinctive feature of the public sector, compared to the private sector, is that it exists to meet needs. We should, of course, acknowledge here the division of needs into existing, latent, or perceived.

In setting about a social audit of library services our objectives were:-

To develop a tool for measuring the social impact of library activities in relation to stated objectives.
To investigate the social and economic impact of public libraries.
To investigate how far a library's activities, in practice, contribute towards the achievement of its social objectives.

Given the primary interest of this section, this paper concentrates on the first of these. That is the methodological approach we used.

The basic research premise was to use a 'social process audit' (as described by Blake et al, 1976) to evaluate the social impact of libraries. The project sought to analyse the goals (aims), inputs (resources), outputs (the programme or service) and outcomes (actual experience) of the public library and information service. The research was undertaken through two separate but related studies in the City of Newcastle upon Tyne and the County of Somerset. These two local authorities offer considerable contrasts in terms of geography, demography, politics and the local economy. The methodology has therefore been tested in both rural and inner city locations. The overall research design is shown in Figure 1.

The 'starting point for any social audit process is to identify and clarify the values against which the activities of the business or other type of organisation are to be judged.' (Zadek and Evans, 1993) In this project, our starting point was the stated social objectives of the two authorities.

The literature search examined the library and information science field and the broader social science literature dealing with outcomes in the not-for-profit sector. We were also aware of several other projects related to the social impact of libraries; the findings from these, where appropriate, have been integrated into the thematic discussion that forms our final report. Policy statements and local authority information about the case study areas were also examined to inform the design of research instruments.

The geographical areas to be examined were agreed with the staff in Newcastle and Somerset. Before undertaking the fieldwork the project worker visited the two authorities, and undertook accompanied visits to the case study locations. This offered the opportunity to profile the areas using more informal observational techniques. Following this, profiles describing the nature of the areas, were written, with reference to the observations and material supplied by the two authorities. (Newcastle City Council, 1993; Somerset County Council 1995)

Identification of Stakeholders

Social auditing has been described as a "means of assessing the social impact ... of an organisation in relation to its aims and those of its stakeholders." (Zadek, 1994) The identification of stakeholders is therefore an integral part of the technique. Stakeholders are defined as including "all individuals and groups who are affected by, or who can affect, the organisation." (Zadek, 1994) In using the method, our approach was to cross-check the views and perceptions of different stakeholders in order to obtain as full a view as possible of the social aspects of public library activity. The key stakeholders for our study were;- elected members; (that is local politicians), library staff; and groups of library users and non-users.

Steering Group

Many of the concerns of this study involved public policy and we recognized that such matters can raise difficulties of bias and objectivity. We were also aware of the problems in assessing evidence, and decided from the outset to work with a steering group that reflected a wide range of professional views and opinions. This Steering Group, met four times in the course of the project. In addition to representatives from the two case study authorities, the group included representatives from the Audit Commission and the New Economics Foundation. The NEF has carried out social audits for a number of organisations. Together the members of the group brought a range of professional and political perspectives to the research.

Pilot

A small pilot study was undertaken to test and refine the interview schedule prior to its use. The data from the pilot were used, together with the literature, to inform the design of interview schedule for the main research. The key amendment made to the interview schedule, as a result of the pilot, was the inclusion of a question on the impact of the closure of a library. This was suggested by references to the 'intangible' benefits of the library, with one respondent suggesting that a community would only appreciate the true value of a library if a service was withdrawn.

Interviews with Stakeholders

The interview schedule for elected members and library staff was developed with reference to the objectives of the respective authorities, and to the published literature. For local politicians and library staff, a semi-structured interview schedule, with prompts, was used to obtain the relevant qualitative data. This allowed the freedom for 'open-ended' responses to questions and for the interviewer to explore these responses. (McCracken, 1988)

In Newcastle, interviewees included the Vice-Chair of the Leisure Services Committee, all but one of the elected members in the selected wards, and an opposition member of the Leisure Services Committee. Gaining the viewpoint of an opposition politician was regarded as important, in terms of balance, because the case study wards were all represented by the same political party, whereas the City as a whole had a quite substantial opposition group.

A cross-section of library staff was also interviewed. This included staff based in the four branch libraries in the selected wards, professional staff whose responsibilities covered these wards, and staff based in the City Library. Additional interviews were also carried out with the Community Co-ordinators for the wards in the study. This added dimension was suggested during an interview by one of the elected members, and proved valuable in complementing the data gathered from politicians and library staff. The interviews were semi-structured in nature, although some flexibility had to be built in to reflect individual factors such as library staff having responsibilities covering more than one ward. A 'City-wide' adaptation of the interview schedule was used for the two Leisure Services Committee members who represented wards from outside the case study area.

A similar approach was subsequently taken in Somerset. There, all but two of the elected members for County Council divisions in the case study districts were interviewed. Interviews were also carried out with the Leader of the Council and the Chair of the Libraries, Museums and Records Committee. Eighteen Somerset staff were interviewed, including three mobile library staff. The relatively large number of interviews reflected the size of the case study area. All the interviews were recorded and transcribed in full.

Focus groups

Following the completion of the interviews with elected members and library staff, and analysis of the data collected, focus groups were carried out with users and non-users of library services. Again, the research instruments were designed with reference to the authorities' objectives and the literature. Holding focus group allowed data to be obtained from a larger number of respondents and for people to develop their original responses after they heard other people's views. (Patton, 1990)

In Newcastle, the recruitment of the focus groups was undertaken by the research section of the Chief Executive's Department. These focus groups included both library users and non-users. Names of potential participants were collected from outside local housing offices in each ward. Whether individuals used libraries or not determined which focus group they were invited to, with groups held for both users and non users in each ward. In addition to these 'general' focus groups, a specific group was identified in each ward, with reference to Newcastle City Profiles. The groups chosen were in line with Newcastle Library and Information Service's vision statement, and therefore involved ethnic minorities, elderly people, young people, and lone parents.

In Somerset, there were not the same kind of research and community infrastructures and the groups were therefore recruited in different ways. Three were recruited through staff in Somerset County Council Information Points and community schools in West Somerset. An attempt was made to make these groups reflect the age and employment profiles of the local community. A further group was recruited through a Community Education Office. Unlike the participants recruited through community schools, most of this group were undertaking some form of continuing education and were all library users. The research in Somerset also involved an older people's discussion group, a parents' support group, a group drawn from a detached youth project, and a group of disabled people and carers, recruited with the help of a voluntary worker. The focus group discussions like the interviews were recorded, and subsequently transcribed.

All participants, other than the discussion club in Somerset, were given expenses for taking part In Newcastle, 90 people attended 14 groups, with library users being keener to take part than non-users. In Somerset, a similar number attended just eight groups. The relatively poor attendance in Newcastle may reflect a degree of cynicism or apathy in communities which may feel they have been 'over-researched.'

Workshop

An integral part of the research was to discuss and review our findings with library professionals and politicians. To this end, we produced a preliminary report which provided the basis of a workshop held with invited politicians and practitioners. All participants were given a copy of the preliminary report. Delegates were divided into small groups to discuss the major themes arising from the research. Comments from the day's discussions were used to inform further consideration of the data. The final report takes account of the views expressed at the workshop session. Workshop participants also identified issues that were felt to require further consideration. These were:- a need to involve other partners who influence library objectives, for instance, in the field of economic development, TECs and Chambers of Commerce; a concern to include more stakeholders in a wider sense, notably voluntary and community groups; a need to evaluate the impact of large central libraries as well as the smaller, community libraries, and the economic benefit of libraries.

An extension to the project was designed to address these issues. The additional time was used to evaluate specific services across a wider geographical area. The choice of the two services investigated; business information service and local studies, was largely informed by the workshop discussion.

In Newcastle, focus group discussions were held with three local history groups. For the business information strand, it proved difficult to organise focus groups with business organisations, but a discussion was held with an individual member of Business Link staff. Focus groups were held at a Job Club and with a training group for people seeking to improve their IT skills. Considering the viewpoints of those seeking work and receiving training was considered particularly relevant in Newcastle given the authority's emphasis on skills and employment.

In Somerset, one local history focus group was held. A written response was provided by another local history society, with whom it was not possible to arrange an interview. One business information focus group was held, with a defence contractors' group. This was supplemented by interviews with business link staff.

Data analysis

In practice, the data collection, analysis and writing blended into one another (Mellon, 1990). The diagrammatic representation of the research design (fig 1). is therefore in a sense, artificial, but it does give an overview of the work undertaken. Initial findings were noted whilst the fieldwork was still being carried out and these were presented to the Steering Group. All the steering group meetings were used to develop ideas and refine findings.

Reports on the Newcastle focus group data were prepared by the City Council's research services department, and the themes described therein were used to inform the data analysis. (Research Services, 1997) Initially, the Newcastle and Somerset material was written as two separate case studies. This was reflected in the interim report which reported the two sets of finding and then drew out some common themes. The workshop discussed both the interim report and four short papers prepared to stimulate discussion of particular issues. Notes taken at the four group discussions and the following plenary session were used to further inform the data analysis. The raw data were coded into key categories which were then developed into an outline. The outline was used as a guide to arrange and present the data. (Mellon, 1992) The final steering group meeting was used to discuss the themes emerging, before detailed conclusions were reached.

Validity of the findings

In case study research, variation of method is used to improve the validity of the data. In this study data were collected from a variety of sources, using different techniques. These included, one-to-one interviews, focus groups, and pre-existing written material. Para-professional and professional staff were interviewed, and two case studies locations were used. The interviews were carried out by the project researcher. Focus groups were facilitated by her, and members of Newcastle City Council's research services department. At the same time, the Steering Group brought additional perspectives to the research process.

Generalising from qualitative findings has been described as an 'impossibility.' (Patton, 1987) Our findings reflect what was found in two specific locations. However, we do suggest that reasonable 'extrapolation' is possible. (Cronbach, in Patton, 1987) and that they can be applied to other situations under similar, but not identical, conditions.' (Patton, 1987)

As with a financial audit, so with a social audit one has to trust the auditors, and the use they make of the data. In conducting this social audit we have made use of mainly qualitative data. We have used it in a way that honestly reflects the situations we found. Illustrative quotations have been chosen on that basis. They have been widely used to exemplify our key findings. Although it would not have been appropriate to reproduce them all in the final report the transcripts of our focus groups can be made available to interested colleagues. However, the individual interview transcript interviews will not be released so as to guarantee the confidentiality of individual respondents. The methods used in this study can be used by staff in public libraries to help assess the impact of the services they provide. However, it should be noted that social auditing, as developed by the NEF, does include an external verification process. (Zadek, 1994), In this project the researchers were from outside the two services being audited, and, the research was overseen by the independent steering group. If public library services were to use this technique to evaluate the impact of their own services, some means of external verification might be thought desirable..

It was tempting to provide a model for measuring the social impact of the public library. It is a temptation we have resisted because as Smith (1996) suggests there is a:-

"... need for very careful modelling before drawing any conclusions on the efficacy of a particular management team or programme. In practice, it is almost always impossible to measure all the relevant variables, and it must be accepted that the analysis is both incomplete and partisan, in the sense that a particular set of stakeholders' values are being used."

Moreover in our original proposal we stated that "while an objective quantification of concern or social need is not possible, it is practical to develop a framework for an informed value judgement.".. We believe that we have made some progress towards that framework, and developed a practical tool that can be used by practising librarians. The success or failure of this will be influenced by the research culture and infrastructure of the organisation concerned. Figure 2 represents our current thoughts on what we term a framework for an informed value judgement.

Such a framework enables us to identify some of the matches, mismatches, and differences between the local authority's social objectives and the intermediate and final outcomes of the service. These will be discussed with the politicians and professionals responsible for the service, and the reasons for the matches, mismatches, and differences analysed in terms of the various components of the audit. Managers, armed with this information, can then take appropriate action to maximise the matches and minimise the mismatches.

The process has also enabled us to identify differences between objectives and outcomes. That is to say there may be beneficial outcomes which may not have been identified as library or even local authority objectives. The role of the public library in developing social cohesion and building individual and community confidence may be examples of such benefits. At the same time the technique will enable us to identify unintended disbenefits that may result from social objectives.

Our work has provided us with some fascinating data on the social impact of the library. We are able to say some very positive things about:

The social role of the public library
The sense of ownership that communities feel for their library service
The educational role of the library
Its economic impact
Its impact on reading and literacy
The part played by the public library in developing community identity and confidence.

Managing public library services for equity

On the basis of our data it is reasonable to claim that public libraries help individuals and communities 'get started' and 'keep going' on a wide range of activities. In addition, sometimes with the help of other agencies, libraries help advance and maintain individual and community development. The recognised and established functions of the public library in terms of education, information, culture, and leisure, remain important. In particular, respondents suggest that the public library is a significant resource for school children and adult learners, and an important source of information on careers and training opportunities. The library is perceived as providing equity for older people, those with disabilities and people from ethnic minorities. The equity impact is less strongly felt by lone parents and unemployed young people. In addition, some groups, especially lone parents, tended not to welcome being identified as a priority group. That being said, the data do suggest that library use improves the life chances of individuals, in terms of education and job opportunities. Moreover the provision of public library services helps promote social cohesion and community confidence. In both authorities libraries were perceived as, 'part of the cement in the social fabric.'

They are also seen as community landmarks that reinforce community identity. In particular it was found that local studies services play a significant part in sustaining community identity. For example in Newcastle, this record of community life was especially important given the decline in the industries that have traditionally helped shape the local regional identity. For many people public libraries are, 'the heart of the community'. Respondents, including local politicians saw the local library as having a symbolic value that was well regarded by non-users as well as users.

Researchers often talk of the importance of looking at data and telling a story. Our respondents have provided us with many stories, and many of these will be included in the final report of the project which is to be published shortly. (Linley & Usherwood forthcoming.) They show that the public library helps individuals and communities "get started" on a range of activities. It enriches the lives of many people and we believe that the social audit technique makes that enriching process visible. Many of our conclusions are based on qualitative, if you like, anecdotal evidence. This is real world data that has been obtained in a rigorous way. We make no apology for our approach, and if there is one message that we want you take away from today's presentation it is that, qualitative data, properly gathered, are valid evidence and should be treated as such by politicians and professionals alike.

The social audit technique makes known individual and community experiences of using the library service. In this contribution I have tried to explain our methodology in some detail and would welcome comments on it. The use of participants' accounts to explain causes and consequences of actions has a long history in the social sciences. As long ago as 1928 Thomas (1928 p572) wrote "if men define situations as real they are real in their consequences." Analysing the way 'men' define situations can help us understand the situation better. We believe that maxim can also be applied to LIS research.
Moreover we hope that the social audit approach will be used by staff in public libraries, and other public and voluntary sector information organisations, to assess the impact of the services they provide, and to enable them to identify the reasons for their success or failure. It is a practical tool, which can, help library managers monitor and guide the service, improve the way the value of the service is reported to policy makers, enable stakeholders to make a judgement on the service, and affect organizational behaviour. In short, a social audit can help improve the social performance of the public library

Working within the constraints of the present project we have only been able to involve a limited number of stakeholders. The perceptions of a greater variety of people may well have identified additional indicators. That said, we feel justified in suggesting that the process we have described is one that can be used and developed by professionals and policy makers. It is not yet perfect but to quote Blake et al (1976):-

"waiting for perfection ... in the field ... is like 'waiting for Godot' .... Professional managers need and deserve pioneering and ... bold experiments if they are to confront today's and tomorrow's social problems and pressures with any degree of success and humaneness." We hope that other librarians will follow the lead of our colleagues in Newcastle and Somerset and be willing to participate in some more "pioneering and bold experiments".

References

Blake, D.H., Frederick, W. C, Myers, M.S. (1976) Social auditing: evaluating the impact of corporate programmes . Praeger

Borrowed time? The future of public libraries in the United Kingdom .(1993) Bournes Green Comedia.

Linley, R & Usherwood, B. (forthcoming) New Measures for the New Library. A social audit of public libraries .

Matarasso, F. (1997) Beyond book issues: the social potential of library projects . Comedia: Bournes Green, Stroud.

Mc Cracken, G (1988) The long interview. (Qualitative research methods series no.13) Newbury Park, CA. Sage.

Mellon, C. A. (1990) Naturalistic inquiry for library science . New York Greenwood Press

Newcastle upon Tyne City Council (1993) Newcastle upon Tyne 1991 City Profiles; Results from the 1991 Census . Newcastle upon Tyne: Chief Executive's Department, Newcastle upon Tyne CC.

Patton,M. (1987) How to use qualitative methods in evaluation London:Sage,

Patton, M.Q.(1990) Qualitative evaluation and research methods . Newbury Park, CA Sage.

Percy Smith, J (1992) Auditing social needs Policy and Politics 20(1) 29 -34

Smith, P.ed. (1996) Measuring outcome in the public sector London:Taylor & Francis

Somerset County Council(1995) Somerset rural needs analysis project. Taunton: Somerset CC, Department for the Environment.

Thomas, W. (1928) The child in America . New York Alfred A Knopt.

Usherwood, B. (1996) Rediscovering public library management . London Library Association Publishing.

Zadek, S. (1994) Ethical accountability and strategic planning: the social audit approach. Website at:
http//sosig.ac.uk/NewEconomics/corpcit.html.

Zadek, S. & Evans, R. (1993) Auditing the market: a practical approach to social auditing . Gateshead: Traidcraft /New Economics Foundation.)

Bob Usherwood is Reader in Librarianship & Information Studies at the University of Sheffield and President of The Library Association. Rebecca Linley, a graduate of Sheffield University, was the research worker on the Social Audit project.

Chart

64th IFLA General Conference August 16 - August 21, 1998