65th IFLA Council and General
August 20 - August 28, 1999
Code Number: 102-163(WS)-E
Division Number: II
Professional Group: Science and Technology Libraries: Workshop (King Mongkut Institute of Technology Library)
Joint Meeting with: User Education Library
Meeting Number: 163
Simultaneous Interpretation: No
Evaluation of web-based library instruction programs
Tess Tobin & Martin Kesselman
Rutgers University Library Media Services
Piscataway, New Jersey
The World Wide Web (WWW) is changing the way academic libraries teach and learn. Academic libraries have embraced the potential of the WWW developing innovative ways to meet the needs of users. One of these ways is to deliver automated bibliographic instruction in the form of web-based tutorials. This paper will review recommended guidelines for web-based instruction and will provide a framework for libraries worldwide in designing web-based instructional programs.
The stated purpose is to identify important design and evaluation criteria for web-based literacy/information instruction. Building on an initial investigation, the author examined library instruction literature to see what criteria was available for those wanting to create effective web-based training modules. The author also looked at non-library literature to find web-based training criteria, which was then compared to the criteria developed by library professionals. Over a two-year period, the author was able to note a progression and shift in the guidelines developed by library professionals, which reflects, not only, improvements in the technology itself, but a better understanding by librarians as to what elements are needed to structure an effective online bibliographic instruction program. The library guidelines were examined within the context of current research on bibliographic instruction to determine the usability or deficiency of the established criteria.
To date there are very few end-user studies in the library instruction literature regarding online tutorials. The last section of this paper will report on assessment and usability studies of web learning modules.
Computer scientists have been researching the principals behind effective interface design of computer applications since the 1940's. Stover and Zink report that with the emergence of the World Wide Web in the early 1990's, organizations of all types are recognizing the importance of the World Wide Web as a tool, not only for gaining access to information, but also as a means of disseminating information about their activities, products, and services. 1
Academic librarians have endorsed the possibilities of the WWW and first began to use web technology to create home pages, as starting points, or gateways for searching for information about the library. A home page reflects the institutional character of the university or college and is unique in that it gives the library the opportunity to express its own mission and philosophy.
With the growth of distance learning and the need to offer services to patrons off campus as well as in the library, the libraries needed to explore other ways to deliver library information skills. The WWW provides a dynamic environment for distributing information over a large network and web-based instruction is becoming the desired tool for these new users. In the web environment it doesn't matter if the user is connecting to library resources on the computer in the library, elsewhere on campus or from his/her home.
Developing an online module can be a challenge for many libraries. As librarians approach this task they should not overlook the groundwork laid by computer scientists in the field of computer interface design and computer/human interaction. The knowledge the instruction librarian brings to the task gathered from their classroom experiences and the findings from computer-assisted learning studies need to be considered when planning the design, content and evaluation of a web-based tutorial.
What is web-based instruction?
Web-based training (WBT) is an innovative approach to distance learning in which computer-based training is transformed by the technologies and methodologies of the World Wide Web (WWW), the Internet, and Intranets. 2 It allows self-directed, self-paced instruction in any topic. WBT is media-rich training fully capable of evaluation, adaptation, and remediation that can provide the available tools to organize and deliver content into well-crafted teaching systems.
WBT has certain distinguishing characteristics, which makes it an ideal tool for bibliographic instruction. WBT is an interactive tool. It expands upon computer-based training with activities like discussion forums, mail lists and chat sessions. The user sets his/her pace and direction.
The library instruction literature points to active learning as the best means to deliver bibliographic instruction. 3 The interactivity component of WBT is a two-way interaction. The user inputs to the system and the system provides feedback that either reinforces the user or provides guidance for learning. This can be measured by having the user select the correct response and guide the user to continue if an incorrect response is made. WBT can also meet the needs of diverse populations and learning styles. The multi-media used with online training modules can help all types of learning styles - visual, verbal and aural.
Web-based training is not a collection of information pages. Presently, an examination of many library instruction pages shows that libraries have rushed to embrace this new technology and they are using it effectively. Some librarians have created multiple pages full of text with complete verbal explanations of the objectives of the desired library skills. These sites are uninteresting and do not involve the user. Creating a table of contents with links to full page documents is a misuse of the technology and not an interactive learning tool.
An interactive component such as a quiz or feedback form tests the user's knowledge and allows the librarian to determine the effectiveness of the learning. Without this component, the librarian cannot monitor the learning, and thus cannot adapt the module to best suit the learner. Assessing the modules and reviewing them to alleviate confusion and frustration will go a long way to help keep the learner motivated and more willing to work through the objectives and attain information literacy.
The content and objectives of the web-based tutorials examined for this study varies greatly. Some tutorials take the user through the physical layout of the library with maps and pictures. The next type of tutorial explains basic library skills and can offer a glossary of library terms. Another type of tutorial offers instruction on using the library's OPAC, and leads the user through the steps of locating books, magazines and other library materials. To be effective these tutorials should be learning modules that promote the institutional needs of the library and relate the information library skills the library wises its users to achieve.
The next level of tutorial presents the more conceptual aspects of information literacy. These are tutorials on the research process and evaluation of sources. Some of these tutorials review various library networked databases while explaining Boolean and keyword and subject searching techniques. Searching the Internet is a focus of many tutorials, along with interactive modules that offer guidelines on how to evaluate Internet sources. Each tutorial is different in design, content and instruction level, reflecting the personality and nature of each library.
Web Design and Evaluation Criteria by Library Professionals
For librarians developing online instructional tutorials to supplement their bibliographic instruction programs, the web guidelines developed by the Library Instruction Round Table (LIRT) 4, of the American Library Association should be considered. LIRT advocates library instruction as a means for developing competent library and information use as a part of lifelong learning. The LIRT homepage has links to the Top Ten Instruction Articles for the past three years, and links to Library Instruction Tutorials. In July, 1997 the LIRT Research Committee posted a page on Website Evaluation Criteria. At that time the three basic elements recommended by LIRT to include in a web page were access, design and content. This page has evolved and now is called Web Standards and Guidelines. The section on access is no longer an element since web browsers and servers have become quicker and more stable. The content and design elements are more brief and list technical points. The importance is on elements to avoid rather than offering standards on what makes up a good web page. The new addition to the page is "Procedural/Technical." This new element refers to HTML coding and resolution requirements and settings. The Web Standards and Guides are useful from a technical standpoint and instruction librarians should review them.
The Instruction Section Teachings Methods Committee,5 a committee of the American
Association of College and Research Libraries (ACRL) has also prepared web criteria for instruction. This committee is charged with providing a forum for librarians interested in both the theoretical and practical aspects of teaching methods. These guidelines are intended to help libraries' efforts in web-based instruction and are more pedagogical in nature and reflective of the committee's mission to have librarians be successful in their instructional endeavors. Since online instruction is a new format for many, the guidelines are thorough, and the design standards are in keeping with what constitutes a successful web-based training program. These guidelines, updated three times over the last two years, are keeping up with advances in the technology.
The original page included Access, Design and Layout, Content and Audience. Updates have been made and the standard on content now states that pages should be tight and concise - they convey 'just the facts' without excess verbiage. Many library tutorials are heavy on text. Web-based material should not read like a book. The presentation and delivery of online information have their own unique characteristics. Research indicates that the page will either capture the attention of the user or it will not.
The evaluation criteria for design is now divided into two parts. General Design and Layout covers some basic design elements to web page development. Large pages should be broken down into discrete pages. Graphics should help to convey information and not just be decorative; each page should have a clear title and includes headers and footers.
The newest standard, Design and Layout of Interactive Tutorial Material, incorporates interactivity into the tutorials. Exercises, which provide feedback and test whether the student has grasped the concepts, are needed. This supports the learning theory behind bibliographic instruction that the modules need to engage and promote active learning. Navigational mechanisms which allow learners to repeat or move around within the tutorial have proven to be more effective ways to present information so it is not only accessible, but it is retained.
Last but not least, the criteria considers the audience as a necessary element. The content and links should be suitable to the user. The site should address a particular facet of library instruction. This would mean that the page should clearly state the objectives and goals of the instruction unit being presented. It is important to note that the IS Teaching Methods Committee referred to the LIRT Web Standards along with the guidelines and evaluation criteria used by Argus Clearinghouse6, Cyberstacks7and the Reference Collection Development: A Manual 8 in selecting and revising their criteria.
Another ACRL committee, the Instruction Section Emerging Technology Committee 9 has acted more as an information sharing vehicle and facilitator for librarians interested in sharing instructional materials that they have designed in a networked environment. One of the committee's projects is the ACRL/CNI Internet Education Project10, which has developed selection criteria used to review and evaluate instructional materials libraries wish to post on the web. The ten criteria statements require that the information is accurate, the material, content and language is clear, the material provides instruction using technology, the material makes effective use of good web design (graphics, examples, interactive elements) and the material enhances the learning experience. The committee has also created a Training Resources Bookmark11 with links to numerous sites on web education, library instruction research and web design.
Web Design and Evaluation Criteria in Non-Library Sources
Business organizations delivered instructional models via the web long before the library community. Large corporations, which have spent millions on training, have found more cost-effective and successful ways of teaching employees using this platform. Their experience and research in this arena is valuable to librarians planning web-based tutorials. There are many sites that cover web design for training, but there are two worth mentioning for this study.
Walt Howe has posted a series of articles called Web Page Quality12 including Evaluating Quality, Producing Quality Web Page Content, and Creating Web Page Design which have been covered in the library guidelines. Test Your Pages is an important component that does not appear in any of the library guidelines up to now.
Another site useful for this study is the WBT Information Center13. The section Primer has rules for Good Web Design and The Development Process guides one through the essential steps for developing web-based training from user analysis, interface design, usability testing, instructional design, scripting and coding. The Center also has links to Web Style Guides, Discussion Groups and Articles and Position Papers on web-based training. This site is attractive and is one stop shopping for those interested in creating web instruction guidelines and materials.
Assessment and Usability Studies
Evaluating instructional programs is essential. This appears to be the step lacking from the library guidelines and is absent in 90% of the library tutorials examined. There is no one way to evaluate student learning, and librarians have struggled with this issue long before web-based instruction.
Assessment can take many forms. Librarians could do a pretest and a posttest. It could be an evaluation form distributed after the class or it could simply, be a question and answer session. These formats cannot be included in many web-based training programs due to constraints of the software.
Usability studies are one way to make formal evaluations. Time, money and staffing are required. At the recent Association of College and Research Libraries Annual Conference in Detroit, Michigan in the presentation, "The User is the Expert, " 14 three librarians reported on their experience using usability studies to inform gateway and tutorial web designs. The design should be user-centered and the methods to achieve can include the more costly but accurate testing methods of formal usability testing, focus groups, group testing and survey. Less costly and less accurate means are design walk-throughs, heuristic evaluation, card sorting, matching test or icon intuitiveness evaluation and field tests. Some of the results indicate that students try to assign meaning to colors, navigation through frames is tricky, and links outside the tutorials are confusing.
The IFLA Study
The IFLA Study and presentation includes a detailed examination of these criteria while highlighting worldwide library websites that demonstrate effective use of these design features. The presentation will also review the library use of new web-based courseware products now on the market such as WebCT.
The Web is not a static environment and so any guidelines drawn up for web-based instruction have to change with the technology. The fact that these guidelines are being updated is a healthy sign that librarians are sharing the successes and the mistakes they have encountered when developing a web-based library. This is one of the best forms of evaluation. This is a sound progression and it is important that the focus of library instruction remains developing life-long learners and technology is the means to achieve this end.
Notes and References
1. Stover, M. & Zink, S. World Wide Web Home Page Design Patterns and Anomalies in Higher Education Library Home Pages. RSR: Reference Services Review 24 (Fal1996). p. 8
2. Web Information Center. http://www.filename.com/wbt/pages/primer.htm
3. Designs for Active Learning: A sourcebook for Classroom Strategies for Information Education. Chicago: ALA. http://www.bk.psu.edu/academic/library/istm/Designs.html
4. American Library Association. Library Instruction Round Table. LIRT Library Instruction Tutorials: General Guides to Research. http://diogenes.baylor.edu/Library/LIRT/lirtcrit.html
5. American Library Association. ACRL. Teaching Methods Committee. http://www.bk.psu.edu/academic/library/istm/index.html
6. Argus Clearinghouse Rating System. http://www.clearinghouse.net/ratings.html
7. Cyberstacks. http://www.public.iastate.edu/~CYBERSTACKS/signif.htm
8. American Library Association. RUSA. Reference Collection Development: A Manual. RASD Occasional Papers, Number 13 (1992.)
9. American Library Association. ACRL. Emerging Technologies Home Page. http://www.rci.rutgers.edu/~scholzcr/etech/
10. ACRL/CNI Internet Education Project. http://www.cwru.edu/affil/cni/base/goals.html
11. Training Resources Bookmark. http://www.rci.rutgers.edu/~scholzcr/etech/training.html
12. Web Quality Pages. http://www.delphi.com/navnet/quality.html
13. Web Information Center. http://www.filename.com/wbt/pages/primer.htm
14. The User is the Expert. Presentation ACRL Annual Conference, Detroit, MI, April 1999. http://www.tc.umn.edu/~jveldof/ACRL99/iserdesin.html
Addendum: Standards and Evaluation Criteria Outside the United States
The following are some projects outside the United States that are involved with teaching and learning in a networked environment:
- The Higher Education community of the United Kingdom has an initiative called the The Electronic Libraries Programmes (eLib). (http://www.ukoln.ac.uk/services/elib/) They have posted standard guidelines and recommendations for the selection and use of standards in eLib projects. (http://www.ukoln.ac.uk/services/elib/papers/other/standards/)
- The Tavistock Institute, which undertakes evaluation studies, produced the document, Guidelines for eLib Project Evaluation. Prepared by John Kelleher, Elizabeth Sommerlad and Elliot Stern this document provides useful guidelines for developing and implementing evaluation in individual eLib projects. (http://www.ukoln.ac.uk/services/elib/papers/tavistock/evaluation-guide/intro.html#defining.)
- NetLinkS, a U.K. Training and Awareness project supported by the Joint Information Systems Committee of the Higher Education Funding Council of England. Initially funded for one year, from September 1995, the project received further funding until May 1998. The NetLinkS Research Report is a detailed study of network learning. A link to other eLib projects throughout the United Kingdom is available from their home page. (http://netways.shef.ac.uk/about/overview.htm)
- Academic partnership in NLS resource design: a European case study by Jo Pye is a case study of European partner institutions concerned with electronic learning. This is a user needs analysis report addressing networked learner support in European partner institutions and development of an online course for librarians. This article was presented at the 2nd International Symposium on Networked Learner Support held in Sheffield, England in June 1997. (http://netways.shef.ac.uk/rbase/papers/pye.htm)
- DEDICATE is a good site with information on the development of distance education in Eastern Europe. The aim of the DEDICATE project is to develop cost-effective distance education courses in information literacy. The universities involved in this project are four library sites in Technological Universities in Estonia, Hungary, Latvia and Lithuania and at the International Center for Information Management, Systems, and Services, in Torun, Poland. (http://educate.lib.chalmers.se/DEDICATE/dedindex.html)
- The Multimedia Education Unit at the University of Melbourne provides assistance to major educational projects aimed at transforming units, subjects or whole courses with the use of multimedia and educational technology. They are developing some distance education course. (http://www.meu.unimelb.edu.au/)
- A web-based library education project worth looking at in Finland is one created by Irma Pasanen-Toumainen at Helsinki University of Technology. (http://www.hut.fi/Yksikot/Kirjasto/Palvelut/Koulutus/Informatiikka/)