ellie <3 libraries

sooooo much!

15th Annual New Reference Research Forum July 12, 2009

Filed under: ALA 2009,Conferences — ellie @ 3:00 pm

15th Annual New Reference Research Forum
The Research Forum is one of the most popular programs at ALA Annual, where attendees can learn about notable research projects in reference service areas such as user behavior, electronic services, and reference effectiveness. This year’s Forum features three presentations: Building a Model of Excellent Reference Service Based on WOREP (Wisconsin-Ohio Reference Evaluation Program) Data, the recipient of RUSA’s 15th Anniversary Reference Research Grant; “Teachable Instants” in Instant Message Reference: Taking the Opportunity or Taking a Pass?; and Measuring the Effectiveness of Online Tutorials: A Pragmatic Approach.
Speakers: Julie Gedeon, Kent State University; Carolyn Radcliff, Kent State University; Megan Oakleaf, School of Information Studies, Syracuse University; Amy VanScoy, North Carolina State University Libraries; Cindy Craig, Wichita State University Libraries; Curt Friehs, Kansas Public Library

What WOREP Results Say About Reference Service, Patron Success and Satisfaction

WOREP = Wisconsin-Ohio Reference Evaluation Program

This is a survey in which patron and staff both fill out survey after the transaction. They had a huge amount of data. There were over 100 participating libraries and years and years of participation (1984-2008).

They gave an overview of what methods they used. This part had a lot of statistical jargon that I didn’t know.

They had mostly positive results. Patrons felt they had personal attention and librarians were professional. Their final thoughts: play to your strengths – librarians ranked highly in personal attention and professionalism. Also, continue to do the things that are highly correlated to success: offering enough help, enough time, giving clear explanations, and “librarian appears knowledgeable.” They offered that they didn’t know what that last one meant, maybe everyone should wear glasses. I imagine it has more to do with confidence and skill.

Measuring the Effectiveness of Online Tutorials: A Pragmatic Approach

They started with some assumptions and refutations:

  • “Interactivity is the online hallmark of active learning.” ~ Nancy Dewald
    • Maybe not so if interactivity turns into interruption.
  • Technology is naturally intuitive to college students and young adults.
    • Not really true. Students claimed high satisfaction with the tutorials, but weren’t able to complete the tasks.
  • Tutorials are too hard to make and “I’m too old.”
    • Then you’re down before you start
  • Why can’t we just use the vendor tutorials?
    • You ask what time it is, the vendor is going to teach you how to build a watch. You want something to get to the point quickly. They don’t need to know every feature.
  • “I’ll just put my handouts online. It’s the same thing as an online tutorial.”
    • No, it’s not.
  • “Online tutorials are going to make my job obsolete.”

Some conclusions:

  • Students learn more from video tutorials than from html tutorials
  • Interactivity is not necessary for learning, may even hinder

The Research:

  • surveyed 140 finance students after watching Value Line Online tutorial
  • tutorial left students wanting to know even more
  • more research was needed

The second part of the research was with lower level biology students who came in to the library for instruction with their class – a captive audience.

They looked at some principles in terms of html vs. streaming media tutorials. The principles were modality effect, dual coding theory, and constructivism. For those first two html does poorly since it is visual only while media has video and audio. For the third principle, html looks good because navigation allows for choice while video is linear with (minimal interactivity).

They made 2 tutorials – video and html. The information was the same, only the format was different. After watching the tutorial, students got a survey with an opportunity for feedback and a quiz. They found that confidence increased more with students who watched the video, but more importantly people who watched the video scored way better on the quiz questions. They hope to expand on it more in an upcoming article. They feel there need to be more studies on the effectiveness of different types of tutorials and that libraries should be creating more animation plus narration brief tutorials.

“Teachable Instants”: Taking the Opportunity or Taking a Pass

Megan Oakleaf and Amy Van Scoy

This was a particularly fantastic presentation with wonderful ideas and examples of things we should all be trying.

They took an academic and a teaching and learning focus. They looked at a few models of educational theory in particular.

  • metacognition
  • constructivism & active learning
  • social constructivism

Then they tried to create catchier titles that would be easier to apply:

(great slides with examples)

  • catch them being good – reinforcing positive behaviors
  • think aloud – describe cognitive process, they can learn from our coping tstrategies perhaps even more than when it works out perfectly
  • show, don’t tell
  • chunk it up
  • let them drive – examples “what do you think of those results”
  • be the welcome wagon
  • make introductions – to other staff, to come in in person
  • share secret knowledge

Their methodology included coding 1 year of virtual reference transcripts. They looked at how often the above strategies were used.

62% had at least 1 of the strategies. Show don’t tell had the highest at 43%, but only 15% if they omit page pushes. The lowest was 2% – catch them being good. Also very low at 3% was “chunk it up.” They felt chunk it up was particularly important – showing that this is a process with steps.

Another interesting discovery was that they found a lot of students asking “how?” Students really wanted to be taught.

They had a handout with examples of chat transcripts. The last page was an example of a librarian who took a pass. It was clearly a librarian who wanted to be helpful, but missed an opportunity for instruction.


  • reference transactions are instructional opportunities
  • many librarians take a pass on the opportunity to teach

meganoakleaf.info/teachwithtech.pdf – article

Their handouts will be on the research and statistics website.

There was a strong question and answer section at the end of the presentations. One audience member asked how long people took with the video vs. the html tutorials. He suggested that might have impact. He also suggested they use a task based exam at end rather than a knowledge quiz. I agree with this questioner. The html group might have breezed over it and not bothered, as opposed to when a student chooses to go to the tutorial because they want that information. The captive audience made it an artificial study IMO, since the students don’t have a self directed reason to be going through the tutorial.

An audience member suggested that librarians establish a rapport with students by talking their language.

An audience member said that his library has a student advisory board. They wanted all video tutorials. And just for one little skill. Ex: Find the x date issue of x magazine.

There was a suggestion that 3-5 minutes is appropriate for video length.

There was also a discussion about when to suggest the student come in to the library. You don’t know why they’re on chat. One person had whooping cough. They might be a distance education student, etc.


3 Responses to “15th Annual New Reference Research Forum”

  1. Carlyn Gray Says:

    Thanks, Ellie, great summary of this ALA Wkshp, and I like your opinions and comments. Sounds like a great session.

  2. Cindy Craig Says:

    Hi Ellie! Great write-up! Thanks for attending the session. I wanted to clarify this comment:

    “The captive audience made it an artificial study IMO, since the students don’t have a self directed reason to be going through the tutorial.”

    Actually, the students did have a library assignment which was graded by the biology GAs. One question on the assignment required them to successfully use the BIOSIS Previews database. So, students (perhaps) would have been motivated to pay attention to the tutorial. The assignment was different than the short quiz I gave them as part of the study (which was not graded). I don’t think I made that clear during our presentation.

    Thanks for your comments!

    Cindy Craig

  3. ellie Says:

    Thanks for the clarification!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s