15th Annual New Reference Research Forum
The Research Forum is one of the most popular programs at ALA Annual, where attendees can learn about notable research projects in reference service areas such as user behavior, electronic services, and reference effectiveness. This year’s Forum features three presentations: Building a Model of Excellent Reference Service Based on WOREP (Wisconsin-Ohio Reference Evaluation Program) Data, the recipient of RUSA’s 15th Anniversary Reference Research Grant; “Teachable Instants” in Instant Message Reference: Taking the Opportunity or Taking a Pass?; and Measuring the Effectiveness of Online Tutorials: A Pragmatic Approach.
Speakers: Julie Gedeon, Kent State University; Carolyn Radcliff, Kent State University; Megan Oakleaf, School of Information Studies, Syracuse University; Amy VanScoy, North Carolina State University Libraries; Cindy Craig, Wichita State University Libraries; Curt Friehs, Kansas Public Library
What WOREP Results Say About Reference Service, Patron Success and Satisfaction
WOREP = Wisconsin-Ohio Reference Evaluation Program
This is a survey in which patron and staff both fill out survey after the transaction. They had a huge amount of data. There were over 100 participating libraries and years and years of participation (1984-2008).
They gave an overview of what methods they used. This part had a lot of statistical jargon that I didn’t know.
They had mostly positive results. Patrons felt they had personal attention and librarians were professional. Their final thoughts: play to your strengths – librarians ranked highly in personal attention and professionalism. Also, continue to do the things that are highly correlated to success: offering enough help, enough time, giving clear explanations, and “librarian appears knowledgeable.” They offered that they didn’t know what that last one meant, maybe everyone should wear glasses. I imagine it has more to do with confidence and skill.
Measuring the Effectiveness of Online Tutorials: A Pragmatic Approach
They started with some assumptions and refutations:
- “Interactivity is the online hallmark of active learning.” ~ Nancy Dewald
- Maybe not so if interactivity turns into interruption.
- Technology is naturally intuitive to college students and young adults.
- Not really true. Students claimed high satisfaction with the tutorials, but weren’t able to complete the tasks.
- Tutorials are too hard to make and “I’m too old.”
- Then you’re down before you start
- Why can’t we just use the vendor tutorials?
- You ask what time it is, the vendor is going to teach you how to build a watch. You want something to get to the point quickly. They don’t need to know every feature.
- “I’ll just put my handouts online. It’s the same thing as an online tutorial.”
- No, it’s not.
- “Online tutorials are going to make my job obsolete.”
- Students learn more from video tutorials than from html tutorials
- Interactivity is not necessary for learning, may even hinder
- surveyed 140 finance students after watching Value Line Online tutorial
- tutorial left students wanting to know even more
- more research was needed
The second part of the research was with lower level biology students who came in to the library for instruction with their class – a captive audience.
They looked at some principles in terms of html vs. streaming media tutorials. The principles were modality effect, dual coding theory, and constructivism. For those first two html does poorly since it is visual only while media has video and audio. For the third principle, html looks good because navigation allows for choice while video is linear with (minimal interactivity).
They made 2 tutorials – video and html. The information was the same, only the format was different. After watching the tutorial, students got a survey with an opportunity for feedback and a quiz. They found that confidence increased more with students who watched the video, but more importantly people who watched the video scored way better on the quiz questions. They hope to expand on it more in an upcoming article. They feel there need to be more studies on the effectiveness of different types of tutorials and that libraries should be creating more animation plus narration brief tutorials.
“Teachable Instants”: Taking the Opportunity or Taking a Pass
Megan Oakleaf and Amy Van Scoy
This was a particularly fantastic presentation with wonderful ideas and examples of things we should all be trying.
They took an academic and a teaching and learning focus. They looked at a few models of educational theory in particular.
- constructivism & active learning
- social constructivism
Then they tried to create catchier titles that would be easier to apply:
- catch them being good – reinforcing positive behaviors
- think aloud – describe cognitive process, they can learn from our coping tstrategies perhaps even more than when it works out perfectly
- show, don’t tell
- chunk it up
- let them drive – examples “what do you think of those results”
- be the welcome wagon
- make introductions – to other staff, to come in in person
- share secret knowledge
Their methodology included coding 1 year of virtual reference transcripts. They looked at how often the above strategies were used.
62% had at least 1 of the strategies. Show don’t tell had the highest at 43%, but only 15% if they omit page pushes. The lowest was 2% – catch them being good. Also very low at 3% was “chunk it up.” They felt chunk it up was particularly important – showing that this is a process with steps.
Another interesting discovery was that they found a lot of students asking “how?” Students really wanted to be taught.
They had a handout with examples of chat transcripts. The last page was an example of a librarian who took a pass. It was clearly a librarian who wanted to be helpful, but missed an opportunity for instruction.
- reference transactions are instructional opportunities
- many librarians take a pass on the opportunity to teach
meganoakleaf.info/teachwithtech.pdf – article
Their handouts will be on the research and statistics website.
There was a strong question and answer section at the end of the presentations. One audience member asked how long people took with the video vs. the html tutorials. He suggested that might have impact. He also suggested they use a task based exam at end rather than a knowledge quiz. I agree with this questioner. The html group might have breezed over it and not bothered, as opposed to when a student chooses to go to the tutorial because they want that information. The captive audience made it an artificial study IMO, since the students don’t have a self directed reason to be going through the tutorial.
An audience member suggested that librarians establish a rapport with students by talking their language.
An audience member said that his library has a student advisory board. They wanted all video tutorials. And just for one little skill. Ex: Find the x date issue of x magazine.
There was a suggestion that 3-5 minutes is appropriate for video length.
There was also a discussion about when to suggest the student come in to the library. You don’t know why they’re on chat. One person had whooping cough. They might be a distance education student, etc.