formative evaluation for educational product development
DESCRIPTION
Presented to the Badges for Lifelong Learning grantees of the DML Competition, February 18, 2013.TRANSCRIPT
![Page 1: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/1.jpg)
Formative Evaluation for Educational Product
Development*Or, how to tell if this thing you made is worth its salt
Vanessa Gennarelli Learning Lead, P2PUMaster’s Candidate, Harvard University@mozzadrella
![Page 2: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/2.jpg)
Why Should I Test My Product?Can folks use it?
Evaluate usability and nomenclature.
Do users like it?
Measure appeal and engagement.
Does this audience get it?
Test comprehension and recall.
![Page 3: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/3.jpg)
What is Formative Evaluation?
Testing an educational product with users during development.
The results inform the direction of the product.
Process can be conducted at any time.
Speedy stuff--time is of the essence.
![Page 4: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/4.jpg)
Who Uses Formative Evaluation?
![Page 5: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/5.jpg)
Who Uses Formative Evaluation?
![Page 6: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/6.jpg)
Who Uses Formative Evaluation?
![Page 7: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/7.jpg)
That’s peachy, Vanessa.
![Page 8: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/8.jpg)
That’s peachy, Vanessa.Can I do this myself?
Do I have time for formative evaluation?Is there an easy way to get started?
![Page 9: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/9.jpg)
The answer is “yes”...And you will crush it.
![Page 10: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/10.jpg)
Methods: InterviewsA structured protocol of questions with a target user.
Benefits:
Learn the vocabulary and logic of your users.
Deep investigation of appeal, engagement and comprehension.
Limitations:
Recruitment, research and analysis is time-intensive.
![Page 11: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/11.jpg)
Methods: InterviewsSample questions:
Appeal:
What badge would you want to share with a friend?
Which would you apply for? Not apply for?
Comprehension:
[Stop activity]: What just happened?
What does this badge mean?
![Page 12: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/12.jpg)
Methods: Think Aloud ProtocolAs a user walks through your product they narrate what they notice, think, feel, look at, and decide.
Benefits:
Observe hesitations, hiccups and trouble spots.
Observe positive and negative reactions.
Limitations:
Interviewer needs to stay quiet, can be awkward.
![Page 13: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/13.jpg)
Methods: Think Aloud ProtocolSample Questions:
You have completed a project and are applying for a badge. Apply for a badge, describing what you notice and think at each step of the way.
Select a badge you’d like to apply for. Describe and explain each action you make.
![Page 14: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/14.jpg)
Example: P2PU Course UXUsertesting.com: asked folks to create a course3 respondentsVideo recording of users--where do they get stuck?
![Page 15: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/15.jpg)
Example: P2PU Course UX
Results informed our new Course UXFull report: http://www.slideshare.net/VanessaGennarelli/design-research-course-creation
![Page 16: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/16.jpg)
Methods: Focus GroupsA structured protocol with 2+ more targeted users.
Benefits:
Feedback from several users at once.
Limitations:
In a group setting, bias can be an issue.
Recruiting several users can be difficult.
![Page 17: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/17.jpg)
Methods: Focus GroupsSample questions:
Appeal: What did you like most? Least?
Engagement: What would you like to know more about?
Comprehension: What do you understand the difference between [Badge X] and [Badge Y] to be?
![Page 18: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/18.jpg)
Methods: QuestionnairesSeries of questions that may have standardized answers.
Benefits:
Lightweight in regards to collection and analysis.
Limitations:
Tricky to ask precise and reliable questions.
No observational data, questionnaires are all self-reported data.
![Page 19: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/19.jpg)
Methods: Questionnaires
Sample QuestionsAppeal:
Rate how much you like/dislike [Badge X] on a 1-5 scale (1=least, 5=most)
Comprehension: Rate the clarity of [Badge X] on a 1-5 scale (1=least clear, 5=most clear)What are 3 things you learned about from [X]?
![Page 20: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/20.jpg)
Methods: Click-testingVeryifyapp asks users to click on their preferred design, annotate the screen, or test the labels on your pageBenefits:
Great for expectation questions “Where would you expect to find...”Testing navigation & nomenclatureInexpensive & fast
Limitations:Adult testers only
![Page 21: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/21.jpg)
Example: P2PU Home Page
Users were asked to highlight where they could click to create a course538 viewers, 62 responses
![Page 22: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/22.jpg)
Example: P2PU Home Page
Results informed our new P2PU.org home pageFull report: http://www.slideshare.net/VanessaGennarelli/user-research-p2puorg-home-page
![Page 23: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/23.jpg)
1
Sample Project ScheduleIdentify research areas--what are you evaluating?
Decide upon tools and methods.
Recruit participants from target user base.
NB: You may need a release.
Conduct research.
Analyze results & present to project team.
Target timeframe: 1 week.
![Page 24: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/24.jpg)
Mad Props1Karen Brennan, Assistant Professor, Harvard Graduate School
of Education
Mindy Brooks, Director of Education and Research, Sesame Workshop
Brad Felix, Chief Products Officer, Sangari Global Education
Allison Gevarter, Interactive Producer, CloudKid Media
Ilona E. Holland, Lecturer, Harvard Graduate School of Education
![Page 25: Formative Evaluation for Educational Product Development](https://reader034.vdocument.in/reader034/viewer/2022051209/547d565cb4af9fce158b534c/html5/thumbnails/25.jpg)
Ask things!
Vanessa Gennarelli Learning Lead, P2PUMaster’s Candidate, Harvard University@mozzadrella