“Strategies for
On-Line
Assessment”Presented by:
JulieAnne Adamich Ph. D., CPA
St. Petersburg College
E-mail address: [email protected]
Florida Association of Accounting
Educators
Fall 2004
Define the purpose of the Assessment Process
• It is a Formative evaluation process
• based on the Continuous collection and analysis of data
• with the primary purpose of emphasizing Improvement of:- student learning- student development- the program within the institution
Assessment accomplishes two important goals
“Improvements in learning will depend on how well assessment, curriculum, and instruction are aligned and reinforce a common set of learning goals” Pellegrino, et al (2001,p. 25)
It also is a way of ensuring that students have been exposed to and have “acquired” the major learning outcomes …. essential for successful job placement.
Assessment is an integral part of the Planning and Evaluation
Process
Program/Course Assessment
Program ReviewsStrategic Planning
(Budgeting)
Formative Goals: Continuous Improvement
Summative Goals: Accountability, Performance Evaluation
Both Formative and Summative Goals:
Focus: Allocating Resources
Another reason- “Accreditation” Requirements for
SACS Reaffirmation• “The institution engages in ongoing, integrated, and
institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission.” (Core Requirement 5)
• “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” (Comprehensive Standard 16)
Goals for Developing On-Line Assessments
• Select questions that assess learning objectives taught in a course/program.- Textbook End of Chapter assignments and test banks are typically categorized by learning objective.
• Select questions that match the taxonomy level of the student’s learning development.
• Utilize technology incorporated into Learning Management Systems (such as WebCT) which include quizzing modules to minimize instructor intervention during the assessment process to permit time for instructor focus on the analysis of detailed reports available to the instructor and the provision of student feedback.
• Incorporate quiz features/settings that discourage or prevent cheating.
On-Line Assessment Resources
Quizzing Assessments using WebCT (which is similar to other Learning Management Systems such as Blackboard, eCollege, ANGEL etc.)
Pre-formatted matching and multiple-choice
quizzes are provided to assess each chapter in
Thomson Learning’s Course E-Packs.
Quizzes can also be created directly in WebCT (or imported from a Test Bank, Respondus etc.)
For matching questions, select
the correct answer by clicking on the drop down box.
Answers are saved by clicking on “Save Answer”
Students know the status of
each question
at all times.
For multiple choice questions, students
click on the button to the left of the correct
answer.
On-Line Quizzing Security Features
• Develop question sets and randomize answers to generate a “unique” quiz for each student.
• Deliver questions one at a time to discourage students from printing the entire quiz.
• Limit the quiz duration to discourage students from using/sharing resources.
• Passwords, quiz availability, selective release settings limit the availability of the quiz.
There are a variety of question delivery options to promote a
secured environment.
Question sets can also be developed in
WebCT. For example, the system randomly
selects either MC#1 OR MC#2. Each exam is therefore unique to
discourage cheating.
There are also settings for each question, where the answer order
can be randomized.
Students verify that all
questions have been
answered …
… and upon completion of the quiz click “Finish”
Webct provides the opportunity for students to
receive immediate feedback.
There are a variety of options for
student feedback.
Scores can be released at various stages
Question feedback (such as
the correct answer, the total
score, and grader’s
comments) are available options.
Instructors can also view statistical information
regarding the performance of the entire class on each quiz.
Test Item AnalysisReporting Features available in WebCT
* Item Difficulty Index
* Item Discrimination Index
• Group Performance Indices:
- Standard Deviation
- Mean
Item Difficulty IndexUpper 25%: the percentage of students in the upper 25% of
quiz scorers who answered this question correctly.
Whole Group: the percentage of students
who answered the question correctly.
Lower 25%: the percentage of students in
the lower 25% of quiz scores who answered this
question correctly.
Item Discrimination Index
Discrimination: an indication as to how well the question relates
between high versus low scorers. Higher values indicate better
discrimination. A negative value indicates that the lower-scoring portion of the class performed better on this question than the
high-scorers.
Good Discrimination
Low Discrimination
Group Performance
N: the total number of all
student responses for the
question.
Mean: the average score of all students
for the question.
SD: the standard deviation of the results
from the average score.
Analyze assessment results and prepare an
assessment report
Analyze each major learning outcome, based on the statistical results. Include the following:
• Criteria for Success, i.e., expected statistical results
• Summary of Assessment Findings, i.e., actual statistical results
• Use of Results, i.e., analyze results to determine if planned program improvements are deemed necessary
• Action Plan and Timetable for Implementation• Budgetary and Planning Implications
AICPA Educational Competency Assessment (ECA) Web Site
The Educational Competency Assessment (ECA) Web site is an online resource for educators. The software is designed to guide accounting educators through the process of assessing a program or course.The purpose of the site is to help educators develop or reform curricula and courses to support the development of a set of competencies, consistent with the findings of the CPA Vision. Register free at:
http://eca.aicpaservices.org
Organizers(A series of templates to develop a database for your
institution with the capability to sort by selected course and/or program)
• Evaluate Competency Performance (ECC) Organizer:
- Documents your comparison of the actual level to your desired level of coverage of competencies to determine the degree to which the AICPA Core Competencies are covered in your courses and programs.
- Organized by Competency Categories, evaluate your perceived degree of:* Actual Competency Coverage* Desired Competency Coverage* Desired Learning Outcome* Overall Conclusion
• Assess Student Performance (ASP) Organizer:
- Documents the process of gathering, analyzing and using assessment information to determine how well students are developing the AICPA Core Competencies.
- Areas of input include:* Describing the Assessment Activity* Documenting Findings* Drawing Conclusions* Recommending Improvements
Quizzing Assessments Major Pros
• Explicit demonstration of knowledge
• Efficiently examines a wide range of content knowledge
• Can use well-established practices to evaluate reliability and validity
• Inexpensive to administer and evaluate for a large number of students
• If embedded in courses, allow for more representative or complete student samples
• If test is used multiple times, can obtain longitudinal/trend or pre/post data
Source:
AICPA Educational Competency Assessment (ECA) Web Site
Register free at:http://eca.aicpaservices.org
Major Cons – Mitigated using On-line Assessments
• Labor-intensive to develop valid and reliable questions – Test Bank – Test Bank questions can be imported into WebCT.questions can be imported into WebCT.
• May be unable to generalize performance to other tasks/ Less “authentic” because they do not simulate conditions of work – – Personal Trainer “simulates” detailed exercises and problems. Personal Trainer “simulates” detailed exercises and problems.
• Student participation rates and motivation levels may be low for non-graded examinations. Limited feedback to students if they are not allowed access to results for individual questions – Quiz settings – Quiz settings provide options of hints, solution, and IMMEDIATE feedback upon provide options of hints, solution, and IMMEDIATE feedback upon completion of the quiz. completion of the quiz.
• Standardized tests may encourage instructors to ‘teach to the test’ and direct student focus away from competencies – Question sets – Question sets may be developed to randomize questions based on criteria (such may be developed to randomize questions based on criteria (such as learning outcome, competency level, etc.).as learning outcome, competency level, etc.).
Resources for Faculty
* BEEP – Best Educational E-Practiceshttp://www.spcollege.edu/eagle/research/beep/beep38.htmArchive includes information associated with the following
resource categories:- Assessment and Evaluation- Innovative Technologies- Instructional Resources
• Merlot. Well-known and growing collection of peer-reviewed online learning materials and more. Officially titled Multimedia Educational Resource for Learning and Online Teaching. http://www.merlot.org