technology: access to the future regional meetings
DESCRIPTION
Technology: Access to the Future Regional Meetings. Ellen B. Mandinach Naomi Hupert EDC Center for Children and Technology WWW.edc.org/CCT November 21, 2003 January 23, 2004 February 20, 2004 March 19, 2004. - PowerPoint PPT PresentationTRANSCRIPT
Technology: Access to the Future
Regional Meetings
Ellen B. Mandinach
Naomi Hupert
EDC Center for Children and Technology
WWW.edc.org/CCT
November 21, 2003
January 23, 2004
February 20, 2004
March 19, 2004
“The problem we face in inserting new technologies into education is that they partly represent new ways of teaching
and partly represent new content that ought to be taught now.”
Lesgold (2000)
“A paradox gradually became evident: The more a technology, and its usages,
fits the prevailing educational philosophy and its pedagogical
application, the more it is welcome and embraced, but the less of an effect it has. When some technology can be smoothly assimilated into existing
educational practices without challenging them, its chances of
stimulating a worthwhile change are very small.”
Salomon and Almog, 1998
According to Papert (1987), if an instructional technology is
harmless that it is easily integrated into existing
pedagogical practices without many changes, then it will be equally harmless in making an
instructional difference.
Question:
What is the difference between elephants mating and the implementation of educational technology?
Answer:
There is a lot of dust and noise, and nothing happens for a very long period of time!
Think of a pointillist painting by Seurat or an impressionist
work by Monet
Step up to the painting.
Step all the way back.
Compare what you see.
That is precisely what evaluation and assessment require - taking multiple perspectives of the same phenomena and getting different feedback.
Question:
How can the accountability issue be addressed?
Answer:With great difficulty
Use a number of measurement strategies, asking different, but related questions, all concerned with various aspects of the learning process and outcomes.
Mandinach & Cline (1994)
Assessment, Evaluation, and Research
Different purposes, questions, and objectives:
Assessment - the measurement of learner performance. Can be part of an evaluation, but not synonymous.
Evaluation - the examination of a system, product, or program - formative, summative, or both - to determine how it is functioning, being implemented, and how it can be improved.
Research - encompasses assessment, evaluation, and much more.
Evaluation and Research:Some Tradeoffs
Is likely to be most effective when it is both formative and summative.
Should be used for planning as well as a systematic research tool.
Must consider the information desired by the stakeholders.
Must use measures that will maximize the potential for detecting impact.
Must use standardized tests or targeted assessments.
Must use experimental versus other designs. Must use comparison/control or matched
groups.
Some Caveats
Need for continuous adjustments in: pedagogical philosophy.
assessment techniques.
strategies, roles, priorities, and schedules.
All parties involved need to change their conceptions of proof of successful implementation of technology and its impact on teaching and learning.
What does it mean to say, “it works”?
Is there enough implementation of the technology to enable
measurement?
There must be enough technology implementation to produce the desired outcomes.
If you are only using technology for a small amount of instructional time, there will be limited exposure for each student each day. Therefore the targeted outcomes are not likely to show substantial effects size.
Evaluation/Research Issues Experimental paradigm
– eliminate possible explanations– verify hypothesized causal relationship– control for contaminating influences– random assignment of students to experimental
and control groups– controlled application of the stimuli
Barriers to and issues in the use of an experimental design.
– selection, acquisition, and installation of hardware and software
– training and ongoing support of teachers– enlisting support, encouragement, and
participation of students, teachers, administrators, school board members, and parents
– actual classroom implementation (for as long as it takes)
Quasi-Experiment Versus Formative Experiment
Not sufficiently powerful
Not sufficiently sensitive to changes
Inadequate evaluative question -
Does it work?
“Rarely does one study produce an unequivocal and
durable result: multiple methods, applied over time
and tied to evidentiary standards, are essential to
establishing a base of scientific knowledge.”
Shavelson & Towne, 2002
Methodological Implications for Technology-Based Educational
Reform Efforts
Longitudinal Design
Multiple Methods
Hierarchical Analyses
System Dynamics
Longitudinal Design
Reasons that quasi-experiments are difficult to implement in classroom settings:
– pre and post measures
– experimental and control groups
– random assignments of students
– control of how and when the “stimulus” is administered
Sacrifice the quasi-experimental design in favor of ongoing longitudinal data collection and analyses that are carried out continuously and indefinitely.
Multiple Methods
Think aloud protocols Classroom observations Peer observations In-depth interviews with teachers, students, and
administrators Content analyses of essays Paper and pencil assessments Performance assessments Notebooks and portfolios Performance on traditional tests, assignments, and projects Many data items routinely generated in the
operation of any school (e.g., GPA, tardiness, attendance, drop-out rates, course taking patterns)
Gather data at multiple time points Case studies Bottom line: Identify consistent patterns across
data sources
Educational innovation efforts often generate multiple outcomes that require the triangulation of traditional and nontraditional data collection and assessment methods. For example:
HIERARCHICAL ANALYSES
Student Learning - Processes and Outcomes
Classroom Dynamics - the Changing Patterns of Interactions between Students and Teachers
The School as a Social Organization - its Structure and Functions
Examine impact at different levels
of analysis:
SYSTEM DYNAMICS
A school is an interdependent, multilayered system. One has to understand what is happening across layers and levels.
Reform efforts take place in real world contexts that are composed of many interrelated, dynamic factors.
Any attempt at educational reform will have multiple components and multiple impacts, and they will interact across levels of organization over time.
Asking a single question about impact or outcome is naive.
Thus, we need to approach educational reform systemically.
“The character of education not only affects the research
enterprise, but also necessitates careful consideration of how the understanding or use of results
can be impeded or facilitated by conditions at different levels of
the system. Organizational, structural, and leadership qualities
all influence how the complex education system works in
practice.”
Shavelson & Towne, 2002
Learning Organization
Administration
Students Teachers
Community
Interactive Learner Directed
Technology
Role ChangesTransformation
Accountability
Curriculum
ResourcesSupport
Assessment
Student Motivation
Procedural Traditional Instruction
Systems InstructionNoname 1
Enhancement
ORGANIZATIONAL CHANGE LEVEL
CLASSROOM PROCESS LEVEL
STUDENT LEARNING LEVEL
How do you know if the technology is “working”?
Questions - better, more, what ifs, what and how they ask
Deeper understanding More engaged and on task How they interact Challenged Argumentation Application Independence and self-directedness Planfulness and organizational skills Increased problem solving, logical analysis Reponses in discussion Decision making See the light bulb go on - the aha experience See the obvious Students in charge of their own learning Success is being able to handle failure and learn
from it
Mandinach & Cline (1994)
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Source: Trenton Tim es, September, 2003
WHAT Does NCLB Want?
To determine with scientific rigor:
WHAT WORKS.
Translation: The impact of the intervention must be to:
INCREASE TEST SCORES!
What is Required by the What Works Clearinghouse and NCLB
Randomized, controlled, experimental studies, using the medical model of research.
Not matched comparisons. Not quasi-experimental designs. Must establish causality, ruling out plausible
explanations. Small, focused “interventions.” Limited teacher professional development
components. Short-term. School patterns are not changed. Students are the unit of assignment, not classrooms
or schools. No contextualization.
Foremost, there must be valid and reliable evidence that the intervention improves student achievement through scientific evidence.
The Medical Model as the Gold Standard
The Institute for Educational Sciences (IES) in the US Department of Education invokes the medical model of research as the standard toward which all research should strive.
Yet is this gold standard achievable? Is it the right gold standard or a silver
bullet? For example, can an instructional
“intervention” be examined in the same way as a course of pharmaceutical treatment?
Research and Evaluation Methodology Required by NCLB:Randomized Field Trials (RFT’s)
The rationale for RFT’s is the quest for unambiguous information in education.
“To be scientific, the design must allow direct, empirical
investigation of an important question, account for the context in which the study is carried out,
align with a conceptual framework, reflect careful and
thorough reasoning, and disclose results to encourage debate in the
scientific community.”
Shavelson & Towne, 2002
The Six Guiding Principles of Scientific Inquiry
(Not the Seven Deadly Sins)
1. Pose significant questions that can be investigated empirically (ruling out counter interpretations and bringing evidence to bear on alternative explanations)
2. Link research to relevant theory3. Use methods that permit direct
investigation of the question4. Provide a coherent and explicit
chain of reasoning5. Replicate and generalize across
studies6. Disclose research to encourage
professional scrutiny and critique
Which Really is the Driving Factor -
Research Questions or Methods?
The question should drive the research methodology, not the research methodology driving the questions.
Unfortunately, all too often the reverse has been happening because of political pressures.
Mandated questions, methods, and potentially answers as well.
The Question Should Drive the Research Design
What is happening (e.g., descriptions of population characteristics)?
Is there a systematic effect (i.e., systematic means causal)?
How or why does it happen?
Need to account for contextual factors.
Replicability of patterns across groups and time.
Evaluation
Should be meaningful and constructive. The results and information should benefit the students, teachers, school, and district.
Should not be punitive.
Should be informative, providing information on what is going on, how to improve, or other important questions.
Designing Evaluations
Use targeted evaluations.
Match your goals to data collection activities - that is, let the questions drive the methods.
Use measurable components.
Consider the design before it is implemented.
Be flexible. Things change and the evaluation design must change accordingly.
Numerous Caveats to RFT’s
Fidelity of implementation
Variability of treatment
Overlap between treatment and control groups
Adequacy of outcome measures
Multiple treatment interference
Relevance of control condition to policy issues
External validity
NCLB Goals:Impact on Students
Primary Goal - To improve student academic achievement through the use of technology in elementary schools and secondary schools.
Additional Goals – To assist every student in crossing the digital
divide by ensuring that every student is technologically literate by the time the student finishes the eighth grade, regardless of the student’s race, ethnicity, gender, family income, geographic location, or disability.
– To encourage the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by State educational agencies and local educational agencies.
NCLB Questions:Impact on Students
Is academic achievement improving with effective technology use?
Are students acquiring 21st century skills through effective technology use?
Are students more engaged in learning through effective technology use?
Necessary Conditions
Effective Practice - Is classroom practice characterized by powerful, research-based strategies that effectively and appropriately use technology?
Educator Proficiency - Are educators proficient in implementing, assessing, and supporting a variety of technology-based teaching and learning practices?
Robust Access Anywhere Anytime - Do students and staff have robust access to technology anywhere, any time, to support effective designs for teaching and learning?
Digital Age Equity - Is the digital divide being monitored and addressed through resources and strategies aligned to 21st century vision?
Vision and Leadership - Is there a 21st century vision? Is the education system transforming into a high-performance learning organization?
QuickTime™ and aPhoto - JPEG decompressor
are needed to see this picture.
Information and Communication Technology Literacy
ICT literacy is more than just the mastery of technical skills. It also includes:
Cognitive skills.
The application of cognitive skills and knowledge.
ICT literacy is seen as a continuum of skills and abilities from simple, everyday tasks to complex applications.
ICT literacy is using digital technology, communications tools, and/or networks to access, manage,
integrate, evaluate, and create information in order to function in a
knowledge society.
Source: ICT Literacy Panel, 2002.
A Working Definition
ICT Proficiency Skills
ACCESS - knowing about and knowing how to collect and/or retrieve information.
MANAGE - applying an existing organizational or classification scheme.
INTEGRATE - interpreting and representing information. It involves summarizing, comparing, and contrasting.
EVALUATE - making judgments about the quality, relevance, usefulness, or efficiency of information.
CREATE - generating information by adapting, applying, designing, inventing, or authoring information.
Three Proficiencies
Cognitive Proficiency - the desired foundational skills of everyday life at school, at home, and at work. Literacy, numeracy, problem solving, and spatial/visual literacy demonstrate these proficiencies.
Technical Proficiency - the basic components of digital literacy. It includes a foundational knowledge of hardware, software applications, networks, and elements of digital technology.
ICT Proficiency - the integration and application of cognitive and technical skill. Seen as enablers, they allow individuals to maximize the capabilities of technology. At the highest level, ICT proficiencies result in innovation, individual transformation, and societal change.
The SETDA Technology Literacy Working Definition
Technology literacy is the ability to responsibly use appropriate
technology to communicate, solve problems, and access, manage, integrate, evaluate, and create
information to improve learning in all subject areas and to acquire
lifelong knowledge and skills in the 21st century.
Sources of 21st Century Skill Definitions
www.ncrel.org/engauge/skills/sources.htm
The enGauge 21st-Century Skills National Education Technology Standards SCANS (Secretary’s Commission on Achieving
Necessary Skills) A Nation of Opportunity: Building America’s 21st
Century Workforce Preparing Students for the 21st Century Standards for Technological Literacy, Content for
the Study of Technology Being Fluent with Information Technology Information Literacy Standards for Student
Learning Growing Up Digital
Interactive Assessment and Evaluation
Assessment can be used both as a teaching tool and an evaluation mechanism.
Assessment and instruction should be iterative - a feedback loop to provide information to both student and instructor - continuous and interactive.
Assessments should assist students to evaluate their learning processes and outcomes. They should help to facilitate learning.
Techniques should capitalize on the affordances of the technology.
Online assessment should not be restricted by time constraints or resources.
Interactive Assessment and Evaluation
(Continued)
Techniques should be creative (e.g., games, puzzles, competitions).
Group assessments can be effective tools. Consider collaborative learning groups and team projects.
Information from assessments should provide the teacher with information about a student’s strengths and weaknesses.
Design thought provoking questions that stimulate interesting debate and threaded discussions.
Interactive Assessment and Evaluation
(Continued)
One form of assessment effective online is project-based learning - intensive, long-term, and focused on specific, real-world topics and authentic problems - accomplished individually or collaboratively.
Another form is a short, iterative instruction - assessment feedback cycle.
Use multiple methods.
Focus on the formative, not just summative - processes, not just outcomes, with chances for reassessment, acknowledging that every student has a different learning curve.
Interactive Assessment, Evaluation, and Instruction:
Issues
Collaborative learning necessitates new paradigms for learning, instruction, assessment, and honesty.
Rules for assessment need to be made clear to the students. If assessments are conducted properly, cheating may become a moot issue.
Expectations should be made clear in terms of participation online, including threaded discussions.
Teacher training is a key issue in terms of the technology, paradigm shifts, and role changes.
Interactive Assessment, Evaluation, and Instruction:
More Issues
Encourage interaction among student and teacher.
Provide opportunities for peer review, not just teacher feedback.
Encourage active participation and learning.
Provide prompt feedback.
Recognize diversity in learning - different paths, different paces, different learning styles.
Tailored to individual needs.
Provides a learning trail.
Some Existing K-12 Products and Companies
Smarterkids.com
LeapFrog
K12 - William Bennett’s diagnostic tests
Classroom Connect NOW
ETS K-12 Works
Ignite! - Neil Bush’s company
Many others
A mega-market now at $105 billion (NY Times, 1/21/01)
A Major Caution
Many of the companies are jumping onto the bandwagon and the testing craze.
Tests should be professionally developed and psychometrically sound. Many companies are cutting corners in the rush to market.
Consider in the selection process the pedagogical theory that underlies the tests and the psychometric models. Are the tests even grounded in theory????
BE CAREFUL!
Interactive Assessment, Evaluation, and Instruction: A
Paradigm Shift
Any time, any where, any pace. Immediate knowledge and infinite resources. Global learning. Multimedia. Interactive rather than lecturing or didactic. Learner-directed, not teacher-directed. Active versus passive. Student engagement. Role changes and shifts in responsibility for
learning. Collaboration versus individual learning. “Guide on the side” versus “sage of the stage”.
The teacher as a facilitator and coach, not as the primary transmitter of information.
Intellectual exploration.
Recommendations
Implementation
Don’t depend on the technology itself to improve teaching.
Teachers change their practices more readily when they can work in teams and have the support of administrators.
Teachers are more likely to use technology effectively when they have a computer at home on which to practice.
Technical support for teachers should be available at the building level, not just at the district level.
Recommendations (continued)
Use the technology to support existing activities instead of designing the activities around technology. Develop activities for the sake of learning, not for the chance to use the technology.
Don’t rely on traditional tests to determine what students have learned. Create activities that allow students to demonstrate what they’ve discovered through their projects to enable a more accurate assessment of their knowledge and skills.
Be patient. Teachers need time to learn how the technology can be used in the classroom. Students then need time to discover the knowledge by themselves and in collaboration with classmates.
Recommendations (continued)
Teacher Professional Development
Training should focus on how technology can enhance learning – and be embedded in real projects – rather than simply on how to work the machines.
Teachers need time to rework lessons to use technology effectively and experiment with new teaching styles.
If teachers are encouraged to share with other teachers what they are trying in the classroom, momentum for change can grow.
Teachers need professional development opportunities and continuing support from a facilitator or coach to learn how to use the technology.
Agenda for Research and Action
Conduct targeted case studies that address effective implementations in diverse educational settings.
Conduct research that addresses the “ramping up” issue.
Provide collaborative outreach and research opportunities to schools to begin to bridge the digital divide.
Shift the focus of teacher professional development to include technology as an important component of the emerging educational paradigms and pedagogical philosophies.
QuickTime™ and aPhoto - JPEG decompressor
are needed to see this picture.
The Center for Children and TechnologyA division of the Education Development Center
WWW.edc.org/CCT
Resource:Identifying and Implementing Educational
Practices Supported By Rigorous Evidence: A User Friendly Guide
www.excelgov.org/evidence