edtech 505 jordan final evaluation of ate

8/3/2019 EDTECH 505 Jordan Final Evaluation of ate

1/25

Final Evaluation Report ___________________________________________

_____________

Evaluation of theOnline

Teaching ToolElluminate

___________________________________________ _____________

Norma JordanEDTECH 505Section 4172

July, 27 2011


2/25

Table of Contents

Section Description Page1 Cover page2 Table of Contents3 Learning Reflection 34 Executive Summary 45 Purpose(s) of the Evaluation 56 Background Information 67 Description of Evaluation Design 78 Results & Discussion 9

9 Conclusions & Recommendations 1510 Appendices (includes EPD & Timeline) 16

2


3/25

Learning Reflection

Before entering this class I knew very little about evaluation. It hasbeen a challenge to grasp all of the information over the past eight weeks, butempowering at the same time. I feel like I am better prepared in my currentcareer as an educator, but the knowledge I have gained also gives me leveragein any career that I choose to pursue from here.

I have learned so much about evaluation this semester. Telling you

everything I have learned would take far more than a page, so I have chosen afew aspects that really stand out to me. First, I have gained an understandingof why we evaluate. It is important and necessary to understand how programsare working and if they are meeting the needs they were designed to meet.We evaluate to bring about a better product in the end. Second, I gained anunderstanding of the difference between evaluation and research. The linebetween the two is not incredibly large, but now I can see the difference. Inow understand that we do research to collect data, create a hypothesis, andcreate new knowledge. We evaluate to gain a better understanding of what isalready in place. I also gained an understanding of the differences betweenqualitative and quantitative data, and how to go about collecting each. Years

ago, right after my bachelors I took a research and writing class. I left thatclass with an A, but I can honestly say that I was confused about thedifferences between the two data forms. I think that two factors made thedifference in my new understanding, one being the fact that I am more matureand have a little life experience behind me, and two that the information waspresented in such a way that the difference made sense to me. Although Ihave learned many things, the last thing that I want to address here is thatEvaluation has different models to follow. I dont know what I thought beforethis class, I think I believed that evaluation was mostly about opinion, butreading about and learning more about the models to follow has been a greatlearning experience.

By doing the final project, I learned that even when you think you havecovered everything, there will always be something that was missed, and datacollection needs to be from several devices to get the most accurateinformation possible. I also learned that evaluators can not control all of thefactors in an evaluation, nor is it their job to attempt to control these factors,but simply to state what was found from the data gathered.

The information that I have gained here will be useful and something Ican refer to often in my future, both as an educator and an educational

3


4/25

technologist. I enjoy writing grants, and I think that now I know what I doabout evaluation my skills will be improved. I plan to revisit this informationoften in the hopes of building my skills and becoming better at it.

Executive Summary

Synchronous Teaching and learning on the Internet is quickly becoming astandard means of educating K-12 students online. This is a way of offeringyounger students an alternative to education in their local school. Schools thathold such synchronous teaching sessions include K-12 Inc, Desire2Learn, andvarious other private and public schools. One tool that is available to host

these synchronous sessions is Elluminate Live, now known as BlackboardCollaborate. Elluminate is a collaborative web conferencing tool that enablesmoderators (teachers) and participants the ability to share voice, video, andtext with each other in real time. The interface contains a whiteboard, chat,and desktop applications for teaching and learning.

Currently 29 of 50 states have online schools that use K12 curriculumand host synchronous teaching sessions for their K-8 Students using elluminate.The large volume of students nationally that are being schooled in this mannercontinues to grow. These circumstances encourage an evaluation of suchsoftware to determine if students are able to get the same education that theywould in a regular classroom with a teacher looking over their shoulder. Does

elluminate offer the right tools to keep student attention, and are teacherspleased with the software. What could be changed to improve its use?

It was determined that data would be collected in two ways for thisevaluation. A survey would be sent to teachers and administrators whocurrently use elluminate as a primary way of teaching their students. Thesurvey would determine if teachers felt elluminate live was able to meetstudent needs and if elluminate provides the right kinds of tools for teachingstudents in the K-8 age range. The other evaluation tool will be a checklist of observations made while watching old class recordings. The checklist andobservations will be used to determine which tools are used most frequentlyand to observe if the tools appear to be meeting the needs of the students.

Class recordings and data collected from the teachers survey indicatethat some tools meet the needs of students. It was also determined that manytools are engaging to students and enable teachers to teach from a distanceand still offer a personal touch. Teachers were very happy with the overallelluminate software, but would make small changes if possible. It wasdetermined that elluminate does offer a great collaboration tool for K-8students, but that more research needs to be done in this are to determine if elluminate meets all of the needs of K-8 students.

4


5/25

Purpose(s) of the EvaluationOnline teaching of younger students is becoming more and more popular

in the United States. Elluminate is currently used by over seven millionteachers K-16 and continues to grow at a rapid pace. It was developed to beused to extend the physical boundaries of a traditional classroom to studentswho are being schooled at home. Elluminate is said to give students liveinteraction with a teacher, promote active learning, improve studentperformance, facilitate online communities, and improve comprehension for

younger students who choose this method of education. Teachers are given theuse of elluminate to host their live online classrooms. Most schools use liveasynchronous sessions for younger age students.

The purpose of this evaluation is to see if elluminate provides thetraining and tools needed by online elementary school teachers to effectivelyteach students online. The evaluation will determine if elluminate providestools that help elementary teachers teach k-6 students effectively. Doeselluminate provide the tools needed to engage and enhance distance learningfor younger students? What tools could be changed or enhanced to make theprogram better.

Central Questions:1) What kind of training do teachers receive before hosting a live classroom

on elluminate?2) How often do teachers use certain tools within their live classes?3) Which tools do teachers perceive as helping them be a successful

teacher?4) Which tools meet the needs of the class members?5) Which tools keep students engaged?6) What technical difficulties does elluminate need to address in the

software to make it better?

Who is impacted by the results?Teachers and administrators of schools that use elluminate to teach

online classes to K-6 grade students would be the most impacted by thisinformation. Followed by parents of students that are schooled throughelluminate. Lastly K12 Inc. may be interested in the findings because they arecurrently partners with many schools that use elluminate.

What was the origin of the program or product?

5


6/25


7/25

presentations and control what the participant sees. The moderator can alsogive the participants the ability to navigate on their own. The elluminateclassroom can be controlled by the moderator. Class tools can be givenimmediately upon entering the class, or handed out by the teacher at differentintervals. Tools can also be taken away if necessary. Elluminate will also

allow participants to work in groups on the same screen. The participants canhave discussions related to their project using the microphone tool.Elluminate's virtual meeting room has a shared whiteboard to write and

draw on if given teacher permission. The participants can mark up thepresentation with answers or notes. The options that elluminate provides goeson and on with the use of more tools like: raise hand button, happy facebutton, confused face button, polling tools, leave class button, and many otheroptions.

Moderators can also give students the flexibility to navigate through thepresentation on their own, they can review a presentation at their own pacewith the option of going back and forth between slides and the option of taking

notes if needed without feeling rushed.

Evaluation DesignThe model that the evaluator used for this project was the goal-free

model. The objective was to observe and measure actual data related toelluminate and its current use with K-8 grade students. The evaluator wanted todetermine what the program currently provided for K-8 students, and tried tocarefully distinguish between what the program is actually doing vs. what theprogram was intended to do. The evaluator wanted to determine if elluminate metthe needs of the current K-8 population, and wanted to provide information to the

users so that they could look at the data collected and arrive at a judgment of their own based on the findings. The evaluator has taught using elluminate, sochose the goal free model trying to avoid bias of any kind. It seems that thismodel offers fewer opportunities for this, so it was the model the evaluator feltbest met the needs. The goal free model was also used because it may be usedto identify unintended pros and cons.

The objectives of this project are as follows:

Objective 1: Teacher ExperienceDetermine how many years the interviewees have had teaching online using

elluminate.

Objective 2: TrainingDetermine what kind of training the k-8 teachers are currently receiving onelluminate.

Objective 3: Use of toolsDetermine which tools in the elluminate suite are used by K-8 teachers.

7


8/25

Objective 4: Successful TeachingDetermine which tools teachers perceive as the best tools for meeting theneeds of their K-8 classes.

Objective 5: Meeting NeedsDetermine which tools meet the needs of the students in k-8 classes.

Objective 6: Student EngagementDetermine which tools teachers perceive as being able to keep their k-8students engaged.

Objective 7:Determine changes that teachers would like to see to help them better meetthe needs of their k-8 students.

InstrumentationIn order to collect information to be used in the evaluation, theevaluator designed and sent out an online survey to teachers andadministrators who currently teach K-8 students online usingelluminate. The purpose of the survey is to collect information aboutteachers and their perceptions of their online classes. The results of the questionnaire were collected and used to form an overall picture of how teachers use elluminate and if they feel it is meeting their currentneeds.

The evaluator also watched class recordings to determine if teacher perceptions were accurate, or if there was a discrepancy

between tools teachers believed that they used verses tools that theyactually used.

Sample: The evaluator sent the survey to teachers that work for a public

school in Idaho that uses K-12 curriculum. All teachers within thisschool use elluminate to host their online collaboration teachingsessions. Going into the evaluation the evaluator did not know theyears of experience that the teachers had. It was found that themajority of teacher polled had taught for an online public school usingelluminate for between two and six years.

Data Collection Procedures: The evaluator developed the survey and sent it to online

teachers using two different methods. The evaluator posted thesurvey on facebook, and also sent the survey by e-mail to colleagues.

The evaluator asked the participants to forward the link to anyadditional teachers that might be willing to participate in the survey.

The survey was left open for a two-week time frame to collect data.

8


9/25

During this time the evaluator also collected class recordings from theteachers. The evaluator watched class recordings and collected dataon which tools were used in k-8 classes, and which tools engaged thestudents.

Data Analysis The evaluator collected all data into an Excel spreadsheet. Fromthere the data was turned into percents based on the number of responses. The evaluator then took the data and shifted it so that alltools that fell within a use are were together so that the graphs wouldbe easier to read. At this point the evaluator chose the specific cargraph that would be used to show the data, and turned the graphs anddata into charts. The evaluator followed this same procedure for all of the graphs in the results section.

Results & Discussion of the Results

The results of the survey and class observations were compiled and reviewed toform the following conclusions about elluminate as an online teaching tool of K-8 grade students.

Objective 1:Teacher ExperienceThe majority of participants for this survey reported that they had taughtonline for 4-6 years, followed by participants that had taught for 2-3 yearsonline as seen in chart 1 below. The details of this survey were not surprisingbecause the increase of K-8 students learning online is fairly new.

Chart 1

9


10/25

Objective 2: TrainingThe majority of participants reported that their training was a face-to-face,hands on training either offered by K12 inc. or another teacher at the schoolthat they are employed with. Two other types of training that were reported

were online training, and being self-taught (Chart 2).

Chart 2

10


11/25


12/25

Chart 3- How often do you use the following tools in your K-8 classes?

Objective 4: Successful TeachingThe survey question and class observations were used to determine which tools

helped teachers be successful at their job. Both forms of data showed that thewhiteboard tool was the most effective, closely followed by the microphonetool and the chat tool. This is a significant piece of information because thesethree tools are they ways to communicate within the online class. Other toolsthat were also considered to help the teacher be successful were the handraise, record class, smiley, and thumbs down icons. These too are also formsof communicating within the online class. Other tools that were seen assomewhat helpful are the tools that allow for interaction with other computer

12


13/25

programs, like the web tour, multimedia, and file transfer. Also coming in thiscategory were the tools that allow students the ability to interact with eachother, like the breakout rooms. Again we see the following tools as teachersreported as not useful: the notes, shout outs, and invite new student tools.

Objective 5: Meet Needs

The data for meeting student needs shows that the whiteboard tool,microphone tools, and the chat tool that are most likely to meet studentneeds. Sixteen of the nineteen tools addressed in the survey and evaluatorsobservation are said to meet the need of students to some degree. Again wesee the three tools that are not used are the notes, shout outs, and invite newstudent tools. Which tools do you feel help you best meet the needs of your class?

13


14/25

Objective 6: Student EngagementThe results show that the tools that keep students most engaged are thewhiteboard tool, chat tool, and microphone tool, smiley icons, and thumbs upthumbs down tool. Again we see that online teachers reported that themajority of the tools were at least somewhat helpful at keeping their studentsengaged. Teachers feel that the shoutouts actually move out of the bottomcategory on this section, and leave the least helpful tools as the note tool andthe invite new students tool.

14


15/25

Which tools keep your students engaged.

Objective 7: ChangesAll teachers did address something that they would like to see changed.

The majority of teachers had complains about the speed of application sharing,and multimedia tools. They reported that these were difficult to use becausethe users had big delays in when the teacher derived the content and when theusers actually received the content.

Another complaint was that when the microphone tool has more thanone user there is a horrible echo and they would like to see that improved.Although teachers had things they would like to see changed, they felt that

15


16/25

elluminate was a good tool and they were very pleased with its use.

Conclusions & Recommendations

Immediate Conclusions:

It was determined based upon the data collected that the most useful tools forthe k-8 online teacher are the whiteboard tools, the chat tool, and themicrophone tool. This data can be seen in every chart in the result section.Most of the tools were seen as somewhat helpful to the teacher, and a fewtools were seen as not helpful at all. The following are conclusions based onthe data:

The tools show a bell curve, three tools are very useful, three of thetools were seen as not useful, but the majority of the tools were seen assomewhat useful by all of the teachers.

The evaluator does not feel that elluminate is meeting all the needs of online students at this time.

The evaluator still feels elluminate is one of the best programs currentlyavailable to educators, but according to the survey and data collected itneeds to be improved upon so that more of the tools fall into theAlways category.

Teachers may need additional training on the tools to make them moreeffective in class. The evaluator determined that maybe some of thetools are not familiar to the teachers and that is why they are not usedas often.

Long Range PlanningIt was determined based on the data collected for long range planning anddevelopments that:

Teachers may need more training on elluminate. Elluminate may need to communicate more with teachers to determine

how to make better use of the tools that are available. Elluminate may need to conduct more research and user tests for

younger students to determine their needs.

Evaluation InsightsIf I could do this evaluation differently I would ask more questions on thesurvey, and get more specific data about certain tools. I would ask questions

about teacher comfort level with some of the tools to see if the tools are notused because they are not useful, or because teachers do not know how to usethe tools. I think some of the data is skewed in this manner. I was not able tocollect as much data as I would have liked. Only 25 teachers took the onlinesurvey, I was hoping to have a minimum of 50. I also would have spent moretime having users give feedback on the collection tools to make sure I hadreally thought of all the pros and cons before sending out the survey for datacollection. I would have also had more teachers edit the survey and come up

16


17/25

with ideas for it.

Appendices

Time Line

6/27/11 Monday Brainstorm ideas/make plans for the overall project6/28/11 Tuesday Send project description to Ross Perkins

6/29/11 WednesdayGather information and collect e-mail addresses andphone numbers of sample population.

6/30/11 ThursdayGather information and collect e-mail addresses andphone numbers of sample population.

7/1/11 FridayCompile and write Evaluation's Program Description(EPD)

7/2/11 Saturdaywork on instruments: survey questions, interviewquestions, etc.

7/3/11 Sundaywork on instruments: survey questions, interviewquestions, etc.

7/4/11 Mondaysend drafts of the instruments: survey questions,interview questions, to peers for evaluation.

7/5/11 Tuesday7/6/11 Wednesday

7/8/11 Friday Conduct teacher interviews/ collect surveys7/9/11 Saturday Conduct teacher interviews/ collect surveys

7/10/11 Sunday Conduct teacher interviews/ collect surveys7/11/11 Monday Watch teacher class recordings

7/12/11 Tuesday Watch teacher class recordings7/13/11 Wednesday Watch teacher class recordings7/14/11 Thursday Analyze Data that was collected7/15/11 Friday Analyze Data that was collected7/16/11 Saturday Analyze Data that was collected7/17/11 Sunday Begin to write final report- draft outline7/18/11 Monday7/19/11 Tuesday Write Purpose and Background7/20/11 Wednesday Write Methodology Section7/21/11 Thursday Write Results Section7/22/11 Friday Write Discussion and Conclusion Sections7/23/11 Saturday Write the Summary section7/24/11 Sunday7/25/11 Monday Work on revising and working on the final report7/26/11 Tuesday Work on revising and working on the final report7/27/11 Wednesday Wrap-up7/29/11 Friday Final Project Due

17


18/25

Evaluators Program Description

Link to EPD

Data Collection Tools:Online Survey

18
http://bit.ly/rqaSkdhttp://bit.ly/rqaSkd


19/25

19


20/25


21/25

21


22/25

22


23/25

Observation Checklist

23


24/25

Tools Frequency of Usewhiteboardtoolssmiley icons

hand raisetoolclap/thumbsdown toolclip artchatmicrophonerecord classshout outsweb tourstime clockexplorescreensquiz toolmultimedia(video,MP3,etc...)file transfer notesinvite newstudentsweb cambreak outrooms

Other tools used by teachers that are not on the survey:

24


25/25

edtech 505 jordan final evaluation of ate

Documents