dyknow training evaluation

7/30/2019 DyKnow Training Evaluation

1/27

DyKnow Training Evaluation

Prepared for

Culver Educational Foundation

By

Culver Technology Support Committee

Phillip Cook, Director

October 9, 2012


2/27


3/27

3

Executive Summary

As a result of past discussions amongst faculty at Culver Academies, the Instructional

Technology department at Culver has installed a server to run DyKnow Vision and Monitor

software and purchased licenses for student, faculty and staff machines at Culver, beginning in the

spring semester of the 2012-2013 school year. All faculty and staff at Culver are trained on proper

use of DyKnow Monitor, with faculty receiving additional training in the use of DyKnow Vision

over the course of a two-day training. All staff attends the first day of training, focused on DyKnow

Monitor. The primary goal of the training is to ensure that faculty and staff are able to

appropriately implement DyKnow software within the academic activities that occur on campus.

Training is designed to be interactive, with both sessions focused on using the software within the

context of a classroom setting. This evaluation is designed to determine the effectiveness of

training, as well as gauge the impact of DyKnow software upon the academic community at Culver.

Evaluation planning began in August of 2012, headed by the Director of the Technology

Support Committee on campus at Culver. The Technology Support Committee (TSC) is comprised

of fourteen faculty and staff and is charged with the design and implementation of technology

related trainings and evaluation on campus at Culver. In early August, four members of the TSC

received training from DyKnow representatives; those members then trained the remaining

members of the TSC in the use of DyKnow through designated TSC meetings in September and

into early October. As a result of the initial trainings within the TSC, the following three objectives

were identified: faculty and staff competency in the use of DyKnow Monitor to manage classroom

and related academic settings, the use of DyKnow Vision by faculty to create digital collaborative

situations within the classroom, and the ability to create DyKnow Vision compatible presentations

for use within the academic classroom. Based upon these objectives, competencies of

management, design and implementation were developed. Utilizing Kirkpatricks evaluation

framework, instruments were created to evaluate the effectiveness of the training program in

regards to these competencies related to the first four levels of the framework: reaction, learning,

behavior and results. Level one (reaction) survey gauges the effectiveness of the training in

regards to training materials, training methodology, instructor implementation and setting. Level

two (learning) evaluates comprehension of basic terminology and skills, and consists of a real

world situation which is peer evaluated using a supplied check sheet. Level three (behavior)


4/27

4

involves an observation of the use of DyKnow within a classroom or related academic setting by a

member of the TSC. Level four (results) consists of a written self-reflection as well as

implementation and usage data collected via student experience questionnaires administered at

the end of term 3 and term 4 on campus.


5/27

5

Introduction

DyKnow Vision and Monitor software provides a digital means for faculty and staff to

collaborate and monitor academic activities on campus at Culver Academies. The DyKnow

training program was developed based upon the need for a diverse group of employees to be able

to utilize appropriate feature sets during academic class and study periods. DyKnow Monitor

provides a much desired means to observe student laptop screens remotely, and allows for the

enabling or disabling of specific applications and websites. DyKnow Monitor is viewed as an

additional tool to ensure that students are using their school-issued laptops appropriately and are

making good use of instructional time. In addition to the observational benefits that DyKnow

Monitor provides, DyKnow Vision allows for the creation of digital collaborative learning

environments, both face to face or across campus. Faculty can easily present class material

remotely to students, and students can work collaboratively on projects using the team-centered

features offered within the DyKnow Vision software package. Overall, the ability to create

managed collaborative digital learning environments is very attractive, and makes good use of the

schools existing 1:1 laptop program.

Of primary interest regarding this evaluation is to determine the effectiveness of training,

increased fluency in terminology and skills related to the implementation of appropriate use of

DyKnow Monitor and Vision software within the academic classrooms at Culver Academies. As

mentioned previously, Kirkpatricks evaluation framework serves as the basis for the preparation

of each of the four evaluations developed. These levels and corresponding instruments are

described in the evaluation framework that follows.


6/27

6

Evaluation Framework

Level 1: Reaction

A Level 1 reaction survey was designed in order to evaluate faculty and staff response to

training in regards to learning environment, relevance, delivery, and overall training experience.

The survey uses a four point Likert scale. Faculty and staff will be administered the survey

immediately following the conclusion of each of the two day training sessions. The level 1 survey

can be found in the appendix.

The survey provides timely feedback regarding the two-day training session. Specifically,

evaluation questions focus on classroom environment, utility of training, and effectiveness of the

instructor and instruction related materials. This data is important to ensure that training is

identified as useful within the classroom and similar academic settings and was clearly

communicated within an effective learning environment. Results from the level 1 instrument will

be reported using mean scores as well as standard deviations. Positive scores will indicate faculty

and staff feel prepared and able to use DyKnow Monitor and Vision appropriately on campus at

Culver.

Level 2: Learning

DyKnow Monitor training is scheduled for Saturday, November 24th, 2012. The learning

evaluation is administered during the final hour of the training. Participants are assigned a peer

reviewer and given one of two related tasks involving the use of DyKnow, as well as a scoring

check sheet. The peer reviewer will provide the scored check sheets to the training instructor.

Level 2 instrument scores will be reported using mean scores as well as standard

deviations. Mean scores on individual check sheet components are useful in determining topics in

need of follow-up with future trainings and can help to identify areas in need of greater coverage

and illustration during training.

Level 3: Behavior

The measurement of behavior occurs via academic observations through the submission of

an observation sheet. Individual faculty and staff are each observed within their respective

academic setting (either classroom or within the dorms & barracks on campus) by a member of


7/27

7

the Technology Support Committee. The observing TSC member records an overview of the

observation and uses an observation sheet to mark observed uses of DyKnow Monitor and/or

DyKnow Vision (see the appendix for observation sheet). Each TSC member will observe trained

faculty and staff on two separate occasions, with each trained faculty and staff member being

observed a minimum of two times. All observations will occur after training during terms 3 and 4

in the spring 2013 semester.

Observational data within context is ideal for showing the extent to which DyKnow

software usage is being implemented on campus at Culver. Results from the observations are

compiled at the end of term 4 of the 2012-2013 school year and analyzed for frequency of

observed usage, broken down by item from the observation sheet. The results from the behavior

instrument serve to identify departments that may require follow-up regarding implementation of

DyKnow as well as identifying departments on campus that are making exceptional use of

DyKnow.

Level 4: Results

Results are determined via two methods: self-reflection and student feedback. The self-

reflection (located in the appendix) is administered to faculty and staff during June week

(beginning June 10, 2013) at the conclusion of the 2012-2013 school year, and consists of prompts

related to the change of classroom or academic environment as the result of the use of DyKnow

Monitor and Vision, with responses to prompts consisting of rated values and written comments.

In addition to the self-reflection, students will provide feedback in regards to the impact that

DyKnow Monitor and Vision software has had within their academic courses and within academic

study times on campus during the spring semester. Questions addressing impact are inserted into

the existing Student Evaluation Questionnaire (SEQ) that is administered to students at the end of

term 3 and term 4(attached for reference within the appendix).

TSC members will collate results from the faculty and staff self-reflections, and summarize

common themes expressed within the written portions of the instrument. Rated prompts will

have data collected and analyzed, with mean values and standard deviations presented. Results

will also be broken down by department to identify regions of campus that experienced the most

significant results. This analysis is particularly useful for future training planning for the 2013-

2014 school year, where training can be focused within departments that have had lesser results


8/27

8

during the initial rollout of the software. The results data will also be shared with the Head of

Schools, the Director of Instructional Technology, and the Co-chairs of the Academic Instructional

Technology Department, all of whom are stakeholders in the funding and decision to bring

DyKnow to campus at Culver.

Conclusion

In education, it is always a key concern to maximize instructional time. DyKnow Monitor

and Vision software provide a means by which faculty and staff can minimize distractions during

academic time, and as a result increase learning on campus. The proper training of faculty and

staff is critical for ensuring that the software is used appropriately and has maximum impact upon

the learning environment at Culver. Being able to conduct appropriately designed Monitor

sessions and create interactive collaborative Vision sessions are key skills that are a focus of the

DyKnow training program.

DyKnow software is not cheap, and the initial investment in software licenses and server

equipment must be justified through increased on-task behavior of students during academic

times. Building a community that leverages DyKnow software for good and enhancing the

technology skills of an already talented faculty and staff serves to improve not only the overall

quality of education of students at Culver, but also opens the possibility for future thought

leadership in the thoughtful implementation of technology programs amongst accredited college-

preparatory boarding schools.


9/27

9

Appendix

Evaluation Plan

Evaluation Plan

Project DyKnow Training EvaluationScope Evaluation of two-day training program administered to faculty and staff at

Culver Academies

Identification DyKnow Training Evaluation Plan, October 9th, 2012

Goals

Purpose Determine the effectiveness of DyKnow training

Levels of

Evaluation

Levels 1-4: Reaction, Learning, Behavior & Results

Objectives that

will be

measured

1. Teach faculty & staff proper protocol in the setup of DyKnow Monitorsessions and plans.

2. Teach faculty & staff skills related to facilitating DyKnow Monitorsessions during academic time.

3. Teach faculty skills and techniques in using DyKnow Vision to createdigital collaborative sessions.

Process

Item Date Due Who

Create evaluation plan 8/9/12 TSC

Develop instruments 10/30/12 TSC

Observe DyKnow training 11/24/12;

12/1/12

TSC

Administer and collect level 1 data 11/24/12;12/1/12

Phil

Administer and collect level 2 data 11/24/12;

12/1/12

Peer evaluators

Observe DyKnow usage by faculty &

staff within academic setting

5/31/12 TSC

Collect level 3 data 5/31/12 TSC

Analyze data from levels 1-3 6/1/12 Phil, Megan, Oliver, Lauren

Administer and collect SEQ data 6/7/12 All faculty & staff

Administer and collect level 4 data

from faculty & staff

6/7/12 TSC

Analyze data from SEQ and level 4 6/10/12 TSCPrepare final report 6/11/12 TSC

Present results to Head of Schools,

Director of IT and AITD

6/14/12 Phil

Evaluation Budget

N/A; Evaluation completed in-house


10/27

10

Data Collection

Data Item When How Collected From Whom By Whom

Reaction to

training

After training Survey Faculty & Staff TSC Members

Learning After training Peer evaluation Faculty & Staff Peer evaluators

Behavior After trainingduring terms 3

and 4

Observationsheet

Faculty & Staff TSC Members

Results June week, 2013 Self-reflection;

Student SEQ

survey

Faculty, Staff &

Students

TSC Members

Data Analysis

Target

Measures

Methods of

performing

calculations

How to control

for other factors

Who will perform analysis?

Reaction Mean scores; SD Trainers leave

room

Phil

Learning Mean scores; SD Testing scenarios

similar to

training

scenarios

Phil, Megan, Oliver, Lauren

Behavior Observation;

frequency

Observation via

more than one

TSC member

Technology Support Committee

Results Frequency; Mean

scores; SD

Self-reflection

and SEQ data

weighted equally

Technology Support Committee

Report of Findings

What results are reported To whom How communicated

1. Reaction results Head of Schools,

Director of IT &

AITD

Written report

2. Learning results Head of Schools,

Director of IT &

AITD

Written report

3. Behavior results Head of Schools,

Director of IT &

AITD

Written report

4. Self-reflection & SEQ results Head of Schools,

Director of IT &

AITD

Written report


11/27

Gantt Chart


12/27

12


13/27

13


14/27

14


15/27


16/27

16


17/27

17


18/27

18


19/27

Level 1 (Reaction) Instrument

DyKnow Training Survey

Level 1

Instructions: Thinking about the training that you just completed, please indicate to what degree

you agree with each statement using the following scale:

Please provide comments along with your rating to help the TSC improve this training in the

future.

Learning Environment

1. The training environment helped me to learn 1 2 3 4

2. There were no major distractions that interfered with my learning 1 2 3 4

Relevance

3. The training will be helpful for my success using DyKnow in the future. 1 2 3 4

4. I will be able to immediately use what I learned. 1 2 3 4

1= Strongly Disagree 2= Disagree 3= Agree 4= Strongly Agree

Comments:

Comments:


20/27

20

Delivery

5. I was well engaged with what was going on during the training. 1 2 3 4

6. The activities and exercises aided in my learning. 1 2 3 4

7. I was given adequate opportunity to demonstrate what I was learning. 1 2 3 4

Overall

8. The training met my expectations. 1 2 3 4

9. I am clear on how to apply what I learned on the job. 1 2 3 4

10. I would recommend this training to my colleagues. 1 2 3 4

Comments:

Comments:


21/27

21

Level 2 (Learning) Instruments

DyKnow Training Scenario

DyKnow Monitor Training

Peer Evaluation Scenarios

Instructions: Select a partner within your training group to serve as a peer evaluator. Next,

select one of the two scenarios, and respond to the prompts.

Scenario A

You are an instructor in a humanities class and you have designed an activity where students use

online resources to research the life of Antoine Laviosier. To ensure students remain on task, you

would like them to be able to access the internet, and use only Microsoft Word or Microsoft

OneNote to record their research findings.

1. Identify and record the settings within DyKnow Monitor that you would utilize toaccomplish this task in the space below.

2. Create an appropriate plan with these settings within DyKnow Monitor.3. Demonstrate proper use of the plan to your peer evaluator.

Scenario B

You are a dorm supervisor in Beason in charge of facilitating academic study time from 8pm to

10pm. Lately there have been issues with students using their computers for non-academic tasks

during academic study time. Specifically, the use of Skype, twitter and facebook have been causingsignificant distractions.

1. Identify and record the settings within DyKnow Monitor that you would utilize to minimizedistractions during academic study time.

2. Create an appropriate plan with these settings within DyKnow Monitor.3. Demonstrate proper use of the plan to your peer evaluator.


22/27

22

DyKnow Vision Training

Peer Evaluation Scenarios

Instructions: Select a partner within your training group to serve as a peer evaluator. Next,

select one of the two scenarios, and respond to the prompts.

Scenario A

You are an instructor in a science class and you would like to conduct a DyKnow Vision

collaborative session where you will present a series of slides illustrating the changing models of

the atom. You have created the PowerPoint already, but need to convert the presentation to be

able to use it with DyKnow.

1. Within Microsoft PowerPoint, create a DyKnow compatible version of the suppliedPowerPoint file.

2. Demonstrate how you would import the file into DyKnow to your peer evaluator.3. Demonstrate proper use of Vision, using the converted PowerPoint to your peer evaluator.Scenario B

You are a dorm supervisor in Argonne and several of the girls from the dormitory have requested

a collaborative session using DyKnow Vision to discuss dormitory issues.

1. Identify and record the settings within DyKnow Vision that you would utilize to create acollaborative session.

2. Create an appropriate session with these settings within DyKnow.3. Demonstrate proper use of the session to your peer evaluator.


23/27

23

Peer Evaluation Checksheet

DyKnow Training

Peer Evaluation Checksheet

Name of person being evaluated:_________________________________________

Name of evaluator:______________________________________

DyKnow Monitor

Faculty or staff member being evaluated utilized:

Start/Stop Monitor Allow Applications Block Applications Allow URL Block URL Attention Send File

Remote Control Shutdown/Restart Poll Status Monitor class view Monitor individual view

DyKnow Vision

Faculty or staff member being evaluated utilized: Chat Group session Presentation Quiz inserted within presentation Status Request file Collect file Return file Share screen Presenter control Presentation view

Training: Monitor Vision Scenario: A B


24/27

Level 3 (Behavior) Instrument

DyKnow Training

Observation Sheet

Name of person being observed:______________________________________

Department:_______________________________

Observer:____________________________________

Instructions: Record the frequency of DyKnow skills and features that are used during your

observation using tick marks.

DyKnow Monitor

Faculty or staff member being observed utilized:

Start/Stop Monitor Allow Applications Block Applications Allow URL Block URL Attention Send File

Remote Control Shutdown/Restart Poll Status Monitor class view Monitor individual view

DyKnow Vision

Faculty or staff member being observed utilized:

Chat Group session Presentation Quiz inserted within presentation Status Request file

Collect file Return file Share screen Presenter control Presentation view


25/27

25

Level 4 (Results) Instruments

Faculty & Staff Self-Reflection

DyKnow Training Self-Reflection

Name (optional):______________________________________

Instructions: Reflecting back upon your use of DyKnow Monitor and Vision over the past two

terms, respond to the following questions. Your responses will help the TSC to assess the impact

that DyKnow training has had upon .

What is your current position at Culver?

a. Faculty memberb. Staff member

Which department are you a member of at Culver?

a. Fine Artsb. Humanitiesc. Scienced. Mathe. Modern & Classical Languagesf. Student Lifeg. Other:______________________________

1. I use DyKnow Monitor during academic settings on campus at Culver.a. Alwaysb. Frequentlyc. Sometimesd. Occasionallye. Never

2. I use DyKnow Vision during academic settings on campus at Culver.a. Alwaysb. Frequentlyc. Sometimesd. Occasionallye. Never

3. The use of DyKnow Monitor has resulted in fewer technology related distractions.a. Strongly Agreeb. Agreec. Disagreed. Strongly Disagreee. I do not use DyKnow Monitor


26/27

26

4. The use of DyKnow Vision has resulted in greater classroom collaboration.a. Strongly Agreeb. Agreec. Disagreed. Strongly Disagreee. I do not use DyKnow Vision

5. As you reflect upon your use of DyKnow Monitor, how have you observed the use ofMonitor affect the learning environment at Culver?

6. As you reflect upon your use of DyKnow Vision, how have you observed the use of Visionaffect the learning environment at Culver?


27/27

Student Experience Questionnaire

Excerpt of DyKnow prompts from the Student Experience Questionnaire

Section E:DyKnow Software

Instructions: The questions that follow relate the to use of DyKnow by your instructor as well ascounselors and other staff during academic time on campus at Culver. Read each statement

carefully, and then indicate to what degree you agree with each statement using the following

scale:

1. Many faculty and staff are using DyKnow on campus during academic time.2. Faculty and staff use DyKnow in helpful ways.3. DyKnow helps me to stay focus on learning within the classroom.4. DyKnow helps me to stay focused on learning during academic study time.5. My study behaviors have changed as a result of DyKnow usage on campus.6. DyKnow has improved the quality of instruction on campus at Culver.7. I am using technology more frequently in class as a result of DyKnow.

1= Strongly Disagree 2= Disagree 3= Agree 4= Strongly Agree

dyknow training evaluation

Documents