mole testing evaluation final report-out documents/mole testing eval… · the testing &...

This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).

MoLE Testing & Evaluation Results

1

11 – 12 September 2012Testing & Evaluation Team


PRIMARY OBJECTIVES

The Testing & Evaluation Working Group goals include:

• Conduct an operational evaluation of mobile learning utility, usability and suitability in a multinational environment

• Ensure the guidelines regarding the Human Research Protection Program are implemented

• Develop operational measures of effectiveness and performance

• Develop data collection and analysis plans to ensure the data is collected sufficiently to fully evaluate the project’s technical components

• Make recommendations on how the project’s technical lessons learned may be incorporated into future multinational mobile learning initiatives

2


FY11 ACTIVITIES

• Human Research Protection Program• Established a small group to focus on US, EU and International Research

Ethics Requirements • Developed an initial draft of the MoLE Proof of Concept (PoC) Informed

Consent based on post-MoLE Kick-Off Workshop discussion

• Questionnaire/Survey Development• Drafted Terms of Reference (TOR) for utility, usability, suitability and self-

efficacy• Sent an email to each international delegate seeking input regarding

questions to ask for each TOR

• Testing & Evaluation Workshop (September 2011)• Conducted a final review/edit on the MoLE Proof of Concept (PoC)

Informed Consent• Formed two subgroups (i.e., Questionnaire Development & Data Analysis)

3


FY12 ACTIVITIES

• Human Research Protection Program• Submitted the final MoLE PoC Informed Consent to international

delegates for review/comment• MoLE PoC Informed Consent translated into French and Spanish

• Questionnaire/Data Analysis• Finalized the questions based on their TOR• Finalized the data analysis approach

• Testing & Evaluation Workshop (December 2011)• Conducted a final review/edit on the MoLE PoC Questions • Outlined the process for integrating the questions into the MoLE MedAid

App• Finalized the Demographic Questions for the PoC

4


PROOF OF CONCEPT APPROACH

5


TESTING & EVALUATION PROCESS

6

Email announcement

Online registration App install Activate Try it out Evaluation

Your personal pin number is 00000

Your unique start page: http://mole.tribalhosting.net/register/00000/35400ee9d41f252e3e2d7e3a41708847aace01475375ee272f56363a1e565da2/8abf02dbe063858fc1dd7adb20277addd28cf375f6a2ae45e4fb4871a2ecf16b


WELCOME EMAIL

7

• A “Welcome Email” was initially sent to

approximately 600 volunteer on the first

day of the MoLE Proof of Concept

• The announcement email provided

additional information on installation, the

user guide and an introduction video.


REMINDER EMAIL

8

• A “Welcome Reminder Email”

was initially bi-weekly to remind

the volunteers.

• The reminder email also

provided additional information

on installation, the user guide

and an introduction video.


INFORMED CONSENT

9

• The Informed Consent was provided in three language, which requires the consent process to be conducted in their native language if they have English proficiency concerns;

• Before participating in the MoLE Proof of Concept, each volunteer acknowledged they understood the informed consent in its entirety and understood their rights.


DEMOGRAPHICS

10

Question Responses

Age less than 20, 20-29, 30-39, 40-49, 50+, no answer

Gender Male, Female

How proficient are you in English Beginner . . . Advanced

Are you using your own personal smartphone for the purpose of this trial?

Yes, No, No Answer

How comfortable are you with using the mobile device that's running the MoLE app? [Beginner to Advanced]

Beginner . . . Advanced

Have you previously been involved in humanitarian assistance or disaster relief operations?

Yes, No, No Answer

What is your professional expertise? Medical, Rescue, Training, E-learning, Other

Have you taken the Trafficking in Persons (CTIP ) course within the last two years?

Yes, No, No Answer


INSTITUTIONAL REVIEW BOARD (IRB)

11

• An independent review was conducted at the

request of the CNE-CNA-C6F Medical Officer

• Review confirmed MoLE Research Protocol

complied with US/EU/International Research Ethics

Requirements

• Email to SPAWAR SSC Pacific confirmed SPAWAR

SSC Pacific (DoN Research Protocol Office)

confirmed Independent IRB Findings – except for

Tribal’s Usability Study

• SPAWAR SSC Pacific stated they would provide a

letter; however, subsequent follow-up requires

Letters of Support from key US organizations that

identified US Government/Contractors as

volunteers.


PROOF OF CONCEPT DATA ANALYSIS

12


PARTICIPANT TOTALS

13

• Launched: 268• Started Proof of Concept: 177 (66.4%)• Completed Proof of Concept: 137 (51.12%)


AGE GROUP

14

Age Group Breakdown< 20 years 5 1.87%20+ years 51 19.03%30+ years 83 30.97%40+ years 75 27.99%50+ years 52 19.40%No Response 2 0.75%


GENDER

15

Gender BreakdownMale 181 67.54%Female 87 32.46%


ENGLISH PROFICIENCY

16

English Proficiency LevelsLevel Response Percentage

Level 1 81 30.22%Level 2 11 4.10%Level 3 10 3.73%Level 4 38 14.18%Level 5 35 13.06%Level 6 38 14.18%


English Proficiency Levels Defined

Level 1—Beginning/Preproduction [WIDA level = Entering]:• Does not understand or speak English with the exception of a few isolated words or expressions.

Level 2—Beginning/Production [WIDA level = Beginning]:• Speaks conversational and academic English with hesitancy and difficulty• Understands parts of lessons and simple directions• At a pre-emergent or emergent level of reading and writing in English, significantly below grade level

Level 3—Intermediate [WIDA level = Developing]:• Understands and speaks conversational and academic English with decreasing hesitancy and

difficulty• Is post-emergent, developing reading comprehension and writing skills in English.• English literacy skills allow the student to demonstrate academic knowledge in content areas with

assistance

17

The WIDA Consortium (World-Class Instructional Design and Assessment) is a not-for-profit educational consortium of state departments of education. WIDA designs and implements proficiency standards and

equitable educational opportunities for grade K-12 students who are English language learners.


English Proficiency Levels Defined

Level 4—Advanced Intermediate [WIDA level = Expanding]:• Understands and speaks conversational English without apparent difficulty, but understands and

speaks academic English with some hesitancy• The pupil continues to acquire reading and writing skills in content areas needed to achieve grade

level expectations with assistance.

Level 5—Advanced [WIDA level = Bridging]:• Understands and speaks conversational and academic English well• Near proficient in reading, writing, and content area skills needed to meet grade level expectations• The pupil requires occasional support

Level 6—Formerly Limited-English Proficient/Now Fully-English Proficient:• The pupil was formerly limited-English proficient and is now fully English Proficient• Reads, writes, speaks and comprehends English within academic classroom settings

Level 7—Fully-English Proficient/Never Limited-English Proficient:• Never classified as limited-English proficient and does not fit the definition of a limited-English

proficient student outlined in either state of federal law

18


ENGLISH PROFICIENCY (LEVEL 1)

19

Unusually high numbers of Level-1 Proficiency for English speaking countries (US/UK) can be attributed some users not answering the question (within their right under the research ethics

guidelines) and accepting the default value of ‘1'


“BRING YOUR OWN DEVICE (BYOD)”

20

Bring Your Own Device (BYOD) Analysis

No 78 29.1%Yes 190 70.9%


MOBILE DEVICE PROFICIENCY

21

Mobile Device Proficiency AnalysisLevel Response Percentage

Level 1 81 30.22%Level 2 11 4.10%Level 3 10 3.73%Level 4 38 14.18%Level 5 35 13.06%Level 6 38 14.18%


MOBILE DEVICE PROFICIENCY (LEVEL 1)

22


HUMANITARIAN ASSISTANCE PROFICIENCY

23

Humanitarian AssistanceResponses Number PercentageNo 77 28.73%Yes 191 71.27%


PROFESSIONAL EXPERTISE

24


TRAFFICKING IN PERSONS (CTIP) COURSE

25

CTIP Course AnalysisYes 45 16.79%No 223 83.21%


DEVICE TOTALS

26

Mobile Device Analysis

Phone Type Number Percentage

iPhone 4 23 8.58%

iPhone 5 146 54.48%

android 2 32 11.94%

android 2.3 61 22.76%

android 3 or 4 6 2.24%


TESTING & EVALUATION APPROACH

27

EVALUATION QUESTIONS Category1.1 Rate your confidence in using the checklist. Self-efficacy2.1 How useful are mobile devices for this type of training? Utility2.2 How easy was the CTIP content to navigate? Accessibility3.1 How useful is access to digital reference materials on a mobile device? Utility4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy

5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness6.1 How helpful are collaboratively updated contact details? Self-efficacy7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility

7.5 How useful are collaborative contact pages to access real time information? Utility7.6 How easy was the Global MedAid app to navigate? Accessibility

7.7 Please write up to five single words which best describe your overall experience of using the mobile device as a tool for learning.

7.8 What did you like most about the mobile device as a tool for learning?7.9 What did you like least about the mobile device as a tool for learning?8 Any additional comments


OVERALL MEDAID APP RATING

28

Likert Rating Question All-Users Android iPhoneN/A or Skipped 8 3 5

1 Not useful at all 9 1 82 Not useful 9 4 53 Not very useful 20 14 64 Neither useful nor not useful 18 6 125 Quite useful 33 15 186 Useful 60 23 377 Very useful 22 5 17

n: 179Minimum: 0Maximum: 7Mean: 4.7 +/-0.3Median 5.4Mode: 6Standard Deviation: 1.9Variance: 3.5

0102030405060

0

1

2

3

4

5

6

7

All Users Responses

05

10152025

0

1

2

3

4

5

6

7

Android Users Responses

010203040

0

1

2

3

4

5

6

7

iPhone Users Responses


QUESTIONS & LIKERT SCALE

29

EVALUATION QUESTIONS Category1.1 Rate your confidence in using the checklist. Self-efficacy2.1 How useful are mobile devices for this type of training? Utility2.2 How easy was the CTIP content to navigate? Accessibility3.1 How useful is access to digital reference materials on a mobile device? Utility4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness6.1 How helpful are collaboratively updated contact details? Self-efficacy7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility7.5 How useful are collaborative contact pages to access real time information? Utility7.6 How easy was the Global MedAid app to navigate? Accessibility

Rating 1 Rating 2 Rating 3 Rating 4 Rating 5 Rating 6 Rating 7Not confident at all Not confident Not very confident

Neither confident nor unconfident Quite confident Confident

Very confident


1.1: RATE YOUR CONFIDENCE IN USING THE CHECKLIST

30




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


010203040

Skipped orN/A

1

2

3

4

5

6

7



2.1: HOW USEFUL ARE MOBILE DEVICES FOR TRAINING?

31




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

10152025

Skipped orN/A

1

2

3

4

5

6

7


010203040

Skipped orN/A

1

2

3

4

5

6

7



2.2: HOW EASY WAS THE CTIP CONTENT TO NAVIGATE?

32



n: 179Minimum: 2Maximum: 7.0Mean: 4.8 +/-0.3Median 6.0Mode: 6Standard Deviation: 2.3Variance: 5.2

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7



3.1: HOW USEFUL IS ACCESS TO DIGITAL REFERENCE MATERIALS ON A MOBILE DEVICE?

33




01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



4.1: HOW EASILY COULD YOU EXPLAIN TO A COLLEAGUE HOW TO LIIK UP NGO LOGOS?

34




01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7



5.1: HOW USEFUL ARE VIDEOS AS A TOOL TO ENHANCE YOUR UNDERSTANDING OF USAID?

35




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



6.1 HOW HELPFUL ARE COLLABORATIVELY UPDATED CONTACT DETAILS?

36




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7



7.1 HOW USEFUL ARE MOBILE DEVICES FOR TRAINING?

37




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



7.2 HOW USEFUL ARE MOBILE DEVICES FOR REFRESHER TRAINING?

38




020406080

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

01020304050

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



7.3 HOW USEFUL ARE MOBILE VIDEOS TO ENHANCE UNDERSTANDING?

39




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

101520

Skipped orN/A

1

2

3

4

5

6

7



7.4 HOW USEFUL ARE DIGITAL REFERENCE MATERIALS ON MOBILE DEVICES?

40




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

01020304050

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



7.5 HOW USEFUL ARE COLLABORATIVE CONTACT PAGES TO ACCESS REAL-TIME INFORMATION?

41




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



7.6 HOW EASY WAS THE GLOBAL MEDAID APP TO NAVIGATE?

42



n: 179Minimum: 0Maximum: 7Mean: 4.0 +/-0.4Median 5.0Mode: N/A or

SkippedStandard Deviation: 2.6Variance: 6.9

01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



CATEGORY ANALYSIS

43

EVALUATION QUESTIONS Category2.2 How easy was the CTIP content to navigate? Accessibility7.6 How easy was the Global MedAid app to navigate? Accessibility1.1 Rate your confidence in using the checklist. Self-efficacy

4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy

6.1 How helpful are collaboratively updated contact details? Self-efficacy

5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness

2.1 How useful are mobile devices for this type of training? Utility

3.1 How useful is access to digital reference materials on a mobile device? Utility

7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility

7.5 How useful are collaborative contact pages to access real time information? Utility


ACCESSIBILITY

44



n: 179Minimum: 0Maximum: 7Mean: 4.4 +/-0.3Median 5.0Mode: 6, 7Standard Deviation: 2.0Variance: 4.2

010203040

0

1

2

3

4

5

6

7

All Users Responses

05

101520

0

1

2

3

4

5

6

7


010203040

0

1

2

3

4

5

6

7



SELF-EFFICACY

45




0102030405060

0

1

2

3

4

5

6

7

All Users Responses

05

1015202530

0

1

2

3

4

5

6

7


05

1015202530

0

1

2

3

4

5

6

7



USEFULNESS

46




0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

010203040

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



UTILITY

47




0102030405060

0

1

2

3

4

5

6

7

All Users Responses

05

1015202530

0

1

2

3

4

5

6

7


010203040

0

1

2

3

4

5

6

7



7.7 WRITE FIVE SINGLE WORDS THAT DESCRIBE YOUR OVERALL EXPERIENCE?

48

9971

51 4329 23

2

31.1%

53.5%

69.5%

83.0%92.1%

99.4%

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%100.0%

0

50

100

150

200

250

300

Num

ber o

f Res

pons

es

Categories

Descriptive Terms

This chart is based on the frequency of the words using synonym analysis


7.7 WRITE FIVE SINGLE WORDS THAT DESCRIBE YOUR OVERALL EXPERIENCE?

49

This chart is based on pairwise high frequency analysis


7.8 WHAT DID YOU LIKE MOST ABOUT THE MOBILE DEVICE FOR LEARNING?

50


7.8 WHAT DID YOU LIKE MOST ABOUT THE MOBILE DEVICE FOR LEARNING?

51



7.9 WHAT DID YOU LIKE LEAST ABOUT THE MOBILE DEVICE FOR TRAINING?.

52


7.9 WHAT DID YOU LIKE LEAST ABOUT THE MOBILE DEVICE FOR TRAINING?.

53



8. ADDITIONAL COMMENTS

54


8. ADDITIONAL COMMENTS

55



USER-FEEDBACK

• A very good and innovating idea

• I love how you enable on/offline and the graphs are really nice. I also like the in-app evaluation layer. An interesting combination of action and feedback

• Great to access all this resources in a mobile device

• Quick access to byte sized and chunked information for just in time learning

• This was interesting. I learned a lot about emergency medical care and some things about NGOs

• Overall I thought the information in the tool was very well chunked. I didn't have to dive too deep into anything or scroll too much

• The training modules were very well put together• I tried several ways to take the survey but most of the selections were greyed out in

which only the “prefer not to say” was answered• The mission packs did not seem to work. I can open it and it gives me a list but nothing

opens• Mission tools should be tailored for different regions vs. a generic list; especially, in light

of the immunizations section56


USER-FEEDBACK

• Information is at my fingertips and a tool that is always there when I need it

• I envision the mobile device will be a big part of education in the very near future

• I think the application and the contents are very useful for the purposes that have been developed

• I am a big believer in the potential for mobile devices for military training, keep up the good work

• I just want to say that I am impressed! The videos are clear, as are some of the medical procedures!

57


USER-SUGGESTIONS

• Mission tools should be tailored for different regions vs. a generic list; especially, in light of the immunizations section

• Some users had difficulties submitting evaluation forms

• Request for more information in app

• More off-line video (vs faster install time)

• Use GPS to better target content

• App still needs a little work; great concept

• Would like more interactivity

• A combination of smart phone and larger screen apps might be ideal with real learning on the larger screen and look-up and refresher learning on the smartphone

58


CONCLUSIONS

59

• A suitable representation of the population (e.g., professional expertise, countries, etc)

• “BYOD” did not present any obstacles in the evaluation

• Post-MoLE Questionnaire confirms international partners intend on developing mobile capabilities

• Data worksheet will be cleansed to ensure no identification among volunteers and nations

• Technology works – where is the data to show effectiveness?

mole testing evaluation final report-out documents/mole testing eval… · the testing &...

Documents