basic iraqi survival (bis) language guide california state...

1 Assessment and Evaluation

Basic Iraqi Survival (BIS) Language Guide

California State University Monterey Bay

Selwa Alkadhi

Magdi Ahmed

Assessment and Evaluation IST622

Bude Su, Ph.D

May 2009


Contents

Introduction .................................................................................... Error! Bookmark not defined.

Methodology ................................................................................. Error! Bookmark not defined.

Rationale and Target Audience .................................................. Error! Bookmark not defined.

Prototype Design ........................................................................ Error! Bookmark not defined.

Learners ...................................................................................... Error! Bookmark not defined.

Study Conditions ........................................................................ Error! Bookmark not defined.

Process ....................................................................................... Error! Bookmark not defined.

Entry Conditions ........................................................................ Error! Bookmark not defined.

Results ......................................................................................................................................... 8

Instruction .................................................................................. Error! Bookmark not defined.

Outcomes ................................................................................... Error! Bookmark not defined.

Recommendations ...................................................................... Error! Bookmark not defined.

Summary ........................................................................................ Error! Bookmark not defined.

Appendices ..................................................................................... Error! Bookmark not defined.

Appendix A --Informed Consent ............................................... Error! Bookmark not defined.

Appendix B – Module Assessment Instrument ........................ Error! Bookmark not defined.

Appendix C – Pre-Test Startification Survey ............................ Error! Bookmark not defined.

Appendix D -- Raw Data Table#1 ............................................. Error! Bookmark not defined.

Appendix E -- Raw Data Table#2 ............................................. Error! Bookmark not defined.


I. Introduction

The manner in which technology is integrated in instructional design of language

development often impacts the efficiency and effectiveness of relaying the new

information to students. Prior to this MIST program, the designer had relied on

conventional means to deliver language tools to students through face-to-face

instruction. The purpose of this evaluative paper is to assess the extent at which

technological-based teaching achieves instructional goals in comparison to teaching

approaches that couple both technology and conventional student and instructor

interaction. This is especially relevant to the teaching/learning atmosphere at the CE

(Continuing Education) program. The students depend highly on face expressions,

gestures, and body language in acquiring enough survival language skills to succeed

in their military missions. The goal of the study is to compare the extent at which the

prototype “Basic Iraqi Survival (BIS) Language Guide” achieves its instructional

objectives with or without the inclusion of student instructor interaction.

II. Methodology

Rationale and Target Audience

The target audience of the program is military personnel who are being deployed

to Iraq within a month of completing the given course. The typical class size varies

from 15-30 students. There exists a diversity in students characteristics in terms of

age (ranging from 21-55), rank, educational backgrounds, and social backgrounds.


The prototype is part of a program that focuses on survival language and culture

familiarization in a condensed and fast-paced presentation. The students are military

personnel who have limited training or no training in either the Arabic language or

the Arabic culture. They may or may not have had exposure or direct contact with the

Middle East; its language or its culture. The LFAST (Language Familiarization and

Area Studies Training) is often the primary means of exposing them to this type of

information in a manner that prepares them for their specified objective when

deployed. Despite the dire need for such exposure and familiarization, the curriculum

is terse and limited to between eight and thirty hours of instruction. Hence, it is not a

surprise that the students are overwhelmed with the large amount of vocabulary,

phrases, and cultural elements that they must retain in a given time. Students often

memorize provided phrases and vocabulary without a notion of when and how to

manipulate these language tools in real situations. The new information does not

cohere with any pre-existing knowledge. Rather than establishing these vocabulary

lists as the base of their developing knowledge of the survival language, this module

provides the students with a more practical base on which the previously provided

vocabulary lists and phrase lists can be supplemented. These students receive

booklets that they use to find vocabulary needed to communicate. The military’s duty

is mainly to train and/or guide the Iraqis in different areas of need. Hence, simply

memorizing words and/or phrases alone will not be sufficient for specific military

objectives. When it comes to grammar, students have a difficult time making sense of

new knowledge attained. Sentence structure is most efficiently learned when it is


delivered in context of situations during which students can apply their new learned

information.

Students need to be given the opportunity to apply basic new language skills to their

everyday lives before they relay these skills to the complex situations of their military

profession. A basic foundation for everyday conversation and language usage is vital

to build upon.

The manner in which new knowledge is presented plays a pertinent role in the extent

at which the students will grasp and appreciate their newly acquired language skills.

Demonstrating the manner in which their new language abilities can be applied is an

essential means of motivating them to consistently develop their skills. With regard to

language learning, the intertwining of culture and language is a valuable motivating

factor to the students. Exposing the students to a variety of representations of the

situations they learn about in the new language fosters recognition learning and will

achieve significant results in the affective domain.

The Prototype

The “Basic Iraqi Survival (BIS) Language Guide” prototype is designed to be

launched as a link to a webpage. Minimal computer skills required to launch the

prototype

Tryouts and Sample Populations

A pool of fifty randomized students is selected for the experiment in accordance

to the characteristics sought in the target population. As an attempt to stratify our

population, these fifty students are provided a pre-test. The purpose of the pre-test is


to eliminate any confound that may alter our results or what we are actually testing.

For example, characteristics such as prior knowledge of the language, prior

deployment experience in the Middle East region, prior experience with learning

languages, etc. can influence the extent at which the prototype has on instructional

efficacy. Such characteristics are equally stratified among the two blocks of samples

which are (sample 1) students who utilize the prototype only and (sample 2) students

who utilize the prototype coupled with instructor interaction. Once the two samples of

25 students are structured, the experiment proceeds to test the effect of student

instructor interaction on the results of the prototype in achieving its objectives. The

variable condition in the two samples is the presence or lack of an instructor during

the 30 minute prototype. While close observations are made to assess student-student

interaction, student-teacher interaction and student-module interaction especially

during the instructor's presence in class, the most valued tool in evaluating our

subjects is through a comparison of the sample’s performance on their post-tests in

comparison to that in their pre-tests. These samples ultimately play the role of the

critics/analysts of the prototype and whether or not it must rely on instructor

intervention to most effectively reach its instructional goals.

Domains Evaluated

The pre- and post tests for the evaluation will address the user’s knowledge in both

the cognitive/knowledge and cognitive/application. The questions featured in the pre-

tests and post-tests address the domains according to the following:


Table #1: Pre-tests and Post-tests Domains

Topic/task

Objective

Domain and level

Confirm whether or not

the student has had prior

Arabic language

instruction.

to accurately assess the prototype’s

effect on student learning as well as

for stratification purposes. (survey

only)

Cognitive/knowledge


the student has had prior

Middle East deployments.

to accurately assess the prototype’s

effect on student learning as well as

for stratification purposes. (survey

only)

Cognitive/knowledge


the student is bilingual,

trilingual, etc.

to stratify those students who are more

experienced in language learning.

(survey only)

Cognitive/knowledge

Identify feminine and

masculine forms of Arabic

objects.

users will be able to identify

appropriate feminine and masculine

designations for objects.

Cognitive/knowledge

Identify correct use and

placement of pronouns in

varying sentence forms.

users will be able to identify the

correct designation of pronouns in

sentences.

Cognitive/knowledge

Identify correct sentence

structure in imperative

form sentences.

users will be able to identify correct

imperative sentence structure.

Cognitive/knowledge

Identify definitions of

vocabulary.

users will have a broadened pool of

new basic and practical vocabulary.

Cognitive/knowledge

Identify vocabulary within

common relationships. Ex.

Antonyms, synonyms, etc.

a learner will be able to identify

vocabulary in terms of relationships.

Cognitive/knowledge

Distinguish between the

sounds of letters that are

found in both the Arabic

and English language.

a student will be able to identify and

pronounce different sounds of the

Arabic alphabet.

Cognitive-application


Construct basic survival

sentences.

a student will be able to construct

basic survival sentences using newly

learned vocabulary.


Ability to change forms of

pronouns to feminine,

masculine, plural,

singular, etc.

a student will be able to direct

commands to different pronouns by

changing the forms of pronouns.


III. Results

Upon the start of this trial, our hypothesis was that students who are in the sample that is exposed

to both the prototype and the face-to-face instruction will demonstrate more improvements in

their post-test results in comparison to their pre-test results. This conclusion would be measured

by a quantitative comparison of percentages attained on the pre-test and post-test of the two

sample groups. In such a trial, we recognized that it is particularly vital to stratify our population

to be tested in order to get rid of confounds that may affect a change in test results besides the

variable we are actually testing- the instructors influence on learning when coupled with the

prototype. Essentially, this trial not only tested the efficacy of the prototype itself, but also its

dependence on face-to-face instructor interaction for optimal learning results.

A pre-test was given to all 50 students and results were utilized to block and stratify the pool of

fifty students into two samples of 25 students each. In reference to Table #1.

Entry Conditions

The average scores attained by all 50 students in the pretest is 18.4%. The score represents

students’ performance prior to exposure to the prototype.

Most of the students in the population had no history of knowledge of the Arabic language.

Some students may have demonstrated a score higher than the average due to circumstances


unrelated to our trial. Hence, the population was stratified so that students who have been

deployed and have had exposure to the language are evenly distributed among the two samples.

In assessing the effectiveness of the prototype itself.

Observing the results, we noticed that although some students received prior instruction they still

scored low to very low scores because the kind of

Arabic instruction they had received was

Modern Standard Arabic (MSA) which is

considerably different from the Iraqi dialect

especially in pronunciation.

Some students with no measurable background,

scored higher than zero on the pretest. Upon

further discussion with them, we found that they had learned some Arabic words or phrases from

either their casual Arabic friends, from following up with the media on the Iraqi war, or from

their fellow soldiers who have been deployed. However, as the results show, the scores were not

significant.

Outcomes

The test scores of all 50 students with or without an instructor demonstrate that the prototype

improved their awareness of the language and culture. Looking at the correlation results we see

that students did considerably well in general. Scores showed that the prototype was effective

even when used alone. This is exhibited in the graphs #1 and #2 shown below.


The average score of the posttest for students without the instructor and the average for the

posttest score with an instructor are 73.4% and 82.16% consecutively. Hence, it is evident that

there is an improvement in teaching and learning.

Graph #1 Graph #2

The main objective of this trial was to evaluate the extent of the effectiveness of this

prototype with the inclusion of face to face instructor interaction. Although the prototype

achieves its objectives alone, our study shows that the students who utilize the prototype in

conjunction with instructor exposure did notably better.

Mean 73.04 Standard Error 2.3017964 Median 73 Mode 75 Standard Deviation 11.508982 Sample Variance 132.4566667 Kurtosis -0.101805279 Skewness 0.078180312 Range 45 Minimum 50 Maximum 95 Sum 1826

Count 25

Mean 82.16 Standard Error 2.387243878 Median 82 Mode 79 Standard Deviation 11.93621939 Sample Variance 142.4733333 Kurtosis 1.719178611 Skewness -1.037186053 Range 49 Minimum 51 Maximum 100 Sum 2054 Count 25


Graph #3 Comparison between Pre and Post Test: Prototype Without Instructor

Graph #4 Comparison between Pre and Post Test: Prototype With Instructor

Recommendations

Conducting this tryout has proved to be useful in providing tips that may enhance the learning

process for the deploying troops. It is vital for the military personal to interact with an instructor

especially if only to understand the culture of the language they are learning.


This tryout has helped us realize that the prototype is only a catalyst to the learning process and

that instructors will always have a pertinent roll in language teaching.

t-Test: Two-Sample Assuming Equal Variances

Post-Test Score % Without

Instructor Post-Test Score % with

Instructor

Mean 73.04 82.16 Variance 132.4566667 142.4733333 Observations 25 25 Pooled Variance 137.465

Hypothesized Mean Difference 0

df 48 t Stat -2.750133502 P(T<=t) one-tail 0.004187424 t Critical one-tail 1.677224197 P(T<=t) two-tail 0.008374848 t Critical two-tail 2.010634722

Observing the p-value of the T-Test for scores of Post-Test with and without instructor we can

conclude that the null hypothesis is discredited and our original hypothesis is supported.

Comparison between Post Test Scores for taking Prototype with and without Instructor


Appendix A

CONSENT TO PARTICIPATE IN RESEARCH

Title of Program: Basic Iraqi Survival Language Guide

You are invited to participate in a research study conducted by Selwa Alkadhi, student in IST 695, Educational

Assessment and Evaluation at California State University, Monterey Bay and instructor at the CE-LFAST

Program at the DLIFLC in Monterey, CA.

The purpose of this research is to evaluate the effectiveness of an instructional prototype on Language

familiarization and cultural training to evaluate the effect of instructor inclusion in a module presentation. You

were selected as a prospective participant in this study because you are currently a sample of a large population of

Iraqi Survival language learners. No particular level of computer experience is assumed or required for this study.

Taking part in this project is entirely up to you. You can choose whether or not to be in the study. If

you volunteer to be in this study, you may withdraw at any time without consequences of any kind. You

may also refuse to answer any questions you do not want to answer and still remain in the study. The

investigator may withdraw you from this research if circumstances arise that warrant doing so.

If you decide to participate in this research, you will be asked to do the following: Answer survey questions (three

general demographic questions, in addition to language based questions. The process will take between 30

minutes to 45 minutes to complete.

You are not being asked to sign any portion of the evaluation or to identify yourself in any way. Any information

that is obtained in connection with this study and that could be identified with you will remain completely

confidential. After the results are compiled, the paper surveys will be promptly destroyed.

If you want to know more about this research project or have questions or concerns, please contact Selwa at

[email protected]

You will get a copy of this consent form. Thank you for considering participation.

Sincerely,

Selwa Alkadhi

Consent Statement

I understand the procedure described. My questions have been answered to my satisfaction and I freely agree to

participate in this study. I know what I will have to do and that I can stop at any time.

I have been given a copy of this Consent Form.

Signature Date


Appendix B

Module Evaluation

The purpose of this evaluation is to assess the effectiveness of the presented material throughout

this course and to make necessary revisions both in the content and structure of the course. Your

feedback is valuable to us and will be utilized to make necessary revisions to improve this

course. Please do not hesitate to make any comments or suggestions

1. Strongly Agree

2. Agree

3. Neutral

4. Disagree

5. Strongly Disagree

1 2 3 4 5

1．The training meets the course objectives.

2．The course materials were well prepared and

organized.

3. The concepts were delivered in a clear manner.

4. The length of the program was adequate in order to

meet goals.

5. The tasks and/or activities were well organized and

relevant to each subject.

6. Training units were detailed thoroughly.

7. The organization and layout of the design were easy

to follow.

8. Technology was well integrated into the course.

9. I will be able to apply what I learned here to my job.


Appendix C

Pre-Test Stratification Survey

Please circle a suitable answer to the following questions:

1- Have you previously been deployed to the ME region during your military career?

2- Have you had any previous instruction in the Arabic language?

3- Have you had any other language instruction?

4- Have you been to the ME region for reasons other than the military?

Appendix D

Table#1 Raw data

Appendix E

yes no

yes no

yes no

yes no

Student Prior

Deployment Prior

Instruction Other

Languages Pre-Test Score %

Post-Test Score % Without

Instructor

1 no no no 0 75 2 yes no no 5 72 3 no no no 8 50 4 no no no 0 63 5 no no no 0 62 6 no no no 0 70 7 no yes no 25 68 8 no yes yes 55 87 9 no no yes 38 80

10 no no no 0 71 11 yes no no 10 68 12 yes no no 23 74 13 no no no 5 63 14 yes no no 0 75 15 no no no 0 63 16 yes no no 30 78 17 no yes yes 68 85 18 yes yes no 58 93 19 no no no 0 78 20 yes yes yes 89 95 21 no no no 0 52 22 yes no no 5 75 23 yes no no 8 73 24 no no no 0 65 25 yes yes yes 75 91


Table #2 Raw data

Student Prior

Deployment Prior

Instruction Other

Languages Pre-Test Score %

Post-Test Score % with

Instructor

1 no no no 0 53 2 no no no 0 51 3 no no no 0 82 4 yes no no 6 79 5 no no no 8 77 6 no no no 0 76 7 yes yes yes 73 96 8 yes yes no 26 92 9 no no yes 15 98

10 no no yes 10 84 11 yes no no 15 77 12 no no no 0 81 13 no no no 0 79 14 yes no no 35 88 15 yes yes yes 78 100 16 no yes no 36 93 17 yes no no 12 86 18 yes yes yes 68 94 19 yes no yes 16 94 20 yes no no 5 75 21 no no no 0 80 22 no no no 0 79 23 yes no no 10 83 24 yes no no 5 85 25 no no no 0 72

basic iraqi survival (bis) language guide california state...

Documents