running head: assessing second language skills: test

Running head: ASSESSING SECOND LANGUAGE SKILLS: TEST PROPOSAL 1

Assessing Second Language Skills: Test Proposal

Bridget Schuberg

Colorado State University

ASSESSING SECOND LANGUAGE SKILLS: TEST PROPOSAL 2

Assessing Second Language Skills: Test Proposal

Assessments and tests have become an essential component of Western education in the

twentieth century. In fact, the use of objective tests is often considered one of the most influential

and crucial steps in the “modernization” of education (Spolsky, 1978, p. v). Though they are used

for a variety of reasons, two of the most common are to provide teachers and researchers with

information about effective forms of instruction and to help in decision-making processes related

to student ability (Miller, Linn, & Gronlund, 2000; Alderson, 2000). The ultimate goal of a test is

to be able to derive an inference from a student's performance on a test task about his/her ability

to perform a real-life task (Bachman & Palmer, 1996). The need for authenticity, or

correspondence between the test task and a real-life task, is particularly central in determining

the usefulness of a test.

I previously served as a project assistant to the development of CSU's Kitchen Resource

Guide: An Introduction to the Kitchen for Non-Native Speakers of English. To complete this

resource guide, project leader Andy Fuller analyzed the entire recipe and instruction book used in

Colorado State University kitchens, and identified the most frequent off-list, kitchen-specific

vocabulary with which English language learners (ELLs) would most likely lack familiarity.

Using this information, the project team created a picture dictionary separated into different

categories: general kitchen; food and ingredients; equipment; time, temperature, and weight;

holding, serving, and storing, and cooking and preparation. Many of the non-native workers who

benefited from the creation of this dictionary also have taken part in the semester-long

Workplace English Training (WET) program at CSU, which aims to provide international

workers in CSU kitchens with the language skills they will need to succeed in their jobs. Due to

both my participation in the development of the kitchen resource guide and my commitment to

intern for the WET program in the future, I have an interest in creating a test that would be useful


to this same group of ELLs. In this particular context, vocabulary is the most necessary to assess,

as a majority of the tasks these workers must perform require an understanding of recipes they

are given to follow.

Vocabulary knowledge is often assumed to be the foundation of language ability. While

competence in other language areas (e.g. grammar, phonology) is not necessarily a prerequisite

in either understanding or being understood in a language, vocabulary knowledge is absolutely

critical in both receptive and productive tasks. Though for a while, vocabulary tended to be

relatively ignored in favor of a greater focus on grammatical and communicative competence,

the view of the role of vocabulary as an index of language ability has been renewed once again

(Read, 1997).

Due to the relative ease with which they can be written, vocabulary items have been “one

of the most common components in objective language testing” from the beginning (Read, 1997,

p. 99). Popular test types for assessing vocabulary include multiple-choice tests, cloze tests, in

which words are deleted from a reading passage at either a fixed ratio or based on their lexical

content, and tests which require test-takers to match vocabulary words with short definitions,

such as Nation's Vocabulary Levels Test (Nation, 1990, pp. 261-272). The scores from such

objective test items are also desirable as they are considered to have a high rate of test-retest,

inter-rater, and intra-rater reliability (Read, 1997, p. 99).

However, testing vocabulary is not without difficulty. Some argue that vocabulary

knowledge is not truly indicative of language ability, as it is only a “lower-order enabling skill'

(Read, 1997, p. 104), and as such, is not as valuable as other skills in the development of

communicative competence. Furthermore, certain scholars such as Nattinger and DeCarrico

(1992, as cited in Read, 1997), believe that groups of words and lexical units play a more

important role in spoken and written discourse than do individual words in isolation.


Additionally, context plays an important role in the definition of a word. Therefore, to present

vocabulary items in a more developed discourse context decreases the chances that a “pure

measure of vocabulary” untainted by reading ability has been measured (Read, 1997, p. 105).

Consequently, the two abilities serve complementary roles at times. For the purposes of my test

and the vocabulary my test will include, the context is already established and known for the

test-taker. Because I will be testing specialized vocabulary that has been found to be highly

frequent in CSU kitchens (CSU’s Kitchen Resource Guide, 2012), the necessary step of selecting

target words to both teach and test is already complete.

Organization

In the rest of this paper, I will provide a description of the test, including its purpose, its

type, how scores from the test are to be interpreted, the TLU domain the test is meant to

represent, the definition of the construct my test aims to assess, how the test is designed, the test

task, and how the items were written. Next, I will discuss how the procedures involving a

hypothetical piloting of the test. Then, I will examine the types of analysis that would be

appropriate if I were to administer the test. Lastly, I will look at the types of data that would help

me argue for the validity of my test.

Description of the Test

Test Purpose

The language purpose of this test is to determine whether or not the test-taker has control

of the general skills necessary to succeed in his/her work environment (in a CSU kitchen or

dining hall). More specifically, the test-taker will need to demonstrate that he/she is able to

recognize words related to food, ingredients, equipment, time, temperatures, weights, holding,

serving, storing, cooking, and preparation. More broadly, this test will facilitate the process of

transitioning into a new job for English language learners.


Inference. Because the vocabulary test-takers are asked to translate constitutes over 90%

of the words they will encounter in the cookbooks, one should, in theory, be able to infer that the

test-takers who pass the test have an understanding of a majority of the words needed to

complete the tasks demanded of them and will therefore be able to complete the task of preparing

food effectively.

Use. As defined by Stoynoff and Chapelle (2005), a score “represents a summary of an

examinee's performance across one or more tasks on a test” (p. 18). The scores from this test

should be used only for placement purposes—to decide whether an employee possesses the

language skills necessary to fulfill his/her workplace responsibilities, or whether he/she needs to

enroll in the Workplace English Training program sponsored by Colorado State University. The

Workplace English Training program is a 2 hour/week, 10 week course offered for free to

English language learners every semester.

As Bachman and Palmer (1996) assert, it is extremely important that the inferences meant

to be made on the basis of test scores is clearly established prior to administering a test. It is

critical that this test not be used for selection of employees; to do so would greatly diminish the

chance of non-native speakers being selected. Furthermore, to use this test as a selection aid

would create a high-stakes testing environment, increasing the anxiety of the test-taker and

thereby potentially reducing both the construct validity and test-retest reliability.

Impact. A test's impact is defined as the positive and/or negative effects a test has on the

individuals who take the test, a society, and on educational systems (Bachman & Palmer, 1996).

On a micro-level, I believe the potential negative effects of my proposed test would be

minimal for many reasons. First of all, the test is merely a placement test, not a selection test;

failure to pass the test does not result in unemployment, but merely in a short course that comes

at no financial cost to the test-taker. Completion of the course is not a prerequisite to


employment; in other words, the learner does not need to have finished the Workplace English

course before beginning his/her job. Furthermore, the course does not assume any English

proficiency; therefore, test-takers need not worry about his/her English level as they might in a

typical English classroom. Finally, the aforementioned CSU Kitchen Resource Guide is available

to at least all employees who do not speak English as their native language; it is from this

resource guide that the majority of test items are based. I believe that so long as the test-takers

are made aware of the low-stakes situation of the test, the impact of this test on individuals

should not be negative.

I also believe this test could have positive effects at the macro-level, as it could result in

greater employment for non-native speakers of English, benefiting both traditionally

underprivileged groups and thereby the larger Fort Collins community. Moreover, an increased

participation in the Workplace English Training program could provide more opportunities to

aspiring teachers, while helping both CSU educators and employers to more precisely identify

specific areas of need for language learners, particularly regarding workplace English.

Type of Test

Because this test aims to measure what a person already knows about workplace

language and is not based on any particular syllabus, it is by definition a proficiency test. This

test is based on the belief that reading ability and vocabulary knowledge are necessary

components of working in a CSU kitchen, and that proficiency in these areas increases an

employee's chance of success in the workplace.

Interpretation of Scores

Test takers will not be scored according to their relative knowledge in comparison to each

other or to an external reference group, as no selection or admission decisions will have to be

made on the basis of the test results. Instead, this test is designed to “provide information on the


specific knowledge or skills possessed by” an employee (Mitchell, 1982, p. 2), and is therefore a

criterion-referenced test. Seventy-five percent of items will need to be answered correctly in

order for the test taker to be eligible to opt out of Workplace English Training.

Target Language Use Domain

Language test developers must have a specific purpose and group of test takers in mind

when deciding to create a test if they wish it to be useful. Most importantly, they must specify a

“situation or context in which the test taker will be using the language outside of the test itself”

(p. 18) to make the tasks truly useful. This outside situation is commonly referred to as the Target

Language Use, or TLU, domain (Bachman & Palmer, 1996).

The TLU domain that my test aims to represent is the workplace environment,

particularly that of a kitchen or dining hall. The test will attempt to predict the test-taker's ability

to successfully complete one TLU task: to prepare food according to a recipe book written in

English while simultaneously following health and sanitation procedures (see Appendix B).

Definition of Construct

Explicitly delineating the “precise nature of the ability we want to measure,” also known

as defining the construct, is a necessary step if we wish to be able to interpret test scores and

make inferences about learner ability (Bachman & Palmer, 1996, p. 89). In relation to

organizational competence, grammatical knowledge in the form of lexis will be tested, as the all

of the test items will be focused on identification and recognition of vocabulary related to food

and the workplace. Vocabulary breadth, not depth, will be the focus of this test. Familiarity with

all English phonological and graphological units will be assumed, as will minimal topical

knowledge of how to prepare food. Strategic competence, sociolinguistic knowledge, textual

knowledge, and writing ability will not be tested.


Design of Test

A Table of Specifications (TOS) was used in the development of the test to help with the

sampling of items and to align content with objectives (Miller et al., 2009). Because the final

vocabulary test called for only one task, the TOS indicates that this task composes 100% of the

test. The Table of Specifications used for this test (Appendix A) delineates the objective

regarding the vocabulary used in the test questions according to their part of speech. This hopes

to be a useful distinction for those who will have access to the scores (most likely those running

the Workplace English Training program); if a test-taker misses mostly verbs, the emphasis of

the training should be on general kitchen actions; if the employee misses mostly nouns, trainers

should attempt to focus on products and ingredients; if they seem to have the most trouble with

adjectives, they should receive more descriptions of products; if the test-taker misses questions

asking about adverbs, specific manners of carrying out processes should be emphasized; lastly, if

he or she mostly misses questions regarding prepositions correctly, he/she should receive more

instruction regarding locations. The content column of the TOS uses the K1, K2, and K3 word

lists to organize these items. (i.e. their classification according to their frequency in English).

Number of parts. There is only one part to this test. As the principle jobs given to

English language learners in CSU kitchens are food preparation and food storage, and because

employees are expected to use the instructions on how to adequately perform these tasks that are

written in cookbooks and handbooks, vocabulary knowledge is the fundamental basis of the

employee’s ability to fulfill his/her workplace tasks. Therefore, only vocabulary knowledge is

tested.

Sequence of parts. The items are organized according to the frequency with which they

appear in the cookbooks that all employees must use when preparing food (from high to low

frequency). Should test-takers run out of time, if they complete the items in order, the questions


aiming to assess their knowledge of the most high-frequency words (and therefore the ones that

are likely to be the most important to successful task performance in the TLU domain) will have

been completed first.

Total time to administer test. The total time needed to administer this test is 95 minutes.

Five minutes will be set aside before the test is administered for test-takers to read the directions

and ask any pertinent questions they may have about the instructions, purpose, scoring, etc.

Then, test-takers will be given 90 minutes to complete the test.

Description of Test Task

Instructions. The instructions are written on the test in the native language of the test-

takers just before the actual test items. This is to eliminate any potential confusion about how to

mark the correct answer, what the purpose of the test is, how much time they will have to take

the test, and how the test will be scored. As the test contains only one task, the general directions

apply to the entire test. Following the instructions, an example will be given for test-takers to

follow should they still not understand.

Input. Because the purpose of this test is to measure the employee’s ability to function in

his/her place of work, the test does not ask the test-takers to define English words in terms of

more English words; rather, test-takers must select the most accurate English translation of the

word in their native language from a set of provided choices. As the overwhelming majority of

employees in need of workplace English training have had an L1 of Spanish, the alternatives are

written in Spanish.

The focus of this test is on meaning, since these employees will be asked to perform real-

life tasks based on their comprehension of the recipes they read in their cookbooks. Although

dictionaries explaining kitchen-specific English words can now be found in all CSU kitchens,

these were formed using Compleat LexTutor to target topic-specific, off-list vocabulary that


would be considered essential to the successful completion of the tasks performed by the

employees. While it is important to prioritize the words these employees are least likely to know

but very likely to use frequently, it cannot be assumed that the non-native employees have

knowledge of the meaning of all the K1 or K2 words used in the cookbooks. Moreover,

employees must already understand certain words in English in order to be able to use the

information provided in the CSU kitchen dictionaries all employees are provided with.

Furthermore, cooking and baking are highly specific processes; misinterpretation of any word

given in a recipe could potentially significantly alter the end result. Knowledge of the vocabulary

words on this test, when combined with the words present in the kitchen dictionaries that the

employees have access to, comprise almost 90% of the total words found in the cookbook.

Expected response. Because the test-takers are at a beginning level in their English

language learning, the output required in response to the input is minimal; test-takers are merely

responsible for circling the best answer. As participants are not required to create his/her own

language, the design of the test reflects the fact that the test-takers typically do not have to create

their own language to complete their TLU tasks.

Relationship between input and response. As the input is the same for all test-takers,

this task is non-reciprocal; there is no interaction or feedback between the test-takers and the

proctor/s. Only a sentence of context needs to be processed in order to answer the question

correctly; therefore, the scope of the relationship is narrow. Finally, as the input supplies all the

information needed for the test-takers to respond correctly, the relationship between the input and

the response is direct.

Scoring. Because this is an objective test in which test-takers can choose the correct

answer from a set of alternatives, no scale or rubric is needed. Each correct response will earn

one point, while an incorrect response will not earn any points. The total score possible is 90


points; test-takers must answer a total of 68 items correct for a total of 68 points (75%) in order

to pass. After an administration of the test, the appropriateness of this cut-score could be re-

assessed by analyzing whether those who were considered “masters” of the test (i.e. those who

passed) are also considered masters of the TLU task.

Time for task by test-takers. Test-takers will be given 90 minutes to complete the test.

The amount of time allotted was chosen according to the suggestion that beginning-level

language learners (the target audience) typically are capable of answering around one multiple-

choice question per minute.

Item-Writing and Editing

The CSU Kitchen Resource Guide includes off-list, kitchen-specific terminology.

However, I wanted to create a test that would determine whether employees understood the most

frequent K1, K2, and K3 words as well. In order to decide which vocabulary words should be

included, I uploaded the .txt file of the cookbook workers at Corbett Kitchen are required to use

when preparing, serving, holding, and storing food to Compleat Lex Tutor so as to determine

which words that were not kitchen-specific (and were on the K1, K2, and K3 lists) occurred most

frequently.

Next to each vocabulary item on the test is an English sentence from the actual Corbett

Kitchen cookbook to provide the test-takers with context for the word they must translate. In

coming up with distracters, I often chose English words which look or sound similar to the

correct answer translations based on inappropriate senses of the word. Alternatives are listed in

alphabetical order. The proper letter choice does not appear more than three times in a row. When

I finished constructing the items, a native Spanish-speaker looked at my answers to ensure the

alternatives were logical, and that the answers were obvious only to those who truly knew the

correct translations.


Pilot Test Procedure

Participants

The participants in a piloting of this test would have an L1 background of Spanish, in

which they would most likely be of advanced proficiency (at least verbally). In order to be in

accordance with the target demographic of the actual test-takers of operational administrations of

the test, many of them should be immigrant women in their 30s and 40s. The participants of the

study may or may not have had prior exposure to the material that will be assessed to align with

the background of the takers of the operational test.

Administration

The WET placement test could be piloted at before the beginning of the upcoming

semester (Spring 2013), as the greatest number of hires in the CSU kitchens occurs at the

beginning of each semester. This is feasible for both test administrators and test scorers; the

WET program does not begin until approximately the fifth week of each semester, allowing

ample time for the hiring, test administration, test scoring, item analysis, and item revision

processes.

The WET program takes place in the Palmer Center, the building dedicated to Housing

and Dining Services, as this sector is the largest employer of non-native speakers on the CSU

campus. A room in the Palmer Center is therefore where the test will be piloted, as the

administrators in this building work in conjunction with the WET program and a room is

typically already designated for these classes.

Prior to the administration of the test, the test-takers will be informed about how the

scores from the test will be used. They will be given five minutes to read the directions and ask

any pertinent questions. Then, they will be given 90 minutes to complete the test.

Scoring Procedures


As the test is composed solely of multiple choice items, no special training is required to

to administer, proctor, or score it. After the 90 minutes allotted for the test has passed, the tests

will be given to the coordinator of the WET program, to score using the answer key. After she

determines how many each participant has answered correctly, she will fill out a score report

form (Appendix D) to be handed back to the student which will indicate whether or not he/she

has surpassed the cut score (in this case, 75% or 68/90) necessary to test out of the WET

program.

Proposed Test Results

Item Statistics

Item statistics are necessary for making decisions about the effectiveness of certain items

so that a test may be revised to be more useful to all stakeholders. The basic purpose of a

criterion-referenced test such as this one is to determine the amount of material that test-takers

know or can do (Brown, 2003) for the purpose of making decisions with the results. Item

difficulty would be calculated by indicating the percentage of students who got each item right.

An item’s discriminating power could be determined by calculating the numbers of test-takers

who got each question right in both the upper and lower groups. Using the same formula for item

discrimination, the effectiveness of the distracters of each item could also be analyzed. Using the

B-index, one could also determine how well each item contributes to the pass-fail decision made

with this particular test. This information could also help to determine if the cut score for this test

is appropriate or if it needs to be revised.

Descriptive Statistics

Descriptive statistics describe a set of scores; they help summarize performance so that

interesting trends may be seen. They also serve as the basis for more powerful analyses (Miller et

al., 2009). To calculate central tendency (e.g. how test-takers group), I would find the mean, or


the average score of data, by taking the sum of all scores and dividing it by the total number of

points possible; then, I would plot the participants’ scores on a frequency polygon (line graph).

To calculate variability, I would determine the standard deviation to see to what degree each

score deviates from the mean. Ideally, two thirds of the cases will have scores between the mean

minus one standard deviation and the mean plus one standard deviation (Miller et al., 2009).

Reliability and Standard Error of Measurement

Reliability is a necessary component of an assessment, as it contributes to an

assessment’s validity. Reliability measures attempt to reflect the generalization we can make

across circumstances, tasks, and raters. In order to collect evidence of the internal consistency of

this single assessment, the split-half method could be used to determine to what extent the items

in the assessment measure similar characteristics (Miller et al., 2009). Test-retest reliability and

inter-rater reliability are not relevant to this test. Lastly, I would calculate the standard error of

measurement (SEM), which represents a variation of scores around the “true score” (i.e. if the

test were free from error, what would a particular test taker’s score be?) Calculating the SEM is a

way to gauge our confidence with the interpretation of an assessment; the higher the SEM, the

less precise or reliable the assessment may be.

Description of Masters/Non-Masters

Criterion-referenced tests typically involve the classification of test-takers as either

masters or non-masters (those who pass or those who fail). The agreement coefficient can be

used to show the proportion of examinees who are considered masters or non-masters on

multiple administrations of the same test. However, the proposed test would only be piloted

once; therefore, the chart presented by Subkoviak (1998, p. 3) for calculating the agreement

coefficients from a single administration (provided one has the z score and the reliability


coefficient r) is particularly helpful in determining the overall consistency of those test-takers

who would have supposedly mastered the objectives.

Discussion

Critique of Item Performance

To argue for validity, the individual items on my test would have to be well-constructed.

In order to determine this, I would calculate each item’s difficulty and discriminating power.

Items on this test should ideally have a difficulty between .6 and .8 as there are four alternatives

(Miller et al., 2009). For an item to be classified as discriminating positively, it would have to be

answered correctly by the test-takers in the upper group than those in the lower group (Miller et

al., 2009). The formula for item discrimination can also be used to determine the effectiveness

of distracters; if the distracter attracts more students from the lower group than the upper group,

it can be deemed effective. Lastly, the closer the B-index value is to 1.00, the more it contributes

to the final pass/fail decision; the closer it is to -1.00, the less effective the item is in making

these decisions. Once the test was piloted and these values could be calculated, one could speak

more to teach item’s individual performance, and therefore, its effectiveness.

Evaluation of Test Usefulness

Reliability. The statistics produced from an analysis using either the split-half method or

coefficient alpha could be examined to determine this test’s internal consistency. The closer the

correlation coefficients are to 1, the more reliable the test may be considered. The SEMs for each

score range would also have to be analyzed in determining the test’s reliability; the smaller the

SEM, the higher the reliability. Once the test was piloted and these values could be calculated,

more evidence for reliability could be accrued.


This test is a selection test; therefore, its objectivity contributes to reliability. Its large

number of items (90) for its one construct could also be seen as contributing to reliability (Miller,

Linn, & Gronlund, 2009).

Construct-related evidence for validity. The construct that my test was purported to

measure was the test-taker’s ability to understand and follow a written recipe. Because the items

that the test task is based on are taken directly from the actual material employees use when

completing the TLU task, we may consider the test task to be highly representative of the TLU

domain; it therefore should theoretically have a high degree of authenticity.

In order to argue for a high degree of construct validity, one would have to determine the

success of the employees in this task and then correlate these employees’ scores on this test. In

this way, one could see if the highest-scoring students with their rank in terms of the quality of

their performance in their completion of the TLU task. Additionally, one could compare test

scores between two groups: one that has been deemed unprepared to prepare food in a CSU

kitchen, and one that has been successfully completing the tasks required of them; the scores of

the latter group should be higher than the cut-score set for this test. Along these same lines, one

could administer the same test to the same group of people at two different times: once before

they worked in a CSU kitchen, and once after they have successfully been carrying out their

tasks there; if the scores from the second administration are passing, we might be able to say the

test is highly reliable.

Consequential evidence for validity. As validity is defined as the appropriateness of the

interpretation and use made of assessment results, impact is extremely important to consider

when determining the degree of validity of a certain test use. In order to argue that this test’s

impact contributes to validity rather than hinders it, the aforementioned aspects of construct

validity and reliability would need to be determined; in this way, one could argue that the


assessment scores are consistent, relevant to the decisions that are to be made, and are

generalizable to the TLU task. A questionnaire about the test-taker’s feelings about their

perceptions of the test could be used to collect feedback about the impact of the test on their

attitudes toward their job and toward their language learning. The kitchen managers and those

who work with the international employees could also provide feedback on the performance of

the test-takers who tested out of the WET program in order to determine if they believe the test is

placing workers appropriately.

Practicality. This test may be deemed practical based on the availability of employees to

take the test, the availability of proctors to administer and monitor the test, the cost of printing

and distributing the test, and the ease with which a room may be reserved for the administration

of the test. This information could be ascertained after several attempts to administer the test.

Additionally, the scores of the test are easily interpreted, and the score report form contributes to

clarity.

Achievement of Test Purpose

The purpose of this test, as defined earlier, is to determine whether or not the test-takers

have the vocabulary knowledge necessary to follow a recipe according to the sanitation

guidelines. In order to decide whether or not the purpose has truly been achieved, then, one

would have to run statistical analyses of the reliability and validity of the test items and results

according to the aforementioned methods, and determine whether or not these numbers fall into

the acceptable ranges.

Reflection

The most prominent theme that comes to mind when reflecting on my own test

development process is that I never realized all of the components that are required of a well-

made test. I often just assumed the competence of test developers and those who make inferences


based on test scores, and thus never questioned whether the items truly tested the defined

construct, whether the test was reliable, and whether the use of the test could be considered valid.

However, now that I know the many factors that must be considered when creating and justifying

a test’s use, I will look much more critically at the tests I am given or the tests I may be told to

give to students.

In the future, when I create tests, I know now to ensure that I allow enough time for its

development, editing, piloting, analysis, and revision. I will also pay closer attention to the Table

of Specifications. For this particular test, I felt that the frequency with which the words appear in

the employee cookbook was sufficient justification for the items I wrote. However, the

organization of the content and the objectives sections of my TOS might be seen as weak.

Furthermore, since the items are organized from high to low frequency, perhaps it would be

possible to weight the items differently, as understanding of certain words on the test may be

more significant than others with regard to the TLU domain.


References

Alderson, J. C. (2000). Chapter 7: Techniques for testing reading. In Assessing reading (pp. 202-

270). New York: Cambridge University Press.

Bachman, L., & Palmer, A. (1996). Language testing in practice. New York: Oxford University

Press.

Brown, J.D. (2003). Criterion-referenced item analysis (online). JALT, 7, 18-24. Retrieved from

http://jalt.org/test/bro_18.htm

Miller, M.D., Linn, R., & Gronlund, N. (2008). Measurement and assessment in teaching.(Tenth

Edition). Upper Saddle River, NJ: Merrill, Prentice Hall.

Mitchell, B. (1982). A glossary of measurement terms. (Test Service Notebook 13). NY: Harcourt

Brace Jovanovich, Inc.

Nation, I.S.P. (1990). Teaching and learning vocabulary. New York: Heinle and Heinle.

Nattinger, J.R. & DeCarrico, J.S. (1992). Lexical phrases and language teaching. Oxford:

Oxford University Press.

Read, J. (1997). Assessing vocabulary in a second language. In C. Clapham and D. Corson

(Eds.), Encyclopedia of language and education: Vol. 7. Language testing and

assessment (pp. 99-107). Norwell, MA: Kluwer Academic Publishers.

Spolsky, B. (1978). Introduction: Linguists and language testers. In B Spolsky (Ed.), Approaches

to language testing, (pp. v-x). Washington, D.C.: Center for Applied Linguistics.

Stoynoff, S., & Chapelle, C. (2005). ESOL Tests and Testing. Alexandria, VA: TESOL, Ch. 1

Subkoviak, M. (1988). A practitioner’s guide to computation and interpretation of

reliability indices for mastery tests. Journal of Educational Measurement, 25, 47-55.


Appendix A

Table of Specifications

Text Understanding meaning of vocabulary items in context

# of items

% of items

Descriptions(Adjectives)

General Actions(Verbs)

Objects(Nouns)

Specific Processes(Adverbs)

Locations(Prepositions)

K1 wordlist

15 13 16 11 9 64 71.1%

K2Wordlist

8 7 2 0 1 18 20%

K3 wordlist

2 2 3 1 0 8 8.9%

# of items per section

25 22 21 12 10 90 100%

% of items per section

~27.8% ~24.4% ~23.3% ~13.3% ~11.1% 100%

Items According to Type and Category

Adjectives Verbs Nouns Adverbs Prepositions

K1 wordlist 5, 12, 18, 19, 32, 33, 34, 43, 53, 57, 62, 63, 69, 73, 79

1, 4, 7, 8, 46, 48, 51, 70, 76, 77, 83, 84, 90

3, 16, 24, 25, 27, 29, 30, 47, 49, 65, 66, 74, 82, 86, 87

31, 52, 54, 56, 59, 61, 71, 75, 80, 81, 85

6, 10, 11, 21, 23, 37, 58, 64, 88,

K2 wordlist 13, 26, 38, 41, 42, 50, 55, 60

14, 15, 20, 22, 35, 44, 72

17, 28, 78 9

K3 wordlist 36, 39 40, 45 2, 67 68


Appendix B

TLU Domain Description

TLU Task #1:Using a recipe book to prepare food while following health and sanitation procedures.Characteristics of the setting physical characteristics kitchen Participants employees time of task morning, afternoon, eveningCharacteristics of the input Format Channel visual Form language Language target language (English) Length short sentences Type series of “items” degree of speededness normal speed (~160-180 wpm) Vehicle reproduced language of the input language characteristics organizational characteristics Grammatical full set of graphological features; simple and

frequent vocabulary; statements and commands

Textual pragmatic characteristics Functional instrumental Sociolinguistic topical characteristics food-related or workplace vocabularyCharacteristics of the expected response Format Channel tactile (producing the end food product) Form non-language Language Length Type degree of speededness language of the expected response language characteristics organizational characteristics Grammatical


Textual pragmatic characteristics Functional Sociolinguistic topical characteristics food-related and workplace-related

vocabularyRelationship between input and response Reactivity non-reciprocal scope of relationship narrow directness of relationship direct


Appendix C

Test

Nombre: ______________________________________________ Fecha: _______________

El propósito de este exámen es evaluar su saber del significado del vocabulario básico que necesita para trabajar en una cocina de CSU. Tiene 90 minutos para tomar el exámen. Cada

pregunta será marcado correcto o incorrecto. Los resultados serán usados para determinar si necesita participar en el curso de Workplace English Training.

Instrucciones:Lea cada oración ingles. Después de leer el oración de contexto, trazar un círculo alrededor de la palabra/frase que representa la mejor traducción de la palabra/frase en negrita. No adivine si no

sabe la respuesta.

Por ejemplo: 0. cup Put the liquid in the cup.a. cuencob. ollac. tazad. vaso


1. to need Refrigerate and serve as needed.a. deber b. exigir c. necesitar d. querer

2. recipe Ensure that all preparation equipment and utensils needed for this recipe are cleaned and sanitized.

a. advertencia b. lista c. mensaje d. receta

3. hands Wash hands prior to beginning preparation.a. dedos b. manos d. platos d. utensilios

4. to begin Wash hands prior to beginning preparation.a. cocinar b. continuar c. empezar d. terminar

5. clean Place product in clean sanitized serving pans and cover. a. limpio b. modificado c. nuevo d. sano

6. until Keep chilled at 41F or below until ready to use.a. antes b. después c. hasta d. mientras

7. to add Add lemon juice, water, and salt. a. añadir b. exprimir c. medir d. sacar


8. to keep Keep chilled at 41F or below until ready to use.a. calentar b. congelar c. conservar d. mantener

9. below Keep chilled at 41F or below until ready to use.a. bajo b. más c. por d. sobre

10. when Follow all appropriate sanitation procedures when handling and serving product.a. antes de b. cuando c. después de d. donde

11. during Maintain temperature at 140F or above during holding and serving.a. antes de b. durante c. después de d. si 12. ready Keep chilled at 41F or below until ready to use.a. disponible b. listo c. deseado d. necesario

13. internal Cook to internal temperature of 135F.a. alto b. ambiente c. extremo d. interior

14. to maintain Maintain temperature at 140F or above during holding and serving.a. anotar b. mantener c. medir d. observar


15. to reach Cooking time will take from 5-7 minutes to reach 165F internal temperature.a. extenderse b. contactar c. llegar a d. sobrepasar

16. seconds Ensure that skillet reaches an internal temperature of 165F for 15 seconds. a. secondarios b. secos c. segundos d. siguientes

17. directions Prepare crepes according to recipe directions. a. direcciones b. distribuciones c. instrucciones d. introducciones

18. hot Add hot water. a. caliente b. congelado c. frío d. tibio

19. cold Keep cold. a. caliente b. congelado c. frío d. tibio

20. to remove Remove from oven.a. agregar b. reemplazar c. remover d. sacar 21. over Spread 2lb 4oz mixture over each brownie. a. abajo de b. al lado de c. dentro de d. encima de


22. to cool Remove starch and cool completely. a. agitar b. calentar c. enfriar d. mezclar

23. before Pan roast allowing golden color to form before turning.a. antes de b. después de c. mientras d. si

24. top Spread a thin layer of sour cream over the top.a. centro b. fondo c. parte de abajo d. parte superior

25. time The amount of time should not exceed 4 hours. a. época b. medidas c. tiempo d. veces

26. fresh Prepare fresh lime juice according to recipe direction.a. ácido b. delicioso c. dulce d. fresco

27. pieces Divide dough into 2 lb 2 oz pieces. a. filas b. pedazos c. pilas d. tiras

28. bowl Combine salt and sugar in a bowl. a. cacerola b. cuenco c. olla d. taza

29. half Cut in half. a. mitad b. cuarto c. tercio d. todo


30. amount Cooking time will vary depending on the amount of product in the oven.a. calidad b. cantidad c. montón d. poquitín

31. well Mix well.a. bien b. bueno c. lo mejor d. mejor

32. best Best served fresh.a. bien b. bueno c. lo mejor d. mejor

33. green Prepare chopped green onions.a. amarillo b. blanco c. púrpura d. verde

34. yellow Prepare recipe for diced yellow onions.a. amarillo b. blanco c. púrpura d. verde

35. to select Select "pan fried" from SCC menu.a. empujar b. escoger c. tirar d. tocar

36. smooth Puree until smooth. a. con azúcar b. con grumos c. sin azúcar d. sin grumos


37. under Run under cold water.a. al lado de b. alrededor de c. debajo de d. encima de

38. golden Bake at 375F for 10 to 12 minutes, or until puffed and golden. a. crudo b. dorado c. medio crudo d. quemado

39. medium Heat dry saute pan over medium heat for 2 minutes. a. bajo b. máximo c. mediano d. mínimo

40. layer Spread a thin layer of sour cream over the top. a. capa b. parte c. región d. sección

41. red Prepare red pepper strips.a. amarillo b. naranja c. rojo d. verde

42. dry Strawberries should be dry.a. blando b. firme c. mojado d. seco

43. small Cut tofu into small rectangles.a. corto b. grande c. largo d. pequeño

44. to spread Spread across sheet pan evenly in single layer. a. doblar b. mezclar c. partir d. untar


45. to toss Toss and stir until all ingredients are hot.a. lanzar b. mezclar c. organizar d. tirar

46. to check Check internal temperature each time.a. controlar b. determinar c. servir d. verificar

47. day Best served same day.a. día b. mañana c. noche d. tarde

48. to make Make sauce by whisking soy, sriracha, fresh ginger and olive oil together.a. freír b. hacer c. hornear d. marcar

49. side Brown well on both sides until 165F internal temperature is reached.a. dorso b. frente c. lado d. rincón

50. thick Slice onions and peppers in thick strips about 2 inches wide.a. delgado b. duro c. fino d. grueso

51. to turn off Turn off heat and let steep for 1-2 hours. a. abrir b. apagar c. cerrar d. encender


52. very Drain tofu very well.a. demasiado b. muy c. nunca d. siempre

53. done When noodles are done, drain and rinse with cold water.a. caliente b. cocido c. seco d. mojado

54. lightly Lightly season chicken with salt and pepper.a. claramente b. ligeramente c. oscuramente d. suavemente

55. brown Deep fry at 350F for 3 minutes, until crisp and lightly brown. a. correoso b. crocante c. dorado d. quebradizo

56. fully Bake shell for 20 minutes until almost fully baked. a. a mitad b. achicharradoc. enteramente d. parcialmente

57. same In same pan, saute onions and garlic. a. diferente b. limpio c. mismo d. nuevo

58. through Strain sauce through china cap.a. al lado de b. a través de c. con d. encima de

59. evenly Evenly distribute the carrots, cucumbers and green onions.a. equitativamente b. finalmente c. lentamente d. rápidamente


60. soft Saute onions until soft and fragrant over medium heat.a. aromático b. blando c. crujiente d. dorado

61. briefly Warm in the oven briefly to re-crisp pastry.a. brevemente b. cuidadosamente c. lentamente d. rápidamente

62. high Heat nonstick pan over medium high heat.a. alto b. extremo c. lento d. mediano

63. low Mix on low for 3-4 minutes.a. alto b. extremo c. lento d. mediano

64. while Add pasta while gradually stirring.a. antes de b. después de c. mientras d. por

65. bag Thaw chicken in bags in cooler.a. bolsa b. caja c. nevera d. congelador

66. bottom Scrape bottom of pan to incorporate any browned bits.a. fondo b. lado c. parte superior d. rincón


67. flavor Full cake flavor is best at room temperature.a. aparición b. color c. olor d. sabor

68. gently Gently smooth glaze along top and sides of cake.a. cuidadosamente b. generosamente c. rápidamente d. totalmente

69. large Scale 4 pounds mixture into large roasting pans.a. corto b. grande c. largo d. pequeño

70. to leave Leave at room temperature for immediate service.a. calentar b. cuidar c. dejar d. salir

71. more Continue cooking for 10 more minutes.a. más b. menos c. muchos d. pocos

72. to transfer Then transfer pizza to cutting board.a. cambiar b. pasar c. transferir d. reemplazar 73. warm Pour in warm water.a. caliente b. frío c. helado d. tibio

74. box Cook according to box instructions. a. bolsa b. caja c. paquete d. libro


75. carefully Carefully fold eggs into noodles. a. cuidadosamente b. finalmente c. lentamente d. rápidamente 76. to cover Cover bowl and let set at room temperature.a. dejar b. mezclar c. sacar d. tapar

77. to lay Lay sausage on sheet tray.a. cortar b. estirar c. poner d. separar

78. plate Turn the pan upside down, rolling the omelet onto the plate.a. bandeja c. bol c. plata d. plato

79. flat Grill on flat top until golden brown and cheese is melted.a. blando b. duro c. liso d. plano

80. at least Preheat oven for at least 1 hour before using.a. más de b. menos de c. por lo máximo d. por lo menos

81. directly Roast directly on oven racks.a. inmediatamente b. directamentec. luego d. primero


82. ends Cut off ends. a. macas b. moho c. pedúnculosd. puntas

83. to form Form into 1.75 oz patties.a. formar b. hacer al grill c. hornear d. separar

84. to order Assemble and serve when ordered.a. pedir b. poder c. poner d. preguntar

85. quickly Cook quickly until just donea. cuidadosamente b. generosamente c. lentamente d. rápidamente

86. size Medium Time: 17-25 minutes (depending on batch size.)a. altura b. distancia c. longitud d. tamaño

87. speed Beat for 2-3 more minutes on low speed.a. calor b. grado c. nivel d. velocidad

88. between Place wings on pan with space between them.a. al lado de b. enfrente de c. entre d. atrás de

89. center Cake should be just firm in the center when done.a. centro b. fondo c. lado d. parte superior


90. to continue Continue baking for 5-10 minutes until cheese is melted.a. continuar b. empezar c. terminar d. parar


Answer Key

1. c. necesitar 2. d. receta3. b. manos 4. c. empezar 5. a. limpio 6. c. hasta 7. a. añadir 8. d. mantener 9. a. bajo 10. b. cuando 11. b. durante 12. b. listo 13. d. interior14. b. mantener 15. c. llegar a 16. c. segundos17. c. instrucciones 18. a. caliente 19. c. frío 20. d. sacar21. d. encima de22. c. enfriar 23. a. antes de 24. d. parte superior 25. c. tiempo 26. d. fresco27. b. pedazos 28. b. cuenco 29. a. mitad 30. b. cantidad

31. a. bien 32. c. lo mejor33. d. verde34. a. amarillo 35. b. escoger 36. d. sin grumos37. c. debajo de38. b. dorado 39. c. mediano 40. a. capa 41. c. rojo 42. d. seco43. d. pequeño 44. d. untar45. b. mezclar 46. d. verificar47. a. día 48. b. hacer 49. c. lado 50. d. grueso51. b. apagar 52. b. muy 53. c. cocido 54. b. ligeramente 55. d. dorado 56. c. enteramente 57. c. mismo 58. b. a través de 59. a.equitativamente 60. b. blando

61. a. brevemente 62. a. alto 63. c. lento 64. c. mientras 65. a. bolsa 66. a. fondo 67. d. sabor 68. a. cuidadosamente 69. b. grande 70. c. dejar 71. a. más 72. b. pasar 73. d. tibio74. b. caja 75. a. cuidadosamente 76. d. tapar 77. c. poner 78. d. plato79. d. plano80. d. por lo menos81. b. directamente 82. d. puntas 83. a. formar 84. a. pedir 85. d. rápidamente 86. d. tamaño87. d. velocidad 88. c. entre 89. a. centro 90. a. continuar


Translation of TestName _________________________________________________ Date __________________

The purpose of this exam is to test your knowledge of the meaning of basic kitchen vocabulary you will need to work in a CSU kitchen. You will have 90 minutes to take the test. Each response

will be marked either correct or incorrect. The results will be used to determine if you need to participate in the Workplace English course.

Directions: Read each English sentence. After reading the sentence of context, circle the answer which represents the best translation of the word/phrase in bold. Do not guess if you do not

know the answer.

For example : 0. cup Put the liquid in the cup.a. bowlb. potc. cupd. glass

1. to need Refrigerate and serve as needed.a. to have to b. to demand c. to need d. to want

2. recipe Ensure that all preparation equipment and utensils needed for this recipeare cleaned and sanitized.

a. warning b. list c. message d. recipe

3. hands Wash hands prior to beginning preparation.a. fingers b. hands c. plates d. utensils

4. to begin Wash hands prior to beginning preparation.a. to cook b. to continue c. to begin d. to finish


5. clean Place product in clean sanitized serving pans and cover. a. clean b. modified c. new d. sanitary

6. until Keep chilled at 41F or below until ready to use.a. before b. after c. until d. while

7. to add Add lemon juice, water, and salt. a. to add b. to squeeze c. to measure d. to take out

8. to keep Keep chilled at 41F or below until ready to use.a. to heat b. to freeze c. to conserve d. to keep/maintain (something cold)

9. below Keep chilled at 41F or below until ready to use.a. below b. more c. through d. above

10. when Follow all appropriate sanitation procedures when handling and serving product.a. before b. when c. after d. where


11. during Maintain temperature at 140F or above during holding and serving.a. before b. during c. after d. if

12. ready Keep chilled at 41F or below until ready to use.a. available b. ready c. desired d. necessary

13. internal Cook to internal temperature of 135F.a. high b. room c. extreme d. internal

14. to maintain Maintain temperature at 140F or above during holding and serving.a. to write down b. to maintain c. to measure d. to observe

15. to reach Cooking time will take from 5-7 minutes to reach 165F internal temperature.a. to spread b. to contact c. to reach d. to surpass

16. seconds Ensure that skillet reaches an internal temperature of 165F for 15 seconds.

a. secondary b. dry c. seconds d. following

17. directions Prepare crepes according to recipe directions. a. addresses b. distributions c. directions d. introductions


18. hot Add hot water. a. hot b. frozen c. cold d. warm

19. cold Keep cold. a. hot b. frozen c. cold d. warm

20. to remove Remove from oven.a. to add b. to replace c. to stir d. to remove 21. over Spread 2lb 4oz mixture over each brownie. a. under b. next to c. inside d. over

22. to cool Remove starch and cool completely. a. to shake b. to heat c. to cool d. to mix

23. before Pan roast allowing golden color to form before turning.a. before b. after c. while d. if

24. top Spread a thin layer of sour cream over the top.a. center b. end c. bottom d. top


25. time The amount of time should not exceed 4 hours. a. time (in history) b. measurements c. time d. times (occurrences)

26. fresh Prepare fresh lime juice according to recipe direction.a. acidic b. delicious c. sweet d. fresh

27. pieces Divide dough into 2 lb 2 oz pieces. a. rows b. pieces c. piles d. strips

28. bowl Combine salt and sugar in a bowl. a. saucepan b. bowl c. pot d. cup

29. half Cut in half. a. a half b. a quarter c. a third d. a whole

30. amount Cooking time will vary depending on the amount of product in the oven.a. quality b. amount c. a lot of d. a little bit of

31. well Mix well.a. well b. good c. the best d. better


32. best Best served fresh.a. well b. good c. the best d. better

33. green Prepare chopped green onions.a. yellow b. white c. reddish-purple d. green

34. yellow Prepare recipe for diced yellow onions.a. yellow b. white c. reddish-purple d. green

35. to select Select "pan fried" from SCC menu.a. to push b. to select c. to pull d. to touch

36. smooth Puree until smooth. a. with sugar b. not smooth c. sugarless d. smooth

37. under Run under cold water.a. next to b. around c. under d. above

38. golden Bake at 375F for 10 to 12 minutes, or until puffed and golden. a. raw b. golden c. semi-cooked d. burned


39. medium Heat dry saute pan over medium heat for 2 minutes. a. low b. maximum c. medium d. minimum

40. layer Spread a thin layer of sour cream over the top. a. layer b. part c. region d. section

41. red Prepare red pepper strips.a. yellow b. orange c. red d. green

42. dry Strawberries should be dry.a. soft b. firm c. wet d. dry

43. small Cut tofu into small rectangles.a. short b. large c. long d. small

44. to spread Spread across sheet pan evenly in single layer.a. to turn b. to combine c. to cut d. to spread 45. to toss Toss and stir until all ingredients are hot.a. to throw b. to toss c. to organize d. to throw out


46. to check Check internal temperature each time.a. to control b. to determine c. to serve d. to check

47. day Best served same day.a. day b. morning c. night d. afternoon

48. to make Make sauce by whisking soy, sriracha, fresh ginger and olive oil together.a. to cool b. to make c. to bake d. to mark

49. side Brown well on both sides until 165F internal temperature is reached.a. back b. front c. side d. corner

50. thick Slice onions and peppers in thick strips about 2 inches wide.a. thin b. hard c. fine d. thick

51. to turn off Turn off heat and let steep for 1-2 hours. a. to open b. to turn off c. to close d. to turn on

52. very If carrot is very slender, don't cut it lengthwise.a. too b. very c. never d. always


53. done When noodles are done, drain and rinse with cold water.a. hot b. done c. dry d. wet

54. lightly Lightly season chicken with salt and pepper.a. lightly (in color) b. lightly c. darkly d. gently

55. brown Deep fry at 350F for 3 minutes, until crisp and lightly brown.a. crunchy b. toasted c. crumbly d. brown

56. fully Bake shell for 20 minutes until almost fully baked. a. halfwayb. burnedc. fully d. partially

57. same In same pan, saute onions and garlic. a. different b. clean c. same d. new

58. through Strain sauce through china cap.a. next to b. through c. with d. on top of

59. evenly Evenly distribute the carrots, cucumbers and green onions.a. evenly b. finally c. slowly d. quickly


60. soft Saute onions until soft and fragrant over medium heat.a. fragrant b. soft c. crunchy d. golden

61. briefly Warm in the oven briefly to re-crisp pastry.a. briefly b. carefully c. slowly d. quickly

62. high Heat nonstick pan over medium high heat.a. high b. extreme c. low (heat) d. medium

63. low Mix on low for 3-4 minutes.a. high b. extreme c. low d. medium

64. while Add pasta while gradually stirring.a. before b. after c. while d. through

65. bag Thaw chicken in bags in cooler.a. bag b. box c. refrigerator d. freezer

66. bottom Scrape bottom of pan to incorporate any browned bits.a. bottom b. side c. top d. corner


67. flavor Full cake flavor is best at room temperature.a. appearance b. color c. smell d. flavor

68. gently Gently smooth glaze along top and sides of cake.a. gently b. generously c. quickly d. totally

69. large Scale 4 pounds mixture into large roasting pans.a. short b. large c. long d. small

70. to leave Leave at room temperature for immediate service.a. to heat b. to care for c. to leave (something) d. to leave (a place)

71. more Continue cooking for 10 more minutes.a. more b. fewer c. many d. few

72. to transfer Then transfer pizza to cutting board.a. to change b. to transfer (something) c. to transfer (a person) d. replace 73. warm Pour in warm water.a. hot b. cold c. frozen d. warm


74. box Cook according to box instructions. a. bag b. box c. packet d. book

75. carefully Carefully fold eggs into noodles. a. carefully b. finally c. slowly d. quickly 76. to cover Cover bowl and let set at room temperature.a. to leave b. to mix c. to take out d. to cover

77. to lay Lay sausage on sheet tray.a. to cut b. to stretch c. to lay d. to separate

78. plate Turn the pan upside down, rolling the omelet onto the plate.a. tray b. bowl c. silver d. plate

79. flat Grill on flat top until golden brown and cheese is melted.a. soft b. hard c. smooth d. flat

80. at least Preheat oven for at least 1 hour before using.a. more than b. less than c. at the most d. at least


81. directly Roast directly on oven racks.a. immediately b. directly c. later d. first

82. ends Cut off ends.a. bruises b. mold c. stems d. ends

83. to form Form into 1.75 oz patties.a. to form b. to grill c. to bake d. to separate

84. to order Assemble and serve when ordered.a. to order b. to be able to c. to put d. to ask

85. quickly Cook quickly until just done.a. carefully b. generously c. slowly d. quickly

86. size Medium Time: 17-25 minutes (depending on batch size.)a. height b. distance c. length d. size

87. speed Beat for 2-3 more minutes on low speed.a. heat b. degree c. level d. speed


88. between Place wings on pan with space between them.a. next to b. in front of c. between d. behind

89. center Cake should be just firm in the center when done.a. center b. bottom c. side d. top

90. to continue Continue baking for 5-10 minutes until cheese is melted.a. to continue b. to begin c. to finish d. to stop


Appendix D

Administrative Score Report Form

The following score report is from the Workplace English Training Placement Test taken by

___________________________________________ on ________/ ________ / ________.Employee Name / CSU Employee ID # month day year

The employee was evaluated according to the correctness of his/her selections.

Each item answered correctly was worth 1 point. There were a total of 90 points.

Number of items correct involving:

Adjectives: ______ / 27

Verbs: _____ / 22

Nouns: ______ / 20

Adverbs: ______ / 11

Prepositions: ______ / 10

Total Raw Score: ______ / 90

Percentage Score: ________

Raw score required to test out of Workplace English Training: 68/90

Percentage score required to test out of Workplace English Training: 75%

Does this employee test out of Workplace English Training? Yes _____ No______

running head: assessing second language skills: test

Documents