REPOR T RESUMESED 018 644 VT 004 863

USING PROGRAMED INSTRUCTION IN OCCUPATIONAL EDUCATION.

BY- NORTON, ROBERT E.NEW YORK STATE EDUCATION DEPT., ALBANYSTATE UNIV. OF N.Y., ITHACA

PUB DATE SEP 67

EDRS PRICE MF-S0.25 HC -$2.32 56P.

DESCRIPTORS- *PROGRAMED INSTRUCTION, *VOCATIONAL EDUCATION;

*EDUCATIONAL PRINCIPLES, RESEARCH REVIEWS (PUBLICATIONS) ,*SELECTION, INFORMATION SOURCES, BIBLIOGRAPHIES, EVALUATION,

ALTHOUGH THIS REPORT ON THE STATUS OF PROGRAMEDINSTRUCTION AND THE POTENTIAL IT HOLDS FOR SCHOOLS EMPHASIZES

THE APPLICATIONS OF PROGRAMED INSTRUCTION TO OCCUPATIONAL

EDUCATION, MUCH OF WHAT IS SAID IS ALSO APPLICABLE TO GENERAL

EDUCATION. AN EXAMPLE OF PROGRAMED INSTRUCTION, AN OVERVIEW

OF SUPPORTING PSYCHOLOGICAL PRINCIPLES, RESEARCH FINDINGS,

SOME CONTROVERSIES, SOME ADVANTAGES, AND SOME LIMITATIONS OF

PROGRAMED INSTRUCTION ARE PRESENTED. A DISCUSSION OF THE

SELECTION AND USE OF PROGRAMED MATERIALS INCLUDES WHAT THE

MATERIALS CAN TEACH, HOW THEY SHOULD BE USED, THE KINDS

AVAILABLE, THE ECONOMICS OF USING THEM, AND RECOMMENDED

SELECTION PROCEDURES. CASE STUDIES FROM SCHOOLS AND

INDUSTRIES USING PROGRAMED INSTRUCTION SUCCESSFULLY INOCCUPATIONAL EDUCATION ARE PRESENTED FOR BASIC INDUSTRIAL

SKILLS, BUSINESS, ENGLISH, MEDICAL EDUCATION, HEALTH AND

SAFETY, HOME ECONOMICS, INDUSTRIAL TRAINING, MEDICAL SALES,

PERSONNEL PRACTICES, TRADE AND INDUSTRIAL, SELLING ANDRETAILING, SPELLING, VOCATIONAL AGRICULTURE, AND 14VOCATIONAL AREAS. ALTHOUGH THE SUCCESSFUL USE OF PROGRAMED

INSTRUCTION HAS BEEN WIDESPREAD, ADMINISTRATORS AND TEACHERS

SHOULD LEARN AS MUCH AS POSSIBLE ABOUT IT BEFORE ATTEMPTING

TO USE IT, INVOLVE QUALIFIED TEACHERS, SELECT MATERIALS

CAREFULLY, TRY IT ON A SMALL SCALE UNTIL IT IS PROVED

SUCCESSFUL, FOLLOW 000D MANAGEMENT PROCEDURES, AND INTEGRATE

IT WITH OTHER METHODS OF INSTRUCTION. SOURCES OF PROGRAMED

TEXT MATERIALS, RECOMMENDED REFERENCES, CRITERIA FOR

ASSESSING PROGRAMED INSTRUCTIONAL MATERIALS, RECOMMENDATIONS

FOR REPORTING THE EFFECTIVENESS OF THE MATERIALS, AND A

BIBLIOGRAPHY ARE INCLUDED. (PS)

:7)) USING PROGRAMED INSTRUCTION

IN

OCCUPATIONAL EDUCATION

The University of the State of New YorkThe State Education Department

Bureau of Occupational Education Research

and

Department of EducationNew York State College of AgricultureCornell University, Ithaca, New York

THE UNIVERSITY OF THE STATE OF NEW YORK

Regents of the UniversityWith years when terms expire

1968 Edgar W. Couper, A.B., LL.D., L.H.D., ChanCellor

1970 Everett J. Penny, B.C.S., D.C.S., Vice Chancellor

1978 Alexander J. Allan, Jr., LL.D., Litt ,D,

1973 Charles W. Millard, Jr., A.B., LLD,

1972 Carl II. Pforzheimer, Jr., A.B., M.B.A., D.C.S.

1975 Edward M. M. Warburg, B.S., L.H.D.

1969 Joseph W. McGovern, A.B LL.B., L.H.D., LL.D.

1977 Joseph T. King, A.B LL.B.

1974 Joseph C. Indelicato, M.D.

1976 Mrs, Helen B. Power, A.B., Litt.D,

1979 Francis W. McGinley, B.S., LL.B.

1981 George D. Weinstein, LL.B.

1980 Max J. Rubin, LL.B., L.H.D.

1971 Kenneth B. Clark, A.B., M.S., Ph ,D,

1982 Stephen K. Bailey, A.B., B.A., M.A., Ph.D., LL.D.

President of the University and Commissioner of EducationJames E. Allen, Jr.

Deputy Commissioner of EducationEwald B. Nyquist

Associate Commissioner for Elementary, Secondary and Continuing EducationWalter Crewson

Associate Commissioner for Research and EvaluationLorne H. Woollatt

Assistant Commissioner for Research and EvaluationWilliam D. Firman

Director, Division of ResearchCarl E. Wedekind

Chief, Bureau of Occupational Education ResearchLouis A. Cohen

Binghamton

White Plains

Troy

Buffalo

Purchase

New York

New York

Queens

Brooklyn

Rochester

Glens Falls

Hempstead

New York

Hastings on H

Syracuse

USING PROGRAMED INSTRUCTION IN OCCUPATIONAL EDUCATION

U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE

OFFICE OF EDUCATION

THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE

PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIONS

STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION

POSITION OR POLICY.

Prepared by

Robert E. Norton

Research AssociateDepartment of Education

New York State College of AgricultureCornell University, Ithaca, New York

The University of the State of New YorkThe State Education Department

Bureau of Occupational Education ResearchAlbany, New York

September 1967

"Programed Instruction in Occupational Education" is theresult of interest shown in the previous publication, "Using Pro-gramed Instructions With and Without Self-Instructional Practiceto Teach Psychomotor Skills" by the same author.

The bulletin is intended as a guide to occupational teachers,supervisors, and administrators in planning, developing, and usingvarious types of programed instruction as a guide in the learningprocess. It is desired that the information presented herein willresult in units of instruction where students and teachers maybenefit from such facilities useful in learning new skills to increasecognitive faculties in specific occupational areas.

The writer wishes to acknowledge the supervisory assistanceof Professor Joe P. Bail, Chairman of the Agricultural EducationDivision at Cornell, who served as Project Director, and to Pro-fessor Frederick K. T. Tom, of the Agricultural Education Divisionat Cornell, who served as Project Director during the planningstages.

Similarly, gratitude is due Lyle Wicks, Instructional MaterialsSpecialist, Division of Agricultural Education, for his many helpfulsuggestions and his assistance with the production of this publica-tion.

Appreciation is expressed to Dr. Alan Robertson, formerlyChief of the Bureau of Occupational Education Research, who wasresponsible for initiating the project, and to Charles Meislin, anAssoctate in the Bureau, who served as project officer. Thanksare also due Dr. Louis Cohen, presently Chief of the Bureau, forhis assistance with the final stages.

I. INTRODUCTION

TABLE OF CONTENTS

Purpose and ScopeHistorical SketchWho Is Using Programed Instruction?The Need and The ResponsibilityTraditional Instruction versus an Instructional System

111224

II. PROGRAMED INSTRUCTION - AN EXPLANATION AND OVERVIEW OFA NEW TECHNOLOGY OF TEACHING 5

What Is Programed Instruction? 5

Illustration 6

Underlying Educational and Psychological Concepts 6

Distinguishing Characteristics 8

Research Evidence 8

Controversies 11

Some Advantages 12

Some Limitations 13

Student Reactions 13

III. SELECTION AND USE OF PROGRAMED "ATERIALS 15

What Can Programed Materials Teach? 15

How Should Programs Be Used? 15

What Kinds of Materials Are Available? 16

What Are the Economics of Using Programed Materials 16

What Selection Procedures Are Recommended? 17

What Administrative Changes Will Require Attention? 18

What Changes May Occur In The Teacher's Role? 18

IV. SOME SUCCESSFUL APPLICATIONS OF PROGRAMED INSTRUCTION 19

1. Basic Industrial Skills 19

2. Business 19

3. Elementary 19

4. English 19

5. General 19

6. Health-Medical Education 20

7. Health and Safety 20

8. Home Economics 20

9. Industrial Training 20

10. Industrial Training 20

11. Medical-Nursing-Sales 20

12. Personnel Practices 21

13. Trade and Industrial 21

14. Selling and Retailing 21

15. Selling and Retailing 21

16. Spelling 21

17. Vocational Agriculture 21

18. Vocational 22

iii

TABLE OF CONTENTS (continued)

V. RECOMMENDATIONS AND CONCLUSIONS 23

Recommendations23

Conclusions24

APPENDIX A25

APPENDIX B32

APPENDIX C35

APPENDIX D49

CHAPTER I

INTRODUCTION

In an age where the acquisition of knowl-edge is the way of life for most people, pro-gramed instruction offers much: individual-ized rate of instruction, faster and moreeffective learning, immediate confirmationof student responses. Many students, espe-cially those with low ability, experience forthe first time a new feeling of success andconfidence in their ability to learn. By mean:;of carefully constructed sequences, the pro-gram takes each learner from what he knowsto what we want him to know.

Today professional literature aboundswith reports attesting to the fact that pro-gramed instruction has proved itself inindustry, in the colleges, and in the armedforces. It is high time that programed in-struction redeemed its promises in ourschools. What can be done more effectivelyand more efficiently with this method shouldbe done!

Purpose and Scope

The purpose of this publication is to pre-sent an up-to-date report on the status ofprogramed instruction and the potential itholds for our schools. Most educators havenot yet been convinced of the value andeffectiveness of programed instruction, norare they aware treat it is a direct effort toimplement modern learnin theories and theresults of current research by uildin theminto a more effective, more scientific tech-noloirorreins and learning.

The emphasis here is on the applicationof programed instruction to occupational edu-cation, but much of what is said is also appli-cable to general education.

An explanation of what programed in-struction is, and an overview of supportingpsychological principles, research findings,some controversies, some advantages, andsome limitations of programed instructionare presented in Chapter 2. Chapter 3 givespractical and useful information on the selec-tion and use of programed materials, and the

fourth chapter cites numerous case studiestaken from the experiences of schools andindustry where programed instruction hasbeen used in occupational education.

Three of the four appendices will be ofspecial interest to teachers, administrators,and others who want to: (1) learn what mate-rials are available commercially, and theirsources (Appendix A), (2) read further aboutprogramed instruction (Appendix B), and (3)learn more about recommended proceduresfor selecting programed materials (AppendixC).

Historical Sketch

No single person can claim responsibilityfor the introduction of programed instruction.Instead, three persons - Sidney L. Pressey,B. F. Skinner, and Norman Crowder - arerecognized as the outstanding contributors toits development. Teaching machines werefirst discussed by Dr. Pressey at an Amer-ican Psychological Association meeting heldin 1924. Pressey's approach is based onrecognition of a correct response.

With the exception of research scientistsand a few university educators, very littleattention was paid these devices until the1950's. A paper on conditioned learning byHarvard's Dr. Skinner, read before a con-ference of psychologists in 1954, is believedto have created a stir far beyond his im-mediate audience. Skinner, a well-knownexperimental psychologist, began to publisharticles about the teaching machines he andhis colleagues had developed. Much of hiswork was based on the principle of recall ofinformation, a major feature of linear pro-graming.

Because of serious teacher shortages andthe need for new and improved methods ofinstruction, his writings were widely dis-cussed and stimulated many people. One ofthe more notable of these was Norman Crowd-er who initiated and carried out extensive

work with the branching technique, known asintrinsic programing. During the last fewyears so many researchers have turned theirattention to programed instruction that pro-fessional literature in education has beenflooded with articles and reseanh reportson it.

Who is Using Programed Instruction?

Unfortunately, there is no up-to-date in-formation on the percentage of schools in theUnited States that is actually usingprogramspublished commercially or produced locally.Kenneth Komoski, director of the Center forProgramed Instruction, Columbia University,stated in 1965 that a conservative estimatewould probably not go beyond 10 per cent,but that an estimate of 20 per cent might notbe far from the mark either. He also re-ported a great deal of international activityin programed instruction. Including work byeducational ministries, other governmentagencies, universities, and indvIstries, in1965 between twenty and thirty countrieswere involved in researching, developing,and/or using programed instruction. Itsproliferation in many directions is such thatit must surely stand as one of the mostrapidly developing and diffusing educationalinnovations of all time.

Far more startling is the use of pro-gramed instruction by industry. All types oforganizations - from the industrial giants(Gulf, Ford, General Motors, DuPont, Amer-ican Telephone and Telegraph, American Oil,U. S. Steel, IBM) down to local businesses(auto dealers, supermarkets, banks, tool anddie shops, and department stores) - areusing programed instruction wherever thereis a training job to be done.

Illustrative of the extensive use of pro-gramed instruction by companies are tworeports, one by the DuPont Company and theother by the Argyle Publishing Corporation.According to Dr. Arthur Santora, head ofDuPont's Industrial Training Service, whatDuPont did not intend to do, at the time theybegan working with programed instruction in1959, was to get into the teaching business.But the same factor that got them started -a lack of commercially available courses inthe desired skills - created a demand fortheir courses outside the company. Above

-.

2

and beyond their own internal demand of55,000 course units in the last five years,there has been an outside demand for anequal number of course units in less thantwo years! Courses have been supplied to1,200 other companies, many governmentaael state agencies including the Bureau ofPrisons and the Job Corps, hospitals, insti-tutions, and vocational schools. Similarly,an Argyle sales brochure lists 132 majororganizations including the Army, Navy, andAir Force and states that these are a few ofthe more than 9,000 companies now usingArgyle programed instruction.

Education, however, is not keeping pacewith industry. A 1966 National EducationAssociation study of programed instructionin large school systems in the United Statesreports the grades and number of programsused in 126 school systems. A glance atFigure 1 provides an idea of where mostprogramed instruction is presently beingused in our elementary and secondaryschools.

The largest percentage of use has been atthe secondary level, with the ninth gradebeingthe level at which the most extensive useoccurred.

The Need and the Responsibility

Modern society is characterized by ahighdegree of technological innovation and sophis-tication. Joggled by a technology that changesfaster than feminine fashions, industry isdevoting ever more attention to training, re-training, and upgrading the abilities of itsworkers. Correspondingly, we might expectthat through research and development edu-cation would also have turned to technologyfor assistance in meeting its increasinglyimportant objectives, but actually educationhas not been characterized by technologicalinnovation. Only recently have perceptive andforward-looking educators turned to tech-nology al a means to improve education.Education has witnessed a technological lagrather than a technological revolution.

Are yesterday's methods and techniquescapable of meeting today's needs? In the nottoo distant past, a man could learn a tradeand look forward to practicing it the rest ofhis life without having to learn much of any-thing new. Today vocations change in many

Figure 1. Grades in which programed instructionwas used in 126 large school systems

Grade

Numberof

Systems

Percentof TotalSystems

Numberof

Programs

Percentof 378

Programs

1

K 2 1.6% 2 0.5%

1 20 15.9 23 6.1

2 14 11.1 17 4.5

3 9 7.1 12 3.2

4 28 22.2 41 10.8

5 37 29.4 56 14.8

6 38 30.2 58 15.3

7 45 35.7 70 18.5

8 50 39.7 77 20.4

9 67 53.2 121 32.0

10 57 45.2 96 25.4

11 48 38.1 74 19.6

12 43 34.1 76 20.1

important ways during eachman'sproductiveyears. Every man must be prepared tolearn and relearn a living throughout hislife.

Educators have long idealized but neverrealized the educational goal of adaptinginstruction to the ability of the individualstudent. We are constantly pitching ourclassroom instruction to the level of themythical average student while those ofhigher ability are bored, and those of lowerability, baffled. Can we allow this injusticeto so many students to continue?

The findings of multicounty surveys inNew York State indicate another definiteneed to expand and improve occupationaleducation in New York State. Three of thesefindings, as presented in Education for Occu-pations, are as follows:

1. Statewide trends indicate a continuingneed for development of higher skilland technological knowledge compe-tencies among the work force.

2. Statements by employers and Anionsin the area studies certify thai,p2rsonswith vocational-technical skills neededin the work force are in short supply.

3

3. The multiple area studies confirm thatmany present school programs areinadequate in providing needed salableskills.

A representative of many industries, theNational Association of Manufacturers, sup-ports these findings in a case study whichreports that many jobs continue to gobeggingbecause thousands of untrained oruntrainableunemployed lack necessary basic educationalskills.

What is our responsibility as educators?A statement printed in the NEA's January1963 issue of Audiovisual Instruction relatesour responsibility so well that it isrepeatedhere.

Education is a matter of individualhuman growth and development; there-fore technological methodology mustbe introduced with care. Our primaryconcern is and must be the individualhuman personality - the nurturing ofsocial and moral values, and, perhapsmore too the point in our case, thedevelorment of rational thinking, ofintellectual competence, of respon-sible action, and of productive ability.

Such development involves the trans-mitting (teaching) and mastering(learning) of a great deal of informa-ton and many complex skills.

The responsibility is clear and the needis evident. With full realization of the taskto be accomplished, and the ever increasingshortage of qualified teachers, educatorsmust turn to technology. Again to quoteAudiovisual Instruction, "A technologicalleap forward is required in education."

Traditional Instruction versus an Instruc-tional System

The very common direct teacher-pupilrelationship, the basis of traditional educa-tion, is illustrated in Figure 2, taken fromthe NEA's January 1963 issue of AudiovisualInstruction.

Objectives

This approach presents a scientifically de-veloped arrangement of instructors, mate-rials, and technological media for providingoptimum learning with a minimum of routinepersonal involvement by the teacher. Theresult, as depicted in Figure 3, is a carefullyplanned system consisting of subject matter,procedures, and media coordinated in a pro-gram-unit design which is directed towardspecific behavioral objectives.

The instructional system approach pro-vides three alternate instructional routes andallows the teacher to select the most effectiveone for each learning situation. The teachermay present new material himself, selectvarious media to help him with the presenta-tion, or utilize media alone to make the pre-sentation.

Another feature of the instruction systemapproach is that it provides for regular feed-

Figure 2. Traditional Instruction

Contentand

MethodDecisions

Although printed materials, chalk, and afew other devices came into play, there islittle real technology involved in this in-structional organization.

The new approach to a systems conceptof instruction requires our earnest attention.

Teacher m0041011110

back and evaluation at all stages of the in-structional process. Such feedback and eval-uation, properly utilized, provides valuableinformation for revising objectives, alteringcurriculum content, and selecting the bestmethods.

Figure 3. Instructional System

Contentand

MethodDecisions

Feedback & Evaluation

Pupil

The NEA article summarizes the functionof educational media as twofold.

The first function of technologicalmedia is to supplement the teacherthrough increasing his effectiveness inthe classroom. Educational media areboth tools for teaching and avenues forlearning, and their function is to servethese two processes by enhancingclarity in communication, diversity inmethod, and forcefulness in appeal.Except for the teacher, these mediawill determine more than anything elsethe quality of our educational effort.

The second function of media isone in which the media alone maypresent and, in a sense, teach certaincontent to pupils. Here, the teacherdetermines objectives, selects meth-ods and content, and evaluates the final

learning outcomes. The presentationof information, and even the directionof routine pupil activities, may beturned over to such new media asprogramed learning materials, tele-vision, or motion pictures. FunctionNo. 2, then, is to enhance overall pro-ductivity through instructional mediaand systems which do not depend uponthe teacher for routine execution ofmany instructional processes or forclerical-mechanical chores.

As the training director of a large NewYork City bank points out, "No one trainingmethod meets all requirements. A profit-oriented company or an educational systemmust provide the most effective training andeducation, using the most up-to-date andeffective methods, and at the least cost con-sistent with good judgment."

CHAPTER II

PROGRAMED INSTRUCTION - AN EXPLANATION AND OVERVIEW

OF A NEW TECHNOLOGY OF TEACHING

What is Programed Instruction

Programed instruction is one of severalrelatively new technological innovations ineducation. It is a process or technique forthe design and development of self-instruc-tional materials. A program of instructionis simply an instructional unit developed bymeans of this process. The process con-tains these essential steps: (1) establish-ment of specific behavorial objectives, (2)analysis of instructional content, (3) pro-gram production, (4) preliminary testing, (5)revision, (6) field testing, (7) final revision,and (8) validation.

It is a way of learning and a way of pre-senting materials to be learned. It is essen-tially self-instructional, meaning that basic-ally the student is learning by himself and athis own pace with the help of a programedtext or machine. Because it enables studentsto proceed independently, programed instruc-

5

tion is the first teaching technique that per-mits breaking the traditional classroom lock-step procedure.

Programed materials are based on a care-ful and detailed definition of learning objec-tives. To provide the learner maximumopportunity to attain these objectives, thesubject-matter is broken into small, easilyabsorbed increments called "frames," whichare carefully selected and sequenced to buildon preceding units.

In each frame the learner must do some-thing (active response) to show that he hasgrasped the information presented. Thestudent can immediately check his answeragainst the correct one (confirmation). Thusthe student knows whether he has graspedthe information. If his response is correct,the program reinforces his learning throughrepetition; if incorrect, he can study theinformation again or ask his teacher wherehe went wrong. Having to act maintains his

attention, and knowing that he has actedcorrectly Maintains his confidence.

Russell Pease, consultant for Dupont, ina talk that he gives on the subject, explainsthe value of this approach, "The incrementsof knowledge are readily understood andeasily digestible. They are prepared so thata student's reply is almost always correct.The fact that the student is almost alwayscorrect encourages him. He enjoys his work.Learning becomes easy, pleasant and self-sustaining. The simplicity of this method isdeceptive and the results are amazing."Educational psychologists have found that manlearns better and faster when he is con-fronted with a minimum of errors. Mostprogramed instruction is geared to this end.

In a very real sense, this techniquenarrows the communications gap betweenthe students and the technical expert whowrote the course. It is almost like a tutorialrelationship. Differences in age, education,job experiences, and learning speed tend tobe neutralized - the learners emerge withmore uniform levels of understandingbecauseeach sets his own pace and checks his ownprogress. By permitting flexibility hereto-fore impossible, programed instruc ti onbrings us closer than ever before to the goalof individualizing instruction.

Illustration

The best way to become familiar with thetechnique of programed instruction is toexperience it yourself, if you have not alreadyhad the opportunity. With this thought inmind, the next page illustrates one type ofprogramed learning.

Underlyiigi Educational and PsychologicalConcepts

Programed instruction provides the firstreal penetration of psychological theory intothe educational process. Here, in one of thefew instances where the product of learningtheory has found direct and practical appli-cation in the classroom, educational theoryand a method of instruction converge.

Ernest R. Hilgard, professor of psy-chology and education at Stanford University,has described six established principles fromthe psychology of learning which support

6

programed instruction.1. Programed instruction recognizes in-

dividual differences by beginning where thelearner is and by permitting him to proceedat his own pace. Individual differences inlearning ability have long been observed inall areas of intellectual and motor activity.It is possible that programed learning maysucceed in reducing large individual differ-ences because of these features.

2. Programed instruction requires activeparticipation on the part of the learner.Learning by doing is an old educationalslogan, and is still a good one. Active par-ticipation by the learner leads to faster andmore effective learning than does the moreconventional learning situation.

3. Programed instruction provides forimmediate knowledge of results. Whether itis because programed learning provides re-inforcement, reward, or cognitive feedback,there is abundant evidence that immediateknowledge of results is important in learn-ing. It favors learning the right thing whilepreventing the repetition and fixation ofwrong answers.

4. Programed instruction reduces anxietythat so often is connected with the learningsituation. The learner is not threatened bythe task because he knows he can learn andis learning. This knowledge brings himsatisfaction and increased confidence in hisability. The major principle involved is thatlearning motivated by success is more effec-tive than learning motivated by failure orfear of failure.

5. Programed instruction provides spac-ed review in order to guarantee the highorder of success that has become a standardrequirement of good programs. If properlyarranged, review permits a high degree oflearning on the first run through a program.

6. Programed instruction emphasizes theorganized nature of knowledge so often lack-ing with other methods of instruction. Goodprograms are written to provide for goodcontinuity between the easier (earlier) con-cepts and the harder (later) ones. Theproc-ess of programing requires examining thesubject matter very carefully in order to findout what has to be known before somethingelse can be learned, and eliminating sideissues that do not lead to cumulative learn-ing.

AN EXAMPLE OF LINEAR PROGRAMED INSTRUCTION

This page will demonstrate how programed learningworks. Cover up the answers in the outside columnsas you work through the "frames" of information.Check answer to each frame before going on to thenext. (Reprinted, with minor changes, by permissionof Office Equipment and Methods Magazine.)

1. Programed instruction learning ofthe linear type, as originated by Dr.B. F. Skinner of Harvard, breaks downsubject matter into small chunks ofinformation called frames. The unit ofinstruction in such a system is calleda

2. Each frame presents some newpiece of information and calls for aresponse from the student. Thereforeeach frame calls for two actions. Thestudent must read the

frame

and make his

3. This participation, or learning bydoing, is the most efficient form oflearning. So the student is assisted byhis own

informationor frame

response

4. One way of getting a response fromthe student is to ask him to completethe blank space in a sentence. By fill-ing in the space he is made

participation

to

5. When a student has answered thequestion in a frame, he is given rein-forcement. After answering the ques-tion, his learning is by the

blankrespond

correct answer.

6. Periodically, a program will includea frame reviewing information that hasgone before and calling for a series ofresponses to help fix the subject matterin the student's mind. Just like this:The unit of instruction in a program iscalled a . It presents

reinforced

and calls Tii afrom the student. This may be done byfilling in the in a sentence.The student's earning is thenby the correct answer.

7. The correct answer in a programedtext can be conveniently placed along-side the next frame, as it is here.When the student has answered a ques-tion he turns to the to

frameinformationresponseblank spacereinforced

check his answer.

1

8. A well-written program will drawthe correct answer from a studentmost of the time. The student shouldanswer almost all the questions

next frame

9. A program should do more thanmerely present information. It mustteach the student. A program fails ifit does not and it

correctly

must be revised.

10. Programed learning encourageseach student to work at his own speed.He is not held back or rushed. He canmove along atemdily at

teach

11. Programs can be written for pre-sentation in text form or for a teach-ing machine. A student can derive thesame benefits from a as he

his own speed

can from a

12. Either way he is given reinforce-ment after his response by being toldthe correct answer. His

text

teachingmachineis obtained from the correct answer

which he checks after making his

13. Whichever method of presentationis adopted, it is the program not thepresentation that is important. How-ever slick the presentation, it will failif the is badly written.

reinforcemen ,response

14. Programed learning is as good asthe program and the program is asgood as the programer. So the successor failure of a programed coursedepends on the skill of the

program

15. There are other forms and varia-tions - but this small program shouldgive you an idea of some of the basic, rinciples.

programer

Distinguishing Characteristics

At this point you may ask: "What featuresdistinguish programed materials from text-books, workbooks, and other common teach-ing materials?" Persons well versed inprogramed instruction would describe it as ameans of instruction with most, if not all, ofthe following characteristics:

1. Precise definition of objectives interms of learner behavior, the achievementof which can be measured.

2. Presentation of carefully organized andsequenced information.

3. Arrangement of material in relativelysmall steps, each building on the precedingone, using steps that have been researchedto provide almost errorless learning enroute.

4. Individualized instruction - the oppor-tunity IFFeach student to proceed at his ownpace.

5. Structuring so as to elicit frequentresponses, making the student an active par-ticipant throughout the learning experience.

6. Provision for immediate feedback, al-lowing for confirmation or correction ofresponse.

7. Repeated testing or tryout with thetarget population and consequent revision, toinsure that learning outcomes have beenachieved.

8. By reason of the above, an almosterrorless, highly successful path to thedesired learning.

These characteristics allow most usersto specify a minimum acceptable standardfor programed instruction of "90/90." Thismeans that 90 per cent of the students mustpass the final test with a grade of 90 orbetter. If they do not, there is somethingwrong with the program - not with the stu-dents.

These characteristics also allow a studentof lower I.Q, usually to make as good a scoreas his classmates; it just takes him longer.This does not mean that "anybody can learnanything" but it is abig step in that direction.

Research Evidence

No method of instruction has ever comeinto use surrounded by so much researchactivity. Wilbur Schramm, director of the

8

Institute for Communication Research atStanford University, reviewed the researchon programed instruction and summarizedhis findings in an annotated bibliography,"The Research on Programed Instruction."Many of the research conclusions presentedhere are drawn from his publication.

Do students learn from programed in-struction? Schramm concludes "the re-search leaves us in no doubt of this." Theydo, indeed, learn from programs onmachinesand from programs in texts. Every kind ofstudent that programs have been tried on hasbeen able to learn - pre-school, elementary,secondary, college, adult, professional, skill-ed, unskilled, military, deaf, retarded, andprisoners. Using programs, these studentshave been able to learn math, science, history,spelling, electronics, business skills, read-ing skills, basic industrial skills, and manyother subjects.

How well do students learn from pro -ramed as com ared to conven-

of It is di icult teageneralize from comparative research be-cause it is impossible to define a "normalclassroom teaching situation." The outcomeof such research often depends on what kindof teacher is being compared to what kind ofprogram.

Keeping these limitations in mind, weturn to reporting the results of the evaluativestudies made. An analysis by Hartley (1966)indicates that existing programs are over-whelmingly as effective as, or more effective,than traditional forms of instruction. (Figure4)

Such results are even more astoundingwhen one realizes that a majority of thecommercially published programs have notbeen adequately tested. Komoski (1966) re-ported on the testing experience of 291 self-instructional programs currentlymarketed inthe United States. Records show that 36 percent were given no field testing, 32 per centwere tested only once, and 32 per cent weretested more than once.

What is the effect of immediate knowledgeof results? A majority of the studies on pro-gramed learning support the theory thatimmediate knowledge of results contributesto learning.

There are many areas in which the

Figure 4. Results of 112 Studies Comparing Programed and ConventionalInstruction on General Effectiveness (Achievement and Efficiency)

No signi:t?,m difference between 1program and conventional

instruction.

Program significantlymore effective.

41%

research is not conclusive. Logic and cau-tion are needed when making decisions onmatters where only limited research isavailable. Following is a brief summary ofsome of these problem areas.

11

Some schools have constructed study carrelsto help minimize distractions that may in-terfere with the study of programed mate-rials. Photo courtesy of Albert Salerno,Mahoning Valley Vocational School.

9

What type of programing technique is best?By far the most common are the linear andbranching types, each of which may employnumerous variations. These two and a lesscommonly known third type called "mathe-tics" are discussed in more detail later inthis chapter. The great majority of studieson programed instruction has been done withlinear programs. Where comparisons havebeen made, the evidence does not suggestany clear-cut superiority of the linear pro-graming method over the branching, or viceversa. Rather, both approaches appear tohave certain strengths and weaknesses de-pending on the individual student and thetasks to be learned.

Are machine presented programs moreeffective than programed texts? Some pro-grams require a machine for their presen-tation, while some are available in twoseparate versions, one in book form and theother for use with a machine. In any case,it should be emphasized that the so-calledteaching machines, in themselves, do notteach. The teaching is dependent on theprogramed materials which are presentedthrough the machine. Research has notprovided conclusive evidence of outrightsuperiority of either machine or non-machine

presentation of printed materials. The choiceof text or machine presentation probably willhave to be made along pragmatic lines andin terms of the educator's specific purposes.

Much research on programed instructionhas dealt with questions concerning the char-acteristic variables of a linear progrem.What does the research say about thesecharacteristics?

(a.) Size of ste . When significant differ-ences have een ound in learning from pro-grams utilizing different step size, programswith short steps are generally favored. Somereports indicate, however, that superiorstudents often become impatient and boredwith long programs made up of short steps.

*ft

(b.) Overt versus covert ret sc. Theovert response technique requires the stu-dent to recall and write out his answer,while the covert response technique requiresrecognition or selection of the correct re-sponse from a multiple choice of answers.Most studies of this variable report no sig-nificant differences.

(c.) Yndiyiduail pacing. Most of us wouldagree intuitively that students will learnmore efficiently at their own pace. Never-theless, the experimentors have not yet beenable to demonstrate as much advantage forindividual pacing as mightbe expected. Somestudies have indLlated that self-pacing workedbest for superior students.

L

.11,011111141111111.11.,-

trire7 fdl*I..2..1

is ;h; 1,Z.

',6,410 4

2ttoir a i'LAr.ft 4,

#1"1 " -4014vg'gror

4

_At

I

1

_ "rnotor1111114114L,_

Ott '9111111106.

.".411004.-

While adult trainees study their programed texts an instructor assists individual students

whenever help is needed.

Photo courtesy of Dupont Company.

10

Controversies

Disagreements surround almost all newtechniques. Programed instruction is noexception, and many controversies havearisen. Two of these issues merit furtherconsideration here.

(a.) Books versus machines. Basically,a teaching machine is simply a device usedto convey information using the technique ofprogramed instruction. A common view,especially a few years ago, was that pro-gramed instruction required the purchase ofexpensive machines, As Komoski says, "Itis regrettable that audio visualists, out of anhonest desire to positively exploit any and alldevices that might improve instruction, werethe first group within education to activelypromote programed instruction (or as theysaw it, the teaching machine). The fact thatthis group, with its strong machine orienta-tion, was closely associated with the newtechnology served to create the impressionthat the technology was necessarily machine-based. This inevitably helped fortify theapprehension of the rest of the educationalcommunity that the Incipient technology con-tained more of what was threatening than itdid of what was hopeful and promising."

Several machine manufacturers todaypublish their programs in two forms, assheets to be used in their machines and asprogramed textbooks for use where a machineis not necessary or desirable. One supplier,Grolier, provides the following evaluation:"The primary utility of the machine is inthe area of motivation and control. Themachine is often effective in motivating theslow or reluctant learner. The controlfeature is most useful in unsupervised studysituations and in educational research appli-cations."

Certain problems must be confrontedimmediately by anyone desiring tobuyteach-ing machines. A great variety of machinesis available, and there is little standardiza-tion among them. In evaluating any make ormodel of teaching machine, you must alsoassess the type and quality of programsavailable for use in it. The mechanicaldependability of many machines cannot betaken for granted. Potential purchasersshould determine what maintenance problemshave been encountered and the extent to

11

which parts and service are locally available.Although programed textbooks may cost sev-eral dollars for a semester-length course,prices for teaching machines range fromseveral dollars to several thousand dollars.

t

An instructor assists one student in the load-ing of her teaching machine while otherstudents continue to study at their own pace,

Photo courtesy of Welch Scientific Com-pany.

Although many kinds of "machines" -computers, television, tape recorders, andfilm projectors, among others - are beingused to present programed materials, it mustbe emphasized that machines used in theautomation of instruction are only vehiclesfor presenting the instructional program tothe learner.

(b.) Types - Linear, Branching, andMathetics. Throughout most of its shorthistory, there has been general acknowledg-ment of two techniques of programing, thelinear method and the branching or intrinsicmethod. Lately another technique, known asmathetics, has begun to gain acceptance asan efficient and tested method for impartingknowledge and skills. Although there areproponents for all three techniques who willargue that their particular approach is best,the research thus far available fails to indi-cate superiority of any one technique for allsituations. No two tasks present the sameteaching problem and no single teachingtechnique can solve every problem.

The majority of current programs arelinear. Characteristics oi linearprogramingare: (a) frames are very short, usually notmore than two or three lines, (b) studentmust use recall to construct the response,(c) basic premise is that the learner shouldbe successful at all times, (d) student pro-ceeds through each step from beginning toend. Some linear programs incorporate an"express-stopping" technique which directsa student to skip ahead if he has demonstratedadvanced knowledge.

The advocates of branch programing be-lieve that the student can learn efficientlyfrom errors as well as from respondingcorrectly, and that he can absorb a relativelylarge quantity of information at one time.Characteristics of the branching techniqueare: (a) frames are large, often includeentire paragraphs, (b) student is asked torecognize the correct response usually froma multi-choice type question, (c) assumptionmade that student learns from successes andmistakes, (d) student is individually handledby use of sub-sequences which branch offfrom the main line; a failure at a crucialpoint leads to alternate remedial materials.In most branching programs the program isconstructed so the choice of a particularanswer to a diagnostic question determineswhich frame will be presented next.

Mathetics is a relatively new programingtechnique being popularized by Teco Instruc-tion (Ft. Lauderdale, Florida). Its format,when frames are used, resembles the branch-ing technique. Individuation exercises areincluded, which may be skipped entirely bythose already proficient but permit moreinstruction for those who need it. A majorfeature of this technique is the degree oftask simulation used. Advocates hold thatthe greater the simulation, the greater thetransfer. Therefore, many mathetical pro-grams employ kits or simulators whichinclude components quite similar to thosewhich are used in the actual task. High costsof simulation have presented some economicproblems. Printed matter in mathetics formis usually profuse with diagrams andpicturesin varying stages of completion. Accordingto its advocates, the active responses re-quired of the student during the program arealways, to the degree required, simulatedperformance oZ the task he is learning.

12

For example, if the task were inspectingsteel drills, the kit accompanying the pro-gram would include a drill gauge and someactual drills ground in various ways. In-corporated into the program at appropriateplaces would be instructions which tell thestudent to pick up his drill gauge and checkspecific drills for lip clearance, lip angle,or whatever, depending on the point beingmade by the program.

There is little basis at present to favorany one of these general types over theother. Specific types of programs are likelyto be useful for particular teaching problems.In some instances the most effective programmay involve a combination of these differentstyles.

Some Advantages

Although many of the advantages of pro-gramed materials have already been men-tioned, others have not been. For the con-venience of the reader, those consideredworthy of special note are summarizedhere.

1. They are self-paced, permit eachlearner to move through the sequence ofresponses at his own speed, and thus ap-proach private tutoring.

2. They can free the teacher from theroutine and drillmaster tasks of instructionand provide him with more time for creativeand interpersonal activities with students.

3. They can successfully teach most kindsof information and certain skills.

4. They are efficient - unnecessary verb-iage is eliminated and only information cru-cial in terms of the program's stated objec-tives remains.

5. They are based on sound theories ofeducation and psychology.

6. The information presented is organizedand sequenced for individual readiness.

7. Once oriented and motivated by theteacher, the student can learn alone with agood program.

8. A single teacher can monitor and helpindividual students who are working on avariety of subjects at the same time.

9. Learning tends to be of higher qualityfor all students because of the individualpacing and better control over what is pre-sented and how it is presented.

10. The low error rate of most programed

materials is a great motivational tool initself - especially for slower students whomay be successful at learning for the firsttime.

Several other benefits of programed in-struction have been realized by industrialgiants like Dupont and the Proctor andGamble Company. Most industrial traininginvolves adults, who, for the most part, areseriously motivated by the desire to improvetheir rank and stature within the company.Whether these gains will be possible inoccupational and other public educationalsituations still remains to be seen. Withthese precautions aired as a check againstoverenthusiasm the following industrialbene-fits are listed:

(a.) A 25-50 per cent reduction in train-ing time.

(b.) 10-25 per cent more effective learn-ing.

(c.) Savings of $30 to $60 per student percourse, as compared to conventional trainingmethods.

(d.) Uniform high quality instructionwhich is transmitted to job performance.

Some Limitations

Despite the fact that programed instruc-tion has shown great potential, it is likemost any other technological improvement inthat it also has some limitations and dis-advantages.

1. It cannot take the place of goodschoolsor skilled teachers.

2. It cannot solve the problems of over-crowded classrooms.

3. It cannot succeed in classrooms whereteacher attitudes toward it are hostile.

4. It cannot provide effective instructionunless the materials selected are properlyprepared and tested.

5. Some students become bored afterworking with programs for some length oftime.

6. Administrative problems of schedulingmay arise when students using programedinstruction and finishing at different timesare scheduled for subsequent training as anintact group.

13

7. Teachers who will use programed in-struction must be trained in the use andclassroom management of such materials.

8. Selection of good programs that willfit in or complement the existent curriculumis not easy.

9. As with any technology, certain costsare involved - acquiring the program, train-ing teachers to use it, and evaluating it.

10. The number of good programs availablecommercially is quite limited in some con-tent areas. One survey indicated that approx-imately sixty per cent of the programs avail-able were in the areas of mathematics,science, and English.

Student Reactions

Student evaluation of and comments abouta new medium of instruction provide a goodindication of how well they accept it and, inturn, an indication of how well they are likelyto learn and like learning from it. The eval-uation summary and student comments quotedbelow are reprinted through the courtesy ofMahoning Valley Vocational School, Vienna,Ohio, and its basic education supervisor,Albert A. Salerno. This school was createdto provide saleable vocational skills foryoung men aged 16-21 who are either un-employed or underemployed. The ProgramedLearning Center at Mahoning was opened inAugust 1965. Using programed instruction inthe Center atmosphere was another steptoward "customized education" writes Mr.Salerno, Their philosophy of customizededucation is "an attempt to design a com-plete educational program for each of thestudents at the student's achievement level."

The student evaluation sheet used by arandom sampling of 100 students at Mahoningat the end of one year is reproduced in Fig-ure 5. The figures represent a summary ofthe students' responses. The evaluation sheetincludes two items each on magazines andmusic which were included to sample studentreaction to vocational magazines available intheir library and to the playing of classicalor semi-classical music three to four hoursduring the day.

Figure 5: STUDENT EVALUATION SHEET - PROGRAMED LEARNING CENTER

Please check as many boxes concerning your thoughts on the Programed Learning Centeras you wish. We are very interested in your reactions to Programed Learning and to ourCenter. You may also want to add some of your own ideas in the space allotted at the bottom

of the page. Naturally it isn't necessary for you to sign your name. Mr. Albert Salerno

"It helped me, and Ilearned a lot"

77

"Didn't help" 0

"Liked it" 89

"Waste of time" 0

"Interesting" 84

"Boring" 4

"Prefer to classroom 32

"Good Review" 60

"Efficient" 58

"Explained clearly"

"Would like P.L. inother classes."

"Made me think"

"Know immediately whyPini t or wrong"

"Liked working alone"

"Helped in my Vocationalclass"

"Didn't really help inmy Vocational

"Good study atmos-

"Poor place to do anystud n "

59 "Don't like the music" 4

52 "Like the music" 88

61 "Am in P.L. too manyhours"

3

57 "Should be open atnite"

59

72 "No need to open at nite" 1

64 "Like the magazines" 79

9 "Don't like the maga-zines"

2

69 "Am in P. LC. too fewhours"

43

1

"Didn't like it" 0 "Not enou h teacher help" 6ligievt.uation sheet was i ec out by a random sampling of 100 students.

THIS SPACE IS FOR WRITING ANY ADDITIONAL COMMENTS THAT YOU MAY HAVE CON-

CERNING THE CENTER: (Note: This is a summary of the actual comments written by the

100 students.)

"This kind of program in high school would have helped the slow student gain confidence.""Best equipped study center I have ever been exposed to.""Quiet and convenient enough to do anything.""Very helpful to me in Algebra.""Teaches the student how to help himself.""To the rest of the guys I've talked to P.L. has helped in their vocational areas.""Need more books.""Good chance to review.""Get a greater variety of magazines.""There is no one on my back, if I feel uninterested I can do something else.""Can learn better at my own speed.""First time I've had a self-tutor program and it is challenging.""All trainees should take advantage of this. It teaches the fundamentals of a subject area."

14

CHAPTER III

SELECTION AND USE OF PROGRAMED MATERIALS

This chapter will cover only brieflycertain topics related to the selection anduse of programed materials. The brevityshould not be regarded as an indication ofthe importance of the points covered. Se-lecting the best program available for yourteaching objectives is highly important andnot an easy task. And even the best pro-gram needs to be properly integrated intothe existing curriculum and properly ad-ministered if its full potential is to berealized.

What Can Pros ramed Materials Teach?

In theory, any subject that can be verbal-ized and/or illustrated in one way or anothercan be programed. Programing helps peoplelearn simple skills such as accurate meas-urement with a micrometer, and complexskills such as calibration of a precisioninstrument. It is possible through a self-instructional program to teach a rathercomplex combination of cognitive and motorskills - the reading and interpretation of anelectrocardiogram tracing, for example.

Programed instruction has been mostsuccessful to date in teaching cognitivematerial - facts, principles, concept s.These programs can be used from elemen-tary to college levels for a great variety ofsubjects. One guide to available programswhich classifies them by subject lists over80 different subject matter areas.

Industry has made extensive use of pro-gramed materials in teachingbasic industrialskills. A study in which this writer was in-volved raised serious question as to whetherprogramed materials alone can satisfactorilyteach a complex psychomotor skill such asregrinding a drill. The students had nodifficulty in learning the necessary cognitivematerial from the program but had con-siderable difficulty performing the motionsrequired for proper regrinding.

When teaching a skill, some programscall for the student to read each step in thebook and then perform it with the actual

15

equipment or simulated equipment. Similar-ly, the program may be supplemented withillustrations, photographs; samples, or otherdevices. These are usually called panels orexhibits and are designed as an integral partof the program. This procedure is basic tothe mathetics programing technique and tothis writer's knowledge is very effective.After such self-instruction, the learner us-ually performs the task in a real situationunder the observation of a qualified instruc-tor.

we

SW/

Visual techniques are sometAnes used to pre-sent vocational subject mater.Photo courtesy of Albert Salerno, MahoningValley Vocational School.

How Should Programs Be Used?

Much remains to be learned about how aprogram can best be used in a course. Onefact is certain - programs can be used in avariety of ways. They may be employed asthe main source from which students areexpected to learn facts, principles, or certainskills. Some feel there is a place for pro-grams which will even teach a subject with-out an instructor. These might be appropriatefor adults or for remedial courses designedto provide review for students enteringcollege without sufficient preparation.

Other programs should be used to teachparts of courses, to introduce other instruc-tion, or to provide enrichment material forstudents with special interests and abilities.In most schools, programs should be used inconjunction with other media as one of themeans of teaching a course.

The NEA sound filmstrip, "Selection andUse of Programed Materials," summarizesthe seven most common uses of programs asfollows:

1. Basic instruction.2. Incidental instruction.3. Correlated instruction.4. Remedial instruction.5. Enrichment.6. Substitute instruction.7. Research.

What Kinds of Materials Are Available?

There has recently been a dramatic in-crease in the number of commercially avail-able programed texts in a variety of subjectmatter areas. More will be said later, butmere availability is no guarantee of quality.

The most comprehensive list of pro-gramed courses available in this country hasbeen produced by Carl Hendershot of DeltaCollege, Bay City, Michigan. His publication,"Programmed Learning: A Bibliography ofPrograms and Presentation Devices," in-cludes programs covering over 80 subjectmatter areas. The current revision (1967)lists the addresses of 150 program publish-ers and includes over 2,500 programs withbrief descriptive information about each.Pertinent data given include the title, author,approximate length, intended student popula-tion, and price. No attempt has been madeto evaluate the programs, but the compilerhas pointed out prerequisite knowledge andother facts to help determine the program'spossibilities for various instructional situa-tions.

In Appendix A this writer has compiledlists of publishers of programs relevant tooccupational education. .The listing does notinclude prices, titles, or other descriptiveinformation. Because new programed mate-rials are appearing rapidly, all availableprograms may not be listed here. To staywell informed, you must subscribe to one ormore of the general listings and/or request

16

that individual publishers keep you on theirmailing list.

What can be done if there are no highquality programs available in the desiredsubject area? The answer of course is toconsider the possibility of writing your own.

er, however, that preparing a pro-gramed instruction course is a demandingand exacting task - and usually a long one.The typical program author needs about sixmonths to complete his research, writing,testing, revisions, editing, and productionbefore the course is ready to be used. Shortcourses in how to program (offered by theUniversity of Rochester and the Universityof Michigan) are highly recommended to any-one planning to do much programing.

The Denver, Colorado, school system hasdiscovered certain advantages to encourag-ing some local production even though itdepends on commercial materials for mostneeds. Locally produced programs may bemore tailored to the needs of the particularschool. And more importantly, they conclude,the principles of programing will tend tospill over and improve all teaching and allteaching materials.

What Are the Economics of UsingProgramedMaterials?

Figure 6, which shows the trend in costsof programed texts, was prepared by theeditors of Programed Instruction. While in1963 slightly more than half of all programssold for school use were priced at $4.00 orless, during 1964-65, 73 per cent of the pro-grams produced were sold in this cost range.This trend occurred in spite of the fact thatduring the same period the average length ofprograms increased. Most publishers areattempting to make programed materialsmore economically competitive with otherkinds of instructional materials. One waythis can be accomplished is by producingreusable programs which require only thepurchase or duplication, if permitted, of stu-dent answer sheets in succeeding years.

Teaching machines vary in cost fromSeveral dollars to several thousand dollars.Their costs have remained substantially thesame probably because the high cost of somehardware and its fragility in student handshave resulted in a low sales volume. The

cost of programs designed for machine usehas declined somewhat.

Figure 6. Programed Text Costs

5%

5%

10%

9%

14%

3

27%

More than $17.00

less than$2.00

11%

35%

40%

1963 1964-65

Local or in-house production of testedhigh quality materials can be quite costly.Dupont estimates that it costs them approx-imately $3,000 to $4,000 per course hour formaterials developed by company personnel.Most school teachers and administratorsshould carefully scrutinize commerciallyavailable programs before attempting towrite their own.

The cost of industrial training by pro-gramed instruction, as reported by the Re-sources Development Corporation, is sur-prisingly low - less than $1.00 per hour oftraining for materials developed for industry-wide audiences. This, they report, comparesvery favorably with the cost of assembledtraining - instructor time, classroom facili-

17

ties, training materials, and other overheadwhich starts at $11.00 per hour per man.Needless to say, public educators have notbeen able to realize similar economic bene-fits.

What Selection Procedures are Recommend-ed?

Program evaluation is difficult. Thebenefits of programed instruction can berealized only if users have adequate infor-mation with which to evaluate self-instruc-tional materials. Since many programs ofvarying quality (from poor to excellent) areavailable, careful selection is crucial. Somepractical suggestions are presented herewhich will assist teachers and administratorscharged with this responsibility. Because ofspace limitations, this discussion is neces-sarily brief; however, evaluation is of suchimportance that in Appendix C a classicstatement on "Criteria for Assessing Pro-gramed Instructional Materials" is repro-duced. This statement was prepared by ajoint committee made up of representativesfrom several professional organizations con-cerned about providing direction to the pro-duction, dissemination, and utilization ofprogramed materials.

The following steps are recommended asa procedure for evaluating a program.

1. Read the program carefully and thor-oughly.

2. Compare the compatibility of its con-tent with your instructional objectives.

3. Examine the adequacy of programing.The frames should present a careful, logicalprogression toward mastery of the subjectmatter. Students should be required to re-spond to critical aspects of each item or toperform the operation that step was meantto teach. The format should be an attractiveone, appropriate to the subject matter andto your students. The reading level must besuitable.

4. Examine the field test and other vali-dation data available. They should provideinformation on the program's effectivenessand efficiency.

5. Experiment with the program on asmall group of students before adopting iton a wide scale.

What Administrative Changes Will Require What Changes May Occur in the Teacher'sRole?

Attention?

Effective use of programed instruction inany school system will depend upon admin-istrative assistance, guidance, and support.Charged with the responsibility for guidingand unifying the efforts of the school systemtoward specific objectives, the administratorwill want to consider the following questionsrelating to the use of programed instruction.

1. What orientation or training should beprovided teachers before they use programs?Consideration should be given to holding in-service workshops for the faculty to acquaintthem with the medium.

2. What orientation should be providedstudents and parents? Some publicity aboutthe nature of programed instruction andreasons for introducing it is recommended.

3. What effects will the use of programedmaterials have on curriculum and schedul-ing? Certain curriculum changes may bedesirable because of the greater opportunityfor individualized instruction. In d i v i du a 1pacing may also require more flexible sched-uling; students starting a program at thesame time will not all finish at the sametime.

4. What will be the financial needs?Probably an overall financial plan should beestablished, one which looks ahead to con-tinued use of programed materials. Considerthe costs of acquiring programs, trainingteachers to use them, and evaluating them.Financial assistance is available from thefederal government under several educationacts.

5. What logistical problems - s p a c e,building design, etc. - will need attention?Some schools are establishing special pro-gramed learning libraries where studentscan come, check out a programed text, andstudy during their free time. Most use pro-grams in their regular classrooms. Othershave built carrels for individual study.

6. What changes may be necessary inreporting pupil achievement? In most cases,regular course credits probably will begranted upon passing course exams, butadditional credit may be granted to studentswho complete extra programed course mate-rial.

18

Though his role may change considerablyas more and more technology finds acceptancein the classroom, the teacher always hasbeen and will continue to be the centralfigure in any instructional program. If,through programed instruction and othertechnology, we free the teacher of the burdenof routine instruction in facts, computations,and the like, the teacher will then have moretime to do the things that only he can do.Programed instruction and teachers havedifferent purposes.

The following questions reflect some ofthe changes likely to occur in the teacherrole as a result of using programed instruc-tion.

1. How will the self-pacing quality ofprogramed instruction affect classroom pro-cedures? Teachers will have to learn howtomanage programs effectively, lest they be-come a burden instead of an aid. If a classof students are to finish a program at aboutthe same time, then there will have to beconsiderable difference in the starting times.Use of programed materials must be planned.

2. How will use of programed instructionaffect the teacher's work in the classroom?Although programed materials will relievethe teacher of much that is routine, theteacher must still supervise use of the mate-rials. More time will be spent working withindividuals or small groups when usingprogramed instruction than was possible withconventional instruction.

3. How can the teacher reinforce theprogram? By being thoroughly familiar withits content, by using good management pro-cedures, and by displaying a positive attitudetoward its use, a teacher can contributegreatly to successful use of a program. Aswith any type of instruction, the teachershould develop an enthusiastic climate forlearning.

4. How will the teacher evaluate studentprogress? The method of grading and test-ing should be carefully explained to students.They must understand they are not beingtested while working on the program itself.Regular testing and reporting proceduresare easily adapted for use with programedinstruction.

5. What should be the teacher's responsi-bility with regard to programed instruction?Certainly the teacher should strive tobecomewell informed about programed instructionand its uses. In most schools, teachers usingprogramed materials will have a voice in

their selection. Thus, the teacher must learnto distinguish between a poor program and agood one. He should want to attend in-serviceworkshops to learn about program develop-ment, to get experience in writing frames,and to discuss their use in the classroom.

CHAPTER IV

SOME SUCCESSFUL APPLICATIONS OF PROGRAMED INSTRUCTION

This chapter reports on several casestudies where programed instruction has beensuccessfully used. Case histories involvingas many occupational education areas aspossible have been included, although casesare not specifically identified as to vocationalfield. The histories include situations involv-ing elementary and secondary education, in-dustrial training, general and vocationalsubjects.

1. Subject Area: Basic Industrial Skills

Program(s): VariedResults: Dupont's experience withprogram-ed instruction courses in 106 skills since1959 shows that this new technique trainsoperators 25-50 per cent faster and 10-25

per cent more effectively. In the new tech-nique of programed instruction, they found atraining tool that costs $30-$60 less perstudent per course compared to conventionaltraining methods.Source: Dr. Arthur Santora, Head, IndustrialTraining Service, Dupont Company, Wilming-ton, Delaware.

2. Subject Area: Business

Program(s): Several texts using themathet-ics design (including simulated practice ex-ercises) to train bank tellers.Results: Retention immediately imp rovedand the amount of verbal instruction wasreduced from 48 hours to 6 hours; resultingin a shortening of the average training per-iod from five weeks to as little as threeweeks.Source: Gordon Rhodes, Director of Train-

19

ing, First National City Bank, New York,New York.

3. Subject Area: Elementary

Program(s): VariedResults: "Experimentation conducted thusfar supports the expectation that good pro-grams, carefully developed, can significantlyimprove the quality and economy of instruc-tion. Whether any particular program willdo so is subject to question until establishedby adequate tests of that program."Source: Programed Instruction Project -Annual Report, 1965-66, Board of Educationof the City of New York, New York.

4. Subject Area: English

Program(s): Grammar from commercialsource, one year length.Results: Respondent reported, "Programused with accelerated classes. This methodof teaching English grammar was found to beas good or better than the conventionalmethod for these classes. The method con-sumed less instructional time."Source: Warren, Ohio Public School systemas quoted in NEA's Educational ResearchService Circular #7, 1966.

5. Subject Area: General

Program(s): VariedResults: Dr. Carl Hendershot, consultant tobusiness, industry, colleges, and schools,has experienced extensive association withthe selection and use of programed instruc-tion. At Delta College, programs have been

used under his direction since 1961 as basictexts, supplementary texts, and for both adultand student self-instruction. In all of theseapplications, programed texts selected tomeet the requirements of instruction and theneeds of the student have produced highlydesirable results.Source: Coordinator of Improvement, DeltaCollege, University Center, Michigan.

6. SubctAxea: Health-Medical Education

Program(s): A linear program on "Allergyand Hypersensitivity"Results: Announced in 1963, the demand forthis publication by individuals, m e di c a 1schools, local. medical societies, specialtygroups, and others has been so extensivethat over 200,000 copies have been dis-tributed. More than two-thirds of the med-ical colleges are using the program as aninstructional segment of their curricula.Source: Dr. Jerome Lysaught, College ofEducation, University of Rochester, Roch-ester, New York.

7. Subject Area: Health and Safety

Program(s): A two-hour program on how to"positively reinforce" employees for properlifting.Results: Program was administered to su-permarket supervisors in two widely sep-arated parts of the country. Before thetraining, 39 back injuries were reported ina six-month period; only three were re-ported in the six months following training.(Average cost of each back injury estimatedat $4,050.)Source: Brochure prepared by Center forProgrammed Learning for Business, TheUniversity of Michigan, Ann Arbor, Michigan.

8. Subject Area: Home Economics

Program(s): Nutrition, from commercialsource, one semester length.Results: Respondent reported, "We receivedreasonably good results from this programin meeting a specific need. Programs cen-tering on a specific topic seem more suc-cessful than total programs."Source: Gary, Indiana, Public School system,as quoted in NEA's Educational Research

20

Service Circular #7, 1966.

9. Subject Area: Industrial Training

Program(s): VariedResults: Dr. Garland S. Wollard, directorof education for the Federal Bureau of Pris-ons, reports that the experiences gained atthe Petersburg Reformatory have been soencouraging that he plans to introduce pro-gramed instruction to the other five federalreformatories as well as to two federalpenitentiaries. One of the by-products of thePetersburg experiment was that inmatesfelt they could do something to improvethemselves. This replaced a feeling thatthey were being "used" by the institution.Source: Article in New York Times by GerdWilcke, spring 1967.

10. Subject Area: Industrial Training

Program(s): VariedResults: The Boeing Company first offeredprogramed instruction on a voluntary trialbasis in January 1966, Results have beensuch that they expect to enroll 10;000 stu-dents in 50 courses in 1967. Employees haveto pay for their own text, study the courseoff-hours at the time and place of their ownchoosing, and at their own pace. Some over-all conclusions have already been made: (a)A large majority of their employees likes theprogramed instruction method. They want tolearn whenever and wherever they choose, attheir own speed, away from the classroom.(b) Future course enrollment and interest inprogramed instruction and adult educationwill continue to accelerate.Source: Article in "Training in Businessand Industry" by Paul Watson, ProgramedInstruction Coordinator, The Boeing Com-pany, Seattle, Washington.

11. Subject Area: Medical - Nursing Sales

Program(s): Two programs, each approxi-mately six hours long, one on a complexdisease and the other on a drug to combat it.Results: New company salesmen were re-luctant to discuss this drug with physiciansuntil they had been on the job six to ninemonths. After completing the program, newsalesmen now approach the physician on the

particular drug within the first month theyare on the job, resulting in new sales per-sonnel being productive six months sooner.Source: Brochure prepared by Center forProgrammed Learning for Business, TheUniversity of Michigan, Ann Arbor, Michigan.

12. Subject Area: Personnel Practices

Program(s): One and a half hour programdeveloped to train supervisors in the use ofa new complex computer form for personnelrecords.Results: Program was distributed to 5,000automobile manufacturing employees acrossthe country before switching to new system.New system is functioning with no problems,even though errors had to be non-existent inorder to insure its success.Source: Brochure prepared by Center forProgrammed Learning for Business, TheUniversity of Michigan, Ann Arbor, Michigan.

13. Subject Area: Trade and Industrial

Program(s): Machine Shop - programedmaterials include slides, tapes, and printedmaterials - all teacher produced.Results: Respondent reported, "Has trulyindividualized instruction and made instruc-tor available for greater personal super-vision. Also makes last hour class as effec-tive as first."Source: Independence, Missouri, Pub 1 i cSchool system, as quoted in NEA's Educa-tional Research Service Circular #7, 1966.

14. Subject Area: Selling and Retailing

Program(s): "Making Courtesy Work forYou" and "Making Suggestions to IncreaseSales."Results: The actual sales increases record-ed have convinced Allied Stores of the pro-grams' effectiveness. Tests showed an in-crease in sales dollars per hour over thecontrol group of 25 per cent by the pro-gramed group and 8.3 per cent by the class-room group. As a result of these tests,store personnel and trail:ling directors havebeen advised to use the programed manualsas the basic tools of retail sales training.Source: Article in "Department Store Econ-omist," written by Frederick Finigan, Per-

21

sonnel Director of Allied Stores, April 1965.

15. Subject Area: Selling and Retailing

Program(s): Thirteen separate self-studyunits on retail food store operations.Results: A research study was conducted byWestern Michigan University to determinethe effectiveness of the program. Testswere administered to a typical group of storeemployees before and after they studied theunits. A substantial and significant improve-ment was found in the knowledge level of thegroup. The employees found the materialsinteresting, they found the learning easy, andthey felt they had significantly increasedtheir understanding of store operations.Source: Announcement brochure from SalesTraining Department, The QuakerOats Com-pany, Merchandise Mart Plaza, Chicago,Illinois 60654.

16. Subject Area: Spelling

Program(s): TMI-Grolier SpellingResults: A controlled experiment was con-ducted using two classes of 30 students eachselected from a cross-section of abilitylevels. One sixth grade group was given theprogram and the other was taught conven-tionally with textbooks. Students who usedthe program scored higher gains after eightweeks of study than the conventionally-taughtclass after twelve weeks of study. Pro-gramed learning saved time for above averagestudents and eliminated the problem of lossof instruction resulting from illness as wellas the problem of how to work with a newstudent in the class.Source: Research Department, EducationalDivisions, Grolier Inc., New York.

17. Subject Area: Vocational Apiculture

Program(s): Parliamentary Procedure, fiveto seven hours.Results: A research study conducted at Cor-nell University studied the relative effective-ness of supplementing programed instructionwith blocked versus spaced review. Theprogram was administered to 279 first yearvocational agriculture students in New YorkState. Conclusions drawn were that neitherspaced nor blocked review resulted in sig-

nificantly better performance than was ob-tained from the program alone.Source: Final Report, "The Relative Effect-iveness of Supplementing Programed Instruc-tion with Blocked Versus Spaced Review,"by James A. Scanlon, Cornell University,May 1967.

18. Subject Area: Vocational

Program(s): Varied - 14 vocational areasResults: Programed courses at MahoningValley Vocational School offered students achance to pursue vocationally oriented sub-

jects at their own pace, with a minimum oftrouble or error. The low error rates andquestions-response-check format of pro-gramed instruction was a great motivationaltool in itself. This type of learning situationactually built confidence and gave directionto students. Changes in skill behavior coulddefinitely be observed in their vocationalclasses. An increase in the knowledge andskills experienced by our students was mostevident on our post-test results.

Three conclusions drawn by the coordin-ator of the Mahoning Programed LearningCenter at the end of the first year were as

.rams..w-Aomr

SELF.STODYI

yA9UAlS j:

Some vocational schools and many companies are establishing programed learning centers orlibraries. Students may be allowed to select materials according to their interests or beassigned specific materials. Photo courtesy of Dupont Company.

22

follows:1. That programed texts were successful

in vocational training.2. That students who are properly ori-

ented and counseled can work for extendedperiods of time on programed courses.

3. That both basic education and voca-tional skills can be developed or improvedupon through programed instruction.

According to the basic education super-visor, experimentation in this new educa-tional device has brought many benefits toMahoning Valley and the vocational move-ment in Ohio. Vocational trainees are nowbetter prepared for jobs as a result of thissuccess, joint vocational schools in Ohio arenow incorporating programed learning cen-ters in building plans, and Manpower De-velopment and Training Act city programsare incorporating programed learning cen-ters in Ohio.Source: Albert Salerno, Mahoning ValleyVocational School, P. O. Box 278, Vienna,Ohio 44473.

Recommendations

Programed instruction can be a successor a failure in any school system. Althoughthe potential is great, programing is not acure-all. Teachers and administrators mustbe careful in the use of programed instruc-tional materials. The following recommen-dations can help in effectively developing thepotentialities of programed instruction.

1. Learn as much as possible about pro-gramed instruction before you attempt to useit. See Appendix B for suggested readings.

2. Seek the opinions of persons usingprogramed instruction in your area, prefer -.eably those teaching your subject. If this isnot possible, write to teachers in otherlocations.

3. Involve qualified teachers who arewilling to learn about and try something new.

4. Inform faculty members, students,

It is impossible to present a completepicture of current developments and applica-tions of programed instruction, The inter-ested reader is referred to the following tworeports which contain additional case studies.

"Programed Instruction in Large SchoolSystems"

Educational Research Service Circular No.7, September, 1966.

National Education Association, Washington,D.C.

(Available for $1.75 each from EducationalResearch Service, 1201 Sixteenth Street,Washington, D. C.)

Four Case Studies of Programed InstructionFund for the Advancement of Education477 Madison Avenue, New York, New York

10022(Dated June 1964, available free.)

CHAPTER V

RECOMMENDATIONS AND CONCLUSIONS

23

parents, and others in the community aboutprogramed instruction and seek their help inevaluating it.

5. Select materials very carefully. Goodprograms are challenging, not boring orinsulting. The potential user of programedinstruction should learn to discriminate be-tween an actual program and conventionalinstruction presented in aprogramed format.Remember, evidence based on student per-formance is needed to demonstrate that aprogram actually teaches.

6. Try programed instruction on a smallscale. Consider the project an experimentuntil it is proved successful in your school.Wide scale adoption can follow.

7. Follow good management procedures.The programed instruction effort must beeffectively managed. Without direction orpurpose, its effects are unpredictable -sometimes it will work well, at other times

the effect may be unclear, and sometimes itwill fail miserably.

8. Integrate programed instruction withother methods of instruction. Programedmaterials should be viewed as part of a vastarray of instructional materials available toteachers and students. The full potential ofprogramed instruction cannot be realized byits use alone; instead, the teacher must useit and other methods to supplement his effortsin the best possible way.Conclusions

Programed instruction shows outstandingpromise of making a definite contribution toeducation. An uncritical acceptance of pro-gramed materials or a premature or unin-formed rejection of them would be mostunfortunate. All indications are that withimagination, resourcefulness, and adequatepreparation in the use of this medium, allconcerned will benefit. As teachers gainexperience with the technique, the benefitsand learning principles involved in programedinstruction also may be incorporated intoother media of instruction.

Programed instruction has been called"infinite consideration for the learner." Itrepresents the first real penetration of psy-chological theory into the educational proc-ess. Research conducted thus far supportsthe contention that good programs, carefullydeveloped and properly administered, cansignificantly improve the quality and economyof instruction.

The advantages of programed instructionoutweigh its disadvantages. Its limitationsmust be recognized; it is not a panacea forall of education's ills, nor will it revolu-tionize education overnight. However, itsmain advantages - individualized rate of in-struction, shortened learning time, moreeffective learning, and ability to relieveteacher fatigue - cannot be denied or ig-nored by educators.

Programed materials offer a greatamount of versatility in occupational educa-tion. They may be used as developmental,remedial, or accelerated tools dependingupon the students and their needs. Adultsas well can profit a great deal from self-instructional materials. The logical struc-ture and built-in success factor of prograinedmaterials offer students a chance to increaseknowledge and skills at the same time mo-

24

tivation to learn is being enhanced.The successes already reported by its

users in both industry and education arewidespread. It has gained acceptance inover 9,000 companies, an increasing numberof general, technical, and vocational schools,the armed forces, and several governmentagencies, as well as in several foreigncountries.

At present programed texts offer themost economical route available to educatorswho want to move toward individualized in-struction. Although the costs of producingprograms locally may be prohibitive formost school systems, the purchase of com-mercially prepared materials is not. Withthe latter, the problem is more likely to befinding a program of high quality in thespecific subject matter area desired.

A very skillful teacher can do whatever aprogram can do and at least as well as theprogram can do it - with one exception. Theteacher can tutor only one student at a time.Programed instruction is one of a variety ofinstructional devices that should be put atthe command of the teacher to help him dohis job better. The teacher has been andwill continue to be responsible for decisionsabout what is taught and how it shall betaught. Neither the teacher alone nor theprogram alone can do the effective job thatthe proper combination of both can do.

Following are a few predictions of whatthe future may hold for programed instruc-tion.

1. History will someday record that pro-gramed instruction was the most effectivedevelopment in giving access to learningsince the invention of printing.

2. 1975 will find programming a majortechnique of instruction at all educationallevels, in all parts of the world.

3. Although some school% and teacherswill prepare their own programs, in mosteducational systems teachers will use com-mercially prepared programs.

4. Lastly, this writer agrees completelywith the superintendent who said, "Althoughit is becoming evident that programed in-struction can teach without a teacher's as-sistance, it seems to us that its greatestpromise lies in its uses by an able, enthusi-astic teacher as one of several tools."

See for yourself - try programed in-struction in your school.

APPENDIX A

PROGRAMED TEXT MATERIALS AVAILABLE IN OCCUPATIONAL EDUCATION

SUBJECT MATTER AREAS AND THEIR PUBLISHERS

Since programed text publishers often produce and sell materials in several subject matterareas, this listing has been divided into two parts to avoid unnecessary duplication of sources.Part I lists the subject matter areas and the code numbers given to publishers who havematerials available in that general area. These code numbers in turn correspond to a masterlist of 69 program publishers contained in Part II.

No claim is made that all available subject matter areas and their publishers are listed.Neither does listing here imply that the author has evaluated the program. Rather, listingswere made on the basis of general announcements made available to the author by publishers.

PART I

Subject Areas and Sources

Adult Basic EducationSource(s): 24, 41

Agriculture - GeneralSource(s): 18, 19, 58, 67

Agricultural MechanicsSource(s): 9, 18, 21, 32, 53, 67

Banking and FinanceSource(s): 1, 2, 17, 22, 39, 64

Blueprint ReadingSource(s): 41, 69

Bookkeeping and AccountingSource(s): 1, 3, 19, 21, 38, 41, 49, 66

Business and Office Education - GeneralSource(s): 3, 8, 28, 56, 59, 63

Business EnglishSource(s): 8, 9, 17, 19, 41

Business LawSource(s): 1, 17

Business-Machine OperationSource(s): 13, 22, 37, 46

25

Business ManagementSource(s): 8, 9, 17, 21, 22, 27, 38, 41, 50, 63, 68, 69

Business MathematicsSource(s): 41, 59, 63

Business-Office PracticeSource(s): 8, 38

Business-Typing and ShorthandSource(s): 3, 29, 38, 41

ConstructionSource(s): 64, 69

Data ProcessingSource(s): 5, 9, 17, 19, 31, 37, 41, 46, 48, 49, 68, 69

Drawing and GraphicsSource(s): 3, 18, 41

EconomicsSource(s): 9, 10, 20, 21, 22, 41, 43

ElectricitySource(s): 1, 3, 17, 18, 23, 26, 29, 31, 38, 41, 49, 53, 68, 69

ElectronicsSource(s): 3,,9 15, 17, 20, 31, 38, 41, 49, 51, 65, 69

HealthSource(s : 10 21

Home EconomicsSource(s): 21, 50

Industrial, Trade and Technical EducationSource(s): 3, 5, 6, 8,

InsuranceSource(s): 1, 17, 52

Mathematics-ArithmeticSource(s): 1, 14, 17,

Mathematics-ModernSource(s): 7, 10, 21,

Mathematics-Slide RuleSource(s): 25, 30, 41

9,

18,

29,

18,

21,

31,

23,

28,

33,

27,

29,

34,

38,

30,

39,

41,

34,

41,

53,

39,

43,

69

41,

56,

42,

68

53, 56, 64

Mathematics-Other AreasSources(s): 1, 7, 10, 14, 17, 18, 20, 21, 25, 28, 29, 30, 33, 34, 41, 42, 48, 49, 51, 53, 57,

68

MeasurementSource(s): 18, 38, 44, 53, 69

Medical and NursingSource(s): 7, 9, 16, 20, 21, 29, 36, 40, 42, 45, 47, 55, 60, 61, 62, 63, 68

Parliamentary ProcedureSource(s): 17, 30, 42, 57

Peromnel Practices and SupervisionSources): 1, 5, 8, 19, 35, 38, 41, 53

SafetySource(s): 10, 18, 53

Science-BiologySource(s): 3, 7, 12, 14, 17, 19, 20, 21, 28, 29. 41, 42, 48, 55, 61

Science-ChemistrySource(s): 1, 3, 7, 11, 12, 14, 17, 19, 23, 28, 29, 30, 33, 45, 499 55, 61, 68

Science-OtherSource(s): 3, 7, 10, 14, 21, 23, 25, 28, 29, 31, 33, 34, 39, 40, 41, 43, 48, 53, 56, 68

Selling and RetailingSource(s): 8, 9, 22, 37, 39, 46, 50, 63

PART II

Directory of Programed Text POlishers -

W Abject Matter Relevant to Occupational Education

1. Addison-Wesley Publishir,g Company, Inc.Reading, Massachusetts 01867

2. The American Bankers Association90 Park Avenue, New York, New York 10016

3. American Book Company55 Fifth Avenue, New York, New York 10003300 Pike Street, Cincinnati, Ohio 45202

4. American Journal of Nursing, P. I. Reprints10 Columbus Circle, New York, New York 10019

5. American Management Association, Inc.135 West 50 Street, New York, New York 10020

6. American Society of Tool and Manufacturing Engineers20501 Ford Road, Dearborn, Michigan 48128

27

7. Appleton-Century-Crofts440 Park Avenue South, New York, New York 10016

8. Argyle Publishing Corp.200 Madison Avenue, New York, New York 10016

9. Basic Systems Inc.880 Third Avenue, New York, New York 10022

10. Behavioral Research LaboratoriesLadera Professional Center, Box 577, Palo Alto, California

11. W. A. BenjaminOne Park Avenue, New York, New York 10016

12. Burgess Publishing Company426 South Sixth Street, Minneapolis, Minnesota 55415

13. Burroughs CorporationDetroit 32, Michigan

14. Central Scientific Company1700 Irving Park Road, Chicago, Illinois 60613

15. Cleveland Institute of Electronics1776 East 17th Street, Cleveland, Ohio 44114

16. Columbia University, Teachers College, Bureau of Publishers525 West 120th Street, New York, New York

17. Doubleday & Company, Inc.501 Franklin Avenue, Garden City, New York 10017

18. Du Pont Industrial Training ServiceRoom 7450, Nemours Building, Wilmington, Delaware 19898

19. Educational Methods, Inc.20 East Huron Street, Chicago Illinois 60611

20. Educational Systems Development31270 Stephenson Highway, P. 0. Box 457, Royal Oak, Michigan 48068

21. Encyclopaedia Brittanica Press425 North Michigan Avenue, Chicago, Illinois 60611

22. Entelek Incorporated42 Pleasant Street, Newburyport, Massachusetts 01950

23. Fearon Publishers, Inc.2163 Park Boulevard, Palo Alto, California 94306

24. Follett Publishing Company1010 West Washington Boulevard, Chicago, Illinois 60607

28

'-

25. W. H. Freeman and Company660 Market Street, San Francisco, California 94104

26. Friden Inc. Service Training Center421 University Avenue, Rochester, New York 14607

27. General Programmed Teaching. Division of Commerce Clearing House, Inc.424 University Avenue, P. 0. Box 402, Palo Alto, California

28. Graflex Inc.3750 Monroe Avenue, Rochester, New York 14603

29. Grolier Educational Corporation575 Lexington Avenue, New York, New York 10022

30. Harper & Row, Publishers Incorporated49 East 33rd Street, New York, New York 10016

31. Hayden Book Co., Inc.116 West Fourteenth Street, New York, New York 10011

32. Hobart Welding SchoolTroy, Ohio

33. Holt, Rinehart & Winston, Inc.383 Madison Avenue, New York, New York 10017

34. Honor Products Company22 Moulton Street, Cambridge, Massachusetts 02138

35. Human Development Institute1299 West Peachtree Street, N.E., Atlanta, Georgia 30309

36. Instructive Communications UnitTraining Branch, Communicable Disease Branch, Public Health Service, D.H.E.W.,Atlanta, Georgia 30333

37. International Business Machines Corporation6 Roosevelt Avenue, Endicott, New York 13760

38. International Educational Services, Inc., Division of International Textbook CompanyDepartment 852A, Scranton, Pennsylvania 18515

39. Learning Incorporated131 East Sixth Avenue, Scottsdale, Arizona 85215

40. J. B. Lippincott CompanyEast Washington Square, Philadelphia, Pennsylvania 19105

41. McGraw-Hill Book Company330 West 42nd Street, New York, New York 10036

42. McMahon Electronic Engineering, Research and Development Laboratory381 West Seventh Street, San Pedro, California 90731

29

43. The Macmillan Company866 Third Avenue, New York, New York 10022

44. Model Publishing and School Supply Company1604 Hodiamont Avenue, St. Louis 12, Missouri

45. C. V. Mosby Company3207 Washington Boulevard, St. Louis, Missouri 63103

46. National Cash RegisterMarketing Services, Dayton, Ohio 45409

47. National League for Nursing10 Columbus Circle, New York, New York 10019

48. Pergamon Press Inc.44-01 - 21st Street, Long Island City, New York 11101

49. Prentice-Hall Inc.Englewood Cliffs, New Jersey 07632

50. Quaker Self-Study Program345 Merchandise Mart, Chicago, Illinois 60654

51. RCA Institutes Inc.350 West Fourth Street, New York, New York 10014

52. Research & Review Service of America, Inc.123 West North Street, Indianapolis, Indiana 46209

53. Resources Development CorporationBox 591, East Lansing, Michigan 48823

54. University of Rochester, Rochester Clearinghouse for Information on Self-Instruction inMedical Education

University of Rochester, Rochester, New York

55. W. B. Saunders CompanyWest Washington Square, Philadelphia, Pennsylvania 19105

56. Science Research Associates, Inc.259 East Erie Street, Chicago, Illinois 60611

57. Scott, Foresman and Company1900 East Lake Avenue, Glenview, Illinois 60025

58. Scott Visual Aids Service110 Brantford Lane, Greenville, South Carolina 29605

59. South-Western Publishing Company5101 Madison Road, Cincinnati, Ohio 45227

60. Springer Publishing Company, Inc.200 Park Avenue, South, New York, New York 10014

30

61. Stipes Publishing Company10 Chester Street, Champaign, Illinois

62. Superintendent of DocumentsU. S. Government Printing Office, Washington, D. C. 20402

63. Teaching Systems Corporation334 Boylston Street, Boston, Massachusetts 02116

64. TECO Instruction Inc.3236 N. E. 12th Avenue, Ft. Lauderdale, Florida

65. Tektronix, Inc.P. 0. Box 500, Beaverton, Oregon 97005

66. TutorincP. 0. Box 432, La Porte, Colorado 80535

67. Vocational Agriculture Service434 Mumford Hall, Urbana, Illinois 61801

68. John Wiley & Sons, Incorporated605 Third Avenue, New York, New York 10016

69. Xerox Education Division600 Madison Avenue, New York, New York 10022

APPENDIX B

SELECTED BIBLIOGRAPHY OF RECOMMENDED REFERENCES AND

LIST OF SOURCES FOR FURTHER INFORMATION

Both the bibliography and the list of sources are highly selective. The lists contain val-uable reference materials and resource information. Several types of references are includedand many of these contain extensive listings of additional programed instruction materials.

Selected Bibliography

A. Books

1. Forbes, Jack E. A New Look at Programed Texts and Their Use. EncyclopaediaBritannica Press, 1965. 19 pp.

2. Glaser, Robert (ed.). Teachin Machines and Pro_ramed Learnin II: Data andDirections. Department of Audiovisual Instruction, National Education Association,Washington, D. C., 1965. 831 pp.

3. Hendershot, Carl H. PrmiammecIIearning!A Bibliography rams and Pre-sentation Devices; 4th ed. Hendershot, 4114 Ridgewood Drive, Bay City, Michigan48707, 1967.

4. Jacobs, Paul I., Milton H. Maier, and Lawrence M. Stolvrow. Guide to EvaluatingSelf-Instructional Programs. Holt, Rinehart and Winston, Inc., New York, 1965.84 pp.

5. Lysaught, Jerome P., and Clarence Williams. Guide to Programed Instruction. Wiley,New York, 1963. 180 pp.

6. Mager, Robert F. Preparing Objectives for Programed Instruction. Fearon, SanFrancisco, California, 1962. 62 pp.

7. Programed Instruction in Large School Systems. Educational Research Service Cir-cular, No. 7, National Education Association, Washington, D. C., 1966. 60 pp.

8. Rummler, Geary A. Programs: Development and Selection. Occasional Paper No.III, Center for Programmed Learning for Business, University of Michigan, AnnArbor, 1965. 24 pp.

9. Ryan, William F. A Handbook of Programed Learning Information. The State Educa-tion Department, Albany, New York, 1964. 59 pp.

10. Salerno, Albert. An Evaluative Re ort on the Pro ramed Learnin Center. MahoningValley Vocational School, Experimental Project, Mimeo Report, Vienna, Ohio,1965-66, 63 pp.

32

11. Schramm, Wilbur. The Research on Programed Instruction - An Annotated Bibli-ography. Office of Education, U. S. Department of Health, Education, and Welfare,Bulletin 0E-34034, U. S. Government Printing Office, Washington, D. C., 1964.114 pp.

12. Selection and Use of Pros ramed Materials: A Handbook for Teachers. Division ofAudiovisual Instructional Service, National Education Association, Washington, D.C.,1964. 71 pp.

B. Periodicals and Others

1. A. V. Communication Review. Department of Audiovisual Instruction, National Educa-tion Association, 1201 Sixteenth Street, N.W., Washington, D. C. 20036.

A quarterly which includes articles and reviews of publications of interest to thoseusing programed instruction. Subscription $6.00 per year.

2. Audiovisual Instruction. Department of Audiovisual Instruction, National EducationAssociation, 1201 Sixteenth Street, N.W., Washington, D. C. 20036.

Published ten times a year; subscription $6.00 per year. Articles provide infor-mation regarding educational media materials, techniques, and instructionaldevelopments.

3. Bulletin on Programmed Learning. Center for Programmed Learning for Business,The University of Michigan, Ann Arbor.

These bulletins are published regularly and are available upon request.

4. Bulletin - The Clearinghouse on Self-Instructional Materials for Health Care Facili-ties. The University of Rochester, School of Medicine and Dentistry.

These bulletins are published regularly and are available to those in the healthfield upon request.

5. NSPI Journal. National Society for Programmed Instruction, Trinity University, 715Stadium Drive, San Antonio, Texas 78212.

This journal is published monthly; annual subscription $6.00.

6. Programed Instruction. The Center for Programed Instruction, Institute of Educa-tional Technology, Columbia University. (No longer published.)

Reported research findings and other news about programed instruction.

7. Training in Business and Industry. Gellert Publishing Corporation, 33 West 60thStreet, New York, New York 10023.

Contains up-to-date reports on what techniques business and industry are using,as well as advertisements by many producers'of programed instruction and othereducational media. Magazine is published monthly, is free to qualified personneland available by subscription at $5.00 per year to others.

33

Selected List - Sources of Information

1. Center for Programmed Learning for BusinessThe University of Michigan, 340 South State StreetAnn Arbor, Michigan 48104

Department of Audiovisual Instruction, National Education Association1201 Sixteenth Street, N.W., Washington, D. C. 20036

3. Educational Media Branch, Office of EducationU. S. Department of Health, Education and WelfareWashington, D. C. 20202

4. Institute of Educational TechnologyTeachers CollegeColumbia University, New York, New York 10027

5. National Society for Programmed InstructionTrinity University715 Stadium Drive, San Antonio, Texas 78212

6. The Clearinghouse on Self-Instructional Materials for Health Care FacilitiesUniversity of Rochester260 Crittenden Boulevard, Rochester, New York 14620

34

APPENDIX C

CRITERIA FOR ASSESSING

PROGRAMED INSTRUCTIONAL MATERIALS

1962 Interim Report of the Joint Committeeon Programed Instruction and Teaching Ma-chines

American Educational Research Association

American Psychological Association

Department of Audiovisual Instruction, NEA

Prepared with the cooperation and supportof the Educational Media Branch, U. S.Office of Education, under NDEA Title VII

THE members of the AERA-APA-DAVI JointCommittee include Harry F. Silberman,Evan R. Keislar, Robert Glaser, and ArthurA. Lumsdaine, Chairman (American Educa-tional Research Association); Richard S.Crutchfield, James G. Holland, and LawrenceM. Stolurow (American Psychological Asso-ciation); and Jack V. Ed ling, Edward B. Fry,Wesley C. Meierhenry, and Paul R. Wendt(Department of Audiovisual Instruction, Na-tional Education Association). Helpful con-tributions were made to the preparation andreview of the present statement by a cooper-ating committee of the American Society ofTraining Directors, whose members areLeonard C. Silvern (chairman), Robert L.Craig, Stanley L. Levine, Leonard Nadler,and Gerald H. Whitlock. Also contributingwere several consultants and staff assistants,including Lloyd 0. Brooks, Martin V. Cov-ington, H. J. A. Goodman, Bert Y. Kersh,Susan M. Markle, Ernst Z. Rothkopf, andDavid G. Ryans. Further suggestions fromprogram writers, publishers, or users areinvited for the committee's use in the prep-aration of subsequent reports. This presentarticle, as an interim report, can be in-terpreted as expressing a consensus ofJoint Committee members rather than anofficial policy statement of AERA, APA, andDAVI.

35

This statement by the AERA-APA-DAVIJoint Committee on Programed Instructionand Teaching Machines is concerned witheducational techniques that are variouslycalled "programed instruction," "auto-in-structional" methods, and "programmedlearning."

The present statement amplifies and ex-tends the previous guidelines published in1961 by the Joint Committee.' This report,like the previous one, is addressed pri-marily to the non-technical reader inter-ested in the possible purchase of programs.It summarizes some basic aspects of thenature and current status of programed in-struction, and also presents some suggestionsand cautions concerning the assessment ofprograms.

A subsequent, more technical report willprovide supplementary information and rec-ommendations addressed to the technicalspecialist who is directly concerned withobtaining or interpreting quantitative data toindicate the effectiveness of programs Lcontributing to specified instructional goals.

Programed instruction. As used herein,programed instruction refers to the use ofmaterials or procedures which incorporatean "auto-instructional" (or self-instruc-tional) program. Such a program commonlyattempts to provide conditions under whicha student can learn something efficientlywith little or no outside help. Current pro-grams typically employ a pre-arranged se-quence of material that is presented to thestudent one small unit at a time (e.g., asentence or paragraph). Most programsrequire the student to respond actively at

1. This earlier statement by the JointCommittee was published in 1961 in theJuly-August issue of the AV CommunicationReview, the September issue of AudiovisualInstruction, and the November issue of theNEA Journal, as well as in a number ofother educational periodicals.

least once for each unit (or "frame") ofmaterial - for example, by composing orselecting an answer to a question. Pro-grams also commonly provide prompt con-firmation or correction, as the case may be,for each response the student makes. Insome cases, the program is presented by amechanism or device called a "teachingmachine"; in other cases it is prsented bya specially designed form of book.4

Some Basic Considerations Concerning Pro-gramed Instruction

With or without the use of "teachingmachines" for controlled presentation ofprograms, individual instruction by pro-gramed materials offers a very importantpotential resource for education. Attentionto the following guidelines is suggested,however, in order that the potentialities ofprogramed instruction may be effectivelydeveloped and utilized.

Experimentation and planning for schooluse. Programed instruction represents arelatively new and thus far largely experi-mental resource for education. Experimentaltryout in schools, of both locally and com-mercially developed programs, is stronglyencouraged. Wide-scale adoption of anyparticular program may well await the eval-uation of one or more provisional tryouts ofthat program.

Curriculum planning. An important po-tential advantage of individual programedinstruction is that abler learners can proceed

2. Detailed information about the develop-ment of teaching machines and programed-instruction concepts up to 1960 is containedin the source book, Teaching Machines andProgrammed Learning, edited by Lumsdaineand Glaser and published by DAVI (724 p.;$7.50). A briefer treatise is Teaching byMachine, by L. M. Stolurow, published in1961 by the U.S. Office of Education and ob-tainable from the U.S. Government PrintingOffice (173 p.; 65 cents). Reviews of severalmore recent books on programed instructionmay be found in professional Journals suchas Contem oip:LyytrPs ch(1)1E, published bythe APA, and the AV Communication Review,published by DAVI.

36

at an accelerated rate through basic coursematerial and thereby qualify sooner for ad-vanced instruction. On the other hand, suit-able programing may enable the slow learnerto attain higher levels of proficiency thanwould otherwise be possible. Planning foradaptation of curricula to accommodate thesepossibilities needs to be undertaken as pro-gramed materials of demonstrated qualitybecome available.

Perfecting programs through tryout andrevision. Programed instruction affords out-standing opportunities for perfecting instruc-tional sequences through successive revisionbased on detailed records of student responseto preliminary forms of a program. Thedevelopment of high-quality programs willgenerally entail considerable effort and ex-pense. However, if costs can be pro-ratedover a large number of students, a greaterresearch and development effort can be in-vested in a program than might otherwise beconsidered feasible.

Tests of program effectiveness. Althoughthe content Miich a program is designed toteach may be inferred from careful inspec-tion of the program itself, external evidencebased on student performance is needed todemonstrate how well the program actuallyteaches. However, the value of a method ofinstruction can not be tested in the abstract.For example, evaluation of a particular text-book is not an assessment of the usefulnessof textbooks in general. A properly conductedexperimental tryout or field test of a pro-gram may provide an assessment of thatparticular program, but does not affordproofor disproof of the value of a general "method"of programed instruction.

Experimentation conducted thus far sup-ports the expectation that good programs,carefully developed, can significantly im-prove both the quality and economy of in-struction. Whether any particular programwill do so is subject to question until estab-lished by adequate tests of that program.Unfortunately, programs may be offered forsale that will fall short of the potential valueof programed instruction - for example, be-cause they have not been carefully developedthrough procedures that include sufficienttryout and revision to assure their effective-ness.

"TEACHING MACHINES"

Some programs require a machine fortheir presentation, while some are availableonly in book form. Other programs areavailable in two separate versions, one inbook form and the other for use with amachine. In any case, it should be em-phasized that so-called teaching machines,in themselves, do not teach. Rather, theteaching depends on the program of instruc-tional materials that may be presented by amachine. The comparative merits of ma-chine and non-machine presentation of printedprograms for use in schools is as yet anunresolved issue. Any advantage for machineover book presentation can not be tested inthe abstract but would depend on the char-acteristics of a particular machine. Somemachines have demonstrable advantages forcertain uses, including research; and suitablemachines are required for programs thatutilize audio materials.

Machine characteristics. The variety oftypes of teaching machines continues to pro-liferate, with little standardization. In eval-uating any make or model of teaching ma-chine, a continuing necessity is thus to assessthe number and quality of programs avail-able for use in it. For some machines theuser who has sufficient time and skill canprepare his own programed materials; forother machines this may not be feasible.With some machines, a program can be re-used indefinitely; for others a new copy ofthe programed material may be required foreach student.

For many machines, mechanical depend-ability can not yet be taken for granted. Aswith any new mechanical device, potentialpurchasers of teaching machines are welladvised to seek reliable information on howextensively the device has been used inschools, what maintenance problems havebeen encountered, and the extent to whichparts and service are locally available atreasonable cost.

Availability of machines. Existing ma-chines differ greatly in complexity and cost;prices for most of them range from a fewdollars to several hundred dollars per ma-chine. Any catalog of teaching machines islikely to be obsolete as soon as it is printedbecause the field is developing so rapidly.

37

New machines appear, and some advertisedmodels fail to get into production. Severaldozen different machines are briefly de-scribed and illustrated in a 1962 publicationby Finn and Perrin.3 A number of these arecommercially available at present. Othershave been withdrawn from the market orwere experimental models that have nevergotten into production.

PROGRAMS

Availability of Programs

An increasing number of programs isbecoming commercially available in a varietyof subject-matter areas. Mere availabilityis no guarantee of quality, however. Inaddition, programs (as well as machines)are sometimes announced long before theyare actually available; also, as ncted above,some programs are in a format that can bepresented only with a particular kind ofmachine.

A useful guide to available programs forschool subjects is a 383-page governmentpublication entitled Programs, '62.4 Thispublication lists some 120 programs reportedto be commercially available by September1962. These programs span the curriculumfrom elementary to college levels and covera variety of subject matter, including lan-guage arts, mathematics, music, physicaland biological sciences, social studies, andbusiness education. The report cited in-cludes descriptive information and one or

3. Finn, J. D., and D. G. Perrin. Teach-ing Machines and Programed Learning 1962:A Survey of the Industry. Occasional PaperNo. 3, NEA Technological Development Proj-ect. Washington, D. C.: U.S. GovernmentPrinting Office, 1962 (Publication No. OE-34019; 85 p., 55 cents). See subsequent foot-note concerning information presented inthis report.

4. The Center for Programed Instruction,Inc. Programs, '62: A Guide to ProgramedInstructional Materials. Washington, D.C.:U.S. Government Printing Office, 1962 (U.S.Office of Education, Publication No. OE-34015; 383 p., $1.50). See following footnoteconcerning information presented in thisreport.

more sample sequences from each program.Pertinent data given include the intendedstudent population, the number of "frames"in each program, and its price, but noattempt is made to evaluate the programs.5It is anticipated that this compilation will beupdated by similar guides for subsequentyears.Types of Programs

Programs are being produced in a varietyof forms. Thus far they have tended tocluster around two or three main types; how-ever, new variants or mixtures of types arealso being introduced. The majority of cur-rent programs break the subject matter downinto a large number of small steps or"frames," requiring the student to make oneor more responses to each step. Such aprogram can be so designed that the studentwill respond to critical aspects of each frameor will get practice in performing the specificoperation that each frame is meant to teach.Careful programing requires the programerto take great pains to insure that these stepsembody a logical, well-sequenced progress-ion of the subject matter. This appliesespecially to programs that are intended toserve as sole or independent sources ofinstruction rather than only as supplementsto other material. Such programs often pro-vide a number of examples to illustrate eachprinciple, concept, or act that is to be learn-ed.

Programs of the kind described aboveare designed to adapt to individual differ-ences by allowing each learner to proceed athis own rate. In addition, some types ofprograms further adapt by providing for"branching" to alternate materials. Forthis purpose, frames may include questionsdesigned to diagnose the learner's needs,with directions taking him to alternate mate-rial suited to these needs.

5. The reports on devices and programsidentified in the two preceding footnotes arecited solely for the convenience of the reader,and statements concerning them are not tobe construed as an endorsement by the JointCommittee or its parent organizations, eitheras to completeness or accuracy of the infor-mation presented, or of the quality of thedevices and programs listed in these publica-tions.

38

In most of the current "branching" pro-grams, the program is so constructed thatthe choice of a particular answer to a diag-nostic question determines which frame willbe presented next. Incorrect answers maytake the student to frames containing infor-mation designed to correct the error beforeallowing him to continue through the se-quence, or to frames that provide supple-mental information or practice.

There is little empirical basis at presentto favor one general type of program overanother. It may be anticipated that differenttypes of programs will eventually prove tobe especially useful for particular kinds ofeducational purposes, and thatdifferent stylesof programing may be combined effectivelyin a single program. At the present time,however, the general pattern of one type oranother of prograrr.Ing may be superficiallyfollowed without necessarily capitalizingfullyon its potential advantages.

CRITERIA FOR ASSESSING PROGRAMS

"Internal" and "External" Sources Of Infor-mation About Programs

A useful distinction can be made between"internal" and "external" characteristicswhich might serve as possible criteria forprogram evaluation,

"Internal" characteristics refer to fea-tures that can be revealed through visualinspection of the program. These includeboth the content of the program and the waythe program is constructed. Content mightbe described in terms of relative emphasisgiven to various topics as well as ,;enerelorganization of the material. Descriptivecharacteristics of program constructionmight include information about the lengthof frames, use of branching sequences,techniques of prompting, patterns of repeti-tion and review, modes and frequency ofresponse called for, procedures and sched-uling of reinforcement, and the like.

"External" information about a programrefers to which cannot be observedmerely by inspecting the program itself, suchas the way it was developed and character-istics of its performance as a teaching in-strument. External information of interestto a potential purchaser could include such

things as the source of program content,qualifications of authors, history of the pro-gram's development, tryout and revisions,and test data indicating gains in achievementproduced by the use of the program. Thisinformation, as indicated more fully below,may be presented in a descriptive manualsupplied by the program publisher.

Critical reviews of programs may furnishan additional basis for evaluation. Such re-views are beginning to appear in professionaljournals along with reviews of textbooks.(Some reviews include data on achievementattained by using the program as well as thereviewer's opinion about program content andstyle.)

Programs as Related to Textbooks and toTests

The applicability of internal and externalkinds of information as possible criteria forevaluating programs may in part be seen bycomparing programs with textbooks and alsowith educational or psychological tests.

Pro:. rams 'ed with textbooks.oks.Both programs arkitextWAomay e in-spected to determine what topics are coveredand the relative amount of attention given toeach. Such inspection would also indicatewhether the subject matter is factually cor-rect, whether it is current, etc. However,despite their similarities, programs differfrom textbooks in several important respectsthat may affect their evaluation. A program'srequirement for frequent student responsegenerates a special source of data usefulfor revising the program in detail. Thetendency to empirically guided developmentof programs is coupled with an orientationtoward testing the specific effects producedby a program, and toward more sharplyfocused objectives defined in terms of speci-fied behavioral outcomes. In addition, theprogram is intended to generate a more pre-dictable pattern of student behavior than doesthe study of a textbook, which generally hasa less specialized purpose in aimingto serveas a reference source as well as a sequenceof instruction.

Programs compared with psychologicaland educational tests. Although programsainrTrinWtiiTigtruct students ratherthan to test them, programs and tests share

39

some important attributes. Since both gen-erate student-response data as an inherentfeature, both tend to be developed in termsof empirical procedures. The difficulty ofeach item in a program, as in a test, can beinvestigated by presenting the program toappropriate samples of students and record-ing their responses. Both the program andthe test have limited ranges of usefulnessthat can be described to the potential userin terms of empirical evidence; and in bothcases it is possible to specify an externalcriterion to indicate the extent to which someintended outcome is achieved, as evidencedby the kinds of behavior that have been de-veloped or differentiated.

Uses of Internal and External InformationFor Assessing Programs

Inspecting the subject-matter content ofprograms. At the present time, the principalrecommended use of internal data obtainedfrom inspection of the programed materialsis for determining whether program contentis appropriate to the educator's objectives.As with other educational materials, pro-gram titles often are not definitive. Pro-grams labeled with the name of a particularsubject matter can vary widely in terms ofcontent and associated instructional objec-tives. The prospective purchaser of a pro-gram should, therefore, inspect the contentof the program at least as carefully as hewould that of a textbook. Preferably heshould go through the entire program todetermine what aspects of the subject aretreated or omitted, and the extent to whichparticular sub-topics are developed.

Limitations of program ins ection. Arisk in relying on inspection for eva uating aprogram is that one's perception of its valuemay be inappropriately influenced by hisreaction to particular structural features ofthe program, For example, certain framesor items may seem too difficult or too easy.However, the difficulty and appropriatenessof items in a program, like those in a test,generally cannot be judged accurately byinspection alone. External data are needed- data from an actual tryout of the programon students who are representative of thepopulation of intended users.

The need for test data to assess a pro-gram's effectiveness. Empirical evidence onwhat is learned from the program can alsobe a better basis than mere inspection foranswering such questions as whether programsequences have too much or too little repe-tition, review, prompting, overlap of steps,etc. At present, the scientific evidence isnot considered sufficient to permit accurateprediction in these respects or to justifyrecommendation that adherence to specificrules of program construction be used as abasis for program evaluation. External evi-dence is recommended as the main basis forthe evaluation of program effectiveness - inparticular, test data obtained from using aprogram under specified conditions whichprovide dependable measures of gains pro-duced in student achievement and of the timestudents require to achieve these gains.6

Uses and assessment ofpxruoprograms. Pro-grams may have a variety of uses. Forexample, they may be intended to providethe main source from which students areexpected to learn facts, principles or skills- or they may be intended only to review orintroduce other instruction. Inmost schools,programs will probably be used in conjunc-tion with other media of instruction. How-ever, no matter what eventual use is con-templated for a program, it will generallyhelp a prospective user to know what theprogram itself actually contributes to thestudents' knowledge or proficiency - in addi-tion to what is contributed by other elementsin the instructional situation.

The kinds of effects that can be revealedthrough empirical tryout are limited by thecontent of the achievement tests or othermeasures used to assess these effects. In-spection of the program by the prospectivepurchaser, supplemented by independent pro-

6. This emphasis external criteria forassessing program effectiveness is con-sistent with the position earlier advocated inLumsdaine and Glaser's 1960 "ConcludingRemarks" (op. cit., p. 566), and in Roth-kopf's (1961 conference report, "Criteria forthe Acceptance of Self-Instructional Pro-grams," in Improving the Efficiency andQuality of Learning, A. E. Traxler, editor.Washington, D. C.: American Council onEducation, 1962, p. 30-38.

40

_ 02' ':',"!".?1:Vit4-irleitrf

times suggest additional uses for which aprogram might be suitable or kinds of pro-gram effects which are not indicated byfield-test data because they were not con-templated in the programer's original pur-pose.

Inspection of achievement-test content.Aside from the data obtained in texting aprogram's use under laboratory or fieldconditions, inspection of the program itselfas a basis for appraisal can be supplementedif the author or publisher has spelled out theprogram's purpose bydescribing and exhibit-ing in full the achievement-test items whichpurport to exemplify what the program isintended to teach. These criterion-testitems, as well as responses called for by theprogram and test, can be examined to seewhat the learner is required to be able to do,and whether this reflects the kind of compe-tence which the educator wishes to achieve.Such an analysis of test content as a basisfor determining a program's objectives maybe particularly helpful for programs whichare intended to serve as a primary sourceof instruction rather than merely as anadjunct to other instructional material.

REPORTING DESCRIPTIVE INFORMATIONABOUT PROGRAMS

Manuals for Providing External Data

"Manuals" for tests and programs. Be-cause some of the characteristics needed toappraise educational and psychological testsare revealed only through data obtained byusing them, it has become accepted practiceto furnish information about test character-istics in a manual supplied by the test pro-ducer. It appears both desirable and feasibleto provija a similar manual for programsas a vehicle for presenting relevant externalinformation about properties which are notapparent on inspection.

Questions that might be answered abouta program. Information presented in a man-ual can help program producers or dis-tributors to answer questions which theprospective purchaser may wish to ask as abasis for selection. Several areas of suchquestions concerning external informationabout a program may be identified. Thesequestions might deal with (1) the program's

purpose and intended use, (2) the source ofprogram content, (3) the way the program wasdeveloped, including tryout and revision, and(4) the conduct and results of testing todetermine empirically the effectiveness, or"performance characteristics," of the pub-lished program. The last of these kinds ofinformation will generally be considered themost important; however, it also involvesthe kind of data which may be hardest toevaluate as to adequacy without specializedtechnical advice.

Purpose and Scope of a Program Manua1

The kind of manual here suggested couldapply to all types of programed materials.However, some of the details appropriatefor some programs probably would not applyto others. For example, less test data wouldprobably be needed in the case of very shortprograms.

It is expected that the main user of sucha manual would be the school district orother large-scale purchaser interested inbuying programs in considerable quantity.To evaluate fully some of the data that couldappropriately be included would generallyrequire advice from a technical consultantwho has professional trainingand competencein testing and measurement techniques aswell as in statistics andexperimental design.However, the manual also could well supplygeneral interpretive information to help thenon-technical purchaser determine the pro-gram's relevance to his educational purposes.Such information could precede and refer to,when appropriate, the presentation of thetechnical detail needed for the specialist toappraise a program's effectiveness.

Program "labels." In addition, a digestof the information in the manual might bepresented as a brief preface or "label"attached to individual copies of the program.Such a label could, at a minimum, indicatethe purpose and intended use of the program,who was primarily responsible for its con-tent and preparation,/ and the source of pub-lications in which further data on its de-velopment and effectiveness might be found.This information should include the age- orgrade-level(s) of the learners for whom theprogram is designed, and the prerequisiteskills and abilities these learners are

41

assumed to have. The publisher could thencharacterize and briefly illustrate the kindof competence s the programhas been demon-strated to produce when used in the mannersuggested.

Further Information on Source andDevelopment of Program Content

7IMNIM

A more detailed manual which could besupplied by the program publisher to pros-pective users on request might elaboratethis minimum information in relation tofurther questions, such as the following,which concern the source and developmentof program content.

Sources of content. What textual or cur-riculum sources were used in the selectionand development of the content? How currentwere these sources? Who were the pro-gramer(s) and the collaborating curriculumspecialists or subject-matter consultants (ifany), that prepared, edited and reviewed theprogram materials? What are their aca-demic and experience qualifications withrespect to competence in the subject matterand techniques of programing? To what kindof review was the program material sub-jected during its development?

Development, tryout, and revision. Aspreviously noted, records of learners' re-sponses to preliminary versions of a pro-gram can provide a basis for its progressiverevision and improvement prior to publica-tion. Accordingly, theprospective purchasermight wish information about the extent towhich such tryout and revision has beenconducted, the kind and amount of student-response data obtained, and the way in whichthe data were used in revising the program.The manual might also indicate the criteriaused to determine when the program wasready for final release and printing prior tothe effectiveness testing on which the per-formance characteristics of the publishedprogram are based. As supplementary in-formation, the producer might also wish toindicate the assumptions made and principlesused in constructing the program.

Information About the DemonstratedEffectiveness of a Program

It is to be hoped that the manual for a

program, at least for major programs ofconsiderable scope, will furnish evidence onthe program's effectiveness based on meas-urement of student performance on pre- andpost-program criterion tests. These testsshould be exhibited either in the manual orin an available supplement, so as to exemplifywhat the producers expect the student tolearn as a result of program use.

Program producers are strongly en-couraged to support any claims for theeffectiveness of the programs in terms ofgains in student performance produced bythe final, published version of the program,as revealed by appropriate criterion tests.A clear distinction should be made betweenthis effectiveness-test data for the finalprogram and any test data obtained in earliertryouts of preliminary versions used as abasis for revision. (Changes made in theprogram after the later effectiveness-testdata are obtained could throw doubt on thevalidity of these data for a demonstration ofthe program's effectiveness.)

The manual should present whateverfurther information would seem helpful inevaluating the reported effects of the pro-gram or the adequacy of the evidence onwhich they are based. It should in all casespresent evidence to document for the tech-nical reader that the gains in achievementreported can rightly be attributed to theeffect of the program's use rather than toextraneous causes. In addition, it shoulddescribe the physical and social conditionsof the program's use and effectiveness-testing procedures in sufficient detail so thattheir essential features could be reproducedby another investigator if desired. Thisinformation would include details of super-vision and incentives used, other instructiongiven, size of student groups, and physicalarrangement of rooms during program useand testing. Any material discrepanciesbetween recommended conditions of use andthose that were employed in obtaining theeffectiveness-test data should be noted. Stu-dents/ prior experience with programs andteaching machines, if any, should be noted inview of spuriously large temporary gainsthat can sometimes result as a novelty effectwhen a new device or procedure is firstintroduced.

The manual should indicate how many of

42

the students started and completed the pro-gram, the average length of time they re-quired to finish it, the average level of per-formance on the specified pre- and post-program tests of achievement, and the rangeor variability with respect to these meas-ures. Relevant further temporal data wouldinclude the amount of time learners ofdiffer-ent ability spent on various portions of theprogram, how this time was distributed(especially for long programs), and the timelapse between the completion of the programand the criterion test.

Effectiveness tests could of course beconducted so as to include post-programmeasures other than the test that specifiesthe programer's objectives. The program'seffect on secondary objectives not originallyaimed at could thus also be revealed. How-ever, whether or not such tests are conductedby the producer or by others (e.g., by aprospective user or by an independent re-search agency), it is to the programer'sinterest to specify what he intended as theprogram's principal objectives. Finding aprogram to be ineffective or of only limitedeffectiveness for contributing to a secondaryor unintended objective might be helpful tothe user in making a decision about the useof the program for that purpose, but cannotproperly be held as a criticism of the pro-gramer's effort.

It is anticipated that a school districtcontemplating the use of a program will beinterested in its effect on performancesother than those tested by the program pro-ducer. Particularly in this case, it isrecommended that, when possible, potentialusers assess a program by their own fieldtests, guided by suitable technical consulta-tion, before deciding on adoption of a pro-gram for wide-scale use. Performance char-acteristics of a program could specify itseffectiveness in affecting behavior of stu-dents describable as changes in knowledge,understanding, skill or other outcomes, in-cluding beliefs, interests, and motivations.

Learner characteristics. Specification ofprior knowledge and ability of learners canserve both to identify the pre-program baseline from which gains may be measured, andalso to indicate what prerequisites are neededin order to learn effectively from the pro-gram. Learner characteristics may be

specified as an aspect of the program'spurpose and intended use. The correspond-ing characteristics for the samples of stu-dents used in preliminary tryouts or, par-ticularly, in the effectiveness-testing of theprogram, should be separately specified soas to indicate the degree to which theselearners were typical or atypical of thelearners for whom the program is intended.The producer should also indicate the limits(particularly the lower limits) of the popula-tion for whom the program is intended, andof the samples used in testing its effective-ness.

Technical information concerning the con-duct of effectiveness-testing. Valid assess-ment of what is taught by the use of a pro-gram generally involves special technicalproblems. Evaluation should, whenever pos-sible, utilize the assistance of technical

specialists having recognized competence ineducational measurement and experimentaldesign. The analogy of programs with stand-ardized educational and psychological testsalso suggests a precedent for preparation oftechnical recommendations by members ofrelevant professional organizations. Theserecommendations can serve both to helpinsure the technical soundness of effective-ness-testing procedures, and to promotecomparability and interpretability of data byfostering consistently high standards of prac-tice in reporting the results of tests. Thefurther interim report to be published at alater date by this Committee will discuss inmore detail some of the technical problemsof assessing what the use of a program, inand of itself, contributes to definable in-structional goals.

RECOMMENDATIONS FOR REPORTING

THE EFFECTIVENESS OF PROGRAMED INSTRUCTION MATERIALS

(Adapted from October, 1965 revision)

I. INTRODUCTION

This report has been prepared by the Joint Committee on Programmed Instruction andTeaching Machines of the American Educational Research Association, the American Psy-chological Association, and Department of Audio-Visual Instruction (National EducationAssociation).1 The work of this committee has been supported by the Educational MediaBranch, Office of Education, U. S. Department of Health, Education, & Welfare, under TitleVII of the National Defense Education Act. The parent associations have charged the com-mittee with providing useful guidance to publishers and purchasers of programed instructionmaterials.2 The present recommendations are intended to help in improving the effectivenessof program selection and utilization. This report supplements the 1962-63 committee report,to which the reader is referred for background.3

Purpose and Content

The report provides assistance primarilyto potential users concerned with the selec-tion and effective use of instructional pro-grams. It also provides guidance to thosewho publish programs or report data onprogram effectiveness. The present reportdeals with recommendations concerning in-formation on the effects that a given pro-gram can be shown to produce, regardless

43

of how these effects may relate to the user'spurposes. Supplement I to this report con-tains suggestions for information to be in-cluded in a program manual for teachers andother users who require information aboutprogram characteristics. Supplement II con-tains recommendations intended to serve asa guide for those who are preparing tech-nical documentation in support of statementsabout the outcomes that a program canproduce.

RECOMMENDATIONS ON REPORTEDEFFECTIVENESS OF PROGRAMS

The basic premise of this report is thatinstructional effectiveness must be judgedfor each program, according to its demon-strated merits. Evidence for the effective-ness of a program should be based on adetailed study which has been fully docu-mented in a technical report. As empha-sized in its 1962-63 report, the committeetakes the position that effectiveness of eachprogram must be determined by measure-ment of the instructional outcomes whichthe program's use can be shown to bringabout. At the present state of the art, usersgenerally cannot assess the effectiveness ofa particular program reliably by mere in-spection of the program or by reference tostatements about its developmental history.

A. General Recommendations

Recommendation 1:

Evidence for the effectiveness of a pro-gram should be based on a carefully con-ducted study which shows what the program'suse accomplished under specified conditions.Such a study must employ suitable beforeand after measurements, with control pro-cedures to insure that effects attributed tothe program can be clearly distinguishedfrom the effects of other instruction.

Recommendation 2:

The results of the evaluation study shouldbe carefully documented in a technical re-port prepared in keeping with accepted stand-ards for scientific reporting. (S p e c i f i crecommendations for the preparation of suchdocuments are presented in Supplement II tothis report.)

Recommendation 3:

All claims or statements about the effec-tiveness of a program should be supportedby specific reference to the evidence con-tained in the technical report.

B. More Specific Recommendations

It is assumed that data on the effective-ness of programs will be obtained and re-ported by: (1) program producers, (2) usingagencies, including school systems, and (3)projects conducted by universities and otherresearch agencies. Accordingly, some fur-ther specific recommendations and sugges-tions, given below, are addressed to pros-pective users, program publishers, review-ers, and research agencies or institutionswhich conduct or report assessment studies.

The uses of program-assessment datadiffer depending upon the needs and technicalexperience of the user. For most teachersand school administrators, reports are need-ed which report the effectiveness of aninstructional program in fairly straight-forward terms that are quickly compre-hensible without examining detailed technicaldata. On the other hand, a detailed accountof all experimental procedures and instru-ments used in assessment is needed for thetechnical evaluator, who must criticallyanalyze the study to see if the summarizedresults and interpretations are warranted.

1. Recommendations for the ros ective ur-c aser or user:

a. Prospective users should evaluate eachprogram on its own merits according to itsdemonstrated effectiveness in producingspecified outcomes.

b. In determining the suitability of anyprogram for a particular purpose, the pros-pective user should first formulate his ownobjectives in as much detail as possible andthen evaluate the program in relation tothese objectives in the light of three things:

(1) The apparent appropriateness ofthe program content for his pur-poses, as based on inspection ofthe program itself and of the pro-ducer's statement of the program'sobjectives. These objectives maybe inferred from tests supplied bythe producer for measuring theintended outcomes of the program.

(2) Consideration of factors affectingpracticality, or feasibility of use,such as the unit cost of the pro-gram, initial and maintenance costof a machine (if required), andfactors affecting sup e r v i s ion,scheduling, and other aspects ofadministration,

(3) Evidence on the demonstrableeffectiveness of the program interms of outcomes relevant to theuser's objectives. (These may in-clude motivational, or attitudinal,effects as well as subject mattercompetences,)

c. The prospective user is advised toignore all claims for the effectiveness of aprogram which are not backed up by appro-priate data that have been subjected to com-petent evaluation. Advice on the soundnessof claims for program effectiveness shouldpreferably be obtained from a technicaladvisor or reviewer who has competence inthe fields of educational psychology, meas-urement, and experimental design and whohas reviewed available reports on the effectsof the program in the light of technicalrecommendations identified in Section 5herein.

d. In addition to consulting reviews pub-lished in professional journals, users shouldseek all available data on demonstratedperformance characteristics of the program,not only from information supplied by theproducer but also from reports preparedindependently: for example, reports pre-pared by school systems, research projects,or other agencies that have conducted pro-gram assessment studies of the particularprogram.

2. Recommendations for program publishers:

The following recommendations, plusrecommendations 4a-4e following, are offer-ed to assist the program producer in pro-viding necessary information which will helpusers make intelligent choices among avail-able programs.

a. The publisher should state in detailthe minimum objectives of his program,preferably in terms of specific behaviors orcompetences which its use is intended toachieve for specified kinds of learners ,

45

b. Publishers should refrain from pro-moting a program in terms of general state-ments about the value of programed instruc-tion as a general "method," or on the basisof statements about its effectiveness notsupported by detailed data, as recommendedherein.

c. Publishers should provide a programmanual, preferably one that can be updatedor supplemented as new data on the pro-gram become available. (See suggestionsfor the content of such a manual provided inSupplement I to this report.)

d. Preliminary limited editions of pro-grams, prior to validation by definitiveevaluative studies meeting the conditions oftechnical adequacy indicated herein shouldbe issued to facilitate collection of evalua-tion data. These should always be clearlyidentified to the purchaser as experimentalor preliminary editions.

e. Publishers should use a suitably de-scriptive title for the program, one whichappropriately delimits the scope of the sub-ject matter and skills taught. Relativelylonger titles and use of subtitles and detailedtables of contents are recommended.

3. Recommendations for reviewers:

To assist users in evaluating programs,those who prepare reviews might, in additionto expressing their opinions about the suita-bility of the program content and objectives,be guided by the following suggestions andbyrecommendations 4a-4d.

a. Take into account all available assess-ment data.

b. Evaluate end interpret such data in thecontext of techncal considerations such asthose set forth in following sections andamplified in Supplement II to this report.

c. Make available (for example, in asupplementary report or by deposit withthe American Documentation Institute) anyrelevant details of his analysis of assess-ment data which require more space than isappropriate for a published review in pro-fessional journals.

d. Utilize a procedure and format ofreporting which provides a thorough analysisrequired of the program. (When appropriate,reviewers might consider employing a set oftopical headings related to program appro-

priateness, such as producer's statementof program objectives, appropriateness ofobjectives to current curricular concepts,suitability of objectives and program to thedesignated student population, etc. Wheredata from a formal study to assess programeffects are available, the reviewer shouldevaluate the data in the light of the criterionmeasures used, adequacy of description oftest populations, description of conditions ofexperimental and intended program use,degree of correspondence between conditionsof testing and of intended application, etc.)

4. Recommendations addressed jointly toprogram producers, reviewers, and technicaladvisors:

The following recommendations recognizethat data on the effects of programs mayvary from impressions based on observa-tions of one or two subjects, as they workthrough a program, to a full-scale, formalstudy in which the program's specific effectson learning, retention, motivation, and appli-cation of knowledges and skills are de-termined for representative populations ofstudents under varying conditions of use.In the formal study, these data may beanalyzed to show differing effects for sub-groups of varying aptitude and background.

Informal tryouts and subjective impres-sions can be useful when intended to serveas guidance to the programer in revisingearly versions of a program. The teacheralso receives some value from informaltryout when a rough overall "screening test"is desired to help decide whether or not aprogram seems to "work" (in the sense ofbeing generally suitable for its intended pur-pose).4

The recommendations that follow areconcerned primarily with the reporting offormal and rigorous assessment studieswhich are required for determining in somedetail the performance characteristics of aprogram - that is, the specific outcomeswhich a program can be shown to be reliablycapable of producing.

a. Reported data on effectiveness shouldrefer to the effects produced by the programitself, unless other instructional sources areclearly identified, and their contribution isassessable. (Although most schools use

46

programs in conjunction with other media ofinstruction, it will generally help a prospec-tive user to know what the program aloneactually contributes to the student's knowl-edge or proficiency, apart from what iscontributed by other elements in the in-structional situation.)

b. Program producers should cite theavailable evidence from their own studies ofthe program to document any claims theymake about the effectiveness of the program.Publishers as well as reviewers should alsocite any pertinent evidence available fromall documented studies of the program thatare known to them.

c. Publishers and reviewers shoulddifferentiate clearly and explicitly between(1) mere opinions, of experts or others,about the probable effectiveness of the pro-gram, and (2) documented evidence on theoutcomes its use has been actually shownto produce.

d. While brief summary statements mayoften appear in advertising copy or bro-chures, or on the cover or label of a pro-gram, or in a program manual, or a review,such summary statements should alwaysreference the technical report on which theyare based, so that the correspondence be-tween interpretive statements and underly-ing data is made explicit.

e. Summaries of information concerninga program's demonstrated effectivenessshould be made widely available in publishedsources of general distribution. It is sug-gested that program producers report suchdata in a program manual and that usingagencies (such as school systems and re-search projects) also publish the resultsthey obtain, on any commercially availableprogram, in appropriate professional jour-nals. These agencies should provide copiesof their reports to the publisher of the pro-gram as soon as they are completed andready for publication.

f. To insure continued availability oftechnical reports and of data not publishedin full in standard books or journals, atleast one complete copy of the technicalreport and of all basic data tabulations shouldbe furnished to a suitable depository suchas the American Documentation Institute orUniversity Microfilms, and this fact shouldbe noted in the teacher's manual.

5. Recommendations Concernin: TechnicalReporting:

These recommendations, in addition toRecommendations 4a-4f above, are addressedto those concerned with obtaining, reporting,or evaluating the adequacy of informationfrom empirical studies of the effects ofprograms. The recommendations summar-ized here are amplified in Supplement II tothis report.5

a. In accordance with basic criteria ofscientific reporting, the entire evaluationprocedure should be reproducible. Thisapplies to the derivation, administration,and description of criterion measures aswell as to the selection of the experimentalpopulation and all aspects of experimentaldesign and procedure. The technical reportshould describe the procedures used and theresults obtained in such a way that a tech-nically-qualified person (1) can assess thevalidity of the statements concerning whatoutcomes the program's use will achieveand (2) could replicate the study in substan-tially identical fashion.

b. To satisfy this basic requirement, thetechnical report should give full details onall relevant aspects of the evaluation study,including criterion measures, 'character-istics of subjects, conditions of program useand data collection, experimental design,and data obtained. Some of the more specificaspects of such topics dealt with in Supple-ment II are:

(1) Procedures employed in measuringof retention, transfer, and attitude,and other dependent measures suchas time to reach a criterion.

(2) Characteristics of students as in-dicated by measures of prior knowl-edge, competence, intelligence, ap-titude, etc.

(3) Procedures and scheduling of pro-gram use, related instruction, con-ditions of administration.

(4) Procedures used in sampling andassignment of subjects, use of con-trol and comparison groups, con-trols for so-called Hawthorne andrelated spurious (in the sense ofextraneous - Ed. note.) effects,and account taken of "pall" effects.

(5) Processing, tabulations, analysis,and summarization of data, testsof significance and reporting fidu-cial limits, etc.

FOOTNOTES

1. Committee members are: Harry F.Silberman, Evan R. Keislar, Robert Glaser,and Arthur A. Lumsdaine, Chairman (AERA);Richard S. Crutchfield, James G. Holland,and Lawrence M. Stolurow (APA); and JackV. Edling, Edward B. Fry, Wesley C. Meier-henry, and Paul R. Wendt (DAVI). Ernst Z.Rothkopf served as consultant, and Brett B.Hamilton as staff assistant, in the prepara-tion of this report. Helpful contributionswere made to the preparation of the presentstatement by members of a cooperating com-mittee of the American Society for Trainingand Development (formerly American Societyof Training Directors) under the chairman-ship of Leonard C. Silvern. The presentreport represents a consensus of the JointCommittee members rather than officialpolicy of AERA, APA, or DAVI. Furthersuggestions from program writers, publish-ers, or users are invited.

2. A useful guide to available programsfor school subjects is the government pub-lication entitled Programs '63, prepared bythe Center for Programed Instruction (U.S.Office of Education. Publication No. OE-34015-63; 814 pp., $2.50). This publicationlists some 350 programs reported to becommercially available by the end of 1963,and includes descriptive information, price,and one or more sample sequences fromeach program, though with no attempt toevaluate the programs. Another useful com-pilation of programs, which includes moreprograms but gives less detail on each pro-gram is Programed Learning: A Bibliog-raphy of Programs and Presentation Devices,edited by Carl H. Hendershot, and publishedby Delta College, University Center, Mich-igan. This listing has been updated quarterly(L/C Cat. No. 64-11824; $3.50).

3. The 1962-63 report was entitled "Cri-teria for Assessing Programmed Instruc-tional Materials," and was published inAudiovisual Instruction, February, 1963, pp.

84-89; it has also been reprinted in severalother educational Journals and books.

4. Suggestions concerning such class-room tryouts by the teacher are offered inthe booklet Selection and Use of ProgramedMaterials: A Handbook for Teachers, pub-lished by the Division of Audiovisual In-struction Services of the National EducationAssociation, Washington, D. C. (1964: Li-brary of Congress catalogue number 64-23523); 500 per copy.

48

5. For further background on rationale,techniques, and problems in assessing theeffectiveness of instructional programs thereader may wish to consult the chapter byA. A. Luxnsdaine entitled "Assessing theEffectiveness of Instructional Programs"in the book Teaching Machines and Pro-gramed Learning II; Data and Directions(Ed. by R. Glaser, Washington, D.C.; Na-tional Education Association, 1965).

APPENDIX D

BIBLIOGRAPHY

BOOKS

A Guide for Wiley Authors in the Preparation of Auto-Instructional Programs; 3rd ed. JohnWiley & Sons, Inc., New York, 1967. 14 pp.

Currie, Dorothy A. Selected Reading ramec Learning: A Bibliography. Bureau ofSecondary Curriculum Development, The State Education Department, Albany, NewYork, 1966. 66 pp.

Fanning, Robert J. Programed Instruction Project--Annual Report, 1965-66. Board ofEducation, New York, 1966. Various paging.

Forbes, Jack E. A New Look at Programed Texts and Their Use. Encyclopaedia BritannicaPress, 1965. 19 pp.

Glaser, Robert (ed.). Teaching Machines and Programed Learning II: Data and Directions.Department of Audiovisual Instruction, National Education Association, Washington,

D.C., 1965. 831 pp.

Hendershot, Carl H. Pro rammed Learnin : A Biblio raph of Pro rams and PresentationDevices; 4th ed. Hendershot, 4114 Ridgewood I rive, Bay City, Mic igan 8707, 1967.

Hughes, J. L. (ed.). Programed Learning: A Critical Evaluation. Educational Methods,Chicago, 1964. 238 pp.

. Programed Instruction for Schools and Industry. Research Associates, Chicago,

1962. 299 pp.

Jacobs, Paul I,, Milton H. Maier, and Lawrence M. Stolvrow. Guide to Evaluating Self-Instructional Programs. Holt, Rinehart and Winston, Inc., New York, 1966. 84 pp.

Keiserman, Stanley W., and Peter T. Harkness. Education for Occu ations--Grass RootsThou' is and Beliefs in New York State--A Summa of the Mu ticounty rea to ies.Bureau of Occupational Education Research, The State Education Department, giginy;1967. 82 pp.

Lunsdaine, A. A., and Robert Glaser (ed.). Teaching Machines and Programmed Learning:A Source Book. 1960. 724 pp.

Lysaught, Jerome P., and Clarence Williams. Guide to Programed Instruction. Wiley, NewYork, 1963. 180 pp.

Magen, Robert F. Preparing Objectives for Programed Instruction. Fearon, San Francisco,California, 1962. 62 pp.

Margvlies, Stuart, and Lewis D. Eigen (eds.). Applied Programed Instruction. John Wiley &Sons, York, 1962. 387 pp.

49

Mechner, Francis, Donald A. Cook, and Stuart Margvlies. Introduction to Programed Instruc-tion. Basic Systems, Inc., New York, 1962.

Norton, Robert E. Using Programed Instruction With and Without Self-Instructional Practiceto Teach Psychomotor Skills. Ph.D. Thesis, Cornell University, Ithaca, 1967. 104 pp.

Ofiesh, Gabriel D., and Wesley C. Meierhenry (eds.). Trends in Programmed Instruction.National Education Association, Washington, D. C., 1964. 289 pp.

Programed Instruction in Large School Systems. Educational Research Service Circular,No. 7, National Education Association, Washingtori; D. C., 1966. 60 pp.

Rosen, Ned A., and Emil A. Mesics. Programmed Instruction. New York State School ofIndustrial and Labor Relations, Reprint Series No. 141, Cornell University, Ithaca,1963. 10 pp.

Rummler, Geary A. Programs: Development and Selection. Occasional Paper No. III,Center for Programmed Learning for Business, University of Michigan, Ann Arbor,1965. 24 pp.

Ryan, William F. A Handbook of Programed Learning Information. The State EducationDepartment, Albany, 1964. 59 pp.

Salerno, Albert. An Evaluative Report on the Programed Learning Center. Mahoning ValleyVocational School, Experimental Project, Mimeo Report, Vienna, Ohio, 1965-66. 63 pp.

Scanlon, James A. The Relative Effectiveness of Supplementing Programed Instruction WithBlocked Versus Spaced Review. Ph.D. Thesis, Cornell University, Ithaca, 1967.

Schramm, Wilbur, et al. Four Case Studies of Programed Instruction. The Fund for theAdvancement of Education, New York, 1964. 120 pp.

Programed Instruction - Today and Tomorrow. The Fund for the Advancementof Education, 477 Madison Avenue, New York, 1962. 74 pp.

The Research on Programed Instruction - An Annotated Bibliography. OfficeoiEducation, U. S. Department of Health, Education and Welfare, Bulletin OE-34034,

U. S., Government Printing Office, Washington, D. C., 1964. 114 pp.

Selected NSPI Talks, 1965. Center for Programmed Learning for Business Staff, Universityof Michigan, AnnAFFor, 1965. 78 pp.

Selection rd Use of Programed Materials: A Handbook for Teachers. Division of Audio-visual Instructional Service, National Education Association, Washington, D. C., 1964.71 pp.

Stolvrow, Lawrence M. Teaching by Machine. U. S. Department of Health, Education andWelfare, Office of Education, Cooperative Research Monograph No. 6, OE-34010, U. S.Government Printing Office, Washington, D. C., 1961. 173 pp.

50

PERIODICALS AND OTHER SOURCES

Bulletin on Programmed Learning. Center for Programmed Learning for Business, The

University of Michigan, Ann Arbor. Several issues.

Bulletin - The Clearinghouse on Self-Instructional Materials for Health Care Facilities.

The University of Rochester School of Medicine and Dentistry. Several issues.

Finigan, Frederick. "Programed Sales Training," Department Store nist, April 1965.

Dupont Magazines including Agricultural Newsletter, Dupont Magazine and others. E. I.

Dupont de Nemours Co., Wilmington, Delaware. Several issues.

Hartley, J. "Research Report," New Education, Vol. 2, No. 1, 1966.

Hilgard, Ernest R. "What Support from the Psychology of Learning?," NEA Journal, Nov.

1961, pp. 20-21.

Komoski, P. Kenneth. "Programed instructional Materials," Programed Instruction, Vol. 5,

Nos. 3 and 5, 1966.

. "Programed Instruction and Modern Educational Objectives," an address pre-

sented at the 30th Annual Education Conference, October, 1965, mimeo. 7 pp.

Langewieshie, Wolfgang. "Now You Can Double Your Learning Power," Readers Digest,

Vol. 91, No. 500, April 1967, pp. 112-116.

Lysaught, Jerome P. "Research on Programmer Training: What Has Been Done; What Needs

Doing," NSPI Journal, Vol. VI, No. 5, 1967, pp. 6-9.

Mathetics Su lement reprinted from "Programed Instruction" issued by the Center for

Programed struction, Inc., Vol. 3, Nos. 3 and 4, December 1963 and January 1964,

15 pp.

Morris, Barry (ed.). "The Function of Media in the Public Schools," A position paper,

National Education Association, Washington, D. C., 1962. 8 pp.

NSPI Journal (National Society for Programed Instruction). Trinity University, San Antonio,

Texas. Several issues.

Programed Instruction. The Center for Programed Instruction, Institute of Educational Tech-

nology, Columbia University. (No longer published.) Several issues.

"Programmed Instruction: New Path to New Skills," Moonbeams, The Proctor and Gamble

Company, July 1966.

Training in Business and Industry. Gellert Publishing Corp., New York. Several issues.

Training and Research. Newsletter of Resources Development Corporation, East Lansing,

Michigan. Several issues.

"What is a Teaching Machine," Promotional Brochure, Grolier Educational Corporation,

New York, 1967.