76 ieee transactions on education, vol. 51, no. 1 ... · enhancement of student learning in...

76 IEEE TRANSACTIONS ON EDUCATION, VOL. 51, NO. 1, FEBRUARY 2008

Enhancement of Student Learning in ExperimentalDesign Using a Virtual LaboratoryMilo D. Koretsky, Danielle Amatore, Connelly Barnes, and Sho Kimura

Abstract—This paper describes the instructional design, imple-mentation, and assessment of a virtual laboratory based on a nu-merical simulation of a chemical vapor deposition (CVD) process,the virtual CVD laboratory. The virtual CVD laboratory providesa capstone experience in which students synthesize engineering sci-ence and statistics principles and have the opportunity to apply ex-perimental design in the context similar to that of a practicing en-gineer in industry with a wider design space than is typically seenin the undergraduate laboratory. The simulation of the reactoris based on fundamental principles of mass transfer and chem-ical reaction, obscured by added “noise.” The software applica-tion contains a 3-D student client that simulates a cleanroom envi-ronment, an instructor Web interface with integrated assessmenttools, and a database server. As opposed to being constructed as adirect one-to-one replacement, this virtual laboratory is intendedto complement the physical laboratories in the curriculum so thatcertain specific elements of student learning can be enhanced. Im-plementation in four classes is described. Assessment demonstratesstudents are using an iterative experimental design process reflec-tive of practicing engineers and correlates success in this projectto higher order thinking skills. Student surveys indicate that stu-dents perceived the virtual CVD laboratory as the most effectivelearning medium used, even above physical laboratories.

Index Terms—3-D computer simulation, chemical vapor deposi-tion (CVD), experimental design, instructional scaffolding, think-aloud, virtual laboratory.

I. INTRODUCTION

THE traditional undergraduate laboratory class is deliveredin a mode where small student teams work with dedicated

equipment physically located in a laboratory. Students are typ-ically tasked with taking a set of experimental measurements,analyzing the data, often in the context underlying theory in thecurriculum, and reporting the findings. The pedagogical valueof the hands-on experience that a laboratory provides is ubiqui-tously endorsed by educators [1]. However, in practice the engi-neering laboratory has limitations as well. The traditional modeof delivery requires large amounts of resources for a high qualitystudent experience since students must be supervised and equip-ment is expensive to purchase and maintain. Moreover, versatile

Manuscript received August 21, 2006; revised August 1, 2007. This work wassupported by the Intel Faculty Fellowship Program and the National ScienceFoundation’s Course, Curriculum and Laboratory Improvement Program, Ed-ucational Materials Development, under Grant DUE-0442832. Any opinions,findings, and conclusions or recommendations expressed in this material arethose of the author(s) and do not necessarily reflect the views of the NationalScience Foundation.

The authors are with the Department of Chemical Engineering, Oregon StateUniversity, Corvallis, OR 97331-2702 USA (e-mail: [email protected]).

Digital Object Identifier 10.1109/TE.2007.906894

laboratory experiences are needed that can accommodate stu-dents enrolled via distance education.

A possible way to overcome the limitations of the traditionalphysical laboratory is to use alternative modes of delivery.One alternative approach is the use of Web-based remotedelivery and virtual laboratories. In the remote delivery mode,students off site can control the equipment and collect datausing a Web-based interface. Examples have been reportedin process control, power electronics, networking, fluid me-chanics, photonics, and digital signal processing [2]–[8]. In avirtual laboratory, simulations based on mathematical modelsimplemented on a computer can replace the physical labora-tory. In the most straightforward manner, the virtual laboratoryis used as an alternative mechanism for achieving the samelearning outcomes as in the corresponding physical laboratory.Virtual versions of standard laboratories have been developedand integrated into engineering [9]–[11] and science [12]–[14]curricula.

The effectiveness of Web-based remote laboratories and vir-tual laboratories as a direct replacement of their physical coun-terparts has been assessed. Several researchers have concludedthat student learning is not detrimentally affected by using Web-based remote laboratories; in fact, students appreciate the flex-ibility that they offer [2], [4]–[8]. Analogous studies indicatethat no significant difference in student learning results fromusing virtual laboratories versus physical laboratories [15], al-though feedback from students indicates that the complete re-moval of physical laboratories from the curriculum would notbe welcome [10]. One study examined a laboratory delivered tomechanical engineering juniors in all three modes [16]. The au-thors describe differences in the achievement of student learningoutcomes based on mode of delivery. While a given mode im-proves some learning outcomes, others diminish. For example,the authors report that students who experienced the nonprox-imal (Web-based or virtual) modes showed deeper levels of cog-nition in aspects of analysis of the data, both being more likely toidentify nonidealities in the experimental results and more likelyto demonstrate an understanding of the consequences of thenonidealities. However, as compared to the Web-based remotemode, students experiencing the virtual mode displayed a lessergrasp of the real context, even with both groups using iden-tical interfaces. The authors suggest that if instructors choosean alternative mode as a deliberate educational choice, ratherthan a convenient alternative, they can improve certain specificlearning outcomes in laboratories.

Indeed, a few cases have employed an alternative paradigmwhere the virtual laboratory was constructed as a complementto the physical laboratory. When viewed in this way, a virtual

0018-9359/$25.00 © 2008 IEEE

KORETSKY et al.: ENHANCEMENT OF STUDENT LEARNING IN EXPERIMENTAL DESIGN 77

laboratory can be designed to extend the range of the learnerspecifically, allowing completion of tasks not otherwise obtain-able. A virtual laboratory in circuits was used as a “primer” tocomplement the corresponding physical laboratories [17]; theintent was to orient the student for the physical laboratory tocome. The authors suggest that this approach can elicit learningoutcomes that are not possible from completion of a standalonephysical laboratory, such as allowing students to go through adebugging process for which they typically do not have time. Avirtual laboratory has been developed to allow students to applythe Taguchi method to determine the 14 design parameters of anextrusion process in just 90 minutes [18]. The simulation, whichwas based on an arbitrary mathematical model, performs the ex-periments and calculates the relevant statistics, freeing studentsto do other tasks. In particular, students compare their resultsfrom a trial and error approach to results from the more struc-tured Taguchi method. In contrasting scope, a capstone environ-mental engineering design project used a virtual laboratory toallow students to perform as a field engineer, drilling boreholes,collecting core samples, constructing wells, collecting ground-water samples, submitting samples for laboratory analysis, andexecuting hydraulic and transport experiments at a virtual haz-ardous waste site [19]. The assessment indicates that studentsgained deep content knowledge, linking theory to real-worldapplications; additionally, students reported an improvement intheir ability to handle complex projects and problem-solvingskills [20].

This paper describes the virtual chemical vapor deposition(CVD) laboratory, which has been designed to promote studentlearning in ways that are difficult to address with the physicallaboratories at the university. First, because of its structure, thevirtual CVD laboratory allows students to experience the com-plete, iterative experimental design process typical of practicingengineers. Second, the context of the project is reflective of thephysical and social context found in industry, and the solutionprocess encourages students to synthesize and toggle betweenstatistics and engineering science concepts. These aspects fa-cilitate integration and transfer of knowledge. The instructionaldesign, software design, implementation, and assessment of thevirtual CVD laboratory are presented.

II. INSTRUCTIONAL DESIGN AND PEDAGOGY

An overview of the virtual CVD laboratory with a focus onhow it applies relevant educational theory to promote studentlearning of experimental design is presented in this section.

A. The Virtual Chemical Vapor Deposition Reactor

CVD is an important unit process used to grow thin films inthe manufacture of integrated circuits and other devices [21].While laboratories in semiconductor processing have been im-plemented in the university environment [22]–[24], they tendto be resource intensive and utilize equipment many genera-tions older than found in industry. The virtual CVD laboratoryis based on a state-of-the-art industrial scaled vertical reactor,schematically shown in Fig. 1. The particular reaction used isthe deposition of silicon nitride from ammoniaand dichlorosilane ( or DCS) gases. The virtual labo-ratory is based on a simulation using fundamental principles of

Fig. 1. Schematic diagram of a CVD reactor.

mass transfer and chemical reaction, obscured by added “noise.”More detail of the mathematical model is provided elsewhere[25]. Rather than having access to the entire output of the model,the film thicknesses are provided to students only at the pointsthat they have decided to “measure.” Since the reactor outputsare reflective of the chemical and physical phenomena occur-ring within the system, those students who integrate their fun-damental engineering science knowledge should be more effi-cient with the experimental design. This approach is intendedto allow conceptual integration of statistics and engineering sci-ence and lead to deeper understanding of both topics.

Students use the virtual reactor in the same manner as theywould use a reactor in industry. They are tasked with optimizingreactor performance based on experimentation. In completingthis project, they need to choose the values for the nine param-eters for each of the runs they perform and choose the wafersand wafer locations on which film thickness measurements aremade. Since student runs and measurements have a cost associ-ated with them, a realistic economic constraint is present. Thisaspect discourages the use of unstructured trial and error ap-proaches. The most effective performance will be a balance be-tween optimization of performance and cost. Not only must stu-dents learn how to apply experimental design, they must alsodetermine when their results are good enough, so that they cansubmit their “recipe” for production.

B. Experimental Design in Engineering Education

Experimental design is an essential but difficult skill to in-tegrate into the engineering curriculum. The complete experi-mental design process used by those people who can be con-sidered experts, for example, experienced practicing engineersor research scientists, is generically depicted in the flowchartshown in Fig. 2. The expert conducts experiments to obtain in-formation about a problem that has been identified. The nextstep in the process is to design the experiment. This step includesselection of the dependent variables (responses) and the inde-pendent variables (factors) to be investigated. An appropriatesequence of experiments, often in the form of a design of exper-iments (DOE) array, is then created. The experiments are thenperformed where the independent variables are set to specifiedvalues and the dependent variables are measured. The data are


Fig. 2. Flowchart of expert approach to experimental design.

Fig. 3. Schema of the exposure of an undergraduate student to experimentaldesign in the physical laboratory. The shaded boxes represent activities that con-sume the majority of the students’ cognitive capital.

then summarized and analyzed using the appropriate statisticalmethods. The physical meaning of these data is inferred. Basedon the analysis, the practical significance of the findings is de-termined; conclusions are drawn; and decisions are made. Thedecision often consists of identifying the next set of experimentsthat need to be conducted to answer the question. In this case,the design/analysis process is repeated, as represented by thecurved arrow in Fig. 2. Being adaptable and creative as theyiterate is a trademark of successful experts. Finally, when theiterative process is complete, the findings are communicated inwritten or oral form.

A desired learning outcome in providing undergraduate labo-ratory classes is to get students to learn the experimental designmethodology as practiced by experts and shown in Fig. 2. How-ever, the experience students have in the undergraduate labo-ratory can be dramatically different. A schema representing acommon experimental process flow applied to physical engi-neering laboratory experiments is depicted in Fig. 3. Because ofthe structure of the physical laboratory, certain portions of theexperimental process are emphasized over others. For purposesof this work, cognitive capital is defined as the finite amount ofmental energy and time that a student has to invest in the com-pletion of a laboratory or project. Those steps that occupy themajority of the student’s cognitive capital are shaded in Fig. 3.To the students performing the actual experiment, getting thephysical equipment to operate properly and obtaining the dataare viewed as the paramount goals of the laboratory experience.

Students tend to view the laboratory as complete when theysuccessfully use the laboratory equipment to obtain a mean-ingful set of data. This step often consumes a majority of thestudent’s cognitive capital. The other major activity is the lab-oratory report upon which they are assessed and receive theirgrade. When their cognitive capital is spent on these activities,little remains for the critical steps of experimental design andanalysis of data. In fact, the experimental design is often deter-mined by the instructor and provided for the student. Further-more, resource and time limitations diminish the likelihood that

Fig. 4. Schema of the exposure of a student to experimental design in the virtualCVD laboratory.

students can use their results to identify and perform further ex-periments. Thus, the process depicted in Fig. 3 is linear, as op-posed to the iterative process shown in Fig. 2.

C. Problem-Based Learning Through Instructional Scaffolding

The instructional design of the virtual CVD laboratoryapplies problem-based learning (PBL) to promote studentlearning of experimental design. PBL focuses on student-cen-tered learning where instructional scaffolding is used to enablestudents to complete an authentic, open-ended problem suc-cessfully. Educational research into the effectiveness of PBLsuggests four principles of good design for PBL: settinglearning appropriate goals, providing scaffolding that promotesboth student and teacher learning, providing formative assess-ment opportunities, and using social interactions to promoteparticipation [26], [27].

Instructional scaffolding is an instructional tool that providessupport, extends the range of the learner, and permits the com-pletion of tasks not otherwise possible. The scaffolding is de-signed to be gradually removed as the students become morecompetent [28]. Scaffolding can be likened to training wheelsin that scaffolding enables students to engage in more advancedactivities that require a deeper level of cognition than would bepossible without the presence of the scaffolding [29]. The virtualCVD laboratory acts as scaffolding by simulating the physicaloperation of the process and metrology equipment. This scaf-folding allows a student to experience a different emphasis onthe experimental design process than they typically encounterin a physical laboratory. A schema of the experimental processflow utilized in this virtual laboratory is depicted in Fig. 4. Thestudent is provided with a problem statement which includesa set of independent variables to explore and a mechanism tomeasure the response variable. Student effort is then focusedon developing an experimental strategy (often in the form of aDOE design array) to explore the design space and solve theproblem. In contrast to a physical laboratory experience, thenext step in the process is performed virtually by the simula-tion. Therefore, this step consumes a relatively small amount ofthe student’s cognitive capital and is not shaded in Fig. 4. Thus,the student can invest cognitive capital on the analysis and in-terpretation of the data, and also on applying their conclusionsto iterate on the experimental design. This process continuesuntil the student decides that the problem is solved, at whichtime he or she reports the findings. In these ways, the experi-mental approach offered by the virtual CVD laboratory is reflec-tive of the approach of practicing engineers depicted in Fig. 2.


Furthermore, the cognitive processes needed to complete thesetasks complement the haptic skills emphasized in typical phys-ical laboratories. Specifically, students will need to operate atthe highest cognitive levels of Bloom’s taxonomy [30]—anal-ysis, synthesis and evaluation—to navigate the experimental de-sign process successfully.

The virtual CVD laboratory also applies the idea of situatedcognition by replicating key aspects of the social and physicalenvironment prevalent in industry. Situated cognition contendsthat situations coproduce knowledge with activity. Therefore,the method and content should be properly situated and reflectthe desired outcomes of the learner [31]. Applying situated cog-nition to the engineering classroom suggests that by utilizinglearning tools that replicate the social and physical context typ-ically found in the workplace, engineering students will be ableto more effectively transfer their cognitions and apply themto realistic engineering project work. One such aspect is boththe goal and approach to experimentation. A practicing engi-neer does not routinely conduct experiments to elucidate funda-mental principles, as is often the intended outcome in the under-graduate laboratory. More likely, the practicing engineer seeksto optimize performance of a process by investigating the effectof the input variables. Similarly, optimization of film uniformityis the direct student deliverable in the virtual CVD laboratoryproject. Students must account for cost constraints since they arecharged virtual money for each run and measurement. Further-more, the physical equipment modeled in this virtual laboratoryis reflective of equipment and throughput currently found in cut-ting-edge semiconductor factories in that two hundred 300 mmwafers are processed in a batch. In contrast, a teaching labora-tory at the university typically uses one to five 50 mm wafers.Therefore, specific cognitive tasks that a practicing engineerperforms in a high-volume manufacturing plant, such as maxi-mizing axial uniformity in the reactor, are not elicited by typicaluniversity laboratories. Even the physical look of a cleanroomis replicated in the 3-D student client, which allows a student tobecome familiar with this unique manufacturing environment.

Clearly, many benefits of integrating team work into engi-neering education exist. However, the sociocognitive interac-tions among members of a team of practicing engineers can bedistinctively different among students in a physical laboratoryat the university. The term sociocognitive interaction refers toboth how the social dynamic of the group affects the cognitiveactivities of the individual, and how the cognitive tasks affect thesocial dynamic. The effective team of engineers collaborates toresolve high level cognitive tasks, such as experimental design,data interpretation, and the iterative process of redesign shownin Fig. 2. When student teams operate a physical laboratory inthe mode depicted in Fig. 3, the haptic skills in collecting dataand the verbal skills of communicating results are where collab-oration inevitably occurs. The nature of the respective cognitivetasks drives different types of interaction between team mem-bers. Moreover, this social interaction is an important factor indetermining the direction of the project. Inspection of Fig. 4shows that the virtual CVD laboratory emphasizes the high levelcognitive tasks of design and analysis, and consequently, pro-motes the type of sociocognitive interactions found in teams ofpracticing engineers.

Fig. 5. Main software components of virtual CVD laboratory.

III. SOFTWARE DESIGN AND FEATURES

The virtual CVD laboratory software is designed to ensureease of transportability and adoptability to different educationalprograms. Instructors who want to use the software in a coursecan sign up for an instructor account via the virtual CVD lab-oratory Web page [32]. The software is designed as three in-dependent components: the 3-D student client, the instructorWeb-based interface, and the data server. The relationship be-tween these components is shown in Fig. 5. More details of thesoftware can be found elsewhere [25].

The 3-D student client is a three-dimensional graphical inter-face that provides the look-and-feel of a typical semiconductormanufacturing environment. From here, the students can makereactor runs, take measurements, and obtain output data. Fig. 6displays screen-shots of this interface. The reactor bay is shownin Fig. 6(a). The reactor input screen [Fig. 6(b)] allows the stu-dent to specify the nine operating parameters needed to grow the

thin films (five temperature zones, flow rates of ammoniaand dichlorosilane feed gases, the reactor pressure and the reac-tion time). After the student has run a batch of wafers throughthe virtual reactor, that student can navigate to the metrologybay where access is available to a virtual ellipsometer to mea-sure the film thicknesses. Fig. 6(c) displays a view of the ellip-someter console used to choose measurement sites for a specificwafer. From these data, students can estimate the overall filmuniformity. After the students have their final optimized reactorsettings, they can submit the process recipe to production viathe 3-D student client.

The second component, the instructor Web interface, allowsthe instructor to set up student accounts, view the students’ inter-actions with the virtual CVD laboratory, and customize reactorparameters, if so desired. The difficulty of real-time assessmenthas been identified as a limitation to the effectiveness of physicallaboratories [33]. To address this issue, the virtual CVD labora-tory is designed with integrated assessment tools. The instructorWeb interface contains an overview page, which shows the totalnumber of furnace runs, measurement sets, and total cost. Byclicking on a student’s account name, the instructor can viewthe student’s chosen reactor parameters for each run, and thestudent’s measurement points and resulting measured values foreach measurement set. The time at which the run or measure-ment set was made is also shown. An example of a set of stu-dent data is shown in Fig. 7(a). These data can also be exported


Fig. 6. Screen shots of the virtual CVD 3-D student client. (a) CVD reactor bay in the virtual factory. (b) Virtual CVD reactor parameter inputs. These parametersmust be input by the student to run the reactor. (c) Selection of measurement points on a wafer.

Fig. 7. Instructor Web interface. (a) View of a student’s furnace runs and measurement data. (b) Reactor parameters which may be customized by the instructor.

to a Microsoft Excel spreadsheet, as individual runs, measure-ment sets, or as a single combined spreadsheet. This feature en-ables formative assessment by allowing the instructor to providereal-time feedback to the students.

Summative assessment, or assessment of individual achieve-ment, is automatically generated by the learning tool upon stu-dent completion of the assignment. First, a summary of the per-formance for each submitted final recipe is available in tabularform. This table reports the overall uniformity, standard erroroff target, the gas utilization, and the total cost for each studentgroup with the chosen reactor parameters. Second, the standarderror off target is plotted in graphical form for different axial po-sitions along the reactor. Interpreting this graph and relating thegraph to the table summarizing students’ results, an instructorcan assess how well a given group optimized the reactor. As-sessment results from the instructor Web interface are presentedelsewhere [25].

Additionally, the instructor can customize the reactor be-havior using the instructor Web interface. Instructors who donot wish this level of specificity can simply use the defaultvalues. Default reactor settings are shown in Fig. 7(b). Fromthis interface, the instructor may change the number of wafers,

wafer radius, wafer spacing, and the virtual “cost” that the stu-dent is charged for making each furnace run and measurementset. The instructor can also change the number of temperaturezones, the position of the thermocouple within each zone,and the “bias” associated with it. This feature allows uniquesolutions from year to year. Experimental errors are added tothe students’ measured film thicknesses via a normal distribu-tion with standard deviation, . The instructor can change theparameter , so that smaller or larger experimental errors areobserved by students.

The data server performs all of the calculations, stores data,carries out actions requested by users, and sends results back tousers. The data server has several subcomponents: a mathemat-ical model which is used to determine the film thicknesses onthe wafers, a database used to store user relationships and datameasured within each user account, and an XML-RPC serverwhich runs as a daemon and handles requests from the 3-D stu-dent client and the instructor Web interface.

IV. IMPLEMENTATION

A summary of the experimental activity in the classes thathave used the virtual CVD laboratory is shown in Table I. In


TABLE ISUMMARY OF EXPERIMENTAL ACTIVITY OF THE VIRTUAL CVD LABORATORY

winter 2005 (W 05) and winter 2006 (W 06), the virtual lab-oratory was used in a senior-level undergraduate and graduateelective chemical engineering course at Oregon State University(OSU), Corvallis, Thin Film Materials Processing (ChE 444/544), where CVD is a core topic. Beginning fall 2006 (F 06),the virtual CVD laboratory was integrated into the first courseof the required chemical, biological, and environmental engi-neering capstone laboratory sequence (ChE 414). In this course,two lectures were delivered to provide students the necessarybackground knowledge of the CVD process. In addition to theclasses at OSU, the virtual CVD laboratory was implemented re-motely in the Semiconductor Processing and Device Character-ization MS Program at the University of Oregon (UO), Eugene.In summer 2006 (Su 2006), 11 graduate students completed thevirtual CVD laboratory. This case illustrates a strength of thevirtual CVD laboratory as an educational tool; the virtual labo-ratory can be creatively adapted to fit in and support any numberof different curricular structures at remote locations. The vir-tual CVD laboratory was also used in a summer outreach pro-gram called Summer Experience in Science and Engineering forYouth (SESEY).

In the capstone laboratory sequence, the virtual laboratorywas the first laboratory the students completed. The intent wasto have every ChE, BioE, and EnvE undergraduate gain criticalexperience in the entire cycle of applying experimental designmethodology prior to their experience with real equipment inthe laboratory. The student teams next engaged in three rela-tively structured physical unit operation experiments. These lab-oratories focused on developing haptic skills. Finally the lab-oratory sequence culminated in an open-ended project, wherethe focus is on working independently, developing a projectproposal, designing the experiment, completing experimentalwork, analyzing the data, and writing a final technical report thatincluded recommendations for future work. In terms of instruc-tional scaffolding, the final project reflects the case of removedscaffolding.

Before being granted access to the reactor, each group neededto develop a “preliminary design strategy” which had to be re-viewed by the group and the instructor. To determine their initialreactor parameters, the students had to apply concepts learned incore engineering classes to this open-ended problem. This taskwas a direct indication of their cognitive ability to synthesize.In addition, the students needed to review the literature to deter-mine realistic operating conditions; this task gave insight intotheir critical evaluation skills. The “preliminary design strategy”

process ensured that the groups had a well planned approach be-fore making runs, and was used so that students did not approachthis project as a video game. As an application of situated cog-nition, this design strategy process is in analogy to obtainingmanagement approval in industry.

In total, over 1,300 reactor runs have been made, and over50 000 measurements have been taken. This high level of ac-tivity is unusual in a physical laboratory, even for much simplerexperiments, and indicates that the virtual CVD laboratory isachieving its intended design of instructional scaffolding.

V. ASSESSMENT

A behavioral objective evaluation approach was implementedto measure the degree to which the intended learning outcomesof the virtual CVD laboratory had been met. Specifically, the as-sessment addressed the alignment between the students’ exper-imental approach and the experts’ approach (depicted in Fig. 2)and quantified the higher cognitive activity demonstrated by thestudents. Students’ perception of the effectiveness of the virtuallaboratory and their learning achievement was also measured.

A. Assessment of the Experimental Approach Used by Students

Traditional assessment methods, such as quizzes and tests,are effective for measuring the acquisition of facts, concepts anddiscrete skills. However, these methods have been shown to beless effective in assessing higher order cognitive activity thanperformance tasks that are designed to simulate real-world sce-narios [34]. Because of the situated nature of the virtual CVDlaboratory, completion of the project itself consists of such a setof performance tasks. Therefore, a task analysis of the students’approach to the project is an essential aspect of the virtual CVDassessment.

Six student groups participated in a “think-aloud” study tocollect assessment data. In the “think-aloud” technique, stu-dents are observed while they verbalize their thought processes.This technique has been shown to give insight to their cogni-tive processes, especially in situations where higher order crit-ical thinking ability is needed [35]. The “think-aloud” sessionsentailed observation of the student groups throughout the en-tire virtual CVD project. All sessions were captured with anaudio-recording device and later analyzed and coded to provideinsight into the experimental design approach. Additionally, sig-nificant sociocognitive interactions and their effect on makingkey decisions were recorded. A task analysis was performed onthe qualitative data to represent the unique experimental path


Fig. 8. Experimental paths derived from analysis of think-aloud sessions from two representative student groups. (a) Group A. (b) Group B.

taken by the student group. From this analysis, alignment withthe desired experimental approach can be measured.

Schemas representing the experimental path taken by two stu-dent groups are depicted in Fig. 8. The sequence of tasks com-pleted by the groups is indicated numerically in the schemas. Forexample, Group A, represented in Fig. 8(a), started the projectwith an experimental design based on a 2 level, 3 factor, fullfactorial DOE design array (task 1). After completing the firstrun in the array (task 2), they analyzed the data (task 3) andconcluded that they were too far away from the target thick-ness (task 4). Their experimental approach degenerated to one-factor-at-a-time (task 5), and they proceeded to perform a seriesof single experiments, each followed by an analysis step (tasks6–32). They continued this cycle of experimentation and anal-ysis until they concluded that they had successfully optimizedthe process (task 32), at which time they submitted their finalprocess recipe (task 33).

The experimental path of Group B, shown in Fig. 8(b), re-veals two distinct differences from Group A. First, Group Bcompleted all eight of the runs prescribed in their DOE (tasks2–9). Second, this group expanded the scope of the project toinvestigate some additional issues to evaluate if better unifor-mity could be achieved (task 49). Specifically, the group per-formed identical measurements on different ellipsometers andperformed a replication run on the reactor using the final con-ditions (tasks 50–51); these experiments allowed them to deter-mine the repeatability and reproducibility of the equipment, andlet them know the statistical uncertainty of the process and themeasurements (task 52).

Group A abandoned their planned DOE for one-factor at atime experimentation, and, therefore, needed to expend cogni-tive capital in the analysis which followed each run. In contrast,Group B completed their DOE, and, in addition, performed ameasurement system analysis and a confirmation run. Group Bwas able to complete these additional tasks since they still hadcognitive capital in reserve. This type of initiative is reflective of

high performing engineers in industry and confirms the situatednature of the cognitive activity. Moreover, the fact that GroupB completed these extra tasks is reflective of their perception ofthe virtual laboratory. If they did not consider this laboratory as“real”, they would not pursue this type of statistical verification.Hence, this virtual laboratory appears to be capable of engagingstudents in just as “real” a manner as a physical laboratory.

Inspection of the schemas in Fig. 8 shows both Groups A andB achieved several iterative cycles. Group A completed one de-sign cycle (outer loop) and eight analysis cycles (inner loop),while Group B completed two outer loops and 11 inner loops.Similarly, results from the task analysis from the other groupsparticipating in the think-aloud sessions have been compiledand are shown in Table II. In all six cases, the groups demon-strated an iterative approach to experimental design, completingan average of 3.3 design loops and 11.5 analysis loops. This ev-idence suggests that students were engaged in the intended ex-perimental approach that is depicted in Fig. 4.

This analysis of the “think-aloud” sessions validates that thevirtual CVD laboratory provides instructional scaffolding. Thevirtual laboratory minimizes the cognitive capital required toperform an experiment mechanically and, therefore, gives stu-dents a much broader experience in experimental design. Ad-ditional analysis of the think-aloud sessions was performed togive insight into sociocognitive interactions and their effect onkey decision points. Primary decision nodes were influencedjust as often by the personality dynamics of the team membersas by technical veracity of the different approaches, if not moreso. This result suggests that the social nature of group interac-tion plays a significant role in the experimental path, the overallachievement, and the degree of learning. Further investigationof the effect of sociocognitive interactions is warranted.

B. Assessment of Higher-Order Cognitive Activity

Evidence of cognitive activity requires a cognitive model thatspecifically reflects what students are intended to learn [36].


TABLE IITASK ANALYSIS SHOWING EXPERIMENTAL APPROACH

TABLE IIIASSESSMENT DATA STUDENT COURSE EVALUATIONS USING A LIKERT-SCALE

The development of an appropriate cognitive model becomes in-creasingly more difficult as increasingly higher level thinking isrequired. One of the first taxonomies of learning, developed byBloom in 1956, describes six successively deeper levels of cog-nition that culminate with the highest level of cognitions—anal-ysis, synthesis, and evaluation [30]. More recently, Shavelsondeveloped a taxonomy of learning that distinguishes betweendeclarative, procedural, and schematic knowledge [37]. The ex-perimental approach that the virtual CVD laboratory intends toelicit has inherent alignment with Bloom’s higher level cog-nitions. The steps in the expert’s experimental design process(Fig. 2) “Design the Experiment,” “Analyze & Interpret Data,”and “Conclusions/Decisions” correspond directly to the syn-thesis, analysis, and evaluation levels of Bloom’s taxonomy.

The written reports of eight student groups were analyzed forevidence of higher-order cognitive activity. For each group, acount of the number of statements present in the written reportswhich demonstrate analysis, synthesis, and evaluation was per-formed. Fig. 9 shows a plot of the number of statements versusthe group’s written report score and the instructor assessment ofthe achievement of learning outcomes, as demonstrated both inthe written and oral reports. Both scores track proportionatelywith the number of high cognition statements, with the formershowing saturation because scores could not exceed 100. Onecan infer that more activity in the higher cognitive domain re-sults in better project performance.

C. Assessment Through Course Evaluations

Assessment data were collected from Likert-scale statementsthat were part of the overall ChE 444/544 course evaluations in

Fig. 9. Evidence of higher cognitive activity and the impact of such activity onwritten report score and achievement of learning outcomes.

winter 2005 and in winter 2006. The evaluation statements weredesigned to assess how well the student felt the course learningoutcomes were satisfied and to measure the effectiveness of thedifferent learning media used in the course. One of the six courselearning outcomes was specific to the outcome resulting fromcompleting the virtual CVD laboratory. In addition, the effec-tiveness of the five learning media that were utilized in thiscourse (physical laboratories, seminar speakers from industry,traditional homework assignments, a literature review, and thevirtual CVD laboratory) were rated by the students. These eval-uation statements and results are shown in Table III.


Analyzing these assessment results leads to an interestingjuxtaposition. Students felt somewhat less proficient with thelearning outcome concerning the virtual CVD, as comparedto the other learning outcomes (average scores of 72% versus75.5%). However, they felt that the virtual CVD was the mosteffective learning medium used in the course—even more sothan physical laboratories (81% versus 75.5%). Although thestudents perceived the virtual CVD to be effective, after theycompleted the assignment, they only felt moderate mastery ofthe associated learning outcomes. These results highlight thecomplexity of the cognitive processes necessary to perform thistask, which is precisely the benefit of applying instructionalscaffolding.

VI. CONCLUSION

A virtual CVD laboratory has been developed with the in-tent of complementing, not replacing, existing physical labora-tories. The laboratory is designed to allow students to engagemore fully in certain aspects of the experimental design process;specifically, the experimental strategy, the analysis and interpre-tation of data, and the iterative process of redesign. These as-pects require higher level cognitive skills. The simulation of theCVD reactor is based on fundamental principles of mass transferand chemical reaction, obscured by added “noise.” The softwareapplication contains a 3-D student client that simulates a clean-room environment, an instructor Web interface with integratedassessment tools, and a database server complete with calcula-tion engine. The application has integrated assessment tools di-rectly available enabling formative and summative assessmentstrategies.

The virtual CVD laboratory has been implemented at OSUand remotely at the UO. Assessment has been performed to mea-sure ways in which the virtual CVD laboratory promotes stu-dent learning. Task analysis of “think-aloud” sessions verifiesthat students are engaged in the intended, iterative experimentaldesign approach of practicing engineers. Additionally, person-ality dynamics of the team members played a significant role inthe project direction. Student performance was shown to be pro-portional to the number of higher level cognition statements asclassified by Bloom’s taxonomy. As measured by end of coursesurveys, students felt somewhat less proficient with the learningoutcome concerning the virtual CVD laboratory, as comparedto the other learning outcomes. However, they felt that the vir-tual CVD was the most effective learning medium used in thecourse, even more so than physical laboratories.

ACKNOWLEDGMENT

The authors would like to thank Northwest Regional Edu-cational Laboratory’s E. Gummer, who consulted on the as-sessment design and implementation, and D. Meyers-Grahamand J. Noffsinger, computer science undergraduates at OSU, fortheir contribution to the 3-D student client.

REFERENCES

[1] P. C. Wankat and F. S. Oreovicz, Teaching Engineering. New York:McGraw-Hill, 1993.

[2] J. Henry, “Running laboratory experiments via the world wide web,”presented at the Annu. Amer. Soc. Engineering Education Conf. Ex-position, Washington, DC, 1996.

[3] D. Gillet, H. A. Latchman, C. H. Salzmann, and O. D. Crisalle,“Hands-on laboratory experiments in flexible and distance learning,”J. Eng. Educ., vol. 90, pp. 187–191, Apr. 2001.

[4] M. Ogot, G. Elliot, and N. Glumac, “An assessment of in-person andremotely operated laboratories,” J. Eng. Educ., vol. 92, pp. 57–64, Jan.2003.

[5] W. G. Hurley and C. K. Lee, “Development, implementation, and as-sessment of a web-based power electronics laboratory,” IEEE Trans.Educ., vol. 48, no. 4, pp. 567–573, Nov. 2005.

[6] S. Sivakumar, W. Robertson, M. Artimy, and N. Aslam, “A web-basedremote interactive laboratory for Internetworking education,” IEEETrans. Educ., vol. 48, no. 4, pp. 586–598, Nov. 2005.

[7] G.-W. Chang, Z.-M. Yeh, H.-M. Chang, and S.-Y. Pan, “Teaching pho-tonics laboratory using remote-control web technologies,” IEEE Trans.Educ., vol. 48, no. 4, pp. 642–651, Nov. 2005.

[8] A. Spanias and V. Atti, “Interactive online undergraduate laboratoriesusing J-DSP,” IEEE Trans. Educ., vol. 48, no. 4, pp. 735–749, Nov.2005.

[9] P. J. Mosterman, M. A. M. Dorlandt, J. O. Campbell, C. Burow, R.Bouw, A. J. Brodersen, and J. Bourne, “Virtual engineering laborato-ries: Design and experiments,” J. Eng. Educ., vol. 83, pp. 279–285, Jul.1994.

[10] T. Wiesner and W. Lan, “Comparison of student learning in physicaland simulated unit operations experiments,” J. Eng. Educ., vol. 93, pp.195–204, Jul. 2004.

[11] J. Richardson, N. Adamo-Villani, E. Carpenter, and G. Moore, “De-signing and implementing a virtual 3D microcontroller laboratory envi-ronment,” in Proc. 36th Annu. Frontiers in Education Conf., San Diego,CA, Oct. 2006, pp. 1–5.

[12] D. Rosenthal and D. Arnold, “Simulation of experimental data, thedesign of experiments, and the analysis of results: Kinetics of cat-alyzed ester hydrolysis,” J. Chem. Educ., vol. 54, pp. 323–325, May1977.

[13] R. Stolzberg, “Screening and sequential experimentation: Simulationsand flame atomic absorption spectrometry experiments,” J. Chem.Educ., vol. 74, pp. 216–220, Feb. 1997.

[14] B. Woodfield, M. Andrus, T. Andersen, J. Miller, B. Simmons, R.Stranger, G. Waddoups, M. Moore, R. Swan, R. Allen, and G. Bodily,“The virtual ChemLab project: A realistic and sophisticated simulationof organic synthesis and organic qualitative analysis,” J. Chem. Educ.,vol. 82, pp. 1728–1735, Nov. 2005.

[15] J. Campbell, J. Bourne, P. Mosterman, and A. Brodersen, “The effec-tiveness of learning simulations for electronic laboratories,” J. Eng.Educ., vol. 91, pp. 81–87, Jan. 2002.

[16] E. D. Lindsay and M. C. Good, “Effects of laboratory access modesupon learning outcomes,” IEEE Trans. Educ., vol. 48, no. 4, pp.619–631, Nov. 2005.

[17] H. Hodge, H. S. Hinton, and M. Lightner, “Virtual circuit laboratory,”J. Eng. Educ., vol. 90, pp. 507–511, Oct. 2001.

[18] J. M. Allwood, B. M. Cox, and S. S. Latif, “The structured devel-opment of simulation-based learning tools with an example for theTaguchi method,” IEEE Trans. Educ., vol. 44, no. 4, pp. 347–353,Nov. 2001.

[19] T. C. Harmon, G. A. Burks, J. J. Giron, W. Wong, G. K. W. K. Chung,and E. Baker, “An interactive database supporting virtual fieldwork inan environmental engineering design project,” J. Eng. Educ., vol. 91,pp. 167–176, Apr. 2002.

[20] G. K. W. K. Chung, T. C. Harmon, and E. Baker, “The impact of asimulation-based learning design project on student learning,” IEEETrans. Educ., vol. 44, no. 4, pp. 390–398, Nov. 2001.

[21] S. Wolf and R. N. Tauber, Silicon Processing for the VLSI Era. SunsetBeach, CA: Lattice, 2000.

[22] A. J. Muscat, E. L. Allen, E. H. Green, and L. S. Vanasupa, “Interdis-ciplinary teaching and learning in a semiconductor processing course,”J. Eng. Educ., vol. 87, pp. 414–421, Oct. 1998.

[23] J. P. Chang, “A hands-on laboratory in the fundamentals of semicon-ductor manufacturing: The capstone course of a new undergraduate op-tion,” Chem. Eng. Educ., Winter, 2002.

[24] C. Chang, M. D. Koretsky, S. Kimura, S. Rochefort, and C. Shaner,“Lab-based unit operations in microelectronics processing,” Chem.Eng. Educ., pp. 188–195, Summer, 2003.


[25] M. D. Koretsky, C. Barnes, D. Amatore, and S. Kimura, “Experientiallearning of design of experiments using a virtual CVD reactor,” pre-sented at the Annu. Amer. Soc. Engineering Education Conf. Exposi-tion, Chicago, IL, 2006.

[26] B. J. S. Barron, D. L. Schwartz, N. J. Vye, A. Moore, A. Petrosino, L.Zech, and J. D. Bransford, “Doing with understanding: Lessons fromresearch on problem and project-based learning,” J. Learn. Sci., vol. 7,pp. 271–311, Nov. 1998.

[27] D. R. Woods, Problem-Based Learning: How to Gain the Most FromPBL. Waterdown, ON, Canada: McMaster Univ., 1994.

[28] R. H. Burning, Cognitive Psychology and Instruction. Upper SaddleRiver, NJ: Merrill, 1999.

[29] J. Bransford, A. Brown, and R. Cocking, How People Learn. Wash-ington, DC: National Academies Press, 2000.

[30] B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krath-wohl, Taxonomy of Educational Objectives: The Classification of Ed-ucational Goals. New York: David McKay, 1956.

[31] J. S. Brown, A. Collins, and P. Duguid, “Situated cognition and theculture of learning,” Educ. Res., vol. 18, no. 1, pp. 32–42, Jan.–Feb.1989.

[32] [Online]. Available: http://cbee.oregonstate.edu/education/Virtu-alCVD, Nov. 21, 2006

[33] L. D. Feisel and A. J. Rosa, “The role of the laboratory in undergraduateengineering education,” J. Eng. Educ., vol. 94, pp. 121–130, Jan. 2005.

[34] G. Wiggins and J. McTighe, Understanding by Design. EnglewoodCliffs, NJ: Prentice-Hall, 2000.

[35] M. A. Ruiz-Primo, S. E. Schultz, M. Li, and R. J. Shavelson, “Compar-ison of the reliability and validity of scores from two concept-mappingtechniques,” J. Res. Sci. Teach., vol. 38, no. 2, pp. 260–278, Feb. 2001.

[36] J. W. Pelligrino, N. Chudowsky, and R. Glaser, Knowing What StudentKnow: The Science and Design of Educational Assessment. Wash-ington, DC: National Academies Press, 2001.

[37] R. Shavelson, M. A. Ruiz-Primo, M. Li, and C. C. Ayala, EvaluatingNew Approaches to Assessing Learning, Univ. California, Los An-geles, 2003, Center for the Study of Evaluation Rep.

Milo D. Koretsky received the B.S. and M.S. degrees in chemical engineeringfrom the University of California, San Diego, and the Ph.D. degree from theUniversity of California, Berkeley, in 1984, 1985, and 1991, respectively.

He joined Oregon State University, Corvallis, in 1992. His current researchactivity includes areas related to thin film materials processing and engineeringeducation. He authored Engineering and Chemical Thermodynamics, which in-cludes an integrated, menu-driven computer program, ThermoSolver.

Dr. Koretsky has won awards for his work in engineering education at theuniversity and national levels, and is a six-time Intel Faculty Fellow.

Danielle Amatore is working towards the Ph.D. degree in chemical engineeringat Oregon State University, Corvallis.

Her research interests involve the simultaneous design of learning tools forengineering education while conducting research that contributes to learningtheories.

Connelly Barnes is working towards the Ph.D. degree in computer science atPrinceton University, Princeton, NJ.

He was an undergraduate student at Oregon State University, Corvallis, whenhe developed the architecture for the virtual CVD using Unix, Python, C++, andOpenGL.

Sho Kimura received the B.S. degree from Osaka University, Osaka, Japan,in 1967, the M.S. degree from Oregon State University (OSU), Corvallis, in1976, and the Ph.D. degree from Osaka University in 1982, all in chemical en-gineering.

He is currently a Professor of Chemical Engineering at OSU. Prior to joiningOSU in 1989, his work experience included 11 years in industry and 10 years inacademia in Japan. He developed the mathematical model for the virtual CVDreactor. His research activities cover applications of fluidization technologies inhigh-temperature gas-solid reaction systems.

76 ieee transactions on education, vol. 51, no. 1 ... · enhancement of student learning in...

Documents