can students and faculty learn to respond to challenges facing engineering education? jeff froyd...
TRANSCRIPT
Can Students and Faculty Learn to Respond to Challenges Facing Engineering Education?
Jeff FroydDirector of Academic Development
Texas A&M University
Purdue University
6 October 2003
"There is an incredible evolution of learning or education as almost the sole source of competitive advantage in an economy that has changed so much."
Howard Block, Managing Director
Banc of America Securities
An investment-bank and brokerage subsidiary of Bank of America.
Challenges in Engineering Education
Challenges Challenge of lifelong learning Challenge of problem solving Challenge of engineering design Challenge of transfer Challenge of conceptual
understanding
LASSI SCALE ENGR111(Mean)
CVEN349(Mean)
Significance*
Skill Component
Information Processing 60.68 60.29 0.930Test Strategies 64.33 63.27 0.794Selecting Main Ideas 55.18 59.29 0.342Will Component
Anxiety 60.52 67.12 0.147Attitude 42.47 34.56 0.080Motivation 63.30 59.29 0.397Self-regulation Component
Concentration 61.31 54.56 0.144Self-testing 52.47 37.92 0.006Study Aids 60.24 45.04 0.005Time Management 55.23 47.65 0.134
“Although the data suggest a slight upward upward trend, the trend proved not to be statistically significant based upon an analysis of variance (ANOVA). Thus the cross-sectional study did not find evidence of an increase in readiness for self-directed learning, even for students in the later semesters who are taking elective courses and their capstone courses.”
Litzinger, T., Wise, J., Lee, S., and Bjorklund, S. (2003) Assessing Readiness for Self-directed Learning, Proceedings, ASEE Annual Conference
Lifelong Learning at Penn State
Self-Directed Learning Readiness Survey (SDLRS), Guglielmino & Associates, http://www.guglielmino734.com/prod01.htm, March 2003. 27
Challenge of Problem Solving
“Despite individual professors’ dedication and efforts to develop problem solving skill, “general problem solving skill” was not developed in the four years in our undergraduate program. Students graduated showing the same inability that they had when they started the program. Some could not create hypotheses; some misread problem statements. During the four-year undergraduate engineering program studied, 1974-1978, the students had worked over 3000 homework problems, they had observed about 1000 sample solutions being worked on the board by either the teacher or by peers, and they had worked many open-ended problems. In other words, they showed no improvement in problem solving skills despite the best intentions of their instructors.”Woods, D. et al (1997) “Developing Problem Solving Skills: The McMaster Problem Solving Program,” Journal of Engineering Education,
Challenge of Problem Solving Ineffective approach #1. Give the students
open-ended problems to solve; This, we now see, is ineffective because the students get little feedback about the process steps, they tend to reinforce bad habits, they do not know what processes they should be using and they resort to trying to collect sample solutions and match past memorized sample solutions to new problem situations.
Challenge of Problem Solving Ineffective approach # 2: Show them how you
solve problems by working many problems on the board and handing out many sample solutions This, we now see, is ineffective because teachers
know too much. Teachers demonstrate "exercise solving". Teachers do not make mistakes; they do not struggle to figure out what the problem really is. They work forwards; not backwards from the goal. They do not demonstrate the "problem solving" process; they demonstrate the "exercise solving" process. If they did demonstrate "problem solving" with all its mistakes and trials, the students would brand the teacher as incompetent. We know; we tried!
Challenge of Problem Solving Ineffective approach #3: Have
students solve problems on the board Different students use different
approaches to solving problems; what works for one won't work for others. When we used this method as a research tool, the students reported "we learned nothing to help us solve problems by watching Jim, Sue and Brad solve those problems!"
Challenge of Problem Solving Through four research projects we identified
why and how these and other teaching methods failed to develop process skills and which methods were successful in developing the skills
Woods, D.R., J.D. Wright, T.W. Hoffman, R.K. Swartman and I.D. Doig (1975) "Teaching Problem Solving Skills," Annals of Engineering Education, 1, 1, 238-243.
Woods, D.R. et al. (1979) "Major Challenges to Teaching Problem Solving" Annals of Engineering Education, 70, No. 3 p. 277 to 284, 1979 and "56 Challenges to Teaching Problem Solving" CHEM 13 News no. 155 (1985).
Woods, D.R. (1993a) "Problem solving - where are we now?" J. College Science Teaching, 22, 312-314.
Woods, D.R. (1993b) "Problem solving - what doesn't seem to work," J. College Science Teaching, 23, 57-58.
Woods, D.R. (1993c) "New Approaches for developing problem solving skills," J. College Science Teaching, 23, 157-158.
Challenge of Engineering Design
The literature is filled with positive comments from students, instructors, and industrial sponsors who have participated in capstone design courses. The vast majority of participants feel that the course benefited all involved.The nature of capstone design courses, however, often leads to a purely subjective evaluation with little or no “hard evidence” of actual benefits. Born, for example, does not attempt to prove the value of senior level design courses. He simply states that he is convinced from his experiences that such courses are valuable. Other educators have similar “feelings” as to the relative costs and benefits of capstone design courses.
Dutson, A.J., Todd, R.H., Magleby, S.P., Sorensen, C.D., (1997) “A Review of Literature on Teaching Engineering Design Through Project-Oriented Capstone Courses.” Journal of Engineering Education
Challenge of TransferResearches posed this problem to people
"Suppose you are a doctor faced with a patient who has a malignant tumor in his stomach. It is impossible to operate on the patient, but unless the tumor is destroyed the patient will die. There is a kind of ray that can be used to destroy the tumor. If the rays reach the tumor all at once at a sufficiently high intensity, the tumor will be destroyed. Unfortunately, at this intensity the healthy tissue that the rays pass through on the way to the tumor will also be destroyed. At lower intensities the rays are harmless to healthy tissue, but they will not affect the tumor either. What type of procedure might be used to destroy the tumor with the rays, and at the same time avoid destroying the health tissue?"
Challenge of TransferConsider the following story
"A small country was ruled from a strong fortress by a dictator. The fortress was situated in the middle of the country, surrounded by farms and villages. Many roads led to the fortress through the countryside. A rebel general vowed to capture the fortress. The general knew that an attack by his entire army would capture the fortress. He gathered his army at the head of one of the roads, ready to launch a full-scale direct attack. However, the general then learned that the dictator had planted mines on each of the roads. The mines were set so that small bodies of men could pass over them safely, since the dictator need to move his troops and workers to and from the fortress. However, any large force would detonate the mines. Not only would this blow up the road, but it would also destroy many neighboring villages. It therefore seemed impossible to capture the fortress. However, the general devised a simple plan. He divided his army into small groups and dispatched each group to the head of a different road. When all was ready he gave the signal and each group marched down a different road. Each group continued down it road to the fortress at the same time. In this way, the general captured the fortress and overthrew the dictator."
Challenge of Transfer After the subjects read and summarized this
story, they were asked to solve the tumor problem under the guise of a separate experiment.
Given the clear analogy, you might think that performance would be near ceiling. Surprisingly, only 30% of the subjects offered a convergence solution.
Moreover, when these same subjects were given the suggestion that they should use the General story, 80% provided a convergence solution.
This finding demonstrates that half the subjects could apply the General story to the tumor problem when they were instructed to but did not do so on their own.
Challenge of Conceptual Understanding Force Concept Inventory
Signals and Systems Concept Inventory
Heat Transfer Concept Inventory
Materials Concept Inventory
Hestenes Force Concepts Inventory ResultsHistogram of Number of Courses vs.
Their Normalized Gain ( )N
um
ber
of
Co
urs
es(6
2 C
ou
rses
En
rolli
ng
654
2 S
tud
ents
)
Interactive EngagementInteractive EngagementTraditionalEngagementTraditional
Engagement
Bin Centers for Normalized Gain (%)From: R. Hake, Am. J. Phys. 66, 64-74 (1998),
or see hitchcock.dlt.asu.edu/media2/cresmet/hake/
100%0%Pretest Posttest<g>
<p>
<g><p>
x 100
FIPE‘98-99
FIPE‘96-97,‘97-98 CC class,
Spring ‘99
FIPE‘99-00
FIPE‘99-00
Hestenes, Wells and Swackhamer, The Physics Teacher 30, 141 (1992); Revised edition available at http://modeling.la.asu.edu/R&E/Research.html
One of the Questions from the Force Concept Inventory
Hestenes, Wells and Swackhamer, The Physics Teacher 30, 141 (1992); Revised edition available at http://modeling.la.asu.edu/R&E/Research.html
Signals and Systems Concept Inventory (SSCI) Continuous-time SSCI Discrete-time SSCI Analysis of the 174 CT-SSCI exams revealed a
normalized gain between pre- and post-test scores of 0.22 +/- 0.07 which is consistent with the results Hake reported for other concept inventory studies of traditional lecture courses.
http://ece.gmu.edu/~kwage/research/ssci/
Heat Transfer Concept Inventory
Another student was confident of “energy balances!” but asked “Should I understand the derivation of the heat diffusion equation and where each term comes from?” The heat diffusion equation is an energy balance equation.
One topic that is typically covered in depth over a one- to two-week period in a heat transfer is the use of fins to increase heat transfer. All of the students expressed uncertainty about this subject.
They appear unable to connect to what they know about thermodynamics (i.e. heat cannot be transferred without a temperature difference) and apply it to the problem at hand (two surfaces at the same temperature).
Materials Concept Inventory Jointly developed between Arizona State
and Texas A&M Preliminary results show normalized
gain of about 0.2 in lecture classes and 0.4 in a class with some active learning
Both developers are disappointed in the pre- to post-gains shown by their classes and are actively revising their approaches to teaching
C I Questions - MCI Example
b*
3 spoonfuls of salt are stirred into a glass of water, leaving a spoonful of water-saturated salt on the bottom. If a small amount of salt is stirred in, the change in concentration of the salt in the solution is given by curve: _______
Pre-test - ASU 39%, TAMU 39%; Post-test -TAMU 66%
Critical concept - Solubility limits in phase diagrams
Student-Generated Distracters
After a piece of copper wire from a hardware store is heated it becomes softer. This is because:
a. The bonds have been weakenedb. It has fewer atomic level defects * c. It has more atomic level defectsd. The density is lowere. There’s more space inside crystal lattice
Student-generated distracters are underlinedCritical concept is role of defects in metal strength and in
metal processing.
a 39% 18%
*b 8% 29%c 6% 21%d 9% 4%e 48% 28%
Pre Post
C I Questions - MCI Example
A “Favorite” Misconception from the MCI
Nickel, and probably most other metals, cannot exist as a gas, but only as a solid or a liquid.
Nickel can exist as: ____a) solid onlyb) liquid onlyc) gas onlyd) liquid or solid onlye) liquid or solid or gas *
ASU Pre
TAMU Pre
TAMU Post
a 5% 6% 2%
b 0% 0.4% 0%
c 0% 0.4% 0%
d 46% 57% 43%
e 49% 36% 53%
Change in Engineering Education Initiating and Sustaining Change
Resistance
Leadership
Culture
Learning, Assessment, Evaluation
Initiating and Sustaining Change “It matters to me.”
Personal mastery and learning “It matters to a network of my peers.”
Knowledge management Communities of practice
“It matters to constituents and stakeholders.” Faculty and curriculum development Technology transfer and knowledge
management
Resistance Resistance is inevitable and
comprehensible Responses to resistance
Ignore Steamroll Understand and address Anticipate, understand, and address
Possible Sources of Resistance to Change
“Whatever changes we make, we must be sure to teach the fundamentals!”
Possible Sources of Resistance to Change
“Why do we need to change? Our graduates are being hired.”
Leadership Leadership takes place every day. It
cannot be the responsibility of the few, a rare event, or a once-in-a-lifetime opportunity.”
Heifetz, Ronald and Donald Laurie, “The Work of Leadership,”
Leadership is too important to be left in the hands of the few people near the top of the organizational hierarchy.
Leadership for Change Change is hard work. Leadership begins
with values Intellectual leads
physical Real changes takes
real change Leadership is a team
sport Expect to be surprised
Today competes with tomorrow
Better is better Focus on the future Learning from doing Grow people Reflect
Sullivan and Harper, Hope is not a Method
What is the organizational culture?
The culture of a group is a pattern of shared basic assumptions that the group learned as it solved problems … that has worked well enough to be considered valid, and therefore, to be taught to new members as the correct way to perceive, think, and feel in relation to those problems.
Artifacts
Visible structures and processes
Espoused Values
Strategies, goals, philosophies
Basic Underlying Assumptions
Unconscious beliefs, perceptions, thoughts, feelings
Changing Culture
“You cannot create a new culture. You can immerse yourself in studying a culture ... Until you understand it. Then you can propose new values, introduce new ways of doing things, and articulate new governing ideas. Over time, these actions will set the stage for new behavior. If people who adopt the new behavior feel that it helps them ... The organizational culture may embody a different set of assumptions, and a different way of looking at things ...”
Edgar Schein, in Senge, Peter, The Dance of Change
Changing Culture
“… culture is the result of all the daily conversations and negotiations between the members of an organization. They are continually agreeing (sometimes explicitly, usually tacitly) about the ‘proper’ way to do things and how to make meanings about the events of the world around them. If you want to change a culture you have to change all these conversations—or at least the majority of them.”
(Richard Seel, 2000)
Questions?
References Fowler, D., Maxwell, D., Froyd, J. (2003) Learning Strategy Growth Not What Expected After
Two Years through Engineering Curriculum, Proceedings, ASEE Annual ConferenceAbstract As the pace of technological development continues to increase, consensus has emerged that undergraduate science, technology, engineering and mathematics (STEM) curricula cannot contain all of the topics that engineering professionals will require, even during the first ten years of their careers. Therefore, the need for students to increase their capability for lifelong learning is receiving greater attention. It is anticipated that development of this capability occurs during the undergraduate curricula. However, preliminary data from both first-year and junior engineering majors may indicate that development of these competencies may not be as large as desired. Data was obtained using the Learning and Study Skills Inventory (LASSI), an instrument whose reliability has been demonstrated during the past fifteen years. The LASSI is a ten-scale, eighty-item assessment of students’ awareness about and use of learning and study strategies related to skill, will and self-regulation components of strategic learning. Students at Texas A&M University in both a first-year engineering course and a junior level civil engineering course took the LASSI at the beginning of the academic year. Improvements would normally be expected after two years in a challenging engineering curriculum. However, data on several different scales appears to indicate that improvements are smaller than might be expected.
ReferencesLitzinger, T., Wise, J., Lee, S., Bjorklund, S. (2003) Assessing Readiness for Self-
directed Learning, Proceedings, ASEE Annual ConferenceConclusions The goals of this study were to determine whether students’ readiness for self-directed learning increases as they proceed through an undergraduate engineering program and to determine whether capstone courses increase their readiness for self-directed learning. A related goal was to determine if there are any gender differences in readiness for self-directed learning. The cross-sectional study showed no statistically significant increase in SDLRS scores. This result suggests that most courses that students take in the undergraduate engineering programs do not ask them to undertake tasks that increase their readiness for self-directed learning. No gender differences were present in the cross-sectional study. Although some of the data show an increase in SDLRS scores between a pre-test and post-test in capstone courses in Mechanical and Electrical Engineering, the results failed to support the hypothesis that the complex problem solving required in these courses would lead to increases in readiness for self-directed learning. The lack of a statistically significant increase in SDLRS scores is of concern in an era when lifelong learning is more critical than ever for engineers. The results of the present study will be extended in future work to increase the sample size in the pre-test/post-test study and to document carefully the type of activities that the capstone courses require the students to undertake
ReferencesWoods, D. et al (1997) “Developing Problem Solving Skills: The McMaster Problem
Solving Program,” Journal of Engineering Education, Abstract This paper describes a 25-year project in which we defined problem solving, identified effective methods for developing students’ skill in problem solving, implemented a series of four required courses to develop the skill, and evaluated the effectiveness of the program. Four research projects are summarized in which we identified which teaching methods failed to develop problem solving skill and which methods were successful in developing the skills. We found that students need both comprehension of Chemical Engineering and what we call general problem solving skill to solve problems successfully. We identified 37 general problem solving skills. We use 120 hours of workshops spread over four required courses to develop the skills. Each skill is built (using content-independent activities), bridged (to apply the skill in the content-specific domain of Chemical Engineering) and extended (to use the skill in other contexts and contents and in everyday life). The tests and examinations of process skills, TEPS, that assess the degree to which the students can apply the skills are described. We illustrate how self-assessment was used.
ReferencesDutson, A.J., Todd, R.H., Magleby, S.P., Sorensen, C.D., (1997) “A Review of
Literature on Teaching Engineering Design Through Project-Oriented Capstone Courses.” Journal of Engineering Education
Abstract Senior project or Capstone-type courses have existed at engineering schools for many years. Capstone courses provide student engineers the opportunity to solve real-world engineering projects, and have been highly regarded as important learning activities. A survey of Capstone courses in engineering departments throughout North America was conducted in order to understand current practices in Capstone education. This study was conducted for presentation at the 1994 Advances in Capstone Education Conference, held at Brigham Young University, which brought together engineering educators interested in improving Capstone experiences. This conference was sponsored by ASEE, ASME, SME, and NSF. Response to the study was very high, with 360 departments from 173 schools responding to the survey. Survey results were categorized into five major areas of interest: Profile of the Respondents, Course Description Information, Faculty Involvement in Capstone Education, Project Information, and Industrial Involvement in Capstone Education. Graphs provide the results of responses to survey questions.
ReferencesMedin, D., Ross, B., Markman, A. (2001)
Cognitive Psychology, third edition, Fort Worth: Harcourt College Publishers, 470-471
Gick, M., Holyoak, K. (1980) Analogical problem solving. Cognitive Psychology, 12, 306-355
Gick, M., Holyoak, K. (1983) Schema induction and analogical transfer. Cognitive Psychology, 15, 1-38
ReferencesHake, R.R. (1988) "Interactive-engagement vs traditional methods: A six-thousand-
student survey of mechanics test data for introductory physics courses," Am. J. Phys. 66, 64- 74, http://www.physics.indiana.edu/~sdi/ajpv3i.pdfAbstract A survey of pre/post test data using the Halloun-Hestenes Mechanics Diagnostic test or more recent Force Concept Inventory is reported for 62 introductory physics courses enrolling a total number of students N = 6542. A consistent analysis over diverse student populations in high schools, colleges, and universities is obtained if a rough measure of the average effectiveness of a course in promoting conceptual understanding is taken to be the average normalized gain <g>. The latter is defined as the ratio of the actual average gain (%<post> – %<pre>) to the maximum possible average gain (100 – %<pre>). Fourteen "traditional" (T) courses (N = 2084) which made little or no use of interactive-engagement (IE) methods achieved an average gain <g>T-ave = 0.23 ± 0.04 std dev). In sharp contrast, forty-eight courses (N = 4458) which made substantial use of IE methods achieved an average gain <g>IE-ave = 0.48 ± 0.14 (std dev), almost two standard deviations of <g>IE-ave above that of the traditional courses. Results for 30 N = 3259) of the above 62 courses on the problem-solving Mechanics Baseline test of Hestenes-Wells imply that IE strategies enhance problem-solving ability. The conceptual and problem-solving test results strongly suggest that the classroom use of IE methods can increase mechanics-course effectiveness well beyond that obtained in traditional practice.
References Halloun, I., Hestenes, D. (1985)
The Initial Knowledge State of College Physics Students, Am. J. Phys. 53, 1043-1055
Halloun, I., Hestenes, D. (1985) Common Sense Concepts about Motion, Am. J. Phys. 53, 1056-1065
More information about the Force Concept Inventory is available at http://modeling.la.asu.edu/R&E/Research.html
ReferencesWage, K., Buck, J., Welch, T., Wright, C. (2002) The Signals and Systems Concept
Inventory, Proceedings, ASEE Annual ConferenceAbstract This paper describes the development of continuous-time and discrete-time signals and systems concept inventory exams for undergraduate electrical engineering curricula. Both exams have twenty-five multiple choice questions to assess students’ understanding of core concepts in these courses. The questions require little or no computation, and contain incorrect answers that capture common student misconceptions. The design of both exams is discussed, as are ongoing studies evaluating the exams at four campuses. Preliminary results from administering the continuous-time exam as a pre-test and post-test indicate a normalized gain of 0.24 ± 0.08 for traditional lecture courses, consistent with reported results for the Force Concept Inventory exam in lecture courses for freshman physics.
ReferencesJacobi A., Martin, J. Mitchell, J., Newell, T. (2003) A Concept Inventory for Heat
Transfer Proceedings, Frontiers in Education Conference, http://fie.engrng.pitt.edu/fie2003/papers/1192.pdfAbstract Students enter courses in engineering with intuitions about physical phenomena. Through coursework they build on their intuition to develop a set of beliefs about the subject. Often, their understanding of basic concepts is incomplete and their explanations are not “correct.” Concept Inventories are assessment tools designed to determine the degree to which students understand the concepts of a subject and to identify the bases for misunderstandings. A cooperative effort between faculty at the Universities of Wisconsin and Illinois has been undertaken to develop a concept inventory for heat transfer. The process initiated with student identification of the conceptual problems rather than with faculty perceptions of student misunderstandings. Students then explored areas of conceptual difficulty and phrased questions that would test understanding of the concepts. Students working together with faculty developed a concept inventory for heat transfer. The presentation will report on the experience with using student groups and the resulting concept inventory.
ReferencesKrause, S., Decker, J., Griffin, R. (2003) Using a Materials Concept Inventory to
Assess Conceptual Gain in Introductory Materials Engineering Courses, Proceedings, Frontiers in Education Conference, http://fie.engrng.pitt.edu/fie2003/papers/1160.pdfAbstract A Materials Concept Inventory (MCI) has been created to measure conceptual knowledge gain in introductory materials engineering courses. The 30-question, multiple-choice MCI test has been administered as a pre and post-test at Arizona State University (ASU) and Texas A & M University (TAMU) to classes ranging in size from 16 to 90 students. The results on the pre-test (entering class) showed both “prior misconceptions” and knowledge gaps that resulted from earlier coursework in chemistry and, to a lesser extent, in geometry. The post-test (exiting class) showed both that some “prior misconceptions” persisted and also that new “spontaneous misconceptions” had been created during the course of the class. Most classes showed a limited, 15% to 20%, gain in knowledge between pre and post-test scores, but one class, which used active learning, showed a gain of 38%. More details on these results, on differences in results between ASU and TAMU, and on the nature of students’ conceptual knowledge will be described.
References Senge, P., Kleiner, A., Roberts, C.,
Roth, G., Ross, R., Smith, B. (1999) The Dance of Change: The Challenges to Sustaining Momentum in Learning Organizations, Double Day
References Mauer, R. (1996) Beyond the Wall of
Resistance: Unconventional Strategies That Build Support for Change, Bard Press, see also http://www.beyondresistance.com/index.html
References Heifetz, R., Laurie, D. (1997). The work of
leadership. Harvard Business Review, 75(1), 124-134.
References Sullivan, G., Harper, M. (1996)
Hope is Not a Method, New York: Random House
References
Seel, R. (2000) “Culture and Complexity: New Insights on Organisational Change,” Organisations & People, 7(2) 2-9, see also http://www.new-paradigm.co.uk/culture-complex.htm
McMaster Problem-Solving Skill Sets Self-awareness, self confidence, self-
management Personal problem solving
Well-defined problems Ill-defined problems
Interpersonal and group skills Self-assessment Change management Lifelong learning skills
MPS Self-awareness, Self-confidence, Self-management Skills
Stress management Time management The unique you Managing anger Awareness Self assessment Personal enrichment
MPS Well-defined Problem Solving Skills
Awareness Strategies Analysis
(classification) Analysis (consistency) Creativity Drawing diagrams Engaging in the
problem Defining the stated
problem Getting unstuck
Broadening perspectives
Defining real problems Creating the look back Exploring the problem Tactics and heuristics Criteria Decision-making Learning skills Stress management Time management The unique you
MPS Ill-defined Problem Solving Skills Mission Vision and goals Trouble shooting Coping with ambiguity Optimum sloppiness and successive
approximation Project management
MPS Interpersonal and Group Skills
Assertiveness Asking questions Listening Interpersonal skills Group skills Chairperson skills
Conflict resolution Giving and receiving
feedback The unique you Managing anger Stress management
MPS Lifelong Learning Skills Knowledge structure Tacit or experience knowledge Learning skills Self-directed learning/PBL The unique you
MPS Change Management Skills Managing change Stress management Managing anger
MPS Self-assessment Skills Self assessment Personal enrichment
Attributes of Effective Leaders Inner drive/energy: necessary to initiate
and sustain leadership of change over extended periods of time.
Intellectual capacity: necessary to listen to input from diverse sources and synthesize vision and strategy
Integrity: necessary to synthesize vision and strategy that benefits the organization first and the individual second
Mental/emotion health: necessary for self-confidence and interpersonal skills
Kotter, John P., A Force for Change: How Leadership Differs From Management,