Mathematics and Science Indicators:

Deciding on Measures that Matter

Val PliskoNational Center for Education

Statistics

14th Annual Management Information Systems

ConferenceFebruary 27, 2001

2

Current Status of Indicators Affecting Student Learning

• Current research points to three broad aspects of school quality that affect student learning:– Training and talent of the teaching

force– Classroom activities– School culture and atmosphere

• Indicators in each of these three areas are currently of differing levels of quality

From Monitoring School Quality: An Indicators Report (Dec 2000) http://nces.ed.gov/pubs2001/2001030.pdf

3

Determining Factors in Quality of Current Indicators

• High quality indicators usually result from easily assessed dimensions or a long history of data collection on the dimension

• Moderate quality indicators generally lack information on an important facet of the indicator, but still provide some value

• Poor quality indicators generally suffer from the indicator being more complex than the data

4

Current Quality for Student Learning Indicators

High Quality Indicators• Teacher Assignment• Teacher Experience• Teacher Academic Skills• Class Size

Moderate Quality Indicators• Professional Development• Technology• Course Content• Discipline• Academic Environment

Poor Quality Indicators• Pedagogy• Goals• School Leadership• Professional Community

5

Data on Teacher Preparation

• Teacher subject matter preparation in mathematics and science is related to student achievement

• Higher student performance associated with more experienced teachers than novice teachers

• Higher student learning has been connected with teachers who attended higher quality undergraduate institutions, as measured by admissions selectivity

• Teachers with higher scores on standardized tests have been linked to students with higher test scores

6

Data on Class Size

• Most analyses of class size have found that smaller classes lead to higher student test scores, particularly for primary-grade students who are minorities or who come from economically disadvantaged backgrounds

• Yet lowering class size may not enhance student learning without changes to instructional practices or if unqualified teachers are used to reduce class size

7

Data on Goals

• Researchers agree that successful schools begin with identifying and communicating ambitious goals, then implementing and institutionalizing those goals with broad consensus from key stakeholders

• Data on school goals are generally lacking– Only nationally representative data

come from how public and private school principals responded on a questionnaire to a list of seven general goal statements

– Data lack specifics on how these goals are identified, communicated, implemented and institutionalized

8

International assessments offer a macro perspective

• Enable the United States to benchmark student performance against international averages or countries of particular interest (e.g., G-8 countries)

• Enable us to think outside of the U.S. box for alternatives to the content, methods and context for teaching

9

Average Eighth-Grade Mathematics and Science Achievement, by Nation:

1999Nation Average Nation AverageSingapore 604 Chinese Taipei 569Korea, Republic of 587 Singapore 568Chinese Taipei 585 Hungary 552Hong Kong SAR 582 Japan 550Japan 579 Korea, Republic of 549Belgium-Flemish 558 Netherlands 545Netherlands 540 Australia 540Slovak Republic 534 Czech Republic 539Hungary 532 England 538Canada 531 Finland 535Slovenia 530 Slovak Republic 535Russian Federation 526 Belgium-Flemish 535Australia 525 Slovenia 533Finland 520 Canada 533Czech Republic 520 Hong Kong SAR 530Malaysia 519 Russian Federation 529Bulgaria 511 Bulgaria 518Latvia-LSS 505 United States 515United States 502 New Zealand 510England 496 Latvia-LSS 503New Zealand 491 Italy 493Lithuania 482 Malaysia 492Italy 479 Lithuania 488Cyprus 476 Thailand 482Romania 472 Romania 472Moldova 469 (Israel) 468Thailand 467 Cyprus 460(Israel) 466 Moldova 459Tunisia 448 Macedonia, Republic of 458Macedonia, Republic of 447 Jordan 450Turkey 429 Iran, Islamic Republic of 448Jordan 428 Indonesia 435Iran, Islamic Republic of 422 Turkey 433Indonesia 403 Tunisia 430Chile 392 Chile 420Philippines 345 Philippines 345Morocco 337 Morocco 323South Africa 275 South Africa 243

488

Science

Average is significantly higher than the U.S. averageAverage does not differ significantly from the U.S. average

International average of 38 nations

Mathematics

487 International average of 38 nations

Average is significantly lower than the U.S. average

10

Comparisons of Average Mathematics Achievement, by Nation: 1995 and 1999

1995 average

1995-1999 difference

488 505 17521 531 10468 476 9569 582 13529 540 11472 482 10492 502 9550 558 8581 587 6519 525 6527 532 5418 422 4524 526 2534 534 0531 530 -1474 472 -1498 496 -1581 579 -2609 604 -4491 485 -6501 491 -10527 511 -16546 520 -26

International average of 23 nations 519 521 2

The 1999 average is significantly higher than the 1995 averageThe 1999 average does not differ significantly from the 1995 averageThe 1999 average is significantly lower than the 1995 average

(Bulgaria)Czech Republic

Hong Kong SAR

(Slovenia)(Romania)(England)

United StatesBelgium-FlemishKorea, Republic of(Australia)

ItalyNew Zealand

Russian FederationSlovak Republic

Japan

1999 averageNation

(Latvia-LSS)

Singapore

CanadaCyprus

(Netherlands)(Lithuania)

HungaryIran, Islamic Republic of

11

Comparisons of Average Science Achievement, by

Nation: 1995 and 1999

1995 average

1999 average

1995-1999 difference

476 503 27464 488 25514 533 19537 552 16510 530 20527 540 14452 460 8523 529 7533 538 5541 545 3532 535 3546 549 3513 515 2533 535 2471 472 1497 498 1511 510 -1554 550 -5541 533 -8580 568 -12463 448 -15555 539 -16545 518 -27

518 521 3The 1999 average is significantly higher than the 1995 averageThe 1999 average is not significantly different from the 1995 averageThe 1999 average is significantly lower than the 1995 average

Iran, Islamic Republic ofCzech Republic(Bulgaria)

International average of 23 nations

New ZealandJapan(Slovenia)Singapore

United StatesBelgium-Flemish(Romania)Italy

Nation(Latvia-LSS)(Lithuania)Canada

Korea, Republic of

Hungary

(England)(Netherlands)Slovak Republic

Hong Kong SAR(Australia)CyprusRussian Federation

12

Mathematics Achievement Relative to International

Average, Fourth Grade TIMSS 1995 and Eighth Grade

TIMSS-R 1999

73 Singapore 8063 Korea, Republic of 6350 Hong Kong SAR 5840 Japan 5532 Netherlands 1623 Hungary 88 Canada 74 Slovenia 60 Australia 10 Czech Republic -4-7 Latvia-LSS -19-12 United States -22-18 England -28-33 New Zealand -33-42 Italy -39-48 Cyprus -48-130 Iran, Islamic Republic of -102

517 International average of 17 nations 524

Average is significantly higher than the international average

Average does not differ significantly from the international average

Average is significantly lower than the international average

(Australia)(Italy)

1999 Eighth grade

1995Fourth grade

Singapore

Czech Republic(Slovenia)

United States(Hungary)

Korea, Republic ofJ apanHong Kong SAR(Netherlands)

Canada(Latvia-LSS)(England)Cyprus

International average of 17 nations

New ZealandIran, Islamic Republic of

13

Science Achievement Relative to International

Average, Fourth Grade TIMSS 1995 and Eighth Grade

TIMSS-R 1999

62 Singapore 4439 Hungary 2828 Japan 2528 Korea, Republic of 2418 Netherlands 2117 Australia 1614 Czech Republic 1512 England 1410 Canada 910 Slovenia 98 Hong Kong SAR 5-6 United States -9-6 New Zealand -15-9 Latvia-LSS -21-27 Italy -26-64 Cyprus -64-134 Iran, Islamic Republic of -76

514 International average of 17 nations 524

Average is significantly higher than the international average

Average does not differ significantly from the international average

Average is significantly lower than the international average

(Latvia-LSS)CyprusIran, Islamic Republic ofInternational average of 17 nations

(SloveniaHong Kong SAR(Hungary)New Zealand

1995Fourth grade

Korea, Republic of

1999Eighth grade

JapanUnited States(Australia)Czech Republic

Singapore

(Netherlands)(England)Canada(Italy)

14

• The materials presented in U.S. mathematics classrooms is at a lower grade level than that presented in German and Japanese classrooms.

• The mathematical content presented to U.S. eighth-grade students is of a lower quality than that presented to Japanese eighth-grade students.

• U.S. eighth-grade mathematics teachers’ typical goal is to teach students how to do something; Japanese teachers’ typical goal is to help students understand mathematical concepts.

• U.S. eighth-grade mathematics lessons appear to be less coherent than Japanese mathematics lessons.

• U.S. curriculum covers more topics and spends less time on each topic than in other nations.

• U.S. curriculum retains topics throughout the K-12 learning experience, whereas other nations introduce and then move to more advanced topics.

Factors related to mathematics and science achievement: TIMSS and

TIMSS-R Findings

15

Percentage of Lessons Rated as Having Low, Medium, and High Quality of Mathematical

Content

38

89

11

34

11

51

0

39

28

0

20

40

60

80

100

Germany Japan U.S.

Perc

enta

ge o

f Les

sons

Low

Medium

High

16

Eighth-Grade Mathematics Teachers’ Academic Preparation: 1999

4137

54

16

46

71

31 3235

32

0

20

40

60

80

100

Mathematics* MathematicsEducation

Education* Science/ ScienceEducation*

Other*

Bachelor's or master's degree major or main area of study

Perc

enta

ge o

f stu

dent

s

United States

Internationalaverage

*Significant difference between U.S. average and international average in this category.

17

Eighth-Grade Science Teachers’ Academic Preparation: 1999

47

13

21

43

56

14

4542

23

30

44

3025

29

0

20

40

60

80

100

Biology Physics* Chemistry ScienceEducation

Education* Math/ MathEducation*

Other*

Bachelor's or master's degree major or main area of study

Perc

enta

ge o

f stu

dent

s

United States

Internationalaverage

*Significant difference between U.S. average and international average in this category.

18

• It is not the implementation of a national curriculum.

• It is not the amount of instructional time.

• It is not the amount of homework assigned by teachers.

• It is not the presence or use of computers in the classroom.

• It is not average class size.

• It is not the amount of time spent watching TV or videos.

* This is not to say that such factors do not matter nationally.

Factors not* related to mathematics and science achievement among nations in

TIMSS

19

TIMSS-R State and District Benchmarking

Purpose: to provide data on the mathematics and science achievement of 8th-grade students in participating states and districts in comparison to students nationally and in 37 other nations Benchmark performance against

international average Compare state/district instructional

practices of teachers, student attitudes, and curriculum with that of other nations

Allow state/district to assess the rigor and effectiveness of local math and science programs in an international context

20

TIMSS-R Benchmarking Participants

States Connecticut Idaho Illinois Indiana Maryland Massachusett

s Michigan Missouri North

Carolina Oregon Pennsylvania South

Carolina Texas

Districts and Consortia Academy School District #20 (CO) Chicago Public Schools (IL) Delaware Science Coalition First in the World Consortium (IL) Fremont/Lincoln/West Side Public

Schools (NE) Guilford County Schools (NC) Jersey City Public Schools (NJ) Miami-Dade County Public Schools

(FL) Michigan Invitational Montgomery County Public

Schools (MD) Naperville School District #203

(IL) Project SMART Consortium (OH) Rochester City School District (NY) SW Pennsylvania Regional

Mathematics and Science Collaborative

21

Program for International Student Assessment (PISA)

Purpose: to measure 15-year-oldstudents’ knowledge, skills, andcompetencies in reading, mathematics and science.

• Nations can compare how their students perform on reading, mathematics, and science in relation to students in other nations

• Nations can compare themselves to other nations on contextual variables related to reading, mathematics, and science education

• Nations can gauge the impact of policy changes and reform efforts over time

22

PISA Nations

AustraliaAustriaBelgiumBrazilCanadaChinaCzech RepublicDenmarkFinlandFranceGermanyGreeceHungaryIcelandIrelandItaly

JapanKorea, Republic ofLatviaLuxembourgMexicoThe NetherlandsNew ZealandNorwayPolandPortugalRussian FederationSpainSwedenSwitzerlandUnited KingdomUnited States

23

Complementing Indicators with Research on What

Matters• Indicators should be framed by

current research on areas that affect school and teacher quality

• Indicators can also inform research in these areas, but research needed on macrolevel policies that contribute to quality teaching and learning

• International assessment can help inform the development of indicators that are most valuable to capture

24

Upcoming International Releases

• April 4, 2001: TIMSS-R Benchmarking Results

• Late 2001: Mathematics Results from the TIMSS-R Video Study

• Late 2001: PISA Results

• Late 2002: Science Results from the TIMSS-R Video Study

25

For More Information

• http://nces.ed.gov/timss

• http://pisa.oecd.org

• Val PliskoT: (202) 502-7434Email: vplisko@ed.gov