NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESSAshley Singer

University of Central Florida

ARE 6905

April 16, 2013

PURPOSE

Teachers demand for creating applications from results

Add more value and clarity to existing NAEP testing by addressing the lack of teacher questionnaire in the visual arts assessment that is present in nearly all other NAEP subjects.

WHY IT’S IMPORTANT Exists as a result of lack of lucidity that accompanies

NAEP visual arts assessment data Nothing offers possible explanations for results Teachers are left to interpret the numbers without

guidance Add some context to the data Be able to see what is current or trending in the

classroom What could be lacking in their curriculum What is proving successful in their practice Universities and schools could use to see what educator

training programs have been successful in equipping teachers for their fields

What teachers may be lacking in their classrooms and how to give it to them

RESEARCH QUESTIONS

How can NAEP clarify the results of the visual arts assessment by adding a teacher questionnaire with common practice and teacher background similar to existing teacher questionnaires of other subjects?

How can the demographic and background information be applied to understanding knowledge and experience as well as hiring trends?

Do the findings suggest certain training and specialties lead to classroom achievement?

What areas of art education are being concentrated on and what areas are being neglected?

How could we take the results to further develop a NAEP curriculum and understand best practices?

WHAT I’VE LEARNED

Creating a well-done questionnaire is difficult

Basic structure and style is simple Scientific approach is tedious and thought-

provoking Various steps to developing a questionnaire

Not just writing whatever questions you want answers to and expecting reliable results from them

Overwhelming to develop questions that would yield best applications for educators while answering my questions

WHAT I’VE LEARNED

Critical to review who is writing tests, papers, and surveys

What do you want to know? Likely based on what they know, their

experience, or what they want to know May not be true representation of the information

Boards and panels are important Can also be influenced by central philosophy or

philanthropist Reduce bias based on multiple experiences and

perspectives

WHAT I’VE LEARNED

Objectivity and Adjustments

Analyzing previous and test-specific data Research’s ultimate progress

Minor and major changes made to improve tests Discussion of limitations shows what could be better Changes are not personal – just progress

Adjustments to create another test Designed around teacher’s training and preparedness

Areas of focus, certification process, work history, etc

What is making teachers ready for the classroom

WHAT I’VE LEARNED

NAEP

Obvious need for more clarity If complaints are lack of application, they have to

find ways to make it relevant Educators must be a part of the process

Either in test development, research, or advocacy NAEP could find more ways to reach out to

teachers

Whether you are a researcher or a teacher, you cannot continue doing things the same way and expect different or better results

REVIEW OF LITERATURE “Finally, the arts assessment reminds us once again that arts

education is for all students, not just for the talented. No one has suggested that math or science should be taught only to students with talent in those disciplines. The arts, similarly, provide long-term benefits that are important for every student. Experience has demonstrated to arts educators that all children can learn basic arts skills and knowledge, provided that they begin instruction early enough.” (Lehman, 1999)

“Most NAEP assessments” have teacher questionnaire (NAEP, 2012) The common education practitioner often has difficulty gleaning

consequence and meaning from the scores – must ask what we know about these teachers (Eisner, 1999)

“Test performance, like paintings, needs to be ‘read,’ not only seen. Information needed to give test scores a deep reading is very limited” (Eisner, 1999

Recent study - “revealed that untrained people do not simply walk into classrooms and become successful” prepared and certified teachers are more successful than the untrained ones (Hatfield, 2007)

Test results only leave readers with “value without clarity” (Diket & Brewer, 2011)

REVIEW OF LITERATURE “While teachers’ completion of the questionnaire is

voluntary, NAEP encourages their participation since their responses make the NAEP assessment more accurate and complete” (Teacher Questionnaire, 2011)

Covers: “teaching experience, certifications, degrees, major and

minor fields of study, coursework in education, course work in specific subject areas, the amount of in-service training, the extent of control over instructional issues, and the availability of resources for the classroom” (Teacher Questionnaire, 2011)

“pre- and in-service training, the ability level of the students in the class, the length of homework assignments, use of particular resources, and how students are assigned to particular classes” (Teacher Questionnaire, 2011)

METHODOLOGY

Population

Similar to NAEP sample selection Need to be directly related to the test results

Teacher questionnaires must match up with NAEP participants’ classrooms, schools, districts, etc.

NAEP participation is entirely voluntary Teachers survey would also be voluntary No way to accurately forecast who will be undergoing

the research and how they represent the actual population of the United States visual arts classroom

NAEP visual arts exam only covers eighth grade students Only be administered to corresponding eighth grade

teachers of the visual arts program

METHODOLOGY

Procedures

Similarly follow NAEP testing to adhere to procedural protocol

Teachers will be given a general background questionnaire and a subject-area specific questionnaire Consists of a series of select-response questions

Teachers will mark their answers in their booklet or record answers online as accurately as possible

Once the survey is finished the online answers will be saved or the booklet can be given to the NAEP school coordinator

Methodology – Descriptive/Quantitative Used to look for trends and graph opinions, facts and

demographic data Used to make recommendations for classroom application Could prove to be effective information for correlation tests

INSTRUMENTATION

Development based on: Other teacher questionnaires

Reading and writing teacher questionnaire. (2011). National Assessment for Educational Progress.

Writing teacher questionnaire. (2010). National Assessment for Educational Progress.

Teacher data in NAEP Data Explorer NAEP 1997 national theatre results. (2002). National

Assessment of Educational Progress. Considered questionnaire development resources

Gillham, B. (2000). Developing a questionnaire. New York, NY: Continuum.

NAEP teacher questionnaire overviews Teacher questionnaire. (2011). National Assessment of

Educational Progress.

DATA ANALYSIS

Best done by professional statistician Per advice for collaboration within quantitative

research (Brewer, 2013)

Descriptive Analysis will show trends, demographic data, etc.

Correlation Correlation testing to note potential relationships

between student results and teacher questionnaires

RESULTS AND IMPLICATIONS Speculative in nature Descriptive and correlation research

Whatever results are reported, they will be limited to: Making recommendations, not judgments Seeing relationships, not causes

Add transparency to results Show that specific subjects are highly promoted or often

neglected in classrooms See what practices (i.e. writing, production, assessment,

presentation, critical analysis) are being done in classrooms and which are not

Educational background and current practice and training in the field of the teachers

With that information, we can compare the educators with the “ideal practices” and see how their classrooms performed on NAEP testing and determine possible explanations for success

or failure by looking for patterns.

RESULTS AND IMPLICATIONS More background information = results will likely be more

generalizable and reliable (Brewer, 2013) Results from NAEP follow principle with teacher background Generalizability usefulness

Step towards examining school structure and culture that Eisner calls for in order to make improvements in student achievement (1999)

Could likely affect the qualifications for hiring and successful preparation programs if Hatfield is correct

Relationships between student success and certain visual arts subjects and practices individual classroom structures may progress and a possibility for curriculum improvements Rationale for teacher adjustments

The call for direct applications may finally be heard and answered.

LIMITATIONS

Not having a board or a panel creating the survey Solely developed by me

Based on what I want to know – no hidden agendas No other perspectives or experiences Based on my experience or lack thereof

Cause assumptions because of what I think I know about the issues (Gillham, 2000)

Quick development No pre-pilot or pilot stage

Affects wording and understanding (Gillham, 2000) Assumed done from other questionnaires

Sample population variable

INSTRUMENTATION

INSTRUMENTATION

INSTRUMENTATION

INSTRUMENTATION

INSTRUMENTATION

INSTRUMENTATION

INSTRUMENTATION

REFERENCESBrewer, T. (Forthcoming, 2013). A primer for today’s quantitative research in art education. In K. Miraglia & C.

Similian (Eds), Inquiry in Action: Research Methodologies in Art Education. Reston, VA: National Art Education Association.

Diket, R. M., & Brewer, T. M. (2011). NAEP and policy: Chasing the tail of the assessment tiger. Arts Education Policy Review, 112(1), 35-47. Retrieved from http://www.informaworld.com.ezproxy.lib.ucf.edu/openurl?genre=article&id=doi:10.1080/10632913.2011.518126

Eisner, E. W. (1999). The national assessment in the visual arts. Arts Education Policy Review, 100(6), 16-20. Retrieved from http://ezproxy.net.ucf.edu/login?url=http://search.ebscohost.com.ezproxy.net.ucf.edu/login.aspx?direct=true&db=eric&AN=EJ624037&site=ehost-live

Gillham, B. (2000). Developing a questionnaire. New York, NY: Continuum.

Hatfield, T. A. (2007). The unevenness of arts education policies. Arts Education Policy Review, 108(5), 9-13. Retrieved from http://ezproxy.net.ucf.edu/login?url=http://search.ebscohost.com.ezproxy.net.ucf.edu/login.aspx?direct=true&db=eric&AN=EJ771257&site=ehost-live

Lehman, P. R. (1999). Introduction to the symposium on the "NAEP 1997 arts report card.". Arts Education Policy Review, 100(6), 12-15. Retrieved from http://ezproxy.net.ucf.edu/login?url=http://search.ebscohost.com.ezproxy.net.ucf.edu/login.aspx?direct=true&db=eric&AN=EJ624036&site=ehost-live

Mathematics teacher questionnaire. (2013). National Assessment for Educational Progress. Retrieved from http://nces.ed.gov/nationsreportcard/bgquest.asp

NAEP 1997 national theatre results. (2002). National Assessment of Educational Progress. Retrieved from http://nces.ed.gov/nationsreportcard/tables/art1997/sdt02.asp

National Assessment for Educational Progress (NAEP). (2012). Questionnaires for Students, Teachers, and Schools. Retrieved from http://nces.ed.gov/nationsreportcard/bgquest.asp

Reading and writing teacher questionnaire. (2011). National Assessment for Educational Progress. Retrieved from http://nces.ed.gov/nationsreportcard/bgquest.asp

Teacher questionnaire. (2011). National Assessment of Educational Progress. Retrieved from http://nces.ed.gov/nationsreportcard/tdw/instruments/noncog_teach.asp

Writing teacher questionnaire. (2010). National Assessment for Educational Progress. Retrieved from http://nces.ed.gov/nationsreportcard/bgquest.asp