the evaluation of rsc materials for schools and colleges · 2019-04-26 · patricia portsmouth,...

The Evaluation of RSCMaterials for Schools andCollegesA report for the Royal Society of Chemistry

ISBN 0-85404-276-8Royal Society of Chemistry

Burlington HousePiccadillyLondon W1J 0BA

T +44 (0)20 7440 3344F +44 (0)20 7287 9825E [email protected]

Registered Charity Number 207890

Patricia MurphyHelen JonesStephen Lunn

November 2004

The RSC is the largest organisation in Europe for advancing the chemical sciences. Supported by a network of 45,000 membersworldwide and an internationally acclaimed publishing business, our activities span educations and training, conference andscience policy, and the promotion of the chemical sciences to the public.

Evaluation of RSC Materials 23/11/04 8:47 am

1

THE EVALUATION OF RSC MATERIALS FOR SCHOOLS AND COLLEGES

A REPORT

Patricia Murphy Helen Jones Stephen Lunn The Faculty of Education and Language Studies

2

Report of Evaluation of RSC materials for Schools and Colleges

September 2004

Patricia Murphy Helen Jones

Stephen Lunn

Centre for Curriculum and Teaching Studies Faculty of Education and Language Studies

The Open University Milton Keynes

MK7 6AA

Tel. 01908 652587 [email protected]

© Copyright 2004

3

Acknowledgements The evaluation study and the production of this report has only been possible with the assistance of a large number of people. The authors and the Royal Society of Chemistry would like to acknowledge and thank the following: Dr Colin Osborne, Education Manager, Schools & Colleges, Royal Society of

Chemistry, London

Lorraine Hart, Education Department, Royal Society of Chemistry, London

Fiona Walker, Senior Web Editor, Royal Society of Chemistry, London

Barry Anderson, Manager, Sales & Customer Care Department, Royal Society of Chemistry, Cambridge

Dr Wilson Flood and Dr Nigel Botting, organisers of the Eighth National Scottish Meeting for Teachers of Chemistry

Dr Robert Maguire, CCEA, Belfast

Patricia Portsmouth, Research Secretary, The Open University, Milton Keynes

We would also like to acknowledge and thank all the teachers who volunteered their time and their insights in spite of their daily pressures. Without their involvement the study could not have succeeded. We are particularly indebted to the teachers and the students who allowed us into their classrooms to observe and interview and RSC in-service training providers, especially those who allowed us into their workshops and the teachers who were their students. Thank you.

4

Contents Acknowledgements....................................................................................................3 Contents.....................................................................................................................4 Executive Summary ...................................................................................................5

Background............................................................................................................5 Key findings...........................................................................................................6 Recommendations ................................................................................................13

1.0 Introduction..................................................................................................15 1.1 Background to the study ...........................................................................15 1.2 The report of the study..............................................................................15

2.0 The Evaluation Study ...................................................................................16 2.1 The aims of the evaluation study...............................................................16 2.2 The Design of the study ............................................................................17 2.3 Data collection methods............................................................................19

3.0 Dissemination...............................................................................................21 3.1 Volunteer sample ......................................................................................21 3.2 Cold call sample .......................................................................................25 3.3 Issues Emerging re Dissemination ............................................................40

4.0 Product Evaluation .......................................................................................42 4.1 Some general findings across products......................................................42 4.2 Paper-based products................................................................................46 4.3 Multimedia product ..................................................................................70 4.4 INSET product .........................................................................................82 4.5 Web-based product .................................................................................105 4.6 Issues emerging from the product evaluation ..........................................111

References .............................................................................................................120

5

Executive Summary

Background The RSC Education Department has a history and reputation for providing quality support for science and chemistry teachers and their students across the phases of education. The majority of publications are targeted at the age range from 11–19 in all schools including public, private, selective and comprehensive. The materials are made available in England, Wales, Northern Ireland and Scotland and latterly in Eire. The RSC publications include paper-based products, multimedia packages, web-based products and INSET provision. The RSC is currently engaged in a new planning cycle and for this purpose it was decided to fund an independent evaluation study. The intention was to extend the depth of evaluation of a small number of products to provide the RSC with insights about the success of their products in meeting teachers’ and students’ needs. A team based at the Open University was commissioned to conduct the independent evaluation. The study began in September 2003. Data collection commenced in January 2004 and continued to mid September 2004 with data analysis continuing through September 2004. The aims of the evaluation study were twofold, to evaluate: • the effectiveness of the distribution and dissemination of products; • the use and effectiveness of the RSC materials for schools and colleges. To address the first aim the following issues were considered: - teachers’ access to dissemination routes used by the RSC; - how teachers found out about and got hold of RSC products; - how effective teachers felt the RSC product distribution was; - teachers’ perceptions of the problems in dissemination of RSC products; - how RSC resources are made available within schools. To address the second aim the following issues were considered: - teachers’ access to and use of selected products; - teachers’ specific uses of the products; - teachers’ and students’ rating of the products; - impact on students’ learning; - teachers’ preferred choice of medium for products. The intention of the study was to provide a breadth and depth of data about a single product within the four main categories of products produced by the RSC. The evaluation was restricted to products intended for the main audience of teachers of the 11–19 age-range. The following products were selected:

• Paper-based: Chemical Misconceptions: Volume 1 and Chemical

Misconceptions: Volume II; Classic Chemistry Experiments and Classic Chemistry Demonstrations; Ideas and Evidence series.

• Multimedia: Alchemy? CDROM.

6

• INSET: Improving Teaching and Learning Chemistry using ICT; Using Assessment to Improve Learning in Chemistry and Science.

• Web-based: Joint Earth Science Education Initiative (JESEI). To examine the effectiveness of the dissemination systems a sample of teachers – the cold call sample – were approached directly by telephone or questionnaire and included teachers in Scotland, Northern Ireland and three LEAs in England and South Wales. For the product evaluation a volunteer sample of teachers that included users and non-users of RSC products across school type in England and Wales was invited to be involved The main data collection was via questionnaire surveys and interviews. 610 questionnaires were circulated and 70 questionnaires (12%) were returned by the cold call sample and five of these teachers were interviewed. 248 questionnaires were circulated and 168 questionnaires were returned (68%) for the product evaluation. After the product questionnaire survey a further sample of teachers (N=16) was selected for interview within each product type and then from this sample a case study teacher was selected to observe the use of the product with students. The case studies were restricted to three product types. In each case study the teacher was observed and interviewed and a sample of students interviewed (N =14). Student questionnaires were used in two cases and administered directly by the evaluation team so return was 100% (N=73).

Key findings

Effectiveness of dissemination • Overall for the majority of teachers the dissemination of, and access to, RSC

products is successful. • The RSC publication Education in Chemistry, sent free to all schools, is the key

means by which teachers find out about RSC products. The ASE publications are significant in this respect particularly for teachers in England and Wales. The RSC websites are frequently used by the majority of teachers with the exception of teachers in NI secondary schools.

• The majority of teachers report that access to RSC products is unproblematic. • However significantly more teachers in NI secondary schools than other teachers

experience difficulties in getting hold of products. More teachers in English and Welsh schools than teachers in NI grammar schools and Scottish schools do not consider getting hold of RSC products to be easy. These findings may reflect the lower proportion of RSC non-members in these two samples.

• Teachers reported that they felt privileged to receive the RSC products and

publications directly and without charge. It singled them out from their colleagues. • The main route for getting hold of products is the direct mailing or from

departmental libraries.

7

• There is an expected link between teachers’ access to the main RSC information sites and their reported difficulties with access to products. Teachers who reported no difficulties with access not only frequently access main information sites they also have organised access to RSC products in their departments and tend to have colleagues who help keep them informed.

• A third of teachers reported that problems in dissemination occurred within the

school, college or department. These problems could not be solved by the RSC. The evaluation found that turnover of staff was a significant issue and therefore the present mailing system was appropriate.

• NI secondary school teachers appear to be less aware of the direct mailing of

products or are not the member of staff who receives them. This may reflect the departmental structures in the schools and the curriculum focus. These problems also seem to affect lecturers in FE colleges.

• The RSC websites are frequently used by about a third of teachers to get products

and this reduces to about a fifth of teachers in NI schools. Overall nearly two fifths of teachers reported only occasional use of the websites to access products.

• The teachers who reported using web-based resources for chemistry, cited the

RSC sites slightly more often than other sites. • A large proportion of the cold call teacher sample used web-based resources for

teaching chemistry. About a third of teachers reported frequent access to web-based resources during lessons though this figure rose to around two-fifths to a half, accessing resources from home.

• A notable exception to access to the web was noted for teachers in NI secondary

schools with the majority reporting only occasional or rare access during lesson time. This reflects resource constraints of both equipment and time and is therefore not something that the RSC can address.

• Overall teachers wanted the current RSC systems of dissemination to schools and

colleges to be maintained

Barriers to access • Around two thirds of teachers considered the RSC sites to be easy to navigate.

This figure falls to a third for teachers in NI secondary schools. A third of NI secondary school teachers considered they needed training support to improve their access. There is an expected link therefore between frequency of use and familiarity with the site, which can result in a barrier to access for some teachers.

• Time constraints were identified as a barrier to using the web. This compounded

the problem of web access for teachers who were less familiar with the sites. There are teachers therefore who would benefit from wider access to RSC products that are currently not aware of what products are available.

8

• Generally teachers reported that their access to RSC products was constrained by

the lack of direct cross-referencing to exam and curriculum specifications. This affected their use of the web and of the products themselves.

• Significant numbers of teachers in England and Wales and NI secondary schools

wanted more guidance about age range of use and difficulty level to increase their access to products.

• Overall teachers tend to see the RSC products as suited to older (KS4 and 5) and

more able students. This perception shapes teachers’ engagement with the RSC products and can act as a barrier to access for some groups. These groups are those in non-selective schools or secondary schools who predominantly teach combined science to KS3 and teach double award science to KS4 students as a single subject rather than as separate disciplines in rotation.

• The prescribed curriculum and examination system is seen as a barrier to some

teachers’ use of products. • Most teachers would be more encouraged to use the RSC sites if they could (i) see

how the products and their particular contents mapped on to the specifications that they have to work to and (ii) were able to narrow their searches to their particular needs. This is already being addressed by the RSC in their development of a content management system for the websites.

Use and effectiveness of products • Teachers involved in the evaluation of products were generally very committed to,

and appreciative of, the RSC’s contribution. • Members and non-members alike commented on the quality of the RSC provision.

‘In general I find RSC materials fantastic’ was typical of the comments received. • The paper-based products were all rated highly for their content and clarity.

Effectiveness in meeting needs was considered to be good to excellent. • The vast majority of teachers reported having access to the Classic Chemistry

texts. This fell to a half for the Misconceptions texts and the multimedia product Alchemy? Only a fifth of teachers were aware of the Ideas and Evidence series. These findings reflect in some cases, the length of time the products have been available and the opportunities for them to become integrated into schemes of work. It is likely, however, to be a combined effect of length of time available and teachers’ views of the relevance of the products to core curriculum and examination specifications.

• The majority of teachers wanted products in multiple formats. CDROMs were

particularly favoured with DVDs second. Web resources were rated highly by

9

about half of the teachers. The current policy of the RSC to adopt an intermediate media position with regard to products is in keeping with teachers’ preferences.

• Across the paper-based products ratings of ease of use were slightly depressed.

This reflected a general need expressed for more cross-referencing of product components to examination and curriculum specifications. A significant number of teachers also wanted more guidance about the age range of use and the difficulty level of specific student activities, experiments and demonstrations.

Paper-based products • Teachers welcomed having access to relevant research evidence in the Chemical

Misconceptions product and reported that they selected Misconceptions Volume I to develop their practice and their understanding of students’ learning.

• Teachers were using the activities and guidance across the key stages. A common

comment was that the product was a ‘very useful tool’. • The student resources and guidance for teaching in Volume II were rated

exceptionally highly against all criteria. Teachers found the guidance about target level i.e. age and ability very effective in supporting their planning.

• Teachers reported a very high level of success in improving students’ learning

across the vast majority of the most frequently used probes in Volume II. The feedback from interviews supported this view.

• Students observed and interviewed had extended their understanding of key

concepts about the nature of matter. Where confusions remained this was largely due to the way in which the teacher had used the probe.

• All teachers’ using the Misconceptions texts had altered and extended their view

of learning irrespective of their starting point perspectives. This was confirmed in interviews:

‘It really did enlighten me.’

‘You suddenly realise a bit more about how the children are thinking rather than how we are thinking.’

• Teachers who no longer used the product had incorporated it into their practice

and their schemes of work or had moved on and sought out additional research-based material. Non-users who were introduced to the product via the study were keen to make use of it in the future.

• The teachers found the product an effective tool for mentoring student teachers. • The structure of the probes reflects an embedded pedagogic strategy that is

successful. However when teachers amend the probes and are not aware of the pedagogic approach opportunities for learning can be missed. Many teachers would benefit from additional information about the thinking behind the structure of the probes in relation to the pedagogic approach and how students learn.

10

• The Classic Chemistry Experiments and Classic Chemistry Demonstrations

product was rated very highly as an effective resource.

‘I haven’t found a single resource of demonstrations like this, good ideas, succinct and well written.’

• There was an impressive use of both Experiments and Demonstrations spread across the age ranges but predominantly use is in key stages 3 and 4. Teachers commented on the high quality of the texts as a teaching resource and the clarity and quality of the instructions for students.

• A third of teachers wanted cross-referencing to curriculum and exam

specifications for this product particularly. • The majority of the teachers rated the product highly in terms of impact on

students’ learning. They reported that all the major learning objectives targeted were well met.

• Teachers rated the Demonstrations text very highly in developing students’

motivation and interest in chemistry and enabling them to understand the link between theory and practice.

• Teachers rated the Experiments text highly in terms of developing students’

knowledge of, and practice in, experimental skills, the effective use of equipment and materials and the ability to follow instructions.

• Teachers tended not to select the more open approaches to experiments, which

allow students’ more autonomy and decision–making to develop investigative and problem solving skills. These skills are a central part of science and chemistry curriculum specifications. Some teachers were not clear about how the choice of approach allowed for different learning opportunities.

• Only about a quarter of teachers in the whole sample reported that they had access

to the Ideas and Evidence series. This was low in comparison to other products published at a later date. The cold call interviews supported this finding.

• The majority of the teachers who had access rated the product as good to

excellent. • The findings suggest that teachers who are aware of the series are using those

books that are more obviously part of the curriculum and exam specifications. Hence perceived relevance is a key determinant of access.

• The relevance of other books such as Green Chemistry and Climate Change is less

obvious to teachers because of the innovative nature of their content and the lesser extent to which it is signalled in curriculum and exam specifications.

• The series of books are very much in tune with future policy directions for the

science and chemistry curriculum. The findings indicate that there is considerable potential in the product that is not yet being exploited.

11

Multimedia product • All teachers rated the multimedia product Alchemy? as excellent or good.

Feedback from the cold call interviews supported this view. Teachers choose to use Alchemy? and not other resources because it is ‘clear, concise, includes diagrams, and is relevant to the specification’.

• Teachers found the multimedia nature of the product particularly valuable in

motivating students and in enabling them to support a variety of ways of teaching that included whole class work, group work and independent study.

• The product was used predominantly at key stages 4 and 5. The more effective

teachers found a topic in its impact on students’ learning the more frequently they used it.

• Teachers gave very high ratings about how well the product met its main learning

objectives. The case study teacher commented about Alchemy?

‘From somebody that loves using digital video in so many different ways, you know, the Alchemy? stuff has really hit the spot with Year 10 and Year 11, and Year 12 as well.’

• The objectives less well met concern making links between science and school

learning and practices in the chemical industry and between science and everyday artefacts. Both objectives are types of learning that it is hoped the future science curriculum will better address. There is potential to highlight the way that this product can be used to achieve these objectives and so enhance its impact.

• The CDROM format and use of short video clips is very popular with teachers,

who would like more of the same: as one said,

‘More! Digital video clips are the way forward in science education’.

• All teachers reported using the video clips, which compared with about half to two

thirds using the other components of the product. The web links were the least well used component for reasons discussed in the dissemination section above.

INSET • The RSC INSET events are very popular and, in the view of one of the trainers

that is because: ‘the RSC has got such a high reputation ... it means that people expect that it will be good so they will come on it.’

• Teachers rated the training exceptionally highly in terms of how well it met their

needs. This is the result of a combination of factors: well qualified and committed trainers who have high credibility with other teachers; well designed activities; and perhaps most important of all a focus on practice and how to do things with students. One teacher described the RSC INSET as ‘very effective in making me think, reflect on my teaching and on what I was going to do next. Because I

12

always like to do something that I know I can implement in the classroom and I think this one was very much like that’.

• Teachers’ feedback about actual and planned implementation showed that there

had been a very significant impact on practice. The case study data supported this. • Over 90% of the teachers attending the ICT event reported that students’ learning

had been enhanced. A fifth of the teachers attending the Assessment event reported that students’ motivation and enjoyment had improved. Teachers also felt that the techniques worked well across students of all abilities.

• The evidence from the case study revealed that students were broadening their

understanding of assessment and its role in improving their learning. Over a third of students considered the purpose of the new assessment techniques was to help them to better understand science; what constitutes good work; and their learning objectives. Over half considered that being a marker was helpful for their learning and a slightly smaller proportion reported that peer marking was beneficial.

• Teachers thought that resource books or materials with examples of techniques

and pupils’ and teachers’ work would enable them to sustain their professional development. They would also value further events to extend their understanding, repeat events for colleagues and more opportunities for networking to share good practice, ideas and experiences.

Web-based product • Those teachers providing feedback on the JESEI site and materials rated it very

high quality, very useful, clear and easy to use. The topics were widely used across key stages 3, 4 and 5.

• Teachers valued the flexibility, quality, innovative approach and usefulness of the

JESEI product. • The teachers’ rating of how well objectives were met for their own professional

development were very high with the majority reporting that it improved their understanding, interest and motivation in the earth sciences. As one commented ‘Overall they make a potentially very dry subject far more enjoyable.’

• Two-thirds of the teachers said the product gave students a better idea of the

subject and half of the teachers felt it increased students’ motivation and engagement in the subject.

‘Students enjoyed the opportunity to work in a very different way to that normally found in science. They were well engaged.’

• Some of the teachers were using the site and the materials extensively. They had integrated them into their core teaching and as homework resources. Other teachers described their use as a matter of ‘chance’.

• Only 20% of the teachers in the volunteer sample were aware of the JESEI site.

The feedback suggests that this is an excellent resource but its potential for

13

impacting on teaching and learning is not realised because significant numbers of teachers continue to be unaware of it.

Recommendations • The current systems of dissemination via mailing of Education in Chemistry and

products direct to schools and colleges is effective and no alternative methods would increase efficiency as barriers to distribution are within institutions and often are a consequence of staff turnover.

• Teachers who access the websites do not experience difficulties with navigation of

the sites but access is constrained by the limited cross-referencing to curriculum and exam specifications for all teachers. The content management system that is currently under development should continue as it is key to extending access.

• The evaluation feedback should be considered in the development of the content

management system as it indicates which type of referencing is needed for certain products and for which groups of teachers.

• Teachers’ specific feedback about which aspects of products they use and with

which students, and their judgements about their impact should be used to alert teachers who are accessing the websites to relevant resources and to inform their selection.

• Many teachers see themselves as science teachers who teach chemistry. A

significant barrier to access is these teachers’ perceptions of the relevance of RSC products in relation to curriculum and exam specifications and the ability range of their students. Marketing and re-marketing of products should direct these teachers first to the age range and the curriculum specifications that aspects of products address.

• There are a significant number of science teachers who would benefit from the

RSC’s expert advice and resources who do not use the current systems of dissemination for the reasons given above. Alternative ways of increasing awareness of the potential of RSC products is needed and collaboration with the Centres for Science Learning in England may be one way of achieving this.

• The study has provided a firm basis for the recommendation that extending the

access to and use of the products will significantly benefit the teaching and learning of science and chemistry and build on the already significant and impressive contribution made by the RSC.

• Teachers want more products. In particular they want products that support

implementation in classrooms by having exemplars of teachers and students work. INSET events are one potential source for these and warrant further exploration.

14

• Teachers prefer products in multiple formats particularly with CDROMs. The RSC is already moving to widen its use of intermediate media and this should continue.

• The evaluation revealed that for at least some of the paper–based products and the

multimedia and web-based product there was significant potential for wider impact. The cross-referencing by curriculum and exam specifications, target age and level of difficulty will address this to some extent. One further recommended way of achieving this is to include as part of the website some explicit guidance about the pedagogy assumed in student activities and how the structure of activities allows for particular learning opportunities.

• Alchemy? is very well received by teachers and students in its approach and

content. The multimedia format allows a range of learning situations including group work and independent study which fits well with the current emphasis on key skills in the 14-19 curriculum. It is a high quality product and a timely one and it warrants further publication through RSC dissemination systems.

• The Ideas and Evidence series and the JESEI site and resources contain both

innovative content and teaching strategies in line with future curriculum developments. These products should be considered for re-marketing perhaps in an alternative format, which highlights their relevance to new curriculum and examination developments and their appropriateness for a wide age range and ability of students.

• RSC INSET is very successful. Teachers want more of the same for other

colleagues, extensions of ongoing events to develop their own practice further and new events. The RSC is widening access to this product and this is welcomed.

• The findings of the study indicate that the RSC is a provider of excellence in

INSET and unlike many providers has evidence of impact to support this claim. There is considerable potential for the RSC to extend its provision, however, staffing resource is a limitation. Provision for teachers could be extended through close collaboration between the RSC and other providers in particular the Centres for Science Learning. It is recommended that this collaboration is fostered to enable wider access to the RSC product.

There are clear directions from teachers about how to widen their access and use of RSC materials for schools and colleges. These directions have to a large extent been anticipated by the RSC in their ongoing developments. There is an issue though of which teachers and which students in which schools that access should be available to. The evaluation has highlighted groups of teachers who through a combination of factors have restricted access to RSC products. It is recommended that serious consideration is given to widening access for these groups who would benefit in particular from the excellent support available in the form of RSC products for their professional development and their students’ learning.

15

1.0 Introduction

1.1 Background to the study The RSC Education Department has a history and reputation for providing quality support for science and chemistry teachers and their students across the phases of education. The majority of publications are targeted at the age range from 11–19 in all schools including public, private, selective and comprehensive. The materials are made available in England, Wales, Northern Ireland and Scotland and latterly in Eire The main dissemination routes are through direct mailing to schools and colleges and key publicity sources, the main being the RSC websites, e-mail list, teacher conferences and the publication – Education in Chemistry; and the Association for Science Education publications and conferences. The RSC is a well-established and well-known source of expertise that is drawn on at national policy level and local level. The recent mapping review of science, technology, engineering and maths (STEM) initiatives (DfES 2004) was carried out to address concerns about the decline in demand of university places in those subjects. The review was to identify gaps in provision and to assess the activity going on externally in order to inform future policy. One of the findings of the review was the limited availability of evaluation data to assess impact with only 7.9% of initiatives appearing to have been evaluated and to be working. A further 16.3% had some evaluation and some indication of impact in increasing pupils’ participation. The RSC has identified the need to get government to recognise that investment in chemistry education and the sciences generally is vital to ensure an adequate number of good university students in the chemical sciences. It is supporting a programme to revolutionise educational outreach and to improve teachers’ access to chemistry teaching material. The RSC publications include paper-based products, multimedia packages, web-based products and INSET provision. These are routinely evaluated by the Education Department in a number of ways including questionnaires and web-based surveys. The RSC is currently engaged in a new planning cycle and for this purpose it was decided to fund an independent evaluation study. The intention of this evaluation was to extend the depth of evaluation of a small number of products to provide the RSC with insights about the success of their products in meeting teachers’ and students’ needs. To this end it was necessary to establish teachers’ and students’ views of the usefulness and appropriateness of products and some indication of how products were being used and the impact of their use on students’ learning. Of concern too was the effectiveness of the dissemination of materials and what, if any, were the barriers to teachers getting access to, and using RSC materials. Where barriers did exist it was also of interest to establish if these were experienced generally or by particular teachers in particular schools and why this might be the case.

1.2 The report of the study The report is in five parts. The Executive Summary reports the main findings and recommendations of the study. The report of the study and the findings is in four parts. Part 1 provides some background to the study. Part 2 describes the study and its design and data collection. Part 3 looks first at dissemination issues emerging from

16

the product evaluation data generally and this is followed by a discussion of dissemination findings from the cold call sample in Scotland, Northern Ireland, England and Wales. Part 4 focuses on the evaluation of specific products - that is access to them, their use and impact on learning. Each part concludes with issues emerging, which are summarised in the Executive Summary.

2.0 The Evaluation Study A team directed by Patricia Murphy at the Open University, Centre for Curriculum and Teaching Studies in the Faculty of Education and Language Studies was commissioned to conduct the independent evaluation of RSC provision. Colin Osborne, Education Manager, Schools and Colleges of the RSC Education Department managed the project. The study began in September 2003. Data collection commenced in January 2004 and continued to mid September 2004 with data analysis continuing through September 2004.

2.1 The aims of the evaluation study The aims of the evaluation study were twofold, to evaluate: • the use and appropriateness of the RSC materials for schools and colleges; • the methods of distribution and dissemination used. To address the first aim the following issues were considered: • teachers’ access to and use of selected products; • teachers’ and students’ access to the products and any difficulties with this; • teachers’ specific uses of the products and with which students, and the reasons

for this in terms of their own teaching needs and the learning needs of their students;

• teachers’ and students’ rating of the products in terms of ease of use and effectiveness for a range of students;

• impact on students’ learning; • teachers’ preferred choice of medium for products; • alternative resources used by teachers and their reasons for this. To address the second aim the following issues were considered: • teachers’ access to dissemination routes used by the RSC; • teachers’ views of the use of RSC products in their departments; • the reasons for teachers’ use of the products and their non-use; • how teachers found out about and got hold of RSC products; • how effective teachers felt the RSC product distribution was; • what teachers’ perceptions of the problems in dissemination of RSC products

were; • how RSC resources are made available within schools; • teachers’ use of web-based resources for supporting learning in science and

chemistry; • teachers’ views of ways in which use of RSC products could be made more

effective.

17

2.2 The Design of the study

2.2.1 Selection of products The intention of the study was to provide a breadth and depth of data about a single product within the four main categories of products produced by the RSC. It was agreed to restrict the evaluation to products intended for the main audience of teachers of the 11–19 age-range. The selection of products from the full range of RSC materials was designed to provide a spread of formats, target ages, popularity (as judged by sales) and purpose. Note was also taken in making the selection of emerging policy trends in science and chemistry education, which the RSC materials were addressing, for example the use of ICT to support learning, and the introduction of formative assessment. A further factor taken into account was national policy initiatives, which had highlighted specific texts for all schools. For example Chemical Misconceptions Volumes I and II featured in the Key Stage 3 science strategy in England as recommended materials. The RSC tender document that outlined the scope of the evaluation recommended that the evaluation examine one product in each category of product. It was however decided to extend this for the paper-based products to include traditional texts, current recommended texts and texts that addressed innovations in the science curriculum. The data to be collected across the paper-based products was varied in terms of depth of coverage of issues to allow the necessary data collection and analysis within the time and cost constraints of the study. The following products were selected: Product Type • Paper based: Chemical Misconceptions: Volume 1 and Chemical Misconceptions:

Volume II, Classic Chemistry Experiments and Classic Chemistry Demonstrations.

• Mixed format, web and paper: Ideas and Evidence (the five booklet series Chemists in a Social and Historical Context; Climate Change; Green Chemistry; Health, Safety & Risk; The Nature of Science).

• Multimedia: Alchemy? CDROM. • INSET: Improving Teaching and Learning Chemistry using ICT; Using

Assessment to Improve Learning in Chemistry and Science. • Web based: Joint Earth Science Education Initiative (JESEI).

2.2.2 Sampling of teachers and schools Two samples were identified to meet the two aims of the study. For the product evaluation a volunteer sample of teachers across school type was needed that included users and non-users of RSC products. Volunteers to review products, both RSC members and others, were sought in a variety of ways mainly in England and Wales. First an invitation was put up on the RSC website, and on the JESEI site and in Education in Chemistry. Second, to extend the range of schools involved and the potential for non-RSC members and non-users of RSC products, an invitation was circulated to all teachers attending the ASE annual conference. At the conference further invitations were handed out at the RSC stand which a team member attended. All those responding were sent a brief questionnaire to indicate the type of involvement they would be prepared to undertake and their preferred method of contact. Finally two RSC INSET events were attended by team members who briefed teachers about the evaluation study and circulated an invitation to volunteer to

18

evaluate their INSET experience and review products. This invitation was circulated to attendees at prior INSET events too. There was therefore some overlap between the INSET sample and the product evaluation sample. Over two hundred volunteers came forward, which was sufficient for the study design and allowed for a degree of non-return. To examine the effectiveness of the dissemination systems a sample of teachers were approached directly by telephone or questionnaire – the cold call sample. This sample included teachers attending a RSC conference in Scotland, teachers in schools in Northern Ireland and teachers in three LEAS in England and South Wales. Seventy responses (12% return rate) were received across these samples. Table 2.1 provides a breakdown of the volunteer sample by product, by school characteristics and teacher characteristics. It should be noted that the sample available for JESEI was small. Across the other product categories there is a spread of school types and non-members and members of the RSC. In each product category the overwhelming majority of teachers volunteering had chemistry as their specialist subject. Table 2.1 Volunteer sample profile

School characteristics Teacher characteristics

Product Type Sex Age Subject

specialism

Age group(s) taught

RSC Membership

Chemical Misconceptions

76% State 19% Ind. 5% FE

62% Comp 33% Select 5% FE

71% Mixed 10% Girls 19% Boys

90% 11-18 5% 11-16 5% FE

95% Chemistry 5% Physics

85% 11-18 10% 11-16 5% 16-18

48% RSC 52% non-RSC

Classic Chemistry Experiments and Demonstrations



92% Mixed 8% Girls

76%11-18 12%11-16 12% FE

92% Chemistry 4% Physics 4% no response

80% 11-18 8% 11-16 12% 16-18

32% RSC 68% non-RSC

Ideas and Evidence

90% State 10% Ind.

80% Comp 20% Select


80%11-18 20%11-16

90% Chemistry 10% Biology

Alchemy?




75%11-18 9%11-16 17% FE

95% Chemistry 5% Biology

70% 11-18 13% 11-16 17% 16-18

41% RSC 59% non-RSC

INSET




87% 11-18 7% 11-16 7% FE

100% Chemistry

81% 11-18 10% 11-16 10% 16-18

35% RSC 66% non-RSC

JESEI web resources

100% State

100% Comp

100% Mixed

100% 11-18

67% Chemistry 17% Physics 17% Biology

100% non-RSC

Figure 2.1 shows the breakdown for the sample by years of teaching experience. This data was not available for a small number of the teachers attending the INSET events. .

19

Figure 2.1 Teachers’ experience

Table 2.2 provides a similar breakdown by school characteristics and teacher characteristics for the cold call sample. Table 2.2 Cold call sample profile Region State/

Independent Comprehensive/ Selective

Sex Age Subject Specialism

RSC membership

England and Wales

85% State 15% Ind.



75% 11-18 10% 11-16 5% 14-18 10% FE

95% Chemistry 5% Physics

30% RSC 70% non-RSC

NI (Grammar)

100% State

100% Select


100% 11-18

94% Chemistry 6% Biochem/ Physiology

41% RSC 59% non-RSC

NI (Secondary)

100% State

100% Comp

93% Mixed 7% Boys

60% 11-18 40% 11-16

73% Chemistry 20% Physics 7% Biology

100% non-RSC

Scotland

87% State 13% Ind.

100% Comp

85% Mixed 15% Girls

94% 11-18 6% 16-18

100% Chemistry

44% RSC 56% non-RSC

Table 2.2 reveals a similar preponderance of chemistry specialists among respondents. However, it is notable that the balance of RSC members versus non-members is significantly different for teachers in NI secondary schools than in other school types and in comparison to the volunteer sample there are also more teachers responding who are not chemistry specialists from these schools.

2.3 Data collection methods

2.3.1 Product evaluation As part of the design of the study the sample of volunteers were asked to indicate for the products selected whether they were past or current users, if they planned to use the product in the future or were non-users. We considered the profile of teachers and

0

5

10

15

20

1 to 5 yrs

6 to 10 yrs

11 to 15 yrs

16 to 20 yrs

21 to 25 yrs

26 to 30 yrs

Teachers' experience

Number of teachers

20

schools for users and non-users and found that non-users were generally not from selective schools. The design was such that a spread of different users was selected for the paper-based and multimedia product types and questionnaires relevant to these users were developed (see Appendix 1). We mailed and sent questionnaires electronically and sent two additional mailings to ensure the highest return rate (see Table 2.3). Each questionnaire was sent with a copy of the product to be evaluated. A questionnaire based on observations of each of the INSET events was developed (see Appendix 2), we sent one additional mailing to this group. A simple questionnaire was sent electronically and mailed to the JESEI group (see Appendix 3). This group was sent an additional mailing, however the response rate was the lowest for all the product types. The decision to extend the data collection about the paper-based products was taken quite late and so a simple questionnaire was placed on the website itself for the Ideas and Evidence group (see Appendix 4). Nor did the respondents get sent copies of these products but they could access the majority of the paper-based product via the website. After the questionnaire surveys a further sample of teachers was selected for interview within each product type and then from this sample a case study teacher was selected to observe the use of the product with students. In each case study the teacher was observed and interviewed and a sample of students interviewed in small groups. When possible evidence of impact through student questionnaires and examples of work was also collected. The case studies were restricted to three product types as volunteers for JESEI were not forthcoming in the period of the study. Only one paper-based product was the subject of a case study, Chemical Misconceptions Vols. I and II. Table 2.3 provides a breakdown of the sample by product that shows the number sent questionnaires, the number responding and the numbers interviewed, both teachers and students, and the sources of other student data.

21

Table 2.3 Data sources by sample Interviews

Volunteer Sample Questionnaires sent

Teacher Questionnaires returned Teachers Students

Student Questionnaires

Misconceptions Vols. I and II

29

21 5 4 20

Experiments and Demonstrations 29 25

Paper based Web & Paper

Ideas and Evidence open 26

Multimedia Alchemy? 30

23 3 4

ICT in Chemistry 60

30

1

INSET Assessment for Learning

58

34 3 6 53

Web based JESEI 32 9

4

2.3.2 Regional dissemination

The cold call sample were either contacted by phone or sent or circulated with questionnaires (N=610). The Scottish teachers were attending the National Scottish Meeting for Teachers of Chemistry conference and were invited to take a questionnaire. The Northern Ireland schools were mailed. We mailed as many schools as possible as we expected a low return because of the earlier closure of schools compared with England and Wales. The questionnaires were generally the same with slight modifications to take account of national contexts. An exemplar questionnaire is given in Appendix 5. Overall 610 questionnaires were circulated with a return rate of 12 % (N=70). The response rate was between 9% and 11% for all groups with the exception of Northern Ireland grammar schools whose response rate was 24%. Table 2.4 shows the number of questionnaires returned from the cold call sample. There were in addition telephone interviews with five teachers to supplement this data. Table 2.4 Cold call sample respondents

Cold call sample questionnaires

England and Wales 20 Scotland 18 N I (grammar) 17 N I (secondary) 15

3.0 Dissemination

3.1 Volunteer sample The product evaluation questionnaires raised issues about dissemination that are reported first. The data is drawn across all questionnaires sent to the volunteer sample but the detail available varies, in particular only a small amount of dissemination data was available from the INSET sample because the focus of this questionnaire was on

22

evaluation of the provision. Figures of .5 and above are rounded up and of .4 and below are rounded down so sometimes this means the totals in tables are slightly under or over 100%. Table 3.1 indicates how teachers rated the ease of access to RSC products. Table 3.1 Ease of access of RSC materials

Product Easy Neither easy or difficult

Difficult

Chemical Misconceptions Volume II 94% - 6% Classic Chemistry Experiments and Demonstrations 74% 20% 6% Alchemy? 78% 22% - JESEI web resources 50% 50% -

Teachers were asked to rate how easy they found it to get hold of RSC materials. The vast majority of teachers reported they find it easy to access RSC materials, with a few finding it neither easy nor difficult, except with the JESEI web resources where half of respondents found it neither easy nor difficult. Only two teachers in the sample had found it difficult to access RSC materials. Teachers were asked how they got hold of the product they were evaluating, whether it was sent directly to them, already available in their department, or they had to track it down. As Table 3.2 shows for nearly half of product users and half of product non-users the product had been sent to them directly. Of the users who had to track the product down, three were Alchemy? reviewers and two were Classic Chemistry Experiments reviewers, there is no obvious connection between the five to explain why they should not have received the products more easily, although it may be worth noting that none of them are RSC members. Feedback from cold call interviews indicated that some teachers had also not seen the Alchemy? CDROM but this seems to be an issue that is school-specific and not related to the RSC distribution system. For example one of the Alchemy? reviewers, in a follow up interview, described how he had come into a new school and found a group of resources stored in a cupboard, one of which had been the Alchemy? CDROM. Table 3.2 How products have been obtained

Sent directly In department Had to track it Direct and in department

Product users 46% 34% 10% 10% Non-users of products 50% 36% - 14%

Teachers were asked whether they would have found it helpful to have the product under review in a different format. What is clear from Table 3.3 is that teachers welcome multiple formats, which allow ease of access to photocopiable or downloadable resources and which therefore save them time. Ring binders with text were seen as a popular addition for both paper-based and multimedia products. CDROMs were seen as a very useful addition to paper-based resources and the web-based resources. Teachers expressed concern about time and access in relation to their preference for particular formats and these were their reasons for not going for others such as web based. This may reflect their beliefs rather than their experience although

23

the JESEI users did indicate some difficulty with access. However it should be noted that the multimedia users would have welcomed materials on the web. Table 3.3 Alternative formats preferred

Product Book Ring binder

Web Ring & web

CDROM CDROM & ring

CDROM & web

All ticked

Misconceptions - 19% - 25% 19% - 31% 6% Expts + Demos - 32% 5% 23% 27% 9% - 5% Alchemy? 9% 55% 36% - - - - - JESEI - 20% - 40% 40% - - -

To explore the issue of web-based resources teachers were asked generally about their use of web resources for chemistry. Of those who responded (N=70) 67% did and 33% did not make use of web-based resources. The main reasons given for not making use of web-based resources were teachers' lack of access to a computer (41%), lack of access to the Internet (23%), and lack of time to locate and try out materials (23%). Other reasons mentioned were the materials being in unsuitable formats, lack of training, lack of class access to Internet for student use, and there being easier ways to acquire suitable resources. One teacher said, 'With other good resources e.g. Alchemy? and Spectroscopy it [Internet access] is unnecessary. It is also more trouble than it is worth and fairly unreliable.' The 47 teachers reporting making use of web-based resources in their chemistry teaching made 145 citations of websites. As far as the level of detail given by teachers allows, these are analysed below in terms of the type of organisation behind the sites, the purposes they serve for teachers and pupils, and the areas of science covered. Table 3.4 Types of organisation

Type of organisation No. citations % of citations (N=145)

RSC sites (including JESEI) 51 35% School-oriented science/curriculum support sites 43 30% 'Private' sites (e.g. school science departments, individuals) 15 10% H.E. sites (60% U.S., 33% U.K.) 14 10% BBC sites (including Bite-size revision) 9 6% Exam boards/curriculum developers 6 4% Educational professional sites (ASE, ESTA) 5 3% General .com sites 2 1%

Table 3.5 Purposes served

Purposes No. citations % of citations (N=91)

Science content 33 36% ICT-based resources for teacher use 20 22% Worksheets, materials for paper-based student use 15 16% Revision material for students 14 15% Specification/curriculum information for teachers 6 7% ICT-based resources for student use 3 3%

24

Table 3.6 Areas of science covered Areas of science No. citations % of citations

(N=45)

Periodic table/elements ( RSC/ChemSoc) 18 40% Earth science (JESEI + 2 others) 14 31% Practical/laboratory 5 11% Biochemistry/medicine 2 4% Chemical reaction mechanisms 2 4% Balancing equations 2 4% Spectroscopy 2 4%

Teachers were also asked whether they had viewed the web resources related to the product they were evaluating. Reviewers of the Classic Chemistry and Chemical Misconceptions titles were also asked whether they had downloaded any of these resources, and if so which. Whilst Alchemy? reviewers were asked whether they had followed any of the links on Alchemy? RSC website, and how they had made use of these. Table 3.7 Viewing product-related websites

Viewed product- related website

Alchemy? (19)

Chemical Misconceptions (16)

Classic Chemistry Experiments (19)

Classic Chemistry Demonstrations (22)

Yes 53% 19% 26% 18% No 47% 81% 74% 82%

Table 3.8 Downloading resources and exploring links

Downloaded resources/ Explored links

Alchemy? (12)

Chemical Misconceptions (7)

Classic Chemistry Experiments (4)

Classic Chemistry Demonstrations (4)

Yes 33% 14% 75% - No 67% 86% 25% 100%

There was a relatively low proportion of teachers seeking additional information and resources via the web, which reflects the comments made above. More teachers using Alchemy? report making use of the web but only just over 50%. Only a sub set of those who browse actually download or explore links. Those who do browse reported they did so for student worksheets which some found invaluable, and extension work. Teachers’ comments suggested that what they searched for was not necessarily on the site, particularly extension work for students. In the product evaluation section of the report we reconsider the issues around web-based use with reference to the multimedia and web-based products. There is a common conflict in meeting teachers’ needs that all providers have to consider. Teachers value flexibility in resources so that they can choose to use them to fit their context and their student needs. Flexibility can however create problems of access as teachers fail to locate resources for particular students or particular curriculum issues. This is particularly the case for new teachers and/or new users of products. The RSC products cover a wide range of age ranges and have to allow use across a range of national curricula and exam specifications. Typically the RSC products specify quite an extended age range for use and allow teachers to plan how to use the products. Detail is given about the purpose and learning outcomes within products for different activities to enable this planning. Teachers’ views about whether products would be more accessible if they were cross-referenced more specifically were sought. These responses are predominately from English teachers.

25

The ratings were on a five point scale and for reporting purposes figures have been combined for those expressing a positive response and those expressing a negative response. Table 3.9 Cross-referencing with exam specifications

Product Yes Don’t mind No

Chemical Misconceptions Volume I 46% 47% 9% Chemical Misconceptions Volume II 36% 46% 18% Classic Chemistry Experiments 67% 28% 6% Classic Chemistry Demonstrations 55% 40% 5% Alchemy? 53% 47% -

Table 3.10 Cross-referencing with National Curriculum

Product: Yes Don’t mind No

Chemical Misconceptions Volume I 55% 46% - Chemical Misconceptions Volume II 36% 55% 9% Classic Chemistry Experiments 67% 17% 17% Classic Chemistry Demonstrations 71% 24% 5% Alchemy? 13% 67% 20%

The findings in Table 3.9 suggest that for those teachers working with KS4 and KS5 students a significant proportion would welcome activities and content referenced to the exam specification that they use. The figure for those teachers valuing cross-referencing to the curriculum and programmes of study increases slightly for the paper-based texts and notably not for Alchemy?. This reflects the quite distinct nature of the content in the multimedia resource compared with the wider ranging content in the paper-based products. The evidence here provides support for the developments already being undertaken to provide a content management facility to the RSC websites that will allow this cross referencing.

3.2 Cold call sample The cold call sample provides a different perspective on dissemination and access to that provided by the volunteer sample who responded in the context of a specific product and were a self-selected group. In Northern Ireland pupils aged 11–14 are in years 8–10, and pupils aged 14–16 are in school years 11–12. Questions in the Northern Ireland questionnaire reflected these differences, however in this report all pupils aged 11–14 are described as in years 7–9, and pupils aged 14–16 are described as in school years 10–11. RSC events and conferences are also not available in Northern Ireland though the provision is being extended to NI in 2004–05. We report the questionnaire findings first and then supplement these with insights from interviews with seven additional teachers from an LEA in England. It is clear from Table 3.11 and 3.12 below that teachers’ views of the ease of access to RSC materials and their schools’ ability to get hold of products is similar to that expressed by the volunteer sample. The majority of teachers in selective schools in NI and in schools in England, Wales and Scotland report no difficulties with access. This is in marked contrast to the response of teachers in secondary schools in Northern Ireland where over half the teachers have reservations about their schools’ ability to access products. Note that whereas about a third or more of teachers in the sample are RSC members there were no RSC members in the NI secondary teachers group.

26

3.2.1 Ease of finding out about and getting hold of RSC products Table 3.11 How easy is it for teachers to find out about RSC materials?

Region Very easy 2 3 4 Very difficult

England and Wales 37% 37% 16% 5% 5% NI – Grammar 35% 47% 12% - 6% NI – Secondary 7% 29% 43% 14% 7% Scotland 56% 33% 11% - -

Table 3.12 How easy is it for teachers to get hold of RSC materials?

Region Very easy 2 3 4 Very difficult

England and Wales 30% 20% 40% 5% 5% NI – Grammar 35% 35% 18% 12% - NI – Secondary - 43% 29% 21% 7% Scotland 50% 28% 17% 6% -

It is significant that the teachers responding in NI secondary schools also report difficulties in getting hold of the products within their schools. More teachers in the England and Wales sample compared with Scotland and NI grammar teachers also reported that it was not that easy to get hold of RSC products. This question was therefore followed up by asking teachers about their personal access to and use of RSC information sites (see Table 3.13) and which of these they use to get hold of RSC products (see Table 3.14). (Note that RSC events are not available in Northern Ireland.) The RSC publication, Education in Chemistry, sent free to all schools is by far and away the key means of finding out about products for teachers and to a lesser extent the ASE publication. Table 3.14 indicates that except for NI secondary teachers the publication is frequently used by 40%- 60% of teachers. The RSC websites are the next most highly accessed and used site of information with the notable exception of NI secondary teachers. Only a fifth of respondents in NI secondary schools compared with four fifths of the Scottish teachers use the RSC website. They therefore do not use this as means of finding out about products in contrast to the Scottish teachers. Scottish teachers and NI grammar school teachers use the website more frequently than their counterparts in England and Wales. Secondary teachers in NI are more likely to use general science conferences and information sites to find out about products than ones dedicated to chemistry which may well reflect the emphasis on double award science in the science curriculum of these schools. The ASE publication is accessed and used by a significantly higher proportion of teachers in England and Wales than teachers in Scotland and NI. It appears to be the third most popular method of access to information but not for teachers in NI grammar schools. From Table 3.13 it appears that approximately the same proportion of teachers in NI secondary schools reporting difficulties with access and getting hold of products do not have access to the main RSC information sites. More teachers in Scotland and in NI grammar schools use direct means such as e-mail than teachers in England and Wales and secondary schools in NI.

27

Table 3.13 Teachers’ access to and personal use of sources of RSC product information

Access to Personal use

England and Wales 70% 55% NI – Grammar 35% 18% NI – Secondary 53% 40%

School Science Review (ASE journal)

Scotland 56% 39%


Education in Chemistry (RSC journal)

Scotland 94% 83%


RSC e-mail list

Scotland 44% 28%


RSC and chemsoc websites

Scotland 94% 83%

England and Wales 45% 15% NI – Grammar 35% 6% NI – Secondary 13% -

RSC CPD

Scotland 61% 39%

England and Wales 55% 20% NI – Grammar 53% 6% NI – Secondary 13% -

RSC teacher conferences

Scotland 83% 61%


Other meetings/ conferences (e.g. ASE)

Scotland 67% 61%

28

Table 3.14 How teachers find out about RSC materials Very frequently Frequent Occasional Rare Never

England and Wales 29% 18% 12% 29% 12% NI – Grammar 8% - 33% 33% 25% NI – Secondary - 27% 55% - 18%

School Science Review

Scotland - 36% 21% 14% 29%

England and Wales 32% 11% 37% 16% 5% NI – Grammar 18% 29% 53% - - NI – Secondary 17% 25% 33% 8% 17%

Education in Chemistry

Scotland 24% 35% 29% 6% 6%

England and Wales - 7% 7% 7% 80% NI – Grammar - 9% 18% 9% 64% NI – Secondary - - 11% - 89%

RSC e-mail list

Scotland 8% 15% 15% 8% 54%

England and Wales - 11% 61% 17% 11% NI – Grammar 7% 29% 43% 14% 7% NI – Secondary - - 60% 20% 20%

RSC websites

Scotland 12% 35% 53% - -

England and Wales - 14% 7% 21% 57% NI – Grammar - - 14% 21% 64% NI – Secondary 10% - - - 90%

RSC CPD courses

Scotland - 14% 36% 7% 43%

England and Wales 7% 7% 13% 20% 53% NI – Grammar - - - 33% 67% NI – Secondary 10% - - - 90%

RSC teacher conferences

Scotland 14% 21% 50% 7% 7%

England and Wales 7% 27% 33% 20% 13% NI – Grammar - 15% 23% 23% 39% NI – Secondary - 10% 50% 10% 30%

Other teacher meetings/ conferences

Scotland 23% 15% 46% - 15%

England and Wales 6% - 38% 13% 44% NI – Grammar - 14% 43% 21% 21% NI – Secondary - - - 22% 78%

RSC printed catalogue

Scotland - 33% 13% 7% 47%

England and Wales - - - 7% 93% NI – Grammar - - - 31% 69% NI – Secondary - - - - 100%

RSC telephone sales line

Scotland - - 8% 8% 85%


Colleagues

Scotland 14% 14% 29% 14% 29%


Dept resource library

Scotland 25% 17% - 8% 50%

The teachers’ responses suggest that RSC products are more likely to be available for use in departmental resource libraries in schools in England and Wales and Scotland. Teachers tend generally not to rely on phone or catalogues as a means of finding out about RSC products. Table 3.14 indicates there is some reliance on colleagues and departmental resources for finding out about RSC products particularly for teachers in England, Wales and Scotland. Hence in schools where RSC products have a lower profile there is less likelihood of materials being disseminated in these ways. Table 3.15 shows the main ways the teachers responding obtain RSC products. The method relied on by the majority of teachers is the direct mailing to schools. It is

29

notable that whilst about 80% of the teachers report obtaining products in this way this compares with only 50% of the teachers in NI secondary schools. This adds further support to the earlier suggestion that there may be school-based barriers to dissemination. ‘Sometimes we do not receive new material. Don't know why.’ (Scotland). The findings confirm that about a third of teachers in England, Wales, and Scottish schools are using the website frequently but this reduces to about a fifth of teachers in NI schools. Overall nearly two-fifths of all the teachers reported only occasional using the websites. RSC events tend not to function as routes for obtaining products. Table 3.15 How teachers get hold of RSC materials

Very frequently Frequent Occasional Rare Never

England and Wales 57% 18% 28% - 6% NI – Grammar 47% 35% 18% - - NI – Secondary 25% 25% 42% - 8%

Materials sent direct to school

Scotland 40% 40% 7% - 13%

England and Wales 6% 25% 38% 19% 13% NI – Grammar 6% 13% 38% 31% 13% NI – Secondary - 22% 44% - 33%

RSC websites

Scotland 6% 31% 38% 6% 19%

England and Wales 14% 7% - 29% 50% NI – Grammar 7% 21% 29% 14% 29% NI – Secondary - 20% 30% 20% 30%

Dept resource library

Scotland 15% 31% - 15% 39%

England and Wales 8% - 39% 15% 39% NI – Grammar - 13% 27% 13% 47% NI – Secondary - 11% 33% 22% 33%

Colleagues

Scotland 14% 29% 14% 7% 36%

England and Wales - 13% 44% 13% 31% NI – Grammar 7% 7% 29% 14% 43% NI – Secondary - 11% - 22% 67%

RSC mail order

Scotland 13% 13% 31% 6% 38%

England and Wales - 18% 46% 18% 18% NI – Grammar - - 43% 14% 43% NI – Secondary 13% - 25% 38% 25%

Other teacher meetings/ conferences

Scotland 8% 15% 39% 15% 23%

England and Wales - 17% - 8% 75% NI – Grammar - 7% 7% 20% 67% NI – Secondary - - - 13% 88%

RSC telephone sales line

Scotland 7% 7% 7% 14% 64%

England and Wales 8% 8% 18% 8% 58% NI – Grammar - - 21% 7% 71% NI – Secondary - 13% - 13% 75%

RSC e-mail list

Scotland - - 31% 8% 62%

3.2.2 Use of RSC products Teachers were asked to indicate the frequency of use of different product types in their departments Table 3.16 lists these and it shows that posters are very popular followed by paper-based products. Multimedia materials are increasingly used in similar proportions to web-based resources but neither of these by the NI secondary schools. The level of reported use of events for pupils is low however in the interviews with teachers reported later there were indications that this would increase with specialist science schools becoming involved in helping organise local events for schools.

30

Table 3.16 Departmental use of RSC materials and services

Very frequently Frequent Occasional Rare Never

England and Wales 35% 15% 20% 25% 5% NI – Grammar 63% 31% 6% - - NI – Secondary 27% 47% 13% 7% 7%

RSC posters, wall charts, etc.

Scotland 44% 28% 22% - 6%

England and Wales - 25% 55% 20% - NI – Grammar 6% 41% 53% - - NI – Secondary - 7% 60% 20% 13%

RSC books, booklets

Scotland 22% 22% 39% 11% 6%

England and Wales 5% 35% 30% 20% 10% NI – Grammar 12% 24% 59% 6% - NI – Secondary - - 36% 36% 29%

RSC CDROMs, multimedia

Scotland 17% 22% 33% 22% 6%

England and Wales - 25% 30% 35% 10% NI – Grammar 6% 25% 50% 13% 6% NI – Secondary - - 36% 29% 36%

RSC www sites

Scotland 6% 39% 44% 6% 6%

England and Wales - 5% 20% 20% 55% NI – Grammar - - 13% 38% 50% NI – Secondary - - - 7% 93%

RSC INSET

Scotland 7% 7% 44% 17% 28%

England and Wales 5% - 5% 35% 55% NI – Grammar 6% 6% 31% 38% 19% NI – Secondary - - - 14% 86%

RSC events for pupils

Scotland 6% - 33% 28% 33%

Teachers gave a number of reasons for not using RSC products. Table 3.17 lists these and the number of teachers making the comment. These teachers were in the minority although a significant number of teachers from NI secondary schools stated that they had difficulty with finding out about products and gaining access. Some teachers not using events for pupils suggested that either the department or the school did not emphasise school visits or actively discouraged them.

‘Not keen on pupils being out of school.’

31

Table 3.17 Why departments don’t use RSC materials No. of teachers

England and Wales 2 NI – Grammar 2 NI – Secondary 5

We find it difficult to get information on what is available

Scotland 2

England and Wales - NI – Grammar - NI – Secondary 3

We have no access to RSC materials

Scotland -


The materials are not appropriate for the pupils we teach

Scotland 2


The materials are not appropriate for the subjects/curriculum we teach

Scotland 1

England and Wales - NI – Grammar - NI – Secondary 1

Departmental and school policy make it difficult to use RSC materials

Scotland 1


We choose to use other, non-RSC resources

Scotland 1

There were teachers in each group who reported that the products were not appropriate for pupils though more teachers in schools in England and Wales and NI secondary schools thought this. The reasons given related to curriculum relevance to, and accessibility for, students.

‘Do not follow CEA specification.’ (NI – Grammar)

‘Much is not relevant to already full SQA courses.’ (Scottish teacher)

‘More material needed with a Scottish dimension.’

‘Text-books from other sources better match Scottish S1–S4 curriculum.’ (Scottish teacher)

‘Any material I have seen has been too difficult.’ (NI – Secondary)

‘More useful to teachers than pupils.’ (NI – Secondary)

‘Materials selectively used, as many pupils low ability.’ (NI – Secondary)

‘Work not appropriate for low ability pupils.’ (NI – Secondary)

‘Most aimed at higher ability.’ (England)

Nearly a third of teachers in the England and Wales sample also reported using other resources. These teachers’ decisions to use other resources reflect their views about suitability of the materials for their particular students’ needs. This perception of curriculum suitability appeared to colour teachers’ views about the degree of fit between their schools decision-making and distribution processes and the RSC’s dissemination and distribution systems. About a third of the NI teachers in grammar schools chose to comment about the lack of access to RSC INSET events, commenting on the difficulty of getting to locations.

32

‘Funding limits use & attendance of course abroad.’

‘No meetings/courses in NI. No local branch active.’ From the school year 2004–05 events will be presented in NI which will meet these teachers’ needs and concerns. The later evaluation of the RSC INSET events indicates that they are very well attended and rated as excellent. The concern with access to INSET was also commented on by one of the Scottish teachers ‘Timings & locations are usually unhelpful.’ One of the volunteer group sample who was interviewed was directly involved in supporting RSC INSET but despite his involvement, was unaware of other INSET. He suspected that this was because he was not an RSC member: he did not know how much membership cost, but 'as a young professional, joining a number of organisations is financially difficult and that would probably be the reason that I don’t know about these things'. The success of the RSC INSET events found by this study (see section 4.4) suggests that wider access would benefit chemistry and science teachers and their students.

3.2.3 Distribution of RSC products We asked teachers to indicate how effectively the RSC’s dissemination and distribution systems fitted with those systems in their schools. It was clear that while some teachers had experienced no problems others had. The volunteer teacher referred to above described how when he arrived at his current school he found a number of resources 'gathering dust' in a cupboard, including Alchemy? and other RSC publications. In his view this was hard to avoid unless materials are addressed to specific individuals. This was view held by 20% of the England and Wales sample too. Several teachers who thought a problem with the distribution of RSC materials was with the school or college structure commented:

‘It goes through a hierarchy of people before reaching myself.’ (Scotland)

‘Not connected to Internet in ANY of science labs.’ (NI – Secondary)

‘Goes to head of department and rarely distributed beyond.’ (NI – Secondary)

‘Material gets "lost" and not made available to dept.’ (NI – Secondary)

‘Very big school. Difficult to pass round.’ (England) 35% of the teachers in England and Wales saw the problem of dissemination as located in the school or college and departments. They highlighted the need for clearing internal routing and the problem of delays, which affected departmental decision-making. Several teachers commented further about problems they thought arose within their departments:

‘Difficult to keep track of who has what.’ (Scotland)

‘I tend to hold on to too much stuff for my own use!’ (Scotland)

‘May be in dept but other members do not know.’ (NI – Grammar)

33

‘Staff do not visit the Dept Library frequently.’ (NI – Secondary)

‘Storage is limited.’ (Scotland)

‘Head of Department sometimes don’t pass on the resources.’ (England) Others saw the constraints as resting with the departmental policy.

‘Work schemes rigidly referred to. Allows little flexibility.’ (Wales)

‘Having the time to incorporate ideas in the scheme of work.’ (England) In spite of the problems raised teachers overwhelmingly rated very positively the RSC’s method of distribution (see Table 3.18). Table 3.18 Free mailing to each school

No. of teachers


A convenient and effective way of receiving the materials

Scotland 17

England and Wales 2 NI – Grammar - NI – Secondary 3

An inconvenient or problematic way of receiving the materials

Scotland 1

Many teachers commented further on the convenience and effectiveness of receiving the free mailing of a single copy of RSC resource to their school:

‘By arriving on our doorstep we're 'forced' to evaluate/use them.’

‘Easy to review.’

‘Good way to evaluate materials for general use.’

‘Have previewed and then bought some copies of materials that were very suitable.’

‘They are made available in Dept Resource Room.’

‘Makes us aware of all new material, even if not directly relevant to us.’

‘Very helpful.’

‘Quick, don't need to hunt for it.’

‘Such materials are more likely to be passed on.’

‘I am the only chemistry teacher so everything comes to me!’

‘It is the envy of other science colleagues.’

‘Excellent.’

‘Arrives in the pigeon-hole in staff room.’

‘Please carry on.’ Teachers were asked how the RSC system might be improved and several teachers saw the need was for them to make more frequent use of the website. Needless to say

34

many wanted multiple copies and more regular mailings of what was in the pipeline. Widening access to the websites would meet most of the teachers’ needs. The teachers indicated that it was general to have the RSC resources in a fixed collection in the department, a small number (20%) reported that they were dispersed to teachers of particular age groups. For a similar small number of teachers resources were distributed by personal choice. Teachers were asked if RSC products were distributed to all teachers or mainly those with chemistry as a specialism. It is clear that in most schools, even where double science is taught and often by one teacher, the policy is to distribute to chemistry specialists presumably with a view that they will review these. The feedback from teachers indicated that on the whole the RSC products were used mainly with KS4 and KS5 students.

3.2.4 Use of web-based resources for chemistry Teachers were also asked if they used other web-based resources to support chemistry teaching and Table 3.19 shows that they do. What is interesting is that three quarters of NI secondary teachers report using these resources compared with a fifth who reported using the RSC sites. The sites used are similar to those listed by the volunteer group though with different curricula and awarding bodies referred to. Teachers report their general practice is to use the first website that search engine returns therefore if the RSC is not towards the top of the list ‘you may not visit’. Table 3.19 Departments that use web-based resources to support chemistry teaching

Region Yes No

England and Wales 83% 17% NI – Grammar 73% 27% NI – Secondary 79% 21% Scotland 94% 6%

To further explore the issue teachers were asked about the type and frequency of access and use of web-based resources (see Table 3.20). In Table 3.20 similar proportions of teachers across schools have access to the web at school outside of lessons and at home and make use of it. The significant difference is in access during lesson time, which is common for about a third of teachers in England, Wales, Scotland and NI grammar schools and only occasional for teachers in NI secondary schools.

35

Table 3.20 Teachers’ frequency of access to web-based resources Very fre-

quently Fre-quently

Occa- sionally

Rarely Never

England and Wales - 35% 53% 12% - NI – Grammar 7% 29% 43% 14% 7% NI – Secondary - 9% 64% 18% 9%

Teachers’ access to the web during lessons

Scotland 6% 31% 31% 25% 6%

England and Wales 6% 47% 35% 6% 6% NI – Grammar 7% 50% 43% - - NI – Secondary 10% 40% 40% - 10%

Teachers’ access from school in non-contact time

Scotland 31% 44% 13% 13% -

England and Wales 6% 35% 35% 24% - NI – Grammar 14% 36% 36% - 14% NI – Secondary 27% 27% 36% - 9%

Teachers’ access from home

Scotland 38% 25% 13% 13% 13%

Table 3.21 Teachers’ frequency of access to RSC websites

Very frequently

Fre- quently

Occa- Sionally

Rarely Never

England and Wales - 7% 47% 40% 7% NI – Grammar - 7% 64% 21% 7% NI – Secondary - - 44% 44% 11%

Teachers’ access to the RSC websites

Scotland 13% 38% 31% 19% -

England and Wales 6% 11% 39% 28% 17% NI – Grammar - 7% 64% 21% 7% NI – Secondary - - 11% 67% 22%

Teachers’ access to the RSC websites to locate a specific known resource

Scotland 6% 19% 44% 31% -

England and Wales - 13% 19% 44% 25% NI – Grammar - 21% 43% 21% 14% NI – Secondary - 10% 40% 30% 20%

Teachers’ access to the RSC websites to look for potential new resources

Scotland 13% 25% 25% 38% -

In Table 3.21 teachers were asked about the rate of use of the RSC sites, as before this confirms a much higher use by teachers in Scotland. Use by teachers in England and Wales is relatively low compared with Scottish teachers and those in NI grammar schools. What is noteworthy is the relatively low level of teachers in England and Wales using the website to look for new resources compared with all other teachers. Teachers were also asked about the ease of navigation of the site. It can be seen (Table 3.22) that there is an expected relationship between frequency of use and views about ease of navigation for teachers in NI secondary schools. No doubt familiarity enhances teachers’ ability to use any website. The teachers’ response suggests that there are no particular problems with the RSC website beyond its undoubted complexity because of the richness and range of the resources. It is not problems of ease of use that explain why teachers in England and Wales are making less use of the site to find new resources than other teachers, however. This is perhaps more a reflection of teachers’ views of the possibility for change and innovation in the current centrally prescribed science curriculum and associated national assessment arrangements.

36

Table 3.22 Ease of navigation of RSC websites Region Very

easy 2 3 4 Very difficult

England and Wales 15% 46% 39% - - NI – Grammar 17% 42% 42% - - NI – Secondary - 33% 50% 17% - Scotland 44% 19% 31% 6% -

Ongoing developments to enable content management of the web-based resources will go a considerable way to extend teachers’ access to the website. However, as the discussion of web-based products in section 4 suggests, the site is already very well used. The more restricted degree of use reported in this section tends to reflect general issues to do with access and time related to barriers in the infrastructure at school level which policy initiatives are currently attempting to address. For example a third of teachers in the NI group reported a lack of access to a computer and to the Internet for school purposes. 20% of teachers in England reported problems of access for whole class use either because of location, numbers or network problems. A third of the NI secondary teachers felt they needed training as did 15% of the teachers in the England and Wales sample. The responses of the NI teachers suggest a more general lack of awareness of what is available on the RSC sites and this is compounded by other findings about access, time pressures and beliefs about the suitability of the materials. This was particularly the case for NI secondary teachers and some teachers in England and Wales.

3.2.5 Importance of particular features of resources for supporting chemistry Teachers were asked to rate how satisfactory they found current RSC resources in their curriculum focus, differentiation for age of pupils and ability of pupils, their views are reported in Table 3.23. Table 3.23 Teachers’ ratings of how satisfactory RSC resources are in relation to curriculum focus and differentiation

Very Not Satisfactory at all Focus Sample 1 2 3 4 5

England and Wales - 26% 53% 21% - NI – Grammar NI – Secondary - 25% 42% 33% -

Curriculum

Scotland 6% 29% 47% 12% 6%

England and Wales - 19% 44% 25% 13% NI – Grammar - 25% 75% - - NI – Secondary - 18% 55% 27% -

Differentiation by age

Scotland 13% 27% 40% 20% -

England and Wales 12% - 41% 29% 18% NI – Grammar - 17% 58% 25% - NI – Secondary - 18% 27% 55% -

Differentiation by ability

Scotland - 13% 40% 47% -

The figures for curriculum focus show marked similarities in views across the groups with most teachers indicating some need for more attention to how the RSC resources target different aspects of the specified curriculum. With respect to age differentiation teachers in Scotland and to a lesser extent NI grammar schools found this satisfactory but teachers in England and Wales and NI secondary schools wanted greater focus on

37

differentiation to target particular students. To follow up these views, teachers were asked to rate the importance of particular features of resources to inform the future planning of the RSC.

Format The majority of teachers across the groups in reporting their preferred format for resources gave CDROMs the highest importance rating. DVDs were rated highly but less so than CDROMs again indicating problems with access to the hardware. Ring binders were as popular as DVDs indicating the teachers’ views of the need for multiple formats to meet their different needs. Web resources were rated very highly by over 50% of teachers in England, Wales and NI grammar schools, less so (40%) by Scottish teachers. Only one teacher in a NI secondary school considered these resources were very important but 40% thought they were important. Teachers were still giving a high priority to text-based resources with over half in England, Wales and NI of the view that they were important.

Teaching focus of materials As expected from the findings in Table 3.23 the majority of teachers in the Northern Ireland groups thought it was very important that the RSC materials should focus on the Programmes of Study, particularly secondary school teachers. 60% of the England and Wales sample thought this was necessary too. The Scottish teachers were less concerned about this. NI grammar school teachers rated a focus on exam specifications equally highly, less so secondary school teachers and teachers in England and Wales though about 50% felt this to be very important. The majority of Scottish, English, Welsh and NI grammar school teachers thought a focus on curriculum enrichment and extension in teaching resources was very important or important. Teachers in NI secondary schools were less concerned about this as a need for their students, which reflects their views that existing products challenged their students. Cross curricular links were seen as important by half of the Scottish teachers and the NI secondary teachers compared with less than a third of NI grammar school teachers and teachers in England and Wales. This reflects the different curriculum emphases in the schools and the age ranges the teachers target in their use of RSC products.

Age range of students It has been noted that the teachers tend to perceive the RSC products as being appropriate for KS4 and 5 mainly and this is further compounded in NI secondary schools where teachers view the materials as challenging for their particular pupils. It has also been noted that currently teachers in the volunteer and cold call groups tend to use the web for finding student resources e.g. worksheets and/or enrichment /extension materials. It is therefore not surprising that teachers tend not to consider that the RSC resources should focus on years 7 and 8, however about half of the teachers in NI grammar schools and 40% of teachers in the England and Wales sample do value this focus. More teachers in Scotland (30%), England and Wales (50%) and NI secondary schools (60%) consider that resources for Y9 are important. The numbers of teachers that consider the resources should focus on older students rises from Y10 to Y12 with the majority of English, Welsh and Northern Irish teachers considering this to be very important. Scottish teachers seem less concerned presumably reflecting that they consider the current focus to be appropriate in this respect which is borne out by the data in table 3.23. The majority of teachers in

38

England and Wales and the NI grammar school teachers considered resources for Y13 students to be very important. Teachers were also asked to indicate the ability of students that they felt the RSC products should target. All NI secondary school teachers wanted a focus on mixed ability students whilst the majority of other teachers wanted this too, less so in NI grammar schools which is unsurprising. The majority of English, Welsh, Scottish and NI grammar teachers wanted materials for gifted students and this fell to about 40% for secondary teachers in NI. 40% of English, Welsh and Scottish teachers and two-thirds of teachers in NI secondary schools considered materials for special needs students and those with learning difficulties important. (This may well be a Teacher Fellow project in 2006). Teachers were asked how they would like to hear about RSC materials in the future and overwhelmingly they cited e-mail and post or through Education in Chemistry, which are the normal routes used by RSC. The current system is therefore appropriate for members and users of RSC materials. It is not feasible for non-members who are also non-users to expect the RSC to be able to contact them via e-mail. Teacher feedback did indicate that the web was not the best method for extending access. It appeared that most teachers wanted the existing methods to be maintained whilst they began to adapt to using the websites.

‘By post and on the RSC website.’

‘ASE journals and the website.’

‘By catalogue and mailed books but I will have to start using the INTERNET now I know it is there.’

3.2.6 Feedback from interviews We interviewed by telephone five teachers in the cold call sample from one LEA in England. The teachers all taught in schools that were close geographically, only two of the schools had Heads of Chemistry and taught separate sciences the others taught mainly double award science. One teacher from an independent school received all of the RSC materials directly in that everything was opened and directed to her internally. She was aware of and used most of the products evaluated referring to Alchemy? as ‘wonderful’, though she did not refer to the Misconceptions texts and had not heard of the Ideas and Evidence texts. She did use the web but ‘not very often’. In another state secondary comprehensive school, which offered separate sciences the Head of Chemistry used the RSC website. The RSC products were in the Department resource library but little general use was made of them although it was clear that products like the Ideas and Evidence series were referenced in the schemes of work across a key stage. The explanation the teacher gave about why wider use was not made of the RSC resources confirms earlier speculations about why some teachers in England and Wales were not tending to use the website to search for new resources (see Table 3.21).

‘Time pressure is the main concern, there’s quite a prescriptive amount of work you’ve got to get through now, so any new materials that we try and integrate would have to be relevant to that.’

The teacher did plan to attend the RSC ‘Chemistry at Work’ day with the Y10 groups.

39

In another school the teacher was aware and made use of the RSC products but constraints in his current school meant he could not continue with the use of some. For example he no longer used Alchemy? as the ‘system here doesn’t easily allow us to run CDROMs.’ He did use the website but said not as a way of keeping up to date: ‘I’d love to, but no, that’s just the time thing’. He reported that the RSC resources were held in one of the prep rooms ‘near where the majority of chemistry teaching happens.’ He described his use of products Classic Chemistry Experiments and Demonstrations as ‘fairly regular’ though his use of the Misconceptions texts was more infrequent and largely as reference texts. He attended events for pupils, which he found very useful. He reported that he had not seen the Ideas and Evidence series. He knew of JESEI but had yet to make use of it. At his last school the RSC resources ‘were written into the schemes of work’ at his present school he described their use as more individual though felt this could change when they revamped the schemes of work. He felt that there was little space to incorporate new materials into KS3 and that it was ‘very difficult’ to add to the KS4 curriculum whereas it was ‘relatively straightforward at ‘A’ level.’ The Head of Science at another school, although not an RSC member, looked at the websites regularly. He had always been aware of the RSC resources but now routinely relied on the website to keep up to date. He described the distribution system in the school as ‘a bit hit and miss’ because the secretaries open the mail and ‘see who they think they’re best suited for.’ It appeared that he made use of the Classic Chemistry texts but knew little about the more recent ones, which seemed inconsistent with his reported use of the website. For example he had not heard of the Misconceptions texts, Alchemy? or the Ideas and Evidence series. He argued the need for more materials on CDROMs that were more directly integrated with the science curriculum. He was very positive about the quality of RSC resources and described the ‘Chemistry at Work’ event as ‘brilliant’. Another Head of Science who was too busy to be interviewed also commented on the quality of the ‘Chemistry at Work’ event. His school is a specialist science school and he was organising the event at his school next year and encouraging other local schools to attend. The last teacher we interviewed used some of the RSC resources but like some other teachers appeared unaware of the more recent products. He was not accustomed to using the Internet and found it quite time consuming. He described the search process as one of ‘pot luck’, which gives further support for the need for more direct routes of access on the websites that target pupils and topics as well as curriculum and exam specifications. He felt he would benefit from joining the RSC but had found the form very difficult and had given up. He was the only teacher to offer this view of the online application process, which may indicate his experience of using web resources. The interviews backed up the questionnaire data in that there was routine and regular use of RSC resources and this was widened if the resource was integrated into a school’s schemes of work. Knowledge of resources varied between teachers. Some teachers were not finding out about relevant resources often because of lack of time coupled with lack of familiarity with the RSC websites. Other barriers included inadequate distribution systems in school which were compounded by the rate of turnover of staff in science and chemistry departments and to an extent teachers’

40

preconceptions about the relevance of RSC products. Often these preconceptions were based on an outdated view of what the RSC had to offer which suggests a need to increase these teachers’ awareness of more recent publications. It was clear that the process of being questionnaired or interviewed had served to prompt many of the teachers to review and become excited about the potential of the RSC resources. For example one teacher interviewed provided an almost immediate review for his department of the five Ideas and Evidence books that he had previously not seen, indicating what was relevant, where it would fit, and for which pupils.

3.3 Issues Emerging re Dissemination Overall for the majority of teachers the dissemination of, and access to, RSC products is successful. There are clear directions from teachers about how to widen their access and use. These directions have to a large extent been anticipated by the RSC in their ongoing developments. There is an issue though of which teachers and which students in which schools that access should be available to. The evaluation has highlighted groups of teachers who through a combination of factors have restricted access to RSC products. Some factors relate to school systems and resources that the RSC cannot address. However, we recommend that serious consideration is given to widening access for these groups who would benefit in particular from the excellent support available in the form of RSC products for their professional development and their students’ learning. The majority of teachers from both samples report that access to RSC products is unproblematic. Significantly more teachers in NI secondary schools than other teachers experience difficulties in getting hold of products. More teachers in England and Wales than teachers in NI grammar schools and Scottish schools do not consider getting hold of RSC products to be easy. These findings may reflect the lower proportion of RSC non-members in these two samples. The RSC publication Education in Chemistry, sent free to all schools is the key means by which teachers find out about RSC products. The ASE publications are significant in this respect particularly for teachers in England and Wales. The RSC websites are also frequently used by the majority of teachers with the exception of teachers in NI secondary schools. There is an expected link between teachers’ access to the main RSC information sites and their reported difficulties with access to products. Teachers who report no difficulties with access not only frequently access main information sites they also have organised access to RSC products in their departments and tend to have colleagues who help keep them informed. Teachers felt privileged to receive the RSC products and publications directly and without charge. It singled them out from their colleagues and they valued this highly. Across both samples teachers report that the main route for getting hold of products is the direct mailing or from departmental libraries. A third of teachers reported that problems in dissemination occurred within the school, college or department. These problems could not be solved by the RSC even though teachers tended to assume that more personalised mailings were possible. The evaluation found that turnover of staff was a significant issue and therefore the present mailing system was appropriate. NI secondary school teachers appear to be less aware of the direct mailing of products or are not the member of staff who receives them. This may reflect the departmental

41

structures in the schools and the curriculum focus, which is double science. The RSC websites are frequently used by about a third of teachers to get products and this reduces to about a fifth of teachers in NI schools. Overall nearly two fifths of teachers reported only occasional use of the websites. The volunteer sample teachers, who reported using web-based resources for chemistry, cited the RSC sites slightly more often than other sites. Their use of the web resources related to the products largely reflected the nature of the product with over half of the multimedia (Alchemy?) product users making use and about a quarter of the teachers using Classic Chemistry Experiments. The teachers’ searches were mainly for student related materials which they found very valuable. High proportions of the cold call teacher sample used web-based resources for teaching chemistry. About a third of teachers reported frequent access to web-based resources during lessons though this figure rose to around two-fifths to a half accessing resources from home. A notable exception to access was noted for teachers in NI secondary schools with the majority reporting only occasional or rare access during lesson time. Such barriers reflect resourcing in schools and are therefore not something that the RSC can address. Barriers to access beyond access to equipment were explored. Around two thirds of teachers considered the RSC sites to be easy to navigate. This figure falls to a third for teachers in NI secondary schools. A third of NI secondary school teachers considered they needed training support to improve their access. There is an expected link therefore between frequency of use and familiarity with the site, which can result in a barrier to access for some teachers. Another barrier reported by many teachers was time constraint and this compounds the problem for teachers who are less familiar with the sites as they need more time to first find out about the sites and then to conduct searches. There are teachers therefore who would benefit from wider access to RSC products that are currently not aware of what is available. Other barriers to access that teachers reported were the absence of more direct cross-referencing to exam specifications and to curriculum specifications. This was a fairly general view of all the teachers. Some teachers who tend not to use the products consider that there is not enough opportunity to innovate in a tightly prescribed curriculum and examination system. Most teachers would therefore be more encouraged to use the sites if they could (i) see how the products and their particular contents mapped on to the specifications that they have to work to and (ii) were able to narrow their searches to their particular needs. This is already being addressed by the RSC in their development of a content management system for the websites and should have a major impact on extending dissemination and access. Teachers in England and Wales and NI secondary schools also wanted more guidance about age range of use and difficulty level. These teachers also wanted more products to be targeted at mixed ability students and key stage 3 students. Again both those needs can be addressed. However, this is not just a matter of guidance it requires some cultural shift in perceptions about the RSC products, their nature and aims. Overall teachers tend to see the products as suited to older (KS4 and 5) and more able students. This perception is confirmed by some of the guidance in the products themselves. This affects teachers’ engagement with the RSC products more generally as those in schools with selected intakes either through parental choice, catchment

42

area or the school system and with a curriculum to meet such students’ perceived needs expect to find appropriate materials. The converse is the case for those teachers in schools that have a restricted ability intake at the lower end and who teach predominantly single and double award science. However, there is much in the products that were reviewed that would be of great benefit for these teachers and their students. In addition the INSET events observed would have been of benefit to a very wide range of science teachers and their students. This barrier is one that can be addressed by the RSC and we would recommend that this should be considered. Cultural shifts are not easy to manage but with the extension of RSC events to Northern Ireland and the establishment in England of regional science centres there is an important opportunity to raise and extend the profile of the RSC and its impact. In making this recommendation we are considering the needs of the teachers and their students and it must be noted that there would be consequences in extending access for the RSC and its management. What is achieved and maintained with current staffing provision is exceptional in our view. In conclusion teachers wanted the current systems of dissemination to schools and colleges to be maintained. They wanted wider access to existing products through structured guidance about how to match their needs to RSC products. When asked about their preferred formats the majority of teachers wanted multiple formats. CDROMs were particularly favoured with DVDs second to this. Web resources were rated highly by about half of the teachers. The problems at the moment with equipment across schools indicates that the current policy of the RSC to develop the management of the websites and to adopt an intermediate media position with regard to products is the right one.

4.0 Product Evaluation The selection of products for evaluation was based in part on sales figures. The products therefore are popular which is an endorsement in itself. However, the evaluation sought to establish how well the products were valued in their use and impact. Section 4.1 provides some general findings and this is followed by more detailed product by product reports.

4.1 Some general findings across products The feedback on specific products came from the volunteer group only. Initially teachers were asked to complete a user survey to indicate whether they had access to the selected products for evaluation and what was the status of their use of products i.e. past, current, planned or non-user. The selection of the sample to evaluate each product took this into account and ensured that teachers from each category of user were involved. All of the products selected except the JESEI website have been mailed to all schools and colleges. The publication dates are given in Table 4.1. The information from the survey provides some initial indication of teachers’ awareness of the availability of products (see Table 4.1 below).

43

Table 4.1 Proportion of total volunteer sample access and use across products

Reported access and use by product (N=73)

Publication date Access Current user Past Planned Non-use

Chemical Misconceptions Vol. I 2002 42% 26% 15% 14% 45% Chemical Misconceptions Vol. II 2002 41% 26% 12% 19% 43% Classic Chemistry Experiments 2000 83% 73% 12% 1% 14% Classic Chemistry Demonstrations Ideas and evidence:

1995 83% 75% 12% - 12%

Chemists in a Social and Historical Context 2001 28% 10% 10% 12% 68% Climate Change 2001 25% 8% 14% 8% 70% Green Chemistry 2001 28% 11% 14% 11% 64% Health, Safety & Risk 2001 28% 15% 6% 10% 70% The Nature of Science 2001 20% 6% 10% 10% 75% Alchemy? 2002 52% 36% 8% 8% 48% JESEI 20% 6% 3% 6% 86%

The publication dates are significant when considering access but even then it is clear that a large proportion of the sample of which about 40% are RSC members state that they do not have access to these products. As noted earlier the resources may well be in schools but not referred to by the teachers or teachers may not be aware of their location. Eight products were selected for evaluation, five of which were paper-based. Ideas and Evidence, which we refer to as a paper-based product, is a collection of five books. In the data collection a five point rating scale was used but this is reduced for reporting as the overwhelming majority of teachers used the positive end of the scale. Table 4.2 Teachers’ ratings of the products

Product Excellent and good Satis- factory

Poor

Chemical Misconceptions Vol. I for teachers 81% 19% - Chemical Misconceptions Vol. II for pupils 83% 17% Classic Chemistry Experiments 94% 6% - Ideas and Evidence booklets 82% 11% 7% Alchemy? 87% 13% - JESEI web resources 100% - - INSET 79% 14% 7%

Teachers rate all of the products very highly with a very small number rating products as satisfactory only. The only teachers in the overall sample that considered a product to be less than satisfactory were two teachers of the thirty attending the INSET Improving Teaching and Learning in Chemistry using ICT. The more detailed analysis of feedback for this particular product indicates that these teachers had more basic needs with respect to ICT use than was planned for or feasible to deal with in the plans for the day. The feedback from teachers evaluating the multimedia product Alchemy? also suggests that this is a factor for some teachers that is further exacerbated by their lack of access to hardware and equipment to develop and apply their skills. Teachers were asked to what extent they found the products they were reviewing clear and accessible. This question was not included in the INSET evaluation questionnaires as the focus here was on the usefulness of different activities. The five point rating scale was again reduced as teachers used the positive end of the scale only. Across the products the ratings on clarity and accessibility of the RSC products

44

was exceptionally high with the majority of teachers rating products as very clear or clear. This is a significant comment about the teaching quality of the products given the complexity of some of the content and issues considered. Table 4.3 Teachers’ ratings of the clarity and accessibility of the products

Product Clear Neither clear or unclear Unclear

Chemical Misconceptions Vol. I 63% 19% 19% Chemical Misconceptions Vol. II 100% - - Classic Chemistry Experiments 94% 6% - Classic Chemistry Demonstrations 86% 10% 5% Alchemy? 93% 7% - JESEI web resources 100% - -

Chemical Misconceptions Vol. I had the lowest ratings for clarity and access. This reflects the nature of the text, which is substantial and theoretical. It requires teachers to follow ideas developed through the text and is not a book that you can dip into easily until you are familiar with the theoretical ground. The theoretical ground covers learning issues generally as well as examining how they apply to the construction of key chemical concepts so it is not possible to make this an ‘easy’ read. The feedback that over two-thirds of teachers had no problems with access is actually a very positive finding. The sister companion text was highly rated and this is a teaching text that is very well structured for teachers to select from to an extent whatever their grasp of the ideas in Volume I. However, how well they implement activities in Volume II depends crucially on their understanding of Volume I. Across products teachers were asked to rate the extent to which the products met their needs for support for teaching and learning science and chemistry. For Chemical Misconceptions this was restricted to Volume II designed specifically for teaching purposes. The ratings for the paper-based products and the INSET are very high. There is a small decline in ratings for the multimedia product, Alchemy?. Table 4.4 Teachers’ ratings of how well needs were met

Product Needs met Neither Needs not met

Chemical Misconceptions Volume II 91% 9% - Classic Chemistry Experiments 83% 17% - Classic Chemistry Demonstrations 86% 10% 5% Alchemy? 73% 13% 13% JESEI web resources 67% 17% 17% INSET 93% 7% -

The detailed reporting on Alchemy? (see section 4.3) indicates that some teachers experience technical problems either with the resource itself or with the technical demands of the resource, which might influence how well they consider their needs to be met. In relation to JESEI the web-based product (see section 4.5) the constraints of time and access to web-based resources have been discussed earlier. It is also the case that a large number of teachers search the index and glossary but do not follow this up further suggesting that what they consider they need is not available which probably reflects teachers’ perceptions of their specific curriculum needs. Once accessed both Alchemy? and the JESEI materials get very high ratings for ease of use (see Table 4.5).

45

A sub-set of the volunteer sample was asked to rate overall how easy it was for them to use the products to support their teaching. This question was not appropriate for the INSET sample. In the case of the Misconceptions volumes these are complex and wide-ranging texts and so questions about use were more detailed and are reported on later (section 4.2). Again the ratings are very high particularly so for the web-based and multimedia products. Table 4.5 Teachers’ ratings of ease of use

Product Easy to use Neither Difficult

Classic Chemistry Experiments 78% 22% - Classic Chemistry Demonstrations 72% 19% 10% Alchemy? 93% 7% - JESEI web resources 100% - -

Nearly four fifths of teachers consider the Classic Chemistry Experiments to be an easy text to use with slightly less finding this to be the case with Classical Chemistry Demonstrations. The content in both of these texts covers a range of pupils and a very wide range of the curriculum topics, there is therefore more searching for teachers to do. The evidence from the dissemination (see Tables 3.9 and 3.10) showed that far higher numbers of teachers wanted cross-referencing to exam specifications and national curriculum (between 50%–70%) for these particular texts. The introduction of content management on the RSC website, currently under development, will therefore be very helpful and should increase the ease of use and access to these products.

4.1.1 Issues emerging Overall feedback indicates a very impressive confirmation by teachers that the RSC products are of very high quality, clear and accessible, easy to use and meet needs well. Where problems emerge that limit ease of use these relate to the wide-ranging nature of products and indicate that teachers need more help with searching some products to find suitable materials. The developments already planned to support content management of resources for teachers on the RSC websites will go a long way to meet this need. The technical problems that some teachers refer to are small but do affect their use of multimedia materials and these will be heightened for teachers who feel they lack skills in the use of ICT and for those teachers who have limited access to appropriate hardware. Nevertheless there is a clear indication from the dissemination feedback that CDROM format is considered very valuable and useful by teachers and this is confirmed by the more detailed evaluation of the multimedia product in section 4.3. Teachers that access and use web-based resources give them very high ratings. It appears that these teachers want more of these resources. The dissemination feedback suggests this should be extension and enrichment material and more student material i.e. worksheets etc. Finding web-based resources is problematic for some teachers as ease of access increases with familiarity, which requires some persistence and awareness of what is available. The constraints of time on teachers and lack of access

46

to the Internet in some schools will continue to impact on the extent to which teachers use any website. However this way of accessing materials will continue to increase.

4.2 Paper-based products This section includes the more detailed evaluation of the paper–based products.

4.2.1 Chemical Misconceptions Volumes I and II Two related products Chemical Misconceptions Volumes I and II were selected for the most detailed consideration. The reasons for this choice were that the texts were current, published in 2002, and importantly feature in the national Key Stage 3 strategy for science for pupils aged 11–14 in England as recommended texts. The texts also contain materials appropriate for students in KS4 and 5. Both of the volumes are for teachers and provide a professional development resource for teachers as well as activities for students. The second volume contains the student activities in the form of photocopiable worksheets which are also on the web. The theoretical issues about learning and their consequences for teaching discussed in Volume I have relevance across age ranges though the examples suggest that the targeted audience is teachers in secondary schools and colleges. This is further supported by the activities in the companion volume. Both volumes are based on a very particular view of learning called constructivism, which challenges traditional views of how students learn. Key to the view is the notion that students do not receive knowledge, they construct knowledge. The sense and meaning that students develop is dependent not only on what is made available in the classroom but also crucially on what students bring to each learning situation. Students therefore come to chemistry activities with concepts and ideas about phenomena based on their prior experiences within and without school. As traditional pedagogy has not considered students’ prior knowledge these prior ideas are often unchallenged and therefore influence the new knowledge that students’ construct. In this way strongly held views that conflict with scientifically accepted views can develop and limit students’ future learning. The two volumes focus on explaining and exemplifying the nature of students’ misconceptions (Volume I) and providing ways of establishing these and moving students’ ideas towards more scientifically appropriate ones (Volume II). There are ten chapters in Volume 1, some focus on teaching and learning issues and methods i.e. chapters 1, 2, 3, 4 and 5 and others deal with broad chemical concepts e.g. chemical axioms, chemical structure. The chapters are seen as the theoretical background to the student activities but this is assumed to be understood rather than made explicit as advice to teachers. The section ‘How to use this resource’ lists the chapters and cross-references the student activities and the website address for the worksheets in the activities. The same cross-referencing is given in Volume II. Each chapter shares a common structure, the opening section sets out the theoretical constructivist perspective on the topic under consideration whether learning focussed or chemistry focussed. This is followed by general examples drawing on research evidence for a number of chemical concepts that illustrate the topic and detailed discussion and teaching techniques to address the issues. Most chapters include a description of a different teaching technique. As noted earlier this is a scholarly book that is written in an accessible style for any teacher prepared to ‘work at it’. It does

47

not offer a ‘quick fix’ to problems with learning that a teacher may have encountered. To be used effectively it would require a thorough reading and trying out of some of the activities in Volume II. For these reasons we report the finding for both Volumes together where possible. Chemical Misconceptions Volume II provides classroom resources in three parts. The first provides guidance notes for teachers, followed by activities or ‘probes’ to be used by students and then, in most cases, answers to questions posed. The answers are generally intended for use by the teacher and may provide guidance for class discussion. The student activities ‘probes’ are grouped with group A identified for use with students aged 11–14, Group B for students aged 14–16 or for post 16 courses, Group C for post 16 only while Group D is considered to be applicable across the age ranges. The teacher guidance is provided in a standard form that includes target level, chemistry topics covered, rationale for the learning difficulty to be addressed, instructions about how to use the materials, resources needed, feedback for students and answers. No indication is given about the amount of time needed. The student activities do not have a standard form. In addition to including material for students’ learning about chemical concepts they model for teachers a variety of ways in which probes can be constructed and used and serve a professional development role in this regard.

Feedback from the Questionnaires The questionnaires were sent to 29 volunteers and there was a 72% return. The overall feedback from teachers was that the products were highly rated for quality and meeting needs. Volume II was very highly rated for clarity and access but a third of teachers had difficulties accessing Volume I.

Teachers’ theoretical positions about students’ learning As both texts require teachers to understand a constructivist view of learning and knowledge the questionnaires sought their initial understanding of this. As it was assumed that teachers would consider Volume I first these teachers were asked to indicate what their understanding of constructivism was before using the text and after using the text on a ten point scale where 1–2 indicated that the approach was both very familiar and used in teaching. The teachers’ initial positions and changed positions are shown in Table 4.6. The table also shows the position of the Volume II sample before using the text assuming that they had already engaged with Volume I to some extent Table 4.6 Teachers’ ratings of their awareness of constructivism

Awareness of the theory Volume I evaluators Before After

Volume II evaluators Before

No awareness 25% 0% 13% Heard of but not familiar with it 13% 6% Heard of it and interested to use it 44% 13% 33% Getting familiar with the approach and using it 25% 13% 20% Very familiar and use this approach in my teaching 6% 61% 27%

All of the teachers showed a significant development in their thinking about students’ learning and the number feeling equipped to use the approach in their teaching doubled to include over two-thirds of the teachers. The figures for the Volume 11 evaluators showed that they had more awareness of the theory and more were using

48

the approach than the Volume 1 sample before the evaluation. This suggests that they had developed some familiarity possibly through prior use of Volume 1 as the sample included current users. Teachers from both samples were asked if they combined their use of the two volumes and if so how they did this. Their responses are shown in Tables 4.7 and 4.8. Table 4.7 Teachers’ approaches to the texts

Approach to the texts % teachers (Vol. I sample)

I read Vol. I completely, and then used some of the Vol. II probes 19% I read Vol. I in stages and tried out the Vol. II probes as I worked through the book 31% I skimmed Vol. I and then used the Vol. II probes 50%

Table 4.8 Teachers’ approaches to the texts

Approach to the texts % teachers (Vol. II sample)

I read Vol. II first and used the probes, referring to Vol. I as I felt the need 62% I only read Vol. II and used the probes 23% I started to read Vol. I but found it too time consuming/not useful and did not use Vol. II 15%

All of the teachers evaluating Volume I report using the probes in Volume II which probably accounts for the significant shift in their understanding and teaching reported in table 4.6. A third of the teachers worked in a way that integrated the two volumes allowing a progressive development of their understanding. A high proportion (62%) of the Volume II evaluators reported using a similar integrated approach.

Why teachers chose Chemical Misconceptions Teachers were asked what prompted them to choose to use the Chemical Misconceptions texts. For both Volumes I and II teachers made their choice based on pupil needs and their own personal development, although one teacher said they chose it because of the Key Stage 3 Strategy:

‘An interest in alternative methods of teaching difficult concepts.’

‘fascinated by the research and found it a very useful tool.’

Volume 1

Teachers’ views of the effectiveness and accessibility of the text The teachers gave Volume I a high rating on quality but a lower rating on clarity and accessibility with about a third of teachers suggesting there were some difficulties with using the text. To explore this further teachers were asked to consider each chapter in turn to rate how well it met their aims, was accessible and useful. The findings by chapter are given in two tables. In Table 4.9 the chapters concerned generally with learning and teaching are listed. In Table 4.10 the chapters concerned with learning particular chemical concepts are included.

49

Table 4.9 Teachers’ ratings of the chapters concerned with student learning

Very Not at much all

1 2 3 4 5

Aim met for you 19% 44% 19% 13% 6% Accessibility of text 21% 43% 14% 14% 7%

Chapter 1 The nature of and significance of pupils’ ideas and alternative conceptions Usefulness of information 21% 43% 29% 7% -

Aim met for you 38% 46% 8% 8% - Accessibility of text 18% 55% 18% 9% -

Chapter 2 Pupils’ difficulties in learning chemical concepts Usefulness of information 25% 50% 17% 8% -

Aim met for you 15% 62% 15% 8% - Accessibility of text 15% 38% 15% 23% 8%

Chapter 3 Distinction between conceptual and cognitive knowledge; conceptual structure of chemistry Usefulness of information 31% 31% 23% 15% -

Aim met for you 31% 38% 23% - 8% Accessibility of text 33% 42% 8% 8% 8%

Chapter 4 Overcoming learning impediments

Usefulness of information 38% 31% 15% 8% 8%

Aim met for you 18% 73% 9% - - Accessibility of text 18% 55% 9% 9% 9%

Chapter 5 Scaffolding learning in chemistry

Usefulness of information 55% 27% 9% 9% -

Each of chapters 1–5 received positive ratings from over two-thirds of the teachers for needs met, accessibility and usefulness. The ratings for the chapters on pupils’ conceptual difficulties and scaffolding students’ learning received very high ratings although accessibility was an issue for some teachers in the chapter on scaffolding. The questionnaire asked specifically how clear the teachers found the explanation about the teaching method in this chapter and a quarter expressed a view that it lacked clarity. These may well be the teachers who began the evaluation with no knowledge of the theory and its associated practices, which would make the examples and explanations less transparent for them. Accessibility of the text was considered problematic by nearly half the teachers for chapter 3, which dealt with distinctions between knowledge types and the conceptual structure of chemistry. These are very demanding topics and so the feedback is to be expected. It is notable that many teachers, over two-thirds, still rated the chapter highly in terms of aims met and usefulness

50

Table 4.10 Teachers’ ratings of the chapters on chemistry concepts Very Not at much all

1 2 3 4 5

Aim met for you 15% 62% 15% 8% - Accessibility of text 23% 46% 23% - 8%

Chapter 6 Chemical axioms


Aim met for you 25% 42% 33% - - Accessibility of text 18% 64% 9% - 9%

Chapter 7 Chemical structure

Usefulness of information 33% 25% 42% - -

Aim met for you 33% 58% 8% - - Accessibility of text 25% 67% - - 8%

Chapter 8 Chemical bonding

Usefulness of information 80% 20% - - -

Aim met for you 46% 38% 8% 8% - Accessibility of text 39% 39% 8% 8% 8%

Chapter 9 Chemical reactions


Aim met for you 33% 42% 25% - - Accessibility of text 33% 25% 25% 8% 8%

Chapter 10 Constructing chemical conceptions

Usefulness of information 33% 33% 33% - -

The chapters on teaching and learning specific chemical concepts received equal or higher ratings than the general chapters. Those receiving very high ratings across the three issues included Chemical bonding, Chemical reactions and Constructing chemical conceptions.

Teachers’ use of the product with students Teachers were asked to indicate if they used two of the key teaching techniques discussed in Volume I and if they did with which students. The findings in Table 4.11 show that nearly half of the teachers had not used the similarity and differences technique comparing two stimuli to elicit students’ ideas. This might reflect the teachers’ views of the suitability of the technique for the older age groups that many of them choose to use the product with. The 50% of teachers that use the technique do use it across KS3 and 4 and not with KS5 students. Concept maps were used more frequently and by over 80% of teachers. They were used for both diagnosis and revision purposes. Table 4.11 Teachers’ use of teaching techniques

Frequently Not Used used

Frequency of use of techniques

1 2 3 4 5 Similarity and differences between two stimuli e.g. pictures for eliciting pupils’ ideas 14% 7% 29% 7% 43% Concept maps for diagnosis, teaching and revision 21% 29% 36% 14% -

With which students? KS3 KS4 KS3 &4

KS5 All

Similarities and differences 11% 33% 33% - 22% Concept maps 6% 31% 31% 12% 18%

Concept mapping is a technique that can be used by children, young people and adults. This makes it a more flexible technique for teachers and this is supported by the teachers’ use of the technique with students with over a third using it with KS3 students and a similar proportion using it with KS5 students.

51

Teachers’ reasons for not using the product The reasons teachers gave were that the approach had become incorporated into their teaching and so they no longer needed recourse to the text. One teacher had moved beyond this approach in his view and was incorporating more recent research into his teaching. For several teachers other texts or ‘more appropriate material’ had replaced the product. Teachers were asked what alternative resources they used. Their responses did not reveal alternative resources as such but referred to courses like Salters’ which incorporate a similar approach to learning and teaching, although not the research evidence and detailed probes as in the RSC products. One teacher used the web to tap into particular points of interest without becoming ‘overloaded’. Another teacher referred to using the School Science Review for articles on specific topics and on-line articles that deal with research evidence about students’ learning.

Volume II

Teachers’ views of the effectiveness of the text The sample evaluating Volume II were asked how useful they found the guidance for teachers about how to use the activities and with whom. Table 4.12 shows their ratings across the different elements of the guidance. Table 4.12 Teachers’ ratings of the usefulness of the teacher guidance

Very Not at all Elements of the Guidance 1 2 3 4 5

Target level for use 53% 40% 7% Rationale for use 44% 44% 13% Instructions for use 50% 31% 13% 6% Feedback for students 44% 31% 19% 6% Answers 50% 31% 13% 6% How useful did you find the structure of the guidance overall 38% 62%

The ratings across all the elements of the guidance were very high with 75%–100% of teachers judging them as very useful or useful. Teachers were asked to provide feedback about their use of individual student probes and how well they worked and with which students. The findings are reported in two tables. Table 4.13 gives the topics that are used by over 50% of the teachers and their ratings of their impact and age group used with. Table 4.14 gives the topics that are used by less than 50% of the teachers, their ratings of their impact, and age groups used with. Table 4.13 Teachers’ use of probes and ratings of impact

Well Not well

Age group used with Probes Use

1 and 2 3 4 and 5 KS3 KS4 KS5 3&4 4&5

B Ionic bonding 100% 100% 56% 33% 11% C Ionisation energy 100% 100% 100% A Changes in chemistry 86% 67% 33% 75% 25% A Revising acids 86% 86% 29% 14% 40% 50% 10% B Word equations 86% 100% 50% 13% 38% A Elements, compounds and mixtures 71% 88% 13% 57% 43% B Spot the bonding 71% 100% 60% 40% A Mass and dissolving 57% 100% 100% B Types of chemical reaction 57% 100% 33% 33% 33% B The melting temperature of carbon 57% 67% 33% 100% C An analogy for the atom 57% 80% 20% 17% 33% 17% 33%

52

Eleven probes were used by over 50% of the teachers, four from the 11–14 age group, five from the 14–16 age group and two from the post 16 group. Six probes were rated as very effective by all teachers these spanned the groups. The other probes were considered effective by two-thirds or more of the teachers. The probes in group A were targeted at 11–14 year old and were predominantly used at KS3 but also at KS4. Other probes were also used for the target audience with the exception of word equations that was used predominantly at KS3. Table 4.14 Teachers’ use of products and their ratings of impact

Well Not well

Age group used with Probes Use

1 and 2 3 4 and 5 KS3 KS4 KS5 3&4 4&5

B Definitions in chemistry 43% 25% 75% 20% 40% 40% B Revising the Periodic Table 43% 60% 40% 17% 50% 17% 17% B Iron – a metal 43% 67% 33% 75% 25% B Precipitation 43% 100% 50% 25% 25% C Stability and reactivity 43% 100% 100% C Acid strength 43% 100% 33% 33% 33% D Chemical comparisons 43% 50% 50% 25% 25% 25% 25% C Hydrogen fluoride 29% 100% 50% 50% C Interactions 29% 100% 67% 33% C Reaction mechanisms 29% 100% 100% D Learning impediment diary 29% 50% 50% 33% 33% 33% C Scaffolding explanations 29% 100% 20% 40% 40%

Six of the twelve less frequently used probes in Table 4.14 were considered by the teachers to be very effective. Probes that were considered to work less well included Definitions in Chemistry (14–16), Chemical Comparisons (across ages), Hydrogen Fluoride (14–16) and Learning Impediment diary (across ages). Table 4.15 Teachers’ approach to the probes

Approach to the probes % teachers

As an integrated part of teaching 100% In a sequence i.e. matching scheme of work/specification 67% To diagnose student problems 8% As revision for students 17% As both diagnosis and revision 100%

Extension work 8% Homework 8% Stand alone pupil use only Revision 17%

As expected teachers select activities from the product to integrate into their planned curriculum. Two-thirds do this in an organised way sequencing the probes in relation to schemes of work and specifications. All teachers responding to this question use the probes for the purposes of diagnosis of needs and revision to consolidate learning. A very small number use them as stand-alone student materials. The teachers evaluating the multimedia product tend to use this format to support independent student work with older students.

Teachers’ reasons for not using product The teachers who no longer used the product generally gave time constraints as the reason. Though one reported that his practice had developed beyond constructivist approaches and another reported that more appropriate texts were being used in the department.

53

Two volunteers who had classified themselves as ‘non-users’ of the Chemical Misconceptions texts said they had not had access to the texts previously. Both gave positive feedback about the format and organisation of the texts. The first impressions of these reviewers with regard to the appropriateness of the subject and the level for students were again both positive. One of the reviewers thought the material was compatible with exam specifications, however the other, a further education college teacher, did not. Both reviewers found the material compatible with their teaching and learning approach. Both reviewers definitely planned to use the two texts in the future. The further education college teacher indicated that he might use the majority of probes with his mixed KS5 groups. The other reviewer was also extremely positive about his planned use of the probes in the future and indicated that he would use at least part of each probe with appropriate groups of pupils in Key Stages 3, 4 and 5. It was clear that being involved in the evaluation of these products had made some teachers newly aware of them and their potential and had reminded past users of this and they also expressed their intention to begin to reuse the products.

Chemical Misconceptions Volume II: feedback from interviews and observations Five teachers agreed to be interviewed, all were based in mixed 11–18 comprehensive schools in central England, the south west, London and Wales.

Reasons for using the product One teacher had used both products in the past but was involved in a postgraduate course and had chosen for his project to examine students’ misconceptions. He also saw the author of the texts give a workshop at the ASE conference. He described how he was disseminating the product particularly through the school’s involvement in SCITT. He was mentoring a graduate trainee and he had supported her to use the probes too. He had yet to share the products with colleagues but was planning to do so:

‘I’ll probably ask my Head of Department if I can have a training session at a Departmental meeting. I was responsible for writing that scheme of work and I did try to put stuff [from the probes] in it. I don’t know how much notice they take of it, we’ve got a non specialist teaching GCSE chemistry. But I am quite fired up by it now.’

Another teacher had produced laminated copies of some of the probes so she could reuse them with student groups particularly KS3 students. She recalled one of her colleagues, another chemistry teacher had commented on ‘what a good resource it is’ but they had never discussed how they used it. Like other teachers one of the interviewees described how he used the product because of his ‘perception of teaching, you know, when you suddenly realise hey there are these misconceptions that the children can have and you suddenly realise a bit more about how the children are thinking rather than how we are thinking.’ This teacher worked in a small chemistry department and had also used the products with students rather than colleagues. He did feel that the use of the products could be extended. Like the first teacher he felt he had integrated the product ideas and activities into his teaching.

54

Another teacher explained how the products got to him via the Head of Chemistry. He particularly valued the way the RSC distributed resources.

‘One of the best things about the RSC is that as soon as stuff for teachers does come out we get sent it which is really good. And you know it’s free copies and so it kind of gives you the chance [to use new resources].’

He described how he went through the products and chose activities and put them into the scheme of work and this ensured that colleagues used them.

‘Rather than photocopy loads and give it as a resource which probably most people wouldn’t have ever looked at I actually put some of it in the scheme of work.’

One teacher interviewed worked in an FE college. Some feedback from the cold call questionnaire suggested that teachers felt that colleges were not necessarily mailed with RSC products. This teacher too was not aware of having access to the products. What he had done was access some of the probes on the website recently and used them with a KS5 group. This was just before ‘the books were sent to me. That’s why I was really excited to get them’. He explained how he came to be interested to use the products in his teaching:

‘It really did bring out the fact that there are these misconceptions, and it wasn’t just people who came in with GCSE Science who had a misconception, I’d actually got a student in the group who’d done AS Chemistry and someone who had already done full A level and they still hankered after the misconception.’

The teacher only had one colleague based on a different physical site and he had already sent him a photocopy of the content page commenting: ‘I’ll add that to the list of things to discuss next time I see him.’

Teachers’ professional development Teachers were asked if the texts had supported their own professional development and helped with their understanding of chemical concepts. They were all agreed that both aspects had been addressed well by the products.

‘Oh very much so. And as I say I have used them for the professional development of student teachers as well. It’s made me focus on and you know think seriously about [the concepts].’

‘Definitely...it really did enlighten me...it opened my eyes a little bit. Sort of get an insight into the psychology of things.’ (student teacher)

‘Yes, because they made me think. I’ve been teaching, this what the end of my third year so it’s not all that long since I did my PGCE and we talked about misconceptions and things, but it [the products] made me think more about misconceptions and ways of addressing them. They have [helped with understanding chemical concepts]. Particularly like [bonding] ionic and iron [metallic] and covalent bonding as an example, so for more challenging things. Because it’s a long time since I did my A level chemistry.’

‘In terms of ensuring that I was aware of where my students were in terms of the initial assessments, and also in terms of knowing where the blockages

55

were likely to be. I'm not thinking dumb student, doesn’t get it, but thinking actually we’re unpicking four or five years of poor learning here, and then turning round to the students and being able to say look, you’ve got that wrong not because you’ve been badly taught at all, but because you learned the wrong thing and you’ve been allowed to keep with it, and the longer you’ve had that misconception the more difficult it's going to be to shift.’

Context of use The teachers were asked to give some examples of how they had used the probes and their views of their effectiveness for them as teachers and for their students. The student teacher had used probes to elicit students’ prior ideas and to build on those ideas, using concept maps and mind maps etc. The probe he had used recently was on ‘acid strength’ because he found that that presented more misconceptions than the other topics. He used the probe as part of a scheme of work with a Y8 group of mixed ability students ‘so we were just listing their prior ideas and seeing what they’d done from the Year 7, building on from that work.’ He used the activities as they were presented and found that the students ‘appreciated that there was more to acids and alkalis than they originally thought, and obviously they used to associate acids with strong corrosive elements, you know, whereas you’ve got the weak acids as well. He considered that the lesson went very well:

‘When I actually took those ideas from the start and incorporated them into the lesson you know, you could just see from the expressions from a few of them that they didn’t realise the extent that acids and alkalis are used in society. So I think they took quite a lot away from the lesson from seeing what they did originally know.’

He described the students’ reactions to the approach:

‘I think it was nicer. I mean it was something different. I think in any subjects they’re usually just taught something, you know, then they’re not expected to actually think about what they already know about the topic. So I think it was a nice tool to use. And I think I’ll definitely use it again in the future.’

The FE teacher was trying out the products for the first time and had used the sodium probe and the hydrogen fluoride to try and find out what his group of KS5 students already knew. He felt the activities were effective in informing him but not in changing the students but this was because:

‘I wasn’t pushing it far enough. And in many ways I wasn’t prepared for them to have the misconception to the extent that they did. I mean you’ve got 18 students and 17 of them say sodium + will be more stable than sodium, and one said that sodium 7- would be more stable than sodium. I think that’s a bit of a shock there, especially when you’ve got two who are experienced already. It was the one who’d done the AS Chemistry who thought sodium 7- was more stable than sodium. Full A Level, yes. And got a grade C.’

He found that the students enjoyed the style of working and that ‘the style hit them in the face, and it was very easy for them to comprehend what they were looking for. The fact they didn’t get the right answer doesn’t seem to matter. No, I wouldn’t look to change the format.’

56

Two other teachers used the ionic bonding probe. One worked with a Y11 group and focused on the ionic lattice using a model of his own to help students understand that ‘an ionic structure is a giant structure and it’s a 3D structure.’ In his words through using the probe he had ‘changed the emphasis of the way I teach it.’ The other teacher valued the approach and thought the probes were effective but more so for able students than others. For able students he thought the approach provoked debate and allowed them to be challenged to take their learning to another level. One of the teachers interviewed also gave us permission to observe a lesson and talk with students. She worked in a small department with another chemistry colleague. She planned to use two probes Elements, compounds and mixtures and Chemical comparisons with a Y9 class beginning KS4 study. The teacher intended to use the probe on elements, compounds and mixtures ‘to get them to discuss and while they are doing that then I could go round and see what their conceptions or misconceptions were in that area. So it is a bit of feedback from them to me as I went round I could see what particularly they weren’t understanding, then it was a chance for them to discuss it with each other before they actually, you know, so they could learn from each other as well.’ The teacher adapted the pre-test and reduced it so that students only had to provide definitions of an element, a compound and a mixture. They did not get the chance to consider these in relation to particles in different materials. The pre-test was handed out and the students discussed the questions in groups and recorded their agreed answers. The group’s answers showed that students in three out of ten groups could not describe an element. For the majority the definition was in terms of a single atom. Many of the students struggled to describe a compound. Two groups described it as a mixture of atoms. Four groups extended this to two or more elements or different atoms where the atoms are joined/stuck together. The remaining three groups referred to elements joined or stuck together that cannot be easily separated. One group could not provide a definition of a mixture and another group referred to it as something easily separated. Three groups referred to compounds or elements mixed together. Two groups thought a mixture was two or more elements mixed together and easily separated. Two additional groups referred to different elements or groups of atoms that are not joined. The teacher walked around the class and asked questions to prompt thinking but did not intervene to state if group answers were right or wrong. This was then followed by a class discussion to which only some students contributed.

Element: Pupil: Element, metal, non metal, one type of atom Pupil: Single type of atom Pupil: Single atom or group of one type Teacher: So, in the periodic table – elements – one type of atom. Compound: Pupil: Mixture of 2 or more different elements joined together Pupil: Two or more atoms

57

Pupil: Two or more different elements joined together, but can’t be separated. Mixtures: Pupil: Two or more elements, but atoms not joined Teacher: Can only elements be mixtures? Pupil: Can be separated

The teacher’s reactions suggest that she was expecting students to agree with the definitions given that were validated by her acceptance of them. They were then to use these to reconsider their own understanding, what she saw as them learning from each other. Her response to the idea that mixtures are elements was typical in that she raised questions but did not give answers or guidance about how to think about things. The intention in the probe is to follow up the pre-test with a carefully structured exercise that allows students to reflect on key ideas in a structured way. That is the difference between pure substances and mixtures in terms of particles – atoms and molecules and how these are related to the definition and representation of a pure substance and a mixture. This is then followed by a consideration of elements and compounds and examples of their representation in particulate form. The teacher did not use this as she wanted to move on to the periodic table and so her main concern was with elements. However she did ask the students to complete the post test that follows the full probe. The students’ responses are shown in Table 4.16. Table 4.16 Students’ responses post probe test Elements, Compounds and Mixtures

Answers to questions Group Compound Element Mixture Compound Mixture Element

1 √ √ √ √ √ √ 2 √ √ √ √ √ √ 3 √ √ X √ √ √ 4 √ √ √ √ √ √ 5 √ √ √ √ √ √ 6 √ √ √ X √ √ 7 √ √ √ √ X √ 8 X X √ X X X 9 √ √ √ X √ √ 10 √ √ √ √ √ √

The table shows that the majority of students can make the distinctions. Two groups remain confused about a mixture and a compound and one group is unclear about all three distinctions and the students are not well placed to begin their study of the periodic table. Four students were interviewed and commented that they liked the worksheets and the chance to think about these things together. The students were given further diagrams of mixtures, compounds and elements and each were able to distinguish them correctly. Although the students were successful two of them still thought a mixture was two or more elements, which are not joined together. All four had a hazy notion of the difference between an atom and a molecule, which the probe activity was designed to address as a basic underlying concept. In one student’s words ‘you can’t see an atom you can see a molecule.’ It is expected that teachers will use the products to fit their needs and will use them in ways that reflect their teaching approaches and beliefs about learning. Consequently the teachers and their students always mediate the impact of the products. Overall whatever the views of teaching and learning that teachers held who evaluated the

58

Misconceptions products they were successfully changing teachers’ teaching and making a very significant contribution to professional development. They were also in the main, achieving this whilst enhancing students’ motivation and learning.

4.2.2 Classic Chemistry Experiments: one hundred tried and tested experiments; Classic Chemistry Demonstrations: one hundred tried and tested demonstrations. This section reports on the evaluation of the two products Classic Chemistry Experiments and Classic Chemistry Demonstrations both of these are popular texts and have been available for the longest period of all of the selected products. Classic Chemistry Experiments is a resource book of 100 ‘classic’ Chemistry Experiments presented in the form of guidance notes for teachers and worksheets for students. The notes and worksheets are also available in electronic form via the RSC web site. The author describes the purpose of the book as providing teachers with a resource to use to enable them to ‘communicate the excitement and wonder of the subject to their students so that they also will be captured by the subject and want to take it further’. The author points out in the introduction that while in the National Curricula in England, Wales and N. Ireland much emphasis has been given to investigative practical work ‘the importance of other laboratory activities has recently become somewhat neglected’. He identifies a number of ‘reasons’ why students should do class experiments: Applying knowledge and understanding to their experiences Developing basic skills of selecting and using equipment Learning various techniques Illustrating a concept or process A starting point for investigations, asking questions and making predictions Extending the scope of the curriculum through work in science clubs. Classic Chemistry Experiments is for teachers. It covers aspects of the Science curriculum for KS3 and 4. Throughout the book this curriculum is referred to as Chemistry although some experiments might also form part of a Physics or Biology curriculum and one has been observed in use in a Food Technology curriculum. In addition there are some experiments that would now be carried out in most primary schools such as ‘Separating salt and sand’, although primary schools would be unlikely to use the ‘classic equipment’ described here. It is not a course book that would be given out to students. The photocopiable worksheets, also available electronically, are intended for students, as the teacher deems appropriate for a particular section of a course. The author describes the book as being for ‘both the experienced teacher of chemistry and for those whose first subject is not chemistry’. The experiments are offered in a variety of formats that alter the demands placed on students to follow instructions or to make decisions themselves. Classic Chemistry Demonstrations is intended for teachers of chemistry of a range of levels of experience and disciplinary backgrounds. The intention is to furnish these teachers with ready access to tried and tested demonstrations presented in a standard, easily followed format. Each demonstration is categorised as to its entertainment value; its suitability for the pre- and post-16 curriculum; and which broad area of

59

chemistry it falls under, from the following: acid-base; electrochemistry; equilibria; inorganic; kinetics; organic; periodicity; polymers; quantitative; and thermodynamics. A table at the beginning (p. vii–ix) summarises the categorisation of the 100 demonstrations for ease of access. Also at the beginning a summary of chemical-related safety issues is presented (p. x–xiii).

Feedback from the Questionnaire Twenty nine volunteers were sent a copy of each of the Classic Chemistry texts and a questionnaire for each. The sample (86%) returning completed questionnaires included: 14 current users, 8 past users, 1 planned user, and 2 non-users (N=25).

How well learning objectives were met Teachers were asked for their views of the extent to which the Classic Chemistry products allowed their students to achieve various learning outcomes. In Table 4.17 the teachers’ ratings for Chemical Experiments are presented. Table 4.17 Teachers’ ratings of how well learning objectives are met (Chemical Experiments)

Very Not at all Learning outcomes 1 2 3 4 5

Develop skills in carrying out experiments 37% 58% 5% Ability to follow instructions 32% 58% 11% Use equipment and materials effectively and safely 32% 63% 5% Develop knowledge of techniques 26% 47% 16% Understand purpose for techniques and skills in using them 11% 22% 67% Ability to describe how to carry out particular experiments 16% 63% 21% Ability to make decisions about how to carry out an experiment 32% 47% 16% 5% Understanding of chemistry concepts 11% 47% 42%

The majority of teachers rated the products highly in terms of meeting the learning outcomes of developing skills in carrying out experiments, learning to use equipment and materials effectively and safely and being able to follow instructions. The product was also rated highly in terms of supporting students to understand how to carry out experiments and slightly less so for developing knowledge of practical techniques. The objectives judged to be met less well included understanding the purposes for particular practices and developing students’ decision-making about how to conduct experiments. These objectives can only be achieved in the way that the product activities are presented to the students and the degree of openness allowed for. The text provides teachers with different ways of presenting the same experiment, which allows different degrees of independence for students. Therefore these learning objectives are to a large extent determined by the teacher. As one teacher observed: ‘These experiments are mainly used either (i) to illustrate some concept in a fun way in which case approach B is used for clarity & speed or (ii) to teach in a more investigative/problem solving mode where approach E used.’ Teachers were also asked to rate how well the Chemical Demonstrations text met the learning objectives specified. Their ratings are shown in Table 4.18.

60

Table 4.18 Teachers’ ratings of how well learning objectives are met (Chemical demonstrations)

Very Not at all Learning outcomes 1 2 3 4 5

Develop motivation and interest in chemistry 58% 37% 5% Observe ‘real’ experiences otherwise not available 26% 63% 11% Develop observational skills 16% 58% 26% Experience and understand experimental procedures 5% 42% 53% Understand safety issues in chemistry 11% 53% 32% 5% Observe skilled practice 21% 74% 5% Develop understanding of chemistry concepts 11% 83% 6%

Four of the seven objectives were considered by teachers to be very well met by the product, in particular to motivate and interest students in chemistry. Observing skilled practice was the next most highly rated objective followed by observing ‘real’ experiences. The Experiments text was seen to be much more focused on developing students’ practical capabilities than the Chemical Demonstrations text and this is evident in the different ratings accorded to how well the two texts support the development of students’ understanding of chemical concepts and understanding of experimental procedures. The teachers saw the Demonstrations as a means of teaching conceptual understanding in particular, and commented on the way they ‘Link theory with practice ‘chemistry in action’ and reinforce understanding.’

Teachers’ use of Experiments and their rating of their effectiveness Teachers were asked which experiments they used, how effective they considered the experiments, and what age groups they used the experiments with. Table 4.19 shows the ratings for those experiments used by 40% and more of the teachers in order of use. Table 4.19 Teachers’ use of experiments and their effectiveness

Effective Not effective Age group used with Classic Chemistry Experiments Use

1 and 2 3 4 and 5 KS3 KS4 KS5 KS3+4 KS4+5

Separating a sand and salt mixture 93% 93% 7% 87% 7% 7% The pH scale 67% 100% 70% 30% Exothermic or endothermic? 67% 100% 20% 50% 30% ‘Smarties’ chromatography 67% 60% 30% 10% 90% 10% Displacement reactions between metals and their salts 67% 100% 30% 30% 10% 20% 10% Titration of sodium hydroxide with hydrochloric acid 60% 100% 38% 38% 13% 13% Causes of rusting 60% 78% 22% 44% 22% 22% 11% Change in mass when magnesium burns 60% 44% 56% 78% 11% 11% Cracking hydrocarbons 60% 89% 11% 44% 11% 44% Making a crystal garden 53% 75% 13% 13% 75% 13% 13% Making a pH indicator 53% 88% 13% 88% 13% The effect of temperature on reaction rate 53% 88% 13% 13% 63% 13% 13% The effect of concentration on reaction rate 53% 88% 13% 13% 38% 25% 13% 13% Polymer slime 53% 75% 25% 57% 43% Chromatography of leaves 47% 67% 33% 67% 17% 17% A compound from two elements 47% 57% 29% 14% 57% 14% 29% Reactions of calcium carbonate 47% 86% 14% 14% 86% Electricity from chemicals 40% 83% 17% 17% 33% 17% 17% 17% Energy values of food 40% 67% 33% 67% 33% Testing of pH of oxides 40% 100% 17% 83% Extracting metal with charcoal 40% 83% 17% 33% 17% 33% 17% Testing the hardness of water 40% 83% 17% 100% To find the formula of hydrated copper (II) sulphate 40% 100% 67% 17% 17%

61

Effective Not effective Age group used with

Comparing the heat energy produced by combustion of various alcohols 40% 100% 67% 33% Electrolysis of copper (II) sulphate solution 40% 83% 17% 100%

Experiments that were used by many teachers also tended to be rated highly in terms of their effectiveness. The most commonly used and highly rated were experiments for KS3 students. We noted earlier that one of the difficulties teachers experience with these two texts is accessing them because of their wide ranging content and it is clear from this table that the experiments are well used across KS3 and 4. Other experiments were used but by less than 40% of the teachers. The list below shows those experiments that were not used by any of the teachers responding to this question. Viscosity Rate of evaporation Accumulator Iron in breakfast cereal Reaction of a Group 7 element (iodine with zinc) Sublimation of air freshener Reaction between carbon dioxide and water Particles in motion? The oxidation of hydrogen The volume of 1 mole of hydrogen gas The decomposition of magnesium silicide Colorimetric determination of copper ore Rubber band Teachers were also asked which demonstrations they used, how effective these were and what age groups they used the demonstrations with. Table 4.20 shows the ratings for the demonstrations used by 40% or more of teachers who responded to this question in order of use. Table 4.20 Teachers’ use of demonstrations and their effectiveness

Effective Not effective Age group used with Classic Chemistry Demonstrations Use

1 and 2 3 4 and 5 KS3 KS4 KS5 KS3+4 KS4+5

Making nylon – the ‘nylon rope trick’ 100% 86% 7% 7% 29% 36% 36% Diffusion of gases – ammonia and hydrogen chloride

87% 85% 15% 42% 8% 50%

The ammonium dichromate ‘volcano’ 80% 100% 55% 18% 18% 9% The thermit reaction 80% 91% 9% 8% 42% 8% 42% The non-burning £5 note 67% 70% 20% 10% 67% 22% 11% The combustion of iron wool 60% 67% 11% 22% 44% 11% 44% Catalysts for the decomposition of hydrogen peroxide

60% 100% 13% 88%

Identifying the products of combustion 60% 100% 22% 44% 33% The reaction between potassium permanganate and glycerol

53% 100% 33% 17% 17% 17% 17%

Flame colours 53% 100% 17% 17% 17% 50% Sulphuric acid as a dehydrating agent 53% 88% 13% 14% 29% 14% 14% 29% The fountain experiment 53% 88% 13% 88% 13% Cracking a hydrocarbon/dehydrating ethanol 53% 75% 25% 83% 17% The reaction between zinc and 47% 100% 14% 14% 71%

62

Effective Not effective Age group used with Classic Chemistry Demonstrations Use

1 and 2 3 4 and 5 KS3 KS4 KS5 KS3+4 KS4+5 copper oxide The hydrogen peroxide/potassium iodide clock reaction

47% 100% 29% 43% 29%

‘Magic’ writing 47% 86% 14% 100% The electrolysis of molten lead bromide 47% 86% 14% 100% Turning ‘red wine’ into ‘water’ 47% 100% 100% The oxidation states of vanadium 47% 100% 17% 83% The optical activity of sucrose 40% 67% 33% 100% The real reactivity of aluminium 40% 100% 67% 33% The reduction of copper oxide 40% 83% 17% 17% 50% 17% 17% Reactions of the alkali metals 40% 100% 17% 17% 67% The reaction of magnesium with steam 40% 83% 17% 50% 17% 33%

The list below shows those experiments that were not used by any of the teachers responding to this question. The preparation of nitrogen monoxide and its reaction with oxygen Dalton’s law of partial pressures Gas bags Determination of relative molecular masses by weighing gases A controlled hydrogen explosion The combustion of methane Equilibria involving carbon dioxide and their effect on the acidity of soda water The density of carbon dioxide The reaction of ethyne with chlorine The spontaneous combustion of iron The preparation and properties of nitrogen (I) oxide Light scattering by a colloid (the Tyndall effect) – ‘the thiosulphate sunset’ The liquefaction of chlorine The electrical conduction of silicon – a semiconductor The lead-acid accumulator Making polysulphide rubber For the relatively small sample there is an impressive use of both experiments and demonstrations spread across the age ranges but predominantly use is for KS3 and 4 students.

How the products were used Teachers were asked what prompted them to choose to use the Classic Chemistry texts. Almost all teachers’ choice was made on a personal basis, rather than departmental or in relation to specification requirements. Teachers explained why they chose to use the product in relation to other resources available to them. In particular teachers valued the quality of the teaching resources in Chemical Experiments and the clarity and quality of instructions:

‘Clear instructions but room for pupils to think for themselves.’

‘Clear instructions. Not too much text on page, not dumbed down.’

‘Clear, good teacher and student sheets easy to access.’

‘Excellent students worksheets from web. Good teacher/technician info’

‘Good instructions, clear and good questions at end.’

‘Reliable – can be trusted, clear instructions for technician.’

63

‘The experiments are generally clear, easy to follow and well presented.’ One teacher commented on the nature of the experiments in the following way:

‘Exciting experiments not normally detailed in textbooks.’ Similar reasons were given for their choice of Chemical Demonstrations over and above other resources:

‘Best information and detail.’

‘Detailed procedures outlined, health and safety issues given.’

‘They are clear and info. for technicians is good.’

‘Good ideas/succinct and well written.’

‘Good information, generally know its going to work in the time said!’

‘I have a range of resources available, but this is usually the most comprehensive.’

‘I haven’t found a single resource of demonstrations like this.’

‘The experiments all work really well, the instructions are well written & it is v. handy having a single book with all good demos in.’

‘The experiments are easy to follow and provide excellent tips and theory.’

‘Tried experiments that work are well explained. Theory accompanies them = quality experiences.’

In Table 4.21 teachers’ ratings of how useful they found the different aspects of guidance about the demonstrations in Classic Chemistry Demonstrations are reported. Table 4.21 Teachers’ ratings of the usefulness of guidance about demonstrations

Very Not at all Classic Chemistry Demonstrations 1 2 3 4 5

Where it fits in the curriculum 17% 33% 17% 33% Who it’s for 39% 39% 15% 8% Timing 46% 46% 8% Resources 91% 9% How to use it in your teaching 15% 54% 8% 23% Chemistry theory related to the demonstration 54% 46% Extension activities 15% 46% 31% 8%

As expected teachers rate highly any pragmatic advice about timing and resources. They also put a high value on the referencing to the chemical theory that the demonstrations exemplify. As with other products teachers welcome advice about which students the activities should be used with. However where the demonstrations fit in the curriculum and how to use it in their teaching were areas which they expressed less concern about. What they do seem to want is more direct referencing to national curricula and schemes of work. Teachers were asked which ability groups of pupils they use the two texts with over each of the years 7 to 13. In years 7, 8 and 9 all teachers indicated they used both texts with all ability or mixed ability groups. In addition to these groups, in later

64

years one or two teachers indicated they used the texts with top sets as well as mixed ability groups. 83% of teachers said they used the experiments in a particular order determined by the curriculum.

‘Experiments used wherever they would fit into each course's timeline.’

‘Impossible – I would match them to the schemes of work. At KS3 this would be taught on a rota to avoid equipment clashes.’

‘Loosely – used where relevant but no structure. Possibly an area to develop.’ 67% of teachers also reported that they used the demonstrations in an order determined by the curriculum, courses and schemes of work. Teachers were asked how additional referencing might support their use of the products. Although four teachers would find it helpful to have experiments organised in the text by sequences in age, and two indicated they would like them organised by difficulty, teachers’ comments more generally suggest multiple needs:

‘Again initially sequenced by age and then by difficulty within each age grouping.’

‘By chemicals involved or topic. This is a superb resource for teachers who should be able to use their own judgement over difficulty/age.’

‘By level (e.g. GCSE, AS, A2) and/or by topic (e.g. Group 1, Group 7, Rates etc.)’

‘If organised other than by topic area may lose sight of some. I have just found more useful ones by looking through again.’

‘It would be really useful to have them matched to KS3 QCA schemes of work and at KS4 by main modules for different awarding bodies.’

‘Type/topic – the most frustrating thing about the book is finding relevant ones for each topic. Some have been put on our Schemes but it would still be useful to order them by topic.’

Similar needs were expressed for the referencing of Demonstrations:

‘Firstly in year group sequence then within each year group by difficulty.’

‘Grouped according to the National Curriculum for KS3 then KS4 with KS5 arranged by theme.’

‘KS3 linked to QCA units, KS4 topic based & KS5 topic based (i.e. organic/physical/inorganic).’

One teacher made the point well for the need for the content management that is already under development for the RSC websites: ‘No problem as they are but referencing to QCA KS3 schemes of work, and GCSE areas might be really helpful – especially for searching on a CDROM/web. While some teachers thought referencing by difficulty or level would be beneficial, one teacher expressed the following concern: ‘Sometimes a more difficult/easier one still illustrates concept – would maybe miss some if categorised too much.’

65

As reported elsewhere in the evaluation of the Classic Chemistry texts, although teachers did not want the texts to advocate a particular teaching approach, a clear idea of teaching contexts would be welcomed by many.

‘What would be super would be a suggestion of contexts. I think this would increase their use to teachers.’

‘Ideas on how this fits into the curriculum, any ideas on introduction/links to other topics etc.’

‘OK – good to use as they are but wouldn't mind teaching suggestions.’ Of the teachers who were asked, 83% indicated they use the Classic Chemistry Experiments as set out in the text, whilst 17% adapt them. One teacher explained why he adapted the activities.

‘This would depend on the class/time of day/importance of the practical in the lesson. If there was a template then the teacher could alter the experiment to fit the template (version D in particular), if material was on CDROM/web.’

Teachers were extremely complementary about the Classic Chemistry texts and gave the following reasons why they liked the materials as they are. Classic Chemistry Experiments:

‘Allows staff to fit them into their own plan of the lesson.’

Clear diagrams & instructions.’

‘Experiments are clear with easy to follow instructions and reduce the amount of preparation time.’

‘They are easy for the pupils to use and require no alterations and are easy enough for staff to follow.’

Classic Chemistry Demonstrations:

‘Allows freedom to fit into the lesson as we wish – they are demos which we chose to use to illustrate the scientific theory.’

‘I think it is important that teachers use their own judgement and knowledge to construct their lessons & select appropriate activities.’

‘Sufficient detail, referenced by topic type & age range anyway.’

‘The material is in discrete units, with clear instructions – good for dipping into.’

Reasons for no longer using the products Seven teachers completed a questionnaire asking about their ‘past’ use of Classic Chemistry Experiments, and seven teachers completed a questionnaire about their ‘past’ use of Demonstrations. These teachers were asked to indicate their reasons for no longer using the texts (see Table 4.22).

66

Table 4.22 Teachers’ reasons for no longer using texts Very Not at all Classic Chemistry Experiments 1 2 3 4 5

Now incorporated into my teaching 14% 43% 29% 14% Replaced by more appropriate material 43% 14% 43% No longer relevant to the specification 14% 14% 43% 29% Other texts are used in the department 14% 14% 43% 14% 14%

A pleasing number of teachers have integrated the activities into their teaching and their schemes of work. A small number of teachers consider the texts to be no longer relevant, as with other products they refer to a lack of specific curriculum relevance.

‘Many of the experiments are 'good fun’ but very few are relevant to an AS/A level course. A number can be used as demos, relatively few as class experiments.’

The reasons for non-use of the Demonstrations text were less clear with only a small number suggesting that the demonstrations were integral to their teaching now or that the text was inappropriate but teachers offered the following explanations:

‘Did not always have access to the books, as others 'borrowed’ them. Thanks for my own copy!’

‘Time to review practice and incorporate more/new experiments into lessons.’

‘Time & pressure to constantly refer to other resources as I have far more responsibility.’

Two volunteers who had classified themselves as ‘non-users’ of the Classic Chemistry texts returned completed questionnaires. They both said they had not had access to the texts previously. Both were positive about all aspects of the two texts: the format; the organisation of the text; subject coverage; appropriateness for pupils; compatibility with exam specifications; compatibility with teaching and learning approach. One of the reviewers commented that he had found both the texts ‘well set out with lots of new ideas.’ Both indicated that they would definitely consider using the texts in the future. Of the products selected for evaluation these two related texts were the ones with the earliest publication dates. The teachers’ responses indicated that they were still widely used and highly valued and continued to feature in schemes of work across the key stages. It was also clear that the lack of use reported for aspects of the texts reflected in part, changes in the curriculum and in the amount and type of practical work undertaken in current classrooms. Research has found that this use has narrowed in schools in England, particularly at KS4, in response to the requirements of GCSE coursework (Osborne, 2004). The other finding emerging was that teachers would welcome explicit cross-referencing to the curriculum and exam specifications. Some teachers would like more guidance about the age appropriateness of the materials and their difficulty. To an extent this is embedded in the Classic Chemistry Experiments in the alternative approaches to the student activities offered. It is clear that some teachers are happy with limited guidance and maximum flexibility whereas others need more explicit advice about the target age and difficulty level of the activities. These are two related issues and to an extent any guidance must make assumptions about learning and teaching that are difficult to make explicit in any directory approach to content management. Such guidance might be best offered through

67

exemplars from teachers’ use of materials with their students as part of the RSC website.

4.2.3 Ideas and Evidence The original intention of the evaluation study was to evaluate one of each type of product. The paper-based products were over sampled to include classic texts that had been available for some time (Classic Chemistry Experiments and Demonstrations) as well as more recent ones (Chemical Misconceptions Vol I. and II.). There was, however, some interest to see whether teachers were accessing recent paper-based products, which specifically aimed to meet recent curriculum changes and challenges. Very limited data collection was planned to consider this in relation to the Ideas and Evidence series of texts. The Ideas and Evidence series of books were produced by Dorothy Warren an RSC School Teacher Fellow. Like other products they were recent publications (2001) that were written in response to changes in the science curriculum. These curriculum changes had increased the emphasis on teachers using cross curriculum themes to teach subject knowledge and extended scientific enquiry to include treatment of ‘ideas and evidence in science’ (England and Wales) which in turn was seen to relate to ‘developing informed attitudes’ specified as part of the Scottish 5-14 Environmental Studies curriculum. The books were designed to serve as professional development for teachers providing them with up to date information on cross curriculum themes and to extend their teaching practice by introducing a range of activities that catered for students’ different learning styles and resources that allowed ideas and evidence to be taught as part of ongoing chemistry or science lessons. Each of the five books target the 11-19 age range and can be accessed on the website and student activities and diagrams modified. The books have similar structures and approaches and each emphasises scientists’ roles and ways of working and introduce students to key players in the past and more recent history of scientific developments and theorising. Another shared aspect is a focus on the way that political and social influences can shape science and vice versa. Each book has teacher notes that include background information, sources, teaching tips, resources and answers to students’ worksheets that follow. There is also advice about how to access the product on the website and how to modify activities. There are a variety of student activity sheets sometimes these are in alternate forms to allow different roles for students and to support different learning objectives. There are also a variety of information resources for students. The worksheets and activities are given specific references about the age and ability of students that they can be used with. Chemists in a Social and Historical Context is structured in discrete topics but each deals with the development of scientific theories or other advances in scientific knowledge, the historical role of scientists in this and the impact of the social and cultural context on scientific developments and their impact. The learning objectives targeted in this book are not new to the curriculum but the range and depth of treatment is. The activities provide teachers with ways of addressing this aspect of the curriculum that allow students an active and critical role in thinking about scientific evidence. Climate Change is similar in approach but looks at current scientific ways of working and communicating. The topic covered is global warming, and in relation to this students consider ways of collecting temperature data and reading and

68

interpreting trends in data. They also examine theories about global warming and the role of greenhouse gases, in particular CFCs in the depletion of ozone and how this scientific discovery has influenced politics and the manufacturing industry. Whilst global warming features in most curriculum specifications at KS4 the treatment in the book differs in two important respects. First in the range and depth of treatment and second in the approach. Students are expected to consider the issue in some depth and to relate past and present understandings to scientific discoveries and evidence. They are then required to engage critically with evidence in order to be able to evaluate different positions - their own and others. Green Chemistry is devoted to developing understanding of the twelve principles of green chemistry and green energy resources and students study these in the context of particular industrial applications such as metal extraction processes, desulfurisation, and waste management. Green Chemistry, unlike the other two books, does introduce a major concept i.e. green chemistry that is not particularly emphasised in current specifications. Health, Safety & Risk is somewhat similar in that it introduces and develops concepts such as dread, risk and hazard and looks at scientists’ roles in advancing understanding about public health and safety in industrial practices. However it does include assessment of safety and risk in the context of students’ own investigations and experiments. There are some excellent activities to help students and teachers work with these innovative ideas in the context of their science and chemistry learning. The Nature of Science is perhaps the book that appears to fit more closely with current practice, or at least practice that is advocated, as its main objective is understanding evidence and investigative procedures which is increasingly part of the assessment of KS3 and KS4 science. The first four books include issues and applications that can readily be located in the science curriculum but the treatment of issues is probably in much more depth than is often expected in exam specifications involving students in thinking outside the box both from an historical, social and political perspective. These books too, appear to deal both in their pedagogy and content with science and chemistry in ways that are only just becoming mainstreamed for secondary students through the introduction of the pilot project on 21st Century Science.

Feedback from the Questionnaire A very simple questionnaire was put on the website inviting feedback on the Ideas and Evidence books. There were 26 reponses: Nature of Science (11); Chemists in a Social and Historical Context (8); Health, Safety & Risk (5); Green Chemistry (2) and Climate Change (0). The sample was international including respondents from 9 countries though predominantly English. The responses to an extent reflect what we would argue is the current curriculum relevance of the books with Green Chemistry and Climate Change featuring the least or not at all. One teacher was interviewed and he had not yet used any of the books but in his review selected The Nature of Science and Chemists in a Social and Historical Context as having potential widespread relevance across key stages 3 and 4 giving further support to this. The majority of respondents (82%) judged the product to be excellent (46%) or good. Only two respondents thought the product was not adequate. It was not possible to discern differences between the books but Health, Safety & Risk had the odd negative comment, which supports our view that the issue was teachers’ perceptions of

69

curriculum relevance. The teacher interviewed left this book until last in his review and commented that it appeared ‘dull and not pupil friendly’. We suspect that this is because the core concepts such as risk that are targeted are not well understood by teachers currently. We asked how teachers would like to see the product improved and 50% referred to the need for better cross-referencing to the curriculum, a further 30% wanted age groups more clearly targeted. Teachers were asked how they had heard about the books and only three referred to the direct mailing of the book. Two teachers had heard about it from other RSC products and two from colleagues. The majority had heard about it from the RSC website (30%) or other websites (23%). This is perhaps a reflection of the form of data collection and that a significant number of respondents were outside the UK. Teachers were asked what aspects of the books they used. About a third tended to use the books to increase their own background knowledge with about half using the introductions and teacher guidance. Just over a third used the student worksheets and activities. Use was spread quite evenly across the age ranges targeted, though KS3 students were identified most frequently (58%) followed by KS4 (42%). The opportunity to modify activities and worksheets was welcomed by teachers through the web resource and the majority of teachers reported that they adapted sheets. We asked teachers about their preferred format for the resource and as with other respondents multiple formats was favoured by the overwhelming majority (81%). Only one teacher wanted the web only and two the book only. Although the data is very limited it does appear that the product’s potential is not being fully realised. This is particularly the case where the books focus on topics that are only treated lightly in curriculum and exam specifications. Or where the concepts being explored, and the teaching strategies used, are particularly innovative. When the books are used they are, as with all other paper-based products, very well received. We have already noted that in both content and pedagogy the series is in advance of mainstream curriculum materials. Teachers identified the need for cross-referencing to curriculum and to students. It would be particularly important to cross-reference these books in relation to the new curriculum for KS4 students in England. This cross-referencing should not only be in terms of topics and issues covered but importantly in terms of teaching strategies that have relevance within the chemistry curriculum but also across the science curriculum more generally. These pedagogic strategies would also have wider interest for other teachers. Teachers need considerable support to deliver a curriculum which emphasises public understanding of science as well as academic and vocational learning and these well thought through and teacher-trialled strategies will be of great benefit to them. It would be a particularly valuable focus for future RSC INSET. The paper-based products were all rated highly for their content. Clarity and effectiveness in meeting needs was also considered to be good to excellent. Ratings on clarity were reduced for Chemical Misconceptions Volume I with a third of teachers experiencing some difficulties with access. Ratings of ease of use across the products were slightly depressed reflecting other feedback about the need for cross-referencing to exam and curriculum specifications. The Misconceptions texts are very much a combined professional development and teaching and learning resource. Teachers selected Misconceptions Volume I to

70

develop their practice and their understanding of students’ learning. The feedback indicated that teachers’ difficulty with access was largely restricted to the theoretically challenging chapter in the volume. Even then two thirds of teachers reported no problems with access. The chapters on chemistry topics were highly rated in terms of clarity and usefulness particularly those with direct relevance to key aspects of the chemistry curriculum for KS3 and 4. The detailed evaluation allowed insights into teachers’ use of products with students. It was clear that teachers were using the activities and guidance across the key stages.

4.3 Multimedia product

4.3.1 Alchemy? Chemistry & Industrial Processes for Schools & Colleges The product selected is a double-CDROM Set and supporting web site resource published by RSC in 2002. Alchemy? is for teachers and their students aged 11–18 i.e. KS3, 4 and 5. Its purpose is to support teaching and learning about industrial processes in the chemical and pharmaceutical industries. Material on the CDROMs is structured into 15 topics, each covering an industrial process.

• aluminium extraction • ammonia • chemicals from salt • copper refining • gases from air • iron and steel • nitric acid • Nylon • oil refining • polythene • sodium • sulfuric acid • combinatorial chemistry • computational chemistry • making medicines.

Each topic has a common structure and number of components. These include an introduction accompanied by a slide show of some end uses of the product; a video clip with commentary available as sub-titles enabling use by the ‘hard-of-hearing’; a ‘fact file’; a set of multiple choice ‘focus questions’ which can be answered on screen for automatic marking; a resource file for teachers; video stills; further questions with answers for students; printable pictures and diagrams; and pointers to further resources. Each topic also has a link to the Alchemy? web site, which links on to other relevant sites. How teachers and their students use these elements of the product depends in part on the ICT facilities available.

71

Learning objectives are not made explicit: the objectives suggested by an analysis of the resources making up the product include:

– exposing students to authentic images of the chemical industry, with a good idea of scale and of the functional focus of chemical engineering;

– locating ‘theory’ in authentic industrial contexts; – learning about the ‘scientific’ rationale behind everyday artefacts, thinking and

distinctions, e.g. between drugs and medicines; – learning about some of the roles played by chemists and chemical engineers in

industry and how they use familiar classroom skills; – appreciation of the accessibility of these roles to all; – appreciation of the quality of design thinking and engineering in the chemical

industry; – appreciation of the industry’s efforts towards environmental sustainability.

Feedback from the Questionnaire

Overall views of the product A sample of thirty was identified for the questionnaire survey and a response rate of 76% (N=23) was achieved. The earlier discussion revealed that the majority of these teachers had easy access to RSC products. All teachers rated Alchemy? as excellent to good and all but two considered it to be very easy to use and clear. Over two-thirds considered their needs were well met but six teachers were less positive about this. Two teachers who had used Alchemy? in the past, but no longer did so, both rated the resources highly as means of achieving desired learning outcomes, but cited technical problems, as did a number of current users – for example:

‘Chemicals from Salt section has a CDROM fault. Plays 2 voices over one another.’

‘I had problems getting video clips to run – I felt these were the entry point to the material’

‘... [there’s] the odd software glitch. Odd video link doesn’t work. PDF files troublesome to open.’

Another teacher found the technology too high-level, explaining that she used videos of industrial processes instead of the Alchemy? CDROMs, because a ‘video recorder readily available (2 for 3 Chemistry labs) whereas multimedia projector needs booking (1 for numerous depts – although this will shortly improve)’. One teacher made a practical comment about using the CDROMs, saying that it ‘would be helpful if you could tell which video clips were on each disc BEFORE putting them onto your computer!’ Teachers were asked their reasons for choosing to use Alchemy? and the majority (93%) said it was personal choice with two indicating it was a departmental choice which might influence their approach to the product and their feedback on it. When asked their reasons for choosing the product teachers specified a range of reasons. They valued the currency and relevance of the content and the way that students could access it for themselves.

72

‘Interesting and relevant applications – brings [curriculum] specification to ‘life’.’

‘It is up to date and represents the current situation accurately.’ They also valued the motivation that the product engendered in students and its user-friendliness. For many the interactive nature of the product and the high quality of the resource were particular strengths. They considered that it was a very good teaching tool. Video clips were seen as much more useful than videos.

‘Much more interactive than a video. Better animations than alternatives.’

‘Clear short video clips with good diagrams. CDROM format useful.’ Some teachers who were extending their ICT support for teaching science and chemistry were pleased to have a product that allowed the use of up to date equipment.

‘Different way of engaging students. Have just bought a multimedia projector so are still keen to use it!’

‘Easy to use want to switch to using a data projector rather than videos.’ The CDROM format and use of short video clips proved popular with teachers, who would like more of the same: as one said,

‘More! Digital video clips are the way forward in science education. How about some high impact clips e.g. TNT’.

All four non-users who looked at the resource when prompted to do so by the questionnaire liked what they saw and intend to use it in future. Previously they had not had access to or known about it. In the cold call interviews we also found that some teachers had not been aware of Alchemy? and this appears to be linked to the way materials are distributed in schools and whether or not teachers are members of RSC and therefore have better access to publicity information. About half of the teachers wanted the product to be cross-referenced to exam specifications but very few were concerned about cross-referencing to the national curriculum.

‘Limited value on the AS/A2 courses which are all that we teach. I can see that in a school we would have found much more material directly relevant. Pressure of time means we expect resources to be very focussed on the Syllabus.’

‘Some questions are simply not relevant to any syllabus – way too detailed’.

How well learning objectives were met The product implies a range of learning objectives and teachers were asked to rate the extent to which the Alchemy? resources allowed their students to achieve these.

73

Table 4.23 Teachers’ rating of how well learning objectives are met very well not at all Learning objective 1 2 3 4 5

Develop authentic image of chemical industry. 47% 42% 11% Locate ‘theory’ in authentic industrial contexts. 21% 58% 21% Understand the science behind everyday artefacts and terminology. 11% 37% 52% Understand some of the roles played by chemists and chemical engineers in industry. 21% 42% 21% 16% Awareness of how industrial chemists use familiar classroom skills in these roles. 42% 47% 11% Appreciation of the accessibility of roles to all. 11% 63% 26% Appreciation of the quality of design thinking and engineering in the chemical industry.

5% 32% 37% 21% 5%

Appreciation of industry’s efforts towards environmental sustainability. 11% 26% 58% 5%

Teachers gave very high ratings about how well the product met the main learning objectives of connecting classroom learning to industrial and consumer contexts, and developing authentic images of the chemical industry. Two-thirds of teachers considered that the objective of developing of understanding of the roles of chemists and chemical engineers was well met. Teachers gave lower ratings to the extent to which the product met the objectives concerned with making connections between school learning and practices in the chemical industry and between science and everyday artefacts. These are more difficult objectives to meet and require teachers to emphasise these within their use of the product and more generally in their chemistry and science teaching. The feedback may therefore reflect in part the focus of current specifications. The introduction of 21st Century Science and related specifications should give much more emphasis to these types of learning which positions Alchemy? well for meeting future needs. Teachers’ ratings of the extent to which aims were met was lower for this product for a number of the intended outcomes compared with ratings for the paper-based product. This is in spite of very positive ratings about the quality and relevance of the multimedia materials. Raising students’ awareness of access to roles was not judged by teachers to be as well met as other objectives. This may reflect the teachers’ views about what is required to challenge gender stereotypes. Research indicates that positive images whilst beneficial only go a small way towards challenging gender stereotypes related to careers. Two teachers identified a further learning outcome which was ‘supporting students in carrying out independent research’ and teachers commented on the value they saw in products that allowed students to find things out for themselves.

Teachers’ use of topics, their rating of their effectiveness and which students targeted Teachers were asked which topics they used regularly or occasionally; how well the topics they used worked with their students; and what age groups they used the topics with. Table 4.24 shows the data for the most well used topics in order of use and Table 4.25 shows the data for the least well used.

74

Table 4.24 Most used topics and teachers’ ratings of their impact % users rating

effectiveness of impact very not at all

% users using with age group Topic % respondents using topic

1 and 2 3 4 and 5 KS3 KS4 KS5

Ammonia 57% 85% 8% 92% 23% Aluminium Extraction 52% 75% 25% 100% 33% Iron and Steel 52% 75% 8% 8% 75% 25% Copper Refining 43% 70% 100% 10% Oil Refining 43% 70% 30% 90% Chemicals from Salt 39% 67% 11% 33% 100% 22% Gases from Air 35% 38% 38% 25% 75% 13% Nitric Acid 30% 43% 29% 86% 14% Sulfuric Acid 30% 57% 43% 86% 43% Polythene 26% 50% 33% 83% 33%

Unsurprisingly the topics which teachers rated as the most effective are closely linked to those they used the most: Ammonia Aluminium Extraction

Iron and Steel Copper Refining Oil Refining. An appreciable number of teachers use the following three topics Chemicals from Salt, Sulfuric Acid, Gases from Air and there is a slight decline in teachers’ views of their effectiveness most notable in the case of Chemicals from Salt. Table 4.25 Least used topics and teachers’ ratings of their impact

% users rating effectiveness of impact

very not at all % users using with age group Topic % respondents

using topic 1 and 2 3 4 and 5 KS3 KS4 KS5

Nylon 17% 75% 25% 0% 0% 50% 75% Making Medicines 9% 100% 0% 0% 0% 50% 50% Sodium 9% 50% 50% 0% 0% 100% 0% Combinatorial Chemistry 4% 100% 0% 0% 0% 0% 100% Computational Chemistry 4% 0% 100% 0% 0% 0% 100%

The questionnaire asked teachers more generally about their use of the product for different age ranges and ability groupings and their feedback is given in Table 4.26. This shows a similar reported use to Tables 4.24 and 4.25. The product, in the majority of cases, is used with all ‘levels’ or sets of students. Some teachers report using the product with upper sets only. This reflects the tendency in many schools for Chemistry as part of triple science to be made available only to the top sets of students in Years 10 and 11 (KS4).

75

Table 4.26 Use with different ages and ability groups Year group Used with all sets Used with upper sets only Teachers using with this year

Y7 11% 11% Y8 5% 5% Y9 17% 17% 33% Y10 67% 22% 89% Y11 72% 28% 100% Y12 50% 17% 67% Y13 33% 5% 39%

Teachers’ use of components within topics Each Alchemy? topic has a number of components. Teachers reported using the components to differing degrees. Table 4.27 shows how many teachers used specific components across topics: and the number of cases of teachers citing use of a specific component in teaching a specific topic. Introductions and video clips are far the most widely used components and this reflects the feedback teachers gave about their reasons for choosing the product. The Fact files and Focus questions are also well used as is the auto marking for use by students. The Resource files and www links are reported as used only rarely. Table 4.27 Teachers’ use of components

Component type % using component No. of citations of use of component within topics

Introduction 67% 74 Video clip 100 % 111 Fact file 56% 56 Focus questions 61% 49 Auto-marking 56% 36 Resource file 39% 18 www links 11% 5

70% of the teachers said they found the teaching notes in the resource file for each topic useful. The other teachers who did not were experienced teachers with an average of over 15 years’ experience. One of these teachers also noted that the notes were less useful to him because they ‘go beyond GCSE’. Teachers reported that most of the time components were used as provided. A small number of teachers reported that they adapted components to suit their circumstances. For example the pictures and diagrams were adapted for a PowerPoint display. A fact file was adapted for use as an introduction to a topic. Focus questions for students were also sometimes altered or removed to better fit specific specification requirements. Teachers were asked about the teaching context in which they used the product, that is whether it was used by the teacher with the whole class; used by all the class working as individuals; or used by only some students working as individuals. The responses reflect the different nature of the components for example the introduction and video clips are for use with whole classes whereas other components like focus questions and auto marking are for students' use though they could be used with sub-groups of students too. Where components were used with a subset of students (in 40% of cases) it seems that this latter use is mainly for individual students to carry out personal research. This helps explain why teachers saw support for independent research as a learning objective for the product referred to earlier.

76

Table 4.28 Context of use of product components How resources were used % teachers

Whole class teaching 93 All students 40 Some students 40

All of the teachers who gave feedback about whether more explicit teaching guidance would have been beneficial reported that that they found the fact that no teaching approach was given or advocated for their use of Alchemy? did not limit its usefulness. Teachers expressed appreciation of the flexibility, quality and usefulness of Alchemy?, for example:

‘Able to use parts as I wish/as I find relevant.’

‘Flexible approach to pick & mix from the resources as required. Clear to view. Run easily... .’

‘It is very easy to adapt the material to whatever I want to do in a lesson.’ Teachers planning to use the resources had similar usage profiles in mind to those described above, and appreciated the format, organisation, subject coverage, and compatibility of the resources with their approaches to teaching and learning. They were less convinced of the appropriateness of the level of the material for their students, and its compatibility with their exam specifications.

Feedback from interviews Some of the issues emerging from the questionnaire were explored in more depth through interviews with three teachers and a case study of one class’s use of the product. The teachers interviewed reported that they found the CDROM format convenient, especially if mounted on the school network. The teachers commented on the product being up to date; and for teachers who use ICT extensively in their teaching, the digital video clips are ‘very useful’ and have become embedded across a range of lessons and topics, as the ‘high quality audio-visual stimulus helps to maintain attention’.

Relationship with curriculum and specifications All interviewees saw Alchemy? as contributing to their delivery of the core curriculum, and as providing curriculum extension or enrichment as well. For example one teacher commented that he sometimes uses the product to meet ‘specific objectives from the syllabus, but often just an appreciation of how industrial chemistry really works’. Another saw the intended learning outcomes mainly as ‘knowledge of the chemistry of industrial processes, including the chemical equations, but put in a real context’. Though the questionnaire responses suggested that some teachers see problems in the relationship between the Alchemy? resources and their particular specifications, none of the interviewees agreed with this point. If Alchemy? did not exist teachers would be able to find other resources: they report that they choose to use Alchemy? because it is ‘clear, concise, includes diagrams, and is relevant to the specification’. In one interviewee’s school, she is part of a two-person chemistry department: her junior colleague has followed her example in using the product. In another school the

77

teacher reported that teachers ‘go their own way’, some colleagues use parts of Alchemy? but not necessarily the same parts or in the same way. This was different to the approach in the case study school where teachers across the science department make use of Alchemy? in similar ways, as it has been written into the scheme of work and sufficient ICT resources are available to make such use possible. One interviewee used the components as provided. Another said that topics are ‘backed up’ by ‘a fill-the-gaps worksheet’. In the case study school a sheet of questions and model answers are produced for each topic, which are ‘focussed on information useful/essential for the curriculum’.

Contexts of use The interviews allowed us to explore further some of the ways that teachers used the product with their students. At the time the interviews were carried out, in the first half of summer term, the most recent use of Alchemy? by all three interviewees was in the context of revision. This recapped earlier use in a normal teaching sequence. Consequently the components served a dual purpose - teaching new topics and consolidation of learning. One teacher working in a small independent girls’ school reported using the product for independent research with KS4 and A-level students. Her most recent use was to broaden and deepen understanding coming up to exams with a ‘Y11 class, the separate sciences group i.e. top set, for revision, and getting students to look at the subject matter in a slightly different way.’ The students used the product as independent research for homework. The teacher commented about the content and emphasis of the use of the product:

‘We looked at contact processes and equilibria as the material on the CDROM was at a suitable level for this set. We used the extraction of metals and sulfuric acid sections. Students especially made use of the intro sections, the longer Q&As; and the video clips.’

Her view of the impact of the product was that the lesson

‘went fine, some students found it very useful, it did help understanding as shown by the results of homework. The video sequences are good because they are packed with content and not too long.’

This teacher commented on the professional development aspect of the product which allowed her to update on industrial processes. She felt that it is ‘the sort of material that we’d make a lot more use of if we weren’t so constrained by syllabuses and exams, and could let students work independently a lot more and follow through research projects. I like giving students independent tasks because it opens their eyes beyond the curriculum.’ Along with other teachers including the cold call sample this teacher saw Alchemy? as of limited use for KS3 and Y10. In her view it is ‘generally at too high a level: RSC could perhaps differentiate material more and aim more at supporting independent ‘research and present’ tasks lower down the school.’ She felt that there was a dearth of resource generally available for KS3 and Y10 ‘that isn’t very SATS- and GCSE-specific.’

‘In general I find RSC materials fantastic – especially things like for example the Art/colours in chemistry materials, which we are going to build in to the

78

curriculum next year. Such things are mostly used for independent research done for homework, very rarely as a whole class teaching resource – perhaps 2 or 3 times a year, depending on time and availability of the ICT room.’

A second teacher interviewed worked in a large mixed 11–18 comprehensive school. He had used the product recently with an upper ability Y11 group, for revision on industrial processes. The content covered was the blast furnace, electrolysis of aluminium oxide, and the Haber process.

‘The parts of Alchemy? used were the videos covering the three industrial processes ... Actually we had technical difficulties because we used the CDROM with a smart board, but the sound was not working; so instead my colleague and I took it in turns to talk through different sections. This was surprisingly effective and I would use it again as a technique. We also made use of the opportunity to stop the clip on the diagrams to emphasise what was shown.’

The objective was to revise the chemistry of industrial processes in a ‘real context’. He commented on the impact for students' learning, that ‘the students were very positive and said that the revision session had been useful.’ This teacher identified a particular strength of the product:

‘One of the advantages of the Alchemy? video clips is that they are short. It is important that students are aware of these industrial processes, but there is a limit to how long they will look at pictures of large pipes ... I have used the section on the electrolysis of brine with Y10 students but they found this rather too difficult and as a result they lost interest, even though it is short.’

The third teacher interviewed also gave permission for a revision lesson using Alchemy? to be observed and some of his students interviewed, and gave full access to his teaching materials, schemes of work and lesson planning documentation. He worked in a large mixed 13–18 comprehensive community high school located in an area of socio-economic deprivation. A recent OfSTED inspection talked of a ‘remarkable’ use of ICT, both in teaching and in managing information. Every room in the school has a ceiling-mounted projector connected to a computer with wireless keyboard/mouse. The teacher was in his first year as a newly qualified teacher, though had before that taught for a year at an independent school. He described himself as an ICT enthusiast, and had recently been given whole school responsibility for use of interactive technology in classrooms, as well as becoming KS3 co-ordinator for science. He used several Alchemy? topics during the spring term and was at the time of the evaluation revisiting them more briefly for revision. He often used parts of video clips, or showed them in sections, and was eclectic in putting together material for a lesson from multiple sources. He currently used Alchemy? in four topics in Y10 and another four in Y11. This teacher used the Alchemy? components with all sets of students. He described how he managed this: ‘if I’m differentiating then I’ll differentiate by the questions that I give them’. He adapted the focus questions as he had:

‘not had any success using the Alchemy? focus questions so I’ve written my own to pull out the precise learning outcomes that I want from the video.

79

Differentiated resources in that respect would be good and maybe even using the same video archive, just putting different overlays on top, so word equations rather than symbol equations, if you were looking at [early] Keystage 3’.

He thought the curriculum focus of the Alchemy? materials was appropriate, and appreciated their concise and modular nature:

‘It’s nice to have them bite size, because if somebody gave me a sort of a 50 minute video that was on industrial chemistry I’d never use it, because we’re pushed enough as it is to get through the content. So while it would’ve been interesting to show the basic oxygen steel making process to my Year 10s, it’s not on their syllabus and so when we’re measured by the outcome of our students’ grades rather than their depth of understanding... I am going to very rarely teach students things that they don’t need to know as this stage. And so the fact that they are focused and that they can drop into lessons is very important for me.’

Like other teachers he thought that cross-referencing to the National Curriculum was not very helpful but that cross-referencing to specific exam board specifications would be very useful. He also commented that cross-referencing to the 21st Century Science specification would be very useful when it comes into general use. For KS3 he saw a need to cross reference to the KS3 strategy and QCA schemes of work.

Observed lesson – Year 10 During spring term, the students interviewed had had a lesson in the module on rocks and metals that was concerned with the extraction of iron and focused on the blast furnace. Resources from a variety of sources including Alchemy? and other RSC resources were used in the lesson supplemented by teacher materials. The Alchemy? CDROM2 was used specifically to review facts about iron and steel, and to let students view the video clip on the blast furnace and answer questions on this. The lesson was carefully structured and prior knowledge and learning were considered in relation to reduction reactions. Four students who had taken part in this earlier lesson and the observed follow-up revision lesson on extraction of metals were interviewed, three girls and one boy. They were from an upper set in Y10. The paired interviews took place on the same day as the revision lesson and probed the students’ learning and their views of the usefulness and effectiveness of the product. The interviews found that each of the students displayed an understanding of the processes involved in the extraction of metals at a number of levels. They showed understanding of raw materials and mechanisms involved in the process, understood what reduction and oxidation are and the specific chemical reactions involved in the extraction of iron and how these were examples of reduction and oxidation processes. They were also able to explain the reactivity series and how a metal's position in it dictates extraction methods.

First pair: Limestone goes in the top of the blast furnace along with haematite and carbon, and hot air is blasted into the blast furnace and then molten … at the bottom along with iron … The main process that’s going on ... is reduction ... That something’s losing oxygen ... but gaining electrons ... oxidation would be the reverse ... you get carbon dioxide, so carbon is oxidised. [Other metals

80

that could be extracted from the ore using a similar process would be] anything below carbon in the reactivity series. Apart from things like gold that are found as pure elements. The lower down things are in the reactivity series the less reactive they are, like gold and silver and platinum. The ones above carbon are extracted by electrolysis. Like aluminium.

Second pair: Coke goes in. Limestone. Purified coke. Coke, limestone and the iron ore. And limestone’s for purification and you have like the coke it takes the oxygen ... away from the iron ore. [That’s called] reduction … The coke’s being oxidised, which is carbon … So that with oxygen attached to it, it’s CO2. Carbon dioxide. [That works for] anything below carbon is the reactivity series. Until you get down to gold and things which are too unreactive to happen, like gold ore. [The reactivity series is] like a long list of, is it elements or just … Metals isn’t it? And you have metals that are most reactive at the top and least reactive at the bottom. And carbon’s in that list but it’s not a metal. [The ones above carbon] you have to use electrolysis – magnesium and aluminium.

They recalled watching the video first, as an introduction to the topic, and making notes on it:

We watched the video first and then we went through it in more detail on the board with diagrams and figures and books.

We started with the video and it kind of introduced the topic to us. And we made a lot of notes on it, that was useful and it was easy to take notes.

Their feedback indicated that the video had:

– helped broaden their understanding of the chemistry; – given them a different perspective of the subject; – located their chemical knowledge in authentic settings; – made it more interesting by providing visual images of settings and processes,

and giving insights into interesting areas not covered in lessons such as how coke is made.

For example:

It went broader. A lot of different things, like how coke’s made and things, because we didn’t do that in the actual lesson. It gave you a slightly bigger picture. It showed the formulas and everything for it, which was useful. It showed like the actual factories and how it was being used in factories. We know we couldn’t do without it. How we couldn’t do without it, the chemistry, the chemical knowledge. And it gives you an idea of what the industry is like.

... how some of them are made before they go in, like coke ... how it got transported ... and it had all the pretty colours and all the buildings, colour coding that highlighted the different buildings where processes happened. ... It made it a lot easier because you could actually see it. ... And you could visualise, see how it happened rather than just having to like imagine it. Yes it shows you. Because sometimes, I mean you can read about a process but it’s hard to imagine what it’s really like ... I didn’t really know what coke is, they say you put coke in it but without seeing it how do you know what it’s going to be.

81

The feedback revealed that the particular video did not give these students insight into the sustainability of chemical industry processes; nor into employment in the chemical industry. As they commented: ‘it didn’t look at the human side of it ... It just looked at the actual process’. Both these areas students felt would be interesting, though all four retained a focus on the understanding required by their exam specification, which the video had definitely contributed to. All felt that jobs in chemistry and the chemical industry would be open to them, though none was interested in pursuing such a career path.

‘No I’m not interested in that. Not a very nice place to work really.’

‘I’m more of an artist. ... I’m planning on being a social worker.’

Future directions for development of RSC resources for schools The case study teacher commented about Alchemy?


Given carte blanche to specify his own product, he would ask for ‘Hollywood movie clips related to science ... sort of captivating kids.’ He referred to a clip in the Disney film the Lion King where

‘they’re [the animals] all lying on their backs and looking up at the sky and one of them says to the other “What do you reckon that great big bright light is up there?” And one of them says something like, “Maybe it’s a hole in the sky” or something like this and another one says maybe it’s something else and then one of them says “Maybe it’s a great big ball of burning gas”, and then they all look at him and go “Yeah right”. And it’s just a bit of a joke. But if you were doing the solar system with Year 7...’

‘I love the digital video clips, I think more stuff, more interactive stuff, drag and drop, white board things. That’s happening through ChemIT [another RSC teacher fellow project] anyway.’

Overall, apart from technical problems experienced by some teachers in some schools, the product was very well received and highly rated. Teachers particularly value CDROM format and the use of short video extracts. Teachers commented on the high quality of both the approach and the content. In the cold call interviews one teacher commented without prompting about the high quality of the Alchemy? product referring to it as ‘wonderful’. Feedback from the teachers indicated that the product was well used and use was judged to be very effective. Our observations and interviews with students supported this as we found that there was a positive impact on students’ learning in that their interest in chemistry topics was enhanced, their understanding of chemical concepts and applications broadened and their perception of the subject and its relevance shifted. The product was used by a wide range of teachers and students and in a variety of ways allowing flexibility in delivery that

82

included whole class teaching, group work and individual independent study within school and without.

4.4 INSET products Two INSET events, Improving Teaching and Learning in Chemistry using ICT and Using Assessment to Improve Learning in Chemistry and Science, were observed. Teachers attending both events, and those who had attended a number of prior events of the same INSET, were invited to complete a questionnaire and be involved in the further evaluation. The two events were selected because they were currently running and were addressing very topical issues. They also differed in nature and therefore audience.

4.4.1 Improving Teaching and Learning in Chemistry using ICT 19 teachers attended the event observed. Sixty questionnaires were circulated to these teachers and to teachers who attended earlier events. There was a 50% return.

About the course The Programme had the following structure: 10.30–11.45 Introduction: ICT in relation to teaching and learning 11.00–12.30 Hands-on session 1: exploring existing resources 12.30 Lunch 13.15–15.15 Hands-on session 2: creating new resources 15.15–15.30 Discussion of any issues raised by the day and evaluation 15.30 Close The resources made available to participants included a 35-page hand-out/guide to the workshop and guidance about how to access, use and develop resources; access to the www.ChemIT.co.uk website with downloadable resources, some free software including the RSC Data book and Alchemy?, Spectroscopy and Practical Chemistry CDROMs, and annotated links to useful sites etc. The day was about enabling teachers to use the RSC Chemistry resources and other resources from the web, and through a brief tour increasing their awareness of available resources. The use of Microsoft Office products was encouraged because these were found to be what most people had access to, including students at home, in libraries etc and therefore resources could be customised and students could access them from outside school. The day also covered how to adapt resources in Word, PowerPoint and Excel, to be used as handouts, OHPs or preferably with a projector and tablet/whiteboard. This set up was seen to be the most flexible for teaching. The teachers had individual computer access throughout the day, each loaded with workshop materials. The first session assumed the minimum of a data projector kit was available to teachers and covered how to integrate ICT into classroom sessions and how to use it. As part of this teachers were given templates to develop their own resources and demonstrations and examples of how to develop e-worksheets using Power Point, Word and Excel. Other applications considered apart from Office included Java Applets, Animated GIFs, Flash/Shockwave and drawing software including ChemSketch. The trainer went through these explaining their strengths and weaknesses and in the latter case how to get round these. At the end of this session teachers who needed them were given detailed worksheets with instructions.

83

The second session of the hands-on workshop was organised into three options: - Developing ICT skills for chemistry teaching - Exploring RSC teaching fellow project materials - Exploring web links The worksheets were designed to support these three options. The first option was for the teachers who wanted to develop their own skills rather than explore particular resources and their applications. Just before the workshop began the trainer gave a quick tour of the RSC ChemIT site and ran through the ChemSketch software. At the beginning of the workshop three quarters of the teachers chose to look at ChemSketch and a quarter explored the ChemIT resources. The workshop went on for 40 minutes before lunch. After lunch the trainer demonstrated the use of a tablet for the majority of the teachers, four teachers continued their work at the computer. At 2.00 pm all teachers went back to their computers and continued working until 3.15. The teachers were all engrossed and the trainer circulated throughout the workshop answering queries, interacting and modelling ways of working. The trainer was an RSC teacher fellow who developed the ICT materials as part of his fellowship and was asked to do some INSET which ended up in his devoting about a third of his time to it. He was responsible for planning and running the courses and because of their popularity he was bought out of his post for two days a week at the end of his fellowship to continue the INSET provision. He described the way the INSET evolved: ‘It was very much running one and then picking up on what people said and what works, so it wasn’t a question of having a ready-formed course, this was something that’s evolved and it’s still evolving.’ He described the goals of the course in the following way. ‘It’s awareness-raising about some things that are possible, and it’s also giving people an opportunity to make a start on getting some of the skills because one of the key things is it’s about teachers being active in the process and not passively using material that somebody else has produced for them.’ He commented on the diversity in the skills and needs of the teachers consequently there was no particular target level of expertise assumed apart from the ability to word process a document. This is why the workshop was designed to allow teachers to ‘pick their own path through it and their own level.’ The intention was that the handouts would allow the teachers to work independently but the trainer felt that some of the teachers in the group were not willing to use the handouts and tended to rely on him for instructions and support. He thought this reflected their skill level and experience. He considered that there was an equal emphasis given to the professional development of the teachers through extending their ICT experience and providing them with teaching strategies and activities for students. He thought in practice the emphasis was more on practicalities related to both aims rather than on the transformative potential of ICT.

‘There wasn’t a tremendous amount about, if you like, the philosophy of how this can change the way we work, I mean I think that was implicit really in the presentation at the beginning that it can provide extra dimensions to what we do, but most of the stuff that was coming from people was very practical rather than philosophical in any way.’

84

He offered teachers e-mail support after the course. In his view the main problem for teachers in implementing the course was the time needed to put it into practice.

‘The fact that it involves negotiating things with other people like the technical staff or other managers or whatever, senior managers, and people say that that is what kills it, they just don’t have the time to do that.’

He commented too on the problem of hardware support in schools:

‘There are less people who have access to the facilities within a classroom zone. I mean this group, about half of them have the data projector and the computer in the classroom but a very small number have the interactive facilities, so about half of them could do this in the classroom.’

In commenting on the popularity of the courses he observed

‘…RSC has got such a high reputation that it means that people expect that it will be good so they will come on it. What do I get out of it? I mean it’s a time in my career when it’s good to be doing something different definitely – you know, it’s been revitalising – so that’s the main thing I’ve got out of it.’

Feedback from the Questionnaire 60% of the respondents were RSC members and all taught KS5 students. 17% taught KS5 students only. 70% of teachers taught across the 11–18 age range.

Teachers’ views of the effectiveness of the course and materials used A large majority of the teachers (82%) found the workshop was successful or very successful in achieving its major aim to model how teachers can adapt and use ICT to improve learning. Table 4.29 Teachers’ ratings of the success of the course in meeting aims

Very successful Not at all 1 and 2 3 4 and 5

82% 11% 7%

Teachers were asked to rate how useful they found the particular materials used on the course overall. They gave value to both types of materials but rated the materials for teacher use and dual use very highly. Table 4.30 Teacher ratings of usefulness of materials

Very useful Not useful Materials 1 and 2 3 4 and 5

Mainly for teacher use 85% 15% 4% Mainly for student use 63% 33% Dual-use materials 80% 20%

Although the majority of respondents found materials targeted for all students useful or very useful, those targeted at post-16 students were appreciated by the highest number of teachers (89%) (see Table 4.31). This may reflect both the emphasis of the materials and examples used but also the needs of the teachers in their particular posts.

85

Table 4.31 Teacher ratings of the age range for which materials were the most useful

Very useful Not useful Students targeted 1 and 2 3 4 and 5

Mainly for KS3 55% 32% 14% Mainly for KS4 65% 26% 9% Mainly for post-16 89% 11% Cross key stage materials 65% 24% 12%

Teachers were asked to rate the usefulness of the specific resources used in the course. The ChemIT materials implemented in MS Office and RSC CDROMs were rated very highly in terms of their usefulness by the majority of teachers and over two-thirds gave high ratings to the other resources. Table 4.32 Teacher ratings of specific resources

Very useful Not useful Resource/product 1 and 2 3 4 and 5

ChemIT materials implemented in MS Office 97% 3% ChemSketch 67% 23% 10% Other ChemIT materials 74% 22% 4% RSC CDROMs 93% 7%

The teachers were asked to identify the strengths of the course and these are shown in Table 4.33 in order of teacher citations. Table 4.33 Teacher % citations of course strengths

Strengths of training Category of response

% of respondents

Time to use the materials hands-on 34% Materials 31% Awareness of materials available, how to use and how could be adapted 24% Trainer: credible; engaged with subject; knowledgeable 17% Ideas to develop ICT materials 10% Easy to use 7% Chance to discuss ideas with other teachers 7% Enlightening 7%

The teachers’ feedback on the main strengths of the course very closely match what the trainer considered to be the goals of the course. As the whole course was designed to enable teachers to develop their skills it was not possible to make the course materials easy to use as this was the vehicle for their development. The arrangement whereby teachers worked at computers also made interaction between teachers difficult. Sixteen respondents noted additions to the course that they would have welcomed. The most common request was for more time for hands-on work. This element of the course was discussed in the interview with the trainer and he felt the sessions were about right given the diverse needs of the teachers. This is confirmed by other feedback shown in Table 4.34. There were a variety of needs expressed with some teachers wanting more workshop time and some wanting the converse i.e. more explanation before hands-on.

86

Table 4.34 Teachers’ requested additions to course Category of response Number of responses % of respondents

More time for practising 5 31% More on CDROM (all ChemIT site on CDROM)

3 19%

Another session in 3/6 months 2 12% More structure and guidance before hands-on 2 12%

Other requests included more use of electronic whiteboard/tablets; details of where to purchase different types of electronic whiteboard hardware; and more help in getting the most out of resources.

Impact of the INSET – teachers’ prior practice and future plans The effectiveness of INSET is typically judged by evaluation feedback on the day. However the better measure of effectiveness is the impact on teachers’ practice and students’ learning. To judge the impact on teachers’ practice teachers were asked about their prior use of equipment and ICT resources, their use since the INSET and their planned use. In Table 4.35 teachers’ prior use and their planned use of the ICT equipment used in the workshop are reported. The trainer was of the view that this was available to about half of the group observed but this was not known for the other teachers who attended prior events. Table 4.35 shows that the figure of about half with prior access is true across the sample. The data projector was most commonly used prior to the workshop (55%), whilst only five teachers (18%) used an e-whiteboard and no-one had used a graphic tablet. The increase in use was relatively small whereas planned use showed a major increase in teachers’ intentions to use the whiteboard and the tablet. This supports the view that one of the major barriers to change in this area is availability of hardware. Table 4.35 Teachers’ prior use and planned use of ICT equipment

Whole class presentation equipment:

Used prior to workshop

No plans to use Started to use since workshop

Planning to use or develop

data projector 55% 7% 10% 28% e-whiteboard 18% 36% 4% 43% Graphics tablet 54% 4% 42%

In Table 4.36 teachers’ prior practice and plans for using specific ICT resources are reported. Teachers’ use of ICT resources since the INSET reveal the very significant impact of the INSET on teachers’ practice particularly in the number of teachers using resources since the INSET. The use of ChemIT materials or those developed from it had tripled with half the teachers planning to use these resources in the future. ChemIT materials based on MS-Office, which was a major focus of the course, showed the highest increase in actual use. 85% of teachers either were using or planned to use them compared with 14% using them prior to the event. A significant increase in the number of teachers using the RSC CDROMs and planning to use the RSC websites was a further outcome of the impact of the INSET. ChemSketch had a small increase in use since the INSET but had the largest planned use suggesting perhaps equipment and time constraints in schools.

87

Table 4.36 Use of ICT resources

ICT resources Used prior to workshop

No plans to use

Started to use since workshop


ChemIT materials based on MS-Office 14% 46% 39% other ChemIT materials 12% 8% 31% 50% materials developed from ChemIT 7% 11% 30% 52% materials developed in-house 48% 12% 40% other RSC materials: CDROMs, multi-media 55% 28% 17% other RSC web materials 39% 6% 17% 39% software outside of the workshop: 28% 44% 28% ChemSketch 21% 10% 14% 55% other proprietary materials 58% 21% 5% 16%

In the initial session of the INSET teachers were introduced to different ways that ICT use could be integrated into their teaching. Teachers were asked which of the ways they already used and which they were now using and planned to use (see Table 4.37). Table 4.37 How teachers use and plan to use ICT in their lessons

Used prior to workshop

No plans to use Started to use since workshop


'Starter' for lesson content 13% 20% 17% 50% Background (e.g. intro to new topic) 32% 14% 4% 50% Main Content of learning 28% 21% 52% Context for learning (e.g. video clip sodium)

21% 7% 18% 54%

Modelling 7% 25% 7% 61% General learning support 3% 21% 21% 55% Review of learning 29% 4% 8% 58% Assessment of learning 23% 19% 8% 50%

Prior to the INSET teachers’ use of ICT was across a number of aspects of a lesson. The impact of the INSET was to extend their views of the potential for its use as Starter activities, and particularly to teach science content (21%), to provide context (18%) and as more general learning support (21%). There was a significant increase in teachers’ actual use of ICT to support learning since the INSET. Teachers were asked to provide comment about their implementation of the INSET. There were twenty-nine responses, which were mostly positive describing what they had implemented and difficulties that were encountered, which largely fell into two main categories of shortage of equipment (48%) and lack of time (35%). Fourteen respondents (48%) described problems they had encountered with access to equipment, many had experienced problems using techniques from the workshop with their classes due to the shortage of equipment and competition for what was available.

‘Very inspired by workshop. On returning to school, finding time to follow up and develop ideas is always difficult. Technical support not strong.’

‘One data projector for use with the whole science faculty, requires forward planning.’

‘I have managed to develop quite a few materials. Time has been the main problem (though a grant from GTCW has helped). Also, resources are limited e.g. one data projector/one e-whiteboard per department.’

Some teachers commented on software problems:

88

‘Struggled to get the files to play in Quicktime, otherwise implementation very successful.’

‘Not been able to download “Sodium party” because of school network block.’

Two respondents felt they needed a little more hands-on training with IT generally, one of these had managed to secure funding for the time from General Teaching Council for Wales (GTCW).

Impact on students’ learning Over 90% of the teachers providing feedback about students’ learning reported positive impact of the implementation. This is a further impressive measure of the success of the INSET. Teachers explained why they thought learning was enhanced using ICT:

‘They like it. Experiments are quicker, more efficient (i.e. I give them something else for homework instead of just a ‘write up’).’

‘Sixth form have been guided to web sites and viewed CDROMs and have been enthusiastic.’

‘More involvement of students at the front of the lab, bringing them to the front to answer questions, manipulate information on the screen.’

‘Organic mechanisms PowerPoint presentation – students can access this at home from the internet and have found it very useful to be able to step through mechanisms at own pace and be able to replay them over and over.’

‘AS/A2 level – organic animated mechanisms very helpful. 3D representations of organic molecules using ChemSketch greatly helped visualisation.’

‘Mechanisms are very good – students find concepts much more tangible when animated using ICT.’

‘Investigations on the computer were successful, followed by plotting of data. Hands-on results for all is quick and can replace experimentation from time to time.’

‘… the majority of pupils’ learning has improved by seeing animations e.g. moving electrons in ionic bonding.’

Only two respondents had reservations about improvements to their students’ learning. One of these was surprised to find ‘some students resistant to using computers’. One teacher was interviewed who had implemented the INSET. The teacher was very positive about the INSET and its approach. He found the resources very interesting and the focus on adapting them to individual contexts particularly useful. He also valued being able to get a perspective on equipment needs. As he commented: ‘the requirement to have a fully functional interactive whiteboard is very much at the low end of what’s necessary to make big advances in the use of software.’ He explained how he had fed this back to his school

89

‘to highlight the fact that we don’t necessarily need to spend thousands of pounds when there are other technologies...if we’re looking at the best use of resources certainly putting in lots of projectors is what we want...’

He felt that the INSET had ‘put him in a more powerful position when it came to bidding for IT resources, which we’ve recently had to do.’ The other aspect of the INSET that he particularly valued was the ability to be part of a forum of practitioners for sharing ideas.

‘...to avoid reinventing the wheel and gets us away from reliance on the commercial software.’

Since attending the INSET he had started to use the www.ChemIT.co.uk website fairly frequently as a matter of ‘habit’. The teacher was the only full time chemist in the school but he had fed back to colleagues instructions for developing drop-down menus. He had also passed on to a colleague the Maxwell Boltzmann distribution in disseminating modelling changes using spreadsheets and had got positive feedback. He was recommending to colleagues that they use a PowerPoint presentation for the lattice energy areas of work. In his own work with KS5 groups he found the students’ reactions were very positive as the teacher explained, ‘they like to see links between things that are easy to understand.’ He found that the two teachers he had already involved in that were very enthusiastic about it as were the students. He had also used the resources at an open evening for parents and received very positive comments from parents who commented on how useful it would be for their children and how different to when they were taught. The teacher planned to use ChemSketch but had yet to find the time. He was also unable to use drop-down menus because of a lack of access to computers for all classes. He identified barriers to his wider use of the resource that overlapped in part with teachers’ comments about the multimedia product.

‘There’s some limitations on implementation and actually getting things enshrined in schemes of work depends on equal access to the equipment. And with the best will in the world we can’t re-timetable or do lab swaps every time someone wants to use projector.’

With respect to equipment for students to make wider use of the resources he referred to the limitation of only two IT rooms. He also referred to a trolley of laptops, which were radio linked to the school network. This was to allow students shared access to a laptop in class. However, the teacher pointed out that ‘the sort of software we tend to want to use over the radio link simply takes far too long to load.’ He was however confident that he could and would find the time to push the implementation to the ‘next level’ by which he meant to extend his own and his colleagues use of the INSET resources.

Teachers’ views of future needs to sustain implementation Table 4.38 shows teachers’ views of future support needs.

90

Table 4.38 Teachers’ views of needs to sustain implementation Further support needs

Number of responses

% of respondents

See more examples and updates 6 26% Time (to practise) 6 26% More hands-on training on specific topics e.g. developing animations, drop down tests, interactive worksheets (develop a pre-planned finished product)

5

22%

Development of web resources for student access 3 13% Use of interactive whiteboards and graphics tablet 2 9%

Teachers were asked what further products or materials they would like made available through ChemIT. Sixteen teachers responded to the question and common examples of responses are listed below:

‘More of the same, i.e. material to be used in lessons’

‘Material for use on interactive whiteboards KS3, KS4’

‘Lesson starters – short introductions, things to make children think’

‘Things’ that bring ‘real Chemistry’ into the lab’

‘Would it also be possible to get some of the ‘misconceptions’ stuff in this? The students like this sort of thing’

‘Mechanisms – AS/A2 3-D shapes – structure e.g. tetrahedral orbitals etc’

‘Revision/review activities that really test Chemical understanding’

‘Just extend range of materials – keep it up’

‘More video clips. More of the same – hopefully it will expand rapidly as teachers post ideas onto it’

Eighteen teachers responded when asked what additional product, INSET or support would be beneficial in supporting teaching and learning in chemistry generally. Some of the responses were already addressed by RSC educational activities and products. The responses that RSC could respond to are listed:

‘Continuing support & encouragement essential’

‘More resources!’

‘More examples in PowerPoint’

‘More staff to attend such courses’

‘The instruction booklet we received should be made generally available’

‘Short, snappy demos’

‘Electronic assessment and revision material’

‘More video clips showing practical work for the (A level Edexcel (London) and GCSE AQA double science) syllabi and more exciting experiments that would interest this age group’

‘The support provided to date is very useful and beneficial to students’

91

4.4.2 Using Assessment to Improve Learning in Chemistry and Science Departments 24 teachers attended the course that was observed. These teachers and teachers attending three other courses were sent questionnaires (N=58) there was a 59% return. 56% were RSC members. 67% taught across the 11–18 age range. 10% taught only KS5 students. The teachers came from 5 LEAs in England and Wales.

About the course The course draws on the research of Black and Wiliams at King’s College, which is now enshrined in national policy across the curriculum in England. The research had involved science and maths teachers in developing ways of assessing students’ learning and was based on an action research model. The course was one of the first to be developed from the research and is possibly unique in providing INSET for chemistry teachers in particular on this topic. The course introduces teachers to a new way of thinking about teaching and learning and the role of assessment in that process. As the trainer described it: ‘The day is as much about techniques as it is about changing the school culture.’ The following learning outcomes were provided in the handouts: By the end of the workshop you will be able to: - Explain the difference between assessment for learning and summative

assessment - Identify four marking techniques which are most effective in helping pupils learn - Define and give examples of a ‘good’ question - List at least three strategies for creating a climate and a culture which underpins

thinking and learning in response to questions - Explain how self and peer assessment develops improved learning skills - Identify at least five strategies for using self and peer assessment to develop

learning skills - Formulate an action plan to experiment with some of the ideas from the workshop. As with the other INSET workshop there were two aims which addressed teachers’ own professional development and support for students’ learning. Another common element was the focus on implementation back in schools. The day was devoted to introducing the teachers to techniques that transform the role of students in the learning process. The trainer’s approach was to use the techniques with the teachers in order to model them in practice. The day began with using one of the techniques to establish what teachers understood about key forms of assessment, summative and formative, the latter being the focus of the day. Teachers had to diagnose the state of their understanding using traffic lights where green represented full understanding. At the end of the day they were to reconsider the ‘colour’ of their developed understanding. The trainer used a sophisticated approach to the issues, for example in the first session he chose a task where the content and issues were central to developing assessment for learning, that is ‘What is a good question?’ and ‘How can you set up an environment to help children participate?’. Teachers working in groups had to prepare two OHTs on this. He then asked teachers to engage with this using a new assessment

92

technique that was one of the learning objectives of the course. This technique ‘one +one’ required a group of teachers to review the presentation of another group and identify one thing that they learned and found interesting and one possible improvement. In the plenary session the trainer summarised the teachers’ feedback and gave pointers about providing effective feedback. This session was followed by a video clip of a teacher interacting with a whole class in chemistry. The teachers were to consider themselves as a mentor to the video teacher and identify one piece of advice they would give him about his way of providing feedback and the nature of the feedback. Again the trainer gave teachers a strategy for this and modelled it in his feedback to them. The next session moved on to consider effective marking and the final session focused on peer and self- assessment strategies. The examples used were from chemistry lessons and students’ work in chemistry. In the final session where teachers developed their action plans the trainer asked them to think in terms of low risk change and recommended the approach of ‘experiment, expand and embed’ to allow incremental change and progress in their teaching. The trainer explained how he tried to meet the needs of individual teachers through their action plans:

‘I was really saying, “we’ve picked up all this learning about questioning, so how are you going to make it meaningful to you which is different to everybody in the room?” So I was just using that form around: “this is what you currently do, this is what you’re going to do in the future,” and they were writing that down, I wasn’t telling them – there’s no right answer.’

He was pleased with the workshop, which was very lively and engaged all the teachers throughout and was very interactive. He said he judged effectiveness by observing the teachers:

‘I suppose I was quite positive with their reaction for today – they were involved – and sometimes you just look round and try to get the temperature, which for me is how much energy is there. So when they were doing that little bit of pair work the noise level wasn’t tremendously high but it was they sort of…I don’t know how you describe it, but they were really interested in what they were talking about and keen to explain to people around. So you look for signals like that I suppose really and try to get it relaxed, get a bit of humour in and things like that – I mean it’s gut feeling really.’

We asked what he saw as barriers to an effective workshop such as he ran:

‘I think the biggest issue in any sort of training is attitude and people’s ingrained habits and things like that. So in a way the INSET that we were in here people tend to come because they want to, a few come because they’re being sent. Of course that’s where it’s most difficult because if people come because they want to, they see a reason and they want something back for their time, but if they’re being sent then there are other things to consider. What I really know is if we’re going to make an impact then there needs to be somebody back at the ranch co-ordinating and taking off from what we’ve done. And because there are things like the Key Stage 3 strategy in state schools then we’re hoping we’re contributing to that. In fact all we’re doing is

93

presenting the same messages in maybe a slightly different way but for chemists and chemists working together but supporting what’s happening nationally.’

Feedback from the Questionnaire Before considering the feedback it is worth reflecting on the different challenges that the two INSET events presented teachers with. In the first the challenge was to have the skills to access the workshop materials and apply and develop them in situ nevertheless the teachers had elected to go to the event and therefore had some expertise already. The materials were all designed to meet very specific needs that might be innovative for teachers but would be in a context that they could readily relate to and make links to their curriculum. In contrast assessment for learning is a new concept for most of the profession and challenges teachers’ current ways of thinking about teaching and learning. The techniques have general application across the curriculum so teachers face considerable challenges to personalize their learning and connect it to their teaching contexts in chemistry and science.

Teachers’ views of the effectiveness of the course and assessment issues and techniques used The assessment workshop was very highly rated by almost all respondents (94%) who judged that the workshop met their assessment needs well and very well. Respondents found the approach of the workshop to model assessment issues and techniques for improving learning extremely successful (88%). Respondents reported that they felt well prepared to experiment with assessment with their class (88%) and that they also felt well prepared to develop assessment for learning with colleagues (82%). Table 4.39 Teachers’ ratings of the effectiveness of the approach to, and components in, the workshop

Very Not at all 1 and 2 3 4 and 5

How successful was this for you? 88% 6% 6% How helpful did you find the summary OHT ‘How to experiment’? 69% 24% 7% How well prepared do you feel to experiment with assessment with a class? 88% 12% How well prepared do you feel to develop assessment for learning with colleagues? 82% 15% 3%

In the workshop a number of different assessment issues and assessment techniques to improve learning were explored. Teachers’ ratings of the usefulness of the assessment for learning issues covered are shown in Table 4.40.

Table 4.40 Teachers’ ratings of usefulness of assessment issues covered Very Not at all Assessment issues 1 and 2 3 4 and 5

Questioning – using questions 88% 12% – generating questions 82% 12% 6% – pupil generated questions 78% 13% 9% Establishing a supportive environment 70% 24% 6% Marking – focused marking 91% 9% – use of praise 88% 13% – ‘comment only’ marking 91% 6% 3%

‘Comment only’ marking represents a major shift in the way feedback to students and how they learn effectively is viewed it is therefore very impressive that this

94

challenging aspect of the INSET received the highest rating (91%) with the majority of teachers rating it as very useful. The majority of ratings were high and where these showed a slight decline they were where teachers felt they had already established certain practices such as a supportive learning environment. The majority of respondents considered all of the assessment techniques used in the workshop useful. However, peer assessment and self-assessment techniques were rated extremely highly with 94% of respondents finding peer assessment very useful and 88% of respondents finding self-assessment very useful. Table 4.41 Teachers’ ratings of usefulness of assessment techniques

Very Useful

Not useful

Assessment techniques

1 and 2 3 4 and 5

Traffic lights 71% 21% 9% One +one 78% 22% Experiment, expand, embed 82% 18% Listen, probe, summarise 84% 16% Dealing with ‘wrong’ answers 82% 18% Ground rules/principles (e.g. allowing pupils to say ‘pass’; rewording questions) 82% 12% 6% Peer assessment (e.g. presentations; group marking; jigsaw approach) 94% 3% 3% Self assessment (e.g. finding answers to test questions; marking own work, etc.) 88% 9% 3%

Impact of the INSET – teachers’ implementations and plans Five teachers reported that they had used some or all of the techniques before the INSET, three of these described the workshop as providing greater clarification for them of the techniques:

‘I have a clearer view of my role and the pupils’ role in assessment. It has consolidated some of the views I had been developing regarding this. It has given me a greater impetus to introduce peer and pupil self-assessment.’

Teachers were asked to indicate which techniques they had implemented since the INSET. Table 4.42 below shows that over a third of respondents (38%) had implemented ‘comment only’ marking. Nearly a third had implemented peer assessment and group work. Overall there had been a significant shift in teachers’ practice in response to the INSET. Table 4.42 Techniques implemented

Implemented since the INSET % of respondents

‘Comment only’ marking 38% Peer assessment 31% Group work 31% Self assessment 19% Traffic lights 19% Focused marking 12% Questions for topics 8%

Sixteen respondents included information about the year groups they had implemented workshop techniques with, Table 4.43 shows there was a range of techniques used with different years indicating their individual preferences.

95

Table 4.43 Age group of implementation by technique Year Techniques implemented

Years 7, 8 and 9 Traffic light Peer assessment Group work ‘one+one’ Years 10 and 11 Traffic light Peer assessment Self assessment Presentations Comment only marking Year 12 and 13 Traffic light Peer assessment Self assessment Topic test marking

Plans for developing assessment for learning Teachers provided information about their plans for implementing assessment for learning in their schools (see Table 4.44). Table 4.44 Teachers’ plans for implementation

Plan to implement Teachers %

Peer and self assessment 40% Traffic lights 30% ‘Comment only marking’ 30%

Impact on students’ learning The questionnaire asked teachers to comment on the impact of their implementations on their students’ learning. Twenty-four teachers described impacts on their students’ learning, all of which were positive and are summarised in Table 4.45. Table 4.45 Nature of impact on students

Impact % teachers

Increased pupil motivation and enjoyment 23% Pupils act on comments, willing to discuss 12% Pupils enjoy the extra involvement 23% More work based discussions/peer explanations 7% Pupils question well 7% Instant feedback to pupils 7% Increased pupils’ independence ‘ownership’ 7% Pupils’ more aware of assessment and learning process 7% Traffic lights helps me 7%

Many teachers described successes they had experienced in implementing their learning from the workshop, one respondent had received an ‘excellent response from pupils’, while another had been ‘surprised how many pupils didn’t ask for help’. The table indicates the significant change that the implementation had on the teacher-student relationship with the assessment for learning techniques shifting responsibility for learning towards the students yet at the same time increasing their motivation and enjoyment. Seven teachers who were at the planning stage of implementation described their hopes for an impact on students: a general increase in learning (3), the greater involvement of pupils in their learning (2), an improvement in test scores (1), and pupils taking more note of comments (1). Four teachers described how their implementation plans were working with different ability groups. One had found the techniques worked well with a highly able group ‘after an initial reminder to be positive in criticism’. Another found the techniques worked well with an average ability group. One respondent described success across all abilities and ages, whilst for another the techniques ‘… raise achievement in the weaker end of the class.’

96

Two teachers described problems they had encountered: one felt that younger pupils lacked the maturity for peer assessment; another considered that the ‘one+one’ technique and self- assessment were not effective in making clear where the pupil had difficulties. Difficulties experienced by teachers in their implementation were largely described as pupil resistance to comment only marking (5), lack of time particularly with exam pressures (2), and the tension between ‘comment only’ marking and having to include marks due to school policy (2).

Impact on colleagues The majority of teachers had plans to work with colleagues since the workshop. 50% had already done so: the most frequent form of dissemination was at departmental level (8), although some were working directly with individual colleagues. Several teachers (6) planned to trial the techniques themselves before going on to share with colleagues. Some (3) complained that there was just not enough time at departmental meetings to provide feedback from the workshop.

Teachers’ views of future needs to sustain implementation There was a range of additional support requested by teachers. Many would like a resource book or materials with examples of techniques and methods. Others would value ideas for marking opportunities; yet others wanted examples of good questions; and good comments. Three teachers thought they would value more networking to share good practice, ideas and experiences. The location of the workshop would help one respondent who felt workshops in North Ireland would be helpful, while another hoped for workshops to be held in Devon and Cornwall. RSC already plans to hold INSET in Northern Ireland in 2005 and therefore this need is met. Table 4.46 Teachers’ views of needs to sustain implementation

Future support needs

Number of responses

% teachers

Resource book of materials, OHTs etc with examples of techniques and methods with ideas for marking opportunities, good questions in science and answers, and good comments 8 44%

Further ideas and expansions 3 16% Experiences of colleagues who had already implemented assessment techniques 2 11% More networking to share good practice, ideas and experiences, website with examples and workshop 4 22%

Location: Northern Ireland, Devon and Cornwall 2 11%

Most teachers (97%) said they would find a book on assessment for learning that included techniques and examples of pupils’ and teachers’ work in chemistry useful. The majority (71%) expressed an interest in a further workshop extending techniques for peer and self-assessment.

Feedback from interviews and observations Three teachers were interviewed and one agreed to let us observe a lesson and talk to students. One teacher was interviewed because she was the only teacher who considered that the INSET did not meet her needs ‘at all’. Her reason for going to the INSET was because of ‘the not very good overall results in last year’s GCSE Double Award.’ She went to the INSET to address a very specific ‘problem’ to ‘find out how to make Chemistry/Science more successful for students.’ She interpreted the

97

workshop as concerned with the D/C boundary, which was to quite misunderstand the philosophy of the course and its goals. Her interest was the A/A* boundary. From this teacher’s perspective many of the techniques were inappropriate for very able students; even though she had not implemented them she chose to reject them on these grounds. As she explained in relation to using the ‘traffic lights’ technique, ‘As it is a selective school and this approach with boys would make me a laughing stock and my subject too’. She was interested in the different approach to, and techniques for, marking and feedback covered in the workshop and discussed the possibility of implementing alternative approaches with her department. However while she was given the go ahead to do it herself she had to maintain her existing practice and mark books alongside this and so she did not progress the plan. As the trainer commented earlier, the successful impact of the INSET depends crucially on support back in school. This teacher appeared to have no support to even begin to change practice. An almost polar opposite view came from the second teacher interviewed. She too taught at a selective school, a grammar school. She described how the school became involved with the original research to develop assessment for learning techniques. So the initiative to develop assessment for learning was very well supported in her school at senior management level and across the curriculum. She also had experience of trying out techniques. She gave her reason for going to the INSET as, ‘Yes well my biggest question, I wanted answered by that course is, am I doing things right? Because I’d already done it by then, ... and I wanted some feedback to me on whether what I was doing with the class was right, or whether it was, you know, okay, or whether it was just – I just didn’t have a clue.’ She had tried out the techniques with Y8, 9 and 10 students. She had tried in particular the ‘comment-only’ marking and she had some concerns. ‘I often tend to find that I spend more time feeding back than I do actually teaching new work, and that’s an issue for me. By the time I’ve given them their exercise books back, done some questioning on the homework, gone over what was good and what needed to be improved about their work, I’m like thinking, hold on, I haven’t got any time to do practical now.’ She felt her most successful implementation was with Y10. She was surprised that the students did not question the absence of marks in their exercise books. As she commented:

‘I was quite impressed with how Year 10 took to it, so Year 10 at the moment have probably got about one third of the marks from this term that they would normally have in their books, but nobody’s said anything, no parents have mentioned it, and they’re doing equally as well. I think, as I say, some of them, because it’s not switching them off, it keeps them going and motivating them. The biggest thing I found, with my tutor group as a whole, is it doesn’t switch off the lower end of the group.’

She described how she used peer assessment with her Y8 group:

‘I think I’ve become very aware of over the last few months, how they never get to see anybody else’s work. If you’re working in complete isolation all the time you’re only going to see your own work, and your teacher’s continuously

98

saying to you, well you could have done better. How do you know, how you should have done better? So I gave them each other’s book, and I said, what I want you to do is read through their explanation at home, the method to separate sand from salt and find those six words. If you can find the words and you think it’s used in the right context, can you highlight them with, you know, tick it, underline it, circle it. You decide. If you don’t think it’s been used in the right context, can you write a comment at the bottom. And then, what I’d like you to do is at the bottom, write a list of the words you can’t find, and write some comments about the work. Now if you can think of anything good about the work, then write a comment on what’s good about it, very good diagrams or labels, various other things. But then if you can think of a comment about the way the work could be improved, then write comments about that. And I was amazed. They were all busy giving advice – and they, they seemed to enjoy it.’

She found that the INSET helped clarify for her the purpose of such techniques as ‘comment-only’ marking. ‘I needed somebody to turn round to me and convince me...He [the trainer] was brilliant at that. I walked out of there feeling really confident.’ The third teacher we interviewed taught in a rural comprehensive. The school had already had a whole day INSET session on assessment for learning. He described the way the school-based INSET ‘focused ideas’ but did not lead to significant change in the classroom in contrast to the RSC INSET.

‘I have to say that the Inset that I went on from the Royal Society of Chemistry I really did focus my ideas from that. So the general Inset was an interesting okay, there’s a lot to take on board here and perhaps I’ll try one or two things. I think the Inset I went on in December [the RSC workshop] really crystallised those thoughts and hit on very similar points but it was more focused, you know.’

He felt that the initial INSET just raised his awareness whereas he described the RSC INSET as ‘very effective in making me think, reflect on my teaching and on what I was going to do next. Because I always like to do something that I know I can implement in the classroom and I think this one was very much like that’. He had been experimenting with a range of assessment techniques across the Y7–Y13 age range. Before the INSET workshop he had been familiar with and had used traffic lights as a technique but since the INSET he had used the ‘one+one’ technique and ‘comment-only’ marking as well as peer assessment. He had taken to heart the strategy – ‘experiment, expand and embed’ recognising that the approach to assessment in the INSET represented ‘a massive shift’ and commented:

‘I mean I’ve got classes that it works very well with but I’ve got classes that really struggle with the formative assessment. I’ve got a Year 8 class that are pretty lively and they really find it tricky. They’re the sort of class that if you give them a result, okay then, what’s next and they don’t want to say well what did I do wrong there or what did I miss out, how can I improve this. They’re not good at that at all... .

I’ve got some classes that certainly in my Y7s who don’t know any different, who almost expect, well they don’t expect a mark in the book because I don’t

99

put marks, but they almost expect to have something to do at the start of the lesson to change or add to the homework that they’ve handed in. It’s almost automatic.’

The benefit of the approach was for him the impact on students’ learning:

‘And it’s great to see them think about it and want to know what it is before moving on to something else, and at least they know what they need to know. Because how can people take responsibility for their learning if they don’t know what it’s supposed to be? And I’d like to think that perhaps not every single student that I teach but a lot of them have got a very good appreciation of where they are and what they need to do next. And that’s what I’m trying to do more and I think this is where the INSET helped me a lot, you know.

I was thinking well hang on when I sit down and spend two hours marking a class set of books, am I wasting my time, and I think that’s one big thing came up in the talk, someone said well is it okay to put a tick on books, what have you. He [the trainer] said well are you putting things on books that the students can interact with and can do something with? And you know I’m thinking actually sometimes no. And in a way it’s not that worth it, I mean I’d rather mark the books less, and perhaps when I do mark the books add those sort of things on there.’

The teacher was of the view that he could only do so much himself and that ‘ultimately if anything’s going to happen then you’ve got to have your colleagues on board.’ He was pleased that a lot of things he had brought back from the INSET had been taken up by colleagues and tried out. He described the impact of the INSET on his own practice: ‘I think as a result my lesson plans are more specific in terms of last lesson we learned that, this lesson we’re going to look at, and then perhaps explaining how they will be successful in that. So by the time you leave this lesson you will be able to explain to me and at the end of the lesson coming back to that point and saying right what have you understood from this lesson.’ For his students he thought the impact was that they were improving in their results but more importantly he thought the techniques were ‘helping them to think.’

The observed lesson A Y7 lesson was observed which had assessment for learning techniques integrated throughout. The lesson began with the students being set their homework for the next week in which they had to reflect on and summarise in eight points their learning of the last two topics on Forces and the Solar System. There was a discussion of this that involved the students as they asked questions about how to approach the task and how to present their reflections. The teacher then gave out the students’ portfolios on Forces with his ‘comment-only’ marking on how to improve the work and the students were given an opportunity recognise the improvements that had been made. The main task for the lesson was a peer assessment exercise. The students were given other students’ work to assess. The work required the students to plan a mission to the planet Mars and design a craft to take them there. The learning objectives for the peer assessment task were to increase awareness of the importance of the Earth and key issues surrounding a plan to visit Mars; and to improve pupils’ ability to perform peer assessment. The students had criteria to consider and in groups had to discuss the

100

work and generate comments about what was good about it and how it could be improved. The teacher took the students through the criteria and gave examples of what he was looking for in their responses to the work. The students soon became engaged in the task and were discussing the work seriously. A questionnaire was given to the Y7 students (N=30) to see the extent to which principles of assessment for learning had begun to be understood by them and what their views were of the techniques used by their teacher. We were mindful that the change in ‘habits of mind’ and students’ understanding anticipated was major and also that the teacher had only been working with the techniques for about two terms. Tables 4.47 to 4.59 report the students’ views. Students were asked which ways of assessing their work were the most common. It is clear from Table 4.47 that tests are still seen as the main way that students’ work is judged. However students were beginning to recognise the impact of assessment occurring within their learning and by themselves and their peers. Table 4.47 Y7 students’ views of the ways science work is assessed

Tests Homework In class by the teacher In class by myself In class by other pupils

90% 43% 30% 23% 23%

In Table 4.48 the students reported their views of which were the best ways of having their work assessed, and this indicates that students still recognise the teacher as having authority over outcomes. They also note their dislike of tests. Table 4.48 Which ways Y7 students think are best for themselves

Tests Homework In class by the teacher In class by myself In class by other pupils

23% 50% 53% 17% 17%

In interviews with four of the students they explained why they thought this.

‘I think it’s better for a teacher to mark it. I do sort of enjoy marking other people’s work but if a teacher marks it I think teachers know a bit more than pupils and can write better improvements and pick out more detail.’

‘I think it’s okay. I’d rather the teacher would mark the work because I don’t think any pupils are like qualified science teachers. So I think it’s better if Dr Ferguson marks it. But if we mark it it gives us a chance to reflect on other people’s work.’

Students were aware of the new ways of being assessed but saw it not as a routine practice however they clearly experienced a difference between how they were assessed in science compared with other subjects (see Table 4.49). TheY8 class was given the same questionnaire and an increasing number of the older students (38%) considered that these forms of assessment occurred in most or every lesson in science, rarely in other subjects.

101

Table 4.49 Y7 students’ views of how often they do self- and peer assessment in science

Subject Every lesson Most lessons Sometimes Not often Never

Science 13% 73% 13%

Other subjects 10% 40% 47% 3%

Whilst only about a quarter of students considered the techniques enjoyable very few reported them to be not enjoyable, which in itself is a surprising finding about any assessment. Table 4.50 Y7 students’ views of how enjoyable these forms of assessment are

Very enjoyable Enjoyable So-so Not enjoyable Awful

13% 10% 70% 7%

A significant measure of the impact of the INSET is the extent to which students understand and experience the new assessment techniques as benefiting their learning. In Table 4.51 below over half the students reported these benefits. Students’ responses reveal that they have a healthy cynicism about teachers’ agendas. However when their responses are examined across the different purposes given a different picture emerges. Over a third of the students considered the purpose for the new techniques were to understand science better, what makes good work and what they were supposed to be learning. A further 30% recognised it was to understand science better and 26% of those students cited either to understand good work or to understand what to learn as well. These indicate that a shift in perception is occurring for these students. There were about ten students who cited saving the teacher’s time and that they were not sure what the purposes of the new techniques were. The Y8 group of students expressed similar views. Table 4.51 Y7 students’ views of the purposes for self and peer assessment

To save the teacher’s time 63% To understand the science better 60% To understand better what I’m supposed to be learning 50% To understand better what makes good work different from not so good 50% To understand what I have to learn next 7% Not sure 23%

Students were asked how helpful they found the techniques and nearly half said they were helpful (see Table 4.52). Table 4.52 Y7 students’ ratings of how helpful self and peer assessment is

Very helpful Helpful Not sure Unhelpful Very unhelpful

3% 45% 41% 10%

In tables 4.53 and 4.54 below students’ views of the difficulty of self-assessment are shown. Quite a high proportion of students consider that the technique is not difficult. Those who experience difficulties identify the three aspects that they struggle with – finding what is good about their work, how it could be improved and how to use their

102

marking to improve it. Students seem more aware of what is not good about their own work. It is to be expected that expertise in marking would be something that students would develop over their years in school and given that about a third of students were not sure of the purposes served by self-assessment then their difficulties in carrying it out make sense. Table 4.53 Y7 students’ ratings of the difficulty of self- assessment

Very easy Easy So-so Difficult Very difficult

3% 38% 55% 3%

Table 4.54 Y7 students’ views of the difficulty of self-assessment

Very easy

Easy So-so Difficult Very difficult

Knowing what is good about it 20% 28% 24% 24% 4% Knowing what is not so good about it 20% 24% 40% 8% 8% Knowing how it could be made better 28% 12% 28% 20% 12% Using your marking to make it better 24% 20% 24% 24% 8%

As would be expected the students’ views of the ease or difficulty of self-assessment influences how helpful they experience it to be for their learning (see Table 4.55). More Y8 students (50%) than Y7 considered self-assessment to be helpful or very helpful for their learning. Table 4.55 Y7 students’ views of the value of self-assessment for their learning

Very helpful Helpful Not sure Unhelpful Very unhelpful

7% 32% 50% 7% 4%

The teacher had used peer marking more regularly as a teaching and learning technique. Students were asked about how easy they found this technique and what the sources of their difficulties were. Their views are reported in Tables 4.56 and 4.57. Table 4.56 Y7 students’ views of the difficulty of peer marking

Very easy Easy So-so Difficult Very difficult

21% 17% 55% 7%

There is an increase in the number of students reporting finding this technique very easy compared with self–assessment but similar proportions of Y7 students report experiencing difficulties with both self- and peer assessment (see Table 4.57 below). Students in particular do not find it easy knowing how to give feedback to their peers. In the lesson observed the girls marking a boy’s piece of writing did not know what to say that was good about the writing:

Okay. He hasn’t really used very scientific words. He hasn’t made a point why there is actually life. He just said earth is special because there’s life on it.

103

And he’s not using scientific words.

He is here, he is being like, he is saying kind of why it is but he’s not explaining it that well. So like, it is sort of like, it is sort of working because he’s saying why earth is special, but he’s not saying why. And not explaining and using scientific words.

Also how’s he going to go on Mars? Is he just going to park the spaceship and walk around? Is he going have like a buggy thing or something.

A good thing, let’s put the plus is it looks pretty, the decoration.

It’s not coloured.

Oh yeah. Let’s put he didn’t decorate it properly. [Laughter]. Okay.

I’m struggling to find something good about it.

It’s not like as detailed as it could be. And it hasn’t got any information off the sheet.

Yeah. He’s got loads of information but it’s not really that well researched.

Yes. It’s good information but he doesn’t like do all the points to like explain why you’re going to Mars.

He hasn’t said about Mars’ environment. Give it to E she’s good at finding positive things.

I can’t actually find something that good about it.

I know. I can’t find good points.

Shall I put decoration …?

No, because it’s not about decoration. Decoration won’t get you points. In interview the same boy commented on what he thought about the feedback he received. He was not aware of the girls’ discussion.

‘I didn’t find it very helpful because girls were marking mine and they said good pictures, very good illustrations, but it wasn’t really much about the work. I didn’t really know how I did on my writing. I thought if a teacher was marking it they would’ve focused on the writing more than the pictures and illustrations.’

The girls had considered the writing from a science perspective but had felt that positive comments should dominate the feedback. You can easily imagine how from the boy’s perspective stereotypes about how girls’ are interested in aesthetics might influence how he interpreted the feedback and indeed confirm the belief about girls’ concerns. Developing effective feedback was one of the major learning objectives for teachers in the INSET so it is not surprising that students struggle with this. Teachers also asked for examples of ‘good’ feedback. This small exchange indicates that students too would benefit greatly from such examples.

104

Table 4.57 Y7 students’ views of what makes peer marking difficult Very

easy Easy So-so Difficult Very

difficult

Knowing what is good about someone’s work 19% 35% 31% 11% 4% Finding something that could be made better 11% 31% 27% 31% Writing your comments on what is good about it 11% 31% 31% 19% 8% Writing your suggestions on how it could be made better 11% 15% 46% 12% 16%

Tables 4.58 and 4.59 show students’ views of the benefits of peer marking. One student described how to identify a good point:

‘You pick out good words and information out of other people’s writing. Some people, because like some people have really good ideas.’

An impressive proportion (52%) consider that being a marker is helpful for their learning and a slightly smaller proportion (37%) report that having a peer mark their work was helpful too. This proportion rose to (62%) for the Y8 students. Students in interview explained how they saw the benefits:

‘Well it gives you the chance to learn what other people know and what, if you didn’t get that question you can see what that person wrote and you can learn off marking someone else’s.’

‘Probably you get to know other people’s working level maybe or to help you understand it a bit better than you did in the beginning.’

‘People can often learn off you.’ Table 4.58 Y7 students’ views of the benefits of peer marking

Very helpful

Helpful Not sure Unhelpful Very unhelpful

How helpful do you think your marking was to the person whose work it was?

4% 18% 67% 11%

How helpful was it for your learning to be a marker? 4% 48% 41% 7% How helpful did you find someone else’s marking of your work?

4% 33% 48% 11% 4%

One of the impacts of being a marker in peer assessment is that students begin to get a better perspective on what constitutes good work in science. This seems to be an emergent phenomenon for the students as the majority report understanding just a ‘bit better’ what is good work. There is a shift in their understanding of their science learning but a more limited one in comparison (see Table 4.59) Table 4.59 Y7 students’ views of peer marking for their science learning A lot

better A bit better

No difference

A bit worse

A lot worse

How much has it helped you to know what is good work in science?

7% 82% 11%

How much better do you understand your science learning since you and other pupils have started marking?

60% 36% 4%

Overall the INSET events were exceptionally well received by teachers who considered that their needs were well met they also felt well supported to implement change back in their schools. The trainers’ goals for the events corresponded closely with teachers’ views of the strengths of the events. The impact on teachers’ practice

105

was significant for both events with teachers reporting that they had already implemented changes to their practice and were planning more. The majority of teachers considered that there had been a very positive impact on students’ learning. The case study observations confirmed significant shifts in students’ understanding, motivation and enjoyment since the changes in teachers’ practice.

4.5 Web-based product

4.5.1 Joint Earth Science Education Initiative (JESEI): This section reports on the evaluation of the JESEI website www.chemsoc.org/networks/learnnet/jesei. This was developed as a joint initiative by the RSC in conjunction with the Institute of Biology, the Institute of Physics, the Earth Science Teachers’ Association, the Royal Society, the Association for Science Education, the Geological Society and the UK Offshore Operators Association. JESEI is aimed at teachers of 11–14 and 14–16 age groups. The website ‘aids Chemistry, Biology and Physics specialists with their teaching of Earth Science by providing material within each science specialism’. The site offers support and resources to help teachers teach Earth science well and with enthusiasm, to bring their ‘Earth science teaching to life’ and to ‘highlight the relevance and interest of Earth science’ to their pupils. It contains ‘hands on, interactive practical activities’ for use with students and materials to help teachers ‘build up [their] own background knowledge’. The effect of these is intended to be ‘new approaches’ that will ‘revitalise [their] Earth science teaching’ and ‘give new dimensions to [their] teaching of science’. The main content is a set of downloadable teaching resources. This is supported by a glossary of terms, which can be accessed by clicking on key words throughout the text or directly from the resource list screen. The contents of one resource package, ‘Amazing Earth: facts that fascinate’, also appear at the bottom of the introductory page and facts can be scrolled through one at a time. There are also acknowledgements, feedback and ‘useful websites’ pages. There are 43 items or resource packages available. Each has a common structure which includes: an introductory page with title, learning objectives, English and Welsh National Curriculum links, (links to the Scottish curriculum are available through an alternative contents list) to exam board specifications for 14–16 material, and a time estimate for the activity. An introduction to the activity and cross-referencing to other related JESEI activities follow this. Then there are Teacher’s notes that follow a standard pattern with the following headings: Level; Topic; Description; Context; Teaching points; and Timing. Student materials consisting of worksheets, instructions for activities, question sheets, etc are available in Word or pdf format, or are directly printable. The final element of each resource is a glossary. Subject-specific words, which may be new to students or need their full understanding to fully appreciate the text, are explained in the glossary. Throughout the text these words are highlighted and can simply be clicked to bring up the full definition. At the bottom of each student worksheet a selection of relevant words from the glossary and their explanations are available.

106

Feedback from the questionnaire Teachers were invited to be involved in the evaluation of JESEI via the website itself. The response was relatively low (N=32) and these teachers were sent two mailings. The return rate of 28% was far lower than for any other product. These teachers responding all rated the product as very high quality, very useful, clear and easy to access. The rating for needs met was the lowest of all products and we have discussed how this might reflect teachers’ familiarity with the website and their technical skills. It is also a small sample compared with the others. Teachers were asked their reasons for choosing the product. For two teachers it was a matter of personal choice. Two-thirds of teachers chose the product for their personal needs and those of their students. Teachers were asked why they chose JESEI rather than other resources.

‘I have not yet had time to fully investigate other resources, but I had put this one on my laptop, so it was there when I needed it ...CDROM is easier to access than the web...I can look things up when I’m next teaching an earth science topic.’

‘They are better than the other resources I had access to.’

‘Needed some good ideas to teach the subject and not being a geologist/earth scientist did not have a clue. Came across the JESEI resource first in CDROM form. I was given a copy by a colleague who obtained it from the ASE conference.’

‘I found the ideas were new and exciting engaging students in their work and the subject.’

‘There is plenty of information both for teachers and students to use. Explanations are detailed and easy to follow.’

The comments give further support for the finding reported earlier that teachers like products in multiple formats and CDROMs are particularly valued.

The topics used with which age groups Data from a one-month period of the number of hits on the RSC websites indicate that JESEI is a popular site with nearly 1000 requests for the glossary and around 500 for the index and home page. Hundreds of teachers searched resources such as The chemistry of limestone, The rock cycle and Amazing Earth: facts that fascinate. The teachers were asked about their use of resources and indicate a widespread use of the resources (see Table 4.60).

107

Table 4.60 Teachers’ ratings of their use of different topics Age group Topic Regular

Use Occasional Use KS3 KS4

1. The age of the Earth: can you believe everything you read? 33% 100% 2. The age of the Earth: the changing views of science 23% 11% 100% 3. Amazing Earth: facts that fascinate 23% 33% 40% 80% 4. Carbon cycle: exchanging carbon dioxide between the atmosphere and

ocean 11% 100%

5. Carbon cycle in the lab: carbon products and the processes that link them 11% 100%

6. Carbon cycle: releasing dinosaur breath in the lab 11% 11% 100% 7. Carbon cycle: where is this crucial carbon? 11% 100% 8. The carbon dioxide greenhouse – is it effective?: a lab ICT test 11% 100% 9. Chemical weathering of limestone: my breath and rock chippings from

the car park 11% 11% 50% 50%

10. The chemistry of limestone 23% 23% 100% 11. Crystal size and cooling rate: fast and slow cooling of lead iodide 23% 100% 12. Cycling carbon: seeing how plants use carbon dioxide in the lab 11% 100% 13. Dangerous Earth: a plate tectonic story 33% 11% 100% 14. Earth’s crust: thinner than you think 11% 100% 15. Earth’s crust versus the prep. room: why the differences? 11% 100% 16. Earthquake waves: the ‘find the earthquake’ team challenge 33% 100% 17. Earthquakes or nuclear explosions?: seismic clues to dirty deeds 23% 100% 18. The folding of rocks: lab simulations 11% 100% 19. Igneous processes: matching the evidence they leave behind 11% 11% 100% 20. Igneous rocks: completing the 3D picture 11% 11% 50% 50% 21. Investigating the Earth: the ‘find the Mars bar’ challenge 11% 100% 22. The limestone inquiry, 21st Century 23% 100% 23. Limestone in your everyday life 44% 25% 75% 24. Magnetic patterns: ocean floor pattern plotting 11% 100% 25. Magnetic stripes on the ocean floor: a lab simulation 11% 11% 100% 26. Mantle convection moving plates: the golden syrup / hobnob teacher

demonstration 33% 100%

27. Metamorphic modelling: simulating metamorphic processes 11% 100% 28. Minerals, elements and the Earth’s crust 11% 100% 29. Plate riding: how is the plate you are on moving now? 33% 100% 30. The plate tectonic story: a scientific jigsaw 11% 23% 100% 31. Protecting the Earth: how big is your ecological footprint? 11% 100% 32. Rock cycle in the lab: Earth products and the processes that link them 11% 11% 50% 50% 33. Sedimentary rock from sand: syringe simulation 11% 11% 50% 50% 34. Separating mixtures: how we concentrate natural materials 11% 100% 35. Sequencing of rocks: what was the order of events? 11% 100% 36. Solid mantle in full flow: the DIY potty putty simulation 11% 100% 37. Structure of the Earth: probing anomalous balls 23% 100% 38. Structure of the Earth: the story of the waves 11% 11% 100% 39. Structure of the Earth: teacher demonstrating seismic evidence for the

core 11% 11% 100%

40. Tree rings: a climate record of the past 11% 100% 41. Weathering and erosion: simulating rock attack in the lab 11% 11% 50% 50% 42. Will my gravestone last? an investigative graveyard visit 23% 50% 50% 43. Volcano in the lab: a wax volcano in action 11% 44% 40% 60%

Every topic is used by at least one of the teachers responding. The topics most used on a regular or occasional basis are Amazing Earth: facts that fascinate, The chemistry of limestone, Dangerous Earth: a plate tectonic story, Limestone in your everyday life, and Volcano in the lab. The hits on the website reflect this same profile of topic popularity. Three other topics The age of the Earth: can you believe everything you read?, Dangerous Earth: a plate tectonic story, and Mantle convection moving plates, are used regularly by over a third of the teachers. Nine topics are used with KS3 students whereas all topics are used with KS4 students.

How the resources were used and with whom Teachers were asked to provide some detail about the year groups they used the product with and which sub groups of students. JESEI was used to some degree with all year groups Y7–Y11. Peak usage was in Y10, and in the majority of cases it was

108

used with all ‘levels’ or sets of students. These teachers use the product with a wider range of students than the teachers evaluating the multimedia product did. Table 4.61 gives details. Table 4.61 Use with different ages and ability groups

Year group Teachers using with all sets/levels Teachers using with upper sets only

Y7 23% Y8 33% 11% Y9 11% 11% Y10 44% 23% Y11 44%

Most of the time, the student activities were used as provided. At other times teachers adapted resources to suit their circumstances. Table 4.62 summarises the data available. Table 4.62

Activity type Teachers using as provided Teachers adapting How adapted

Pen & Paper 67% 33% Adapting to school SoW or specification; differentiating for lower ability

Practical 78% 11% Instructions shortened to use less paper Teacher demo 78% 0 Role play 11% 0 ICT 23% 11% Adapted to suit school network

All the teachers who responded said they found the teaching notes for each topic useful. Teachers expressed appreciation of the flexibility, quality, freshness of approach and usefulness of JESEI.

Teachers’ ratings of how well objectives are met The product addresses a range of objectives that include professional development outcomes as well as student learning outcomes. Teachers were asked to rate the extent to which the JESEI resources allowed them to achieve these objectives. Table 4.63 shows the findings. Table 4.63 Teachers’ ratings of how well learning objectives are met

very well not at all Learning Objectives 1 2 3 4 5

Improve your teaching of earth sciences 56% 33% 11% Increase your enthusiasm for teaching earth sciences 44% 44% 11% Build your background knowledge of earth sciences 56% 33% 11% Give students a better idea of the nature of earth sciences 33% 33% 23% 11% Enthuse pupils about and engage them with earth sciences 23% 33% 44%

The majority of teachers considered that the professional development objectives of improving teaching, knowledge of earth science and enthusiasm for teaching it were achieved to a high level. Several teachers remarked on the usefulness of the resources for developing the teacher’s own knowledge. The teachers were perhaps less enthused though still predominantly positive about the effects of the materials on students.

109

Feedback from interviews All four interviewees saw JESEI as contributing to their delivery of the core curriculum, and two as providing curriculum extension or enrichment as well.

Reasons for choosing and alternatives to JESEI The teachers saw JESEI as one of several potential resources for earth science materials, but the quality and format made it more useful than others, both as a source of teacher information and teaching materials that can be assimilated into and developed as departmental resources. As one teacher put it, ‘I am much happier teaching Earth sciences with them than I would be without them’. Another described the resources as 'very important to me. [They] have made the Earth Science element easier to deliver.' The value of JESEI activities to students according to these teachers lies in their unusual ideas and presentation, students find them ‘more engaging’; 'all the students become engaged and learning is taking place'.. Other resources that could be used for earth science, that were available in one teacher’s school, included ‘Spotlight Science’ at KS3 and ‘Chemistry for you’ at KS4, plus other textbooks and videos, but JESEI was seen as ‘more topic-specific and ... more accurate information’. Two of the teachers used the activities as provided; for a third, the only adaptation he has made has been to 'include rock samples to go with the JESEI pictures'. Another ‘supplements a lot, e.g. with a Wegener information sheet supplemented with diagrams and pictures as a homework activity’, and also other PowerPoint materials, textbooks, departmental worksheets and past papers. None of the interviewees knew what their colleagues used for teaching earth sciences, though one has supported colleagues in taking up the use of some JESEI topics, and has been asked by his Head of Department to write the wax volcanoes and other material into the school's scheme of work. This teacher has used all the JESEI activities, explaining that 'The wide choice means lessons are not the same across the year groups. I have included all these in schemes of work as either core teaching methods or to be used/adapted for 'homework for assessment'. The idea is that if the options are available teachers have a choice of excellent resources that can be used for different ability ranges to aid differentiation.' None of the four seemed to have opted for JESEI as the result of a careful investigation. All were rushed for time: two had come upon JESEI ‘by chance’ at a point in time when they needed earth science resources urgently. They were attracted by activities that they felt students would enjoy, were good visually, and added depth to the subject. Two had first been exposed to JESEI during their (recent) PGCE courses, and turned immediately to JESEI when they needed Earth Science material: as one said, 'One of the tutors at [my ITT institution] raved about them. Other options must be available in school but because I am happy with these I have not looked for them.'

Contexts of use At the time the interviews were carried out, in the first half of summer term, the most recent use of JESEI by three interviewees was in the context of revision, recapping earlier use in a normal teaching sequence. In the fourth interviewee's case the most

110

recent use was as part of the scheme of work. The details of use in three of the four schools are covered below (one school did not respond to these questions). One teacher was in his first year of teaching in a mixed comprehensive in the north-east, was a Physics specialist, and was using RSC-sourced materials for the first time. As well as student use, he felt he had benefited from the materials himself, being able to teach with much more confidence as a result of using them. JESEI resources were used with the Y10 upper set throughout the year, for all earth-science related parts of the scheme of work. The JESEI resources provide the main activity(ies) in each earth science lesson. The teacher was pleased with students’ achievements and explained how this was supported by JESEI activities:

‘[Students] achieved well on subsequent testing of the material, using past GCSE questions ... I think adding the background story to scientific discovery always helps, and some of the resources do this. The practicals and demos help engage kinaesthetic and visual learners. The earthquake challenge motivated them as there was a prize for the first team to find it. Overall they make a potentially very dry subject far more enjoyable.’

A second teacher was in her first year of teaching too, in a mixed comprehensive in the south-east. She was a chemistry specialist and used a number of RSC materials in her teaching. JESEI was used with a foundation (lower) tier Y10 group, during the autumn term, as part of the scheme of work section on limestone. There are a number of limestone-related resources on JESEI and these were used to support group activities leading to a debate. Groups had different foci, and were provided with different subsets of the JESEI limestone worksheets ‘to provide the groups with information on their expert area’. The teacher was pleased with the impact of the product and reported that:

‘The debate was very successful. When we came to revise the module students were able to recall the debate in some detail along with all the main uses of limestone. ... Students enjoyed the opportunity to work in a very different way to that normally found in Science. They were well engaged (a normally challenging class behaved well and were clearly involved in the task).’

A third teacher was in his second year of teaching, in a mixed comprehensive in the midlands. He was a chemistry specialist, and made no other use of RSC materials. His most recent use of JESEI material was with year 10 pupils (top and foundation sets), with whom he used the wax volcano activity to 'show intrusions and the 'operation' of the volcano ... students could see the formation of lava intrusions and they really enjoyed this. Lower ability students did not understand the explanation and board diagram that preceded the demo, but 'saw the light' once the demo was completed.' Towards the end of the lesson he had found evidence of learning in that 'students could explain lava, magma crystal formation and intrusions during peer to peer questioning'. He has also found evidence of student enjoyment in that, having used different resources with different sets and year groups, students 'spread the word' about particular activities to each other, and he finds himself being asked for them by other groups: 'I find this encouraging and in my opinion this shows how useful the JESEI material is.'

111

Although the data set is small those teachers responding all rated the product as very high quality, very useful, clear and easy to use. The topics were widely used across key stages 3, 4 and 5. Teachers valued the flexibility, quality, innovative approach and usefulness of the JESEI product. The teachers’ rating of how well objectives were met for their own professional development were very high with the majority reporting that it improved their understanding, interest and motivation in the earth sciences. They were less convinced of the effectiveness for students though two-thirds said it gave students a better idea of the subject only half of the teachers felt it increased students’ motivation and engagement in the subject. Again this may reflect the teachers’ expectations and practice rather than any problem with the product. The interviews provided some support for this view as these teachers saw the value of the product for students was their unusual ideas and presentation, which they felt, enhanced students’ interest and learning. As one commented ‘Overall they make a potentially very dry subject far more enjoyable.’ We examined the JESEI site hits and it is clearly a very popular site but this information is hard to translate in terms of use. Some of the teachers we talked to were using the site and the materials extensively. They had integrated them into their core teaching and as homework resources. However, other teachers we talked to described their use of the site and the materials as a matter of ‘chance’. In the user survey of volunteer teachers only 20% were aware of the JESEI site. The findings suggest that this is another excellent resource that has a huge potential for impacting on teaching and learning but this is not realised because significant numbers of teachers are not aware of it. There is some indication too that some teachers may need additional guidance to deal with the innovative nature of the product, its content and pedagogy.

4.6 Issues emerging from the product evaluation There are general findings that emerged from carrying out the evaluation study that were not designed into the study. First the volunteer sample of teachers spent a lot of time providing thoughtful feedback and from their unsolicited comments this reflected their commitment and appreciation of the RSC generally and their contribution to education in schools and colleges. Although we asked for feedback on specific products members and non-members alike commented on the quality of the RSC provision. ‘In general I find RSC materials fantastic’ was typical of the comments we received. The other finding was that the evaluation study itself drew teachers’ attention to RSC products and their potential in supporting teaching and learning in science and chemistry. Users and non-users reported that they would extend their future use of RSC materials

4.6.1Paper-based products The paper-based products were all rated highly for their content. Clarity and effectiveness in meeting needs was also considered to be good to excellent. Ratings on clarity were reduced for Chemical Misconceptions Volume I with a third of teachers experiencing some difficulties with access. Ratings of ease of use across the products were slightly depressed reflecting other feedback about the need for cross-referencing to exam and curriculum specifications.

112

The Misconceptions texts are very much a combined professional development and teaching and learning resource. Furthermore they are based on research evidence and advocate a particular approach to learning. Teachers welcomed this and reported that they selected Misconceptions Volume I to develop their practice and their understanding of students’ learning. The feedback indicated that teachers’ difficulty with access was largely restricted to the theoretically challenging chapter in the volume. Even then two-thirds of teachers reported no problems with access. The chapters on chemistry topics were highly rated in terms of clarity and usefulness particularly those with direct relevance to key aspects of the chemistry curriculum for KS3 and 4. A common comment was that the product was a ‘very useful tool’. The detailed evaluation allowed insights into teachers’ use of products with students. It was clear that teachers were using the activities and guidance across the key stages. The student resources and guidance for teaching in Volume II were all rated exceptionally highly on all criteria. 100% of teachers were of the view that the structure and the target level i.e. age and ability indicated was of great benefit and effective for their planning. As teachers across the evaluation sample have asked for this type of information across products it is clear that exemplars are already available to support this. Teachers’ feedback about the impact on students’ learning of the activities indicated a very high level of success across the vast majority of the most frequently used probes. The evaluation data about which probes teachers use and with which students would be of interest to teachers and could be something that was put on the website. The positive impact of the product on students’ learning was supported in the feedback from interviews. One teacher who used the probe in line with the guidance considered the lessons to be very effective by making explicit to students their initial ideas and then encouraging them to reflect on their new learning in the light of this. He commented ‘I think they took a lot away from the lesson from seeing what they knew originally...I think it [the approach] was nicer. I think in many subjects they’re usually just taught something, they’re not expected to actually think about what they already know...I’ll definitely use it again in the future.’ In the class we observed the teacher adapted the probe: from our observations and interviews with students we found that the probe helped them clarify their understanding of key concepts about the nature of matter and moved their thinking forward. Where confusions remained this was largely due to the way in which the teacher had used the probe. The teacher had only recently begun to use the resource so it is to be expected that it will take time to incorporate the teaching strategies into her practice. In interview though it was clear that her understanding of learning had shifted and her attention turned towards the learners and their learning processes: ‘you suddenly realise a bit more about how the children are thinking rather than how we are thinking.’ The structure of the probes reflects an embedded pedagogic strategy that is successful. However when teachers amend the probes and are not aware of the pedagogic approach then opportunities for learning can be missed. This is what we observed in the case study classroom. This teacher, along with others, would have benefited from additional information about the thinking behind the structure of the probes in relation to the pedagogic approach and students’ learning. Whilst existing products cannot be amended such guidance could be made available on the website. Alternatively it might be worthwhile to consider an associated resource, a CDROM or DVD which

113

both reminds teachers of the availability of the product, emphasises the pedagogic approach and highlights some of the probes that the evaluation study has shown to be very effective and frequently used by teachers. Teacher feedback from the evaluation study could also be included as teachers welcome colleagues’ view points. The effectiveness of the product for teachers’ professional development was probed and the findings indicated that all teachers had altered and extended their view of learning irrespective of their starting point perspectives. This was confirmed in interviews: ‘It really did enlighten me.’ The teachers also reported that they used the products in their mentoring of student teachers. Teachers who no longer used the product had incorporated it into their practice and their schemes of work or had moved on and sought out additional research-based material. We also found that non-users who were introduced to the product via the study were keen to make use of it in the future. The Classic Chemistry Experiments and Classic Chemistry Demonstrations product was rated very highly as an effective resource. There was an impressive use of both Experiments and Demonstrations spread across the age ranges but predominantly use is for key stages 3 and 4. Teachers commented on the high quality of the texts as a teaching resource and the clarity and quality of the instructions for students: ‘Excellent student worksheets. Good teacher and technician info.’; ‘Exciting experiments not normally detailed in text books.’; ‘I haven’t found a single resource of demonstrations like this, good ideas, succinct and well written.’ Access was seen by some teachers to be problematic because they wanted more cross-referencing to curriculum specifications, age ranges etc. in order to help them make informed selections for their schemes of work. This would be particularly valuable as teachers reported that their use of the experiments and demonstrations was determined by the curriculum - the course they followed and the departmental schemes of work. The feedback from teachers about their use of particular demonstrations and experiments could be used to help inform the cross-referencing. It could also be used to identify a selection of particularly relevant material from the product that might be made available to help some teachers think about how to use the product effectively in their context. The feedback confirmed that some teachers valued the flexibility to choose how to use the product and with which students. There was however clear feedback from some teachers that they wanted the guidance. For some this was to make their use of the product less ‘frustrating’ and for others because they felt they were not in a position yet to make the best judgements about use. One teacher’s comment made the point well for the need for the content management that is under development for the RSC websites. ‘No problem as they are but referencing to QCA KS3 schemes of work and GCSE areas might be really helpful – especially for searching on a CD or the web.’ The majority of the teachers rated the product highly in terms of impact on students’ learning. They reported that all the major learning objectives targeted were well met. Teachers rated the Demonstrations text very highly in developing students’ motivation and interest in chemistry. They also saw the text as a very effective tool for enabling students to understand the link between theory and practice. Teachers rated the Experiments text highly in terms of developing students’ knowledge of and

114

practice in experimental skills, the effective use of equipment and materials and the ability to follow instructions. The materials, like the Misconceptions texts, allowed teachers flexibility in use so that their preferred teaching approaches could be applied. To do this alternative approaches to experiments were offered that altered the degree of autonomy allowed to students and consequently the learning objectives that could be achieved. The feedback indicated that teachers were tending not to use the more open approaches, which allowed students more autonomy and decision–making. The findings suggest that some teachers were not clear about how the choice of approach allowed for different learning opportunities. Again we would recommend that the thinking behind the different structures to the experiments could usefully be made available. This would support teachers to use the more open approaches that correspond well with the investigative approach enshrined in current curriculum specifications and their assessment. Teachers also noted that they would welcome having some examples of the demonstrations and experiments in use. They referred to the learning context and the need for ideas about introductions and links to other topics. Examples of teachers’ use of the product could be used to make explicit the thinking behind the structure of the students’ experiments and provide some examples of learning contexts. These could be made available on the website or on a CDROM or DVD. Non-users in the sample were very positive about the product and reported that they would definitely use it in the future. Minimal data was collected about the Ideas and Evidence series. Our analysis of the product and teacher feedback suggests that there is considerable potential that is perhaps not yet being exploited. This is supported by the feedback from the user survey, which was a large sample (N=73) and yet only about a quarter of teachers reported that they had access to this product which was low in comparison to other products. The cold call interviews also revealed that teachers had not heard about these products even though they were disseminated in the same way as other products. The indications from teachers’ feedback are that the product is highly rated. The use of the product suggests that those books that are more obviously part of the curriculum and exam specifications are being used. Other books such as Green Chemistry and Climate Change because of the innovative nature of their content and the emphasis on students’ thinking about science as practised in the world and as a social and political influence means that their relevance is less obvious to teachers. The series both in its content, aims and teaching strategies is in advance of mainstream curriculum materials and relates closely to the content and approach advocated in the latest science curriculum innovations currently under trial in England. Access to the product will improve with the introduction of content management on the websites. The product, however, has applicability across the science curriculum and key stages and it is not clear that teachers are aware of this. We would recommend that the product is reconsidered and even remarketed as it addresses a very real future need for teachers who are expected to radically alter their orientation to the subject and to students’ learning with the increased emphasis on science for public understanding. The content of the books provides excellent up to date resources for teachers and their students to support them in dealing with curriculum changes planned. The activities provide models for teachers about how to deal with these changes with their students. As these are already trialled and effective

115

we would recommend consideration being given to the possibility of the product being a basis for INSET though this has resource implications.

4.6.2 Multimedia product All teachers rated Alchemy? as excellent or good, unsolicited feedback in cold call interviews supported this view. Teachers chose the product because of its up to date material, its relevance and interest for students. Teachers reported that they choose to use Alchemy? and not other resources because it is ‘clear, concise, includes diagrams, and is relevant to the specification’. Teachers considered that the multimedia nature of the product was particularly valuable in motivating students and in enabling them to support a variety of ways of teaching that included whole class work, group work and independent study. It also supported students working away from the classroom which teachers and students valued. Teachers used a range of topics from the product predominantly at key stages 4 and 5. The more effective they found the topic in its impact on students’ learning the more frequently they used it. Three topics that teachers frequently used were judged to be not as effective with students as others. The feedback from teachers about frequency of use and effectiveness again could inform future content management developments. Teachers gave very high ratings about how well the product met its main learning objectives. The case study teacher commented about Alchemy?


Teachers’ feedback about which learning objectives are less well met indicates the way in which teachers are tending to use the product. The objectives less well met concern making links between science and school learning and practices in the chemical industry and between science and everyday artefacts. Both objectives are types of learning that it is hoped the future science curriculum will better address. We recommend therefore that there is potential to highlight the way that this product can be used to achieve these objectives which will enhance its effectiveness and help meet teachers’ future professional needs. The CDROM format and use of short video clips proved popular with teachers, who would like more of the same: as one said,

‘More! Digital video clips are the way forward in science education. How about some high impact clips e.g. TNT’.

All teachers reported using the video clips, which compared with about half to two thirds using the other components of the product. The web links were the least well used component for reasons already discussed linked to access for teachers and students as well as time constraints. The teachers valued being able to use new equipment like projectors, which feedback from the INSET sample suggests are likely to be increasingly available in schools and seen by teachers to be a teacher- and student-friendly way of teaching with multimedia.

116

Teachers did want cross–referencing but to exam specifications rather than curriculum specifications which reflects the targeted nature of the content of the product and the view that the product is best suited for older students. One teacher made a practical comment about using the CDROMs, saying that it ‘would be helpful if you could tell which video clips were on each disc BEFORE putting them onto your computer!’ Alchemy? is a product that is very well received by teachers and students in its approach and content. Its content linking classroom learning to industrial and consumer contexts and developing authentic images of the chemical industry is in line with future developments in the science and chemistry curriculum. The multimedia approach allows a range of learning situations including group work and independent study which fits well with the current emphasis on key skills in the 14-19 curriculum. It is therefore a high quality product and a timely one and we would recommend that is a product which warrants wider dissemination. Even although it has been mailed to all schools there remains a significant proportion of teachers across the countries in the sample who are unaware of its existence. Feedback from users also indicates that its use could be extended further to the benefit of teachers and students.

4.6.3 INSET The RSC INSET events are very popular and that is because, in the view of one of the trainers: ‘the RSC has got such a high reputation ...it means that people expect that it will be good so they will come on it..’ Across the two events observed – Improving Teaching and Learning in Chemistry using ICT and Using assessment to Improve Learning in Chemistry and Science – teachers’ ratings of how well the training met their needs was exceptionally high. As a team we have evaluated a great deal of nationally delivered INSET and consider that the level of appreciation reported by teachers to be exceptional. This is the result of a combination of factors, well qualified and committed trainers who have high credibility with other teachers, very well designed activities, and perhaps most important of all a focus on practice and how to do things with students within teachers own school contexts. This latter factor allows teachers to move from professional development activity to implementation quickly. Yet the events were not about ‘tips for teachers’ they model effective practice for teachers and focus on transforming it. There was a close correspondence between the trainers’ goals for the events and teachers’ views of what were the strengths of the courses. One teacher at the Assessment event described the RSC INSET as ‘very effective in making me think, reflect on my teaching and on what I was going to do next. Because I always like to do something that I know I can implement in the classroom and I think this one was very much like that’. The evaluation study again provided opportunities for the effectiveness of the INSET provision to be judged in terms of what really matters i.e. its impact on teachers’ practice and students’ learning. We collected data about teachers’ changed use of ICT resources following the INSET event and found significant impact on practice for example use of some had tripled since the event. Teachers’ plans for use suggested that impact would be ongoing and increasing. The teachers particularly valued having a forum to discuss the use of ICT with other teachers and wanted examples of use. There is potential to use this event and teachers’ feedback about their ways of using ICT to develop complementary resources for teachers that could be made available in

117

a number of formats. This is already being explored for the Using Assessment INSET event and there is potential for a model format to emerge from this. The teachers on the Using Assessment event also provided feedback about their implementation and there was a significant shift in practice with nearly a third of teachers implementing three of the main assessment techniques introduced at the event. Again teachers’ plans indicated that impact on practice would extend considerably. Over 90% of the teachers attending the ICT event reported that students’ learning had been enhanced. The feedback provided further support that both teachers and students find multimedia resources interesting and motivating. Detailed feedback on the impact on students’ learning was collected from the teachers attending the Assessment event. A fifth of the teachers reported that students’ motivation and enjoyment had improved. Teachers reported surprise at how well students adapted as one teacher observed ‘excellent response from students’. Teachers also felt that the techniques worked well across students of all abilities.

‘I think, as I say, some of them, because it’s not switching them off, it keeps them going and motivating them. The biggest thing I found, with my tutor group as a whole, is it doesn’t switch off the lower end of the group.’

The evidence from the case study revealed that students’ were broadening their understanding of assessment and its purposes and role in improving their learning. It was particularly noteworthy that over a third of the students considered the purpose for the new assessment techniques were to understand science better; what makes good work; and what they were supposed to be learning. A further 30% recognised it was to understand science better and 26% of those students cited as well either to understand good work or to understand what to learn. An impressive proportion (52%) consider that being a marker is helpful for their learning and a slightly smaller proportion (37%) reported that having a peer mark their work was helpful too. Students in interview explained how they saw the benefits:

‘Well it gives you the chance to learn what other people know and what, if you didn’t get that question you can see what that person wrote and you can learn off marking someone else’s.’

‘Probably you get to know other people’s working level maybe or to help you understand it a bit better than you did in the beginning.’

‘People can often learn off you.’ One of the impacts of being a marker in peer assessment is that students begin to get a better perspective on what constitutes good work in science. This seems to be an emergent phenomenon for the students as the majority reported understanding just a ‘bit better’ what is good work. Teachers in their feedback provided useful comments on how to sustain impact, these included extending the INSET so more staff could attend such courses. Many would like a resource book or materials with examples of techniques and methods. Teachers would value more networking to share good practice, ideas and experiences. The teachers attending the ICT event wanted the instruction book to be made more widely available. Most teachers (97%) at the Assessment event said they would find a

118

book on assessment for learning that included techniques and examples of students’ and teachers’ work in chemistry useful. The majority (71%) expressed an interest in a further workshop extending techniques for peer and self-assessment. A change in location of the workshop would help one respondent who felt workshops in Northern Ireland would be helpful, while another hoped for workshops to be held in Devon and Cornwall. The RSC has already implemented plans to hold events in Northern Ireland in 2005 and therefore this need will be met. On the basis of the feedback from teachers and our evaluation of impact on teaching and learning we can only recommend that the RSC continues its good practice and extends it if resources allow. The evaluation data has already indicated possible foci for future events. The findings of the study indicate that the RSC is a provider of excellence in INSET and unlike most providers now has evidence of impact to support this claim. It would be expected therefore that provision for teachers could be extended through close collaboration between the RSC and the Centres for Science Learning: we would recommend that this collaboration is fostered to enable wider access to the RSC product. Teachers’ have indicated potential for other RSC products to support their teaching. Some of these already exist. One avenue that we recommend exploring is products based on teacher exemplars. INSET events provide an excellent opportunity for collecting examples of teachers’ implementations and if resources were available to develop these, teachers would benefit enormously from them. This is being explored already for the Assessment event but there is scope for more products particularly in multiple format with a CDROM or web component.

4.6.4 Web-based product The sample responding to our invitation to evaluate the Joint Earth Science Education Initiative (JESEI) was very low. Those teachers responding all rated the product as very high quality, very useful, clear and easy to use. The topics were widely used across key stages 3, 4 and 5. Teachers valued the flexibility, quality, innovative approach and usefulness of the JESEI product. The teachers’ ratings of how well objectives were met for their own professional development were very high with the majority reporting that it improved their understanding, interest and motivation in the earth sciences. They were less convinced of the effectiveness for students: though two-thirds said it gave students a better idea of the subject only half of the teachers felt it increased students’ motivation and engagement in the subject. Again this may reflect the teachers’ expectations and practice rather than any problem with the product. The interviews provided some support for this view as these teachers saw the value of the product for students was the unusual ideas and presentation, which they felt enhanced students’ interest and learning. As one commented ‘Overall they make a potentially very dry subject far more enjoyable.’ Another teacher emphasised the different approach too: ‘Students enjoyed the opportunity to work in a very different way to that normally found in science. They were well engaged.’ Another teacher reported that students had so enjoyed the activities that they had told other groups about the activities and resources and he found that other students were asking for them: ‘I find this encouraging and in my opinion this shows how useful the JESEI material is.’

119

We examined the JESEI site hits and it is clearly a very popular site but this information is hard to translate in terms of use. Some of the teachers we talked to were using the site and the materials extensively. They had integrated them into their core teaching and as homework resources. However, other teachers we talked to described their use of the site and the materials as a matter of ‘chance’. In the user survey of volunteer teachers only 20% were aware of the JESEI site. The findings suggest that this is another excellent resource that has a huge potential for impacting on teaching and learning but this is not realised because significant numbers of teachers are not aware of it. We have discussed the access issues that are affecting teachers’ use of the websites and the constraints mainly of time that limit teachers’ use. However, we would recommend that attempts are made to increase teachers’ awareness of this resource through RSC publications and events. Although there is insufficient data about teachers’ use of the product we suspect that the innovative approaches to student activities are a challenge for some teachers. These teachers may need more guidance of the kind we have suggested earlier in relation to extending access and use of the paper-based products. Across the products we have identified the potential for wider access, use and impact and this is particularly the case for specific products. Clearly there has to be a balance between the access aimed for and the resource available to achieve it. Our recommendations cannot be informed by this balance our stance is based on the feedback from teachers and indications of their needs. The study has provided a firm basis for the recommendation that extending the access to and use of the products will significantly benefit the teaching and learning of science and chemistry and build on the already significant and impressive contribution made by the RSC. The evaluation of the products has indicated that teachers value the RSC products highly. They are effective tools and resources and they are achieving their objectives well for both teachers and students. Teachers needless to say would like more of them. We have made recommendations based on the study of what some of these new products might be. These recommendations have emerged within the context of the products selected for evaluation so only represent one aspect of what might be developed in taking the RSC products forward. Again any decisions need to take account of the wider mission of the RSC and the resources available.

120

References Department for Education and Skills (2004) Science, Technology, Engineering and Maths (STEM) Mapping Review [online], Department for Education and Skills. Available from: http://www.dfes.gov.uk/hegateway/hereform/index.cfm?cid=34 [Accessed September 22 2004]. Osborne, J. (2004) ‘Clumsy tests dull passion for science,’ The Times Educational Supplement, 2 January 2004.

the evaluation of rsc materials for schools and colleges · 2019-04-26 · patricia portsmouth,...

Documents