Action research as a methodology used to help assess the effectiveness of sustainable design websites

Peter Simmons

Department of Design and Technology, Loughborough University

Abstract Despite large investment into information communication tools such as websites, the

impact that these websites have within education is rarely assessed. This paper

reviews current action research methodology employed to assess how effective

sustainable design websites have been in influencing the design decisions of student

designers. Website effectiveness encompasses a wide range of elements that

stretches further than just website usability. It includes cognitive psychology, human-

computer interaction, professional writing, linguistics and persuasive value. This

paper shows action research as part of a three year PhD study which looks at the

assessment of website effectiveness using data from interviews, observations,

questionnaires, usability checklists and design folio observations.

The paper outlines three different areas of action research; creating a sustainable

design website, direct support of design project work (both one-to-one and in

groups), and raising awareness. Current sustainable design initiatives in education

that aim to achieve an integration of sustainability principles through websites are

described. These sustainable design initiative websites, such as the Sustainable

Design Award, form the main focus of this investigation.

Keywords Sustainable design, effectiveness, websites, impact, methodology, action research

1 Introduction In the past, the mention of the label ‘action research’ risked conjuring visions of a

research project of low academic value when compared to other research

methodological practices. At one stage there seemed to be an air of unshakeable

controversy surrounding the research method, with many researchers questioning its

validity. Perhaps as a consequence people tended to overlook its longstanding

development: ‘qualitative inquiry and action research have a long and proud tradition

1

that many academics, professors and administrators, and public school teachers and

administrators do not realise’ (Smith 2004:190).

Today however, it is more widely recognised as one of the most productive and valid

forms of research by the leaders in the field (Archer 1992; Norman 1999; Roberts

2000 ; Green 1998). ‘A designerly approach, rather than a scholarly or scientific

approach, can with advantage be made towards educational research and curriculum

development’ (Archer 1992:12). Its history and traditions are now also starting to get

the recognition to support Archer’s assessment of its potential. Take ethical issues

surrounding design research for example. The Revised Ethical Guidelines for

Educational Research by the British Educational Research Association specifically

including guidelines on Action Research for the first time in 2004.

Action research applies ‘all the normal rules governing research practice’ (Archer

2004:29) but it also generates immediately applicable new knowledge. The

methodology used in this paper demonstrates a cyclic method of working within the

research process, as championed by several leading researchers (Elliott 1991, and

Kemmis and McTaggart 1988). This could be considered as action research due to

its practical nature and specific outcomes. The research involves a range of activities

that transcend several common research methods, ‘the labels action research and

qualitative modes inquiry highlight a number of overlapping domains of similar inquiry

modes. Teacher research, classroom research, illuminative evaluation, clinical

research, teacher autobiography, classroom process studies, diachronic inquiry, etc.,

are all tapping into much the same territory’ (Smith 2004:190).

Figure 1: action research method diagram

2

Action research projects are structured to contribute, test and/or refute existing

knowledge systematically through a series of classroom based activities. As Green

writes: ‘the fundamentals of action research involve: the questioning of assumptions;

the clarification of values; the discovery of the mismatches between espoused values

and practice; the understanding of the wider social context in which I work.’ (Green

1998:2). Despite its flexibility, action research, in terms of this research project, takes

its traditional form of consciously taking an action to gain a result. As Burns outlines

in his understanding of action research ‘a total process in which a ‘problem situation’

is diagnosed, remedial action planned and implemented, and its effects monitored if

improvements are to get underway’ (Burns 2000: 443). It is the instigation of change

which makes action research very applicable to research projects such as the PhD

study used as an example in this paper. It is researching with a purpose for an

outcome.

It is not the result of that action that is necessarily applicable to other areas, but the

deeper understanding of the context gained. ‘Action research does not attempt to

produce results that are immediately transferable to other teaching situations… it is

the understanding of the complexities of the particular situation and the recognition of

the different ways in which the familiar can be interpreted that is the aspect that is so

readily transferable.’ (Green 1999: 107)

As part of the study several carefully considered actions were taken to gain some

valuable research information. The cyclic nature of the research enabled minor

alterations to be made throughout this process. Several interim conclusions were

drawn and acted upon. A sustainable design initiative led by Practical Action is

described in this paper which provided the context for the research. Action research

is then described more specifically in relation to the Sustainable Design Award (SDA)

website and the assessment of its effectiveness in influencing design decisions.

3

2 Sustainable design initiatives

Sustainability is a key issue for all educators, designers and students; its rising

prominence is largely due to government legislation and an increasingly

environmentally conscious society. In general terms, sustainability has become a

major user consideration in products, people now choose products based on

environmental and ethical responsibility as much as they do on form, function, price

and aesthetics. Sustainability within design and technology has shifted from a focus

of ‘end of pipe’ techniques to look at the whole product’s life-cycle, a ‘cradle to grave’

approach (Bhamra 2004).

Bhamra (2004) also outlines the challenge that we face in combining technology,

culture and nature, the success of which is largely dependent on effectiveness,

innovation and creativity of implementation. This research focuses on the area of

effectiveness, but also the sustained motivation of the design students and the

communication of the information.

Several groups and organisations have taken significant steps to tackle this often

testing area. Notably work by the Pré Consultancy in the Netherlands who developed

design assessment tools (e.g. the EcoIndicator Manuals for Designers) in the late

90s (Goedkoop 1995), the UNEP Ecodesign manual ‘Ecodesign: a promising

approach to sustainable design and production’ (Brezet and Hemel 1997), the Royal

Melbourne Institute of Technology’s research work and subsequent publication A

Guide to EcoReDesign: improving the environmental performance of manufactured

products (Gerstakis, Lewis et al. 1997), the Demi website developed for

undergraduate students led by Goldsmiths University alongside several UK

organisations (Clare 2001), and most recently Practical Action’s Sustainable Design

Award (SDA) aimed at post-16 students (Capewell and Norman 2003). These could

be seen as actions taken to address the area of sustainability. However this only

becomes ‘action research’ when that action is assessed and evaluated.

The Sustainable Design Award (SDA) is led by a charitable organisation Practical

Action and it operated from 2002-2007 in partnership with Loughborough University

in England, the Centre for Alternative Technology and Twente University of the

Netherlands. These are all organisations with acknowledged expertise in the field.

4

SDA aims to put sustainability on the education agenda in design and technology at

AS/A2 level.

As the author, and indeed Loughborough University, have been involved in the SDA

project from its beginning, this paper and the research in which it is situated have

focused on this scheme and its integration into design and technology education.

Date Sustainable Design Award Description PhD Research Notes Evaluation April 2002

DFID supported projects in the UK and the Netherlands to help to ‘raise awareness and understanding’ in young people of sustainable development.

Phase 1:1

Action 2003

The Sustainable Design Award was introduced by Practical Action to support the sustainable design principles needed within Design & Technology education (Figure 3). Resources were provided such as the teachers handbook and SDA website.

Phase 1:2 Phase 1:3 Phase 1:4

Evaluation 2006

Evaluation of the Sustainable Design Project by James Pitt and Fred Lubben of the University of York. This report evaluated the project through educational resources are key (Figure 4)

Phase 2:5 Phase 2:6 Phase 2:7

Action 2007

Project on-going but now driven through resources and teacher training rather than ‘direct input’ based

Phase 2:8

Table 1: sequence of some of the action-evaluation cycles which took place within the Sustainable Design Award

The scheme can be seen as an action (Table 1) taken to impact upon the education

of students at 16+ level. Each part of the scheme in terms of inputs and resources

could also be seen as actions taken by the SDA to have a specific purpose within

classrooms. After the SDA projects’ pilots studies were completed in 2002-03, 150

schools were enrolled in the 2003-04 and up to 241 schools in 2005-06. The SDA

was awarded to 882 students between 2002-06, and this covered AS and A2 level

(Pitt and Lubben 2007).

The results or success of the SDA scheme, and the actions taken, is perhaps best

measured by the design outcomes of the students who undertook the award and

their design work since then. It could also be measured in terms of awareness and

the educational push towards sustainability within design, at an educator and

legislative level.

5

3 The first action research cycle: the Sustainable Design Award website

The SDA is the first scheme in the United Kingdom to focus specifically on post-16

students and, as outlined by Capewell and Norman (2003), the scheme took several

actions to support classroom education in terms of:

• teacher information days;

• a sustainability pack;

• student study weekends and school visits;

• electronic communication via email and the website.

The scheme has also developed as a result of these initial actions. The Sustainability

Handbook (Daniel 2007) has now been written and sent out to schools free of charge

as a support to teachers and students, it refers back to specific sections of the SDA

website where appropriate. This progression towards a greater reliance on

communication tools such as the website and the handbook is a result of an iterative

process undertaken within the SDA development team in assessing the scheme after

a particular action was taken.

Figure 2: an overview diagram of SDA action research

Judgements were made on the success of the SDA website, and as a result,

recommendations for further developments to have a greater impact and appeal to

teachers and students were made. Figure 2 shows both phases 1 and 2. It is an

6

overview of the process of the SDA introducing information resource tools (phase 1:

1 to 4), and then looking at the use of the SDA website specifically (phase 2: 5 to 8).

These are given in more detail in Figure 3 and Figure 4.

Figure 3: (phase 1) the SDA action research method diagram

Figure 4: (phase 2) SDA website effectiveness action research method diagram

7

Communication has proved a vital component in the development of the scheme to

both teachers and students through different forms of media, from handbooks to

teaching sessions and a dedicated website. From training days to key one-to-one

inputs in schools, the SDA scheme aims to raise the profile of sustainability for the

designers of the future. The need for a website became a viable option to the SDA

Steering Committee, both in terms of funding and use. The website would be a

versatile and accessible resource for both students and teachers to reach key

information to support their educational needs. The Sustainability Handbook and

other SDA resources would support this by advertising the website as a useful

resource tool. As such the website could be seen as an action taken by the SDA to

enable sustainable design practice to become an integral part of 16+ Design and

Technology education for students in England and Wales.

An initial website was set-up and developed through feedback during a number of

study days held by the SDA. Both students and teachers were asked to give

feedback (Figure 3) via group PowerPoint presentations. The feedback proved useful

in refining the website, and as a result key changes were made; splitting the website

into a ‘teachers’ and a ‘students’ section and adding a checklist for students to use

during their project work.

The Sustainable Design Award website (www.sda-uk.org) averages around 13,700

visits a week (MediaHouse 2006), the majority from academic servers. The SDA

website has become a practical way of making sustainable design information

available to students. These website statistics indicated that the website is being

accessed by a number of people at least weekly. Data was also gathered via

questionnaires in classrooms, this looked at website usability, website use in general

and specifically at sustainable design websites when students used websites in their

work, and what information they were looking for.

The research sought to investigate the impact of the SDA and its website at both AS

and A2 level, and decipher how regularly websites in general, sustainable design

websites and the SDA website were used. It also looked at the stages at which the

websites were being used during designing. This information would help to progress

the research study as it would help to assess where the SDA website had been

effective in meeting the needs of its primary target audience.

A questionnaire (Figure 5) was derived from the research questions of the main

8

study. Its focus concentrated on the use of websites within design and technology

and at what stage during designing they were utilised. The questionnaires were

initially trialled at ‘School A’ with 11 advanced level students, and it was then later

refined for the main study.

9

10

Figure 5: Example of the refined website use questionnaire*note ‘all the time’ was explained as every two days

Five schools participated in the study, four who volunteered at a SDA study weekend

and one other, covering 72 students aged 16-18. The results of the questionnaire

indicated that sustainability is a prominent part of design and technology education

with 78% having had some sustainability input. Around two thirds of the students

were intending to take, or had taken, the Sustainable Design Award.

11

79% of students said that they had used websites within their design work and that

this use was more frequent at the start of their design projects. The use of websites

was seen to decrease throughout a project. This was a trend throughout the results

for both sustainable design websites and the SDA website. Around half of the

students had used sustainable design websites but only 28% of the students had

used the SDA website (Figure 6).

Have you used the SDA website to help you during your AS/A2 level major project design work?

Yes28%

No72%

Yes No

Figure 6: Sustainable Design Award website use chart

12

4 The second action research cycle: classroom inputs

Initially there was no action taken as such, the students were merely asked key

questions relating to website use in Design and Technology. This assessment was

deemed important in order to have a foundation upon which other actions could be

implemented. The next phase of the research looked to assess the impact of the

group PowerPoint presentation (see Figure 7 for example of one slide in PowerPoint

presentation) that outlined the areas of the SDA website, pointing the students to

relevant areas for their design folio work. This reassessment embraces the idea of a

cyclic method as used by practitioners in their research work (Figure 1).

Figure 7: An example of a page from the Powerpoint presentation

This section of the paper relates to classroom inputs that are shown as number ‘6’ in

Phase 2 (Figure 4). In this instance the SDA steering committee felt that

communication tools were needed to help implement sustainability in Design and

Technology education. This outlines the actions taken to achieve this through the

introduction and evaluation of the SDA website. The conclusions from this

input/action would help to shape further proposed actions in order to assess website

13

use and website effectiveness.

This second phase indicated that a large number of the students were not

considering sustainability when looking for design information. When using web-

based resources their focus appeared to be on other areas (Simmons and Badni

2007). The students that had accessed the SDA website had used it throughout their

project, but with only 28% of the students using it, this may suggest a greater focus is

needed in motivating designers to access the website in the first place. Several

actions that focus on improving the ‘before use stage’ phase of websites were

therefore proposed (Simmons and Badni 2007).

These actions looked at introducing the website to classes as a group and giving

one-to-one design consultancy with students, advising them on key areas of the SDA

website that were applicable to their work. Students were also introduced to the

Sustainability Handbook by their teachers that referenced many pages on the SDA

website specifically.

After the questionnaires (Figure 5) were completed, a further action was taken by

introducing the students to the SDA website through a group PowerPoint

presentation (Figure 7). This aimed to raise the awareness of the website within the

class so that students could use it to help with their design work. Some of the

students had been on the website but not seen all of the specific areas, such as the

materials selection pages. However the majority had not been on the website at all.

The PowerPoint presentation illustrated the various pages of the website available to

students, highlighting areas that may prove useful in their design work. Further trials

have been planned to assess the impact of this group input on the website use and

content.

14

5 The second action research cycle: consultancy sessions After the second action research cycle, classroom inputs, it was felt that the next

phase of analysis should involve an assessment of whether the actions taken during

classroom inputs had any affect in improving the use of the SDA website (number ‘8’

- Figure 4). The analysis was also designed to enable a comparison of the initial

findings that looked at website use. This information was gathered through

questionnaires similar to those used in the classroom inputs (Figure 5).

Figure 8: An example of the second website use questionnaire

Whilst these questionnaires (example provided in Figure 8) were being completed,

detailed notes were taken when looking at the student’s design folio work for their

advanced level major project. Photographs were taken of this work (Figure 9), and

where possible copies of the work were collected. The notes (Figure 10) highlighted

any reference to use of the internet and any reference to sustainability issues within

the student design folios. The notes were cross-referenced with these photographs

and folio copies.

15

Figure 9: Example of the photographs taken of the folio work

Once these notes were completed further additions were made to the notes by the

students who were asked some follow-up questions to help expand on any key areas

of their work. Again the students were able to point out any moments during their

folio work where they used websites or referred to sustainable design principles.

Another action was then taken by advising the students of specific areas of the SDA

website that would help them with their project work. This advice was again

documented as part of the research data (Figure 10).

16

Figure 10: Example of the notes taken from the folio work

The consultancy sessions enabled a more detailed analysis of website use during

student’s design folio work. It helped to indicate what students were using the

websites for, the type of websites they used and when they used them. It also

revealed how sustainability has started to integrate into mainstream Design and

Technology education. The questionnaires enabled a comparison of the initial

information gathered on student website use with their use after the initial input on

the SDA website. It also helped to indicate what the students look for when

accessing the websites during their design folio work.

17

6 Conclusion and Future actions

As a research method ‘action research’ has proved an invaluable tool in assessing

website use at post-16 education level. Action Research does have its disadvantages

as it can be seen as a very specific form of research of which the findings may only

be applicable to a very narrow set of users or circumstances. For the study outlined

in this paper the method of research has been suitable, as it is within a defined user

group that the study was designed to address. Its nature has enabled an informed

response, with questionnaires and design consultancy feedback being used to help

indicate several key trends in website use with 16+ students.

The action research process in this instance took the form of gathering data, taking

an action, evaluating that action and then gathering more data. This cycle of

evaluation and data gathering allowed for a reassessment of the students work to

help to confirm the findings of the initial studies. The iterative nature of action

research also allowed for several key variables to be altered to help to reach

conclusions. It also indicated areas that need more investigation.

The classroom inputs that were part of the second action research cycle, did indicate

that the actions taken had had an impact on improving sustainable design website

use within classrooms. The consultancy sessions which were part of the second

action research cycle focussed more on one-to-one consultancy but based around

the same questions. This allowed for a comparison between the two trials. Similar

trials could then be made for different audiences to reach more generalised

conclusions across a broader range of designers.

The major benefit of the research method is that the information is gathered through

direct contact with people in a natural environment. This allows for a natural

assessment with the students performing acts which are part of their coursework

rather than the researcher creating staged or experimental studies on which to gain

feedback. It allows for observation, questionnaires and one to one consultancy

without hindering the validity of its outcomes. Action research has enabled a unique

assessment of website use with users and their folio work. The information generated

is in varying forms; photographs, copy of folios, user notes, discussion, focus groups

and questionnaires. This presents the whole picture of designing through differing

methods that reflect the act of designing itself.

18

The main SDA programme funding ran from 2002-2007, and the SDA steering

committee produced a number of resources that aimed to continue beyond this

funding as self-running activities. The scheme would continue but through use of the

SDA website, the Sustainability Handbook and other similar resources. The training

of sustainable design teachers within different areas of the United Kingdom would

also play a role in the success of sustainability within schools.

As indicated in this paper (Figure 6) recent research has indicated that only 28% of

the students questioned have used the SDA website. This is an area that clearly

needs addressing if the SDA scheme is to rely on the website for its continuation.

Future actions will look at the ‘before use’ phase of website usability, focussing on

awareness raising at undergraduate and professional level. This will enable an

iterative style of research using the same actions and analysis, but with a different

target audience.

19

7 References

Archer, B. (1992) 'The Nature of Research into Design and Technology Education'. Design Curriculum Matters. Loughborough, Department of Design and Technology, Loughborough University. Archer, B. (2004) Designerly Activity and Higher Degrees: seminar papers from a staff development short course. Wellesbourne, The Design and Technology Association. Bhamra, T. (2004) 'Ecodesign: the search for new strategies in product development.' Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 218(5): 557-569. Brezet, H. and C. v. Hemel (1997) Ecodesign: A promising approach to sustainable production and consumption. Paris, United Nations Environment Programme (UNEP) - Industry and Environment. Burns, R. B. (2000) Introduction to Research Methods. London, Sage Publications Ltd. Capewell, I. and E. Norman (2003) 'The Sustainable Design Award: Supporting 16 plus Students in Addressing Sustainable Design Issues.' The Journal of Design and Technology Education 8(2): 82-91. Clare, D. (2001) 'DEMI - Supporting learning and teaching for sustainable design.' Art Design and Communication: Learning and Teaching Support Network(February): 4-5. Daniel, K. (2007) The Sustainability Handbook for D&T Teachers. Rugby, UK, Practical Action. Elliott, J. (1991) Action Research for Educational Change. Milton Keynes, Open University Press. Gerstakis, J., H. Lewis, et al. (1997) A guide to eco-redesign-improving the environmental performance of manufactured products. Melbourne, Australia, Centre for Design at Royal Melbourne Institute of Technology (RMIT). Goedkoop, M. (1995) The Eco-Indicator’95. Final report Amerstfoort, The Netherlands, Pré Consultants B.V. Green, K. (1998) 'Action research in a dynamic context: problems and possibilities'. International Conference of Design and Technology Educational Research and Curriculum Development '98, Loughborough University, UK. Green, K. (1999) 'Defining the Field of Literature in Action Research: a personal approach.' Educational Action Research 7(1): 105-124. Kemmis, S. and R. McTaggart (1988) The Action Research Planner. Deakin University, Geelong.

20

Norman, E. (1999) 'Action research concerning technology for design and associated pedagogy.' Educational Action Research - an International Journal 7(2): 297-308. Pitt, J. and F. Lubben (2007) Evaluation of the Sustainable Design Award Project York, University of York: 1-53. Roberts, P. H. (2000) 'Aspects of research concerning design education'. Design and Technology Educational Research and Curriculum Development: The Emerging International Research Agenda. E. W. L. Norman and P. H. Roberts. Loughborough, Department of Design and Technology, Loughborough University. Simmons, P. and K. Badni (2007) 'A review of the literature concerning website effectiveness: before, during and after use'. E-learning in Science and Design and Technology: International Conference of Design and Technology Educational Research and Curriculum Development Online, Loughborough University, The Department of Design & Technology, Loughborough University. Simmons, P. C. and K. Badni (2007) 'The Use of Sustainable Design Websites Within Design and Technology Education at AS/A2 Level'. Linking Learning: The Design and Technology Association Education and International Research Conference 2007, University of Wolverhampton, Telford, The Design and Technology Association. Smith, L. M. (2004) 'Yesterday, Today, Tomorrow: reflections on action research and qualitative inquiry.' Educational Action Research 12(2): 175-195.

21