« Best practices in museum education and cultural programmes »

Planning, developing and evaluating a programme

Marie-Clarté O’Neill and Colette DufresneTassé

ForewordThis document is a practical tool to assist and focus the planning, development and evaluation of education and cultural programmes.Its purpose is to support professionals, students, volunteers and evaluators in identifying and analyzing all aspects of their projects.NB1. The summary acts as a quick guide of the frameworkNB2. The various steps will be both described and illustrated in italics

ContentsIntroduction: education and cultural programmes, definition and role

1. What is an education and cultural programme?2. Role of education and cultural programmes3. What influences the programmes?

I. Design stage1. Background information

Type of museum Type of collection(s) Museum vision and mission Key reasons for developing and delivering a programme2. Incentive to developing a program

2. Target audience

2. Relevance and rationale

How does it relate to the museum's mission, strategies and plans: the institutional relevance, How does it relate to the museum's collections and stories:

the scientific relevance How does it relate to the audiences? What is its perceived social value?

the social relevance What is its perceived economic value?

How is it funded? The budgetary rationale

5. Partnerships

5. Resources

Logistics Financial Human

1

Timing7. Aims and objectives7. Content

What is the content? Awareness of possibilities Selection Presentation

9. Means of deliveryWhat means are used to deliver the programme?

Awareness of possibilities Selection Presentation

10. Audience participation10. Coordinating the different elements of the programme: content, means, audience10. Research and evaluation

Accompanying research Assessment research

13. Projecting operation13. Communication & marketing Internal - External

II. Delivering the programmeOperations

Time structure Equipment Stakeholders Partnerships' agreements Communication & Marketing Budget Logistics Health & safety Finance monitoring Audience management Content and methods of delivery

III. Evaluation1. Recording and analyzing the data

Data collection Data description and analysis Interpretation Validity (criticism) of the data Conclusion and follow up

b. Dissemination of findings Short presentation Full report

IV. Implementing the evaluation Reasons for remediating From evaluation to remediation

Conclusion

2

1. Introduction : education and cultural programmes, definition and roles1. What is an education or cultural programme?

Museum services can be grouped in different ways which include: Academic research and expertiseCollection, exhibition, premises Public programmes

This framework is designed to support programmes that are inspired by, connected to and targeted at both audiences and museum resources.

2. Role of education and cultural programmesThe educational function of the museum is the development and fulfillment of the people. The museum can perform this function in two ways: By offering programmes to which it assigns specific goals and that are conceived to achieve each goal. By convention these programmes are called educational programmes. By offering programmes to which it assigns more general goals, (beneficiaries getting to the content as they wish). By convention, these programs are called cultural programmes.

Such programmes are named differently depending on the geographical areas: education programmes, learning programmes, projects, productions, etc. For example, Arts Council England supports the use of “The Inspiring Learning for All” framework, which is built on a broad and inclusive definition of learning adapted from the Campaign for Learning.

It identifies that:Learning is a process of active engagement with experienceIt is one of the things people do when they want to make sense of the worldIt may involve the development or deepening of skills, knowledge, understanding, values,

ideas and feelingshttp://www.artscouncil.org.uk/whatwedo/supportingmuseums/ilfa/defininglearning/

Other countries will stress more clearly the use of the term “education” as a process coveringnon formal education for all types of audiences.

Whatever their diverse names, these programmes support education for specific target audiences and they can:

Support the engagement with permanent displays and temporary exhibitions Target audiences that are not catered to in the exhibitionsAttract new audiences

Education or cultural programmes can take place inside or outside the museum and can be one-off sessions or a series of sessions.

3. Who and what influence the creation of programmes?Recommendations of international organizations (e.g. ICOM)Governments and other national policies Academic research and findingsSocial trendsSocial pressures (development of events production, marketing, new technologies, considering physical, intellectual and social handicaps)Policies (influence of ideologies or of cultural diplomacy)

3

1. Design stage

This essential phase includes the following: 1) Background information, 2) Incentive to developing a program, 3) Target audiences,

4) Relevance and rationale, 5) Partnership, 6) Resources, 7) Aims and objectives, 8) Content, 9) Means of delivery, 10) Audience participation, 11) Coordinating the different elements of the programme, 12) Research and evaluation, 13) Projecting operation, 14) Communication and marketing.

1. Background informationInformation about the museum (size, staff size, visitor numbers, etc.)Type of collections (Nature of exhibits, permanent collection, temporary exhibition, archeological field, etc.)Museum vision and missionKey reasons for developing and delivering the programme

Content (complex topic)Visitors (type or expectations)Institution (a museum poorly appreciated by its visitors)

2. Incentive to developing a programmeWhat are the starting reasons for elaborating a programme?

The programme has been required by an association.Any more specific reasons?

External requestsA teacher asks for a visit with a particular theme related to

his/her class’s project.Incentive of the Ministry of Culture to offer events related to the "China Year"

Internal requestsStaff decision

Opening of a new exhibitionInterest in developing a programme for a special needs group

Large presence of family groups on certain days in the galleries, without a suitable offer

Evaluation and audience research findingsEncouraging results of an evaluation conducted by colleagues on Treasure Hunt documents

3. Target audiencesThe target audiences are those for whom the programme is conceived. Museums cannot cater to all audiences, in all of their activities, all of the time. An audience development policy identifies the nature of audiences considered as the most important at particular time and in particular venues and how they relate to each other. This audience development policy is linked to the overall policy of the institution.

Target audiences are potential applicants for a programme that is offered by the museum based on its resources. The museum may respond to a need felt by a particular group(s). This can be personal, institutional or political, but needs approval and to support the museum's policy: social relevance, resources in terms of collections, of staff or resources, etc.

Tourism providers in a region ask a local museum to organize guided tours in foreign languages.

4

Or the museum initiates a programme of its own. In all cases, it is necessary to specify the target audience(s)

4. Relevance and rationale After having defined the main objectives of the programme and chosen its target audience, it is necessary to consider the relevance of this program, i.e. its compliance with certain requirements. These can be institutional, scientific, social or budgetary.

Institutional relevanceThis touches on the project's relationship with the policy of the institution and with its means, that is to say:o The strengths and weaknesses of the existing resources in the institution: specificity of collections, scientific and professional resources

Organization of a programme on the ongoing conservation process rather than a proposal of a costly event without a strong cultural link to the collections

o The actual or sought image of the institutionCan a museum with a recognized expertise in new technologies offer a program based on a poor quality video game?

o The institution's management choicesCan a museum seeking to expand its ticketing financial resources provide a free access programme?

o The requirements and external pressuresThe priority to welcome certain school groups (primary, secondary) is selected in accordance with the administrative authority responsible for each grade level (city, region) and the administrative authority governing the museum (Municipal museum, National museum.)

Note: However, even though it is essential that programmes relate to the institution's aims and strategies, they can also act as catalysts for change and be integral to its evolution.

Academic/curatorial relevance _One of the reasons for the development of programmes is to bridge curatorial expertise and academic research to non specialist audiences. In order to maintain museum expertise and subject authority, education programmes are based on current curatorial knowledge and are checked and endorsed by the relevant experts.

Updating information given to the public in relation with the major developments of the archaeological discipline.

The quality of the programme is often based on the quality, uniqueness and the authority of the content.

Social relevanceIt is important to have a clear idea of the society’s characteristics and of the diverse museum and public services representations about the possible social role of a museum.How is the programme relevant to its social environment/the society it serves?

The museum is located in a rural or urban environment, with or without the presence of a multicultural community, in an economically rich or deprived environment, in a touristic area, in a specific political situation, etc.

Economic relevanceHow is the programme funded and how does this relate to the overall funding of the institution? What is its perceived economic value?

5

5. PartnershipWhat collaborations and partnerships are developed as part of the programme and what is their short, medium and long term impact?Partnerships can be internal (e.g. with other departments) and external (e.g. other organizations, community groups, etc)

To decide whether collaboration is beneficial, one can focus on the following questions: What is the purpose of the partnership?What are the benefits of the partnership for the organization, the individuals and the audiences?What are the challenges/limitations of the partnership?

Things to consider include: choice of partner(s), resources, roles and responsibilities, formal or informal agreements and an exit strategy

Is the ethical dimension considered in the same way by the various partners? What relative importance will be given by each of the partners to the economic fallout in comparison with social or cultural benefits?

6. ResourcesThe size and nature of resources depend on the characteristics and profile of the institution (size, geography, finances, collections, nature of personnel, mission, vision and strategies)Some of the things to consider include:

Logistics:Reception facilities tailored to the target audience

Orientation and signageOrganization and availability of space and orientation signs

SpaceMaterial possibility of having a specific space for workshops versus

organizing educational workshops in the museum galleriesCloakroom for groups, catering areas, toilets for disabled, baby

change.

Necessary equipmentSimultaneous translation equipment, artist supplies, costumes

Financial resources:BudgetPossibility for sponsorship (money or in kind)

Invite a celebrity or involve a local actor, favoring the use of professionals rather than volunteersTake advantage of local products available for free.

Human resources:Quantity of trained staff and/or volunteers needed Characteristics and competencies of staff with audiences

Professional staff available internally and/or externally This type of professional is preferred, despite the cost, given their scientific and communication skills and availability. They can be: - Scientific staff of the museum (conservators, curators, librarians, researchers in science museums). Their availability and ability to communicate with the public needs to be addressed. - Practicing teachers: their competence to engage audiences in a non formal learning setting needs to be addressed.

6

Different ways of approaching an historical figure at school or at the museum

- Other museum staff who are skillful/have been trained in communicating with different audiences Museum guards giving tour - External and other specialists that are not only exceptional in their field but also effective communicators.

What impact on the public will the presence of one of the artists have as opposed to the act of creation presented by an art historian?

VolunteersThe importance of their role and expected degree of scientific and communication skills may vary widely between countries and institutions. Volunteers need to have ongoing training, support and monitoring. Timing

Certain periods or specific dates (different in each country) provide museum opportunities for specific programming and attracting particular audiences.

Mother's Day, International Museum Day, school holidays.

7. Aims and objectives

They can be General objectives (developing observation skills)Specific objectives (to recognize the production of an artist)

Different aspects of personal development can be pursued, from sensitivity stimulation to the acquisition of know-how.Outcomes for participants can be perceptual, cognitive, affective and social. Here are some possibilities:

PerceptualTo increase observation skills

CognitiveTo develop comparison skills

AffectiveTo inspire curiosity

SocialTo develop empathy

Other types of outcome can be :

Increased knowledge Improved know-how Keener experiential emotions, etc

Leading, eventually, to the following developments: Perceptual

Increase in form sensitivityKnowledge: what to observe and why? Know-how: how to observe?

CognitiveIncrease in sensitivity in information needsKnowledge: new learning as expected by the programme. Know-how: Compare/how to compare.

Affective Increase in sensitivity to objects’ beauty

7

Knowledge: identifying one’s tastes.Know-how: How to express one’s tastes.

SocialIncrease in sensitivity to others Knowledge: Knowledge about the political structure of a given cultural group.Know-how: how to interact with people from other cultural groups.

8. ContentThe content covers all kinds of information and/or experiences offered throughout the programme.

One may, in a natural history exhibition, choose to offer a guided tour about very different types of information: beauty of plants, botanical characteristics, territories of origin, social use, etc.

One way of working with content includes the following three steps: awareness of possibilities, selection and presentation.

Awareness of possibilitiesThe programme developer becomes aware of the extent and of the variety of collections, stories and facts about the subject matter.

An Egyptian archeology collection can be presented under an anthropological angle, or archaeological, aesthetic, theological, etc.

Selection among the possibleOne guided tour of an extensive Egyptian department will choose to focus on daily life in the time of the pharaohs

This selection may be made according to various criteria: relevance to the target audience, relevance to the content of exhibitions, relevance to the museum policiesIn any case, the information provided in the programme is based on fundamental notions rather than anecdotal notions.

Archaeology presented as a process of interpretation of history versus giving information only on specific results of archaeological research

Relevance of content to the target audience, considering their preexisting knowledge and understanding, needs and expectations and ways of learning

A program on the ”Western representation of women in art” for literacy workshops for newcomers, mostly female

Relevance of content to the displays/exhibitionsSeveral options are at hand:

Stick to the content of the displays/exhibitions Trace the career of an artist in a conference scheduled around a monographic exhibition.

Develop additional contentDevelop anthropological aspects or provide contextual elements in an exhibition presenting the results of recent archaeological excavations

Develop alternative approaches to the contentFor very young children, provide a programme around the

recognition of animals in an exhibition on European still lifes of the 17th century

Relevance to the museum policies.Link policies and strategies (organizational, local, national, international)

The director of a natural history museum chooses to discuss biodiversity rather than the beauty of plants.

8

Relevance to the collections and the non formal learning environment of the museum.

Difference between a programme running in the galleries and another set in an auditorium or in a classroom.

The content needs to reflect the quality and academic authority of the institution without underestimating the needs and abilities of the target audiences.

Do not limit yourself to have very young children count rabbits represented in a tapestry, (to develop observation skills), but also develop other essential skills such as understanding iconography and symbolism.

Presentation of the contentThe aim is to build a dynamic and coherent programmeThe content of the programme, including the use of stories and collections, can be organized in

different ways: The story of an evolution (Painting kings throughout history)., A continuous story told on the base of objects (The Napoleonic era in a historical site). The comparison approach (Comparison of style evolution between Matisse and Picasso). The typological approach. (The shapes and decorations of ancient Greek vases), etc.

9. Means of delivery

Choosing the means of delivery uses similar processes to choosing the content: be aware of the variety of choice, set specific criteria for your choice of means, structure the programme.

Variety of choiceThere are unlimited ways to connect and engage with audiences.

Conferences, workshops, demonstrations by artists or artisans can be used concerning the same content for different audiences

Criteria for your choiceThe criteria when choosing a method, include:

Audience suitabilityWhat is the most suitable way to engage young people?

Content suitability What is the best way to share the aesthetics of stained glass?

Innovation.Innovation should not be an end in itself, but it might be useful and needed when: Attracting new and different audiences

Programme for moms and babies Exploring a new topic

Mime as an interesting vehicle for understanding balance in sculpture

Structuring the programmeUsing several means in a programme requires careful orderingThings to consider are:

The order of the activities offeredDo we choose to place gallery visits or workshops first?

9

The intellectual flow of the activitiesWhat is most effective to do first?

The interrelationship between the sessions within a single program.The programs may consist of one or more sessions. In the case of multiple sessions it is desirable that they be designed in a complementary way, following the same structure, when appropriate.

10. Audience participationAudiences may be asked to be active participants and interact with the museum staff, other participants and/or the activities.

Parents and children working together as a familyParticipation could occur in different ways:

Between public and museum staff Between participants Between participants and delivery means

From questions asked by the education officer to the public, to parents and children working together in a family visit, to treasure hunts or activities around a kiosk.

Through the interaction, participants can share experiences, exchange ideas and experience and understand concepts in different ways.

A group of students creates a collective work after a visit to an exhibition on the sunMime as an experimental approach to the dynamics of the sculpted body

Both frequency and intensity of interaction have to be considered.

11. Coordinating the different elements of the programmeThe ultimate aim is for the public to engage with the programme. This is done through articulating, in the most effective way, content, means of delivery, interaction types and level of involvement of the participants.

12. Research and evaluationAudience research and evaluation is a cycle that needs to take place throughout the programme.

In the anglo saxon tradition, the four main stages are the following: Literature review Front end evaluation Formative evaluation Summative evaluation

See more at: http://www.nhm.ac.uk/aboutus/visitorresearchevaluation.html#sthash.tVrSCpkw.dpufThe Visitor Studies Group in the UK and the Visitor Studies Association in the US have a wealth of information one can use.

The orientation taken here is different. When one thinks of research around a programme, one usually considers the evaluation of its results. This is only one of the many forms of investigation suggested here. One should carry out accompanying research during the program’s design and implementation phases, then assessment research to know its results.

Accompanying researchThis research include (a) simple gathering of information already available, (b) systematic collection of information selected for its rigor and duly synthesized (c) collection of new information through surveys .

1. Simple gathering of information already available

10

In archives, reports, statistics, to precisely understand/ know the museum’s environment In the documents produced by the museum in order to:

Specify the museum orientation and requirements to reach a harmonious insertion of the programme into that orientation. (Financial requirements) Discover opportunities for internal collaboration.

b. Systematic collection and synthesis of rigorous informationThis includes

Scientific information to ensure the accuracy and quality of the programme content Psychological and sociological information, to adapt the programme to the participants

characteristics Museological information to ensure that the various aspects of the programme are

implemented according to the experts’ opinions .c. Collection of new information through surveys, for example: The specific characteristics of visitors to whom the programme is offered Their expectations Their prior knowledge and representations

One can also carry out tests on critical aspects of the programme, these tests often leading to changesduring the design phase. These tests are called formative evaluation.

Assessment researchThis research is based on questions regarding:

1. Aims of the data collection Checking to what extent the aims of the programme have been reached (This is called

summative evaluation)Objectives regarding the participants

Understanding how to fly a planeObjectives regarding the institution

Renewing the image of the museum Identifying other possible impacts (positive, negative or neutral) on the participants or the

museum. Detailing specific benefits for the participants or the museum.b. Moments of data collection During the program:

Observing behaviors or situationsResults of activities (sorting, treasure hunts, etc.)Productions of the participants (oral, artistic, written, photographic, film etc.)

Remark: Any deviation from what was planned (activities, premises, staff) should be noted, as this could explain results.

When the programme is over:Questionnaire, interview or activities that could contribute to the assessment.

c. Means of collecting and processing. Choosing a technique Techniques based on observation Techniques based on questioning: questionnaires, Interviews, focus group Techniques based on “thinking aloud” practices Techniques based on the analysis of the participants’ productions

Note: For detailed information regarding these techniques, use the methodological publications in sociology, psychology or education.

Persons from whom information is collected. Gathering could be done from all the participants Gathering could be done from a sample of participants according to age, social status,

expectations, prior knowledge and expertise in museum visiting, etc. The sample should

11

include enough subjects to allow subdivisions into several categories and comparisons between subgroups.

Ways of processing the information gatheredThe information could be analyzed:

quantitativelyHow many visitors return for a second workshop?

qualitativelyThe explanations given by these visitors to explain their return.

13. Projecting operationsScheduling, taking into account what else is happening externally & internally

Forecasts for the material aspects of the programme imply: Time management: Integration of the programme within the overall calendar of activities of the institution

Will the conference scheduled in the auditorium be finished in time for the museum closing procedures?

Adopted agenda suitable to audiences availabilities When to provide a programme to very young children?

Management of material resources.What audiovisual equipment is needed for the proposed programme?

Managing people:d. Audience management:

It implies projecting organization, both for the public concerned by the programme, as for any otheraudience present in the institution.

e. Management of the professionals involved:Availability and expertise should be considered concomitantly.

When will the best speaker on this subject be available?

14. Communication and marketingThe programme has to be communicated, as part of a wider strategy, both internally and externally.

Internally, it will enable other departments and teams to know what is going on and to cooperate. Externally, it will ensure that potential audiences know about it.

One needs to: find out the best way to communicate with audiences

Choice of internet information for young audiences, selection of posters for schools, public websites for multi handicapped audiences

When to provide a programme and send the information to the public schools?

plan for it accordinglyBudgeting for communication is also very important.

Relative cost of advertising, by mail, by poster, by radio spots, by internet

These aspects of communication may vary depending on: The location of the museum

Relevance of a display of advertising posters according to the territorial coverage of a museum

The financial means available

12

Relative cost of an advertisement, by mail, by poster, by radio spots, by internet

Program circumstancesPossibly different methods of dissemination for occasional or regular programming.

Choosing to communicate to the target audience or to its relaysAssociations of disabled vs. disabled individuals, individual teacher or school administration versus central information to the Ministry of Education

2 . Delivering the programme

This is the implementation of the design/planning and its adaptation to the programme sequence from possible to reality.

OperationsLeading operations making the programme a reality.

Initiate a new Sunday family programme, or Mother's Day programme, with the financial support of a baby food company and the support of grandmothers who are members of the Association of Friends of the Museum.

Items to consider include: Time structure

Creating the timetable and scheduling each task and the length of time it needs Equipment

Identifying the needs. Acquiring the equipmentEnsuring that it is in place and returning it

StakeholdersIdentifying the key stakeholders, communicating with them, training them and debriefing them

Partnership agreementThe convention can cover multiple aspects:

Selection of the partner(s) Legal, administrative, finance, insurance aspects Creation of an agreement/Memorandum of Understanding Communication and marketing

Delivering, monitoring and assessing communications.People communicating internally with other teams

Security officers are informed and they agree to conduct?? a dance workshop in the museum galleries

BudgetRecording and monitoring the budget

LogisticsAssessing the logistical needs on the day

Queue management, orientation, signage Health and safety: Completing a risk assessment – identifying potential risks and the actions taken to reduce/mitigate them.

Finance monitoring

Content and methods of deliveryWhat relationship can exist between the designed scenario and the implemented scenario?

13

Strictly follow the indications about content, exhibits to be used, activities organization, time allotted to each activity. Observe part of previously chosen objectives either in terms of educational approach (asking question, using mime), or concepts to be acquired (stratigraphy in archeology), or exhibits to be presented, activity calendar, time allotment, etc.

Adapting exclusively to desires expressed by visitors

2. 3. Evaluation

1. Recording and analyzing the dataProcessing of the data is organized into five distinct and successive stages: data collection, data description, data interpretation, data validity, conclusions and follow-up to the study.

o Data collectionData collecting means compiling information that has been collected according to the questions raised.

o Data descriptionData description means presenting them systematically.Data can answer questions that were formulated but can also provide information that was not anticipated.

70% of visitors of a guided tour express a desire to return. But 40% of these visitors express the desire for a tour made by another staff member who would give more details and speak more slowly.

One begins by describing data that can answer the formulated questions and then introduces unanticipated items.

o Data interpretationData interpretation should/is used to reveal the meaning of the data by placing them in context.

70% of visitors expressed a desire to return, which is a significant proportion compared with results found previously (55%). The importance of the intervening style is also worth considering as it is mentioned by 40% of the visitors.

o Validity or criticism of dataCritical data should/is used to ask what degree of trust results can provide

The preceding data have been collected from a very small number of visitors surveyed (15). One should check them on a larger sample.

o Conclusion and follow-upIt is appropriate to draw conclusions based on the data collection and on their criticism

For more conclusive results, this study should include a higher number of visitors and check if the staff members are aware that speaking fast is negatively received from a high proportion of the visitors

b. Dissemination of findings

This is the description of the study carried out. This description may take several forms: (a) a summary report for the director and the management team, where one mainly focuses on the results and their significance for the institution; (b) a complete report for the institution's archives and /or publication;

1. Short presentationA text (5 to 10 pages) that briefly recalls the subject of the study and how it was carried out, in order to devote the most possible space to the description of results, their significance and their

14

possible follow ups. Reading and understanding of the results is usually facilitated by a presentation in the form of enumerations.

b. Full reportIt will include a description of the institutional context of the study; a short presentation of it, including questions it tries to answer and the reasons for the decisions; descriptions of tools used and how they were used; an identification of the source of the information gathered (groups, samples, etc); the description, interpretation and criticism of the results, and finally the follow up to the study.

3. 4. Implementation of the evaluation

In relation with the evaluation results, the programme carried out may have to undergo extensive changes, called remediation.

Remediation may relate to:the programme goals

The museum is not, as we had thought, a place appropriate for formal learning,

its content in relation with the target audienceToo abstract, and ask for the reading of too many guidelines

its characteristicsToo long a program leading to the removal of one of its sessions

its stakeholders’ needsThe primary school programme has changed and the teachers are asking for programmes dealing with different subject matters

It may result in immediate , medium term or long term changesFrom evaluation to remediation

Remediation is not necessarily completely dependent on the assessment study:Taking into account the results of satisfaction, but not learning outcomesDesire expressed by visitors to touch the works to better appreciate them

Conclusion

Despite some concerns over the relative heaviness of this framework, its usefulness, for example in the "CECA exemplary programs award " has been clearly highlighted.Benefits include: A greater number of aspects of a programme design and implementation taken into account A clearer and more detailed description of a programme implementation A greater attention paid to the pendant choices A greater importance attached to the consequences A better use of evaluation, of critical thinking, leading to more numerous attempts of remediation

Considering all these aspects allows professionals, at every stage, to take the best decisions possible, given the specific circumstances in which they are working.

15

For the researcher, the detailed analytical description can identify aspects worth thinking about and appraising. It also facilitates the creation of assumptions and enriches the possibilities of interpretation.

This framework provides the elements needed to build/produce/conceive an exemplary project (Best Practices). The variety of aspects and factors considered lead necessarily to the most appropriate decisions, given the resources we have, the people we address, the goals we pursue and, the pressures we can experience. These practices, which proceed by progressive questioning, provide a project the best possible chance of successto succeed and often engender important scientific discoveries.

16