What impact do your science activities have on visitors? Is your institution contributing to science engagement policies? What do staff, scientists, teachers and other actors involved get from participating in science communication events? How do these activities impact on your city or region?

Whether you use the results to improve your offer or prove your relevance to stakeholders and decision makers, impact evaluation can be a powerful tool. And that is exactly what the Impact Assessment Toolkit – developed through the European project PLACES – is all about.

What is PLACES?
PLACES is a four-year EU-funded project aimed at defining, developing and promoting the European City of Scientific Culture. It offers a common platform to dozens of local networks through which stakeholders can combine their efforts to structure science communication activities and help create instruments for the study of science communication initiatives and the impact of policies. PLACES is coordinated by Ecsite (European Network of Science Centres and Museums), in partnership with the European Regions Research and Innovation Network (ERRIN), European Science Events Association (EUSEA) and the Science Communication Observatory from Universitat Pompeu Fabra (OCC-UPF) in Barcelona.

As part of the PLACES project, a group of 28 independent researchers with experience in the field of Science in Society developed a method of assessing the impact of Science Communication Initiatives and Policies (SCIP). This group aimed to investigate and gather evidence about SCIP in Europe to establish common methodologies and collect recommendations for future activities. With these objectives in mind, they created The PLACES Impact Assessment Toolkit under the coordination of the Science Communication Observatory from Universitat Pompeu Fabra (OCC-UPF). As a result, 27 case studies were conducted using the Toolkit in 20 European countries, involving more than 50 investigators from 10 science centres and museums, eight science events and nine cities of scientific culture.

What is the benefit of the Toolkit?
The PLACES Toolkit allows users to evaluate both the individual effects of SCIP and their long-term, additive and cumulative effects. This gives the project a deeper scope, as a broader spectrum is gained when both kinds of impact are considered.

How can you use it?
The Toolkit can be found online at www.occ.upf.edu/places and is free to use. It offers a methodological matrix (see image above), combining quantitative and qualitative instruments that can be used online to explore any kind of impact. These include standardised surveys, semi-structured interviews and focus groups, in addition to institutional sources and documental analyses. All questions and surveys have been written for the user – all you have to do is simply approach your visitors, scientists or stakeholders and start collecting data. The modules have been translated into several European languages.

The Toolkit consists of a 3x3 grid. Depending on who is running the event or which policy you want to evaluate (ie science centre or museum, science event, or cities of scientific culture) and who the target audience is (ie public sphere, political sphere, or factors involved in SCIP), you will be advised to use different modules.

For instance, if you want to gauge the impact that spending a day at your institution has on individual visitors, you will be advised to run interviews immediately after their visit (module A1) and/or ask them to fill in a survey (module A2), whilst also looking into institutional resources. However, if you want to gain an insight into the impact that participating in a science festival has on the actors involved, it’s recommended to conduct in-depth semi-structured interviews (module C1) and/or to organise a focus group with their representatives (module C2).

One of the main features of the interactive version of the online Toolkit is that it can be used in situ, which is very convenient when visitors are being surveyed or interviewed immediately post-visit. Tablets and smart phones can be used to fill in the questionnaires, and results can then be sent directly to the evaluator’s email.

If you’d like to check case studies similar to yours, the series of 27 evaluations previously carried out can be viewed online and downloaded. The content of the Toolkit sections has been adapted to each individual case to make it more accurate and also to ensure it better addresses its target.

What do the results show?
An analysis of the existing 27 case studies has shown a remarkable “socialising” effect of SCIP on visitors of science centres or museums, science events, or cities of scientific culture. Thus, when visiting a science centre or science event:

- Visitors contribute to the “normalisation” of science, as visits to science centres or events become part of the leisure and cultural time of families or groups.

- Ties within families and groups of friends are strengthened.

The main contributions of the research are related to the study of SCIP’s impact on a local or city dimension.

- All actors involved consider that local policies promoting science culture already play – or are going to play – an important role in the economic development and visibility of the city.

- Citizens perceive science centres, museums and events as significant symbols of their town, especially in those regarded as scientifically cultured.

Moreover, results have confirmed that:
- SCIP have a cognitive impact on adults and children (that is, the positive learning effects that they induce).
- SCIP boost intellectual curiosity, increase self-esteem when talking about science issues and enhance scientific vocations.

Regarding the issue of education, a stakeholder involved in one of the case studies stated: “There has to be strong engagement at that early, formative stage. We need to promote a culture of engagement with science from the earliest possible opportunity” (1). In another, a researcher said that school visits to museums “are one of the most important tasks that these centres can accomplish” (2). A teacher involved in a science event stated: “it enhances the feeling of responsibility and democratic citizenship of students inside and outside the classroom“ (3).

The study also showed that there’s a strong impact on actors involved in SCIP (scientists, teachers, business people, journalists, politicians, centre staff). Such activities stimulate networking among actors, promote the creation of new projects, facilitate access to new financial resources, help understanding of audience’s needs and improve professional skills.

A researcher participating in a semi-structured interview stated that science events are a great experience “to learn about the work of my university colleagues and from other institutions” (4), while a media person said: “prejudices and over-expectations towards science can only be changed if science does not stay isolated in laboratories and institutes” (5).

Any recommendations?
As part of the PLACES project, a set of recommendations has been compiled and can be found online. Its aim is to contribute to the planning, implementation and evaluation of future SCIP. Advice is organised into sections: objectives, targets, venues, areas/issues, timing, formats/ways, local dimension, promotion/advertisement/communication, financing and evaluation.

Planning an event soon?
Easily implementable recommendations:
- Clearly define your objectives.
- Explicitly state your evaluation process
- Identify audiences and adapt content/planning to their needs/expectations.
- Science communication from museums and science centres is important to your city – take it into account when planning actions.
- Present contents in an innovative way with a multidisciplinary approach – visitors appreciate real the opportunity to explore.
- Liaise with local scientific communication agents and promote your acitivites with local print and broadcast media.