The New AMEC Integrated Evaluation Framework A Collection of Coverage from The Measurement Standard

The New AMEC Integrated Evaluation Framework

Welcome to this collection of The Measurement Standard’s coverage of the new AMEC Integrated Evaluation Framework. Here you will learn about what the Framework does and doesn’t do, how to use it, what people think of it so far, and the theory behind it.

Contents:

• Chapter 1: An Introduction to the Framework AMEC Debuts Convenient Online Measurement Tool by Bill Paarlberg

• Chapter 2: An Overview of the Framework, with a Review of Public Commentary Your Guide to the New AMEC Integrated Evaluation Framework by Bill Paarlberg

• Chapter 3: A Discussion of the Framework’s Value The Real Value Behind the New AMEC Integrated Evaluation Framework by Mazen Nahawi

• Chapter 4: A Road Test of the Framework, with Tips for Users A Trial Run of the New AMEC Integrated Evaluation Framework, and 7 Tips on How to Use It by Katie Paine

• Chapter 5: The Theory Behind the Framework Interview with Jim Macnamara: The Theory and Practice Behind the New AMEC Integrated Evaluation Framework

• Chapter 6: The Framework’s Relationship to Models and Standards Is the AMEC Integrated Evaluation Framework the Standard Model? by Fraser Likely

© 2016 CARMA. All rights reserved. 2

The New AMEC Integrated Evaluation Framework

Chapter 1: AMEC Debuts Convenient Online Measurement Tool: The Integrated Evaluation Framework

AMEC has just launched a major upgrade to communications measurement tools and theory: the Integrated Evaluation Framework. Remember the AMEC Valid Metrics Framework, introduced in 2011? (We showed you how to use it here, here, and here.) It was a valuable tool to organize and design communication measurement programs, but many users found it awkward to apply.

The new Integrated Evaluation Framework deals with that objection and a lot more. It’s a free, non-proprietary, intuitive, interactive, and easy-to-use online tool that serves as a theory-enhanced overhaul to the original Valid Metrics Framework. That’s an image of it up top. You answer a few questions in the seven indicated areas, and the tool guides you toward proper measurement and evaluation. It doesn’t do the work for you—there’s

© 2016 CARMA. All rights reserved. 3

The New AMEC Integrated Evaluation Framework

still a lot of careful thinking required— but it does help you get organized, with tips and advice on how to do so.

Jim Macnamara, Professor of Public Communications at the University of Technology, Sydney, describes the most salient new features of the new Framework in this video:

1. It’s a convenient online tool,

2. It incorporates standards and theory from a wide range of fields related to PR and communications, and

3. It includes very specific tips and steps.

This is no one-size-fits-all metric or black box. Rather, in the words of project lead and AMEC Board Director Richard Bagnall, it’s “…a common approach that will work for organisations of all sizes but which can be tailored to very specific user cases and objectives… Anyone can use it…”

It is really so straightforward and user-friendly that the easiest thing to do is just take a look at it here. It comes with its own website full of tips, resources, and supporting materials. Congrats to AMEC.

And if you’d like to read more about how to use the original Valid Metrics Framework:

• How to Choose Tools and Benchmark for Social Media Measurement Using the AMEC Valid Metrics

• How to Measure Social Media Using the AMEC Valid Metrics Framework

• The Valid Metrics Framework, Updated to Include Social Media

© 2016 CARMA. All rights reserved. 4

The New AMEC Integrated Evaluation Framework

Chapter 2: Your Guide to the New AMEC Integrated Evaluation Framework By Bill Paarlberg

The new AMEC Integrated Evaluation Framework debuted in June 2016 at AMEC’s Summit in London. Reaction has been overwhelmingly positive. It’s been hailed by many and most as a major milestone in measurement, with a handful of critiques and caveats (see below). There is no doubt that it moves our industry forward by making sophisticated communications measurement and evaluation easier and more accessible. Here’s your quick guide to what it is and what people think about it so far.

What is the new AMEC Integrated Evaluation Framework?

For most people, the new AMEC Integrated Evaluation Framework will be a practical tool to help them get properly organized to do measurement and evaluation that connects communication results to organizational goals. But it’s also a guide to best practice and a theoretical framework. It builds on AMEC’s earlier Valid Metrics Framework by providing an online interactive front end and a more solid theoretical underpinning.

© 2016 CARMA. All rights reserved. 5

The New AMEC Integrated Evaluation Framework

One of the truly valuable things about this new Framework is that it functions both as a practical tool and an educational guide to best practice. On the one hand it is a convenient apparatus with which to work through the measurement and evaluation process. On the other hand it is an instructive benchmark for the quality (and qualities) that the process ought to include. The excellence that our industry aspires to is now obvious, because it’s built into the tool we use to do the work.

Where it came from

AMEC’s original Valid Metrics Framework debuted in 2011. It was a valuable and comprehensive tool, but many would-be users found it complex and awkward to use. So AMEC’s team of measurement heavy-hitters, led by Richard Bagnall, wisely chose to upgrade it with online and interactive convenience.

In addition, AMEC has included a strong theoretical underpinning, see Chapter 5 of this collection. This academic backbone dovetails nicely with work of the Task Force under the leadership of Fraser Likely, see Chapter 6 of this collection. (By the way, here at The Measurement Standard we are very pleased with this attention to theory. For many years we’ve been calling for measurement to take psychology into greater account, see, for instance, “Eye Contact and Public Relations Measurement’s Empty Head.”)

What it does and doesn’t do

The new Framework helps you get your measurement program organized and off on the right foot by asking the vital questions that connect an organization’s communications to its objectives. The new Framework does not do the hardest work for you, which is researching and answering those vital questions.

How to use it

Here’s a look at some of what the Framework requires that you take into account:

© 2016 CARMA. All rights reserved. 6

The New AMEC Integrated Evaluation Framework

The online interactive site is largely self-explanatory, and it includes some helpful case studies. For more practical advice on how to use the Framework, see Chapter 4 of this collection.

Response to the Framework

Public response to the Framework is strongly positive, enthusiastic, and optimistic. See, for instance, Chapter 3 of this collection.

Some response reflects an unfortunate tendency for PR pros to want a simple solution to what is an unavoidably complex process (for example, see Gini Dietrich’s post, below). Far too many people seem to think or assume that AMEC’s Framework, the Barcelona Principles, and/or recent standards-setting efforts have provided or will provide a single standard, metric, or technique that will make measurement simple and easy. (Shades of AVEs.) That is not the point of the new Framework. It does, however, make it simpler and easier to get organized.

© 2016 CARMA. All rights reserved. 7

The New AMEC Integrated Evaluation Framework

Here is a sample of responses to the new Framework:

• Katie Paine has written a helpful critique of the Framework, see Chapter 4 of this collection. She says she loves the new Framework, but:

“…it will be just as challenging to fill out as the earlier version, because all the problems inherent in the old Framework still exist, despite the sexy new front end. …The vast majority of PR is for small to medium-sized businesses… These people, for the most part, ask questions and struggle with answers that are far more basic than those the Framework deals with. And so the new Framework is going to be difficult for many typical PR pros to use.”

• In “AMEC’s Interactive Evaluation Framework” Michael Blowers provides a nice introduction, and scores the Sound Bite of the Year:

“Will it be recorded as the day PR grew up?”

He includes a few caveats about what the Framework doesn’t do, most importantly:

“It won’t ‘evaluate’ your coverage for you. Think about it more as a checklist, an opportunity to see a structure or a series of questions or suggestions to pick from.”

• A similar point is made by Aaron Mann in “The PR Pro’s Guide to AMEC’s Integrated Evaluation Framework”:

“The AMEC Framework is a leap forward, and it can only be fully realized agency-by-agency, and client-by-client. You aren’t off the hook, but have a fantastic roadmap!”

• Lyndon Johnson, in “Thoughts on AMEC Integrated Evaluation Framework” posts an objection:

“The AMEC framework gives no way for entrepreneurs or communications professionals to experiment. It is simply measuring outcomes based on assumptions. Assumptions that content – paid, earned, shared or owned – is the right way to achieve a commercial outcome. It also measures effectiveness after significant time, money and effort has been spent.

So, while it is a start – a good start – it doesn’t help with the most important piece of the communications jigsaw puzzle. Finding what works. It doesn’t help test assumptions before activity at scale.”

• Gini Dietrich’s Spin Sucks enthusiastic introduction to the Framework is a little heavy on the gushing, and a little light on the significant effort required to make it work:

© 2016 CARMA. All rights reserved. 8

The New AMEC Integrated Evaluation Framework

“[AMEC] is going to make it incredibly easy for you to plan, execute, measure, improve, and report a fully integrated communications program.”

• We’d be remiss if we didn’t include comments from strategist Alan Kelly, who always emphasizes the motivation behind measurement:

“What’s missing is what’s most important: Strategy and Motive. What’s measured is what clients will buy — never the strategy. They ask for candy. Vendors make them candy. Breakthru comes when they buy what matters.”

Where it is going

The PRCA, ICCO and AMEC are setting up a global working group with the aim of providing feedback on AMEC’s Integrated Evaluation Framework. AMEC has plans to translate the Framework into Spanish and Portuguese. We look forward to the Framework’s further development.

© 2016 CARMA. All rights reserved. 9

The New AMEC Integrated Evaluation Framework

Chapter 3: The Real Value Behind the New AMEC Integrated Evaluation Framework by Mazen Nahawi

So, how can a grid with seven colored tiles save PR from its inability to demonstrate results? That seems to be the substance behind most criticisms of the new AMEC Integrated Evaluation Framework.

I believe the question itself is misguided, and so are the critics.

It’s up to you to figure out what works for you and your client. If you know your objectives and are honest about measuring them, you will always find a way to do so.

The new AMEC Framework helps you by outlining an approach that builds on well-established principles. It will help anyone, from a junior practitioner to a

© 2016 CARMA. All rights reserved. 10

The New AMEC Integrated Evaluation Framework

seasoned veteran, measure and evaluate their communications and its impact on business results.

The new AMEC Integrated Evaluation Framework is an excellent measurement and evaluation tool. Here are four reasons why:

1. It builds on the Barcelona Principles

The great value of the Barcelona Principles was its ability to unify most of the industry around a common set of measurement values. It also decisively emptied AVEs of any remaining credibility, even among those who still use them. The Principles laid a foundation from which to push for more qualitative and meaningful measurement and evaluation.

In other words, the Barcelona Principles were invaluable in simply helping our industry unify itself around the right way to measure. No easy task, and certainly a huge accomplishment.

The new Framework, which is based on the Barcelona Principles, allows practitioners to build measurement programs that are anchored in ethics and universally accepted principles.

2. It simplifies the process of measurement

Many have questioned why AVEs persist, despite their widely acknowledged lack of credibility and regard as a metric. I have always believed AVEs are here to stay (in some form or another) because they are a supremely simple way of trying to ascertain some form of value.

The interactive and PESO-friendly form of the new AMEC Integrated Evaluation Framework is a huge step in simplifying the process of measurement. What better way to organize hundreds of metrics, oceans of content, various goals, and countless other considerations?

3. It is flexible and easy to use

Some wrongly think the Framework represents complexity or inflexibility.

The reality is you have full freedom to plug in the objectives and data you care about, change those later on, and ensure that any subsequent activity and impact remains measured in a consistent manner.

© 2016 CARMA. All rights reserved. 11

The New AMEC Integrated Evaluation Framework

I find the Framework flexible and easy to use. In my early discussions with clients they all seem to get it quickly and easily, too.

4. It is a foundation for innovation

In its service to the measurement industry worldwide, AMEC has done too many good things to count. But its two biggest achievements are:

Defining what is right in measurement (the Barcelona Principles), and

Providing a tool to help accomplish proper measurement (the new Framework).

These are the most important simply because they have had and will have the biggest impact on our day-to-day work.

I believe the Framework will serve as a foundation for another layer of innovation that will arrive soon from AMEC. This will likely be another small step but with big impact—and will continue to eliminate the lack of certainty which has clouded our profession for generations.

What will that next big layer of innovation be? I don’t know; we’ll have to see what the team at AMEC comes up with. Having said that I would hope to see ideas on how to promote accurate measurement in social data. In a world where automated rubbish seems to be taking the place of AVEs, it would be good to ensure that our industry unifies around principles and approaches that deliver what matters.

© 2016 CARMA. All rights reserved. 12

The New AMEC Integrated Evaluation Framework

Chapter 4: A Trial Run of the New AMEC Integrated Evaluation Framework, and 7 Tips on How to Use It By Katie Paine

I’ve just spent the better part of a day trying to navigate through the hottest thing in measurement this month: the new AMEC Integrated Evaluation Framework, unveiled with great fanfare at AMEC’s International Summit in London in June 2016. It’s an interactive version of the original AMEC Valid Metrics Framework, designed to make it easier for clients to implement Barcelona Principles/standards-compliant measurement in their organizations.

The Framework at first glance

The new Framework has an online interface that features multi-colored squares. Each square requires you to provide information about your organization’s campaign or program. The 7 squares are:

© 2016 CARMA. All rights reserved. 13

The New AMEC Integrated Evaluation Framework

1. Objectives

2. Inputs

3. Activity

4. Outputs

5. Outtakes

6. Outcomes

7. Impact

When you click on each square it asks you questions like, “What are the broad objectives of your organization?” and then, “What are your communications objectives?”

Visually, it is certainly an enormous improvement over the old Framework (which, I will confess, I have never once been able to get a client to comprehend). And the interactive nature of this new Framework is a lot less daunting than the old PowerPoint version.

But in the end, it will be just as challenging to fill out as the earlier version, because all the problems inherent in the old Framework still exist, despite the sexy new front end. Sure, there’s a taxonomy that offers examples of the types of answers they’re looking for, but the confusion will persist.

This is because the very smart people who created the Framework actually live and breathe measurement every day of their working lives. Many if not most of the Framework’s creators primarily work with large organizations that do sophisticated PR and have at least some background in measuring results.

Unfortunately, your typical PR program is neither large nor sophisticated. Despite the impression that Samantha Jones (from Sex in the City) may have left on a generation, the vast majority of PR is for small- to medium-sized businesses, restaurants, NGOs, and government agencies. And it is done by the people who show up at conferences, attend my workshops, and participate in webinars. These people, for the most part, ask questions and struggle with answers that are far more basic than those the Framework deals with. And so the new Framework is going to be difficult for many typical PR pros to use.

© 2016 CARMA. All rights reserved. 14

The New AMEC Integrated Evaluation Framework

A test run of the new Framework

To try out the new Framework I completed it using as an example a PR campaign I recently helped a client measure. At first, I filled it out using the responses that members of the client’s PR team gave me at our first meeting:

Our initial meeting left large gaps between what the client’s team saw as organizational objectives and the goals of the particular event. There was considerable confusion between inputs, activities, and outputs, and no clear connection between the organization’s objectives and what the actual impact would be.

The good news is that, during the course of working with this client, we did eventually identify target audiences and objectives. We were able to connect the dots between the communications activity and the ultimate impact. Thanks to this and other work, that

© 2016 CARMA. All rights reserved. 15

The New AMEC Integrated Evaluation Framework

client now has a useful and convenient dashboard to update management on what is working and what is not working. It just took a bit of time to sort out the definition of “working” and “not working.”

So I went back to the Framework and filled it out again, this second time with the actual information I used to build the client’s dashboard.

7 tips for using the new AMEC Integrated Evaluation Framework

After a long day of checking the taxonomy and making sure I put everything in the right boxes, I’ve learned a lot about how to use the Framework. Here are seven tips for anyone trying to do the same:

#1. Before you go to Google and type in “AMEC Framework,” do your homework.

What all measurement requires, regardless of what framework, tool, or anything else you employ, is a thorough understanding of the business goals of the organization and how PR affects them, e.g., “What is the mission?” and “How does it make money?” and “What is the perceived role of PR in that process?”

For agency folks, this is your biggest weakness and one reason why PR gets no respect when budgets get tight. If you don’t have an agreed-upon definition of how PR contributes to the success of the organization, then you’ll never get past square one (literally).

So meet with your boss, your boss’s boss, or whoever is asking, “What have you done for me lately?” and make sure you have agreement on what the organizational goals are and how communications contributes to them. Click here for a recipe that will walk you through the process. When you are done, you should have a list of organizational goals, and a list of communications goals. (You are going to need them later.)

#2. Bake cookies to find your “Inputs”

Some of the answers to the Framework questions about target audiences and strategy may lie in other departments. Depending on the size of your organization, information on specific target audiences/personas or even overall strategy may well lie in sales, marketing, or customer intelligence. If you’re a nonprofit, answers may reside in membership or development. In government agencies, there may be a data center or a committee that has the answers.

Visit whatever departments hold the clues and bring treats. I’ve always gotten much more information with chocolate chip cookies than with an email. Depending on people’s stress level, a good Scotch whisky can also be an excellent persuader.

© 2016 CARMA. All rights reserved. 16

The New AMEC Integrated Evaluation Framework

#3. Look at the communications budget for “Activities”

What you’re really doing with this Framework (and any measurement program, for that matter) is determining which efforts are worthwhile and which are not. The fundamental concept is “worth,” which implies a financial or resource commitment. So rather than just a laundry list of activities, which could quickly become a nightmare of random metrics, only list those activities that require either a significant amount of budget, time, or resources. (You can get to the others later.)

To help you simplify the process: Since the Conclave on Social Media Measurement Standards has determined that you “earn” a share, I wouldn’t even bother with the “S” column; just include any shared data under “earned.” Also, note that “earned” doesn’t mean what you have already earned, but rather what you plan to do in terms of “earned” media, e.g., what you’re writing, the nature of the media outreach, speech writing, or anything that is going to require resources.

#4. Outputs are what you’ve checked off your to-do list

After you’ve listed all the activities, now you need to see what actually happened, i.e., Did any of that activity reach the agreed-upon target audiences? This is where you can “count” the number of media items that ran or that you “earned.” Tally up the paid media placements and anything that was shared. Add the data on clicks, time on site, or whatever metrics you’ve agreed are important from your web analytics platform. If you’re measuring events, count the number of attendees as well as anyone who used your hashtag. Whatever you do, try to avoid completely inaccurate definitions of “reach” and “impressions.”

#5. If you don’t have good survey or engagement data, skip the “Outtakes” section

Essentially, outtakes are what your target audience actually takes away from all the stuff you’ve listed in #4, above. In order to understand what a member of your audience actually “takes away,” you have to ask their opinion. In other words: Are they more aware or more likely to consider or prefer your brand? While engagement is not the same as awareness, it may be an acceptable proxy for evidence of attention on the part of your target audiences.

#6. “Outcomes” should be the same as your communications goals you listed in #1.

Go back to Section 1 and cut and paste your communications goals in the “Outcomes” section and change the tenses. For example, if the goal was “Increase preference in the new brand by 10%,” then the outcome should be “Increased preference in the new

© 2016 CARMA. All rights reserved. 17

The New AMEC Integrated Evaluation Framework

brand by 10%, as measured by pre/post testing.” If that didn’t happen, prepare a good explanation.

#7. “Impact” should be the same as the organizational goals you listed in #1.

Go back to Section 1 to copy and paste the business goals in the “Impact” section and change the tenses. For example, if your organizational goal is “Increase support for independence by 10% among Scottish women,” then your impact should be “Increased support for independence among Scottish women by 10% as measured by an average of public polls.” If the impact is different, prepare a good explanation.

This Framework asks the right questions

Ultimately, I love this framework, not because it is perfect, or even particularly easy to use, but because it poses the kind of questions that I’ve been answering for thirty years. And I know all too well how hard many of them are to answer.

For those of you interested in frameworks I refer you to the work of Fraser Likely and the Task Force on Standardization of Communication Planning/Objective Setting and Evaluation/Measurement Models, a group that is looking at the dozens of measurement frameworks that currently exist.

(A version of this post originally appeared in Katie Paine’s Measurement Blog.)

© 2016 CARMA. All rights reserved. 18

The New AMEC Integrated Evaluation Framework

Chapter 5: Interview with Jim Macnamara: The Theory and Practice Behind the New AMEC Integrated Evaluation Framework

The Measurement Standard is pleased to welcome back Jim Macnamara, Professor of Public Communication, University of Technology Sydney. We spoke to him in July, 2016 about the new AMEC Integrated Evaluation Framework, for which he provided a theoretical underpinning and evaluation taxonomy. We’ll also find out about his recent move to London and AMEC awards.

The Measurement Standard: Welcome back, Jim. We understand you’ve been quite busy lately. Tell us something about that.

Jim Macnamara: My wife and I have just packed up and moved to London where we’ll be for the rest of this year. I’m on sabbatical from the University of Technology Sydney (UTS) and will spend six months doing research with the UK Cabinet Office and UK government departments. I have been appointed Visiting Professor at London School of Economics and Political Science (in their Media and Communications Department). Plus I am a member of the Evaluation Council of the UK Government Communication Service. And on top of that I’ve just been contracted by Routledge to write a book on evaluation of communication. The book I am planning will be quite different to what is in the market, so I’m looking forward to hopefully doing a good job on that.

TMS: Very busy indeed. We expect a little preview of your book, as soon as you can leak it to us. OK, let’s talk about the new AMEC Integrated Evaluation Framework. To what extent does academic research inform it?

JM: The new AMEC Integrated Evaluation Framework is a great example of senior PR and communication practitioners, academics researchers, and commercial social and media researchers working together. The framework is referred to as ‘integrated’ because it:

© 2016 CARMA. All rights reserved. 19

The New AMEC Integrated Evaluation Framework

Integrates academic, communication practitioner, and commercial research perspectives;

Integrates other models, matrices, and frameworks already in existence and builds on them; and

Allows integrated evaluation of paid, earned, shared, and owned media and communication.

I believe it is fair to say that academic research made a significant contribution by basing the framework on best practice program evaluation knowledge. I prefer to use the word ‘knowledge’ rather than theory because many practitioners have an incorrect view that ‘theoretical’ means ‘hypothetical’ and esoteric. The fact is that there is a whole world of program evaluation knowledge in disciplines where evaluation is widely practiced such as public administration, management (e.g., performance management), and organizational development. It is important for PR and communication to stop re-inventing the wheel and making up new models and terms.

What we did in developing the new AMEC Integrated Evaluation Framework was base the approach on program evaluation concepts, principles, and terminology such as program logic models that are widely used across many industries, as well as social science insights into human communication such as the communication-persuasion matrix drawn from social psychology, and then adapt these to PR and corporate communication practice specifically.

Ultimately, the framework has to be a practical tool for practitioners to use—and I believe it achieves that by offering an online interactive application. But behind the tool is a whole range of information and resources that identifies and describes the various terms, and guides users on what metrics and what methods are applicable to each type of activity and each stage. (Note: For more on the new Framework, read Jim’s Introduction to it here.)

TMS: You’ve said that academic research and practice are “mutually informative.” How does that work, particularly how does industry practice inform academic research?

JM: This is a really important issue that relates back to the first question. The interaction of practitioners and academics is a two-way street. It is not only a matter of practitioners having to learn from academics. There is a huge body of research findings in the academy that can inform practice—and this is ongoing and growing every day. So to me it is incomprehensible that practitioners would not tap into that.

On the other hand, academics need to engage with industry for at least three reasons:

© 2016 CARMA. All rights reserved. 20

The New AMEC Integrated Evaluation Framework

First, engaging with industry and professional practice is important to identify problems and gaps in knowledge which research can explore. While some academic research explores questions raised by academics themselves, another major reason for research is to examine problems and issues encountered in industry and practice. While that leads to applied research rather than pure or critical research, it is very important.

Second, academics want their research to have impact, not simply be published in niche academic journals. In some countries, such as the UK, academic research funding is now awarded and assessed based on impact, not just publication. Impact can be changing government policy, providing solutions for social issues, or it can be improving practices in an industry or profession. So many, if not most, academics want their research taken up and used.

Third, even if academics are not carrying out applied research (i.e., research to solve a practical problem), they do nevertheless need their research to be grounded and relevant to fields of practice. There is little point in coming up with findings and recommendations that are based on misunderstandings of practice or that are impossible to implement such as being hugely costly or too complex for users to understand. So consultation and relationships with industry and the professions are important.

In short, we learn from each other.

TMS: Academic research has been neglected in relation to industry standards for measurement and evaluation, and generally in PR and strategic communication practice. Why?

JM: There are usually at least two sides to every story and this is the case with the gap between academic research and practice in the fields of PR and communication. Both sides have to take some responsibility.

Practitioners are missing out on a lot of valuable insights and research data by not engaging with academic researchers. For instance, very few practitioners read the findings of academic research, and many industry organizations do not invite academic researchers to their conferences as presenters. In my home country, the Public Relations Institute of Australia (PRIA) runs its national conference with two separate events —one for academics and one for practitioners—on different days. That kind of segregation is counter-productive and very short-sighted.

On the other hand, some academics publish in a narrow range of academic journals, only attend academic conferences, and some write in a quite dense way with a Fog Index that requires a PhD to understand them.

© 2016 CARMA. All rights reserved. 21

The New AMEC Integrated Evaluation Framework

I have called on practitioners to open their minds and reach out to academic research. Equally, I regularly call on academics to reach out to the industry and engage with it through conferences, professional publications, and participating in industry organizations. I am pleased to say that many now do.

Also, in terms of bringing practitioners and academics together, I must commend two recent initiatives:

The Task Force set up in the US to review models and develop standards for evaluation. This is chaired by Fraser Likely and has brought together practitioners and academics from a number of countries.

In late 2015 AMEC established an Academic Advisory Group, which I am honored to Chair.

These initiatives extend the work of the IPR Measurement Commission in terms of both the number of academics involved and international representation.

TMS: Tell us about your recent wins at the 2016 AMEC Awards for communication effectiveness.

© 2016 CARMA. All rights reserved. 22

The New AMEC Integrated Evaluation Framework

JM: There is a backstory here, and it’s important to understand that these awards focus on research for planning and evaluation. At the 2015 AMEC Summit on Measurement in Stockholm one of the delegates expressed appreciation for my paper but asked whether I ever did any evaluation, implying that there is a gap between academic research and practice. In fact, I do do research to inform planning of communication campaigns and evaluation—as do many academics who carry out contract research and who are often called in by organizations as independent advisers.

So this year for the first time I entered two research projects in the AMEC awards and was delighted and very honored to win two Gold awards and an overall Platinum Grand Prix award. I think that makes a point very clearly: robust, reliable research using sound methodology can be applied in practice and informs best practice.

TMS: Congrats—but wait a minute. Are you implying that “robust, reliable research using sound methodology” is not typically applied in practice? Are you commenting about the quality of most communication research?

JM: I think we already have the answer to that question. Typically no. While there are pockets of excellence, robust evaluation research is not used in PR practice generally.

© 2016 CARMA. All rights reserved. 23

The New AMEC Integrated Evaluation Framework

That has been shown in studies in the US, UK, Europe, and Asia Pacific—for instance, studies published on the Institute for Public Relations (IPR) Web site, the European Communication Monitor, the Asia Pacific Communication Monitor, and so on. Almost every survey of practitioners finds acknowledgement that we need to do better in this area. I look on it positively – this is an area for improvement to show the value of practice. That is the key to future growth.

TMS: Thank you Jim, for another informative interview. All the best in London and on the new book.

© 2016 CARMA. All rights reserved. 24

The New AMEC Integrated Evaluation Framework

Chapter 6: Is the AMEC Integrated Evaluation Framework the Standard Model? by Fraser Likely

Background: About a year ago, Fraser Likely wrote an article in The Measurement Standard newsletter entitled “Finding a Standard Model for Barcelona Principles 1, 2 & 3: A New Task Force.” In that article he introduced a study group called “Task Force on Standardization of Communication Planning/Objective Setting and Evaluation/Measurement Models.” Just last month the AMEC Integrated Evaluation Framework debuted. A recent comment from Daniel Johnson on Fraser’s article prompted a reply from Fraser which provides a very informative discussion of models vs. standards, and how standards become adopted. Here is their exchange:

— From: Daniel Johnson

© 2016 CARMA. All rights reserved. 25

The New AMEC Integrated Evaluation Framework

Hi Fraser, do you not consider the AMEC Integrated Evaluation Framework to be the standard model?

—From: Fraser Likely

Hi Daniel,

Richard Bagnall, CEO of Prime Research UK, has done a wonderful job of leading the AMEC team that created the new six stage evaluation/measurement model. The important thing about this model is that it is built on a theoretical framework. In doing so, Richard’s team was ably supported by the AMEC Academic Advisory Group, led by Dr. Jim Macnamara, with such noted measurement scholars as Don Stacks, Tina McCorkindale, Tom Watson, Anne Gregory, Brad Rawlins, and Ansgar Zerfass.

AMEC’s Integrated Evaluation Framework is both an initial theoretical framework and a model, with the former underpinning the latter. Having both elements is fundamental to the process of standardization. The fingerprints of the advisory group are all over the framework aspect.

Is it a standard? The marketplace, ultimately, will determine that – and the only marketplace of import is the PR/Communication department within an organization. If a critical mass of CCOs adopt the framework and the model, then yes, it can become a standard. But, there is much water to flow under the bridge first.

Many questions must be decided in that marketplace. For example, is a six stage model appropriate for all situations? Is a six stage model appropriate for a single communication product (news release; tweet; Facebook post; etc.), for a communication campaign (various products and channels with a single objective), and for a communication program (number of campaigns over time, with multiple objectives but directed at the same stakeholder)? What about four or five stage models, do they have utility? Where and when?

For example, is the terminology used for each stage the language that the marketplace will adopt? Is what we’ve taken from management literature important, including inputs/throughputs and outputs, in all situations? Is what we have created—such as the concept and term “outtakes”—important in all situations? Is what we’ve borrowed from the program evaluation literature – outcomes at all stages or levels—important in all situations? Is what we’ve taken from the financial/management literature—outflows; impacts; roi; outgrowths—important in all situations? What are the correct terms? What terms will the C-suite recognize and find valuable?

Is it a standard? The marketplace, ultimately, will determine that. For example, what model—be it a three, four, five, six or more stage model—is better for marketing

© 2016 CARMA. All rights reserved. 26

The New AMEC Integrated Evaluation Framework

communication, internal communication, stakeholder relationships, issues management, etc.? Does the use of digital, social media, or traditional media influence the choice of terms and stages? Or, is the model affected by the choice of PESO channels?

My original piece, on which you’ve commented Daniel, was written in June 2015. The new AMEC framework and model came out in June 2016. Besides AMEC’s work, the Task Force on the Standardization of Communication Planning/Objective Setting and Evaluation/Measurement Models has also progressed down the road to standardization. The Task Force has conducted a major literature review, examining the development of models both historically and comparatively—but also with an eye on the antecedents that we’ve borrowed from other disciplines. Without an appreciation of where models or model stages and terms came from and how they’ve evolved, we risk—as a profession—continually reinventing. Many of the models that exist now are really simple redos of existing models, but with a new coat of paint in the form of a different stage or term. And, most importantly, most of the models are just that, a model drawn on paper but lacking any theoretical underpinning or empirical testing.

The Task Force has entered its second stage. We’ll produce two papers on the evolution of measurement models. We’ll also conduct two or three qualitative research projects examining framework and model uptake in agencies/measurement suppliers and PR/C departments from the vantage points of CEOs, CCOs, and research/measurement heads. Once we have this research at hand, the Task Force will examine all models closely and compare to what the research has told us. We’ll develop a theoretical framework and suggested model(s). Then we will peer review, test with PR/C departments and have the Coalition of PR Research Standardization, on which both AMEC and the IPR Measurement Commission sit, the framework and model(s), examine the reports and approve.

It would be great for Richard’s team and the Task Force members (now including Sophia Volk on the list above) to benefit each other going forward. By the way, Jim Macnamara is a member of the Task Force and a member of the AMEC Academic Advisory Group, and that cross influence has already been felt.

So, have we arrived at a standard? Not yet. But a lot of work has been and is being done.

Comment again in June 2017! Then we’ll see how close we are! —Fraser

— From: Daniel Johnson

Thanks for providing such a comprehensive response Fraser. Good luck with your efforts!

© 2016 CARMA. All rights reserved. 27

The New AMEC Integrated Evaluation Framework

(Thanks to Wikipedia and Yupi666 for the image at the top, which depicts a very basic model of communication, but otherwise does not represent the work of Fraser Likely’s Taskforce or the new AMEC Framework.)

Call for free consultation

1.603.410.5000 T +9714 3564100 Carma PO Box 341138.

F +9714 3564222W Dubai Silicon Oasis, Dubai,

© 2016 CARMA. All rights reserved. 28