developing and pretesting concepts, messages, and...
Post on 11-Jun-2020
5 Views
Preview:
TRANSCRIPT
Developing and Pretesting Concepts, Messages, and Materials
Page 1 of 33
Developing and Pretesting
Concepts, Messages, and Materials
Ms. Williams: Today we are going to talk
about step two, which is developing and
pretesting concepts, messages, and materials.
We are also going to talk about implementing and
managing health communication programs. So,
you will have step one and step two under your
belt, and then you can work on your plan. We
will use the communication plan template that
comes out of the pink book to do that.
But, first, before we get into that, we
wanted to come back to what we talked about just
briefly yesterday towards the end of yesterday.
That was, once you develop this profile and you
have your indicators in place, and you kind of
have a sense of probably where your population
is, how do you make decisions about what
approach you are going to take?
The example we showed yesterday was that
you have southern small state, central small
Developing and Pretesting Concepts, Messages, and Materials
Page 2 of 33
state, and northern small state. They all have
sort of a different demographic profile. How
are you going to make a decision about we are
going to use health communication approaches in
this community and we are going to use policy
approaches in this community, and we are going
to do something else in that community?
It is a process of looking at a number of
different factors. So, you might say southern
small state has high unemployment, low income,
high uninsured, a few barriers, a few providers.
There are a lot of barriers to people making
behavior change.
If you were doing a social marketing
campaign and there is sort of this idea that you
go for the low-hanging fruit, southern small
state is not low-hanging fruit. However,
central small state or northern small state
might be considered more low-hanging fruit.
The challenge for us who work in the
government and use public money is that is not
always how we are expected to spend the tax
Developing and Pretesting Concepts, Messages, and Materials
Page 3 of 33
dollars, right, on people who are ready to
change? We are expected to work on the tough
problems. That is why they have us. Maybe some
of our partners can work on some of the more
low-hanging fruit.
What do you do in these cases of the
barriers are so high and there’s multiple
problems going on? How do you make a decision
about the appropriate approaches?
One place that you might start is back to
your literature review and your environmental
scan. What is that telling you about different
approaches you might take in communities that
face similar challenges?
There are a number of organizations out
there that have put out best practices and
guidelines. One place you can look is the
Community Guide to Preventive Services has a set
of recommendations around health communication
and marketing. You can look at what they have
recommended. Robert Wood Johnson has put out
some work on policy. I think Kellogg has some
Developing and Pretesting Concepts, Messages, and Materials
Page 4 of 33
information about strategies. The Public
Health Institute has strategies.
You can look at Healthy People 2012 now.
What are we on? Whatever the next one, 2020,
you can look at Healthy People, those
recommendations, and see what they are
suggesting in terms of how you approach some of
the issues that you are looking at.
The Community Guide also has
recommendations for diabetes, heart disease,
physical activity. You can look and see across
what all of these sort of people who have been
looking at the literature are saying are what
may be best practices right now.
You can also work with your stakeholders.
If you say this is the picture of where things
are, these are the different strategies we might
take. We might take a policy approach. We
might take a worksite approach. We might take
a communication approach. Discuss amongst
your stakeholders what seems like it is going
to be something that is going to be meaningful
Developing and Pretesting Concepts, Messages, and Materials
Page 5 of 33
and impactful.
Frankly, your own priorities within your
organization are going to dictate a lot of what
you do. You might have categorical funding that
says you must do health communication, and you
are looking at southern small state and saying,
“Well, what can we tell them when they can’t see
a doctor or they don’t have access to healthy
food or access to resources, but we have to do
communication?”
You might look at, then, what are your
options, given the conditions. You might
consider can you apply communication in another
way. Instead of reaching the people who don’t
have access to resources, can you use your
communication funding to reach decisionmakers,
to reach communities that might influence
access, to address some of the policy and
environmental barriers? Can you use your
communication funding to do some community
collaboration or some kind of community
mobilization kind of work? There are a lot of
Developing and Pretesting Concepts, Messages, and Materials
Page 6 of 33
different approaches that you could take, even
in these seemingly difficult areas for doing
communication.
But it is multiple approaches to making
a decision about, given all of these conditions,
what would be our best approach to take? You
can always, also, ask us at NDEP. I mean, you
can run by, “This is what we are thinking. These
are the strategies that we are thinking.” We
are happy to provide some technical assistance
and some advice in terms of how you might
approach any given situation.
Mr. Petty: Another thought that I had is,
looking at these three different profiles, you
are going to have three very different types of
campaigns that you would want to roll out in each
area. So, if I am looking at southern small
state, you are probably looking at more
environmental things, dealing with access,
maybe building parks and walking trails so that
people have the opportunity to do more physical
activity, maybe working a little more with
Developing and Pretesting Concepts, Messages, and Materials
Page 7 of 33
trying to get different policies put in place
that allow for some of these things to improve.
It looks like they have got some of the
highest smoking rates, but their obesity isn’t
too bad. Maybe there is more of an emphasis on
tobacco in that area.
Whereas, if you are looking over at the
northern small state, that is probably more
where you are trying to get down to individuals
and help work on behavior change for those that
aren’t quite there. I mean, just looking at the
different characteristics says to me that you
would have very different types of approaches
in each different area that you would just want
to think through.
Ms. Williams: Another thing you might
find is maybe central small state has tons of
campaigns already going on. They are sort of
in the middle, and they are sort of what people
like that do health communication or do
interventions. There may be lots of other
programs already happening. Maybe the place
Developing and Pretesting Concepts, Messages, and Materials
Page 8 of 33
for you, then, is southern small state because
there is less happening. So, there are more
opportunities to get things done.
There are a lot of environmental factors,
evidence factors. There is a lot that goes into
that decisionmaking process of deciding we are
going to take these approaches in this area with
these people. It is a process that you have to
go through, and you bring a lot of different
factors to bear in that decisionmaking.
Let’s talk about step two, which is
developing and pretesting concepts, messages,
and materials. I was at a presentation once,
and this young woman was developing a breast
cancer awareness campaign for African-American
women. She had done all the things you are
supposed to do in step one. She had done a
series of focus groups with African-American
women. She asked about what their preferred
communication channels were.
One of the channels they mentioned was the
links. She thought that was kind of weird,
Developing and Pretesting Concepts, Messages, and Materials
Page 9 of 33
links being another word for golf course. But
she had developed this whole golf campaign where
they were going to ride around in golf carts.
This is a true story. They were going to ride
around in golf carts and distribute breast
cancer information awareness. She hadn’t
launched the campaign yet, but she was in the
process of putting this all together.
I was like, “I’m sorry, but are you sure
they meant a golf course? Because you know
there is an organization of African-American
women called The Links?” And she just kind of
looked at me. She was like, “Oh, no, I didn’t
know that.”
She missed that, too, which is pretesting
your messages, materials, and concepts. Once
she had put together her golf, her links, and
her focus groups, and once she had decided on
the concept of a golf course intervention, she
should have pretested that. She could have just
run it by three African-American women, and they
would have said, “Well, they probably didn’t
Developing and Pretesting Concepts, Messages, and Materials
Page 10 of 33
mean golf course. They probably meant The Links
organization.” If she had done the pretesting,
she would have saved herself a lot of work.
Especially when you are not as familiar
with the community or you are an outsider to a
community, it is important to do pretesting and
to keep in mind that focus groups, they are good
for a certain amount of information, but you
have to check what you hear in a focus group.
Sometimes the facilitator will latch onto a
novel idea. That doesn’t mean that your
audience is going to like that. But if there
is something that you heard that you didn’t hear
anywhere else, and you thought it was a good idea
and it rises to the top for some reason, you have
to go back and check that.
When I was doing a secondary analysis of
the focus group data for the Surgeon General’s
Report on Physical Activity, they had made a
recommendation that one of the partners we
should work with to disseminate the Surgeon
General’s report for African-Americans was the
Developing and Pretesting Concepts, Messages, and Materials
Page 11 of 33
Nation of Islam. That was in the report from
the focus group, the contractor.
I went back and reviewed the tapes. It was
one person out of – they did probably nine focus
groups with African-Americans – one person in
one focus group and nobody else in the room.
There was no non-verbal agreement with that one
person. But it was something that the
facilitator had not heard before, and so she put
it in as a novel idea.
Can you imagine CDC or the Surgeon General
partnering with the Nation of Islam because they
heard one thing from one person in a focus group
that no one else -- I mean, there was not even
people nodding their heads, “Oh, yes.” There
was none of that.
You have to check what you hear from your
audience. That is why pretesting is so
important. Pretesting allows you to assess the
cultural appropriateness of what you are
developing. It helps you assess whether people
will pay attention to your message or your
Developing and Pretesting Concepts, Messages, and Materials
Page 12 of 33
strategy, whether they understand it.
You can assess how motivating your message
is or your concept, whether people can remember,
whether people are remembering accurately.
You put out a message and people interpreted it
in an entirely different way. You need to know
that before you move forward with your campaign.
Is it meaningful?
It can also help you identify
controversial areas. In one case, there was a
tobacco campaign that had done these focus
groups with teenagers that were smokers. One
of the stories that was told was that, before
you start smoking when you are 14, you think that
being addicted to cigarettes is like really
wanting a piece of chocolate. Once you become
addicted to cigarettes, you realize that it is
more like you are drowning and you need a next
breath of air. That is what it feels like to
be addicted.
And so, they came up with this whole
campaign about drowning, right? Smoking was
Developing and Pretesting Concepts, Messages, and Materials
Page 13 of 33
like drowning. When they launched the
campaign, they got blowback from the people who
had children who drowned, right? They had to
pull the campaign.
They didn’t know that that was going to
happen. That wasn’t their intent. They didn’t
mean to upset people. But you need to check.
Especially when you do something that is
potentially controversial, that is potentially
edgy, that is potentially out there, you just
need to run it by some people to make sure you
are going to be okay, and be prepared if there
is going to be any kind of blowback or outcry
over your message.
Pretesting allows you to assess the
strengths and weaknesses before the campaign
launches. Now keep in mind that there are
formal ways of doing pretesting that are
expensive and complex. You can also just run
things by people. You can do a check with a few
people that represent your target audience,
that represent people in your organization.
Developing and Pretesting Concepts, Messages, and Materials
Page 14 of 33
You can do it with your stakeholders. You can
go to opinion leaders, gatekeepers, and run your
message by them. So, there are also less-formal
ways to do pretesting.
It is not objective. It is not predictive
or statistically-precise. It is not a
substitute for a sound model or theory. You
still have to have your messages based on
something. They still need to come out of some
kind of framework.
Pretesting is not going to make up for poor
planning or flawed execution and
implementation. You still have to do all of the
steps, even if you pretest.
If you go to the pink book, what it is going
to explain to you about pretesting is the steps
that you take are you review the existing
material. So, you go back to your literature
review and review other campaigns. Or you go
back to your environmental scan. You review
what has already been done. You look at
national organizations. You look at what
Developing and Pretesting Concepts, Messages, and Materials
Page 15 of 33
messages are already out there.
You develop and test your message
concepts, which are ways of presenting
information. Are we going to take a funny
approach or a serious approach, a scary
approach? Are we going to use this kind of
person or that kind of person? Are we going to
use photos or illustrations or comics?
There are all different kinds of concepts
that come out, and you test these concepts to
find out which resonate with your audience. You
decide what materials to adopt or tailor. You
develop those messages and materials, and then
you pretest them.
It is a complex process. It takes people
who are really skilled at developing concepts
to do this kind of work. It is oftentimes people
that we don’t have. A lot of times, if you don’t
have a contractor that can do this work for you,
it can be very difficult to do it on your own
because it does take a certain amount of skill
to develop the concepts and then test the
Developing and Pretesting Concepts, Messages, and Materials
Page 16 of 33
concepts in a way that is going to give you
meaningful information.
So, that is kind of a red flag. It can be
an expensive and complex process. However, you
can work with materials that are already out
there where some of this work has already been
done, and it will save you some steps.
You can work with NDEP. NDEP has already
developed and tested the message concepts
around the diabetes prevention and control
materials that we put out. So, review NDEP
materials.
Decide what materials to adopt and tailor.
Based on what you have learned about your
audience and the approaches you are going to
take, and the strategy you want to take, you can
modify and adapt those messages and materials.
Then, you want to pretest. Pretest the
adaptations you made to make sure they are still
on point, to make sure people still are getting
the message. It saves you the step of doing that
concept testing, which can be very complicated
Developing and Pretesting Concepts, Messages, and Materials
Page 17 of 33
and expensive.
So, leverage the materials that are
already out there. Start with NDEP and go on
from there. Okay?
You review our existing materials. You
can search by risk status, ethnicity, language,
what group you are trying to reach. You can also
call us or send us an email because we do that.
People are thinking, “We are trying to do this
and we were thinking these materials you have.”
We will say, “Well, these materials, this much
material, or that material might work better for
what you are trying to accomplish.”
There is a lot of stuff on the website,
and sometimes it can be difficult to navigate.
We are working on that, but we are happy to help
you work through what is there, to identify what
might work.
What you want to look for is, are the
messages relevant? We learned all of this in
our formative research. We think this is the
message that is going to resonate. This is what
Developing and Pretesting Concepts, Messages, and Materials
Page 18 of 33
we are going to try to accomplish in our
community. Which of these materials? Are the
messages relevant in these materials to what you
are trying to accomplish?
Some are prevention messages. Some are
control messages. If you are trying to focus
on control, you may not want to use the
prevention materials. You may want to focus on
the control material. Okay? So, make sure the
messages are relevant.
Are the materials appropriate, the
format, the style? Are there cultural
considerations? The language, the literacy
level, you want to take a look at that. Those
are things that can be adapted, too. Just
because you see something like our “Paso a
Paso,” which is “step by step,” or you might see
one of our American Indian materials, it doesn’t
mean it can’t be adapted for your population if
the message seems to be on target.
Are the materials likely to meet your
communication objectives? Whittle down. Ask
Developing and Pretesting Concepts, Messages, and Materials
Page 19 of 33
yourselves these questions. Ask your
stakeholders these questions, and whittle down
your choices to the ones that are most likely
to be yes.
Decide which materials to adopt. You may
want to test existing materials. You may get
three or four NDEP resources and you may say,
“I’m not sure which of these is going to be the
best fit.” So, you might want to run that by
your target audience just to be sure. “What do
you think about this?” “Oh, this is fine, but
it’s wordy.” Or “This is fine, but I don’t want
to read it on paper. I want to read it on my
phone,” or something like that. So, you might
at this stage want to run something by your
target audience.
For program activities, you might want to
test the concept. If you are deciding to do like
Road to Health, you might want to run that by.
How will we do Road to Health in this community?
What is going to be the best way to make this
happen? Or power to prevent? Or, if we are
Developing and Pretesting Concepts, Messages, and Materials
Page 20 of 33
going to do some kind of provider outreach with
one of our provider tools, how is that going to
work?
You might want to test, actually, that
concept for making that activity work. Does
that make sense? Not testing necessarily the
material itself, but how you are thinking you
are going to implement or disseminate the
activity. Okay? Then, you can decide if any
tailoring is needed or if you are going to need
to add anything to make this really work.
It doesn’t necessarily have to be a whole
big thing, right? You don’t necessarily have
to convene another round of 18 focus groups to
do this. It can be a matter of sitting down with
your stakeholders and going through the
material, sitting down with members of your
target audience, running it by people who are
knowledgeable or who have done this work before,
and working through this.
So, it can be done very effectively. It
can be done without using a lot of resources.
Developing and Pretesting Concepts, Messages, and Materials
Page 21 of 33
It can be done in a very timely fashion.
Step three will be modify or adapt
messages and materials. The first place to
start is the principles established in the NDEP
Guidelines for Partner Collaborations. That
is on our website. There is a link to it from
the phConnect site. You can also find it on our
website.
But, basically, the materials are in the
public domain. There are no copyright
restrictions on them. You can brand them with
your logo or your partner logos. You can add
your contact information. You can tweak the
examples that are used. You can change the
pictures.
I think in Utah they were using “Managing
Diabetes: It Is Not Easy, But It Is Worth It.”
They used local people because they were trying
to tell local stories about diabetes. They just
took the pictures we put on there off and put
the pictures of their local people on.
You can alter the format. We have a
Developing and Pretesting Concepts, Messages, and Materials
Page 22 of 33
brochure. You convert that to HTML for your
website. Or we have a PSA canned. You can have
someone else read the script. That’s fine.
You can adjust for programs the delivery
and the timeline. We have the Power to Prevent
curriculum, which I think is 12 weeks. You can
make that shorter; you can make that longer.
You could add Road to Health to that. You can
do Road to Health in one sitting. You can do
Road to Health in three sittings. You can do
Road to Health in six sittings. If you have a
question, ask. Okay?
You can complement and supplement
existing materials. You might want to tailor
for cultural or local relevance. We have, say,
the Fat and Activity Counter. You might have
some restaurant or something in your community
that everybody goes to, and you want to add that
menu or you want to provide that menu as well,
those nutrition facts for those folks. You can
provide that as a supplement to our existing
materials.
Developing and Pretesting Concepts, Messages, and Materials
Page 23 of 33
We can work with you on literacy if
literacy level of the materials is an issue. We
can work with you on language if the language
of the materials is an issue.
Things that should not be modified: the
health topic. If we have something that is
prevention, it is prevention. Those are
scientifically-approved. We go through a
process of clearance, clearing our messages,
that people look at and say, okay, it is okay
for you to say this. So, if it is a prevention
message, those messages have been crafted for
prevention.
If it is a management message, those
messages have been tailored for management.
You have to be very careful about switching
them, about saying, “Okay, well, we are going
to use these prevention tools in our
self-management class.” You have to be careful
about that because there are differences
between the messages for the two audiences. Ask
us if you have a question or a concern.
Developing and Pretesting Concepts, Messages, and Materials
Page 24 of 33
You can’t really make substantial
alterations to the text that alter the meaning.
Like I said, that has been
scientifically-approved. You shouldn’t be
making substantial changes that really alter
the meaning of what is being said. Really,
attaching and inserting new text is not allowed,
although there is an exception for adding
culturally-relevant examples, for the most
part.
And you can’t imply that NDEP endorses
commercial products. There is language in the
Partner Guidelines about, if you are using a
commercial, you have Walgreens or Pfizer or
something like that who is local you are adding,
what the language is that needs to go on there
to make it clear that NDEP doesn’t endorse that
particular company.
If you adapt the material to the extent
that the NDEP logo has to be removed, generally,
our test is, if you have made changes to the
extent that we would have to get it cleared
Developing and Pretesting Concepts, Messages, and Materials
Page 25 of 33
again. If you run it by us and we say, “We would
have to run this through clearance,” then you
would have to take the logo off. It doesn’t mean
you can’t do it and you can’t use it. It just
means that the logo has to come off because it
is not what has been cleared by CDC or NIH, and
you put this message on it. This is also in the
Partner Guidelines.
I think Utah also changed some of the text
on some of the posters to be preventing and
managing diabetes. They added prevention and
management messages together. So, they took
the NDEP logo off because it was sort of pulling
messages from different places and putting them
into one place. You can always check with us,
always ask.
Let’s do an example. This campaign’s
messages are “Make a plan and manage your ABCs.
Make an appointment to visit your healthcare
provider and keep it. And make a referral to
the small state quit line.”
So, they are using our Managing Diabetes
Developing and Pretesting Concepts, Messages, and Materials
Page 26 of 33
posters. They added the state’s website, and
they used a local couple. Can they keep the NDEP
logo? How many people say yes? How many people
say no? They can keep the logo in that case.
So, it is okay to add your contact information,
add your local agency information. You can
switch out the pictures and still keep the logo.
Okay?
What about they pick this heart disease
flyer. It was developed for American Indians.
They changed the photographs and converted it
to HTML. Can they keep the logo? Yes. Good.
They took Four Steps for Life, and they
reprinted it with information about smoking
cessation in the back. Can they keep the NDEP
logo? No. For one thing, it is not four steps.
No.
(Laughter.)
Participant: Yes, and they changed it to
five steps.
Ms. Williams: And they changed it to five
steps.
Developing and Pretesting Concepts, Messages, and Materials
Page 27 of 33
You can’t attach new information that
hasn’t been cleared. You could add the smoking
cessation information as a supplement,
something that is offered. They get the Four
Steps brochure and they get another thing about
smoking cessation. That’s fine. But you can’t
physically attach new content to one of our
materials without us getting clearance for it.
Or you could go ahead and do it and just take
our logo off. You could change it to five steps,
add the smoking, take the NDEP logo off, add that
it was adapted on the back cover, and go on.
Okay?
And then, this is our poster about working
with your pharmacist and your optometrist and
your dentist and your podiatrist. They added
the phone number for the quit line to this
poster. Can they keep the NDEP logo? Yes.
Yes, you can because all you did was add local
contact information, and that is fine. Does
that make sense?
At the end of the day, just call us, send
Developing and Pretesting Concepts, Messages, and Materials
Page 28 of 33
us an email. Tell us what you are thinking. We
are happy to let you know.
Step four is pretesting the message and
materials. This happens after you have
substantially adapted a material. We took
this, we changed the format, we tweaked the
intervention, we did whatever, and it is
different from what it originally was. You want
to pretest that.
Or you have come up with sort of how we
think this campaign is going to run or this
intervention is going to run, or whatever the
strategy is. You might want to do a small pilot,
just to make sure that it is working the way that
you intended, that those messages are
resonating the way that you intended.
Again, it doesn’t have to be a whole big
thing. It is just running it by people who know
to make sure that you are still on target, you
are still on a message, your messages are making
sense.
It helps you to avoid missteps in
Developing and Pretesting Concepts, Messages, and Materials
Page 29 of 33
execution. So, you are not driving around on
a golf course looking for Black women and not
finding them, and wondering why the people in
your focus group lied to you.
It helps you confirm that messages and
materials are consistent with the audience
research and the testing. It also helps you
gain buy-in. So, we have all these people we
want to carry our message for us. They need to
have some kind of input into the development of
those messages.
This pretesting gives you a chance for
them to say, “Hey, this looks really good. We
can’t wait to start using these materials.” Or
“It would be good if you would tweak this a little
bit for us.” Or “This is on point.” Or “This
could change.”
Pretesting is a way to gain buy-in from
your stakeholders. Again, it is always helpful
to run your messages, your materials, by someone
who knows.
Mr. Petty: Just another quick example on
Developing and Pretesting Concepts, Messages, and Materials
Page 30 of 33
this. I am not sure exactly where this came
from, but I was at a social marketing conference
a year or two ago and heard about an example of
some folks that got some money to do an HIV
campaign. They decided that, through their
environmental scan, they really wanted to
target prostitutes, street workers.
They appropriately discovered that the
government should not be the right message
deliverers for this program because it is an
illegal activity. But, nonetheless, they
recognized that it was a problem.
And so, very creatively, they went out and
got several of them in a room. Basically, they
created this program for themselves. Talk
about getting buy-in and avoiding all these
potential problems, because they recognized we
know nothing about their life and what they do.
But they really went out and got the people who
are in that situation to craft messages to help
them make more healthy life choices and had a
lot of really good success with it.
Developing and Pretesting Concepts, Messages, and Materials
Page 31 of 33
It is kind of out of the box, but a great
example of really pulling in the right people
at the right time to make sure you don’t make
some of these mistakes.
Ms. McDivitt: And that is recommended,
also, for working with kids and youth, is
getting them involved in the development.
Mr. Petty: Early, yes.
Ms. Williams: When you are looking at
developing and pretesting messages and
materials, this step is key to ensuring the
effectiveness of your program materials and
activities. Sometimes you go through all your
formative research and you come up with a great
concept, and you just want to get started. You
just want to run out and implement it. You think
you are okay because you did a pretty good job
with your formative research.
But don’t shortcut step two. Even if it
is just a matter of running your ideas by some
people, it is worth the time and effort to make
sure that you avoid some missteps in the
Developing and Pretesting Concepts, Messages, and Materials
Page 32 of 33
execution of your campaign or your activities.
NDEP materials can serve as the basis for
most campaigns. We are willing to work with you
to make those work. You can look at other
materials, and we are willing to work with you
on that as well.
Test the materials and activities for
relevance to your intended audience. Test them
to make sure they are going to be okay in your
agency, that your stakeholders are okay with
them.
If you adapt materials significantly or
you adapt an approach significantly, pretest or
pilot before launching the program, just to make
sure that everything is working the way you
intended, and not just the messages, but your
800 number, your website, whatever systems you
put in place to make this work. Make sure those
things work.
I worked on a campaign that crashed NCI’s
Call Center. That is not something you want to
do. That didn’t make them happy. But it was
Developing and Pretesting Concepts, Messages, and Materials
Page 33 of 33
because we didn’t have all the right pieces in
place when we launched our campaign. We had no
idea how many people were going to call. We had
no idea that the Call Center was going to be
understaffed the day that we launched our
campaign because half the people were in
training. And so, we crashed the Call Center.
Running through your plan, not just your
message, but how you are going to execute your
plan, is important, too. It is not helpful to
make people in your agency upset with you about
crashing the Call Center. So, this is a really
important step.
top related