developing a data and evaluation framework for urban promise zones - session 3 … ·...

~~~ Noble Transcription Services - 714.335.1645 ~~~

Developing a Data and Evaluation Framework for Urban Promise Zones - Session 3

Janine Cuneo: And welcome, everyone, to our third session in our series of four sessions on the Urban Promise Zones relating to how to develop a data and evaluation framework. We've had -- for the last two sessions, we most likely will continue to have, at this session and the fourth one, participants at all different levels.

The series is really designed to help those without a framework put one in place by the end of these sessions in June, those of you that maybe have a framework but can use this as a time to review it and update it and share your experiences with others.

Part of the goal is for the PZs to get to know each other a bit so that you can continue to support each other after the webinar series is finished and therefore, that's why you're seeing different speakers come on that are your PZ colleagues as well as the peer-to-peer convenings that are happening.

We've had a lot of great participation in session one and two and look forward to continuing that. Again, just to reiterate what Chantel, our host, said, if you're having any troubles with the platform, feel free to chat her in the chat box on the right-hand of your screen.

Or if you're only coming in via audio and you're even having problems seeing the screen, feel free to email trainings -- it's plural, [email protected] and Chantel and our staff will work hard to get you situated. Just to be safe, we also sent a PDF of the slides we're presenting today to everyone who registered. So feel free to follow along by looking at that email.

Just some logistics in technology orientation for you, again, we -- our goal is to have an interactive style. If you're like me, more interactivity the more I retain information. And so you'll see we're going to do polls and quizzes, sharing and questions. So feel free to engage at all levels, we look forward to hearing from you, getting your questions.

But also, if you have any answers to one of your colleagues' questions, please feel free to chime in. The TA provider team today that's going to be presenting is myself, my colleague, Xiaodong and Kasia.

And then you might have also noted, for those of you that have been taken advantage of the office hours in peer-to-peer sessions that our colleagues at AIR have been providing that information and we really want to encourage you guys to continue to use those office hours in peer-to-peer sessions.

Before we dive into session three, though, I really wanted to pause and allow any questions people might have about session two, how that might've gone. If you did -- were able to use some office hours, anything you've gleaned from that that you think your other PZ colleagues might be interested in hearing about.

Let's just take a moment and see if anybody has any thoughts. Just remember, you most likely muted yourself. And so you'll need to unmute yourself to participate in this conversation. So look

Developing a Data and Evaluation Framework for Urban Promise Zones - Session 3, 6/7/18


2

at your name on the right-hand side and you'll go ahead and click the microphone item to unmute yourself and once you do, please feel free to speak.

Q: Will they -- do you think [inaudible] Congressional delegation at all?

Janine Cuneo: I'm so sorry, I don't think I could hear that question well. Could you repeat yourself? Hello. Is someone asking a question? I apologize, I'm not sure if I heard that question. So I'm going to have to, unfortunately, pass on that. But any other people have questions?

So one thing I do want to remind folks of, some things we've been sharing a bit about, is the concerns that people are having as it relates to maybe they feel that they're not ready yet to really take this next step in to building their evaluation framework. They might feel they need to do some more partner engagement and such.

We understand that everyone is really at a different phase in this process. Again, some people came with a full framework you developed and some were at the stage of thinking, how do I do data and evaluation again. So we do recognize that. That said, part of these sessions is really to encourage, cajole, cheerlead for you guys to really move forward on this framework path.

And although it may be daunting to some of you who really haven't started the process before these sessions and really maybe hadn't even thought about it since you guys submitted your applications, but we really want to encourage you to think of this as your best first attempt as evaluation framework.

It does not need to be perfect. What you do want to do is get yourself moving and get yourself starting to write down questions, indicators, etc. You may not have every entity involved at the level you are hoping.

You may not even know exactly who's doing what steps in -- with regards to collecting data as you choose these indicators that we're going to be talking about during this session.

This does not mean you still can't identify those indicators you're seeking in your framework and then simultaneously, you can develop a list of action items you need to undertake to ensure the next steps can be accomplished. Remember, an evaluation framework is, by nature, an interactive process.

So yes, for those that are really starting at ground zero, your first draft may have a lot of items you've highlighted and say, I need more work here, we need further coordination at the local level, I need an increased understanding of what my resources are going to be to actually gather this. But don't let the goal of a perfect framework be the enemy of a good first draft.

So you've got to push yourself through all this information and get things going, write things down and then you can continue to iterate the framework in the next couple of months. I want to let you guys know, again, we mentioned our colleagues AIR are conducting office hours. For the next few weeks, we have multiple sessions of these office hours.



3

Please use the link below and again, if you guys register for this class, you've got a PDF of these slides, so you can check out that link there and sign up. Again, especially if you're at the new stages, they can help you think through if I put this information down, what are my next steps, how do I actually make this come to fruition?

So please make sure you're using those office hours, it's a great tool for you. Let's talk a little bit about what we're going to be going over in this session. First, we're going to be describing data sources and measures as it relates to these four groups, these [inaudible] defined groups, A, B, C and D for indicators.

We're going to discuss different types of data, different systems and practices for collecting and storing data, including data usage agreements and we're also going to share some common challenges and solutions in data collection. So one key for data collection is collaboration, collaboration, collaboration.

You really can't overemphasize the need to collaborate, identify the roles of different partners and how and when and to what degree you're going to be sharing information. It really is the meat to all of this is exactly how you get things done is through this collaboration and relationship forming in the process.

We're going to want to get everyone who needs to be involved not just to say yes in the abstract. That probably happened when you guys were filling out your PZ applications, for example, but to get real buy-in and follow-through takes a whole other level of organization and leadership.

Really, most of your sites -- some people have [inaudible] multiple sites each stakeholder engagement. Without it, they may not have the ability to execute data-sharing agreements, identify in your logic models, for example.

You're going to want to establish ongoing relationship with partners that have invested in the process and that would create the commitment that ultimately results in the partners coming through with the data. So let's talk about really kind of three elements you need to think about to build these collaborative relationships.

One is communication strategies. So it might sound simple, [inaudible] regular meetings, create mechanisms for communicating between meetings, but the reality of the situation is the devil is in the detail and if you don't have these regular meetings, as we all have our own lives, people will focus on other things.

So ensuring that you have regular meetings set up, that as leads, you're committed to that and data leads you are thinking about what is going to be setting the agenda and also, what are the action items out of each one of those meetings and how do I communicate that and ensure people are staying on top of those.

Some information-sharing things that you'll want to think through, if you really want to make sure not just people have the PZ goals, but really making sure people understand what those Promise Zone goals are and how each of the data partners are supporting those efforts. So not just enough that you're sharing information, but you're ensuring that they understand it.



4

One way you can do that is making sure that you're constantly talking about those goals. Even set agendas as it relates to those goals. You also, in information-sharing, want to think about clearly outlining what activities will lead to which outcomes and then also, how they will be documented.

So it's not merely enough to say, okay, we're doing this activity and people keep moving along that road, but keep coming back to how does that activity lead to an outcome and how are you actually documenting that? Because you don't want to get too far down the road of an activity without really making sure you're thinking outcome-based and documenting it.

And last, you want to explain the benefits of collaboration and partnerships for each data partner or each stakeholder, for them to tell their own stories. They've got have some skin in the game. Lastly, sustaining engagement. You want to make sure you have a defined logic model of framework really to help with transition.

The transition many of you guys are facing is new VISTAs coming on. So as long as you have a clear logic model, one that is saved and people understand where it is and what it is. Again, even if it iterates over time that you're constantly updating that and making sure people understand where that is, that therefore, transitions will be made easier.

You'll want to assign concrete roles for each subcommittee if you're choosing a subcommittee process and go [inaudible] commitment that way. And then you'll really want to keep your framework at the center of your work. One of our past speakers even talked about they literally put framework on the table of each meeting to make sure people remembered that as their focus.

Lastly, I just want to mention for folks is don't get paralyzed by the need to collaborate. With a collaborative process, things often need to be iterative with a [inaudible] document first for folks to react to the discussion, debate and refining it.

So remember, collaboration takes several phases and several ways between discussions, debates and then you also then take that document out and literally marking it up. In the process, you're going to be asked to go through now to create or refine your framework in really a short timeframe.

You don't have time to always involve all your partners as deeply as you might want to, but you do have time, once the framework is put together, to circle back, make improvements with input from the partners and gain some collaborative sharing responsibilities at that time. Let's talk about group indicators a bit.

These are the indicators grouped in a four-part model specific to the Promise Zone program. Now, if you start speaking to a non-PZ or about group C indicator, they most likely won't know what group C is, but if you talk to a PZ and if they've taken this class and are engaged, they might understand this group C.

So again, remember, this is the HUD-defined way of thinking about indicators to support you and PZs to ensure you're thinking about a wide range of indicators, not just one example. And so



5

there's really four groups we have, A, B, C and D. A are examples of indicators or measures that are possible to track really from that HUD level or a USDA level.

So jobs created, job sectors, employment by sector. Group B is where you're measuring that required -- the local data. So these are really your building permits or crime rates. Group C is slightly different here and these are indicators in measures that must be informed by what we're calling kind of a local definition of context.

So think here about school quality or workforce development participation and the need for that local definition. And lastly, group D is information really, that's only available to things like crowd sourcing or individual level data collection methods that you survey.

And examples of those are kind of maybe your perception or the community's perception of safety and community trust in law enforcement or their perception of their community connectedness. I want to remind you guys that group D is not the only place the survey is done. Many of these groupings will include a survey.

For example, group A -- the census is a really good example of group A and as we all know, because many of us take it, that's part of a survey methodology. Let's just do a quick example of the L.A. Promise Zone, and thank you, L.A., where they were thinking out these groups, again, by the indicators defined by HUD, A, B, C and D.

And they talked about these in a short-term, a mid-term and a long-term range. So here's some good examples that they were giving. So -- and a short-term for group A, for example, was kind of new neighborhood amenities, new investments and money leveraged by source.

Let's go down then to group D as a survey, some mid-term and long-term that they have with the perception of neighborhood quality, perception of safety, really important for you to understand what your community is feeling on that and that's really that group D that you want to figure out and understand at the local level how they're doing that.

Let's stop and take a poll for a second. Good time for us to engage in how we all are understanding these different indicators. So what is your level of understanding of the distinction between group A, B, C and D indicators? A, what are the group indicators? I don't know, what are they? Group B, I don't know the difference between the groups.

C, I'm familiar with them. D, I know the difference between the groups. And E, I have a good understanding of each group and the sources of the data. You'll see on the right-hand side of your screen, there should be a poll that just came up that you'll be able to click A, B, C, D or E and give us an understanding of what your level of understanding of this distinction is.

And remember, these are specifically HUD-defined groups, but none of these indicators are new to the evaluation process. Thinking about them from a national data level with census, for example, all the way to the local level of trying to understand what people's perception are of the community around them, very common ways to evaluate and think about a place-based initiative.



6

But as we're thinking about them within these groups, you might be a little [inaudible] here. So let's check that out. We have a lot of people in the D and E range, meaning I know the difference between the groups or I even have a good understanding. Some people are in the unfamiliar, but not too many haven't been -- aren't familiar, at least on some level, with that.

So that's great, I'm really happy to hear that there has been a good basis for people of understanding these groups and what they're thinking. I'll just point out I'd like to go ahead and pass to Tanim.

I would assume many, if not all of you, have met or reached out to Tanim in the past, our HUD representative here in the PZ program. He's going to be talking a little bit about HUD's expectations about group indicators. Tanim.

Tanim Awwal: Hi, everyone. Can you all hear me? This is Tanim Awwal.

Janine Cuneo: We can hear you, Tanim.

Tanim Awwal: Great. So essentially, HUD's expectations is laid out in the designation agreement that each Promise Zone has signed under Section 8, data and evaluation. And essentially, it's saying that HUD is committed to working with the local partners to track their efforts and how the community is changing over time.

So the reason why we have these group A indicators was just for us to designate the different types of variables we know that Promise Zones would need to evaluate their progress over time.

If you recall, initially, one of three reasons why we did the group A report is because explicitly, HUD agreed to provide these indicators, which are usually census-level data to the Promise Zones. But when it comes to B, C, D, essentially, I wouldn't look at them as they're specifically HUD defined, but these are just our way of grouping indicators already exist.

You know, you are going to need perception studies, you're going to know -- need to know local police statistics or crime or education statistics. And our hope is through the designation agreement that HUD is able to assist you in how we agreed to. So we are agreeing to provide the group A indicators every year.

And we are agreeing to aid the Promise Zones in sort of understanding how to get those B, C and D indicators. So it's kind of our promise for if there's an issue for Promise Zones trying to find certain data, whether it's education statistics, that HUD can go out of their way from our end to further assist you.

And as you know, one of the reasons why we have this technical assistance in the first place is because we've noticed that there were issues in data gathering. So our hope is that this is, I guess, second step, because the first step was the group A webinars last year, but this is the second step in trying to help the Promise Zones better evaluate their data.



7

Our expectations for these webinars are to have a framework for each Promise Zone that's unique, but it's not for us to force you to provide statistics, but for us to better help the Promise Zones showcase their work.

The more we understand what the Promise Zones are doing the better we can, here at headquarters, show partners or federal partners and other people how important the Promise Zones are and how we can better assist them. I hope that's a good summary of what our expectations are. And feel free to ask me any questions here or afterwards.

Janine Cuneo: Thank you, Tanim. Let's take a couple seconds if anybody wants to go off mute again, click their microphone button on the right-hand side if you have any questions for Tanim. He'll also be on the rest of the webinar if you do pop up and think of any questions later on as well.

If you do think of those questions, again, we'll be pausing a couple more times throughout that you can ask them, but as Tanim said, you can also connect with him afterwards.

At this point and time, I'd like to move into working with you guys on how to use indicators to operationalize outcomes or evaluation questions. My colleague Xiaodong will be taking over this session. Xiaodong.

Xiaodong Zhang: Thank you, Janine. Hi, everyone. This is Xiaodong. We have talked about indicator types as they fit into these four groupings, but for PZs, one key is to come up with indicators that appropriately measure what you care about and [inaudible] a way that allows your system to collect and analyze the data.

There are multiple ways to measure an outcome. Use practical consideration to choose among these possible indicators. Here are a few criteria for you to consider. We're introducing a term called SMART indicator.

It stands for Specific Measurable Attainable Reliable and Timely and we're going to explain them a little bit more in detail and give you some examples what indicators are good and what are some of the bad examples. Let's think about trying to choose some indicators for the food preparation workshop that we discussed in webinars one and two.

Just as a reminder that the program is a workshop that was supposed to help participants to make better food choice with the ultimate goals of decreasing obesity and improving their overall health. For the first one specific, the indicators should be clearly tied to your evaluation questions and simple.

In the first example indicator, while this data about which participants are on the food stamps might be interesting and tangentially related to the project, it does not specifically address your outcome and therefore, it is not a good indicator. If you look at the fourth column, [inaudible] indicator, this is clearly related to the outcome.

For measurables, we have to make sure that we clearly operationalize your indicators. The first indicator is not really operationalized and therefore, you can't really measure it. How are you



8

going to define an increase in knowledge? The next example we're still focusing on the increasing knowledge, but we clearly articulate how this increase is being measured.

The third criterion, obtainable, don't choose too many indicators. Always try to choose those that are readily available and easy to measure. So for example, while the first indicator would be very valuable, you are very unlikely to have access to medical records of the participant and therefore, the second indicator would be much easier to obtain.

Let's look at the next criterion, reliable. By that, we mean that you really want to make sure that if you were to collect your data again, you want to get a similar result. So we have to avoid indicators that might have an inherent bias to them or that are obtained from -- not obtained from a standardized data collection, like in the first example.

And therefore, the second indicator is much more reliable. The last criterion, finally. By that, we mean we -- you really don't want to choose an indicator from available data source if it is only collected once every four years or your program lasted only six months. You also want to consider how likely you are to get the data once the program is over in the example above.

And you are unlikely to be able to get information about participant and wait 10 years after a training. So here are just some of the -- again, a reiteration of what we mean by SMART indicator as some of the examples that you can look into a little bit more closely. The other consideration, deciding on your indicator.

What you really want to emphasize should be a group process and it's not something that can be made in the back by those conducting the evaluation. Consulting your partners as well as stakeholders and those who are taking part in the program will really allow you to make sure that your indicator makes sense and they are the best way to measure your outcomes.

When you have several competing indicators, make sure that your divisions are based on practical considerations and make sure that these indicators are readily available, easy to analyze and collect, they're available in the right timeframe and they're available in the right geographic region.

So for example, keep in mind that selecting indicators and finding their data sources can be an iterative process. For example, in looking for data sources for your desired indicator, you might find that the data are actually much easier to obtain for different indicators. That also could allow you to answer the same evaluation questions.

So in that case, you may really want to adjust your indicator and take the ones that are easier to obtain. In the end, the most important things are to select a few indicators that would allow you to meet your evaluation and data goals.

On this slide, we have a table from our data and evaluation framework pamphlet and this would allow you to help you think through what indicator would help you answer the evaluation questions. From there, you will need to find the most appropriate source of data, which we will talk about next. We could do a whole webinar on data type and we won't.



9

Instead, we're going to hit a few high notes and provide you with slides that have a little bit more detail to look at later. As you get ready to select the data type and collection methods to suit your evaluation questions, you will work -- you have to work with the data partners and evaluators to select the most appropriate approach.

But knowing the basic parameters to think about would help you ask the right questions and link your data collection approach up to your logic model and to your evaluation questions. Here's a chart that we're comparing different data types. We first talk about data types.

There are two types of data, quantitative data, a numeric data, mainly to answer what question and qualitative data, a narrative data, to address the why question. We have added two icons with the number sign for the quantitative data, which are numeric and the question mark for the qualitative data, which answers the why question to the next few slides.

To help those of you who have to mentally translate the qualitative and quantitative when you hear them, especially the way I pronounce them sometimes can really throw you off. Let me just quickly go over these two types of data. So for example, in terms of the scope of data, quantitative data usually has a less index, but can really cover a lot of subjects.

In contrast, qualitative data tends to be more in-depth, but focusing on fewer of these subjects. Quantitative data are usually collected using a standardized data collection method with mainly close-ended questions where as qualitative data can be collected using both standardized or semi-structured protocol with mostly open-ended questions.

Quantitative data are analyzed using statistical approach that would allow you to compare, summarize and generalize to a larger population whereas qualitative data focuses on content analysis and its main goal is to shed -- to provide meaning to a result, an illustrative explanation and the view of the participant.

Going back to our four groupings, quantitative data can be -- can cover all four groupings, group A, B and C. They can all include quantitative data whereas qualitative data, in this sense, only collected as group D only, but they can provide and help explain what's happening in any of the group data.

Let's take a little bit of a closer view of what we mean by quantitative and qualitative data and talk about these different data collection methods. Again, quantitative indicators can cover any of the four groupings, but most of them come either from survey, like senses or from test, like ACT or SAT in the college entrance exams.

You can also do you own surveys or tests, assessments as part of your group D data collection. Quantitative data, as we pointed out, can be really helpful in comparing, analyzing the impact of your project and describing the implementation as well, but there are always risks [inaudible]. So for example, you are likely not getting a representative sample because of low response rates.

Therefore, your survey result would be skewed or encounter bias in the test instrument. I'm not going to go into the details of these pros and cons. You know, you can read them a little bit more



10

after the webinar. Two of the most common approaches to qualitative data collection are interviews and focus groups.

Interviews and focus groups are very similar techniques. Individual or group interviews can be used to gather stakeholder perspectives about their experience with the PZ program. Focus groups can be used to gather information from different stakeholders that were brought together to share their experience about the PZ program, for example.

Both approaches would let interviewer/facilitator explore nuances in-depth, but the focus group has the added advantage of letting participants to build on each other's ideas. On the con side, both methods can be time consuming to plan and implement and they usually will take some various special skills to be able to do it well.

This slide shows two additional types of qualitative data methods. The key to participant observation is that the activity takes place in the natural setting where the observer is sort of like a fly on the wall. Think of classroom observation, for example, where the observer would sit in the back trying to be as unobtrusive as possible.

And this would allow the observer literally to see what's happening, but there's also a lot of challenges to assessing and interpreting the behavior that they observe. And finally, we talk about documented review and documented review can really help understand the context, content and implementation of the program.

And sometimes they're a great step to help developing a survey questionnaire or your interview protocol.

The advantage of having a documented review is that they're usually a low burden for those providing them, because they already existed, but they can be tricky to analyze, because they may not be as complete as you want and they may not contain all the items that you are interested in.

Now let's move onto some of the practical consideration in these data collections. As you decide on these methods, you want to really think about a couple of key considerations. The first one is methodological.

The gold standard, if you're really able to do it, is to triangulate your data by gathering information in a couple of different ways and a couple of different type of respondents as sort of a double-check. So if -- but if the resources type, as they so often are -- and this is -- I would say this is nice to have, but it is not a must when we talk about triangulation.

Reliability and validity are often discussed together, because you always want [inaudible] to demonstrate your instrument to be reliable and valid. Now, these are technical terms, reliability and validity, but they're pretty simple in concept. So for example, you really want to make sure that your instruments measures the same thing in the same way consistently.

You don't want to have a scale that says you weigh 50 pounds today and 100 pounds tomorrow and you really want ones that are reliable to giving you the correct weight each time. You also



11

want to make sure that you measure the things that you care about. This is what we meant by validity.

You want them -- you would not want a scale to find out if you have a fever. So you want the measure to be valid. The next one, generalizability and ideally, you really want your result to let your -- to allow you to make inferences about a broader population, because your result can be generalizable.

So these are the concepts, again, even though they're pretty abstract when you look at the terms. They're pretty intuitive, but you really have to think about them explicitly as you choose your methods. Other than methodological considerations, I also want to point your attention to operational and ethical consideration.

So for example, in terms of populational considerations, you have to be able to realistically get the data you want and you need that talk about the data usage agreement as a tool for working with a partner later and you also want to think about the cost of collecting data and analyzing them.

Finally, at any time you're collecting data, you have to think about the ethical issues and considerations. Privacy and confidentiality issues come up regularly and there are established standards for you to treat data appropriately in different fields. For example, we have HIPAA for the -- for health research, we have FERPA for education research.

If you're conducting human subject research, you will need to go through a review process to ensure the ethical integrity and this is known as Institutional Review Board, also known as IRB most popularly by acronym. The IRB -- these boards will take some time and require a good bit of thought about ethical issues.

So just to be aware that they may be a required step in the process. In working with your university partners, they usually have an IRB process that would guide you through the process.

Janine Cuneo: Thanks, Xiaodong. Why don't we go through and take a poll now. We'd love to know how likely you are to gather qualitative data as part of your framework. Again, you'll see the polling on the right-hand side that just popped up and A being no way all the way to E being we're already there.

Again, qualitative data. So as you're taking the poll, let me just remind you, as Xiaodong stated, two of the most common approaches to qualitative data are collecting interviews and focus groups. Some pros is that -- to this method is that it can export a range in depth of topics in rich data.

Some cons, though, is it may be difficult to analyze and compare data, it could be time consuming for the transcription analysis as well and usually, really most likely requires a trained interviewer or facilitator. So we're just wondering if you guys are engaging or thinking about engaging in an qualitative data as part of your framework.



12

Or we talked to a bunch of folks that are definitely working through the group indicators on a quantitative level, but interested to hear about the qualitative as well. So please make sure you're filling out the poll to your right. Again, A, no way all the way through E, already there. Why don't we go ahead and close the poll and we can see the results in a second.

Again, a couple minutes for it to calculate, my apologies. So our responses -- we are hearing that there are people that are already there, but a couple of our folks have said that and almost certainly we're also hearing that people are almost certainly going to be using qualitative.

Some are still on the fence and not -- and I think we got a lot of no answers or really still not sure exactly. For those that are either already there or almost certain, I'd love to hear from you guys just for one second maybe on what qualitative data you're thinking about.

Could you take yourself off mute and let the team and the group know maybe what qualitative data you're considering and/or have already considered for your framework? Please don't be shy.

Barbara Ann: Hi. This is Barbara Ann [ph]. I'm actually with HUD for the Camden Promise Zone. I have to admit, they've done a really good job. One of our more -- our most active committees really are our health committee, the Get Healthy Camden committee and admittedly, it also -- it serves a dual purpose in addition to being the working group.

It has another purpose. And we've done everything. I mean, we've had townhalls, we've done surveys, we've had listening sessions, actual listening tours and a lot of work has gone into making sure that they're developing a health agenda that is based on what the community needs, because earlier attempts at things like that people were like -- said things like, I need healthier food.

That's true, but what they meant, in most instances, was access to healthy food, how to prepare the healthy food, is the healthy food affordable. So they've looked at the past experiences in trying to collect that kind of information to help inform how they would do it going forward.

So surveys that used to be pages long were shortened to a postcard that they could help them fill out at tables at different events, information they could collect in partnership with the local hospital that is running actual weekly garden -- I can't think, I'm blanking on what we call -- mobile farmer's markets.

They can collect information there. And that's just the one committee. A little bit lesser great idea, but not as successful for housing. We wanted to do something similar with housing. So the Camden Redevelopment Authority launched an online survey that could also be completed in person and it was in English and in Spanish.

It was targeted at existing residents as well as people who work in Camden, but do not currently live there and wanting to know why they're not considering that as an option. So I think that those are just some examples. They've done some similar things with the police too.



13

They realized that in order to make sure they're actually fulfilling the mission, you really had to go to the source to get the information and certain things just aren't available quantitatively. So qualitatively is the only option.

Janine Cuneo: All right. Thank you so much. That sounds really dynamic what Camden is doing. I think especially as you're saying the health space, trying to think of different ways to getting into the community. You mentioned the mobile food gardens, etc. There's so many people that are out in your communities today.

How can you leverage that, work on that, work towards that to get the kind of samples -- sample size you're wanting to and getting out into the community to really find out what they're interested needs are. So thank you so much.

Barbara Ann: You're welcome. Yeah. The key is definitely you've got to go where they are.

Janine Cuneo: Great. Why don't we keep moving forward. I'm going to send it back to my colleague, Xiaodong, to talk a little bit more about data sources and primary and secondary data.

Xiaodong Zhang: Thank you, Janine. So we talk about qualitative data and quantitative data. These are types of data. There's another way to look at the data that it's really the data source and it's what we call primary data and secondary data. The main distinction among group indicators really comes down to the data source.

Primary data are new data that you are collecting and secondary data are existing data that were collected by somebody else. There was just a question about what's the main distinction between group B data and group D data? The main distinction is really the data source. Group A, B and C are all secondary data.

They are collected by the federal agency, they're collected by the local and that you are using them. You're extracting them from another source whereas group D data are all new data that are collected D data are all new data that are collected by yourself. The fundamental choice is really whether you want to collect the data yourself or whether you want to obtain it from others.

So let me first talk about secondary data, because most of your data are going to be secondary data. In our grouping, we [inaudible] all the group A, B and C indicators are secondary. They're existing either from your program data or from external data set.

The advantage of having secondary data is that it's usually less expensive and it's usually more complete than collecting your own data. I mean, usually, that's kind of a -- it's not applicable to all cases. The disadvantage of working with secondary data is that it's not always tailored to your evaluation needs and may not provide exactly what you are looking for.

They could be out of date, the data may not be in the format that you are looking for and sometimes gaining access can be challenging as well. But there are some tips in collecting secondary data.



14

The first one is really look for publicly available data and these data may be free and there may be a fee, but the fee might be worth paying, especially if the alternative is to collecting your own primary data.

And when you work with other organizations, working with a partner, ask them what data that they're collecting while they administer their program, whether they will be able to -- willing to share with you. They may also have access to additional non-public data set. The key point in working with secondary data -- one of the key points is having a data usage agreement sometimes known as Data Use Agreement or data sharing agreement. And not all data requires a DUA, but many do.

What we meant by DUA is that it's a contractual document used for transfer of the data that have been developed by non-public or nonprofit organizations, government or private industries where the data is not public or are not public or are otherwise subject to some restriction on its use. The DUI can be formal contractual type of a document.

Sometimes it can be informal, like in the form of a [inaudible]. And in our last webinar, we included a document that provided some guidance to determine -- to help you determine whether you need a DUA or not. I also want to point it out that the DUA are very often sector-specific. So for example, the DUA in health sectors are often driven by HIPAA.

It's a federal regulation requirement. In education, it's also -- often informed by FERPA requirements. And the criminal justice field also has its own consideration as well. Here's an example, a table summarized what a DUA often entails. We kind of separate it by the responsibility to data providers and those responsibilities to data receivers.

We had wanted to provide a template for a standardized DUA, but we finally decided against it because of the difference in formality and in sectors. So we would encourage you to look for an example and the best way would be to Google, for example, but look for the one in the right field, in your field.

Very often education, health and justice, for example, have very stringent and rigorous type of a requirement as compared to some of the other sectors, such as helping or transportation, etc.

But organizations with what we call complicated type of DUA requirements very often already have existing DUA forms and procedures and in place so you don't have to worry about creating it from the scratch.

Janine Cuneo: Thanks, Xiaodong. That was really interesting to hear about secondary data. In a second, we're going to go even deeper in primary deeper. Before we do that, I'd love to hear and talk a little bit amongst ourselves around the challenges with data usage agreements.

Let's ask some questions, and again, please unmute yourself if you guys have any thoughts about this, additional questions for the rest of the group or any answers. And so some ideas is we know a few of you have definitely established a DUA or multiple DUAs.



15

What challenges have you encountered either in the making of it in trying to negotiate it or actually invoking the data exchange? And then for those who have not yet thought about engagement, have you had difficulty in figuring out what is a DUA and how do I describe it and how do I think about who do I partner with?

Anyone have any good experience with a DUA they'd like to share also with the group? Those are some thoughts for you guys to consider. Please take yourself off mute, I'd love to hear from you.

Tanim Awwal: Hi. This is Tanim Awwal. I actually have a question for the Promise Zones. When we first did our surveys, we noticed that it was getting education data because of the lack of data usage agreements and the [inaudible] from schools. That was the hardest to get data from. Do the people online still have that issue?

And could anyone talk more about some of the challenges they've had getting education data from their partners? Thank you.

Janine Cuneo: Again, please don't hesitate to answer and take yourself off mute. We know you guys have had some struggles with that. So we'd love to hear from you so the rest of the group can engage as well. So we're going to assume everyone's good with data usage agreements, although, I'm sure there's still some struggles out there.

Why don't we do this, keep thinking about that, we'll have another opportunity later in the session to talk a little bit more, have some discussion. And so if you guys do think of anything you're struggling with, we'd love to hear. Also, please don't hesitate to use those office hours to talk about and help craft a data usage agreement during the office hours.

They could be of help to think through who do you need to contact, what are some of the sectors that might be hardest, like education, as Tanim mentioned and some ways around that. So please make sure you're using those office hours to think that through. Before I move back to Xiaodong, anyone have any thoughts or ideas about data usage agreements?

Well, we'll keep moving. And Xiaodong, please go ahead and take the floor and tell us a little bit more about primary data.

Xiaodong Zhang: Yeah. Well, now I'm talking about primary data. In our PZ context, these are all the group D data of primary data. These are data that we collect from survey [inaudible] and interview observations that the PZ develops themselves.

The advantage of having primary data is that they are always -- you know, sometimes you don't even have a choice; this is the only option. If you don't have the data you want, you have to collect it yourself. But in addition to that, they always tend to be more aligned to the evaluation question and to your need.

Obviously, the disadvantage of working with primary data is that it can have a low response rate and it could be burdensome and more costly than working with primary data. And next, I'm



16

going to talk about some of the tips. And there are some -- a few questions coming up. So I'm going to address it as I discuss the tips of primary data.

The first one we really have to think about how to minimize burdens to the participant. For example, try to keep the survey as short as possible, avoid collecting these -- conducting these surveys at a time that are in conflict with people's schedules, such as holiday period or you're collecting when the school is in testing period or spring break, etc.

There was a question about survey, because early on I characterized surveys as mostly quantitative data and there was a person asking, it could be quantitative and you're absolutely right.

And the survey could have both quantitative questions that are -- ask in scale -- ask you to kind of provide [inaudible] certain things or it could have qualitative questions asking open-ended questions, but in the spirit of trying to make it more a broader statement, I said it's mostly -- surveys are mostly quantitative data collection method, but it does allow a few -- a limited number of qualitative questions.

So I just want to amend my question -- my presentation. The second tip is to use sampling to reduce the cost and increase generalizability. What we meant is that sampling, such as random sampling sometimes is better than having a population sampling. It could be more accurate as we often find in the case of working with census data.

Sometimes the data from American Community Survey, ACS, which uses sampling are more accurate than census data. But sometimes you can use purposeful data sampling and this is especially useful in collecting qualitative data, which people sometimes just try to target those best-case scenario or worst-case scenario to make your example a little bit more illustrative.

Again, as one of the participants pointed out, sampling is a way to improve the generalizability of your data -- of your result. And finally, we want to talk about -- we really want to focus on improving response rate, I kind of mentioned it a couple of times, because no response rates can really create bias.

So for example, only people who feel very strongly, either in a negative or positive way, they're likely to respond. So if you only have like a 10 percent response rate, you're going to have a very skewed result and only capture those people who feel strongly about the subject. So how do we improve the response rate?

One way would be to collect the data in person. When you have the training or when you finish the training, make sure that you [inaudible] those satisfaction survey immediately. Don't let people out of the room. Second was to send reminders. People are busy. So you have to kind of send reminders on a regular basis so that you can help remind people to respond.

Or provide incentives, such as gift cards or if you collect focus group interviews, provide pizza if you can afford it. But more importantly, I just want to point out that you really have to explain to people how the result is going to be used to improve the program, to kind of invoke their -- to really let them see how important their effort will be.



17

To wrap up our discussion about primary and secondary data, I'd like to share an external source compiled by Promise Neighborhood Research Consortium, PNRC. It is a consortium funded by NIH to promote neighborhood research that lists some secondary data measures and primary data measures that might be useful to you.

These are, by no means, exhaustive, but we found nonetheless they could be useful. We also included a link to the website in the reference section of this webinar.

Janine Cuneo: Thank so much, Xiaodong. That was really helpful to understand these differences. At this point, we're excited to have Amanda from the West Philadelphia Promise zone here to discuss their use of group A data. Amanda, I'm going to pass it off to you.

Feel free to introduce yourself and dive right into West Philadelphia Promise Zone experiences. Thanks so much.

Amanda Hallock: Hi, everyone. This is Amanda Hallock. I'm the strategy and evaluation VISTA here at -- in West Philly. I started last August. So I'm coming up on the end of my term this August. And today I'll be talking about kind of what I've been doing over the past year in terms of census data and group A data, specifically with the American Community Survey.

So next slide. And you're free to contact me if you want to look at that slide and see the email. Yeah. So I figured I'd start off by saying -- so the thing that I figured out or the thing that I was working on this year was a better way of getting census data.

And so I thought I'd start out just by going over, with one slide, super basic reasons why census data has been useful for some Promise Zones, but I won't spend too much time on it. So one of the main ways that I think it's been useful is just knowing the demographics that people are going to ask you for.

Even just yesterday you sent out poverty rate and unemployment rate and just knowing those things are really important, but then there's also more in-depth things you might want to know that wouldn't be on a basic demographic sheet, like whether people are insured or not or their commute to work or the amount of people that bike.

So all of this information seems really useful and has been really useful. And if you look down at the bottom, there's a link to FactFinder, which many of you might know about and this is the way to get this information in the easiest way possible. So it's really good if you want to pull it just about one thing for one year for a couple of census tracts.

The barrier that I was coming up against was that we are many census tracts in over many years and we have tons of information that we want. So using FactFinder isn't really the most efficient way to do it. So someone in my office recommended that I started with code and I started to learn how to access the census API.

Thank you for moving the slide. So the API, this is where it gets wonky and it gets a little bit past what I understand as well. So the main part you need to know is that an ATI is a part of a server that receives your request and then sends you back information.



18

So when this [inaudible] receives my request to get specific variables for the tracts I want in the time I want and it sends it back to me and I can put it into an Excel doc. And the way that I instruct it how to do that is by using the statistical language R or the programming language R. And so the main benefits to this are that it's reproduceable.

So if I wrote out -- since I wrote out this code, I can send this over to another Promise Zone and they can input their tracts and they can use the same exact code to pull the same variables. So that's kind of the exciting part of it, because if you think about it, 22 Promise Zones, like 60 variables we might want over 10 year the math gets wild.

And the other best part of it is that you can go back and check. And I'll be talking a little bit later about the tool that HUD sent out and the Excel sheet, but it was very helpful, but it was difficult to go back and check.

And code is kind of a way where if you're -- if you want to do it yourself and then your partners come back to you and they want to check where it came from or they want to like ask for sourcing, you can always go back to the code to figure out where the Excel sheet pulled from. Next slide.

So if you look at the bottom image, what I was talking about figuring out exactly where it was pulled from the best way to do this is in column B where you have the variable identifier. And this I found very helpful, because then you can be certain that you're pulling the same exact variable over time.

Otherwise, you could be pulling different ones, which again, you'll see later on, but the American Community Survey is sometimes confusing, because it repeats variables for different purposes. So just by reading the text, you might -- it might get confusing. I know I've been very confused by it in the past.

So these identifiers have been really helpful for me. And then at the top, you can see just why this method is so helpful just because of the large amount of variables that are available from the American Community Survey. So just so many, like what I was talking about, journey to work, occupation, ancestry.

You can find out there are so many things for ancestry. You can figure out how many people are from Croatia or something; you know? It's really extensive. So next slide. So this is a sample of code that I wrote or that I -- actually, I wrote it based off of one of my coworker's codes and I wrote in an R studio, which if you go on this journey, will find out what that is.

But it's basically just a screen that you type things -- that you type the code into and then it helps you run it. And I can send out this template, but it basically runs the code and then it comes out. At the bottom, you'll see write CSV. So it pops out with a CSV of all of your data. Next slide. So on the right is what it came out with.

So you can see for every tract, it has the data for total population, female, black, total for employment status, all of those different things. And then at the bottom of that, you can sum it



19

all up, which is what I did at the bottom here where it says R. And then on the left, I put a HUD table, which is what they sent us.

So my first pull with R, I tried to really base it off of a HUD table just that I could see how close I could get to that. And one of the barriers that I found as a benefit with R was that I knew that the population number I was getting was the population, because without those IDs, it really is difficult. And that slide, I'm good with that. Thank you.

And then the last slide is about getting started. So I won't go through this, but I figured I would add all of it onto a slide just in case you're tempted to go through it. What I will talk about when we get to -- since we've gone through the slide is the drawbacks of using R. It is time-intensive, it does take a lot of personnel -- not time-intensive in typing it, but time-intensive in learning it.

I did not code until last October and November and then I started taping the tutorial that you can see at the bottom on DataCamp. And after that, I was pretty much set to start off on this. And when you have a project, it's really a lot easier to learn it.

So its' something that you'd have to commit to and because of the turnover of VISTAs, it might be daunting to do that, but I think it could be a really great solution for a test problem. So if anyone has any questions, I'd love to hear them or if not, you can email me later.

Janine Cuneo: Thanks, Amanda. Definitely, let's open this up for discussion again. Please make sure you unmute yourself on the right-hand side. This definitely [inaudible], as Amanda said, HUD had passed around some information around data sets in group A, but a lot of discussion on what to do with them.

We have one VISTA, you're great, Amanda kind of diving into the R coding and figuring some of that out. And I'd love to hear what you guys think about that, how difficult it might be on your end or have you done it yourselves.

What are some thoughts here on R? I think there might be someone speaking. Make sure you're unmuted and speaking as loud as you can into your microphone or head set.

Phillip Armstrong: Can you hear me now?

Janine Cuneo: I apologize, I still can't hear you very well. You want to try again? And if not, you can type your answer into the question, we can see if we can grab it from there.

Phillip Armstrong: I already did.

Janine Cuneo: Oh, great. We can hear you now. Go ahead and speak as loud as you can. Let's try that.

Phillip Armstrong: Yeah. Hi. My name is Phillip Armstrong [ph] in L.A. Promise Zone. And prior to starting my VISTA service with the Promise Zone, I had previously done VISTA service in another jurisdiction in the same general area. And I had attended a presentation by a representative of the census bureau.



20

They have people called data disseminators and they are willing to go out and train anyone on using the census data --

Janine Cuneo: You're cutting out a little bit.

Phillip Armstrong: Okay. And specifically, they trained us on how to save queries in American FactFinder. There's a -- when you -- in the advanced search, there's a option there to save your query and using that you can save -- we have 65 census tracts. So we never have to go back and reenter them. We can just put them in the query.

And then also, we can choose the data sets we want and we can choose the data so that we can save those queries and sort of accomplish some of the same functionality or utility that you're getting with the R data without having to learn another program. And I'm assuming that's something that the -- there was a demand for.

And so the census bureau created that. And as I say, they have data disseminators who can come out and train your Promise Zone partners in using census data. We've had a training -- we have actually had two workshops done by different data disseminators for our Promise Zone partners and --

Janine Cuneo: So thanks, Phillip. You're cutting out a little bit, but I think what I heard from you, which I -- sounds like a really interesting and important thing for the rest of the PZ to hear.

So if you mind, I'm going to repeat a little bit of that just to ensure everyone hears that it's -- that you guys had found that you were able to contact your local census bureau and get census bureau data disseminators, I believe was the name in which you used and there are other such data disseminators across the country.

So not just specific to your area of your understanding. And they were actually able to come out -- I believe you had heard workshops and trainings in there even and it allowed you guys to understand data, how to use the census better, but also, how to save queries in American FactFinder and under -- and be able to save your data sets, which allows and accomplishes many of the same things that we are hearing from Amanda as well.

So thank you. But what a great resource for other folks to know about these data disseminators. Contact your census bureau to see if you can't have you and your -- you, being VISTAs out there, bringing that together or PZ leadership out there, seeing if we can't bring some of these data disseminators to the local areas that might need a better understanding on how to facilitate and accomplish the same things Amanda's talking about here maybe in just a different way.

Amanda Hallock: Yeah. We've had a -- this is Amanda again. We've had a lot of good help with that. So that's an excellent point. Like the saving the queries is really -- it just saves everything, because we do have -- I think you said 65, but we have 11. So it's a little bit less saving my life than it is yours, but it's very important for us as well.

And then also, our local census bureau is great too. So probably if two of them are helpful, I would bet a lot of them are.



21

Janine Cuneo: Great. Well, thank you so much. I think we have a couple of great resources here really digging deep into the technology and the applications through using R code, but also, in your local census bureau and connecting with them. So thank you very much, Amanda, we appreciate it.

And again, Amanda did leave, I believe -- I think in the slides that we sent out that you guys have in PDF, her email is on there and I know she had specifically asked to make sure we have it on there so you guys could email her if you have any questions or thoughts.

So thank you so much. Let me go ahead at this point and time and pass it off to my colleague Kasia who will be talking a little bit about data management and collection of data. So who, when, where, why and how. Kasia.

Kasia Razynska: Thank you so much. Hi, everybody. I am now going to discuss a little bit about what you need to do to have some great data management practices and what you need to think about.

So we've already talked about the type of how to choose the data you're going to need for your evaluation and how to be able to make sure that that data answers your evaluation questions. But now that you've gathered data, what we're going to think about is how you're actually going to store that data and maintain it and that's what we mean when we say data management.

Data management's practice is the practice of organizing and maintaining the data once you have collected it. A lot of times people think that data management practices really only apply to primary data, because that's raw data that you're collecting. However, I want to emphasize that data management practices are very important for both primary and secondary data.

And good management includes developing an effective process or consistently collecting the data and recording it, but also, processes for storing your data securely, backing it up, cleaning the data and modifying the data so it can be transferred between different software score analysis.

We discussed data sources, types and instruments. In the next few slides, just like Janine alluded to, I will talk about who, when and how to store and clean your data to protect it. We will go into depth about into -- how to analyze your -- the data in our upcoming workshop. So as I said on the previous slide, the first thing --

Data surge is important for both secondary and primary data, but the questions are a little bit different and the process that you need to go through is a little bit different for both of these types of data. The first thing you definitely want to think about and clearly assign are the responsibilities of your team for your data.

For secondary data, just because you're not responsible for the data collection, it still doesn't mean you don't have to think about the data management. Some data, like we've already alluded to, are available publicly, but you still want to make sure you clearly outline how you're going to clean the data for your particular use and how you're going to prepare it for your analyses.



22

For secondary data that is governed by a Data Use Agreement, or DUA, as we've spoken about to before, you are going to have a list of very specific responsibilities on your end that you have agreed to by taking that data and you want to -- before you do anything with that data, make sure you have a clear plan on who on your team is going to be able to take on those responsibilities and how you're meeting all of your agreed-to rules and guidelines for using of that data.

The worst thing you can do is be ready to publish your results or have to find incredible findings and only to find out that you're not in compliance with your data use agreement and therefore, you're not able to really use the data in the way you thought you were going to be able to do it. So before you do any analysis, you want to make sure you think of those things.

For primary data, on the other hand, you -- there's some specific responsibilities you want to clarify with your staff and with your partners. First, you want to make sure your -- everybody is aware of the instrument that you're using to collect your data.

When collecting primary data and when there are different individuals who are responsible for the data collection, consider whether maybe that person collecting might introduce some sort of bias or perception.

For example, somebody who's an -- one of the persons conducting data collection is an expert on the topic and somebody is a novice, they might -- and you're collecting qualitative data, they might have varying perspectives on what they're hearing. Secondly, you also want to have specific instructions for how the data is going to be submitted.

And you want to make sure that all your data collectors are clearly trained and understand these data submitted practices or otherwise you might be introducing bias into your data collection.

The final consideration, and this goes both for secondary and primary data, but mostly for primary data, is you want to have very clear procedures for protecting Personally Identifiable Information, sometimes referred to as PII. A lot of times people tend to think that the PII only refers to a person's name.

However, PII is any information that can tie that respondent and their responses together. So any information that can make it possible for that -- for the confidentiality of that data collection to be basically -- for somebody to be able to figure out who gave those responses, any information like that, such as name, address and race is considered Personally Identifiable Information and you need to be very, very, very specific about how you're going -- what you're going to do with that data and how you're going to store it to make sure that you don't disclose that data.

The next thing you want to do is you want to talk about -- is you want to think about the timing of your data collection. You can collect your data before your intervention, meaning collecting and the benefit of that is it allows you to get baseline data and you will now have data that you can use for a comparison at a later point.

You can also think of -- you can also collect your data during the intervention. Some data are easier to collect during the intervention, such as for example, sign-in sheets or observations. You



23

can, of course, also collect your data immediately after the intervention and this can help you measure short-term or mid-term effects of an intervention.

Or for -- and a good example of this might be a change in knowledge. Finally, you can collect your data after some period has passed from the time you have done your program and this will help you measure long-term outcomes that might have not manifested themselves immediately after your program.

Next thing you want to consider is data cleaning and checking. You want to, for both primary and secondary data, check for completeness. In terms of primary data, has the survey been filled out completely? Have all sites provided the data? You want to make sure you notice any gaps. The second thing you want to check for is consistency.

Have all the sites been using the same protocol for collecting the data? If one site is offering incentives and another site is not, that might be introducing bias into your data that you want to control for. The next thing you want to think about is the accuracy of your data.

Something very quick, if you're looking at survey data, if there was a respondent scale that's just one to five and somebody accidentally typed in a six, you know that that's a clear typo and you want to make sure that you have a process for dealing with that. And the last thing you want to think about is checking the verifiability of your data.

You want to be able to reliably rerun that data, if need be, and confirm that -- rerun that data and confirm that your data is reliable. So you want to have a process for making sure you can do that. And for example, if you data was collected as a paper -- in paper form, you want to make sure you have a process for maintaining that data.

And the last thing you want to think about is your storage and protection. Again, if your data was collected through a DUA, you might have very specific guidelines for how you're to store and protect your data and also, some data that you might have collected through other means you might have these specific guidelines.

So you always want to look at your data use agreement as a starting place for thinking about your storage and protection, but for primary data, as it was stored in a hard copy, you need to consider the process for transferring it to an electronic form.

If you have data coming from various sources and you're putting them together into one data set that you're going to be using for further analysis, you have to outline that process so that it reliably can be replicated in the future and so that it's documented.

And so somebody comes to you and says, where did this data come from, you have a specific explanation for this is the original data source that we -- where the data came from.

If you have any of that Personally Identifiable Information, or PII, you have to think of a specific protocol of how you're going to make sure that the data -- that the data remains confidential and so that others don't have access to it. And one suggestion is you want to make sure -- you might want to make sure that those files are password protected.



24

Finally, many of your -- the data that you receive will have specific guidelines as to how long you can store them and when and how you should dispose of them. And so you want to make sure you are in compliance and that you have followed those guidelines clearly. I will now pass it back to Janine.

Janine Cuneo: Thanks so much, Kasia. That was really helpful for us to get a sense of managing data systems.

We're going to now go ahead and pivot to a new speaker, Elder Sanabria, from the Los Angeles Promise Zones and he'll be using his local experiences with the PZ to discuss how they're going about data collection, management, evaluation, all the good stuff that we're all talking about here, but being able to see it from the insight of a local PZ partner.

Elder, are you on the line and can you make sure you're unmuted? Hi, Elder. Are you there?

Elder Sanabria: Yeah. Hello. Can you hear me?

Janine Cuneo: We can hear you now great.

Elder Sanabria: Okay. So thanks. My name is Elder Sanabria. I'm the L.A. Promise Zone manager at the mayor's office, recently onboarded a few weeks ago, but I was a former VISTA and VISTA leader for the last two years. So I've definitely been around.

Thank you for allowing me the time to talk a bit about our data collection strategy and the process that we underwent that helped us develop our system for data collection, which eventually led to our creation of our initiative scorecard and performance dashboard.

So I think that by the end of this presentation, everyone will have a better sense of the thinking that our data team went through to decide on 13 indicators to track our progress, but also, the steps that we took and questions that we asked ourselves and how, moving forward, we want to use data to make decisions.

A bit of context, the first initial proposed metrics and indicators that started this whole thing came from the Promise Zone application itself. You know, many of you are aware that the criteria for the first-round applications were very different.

You know, the applications were mostly narrative and applicants were also required to have one of three neighborhood vitalization initiatives, that being the Choice Neighborhoods, Promise Neighborhoods and Byrne Criminal Justice Grant, L.A., of which had all three, and each of those had their own set of metrics and indicators.

Our theory of action and theory of change in the application had the very first set of metrics for the Promise Zone that we proposed to measure.

During that first year, the L.A. Promise Zone operations team, comprised of our community liaison, our director, the VISTAs and a few interns, created the first placemats, which incorporated the strategies and proposed activities in the application, again, which was mostly narrative.



25

These placemats drove the work for the working groups that first year with the groups helping define the activities and strategies that our partners committed to implementing. But it was shortly after that first year that we hired a Harvard fellow to really help us create our data and evaluation framework with the input of our director and liaisons.

That framework refined the application's theory of action and theory of change by focusing on creating the first potential visualization of both theories along with a draft dashboard to track the progress of the initiative.

It was right then and there that our data and evaluation taskforce was formed, which included our data partners, the city, our HUD liaison, our director, aimed at leading the charge for -- to really help selecting our indicators and that taskforce was led by our liaisons. They began collecting a list of potential data related to the strategies and goals in our plans.

But it was through this process that we realized we really needed to be strategic about what we were going to use to measure our impact and our progress. Our taskforce decided that we needed to whittle it down to about 10 to 15 indicators and really tie them into the goals using selected criteria.

So the first steps in the selection process started with us looking at the different types of data that's collected and at the city level, county, state, federal, census and other sources.

We looked at how often the data was being released and updated, was it quarterly or annually, whether or not the data was released consistently and if it was annual, was the data being released around the same time every year, wanting to make sure that we could count on that data being there every year, looking at how accurate and reliable the data sets were and then also, like mentioned earlier in this webinar, how attainable was that data.

Was that data publicly available or were we going to have to count on data sharing agreements to retrieve the information? Was it easy to analyze? Was it easy to visualize? Of utmost importance was also geographic focus.

You know, given that our boundaries and I'm sure that other Promise Zones don't exactly align with council districts or zip codes, but looking at whether or not our data could be looked at at the census track level so we could get an accurate -- accurate data within our boundaries.

I think one of the best ways to summarize this piece and what was key to building our data system was using what's called the CART principles. You know, I think that when we talk about assessing impact, that means that we need more information about what would've happened if the program did not exist, if the Promise Zone was not around?

You know, sometimes getting that information can be incredibly costly and difficult to get and yet that's what we're tasked to do. And with that in mind, I think that we were intentional about ensuring that our systems for data collection around impact was indeed the right data for us and that we were not wasting resources when it wasn't the right time to measure.



26

CART, which stands for Credible, Actionable, Responsible, and Transportable provides organizations with a set of principles that guides them in deciding which data is critical to collect, but also, doing more by fully integrating that data into what they do.

You know, in other words, a right fit system, which when done well, can further internal learning, demonstrate transparency, accountability to the public and provide clarity on how program activities are actually taking place. I won't dive into each one too much, but really briefly, if we can go to the next slide, one more.

And then -- so Credible, just take a look at that one while I'm talking about CART -- Credible is all about ensuring that the data you're collecting is valid and reliable and that you can actually analyze it accurately.

You know, high-quality data, the impact of this program cannot accurately be measured without measuring what would've happened in the absence of a program, like I mentioned earlier. Actionable, which calls on organizations to only collect data that they will use. You know, asking questions with each and every piece of data that you're collecting.

Is there a specific action that will be taken based on the findings? Do you have the resources and a commitment required to take that action? Responsible is about ensuring the benefits of data collection outweigh the costs and not just the direct costs by counting the data collection, but also, opportunity costs, time spent collection data that could've been used elsewhere.

And then there's Transportable, which is really about making certain that the data you collect generates knowledge for other programs, which is particularly important for impact evaluations, which generate evidence that can be relevant for the design of a new program or can support the scaling of the programs that do work.

A quick example based on our data, we know that indicators are valuable, but really, what is that value? You know, in our Promise Zone, graduation rates went up to 91 percent from the low 70s from the beginning of our designation. You know, that's an exciting number. Having an indicator is great, but the question now is how do we use that data to make decisions?

You know, the decision would be what program should we continue, discontinue or invest more in? Evaluation would be what could we replicate? In the long-term, what can we replicate and what are the practices that we would use to replicate? And that's where the design comes in. Can we go to the next slide?

And so this is just a look of what our online scorecard looked like once we selected the 13 indicators and collected all that data. You know, this is kind of the way that we visualized it.

You know, we wanted to make sure that it was easy to read and easy to understand. This is just a snapshot of our public safety scorecard and some of the metrics that we were using and kind of what it looks like. Next slide. One more. There you go.

Another piece of our strategic plan was to create accountability systems in the form of a performance dashboard, which was used to assess if we're engaging our partners effectively



27

through the various forms of communication, our newsletters, our funny bulletins, research and training bulletins, tracking attendance at meetings, clicks and interactions with our social medias, all information that we collect.

Also, grant information, keeping an inventory of the grants to see who's applying and receiving awards. This really helped us inform -- you know, it informed us how we're doing as an operation, but also, identify the areas where we need to improve. You know, maybe there are some partners who aren't as engaged anymore. We need to take a look at why not.

You know, is it because the working groups are not tailored to what they -- you know, to what their interest is in? And this is all data that we have control over. You know, we collected this. So that was kind of the answers of why we created these performance dashboards to kind of really express where we were and how we were doing.

On the topic of identifying areas to improve, the next few slides are about some tools for assessing where you are as a collective impact project and your capacity to evaluate.

You know, four years into our designation we wanted to take a step back and look at ourselves and evaluate where we are as a collective impact project, our organizational capacity and readiness to use data for evaluation. Our community liaison shared some tools, which myself, the director, lead implementation partner also [inaudible].

The first tool that you see on your screen is the evaluation capacity diagnostic tool, which captures information on organizational context and the evaluation experience of the staff. For example, the tool can really highlight strong areas of capacity as well as areas for improvement as well as gauge the changes over time in an organization's capacity.

You know, it also -- this tool encourages staff to think about how the organization or program can enhance the capacity by building on existing experience and skills, but also, serve as a precursor through evaluation activities with an external consultant. The L.A. Promise Zone used this and averaged the scores of all the folks who took it to get an overall score of our efforts.

The next slide. The second tool is a self-assessment and planning tool based on the phases of collective impact with the purpose being to help your collaborative leadership pause and take stock of where you are and the progress you've made in advancing your initiative for your collaborative leadership to also consider what is needed to support the work moving forward.

You know, it will enable you to assess your progress within what they refer to as the core components of success needed to effectively sustain collective impact efforts. And on your screen, you'll see it's collective -- I'm sorry, it's governance and infrastructure, strategic planning, community involvement, evaluation and improvement.

To summarize, our Promise Zone [inaudible] towards developing a data system that made sense for us. Didn't happen overnight, but I think that it was critical -- what was critical to us getting there was really being conscience and knowing how much and what type of data to collect, knowing that a push to demonstrate impact could really waste resources.



28

I'll close with something I read a few days ago, a paper that posed three questions that an organization can ask itself to apply to any data collection strategy, the first one being can and will the cost effectively collected data help manage the day-to-day operations or design decisions for your program?

Is the data useful for accountability to verify that the organization is doing what it said it would? And three, will the organization commit to using data and make investments in organizational structures to do so? I think that's my time.

If we have time for some questions, at this time, I think if our HUD community liaison is on the line, he was also there from the beginning, and involved with the initiative, and kind of supporting the role, and leading the taskforce for data and evaluation. You know, we'd be very happy to hear some questions.

Janine Cuneo: Thank you so much, Elder. This is really, really engaging, I think, for so many of us. I also want to point out you had had -- in some of those, especially in those self-assessment evaluation, you provided some links on PDF and Excel files for folks. So thank you so much for that as well.

Also, we have the contact information for Elder if anybody's interested in finding a little bit more out. But since we do have him on the phone right now, let's see if there's any questions or comments or thoughts that you guys might have about his presentation, any connection with the work you're doing at the local level. Please feel free to unmute yourself.

Has anybody else tried to take the data visualization [inaudible] to what L.A. has done? Maybe there's public. And he said it's -- the goal is to make it easy to use. These scorecards really are dynamic into decrease and increase based on period of time. Anybody else doing that and anybody have any questions for Elder on how they've done that?

Any challenges they might've had, pros and cons? Any other questions overall? We've talked a lot today and we're going to get into the homework in a second, but wanted to see if there's any more verbal questions.

We had a couple that came over through the Q&A that I'm going to try to get to, but before that, any questions that have come up about anything, either questions for Elder or any other topics today on data collection and management?

Well, while you're thinking about that, one question did come in, is that any examples of PZs and frameworks that are conducting evaluations, kind of -- do we have any examples of those, thoughts about those that maybe we haven't talked about already? I wanted to open it up to everyone, not just the other presenters.

But anybody have any thoughts? And if not, presenters, do you guys have any thoughts you want to include there?

Tanim Awwal: Hi. This is Tanim Awwal. I have a question for Elder, actually. It's great to see how you made that scorecard and easy way for leadership to see it's not just a bunch of numbers.



29

How have you seen the success of your work in terms of developing policy changes with your directors or other partners and what are some challenges in presenting that data in a way that's easy for other people to see?

Elder Sanabria: Yeah. So I think that right now we're at that point; right? We're at the point where we haven't really been able to see -- you know, we're only four years in. We haven't really been able to see that much in terms of where are we. And so the -- we're looking at the data point that I mentioned earlier, 91 percent graduation rates.

You know, for us, we're really at that point where we're having to ask ourselves, this number is important, but what do we do with this number?

You know, how do we make data-driven decisions moving forward so that we're not just doing data collection and we're not just looking at this data just for the sake of collecting it, but how are we using this data to transform and move forward a lot of change and stuff?

So we're not quite there right at that policy change area, but what we've done through the evaluations that we did in our self-assessments and -- we realized that we're at that point now where we need to take a step back and look at ourselves and say, how do we use this data and analyze it to improve performance and make recommendations?

And I think at some point, all the other Promise Zones will agree that that's a real big challenge once you're collecting all this data, and you're trying to look at it, and integrate it into your work, and using the logic models. And with everyone's limited capacity, I think that that's definitely a challenge that we still have to address, so to speak.

Janine Cuneo: Here's another question for you, Elder. How did you guys create sustained engagement over this long period of time as it relates to data collection and data management?

Elder Sanabria: In regards to our taskforce?

Janine Cuneo: I think in regards to actually the entire expansion, kind of your partners. Did you create subcommittees and allow for multiple leaders to present themselves? How did this taskforce really sustain itself over the long-term?

Elder Sanabria: Right. So I think obviously, the sustainability is really important and with our HUD community liaison being around for so long and kind of being that push for us to continue with this work. And we have a leadership council, we have working groups and in each of these groups, we're kind of at the very onset of the initiative.

We're really responsible for kind of helping drive forward what we were going to be doing to eventually kind of let us figure out what are we collecting and what are we trying to measure at the end of the day? We have all these activities that we say we're going to do in our strategic plan.

And when we created that, that involved a lot of partners, it involved a lot of people who were involved in the application. You know, what are we doing with all the statistics and all those



30

strategies that we had? You know, are we actually going to be looking at ourselves and developing these scorecards?

And I think that was really the buy-in for all our partners and our data and evaluation taskforce with our partners at USC and our friends at the city, our chief data officer helping us visualize all these things in the long-term.

I mean, I think that because we're in every -- you know, there's so many different phases of this collective impact initiative and we've done this data collection piece and we're there now, but what's really exciting and I think for the next couple of years is our team looking at it now and realizing that what's needed for data review and evaluation is a basis to learning and adapting.

And that's really kind of kept the core group together and kind of figuring out a way to use the leadership council to also inform what's going on at the data level and the expertise that we have at the table.

Erik: Yeah. And this is Erik [ph]. Can you hear me?

Janine Cuneo: Yes. We can hear you. Please.

Erik: Yes. And I wanted to add to Elder's conversation, which was spot on, was that just even just this week we had a convening of our leadership council for the L.A. Promise Zone and part of the discussion, which was a data-driven discussion, was, again, reexamining our theory of action and theory of change, which is focused on the Cradle to College & Career continuum and supporting families and enabling them to have equitable, livable and sustainable communities in supporting the families in the community.

And when we're hearing the dramatic increases in our graduation rates, which is one of our shared measurement system tracking tools, what now begs the question for everybody is if those high school students that we've been improving their lives and having them graduate, now they're going to go to college as we focus on the college experience and then to the career stage, what are our next driven policy-making decisions and focused areas?

Because we directly know that the work that we've been doing in the last four years and a couple years prior with our key neighborhood revitalization initiative grants definitely made an impact on graduation rates in these high-poverty communities. But now what's the policy-making decisions?

What's the strategic vision moving forward with tracking that cohort as they go to the next stage of our continuum? And then reinforcing that same behaviors on the early stage to see if we can make a difference as well.

And then that also begs the question as what could we then do as the group of the leadership council to make policy decisions to help other high-poverty communities?



31

So we had a very robust discussion using this data and using this whole concept that Elder was sharing to drive our initiative and how we're examining where we're looking, where we're focused and how we want to go forward.

Janine Cuneo: Right. How often does that council meet, Elder? Or Eric?

Elder Sanabria: On a quarterly basis.

Janine Cuneo: Great. Someone was also asking about if a PZ has dealt with the collection of data, if your actual PZ cuts through a census tract. Elder, I recall you talked a little bit about, when you're looking at data collection, the geographic focus. And have you guys had any of your PZ cutting through a census tract? And if so, how'd you deal with that?

Elder Sanabria: Actually, I want to turn this question to Phillip if he's on the line, because he's our MAPs guy and he could probably answer that question better than me. Phillip, are you there?

Erik: Yeah. And let me just jump on --

Elder Sanabria: Oh, go ahead.

Erik: -- I can jump in on one aspect of that. I do want -- just have one aspect of that real quickly, because this is [inaudible] around. This is Erik and then I'll turn it over to Phillip.

But what we did when we originally looked -- we do cut through census tracts in our Promise Zone and we actually have a couple census tracts where the boundaries are less than 10 percent -- or the geographic area is less than 10 percent of the entire census tract. And some of those census tracts, unfortunately, are abutting very, very high-income census tracts.

We have this diametric system here where we have like two different Los Angeles's where we have some of our census tracts abut some extremely high-income census tracts with these really low ones.

And so we made a decision, as the data and evaluation taskforce, which tracts we thought did not make sense to include in the census data, because it would skew the data too far. And I'll turn it over to Phillip as well.

Janine Cuneo: Great. Phillip, are you wanting -- can you unmute yourself?

Phillip Armstrong: Oh, am I unmuted?

Janine Cuneo: Yes. I think, again, we're going to have real trouble hearing you. So please talk as loud as you can.

Phillip Armstrong: Okay. So as Erik said, we have some census tract that are in and out of our Promise Zone.

Erik: Can't hear you, Phillip.



32

Janine Cuneo: Unfortunately, we're not going to be able to hear Phillip. Why don't we go ahead -- we'll see if we can't connect with him afterwards any information we can share around with the group during the next session. We apologize for that.

Amanda, if you're still on, did you guys struggle at all with the census tract issue, meaning any of your collection of data actually cuts through a census tract? Make sure you unmute yourself also. We might've lost her already. Why don't we go ahead -- any other questions before I move on?

I don't want to -- I want to make sure I don't drop anybody. Any other questions for Elder or for Erik, who's HUD's liaison, anybody else on the call too?

Angela: Hi. This is Angela from slate B.

Janine Cuneo: Please feel free, Angela.

Angela: Yeah. So my question is I've always been very impressed with L.A. Promise Zone's ability to actually bring together the data partners into this taskforce. Can you kind of go a little bit into how that came about and any kind of tips or advice for Promise Zones who are trying to do something very similar?

Janine Cuneo: Great question. Elder or Erik, would you guys be able to --

Erik: Yeah. I'd be glad to jump in, Angela. This is Erik. So very early on one of the things that we want -- while we were looking at the collective impact framework and we knew that shared measurements was one of the key components of collective impact, we wanted to make sure that we were looking at it from -- as Elder had articulately stated is we wanted to look at it from two frameworks. We wanted to look at the framework from how are we going to collect those key indicators and metrics [inaudible] time-sensitive way to do that and then our performance dashboard.

So that's where we decided on having a scorecard and a dashboard and we set very -- within the first year, we gathered a core team of partners together. So we had the chief data officer from the City of Los Angeles. He had our academic partner, Dr. Gary Painter from University of Southern California and the Price Center for Social Innovation.

We had Allison Becker, who was our Promise Zone director at the time, myself and then our Promise Zone VISTA team and then Dr. Tara Watford who's with the Youth Policy Institute who's with -- she's their data and research director there at the Youth Policy Institute. And our core team got together and decided to have quarterly meetings.

There were sometimes we actually had monthly meetings, but that core team then established that we would need to create this data and evaluation framework. So that's when we started very early on and we didn't have the components in place, but we thought as long as piece it together. So we were very intentional on saying we want to develop this framework.



33

We started with the big picture of collecting the data, but then what would it look like. And so we spent a lot of time saying, we want an online scorecard, we want a dashboard for the operations team to share with the leadership council.

And so that core team is what we developed and used to kind of meet quarterly to talk about what are the goals that we need to do when we set certain deadlines? We had to meet more frequently to make sure we accomplished that, but we did have specific deadlines when we wanted things up.

And of course, when you set those deadlines, sometimes you have to scale your efforts back, but we knew we wanted to have that in place so that we didn't get too far down the initiative if we ever got to an evaluation framework to actually be doing any of the performance or impact or outcome evaluations, we want to have those pieces and measurements in place.

And then finally, the stage we're at now, especially because we're reviewing and revising our evaluation plan, we're thinking of actually expanding that data and evaluation taskforce to be more of a learning community and bringing in other partners that are very interested in data, and evaluation, and other academic partners to participate, and using the Sacramento Promise Zone guide, and their learning community as an expansion to our leadership council.

So that's how the [inaudible], but we started very early on. In fact, the very -- within seven -- six months of our designation, we started -- our data and evaluation taskforce started to meet.

Janine Cuneo: Thanks, Erik. I want to thank all the speakers and also, all these questions that are coming in. Let's go ahead and move to the homework really quickly. So just a status check for all of you guys; right? We're on week three of a four-week series. For the homework assignments, again, the goal here is we're leading to a completed framework by the end of these sessions.

The first week was really the foundation. You guys were developing your theory of change and logic model and if you already had one done, reevaluating it. Week two, the overall design. You're asking your evaluation questions, evaluation methods, what are the resources you have available to you or that you still need to procure, recommending.

Some of you might be in different stages. So you're identifying the resources. It doesn't necessarily mean you had it and some of the timeframes you're thinking about, these evaluation questions to be answered then. Week three, this is where we are at today.

We're talking here about data collection and management throughout this session and we're really needing you guys to establish your indicators and decide how to get the data you need. Again, I can't reiterate enough what I started today's session at is that the goal here is to have a draft framework completed by the end of these sessions.

We recognize everyone's at a very different point. And so we really want to encourage you for those that had not started a lot of these thoughts or conversations yet, is to think of this as your best first attempt. It does not need to be perfect and a goal here, even in the evaluation framework, is that it's an iterative process.



34

So for those of you who started at ground zero, your first draft may have a lot of items that you've highlighted, things you're still thinking about, resources you still need to procure, etc. and that's going to be okay, but you've got to start getting something done on paper to move yourself forward; okay?

And so today's goal is -- this week's homework is to establish these indicators and decide how to get the data you need, [inaudible] the resources you either have and you can tap into or what you don't have and you need to procure, etc.

And then lastly, just to give you guys a little insight what's coming next in week four, that's when we're going to be talking about data analysis and reporting, how do you use that data for reporting and data visualization. So if you guys are keeping up, you're halfway through, this is great.

If you're not keeping up and are a little bit behind, the good news is we have about a three-week break right now until our next session. So you've got some time now to spend, carve out some time to start doing this work. As a reminder, we have provided for you guys some instruments that you may use. I'll give you the link in a second.

But one of it is an actual data and evaluation framework template we provided to you. And so we want to make sure that you guys note in that template, if you're so choosing to use it, you've already filled out a couple areas. Again, last week was the evaluation questions, which is the first column here in the left in evaluation design.

So let's assume you're at that point. If you're not, get there, guys. And then this week we want you to talk through and think about the indicators and the data sources. You know, we talked a lot about group A, B, D, we've shown you examples of those through the L.A. Promise Zone, for example.

Go back through the old examples we've used too and Sacramento, etc. and you can look at that for some help as well. So just as a reminder, here's where everything is located, the hudexchange.info program's Promise Zones' page and you'll both have homework instructions for week three and that data and evaluation framework template, we started using that in week two.

So feel free to grab that. If you haven't already started using that, feel free to. This is purely a template. You might -- some of you might already be using a different template, already at a different process at that point. So it is not required to use the template we're having. It is merely supposed to be a supportive tool for you.

I also want to remind you guys there's various references that we've used throughout this process. Elder has talked about the Promise Neighborhood Resource Consortium, we have the link here, also talked about some data usage agreement. UNC has guidance about that. You might want to look it up and analyze.



35

I want to make sure you guys see those. Lastly, I don't want to forget about what are some of the upcoming dates you want to make sure you have marked on your calendar? There's a next set of office hours that are on the 19th, 20th and 21st that you can sign up for.

And there's two to her sets of those office hours coming up after that. So you've got a lot of office hours. I really want to encourage you guys to take advantage, especially those that might be a bit behind in their homework. Take advantage of those in having some robust conversations around indicators on how you choose [inaudible] think about your resources.

There's also a peer convening meeting number 3 on the 20th from 3:00 to 4:00, get some useful tips from your colleagues and then webinar 4, which is our last webinar is on June 27th. So please make sure you mark your calendar and register for that. At this point and time, I just really want to thank you guys all for your participation, questions, etc.

I think we have a couple questions we were not able to get to today. We'll make sure we review those and if we didn't answer them during the presentation, we'll make sure we get to those and post those Q&As like we did with webinar two, we weren't able to get to all those questions.

And you'll notice if you go to the HUD Exchange, our framework for data and evaluation homepage, you will notice that there are questions from webinar two that we weren't able to get to posted there. We will do the same with this webinar as well. So thank you, everyone. Thank you, speakers.

And again, this is Janine Cuneo on behalf of Tanim and all of his HUD colleagues saying thank you and keep up the great work in getting these evaluation and frameworks done. Have a good evening.

(END)

developing a data and evaluation framework for urban promise zones - session 3 … ·...

Documents