adult population survey (aps) training session yana litovsky gem data team supervisor jonathan...

ADULT POPULATION SURVEY (APS) TRAINING SESSIONYana LitovskyGEM Data Team Supervisor

Jonathan Francis CarmonaGEM Data Manager

THIS SESSION WILL COVER:APS tools & guidelines• APS Questionnaire• Survey Design• Weights• Sample Size

Step in APS Cycle:• Proposal Submission• Data Collection• Data Review• Processed Results

the Adult Population Survey (APS) investigates the role of the individual in the lifecycle of the entrepreneurial process and his or her characteristics or actions

a representative sample of the adult population is used to estimate the entrepreneurial activity that represents the entire country.

The APS is unique because it measuresMotives AttitudesActions Business characteristics

SOME APS FACTS

GEM ENTREPRENEURIAL PHASES

APS QUESTIONNAIRE

APS QUESTIONNAIRE• The APS is comprised of MODULES of different groups of questions, often on different topics

• The modules are distributed within the questionnaire in different BLOCKS for different target respondents

• Introduction: All respondents• Block 1: Nascent entrepreneurs• Block 2: Owner-managers• Block 3: Potential and discontinuing• Block 4: Informal investors• Block 5: All respondents (occupation)• Block 6: Reserved (not used in 2014)• Block 7: All respondents (demographics)

APS BLOCKS

APS MODULES Core APS Module

Remain consistent year to year Used to derive main GEM indicators Must be included by all GEM National Teams

Special Topic Module Set of questions that change year to year On different research topics May be included in one or more blocks May include optional questions

Optional Modules Questions on different topics Suggested by national teams or derived from previous Special Topics Follow placement instructions

REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS

FILTER QUESTIONS

FOR ALL RESPONDENTS

BLOCK 1

QUALIFIED RESPONDENTS

INTRODUCTION

REQUIRED QUESTIONS FOR ENTIRE SAMPLE

OPTIONAL QUESTIONS FOR ENTIRE SAMPLE

BLOCK 1: NASCENT ENTREPRENEURS (FLOW)

UN-QUALIFIED RESPONDENTS

BLOCK 2

OPTIONAL QUESTIONS FOR NASCENT ENTREPRENEURS

SPECIAL TOPIC: REQUIRED QUESTIONS FOR NASCENT ENTREPRENEURS

REQUIRED QUESTIONS FOR OWNER MANAGERS

FILTER QUESTIONS

FOR ALL RESPONDENTS

BLOCK 2

QUALIFIED RESPONDENTS

BLOCK 2: OWNER-MANAGERS (FLOW)

UN-QUALIFIED RESPONDENTS

BLOCK 3

OPTIONAL QUESTIONS FOR OWNER MANAGERS

SPECIAL TOPIC: REQUIRED QUESTIONS FOR OWNER MANAGERS

BLOCK 3

BLOCK 4

BLOCKS 3 & 4

REQUIRED QUESTIONS FOR POTENTIAL & DISCONTINUING

OPTIONAL QUESTIONS FOR POTENTIAL & DISCONTINUING

FILTER QUESTIONS

FOR ALL RESPONDENTS

QUALIFIED RESPONDENTS

UN-QUALIFIED RESPONDENTS

REQUIRED QUESTIONS FOR INFORMAL INVESTORS

OPTIONAL QUESTIONS FOR INFORMAL INVESTORS

BLOCK 5

BLOCK 7

BLOCKS 5 & 7

REQUIRED QUESTIONS FOR ALL (OCCUPATION)

OPTIONAL QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)

REQUIRED QUESTIONS FOR ALL (DEMOGRAPHICS)

REQUIRED VENDOR PROVIDED INFORMATOIN

OPTIONAL QUESTIONS FOR ALL

SPECIAL TOPIC: REQUIRED QUESTIONS FOR ALL (OR BASED ON OCCUPATION STATUS)

• Full questionnaire, with codes and skip patterns.

APS QUESTIONNAIRE FORMAT

APS QUESTIONNAIRE FORMAT• Each block is color-coded

SURVEY DESIGN

• There is no one standard GEM methodology!

• GEM’s only fixed requirement is that a team collects a nationally representative sample of at least 2000 adults

• To ensure this, GEM provides guidelines about:• Proper choice of respondent contact method• Sufficient number of callbacks and initial contact attempts• Appropriate time of survey administration• Appropriate stratification of sample

Some guidelines stable but others differ from team to team

SURVEY DESIGN GUIDELINES

• Sample can be using any combination of:• Face-to-face • Fixed line• Mobile phone

• Choice based on feasibility, budget & bias

• Each methodology can have its own bias, for example:• Fixed-line usually biased toward older respondents• Mobile phone usually biased toward younger respondents • Face-to-face may be biased against people often out of the

house

CONTACT METHOD OPTIONS

• Most teams collect a sample that is proportional to the national population distribution

• Others may add an oversample: may collect disproportionately more responses in a metropolitan area

• Required to meet the n=2000 minimum not counting any oversamples.

• Oversample may be part of Global Report sample or submitted separately

SAMPLE DESIGN OPTIONS

WEIGHTS

• All analysis of GEM APS data MUST use weights.

• National weight mean =1 (do not account for different sample sizes).

• Weights differ by the type and age/range of sample.

• Weighted age/gender distribution must be nationally representative.

• Strata-based weights must match overall strata sizes.

• Very large values for weights (high leverage cases) are not allowed.

WEIGHTS

• RESPONSE WEIGHTS: Used to achieve a better match of the sample surveyed to that of the overall population.• To reduce potential bias due to different responses rates or

normal sampling variability.• Required for every country.

• SAMPLE DESIGN WEIGHTS: Used to adjust a sample that was purposefully designed to have unequal representation• For example, if a team oversamples (by design) in a

metropolitan area, weights are used to not over-represent the city respondents.

TYPES OF WEIGHTS

• WEIGHT_L• Used for the adult population aged 18 to 64 (L = Labor

Force).

• WEIGHT_A• Used for the entire adult population aged 18 to 99 (A

= Adults)

• If national team only collected data for ages 18 to 64, then Weight_L and Weight_A are identical.

GEM NATIONAL APS WEIGHTS

• Only provided for teams who request regional-level analysis.• WEIGHT_LR• WEIGHT_AR

• Each region has its own weights calculated as if they were the entire nation.

• Should be used ONLY for region-to-region comparisons, not for any national-level analysis!

GEM REGIONAL APS WEIGHTS

• Always use weights for any GEM APS individual-level analysis.

• If you want to find comparable Global Report results, use WEIGHT_L

FINAL LESSON

SAMPLE SIZE

SAMPLE SIZES (2013)

COUNTRY SAMPLE SIZE

USA 5698Russia 2029South Africa 3450Greece 2000Netherlands 3005Belgium 2001France 2002Spain 24600Hungary 2000Italy 2052Romania 2021Switzerland 2003United Kingdom 11017

Minimum sampleN = 2000

Appropriate regardless of national population size

RESPONDENTS PER BLOCK (FRACTION OF SAMPLE)

 Block 1: Nascent

Entrepreneurs

Block 2: Owner-

Managers

Block 3: Potential and Discontinuing Entrepreneurs

Block 4: Informal Investors

USA 8.8% 10.1% 3.7% 4.4%

Norway 1.7% 11.2% 1.6% 3.2%

Japan 3.0% 7.9% 1.5% 1.2%

Russia 3.7% 6.4% 1.6% 2.0%

South Africa 6.1% 5.7% 4.0% 1.5%

Argentina 10.8% 12.5% 4.9% 3.6%

RESPONDENTS PER BLOCK (NUMBER IN 2000 PERSON SAMPLES)

 Block 1: Nascent

Entrepreneurs

Block 2: Owner-

Managers

Block 3: Potential and Discontinuing Entrepreneurs

Block 4: Informal Investors

USA 500 578 209 251

Norway 34 224 32 64

Japan 60 158 30 24

Russia 76 129 33 41

South Africa 210 195 137 52

Argentina 238 276 107 80

• But sample sizes vary by nation and year.

• Beware when combining data from nations or years with different sample sizes.

• Weights do NOT adjust for different sized samples

• The weighted total for any national analysis will match the unweighted total (that is why weights have a mean of 1).

SAMPLE SIZE:COMBINING INDIVIDUAL LEVEL DATA

PROPOSAL SUBMISSION PROCESS

RFP PACKAGE• Request for Proposal (RFP) is posted to the GEM consortium website and an announcement is emailed to all teams in February of each year.

• The RFP includes all documents needed by National Team to submit APS/NES proposal, administer the questionnaires, record and submit all data. It also includes valuable reference information.

“RFP” refers to the entire document package as well as the Request for Proposal document itself.

RFP PACKAGE (CONTENTS)Data Delivery Documents:

SPSS Data Input Templates (APS & NES)

Proposal Documents: Instructions for submitting proposals (APS & NES) Survey Reports (APS & NES) Additional NES instructional documents

Questionnaire Documents: Final version of questionnaires (APS & NES; English & Spanish) APS question-by-question commentary

Reference Documents: PowerPoint Training presentation (APS & NES) GEM Timeline Reference Information (additional APS questions, education categories, optional module)

Use provided SPSS Data Input Template to submit final survey data.

Data in any other format will not be processed

RFP PACKAGEDATA DELIVERY: SPSS DATA INPUT FILE

Detailed description of each question

RFP PACKAGEQUESTIONNAIRE: APS COMMENTARY

~ Excel document comprised of several worksheets

~ Unique for each team

Methodology OverviewPrevious year’s Data Quality ReportAPS Questionnaire informationSample strata description & fieldwork reportPopulation statisticsWeight constructionEducation categories

Additional worksheets populated during APS cycle

RFP PACKAGEPROPOSAL: APS SURVEY REPORT

Worksheet A: Methodology Overview

Provide full details of sample design (usually completed by vendor)

Information included in Global Report

RFP PACKAGEPROPOSAL: APS SURVEY REPORT

- Sample size- Interview method- Sampling methodology- Sample stratification- Call backs- Previous data issues

Worksheet C: Strata & Fieldwork Report

List and describe sample strata (if none used, national information)

Complete fieldwork report for each strata after data is collected

RFP PACKAGEPROPOSAL: APS SURVEY REPORT

PREPARING AN APS PROPOSAL

• Download and carefully read the RFP package

• Work with Survey Vendor to decide on appropriate survey methodology and fill out all required RFP documents.• Vendors must use GEM forms except for Vendor Proposal (has no

template)

• Customize your questionnaire and SPSS Data Input Template• Decide which optional modules you will include (if any)• Decide if you will add additional questions (must be approved by GEM)• Establish you demographic variable answer categories• Translate the questionnaire if necessary

Final questionnaire is each team’s responsibility

TIPS FOR TRANSLATING APS QUESTIONNAIRE

• Work with vendor on translations

• National team has more knowledge of question purpose• Vendor can advise how questions will be received by

respondent• Use question-by-question commentary• Always do a back-translation• Can share other team’s translations with permission

TIPS FOR EDUCATION CATEGORIES

~ Team creates own categories for education variable ~ GEM recodes categories into harmonized variables

Keep our recoding rules in mind when constructing your education variable

- Respondent must complete an education level (e.g. earn a degree) to be coded in that category

- Bachelors, Masters and Doctorate degrees are best kept in separate categories

TIPS FOR INCOME CATEGORIES

- Team creates their own categories for income variable

- GEM requires at least 6 income ranges

- GEM recodes variable into lower, middle, upper percentile

Keep national income distribution in mind! If most respondents fall into one income range, this variable is not useful.

- Email the completed forms to data@gemconsortium.org (refer to RFP for instructions on how to label documents).

- The GEM Data team will review submitted documents and proposed sampling methodology and send National Team a review in a new worksheet added to Survey Report document (G. Proposal Review).

- The national team MAY NOT BEGIN APS survey until their proposal has been officially approved (via email) by GEM Data Team.

SUBMITTING AN APS PROPOSAL

EVALUATION OF APS PROPOSAL

• Proposal document review: has all the required material been submitted?

• Every mandatory chart applicable to your proposal must be completed• All information must be submitted in required format

• APS Questionnaire review: do team questions interfere with core APS?• Content and placement of any additional APS questions must be approved by Data Team• Verbal introduction of the survey to a potential respondent are reviewed (see RFP tips)

• Methodology review: does sample/survey design meet GEM standards?• Is sample stratification appropriate?• Is the timing of survey administration appropriate?• Is a sufficient number of call-backs and initial contact attempts proposed?• Have all data issues from previous years been addressed?

EVALUATION OF APS PROPOSAL

If necessary, the National Team will be requested to:

submit any missing informationclarify specifics of the sample designrevise sampling and survey plan

DO NOT WAIT FOR PROPOSAL TO BE COMPLETE BEFORE GETTING IN TOUCH WITH

GEM DATA TEAM IF YOU HAVE QUESTIONS

~ We can give suggestions about choice of vendor~ Pre-evaluate different methodologies~ Help team conduct a pilot test

DATA COLLECTION PROCESS

BEFORE APS DATA COLLECTION

• Mentor Team: New teams are recommended to find a find a mentor GEM country. Contact other GEM teams with same cultural background, especially if they have the same language.

BEFORE APS DATA COLLECTION• Pilot Study: New team or teams with new vendor will be

required to conduct pilot study with a minimum of 100 respondents and submit results before continuing data collection.

• Results (submitted using APS Data Input Template) will be tested for:

• proper skip pattern• correct variable coding• proper use of data templates• no excessive missing data

• Any problems will be quickly communicated• If no problems, pilot respondents can be included in final sample. • A team must guarantee that any will be addressed and corrected

DURING APS DATA COLLECTION: HOW TO CHECK YOUR DATA

• Interim Data Submission: If Pilot Study is not required, teams are still encouraged to submit interim data file to be checked for data format, skip logic or other errors.

• Data Quality Syntax: A national team can use SPSS syntax to check their data before submitting to GEM. These files replicate most GEM data checks.

DURING APS DATA COLLECTION: ERRORS TO CHECK AND AVOID

① Skip logic errors due to incorrect questionnaire programing or poorly trained interviewers: If these result in excessive missing data, team may be asked to recontact participants (if possible) or resample, in order for the data to be approved and included in the Global Report (extra data is also a problem)

DURING APS DATA COLLECTION: ERRORS TO CHECK AND AVOID

② Unrepresentative sample due to incorrectly implemented sampling procedure: If final sample does not reflect the age/gender distribution of country within an acceptable range, team’s data may not be acceptable.

RECORDING APS DATASome basic guidelines:

• All recorded data must correspond to a designated answer category

• Each respondent ID is unique and no greater than 9 digits• Do not delete variables from SPSS template• Do not use any digits or symbols to indicate a blank cell

(e.g. “0”)• Add value labels for demographics with team-created

categories

... see all guidelines in RFP and Data Quality Syntax

OPEN-ENDED TEXT RESPONSES

What kind of business is this?

• Response must describe:• The nature of the product or service as well as• The primary customer base: other businesses or typical

consumers• The detailed sector of activity

• Common businesses need little elaboration:• Restaurant, homebuilder, day care center, hair salon, etc.

COMMON PROBLEMATIC RESPONSES

Unacceptable Response Problem and Solution

Apparel Specific sector needed. Can be apparel manufacturing, clothing store, fashion design.

Electrical Appliances Specific sector needed. Can be manufacture electrical appliances, wholesale electrical appliances, retail electrical appliances, repair electrical appliances, etc.

Machines Must probe to know type of machinery and specific sector such as manufacturing, wholesale, retail, repair or rental.

Services Must probe for more information. Services can be just about anything.

Telephones Specific sector needed. Can be selling mobile phones, repairing phones or working for a call center.

Trade Probe for more specific information. Can be anything; construction (building, plumbing, electrician), manufacturing, wholesale or retail.

VERIFY SURVEY REPORT INFORMATIONConfirm that all Survey Report information submitted before data collection began is still correct:

Education categories are the identical in Survey Report and dataset Population data used to calculate weights same as provided in Survey Report Strata listed in Survey Report have the same names, value labels order Correct weight formula was indicated

If any changes must be made to Survey Report, make the necessary revisions and alert GEM Data Team when revised document is submitted.

SUBMITTING APS DATASET- Submit APS Data in SPSS template (follow document naming

conventions indicated in RFP)

- Complete Strata & Fieldwork Report in Survey Report and submit with APS Data

- Number of answered calls/doors - Number of eligible households contacted- Number of interviews agreed to...

- Send both documents to data@gemconsortium.org

DATA REVIEW PROCESS

INITIAL REVIEW OF APS DATA

Initial data checks

1. Data is correctly recorded and formatted

2. Unique, numeric IDs value for each respondent

3. Proper variable and value labels are used

4. All mandatory questions have been asked

Weights

1. Representative of age/gender distribution in country and, if sample is stratified, representative of strata distribution.

2. No excessive number of high value weights

Open-ended responses

1. Translated into English

2. Both native and English language responses provided in appropriate variables.

Skip logic checks

1. No excessive missing data

2. If data missing, how critical are the errors?

Example of APS Data Review Issues (worksheet H. Data Review, in Survey Report)

DATA REVIEW (SENT TO NATIONAL TEAM)

• Comprehensive review of data quality and team performance

• Included in new worksheet in Survey Report (I. Data Quality Report)

• Includes:• % missing for key variables• % refused for key variables• % skip logic errors• Key indicators for past 3 years• Sample distribution• Deadlines• Exchange rate• Proposal/APS Issues

DATA QUALITY REPORT

DATA QUALITY REPORT

APS PROCESSED RESULTS

APS DATA PROCESSING LEVELS

Submitted Raw Data

HarmonizedData

Processed Individual Level Data

Processed National Level Data

Data submitted by National Team

Reviewed, cleaned, standardized data ready to be merged with all other countries for processing

APS Data with all computed APS indices (TEA rates, Established Business rates, etc.)

APS index rates computed at the national level and weighted by weight_L (for 18-64 population)

COMPUTING APS MEASURES

• Data Team uses cleaned, harmonized and merged datasets to compute indices from core APS data.

• Some individual level data variable names and variables differ from merged national level data.

• Variable codebooks for both individual and national level data made available with processed APS results.

• National level results (weighted by WEIGHT_L) are reported in the Global Report.

TEA CREATION FLOWCHART

During & 1 year after data Collection

1 year after Global Report Launch

Processed individual level data each country Each GEM National Team

All GEM Teams

Processed individual level data all countries

GEM Data, Special Topic Teams,Unilateral Sharing

WHAT DATA IS AVAILABLE?

FebruaryRFP released

mid-MarchNES RFP due

late-AprilAPS RFP (Survey Report and Survey Vendor Proposal) due

mid-JulyAPS & NES data and documentation (including Fieldwork Report) due

mid to late-SeptemberFirst release of APS & NES master dataset on website

mid to late-OctoberSecond release of APS master dataset uploaded to website

TIMELINE

TIMELINESSSUBMIT ALL REQUIRED

MATERIALS ON TIME

DATA QUALITYENSURE THAT YOU FOLLOW ALL

GEM STANDARDS AND PROCEDURES

COMMUNICATIONSTAY IN TOUCH WITH DATA TEAM AT DATA@GEMCONSORTIUM.ORG