A System for Scanner Data

Muhanad Sammar, Anders Norberg & Can Tongur

Statistics Sweden

A System for Scanner Data

• Several years of wanting and canvassing of retail chains paid off very well

• We now have access to vast amounts of data• We receive text files every week• Files contain weekly sales: • sold quantity

• price (w/o VAT)

• EAN

• store name etc.

• Transmission is safe (over FTP)

The Scanner Data Files

• …and files look like this

Market Postal Code YearWeek EAN Brand Quantity Price

ABC 14550 201220 730202… Y-Brand 43 12.90

ABC 14550 201220 7314805… X-Brand 2 8.90

DEF 75424 201220 7310265… X-Brand 174 32.90

Building the SystemWe wanted to incorporate scanner data:• Either by extending the existing production environment, or

creating a new environment• Pity not to use the existing User Interface• Perhaps not so simple to use the regular engine• SAS® seemed extremely attractive & tempting to use…

The Existing System

• The ordinary user interface Pi09 (π09)• Tables in SQL, interface and system in VB.Net

Adding SAS

• We made it simple and added a scroll-down menu• The idea behind this (SAS↔VB.Net) already

established in the house

The Production Flow

Production Initiation

• Initially, an EAN – Product code register must be established for extraction of data

The Monthly Data Extraction Register

Product Code EAN Year Month Product Name Brand

112517 7331010065622 2012 3 BULGUR 1KG DRUVAN

112710 8076804215058 2012 3 SPAGHETTI BARILLA

112737 7310960039211 2012 3 LASAGNE 1 KG FAMILJEN DAFGÅRD

123243 7300208345006 2012 3 BACON SCAN123312

4 5701251501524 2012 3GOL PÖLSER RÖDA HOT DOG PÖLSER 375 G GOL PÖLSER

124079 7313940574504 2012 3 BUFFALO WINGS 500G GULDFÅGELN

922449 4015400347453 2012 3 ALWAYS ULTRA NORMAL PLUS ALWAYS

The Editing Interface

Validation and Selection• Given data – Validate

Perform trivial but crucial checks, e.g.:• Numerics are numeric• Check shop names & that all are there• Check that the period in data (YYYYWW) actually is

the desired week• Given Validity and a Register – Select data

• Revalidate and perform editing and subtraction of deposits etc.

If necessary – use the new interface to make changes/delete prices

Aggregation and Uploading

• Aggregate data before uploading it• Aggregation by geometric mean over 3 weeks• Make a final register check (=mirror) of

EAN – Product in the S.D. sample v.s. the (updated) designated sample

• Upload data to the ordinary system• Perform a market analysis of EAN codes

• Necessary for maintenance and keeping S.D. register up to date

• Pragmatic approach – use MS. Excel

Market Analysis

Market Analysis

Product Code EAN Product Name Brand # stores # packages

112517 733101006.. BULGUR 1KG X 17 876

112710 807680421.. SPAGHETTI Y 4 12

112737 731096003.. LASAGNE 1 KG Z 20 1347

Summing-up

• Note merely SAS but also SQL table views with ”the right stuff” required

• SAS part not so tricky – system integration harder• Ideas have to mature• It works!