essnet “preparation of standardisation”ec.europa.eu/eurostat/cros/system/files/wp1 report... ·...

Essnet “Preparation of standardisation”

Work package 1

Statistical methodological standards

- Review of current international and methodological handbooks and guidelines -

Draft reportVersion 12 August 2011

1. Definition of a statistical standard and of guidance.........................................................................31.1. Introduction...............................................................................................................................31.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics............................................................................................................................................3

1.2.1.1. Method used....................................................................................................................31.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues......................................................................................................................51.2.3. The current use of the words standard and standardisation is not fully consistent..........10

1.3. Taking stock of standardisation in other domains...................................................................111.3.1. Definition of a standard....................................................................................................111.3.2. How should this proposal be understood in the ESSnet STAND-PREP ?.......................12

2. Review and analysis of six methodological manuals.....................................................................162.1. A framework for the analysis..................................................................................................16

2.1.1. Conceptual scheme of the repository for the standards....................................................172.1.2. The IT tool for the management of the repository...........................................................232.1.3. Procedure followed for the definition and refinement of repositories for single handbooks and for the construction of the unified repository....................................................26

2.2. Analysis of the six methodological handbooks.......................................................................272.2.1. Eurostat sampling references guidelines..........................................................................282.2.2. Recommended practices for editing and imputation in cross-sectional business surveys (EDIMBUS)...............................................................................................................................322.2.3. Guidelines for statistical metadata on the Internet...........................................................412.2.4. ESS hHandbook for qQuality rReports............................................................................452.2.5. Recommended practices for questionnaire design and testing.........................................522.2.6. Survey mMethods and pPractices....................................................................................57

2.3. Description of the unified repository for methodological standards.......................................652.3.1. General description of the repository...............................................................................65

Schmidt-P, 2011-08-29,

The official title of WP1 should be mentioned at least.

2.3.2. Examples of compared analyses of standards derived from different handbooks for the same GSBPM sub-processes......................................................................................................72

3. Review of support measures.......................................................................................................82793.1. Introduction.........................................................................................................................82793.2. Support measures related to standards................................................................................82793.3. Overview of support measures............................................................................................8380

3.3.1. Motivation for implementation....................................................................................83803.3.2. Making the implementation easy.................................................................................84813.3.3. Supporting right implementation..................................................................................8683

3.4. Summary.............................................................................................................................87844. Quality of statistical standards and criteria for the assessment..................................................8885

4.1. Introduction.........................................................................................................................88854.2. The concept of quality of standards....................................................................................8885

4.2.1. Applicability.................................................................................................................89864.2.2. Effectiveness................................................................................................................90874.2.3. Efficiency.....................................................................................................................9087

4.3. Relations between the proposed quality criteria on methodological standards...................91884.4. Main problems connected with standards that may have quality character........................9188

5. Conformity to standards.............................................................................................................93905.1. Definition.............................................................................................................................93905.2. Conformity assessment is considered during the development of standards......................93905.3. Standards in the EU and EU conformity assessment..........................................................9592

5.3.1. Framework for EU standardisation..............................................................................95925.3.2. Modules for conformity assessment.............................................................................9693

5.4. How to assess conformity to methodological standards in statistics?...............................100975.4.1. Problem statement in the ESS....................................................................................100975.4.2. Proposal for conformity assessment...........................................................................10299

Annex 1 - Extracts referring to standards with another meaning...............................................105102Annex 2 – Review of literature on standards.............................................................................107104

2

1. Definition of a statistical standard and of guidance

1.1. Introduction

The definition of a standard in the ESSnet STAND PREP is a concern that is felt to be important by all participants. This issue has been tackled several times (even when preparing the answer to the Eurostat “call for interest”!). This is an issue that deserves serious consideration both for this ESSnet but very probably as well for further discussions in the sponsorship and in the ESSC and PG that will discuss the reports of that sponsorship.

This kind of work is not exceptional in the ESS: for instance, the statistical definition of “enterprise” was a lasting concern for several years in Business statistics. The definition of “quality” required some work as well. For these two examples, there were some pre-existing references in other domains or in other statistical offices that helped settle the debate relatively quickly. For the definition of “enterprise”, the existing definitions in national accounts and accounting standards helped a lot. As for the definition of quality in statistics, papers from the US Bureau of the Census, Statistics Canada and Statistics Sweden provided very useful references

In the case of standards, the ESSnet has taken stock of the literature on the subject. Whereas there are many references on standards in domains other than statistics, we did not find references in statistical organisations that could help to solve the issue quickly.

In this document, we refer to the practice in existing statistical texts using the word of “standard” in order to define the underlying agreed concept for statisticians. This method is used below.

However, since the previous method does not bring useful results, we have then used the ISO vocabulary to propose a definition for the concept of “standard” and the corresponding definition of standardisation.

1.2. Current use of the words “standard” (noun or adjective) and standardise/ standardisation in statistics

1 2.21.1. Method used

“Standards” are often referred to by statisticians, with different meanings. This is largely due to the fact that “standard” has several meanings in English. We should therefore select for our purpose only texts in which the meaning of “standard” used by the author refers to standards that are the result of the standardisation activity.

In short, standardisation consists in developing standards.

The definition of standardize given by the Merriam-Webster dictionary is:

1: to reduce to or compare with a standard <standardize a solution>

3

Bode-O, 2011-08-29,

Footnote 2 from page 4 should be put here

Schmidt-P, 2011-08-29,

Does this chapter contain a definition of guidance?

2: to bring into conformity with a standard 3: to arrange or order the component items of a test (as of intelligence or personality) so that the probability of their eliciting a designated class of response varies with some quantifiable psychological or behavioral attribute, function, or characteristic

The last meaning is corresponds to some technical work in statistics, and is not relevant here.

The first two meanings of standardise refer to the word standard with the same underlying meaning of this noun. As a first approximation, such standards would be “a set of rules to be applied”.

We have to take into account the various meanings of “standard” in English, in order to keep for our analysis only the texts that refer to the meaning of standard we are interested in.

Whereas it might be easier to understand for native English speakers, it can be tricky for non native ones.

If we refer to the Cambridge dictionary Online1, the definition of standard is:

1 - meaning “usual”:o a pattern or model that is generally accepted. ex:

This program is an industry standard for computers.o a song or other piece of music which has been popular and often played over a long period

of time2- in the context of quality:

o a level of quality - ex:This essay is not of an acceptable standard - do it again.This piece of work is below standard/is not up to standard.We have very high safety standards in this laboratory.Not everyone judges success by the same standards - some people think happiness is more important than money.Her technique became a standard against which all future methods were compared.

o a moral rule which should be obeyed - ex:Most people agree that there are standards (of behaviour) which need to be upheld, but agreeing on them is rather more difficult.

The definition given by the Merriam-Webster dictionary 2 (“definition for English learners”) is:

1 a : a level of quality, achievement, etc., that is considered acceptable or desirable [count] ▪ high/low standards of quality ▪ His work is not up to our standards. [=is not as good as we require it to be] ▪ By modern/today's standards, the house is just too small. ▪ industry standards ▪ Their standards are slipping. [noncount] ▪ His work is not up to standard. = His work is below standard. [=his work is not good enough] — see also double standard, standard of living b standards [plural] : ideas about morally correct and acceptable behavior ▪ She has high moral standards. [=principles] 2 [count] : something that is very good and that is used to make judgments about the quality of other things ▪ This book is the standard by which all others must be judged. 3 [count] : a fixed official unit of measurement ▪ a standard of weight — see also gold standard 1 British English2 American English

4

http://www.learnersdictionary.com/search/gold%20standard

http://www.learnersdictionary.com/search/standard%20of%20living

http://www.learnersdictionary.com/search/double%20standard

4 [count] : a flag that is used in official ceremonies 5 [count] : a song that has been sung by many different artists ▪ an old standard

The meaning we are interested in is the first meaning listed by the Cambridge dictionary and in the Webster dictionary (reference to “industry standards”).

After taking into account these possible meanings of the word “standard”, we have rejected some references in the statistical literature using the word “standard” (see Annex) and kept the ones below. For each of these, we derive “clues” for the definition of the concept of “standard” that is implicit in the references below. These clues are shaded in light grey.

1.2.2. Some statistical references to standards as “sets of rules to be applied” and the corresponding clues

1.2.2.1) In the report of the “Canadian Advisory panel on Telephone public opinion survey quality” (February 2007):Standards are defined as “Practices that should be requirements for all telephone studies conducted by the Government of Canada ».

Clue 1:We deduce from this definition that:

o there are several statistical standards for just the statistical activity of “telephone opinion survey”. The paper lists a series of such standards. There are more or less as many standards as steps in the GSBPM. These sentences are made of a few sentences (generally 5 to 6).Example: for data collection dealing with call-backs, the standard is defined as:

“Unless circumstances strongly indicate a different number of call-back attempts and scheduling of call-backs is required, there will be a minimum of eight call-backs made, varying the day and time of each call.”

o these standards are requirements, i.e. compulsory rules that have to be applied.

1.2.2.2) In the “ESS code of practice” (CoP), the word “standard” is used several times.

1.2.2.2.1: It is stated that: “The head of the statistical authority and, where appropriate, the heads of its statistical bodies have the sole responsibility for deciding on statistical methods, standards and procedures”.

Clue 2:This would suggest that standards differ from statistical methods (which refer to academic knowledge) and from procedures (which refer to ad hoc organisations of the production process).

1.2.2.2.2: It is also stated that “European and other international standards, guidelines and good practices must be fully observed in the processes used by the statistical authorities to organise, collect, process and disseminate official statistics.”

Clue 3: This would suggest that standards are of a more binding nature than guidelines and good practices. However, since the CoP requires the guidelines and good practice “fully observed” leads to some confusion: what is then the difference between a standard and a guideline?

5

Bode-O, 2011-08-29,

You don´t find 1.2.2.1 or 1.2.2.2 and so on in the table of contents, so it might be better to cancel it here (on the pages 5-9)

Standards may be relevant for the following activities in NSIs: organisation, data collection, processing, dissemination.

1.2.2.2.3: The overall methodological framework of the statistical authority follows European and other international standards, guidelines, and good practices.

Clue 4: Standards may define the methodological framework. There are international standards, international guidelines and international good practices that refer to three different types of entities.

1.2.2.2.4: Procedures are in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical authority.

Clue 5: there is a conformity assessment of standards inside the NSI.

1.2.2.2.5: Revisions follow standard, well-established and transparent procedures.[…]A standard daily time is set for the release of European Statistics.

Clue 6: standards are defined for procedures.

1.2.2.2.6: Statistics are compiled on the basis of common standards with respect to scope, definitions, units and classifications in the different surveys and sources.

Clue 7: standards include some aspects of the definition of statistics like scope, definition (of characteristics to be collected?), statistical units. Standards are defined as well for classifications…These standards have to be consistent (“common”) for the various statistics and classifications.

1.2.2.2.7: Metadata are documented according to standardised metadata systems.

Clue 8: the products than can be standardised include whole systems, for instance metadata system.

1.2.2.3) In the draft regulation on ESA 2010, it is written that:

The objective of the Regulation is to revise ESA both concerning the methodology on common standards, definitions, classifications and accounting rules, and the programme for transmitting for Union purposes accounts and tables compiled according to the methodology.

Clue 9: The ESA includes a methodology (= a set of methods) dealing with standards. These standards have to be distinguished from definitions, classifications, accounting rules.

Article 1 states that: ESA 2010 provides for:(a) a methodology (Annex A) on common standards, definitions, classifications and accounting rules that shall be used for compiling accounts and tables on comparable bases for the purposes of the Union, together with results as required by Article 3;

Articles 3 states that “Member States shall transmit to the Commission (Eurostat) the data and metadata required by this Regulation in accordance with an interchange standard specified by the Commission (Eurostat).”

Clue 10: Standards in statistics will be available for the activity “interchange of data and metadata”.

6

Paragraph 1.21 states that: The concepts are internationally compatible because: a) The concepts in the ESA are consistent with those in the worldwide guidelines on national accounting, i.e. the SNA;b) For the EU Member States, the ESA is the standard for submitting national accounts data to all international organisations;

Clue 11: The whole ESA is considered as one single standard.

It is stated as well that :

“The main concepts in the ESA are well-established and fixed over a long period, because: a) They have been approved as the international standard for many years;

Clue 13: The set of all concepts of the ESA (but not the other rules/ statements) is in itself one single standard.

1.2.2.4) The November 2010 conclusion of the Ecofin Council states that:

The Council CONSIDERS that it is desirable - for administrative purposes – to refer to the agreed set of national accounts standards in the EU when the ESA is revised, in order to reinforce quality, enhance clarity and transparency of communication vis-à-vis the authorities (EFC, Finance Ministers) and other users, and increase efficiency, while reducing burden.

Clue 14: The Esa is not one single standard, but a set of standards.

1.2.2.5) The regulation 223/2009 uses as well the term “standards”:

For the purposes of this Regulation, the following definitions shall apply:1- statistics means […]2- ‘development’ means the activities aiming at setting up, strengthening and improving the statistical methods, standards and procedures used for the production and dissemination of statistics as well as at designing new statistics and indicators;

Clue 15: standards may be produced during the development phase in NSIs.

Please note that in this regulation, “The development, production and dissemination of European statistics shall take into account international recommendations and best practice.

Clue 16: standards set up by the ESS take into account international recommendations.

To guarantee the quality of results, European statistics shall be developed, produced and disseminated on the basis of uniform standards and of harmonised methods.

Clue 17: Standards should be uniform.

1.2.2.6) In the “Communication of the Commission on the vision for the next decade” (404/2009) 3 :

3 In the French version, the word « norme » is always used, referring to the ISO-types standards. In the German version, the word « Norme », but also « Standard/ Standardisierung » are used. The two latters do not belong to the ISO official vocabulary, but seem to refer to the same objective.

7

Bode-O, 2011-08-29,

Clue 12 is missing

In order to guarantee comparability and consistency among the data of all Member States, the output from NSIs is harmonised according to agreed standards.

Clue 18: Since Eurostat describes here the current situation in the ESS (output harmonisation), we assume that such standards include “definitions of statistics” and “methodological handbooks”.

The stovepipe model […] Last but not least, this way of production is highly inefficient and costly, as it does not make use of standardisation between areas and collaboration between Member States.

Clue 19: Logically speaking, this means that standardisation between areas leads automatically to efficiency and diminution of cost.

At the level of the NSIs, statistics for specific domains are then no longer produced independently from each other; instead they are produced as integrated parts of comprehensive production systems (the so-called data warehouse approach) for clusters of statistics. These systems would be based on a common (technical) infrastructure, they would apply as far as possible standardised software, and they would make use of all available data sources which are appropriate in quality.

Clue 20: Software can be standardised.

There are considerable challenges of a technical and methodological nature. Standardisation and integration of formerly separated production processes will demand great efforts and an effective change management.

Clue 20: The production process of NSIs has to be standardised, i.e. standards on various aspects of the production process have to be available.

1.2.2.7) In the Communication on priority setting in statistics:

“Eurostat is also sponsoring, together with other international organisations, development and promotion of the Statistical Data and Metadata eXchange format (SDMX), which has the status of an ISO standard. This XML-based standard permits electronic transfers of statistical data and metadata between organisations and to users. SDMX standards will help enterprises to have a return on the information that they provide to statistical authorities.”

Clue 21: “standard” has the same meaning as in ISO texts.

Direct use of standardised accounting data using electronic tools is another promising way forward. The data are received directly from the enterprise accounts without a special questionnaire. The accounting data can be supplemented by small surveys as in the case of administrative data. The Commission will support Member States by aligning the definitions in the legal acts on those in accounting standards, by developing the necessary electronic tools, and by devising methods for targeted complementary surveys and corresponding estimation techniques.

Clue 22: Since accounting standards are defined by standard setting organisations, the word standard is used in the specialised meaning.

1.2.2.8) The Communication of the Commission on the robustness of statistics (211/2011):

8

Bode-O, 2011-08-29,

Clue 20 appears two times

Recent experience has shown that it is as important to assess the quality of upstream data as it is to continue assessing the quality of the translation of this data into ESA standards. Preventive measures, such as promoting further standardisation in public accounts, and a wider proactive management of risk assessment are needed, including assessing the adequacy of the quality of upstream data for compiling statistics in accordance with ESA rules.

Clue 23: ESA is referred to as a set of several standards. Definitions are classified under standards.

For instance, a set of quality management standards tailored to the specificities of the EDP compilation process could be agreed, implemented by NSIs and certified by the Commission;

Clue 24: Standards can be certified.

1.2.2.9) Invitation to the ESSnet Preparation of Standardisation: (letter MG 447320 from 22 August 2010)

1 - […]: to increase the efficiency and quality of statistical production by further standardisation of processes and methods.

Clue 25: standardisation increases efficiency and quality of statistical production

2- prepare the work of the sponsorship. The ESSnet has to deliver the main results: criteria for the adoption of methodological handbooks as standards classification of standards description of the procedure for adoption and maintenance.

Clue 26: existing methodological handbooks are not considered standards by Eurostat. Adoption of a standard is a formal act, defined by a procedure.

1.2.2.10) Annex I of the call for interest for the ESSnet Preparation of Standardisation:

Standardisation requires the contribution of different actors in the statistical process: statistics production, IT development, methodology, metadata and management.

Clue 27: standards are envisaged for methodology, statistics production, IT development.

The project aims to prepare standardisation in two ways: by reviewing methodological standards embedded in handbooks and by systemising other standards. The difference is not only in subject (handbooks and other standards), but also in angle: the first work package concerns the standards itself, whereas in the second work package the focus is on the process of standardisation.

Clue 28: methodological standards are expected to be either the methodological manual themselves or part of it. This is not very clear.

Although the scope of the standardisation initiative still has to be defined and the proposed action preparing standardisation serves to help define and clarify the scope, it is clear that it concerns statistical standardisation (and not for instance IT or communication standards).

9

Clue 29: statistical standards would be analogous to standards in IT or Communication, i.e. ISO/ IEC4 standards

Work package 1 and 2 will in parallel generate some ideas on future work on standardisation; e.g. the registration of standards, coherence checks, standardisation versus flexibility and development of methods, levels of standardisation and priorities for standardisation.

“Describe procedures for adoption and maintenance of the standards”.

The review document consists of an advice on standardising (parts of) the handbook and of suggestions on necessary changes before standardisation; it also includes a reflection on the current adoption of the standard and on potential support measures;

Clue 30: it is expected that there would be rules for the adoption of standards, that these standards would be binding. Current practice in the adoption of standards is considered not to be satisfactory.

1.2.3. The current use of the words standard and standardisation is not fully consistent

Some clues are very unclear event in isolation:o clue 17 requires standards to be uniform.o clue 20 mentions that software can be standardised. This would mean that: there exist

somewhere standards for software. Existing software has to be brought to conformity with these standards.

o clue 24 mentions that standards can be certified. Certification means that an organisation checks that a product is in agreement with some standards, and that this organisation guarantees the conformity. Standards have to be developed and registered by standard setting organisations, along agreed procedures. What is meant by clue 24?

Some clues give contradictory information:

This clue contradicts clue n°

Why

7 9 standards do/ do not include definitions or classifications 1 8, 28 The scope of a standard is an elementary step of the statistical

process (defined for instance by the GSBPM) in clue 1 and a whole system in clue 8

9, 23 11, 13, 14 The ESA is a set of elementary standards (clues 9, 23) or a single standard (clue 11, 13, 14)

4 16 International standards are available (clue 4) but the ESS standards are only required to take into account recommendations (that would be expected to include guidelines and good practice but not standards).

9 18 Clue 9 excludes definitions from the scope of standards whereas 18 includes them

4 IEC : the International Electrotechnical Commission is in charge of international standardisation in electricity and electronics.

10

15 22, 29 Standards would be the result of some sort of in-house development (clue 15), in contradiction with the formal standards produced by standard-setting organisations (clue 22).

18 26 Methodological handbooks are (clue 18) or are not (clue 26) considered as standards.

Conclusion: the current use of the word “standard” in the ESS does not refer to a very precise concept. It deals with rules, but their origin, their binding nature, their link with various statistical activities is not precisely defined nor uniform in the various texts (and sometimes in the various paragraphs of one single text).

1.3. Taking stock of standardisation in other domains

1.3.1. Definition of a standard

There is a high convergence in the definitions used either by US standardisation organisations, by EU ones (CEN) and by the ISO/ IEC.

The ISO offers a full glossary, defining consistent elements for the standardisation activity.

The group of statisticians in the ESSnet proposes to use the ISO reference vocabulary in order to ease communication with the outside world 5 .

The starting point of the construction is the concept of normative document:

A normative document provides rules, guidelines or characteristics for activities or their results.Note 1: The term “normative document” is a generic term that covers such documents as standards, technical specifications, codes of practices and regulations.Note 2: A “document” is to be understood as any medium with information recorded on or in it.Note 3: The terms for different kinds of normative documents are defined considering the document and its content as a single entity.

A normative document is made of provisions.

A provision is an expression in the content of a normative document, that takes the form of a statement, an instruction, a recommendation or a requirement. A statement is a provision that conveys information.An instruction is a provision that conveys an action to be performed.A recommendation is a provision that conveys advice or guidance.A requirement is a provision that conveys criteria to be fulfilled.A deemed-to-satisfy provision indicates one or more means of compliance with a requirement of a normative documentA descriptive provision is a provision for fitness for purpose that concerns the characteristics of a product, process or service

5 For a full description, see ISO/ IEC Guide 2 - Standardization and related activities - General vocabulary

11

Schmidt-P, 2011-08-29,

Before referring to ISO and CEN it could be useful to introduce these institutions and their relevance for the ESSnet.

A performance provision is a provision for fitness for purpose that concerns the behavior of a product, process or service in or related to use

NOTE These types of provisions are distinguished by the form of wording they employ; e.g. instructions are expressed in the imperative mood, recommendations by the use of the auxiliary “should” and requirements by the use of the auxiliary “shall”.

A standard is a document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context. Note: Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

A code of practice is a document that recommends practices or procedures for the design, manufacture, installation, maintenance or utilization of equipment, structures or products.

Note: A code of practice may be a standard, a part of a standard or independent of a standard.

Standards are classified into various types.

One useful distinction is the following:

product standard : standard that specifies requirements to be fulfilled by a product or a group of products, to establish its fitness for purpose.

process standard : standard that specifies requirements to be fulfilled by a process, to establish its fitness for purpose.

This distinction is relevant mainly for goods. Goods can be standardised either directly, by a product standard, or indirectly, by a process standard defining its production process. Product process are considered in the literature to be much more compatible with innovation than process standards.

Statistics are considered to be services6.

service standard: standard that specifies requirements to be fulfilled by a service, to establish its fitness for purpose.

Note: Service standards may be prepared in fields such as laundering, hotel-keeping, transport, car-servicing, telecommunications, insurance, banking, trading.

It is generally expected that service standards will belong to the category of process standards.

1.3.2. How should this proposal be understood in the ESSnet STAND-PREP ?

The group does not propose to change the meaning of existing recommendations or legal texts; it proposes to adopt future texts on the basis of a vocabulary consistent with other domains, i.e. the ISO definition.6 At least they are classified in this category by the CPA, as all information services.

12

For instance, let us take the example of methodological handbooks in the ESS. We have observed that in general:

o the activity for which the methodological manuals are developed is sometimes/ frequently unclear: for instance, it is often difficult to decide if the manual is intended for general training of statisticians or for defining a process that has to be implemented.

o the adoption procedure is often unclear, or at least is not in line with the usual decision process in the ESS. At some point, such manuals should be adopted by the ESS Committee or even by other kinds of organisation.

o the “optimum degree of order in a given context” is not really dealt with. In particular, it is agreed that standards could aim for various degrees of quality of results or for savings. In general, methodological standards do not deal with this issue.

Hence, in general, existing methodological manuals would be called normative documents, not standards. Very few of them could be considered as proper standards since their adoption is unclear. The set of ESS guidelines on Seasonal Adjustments (2009) is an exception: it has been adopted by the CMFB and the SPC (predecessor of ESS Committee).

Could it be called a standard?

It has to be considered that standards have to comply with some principles. As mentioned in “The ABC’s of standards activities (US National Institute of Standards and Technology- NIST , 2009)”:

“To maximize their utility and to prevent the creation of unnecessary barriers to foreign and domestic commerce, the standards making process should be conducted in accordance with the internationally accepted principles of Consensus, Transparency, Balance, Due Process, and Openness. These principles are particularly important for standards likely to be used in technical regulations. These five principles are explained below: Consensus. Consensus means that all views are heard and the resultant standard is generally agreed to by those involved. Consensus is characterized by the absence of sustained opposition to substantive issues. However, it does not necessarily imply unanimity.Transparency. Transparency means: (a) providing advance public notice of a proposed standards development activity; (b) identifying the scope of work to be undertaken; (c) providing information on conditions for participation; (d) and providing an opportunity for all interested parties to comment prior to final approval and adoption. Balance. Balance means that no one interest, including the government, should dominate. It should be noted that balance can be affected not only by the number of participants in particular categories but also by the funding source. The provider of the funding in standards development work can sometimes end up dominating the process. This is particularly true if the funding is from a government entity. If funding is to be provided by a government agency or other entity, care must be taken to avoid undue influence on the outcome of the process by the funding provider. Due Process. Due process means that any person (organization, company, government agency, individual, etc.) with a direct and material interest has a right to: (a) express a position and the basis for it; (b) have that position considered; and (c) appeal if adversely affected. Due process ensures equity and fair play in the standards development process. Openness. The standards development process should be to open to participation by all materially affected interests.”

The ESS guidelines on Seasonal Adjustments cannot be considered as a standard of the European Economic Area (EU+EFTA) since they do not apply some of these principles. Neither the private producers of statistics, nor academics were associated to this process. Hence the Transparency/

13

http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf


Openness/ Due process criteria are not satisfied here. They can be satisfied only for the population of the national statistical authorities represented by the ESSC (it does not fully apply to the National Central banks, since the ECB/ ESCB has not adopted it).

The US NIST defines a typology of private standard setting organisations, made of five classes: technical and professional societies, industry associations, standards-developing membership organizations, building code organizations, consortia.

It quotes as well in the public sector two types of standards:o Government regulatory standards are those designed to be used by [US] federal regulatory

agencies in rulemaking and related activities.o Federal and Military Specifications, which are official documents used by [US] agencies

and by the [US] Department of Defense respectively, to support government procurement.

If we used the same categories, we would consider that the “ESS guidelines on Seasonal Adjustments” were a standard for the European Commission and Member States administration.

In addition, such standards could be classified according to their binding character:o mandatory standards would be those that are adopted by the co-legislators, i.e. those that

are incorporated in a regulation or in a directiveo voluntary standards would not be adopted by a law-making process.

For instance, the NACE rev.2 classification can be considered as a mandatory standard.

Another important issue in practice is the “scope” of statistical standards that could be developed in the ESS. By “scope”, we mean the activity for which standards would be developed (as mentioned in the definition of a standard, it provides rules for activities or their result) . The ESSnet group has discussed the following issue: should the activity covered be a global one (for instance, the design and production of specific statistics - let us say “Structural business statistics in the manufacturing sector”) or a very focused one (micro-editing of one structural business statistic)?

This concern was linked with the observation that existing methodological manuals can either deal:o with a particular activity defined in the GSBPM (for instance defined by one phase or step

in the model) for all kind of statistics. For instance, the manual on Edimbus: editing and imputation of cross sectional business statistics (GSBPM steps 5.3 and 5.4)

o with a group of statistics for all steps in GSBPM. For instance, the manual on Handbook on the design and implementation of business surveys (1997)

In order to check for consistency of standards - which is desirable- it was considered if developing standards for all “building blocks” of the GSBPM would be the optimal solution…

Although defining a standard for each elementary activity of the statistical process cannot be excluded (no logical shortcoming in this possibility was identified), it was felt that standards (as defined at the ISO or with an EU directive) would encompass a sizeable number of statements (rules, guidelines, etc) in practice. For instance, the ISO 20252-2006 standard on Market, social and opinion surveys is about 35 pages long and covers all steps in the design, production and dissemination phases. It can be noted as well that the ISO has not chosen such an approach to standardisation.This choice might be due to several reasons:

o there are fixed costs for negotiation: so developing a huge number of elementary standards would be costly

14

o in order to cover the whole production process of one statistic, one would have to wait until the whole system of standards is developed.

Adopting such a definition does not mean that statements (as those quoted in paragraph 1.2.2) should be understood using such a meaning of the word “standard”.

Such a choice would be of no consequence on the scope of the work on standardisation in the ESS. There is no ambiguity on this issue, since the ESSnet STAND PREP has no competence on this issue, that can only be dealt with by the Sponsorship on standardisation.

15

2. Review and analysis of six methodological manuals

2.1. A framework for the analysis

The objective of Work Package 1 was to define the concept of standard in statistics, in particular the standards corresponding to “statistical methods”, and to apply these definitions in order to find out, for each of the six selected methodological handbooks, which parts can be considered as standards.Actually, it became soon clear that this work of analysis could yield a value added: the job was not only to excerpt the core of the methodological handbooks, consisting in the provisions that are the basic elements of a standard; but also the relationships with other fundamental items and references that are contained in a handbook, as methodologies, methods, IT tools, classifications, definitions, etcetera. Furthermore, it was also considered the usefulness of linking each provision (or group of provisions) to a given phase or sub-process of GSBPM, in order to have the possibility to collect and compare all provisions available for a given step of the production process of statistical information. The relationships between all the elements cited above are rather complex: it was therefore necessary to model them by defining a conceptual scheme representing all the entities that can be found in a set of methodological handbooks, and the relations connecting these entities. This conceptual scheme was translated in a logical one, the basis for a repository able to contain the knowledge that was to be extracted from the handbooks.It was also necessary to develop an IT tool enabling all partners in the ESSnet to populate the repository in a coordinated and assisted way. To this aim, a procedure was defined, consisting of the following steps:

1. each single handbook was assigned to one or two partners, that conducted the analysis on it;2. if two different analyses had been carried out on the same handbook, a reconciliation step

was performed to get a unique and agreed result;3. the results of all the analyses have been introduced in a final and unified repository, that

contains all the provisions individuated in the six handbooks.This final repository is to be considered as a proof-of-concepts: it is possible to start from the vast amount of handbooks related to the statistical production processes, each of them to be considered as a normative document, and obtain something that can be regarded as an instrument to get information at a detailed level on all rules concerning a particular subject, regardless of the source. This would not be possible if the handbooks had been described only at the aggregated level, without going at the elementary level of the “provision”. For instance, it is now possible to interrogate the repository in order to list all the provisions related to a given GSBPM sub-process obtaining information on the source (a given handbook), the type (indicating the associated degree of freedom), the related methodologies and methods.

16

2.1.1. Conceptual scheme of the repository for the standards

In figure 1, the conceptual scheme of the repository for statistical standards is reported, accordingly to the features of the entity-relationship model7.

Figure 1 – The conceptual scheme of statistical standards in methodological handbooks

In the following, we illustrate the meaning of each entity, their attributes and the relationships among entities. To exemplify, we show also the forms dedicated to each entity that are in the

7 Chen P.P., “The entity-relationship model—toward a unified view of data”, ACM Transactions on Database Systems (TODS), September 22–24, 1975

methodology

contains

handbook = normative document

1,n

classification definition

method

based on

1,n

structured group of provisions

contains1,n

IT tool

implemented in

0,n

1,1

1,1

statisticalactivity (UN)

0,n

refers to

0,n

provision

refers to refers to refers to

0,1 0,1

1,n 1,n 1,n

refers to

0,1

0,n

1,n

0,n

0,1

17

software developed for the management of the repository (which is illustrated in detail in par. 2.1.2).

Normative document

In the model described in the E-R scheme, a statistical methodological handbook may be defined, accordingly to the definitions reported in Section 1 of this report, as a “normative document” (“a document that provides rules, guidelines or characteristics for activities or their results”).

The attributes related to this entity are:1. title: the title of the document;2. version: indication of the version, if available (otherwise, year of publishing);3. type: the normative document can be:

o a “standard” if the document has been established by consensus and approved by a recognized body;

o a “pre-standard” if the document is adopted provisionally by a standardizing body and made available to the public in order to gain the necessary experience from its application;

o a “technical specification” if it prescribes technical requirements to be fulfilled by a product, process or service;

o a “code of practice” if it is a document that recommends practices of procedureso a “regulation” if provides binding legislative rules.

4. level: the area of influence of the document. It can be:o internationalo regionalo nationalo provincial

5. body: the institution(s) that is (are) responsible for the development of the normative document;

6. consensus: the procedure (if any) followed to reach a consensus, required to declare the normative document as a standard;

7. aim: the purpose of the normative document.

In general, the handbooks we are dealing with can be considered as of the kind “code of practice”.

18

Group of provisions

A normative document must contain at least one group of provisions, in general many of them (cardinality = 1,n)8. One group of provisions belongs to only one normative document.In the case of methodological handbooks, the identification of the groups of provisions should take into account the following elements: the structure of the handbook (sections and chapters) and the relations with the most detailed level of GSBPM (sub-processes). A group of provisions must refer to only one sub-process in GSBPM, and should be contained in only one chapter (maybe in more paragraphs).

Attributes of the entity “group of provisions” are:1. collocation: indication of the logical (section, chapter) and physical (from page to page)

collocation inside the handbook;2. description: any important feature to be reported;3. main GSBPM sub-process9;4. auxiliary GSBPM sub-process;5. additional GSBMP sub-process.

Provision

Accordingly to the definitions given in Section 1, a provision is …

A group must contain at least one provision, in general many of them (cardinality = 1,n).

Attributes of the entity “provision” are:1. indication of the physical collocation (page) inside the handbook;2. description: text, as it appears in the document;3. type:

o statement, o instruction, o recommendation, o exclusive requirement, o optional requirement, o deemed-to-satisfy provision, o descriptive provision, o performance provision;

4. sequence: progressive number unique inside the same structured group;8 In a given relationship between entities A and B, the cardinality is a couple whose first element indicates the minimum number of instances of B linked to one instance of A, while the second element indicates the maximum number.9 See the paper “Generic Statistical Business Process Model - Version 4.0” (UNECE April 2009)

19

5. condition(s) of application;6. components of quality10:

a) the ones referred to the product:o relevance;o accuracy;o timeliness and punctuality;o accessibility and clarity;o coherence and comparability;

b) the ones referred to the process:o effectiveness;o efficiency.

A provision may refer to:o statistical activities;o a methodology;o an IT tool;o a classification;o a definition.

Statistical activities

A provision may or may not concern a given statistical activity, or to many of them (cardinality: 0,n). A given statistical domain can be referred by no provision, or by many provisions (cardinality: 0,n). The different statistical activities considered in this scheme are a subset of the complete set that has been for the Global Inventory of UNECE11. In particular, this subset contains all the subject matter areas:

o D1: demographic and social statistics,o D2: economic statistics,o D3: environment and multidomain statistics,

10 We consider the EUROSTAT quality dimensions, as they have been defined in Eurostat (2009) ESS Handbook for Quality Reports, 2009 Edition.11 See the Annex I to the paper “Developing a Global Inventory of Statistical Standards” (UNECE, 18 August 2010).

20

with the exclusion of the other areas as:o D4: methodology of data collection, processing and dissemination;o D5: strategic and managerial issues of official statistics.

This is why the items contained in D4 and D5 have been judged as out-of-the-scope with respect to statistical standards, or better represented by the GSBPM.

The attributes of the entity “statistical activity” are:1. identifier: see the United Nations set of codes;2. description: text (see the United Nations set of descriptions).

Methodology

A given provision may or may not indicate one methodology (cardinality: 0,1). A given methodology, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “methodology” are:

1. identifier: short description;

2. description: extended description of the methodology;3. condition(s) of application.

Method

A given methodology may or may not indicate one method, in general more than one (cardinality 0,n). A given method has to be linked to at least one methodology, or to many of them (cardinality: 1,n).The attributes of the entity “method” are:

1. identifier: short description;

2. description: extended description of the method;3. condition(s) of application.

Relationships between methodologies and methods are difficult to define in a univocal way. Basically, a methodology should indicate a general approach, that may include a set of methods enabling to follow that approach12.

12 For instance, in the field of editing and imputation the so-called “Fellegi-Holt methodology” is a general approach following the principle of the minimum change. Minimum change can be obtained by adopting the Fellegi-Holt (“For

21

Schmidt-P, 2011-08-29,

To better understand the terms methodology and method a short defintion for these terms and their difference could be helpful.

IT tool

A given provision may or may not indicate one IT tool (cardinality: 0,1). A given IT tool, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “IT tool” are:

1. identifier: name or acronym;

2. description: text;3. organisation.

A given IT tool may or may not implement a given method, or many of them (cardinality: 0,n).A given method may or may not be implemented in a given IT tool, or in many of them (cardinality: 1,n).

Classification

A given provision may or may not indicate one classification (cardinality: 0,1). A given classification, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “classification” are:

1. identifier: name or acronym;2. description: text;3. version.

Definition

A given provision may or may not indicate one definition (cardinality: 0,1). A given definition, if present, must be indicated by one provision, in general by many of them (cardinality: 1,n). The attributes of the entity “definition” are:

1. identifier: name or acronym;2. description: text;3. version.

each failed edit record, the Fellegi-Holt approach first proceeds through a step of error localisation in which it determines the minimal set of variables (fields) to impute, as well as the acceptable ranges(s) of values to impute.”) error localisation method and the Fellegi-Holt imputation method (“A single donor is selected from passed edit records by matching on the basis of other variables involved in the edits”).

22

2.1.2. The IT tool for the management of the repository

The conceptual scheme has been implemented in a relational database, whose logical scheme is reported in figure 2.

Figure 2 – Structure of the repository

In order to allow each partner in the ESSnet project to populate the repository in an assisted way, a dedicated software was developed.Here we report only the most important characteristics of the software.

General flow of the operations

Running the application, the first form that will appear is the “general menu” form. The dropdown list located just under the handbook image permits to select the institute (or the specific agent) identifier. At the moment the two section buttons are not enabled.

23

GSBPMsub-process

After the identification of the Institute, the system permits to move in one of the two management sections: “Documents” (or handbooks) and “Methods”.

It was suggested to proceed in this way:1. when analysing a given methodological handbook, start to select the “Documents” section,

and proceed to introduce all necessary information related to the entities in the scheme up to the entity “methods” excluded;

2. once a new method has to be introduced, leave the “Documents” section and, from the “Main menu” select “Methods” section;

3. once the methods have been introduced, then get back to the “Documents” section and proceed to link methodologies to methods.

Documents List Form

The documents list form, as the methods one, is divided in two areas. The search frame permits to display the documents by title, type, body, ID, version, consensus, lever and aim. All these criteria are in “and” search condition.By pressing the “New” button, it is possible to insert a new instance for the “Document” entity.

Provisions Groups List Form

Once a “Document” has been introduced, the Provisions Groups List Form appears. Also in this case, by clicking on the “New” button, it is possible to insert a new instance.

24

Bode-O, 2011-08-29,

Page 23 is covered

Provisions Form

For each selected Provision Group, it is possible to define new instances of “provisions”. Once introduced, for each of them the values of the attributes are reported in the lower part of the form.

In this lower part, it is possible also to associate “Methodologies”, “Tools”, “Classifications” and “Definitions”.If one of these objects does not yet exist, by clicking on the corresponding “…” button it will be possible to introduce a new one that from now on will be available.It is also possible to associate one or more “Statistical activities” by clicking on “List” button: the list of statistical activities appear on the upper part of the form, and it is possible to select the ones required.

Methods List Form

From the initial form, it is possible to select the “Methods” form.In this form all the methods introduced so far will appear.The “New” button permits to add a new method.To select the displayed method the user has to click in any cell of the selected method.When a method has been selected, it will be possible to associate the methodologies to which it belongs.

25

2.1.3. Procedure followed for the definition and refinement of repositories for single handbooks and for the construction of the unified repository

In order to analyse the methodological handbooks, each of them was assigned to different Institutes:

Handbook Organisation(s) carrying out the analysis

Organisation carrying out the reconciliation

Handbook of recommended practices for questionnaire development and testing in the European Statistical System

ONS & Insee ONS

Eurostat sampling reference guidelines: Introduction to sample design and estimation techniques

Destatis -

Survey methods and practice (Stat.Can.) Istat & ONS ISTATGuidelines for statistical metadata on the Internet

Insee -

ESS handbook for Quality reports GUS &KSH KSHEdimbus: editing and imputation of cross sectional business statistics

KSH & Destatis Destatis

In some cases, the analysis (i.e. the introduction into the repository of the information required for each entity in the scheme) has been carried out by a single Institute.In some other cases, the analysis has been performed by two Institutes. That is why we wanted to compare the results, analyse the differences and assess the degree of arbitrariness in the process.In order to build a unique repository for all the handbooks, for the three cases in which the analysis has been done by two partners, to one of them the responsibility for a reconciliation step was assigned. The partner responsible for the compared analysis had to detect:

1. inconsistencies between the two repositories, that could regard• provisions: differences in definition and/or attributes• other objects (methodologies, methods, IT tools, etc.): differences in definitions

and/or in links2. non-correspondences (missing items in one repository with respect to another).

The responsible partner had to list all the identified problems, and discuss them with the other partner, in order to agree on the final result, that is a unique repository of validated provisions and other objects, for the given handbook.

Once for each handbook a unique repository has been defined, then it has been possible to build the final, unified repository containing all the information derived by the analysis of the six methodological handbooks.This final repository has been analysed in order to describe its content and also to illustrate with some examples how it is possible to use it in order to obtain useful information regarding provisions coming from different handbooks and dealing with the same field of application.

27

2.2. Analysis of the six methodological handbooks

In this sub-chapter, the analyses on the different handbooks are reported, accordingly to common structures.In the case where only one Institute was in charge to analyse a given handbook (this is the case for handbooks “Eurostat sampling reference guidelines” and ”Guidelines for statistical metadata on the Internet”), we have

1. a general description of the handbook and a list of the analysed chapters;2. the number of provisions and of the groups in which they have been considered together;3. the indication of the GSBPM main sub-processes;4. a summary of the entities linked to provisions in the handbook (methodologies, methods, IT

tools, classifications and definitions),plus general comments to better illustrate the results of the analysis, or the way it has been carried out.In the case where two Institutes have produced two distinct analyses (the remaining four handbooks), the above items are referred to the “reconciled” version of the analysis, but before of this a compared analysis of the common chapters and divergences encountered have been reported. For each common chapter, if different approaches have been followed, they have been outlined, and examples of divergences are reported. This is an important basis to understand the degree of arbitrariness to which the operation of analysis of a methodological handbook, following the adopted conceptual scheme, is subject.

28

2.2.1. Eurostat sampling references guidelines

General description of the handbook and list of the analysed chapters

The one showed in the following table is the handbook “Eurostat sampling reference guidelines” analysed by DESTATIS alone.

Table 1. Handbook-attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Survey Sampling and Reference Guidelines,

Introduction to sample design and estimation techniques

Version: indication of the version, if available (otherwise, year of publishing)

2008

Type: standard, pre-standard, technical specification, code of practice, regulation

Code of practice, Guidelines

Level: international (=global), regional (e.g.: European), national, provincial

International

Body: organisation(s) responsible for the normative document

Eurostat

Consensus: if any, indication of the agreement EurostatAim These guidelines give an overview of basic

principles and methods of survey sampling. This includes survey planning, survey quality, sampling, estimation and treatment of non-response. Also, references to statistical software (IT tools) are given.

The handbook is composed of 7 chapters, which have been analysed as reported in Table 2.

Table 2. Analysed chapters

Chapters analysed by DESTATIS 1. Introduction2. Survey planning and reporting3. Techniques for sample selection and

estimation4. Treatment of nonresponse5. Software6. References7. Web links

Only chapters 3 to 5 provided relevant information for the identification of provisions, methodology and methods.

29

Overall description of the analysed handbook

Table 3. General characteristics of the chapters

Number of Provision groups

Main GSBPM Number of identified Provisions :

CHAPTER 3 1 4.1 3CHAPTER 4 1 5.6 2

Comment.

In order to detect provisions and provision groups inside the analysed document, a general bottom-up-approach was applied. This means that first the provisions were identified, as these are most elementary items according to the data model of the data base. Secondly, these provisions were logically and according to GSBPM sub-processes grouped into provision groups.The appearance within the chapters was considered as not so important.Of course, top-down would also have been possible instead of bottom-up, but in this case the data model should be more flexible and allow for sub-provisions and sub-sub-provisions etc.The importance of a recommendation on the search strategy is that it should (in theory) lead to similar results when the analysis is carried out in parallel by different people (NSIs).

Table 4. Summary of the entities linked to provisions in the handbook

Entity Total number ListMethodologies 5 Probability Sampling

Sample size Allocation to strata Estimation Techniques Reweighting in case of unit nonresponse Imputation

Methods 21 Cluster Sampling (CLU) Probability-Proportional-to-Size (PPS)

Sampling Simple Random Sampling (SRS) Stratified Sampling (STR) Systematic Sampling (SYS) Equal Allocation Optimal Allocation (Neyman) Power Allocation (Bankier) Proportional Allocation Horvitz-Thomson estimation Ratio Estimation Regression Estimation

30

Post-Stratification GREG-Estimation Direct Reweighting Empirical Response Homogeneity Groups

(RHG) Explicit Response Probability Modelling Calibration of weights Logical Imputation Real-Donor Imputation Model-Donor Imputation

It-tools 31 see belowClassifications

Definitions

Comment.

Also, for the identification of methods and methodologies, the above mentioned bottom-up-approach was applied here. This means that it was started with the detection of methods as the most elementary items in the data model and then the logical grouping of the methods to methodologies was carried out.Also here, a recommendation on the search strategy is considered to be of great importance in order to come to “unique” solutions.

In order to show the linkage of the IT-tools, mentioned in the handbook, to the methods and methodologies, a more complete overview is given.

It-tools

Two large scale statistical programs, SAS and SPSS contain specific modules for sample selection and survey data analysis. In SAS the procedures are included in STAT module from version 8 on. In SPSS there is an add-on module Complex Samples which must be purchased separately, available from version 11 and above.The programming language R is widely used in academic studies and some authors have providedcodes for survey sampling and analysis. Those are found on different web-pages containingR program codes, e.g. www.r-project.org/.

Software for sample selectionSAS/Stat v. 8 and above: Proc SurveySelect (www.sas.com)SPSS/Complex Samples v. 11 and above: CSPLAN together with CSSELECT (www.spss.com)

Software for weight derivationSAS add-on programmes: CALMAR2 (INSEE/France), CLAN97 (Statistics Sweden)SPSS add-on programmes: g-CALIB (Statistics Belgium)BLAISE component BASCULA (Statistics Netherlands)

Software for editing and imputationSAS/Stat v. 8 and above Proc MI and MIANALYZE (multi-imputation)SAS add-on programmes: BANFF/GEIS (Statistics Canada), CONCORD, DIESIS, QUIS (ISTAT/Italy), IVEWARE (www.isr.umich.edu/src/smp/ive/)

31

http://www.r-project.org/

Software for “standard” estimationSAS/Stat v. 9 and above: Proc Surveymeans, Proc SurveyfreqSAS add-on programmes: CLAN97 (Statistics Sweden), POULPE (INSEE/France), GES (Statistics Canada)SPSS v. 11 and above: CSDESCRIPTIVES, CSTABULATESPSS add-on programmes: g-CALIB (Statistics Belgium)BLAISE component: BASCULA (Statistics Netherlands)SUDAAN (www.rti.org)

Software for analytical purposesSAS/Stat v. 9 and above: Proc Surveyreg, Proc Surveylogistic, (GLM, MIXED, NLMIXED,…)SPSS v. 11 and above: CSGLM, CSORDINAL, CSLOGISTICSUDAANWesVAR (www.westat.com)

32

http://www.westat.com/

http://www.rti.org/

2.2.2. Recommended practices for editing and imputation in cross-sectional business surveys (EDIMBUS)


The one showed in the following table is the handbook “Recommended practices for editing and imputation in cross-sectional business surveys” analysed jointly by KSH and Destatis.

Table 5. Handbook attributes

Attributes of the entity “normative document” Title of the document (of the handbook) Recommended practices for editing and

imputation in cross-sectional business surveys


2007


Code of practice


Regional (European)


ISTAT, CBS, SFSO

Consensus: if any, indication of the agreement -Aim Development of a common practical and

methodological framework for editing and imputation applications in business statistics

The handbook is composed of 7 chapters, a subset of which have been analysed as reported in table 6 where the chapters in common are highlighted in bold .


Chapters analysed by Destatis 3. Detection of Errors4. Treatment of Errors5. Subsequent Analysis and Estimation

33

Chapters analysed by KSH 3. Detection of Errors4. Treatment of Errors5. Subsequent Analysis and Estimation

Comparison of the common chapters and analysis of the divergences encountered

First common chapter: “Chapter 3: Detection of Errors”

The chapter regards the identifying of values that are not acceptable with respect to some pre-defined logical, mathematical or statistical criteria. Since different error types may contaminate the observed data in a sample of units, usually the error detection process consists of a set of integrated error detection methods dealing each with a specific type of error.

The standards individuated by the two NSI’s in chapter 3 can be summarized in table 7.

Table 7. Summary

Destatis KSH

Total number of Provision Groups

6 6

Main GSBPM 5.3. Review, validate and edit 5.3. Review, validate and edit

Secondary GSBPM - -

Collocation Chapter 3 Chapter 3 Number of identified Provisionsstatementsinstructionsrecommendations etc.

66 statements11 instructions14 recommendations

75 statements11 instructions15 recommendations1 deemed-to-satisfy provision

Number of Methodologies 5 7

The main difference in Chapter 3 is that KSH specifies the approaches “Macroediting” and “Fellegi-Holt paradigm” as methodologies while Destatis specifies them as methods. Another difference in this chapter is that the provision contained in the provision group “Detection of errors” refers to a classification in the case of Destatis and to a methodology in the case of KSH.

The following table 87 shows the different results for the provision group “Influential errors”. In the specification of Destatis only one provision is contained which refers to the methodology “Detection of influential errors” and this methodology is based on the methods “Macroediting” and “Selective editing”. In contrast KSH specifies two provisions and “Macroediting” is a methodology here which is based on the methods “Aggregate method” and “Graphical editing”.

Table 8. Example of the different approaches

34

Destatis KSHProvision group 5 (Influential errors) 4 (Influential errors)Provision 1 1 and 2Collocation Pages 20-24 Pages 20-24Description Influential errors are errors in variables

that have a significant influence on publication target statistics for those variables

1: Influential errors are errors in variables that have a significant influence on publication target statistics for those variables

2: Macroediting methods allow one to identify suspicious data in individual records on the basis of the analysis of preliminary survey estimates

Type statement statementsSequence none noneMethodology Detection of influential errors Detection of influential errors

MacroeditingDescription Errors in values of variables that have

a significant influence on publication target statistics.

Larger enterprises or those associated with higher weights have a significant contribution to the estimated values.

In general, macroediting methods allow one to identify suspicious data in individual records on the basis of the analysis of preliminary survey esti-mates. These techniques are typically applied to assess the validity of preli-minary estimates and identify possible residual influential errors in data.

Quality dimensions Accuracy and reliability, timeliness and punctuality, coherence and comparability, efficiency

Accuracy and reliability, timeliness and punctuality, coherence and comparability, efficiency

Methods MacroeditingSelective editing

Aggregate methodGraphical editingSelective editing

Second common chapter: “Chapter 4: Treatment of Errors”

The chapter regards the replacement of values that are believed to be in error with more plausible values. The standards individuated by the two NSI’s in chapter 4 can be summarized in table 9 .

Table 9. Summary

Destatis KSH

35


2 1

Main GSBPM 5.4. Impute 5.4. Impute

Secondary GSBPM - -

Collocation Chapter 4 Chapter 4Number of identified Provisionsstatementinstructionsrecommendation etc.

22 statements

22 statements


One main difference in chapter 4 is that KSH specifies the whole chapter 4 as one provision group, while Destatis specifies the two sections of chapter 4 “Imputation” and “Interactive treatment” each time as a provision group. Another main difference is the varying level of detail of specifying the methods the methodology “Imputation” is based on. While KSH specifies 12 methods connected to the methodology “Imputation”, Destatis subsumes them in 4 methods. The following table 10 shows the higher level of detail specified by KSH in this case.


Destatis KSHProvision group 8 (Imputation) 7 (Imputation)Provision 1 1 Collocation Pages 30-34 Pages 30-34Description Missing data resulting from nonre-

sponse or values flagged as erroneous during editing may be treated by imputation. Imputation consists of replacing these values by plausible ones to meet the demands on data quality for analysis and dissemination.

Missing data resulting from nonre-sponse or values flagged as erroneous during editing may be treated by imputation. Imputation consists of replacing these values by plausible ones to meet the demands on data quality for analysis and dissemination.

Type statement statementSequence none noneMethodology Imputation Imputation

Description Missing data resulting from non-response or erroneous data may be treated by imputation.

Missing data resulting from non-response or erroneous data may be treated by imputation.

Quality dimensions Accuracy and reliability, timeliness Accuracy and reliability, timeliness

36

and punctuality, coherence and comparability, efficiency

and punctuality, coherence and comparability, efficiency

Methods Donor-based imputationImputation cellsModel based imputationRule-based imputation

Cold-deck imputationDeductive imputationDonor-based imputationHot-deck imputationImputation cellsMean imputationModel based imputationNearest neighbour imputationRandom donor imputationRatio imputationRegression imputationRule-based imputation

Third common chapter: “Chapter 5: Subsequent Analysis and Estimation”

The chapter regards the estimation and analysis of data which contain imputed values. These values can not be treated as truly observed because the imputations may lead to biased parameter estimates. Additionally sometimes some missing values and outliers are explicitly not treated by the editing and imputation process because it is considered more appropriate to deal with them by special estimators that take missing values or outliers into account; chapter 5 also deals with such estimators. The standards individuated by the two NSI’s in chapter 5 can be summarized in table 11.

Table 11. Summary

Destatis KSH


2 2

Main GSBPM 6.3. Scrutinize and explain 6.3. Scrutinize and explain

Secondary GSBPM - -

Collocation Chapter 5 Chapter 5Number of identified Provisionsstatementinstructionsrecommendation etc.

22 statements

22 statements


The comparison of chapter 5 shows no specification differences at all. Both NSI´s identified identi-cal provision groups, provisions, methodologies and methods.

37

Concluding remarks on the compared analysis

The compared analysis of the Edimbus-handbook “Recommended practices for editing and imputation in cross-sectional business surveys” of KSH and Destatis revealed altogether only small differences. One reason may be that both partners were guided by the example of ISTAT of filling in the ISTAT model on Chapter 6 of the handbook “Survey Methods and Practices”.

Taken together there were very small differences in stated provision groups, small differences in specified methodologies, bigger but still moderate differences in specified methods, slightly to moderate differences in descriptions of methodologies and methods as well as no differences at all in allocation of generic statistical business process models.

The main reasons for the encountered divergences are the varying level of detail in specifying (imputation) methods and the problem of specifying an approach or an entity as a methodology versus specifying it as a method.

Description of the reconciled version of the common chapters

On the one hand the reconciled version follows KSH in a higher level of detail especially for the methods linked to the methodologies "Detection of outliers" and "Imputation". Additionally the specified method "Deterministic checking rules" of KSH (connected with the methodology "Localizing random errors") is incorporated and the name of the methodology "Edit checks or checking rules" by Destatis is changed to "Detection of missing values" by KSH. On the other hand the reconciled version follows Destatis in specifying "Macroediting" and the "Fellegi-Holt paradigm" as methods. Furthermore the provision group “Detection of errors” or more specifically the contained provision refers to a classification and not to a methodology. In addition the reconciled version of the handbook follows Destatis in specifying chapter 4 as containing two provision groups. The following tables so summarize the final versions of chapters 3, 4 and 5 of the Edimbus-handbook.

Final version of “Chapter 3: Detection of Errors”

Table 12. Summary of the final version of chapter 3

Final shared version

Total number of Provision Groups 6

Main GSBPM 5.3. Review, validate and edit

Secondary GSBPM -

Collocation Charter 3Number of identified Provisionsinstructionstatementsinstructionsrecommendations


38

descriptive provisionNumber of Methodologies 5

Final version of “Chapter 4: Treatment of Errors”

Table 13. Final version of chapter 4



Main GSBPM 5.4. Impute

Secondary GSBPM -

Collocation Chapter 4Number of identified Provisionsinstructionstatementdescriptive provision

22 statements

Number of Methodologies 2

Final version of Chapter 5: Subsequent Analysis and Estimation”

Table 14. Final version of chapter 5



Main GSBPM 6.3. Scrutinize and explain

Secondary GSBPM -

Collocation Chapter 5Number of identified Provisionsinstructionstatementdescriptive provision

22 statements





Main GSBPM Number of identified

39

Provisions :

CHAPTER 3 (common) 6 5.3. Review, validate and edit

6 statements

CHAPTER 4 (common) 2 5.4. Impute 2 statements

CHAPTER 5 (common) 2 6.3. Scrutinize and explain

2 statements


Entity Total number ListMethodologies 9 Detection of missing values

Detection of systematic errorsDetection of influential errorsDetection of outliersLocalizing random errorsImputationInteractive treatmentVariance estimation in the presence of imputed dataEstimation in the presence of missing values and outliers

Methods 29 Analysis of fatal editsAnalysis of ratio editsFinite mixture modelsMacroediting Selective editingGraphical editingHidiroglou-Berthelot methodMedian-absolute-deviation ruleOne-step robustified ratio estimatorRegression based modelsWeighted quantilesDeterministic checking rulesFellegi-Holt paradigmCold-deck imputationDeductive imputationHot-deck imputationImputation cellsMean imputationNearest neighbour imputationRandom donor imputationRatio imputationRegression imputationRule-based imputation

40

Analytical methodsMultiple imputationResampling methodsCensored and winsorized meansEM-AlgorithmM-Estimator

It-tools -Classifications 1 Detection of errors

Definitions -

41

2.2.3. Guidelines for statistical metadata on the Internet


The one showed in the following table is the handbook “Guidelines for statistical metadata on the interne” analysed only by INSEE.


Attributes of the entity “normative document” Title of the document (of the handbook) Guidelines for statistical metadata on the

internet Version: indication of the version, if available (otherwise, year of publishing)

2000


Code of practice


International


United Nations Statistical Commisssion and Economic Commission for Europe

Consensus: if any, indication of the agreement UNECE, National Statistical Offices members of the UNECE, FAO, IMF, UNSD

Aim The Guidelines for Statistical Metadata on Internet make recommendations for a further harmonisation accompanying the statistical information on Internet.

The handbook is not formally composed of chapters, however to better organise the analysis we have considered the paragraphs as chapters.


Chapters analysed by INSEE 1. Metadata issues specific to the Internet2. Users of the Internet3. Guidelines for different types of metadata

42

Chapter 1 : “Metadata issues specific to the Internet”

The chapter regards the basic requirements for metadata on the Internet.

The standards individuated can be summarized in table 19 …..

Table 19. Summary


1

Main GSBPM 7.2. Produce dissemination products

Secondary GSBPM 7.3. Manage release of dissemination products

Collocation P7-8 Section IINumber of identified Provisionsstatementinstructionsrecommendation etc.

64 statements2 recommendations


The provision group is actually the chapter. The same information is then to be reported.

Chapter 2 : “Users of the Internet”

Table 20. Summary


1

Main GSBPM 7.4. Promote dissemination products

Secondary GSBPM

Collocation P 8-9 Section IIINumber of identified Provisionsstatementinstructionsrecommendation etc.

21 statement1 recommendation


43

Chapter 3: “Guidelines for different types of metadata”

Table 21. Summary


1

Main GSBPM 7.4. Promote dissemination products

Secondary GSBPM

Collocation P 9-12 Section IVNumber of identified Provisionsstatementinstructionsrecommendation etc.

121 statement7 recommendations2 Decriptive provisions1 instruction1 Requirement






CHAPTER 1 (by NSI: INSEE )

1 7.2. Produce dissemination products

4 statements2 recommendations

CHAPTER 2(by NSI: INSEE)

1 7.4. Promote dissemination products

1 statement1 recommendation

CHAPTER 3(by NSI: INSEE)

1 7.4. Promote dissemination products

1 statement7 recommendations2 Decriptive provisions1 instruction1 Requirement


44

Entity Total number ListMethodologies 0Methods 0It-tools 0Classifications 0

Definitions 2 - Basic definition of the external users of statistics

- Definition of external users of statistics according to their level of skills in statistics

45

2.2.4. ESS Handbook for Quality Reports


The one showed in the following table is the handbook “ESS Handbook for Quality Reports” analysed jointly by KSH and GUS.


Attributes of the entity “normative document” Title of the document (of the handbook) ESS Handbook for Quality ReportsVersion: indication of the version, if available (otherwise, year of publishing)

2009


Code of practice


Regional


Eurostat, Working group on Quality

Consensus: if any, indication of the agreement Aim The general aim of this Handbook (EHQR) is

to provide guidelines for preparation of comprehensive quality reports for a full range of statistical processes and their outputs.

The handbook is composed of 3 parts, 18 chapters chapters, one of which have been analysed as reported in Table 254.


Chapters analysed by HCSOPART II: GUIDELINES FOR QUALITY REPORTS,Chapter 3. Accuracy

Chapters analysed by GUSPART II: GUIDELINES FOR QUALITY REPORTS,Chapter 3. Accuracy

46

Comparison of the common chapters and analysis of the divergences encountered

Common chapter: Part II, Chapter 3. Accuracy

The chapter regards methods and indicators measuring accuracy of different statistical processes.

The standards individuated by the two NSI’s in the chapter 3 can be summarized in table 26.

Table 26. Summary

KSH GUS


8 7

Main GSBPM 10.1 Quality management 4.1 Select sample4.4 Finalize collection6.5 Finalize outputs5.4 Impute

Secondary GSBPM 2.4 Design frame and sample methodology

5.1 Integrate data5.3 Review, validate and edit5.8 Finalize data files

Collocation Part II, Chapter 3 Part II, Chapter 3Number of identified Provisionsstatementinstructionsrecommendation etc.

193

118

75

11

11


KSH and GUS followed completely different approaches: GUS identified some key points of the chapter while KSH tried to identify most of the statements and recommendations included in the chapter.


KSH GUSProvision group 2 3Provision Accuracy for sample surveys Probability samplingCollocation page 35 page 36Description Sampling can be of two types: probability

sampling, meaning that each unit of the frame population has a known, non-zero probability of being selected in the sample, and nonprobability sampling.

Type statement statementSequence - -

47

Methodology - Probability samplingDescription Probability sampling,

meaning that each unit of the frame population has a known, non-zero probability of being selected in the sample.

Quality dimensions Accuracy Accuracy

Methods - -


GUS regrouped the calculation of quality indicators to one provision group, KSH treated the quality indicators as provisions (with an indication „indicator”) in the related thematic provision groups. Division of the chapter to provision groups was also very different: in the case of KSH provision groups cover all the chapter, GUS identified some key points of the chapter. Thus comparison of provision groups is challenging.


At the Rome meeting it was agreed that a detailed analysis is useful, thus the reconciled version was based on the analysis made by KSH, completed with the approach of GUS to include all quality indicators to one provision group. Similarly, recommendations for general description of different aspects of quality to be included in quality reports were regrouped to one provision group.

Final version of the first common chapter

Table 28. Summary of the final version of the first common chapter



Main GSBPM 10.1 Quality management

Secondary GSBPM 2.4 Design frame and sample methodology

Collocation Part II, Chapter 3Number of identified Provisionsstatementrecommendation

19610591


In the reconciled version more methods and methodologies were identified and were linked to provisions. The detailed analysis was kept, but examples were removed from provision groups. Two

48

special provision groups were created: one for quality indicators to be calculated and one for general description to be included in the quality report.





PART II, CHAPTER 3 (by KSH)

8 10.1 Quality management

statement : 118recommendation : 75

PART II, CHAPTER 3(by GUS)

7 4.1 Select sample4.4 Finalize collection6.5 Finalize outputs5.4 Impute

statement : 11recommendation :-

PART II, CHAPTER 3 (common )

10 10.1 Quality management

statement : 105recommendation : 91

As the handbook is about quality reporting the main GSBPM remained “10.1 Quality management”, but it woud fit also in “9. Evaluating”. We agreed that secondary and auxiliary GSBPM would be the GSBPM step corresponding to the referred process, but at the end GSBPM categories were not modified.


Entity Total number ListMethodologies 10 Evaluation of coverage errors

Assessment of accuracy in sample surveyAssessment of biasConsistency studiesDetection of coverage errorsEstimation of expected value and variance of indicatorsEvaluation of nonresponse errorEvaluation of processing errorsQuality assessment for statistical compilationsSampling

Methods 36 Analysis of lag structureAssessment of errors in register variablesAssessment of respondents errorsComparing datasetsDesigning and testing of questionnairesEstimation of the revision size

49

Information from data editingMatching with a different registerReinterviewingReplacement, re-sampling and quality adjustment methods for price statisticsSubject economic data to accounting rulesCoefficient of variation (CV)Estimation of biasRoot mean square error (RMSE)Standard errorQualitative assessment of biasMirror statisticsAssessment of coverage errors in business registersAssessment of coverage errors in household surveyAssessment of under-coverageChecking frame informationDetection of overcoverageHandling of multiple listingsConfidence intervalComplementing with register dataSpecial data collectionsVariations over response wavesStudies of coding variationStudies of effects of editingAnalysis of revisionAnalysis of statistical discrepancyData Quality Assessment Framework (DQAF) of the International Monetary Fund (IMF)Measurement of non-observed economyCut-off samplingNon-probability samplingOther forms of non-probability sampling

It-tools -Classifications 6 Errors

Non-sampling errorsTypes of samplingTypes of coverage errorsCategories of censusRegister-based statistics

Definitions 11 CensusStatistical process using administrative sourcesStatistical process involving multiple data sourcesPrice or other economic index processQuality adjustment for proce price indexesStatistical compilationsRevisionSeasonal adjustmentImputationMistake

50

Code of practice

Based on the last report prepared by ISTAT on the analysis of handbooks, the final analysis should be improved, especially classification of provision groups to GSBPM processes.In the case of this handbook, main GSBPM is “quality management” for all provision groups. As far as secondary and auxiliary GSBPM of provision groups are concerned the main idea was to relate the provision group to the GSBPM it refers to. The problem in our case is that provisions groups identified are too complex, thus they refer to several GSBPM sub processes. In the database only one main, one secondary and one auxiliary GSBPM process can be associated to a provision group. In the table below GSBPM processes are associated to each provisions group.

Table 31. Provisions groups and GSBPM processes.

Provision group Main GSBPM Secondary GSBPM1. General characteristics of and guidelines for accuracy in the QR

10.1. Quality Management 2.4. Design frame and sample methodology4.1. Select sample4.3. Run collection4.4. Finalize collection5.2. Classify and code5.3. Review, validate and edit5.4. Impute5.8. Finalize data files6.5. Finalize outputs9.2. Conduct evaluation

2. Accuracy for sample surveys 10.1. Quality Management 2.4. Design frame and sample methodology4.1. Select sample4.3. Run collection4.4. Finalize collection5.2. Classify and code5.3. Review, validate and edit5.4. Impute5.8. Finalize data files6.5. Finalize outputs9.2. Conduct evaluation

3. Accuracy for Censuses 10.1. Quality Management 4.3. Run collection4.4. Finalize collection5.2. Classify and code5.3. Review, validate and edit5.4. Impute5.8. Finalize data files6.5. Finalize outputs9.2. Conduct evaluation

4. Accuracy for statistical processes using administrative sources

10.1. Quality Management 5.3. Review, validate and edit5.4. Impute

51

5.8. Finalize data files6.5. Finalize outputs9.2. Conduct evaluation

5. Accuracy For Statistical Processes Involving Multiple Data Sources


6. Accuracy for Price and Other Economic Index Processes


7. Accuracy For Statistical Compilations

10.1. Quality Management 5.3. Review, validate and edit5.4. Impute5.8. Finalize data files6.5. Finalize outputs9.2. Conduct evaluation

8. Some special issues concerning accuracy

10.1. Quality Management 5.4. Impute5.8. Finalize data files6.1. Prepare draft output6.5. Finalize outputs9.2. Conduct evaluation

9. Quality and performance indicators for accuracy

10.1. Quality Management 9.2. Conduct evaluation

10. Items to be included in the Quality report for accuracy

10.1. Quality Management 9.2. Conduct evaluation

52

2.2.5. Recommended practices for questionnaire design and testing


The following table describes the “Handbook of recommended practices for questionnaire development and testing in the European Statistical System” analysed by ONS and INSEE.


Attributes of the entity “normative document” Title of the document (of the handbook) Handbook of recommended practices for

questionnaire development and testing in the European Statistical System


Release year 2006


Code of practice


Regional


European Commission

Consensus: if any, indication of the agreement ESS Aim: The recommended practice manual aims at

further specifying the requirements of the European Statistics Code of Practice and defines systematic questionnaire design and testing procedures to minimise measurement error.

The handbook is composed of 7 chapters of which ONS and INSEE have analysed two.

A general comment is that the initial ONS and INSEE analyses were very different and therefore it is difficult to provide meaningful comparisons in any detail. Therefore the two NSIs worked collaboratively to produce the final combined analysis, starting from scratch rather than building on the initial analyses.

Comparison of common chapters

First common chapter: Chapter 2. The cognitive model in answering questions

This chapter describes the application of cognitive research to statistical surveys to provide insight into the cognitive dimensions of the response process.

The standards identified by ONS and INSEE in chapter 2 are summarised in table 33.

53

Table 33. Summary

ONS INSEE


1 1

Main GSBPM 2.3. Design data collection methodology

2.3. Design data collection methodology

Secondary GSBPM 3.1.Build data collection instrument

3.1. Build data collection instrument

Collocation Chapter 2 Chapter 2Number of identified Provisions

1 0 statements0 instructions1 recommendation

107 statements2 instructions1 recommendation


As can be seen from the table, ONS and INSEE took quite different approaches to defining provisions. Additionally, ONS did not complete the methods selection for this chapter. On investigation both NSIs identified ways in which their analyses were flawed and therefore agreed that direct comparisons are not necessary.

Second common chapter: Chapter 3: Questionnaire design

This chapter describes the process of questionnaire design, namely the development of the conceptual frame of the questionnaire, writing and sequencing of questions and establishing the visual design elements of the questionnaire.

The standards identified by ONS and INSEE in chapter 3 are summarised in table 34.

Table 34. Summary for second common chapter

ONS INSEETotal number of Provision groups

6 1



Secondary GSBPM 3.1 Build data collection instrument

None

Collocation Chapter 3. Chapter 3.

Number of identified Provisions



Methodologies 16 36

54

As previously, the two analyses are clearly very different. In this chapter, many of the same terms were identified, but at different levels. For example, many provisions identified by INSEE were identified as methodologies by ONS.


This comparison demonstrated that ONS and INSEE were using different approaches to identifying provisions and methodologies. Following discussion, the NSIs agreed that both approaches could be improved and worked collaboratively towards an agreed analysis.



Table 35. Summary of the final version of the first common chapter: Chapter 2




Secondary GSBPM 3.1. Build data collection instrument

Collocation Chapter 2Number of identified Provisions 1

0 statements1 recommendation0 instructions


Final version of the second common chapter

Table 36. Final version of the second common chapter: Chapter 3





Collocation Chapter 3

55

Number of identified Provisions 175 statements12 recommendations0 instructions




CHAPTER Number of Provision Groups

Main GSBPM Number of Identified Provisions

`CHAPTER 2

THE COGNITIVE MODEL IN ANSWERING QUESTIONS

(Common ONS & INSEE)

1. The Cognitive Response Process

2. Conceptual Frame

2.3 Design Data Collection Methodology

1 Recommendation

CHAPTER 3

QUESTIONNAIRE DESIGN

(Common ONS & INSEE)

3. Data Collection mode4. Define Variables and Draft a Tabulation Plan5. Principles for Writing Questions6.Literature Search

2.3 Design Data Collection Methodology3.1 Build Data Collection instrument1.1 Determine needs for information.

16 Recommendations


Entity Total Number ListMethodologies 11 CASI - Computer assisted self interviewing

Collect and review literatureConceptual and entity relationship schemeData collectionDealing with sensitive questionsFocus groupInterviewer assisted methodsNeed of respondent to appear in a certain wayRecall and memory errorsRespondent’s interpretationSelf interviewing

Methods 11 Output objectives

56

ConceptualisationCAPICASICATIPAPISensitivity effectsIn depth/qualitative researchSocial desirabilityRecall effectBuild data collection instruments

57

2.2.6. Survey Methods and Practices


The following table describes the handbook “Survey methods and practices” analysed jointly by ISTAT and ONS.


Attributes of the entity “normative document” Title of the document (of the handbook) Survey methods and practices Version: indication of the version, if available (otherwise, year of publishing)

2010


Code of practice


National


Statistics Canada

Consensus: if any, indication of the agreement Statistics Canada Aim: This manual is primarily a practical guide to

survey planning, design and implementation. It covers many of the issues related to survey taking and many of the basic methods that can be usefully incorporated into the design and implementation of a survey.

The handbook is composed of 13 chapters which have been analysed as reported in table 40 where the chapters in common are highlighted in bold .


Chapters analysed by ISTAT 4. Data Collection Methods 5. Questionnaire Design 6. Sample Designs 7. Estimation 8. Sample Size Determination and Allocation 10. Processing

58

Chapters analysed by ONS 1. Introduction to Surveys 2. Formulation of the Statement of Objectives3. Introduction to Survey Design4. Data Collection Methods 5. Questionnaire Design

Comparison of the common chapters and analysis of divergences

First common chapter: Chapter 4. Data collection

This chapter describes the process of gathering information in statistical surveys with particular attention to the different methods of data collection.

The standards identified by ISTAT and ONS in the chapter 4 are summarized in table 41.

Table 41. Summary

ISTAT ONS


1 1



Secondary GSBPM 3.1.Build data collection instrument


Collocation Chapter 4 Chapter 4Number of identified Provisions




A general comment is that ISTAT and ONS followed a similar approach: both defined a generic provision regarding data collection as a statement, but while ISTAT identified three methodologies (one for data collection in general, one for self-interviewing and one for interviewer assisted techniques) each linked to the related provisions, ONS identified the general methodology linked to all the provisions. As a matter of fact, as shown in the following table, ONS linked all the identified methods to the general methodology, while ISTAT distinguished the methods to be linked to each methodology.

59


ISTAT ONSProvision group 1 1Provision 5 4Collocation Page 39 Page 43Description Interviewer assisted methods are very

useful for survey populations with low literacy rates or when concepts or questionnaire are complex

In order to use an interview-assisted method sufficient budget must be available to pay for the training, hiring and travelling of interviewers

Type Recommendation instructionSequence None NoneMethodology Interviewer assisted methods Data collectionDescription The questionnaire is administrated by an

interviewerData collection is the process of gathering the required information for each selected unit in the survey

Quality dimension Timeliness, accuracy and efficiency Effectiveness and efficiency

Methods: - PAPI- CATI- CAPI

- PAPI- CATI- CAPI- CASI

Second common chapter: Chapter 5: Questionnaire design

This chapter describes the process of questionnaire design and other related aspects like the guidelines for wording of survey questions, layout of questionnaires, the management of response error in the interviewing phase, etc.

The standards individuated by ISTAT and ONS in the chapter 5 can be summarized in table 43.

Table 43. Summary for second common chapter

ISTAT ONSTotal number of Provision groups

6 1



Secondary GSBPM 3.1 Build data collection instrument


60

Collocation Chapter 5. Chapter 5.

Number of identified Provisions



Methodologies 1 2

As it can be seen, the main difference regards the number of identified Provisions Groups. As a matter of fact, ONS defined only one group (Questionnaire design), while ISTAT split the different aspects concerning the questionnaire design treated in the chapter in separated groups. Furthermore, ONS identified two methodologies, Questionnaire design and Questionnaire testing, while ISTAT did not consider Questionnaire design as a methodology itself.


As a general remark it is possible to conclude that the two approaches were quite similar and that the framework of the repository allowed not only to highlight the differences but also to reconcile them. Finally, an observation valid for both the chapters analysed in common is that ISTAT assigned the attribute “statement” in a smaller number of cases than ONS did.


The analysis of the common chapters led to the final version, combining the two analyses.

Regarding Chapter 4, the final version contains the provisions identified by both NSIs, together with all the other provisions defined separately by each. Three methodologies were chosen as this allows a clearer link to the methods.

With Chapter 5, the integrated version defines two provision groups; the first for Questionnaire design in general (in which the majority of provisions were grouped) and a second group regarding the Questionnaire design process, which corresponds to 7 provisions to be performed in the specified sequence. Provisions referring to concepts identified by both NSIs were maintained, but when pertaining to the same issue they were defined in a single provision.

In general, a number of attributes concerning the type of provision were discussed and sometimes consequently modified.


Table 44. Summary of the final version of the first common chapter: Chapter 4


61





1 statements3 recommendations4 instructions


Final version of the second common chapter

Table 45. Final version of the second common chapter: Chapter 5






10 statements26 recommendations27 instructions






CHAPTER 1 (by ONS)1- Introduction to Surveys


16 statements

CHAPTER 2 1- Formulation of the 1.1. Determine needs 2 statements

62

(by ONS ) Statement of Objectives for information 1 descriptive provision

CHAPTER 3(by ONS )

1- Introduction to Survey Design


18 statements

CHAPTER 4 (common )

1- Data Collection Methods


1 statement4 instructions3 recommendations

CHAPTER 5(common )

1- Questionnaire Design2- Questionnaire Design Process


10 statements27 instructions 26 recommendations

CHAPTER 6(by ISTAT )

1- Non-Probability Sampling2- Probability Sampling

2.4. Design frame and sample methodology

2 statements1 recommendation


1- Estimation 5.6. Calculate weights 3 statements


1- Sample size determination2- Sample allocation for stratified SRS designs


1 statement1 instruction4 recommendations


1- Data editing2- Data imputation3- Evaluation of imputation procedures4- Identification and treatment of outliers

5.3. Review, validate and edit5.4. Impute

10 statements17 instructions11 recommendations1 optional requirement

From the analysis of the considered chapters of the handbook it is possible to highlight that the first chapters (1 to 3) are mainly descriptive of survey design and, consequently, the majority of the provisions identified in these pages are statements. In constrast, the chapters covering data collection, questionnaire design, sample size and editing and imputation are more prescriptive, so a great number of recommendations and instructions were found. Finally, chapters 6 and 7 (about sampling designs and estimation) contain a small number of provisions, almost all defined as statements as these chapters are mainly a review of methodologies and methods.


Entity Total number ListMethodologies 19 Data collection

Self interviewingInterviewer assisted methodsNon-probability samplingProbability samplingDesign weightProduction of simple estimatesEstimating sampling error of survey estimatesSample size determinationAllocation criteriaAllocation methodsSelective editing

63

Deterministic imputationStochastic imputationFellegi/Holt frameworkImputation varianceMethods for outliers identificationMethods for outliers treatmentMethodologies to test questionnaires

Methods 77 Haphazard SamplingVolunteer SamplingJudgement SamplingQuota SamplingModified Probability SamplingSimple random samplingSystematic sampleProbability proportional to sizeWeighting for egual probability sample designsWeighting for unequal probability sample designsWeighting adjustment for nonresponse Using auxiliary information to adjust eights/Post-stratificationUsing Auxiliary Information to Adjust Weights/Ratio EstimationUsing Auxiliary Information to Adjust Weights/CalibrationUsing Auxiliary Information to Adjust Weights/Generalised regressionUsing Auxiliary Information to Adjust Weights/Raking Ratio EstimationUsing Auxiliary Information to Adjust Weights/Calibration FactorsEstimators for Different Types of Data/Qualitative and quantitative data- Population TotalsEstimators for Different Types of Data/Quantitative data- Population AverageEstimators for Different Types of Data/Qualitative data-Population averageEstimators for Different Types of Data/DomainsEstimators for Different Types of Data/Small domainsUsing auxiliary information to Adjust weights/OutliersSampling varianceStandard errorCoefficient of variationMargin of error and confidence intervalDesign effectEstimating the Sampling Variance using Replicated SamplingCluster samplingStratified Sampling (STR)Multi-Stage SamplingMulti-Phase SamplingReplicated SamplingSpecial Topics in Sample Design/Repeated SurveysSpecial Topics in Sample Design/Entry-Exit SurveysSpecial Topics in Sample Design/Snowball SamplingAllocation with fixed sample sizeAllocation with fixed coefficient of variationProportional allocationY- proportional allocationAllocation proportional to square-root of NAllocation proportional to square-root of YOptimum allocation

64

Neyman allocationOptimum allocation with equal variancesTop-down approach for selective editingAggregate MethodGraphical MethodQuestionnaire Score Method (Berthelot and Latouche,1992)Deductive imputationDeterministic Mean Value ImputationDeterministic Ratio/Regression ImputationSequential Hot-Deck ImputationSequential Cold-Deck ImputationNearest-Neighbour ImputationStochastic Ratio/Regression ImputationRandom Hot-Deck ImputationFellegi-Holt error localisationFellegi-Holt imputation methodMultiple imputationRelative distance from the centre of the dataQuartile methodWinsorisationChange the weightRobust estimatorsPAPICASICATICAPIInformal testingCognitive methodsFocus groupInterviewer debriefingsBehaviour coding of interviewer/respondent interactionsSplit sample testsPilot test

It-tools 0Classifications 0

Definitions 0

65

2.3. Description of the unified repository for methodological standards

2.3.1. General description of the repository

The analysis of the unified repository has been carried out through the construction of a set of descriptive tables with the aim to highlight differences and peculiarities of the six handbooks (HB in the following) examined. In these tables the handbooks have been analysed through some synthetic information on the number of identified entities, such as provision groups, provisions, methodologies and listing the statistical activities and the quality dimensions . A further step has been to list all the combination of GSBPM sub-processes indicated for all HB all together, considering the repository as a unique DB.

An important general remark is that the observations deriving from these tables can provide inevitably only a partial view because not all the HBs have been examined thoroughly; nevertheless this overview is useful to highlight some possible deficiencies of the analysis so far carried out, with the aim to improve the future work on the repository.

The first two tables focus on the analysis of provisions and methodologies for each HB, considered in terms of number and type.

Table 1. Counts of entities individuated for each handbookHandbooks # of

Provision Groups

# of Provisions

# of Methodologies

# of Methodsologies

# of IT tools

# of Classifications

# of Definitions

Sampling guidelines 2 5 4 20 3 0 0

EDIMBUS 910 356 9 2930 0 2 0

Guidelines for Statistical Metadata

3 20 0 0 0 0 2

Quality Report 10 197 15 36 0 6 12

Questionnaire Design and Testing

6 17 14 13 0 0 0

Survey Methods and Practices

15 161 19 76 0 0 0

TOTAL 25 258 28 86 3 8 14

Table 2. Distribution of provisions by type for each handbook Type of provision Total # of

Handbooks # of # of # of # of provisions statements instructions recommendations other types Sampling guidelines 0 0 5 0 5

EDIMBUS 101 11 14 0 356

Guidelines for Statistical Metadata 5 1 11 3

20

Quality Report 105 91 1 197

Questionnaire Design and Testing 0 0 17 0

17

Survey Methods and 64 48 47 2 161

66

PracticesTOTAL 121 12 121 4 258

A first consideration regards the total number of provision groups (PG) for each HB (Table 1): generally this number is higher for the multi-topic HB, such as Survey Methods and Practices (SMP), with respect to the HB devoted to specific methodological topic. The same consideration can be done for the number of provisions and methods. It is necessary to underline that for Quality Report (QR) a high number of entities has been indicated, even though only one chapter has been analysed. Another observation regards the relation between the number of PGs and Methodologies on one side and provisions and links to methods on the other side. The analysis of the provision types (Table 2) highlights, in general, a prevalence of statements and recommendations, with a substantial balance between them. Also in this context can be underlined a relation between the nature of the HB (multi-topic or specific) and the prevalence of some types of provision (statement or recommendation)

As far as the quality dimensions are concerned, when analysing Table 3 the prevalent indicated dimensions are accuracy and efficiency: in general accuracy is indicated for the 89% of the provisions, with a small variability among HBs, with the exception of Metadata, while efficiency is indicated for the 27.8% of the provisions but with a high variability among HBs. It is worthwhile to note that for QR only accuracy has been analysed.

Table 3a. Percentage of provisions for quality dimensions for each handbook (product)

Percentage of provisions for dimension of quality (product)

Handbooks % of % of % of % of % of relevance accuracy timeliness accessibility coherence

Sampling guidelines 0.0 80.0 0.0 0.0 0.0

EDIMBUS 0.0 100.0 36.1 0.0 72.2


0.0 0.0 5.0 45.0 10.0

Quality Report 0.5 99.0 0.0 0.0 0.0


41.2 88.2 5.9 0.0 17.6


19.9 87.6 16.1 10.6 14.3

TOTAL 9.2 89.7 9.4 6.0 12.4

Table 3b. Percentage of provisions for quality dimensions for each handbook (process)

Handbooks Percentage of indicated dimensions of quality (process)

% of effectiveness % of efficiency

67

Sampling guidelines 20.0 60.0

EDIMBUS 63.9 77.8

Guidelines for Statistical Metadata 20.0 20.0

Quality Report 0.0 0.0

Questionnaire Design and Testing 0.0 35.3Survey Methods and Practices 30.4 58.4

TOTAL 17.7 31.0

From the analysis of Table 4, where the indicated statistical activities are listed, it is possible to underline that the multi-topic HB refer in general to all statistical activities, while the specific ones, as Edimbus, refer to specific activities both for the survey context and for the survey phase. Moreover, QR indicated all statistical activities but also specific contexts, highlighting that peculiar quality indicators exist for specific survey contexts.

Table 4. List of statistical activities for handbook

Handbook LIST of distinct statistical activitiesEDIMBUS All statistical subjects (D1+D2+D3)

D2 Economic statistics

2.3 Business statistics

4.4 Data editing and data linkage

4.7 Data analysis



4.5 Dissemination, data warehousing

5.5 Management and development of technological resources (including standards for electronic data exchange and data sharing)

Quality report All statistical subjects (D1+D2+D3)

D1 Demographic and social statistics

D2 Economic statistics

2.1 Macroeconomic statistics

2.2 Economic accounts

2.3 Business statistics

2.4 Sectoral statistics

2.4.1 Agriculture, forestry, fisheries

2.4.2 Energy

2.4.3 Mining, manufacturing, construction

2.4.4 Transport

2.4.5 Tourism

2.4.6 Banking, insurance, financial statistics

2.5 Government finance, fiscal and public sector statistics

2.6 International trade and balance of payments

intranet

68

2.8 Labour cost

2.9 Science, technology and innovation

D3 Environment and multi-domain statistics

3.1 Environment


All statistical subjects (D1+D2+D3)

D4 Methodology of data collection, processing, dissemination and analysis


With regards to the GSBPM model, in Table 5 are listed the GSBPM sub-processes indicated as the main ones. Even though the funding of this list depends on the degree of completeness of the analysis conducted for each HB, as a general consideration, the multi-topic HB cover quite a high number of sub-processes, while the other ones a more limited set. In Table 6 it is possible analyse the combinations of sub-processes indicated in the integrated repository with respect to the number of GPs, provisions, methodologies and methods.

Table 5. List of GSBPM procesess for handbook

Handbooks LIST of distinct GSBPM sub-process

Sampling guidelines


4.1. Select sample

5.4. Impute

5.6. Calculate weights

EDIMBUS

5.3. Review, validate and edit

5.4. Impute

6.3. Scrutinize and explain


7.2. Produce dissemination products

7.3. Manage release of dissemination products

7.4. Promote dissemination products

Quality report



2.5. Design statistical processing methodology

4.1. Select sample

10.1. Quality Management


1.1. Determine needs for information

1.5. Check data availability





1.2. Consult and confirm needs

1.3. Establish output objectives

2.2. Design variable descriptions

69




4.1. Select sample


5.4. Impute

5.6. Calculate weights

10.1. Quality Management

70

Table 6. Number of entities for combinations of GSBPM processes in the integrated repositoryGSBPM sub-process combination # of # of # of # of # of # of # of

Main GSBPM Auxiliary GSBPM Additional GSBPM Groups Provisions Methodologies Methods IT tools

classifications definitions


1.2. Consult and confirm needs

1.3. Establish output objectives

1 3


1.5. Check data availability 1 1 1 1


2 34


2.2 Design variable descriptions


2 63 1 7



3 12 6 6


2 7 3 9


4.1. Select sample 2 3 2 16


4.1. Select sample 5.6. Calculate weights 1 3 3 14 2


2 6 5 5



1 6 5 5


7 48 6 18 2


5.4. Impute 5.6. Calculate weights 1 4 2 5

5.4. Impute 3 19 5 19

5.4. Impute 10.1. Quality Management 1 3 1 1

5.4. Impute 5.6. Calculate weights 4.1. Select sample 1 2 1 6 5

5.6. Calculate weights 1 3 3 21

6.3. Scrutinize and explain 2 2 2 4

7.2. Produce dissemination products

7.3. Manage release of dissemination products


1 6


2 14 2

10.1. Quality Management 7 96 8 19 1 11

10.1. Quality Management 2.4. Design frame and sample methodology


1 14 2 2 2


2.5. Design statistical processing methodology

1 8 1 1


4.1. Select sample 1 79 10 30 2

TOTAL 46 436 66 188 7 8 14

72

2.3.2. Examples of compared analyses of standards derived from different handbooks for the same GSBPM sub-processes

With a unique repository containing the results of the analysis carried out on different methodological handbooks, as the one obtained by following the procedure described in par. 2.1.3, it is possible to compare, among other things, all the objects pertaining to a given GSBPM.In particular, it is possible to compare the provisions, that are the elementary items that compose a standard, and all related objects (methodologies, methods, IT tools, definitions and classifications).What is the utility of such a comparison?Let us consider the definition of standard, based on the ISO approach, that has been given in par.1.3.1 (“A standard is a document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.”). Accordingly to this definition, it is clear the methodological handbooks that have been taken into consideration cannot be considered as “standards”, at least for the whole European Statistical System. They are rather “normative documents” of the kind “code of practice”, usually valid locally inside the body that produced them.In case an initiative to produce statistical standards valid for the whole European Statistical System should take place, a recognized body could be instituted, with the aim of defining these standard for each important step of a statistical process, i.e. for each GSBPM sub-process. This body could take into account the most important methodological handbooks currently used inside the National Statistical Institutes and the other international statistical organisations, as a good start for the process of definition of the standards.Each handbook could be analysed following the same procedure adopted here, thus populating a repository with the structure here proposed. Then, for a given GSBPM sub-process, the provisions originated by the different handbooks could be evaluated in order to decide which of them could be promoted to become part of the final “standard” for that sub-process.In the following, we report examples of comparison, one related to “samplingsample selection”, the other one to “editing”.

Case 1: GSBPM sub-process ”4.1. Select sample”

In this comparison, we analyse all relevant objects (from the author’s point of view) in the unified repository, which themselves are the result of the analyses of three methodological handbooks: “ESS Handbook for Quality Reports”, “Survey Methods and Practices” and “Survey Sampling and Reference Guidelines, Introduction to sample design and estimation techniques”.

First, we compare the different (relevant) groups of provision that have been defined from the three handbooks:

ESS Handbook for Quality Reports


Survey Sampling and Reference Guidelines

SamplingNon-probability sampling Probability Sampling

Probability sampling Sample allocation to strata

In fact, there had been defined more Provision groups for the three handbooks (see the analyses within the previous chapters), but these had not been considered here. The reason for this is that GSBPM sub-process 4.1 was only mentioned as secondary GSBPM sub-process and the contained provisions were not considered as relevant in this context.

Now, we consider the descriptions of the different provisions, separately for the three handbooks. In our case of “sample selection”, the provisions may be stated in the general form: “The statistician should apply one method among a list of recommended methods”. However, the descriptions of the provisions in the unified repository, elaborated by different NSIs, do not really reflect this. The formulations appear to be misleading to some extend. Nevertheless, they are listed here:

ESS Handbook for Quality Reports GROUP OF

PROVISIONS PROVISIONS

Sampling

When non-probability sampling is applied, random error can not be estimated without reference to a model of some kind. Furthermore, sampling biases may well be significant and need to be assessed as well. Some further technical points concerning the presentation of sampling errors are: 1) Non-response should be taken into account, i.e., the sample size should be the effective sample, after deduction of non-response; 2) The original stratification should be appropriate.One type of non-probability sampling that is frequently applied in economic surveys and therefore needs special attention is the use of a cut-off threshold.

74


GROUP OF PROVISIONS PROVISIONS

Non-probability sampling

1. Non-probability sampling is a method of selecting units from a population using a subjective (i.e., non random) method.

2. Non-probability sampling can be applied to studies that are used as preliminary steps or follow-up steps.

Probability sampling

Probability sampling is a method of sampling that allows inferences to be made about the population based on observations from a sample. In order to be able to make inferences, the sample should not be subject to selection bias. Probability sampling avoid.

Survey Sampling and Reference Guidelines


Probability Sampling

Sample selection: process of selecting the sampled units from the frame.

Sample allocation to strata


Next, we consider methodologies and methods:

ESS Handbook for Quality Reports Survey Methods and Practices Survey Sampling and

Reference GuidelinesMethodologies Methods Methodologies Methods Methodologies Methods


Cut-off sampling

Non-Probability Sampling

Modified Probability Sampling


Probability-Proportional-

to-Size (PPS)

SamplingOther

forms of non-

probability sampling

Volunteer Sampling

Stratified Sampling

(STR)

Judgement Sampling

Cluster Sampling

(CLU)Quota Sampling Simple

Random Sampling

(SRS)

75

Special Topics in Sample

Design/Snowball Sampling

Systematic Sampling

(SYS)

Modified Probability Sampling

Sample allocation to

strata

Power Allocation (Bankier)

Haphazard Sampling

Optimal Allocation (Neyman)


Cluster sampling Proportional Allocation

Systematic sample

Equal Allocation

Probability proportional to

sizeReplicated Sampling

Simple random sampling


Design/Entry-Exit Surveys


Design/Repeated Surveys

Multi-Stage SamplingStratified

Sampling (STR)Multi-Phase

Sampling

At this level, one can identify overlaps of methods. In the unified repository, multiply mentioned methods and methodologies should be eliminated. The following table gives an example of such a consolidation:

76

Methodologies Methods


Probability-Proportional-to-Size (PPS) SamplingStratified Sampling (STR)Cluster Sampling (CLU)Simple Random Sampling (SRS)Systematic Sampling (SYS)Replicated SamplingSpecial Topics in Sample Design/Entry-Exit SurveysSpecial Topics in Sample Design/Repeated SurveysMulti-Stage SamplingMulti-Phase Sampling


Cut-off samplingModified Probability SamplingVolunteer SamplingJudgement SamplingQuota SamplingSpecial Topics in Sample Design/Snowball Sampling


Power Allocation (Bankier)Optimal Allocation (Neyman)Proportional AllocationEqual Allocation

77

Case 1: GSBPM sub-processes “2.4. Design frame and sample methodology”,” 4.1. Select sample”,” 5.6. Calculate weights”

78

Case 2: GSBPM sub-process “5.3. Review, validate and edit “

In this comparison, we analyse all objects in the repository that are:o the result of the analyses of two methodological handbooks: “Survey Methods and Practices”

and “Recommended practices for Editing and Imputation in Cross-Sectional Business Surveys (EDIMBUS)”;

o related, directly or indirectly, to the main GSBPM sub-process “5.3. Review, validate and edit”.

First, we compare the different groups of provision that have been defined from the two handbooks:

SURVEY METHODS AND PRACTICE EDIMBUS

Data editing

Detection of errorsMissing valuesSystematic errorsRandom errors

Identification and treatment of outliers Influential errorsOutlier

Clearly, there is a more compact way of defining provision groups for “Survey Methods and Practices” than for “EDIMBUS”, but is possible to establish a clear correspondence between the two sets of groups.Now, we consider the different provisions, distinctly for the two handbooks.

SURVEY METHODS AND PRACTICE


Data editing

1.Editing should be performed at several stages of the survey.

2. The purpose of editing is to: better understand the survey processes and the survey data; detect erroneous or missing data; follow-up with the respondent; send a record to imputation; delete a record.3. Editing should be used to provide information about the survey process, either in the form of quality measures for the current survey or to suggest improvements for future surveys.4. When starting a survey, some assumptions are made about the data. During editing, it is possible to test the validity of these assumptions. For example, it may become obvious that some range edits were too strict or that some sequencing edits failed too f5. Editing is the application of checks to identify missing, invalid or inconsistent entries that point to data records that are potentially in error.6. Edits applied throughout collection and processing should be consistent with each other

7. Editing can be automated by means of a computer program.

8. Edits should be developed by staff who have expertise in the subject matter, questionnaire design, data analysis and with other similar surveys.9. There are three main categories of edits: validity, consistency and distribution edits.

10. Validity edits verify the syntax of responses and check that the coded data lie within an allowed range of values.

11. Consistency edits verify that relationships between questions are respected.

79

12. Distribution edits attempt to identify records that are outliers with respect to the distribution of the data.13. Information on the types of edits performed and the impact of editing on the survey data should be communicated to users.14. Quality assurance and quality control procedures should be applied to minimise and correct errors introduced during editing15. Selective editing practices are recommended, particularly for business surveys (i.e., where the population is skewed and a few businesses dominate the estimates).16. Follow-up is generally limited to edits failures identified during collection or arising from selective editing

Identification and treatment of outliers

17. An outlier is an observation or subset of observations that appears to be inconsistent with the remainder of the dataset18. Outliers are detected by measuring their distance from the centre of data

19. Outliers detected at the editing stage of the survey process can be treated in various ways.

20. The goal of outlier treatment is to decrease the impact that the outlier has on the sampling variance of the estimate without introducing too much bias

EDIMBUS


Detection of errors

1. Detection of errors consists of identifying values that are not acceptable with respect to some pre-defined logical, mathematical or statistical criteria. Since different error types may contaminate the observed data in a sample of units, usually the error detection process consists of a set of error detection methods dealing each with a specific type of error

Missing values

2. Missing values stem from questions the respondent did not answer. Nonresponse can be due to several reasons; the respondent may not know the answer, may not be willing to respond or may have simply missed a question.3. Replacing missing values by zero is neither an acceptable imputation procedure nor an acceptable alternative to flagging missing values

4. Variables and cases with many missing values should be studied and a decision on the variables and observations to be imputed should be taken in view of the amount of missingness. It may be necessary to discard certain variables or certain cases from the 5. Appropriate indicators on missing values should be calculated

6. The indicators should be analyzed to gain information on nonresponse mechanisms

Systematic errors

7. A systematic error is an error that is reported consistently over time by responding units. It is a phenomenon caused either by the consistent misunderstanding of a question during the collection of data, or by consistent misinterpretation of certain answers in the course of coding8. Systematic errors should be detected and treated before dealing with random errors, in particular when the Fellegi-Holt method is used and before selective editing

9. The analysis of indicators on edits helps to find systematic errors mechanisms. If systematic error mechanisms are found by examining edits, then appropriate deterministic checking rules to detect errors due to the systematic error mechanism should be add10. If systematic error mechanisms are found, then improvements to the survey process (Questionnaire, interviewer training, coding, processing) should be made to prevent similar errors

Influential errors

11. Influential errors are errors in values of variables that have a significant influence on publication target statistics for those variables.12. Selective editing is an appropriate method to focus attention on critical observations without generating a detrimental impact on data quality

80

13. Priorities should be reflected in the score functions of selective editing. Score functions should include a risk and an influence component

14. The thresholds of score functions should be chosen carefully during the tuning and testing of the E&I process. The thresholds and parameters of score functions should be revised whenever the survey process is changed (Questionnaire, data entry, removal of systematic errors)15. The quality of the anticipated value should be assessed at least for a sub-sample of the survey. E&I flags have to be taken into account if data of earlier periods are used as anticipated values

15. Units, for which the score function cannot be calculated have to belong to the critical stream to prevent biasing the estimates

17. Macroediting should be properly documented

18. Important publication aggregates, publications cells and publication sub-populations should be considered in macroediting

19. The quality of reference data and problems like inflation and structural differences (e.g. definitions) should be taken into account

20. Macroediting should be performed before releasing data for final estimation.

Outliers

21. An outlier is an observation which is not fitted well by a model. The model can be a parametric distribution or the model can be a more loosely defined concept like "close to the center of the data". In the latter case an outlier is an observation which is not close to the center of the data22. Outliers and influential observations should be detected. Such observations may be errors or correct but sometimes their correctness is unclear. Errors should be treated and also correct or unclear influential or outlying observations may have to be treated and also correct or unclear influential or outlying observations may have to be treated to prevent potential bias and high variability23. The influence on important results should be controlled even after selective editing and outlier detection.24. Outlier detection methods should be robust against outliers. Therefore methods based on means or weighted means and standard deviations, which are not robust, should be avoided

25. Simple univariate methods, graphical displays or more complex multivariate techniques should be used for outlier detection depending on possible models among variables. Different models may have to be applied in different sub-populations26. When choosing tuning constants for detection or treatment several tuning constants should be tested and the corresponding impact on estimates and their variances observed. Often the treatment of a few outliers is acceptable.

Random errors

27. Random errors are errors that are not caused by a systematic reason, but by accident. They primarily arise due to in-attention by respondents, interviewers and other processing staff during the various phases of the survey cycle.28. The data in each record should be made to satisfy all edits by changing the fewest possible items of data (fields)29. As far as possible the frequency structure of the data file should be maintained

30. Imputation rules should be derived from the corresponding edit rules without explicit specification

31. If appropriate software is available, random errors should be detected and treated by applying the Fellegi-Holt paradigm

32. If appropriate software for applying the Fellegi-Holt paradigm is not available, random errors may be detected by means of deterministic checking rules. However, the error localization becomes more arbitrary with deterministic checking rules

81

Survey Methods and Practices (SMP) counts 20 provisions, while EDIMBUS contains 32 provisions dealing with sub-process 5.3. It is not surprising, as EDIMBUS is dedicated to the edit and imputation steps of surveys.

By comparing the two sets of provisions, we can classify them in:1. those dealing with the same arguments, and that could be merged (having verified they are not

contradictory): for instance, “selective editing” (provision # 15 in SMP, and provision # 12 in EDIMBUS): these provisions may be potentially merged (“Selective editing practices are recommended, particularly for business surveys (i.e., where the population is skewed and a few businesses dominate the estimates), as they are appropriate methods to focus attention on critical observations without generating a detrimental impact on data quality”);

2. those dealing with the same argument, but not compatible: for instance, provision # 17 in SMP and # 21 in EDIMBUS. They are both statements defining outliers: even they are both correct, the first may be considered too generic with respect to the second, that should be adopted;

3. non overlapping provisions: for instance, provisions from # 1 to # 14 in SMP are more general than those in EDIMBUS; or provisions from #7 to # 10 do not have correspondence in SMP. In this case, once verified their validity, they could be all promoted to be part of a new standard for editing.

Let us now consider methodologies and methods:

Survey Methods and Practices EDIMBUSMethodologies Methods Methodologies Methods

Selective editing

Questionnaire Score Method(Berthelot-Latouche)

Detection of missingvalues

Graphical method

Detection of systematicerrors

Analysis of fataledits

Top-down Analysis of ratioedits

Aggregate method Finite mixturemodels

Methods for outliers identification

Relative distance from the centre of data

Detection of influential errors

Macroediting

Quartile method Selective editing

Detection of outliers

Hidiroglou-BerthelotmethodMedian absolute deviationruleRegression-based modelsGraphical editingOne-step robustified ratio estimatorWeighted quantilesWinsorised mean and trimmed mean

Localising random errors

Deterministic checking rulesFellegi-Holt paradigm

82

Once again, there is a higher number of methodologies and methods defined in EDIMBUS than in SMP.Despite the different naming adopted in the two handbooks, it is possible to establish some equivalences between the defined objects:- “Selective editing” methodology in SMP corresponds to “Detection of influential errors”

methodology, but also to “Selective editing” method, in EDIMBUS;- “Methods for outliers identification” in SMP corresponds to “Detection of outliers” in

EDIMBUS;- the couple “Top-down” and “Aggregate method” methods in SMP correspond to

“Macroediting” in EDIMBUS;- “Graphical method” in SMP corresponds to “Graphical editing” in EDIMBUS (but the former

is linked to the “Selective editing” methodology, while the latter to the “Detection of outliers” methodology).

As an example of integration, we propose to consider the union of all methodologies and methods, with the necessary re-classification in order to avoid inconsistencies (for instance, “Selective editing” should be considered as a methodology, making use of different methods).In the following table a possible integrated solution is reported. This unified list of methodologies and methods could be the starting point for the integration also of the provisions and provision groups, when constructing the final “standard” for this particular sub-process.

Methodologies MethodsDetection of missingvaluesDetection of systematicerrors

Analysis of fatal editsAnalysis of ratio editsFinite mixture models

Detection of influential errors Top-downAggregate method

Selective editing Questionnaire Score Method (Berthelot-Latouche)

Graphical method

Detection of outliers

Hidiroglou-Berthelot methodMedian absolute deviation ruleRegression-based modelsGraphical editingOne-step robustified ratio estimatorWeighted quantilesWinsorised mean and trimmed mean

Localising random errors

Deterministic checking rules

Fellegi-Holt paradigm

83

3. Review of support measures

3.1. Introduction

Application of standards is either compulsory or voluntary. Communication and several support tools are necessary to promote and assist implementation in both cases.

3.2. Support measures related to standards

Implementation can be based only on the excellence of a standard and on the benefits its application provides. This can be observed when a local practice – available in an adequate document – becomes a practical standard implemented by a number of partners without any further external pressure. This was the case of Quality guidelines of Statistics Canada13. Benefits of a standard will get through if it is widely used; the number of users is relatively high. This is why speed is important, hence support measures make sense.

Support measures are different types of activities, connected to standards implementation. Support measures refer to all necessary information, modes and tools for access and special regulations providing scope in better understanding, delivery and application of standards.

Support measures are intended to accelerate, facilitate and smoothen the implementation of standards. They intend to bridge the gap between the optimal expected environment and the local, special environment of implementation. Support measures are “soft” tools, with great variability, optimally adapted to the intended user environments. This is the reason why support measures are not integral part of a standard, however contribute to the realisation of standard development principles14: transparency and openness. In practice, at the moment when a standard is accepted there are at least plans on a set of support measures on the implementation or bodies assigned to be involved in supporting activities. In case of European Standards, CEN (European Committee for Standardisation) manages several regional and national Technical Assistance programmes financed by the European Union. The assistance aims to build a quality infrastructure (standardization, testing and metrology, conformity assessment and certification) in beneficiary countries15. In the field of statistics the general practice of UN Statistical Commission is to take care of support measures, when adapting a new standard. Concerning classifications, as traditional field of international standards, this role is already an agreed rule. “The role of the Statistical Commission and other intergovernmental boards in the area of classifications is to: (a) determine short and medium term objectives for setting standards; (b) work toward harmonisation of existing standards and practices; (c) promote best practices; (d) implement data collection and dissemination strategies that incorporate international classification agreements; and (e) monitor the use of classifications.”, states the UNSD document16.

13 Quality Guidelines, Statistics Canada, Fifth Edition – October 2009 http://www.statcan.gc.ca/pub/12-539-x/12-539-x2009001-eng.pdf14 The ABC’s of Standards Activities, National Institute of Standards and Technology, Gaithersburg, August 2009http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf 15 Compass CEN The World of European Standards ftp://ftp.cen.eu/cen/AboutUs/Publications/Compass.pdf

84

ftp://ftp.cen.eu/cen/AboutUs/Publications/Compass.pdf

http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf%20

http://www.statcan.gc.ca/pub/12-539-x/12-539-x2009001-eng.pdf

The practices of Eurostat are similar when grants, workshops, training courses, coaching, handbooks, collection of best practices are offered to support implementation of new or modified standards.

3.3. Overview of support measures

The direct influence is the motivation using the tools of awarding and sanctioning. The motivation can be cheap and effective if the standard is known and the implementation is easy. Providing support during implementation may help to guarantee success, learning more on standard itself, avoid misuse and fails. In the following we overview all of these types of support measures with examples, although part of them is not yet used for statistical standards.

3.3.1. Motivation for implementation

The decision to use different types of motivation – and in accordance with the type of conformity assessment – can be linked to the level of risk associated with the product/service and the customer requirements. This is not only true in the case of European market but we can observe it in statistics too. We can think about the problems and regulatory consequences of the falsified EDP report (Excessive Deficit Procedure and the financial crisis).

Legal instruments, sanctions:

The strictest motivation is obligation with legal instruments. European examples are the „New Approach” laws and directives in EU standardization process requirement for obtaining CE mark to gain access to European market17. In the field of statistics, an example is the standard for EU national accounts. In the near future: ESA (standard for EU national accounts) methods will be obligatory to be used for calculations of EDP indicators18. The classifications are de jure standards, as usually regulations prescribe their use in the EU, for example NACE Rev.2 and CPA 2008. These regulations state/determine exactly from when and where (in which statistical domains) classifications should be applied. In the field of metadata, there is only one legal instrument available, the 2009/498/EC Commission Recommendation of 23 June 2009 on reference metadata19

for the European Statistical System. This recommendation covers the basic concepts and the structure of the so called ESMS (Euro-SMDX) files (metadata), which have to be sent in line with the corresponding statistical data. This recommendation takes the SDMX standard into consideration, since it might be a world wide standard for metadata and metadata exchange.

At the META Working Group held in Luxembourg in 2011, a new proposal was made for legislation on metadata and metadata exchange – as a cross-cut legislation – in order to make the metadata of Member States more harmonised. Supposedly those instruments will be regulations, therefore compulsory for every Member State. 16 International Standard Classifications: A Draft Statement of Best Practices. http://unstats.un.org/unsd/class/intercop/expertgroup/1997/AC60-15.PDF17 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1998L0034:20070101:EN:PDFDIRECTIVE 98/34/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 22 June 1998 laying down a procedure for the provision of information in the field of technical standards and regulations

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:1998:217:0018:0018:EN:PDFDIRECTIVE 98/48/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 20 July 1998 amending Directive 98/34/EC laying down a procedure for the provision of information in the field of technical standards and regulations18 About sanctions see: COM(2010)0522 Proposal for a COUNCIL REGULATION (EU) No …/… amending Regulation (EC) No 1467/97 on speeding up and clarifying the implementation of the excessive deficit procedure Article 11 and article 12 (page15)19 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:168:0050:0055:EN:PDF

85

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:168:0050:0055:EN:PDF


http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1998L0034:20070101:EN:PDF

Legislation can support implementation of standards not only by obligation but by making reference too. An example from CEN-related activities: “In addition many standards are developed to support European legislation. ‘Reference to standards’ within a legislative text is viewed as a more effective means of ensuring that products meet the essential health and safety requirements of legislation than writing of detailed laws. This allows both processes to support each other, without causing a slowdown.”20 We can find similar examples in statistics: in some cases there is no legal instrument which makes the application obligatory, but according to legislation a set of permissions are necessary preconditions to start certain activities, and the scope is prescribed by referring NACE categories. Awards Beyond obligation and sanctions, positive approach can be useful with rewarding. In the fields, where no obligation is possible, motivation can be supported by giving awards, like international and national quality awards built on quality management standards (ISO 9001, EFQM). Examples: The European Quality Award21, Malcolm Baldrige National Quality Award22, The Deming Prize23.

CommunicationKnowing about standards is a precondition for the decision on implementation. Intended users and their partners should be aware of the existing and forthcoming standards, therefore communication and promotion are crucial, even if implementation is supported by regulations or awards.

3.3.2. Making the implementation easy

Implementation of a standard is easier if it is available, understandable, efficient and consistent and is accompanied by proper training activities.

Availability

Easy availability is a characteristic of standards, however it does not mean the standards are free of charge, especially if they are linked with patterns. Easy availability means free access, downloadable documentation in the case of statistical standards, such as statistical classifications and methodological handbooks (RAMON24 on Eurostat homepage, Methods and classifications on UN homepage25). SDMX standards are downloadable from the SDMX website (www.sdmx.org\standards).

A repository of standards with built-in search features, containing not only actual but different versions could improve availability of standards for potential users. Generally, a “responsible body” has a repository of its own standards (accepted by the body). In general, standards relevant for a unit are originated from different responsible bodies. Potential users are interested in a repository, where all relevant standards are available. In the EU, an NSI has to be aware of the EU, UN, UN-ECE, OECD, IMF, ILO and some other bodies’ statistical standards.

Understandability

20 Compass CEN The World of European Standards21 Established in 1992 and offered by the European Foundation for Quality Managementhttp://www.efqm.org/en/22 Established in 1987 and offered by the National Institute of Standards and Technology in the USA http://www.baldrigepe.org/foundation/23 Established 1951 (International from 1984) and offered by the Japanese Union of Scientists and Engineershttp://www.juse.or.jp/e/deming/24 http://ec.europa.eu/eurostat/ramon/index.cfm?TargetUrl=DSP_PUB_WELC25 http://unstats.un.org/unsd/methods.htm

86

http://www.juse.or.jp/e/deming/

http://www.baldrigepe.org/foundation/

http://www.efqm.org/en/

Understandability can be achieved by using adequate language (plain English, local), by providing additional materials different by length and detail, according to the needs of the intended users (managers, experts, IT people). Example on different complementary, promotion materials are those of the CEN: documents, compass-leaflet, articles, homepage26. Besides European standards, CEN produces other reference documents like: workshop agreements, technical specifications, technical reports and guides.

Another good example for useful support tool is a glossary – definitions specify how the terms are to be understood in the context of the given standard – even if it is not necessarily exhaustive.

Efficiency

The technical implementation may be hindered by the costs of developing tools to implement, to fit in the local IT environment. This burden can be reduced by developing interfaces, IT tools, application methods.

In the field of statistics, the implementation of the recommendations on seasonal adjustment is supported by Demetra, a supporting IT tool, downloadable from Eurostat website, fit to implement recommendations. In the case of SDMX standard, there are guidelines and software downloadable from the SDMX website (www.sdmx.org\standards, and www.sdmx.org\guidelines). These software solutions can be used for the implementation of the SDMX standard for metadata exchange. In order to understand the basic principles and concepts of the standards, guidelines are also available. This handout is the so called Content–oriented guidelines.

At European level, applications are also available, such as the National Reference Metadata Editor (NRME) that can be used by Member States for the production and transmission of metadata files. This is a web application, but there are also other software solutions for utilisation. These can be used for mapping the national metadata to the SDMX concepts, classifications and their items, database table columns, etc.

Consistency

In any business, a number of standards are applied and they have to be able to work as a consistent system. They have to communicate with each other in an efficient, cost effective way. For users, it is more convenient if the connecting standards have mutual recognition, and some kind of complementarities can be expected among them. The consistency of European standards and national standards is assured in the following way: after ratified by CEN, each of the National Standard Bodies adopts the European standard as an identical national standard and withdraws any national standards that conflict with the new European standard.27. Other approach is the case – generally and typically in statistics – when global standards do not go into details and local stakeholders are encouraged to develop additional, more detailed standards or guidelines on their specific activities28. Other examples in statistics are: the derived classification the General Industrial Classification of Economic Activities within the European Communities (NACE) based upon ISIC, the International Standard Industrial Classification of All Economic Activities adopted as UN standard, and Member States are allowed to adopt a national version of NACE. Similar relation links SNA as UN standard and ESA as Eurostat standard. The standards with more restricted

26 http://www.cen.eu/cen/AboutUs/Pages/default.aspx

27 COMPASS Cen The world of European Standards28 Brian A. Harris-Kojetin – Katherine Wallman (2009): Revision and Implementation of Standards for Statistical Surveys for the U. S. Federal Statistical System. http://www.scb.se/Grupp/Produkter_Tjanster/Kurser/ModernisationWorkshop/Final/B_1_QA_systems_Harris-Kojetin.pdf

87

http://www.scb.se/Grupp/Produkter_Tjanster/Kurser/ModernisationWorkshop/Final/B_1_QA_systems_Harris-Kojetin.pdf

http://www.scb.se/Grupp/Produkter_Tjanster/Kurser/ModernisationWorkshop/Final/B_1_QA_systems_Harris-Kojetin.pdf

http://www.cen.eu/cen/AboutUs/Pages/default.aspx

geographical scope are more detailed and more concrete. Generally speaking, the more homogenous the area of application the more detailed and concrete the standard can be.

As conformity analysis of the different standards can be very expensive, users can be supported with information on their conformity. This is included as a requirement in the Policy of standards of Statistics Canada: “A standard shall be accompanied by a statement of conformity to relevant internationally recognized standards or a description of the deviations from such a standard and, where possible, a concordance with the referenced standard.”29 In the case of classifications, the corresponding tables are intended to be used as a key between different classifications, but in the same time they also provide information on their conformity.

Training

Training provides organised support for understanding and implementation of standards. Examples in statistics are the ESTP training courses related to methodological standards, however, ESTP courses make few references to methodological handbooks.

Examples: There are several workshops (e.g.: METIS workshops) in case of SDMX standards. In 2011, it was held in Washington D.C. In the Conference, SDMX sponsors expressed their intentions „to continue providing training and learning opportunities, also supporting for SDMX implementers a “learning by doing” exercise to promote adoption of the standards and guidelines.” 30 There is a course for better understanding the European software solutions for implementing and using SDMX standard and the corresponding software. It consists of the courses: a) SDMX Basic course for Statisticians and b) SDMX for IT developers.

3.3.3 Supporting right implementation

In order to support the right implementation we can find responsible body, which takes care of the standard itself, its promotion, certification, training material, maintenance. This is the case of SDMX, ISO, CEN and to some extent, this is the role of Eurostat in European statistics. The responsible body should provide feedback possibilities for users. Feedback possibilities on success or fails or critics are crucial for further development of standards.

Support during implementation can have different forms, like recommendation on implementation, helpdesk, forum for consultation, financial support or combination of these. An example: on recommendation is the handbook on the methods of implementation of NACE Rev.2 in calculation of time series of different statistical domains. Helpdesk function is not always typical; generally private consultant companies provide this kind of support. In statistics, Eurostat provides temporary support with grants. Further example is the CIRCA Forum of classifications, which operates like a helpdesk and gives prompt assistance in right application of NACE Rev.2 and CPA2008. Usually more than hundred questions are consulted and answered every year, therefore decision can be achieved in one week instead of waiting for the working group meeting once a year.

Conformity assessment and certification provide feedback for users on the right implementation of the standard. Conformity assessment is typical in case of ISO, CEN. “For conscientious manufacturers and service providers, having their products and services assessed in accordance with ISO and IEC International standards allows them to distinguish themselves from less reputable suppliers. When public health, safety or the environment may be at stake, conformity assessment is often made obligatory by government regulations. Without appropriate assessment and approval,

29Policy on standards, Statistics Canada, 2004 http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-eng.htm30 http://sdmx.org/wp-content/uploads/2011/05/SDMX-Global-Conference-2011-Summary-Report-2-4-May-2011.pdf

88

http://sdmx.org/wp-content/uploads/2011/05/SDMX-Global-Conference-2011-Summary-Report-2-4-May-2011.pdf%20%20

http://sdmx.org/wp-content/uploads/2011/05/SDMX-Global-Conference-2011-Summary-Report-2-4-May-2011.pdf%20%20

http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-eng.htm

goods may be barred from sale, or suppliers disqualified from bidding for government procurement contracts.”31 Not only supply providers but consumers and regulators as well, benefit from conformity assessment. See chapter 5 for more details.

In the field of statistics, conformity assessment to international statistical standards is not very common. However, we can find some examples from the last years: Code of Practice, where self assessments were complemented with peer review. In the case of EDP report, audits are the usual tools. At national level there are different practices.

Community of users, regular fora support right implementation, common understanding and further development. Users’ fora with regular meetings, conferences are places to share experiences, discuss problems and motivate reluctant people. In statistics, according to the survey32

carried out in 2010 among methodologists in NSIs of Member States, the mostly used standard handbooks were on quality, seasonal adjustment and disclosure control. There are working groups, expert groups, task forces functioning in these fields as well as METIS workshops and the META Working Group.

In the case of classifications, the Task Force on NACE critical classes were set up to assist right and common interpretation of NACE critical classes, therefore a common application of NACE Rev.2 in EU.

3.4. Summary

Standards become real standards if they are implemented. Having the standards in place, they represent an international consensus on good practices and therefore facilitate the common understandings and mutual recognition. Support measures form the basis for the enhancement and acceleration of implementation.

Support measures related to standards

Motivation Implementationfor implementation Making easy Supporting

Communication, promotion Availability Responsible bodyAwards Understandability Feedback possibilitiesLegal instruments Efficiency Support during implementation

Consistency of standards Conformity assessment, Training Community of users, regular

fora

The forms of support measures collected from the world of standards are summarised in the table above, but still, part of them is used in the field of statistical methodological standards.

31 ISO and conformity assessment, ISO Central Secretariat, Genève, 2005. p 2http://www.iso.org/iso/casco_2005.pdf32 Results were presented at the Eurostat workshop on standardisation October 2010. Analysis is downloadable from the workshop webpage.

89

http://www.iso.org/iso/casco_2005.pdf

4. Quality of statistical standards and criteria for the assessment

4.1. Introduction

One of the sub-tasks of WP 1 is a proposal for the definition of quality of statistical standards and – derived from that – criteria for the assessment of potential standards.

Based on the definition of standard as a normative document the ESSnet has analysed six methodological handbooks in order to identify provisions referring to methodology, IT-tools, classifications and definitions. Additionally, the underlying data model contains (statistical) methods linked with methodology (based on) and IT-tools (implemented in).

The following considerations refer explicitly to quality criteria on statistical methods and methodological standards. How they can be transformed to quality criteria on standards in general will be discussed later.

4.2. The concept of quality of standards

By defining standards the ESSnet follows the ISO concept and its definitions. Besides a definition of standards the ISO also provides a definition of quality as the

‘totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs’ (ISO 8402).

The Qualistat Sub-Group ‘Quality of Norms’33 referred to this ISO definition already. One of their conclusion has been, that the ability of a norm (synonym for standard) to satisfy the needs of its users is the best criterion by which to assess its quality.

Applying this definition to methodological standards or statistical methods as potential standards one has to identify the users of these potential standards as a first step and determine their needs.

As for users of methodological standards the origin of the standardisation initiative is important. The initiative is part of the joint ESS strategy for the implementation of the Commission Communication on the production method for EU statistics (Vision). The idea behind the vision is that interested partners of the ESS community could work together in the development of methodology or other areas of co-operation leading to the application of methodological standards. As expressed in the subscription of the Eurostat grant for the ESSnet Preparation of Standardisation, one of the actions in the Hague action plan is to increase the efficiency and quality of statistical production by further standardisation of processes and methods.

Having in mind the origin of the standardisation initiative, the ESS members as producers of statistics can be seen as the users of methodological standards34. Their needs – in respect to standards in general – may be simply summarised as the production of statistics as good and as efficient as possible (in absolute terms) or to increase quality and efficiency (in relative terms).35

33 The Qualistat Sub-Group was created of the Quality Assurance Group to propose a definition of quality for norms and the means to measure that quality. In July 1998 the Sub-Group presented its final report.

34 There has been general agreement that not only NSIs but all producers of ESS statistics should be users of standards.

35 The Qualistat Sub-Group (see the previous footnote) has identified three main groups of users of norms. Besides producers of statistics, data suppliers and users of statistics were mentioned. In the context of the standardisation initiative described here, the latter ones could be regarded not as direct users of methodological standards. Nevertheless, their needs will be reflected through the quality of statistical output (users) and the response burden (data suppliers).

90

This leads to defining the quality of methodological standards as the ability to contribute to the production of qualitative good statistical output in an efficient way (increase quality and efficiency of statistical production) for the ESS members.

It is clear that a methodological standard as one single element in the whole production process of statistics does not determine output quality and production efficiency alone, but contributes to both in a certain extent. Thus, the marginal extent of this contribution should be the quality criterion. To do so the methodological standard must be applicable for its users. It is much easier to make a standard successful if its users benefit directly36.

The definition above leads to three main quality criteria:

1. The ability for ESS members to produce statistical output by using the methodological standard (applicability)

2. The quality of the statistical output produced by applying methodological standards (effectiveness)

3. The efficiency of the statistical production by applying the methodological standards (efficiency)

In order to define criteria for the assessment of potential standards these criteria have to be specified further. 37

4.2.1. Applicability

The first requirement for methodological standards must be their applicability for the users (fitness for purpose) in one or more specific processes. Statistical methods have to fulfil several conditions to be methodological standards. A good and cost-efficient method for one partner may not be a good method for other partners. The requirements for methodological standards must be higher than for methods in a specific context because they are aimed at “common and repeated use” in the “given context” of the ESS. The given context especially includes the national statistical systems in different aspects (organisational, infrastructural, …).

Several conditions can be derived for the applicability of standards in the ESS as adequate universal conditions. Relevant criteria are:

─ Relevance- A need for the standard exists- The standard is appropriate as far as the application leads to the goal by reasonable effort

(see criteria 2 and 3)- The standard is credible (e. g. has a solid, professional basis to provide trust of users)- The standard is generally accepted

─ Clarity, specificity and accessibility38: the standard is- well-known- completely documented- clearly specified (in terms of preconditions, input, interfaces, functionality, …)- understandable for all users- readily and easily available to all users

─ Conformity and coherence: the standard is

36 Sponsorship on standardisation, Doc PG 2011/40/4.1, 40th Meeting of the Partnership Group, 25 March 201137 The issue about the quality of the standardisation process, which may also influence the quality of

methodological standards, is not dealt here.38 The document ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards, lists some

important features for documents to become standards, which are relevant in this context.

91

- conform with legal requirements (regulations, data protection, …)- conform with technical and organizational requirements- free of contradictions to other standards39

- complimentary with other standards

─ Universality and stability- Application must be possible for all users or at least a relevant group of users- Applicability should be possible over a certain time

─ Exhaustiveness- Degree in which a standard can be applied for a certain task (fitness to the aim of

standard)- The standards covers all necessary features

─ Credibility: the standard is- mutually recognized- the result of common work- state of the art (based on newest results of science, technology and experience)

─ Fitness for conformity assessment- The standard is valuable regarding their conformity with requirements

─ Robustness and simplicity: the standard is- easy to implement and apply- robust in relation to application errors

4.2.2. Effectiveness

Statistical methods are applied with the purpose to produce statistical output for users. Naturally, this implies that they are applicable for the production of the intended output. The degree to which this aim could be achieved is directly related to the quality of the statistical output.

Consequently, the quality criteria for methodological standards are the quality dimensions of statistics to which they commit:

─ relevance,

─ accuracy,

─ timeliness and punctuality,

─ accessibility and clarity,

─ comparability and coherence.

4.2.3. Efficiency

This criterion is directly linked to cost efficiency as one of the expected gains of standardisation. As the increase of efficiency of statistical production is mentioned explicitly, the production costs must be criteria for methodological standards. Costs should include any kinds of efforts which will be generated by the application of methodological standards. Thus, both the costs for the producer (ESS member) as well as the costs for respondents (burden) and other data providers should be involved.

39 This criterion does not exclude to define different alternative standard methods for the same purpose.

92

Given the choice between different methods leading to the same quality of output, one would choose the statistical method with the lowest costs. It is also important to consider here relations between costs and benefits of standards in order to evaluate efficiency.

4.3. Relations between the proposed quality criteria on methodological standards

Whenever different criteria will be applied the question on their relations arises. Additionally, all criteria themselves have more than one dimension. The relations between the quality dimensions of statistics are well known, which is true for the relations between costs and burden as well. These questions will not be dealt here.

As for methodological standards we propose to define minimum requirements for their applicability – independent from quality and efficiency of the statistical methods. Every statistical method which should become a methodological standard has to fulfill these minimum requirements.

Concerning the quality of statistics and the cost-efficiency, there will be a trade-off. It seems clear, that ceteris paribus both criteria apply. The method with higher quality of output will be preferred, if the alternatives have equal cost efficiency. Under methods with equal cost efficiencies the method leading to the highest output quality will be preferred. The choice between two methods, one leading to better quality, the other with better cost efficiency is the most interesting question. The issue, how priorities should be generally set for standardisation has to be discussed at strategic level (Sponsorship Group).

Quality of statistical standards other than statistical methods

Which other types of standards are considered will be determined in WP 2. We therefore propose to deal with the extension of quality criteria for those types of standards afterwards as part of WP 2.

4.4. Main problems connected with standards that may have quality character

To complete proposals on quality criteria regarding standards it is worth mentioning some problems that may have quality character. They are following:

1. Standard was created too late

When national or international standard is created too late, individual creation of the standard is too expensive. When standards are created too late they are not able to respond on current and future users needs.

2. Standard is not adapted to the specific of the information area

Standards should be used in the areas for which they were prepared. This guarantees that they are well adjusted to concrete area.

3. Requirements of the standard are too high

It is difficult to meet the requirements, for example: a requirement for small enterprises to submit statistical declaration with information in constant prices. Standards should be based on realistic requirements. Hence, good standards include at least minimum requirements that are easy to fulfil.

4. Standard does not supply full documentation

93

For example, very often instructions, methodological notes are not complete, clear, comprehensible, or mistaking. Standards should be well-documented in order to fulfil criteria on accessibility and clarity.

5. Standard is out of date (obsolete)

If the process of standard creation is long-lasting, standards may have lost their immediate interest. When standards have been created for a long period, there is a danger for standards to be irrelevant to current and future needs of users.

6. Standard is too general

For example: too broad definition demands additional explanations and interpretations. The level of details of standards should be adequate for their aim. Too general or too detailed standards can bring some problems.

7. Standard is too detailed

Too detailed standard is not applied or is a basis for individual/ local standard). See comment above.

8. Standard is unknown

Standard should be popularised with information concerning advantages from standardisation and disadvantages from lack of standards. One attribute of standards is that they are well-known. Hence, dissemination and promotion actions for standards are very important.

9. Standard is premature

If conditions for the implementation (technical, legal and organizational ones) do not exist, the standard is premature. Creation of standards is connected with sequence of actions which should have be well done. Hence, providing technical, organizational, legal conditions for these actions are key issues.

10. Implementation of the standard is too costly (expensive)

When implementation is beyond financial means of potential standard users, or standards are too costly they can meet financial barriers.

11. Standard is unnecessary (needless)

Standard is needless when it is not connected with users needs

94

5. Conformity to standards

5.1. Definition

The US standard organisations, CEN or ISO consider that conformity assessment is at the core of standardisation. They refer to similar reasons to explain to non-specialists why conformity assessment is important.

For NIST, “Standards promote understanding between buyer and seller and make possible mutually beneficial commercial transactions. A product‘s conformance to accepted standards readily provides an efficient method of conveying complex information on the product‘s suitability.”40

For the ISO, “Products and services are like promises. Business customers, consumers, users and public officials have expectations about products and services relating to features like quality, ecology, safety, economy, reliability, compatibility, interoperability, efficiency and effectiveness. Conformity assessment means evaluating and confirming such features as defined in standards, regulations and other specifications. In this way, conformity assessment makes sure that products and services deliver on their promises”.41

“ISO/IEC 17000:2004 - Vocabulary and general principles”, defines conformity assessment as “demonstration that specified requirements relating to a product, process, system, person or body are fulfilled.”42 Conformity assessment consists in conducting activities that are defined below, at the end of the paragraph on “Standards in the EU and conformity assessment”.

CEN’s glossary states that conformity evaluation means the “systematic examination of the extent to which a product, process or services fulfils specified requirements.”

The conformity of a product (respectively process, service) to a standard means the extent to which this product (respectively process, service) fulfils specified requirements of this standard.

Whereas there is little communication on conformity to standards in itself, the assessment of this conformity is at the heart of many references.

5.2. Conformity assessment is considered during the development of standards

ISO lists among the principles for drafting standards, the principle of verifiability 43 : Whatever the aims of a product standard, only such requirements shall be included as can be verified.

40 : in « THE ABC’s OF STANDARDS ACTIVITIES », NIST.41 : in « ISO and conformity assessment », ISO42 : In the previous version (1996), it was defined as «"any activity concerned with determining directly or indirectly that relevant requirements are fulfilled."43 : In “ISO/IEC Directives, Part 2 - “Rules for the structure and drafting of International Standards”, Annex A”

95

Requirements in documents shall be expressed in well-defined values (see 6.6.11). Phrases such as “sufficiently strong” or “of adequate strength” shall not be used.

Another consequence of the principle of verifiability is that the stability, reliability or lifetime of a product shall not be specified if no test method is known by means of which compliance with this requirement can be verified in a reasonably short time. A guarantee by the manufacturer, although useful, is not a substitute for such requirements. Guarantee conditions are considered to be outside the aspects to be included, being a commercial or contractual, and not a technical, concept.

Further, ISO includes this conformity assessment in its drafting rules, in chapter 6.7 44. It has developed specific rules (standards ISO 17000 and 17025) that apply to the conformity assessment to all standards:

Aspects of conformity assessment

Documents containing requirements for products, processes, services, persons, systems and bodies

All documents containing requirements for products, processes, services, persons, systems and bodies shall be written in accordance with the "neutrality principle", such that conformity can be assessed by a manufacturer or supplier (first party), a user or purchaser (second party), or an independent body (third party).

NOTE 1: First-party, second-party and third-party conformity assessment activities are defined in ISO/IEC 17000.

NOTE 2: The term "documents" is defined in Clause 1.

Such documents shall not include requirements related to conformity assessment other than requirements which are necessary to provide repeatable and reproducible conformity assessment results.Committees wishing to specify additional conformity assessment requirements for the product, process, service, persons, systems or bodies may only do so in a separate document or in a separate part of the document provided that the separate parts can be applied independently. Prior to commencing work on a separate document or separate part, a committee shall seek the approval of the ISO Committee on conformity assessment (ISO/CASCO) […].No document containing requirements for products, processes, services, persons, systems and bodies shall make conformity dependent on a quality management systems standard, i.e. it shall not, for example, make normative reference to ISO 9001.

Conformity assessment schemes and systems

Committees shall not develop documents providing general requirements for conformity assessment schemes and systems. Development of such documents is the responsibility of the ISO policy committee ISO/CASCO […].Committees wishing

44: In ISO/IEC Directives, Part 2 - “Rules for the structure and drafting of International Standards”, chapter 6

96

a) To propose the establishment of a conformity assessment scheme or system, or b) To prepare documents specifying conformity assessment systems or schemes or

sector-specific operating procedures for use by conformity assessment bodies and others for conformity assessment purposes,

shall consult with the secretariat of ISO/CASCO or IEC/CAB or both as appropriate, prior to commencement of the work to ensure that any documents developed are in line with the conformity assessment policies and rules approved by ISO/CASCO and IEC/CAB as relevant.

References to ISO/IEC conformity assessment documents

When a committee develops a document relating to conformity assessment systems or schemes, or any other document addressing conformity assessment aspects, the document shall make normative reference to the relevant published ISO/IEC documents for conformity assessment procedures, including ISO/IEC 17000 and ISO/IEC 17025. The committee may include verbatim text from the ISO/IEC documents for conformity assessment procedures but the committee shall not delete, change or interpret them. Committees shall consult with the ISO/CASCO[…], for advice on correctly referencing the ISO/IEC conformity assessment documents. Any request for addition, deletion, change or interpretation shall be submitted to the secretariats of ISO/CASCO […] for decision.

Since national/ regional standards might be considered by the World Trade Organisation (WTO) as barriers to trade, there are strict rules that have to apply in order to avoid this shortcoming. The definition of cross-cutting rules for conformity assessment makes it easier to guarantee the conformity with WTO rules.

5.3. Standards in the EU and EU conformity assessment

5.3.1. Framework for EU standardisation

The EU with its New approach directives refers extensively to standards for the monitoring of the EU single market45. “Harmonised standards” define the conditions that apply to some kind of goods, for the monitoring of Health and Security in the EU.

The EU has defined its own principles, in the COUNCIL DECISION of 22 July 1993 concerning the modules for the various phases of the conformity assessment procedures and the rules for the affixing and use of the CE conformity marking46. There are about 10 procedures for conformity assessment which are to be used in the technical harmonization directives relating to the marketing of industrial products. The possible assessment modules are listed in this decision.

There are a number of principles that the technical harmonisation directives have to follow when choosing a conformity assessment procedure. Those of particular interest for this work are:

a- the essential objective of a conformity assessment procedure is to guarantee the health and safety of users and consumers;

45 : For a comparison with the US system, and a short summary of the EU system, see A Guide to EU Standards and Conformity Assessment, NIST46 See http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:1993:220:0023:0039:EN:PDF

97


b- conformity assessment can be subdivided into modules which relate to the design phase of products and to their production phase;

c- as a general rule a product should be subject to both phases before being able to be placed on the market if the results are positive;

d- it is necessary to take into consideration, the appropriateness of the modules to the type of products, the nature of the risks involved, the economic infrastructures of the given sector (e.g. existence or non-existence of third parties), the types and importance of production, etc.

e- the cost efficiency of the procedures has to be taken into account

5.3.2. Modules for conformity assessment

Technical harmonisation directives refer to one module in the list below. Modules A, C and H can be used with additional provisions. Module C is designed to be used in combination with module B (called EC type-examination). Modules D, E and F will also normally be used in combination with module B; however, in special cases they may be used on their own.

A summary of the provisions given in the Council directive is given below.

Module A (internal production control)

1. The manufacturer ensures and declares that the products concerned satisfy the requirements of the directive that apply to them. The manufacturer draws up a written declaration of conformity. 2. The manufacturer must establish the technical documentation described in paragraph 3 and must keep it for a period ending at least 10 years for inspection purposes. 3. Technical documentation must enable the conformity of the product with the requirements of the directive to be assessed. It must, as far as relevant for such assessment, cover the design, manufacture and operation of the product.

Module Aa

This module consists of module A, plus the following supplementary requirements: For each product manufactured one or more tests on one or more specific aspects of the product must be carried out by the manufacturer or on his behalf. The tests are carried out on the responsibility of a notified body47 chosen by the manufacturer.

Module B (EC type-examination)

1. This module describes that part of the procedure by which a notified body ascertains and attests that a specimen, representative of the production envisaged, meets the provisions of the directive that apply to it. 2. The application for the EC type-examination must be lodged by the manufacturer with a notified body of his choice. The applicant must place at the disposal of the notified body a specimen, representative of the production envisaged and hereinafter called 'type'.

47 Notified body = an organisation that is specialised in conformity assessment and that is registered by CEN= “third party”.

98

3. The technical documentation must enable the conformity of the product with the requirements of the directive to be assessed. It must, as far as relevant for such assessment, cover the design, manufacture and operation of the product. 4. The notified body must:

4.1. verify that the type has been manufactured in conformity with the technical documentation and identify the elements which have been designed in accordance with the standards, as well as the components which have not.4.2. for the latter components, perform the appropriate examinations and necessary tests to check whether the solutions adopted by the manufacturer meet the essential requirements of the directive; 4.3. perform the appropriate examinations and necessary tests to check whether for the former the standards have actually been applied;

5. Where the type meets the provisions of the directive, the notified body must issue an EC type-examination certificate to the applicant. Module C (conformity to type)1. This module describes that part of the procedure whereby the manufacturer ensures and declares that the products concerned are in conformity with the type as described in the EC type-examination certificate and satisfy the requirements of the directive that applies to them. 2. The manufacturer must take all measures necessary to ensure that the manufacturing process assures compliance of the manufactured products with the type as described in the EC type-examination certificate and with the requirements of the directive that apply to them.

Possible supplementary requirements: For each product manufactured one or more tests on one or more specific aspects of the product must be carried out by the manufacturer. Module D (production quality assurance)1. This module describes the procedure whereby the manufacturer ensures and declares that the products concerned are in conformity with the type as described in the EC type-examination certificate. 2. The manufacturer must operate an approved quality system for production, final product inspection and testing. 3. Quality system

3.1. The quality system must ensure compliance of the products. The quality system documentation must permit a consistent interpretation of the quality programmes, plan, manuals and records. 3.2. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member with experience of evaluation in the product technology concerned. The evaluation procedure must include an inspection visit to the manufacturer's premises. 3.3. The manufacturer must undertake to fulfil the obligations arising out of the quality system as approved and to uphold it so that it remains adequate and efficient.

4. Surveillance under the responsibility of the notified body4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of manufacture, inspection and testing, and storage and must provide it with all necessary information. 4.2. The notified body must periodically carry out audits to make sure that the manufacturer maintains and applies the quality system and must provide an audit report to the manufacturer. 4.3. Additionally the notified body may pay unexpected visits to the manufacturer.

99

Module E (product quality assurance)1. This module describes the procedure whereby the manufacturer ensures and declares that the products concerned satisfy the requirements of the directive that apply to them. 2. The manufacturer must operate an approved quality system for final product inspection and testing. 3. Quality system

3.1. The manufacturer must lodge an application for assessment of his quality system for the products concerned, with a notified body of his choice. 3.2. Under the quality system, each product must be examined and appropriate tests as set out in the relevant standard(s) or equivalent tests shall be carried out in order to ensure its conformity with the relevant requirements of the directive. 3.3. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member experienced as an assessor in the product technology concerned. The assessment procedure must include an assessment visit to the manufacturer's premises.

4. Surveillance under the responsibility of the notified body4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of inspection, testing and storage. 4.3. The notified body must periodically carry out audits to ensure that the manufacturer maintains and applies the quality. 4.4. Additionally, the notified body may pay unexpected visits to the manufacturer.

Module F (product verification)1. This module describes the procedure whereby a manufacturer checks and attests that the products are in conformity with the type as described in the EC-type examination. 2. The manufacturer must take all measures necessary in order that the manufacturing process ensures conformity of the products. 3. The notified body must carry out the appropriate examinations and tests in order to check the conformity of the product with the requirements of the directive either by examination and testing of every product or by examination and testing of products on a statistical basis at the choice of the manufacturer. 4. Verification by examination and testing of every productAll products must be individually examined and appropriate tests as set out in the relevant standard or equivalent tests shall be carried out in order to verify their conformity. 5. Statistical verification

5.1. The manufacturer must present his products in the form of homogeneous lots. 5.2. A random sample shall be drawn from each lot. Products in a sample shall be individually examined and appropriate tests as set out in the relevant standard.

Module G (unit verification)1. This module describes the procedure whereby the manufacturer ensures and declares that the product concerned, which has been issued with the certificate referred to in point 2, conforms to the requirements of the directive that apply to it. 2. The notified body must examine the individual product and carry out the appropriate tests as set out in the relevant standard or equivalent tests, to ensure its conformity with the relevant requirements of the directive. The notified body must affix, or cause to be affixed, its identification number on the approved product and shall draw up a certificate of conformity concerning the tests carried out. Module H (full quality assurance)1. This module describes the procedure whereby the manufacturer declares that the products concerned satisfy the requirements of the directive that apply to them.

100

2. The manufacturer must operate an approved quality system for design , manufacture and final product inspection and testing and shall be subject to surveillance. 3. Quality system

3.1. The quality system must ensure compliance of the products with the requirements of the directive that apply to them. It must contain in particular an adequate description of: - the quality objectives and the organizational structure, responsibilities and powers of the management with regard to design and product quality, - the technical design specifications, including standards, that will be - the design control and design verification techniques, processes and systematic actions that will be used when designing the products, - the corresponding manufacturing, quality control and quality assurance techniques, processes and systematic actions that will be used, - the examinations and tests that will be carried out before, during and after manufacture, and the frequency with which they will be carried out. 3.2. The notified body must assess the quality system to determine whether it satisfies the requirements. The auditing team must have at least one member experienced as an assessor in the product technology concerned. The evaluation procedure shall include an assessment visit to the manufacturer's premises.

4. EC surveillance under the responsibility of the notified body4.1. The manufacturer must allow the notified body entrance for inspection purposes to the locations of design, manufacture, inspection and testing, and storage 4.3. The notified body must periodically carry out audits to make sure that the manufacturer maintains and applies the quality system and shall provide an audit report to the manufacturer. 4.4. Additionally the notified body may pay unexpected visits to the manufacturer. At the time of such visits, the notified body may carry out tests or have them carried out in order to check the proper functioning of the quality system where necessary.

Possible supplementary requirements:

Design examination

1. The manufacturer must lodge an application for examination of the design with a single notified body. 2. The application must enable the design, manufacture and operation of the product to be understood, and shall enable conformity with the requirements of the directive to be assessed. It must include:

- the technical design specifications, including standards, that have been applied, - the necessary supporting evidence for their adequacy, in particular where the standards have not been applied in full. This supporting evidence must include the results of tests carried out by the appropriate laboratory of the manufacturer or on his behalf.

3. The notified body must examine the application and where the design meets the provisions of the directive that apply to it must issue an EC design examination certificate to the applicant.

We can see much similarity with the tools used by ISO that gives the following summary of conformity assessment activities:

Conformity assessment may consist of any one of, some of, or all of the following : sample testing, inspection, process evaluation, supplier’s declaration of conformity, management system

101

certification/registration, personnel certification, product certification, mutual recognition of results and the accreditation and peer assessment of the competence of the organizations conducting these activities – collectively known as “conformity assessment bodies”. ISO and IEC jointly develop standards and guides for all those activities.

5.4. How to assess conformity to methodological standards in statistics?

5.4.1. Problem statement in the ESS

It is good practice to investigate how successful methods can be transposed to our domain. We will consider the EU practice in conformity assessment, even if some limits have to be kept in mind:

the EU harmonised directives refer to the standards for goods, not for services. Some investigation of conformity assessment in the domain of services has been made, but it seems that the framework for such activities is still under development48.

they are dealing with market activities. As in other chapters of the report from the ESSnet on the Preparation of standardisation, we recall that the group of institutions interested in the development of standards for the implementation of Eurostat’s vision for the next decade is a much smaller group and differs from a market.

The methodological standards are intended for the production of statistical specifications, especially in the domain of statistical processing (sampling, detection of reporting errors and imputation of missing values). Such statistical specifications are intermediate products in the production chain under the responsibility of NSIs. These products will have to be assessed for conformity to methodological standards. They are the result of a design activity.

In addition, such products are “one of a kind”. All aspects related to mass production, and that lead to techniques like control of samples are not directly relevant for a design activity.

Among the principles mentioned by the Council decision on conformity assessment, the last two refer to the evaluation of risks and to the efficiency and cost of the assessment procedure. In short, it makes sense to envisage procedures whose cost is proportional to the risk attached to non-conformity.

As a benchmark for the price to pay for conformity assessment, we can consider existing practice. Conformity to standards is already assessed in the following domains:

implementation of the ESS Code of practice production of Gross National Income statistics production of statistics for the Excessive Deficit Procedure, ie. Deficit and Debt of public

administrations.

In the last two domains, the conformity assessment involves very strict procedures. This is due to the strategic nature of these statistics: EDP statistics are vital for the euro currency and the trust in debts of national public administrations. Hence it is as well crucial for the whole EU economy, since many EU economic actors have liabilities connected to these debts. In this case, inspections of 48 See Annex B in “CEN - INTERNAL REGULATIONS - PART 3 - TERMS OF REFERENCE OF THE CEN CERTIFICATION BOARD (2006-9)” or “CEN’s Horizontal European Services Standardization Strategy CHESSS Feasibility Study CONSOLIDATED REPORT”.

102

other administrations than the NSIs have been recently granted to assess the quality of these statistics. Regular visits are organised by Eurostat, which is in charge of the control of conformity to the standard. In this respect, the standard is part of the ESA, which is the the standard for EU natinal accounts.

The production of Gross National Income statistics is very important as well, since it defines the contribution of each Member State to the EU budget. It is regularly controled both by Eurostat and the European Court of Auditors. In addition to the description of methods used by NSIs, some visits can be organised by the assessors on the premices of NSIs.

As for the ESS Code of practice, the assessment of its implementation is currently49 conducted through self-evaluation of NSIs, that answer to questionnaires sent out either by Eurostat or by the European Statistical Governance Advisory Body (ESGAB).

For the production of other statistics, the assessment of conformity to standards (regulations defining statistics are mandatory standards) is in general very light, made of periodic quality reports, and/ or of internal quality reviews, ie. based on self-assessment by the NSI50. These reports tend to focus on specific/ limited aspects of the features of these statistics. For a few statistics - like Harmonised Consumer Price Index - conformity is monitored more closely, with the involvement of Eurostat.

What is the risk attached to non-conformity to standards for statistical specifications, in the framework of the vision, ie. of the integration of the statistical production inside the ESS?

Some hypotheses have to be made on such an integration. We consider here the following possible organisation:

several NSIs would have agreed to integrate their production for one kind of statitics all would share the workload between themselves they would all keep responsibility for data collection (due to linguistic, cultural and legal

reasons) one NSI would be responsible for specifying the statistical aspects of the production downstream, several types of organisation could be envisaged:

1) every NSI implements independently these specifications on its own IT system2) another NSI develops portable IT programs that can be implemented by all NSIs3) any intermediate solution between 1) and 2) could be envisaged.

Whereas delegating the development of specifications might reduce the costs (provided the specifications needed by the cooperating NSIs are similar), every non-conformity in the specification would be implemented in all production systems. Hence, the risk attached to non-conformity in the specifications developed by one singular NSI is highly increased.

This organisation would then require conformity assessment procedures for common interest of ESS:

for Eurostat, a non-conformity in statistics produced in one single NSI can be of limited impact on EU-wide statistics - and can even be partially compensated using the statistics produced by other NSIs. If several NSIs implemented one single set of specifications that is not conform to standards, then this would have larger consequences on EU agregates and it would be much more difficult to adjust for shortcomings in the methods.

49 : This procedure should be revised along the proposals made by the Commission in its Communication 211/2011. 50 : A list of legal requirements for quality reports can be found in the ESS Handbook for quality reports, chapter 3.

103

NSIs involved in the cooperation would inherit the non-conformity due to the NSI in charge of the specification.

As a consequence, it seems quite necessary to look for a conformity assessment scheme that would limit the risk to the levels currently known, ie in the absence of integration of the development of specifications. However, the cost of this assessment has to be limited, so as to keep the benefit of this integration, ie. cost reduction.

5.4.2. Proposal for conformity assessment

We have to deal with two features in conformity assessment: which procedures could be chosen and which “notified body” could be chosen? The ESSnet is not in a position to make proposals in such matters. It limits itself to propose a technical assessment of pros and cons of several options, in order to assist the Sponsorship on standardisation in making the corresponding proposals.

5.4.2.1. Procedure for conformity assessment:

We will consider all procedures used by EU harmonised directives.

First, we have to take into account that statistical specifications can be tested for conformity by a third body only if they are provided with full documentation, and if assistance is provided to those who are in charge of checking the conformity. These conditions are mentioned for the option “design examination” of module H in the Council decision on conformity assessment.

Second, we will take into account the fact that statistical specifications are “one of a type” productions, whereas the types of conformity assessment of the EU decision on conformity assessment cover both mass production and one of a type production.

If we consider that there is no “ESS standard” on the way to write statistical specifications, it is difficult to imagine controls of conformity relying on sampling, or even quality assurance methods. If in the medium or long term such standard on the drafting of specifications were agreed, it might be possible to define homogeneous subsets of one specification51, and to check only a few subsets chosen at random. In a similar way, it would then be possible to define a quality common assurance system for this activity.

We are dealing here only with the assessment of conformity to methodological standards. This conformity will not guarantee that such specifications are fully satisfactory. They have as well to be checked for logical consistency and completeness. This is out of the scope of this chapter. However, developing an ESS standard for the drafting of statistical specifications and defining a common system for quality assurance would as well be useful for this other important concern.

51 : For instance, such standards could lead to group all specifications dealing with one characteristic into one lot. Such a lot would be clearly identified. The whole set of specifications would be made of some pre-processing or post-processing and a series of lots, each one corresponding to one characteristic for which statistics have to be produced. It would then be possible to check conformity independently for each characteristic. Some care should however be paid to the cross-cutting specifications, to be sure that they do not alter this conformity.

104

Bode-O, 2011-08-31,

You don´t find 5.4.2.1. and 5.4.2.2 in the table of contents

Type of conformity assessment/

(module)

Content (summary) Case of “one of a type” production/ statistical specifications

internal production control (A)

self-declaration by producer possible in the short term

EC type-examination(B) with Design examination

examination of a specimen by an independent body

possible if standards on drafting specifications are developed

conformity to type (C) with Design examination

self-declaration of control of the regularity


production quality assurance (D)

production quality assurance audited by an independent body

possible if standards on drafting specifications and if a quality assurance system are developed

product quality assurance (E)

production, inspection and testing quality assurance audited by an independent body

possible if standards on drafting specifications and if a quality assurance system are developed

product verification (F) with Design examination

random testing by an independant body


unit verification (G) with Design examination

every product tested by an independant body

possible in the short term

full quality assurance (H)

identical to production quality assurance since the production activity is a design activity

As a conclusion, it seems that the choice of procedures would be very limited in the short term: either the NSI in charge of the drafting of specifications guarantees himself the conformity, or a third party assesses this conformity, with the assistance of the author so as to understand the specification.

5.4.2.2. Who should be in charge of the assessment of conformity?

We could envisage in theory three kind of actors:1. an organisation independent from the ESS2. a NSI3. Eurostat

It seems difficult to envisage the involvement of an independent organization, a “notified body” as mentioned in the EU Council decision. First, they have not worked with standards in the domain of public statistics up to now. Even in private statistics, their activity is very limited. Since the amount of activity would be very limited, it would be very costly to hire competent people in a position to

105

work in many languages52… The financial issue should as well be addressed (who should pay for the assessment…).

As for relying on an NSI, it would be difficult as well: some languages issues might be met53, NSIs have to reach agreements in the negotiation process inside the ESS, and the

independence between them would be difficult to secure the NSI in charge of the assessment would have to invest heavily in the domain, and a one-

off involvement would be probably very costly due to fixed costs (specific training costs in the domain of conformity assessment)

however, the technique involved are in general practiced by NSIs, and they should have competent statisticians in the domains covered by the methodological standards.

As for relying on Eurostat for this assessment, the situation is the following: it is in the culture of Eurostat staff to have an independent view on work done by NSIs Eurostat would not much less hampered than NSIs if the work had to be conducted in

various languages and located in several NSIs Eurostat has more interest in promoting the implementation of the vision (and such

standardisation) than some NSIs however, in general, Eurostat is not in charge of many aspects of methodology related to

the production of statistics (sampling, editing, imputation). Some investment in Human resources would be necessary as well

there is very few success in the development of long term cooperation between NSIs without the support of Eurostat54. Financial compensation between NSIs would probably be much more complicated than securing money from the EU budget under the responsibility of Eurostat…

52 : Unless there is an agreement to develop these specifications only in English53 : Unless there is an agreement to develop these specifications only in English54 : The exception seems to be found in the cooperation between Nordic NSIs, with a tool like PC-Axis

106

Annex 1 - Extracts referring to standards with another meaning

1: In the US OMB document on “ STANDARD S AND GUIDELINES FOR STATISTICAL SURVEYS” (September 2006):

Standards are very similar to the recommendations of the ESS Code of practice. In the latter, the wording “principle” is used instead of “standard”. The US OMB document states that (see page 1):

Standards for Federal statistical programs serve both the interests of the public and the needs of the government. These standards document the professional principles and practices that Federal agencies are required to adhere to and the level of quality and effort expected in all statistical activities. Each standard has accompanying guidelines that present recommended best practices to fulfill the goals of the standards. Taken together, these standards and guidelines provide a means to ensure consistency among and within statistical activities conducted across the Federal Government.

2: In the ESS Code of practice:

2.1:Statistics comply with the European quality standards and serve the needs of European institutions, governments, research institutions, business concerns and the public generally.

Here, “standard” means “a certain level of quality”, not to rules to measure or assess quality (which do exist), nor to rules that would define the rules that apply to the quality level for each statistic (such rules do not exist)…

2.2: Timeliness meets the highest European and international dissemination standards.“Standard” means here “a certain level of quality”.

3- In the ESA 2010 regulation:

Article 338 of the Treaty on the Functioning of the European Union provides the legal basis for the European statistics. Acting in accordance with the ordinary legislative procedure, the European Parliament and the Council shall adopt measures for the production of statistics, where necessary, on the performance of the activities of the Union. It sets out the requirements relating to the production of European statistics, stating that they must conform to standards of impartiality, reliability, objectivity, scientific independence, cost-effectiveness and statistical confidentiality.

Here, standards mean “levels of quality/ requirement”.

It is particularly surprising that the treaty states that “The production of Union statistics shall conform to impartiality, reliability, objectivity, scientific independence, cost-effectiveness and statistical confidentiality”. There is no mention of standard!!!

Here, standard means level of quality.

4) Ecofin recommandations - Nov 2010

107

These conclusions state as well that “ Council calls upon […] to reinforce the binding nature of the Code of Practice by enshrining in a legal act some of the minimum standards, including professional independence.

standards = level of quality

5) The regulation 223/2009 uses as well the term “standards”:

‘objectivity’, meaning that statistics must be developed, produced and disseminated in a systematic, reliable and unbiased manner; it implies the use of professional and ethical standards, and that the policies and practices followed are transparent to users and survey respondents;

In this paragraph, “standard” means “principles”.

Specific quality requirements, such as target values and minimum standards for the statistical production, may be laid down in sectoral legislation. .

“standards” mean here “level of quality”.

6) The “Communication of the Commission on the vision for the next decade” (404/2009):

The first component remains Community legislation, which will continue to be mainly output-oriented and to set minimum standards for the production of statistics in a particular area.

“standards” mean here “level of quality”.

7) The Communication of the Commission on the robustness of statistics (211/2011):

Recent developments, in particular, the inaccuracy of the Greek government deficit and debt statistics, have however demonstrated that the system for fiscal statistics did not sufficiently mitigate the risk of substandard quality data being notified to Eurostat.

“standard” refers here again to a certain level of quality.

108

Annex 2 – Review of literature on standards

Introduction:

Objectives of the work

Aim: defining the concept of standard and, in particular, infer from the general definition the definition of standard corresponding to statistical methodsFinal objective: making a proposal to define precisely what is a statistical standardHere is the adopted method:

o Taking stock of existing references in the literature for domains when standardisation is well implemented

o Defining a common structure for the analysis of the chosen references.

Discussions took place on the structure and terms of the template. The template was aimed at collecting from the references all information pertaining to questions raised in Eurostat call for proposals.

Consensus was eventually reached on 5 main sections: General description of the document, Concept and main characteristics of standards, Process of standardisation, Possibility to standardise a design activity & Any other relevant content

This work was performed within the scope of WP 1 (but it is also consistent with the tasks foreseen in WP2)

Choice of reference documents

The search for references was organised as follows:o give priority to general reviews on theory or practice in standardisationo look for references recommended by famous standard organisations (nationally or

internationally)o look for EU agreements when possibleo look for references either in statistics or in related domains (services, etc…).

The result covers a large scope in many respects:o An international selection: documents from the EU, USA, Canada, Japan…o Large variety of documents: Sponsorship group’s report, leaflet, Code of practice, contribution at conferences,

real ISO standard, guidelines…o Different types of intended audience: NSIs, businesses, institutional bodies, general public…o Variety of scope of the documents: opinion surveys, multimedia, industrial sector…

I/ Elements to explore for WP1

Definition of a standard

109

Most documents do not offer an explicit definition for the word « standard ». However some definitions were found:

o Technical standards may be described by answering six questions. (The six dimensions of standards)o Documentary standards (…) are written agreements containing technical specifications or other precise criteria

that may contain rules, guidelines, or definitions of characteristics. (The ABC of standards activities)o Standard = Practices that should be requirements for all telephone studies conducted by Government of Canada

(The Advisory Panel on Telephone Public Opinion Survey Quality)

In addition, some documents refer to the ISO definition ( i.e. « a standard is a document, established by consensus, approved by a recognized body, that provides for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context »)

Some documents also provide definitions for « guidelines »:o Guidelines = Practices that are recommended, but would not be requirements; that is, known good practices

or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study (The Advisory Panel on Telephone Public Opinion Survey Quality)

o Guidelines (…) present recommended best practices to fulfill the goals of the standards (Standards and Guidelines for Statistical Surveys)

Binding nature of a standard

Most documents dealt with the possible binding nature of the standard: o no clear description of the binding nature of a standard (it can either be «voluntary», «required», «not

mandatory», « mandatory »…)o 2 documents retain attention because they provide specifications as to the binding nature of standards:

o The ISO defines that a standard is not mandatory in general. It will be mandatory only if a national/regional law requires it. However there are binding requirements that are defined by the standard. Thus they are binding for companies that claim implementing the standard (ISO standard 2052:2006)

o Distinction between « voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. Mandatory standards are set by government regulatory agencies at all levels » (The ABC’s of standards activities)

Other characteristics of standards:- Intended application: it seems that there is no limitation of application for standards in

terms of geography, domains or users. A standard can also be concerned with facilities and equipment, products, activities, customer perception or processes.

- Degree of freedom that remains when the norm is fully implemented: here again, the analysed documents did not provide a clear view, and did not provide an easy answer (examples: “high degree of freedom”, “none”, “it depends on type of standard involved”). However, it is important to note that in the ISO scope, the standard has to be implemented in full in order to gain ISO accreditation (Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control). Moreover the ISO 20252:2006 reads that “there are strict requirements to secure a minimum level of quality except aspects due to quota method. (…) Higher requirements are possible. The quota method leads to much freedom in the assessment of uncertainty.”

- Items to be included in a standard: as noticed before, the scope is very wide. A standard may include precise description of product, methods for reliability, testing and safety, the integration of existing sectoral standards, characteristics of a product, the methodology to assess conformity, or even the conformity assessment process itself.

- Measures of implementation: once again, as the scope of standard is very large, the measures of implementation are numerous. One may retain that control can by secured by a public authority and/or a certification board.

- Quality of standards criteria: The ABC’s of Standards Activities warns about the issues of poorly written standards which can raise costs, reduce product safety and quality and create

110

barriers to trade. Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control is also of interest as it explains what a “good standard” is.

- Conformity to standards: quality assessment is carried out regularly according to defined rules (CEN Workshop Agreement Certification rules for manufactured products in Internal Regulations). Moreover, in the scope of a quality marking scheme, one may identify three assessment options: self-declaration, assessment by an interested party or assessment by an independent body (CEN’s Horizontal European Services Standardisation Strategy, CHESS feasibility Study).

II/ Elements to explore for WP2

Identification of needs, decision on the development of a special standard:One major aspect of standardisation is the origin of such a process. Some of the analysed documents allow for an identification of the needs behind the development of standards. It appears, from the CEN leaflet, that the initiative may be on the part of many different categories such as the industry, consumers, SMEs, boards of certification. In addition, the reasons for this initiative are numerous. One may put forward the need to support legislation (CEN leaflet), to assess quality (Advisory Panel on Telephone Public Opinion Survey Quality) or to maximise profit (International Standardization as a Strategic Tool). It seems obvious that all these reasons are particularly induced by the context of globalisation (Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control).

Identification of the influence of the expected use of a standard on the content: the influence appears to be very strong (Advisory Panel on Telephone Public Opinion Survey Quality) and the need for conformity assessment is very important in matters like health and security (CEN leaflet, A Guide to EU Standard and Conformity Assessment). It is also of a major importance that the production process be modified in order to become modular (International Standardization as a Strategic Tool).

Process of development of a standard: this appears as a major issue as many documents deal with it and offer different elements. First of all, it appears that there is no predefined time in the product cycle for introducing standards. Standards can be anticipatory, participatory or responsive (Six Dimensions of Standards). Whatever the product cycle, the process of development must however include certain steps: proposal, acceptance (after identification of the purpose of the standard and of the requirements to be included - CHESS Feasibility Study), drafting, enquiry, adoption and publication; this process may take up to three years (CEN leaflet). There are certain elements of standard development which are also mentioned, these are consensus, transparency, balance, due process and openness (ABC’s of Standards Activities).

Other elements to be taken into account: - Adoption of standards: votes seem widespread, voters may be from working groups

(Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control) or part of national standard organisations (A Guide to EU Standards and Conformity).

- Maintenance: little is said about this aspect, however ABC’s of Standards Activities warns of the possible problems that may arise from the lack of understanding of reviewers as to how the document was first prepared.

- Conformity Assessment: ISO and Conformity Assessment and The ABC of Standards Activities provide interesting clues regarding this aspect and in particular its signification,

111

the benefits it produces and how it can be performed. One may also want to refer to A Guide to EU Standards and Conformity Assessment which delves deeper into the concepts of risk assesment and self-certification.

- Possibility to standardise a design activity: there seems to be an agreement here, as in International Standardization as a Strategic Tool it is explained that design activity is not standardised but rather kept under control. Advisory Panel on Telephone Public Opinion Survey Quality reads that only very broad standards and guidelines are required for this activity.

Conclusion:This constituted the first work performed in this ESSnet project. The definition of a common structure for the analysis allowed partners to discuss definitions and concepts and to exchange their views on the expected output of the project. This common structure appears to be an efficient tool in order to gather as many relevant elements as possible on standards and the process of standardisation.In conclusion, the analysis of such a variety of reference documents has led the consortium to refine its objectives and to get a clearer picture of what was to be done within the scope of WP1 and WP2.

In general, the underlying concepts in official standardisation seem fairly harmonised on both sides of the Atlantic Ocean (and probably beyond). This can be easily understood: since companies expect their products or services to be accepted worldwide, the trade organisations have engaged in mutual recognition of standardisation, which has lead to similar practices at the regional/ national level. In the EU, the single market has fostered such a convergence.

The analyses are presented below and are organized according to two types of documents: reference documents which have a broad scope and those which are specific to the ESS (documents n° 13 & 14).

112

Bibliography

- “A Guide to EU Standards and Conformity Assessment” - NIST: http://gsi.nist.gov/global/docs/EU_Stds&CA_2000.pdf

- “Draft ESS Quality Assurance Framework” - ESS Sponsorship Group on Quality (SQ): internal draft not to be distributed outside the SQ

- “Independent, Transparent, Externally Audited: the ISO Approach to Survey Process Quality Control” - Bill Blyth: http://www.tnsglobal.com/_assets/files/Paper_Blyth.pdf

- “Architecture-based approaches to international standardization and evolution of business models”, Shintaku, J. Ogawa, K. Yoshimoto, T. (2006), http://www.iecchallenge.org/papers/pdf_iecchallenge/shintaku.pdf “

- The Six Dimensions of Standards - Contribution towards a Theory of Standardization” - E. Baskin, K. Krechmer & M.H. Sherif: http://www.csrstds.com/theory.html

- “ CEN’s Horizontal European Services Standardization Strategy, CHESS Feasibility study” - AENOR, BSI, DIN, DS, EVS, NEN, CMC & Cap Gemini: http://www.cen.eu/cen/Services/Business/Value/CHESSS/Documents/CHESSSBrusselsSeminarGeneralPresentationApril2008.pdf

- “Annex C: CEN Workshop Agreement (CWA) Certification Rules for Manufactured Products in Internal Regulations” - CEN: ftp://ftp.cen.eu/cen/Services/Conformity/Certification/CENIRpart3.pdf

- “ISO and Conformity Assessment” - ISO: www.iso.org/iso/casco_2005.pdf- “European Statistics Code of Practice” - SPC:

http://www.dzs.hr/Eng/international/code_of_practice_en.pdf- “The ABC of Standards activities”, USA National Institute of Standards and Technology:

http://gsi.nist.gov/global/docs/pubs/NISTIR_7614.pdf- “Presentation of CEN - General information” - CEN:

ftp://ftp.cen.eu/CEN/AboutUs/Publications/Compass.pdf- “ISO standard 20252:2006 Market studies, social and opinion surveys” - ISO:

http://www.iso.org/iso/fr/search.htm?qt=20252%3A2006&searchSubmit=Rechercher&sort=rel&type=simple&published=true

- “The Advisory Panel on Telephone Public Opinion Survey Quality” - Public Works and Government Services Canada: http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/documents/comitephone-panelphone-eng.pdf

- “ Standards and Guidelines for Statistical Surveys” - USA Office of Management and Budget : http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf

113

http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf

http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf

http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/documents/comitephone-panelphone-eng.pdf

http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/documents/comitephone-panelphone-eng.pdf



ftp://ftp.cen.eu/CEN/AboutUs/Publications/Compass.pdf


http://www.dzs.hr/Eng/international/code_of_practice_en.pdf

http://www.iso.org/iso/casco_2005.pdf

ftp://ftp.cen.eu/cen/Services/Conformity/Certification/CENIRpart3.pdf

http://www.cen.eu/cen/Services/Business/Value/CHESSS/Documents/CHESSSBrusselsSeminarGeneralPresentationApril2008.pdf

http://www.cen.eu/cen/Services/Business/Value/CHESSS/Documents/CHESSSBrusselsSeminarGeneralPresentationApril2008.pdf

http://www.csrstds.com/theory.html

http://www.iecchallenge.org/papers/pdf_iecchallenge/shintaku.pdf

http://www.tnsglobal.com/_assets/files/Paper_Blyth.pdf

http://gsi.nist.gov/global/docs/EU_Stds&CA_2000.pdf

N°1aSummary of reference document: A Guide to EU Standards and Conformity Assessment

Version prepared by GUS

1. General description of the document

A Guide to EU Standards and Conformity Assessment, NIST (USA)-Special Publication 951The aim of this guide is to present the general principles and concepts behind the European Union’s (EU) “New Approach” laws and directives in order to help business and government officials understand the new laws, the EU’s standardization process and the relationship between the European Commission and the European standardization bodies in the European Union.The document focuses mainly on following aspects of standards: standardization as technical harmonization, process of standardization and conformity assessments for obtaining CE mark to gain access to the European Market.

2. Concept, main characteristics of standards (WP1)a. Definition of

“standard”Direct definition is not mentioned. Some key elements of definition are mentioned e.g. standardisation by consensus and committee (standards bodies), harmonization aspects

b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned!

i. How binding they are? (i.e. voluntary, mandatory)

Presented standards are voluntary and in some cases mandatory with regard to essential health and safety requirements in New Approach Directives.

ii. Form of appearance 1.physical measurement standards

2.documentary standards

Standards have documentary character and in some cases also physical measurement character when they include ergonomic requirements.

iii. Manner in which they specify requirements 1.design

2.performance

Requirements are specified with regard to design (1.) and production process (2.).“Some manufacturers approach risk assessment by using a design standard.”

iv. What it relates to (ie. terminology, classification, product, service, process (if

Presented standards are related with product, service, process in business cycle.

114

statistical process, which GSBPM item), … complex)

v. Link with analysis of architecture of the overall production process

Coverage of production cycle: design phase, production

vi. Intended user group in geographical hierarchy 1. company

2. national 3. regional 4. international

User groups of presented standards are: industry (manufacturers, exporters) and government officials. Hence, we can classified them as company and national. Because of standards are used also in international environment we can distinguish international user group e.g. CEN works with Member state-Standard Organisations.

vii. Intended application domain (i.e. general, industry, statistics, business statistics, SBS)

Industry, government, services

viii. Degree of freedom that remains when the norm is fully implemented (How can we assess it?)

The New Approach Directives are voluntary. They are evaluated on the basis of conformity assessment to obtain product liability CE mark.

ix. Any others to be listed

-

c. Items to be included in a standard

Items that should be included in a standard include technical characteristics of products-essential requirements, quality levels, etc.“A standard can determine, by defining and delineating technical characteristics, what conditions will permit a product on the market”.“There is a vast body of European Standards (ENs) that is not mandated by the Commission; and these standards are not necessarily directed toward essential requirements. In theory, their use is voluntary, just as the use of harmonized standards is voluntary. These standards may or may not address health or safety aspects of products. They may define other characteristics, such as durability, appearance, quality levels, or even cultural preferences. They may be test

115

methods, or measurement guides.”

d. Measures on implementation

There are procedures.For example the procedure for EC type examination:- compiling a technical file- affixing the CE marking“Before affixing the CE mark, therefore, the manufacturer must make sure the product complies with the law by (1) determining which New Approach Directives govern the product, (2) applying the appropriate standards, and (3) carrying out the appropriate method of conformity assessment.The CE mark does not disclose which directive(s) or standards apply to the product, nor will it indicate the method of conformity assessment used to bring the product into compliance. This information is provided by other accompanying documents, such as the Declaration of Conformity.”-drawing up of a Declaration of Conformity-signature the Declaration of Conformity-keeping the Declaration of Conformity available for national surveillance authorities- certification“Most products covered by New Approach Directives can be self-certified by the manufacturer and do not require the intervention of a Notified Body. To self-certify, the manufacturer must assess the conformity of products to the applicable directives, and to the standards if applied.”“There are products, however, which may not be self-certified, but must be subjected to an EC type-examination. This examination involves the inspection of a representative example by a Notified Body within the EEA.”

e. Specificity Specificity can be linked with technical construction files and certain aspects of conformity assessment. “The following main elements should be present in a Technical File:

a general description of the product;

design and production drawing and diagrams;

detailed technical data for essential aspects of the product;

a risk assessment;

a list of standards and or solutions applied;

reports of calculations and tests that have been carried out (calculations might include a

116

bearing, belt, or pulley based on a chart in a catalog);

certificates and inspection reports;

in the case of series production, internal conditions that have been observed to safeguard compliance with the directive;

a user's manual and a Declaration of Conformity.”

“The Declaration of Conformity must contain: the product (name, type or model number, lot,

batch or serial number);

the name and address of the manufacturer or his Authorized Representative;

the name and identification number of the Notified Body if required by the directive;

standards or other normative documents applied to verify compliance with the directive(s). Declaration should also include reference to other directives if applicable;

the date of issue; and

the signature and title.”

f. Quality of standards, criteria

Elements mentioned in the text that are connected with quality criterias of standard: simplification (one law replaced many laws), government of family of products (the same rules for family of products), fulfilment of essential requirements, mutual recognition. These features have impact on following quality dimensions:

a) relevance e.g. acceptance principle- mutual recognition, consensus, the same rules applied for family of products,

b) accessibility and clarity e.g. application of harmonized standards, generic format, government families of products by New Approach Directives, integrated scheme for technical harmonization;

c) comparability and coherence e.g. application of European, harmonized standards, the same rules applied for family of products

3. Process of standardisation (WP2)a. Identification

of needs, decision on the

“The philosophy of standardization by committee and consensus is the same in the EU as it is in the United States. Technical experts and others participate

117

development a special standard (Who will decide?)

voluntarily, and without compensation. The makeup of committees may be organized differently and roles may vary, but they generally follow a pattern that includes input from producers, users, government, and academia. On both sides, committees are fairly autonomous; and both sides have processes for the creation of subcommittees, drafting of standards, disseminating draft documents for comment, voting, and appeals. On both sides, decisions are reached by consensus. And on both sides, standards organizations provide management, administrative, and logistical support for standards activities. They also provide for the editing, printing, publishing, sale, and distribution of standards documents. “

b. Identification of the influence of the expected use of the standard on the content

The identification of the influence of the expected used of the standard on the content is connected with the procedure of conformity assessment

c. Process of the development of a standard (i.e. principles, participants, financing) and related organisational matters

It is possible to distinguish following stages in the process of the development of a standard:

a) the creation of subcommittees;b) drafting of standards;c) disseminating draft documents for comment,

voting and appeals;d) decisions reached by consensus;e) standard organization provides management,

administrative and logistical support for standards activities;

f) editing, printing, publishing, sale, distribution of standards documents.

In addition, process of standardization includes:-design review;-elimination risks;-compiling the technical file of which a list of standards or solutions applied;-affixing the CE marking;-user’s manual;-language;-declaration of conformity;-EC type-examination certificate by notified body within the EEA;-declaration of incorporation“The Declaration of Incorporation includes a statement that protects the subassembly manufacturer, and transfers the responsibility for safe incorporation to the builder of the larger system”.

118

-product liability;- marks;-principle of activity surveillance authorities.

d. Adoption, repository of a standard

Adoption of standards is based on self-certification by producer or is established by a notified body. Repository of a standard is maintained by a standard organization.

e. Maintenance of a standard

-

f. Promotion and support for application

-

g. Conformity assessment

“Conformity assessment is defined by the International Organization for Standardization/International Electrotechnical Commission Guide 2: 1996 as:"any activity concerned with determining directly or indirectly that relevant requirements are fulfilled." Typical examples of conformity assessment activities are sampling, testing and inspection, evaluation, verification and assurance of conformity (supplier's declaration), certification, registration, accreditation, and approval as well as their combinations.Conformity assessment may also be the process by which it is determined that a product's design meets a specification or standard. “

The paper presents conformity assessment procedures with regard to:

a) the module decision – general rules e.g.a. “the essential objective of a

conformity assessment procedure is to enable the public authorities to ensure that products placed on the market conform to the requirements as expressed in the provisions of the directives, in particular with regard to the health and safety of users and consumers;

b. conformity assessment can be subdivided into modules which relate to the design phase of products and to their production phase;

c. as a general rule a product should be subject to both phases before being able to be placed on the market if the

119

results are positive….”

b) design of products- conformity assessment procedures in European Legislation -design:- internal control of production: manufacturer keeps technical documentation at the disposal of national authorities, intervention of notified body;- type examination: manufacturer submits to notified body: technical documentation, type, notified body: ascertains conformity with essential requirements, carries out tests, if necessary, issues EC type-examination certificate,-unit verification: manufactures submits technical documentation,-full quality assurance- EN 29001 manufactures operates an approved quality system (QS) for design, notified body carries out surveillance of the QS, verifies conformity of the design, issues EC design examination certificate.c)production – conformity assessment procedures in European Legislation- production:- internal control of production- manufacturer declares conformity with essential requirements, affixes the CE mark, notified body tests on specific aspects of the product, product checks at random intervals,-conformity to type-manufactures declares conformity with approved type, affixes the CE mark;-production quality assurance- EN 29002, manufacturer operates an approved quality system (QS) for production and testing, declares conformity with approved type, affixes the CE mark, notified body approves the QS, carries out surveillance of the QS;- product quality assurance- EN 29003, manufacturer operates an approved quality system (QS) for inspection and testing, declares conformity with approved type or to essential requirements, affixes the CE mark, notified body approves the QS, carries out surveillance of the QS;- product verification – manufactures declares conformity with approved type or with essential requirements, affixes the CE mark, notified body verifies conformity, issues certificate of conformity;- unit verification- manufacturer submits product, declares conformity, affixes the CE mark, notified body verifies conformity with essential requirements, issues certificate of conformity;-full quality assurance- manufacturer operates an approved QS for production and testing, declares

120

conformity, affixes the CE mark, notified body carries out surveillance of the QS.

4. Possibility to standardise a design activity

“Some manufacturers approach risk assessment by using a design standard.”

5. Any other relevant content Conformity assessment - the process by which compliance with essential requirements is determined. This process can be carried out with or without use of standards.Presumption of conformity – is a legal concept surrounding Harmonized Standards that denotes the relationship between the legislative and standardization processes.The EC type-examination – is a procedure by which the Notified Body ascertains and attests that a specimen representative of production meets the provisions of the directive that applies to it.A Declaration of Conformity - is a document informing Surveillance Authorities that the product meets the essential requirements of the applicable directives, or that the product has been issued an EC type-examination certificate, and meets the essential requirements of the applicable directives.

121

N° 1bSummary of reference document: A guide to EU Standards and Conformity assessment

Version prepared by Insee


A Guide to EU Standards and Conformity AssessmentNIST (USA) -Special Publication 951Aim : This document is designed to help businesses and government officials understand the new laws, the EU's standardization process, and the relationship between the European Commission and the European standardization bodies in the European Union. It also provides information on the EU's approach to conformity assessment and requirements for obtaining CE mark to gain access to the European Market.


“standard”The philosophy of standardization by committee and consensus is the same in the EU as it is in the United States. - (page 19)



For some standards, covered by the so-called “New approach” - defined by a few directives- and when the products are linked to safety, health and environment, there is a high level of requirements and binding conditions..Standardization, when used specifically to translate laws into technical solutions for implementation, is tantamount to technical regulation. (page 19)-

ii. Form of appearance

1.physical measurement standards 2.documentary standards

Not mentioned

iii. Manner in which they specify requirement

1.design 2.performance

Not mentioned

122

iv. What it relates to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex)

Risk assessment, with a view to security and health of users.


Not mentioned

vi. Intended user group in geographical hierarchy

1. company 2. national 3. regional 4. international

This is a two-level organisation: at the EU level, CEN works with Member state - Standard Organisations. Nationally, Standard organisations work with industry representatives or businesses.


All market domains


Risk assessment under the responsibility of the producer (risk-assessment defined by a specific standard) + legal liability for the producer




The specific standards required by the directive on the New approach are controlled by the EU in order to secure that their implementation will be in line with the general market policy of the EU.

e. Specificity

123


3. Process of standardisation (WP2)a. Identification of

needs, decision on the development a special standard (Who will decide?)

Needs are identified by CEN - this is linked with National Standardisation Organisations . In addition, due to agreements with ISO/ IEC, some standards are recognised automatically.


Control by CEN for links with security and health + environmentNeed for conformity assessment in matters like health / security


Same as in the US… (page 19).


vote of national standard organisations at CEN



124


Three levels depending from the risk for health and security:

i. self-certification (Declaration of conformity)

ii. self-certification + quality assurance certification

iii. assessment by an independent institution.

+ definition of standards for risk assessment: EN1050 for machinery and EN1441 for Medical devices

self-certification: (page 24)The following main elements should be present in a Technical File:a general description of the product;design and production drawing and diagrams;detailed technical data for essential aspects of the product;a risk assessment;a list of standards and or solutions applied;reports of calculations and tests that have been carried out (calculationsmight include a bearing, belt, or pulley based on a chart in a catalog);certificates and inspection reports;in the case of series production, internal conditions that have been observedto safeguard compliance with the directive;a user's manual; anda Declaration of Conformity.

In case of an incident or random check, […] the manufacturer, his authorized representative or importer, must be able to provide the Technical File within seven to ten days as of the date of request by the surveillance authority. (page 30)


5. Any other relevant content

product liability: The legal obligation to use good engineering practices in safety matters is not embedded in directives, but exists independently of them.“ (page 36)declaration of incorporation:The general principle behind a Declaration of Incorporation is that the manufacturer of a subassembly cannot be required to foresee all the ways the subassembly may be incorporated, and, more importantly, may not be able to provide all the safety aspects of the larger machine. (page 30)

125

N°2

Summary of reference document: Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control

Prepared by the UK.

1. General description of the document (Who prepared, when, for what purpose)

Independent, Transparent, Externally Audited: the ISO approach to Survey Process Quality Control.

Bill Blyth, TNS plc, UK


The document is not concerned with the standard itself – rather the reasons behind the development of the standard and the process for achieving and assessing compliance with the standard (ISO 20252).

2. Concept, main characteristics of standards (WP1)

ISO 20252 is a Process Quality Standard that is specific to the conduct of Survey Research, both quantitative and qualitative. It covers the process components of survey research.

a. Definition of standard The complete standard is a lengthy document. The core content is sections on:

Quality management system requirements Executive elements Data collection Data management and processing Project documentation

b. Types of standards (Concept of categorisation, and categories)

?

iv. How binding they are? (i.e. voluntary, mandatory)

Voluntary. However a similar UK standard has a constant 90%+ take-up.

v. Form of appearance (physical measurement standards, documentary standards)

Documented process standards (e.g. interviewer recruitment and training, validation of interviews, data entry, coding, etc)

vi. Manner in which they specify requirements (design, performance)

All – very detailed

126



vii. What it relates to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item),…complex)

Statistical processes, covering all GSBPM stages from initial user request to the provision of final deliverables – doesn’t include evaluation (I think – check documentation when available)Examples given in paper are:

Responding to research requests Contents of research proposals Project schedule Assistance by and cooperation with clients/users Managing sampling and data processing Monitoring the execution of research Research reporting Research records Recruitment and training of interviewers (including

content, duration and documentation of training) Ongoing training and appraisal Identification Data protection and confidentiality Validation Data entry Coding Editing File management Analysis Delivery Back-up Retention Data security Project documentation

viii. Intended user group in geographical meaning (company, national, regional, international)

International – specifically designed to be capable of implementation anywhere in the world where survey research is undertaken.

ix. Intended application domain (i.e. general, industry, statistics, business statistics, SBS)

Statistical surveys

x. Degree of freedom that remains when the norm is fully implemented

Standard has to be applied in full in order to gain ISO accreditation

127

(How can we assess it?)

xi. Any others to be listed


Not covered in document


“It is critical to the credibility and success of the standard that it is implemented in a manner that results in consistent and comparable assessment and certification between countries, assessment bodies, applicant enterprises and locations.”“.. there is a need to invest greater thought and effort into the assessment procedure [than for physical specifications]”“One possible approach … is detailed expansion of the content of each section and a description how practically these procedures could be put into place and managed. Where they wished to, companies had available the services of a number of consultants with experience of both the industry and assessment procedures who could provide support to this process. In parallel detailed guidelines were drawn up to specify procedures for inspection agencies.”

e. Conformity assessment

Proposal is to use independent inspection to ensure that the concerns of consistency and comparability are met, using guidelines such as those set out above


?

g. Link with the analysis of architecture of the overall production process

Covers whole production process for statistical surveys – although limited coverage is given to self-completion and internet interviewing

h. Influence of the use of the standard on the content

Consistent and comparable survey research from countries in all regions of the world. Provides a minimum guarantee of quality.

3. Process of standardisation (WP2)

a. Identification of needs, decision on the development a special standard (Who will decide?)

Response to international market and survey research community to the requirement of users of survey research for the specification of minimum common standards for the process components of survey research. This demand was particularly encouraged by the globalisation of the activities of many of the major users of survey research.

b. Process of the development of a standard (i.e. principles, participants, financing)

One of the standards’ organisations led the initiative, providing both the necessary legal, drafting and associated expertise together with the secretarial support.Invitations to participate. A working party of twenty countries. Drafting process was lengthy, detailed and consultative. At the initial meeting a number of broad principles were discussed and identified. Built on existing standards. Consultation was iterative with voting taking place where there were substantive issues that required a clear decision amongst the members. Process took three

128

years and involved five face-to-face meetings.Financing (I think!) is borne by the members of the working group.

c. Adoption, repository and maintenance of a standard

Final draft of the standard was voted on by members of the working group, then by all ISO members.Stored in ISO online catalogue.Not explicit how maintained – think it is driven by users of survey data or the standard requesting an update. The guidelines are currently being revised to cover internet interviewing.

d. Promotion and support for application

Not covered in paper – driven by user demand for consistency and known quality?

e. Conformity assessment (?)

See 2e above

f. Organisational matters See 3b above4. Any other relevant content The agreement of the standard was the culmination of over

fifteen years of work with a history that originated some thirty years ago!


Not obvious from paper. Will investigate further when have copy of ISO standard itself.

129

N°3Summary of reference document: Architecture-based approaches to international

standardization and evolution of business models

Prepared by INSEE


Architecture-based approaches to international standardization and evolution of business models

Junjiro Shintaku, lead author; Koichi Ogawa, Tetsuo Yoshimoto, co-authorsThe university of Tokyo- Manufacturing Management Research Center, Japan

The document is about standards mainly in industry, involving modularization, leading growth in global economy itself . Standardization is seen from a technical and architectural point of view. The benefits of standardization are also analysed.


Business model based on international standards

a. Definition of standard Business model : the product itself, result of the process, is determined as a standard (ex. : DVD). The business model is cut into functions allocated to some components

Modular architecture (as opposed to closed architecture):

Open technology and open design rule Weak interdependency among components World wide installed base Only one function is allocated to each

componentClosed local standards=closed architecture:

integral architecture complex and strong interdependency among

components more than one function is allocated to each

component closed technology & closed design rules


Examples of standards are found in technological industry, as DVD player or different kind of CD’s production. In a full chain of production, newly industrialized economies (NIEs) are specialized in some components and advanced countries cannot operate without : collaboration is needed.

130


Voluntary : open to every firm in the world, not only to advanced countries but also to developing countries.But every participant is obliged to respect the standardized final product and the way to reach it.

ii. Form of appearance (physical measurement standards, documentary standards)

DocumentaryBased on an alliance between firms. They allocate modules according to the expertise of each partner.

iii. Manner in which they specify requirements (design, performance)

Based on the division of labour. The industrial process is cut into sub-processes and then a modular architecture is built. However, it is not clear-cut if it is a process standard or a product standard.

iv. What it refers to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item),…complex)

Industrial processes mainly about new technologies.Related to a global economic growth model.The standardization is about the final product : the rules apply to the product itself through its business process (a set of functions to be realized), related to technical components.

v. Link with analysis of architecture of the overall process

Strongly linked, architecture and business process can’t be separated.

This requires a modular architecture, i.e. the association between one component and one single function of the final machine.Open international standards accelerated modularisation of products and promoted the international division of labour.

vi. Intended user group in geographical meaning (company, national, regional, international)

International Company (in the case of local standards)


Industry


None.

131


categories: International - open Local -closed


Precise description of the product


Market growth

e. Specificity ?f. Quality of

standards, criteria3. Process of standardisation

(WP2)a. Identification of

needs, decision on the development a special standard (Who will decide?)

International market and maximisation of profit - more precisely, how to create a bigger market for the final product, while sharing commercial risks without loosing the know-how in the most advanced technology.


The expected use requires to avoid sharing highly advanced technology, whereas cheap realisation of less advanced technology (including assembly) is crucial for the market development. The process has to be modified in order to become modular. Some modules are then standardised, not all.


As said above, the main principle is labour’s division and participation of several international firms. The leaders are the western company.


The standard of the product has been decided in an international open forum (the DVD forum) by 200 firms from 20 countries.


?


This kind of standard permits to create new business opportunities for NIEs.

g. Conformity assessment (?)


In the paper, the examples are taken from optical storage industry. The advanced design activity is kept under control, and not standardised.


132

N°4Summary of reference document:

The six dimensions of standards - Contribution towards a theory of standardization

Prepared by INSEE


Paper presented at the Seventh International Conference on Management of Technology, February 20, 1998

Authors: E. Baskin, K. Krechmer and M. H. Sherif

This document reviews the different approaches to multimedia communications to understand how this could affect standardization.It notes that the interests of standard seekers and standards creators are at opposite ends. To resolve that, it presents a framework for standards, to help organizations to plan their standard strategy.


The overlap of technologies in multimedia communications has caused many standards groups to address the issues independently and with little coordination leading to application-specific approaches. In such a fragmented environment, a better understanding of the theory of standards becomes more important, to focus resources in a productive way.

a. Definition of standard Technical standards may be described by answering six questions:

The strategic questions are: Why seek a standard? What is the category of product or service to be

standardized? When in the product cycle to standardize? The tactical questions are: Which is the appropriate Standards Development

Organization (SDO)? How will consensus be reached? Where will the standard be used?


The document depicts a layered taxonomy of technical standards into four strata: reference standards, similarity standards, compatibility standards and etiquette standards.


There are no regulatory obligations but a will, for the enterprises, to win or dominate markets. To increase their activity, organizations have to ensure compatibility. Consensus based standards rely on the positive self-interest of the parties involved.

ii. Form of appearance (physical

Reference standards include unit standards which define measurable physical qualities, e.g., ohm, volt, watt,.. or other norms like ASCII, OSI,...

133

measurement standards, documentary standards)

Similarity standards define the nominal value and the allowed variation for a set of similar entities.

Compatibility standards define the "transparency" of an interface between two or more mating elements that communicate with each others. Modem recommendations (V.32, V.34, etc.) or the packet switching recommendation X.25 are examples of compatibility standards: they define interfaces, sufficient aspects of the transmitter and the receiver pairs necessary to ensure communication.

Etiquette standards define the range and open ended variability of protocols.

iii. Manner in which they specify requirements (design, performance)

The applicability of a standard can be assessed in terms of market, industry or geography

iv. What it refers to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item),…complex)

To ensure an access “anywhere any time” for many traffic types, the design and engineering activities of many types of offerings are involved: digital, wireless, multimedia communications with new capabilities and services.

v. Link with analysis of architecture of the overall process

vi. Intended user group in geographical meaning (company, national, regional, international)

International..


Multimedia communications


It depends of the type of standard involved


c. Items to be included in a In the field of multimedia communications, standards specify

134

standard the methods for interaction, the measures of equivalence, the limits of performance, quality, or degrees of compatibility. They may also include safety, reliability and test methods.

d. Measures on implementatione. Specificityf. Quality of standards, criteria3. Process of standardisation

(WP2)a. Identification of needs,

decision on the development a special standard (Who will decide?)

Precursor products or services provide sufficient proof of the technology or market interest to stimulate work on standards.


c. Process of the development of a standard (i.e. principles, participants, financing)

In terms of their position in the product cycle, there are different kinds of standards : anticipatory standards, participatory standards and responsive standards.Anticipatory standards are those standards that must be created before widespread acceptance of the device or services. They can emerge from the cooperation among existing competitors, or by new entrants that offer a precursor of the potential standard product or service.Participatory standards are developed, tested and used in an interactive environment.Responsive standards occur to codify a product or service that has been sold with some success.


e. Maintenance of a standardf. Promotion and support for

applicationg. Conformity assessment (?)4. Possibility to standardise a

design activity5. Any other relevant content

135

N°5Summary of reference document

Prepared by KSH

1. General description of the document Title: CEN’s Horizontal European Services Standardization Strategy, CHESSS Feasibility Study55

Author: Partnership of the National Standards Bodies of Spain (AENOR), Britain (BSI), Germany (DIN), Denmark (DS), Estonia (EVS) and the Netherlands (NEN), and participation from CEN Management Centre (CMC) and Cap GeminiFunded by: European Commission and European Free Trade Association (EFTA)2008Purpose: determine the feasibility of the development of a generic service standardization in Europe


“standard”Concerning the intended standardisation the project team recommends in the feasibility study that the single generic European service standard be written as a requirement specification, because it enables service providers to be assessed against objectively verifiable requirements in the standard and demonstrate compliance



Possible interfaces of legislation and standards: Including legal requirements directly in

service standards Including references to technical

regulations (for example, by mentioning the identification code and the title of the technical regulation, with its date of approval and / or the official journal in which it was published) in the service standards

Standards are developed in order to provide solutions that enhance compliance with legislation

Public administrations, which are responsible for developing

55 The template was filled in according to the general report and to annex G (Guidance Document for the Preparation of Service Standards) of module 1 (Guidance in the preparation of service standards)

136

regulations, can refer to the existing voluntary standards in their technical regulations, if they consider that said documents can be beneficial to society



documentary standards


2.performance

Performance in services


all services in EU

service standard can include requierements which refer to:

Facilities and equipment (hotels, golf courses, etc.)

Products (fish retail trade, translation services – the text, postal services, etc.)

Activities (market research) Customer perception (retail trade,

customer call centres, etc.) Processes (UNE draft standard on

hotels – Tourism)


it covers aspects of service provision that are generic across all service sectors:

design information provision to customers billing complaint & redress innovation & review

vi. Intended user group in geographical hierarchy 1. company 2. national 3. regional 4. international

European

vii. Intended application domain (i.e. general, industry, statistics,

services

137

business statistics, SBS)viii. Degree

of freedom that remains when the norm is fully implemented (How can we assess it?)


Classification: generic service standards

(horizontal standards - a applicable across multiple or all service sectors)

service-specific standards (apply to a single service or a specific aspect within a single service)


According to the feasibility study the aim is the integration of existing sectoral standards in a comprehensible, logical and transparent way to the generic service standardisation. In addition for each services, standard can include requirements which refer to:

Facilities and equipment (hotels, golf courses, etc.)

Products (fish retail trade, translation services – the text, postal services, etc.)

Activities (market research) Customer perception (retail trade,

customer call centres, etc.) Processes (UNE draft standard on

hotels – Tourism)


e. Specificityf. Quality of

standards, criteria

If a quality mark is created according to the generic standard, the certification and quality marking scheme must allow for all three assessment options:

138

― 1st party (i.e. self-declaration) ― 2nd party (i.e. assessment by an interested party) ― 3rd party (i.e. assessment by an independent body)

3. Process of standardisation (WP2)a. Identification of needs, decision on the

development a special standard (Who will decide?)

The aim is that the standard promotes the increase of trade of services in the EU (customers’ confidence, comparability, possibility of assessment). The project was funded by European Commission and European Free Trade Assossiation (EFTA)


In the case of services the assessment of the output is much more difficult than in the case of products, service standards are related to the processes of production of this services and circumstances of services.


1) identifyig in advance the purpose of the standard and subsequently identifying the requirements and recommendations to be considered in the document2) identifying requierements to be included in the standard (according to the service lifecycle)

Service standard can be drafted following different methodologies; these are some of the possible ones: - According to internal processes, business areas or functional areas, e.g., some Spanish or Chilean standards for tourist services, or UNI 11067 Standard on Managing consulting, criteria for supply and check, or AS / NZS Standard on Laundry practice. - According to the service provision sequence, e.g., Restaurant services (UNE 167…), Translation services (EN 15038), Event stewarding (BS 8406). - According to customer needs and expectations;the standard sets requirements in accordance with the different generic customer expectations (reliability, response capability, competence, courtesy, etc.) e.g., UNE Standards on retail trade.

d. Adoption, repository of a standarde. Maintenance of a standardf. Promotion and support for application

139

g. Conformity assessment 4. Possibility to standardise a design activity The service standard can be drafted

following different methodologies; these are some of the possible ones: - According to internal processes, business areas or functional areas, e.g., some Spanish or Chilean standards for tourist services, or UNI 11067 Standard on Managing consulting, criteria for supply and check, or AS / NZS Standard on Laundry practice. - According to the service provision sequence, e.g., Restaurant services (UNE 167…), Translation services (EN 15038), Event stewarding (BS 8406). - According to customer needs and expectations;the standard sets requirements in accordance with the different generic customer expectations (reliability, response capability, competence, courtesy, etc.) e.g., UNE Standards on retail trade.

5. Any other relevant content Scope of sector specific service standard:What does the standard describe?

The entire service provided by the service provider (e.g. an airline service)

One or some of the subservices ( e.g. on-board service on a plane)

All or some service categories (e.g. Business class, safety-related services)

Who shall meet the standard requirements? The service provider Entities subcontracted by the

provider Who shall offer guarantees of

compliance and is held liable in case of non-compliance

Whom is it addressed to? All or some customer segments

(e.g. elderly, children, disabled)

Structure of the service standard: Service is described following a

service provision sequence, Service requirements are

established to directly meet customer‘s needs and expectations,

Service is described depending on

140

the internal organization of the service provider,

Open model, Other models different from those

mentioned above, A mixture of two or more of the

above.

There is a strong interface between service standards and management system standard.

141


Prepared by DESTATIS and KSH


Title: “Annex C: CEN Workshop Agreement (CWA) Certification rules for manufactured products in Internal Regulations PART 3 TERMS OF REFERENCE OF THE CEN CERTIFICATION BOARD”, prepared by CEN (2006) to describe the conformity assessment process.The CWA Certification System is a voluntary European certification system for demonstrating the compliance of products with the requirements of the relevant CWA(s)


“standard”reference to ISO definition (ISO/IEC 17000:2004)?



voluntary



documentary standard


2.performance

n.a.

iv. What it relates to (ie. terminology, classification, product,

to certification of manufactured products on the basis of CEN Workshop Agreements only

142

service, process (if statistical process, which GSBPM item), … complex)


n.a.



European & CEN Management Centre (CMC) ensures the necessary liaisons with the other European and international standardization bodies, and various other organizations and reporting to CCB


Industry.


n.a.


n.a.


n.a.

143


CEN Certification Board is monitoring the implementation in CEN of the CWA Certification System and overseeing its progress and effectiveness with the assistance of the CEN Management Centre (CMC)

e. Specificity n.a.f. Quality of

standards, criteriageneral competence criteria and conditions are specified for certification bodies and their own or sub-contracted testing laboratories and/or inspection/assessment bodies

3. Process of standardisation (WP2)a. Identification of needs,


CEN Certification Board is enlisting the advice of CEN Workshops, where appropriate, and industry associations


n.a.


CEN Certification Board is responsible


common rules are applied to manage the administration of CWA certification

e. Maintenance of a standard CEN Certification Board is responsible.f. Promotion and support for

applicationCEN Certification Board is responsible

g. Conformity assessment CEN Certification Board is responsible.

CCB is responsible for the overall administration of the CWA Certification System among others: - monitoring the implementation in CEN of the CWA Certification System and overseeing its progress and effectiveness with the assistance of the CEN Management Centre (CMC);- examining and deciding on proposals for new CWA Certification schemes and evaluating and deciding on the acceptance of existing certification schemes as being suitable for granting the CWA Certificate of conformity and in compliance with the CWA Certification rules;

In detail:

CWA Certification System Rules, procedures and management for carrying out certification related to products on the basis of CWAs adopted by CEN.

144

CWA Certification scheme Set of specific product-related requirements for the conformity assessment procedure to be applied to the CWA Certification System for carrying out certification of products which are in conformity with (a) specific CWA(s) and to which the same specified requirements, rules and procedures apply. CWA Certification licence Document issued under the rules of the CWA Certification System, by which an empowered certification body grants to a supplier the right to use the CWA Certificate of conformity for its products in accordance with the rules of the relevant CWA Certification scheme. CWA Certificate of conformity Document issued under the rules of the CWA Certification System, providing confidence that a duly identified product is in conformity with the relevant CWA(s).


n.a.

5. Any other relevant content The document deals also with the followings:

content of the certification scheme administration rules certification procedure (including

surveillance procedure) rules managing the modification both in the

production process and in the conformity assessment scheme

correction actions validity of certificates complaints

145


Prepared by DESTATIS (Kai Lorentz) and KSH


Title: “ISO and conformity assessment”, prepared by ISO (2005) to describe the general benefits (user’s point of view) from conformity assessment to standards, regulations or other specifications.

2. Concept, main characteristics of standards (WP1)a. Definition of “standard” nob. Types of standards (Concept of categorisation, and categories) The aim is to

identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned!


voluntary



documentary standard


2.performance

design


Products, services, materials, processes, systems and personnel measures – as scope of conformity assessmentProcess - as Conformity assessment as such is a standard on a process

146


n.a.



all


General, but typically industry and servicesAny type of conformity assessment, especially conformity to ISO/IEC standards


Not general rule, form depending on the results of risk assessment. The ISO standards described originally are voluntary.


n.a.


n.a.


This is a general description on the objectives of conformity assessment and its mechanisms. Conformity assessment items are covered in several ISO norms, which are referred in this document.

e. Specificity n.a.f. Quality of standards,

criterian.a.

3. Process of standardisation (WP2)a. Identification of n.a.

147

needs, decision on the development a special standard (Who will decide?)b. Identification of the influence of the expected use of the standard on the content

n.a.


n.a.


n.a.


n.a.


n.a.


The document presents ISO conformity assessment, concepts, goals, content, types and organisational matters.Conformity assessment means checking whether products, services, materials, processes, systems and personnel measure up to the requirements of standards, regulations or other specifications.Conformity assessment provides benefits for

manufacturers and service providers: allows them to distinguish themselves from less reputable suppliers

consumers: basis for selecting products and services supported by a formal supplier declaration, a mark or certificate of conformity, that attest to quality, safety and other desirable characteristics.

Regulators: gives them means to enforce legislation and achieve public policy goals.

Transparent conformity assessment avoid costly multiple testing and/or certification of products.Conformity assessment may consist of any one of, some of, or all of the following: sample testing, inspection, process evaluation, supplier’s declaration of conformity, management system certification/registration, personnel certification, product certification, mutual recognition of results and the accreditation and peer assessment of the competence of the organizations conducting these activities – collectively known as “conformity assessment bodies”


n.a.

148

5. Any other relevant content This document can be considered as a standard on conformity assessment (in this case WP1 items are relevant) and as a standard-related activity, as conformity assessment of international standards (in this case WP2 is the relevant phase). On the web page other useful pages are available like “How conformity assessment works” – with first, second and third party assessment characteristics, the role of risk in the selection of type. “Mechanisms for performing conformity assessment” includes the references to the concrete ISO norms describing declaration, certification rules in different cases.

149

N°8Summary of reference document: The ABC’s of standards activities

Prepared by Insee


The ABC’s of standards activitiesNational Institute of Standards and Technology (USA)

NISTIR 7614

The report is intended to provide basic information to help educate U.S. government agency officials, legislative staff, industry, and other standards professionals to make informed decisions regarding standards development policies and the effective allocation of limited standards development resources. It is also hoped that this report will serve as a starting point for further study and discussion on some of these important issues.

2. Concept, main characteristics of standards (WP1)a. Definition of “standard”

page 5:„Documentary standards [, which] are written agreements containing technical specifications or other precise criteria that may contain rules, guidelines, or definitions of characteristics. Standards ensure that materials, products, personnel qualifications, processes, and services are:

adequate for their purpose, compatible and/or interchangeable, if necessary;

o ensure public health and safety;o protect the environment;o and/or improve economic

performance.”b. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned!

150


Yet another classification scheme distinguishes between voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. Mandatory standards are set by government regulatory agencies at all levels -- state, local, and federal. They are usually included within the regulations of the government agency with applicable jurisdiction. Such regulations or mandatory ―standards, generally establish requirements for public health and safety, consumer protection, environmental protection, national security, or other similar criteria. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them ―mandatory standards56. (page 7)


2.documentary standardsiii. Manner in

which they specify requirements 1.design

2.performance

Those standards that describe how a product is supposed to function are called performance standards. In contrast, design standards define characteristics or how the product is to be built.It should also be noted that the determination of conformance to performance standards may be more difficult than for design standards. [...] Performance standards are also more difficult to write. Performance standards are also more likely to allow the inclusion of technological innovations in the product and to prevent unnecessary barriers to trade. (page 7)

56 :: Note by Insee: example ( see the EU ”pet regulation” 438/2010 that requires the use of ISO standards 11784 and 11785 for the identification of EU pets).

151


The NIST refers explicitly to the ISO/IEC Guide 2:2004, which is analysed in another form.




Standards may also be classified by the intended user group or by the standard‘s developer. There are company standards, developed for use by a company or organization for its own products or for the products it purchases. There are also international standards, most of which are developed and promulgated by international governmental and nongovernmental organizations, such as the International Telecommunication Union (ITU) (governmental) and ISO (nongovernmental). (page 6)

The U.S. system is also tremendously diverse and the result is a system that is largely sectoral in its focus. This is a logical approach because each industrial sector, such as the information technology, telecommunications, automotive, medical devices, and building technology sectors, is most likely to understand that sector‘s needs and to know what standards best meet those needs. (page11).

152



It should also be noted that the determination of conformance to performance standards may be more difficult than for design standards. [...] Performance standards are also more difficult to write. Performance standards are also more likely to allow the inclusion of technological innovations in the product and to prevent unnecessary barriers to trade. (page7)


page 6:Government regulatory standards are those designed to be used by federal regulatory agencies in rulemaking and related activities. These should not be confused with Federal and Military Specifications, which are officialdocuments used by agencies and by the Department of Defense respectively, to support government procurement. Specifications are a set of conditions and requirements that provide a detailed description of a procedure, process, material, product, or service for use primarily in procurement and manufacturing


Standards can cover many aspects of the conformity assessment process. They can describe characteristics of the product for which conformity is sought; the methodology (e.g., test, inspection or other assessment methods) used to assess that conformity; or even the conformity assessment process itself (e.g., how a certification program or conformity assessment body should be operated). page 22

d. Measures on implementatione. Specificity

153


„However, when standards are poorly written, they can cause significant economic damage. Poorly written standards can raise transaction costs, reduce product safety and quality, and create barriers to trade. They can also constrain innovation; entrench inferior technologies; and impede the development of interoperable products and systems.” (page 2)

„The U.S. standards development system serves the national interest well. In most cases, it supports efficient and timely development of product and process standards that meet economic and public interests.” (page 10)

Problems can also occur when standards are not based on sound science. (page 24)

3. Process of standardisation (WP2)a. Identification of needs, decision on the development a special standard (Who will decide?)

The process is described for CEN (page 20): CEN is a private, not-for-profit organization, composed of the national standards bodies in the EU and EFTA. CEN develops standards in all fields except the electrotechnical area. In addition to its market driven standards efforts, CEN also develops standards in support of EU and EFTA governmental policies and regulations (directives) in response to formal governmental mandates. Under the so-called New Approach, EU regulations (directives) are limited to establishing ―essential requirements. These essential requirements are obligatory and are formulated in general terms. The detailed technical specifications necessary for the implementation of directives are entrusted to European, voluntary standards organizations like CEN.


154


However, certain principles for standards development are widely accepted. Consensus. Consensus means that all views are heard and the resultant standard is generally agreed to by those involved. Consensus is characterized by the absence of sustained opposition to substantive issues. However, it does not necessarily imply unanimity. Transparency. Transparency means: (a) providing advance public notice of a proposed standards development activity; (b) identifying the scope of work to be undertaken; (c) providing information on conditions for participation; (d) and providing an opportunity for all interested parties to comment prior to final approval and adoption. Balance. Balance means that no one interest, including the government, should dominate. It should be noted that balance can be affected not only by the number of participants in particular categories but also by the funding source. The provider of the funding in standards development work can sometimes end up dominating the process. This is particularly true if the funding is from a government entity. If funding is to be provided by a government agency or other entity, care must be taken to avoid undue influence on the outcome of the process by the funding provider. Due Process. Due process means that any person (organization, company, government agency, individual, etc.) with a direct and material interest has a right to: (a) express a position and the basis for it; (b) have that position considered; and (c) appeal if adversely affected. Due process ensures equity and fair play in the standards development process. Openness. The standards development process should be to open to participation by all materially affected interests. (page 8).


155


Unless the original technical experts that developed the standard participate in its revision, the reviewers may not be able to fully understand how the document was prepared, what was eliminated from consideration, and the reasons or assumptions underlying decisions and the resultant provisions. Problems can also occur in the application of specific provisions if the intent behind them is unclear. Rationale statements, which sometimes accompany a standard, are specifically designed to define the purpose and scope of the standard, to explain the criteria used in developing its requirements and to provide all other relevant information at the disposal of the developers. (page 24)

f. Promotion and support for applicationg. Conformity assessment

„A product‘s conformance to accepted standards readily provides an efficient method of conveying complex information on the product‘s suitability.” (page 2)

Conformity assessment includes any activity concerned with determining directly or indirectly that requirements for products, services, systems, personnel qualifications and organizations are fulfilled. Conformity assessment includes: sampling and testing; inspection; supplier's declaration of conformity; certification; and management system assessment and registration. It also includes accreditation of laboratories, certifiers, inspection bodies, and management system registrars, and the recognition of the competence of accreditation bodies. Conformity assessment activities may be conducted by the manufacturer/supplier (first party), by the buyer (second party) either directly or by another party acting on the supplier's or buyer's behalf, or by a body not under the control or influence of either the buyer or the seller (third party). It can also be conducted by a government agency acting in a regulatory capacity. (page 22).

Knowing what aspects of the product will be evaluated in a conformity assessment process and whether there are other aspects which might impact quality, safety, or performance allows the user of the conformity assessment data to evaluate the data‘s significance. (page 23)

156

h. standard developing organisation

Compared with umbrella-type standards organizations that operate in other nations or at the international level, the more specialized U.S. SDO57s also tend be quicker to generate standards needed by industry. (page 11)

Among the private sector bodies, the International Organization for Standardization (ISO) is probably the largest producer of International Standards, having issud over 16,000 standards. (page 16)



57 SDO= Standard developing organisations

157

N°9Summary of reference document: CEN leaflet

Prepared by Insee


Presentation of CEN - general communicationversion September 2010Explains the way it is operating.


No


i. How binding they are? (i.e. voluntary, mandatory)ii. Form of appearance 1.physical measurement standards



2.performanceiv. What it relates to

(ie. terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex)



The CEN is coordinated with the NSOs of Member states, and has agreements with ISO. CEN, CENELEC (electrotechnical standardisation) and ETSI (standardisation for information and communication technology) have close links.


Covers 22 domains. One of them is “Services”:tourism, facility management, postal services, cinematographic works, real estate agents, customer contact centres, supply chain security, (engineering) consultancy services, sheltered housing for the elderly, services of chiropractors, aesthetic surgery

158

services, airport and aviation security services etc.ix. Degree of freedom

that remains when the norm is fully implemented (How can we assess it?)

i. Any others to be listed

c. Items to be included in a standardd. Measures on implementatione. Specificityf. Quality of standards, criteria

3. Process of standardisation (WP2)a. Identification of needs, decision on the development a special standard (Who will decide?)

Most standards are initiated by industry. Other standardization projects can come from consumers, Small and Medium Enterprises or associations, to name some other sources. In addition, many standards are developed to support European legislation. (page 4)


‘Reference to standards’ within a legislative text is viewed as a more effective means of ensuring that products meet the essential health and safety requirements of legislation than the writing of detailed laws. This allows both processes to support each other, without causing a slowdown. (page 4)


page 7:1- Proposal2- Acceptance3- Drafting4- CEN Enquiry5- Adoption by weighted vote6- Publication of the European Standard (EN )duration: 3 years

d. Adoption, repository of a standarde. Maintenance of a standardf. Promotion and support for applicationg. Conformity assessment

specific procedure with the keymark


5. Any other relevant content page 7:Two intermediate products:CEN Workshop agreement: agreement between participants, but not the status of a standard - no obligation for NSOs to adopt them as national

159

standardsCEN Technical specification: pre-standard for innovative features or coexistence of various alternatives.

160

N°10Summary of reference document: Market studies, social and opinion surveys- Vocabulary

and service requirements (ISO standard 20252:2006)

Prepared by Insee

1. General description of the document Since many customers ask for internationally or regionally comparable information, such a standard helps in guaranteeing the comparability of the information. This comparability will then be traceable. The documents contains the definition of 65 terms.


No (there is one ISO definition of s standard).



The standard is not mandatory in general, by definition of an ISO standard. It will be mandatory only if a national/ regional law requires this.However, there are many binding requirements that are defined by the standard. Thus, they are binding for companies that claim implementing this standard.

ii. Form of appearance 1.physical measurement standards 2.documentary standards

documentary.

iii. Manner in which they specify requirements 1.design 2.performance

Mainly process standard, some aspects are of a “product standard” kind: this is the case for the response rate. Overall, ISO would qualify it as a service standard.


Terminology, complete process from design to dissemination. It deals as well with encompassing aspects like a management quality system, training of staff, data protection, quality reviews, continuous improvement, ...


The norm covers quite typically surveys as they are conducted by companies for their customers. When the process is only partially covered by the company - but subcontracted under its responsibility - the company is responsible for the work done by the subcontractor.

161


International


opinion surveys, market analysis, social surveys…


The standard requires much transparency with the client and much traceability. If the client does not require much control, the degree of freedom for the surveying company is higher. There are strict requirements to secure a minimum level quality, except aspects due to the quota method. Higher requirements are possible…The quota method leads to much freedom in the assessment of uncertainty.


c. Items to be included in a standard quality control by the customer, transparency

reliability of results access to the questionnaire and the

guidelines for field staff, the translation of questionnaires

pilot surveys are highly recommended training of interviewers traceability of all actions

d. Measures on implementation all the quality control and the requirements for traceability are closely linked to the implementation.

e. Specificityf. Quality of standards, criteria3. Process of standardisation (WP2)a. Identification of needs, decision on the development a special standard (Who will decide?)b. Identification of the influence of the expected use of the standard on the contentc. Process of the development of a standard (i.e. principles, participants, financing) and related organisational mattersd. Adoption, repository of a standarde. Maintenance of a standardf. Promotion and support for applicationg. Conformity assessment Possible for the data collection, coding, even

imputation. Very few on the choice of estimators and quotas.

162


No clue in this standard.


163

N°11Summary of reference document: The Advisory Panel On Telephone Public Opinion Survey Quality (Final report Feb. 2007)

Prepared by ISTAT


The Advisory Panel On Telephone Public Opinion Survey Quality (Final report Feb. 2007)

The purpose is to provide advice and guidance to Public Opinion Research Directorate (PORD) on survey quality standards and benchmarks appropriate to public opinion telephone survey research conducted for the Government of Canada.For this purpose a Technical Advisory Panel was established, consisting of eight members belonging to:

- Government of Canada- Market Research Industry- Academic Community.

2. Concept, main characteristics of standards (WP1)b. Definition of “standard” Standard = Practices that should be requirements

for all telephone studies conducted by Government of Canada Guidelines = Practices that are recommended, but would not to be requirements; that is, known good practices or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study.

c. Types of standards (Concept of categorisation, and categories) The aim is to identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned!

The report is organized in five sections, corresponding to the five main phases of a telephone public opinion survey:

- Pre-field Planning, preparation and Documentation- Sampling and Data Collection- Response Rate- Data Management and Processing- Data Analysis/Reporting and Survey Documentation.

For each of these sections, it defines Standards and/or Guidelines. i. How binding they

are? (i.e. voluntary, mandatory)

164



Documented process standards.


2.performance

VeryDetailed description for each element (activity, method, etc.) pertaining to each section


It relates to statistical process, to all the GSBPM items, even if not explicitly referred to.


There is a link through GSBPM



Company (commissioned by the Canadian Government)


Statistics


Limited degree of freedom


d. Items to be included in a standard

165

e. Measures on implementation

They are to be included in all calls for tender of the Canadian administration

f. Specificityg. Quality of standards,

criteria3. Process of standardisation (WP2)a. Identification of needs, decision on

the development a special standard (Who will decide?)

The need was to ensure a common approach to assessing data quality and reporting data quality indicators in survey reports commissioned by Government of Canada, as there were not standards or benchmarks concerning these aspects.


Page 10 of the paper:

The objectives for the Panel were to:• Review and discuss current Canadian and international standards, benchmarks, and documentation practices related to telephone survey quality, with a particular focus on response rates and population coverage.• Review and discuss the Government of Canada context for undertaking public opinion research.• Advise on appropriate telephone survey quality standards for Government of Canada public opinion research.• Advise on specific benchmark levels of quality indicators for public opinion research telephone surveys conducted by private sector suppliers on behalf of the Government of Canada.These benchmarks will be considered for inclusion in the 2007 renewal of the contracting tools for public opinion research.

There is a very strong influence of the expected use on the content of the standard itself.


Already explained in box 1

d. Adoption, repository of a standard page 8: “The role of the Panel was also to reach consensus where possible, although this was not an essential outcome of the work of the Panel.”

e. Maintenance of a standardf. Promotion and support for

applicationg. Conformity assessment

166


There is for instance a very interesting summary of this issue on page 18:

There was agreement that for questionnaire design only very broad standards and guidelines are required, and several reasons were given for why this overall approach should be adopted.• It is unnecessary to develop standards/guidelines for questionnaire design because this is primarily the responsibility of the research firms.a) The industry has good “rules of thumb” to follow relating to every aspect of questionnaire design.b) Research firms already need to demonstrate their capabilities in the RFSO or an RFP by the requirement to provide both project credentials and references.• It is both unrealistic and undesirable to impose either standards or guidelines on an activity, i.e., questionnaire development, that:a) Is as much art as it is scienceb) Needs to be tailored to the specific information needs of each GOC survey and the different departmental requirements in certain areas (e.g., inclusion of references to specific pieces of legislation is required in some departments but not in others)• It is best left to the research industry to develop a set of standards and guidelines for questionnaire design, not the POR community in the GOC. Notably, though, the point was also made that the research industry itself does not have enough of a consensus on best practices upon which to base standards.• There is a need to avoid creating a process that becomes unduly burdensome for both GOC and suppliers in terms of the level of effort, time and financial resources that are devoted to developing and finalizing survey questionnaires.


167

N°12Summary of reference document: Office of Management and Budget58 - standards and

guidelines for statistical surveys

Prepared by Insee


page 1:In 2002, the U.S. Office of Management and Budget (OMB), in response to [...] the Information Quality Act, issued government-wide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies” (67 FR 8452-8460; February 22, 2002). Federal statistical agencies worked together to draft a common framework to use in developing their individual Information Quality Guidelines.


No, but they are exhaustively listed, accompanied by guidelines.page 1: Standards for Federal statistical programs serve both the interests of the public and the needs of the government. There is no indication of a consensus, this is required by the government.

Each standard has accompanying guidelines that present recommended best practices to fulfill the goals of the standards.


i.(i.e. voluntary, mandatory)

page 1: These standards document the professional principles and practices that Federal agencies are required to adhere to and the level of quality and effort expected in all statistical activities.

page 4:Federal agencies are required to adhere to all standards for every statistical survey, even those that have already received OMB approval. Agencies should provide sufficient information in their Information Collection Requests (ICR) to OMB under the Paperwork Reduction Act (PRA) to demonstrate whether they are meeting the standards. OMB recognizes that these standards cannot be

58 : USA

169

applied uniformly or precisely in every situation. Consideration will be given to the importance of the uses of the information as well as the quality required to support those uses. If funding or other contingencies make it infeasible for all standards to be met, agencies should discuss in their ICR submissions the options that were considered and why the final design was selected.The agency should also include in the standard documentation for the survey, or in an easily accessible public venue, such as on its web site, the reasons why the standard could not be met and what actions the agency has taken or will take to address any resulting issues.

ii.1.physical measurement standards 2.documentary standards

iii.specify requirements 1.design 2.performance

iv.terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex)

page 2: classical steps of the statistical process:

Development of concepts, methods, and design

Collection of data Processing and editing of data Production of estimates and projections Data analysis Review procedures Dissemination of Information Products.

v.architecture of the overall production process

vi.geographical hierarchy 1. company 2. national 3. regional 4. international

applies to all US statistical agencies

vii.domain (i.e. general, industry, statistics, business statistics, SBS)

cross-domains

viii.remains when the

budget allocation + choice of professional & technical methods

170

norm is fully implemented (How can we assess it?)

ix.c. Items to be included in a standard

page2:The standards and guidelines are not intended to substitute for the extensive existing literature on statistical and survey theory, methods, and operations. When undertaking a survey, an agency should engage knowledgeable and experienced survey practitioners to effectively achieve the goals of the standards. Persons involved should have knowledge and experience in survey sampling theory, [...].

page 4:Additional information relevant to the standards can be found in other more specialized publications, and references to other Federal guidance documents or resources and the work of the Federal Committee on Statistical Methodology are provided in this document.

d. Measures on implementatione. Specificityf. Quality of standards, criteria not relevant here.3. Process of standardisation (WP2)

a. Identification of needs, decision on the development a special standard (Who will decide?)






g. Conformity assessment 4. Possibility to standardise a

design activityTwo standards deal with this issue (page i):

Survey Planning - Standard 1.1: Agencies initiating

171

a new survey or major revision of an existing survey must develop a written plan that sets forth a justification, including: goals and objectives; potential users; the decisions the survey is designed to inform; key survey estimates; the precision required of the estimates (e.g., the size of differences that need to be detected); the tabulations and analytic results that will inform decisions and other uses; related and previous surveys; steps taken to prevent unnecessary duplication with other sources of information; when and how frequently users need the data; and the level of detail needed in tabulations, confidential microdata, and public-use data files.

Survey Design - Standard 1.2: Agencies must develop a survey design, including defining the target population, designing the sampling plan, specifying the data collection instrument and methods, developing a realistic timetable and cost estimate, and selecting samples using generally accepted statistical methods (e.g., probabilistic methods that can provide estimates of sampling error). Any use of nonprobability sampling methods (e.g., cut-off or model-based samples) must be justified statistically and be able to measure estimation error. The size and design of the sample must reflect the level of detail needed in tabulations and other data products, and the precision required of key estimates. Documentation of each of these activities and resulting decisions must be maintained in the project files for use in documentation (see Standards 7.3 and 7.4).

Technically speaking, this is a process standard that lists all steps to be implemented. It refers as well to the documentation of this design, that has to be made partly public.

5. Any other relevant content These standards refer to very general principles, and have some similarity with the ESS Code of practice. However the latter is more general, whereas the former refers to some professional issues is several steps of the process. These US standards refer to the FCSM methodological documents.The ESS code of practice refers to international standards, which should all be used in the ESS, without any limitation. There is no reference to the ESS handbooks.

172


Prepared by DESTATIS and CBS NL


Title: “ESS Quality Assurance Framework”, prepared by the Sponsorship on Quality (SQ)Final Version 1 from June 2011 (internal version, not yet distributed outside the SQ

The QAF is based on the proposal for a new version of the Code of Practise (CoP). Its aim is to assist the implementation of the CoP. While the CoP sets the principles and indicators as standards, the QAF describes, for each indicator, activities/tools/methods that facilitates the implementation of the CoP. The QAF has been developed to support the implementation of principles 4 and 7 to 15 of the CoP in its revised version. For each indicator of these principles the QAF contains:

A. Methods at institutional levelB. Methods at product/survey levelC. Reference Documentation


“standard”b. Types of standards (Concept of categorisation, and categories) The aim is to

identify classifications! Report if any from i.-viii. are mentioned, list categories, and add under ix. if any additional types are mentioned!


The binding character is not clear yet. It may not have a legally binding nature, but be used for peer reviews and other forms of assessment about compliance with the CoP.



2


2.performance

Framework for Statistical Offices (performance?)

iv. What it relates to (ie.

Methods/tools/activities ata. Institutional level of

173

terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex)

ESS membersb. Product/Survey

level


?



Users are all members of the ESS, both Eurostat and the national statistical authorities


Mainly general, in some parts for specific domains (business register, household register)


High degree of freedom. Very often the QAF states, that the NSA should establish a procedure to …(e. g. validate the questionnaire). The choice of the procedure or method is left open.


?


?


Possibly Peer Reviews or Self assessments?

e. Specificity ?f. Quality of standards,

criteria?

8. Process of standardisation (WP2)h. Identification

of needs, The original idea was to find a common language for quality assurance inside the ESS.

174


Decision about the QAF and its application can be expected by the ESSC.

i. Identification of the influence of the expected use of the standard on the content

j. Process of the development of a standard (i.e. principles, participants, financing) and related organisational matters

k. Adoption, repository of a standard

l. Maintenance of a standard

m. Promotion and support for application

n. Conformity assessment



175

N°14Summary of reference document: European Code of Practice

Prepared by CBS NL


The CoP is a set of principles related to different aspects of quality and trust in statistics within the ESS. These aspects are the chapters of the CoP and are related to the highest institutional level till ‘lowest’ methodological aspects.

2. Concept, main characteristics of standards (WP1)a. Definition of “standard” No


x. How binding they are? (i.e. voluntary, mandatory)

The standard is not mandatory (yet).

xi. Form of appearance 1.physical measurement standards


documentary.

xii. Manner in which they specify requirements 1.design

2.performance

Design and organisation of mostly organisational en institutional aspects of production of statistics to ensure trust and reliability.

176

xiii. What it relates to (ie. terminology, classification, product, service, process (if statistical process, which GSBPM item), … complex)

xiv. Link with analysis of architecture of the overall production process

xv. Intended user group in geographical hierarchy 1. company


International, national and company. Focus of CoP is on the national level. A country is responsible for reliable statistics, enough budget etc. Eurostat and NSI’s, the company level, are using the CoP as reference model for their production process. International the CoP is used to build trust in statistics.

177

xvi. Intended application domain (i.e. general, industry, statistics, business statistics, SBS)

general

xvii. Degree of freedom that remains when the norm is fully implemented (How can we assess it?)

Indicators are defined for each principle. The Quality Assurance Framework is needed to assess the degree of implementation.

xviii. Any others to be listed

c. Items to be included in a standardd. Measures on implementation

The QAF is meant to measure the implementation. The QAF is under development. For the time being peer reviews and reports on CoP are measuring implementation.

e. Specificityf. Quality of standards, criteria

3. Process of standardisation (WP2)a. Identification of needs,


ESSC decides. Is owner of the standard.


178


Could be special task force or ESSnets. Now a Sponsorship group is dealing with revision. Participants are ESS members, normally from NSI’s. But also ECB, ESGAB and others are consulted.


Adoption by ESSC

e. Maintenance of a standard Revision on initiative of ESSCf. Promotion and support for

applicationCoP is used as a reference document for all NSI’s.

g. Conformity assessment 4. Possibility to standardise a

design activityNo clue in this standard.


179