statistical quality control the experience of the banco de portugal antónio agostinho maria josé...
TRANSCRIPT
Statistical Quality ControlThe experience of the Banco de
Portugal
António AgostinhoMaria José Valério
Banco de PortugalStatistics Department Statistics Audit Unit
Q 2008 - European Conference on Quality, Rome July 2008
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 2
1. The Statistics Department2. Quality Control Procedures3. Statistics Audit4. Institutional Cooperation5. Conclusions and Prospects
Ahead
Outline
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 3
1.Statistics Department Created in 1997 to cope with rising statistical requirements (in particular in the framework of the Monetary Union)
Centralized statistical function allowing:
The concentration of the available statistical skills;
The increase of capacity to face the challenges resulting from building up the European System of Central Banks;
The improvement of efficiency gains and the enhancing of statistical quality control at all levels of the statistical process (collection, processing, analysis and dissemination);
The adoption of consistent methodological approaches; and,
An easier data sharing and coordination among all the staff in charge of statistical production.
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 4
1.Statistics Department
Employees with university degreeEmployees with university degree 88 ( 56% ) 88 ( 56% )
Other employees Other employees 69 ( 44% ) 69 ( 44% )
Total 157Total 157
National Financial Accounts and Securities Statistics Division
Total: 23 (Univ. Deg. 17)
National Financial Accounts and Securities Statistics Division
Total: 23 (Univ. Deg. 17)
Balance of Payments and Int.Invest. Position Statistics Division
Total: 37 (Univ. Deg. 15)
Balance of Payments and Int.Invest. Position Statistics Division
Total: 37 (Univ. Deg. 15)
Monetary and FinancialStatistics and Central Credit
Register DivisionTotal: 20 (Univ. Deg. 11)
Monetary and FinancialStatistics and Central Credit
Register DivisionTotal: 20 (Univ. Deg. 11)
Central Balance-SheetDivision
Total: 32 (Univ. Deg. 23)
Central Balance-SheetDivision
Total: 32 (Univ. Deg. 23)
Statistics Dissemination UnitTotal: 10 (Univ. Deg. 8)
Statistics Dissemination UnitTotal: 10 (Univ. Deg. 8)
Central Credit RegisterInformation Unit
Total: 14 (Univ. Deg. 3)
Central Credit RegisterInformation Unit
Total: 14 (Univ. Deg. 3)
Statistics Audit UnitTotal: 3 (Univ. Deg. 3)
Statistics Audit UnitTotal: 3 (Univ. Deg. 3)
MethodologicalDevelopments Unit
Total: 3 (Univ. Deg. 3)
MethodologicalDevelopments Unit
Total: 3 (Univ. Deg. 3)
Executive ManagementTotal:2 (Univ. Deg. 2)
Executive ManagementTotal:2 (Univ. Deg. 2)
Staff Support UnitTotal: 13 (Univ. Deg. 3)
Staff Support UnitTotal: 13 (Univ. Deg. 3)
December 2007
Organization Chart
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 5
1.Statistics Department
Four Divisions Four different areas of statistical
production
Vertical organization in each Division
Regular hierarchical statistical production meetings (on
a monthly basis)
Internal technical task forces to deal with transversal
issues
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 6
2.Quality Control Procedures
Quality control procedures along all the phases of
statistical production process: Data collecting Data processing and analysis Statistics dissemination
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 7
2.Quality Control Procedures Data Collecting -
1
Data collecting is based on legal provisions Involvement of the reporting institutions is pursued For more complex changes in collection systems working
groups are created Detailed reporting instructions and handbooks are delivered
to the reporting institutions Appointment of contact persons – respondents for each
specific statistical data submission (training sessions and
regular meetings)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 8
2.Quality Control Procedures Data Collecting -
2
Specific designed software is made available to the
reporting institutions - IT tests and consistency tests on
elementary data BP net – a secure electronic communication system
between the Banco de Portugal and the other financial
institutions Feedback to reporters – the CCR and CBS experiences Statistical reporting indicators
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 9
2.Quality Control Procedures
Quality Assessment Reports – Major Financial Groups
Cross-checking data reported by MFIs: MFIs’ report to money and banking statistics (MBS)
versus accounting data MFIs’ report to MBS versus report for Central Credit
Register MFIs’ report to MBS versus data collected/reported for
securities statistics Securities portfolios versus securities issues MFIs’ report to MBS versus report for BoP statistics
(external assets and liabilities) High level periodical statistical meetings
Data Collecting - 3
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 10
2.Quality Control Procedures Data processing and analysis - 1
Full compliance with the international methodological
framework: concepts and definitions scope of statistics classification and sectorization systems valuation rules recording principles
Adjustments and/or appropriate substitute measures
are developed when the source data are not in
accordance with the standards
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 11
2.Quality Control Procedures Data processing and analysis - 2
Validation tests performed on individual and/or
aggregated data and also at different phases of data
processing – source data, intermediate data and final
statistical data: internal consistency temporal consistency cross-check control
Additional control in the framework of Financial
Accounts compilation
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 12
2.Quality Control Procedures Statistical Dissemination -
1
Meeting the users’ needs and the statistical reporting
requirements of International Organizations Main features of the statistics dissemination:
Release in time according to a pre-announced schedule
Clear and understandable presentation Workshops with media and media references
monitoring Monthly, free of charge, to all users at the same
time, in the BdP website:• Statistical Bulletin (also available in paper and
CD-ROM)• BP stat – allows online access to the interactive
database of BdP Assistance to users – contact points in BdP website and in
the Statistical Bulletin
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 13
2.Quality Control Procedures Statistical Dissemination - 2
BP stat | Statistics online | facilities: Portuguese and English versions; Data under two formats: time series and
multidimensional analysis; Data in hierarchical structure – tree of statistical
domains; Data and metadata may be accessed, downloaded or
printed; Internet links to the relevant metadata references; Predefined and/or personalised tables (“Favourites”
option); “Alert” messages by email to the users; and, Glossary, search, calendar and help options are
available.
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 14
2.Quality Control Procedures Quality Control
Matrix
Outside the
production cycle
Appointment of contact persons Detailed reporting instructions and handbooks
Regular meetings with the reporting institutions
Training sessions Feedback information to the reporters
"Quality Assessment Reports“ High level periodical statistical meetings Internal technical task forces
During the
production cycle
BP net – electronic communication
system
Specific designed software - IT tests -
Validation tests: internal consistency temporal consistency cross-check control
Regular statistical production meetings
Consistency tests on elementary data
Additional control in the framework of Financial Accounts
Totally automated Judgment needed
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 15
3.Statistics Audit Main purposes
(1)
Analyse and evaluate the different phases of statistical production procedures in place, from source data collection up to final statistics compilation and dissemination
Analyse the organizational and functioning aspects, in particular evaluate the burden and the share of functions and responsibilities in the statistical production process
Evaluate the efficiency of the procedures in place, identifying the good practices as well as improvement needs
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 16
3.Statistics Audit Main purposes
(2)
Contribute to enhance methods and applied
techniques and to improve the statistical
production quality by issuing, when it applies,
suggestions/ recommendations on current practices
or by suggesting new procedures and/or
organizational arrangements
Encourage and promote the share and
comparison of good practices and procedures
among the different areas of the Statistics
Department
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 17
3.Statistics Audit Methodological approach – AK
list (1)
A. Initial appointment with the responsible of the
unit in charge of the statistics under audit
B. Collect, organize and analyse the
documentation made available by the unit
C. Submission of the “Self Assessment
Questionnaire”
D. Fieldwork
E. Definition of a flow chart on the information
circuit
F. Preparation of the Check List of procedures
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 18
3.Statistics Audit
G. Joint analysis with the unit’s manager of the Self Assessment Questionnaire
H. Graphical assessment of the statistical production system (radar chart)
I. Report on the statistical audit process (1st draft) – submission to the unit under audit
J. Joint evaluation of the comments from the unit under audit
K. Final report on the statistical audit process – submission to the Head of the Statistics Department
Methodological approach – AK list (2)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 19
3.Statistics Audit
The self assessment questionnaire is the basis for
the presentation of the radar chart
Two sets of answers to the questionnaire – one, in
the beginning of the audit process, from the unit
under audit and another, at the end, from the
Statistics Audit Unit – are contrasted graphically in a
statistical production assessment diagram
Self Assessment Diagram (1)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 20
3.Statistics Audit
Self Assessment Diagram(Securities Issues Unit (SIET)
0
1
2
3
4
5Staff Functions Share
System Adequacy
User Satisfaction
Overall Relevance
Transparency Degree
Overall Coverage
Methodological Soundness Degree
Elementary Data QualityReporting Compliance
Intermediary and Final Data Validation
Dissemination Punctuality
Temporal Consistency
Consistency with other Statistics
Dissemination Clearness and Coverage
Metadata Accessibility
SIET Unit
Accessibility Relevance
Serviceability
Accuracy and Reliability
Integrity
Methodological Soundness
1 - Very low2 - Low3 - Medium4 - High5 - Very high
Self Assessment Diagram (2)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 21
3.Statistics Audit
Self Assessment Diagram(Securities Issues Unit (SIET) versus Statistics Audit Unit (NAE))
0
1
2
3
4
5Staff Functions Share
System Adequacy
User Satisfaction
Overall Relevance
Transparency Degree
Overall Coverage
Methodological Soundness Degree
Elementary Data QualityReporting Compliance
Intermediary and Final Data Validation
Dissemination Punctuality
Temporal Consistency
Consistency with other Statistics
Dissemination Clearness and Coverage
Metadata Accessibility
SIET Unit
NAE
Accessibility Relevance
Serviceability
Accuracy and Reliability
Integrity
Methodological Soundness
1 - Very low2 - Low3 - Medium4 - High5 - Very high
Self Assessment Diagram (3)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 22
3.Statistics Audit Overall assessment of the statistical audit
operations (1)
The overall assessment is quite significant:
1. The deep interest and good collaboration revealed by those intervening in the statistical production systems audited
2. These operations allow a detailed assessment on the current features of the statistical systems in place
3. The possibility of conveying upwards the desires and aspirations as well as the shortcomings faced by the responsibles of the units under auditing, which, otherwise, might encounter more constraints before being acknowledged
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 23
3.Statistics Audit Overall assessment of the statistical audit
operations (2)
4. The share in all Department of procedures and good practices in place
5. Most of the suggestions/ recommendations raised by the statistical auditors and accepted by the units under audit are already being implemented or are included in the unit’s planning for the years ahead
6. The relevance of the follow-up of the implementation of suggestions/ recommendations
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 24
4.Institutional Cooperation
More efficient statistical production processes:
Facilitates data sharing and good practices’
diffusion; Promotes the use of harmonized statistical
methodologies - thus improving statistical data comparability;
Helps reducing reporting burden and data redundancy; and,
Enhances the clarification of responsibilities and areas of competence of the parties involved.
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 25
4.Institutional Cooperation
Some highlights: Banco de Portugal became a national statistical
authority in the same level playing field as Statistics Portugal, without prejudice of their independence;
This new law emphasizes the importance of ensuring an efficient cooperation and the need for them to develop models of cooperation considered as adequate to fulfil their role in the NSS; and,
The principles of the European Statistics Code of Practice are also formally adopted in this new legal framework.
New Statistical National Law (May 2008)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 26
4.Institutional Cooperation
- In the framework of the High Council of Statistics (HCS) – the body on top of the Portuguese NSS;
- Other bilateral/multilateral initiatives:
1) 1998 - Protocol with Statistics Portugal (SP) and Ministry of Finance (MF) in the field of IMF Special Data Dissemination Standard (SDDS);
2) 1998 - Protocol on the compilation of Portuguese National Accounts - shared responsibility between SP (non financial accounts) and BdP (financial accounts) – updated in 2001 and 2007;
3) 1999 – Protocol with SP for the joint conduction from 2000 onwards, in partnership, of a Quarterly Survey on non Financial Corporations;
4) 2004 – Two protocols in the field of Balance of Payments, with SP and General Directorate of Tourism, focusing on a survey of cross-border movement of travellers and a survey on international tourist expenditure (the last was conducted in 2007);
In national terms (1)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 27
4.Institutional Cooperation
5) 2006 - Protocol with SP, (the first was signed in 2000) for the conduction of a Survey to the Households Indebtedness and Wealth;
6) 2006 – Protocol with SP and MF, on the compilation of General Government Statistics, Public Debt Statistics and analysis of Excessive Deficit Report;
7) 2006 – Joint work with SP for the development of a unique File of Statistical Units for the whole NSS;
8) 2006 – Protocol with SP, MF and Ministry of Justice to define a single electronic data submission with data from the financial statements of non financial corporations – IES (formally created by Decree-Law n. 8/2007 of 17th January).
In national terms (2)
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 28
4.Institutional Cooperation
- With ECB and the other EU national central banks in the framework of ESCB;
- With Eurostat and the EU national statistical offices under the umbrella of the Memorandum of Understanding (MoU);
- In the framework of CMFB (currently the Head of the BdP Statistics Department is also part of the CMFB Executive Body);
- With other international organizations namely IMF, BIS, World Bank and OECD;
- Technical assistance – several activities in particular, but not exclusively, with the national central banks ant the national statistical offices of the Portuguese-speaking Countries.
At international level
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 29
5.Conclusions and Prospects Ahead
Further improvements: Define a set of structured quality indicators, to be
compiled on a regular basis, on the quality of the statistical production;
Improve the already existing quality manuals, exploring further these documents both in a user’s oriented perspective and in a statistical quality control framing;
Enlarge the feedback information currently produced and returned to the reporting institutions;
Define mechanisms to have an adequate feedback from statistics’ users (on statistical data already currently available and on the users’ future needs).
The appraisal on the work developed so far is quite significant.
Statistics Quality Control: the experience of the Banco de Portugal
Q 2008 - European Conference on Quality, Rome, July 2008 30
Thank You
for your Attention