uk renal registry welsh data validation exercise … · 2015-10-27 · uk renal registry welsh data...

UK RENAL REGISTRY

WELSH DATA VALIDATION EXERCISE

PROJECT REPORT

Authors: D Ansell, Director Renal Registry

F Benoy-Deeney, Data Manager Renal Registry P Dawson, Data Manager Renal Registry H Doxford, General Manager Renal Registry E Will, Consultant Nephrologist

Date: August 2005 Version: 6c draft Contact information: Hilary Doxford

UK Renal Registry Southmead Hospital Southmead Road Bristol, BS10 5NB

UK Renal Registry Welsh Data Validation Project Report

WelshDataValidationProjectReportAugust2005 of 61

Table of Contents

1. EXECUTIVE SUMMARY ................................................................................... 4 INTRODUCTION .............................................................................................................. 4 PROJECT OBJECTIVES..................................................................................................... 4 SUMMARY...................................................................................................................... 4 RECOMMENDATIONS...................................................................................................... 6

Recommendations for supporting processes......................................................................... 6 Recommendations for data quality issues............................................................................. 7 Recommendations for the Renal Registry ............................................................................ 7

FURTHER DISCUSSION.................................................................................................... 8

2. BACKGROUND TO THE UK RENAL REGISTRY AND DATA COLLECTION 9

3. BACKGROUND TO HOSPITAL RENAL IT INFRASTRUCTURE............. 9

4. STRUCTURE OF THE DATA COLLECTION .............................................. 10

6. RESULTS............................................................................................................. 12 RENAL UNIT SITE VISITS ............................................................................................. 12 RESULTS FROM THE RENAL UNIT QUESTIONNAIRE PRO-FORMA.................................... 13

Organisation Structure ........................................................................................................ 13 Budgetary Control............................................................................................................... 13 Responsibilities and Training ............................................................................................. 13 Internal and External Communications .............................................................................. 14 Good Practice...................................................................................................................... 14 Renal Registry Relationship ............................................................................................... 14 IT Infrastructure and Systems............................................................................................. 15 Miscellaneous ..................................................................................................................... 15

RESULTS ON DATA COMPLETENESS AND ACCURACY .................................................... 15 Demographics ..................................................................................................................... 16 Biochemistry....................................................................................................................... 17 Blood Pressure .................................................................................................................... 17 EPO..................................................................................................................................... 18 Serology.............................................................................................................................. 18 Co-morbidity....................................................................................................................... 18

7. CONCLUSIONS.................................................................................................. 18

DATA QUALITY (DATA COMPLETENESS AND ACCURACY) ............................................ 18 Demographic data ............................................................................................................... 19 Biochemistry and haematology .......................................................................................... 19 Blood Pressure .................................................................................................................... 19 EPO..................................................................................................................................... 19 Serology.............................................................................................................................. 20 Co-morbidity....................................................................................................................... 20

SUPPORTING PROCESSES .............................................................................................. 20 Organisation Structure and Budget..................................................................................... 20 Responsibilities and Training ............................................................................................. 20 Communications ................................................................................................................. 21 Good Practice...................................................................................................................... 21 Renal Registry .................................................................................................................... 21 IT Infrastructure and Systems............................................................................................. 21



8. ACKNOWLEDGEMENTS ................................................................................ 23

APPENDIX 1 PROJECT TERMS OF REFERENCE........................................... 24 APPENDIX 2 RESULTS OF RENAL UNIT STRUCTURE QUESTIONNAIRE........................ 27

APPENDIX 3: RESULTS OF DATA COMPLETENESS AND ACCURACY...... 43 How to use these tables....................................................................................................... 44

APPENDIX 4: RENAL REGISTRY DATA VALIDATION PROCEDURES....... 55 INTRODUCTION ............................................................................................................ 55 COMMON DATA ERRORS............................................................................................... 55 THE RENAL REGISTRY MODALITY TIMELINE................................................................ 56 THE VALIDATION PROCESS........................................................................................... 56

Consistency checks at load ................................................................................................. 56 Consistency checks after loading........................................................................................ 58 Annual data checks ............................................................................................................. 58 SAS analysis ....................................................................................................................... 59

COMMENT.................................................................................................................... 61



1. Executive Summary Introduction This project was initiated by the Project Board leading the development of the National Service Framework in Wales and funded by the National Assembly for Wales, with the project terms of reference listed in Appendix 1. The Welsh Assembly Government has formed a project board to develop and provide policy guidance on management of renal disease in Wales, to improve the quality of renal services in Wales and develop a renal National Service Framework for Wales. The need to assess the information required and the quality of the information currently available is key o demonstrating the success of the National Service Framework and improvements made to the management of renal disease in Wales. This report addresses only the adult dialysis and transplant services, structures supporting the framework for management of chronic kidney and paediatric renal services were outside this project scope. The UK Renal Registry collects and analyses data on the frequency, medical management and effectiveness of treatment of patients on renal replacement therapy in the UK. It monitors the quality, as well as the quantity of activity, to support improvements in the quality of care for dialysis and transplant patients. The UK Renal Registry was commissioned to review and analyse data quality from the Welsh renal units and to provide recommendations on how data quality might be improved. This project is not an assessment of individual renal unit clinical performance or their performance on data quality or returns, but it is to identify both good practice and highlight commonality of problems. These findings will help to prepare recommendations for standardisation of procedures, leading to improvements in data quality and sharing of best practice between the Welsh renal units. Project Objectives The objectives of this project were:

1. To identify the prime causes of incomplete and inaccurate data entry relating to demographic and clinical variables, in order to explore feasible solutions.

2. To investigate all operational processes that may be contributing to the problems, e.g. system management, communication channels, training methods, user documentation, personnel, etc.

3. To provide recommendations on how to address the problem areas at source. 4. To develop and undertake preliminary testing of recommended solutions. 5. To provide support during implementation and provide an effective handover to ensure maintenance

of revised working practices. Summary An assumption has been made that the randomly selected patient records are a representative sample. This on-site data validation exercise has not been previously undertaken by the Renal Registry and we are unaware of a similar exercise having been undertaken by national renal registries in other countries. In the UK, the Myocardial Infarction Audit (MINAP) have published a review on a data quality exercise that they undertook. It is impossible to assign a direct cause / effect to all of the areas where issues have been raised. There are several factors that contribute to incomplete and/or invalid data. Even the data completeness and accuracy of results were assessed subjectively to some extent as it was not always possible to define the primary record (case note or IT system) against which to judge the data. The primary record for an individual data item may



also vary with modality and this fact has not been allowed for in this analysis. The conclusions that have been drawn are to a degree, therefore also subjective. Although staff at the Registry have extensive experience in validation of data sent to the Registry, there is no benchmark against which to assess renal unit performance in these areas and therefore it is difficult to confidently state root causes for all of the data problems. However, some results when reviewed against hospital administration and management procedures certainly indicate areas for further analysis and standardisation. All renal units are following elements of good practice. However where documents exist, they were often not to hand or were unfamiliar to their users. It is apparent that the sites are attempting to improve their data quality and are interested in incorporating validation routines to ensure improvement in data quality. Time and clinical pressures certainly contribute to these data problems. The dialysis service of the Bangor renal unit seemed to be the most structured with the most comprehensive documentation. Transplant patients have been excluded from this centres analysis so that it is not directly comparable with other centres. The separation of IT for the dialysis facility also resulted in Bangor requiring the highest number of additional stand-alone systems to cover other areas unsupported within the IT system (Baxter). Linking a renal unit with the HD providers IT system may reduce short term costs at the expense of reducing flexibility to change their HD provider at a latter date. The new IT system at Clwyd (Fresenius haemodialysis monitoring system), may create similar restrictions. All renal units have at least one stand-alone application holding renal data and one unit had 6 additional systems. This usually resulted in missing data in the renal system. Where data exists in more than one system, there is also the potential for additional errors. As highlighted earlier, the proliferation of stand-alone systems partly reflect inadequacies of the main renal IT system and point to a more integrated renal system being required. The IT support structure for the renal units do not allow for contingency planning for holidays, long term sickness or departure of a key member. This is mostly a cost issue and it is difficult to justify these costs on an individual renal unit basis. Several renal units within the UK have shared their renal IT system with other centres allowing for better contingency planning and improved day to day support. Structure of budgetary control varied for all the renal units and as a result of this, planning and management structures differed within each renal unit. For the renal units who were not holding their own IT budget, there was little dialogue with the IT budget holder to plan for future expenditure. Swansea relies on a charitable trust to address many of their renal IT funding requirements. All renal units had a named individual responsible for running and submitting Registry reports and also for correction of any data errors identified by the Registry. The number of systems containing renal data and the lack of system interfaces suggests that a review of system requirements and functionality be undertaken to determine the best solution for renal service provision. This would determine whether a new system, a single system for Wales, adding functionality to existing systems and/or building interfaces is the most clinically useful and/or the most cost effective answer. The problems caused by the Proton system’s lack of ‘user friendliness’ should not be underestimated. Whilst undertaking the data quality exercise, both renal unit and Renal Registry staff had difficulty locating data items in the Proton system. Until all renal systems have improved technical and user system documentation, data quality will continue to be compromised.



Changes to the local IT system may impact on the Renal Registry data extraction interface. There was poor understanding of what information required communicating to the Renal Registry, should the local renal system database be changed. Four of the five renal units felt that additional IT resources would improve data quality. The main areas of concern were:

• the lack of system interfaces (e.g. to patient administration systems) • the lack of system enhancement plans to enable capture of all requisite data • the existence of poor or non existent validation on data entry.

By job role, awareness of Renal Registry activities varied from 1% being aware of the Registry annual user group meeting to 79% awareness of the Registry annual report. The feedback obtained from all renal units during this project provide a good base upon which to clarify and confirm job roles and responsibilities in the Agenda for Change programme. Four renal units felt this exercise had altered their perception of the Renal Registry. There is scope for further analysis within each of the areas reviewed should more clarification and justification of cause and effect on data quality be required. For the short term, piloting some of the recommendations particularly those relating to the improvement of data validation, the definition of responsibilities and best practice should result in significant change for the better. Recommendations It is to be stressed that there is not yet identified within the renal IT world a ‘best practice’ model. It is important to identify what has worked at other renal units (including those outside Wales) and to use this as the basis for development. It is accepted that any one solution may not work for all renal units. but there is a core of generic process and procedure that all units would recognise as relevant. This is not a static area and few, if any, reporting centres have a comprehensive solution for data capture and quality assurance. A progressive quality improvement programme addressing data concerns at each site is an obvious way forward and such a programme would also support the career development for informatics staff. Active management of the IT activity by defined senior staff is necessary and should occur in collaboration with clinical staff. The most cost effective way to implement change would be to consider addressing relevant shared features centrally and to then add on the necessary local variations (eg. The login password for Swansea users could take them into their own screens)

Recommendations for supporting processes

• To instigate a culture change making the system and personnel integral to the running of the organisation and not peripheral.

• To document system navigation, screens and data entry processes and share this supporting process across Wales.

• To ensure the requirements of the national dataset are encompassed in any system changes. • To establish a training programme for new staff, provide refresher training on data entry and also

Renal Registry issues. • To review existing system security and audit trails to ensure ownership of data entry and

amendments can be attributed to an individual. • To define and implement processes to ensure data errors are corrected at source where possible. • To introduce regular meetings with the main IT department so that the renal units IT needs can be

met.



• To introduce Renal Registry returns as a standard agenda item to a monthly multidisciplinary department meeting each quarter. Items to discuss would include: timescales for quarterly returns, obtaining missing information and dealing with errors found by the Renal Registry.

• To introduce data quality as a standard agenda item at these monthly meeting. Items to discuss would include: data quality developments and ongoing data quality issues with the Renal Registry.. Renal system requirements should become a standard agenda item at the meeting to ensuring that IT requirements are planned and implemented in a timely fashion.

• To ensure that job descriptions and responsibilities are amended so that Renal Registry requirements are included in those posts in regular contact with the Registry .and are culturally considered as part of the multidisciplinary team

• To ensure that cover is available for supporting the renal IT system and also the Renal Registry when the regular contact is absent.

Recommendations for data quality issues • To ensure that relevant renal information held on stand alone systems are incorporated into the

main renal IT system for example by implementation of the Adequest to Proton interface • To change free text entry fields into coded fields where applicable (Timeline and EPO prescription

To establish clear, comprehensive and up to date documentation for staff on their renal systems, procedures and also Renal Registry requirements

• To agree and implement a set of standard range validation requirements to highlight incorrect or suspect data and to identify required data that is missing.

• To ensure where appropriate, their links with satellite units are functioning and that the Renal Registry are informed of any changes with the status of a satellite unit (e.g. a unit opening/closing).

• To inform the Renal Registry of software developments so that any relevant mapping issues can be addressed.

• To agree to provide the Renal Registry with the completed quarterly data file within two months of the agreed start date.

• To agree and share with the Renal Registry an escalation route for unresolved data issues.

Recommendations for the Renal Registry The Renal Registry should:

• Increase awareness and knowledge of Renal Registry purpose and activities. • Provide the dates when they are planning to start quarterly data collection to the sites. • Share future plans and timetables as early as possible to allow sites time to implement any

necessary changes. • Provide relevant feedback to sites to ensure that ongoing data issues are adequately addressed. • Help sites resolve data mapping issues and provide necessary software upgrades where appropriate. • Help sites share information to facilitate best practice and to ensure that data validation is standard.

throughout the sites. • Respond to questions from sites promptly (within three days). • Provide template information sheets for incorporation into induction documentation, job

descriptions, data entry procedures, etc. • Assist with the specification of site data validation rules. • Assist with the specification of routines to identify incomplete and inaccurate data. • Review its remit to see if there is scope to offer additional services.



Further Discussion The Registry have identified several key areas for further discussion, some are indirectly related to this project but were generated through discussions during the process of data collection. Whilst the recommendations for supporting processes and data quality issues are all relevant for further consideration, the Welsh Assembly Government should prioritise and a step change approach is required due to the potential amount of work involved and the level of change required. Should Renal units within Wales move to a centralised, standard renal system ? This would be expected to be the most cost-effective long term solution and would lower operational overheads. It would address many of the problems associated with data quality, staff training, best practice definition, systems support and ownership. While this would result in uniform data there are a number of problems including data confidentiality, cost, timescale and local clinical flexibility within the system. Released investment, particularly in systems analysis, business analysis, staff training and development should pay dividends. Centralising a system has the potential to result in a lowering of overall data quality while data were merged, staff trained but this was not the experience in Bristol or Leeds. Ownership of data would still be retained locally and data entry should be at the point of patient contact. The complexities of IS/IT are transferred to a central point where timely and effective support can be provided. This approach has worked well for several renal units in England and Northern Ireland also has recently taken this approach. If this is adopted an interim solution to the current problems is still required and would need additional investment in local services to achieve improvements sooner rather than later. System scope? The scope of this project was limited to the Registry data, however during conversations with unit staff it became apparent that there will be benefits in adding to system functionality. Areas identified were to hold clinic letters electronically, the production of standard templates, prompts for NSF requirements, the automatic provision of local and national audit information and system prescribing. For example, the Registry monitors haemoglobin standards and has identified inaccuracies in monitoring of EPO prescriptions. GP prescribing of EPO is ‘piecemeal’ and contributes to these data inaccuracies. Organisational culture? Although not specifically reviewed, it became apparent that previous experiences with IT systems are affecting the current culture and attitudes. Clear leadership, direction and support are now needed to overcome the problems. Without a culture of data sharing there is ad hoc development of standalone systems and practices. Data are seen purely as a measure of performance and not as an aide to patient care. Benefits from funding improvements All current IT implementations in the health service and many in the commercial sector have failed to show any significant ‘cost savings’. The UK renal units with well supported IT/IS systems with a high quality of data held within them, have provided their renal unit managers and also commissioners with more timely and accurate data for planning delivery and effective administration of these high cost services. Clinicians and members of the multi-disciplinary team working within these renal units notice the improved benefits for both staff and patients from more readily accessible clinical data. This is principally noticeable in out of hours communication when the consultant is not available or a patient is called in for transplantation. This review has improved the understanding of all the parties involved, in how each individual renal unit is structured and how they function on a day to day basis. It has also been able to highlight areas of good practice seen within each renal unit so that lessons can be shared by other renal units not just within Wales but also within the UK.



2. Background to the UK Renal Registry and data collection From its inception the Renal Registry acknowledged that the central component of setting up a database, it’s maintenance and statistical analysis, was the easier part of the exercise to administer. More difficult was how to monitor and manage data quality issues from the renal unit level, when their development in IT infrastructure, size and complexity was variable. Initially, this was addressed through a formal contract, to supply items according to the Renal Registry Dataset. It was anticipated that this would require regular contact with a specific, senior representative of each unit, who would take responsibility for Registry liaison. For a variety of reasons such contact has been patchy, despite the willingness of both the Registry and renal units to participate. The Dataset has been expanded and refined, which has required active development that has not always been smooth. Renal units have tended to concentrate on the collection of specific subsets of data, sometimes those that carry the most interest for their staff or are easier to acquire and maintain. Part 1 of the Renal National Service framework in England has now formalised the role of the Registry in monitoring the performance of the renal units. The Registry dataset is in the process of being formally approved by the NHS Information Standards Board as part of a ‘National Renal Dataset’.

3. Background to hospital renal IT infrastructure Supporting clinical computing at the hospital speciality level (e.g. nephrology) has never been a mainstream activity for hospital budget holders and IT departments. Resources have tended to be focused on generic trust-wide, administrative and financial solutions rather than speciality-specific support. This has resulted in renal unit computing often having little access to resources, which then limited development and left the workforce vulnerable. Both the national intensive care audit (ICNARC) and the myocardial infarction audit (MINAP) are now utilising specialty level support from administrative staff. Renal unit computing throughout the 1980s and 90s was based on the widespread use of one particular system (Proton) and the Informatics infrastructure was developed more by intuition and experience rather than based on a formal model It is apparent that in renal units nationally there are rather unstructured arrangements for the support of clinical computing and also of feedback to the Renal Registry. Although a ‘best practice’ model has not been piloted, experience suggests that the most complete and accurate records will be achieved by data entry at points of clinical activity, such as the outpatient department or dialysis unit, supported by regular informed and multidisciplinary review of the end data record for missing entries and inaccuracies. The data acquisition task for renal units is complex as patients are treated by a variety of modalities, on a variety of sites (some non-hospital based), by a range of personnel. These circumstances are similar to those of the national programme for IT (‘Connecting for Health’) and have resulted in the creation of an undeclared renal data spine and associated clinical material. Some of these data are numerical and an automated laboratory linkage is seen as an essential, integral part of renal systems. These data vary qualitatively over a wide range from; the patient’s demographic details (typically not linked to the hospital Patient Administration System and so requiring duplicate entry), clinical data relating to the different modes of renal replacement therapy and workup for transplantation, etc. Some of these data are permanent features of the patient (e.g. ethnicity) and other items vary day by day (blood pressure prior to haemodialysis). Clinical records are complex in this environment, where individual treatment-related data are often not registered in the formal hospital case note folder. Depending on the information, the primary record for a given activity is not necessarily the patient case notes, but may be the nursing records,



haemodialysis folder or even the local renal computer system itself. This diversity in the ‘primary record’ source makes any analysis of data quality more complex. Given the rapid rate of expansion of renal replacement therapy over the past two decades, it is unsurprising to find that renal units have not had the time to specifically focus on the area of Informatics, in which they may have limited skills, compounding problems of informatics staff retention and lack of training. The Renal Registry has only an indirect influence on the maintenance and development of renal computing at local sites. The funds received by the Registry through the Registry annual capitation fee have not been directed to the renal unit component of the information network. All these factors lie behind the enquiries that were made on the structure and processes of the Welsh renal units which also relates to the outcome of data completeness and accuracy.

4. Structure of the Data Collection This review was structured round two separate visits to each renal unit. The first visit would enable the Renal Registry to review the operational, administrative and management procedures in the renal units and the second visit would look at data quality. The first site meeting was scheduled to be with both the informatics and clinical staff. A questionnaire pro-forma was sent to each site in advance and the meeting based on a structured interview encompassing:

1. organisation structure 2. IT budgetary control 3. responsibilities of informatics staff 4. IT training 5. communications within the renal unit 6. best practice 7. Renal Registry liaison 8. IT infrastructure and systems.

The second visit was to validate the data held by the Renal Registry against that in the patient case notes and the electronic record from the renal IT system. Renal patients typically have a large set of case notes, which makes data validation a lengthy process. Within the available budget, a time frame of one day was available which allowed for validation of data on 20 patients. The data validation visit was undertaken by a qualified renal nurse in conjunction with a request that local informatics staff provide time to support the cross-checking of Registry data with the local IT system.

The Registry randomly selected 20 patients, to cover the different renal replacement therapy (RRT) modalities and the renal unit was provided with at least 7 days notice to locate the requested patient case notes and supporting documentation. A list of ‘reserve’ patients was also provided so that alternatives were available should an individual case notes be unavailable on the day of the visit. The four patient groups were;

Patient group Numbers New haemodialysis patients in 2003, including 2 from a satellite unit and 2 diabetic

5

Peritoneal dialysis patients, including 2 diabetic 5 Transplant patients 5 Deceased patients in 2003 5



Personnel at first visit

Site Visit 1 date

RR Attendees Unit Attendees

Bangor 15/10/04 P Dawson H Doxford

M Wild, Renal unit manager P Stollery, renal IS systems manager J Jenkins, I Kilburn

Cardiff 28/09/04 H Corderoy F Benoy-Deeney

Dr K Donovan, Consultant nephrologist K Hughes, IS systems manager

Clwyd 13/10/04 P Dawson H Corderoy

Dr A Douglas, Consultant nephrologist B Ledwith, Temporary Data Clerk

Swansea 09/07/04 Dr D Ansell, H Doxford, P Dawson, F Benoy-Deeney M Brealey

Dr A Williams, Consultant nephrologist M Davies, IS systems manager

Wrexham 14/10/04 P Dawson H Doxford

Dr W Ahmed, Consultant nephrologist

Personnel at second visit

Site Visit 2 date

RR Attendees Unit Attendees

Bangor 24/11/04 C Mansell F Benoy-Deeney

M Wild J Jenkins

Cardiff 15/10/04 H Corderoy D Stewart Clwyd 23/11/04 C Mansell

F Benoy-Deeney B Ledwith

Swansea 19/10/04 H Corderoy, M Davies Wrexham 29/11/04 C Mansell

P Dawson J Cripps, Senior Nurse / Practice development W Jones, Data clerk PD S Pierce, Data clerk HD C Clarke, Data clerk HD

The report was reviewed within the structure of the All Wales renal information group. The meeting provided an opportunity for all the renal units to review these results and provide feedback for this report.. Two of the renal units were not represented at this meeting and provided feedback of their comments electronically.



6. Results Renal Unit Site Visits Swansea (Morriston Hospital) At the first visit, all requested information and supporting documentation was provided. At the data validation visit, the wrong patient case notes had been provided due to a misunderstanding between two patient lists. Some patients were therefore only validated against their haemodialysis folders, preventing a full check on the relevant data. Patient case notes were not checked for co-morbidity data as this was not being returned to the Registry.. Cardiff (University Hospital of Wales) At the first visit all requested information and supporting documentation was provided. At the data validation visit, insufficient time was provided by Cardiff for the Registry staff to access the renal unit IT system (Proton). Additionally the local person providing support did not have sufficient knowledge of the system to locate all the necessary data screens required for the validation exercise, resulting in incomplete data evaluation. Patient case notes were not checked to verify co-morbidity data. Wrexham (Maelor Hospital) At the first visit only the consultant was present. The data administrator was unable to attend which resulted in some questions being inadequately covered. Most of the missing information was obtained subsequently. At the data validation visit, all the requested patient case notes were unavailable. While efforts were made to obtain as many patient case notes as possible, it was not possible to validate the complete selected sample. Clwyd (Ysbyty Glan Clwyd) The renal unit was in the process of undergoing a change in the IT infrastructure. Staff were unclear how the new system would integrate and who would be entering and viewing data. This meant that whilst answers could be provided on the basis of the current IT system these responses may not be applicable following implementation of the new system. At the data validation visit, the data entry clerk present was on a temporary contract, finishing that week and did not have full knowledge of the IT system. Similarly to Cardiff, there was insufficient knowledge of the system to locate all the necessary data screens required for the validation exercise, resulting in incomplete data evaluation. Bangor (Ysbyty Gwynedd) This haemodialysis facility is managed by a commercial company (Baxter). Within the UK there are various models for commercial provision of dialysis services. In this renal unit, both the HD machines and the haemodialysis IT system are provided by Baxter. The nursing and support staff are also contracted by Baxter whilst medical staff and the responsibility for patient management are retained within the NHS. The Baxter personnel were primarily involved in liaison with the Renal Registry at both visits. The comments in this report are confined to the areas covered by the scope of the renal service contract between Bangor and Baxter.



The renal unit was well prepared for both visits, with all requested information and supporting documentation being provided. The patient case notes were the most highly organised. Results from the renal unit questionnaire Pro-forma The answers from the structured interviews are tabled in appendix 2.

Organisation Structure Four of the five renal units had an organisation chart, although the Bangor renal unit were only able to provide this for the commercial HD facility and not for NHS employed staff. Informatics staff were included in two of the charts.

Budgetary Control Structure of budgetary control varied for all the renal units and as a result of this, planning and management structures differed within each renal unit. For the renal units who were not holding their own IT budget, there was little dialogue with the IT budget holder to plan for future expenditure. Swansea relies on a charitable trust to address many of their renal IT funding requirements.

Responsibilities and Training All renal units had a named individual responsible for running and submitting Registry reports, loading Renal Registry numbers (the unique patient ID supplied by the Registry) into the local system and also for correction of any data errors identified by the Registry. The role of the Renal Registry contact was perceived by the renal units to be one covering: problem resolution, a point of contact, training, culture change, data quality, data accuracy, data delivery, to look for improvements, liaison, taking the lead. Only one renal unit included the responsibility for the delivery of Renal Registry data in the job description of the designated responsible individual. There was a large variation between renal units over which groups of staff had read-only access to the local renal system (were unable to change any data) and those who were authorised to maintain the data. The number of job roles ‘viewing’ data in the renal IT system varied from 4 to 16 and those roles updating the system ranged from 4 to 11. All the renal units did have at least one job description referencing the Renal Registry although only one of these job descriptions was made available to this review. At none of the renal units was there provision of formal structured staff training for either the Renal Registry requirements or for the local renal IT system itself. The training that was provided, was work based focusing on the operational aspects of using the renal IT system. Training did not include the rationale of the Renal Registry and associated tasks. None of the renal units had an allocated IT training budget, with training only undertaken if money became available. Bangor was the only renal unit specifically allocating 3.5 hours of training time per employee per month (within the HD commercial contract) and the training was funded from a general budget that provided for all company staff training needs. The NHS also provides a notional nursing staff training budget but the extent of this provision was not surveyed at the other renal units. It cannot be inferred that the other units have no training budget. Information systems issues may be included in other parts of in-service training.



Four renal units had some documented user guidance of their renal system. The comprehensiveness of this documentation varied between renal units. Bangor with its more limited and restricted haemodialysis/PD IT system, had the most comprehensively documented processes.

Internal and External Communications No renal unit held a meeting specifically to discuss Renal Registry matters. However, three renal units do have the Renal Registry as an agenda item at an internal meeting. Clwyd are the only renal unit who do not specifically discuss Renal Registry data. The Renal Registry receives the patient data from Clwyd via the Proton system at the Royal Liverpool University Hospital and there was limited communication from Clywd with this site. Wrexham (only recently) and Bangor were actively trying to manage Renal Registry issues actively. Of the two renal units with satellites (Swansea & Cardiff), only Swansea held regular main/satellite unit meetings providing an opportunity to discuss Renal Registry data issues and problems. Swansea and Bangor regularly included informatics staff in their clinical review meetings, in contrast with Wrexham who infrequently included their data administrator. Cardiff and Clwyd had no informatics staff present at these meetings.

Good Practice It is important to restrict the creation of new patient records on an IT system otherwise patient records become duplicated. Within normal working hours the creation of a new patient record may be restricted to specific administration staff, although for a clinical system to remain useful a process is required to manage this during weekends and nights. Only Swansea had trained their nurses on how to enter new patient details into the system out of hours, whilst the other 3 units had to wait. The HD unit at Bangor was not contracted to provide an out of hours service. Three of the renal units have documented the procedures required for data entry. The average time between a consultation and the data entry varied from immediate in Swansea to in excess of a week at Clwyd. Although Bangor also had immediate entry of data, this was limited to the haemodialysis facility and so not comparable with delays of clinic data entry (e.g. change in prescription of EPO dose by consultant). A comparable analysis for Bangor would be judging the time taken for a clinic letter to be received in the HD unit and the changes then made on the IT system. For continued smooth running of clinical IT systems it is important that contingency plans are in place for absence. Only Bangor and Cardiff had such a plan for absence of their systems manager while Wrexham had a contingency plan that covered other IS staff.

Renal Registry Relationship By job role, awareness of Renal Registry activities varied from 1% being aware of the Registry annual user group meeting to 79% awareness of the Registry annual report. There was low awareness of the Renal Registry annual report in Cardiff (33%) and Clwyd (42%); with even less knowledge of other Renal Registry activities. All renal units knew their point of contact in the Renal Registry, with communications for Clwyd provided through Liverpool. Responses by the Renal Registry to renal unit requests were stated to be generally good, although some chasing or follow up was occasionally required. All renal units felt future visits by Renal Registry staff would be of benefit.



IT Infrastructure and Systems All renal units except Clwyd felt that additional IT resources would improve data quality. The main areas of concern were:

• the lack of system interfaces (e.g. to patient administration systems) • the lack of system enhancement plans to enable capture of all requisite data • the existence of poor or non existent validation on data entry.

For sharing of clinical data within a multi-disciplinary team, it is important that the IT system is networked and the data held in a single point of source so that staff do not have to access multiple databases for a data item. All the renal units were holding additional data in at least one stand-alone IT system. All the renal units were using the stand-alone Baxter Adequest system for calculating PD adequacy and this data would therefore not be available for clinical review by others apart from the PD nursing staff. Bangor used the highest number of additional systems (6) which may reflect the limitations of the Baxter IT system. Three renal units had renal systems documentation. Only Cardiff expressed concern about the licence limit on concurrent users of their renal IT system. Changes to the local IT system may affect the Renal Registry data extraction interface. There was poor understanding of what information required communicating to the Renal Registry should the local renal system database be changed. However three renal units were proactive in checking with the Renal Registry should a change to their database be required. There was the facility to manually enter laboratory data in all the five renal units. Cardiff had added validation routines to this data entry process. To maximise the benefit from an IT system it has been recognised that the access to the patient data should be available from the many different places where there may be patient contact. These would include wards, outpatient clinic areas, consultant offices, onsite HD facility, satellite HD facility and peripheral clinics. Only Cardiff provided access at all these patient contact points, while Wrexham & Bangor had the most restricted access.

Miscellaneous All renal units except Clwyd expressed an interest in receiving additional validation routines from the Renal Registry. Clwyd submit their data via Liverpool who were not consulted within the scope of this project. Four renal units felt this exercise had altered their perception of the Renal Registry. Swansea already had a good understanding as a result of the close Registry interaction with one of their consultants who is a member of the Renal Registry committee. Results on data completeness and accuracy The complete list of results are tabled in appendix 3. For the purposes of this analysis, data accuracy needed to be compared against a defined ‘primary record’. The patient case note is not necessarily the primary record as some data may be found in the renal IT system which does not enter the patient case note. The primary record is not necessarily consistent between sites, for example at one site the IT system may be the primary record for erythropoietin (EPO) prescription (updated by the EPO nurse), while others may use the patient case note.



Demographics The surname and first name(s) of patients were usually correctly recorded in the local renal system and Renal Registry database. Details of patients with more than one first name were not always present on the local renal system. Some patients had additional indicators within the name fields, to help with patient identification at the site. Although high accuracy rates would normally be expected for these data items, the Registry data mirrors the accuracy of the local IT system. The impact for the Registry is on the ability to use the NHS tracing service to identify NHS numbers and on links with UK Transplant for joint analyses. Dates of birth were usually correctly recorded on the local renal system and Renal Registry database, although there were some inaccuracies with the day, month or year being incorrect by one digit. This may be due to data input error or due to the necessity to align the date of birth with that held in the laboratory system, to allow automated uploads from the laboratory interfaces. The effect for the Registry is on the ability to use the NHS tracing service to identify NHS numbers and on links with UK Transplant for joint analyses. In the latter case it is possible to accept small errors in dates of birth. Postcodes showed few discrepancies. The Renal Registry validates the address fields received using a commercial postcoding package (QAS systems) that is updated on a monthly basis. Only Cardiff received postcode data via an electronic PAS link that have postcode validation at the source. The local renal IT systems are not able to support validation on this field and even if they were, the financial cost would be prohibitive. Some of the local postcode errors may also be due to the recent re-coding of Welsh postcodes by the Royal Mail, which would not have been updated in the local renal system. The NHS number was often recorded on the patient case notes but had not been entered on to the local system and therefore was not sent to the Renal Registry. The Renal Registry does submit files to the NHS tracing service, which relies on finding an exact match on surname, forename, date of birth and postcode. Wrexham held no NHS numbers on their local renal IT system. Ethnicity is very important in assessing the adequate provision of renal services as the demand is more than four times greater within ethnic minorities. Ethnicity is part of the mandatory PAS dataset, although it was rarely recorded in the patient case notes. Swansea was the only site that comprehensively recorded ethnicity on their renal system. The NHS recommends that ethnicity is determined through asking the patient directly. For those patients allocated an ethnicity it was not possible to determine whether this was through asking the patient or decided on by NHS staff.. The primary diagnosis causing renal failure, had at some sites initially been recorded as unknown for some patients, even though other information was available in the patient case notes. At Wrexham, the primary diagnosis code had historically been poorly completed. This issue was now being addressed by the renal unit. The date of death held by the Registry often showed a small discrepancy of 1 – 2 days from the case notes and this accounts for the low accuracy rates in the Tables. This minor discrepancy would be clinically un-important to analyses by the Registry. In only 2 cases was the discrepancy such as to have a minor effect on calculated survival rates. Cause of death (using the European Renal Association codes) was poorly recorded in the patient case notes. Swansea was the only unit to record this information on the local IT system and hence send this to the Renal Registry. The cause of death and date of death percentage returns for Bangor were low, because when patients at the renal unit ‘stop treatment without recovery of renal function’ (stop haemodialysis) no further details are entered by the Baxter HD staff onto the local renal system. While the start of renal replacement therapy date was often recorded in the patient case note, it was not always recorded on the local renal system within the specific field allocated by the Renal Registry. Many



sites use the date of first treatment modality in the renal replacement therapy timeline to record this information. Validation has been against the timeline data item. There was often a discrepancy of up to 10 days in the dates recorded for the start renal replacement therapy date in the timeline against the case notes and occasionally much greater than 10 days. It is unknown which of these sources is the most valid, although the earliest might be assumed so . The first seen date by nephrologist was not being recorded on the local IT system at some sites. For other sites this was not being received at the Renal Registry again suggesting a data mapping issue. The low accuracy rates were due to minor discrepancies of a few days in the dates recorded, which would not be clinically significant. Height is often not recorded in the patient case notes but was available on the local IT system. This information was not always received by the Renal Registry. Occasionally, there was a slight discrepancy in the figures given. The timeline (RRT modality history) usually included all the information required by the Renal Registry. Where patients were seen by more than one hospital (transplanted patients) there were occasionally slight discrepancies in date of transplant supplied by the different renal units. The last RRT modality was usually accurate and discrepancies were likely to be due to a recent change in modality. At Wrexham, there were examples where the type of transplant differed (cadaveric / live related) as data was also received directly from their transplant centre at Liverpool.

Biochemistry The biochemistry readings for most patients at the hospitals were complete and accurate when compared to the primary record in the local renal IT system, although there were exceptions. The parathyroid hormone (iPTH) measurement gave cause for concern. There are 2 different laboratory units of measurement, which vary by a factor of 10. Combining this item from 2 different laboratory data sources (satellite / main hospital) into a single field in a IT system without adjusting the units, is a source of clinical error. One site did highlight this as a source for concern and the issue needs addressing. HbA1c (only measured for diabetic patients) was only available from one patient throughout the five hospitals. This does not imply that HbA1c was not being monitored in patients, as it may have been measured in the diabetic clinic and not repeated when the patient was seen at a renal clinic. The urea reduction ratio (URR), measured only in HD patients as an indicator of dialysis adequacy, was present for most patients at Bangor and Cardiff. It was identifiable for several patients at Clwyd and not identifiable within the IT systems at Wrexham and Swansea. The Registry data extraction software calculates URR values from the biochemistry data available and the Registry 2003 Report shows that it was able to do this for 70% of the patients on HD at Wrexham. Subsequent discussions with the site indicate that this field is present within the IT system, although was not identified at the time of the site visit. In the compilation of the 2003 Renal Registry annual report, a local IT software error was identified in the storage of the urea reduction ratio at Swansea and this error has since been corrected.

Blood Pressure Systolic and diastolic blood pressure were complete and accurate for dialysis patients at Bangor, although no data were recorded for transplant patients. At the other renal units, blood pressure data were not being recorded on the local renal IT systems. While post haemodialysis systolic and diastolic blood pressure were nearly complete at Bangor, information was either not present or present for only a few patients at the other sites. Where this



information was present in the local IT system, it was not always being picked up by the Renal Registry suggesting a data mapping issue with Proton sites.

EPO Erythropoietin prescription was the data showing the most variation in completeness and accuracy across the sites and there are many different factors governing this.

1. In HD patients monitoring may be via the HD nurses who correct the anaemia in these patients. Recording may vary at satellite units.

2. Some sites employ an EPO nurse whose salary may be funded from the pharmaceutical company. These nurses may keep EPO prescription updated in the company supplied stand alone database, rather than the main renal IT system.

3. Part of the EPO budget at several sites resides with the GPs. Data on prescription of EPO may therefore be absent from the main renal IT system. Monitoring of haemoglobin achievement at renal unit level is difficult as GPs may also refuse to prescribe the EPO dose recommended by the consultant nephrologist.

4. Within an individual renal unit, monitoring procedures for EPO may be by GPs, nephrologists or EPO nurse. depending whether a patient is on haemodialysis, peritoneal dialysis or transplanted.

5. Several sites use a free text (un-coded) field for storage of EPO data within the renal IT system. Registry data extraction routines are prone to error on interpretation of free text fields.

Serology The number of patients where it was possible to determine the hepatitis B and CMV status was low at all sites, particularly for CMV which is clinically only required for patients on the transplant waiting list.

Co-morbidity Co-morbidity prior to the start of renal replacement therapy was only recorded regularly at Bangor and Swansea. Several patients had information recorded at Cardiff but comorbidity was not recorded at the other two sites.

7. Conclusions Data Quality (data completeness and accuracy) Standard IT system design and best practice (technical and procedural) would overcome many issues, with the following helping to address many of the problems relating to incomplete and inaccurate data:

• reviewing which fields should be mandatory • reporting on data out of range • reporting (and/or confirm entry warning) on ‘suspect’ data within range • reporting on missing data • active management of data issues.

The above would help to ensure that as many patients as possible can be uniquely identified and their data correctly recorded on the local system of a centre. They would also help in reducing duplicate recording at, or between sites and enable multi-site links to be introduced.



Demographic data The problems with demographic data suggest that the following could deliver improvements:

• staff training for both new starters and refresher courses for all staff responsible for data entry • a review of documentation for data entry, including procedures to capture treatment modality

changes, patients transferring in and out, etc. • procedures for chasing missing data, including contact with other hospitals and/or GPs • procedures and assignment of responsibility to ensure data is entered in a timely fashion, particularly

relating to weekend and evening registrations • procedures to ensure that patient notes and the local IT system are updated immediately and

concurrently • clarification of when (if ever) interpretation is allowed, (e.g. the inference of a patients ethnic origin

from their name).

Biochemistry and haematology Biochemistry and haematology data were generally complete.

• Non-linkage of laboratory systems at satellite dialysis facilities has lead to reduction in completeness of data. This is currently being addressed through the central Wales laboratory server.

• Missing data items by the Registry, although the data was present at local sites has been addressed by the Registry returning a data completeness file to the sites.

• The measurement of parathyroid hormone (iPTH) is a cause for concern as 2 different laboratory units of measurement (varying by a factor of 10) are in use. Combining this item from 2 different laboratory data sources (satellite / main hospital) into a single field in a IT system without adjusting the units, is a source of clinical error.

Blood Pressure The low number of blood pressure readings suggests this data may be held in paper based systems. Monitoring of blood pressure targets is not possible through paper-based systems and sites need to look at improving capture of the data electronically. The renal unit at Bangor achieve good data capture in their dialysis patients although not in transplant patients. Other dialysis facilities could benefit from examining the structural processes in place within the Bangor dialysis facility. Post haemodialysis systolic and diastolic blood pressure, where recorded, does not appear to be picked up by the Renal Registry from Proton sites suggesting a data mapping issue between the Proton system and the Renal Registry. This problem has now been addressed by the Registry.

EPO EPO is a high cost drug with an estimated £4-6 million being spent throughout Wales. Monitoring of best practice is important to avoid postcode prescribing

• Change from a free text (un-coded) EPO field for storage of EPO data within the renal IT system is essential. Registry data extraction routines are prone to error on interpretation of free text fields as is the local renal unit monitoring of this data. The Registry has an EPO/ general drug screen available for Proton sites and the costs of this upgrade are borne by the Registry.

• Centralisation of the EPO budget within the renal unit may increase overspend within the renal unit if not fully funded, but will remove the risk of GPs holding money for EPO within their budget but refusing to spend it.

• The Welsh renal units are not directly managing their budgets and this provides no incentive to make savings in one area that may then be spent on developing or improving services in other areas. In 2003 the South West Region in England negotiated a discounted contract for EPO with the large savings made being re-invested within the renal services. This would imply that EPO could be contracted for Health Commission Wales in a more cost efficient way, although under the current



Welsh renal budget management system, any costs savings would be clawed back into the general hospital budget, removing all the incentives to undertake this process.

• Provision of EPO prescription within the renal IT system and removing stand-alone data systems is good practice. Holding clinical data in systems which cannot be accessed by clinicians and other nurses increases the potential for clinical error.

Serology Serology has not previously been analysed by the Renal Registry for inclusion in its annual report. Future analysis by the Registry is planned for this area and it is important that sites begin to send this information.

Co-morbidity Collection of co-morbidity data is essential to adjust for differences in survival between centres, identify reasons for differences in take on rates and those patients that require greater service provision. Only Swansea and Bangor are collecting co-morbidity on patients starting RRT. Collection of these data requires the support of consultant nephrologists as these are clinical data items. In addition to motivation, there needs to be a process in place to identify missing data items to consultants early say on a monthly basis so that data can be entered or corrected. Supporting Processes

Organisation Structure and Budget The level of analysis in this project, did not allow informed comment on the organisation structure or the adequacy of the informatics budget. However budget constraints appear to contribute significantly to data quality problems in renal units. Informatics funding would appear inadequate, particularly at Swansea which should not have to rely on charitable funds to support core informatics work within the NHS. The isolation of IT in the dialysis facility also resulted in Bangor requiring the highest number of additional stand-alone systems to cover other areas not supported within the Baxter IT system. Proliferation of stand-alone systems partly reflect inadequacies of the main renal IT system and point to a more integrated renal system being required at Bangor.

Responsibilities and Training With the exception of the Bangor dialysis facility, a lack of stated ownership, responsibilities and tasks appear to contribute to reduced data quality. Managerial responsibilities are known at renal unit level,even though they are rarely documented. The feedback obtained from all renal units during this project provide a good base upon which to clarify and confirm job roles and responsibilities in the Agenda for Change programme. Once implemented, if follow up were maintained through incorporation into the job appraisal process, clearer definition of roles would hopefully improve current achievements and stimulate continual involvement in quality improvement. Training is essential, both when starting a new role and throughout its tenure. The staff consulted generally had a good understanding of how to perform their responsibilities. However four of the five renal units are at risk should a team member leave due to inadequate succession planning, limited training plans and poor training materials for new staff. IT training should become a fundamental process which is integrated into overall staff training programme Clarification of roles and defined staff training would be expected to deliver improved data quality.



Communications Renal units that included the Renal Registry data on a meeting agenda and those meetings that also included informatics staff, (Bangor and Swansea) overall delivered better data quality. Although an interface to the PAS system is recognised as expensive, this may help with NHS numbers, dates of birth, postcodes and ethnicity. Wider use of the NHS number will reduce duplication of patients at the Registry. On-line links from a satellite unit to the IT system in the parent renal unit is essential for monitoring clinical care. Automated laboratory links are also essential and is currently being addressed .as part of the developing All Wales Strategy for IT. A review of communications with satellite units, GPs and other external parties should identify opportunities to improve data quality.

Good Practice Three of the renal units are exposed to potential problems due to the lack of contingency planning for the absence of IT staff. Consideration could be given to the sharing of skilled resources between renal units or within the hospital itself to avoid such a crisis. Various documents exist to facilitate good practice. From observation, it is believed that 80% of processes would be applicable to all renal units. A review and amalgamation of the existing renal unit documents could form the basis of defining data quality best practice. A general document could be adapted for each renal unit’s variables. Ad hoc events e.g. the opening or closure of a satellite unit must also be included. It is recommended that best practice be defined covering the ‘who, what, where, when and how’ questions. A base best practice should be established for Welsh renal units with adaptation to meet local needs.

Renal Registry The Renal Registry found this project extremely informative, enabling the identification of areas where further investigation should improve both internal Renal Registry procedures and data quality, namely;

• improving Renal Registry awareness within renal units, • establishing and maintaining effective communication channels, • assisting in the production of renal unit documentation with respect to data quality, • sharing of data validation rules, • assisting in the definition of renal unit roles and responsibilities • identifying data items that were available in the local system though not being sent to the Registry

Addressing the above should improve data quality by establishing better information sharing and improved working relationships.

IT Infrastructure and Systems Where renal units do not hold an IT budget, problems appear to be exacerbated by poor relations with the main hospital IT/IS department. This is an area that should be reviewed as IT systems are key to effective data quality and critical in the provision of supporting administrative and clinical procedures. Problems are frequently passed back and forth between hospital IT, renal units, hardware and software suppliers. Lack of problem ownership and accountability contributes to frustrations and the inability to resolve issues.



All renal units have at least one stand-alone application holding renal data and one unit had 6 additional systems. This usually resulted in missing data in the renal system. Where data exists in more than one system, there is also the potential for additional errors. An interface between Adequest and Proton is commercially available and the enhancement of the renal systems to hold the additional data would improve the clinical usefulness of the system as well as reducing quality problems. As discussed above in ‘communications’ access to systems from peripheral clinics and satellite dialysis units is important as are interfaces to laboratory systems from satellite units. The number of systems containing renal data and the lack of system interfaces suggests that a review of system requirements and functionality be undertaken to determine the best solution for renal service provision. This would determine whether a new system, a single system for Wales, more effective leadership through shared vision adding functionality to existing systems and/or building interfaces is the most clinically useful and/or the most cost effective answer. The problems caused by the Proton system’s lack of ‘user friendliness’ should not be underestimated. Whilst undertaking the data quality exercise, both renal unit and Renal Registry staff had difficulty locating data items in the Proton system. Improved layout of the Proton menu structures provided to clinical users is recommended. In terms of ‘user friendliness’, a Windows based system is an advantage for new staff due to the more widespread knowledge of these applications which reduces training. This would though require a large investment in new software. Until all renal systems have improved technical and user system documentation, data quality will continue to be compromised.



8. Acknowledgements The Renal Registry staff wish to thank the following for their assistance with this project. The staff at Bangor, Cardiff, Clwyd, Swansea and Wrexham for their assistance in providing the required data, their responses, time and preparation for the renal unit visits. H Corderoy for assistance and advice with the data validation work. C Mansell for assistance and advice during visits to Bangor, Clwyd and Wrexham and input to the project report. Dr. E Will for advice, suggestions in preparing the initial proposal and input to the project report.



Appendix 1 Project terms of reference UK RENAL REGISTRY WELSH DATA VALIDATION EXERCISE Introduction The Welsh Assembly have indicated they have a sum of £10,000 to allocate to a project reviewing data quality in Welsh renal units. The UK Renal Registry have been asked to put forward a proposal as to the productive use of these funds. This document presents that proposal. Background The existing data validation carried out by the Renal Registry suggests both generic and site specific problems – Chapter3 pp13-19, UK Renal Registry Report 2002 (Appendix 3). Common errors fall into two categories, administration errors and errors relating to the clinical data. Typical examples:

Administration; • incorrect Renal Registry numbers assigned • possible duplicate patients, transfers between renal units • incorrect transcription of written notes • demographic related queries • incorrect post codes.

Clinical data errors or queries

• missing disease codes • incorrect ‘timeline’ (clinical event) entries • entries out of range. • incomplete co-morbidity data • errors in data qualifiers

There are many less common errors. The current Registry validation routines run over 150 checks for consistency. An additional factor for consideration is the system being used. Four sites use the Proton system and the fifth (Bangor) uses a Baxter clinical database. One of the sites is in the process of changing to a Fresenius (haemodialysis only) IT system. Proposal Desired Outcome

1. To validate the patient data held by the Renal Registry



2. To improve the quality of data entered into the IT systems in the Welsh Renal units, moving towards complete, valid and accurate data entries and to therefore improve patient care

3. By this means reduce the time required to address errors and queries . Objectives

1. To identify the prime causes of incomplete and inaccurate data entry relating to demographic and clinical variables, to allow consideration of workable solutions.

2. To investigate all operational processes that may be contributing to the problems, e.g.

system management, communication channels, training methods, user documentation, personnel, etc.

3. To provide recommendations on how to address the problem areas at source.

4. To develop and undertake preliminary testing of recommended solutions.

5. To provide support during implementation and provide an effective handover to ensure

maintenance of revised working practices. Scope Description Within scope Excluded from scope Locations covered • Welsh Renal units • Other UK units

• Non renal units System administration

• Electronic • Paper/manual

Application name • Proton, Baxter • Others High level processes • Transcription of existing

patient paper records to electronic format

• Creation and maintenance of electronic new patient records

• None identified

Sub level processes • All processes supporting the high level

• None identified

Areas for review • Data entry • Supporting operational

guidelines • Training materials • Quality controls • Management input • Personnel

• None identified

Data • All data entered for use by the Renal Registry and specified additional items e.g. vascular access

• Other data entered but not analysed by the Registry



Suggested Approach

1. To review the data captured by the Renal Registry (and other organisations) to identify the errors and problems with the data.

2. To visit each unit to understand • the current administration and management processes supporting data entry and • the main data related problems.

3. To undertake a sample random assessment of the database registered data for completeness, validity and accuracy.

4. To review existing documentation and data validation routines relating to the source system(s).

5. To analyse the findings and propose solutions and recommendations for improvement. 6. To obtain approval for the recommendations and pilot.

7. To implement changes in practice where possible. Deliverables

• Agreed Terms of Reference • Interim report:

o defining the problems and causes o providing recommendations

• Documentation to support the implementation of agreed recommendations (subject to sufficient funding)

• Final project report: o results from pilot o suggested way forward

Suggested Timetable 2 weeks introduction and preparation 2 weeks at each of four renal units – 1st week review/analysis, 2nd week piloting change (not continuous necessarily) 1 week consolidation and preparation of report Budget Breakdown Further planning will be required to provide an accurate project budget breakdown. For estimating purposes, assume

• Expenses £1500 • Staff £8500.

£8500 will fund approximately eleven working weeks of Registry staff time, assuming an average of £20/hour.

Appendix 2 Results of Renal unit structure questionnaire Organisation Structure Quest.

No. Description Bangor Cardiff Clwyd Swansea Wrexham

1 Does an organisation chart exist?

Y Y N Y Y

Comment For Baxter staff, not whole NHS staff in the unit.

Considered good idea. Unit run by Fresenius.

2 Are informatics

staff included in the chart?

Y Y N/A N N

Budgetary Control Quest. No.

Description Bangor Cardiff Clwyd Swansea Wrexham

3 Who ‘owns’ the IT budget?

Baxter Renal Unit Fresenius Hospital IT Hospital & Renal Unit

Which items are covered by this budget?

Everything within the contract (not building maintenance)

All IT, including Vitalpulse software and Summerside IT specialists.

Unit budget covers support, maintenance, some training if money left over

4 2004/2005

hardware budget? £15,000 includes all training

Change in progress £0

2004/2005 software budget

Own system, developed on

Change in progress £0



Quest. No.


internal R&D budget 2004/2005 support

& maint. budget? Baxter support themselves at cost

£15,000 Change in progress £0 £5,000

Comment The budget is not ring fenced.

Fresenius not at meeting?

Charitable Trust fund provides hardware and contributes to other informatic core activities.

Hospital IT hold hardware and software budget. Problems getting issues resolved, no ownership of problems by IT and supplier.

5 Is there a dedicated

IT training budget? N See 4 above

N N N N

2004/2005 training budget?

N/A N/A

Responsibilities and Training Quest. No.


6 Is ultimate responsibility for delivery of Registry data assigned?

Y Renal Systems Manager

Consultant & Liverpool Royal Infirmary

N Y only recently

7 Is the responsibility

included in their job description?

N Y Unknown N/A N

What are these responsibilities?

Data quality, delivery, problem resolution, point of contact, look for improvements.

Not stated Training, liaison, accuracy

Not stated Changing the culture, liaison, taking the lead, problem resolution.



Quest. No.


8 Is responsibility

assigned for: Renal system dev. Data quality

Y Y

Y Y

N N

Y Y

Y Y

9 Is responsibility

assigned for: Loading RR nos. Running RR reports Tx RR reports Error correction

Y Y Y Y

Y Y Y Y

Y Y Y Y

Y Y Y Y

Y Y Y Y

10 System users, read

only HD nurses PD nurses Anaemia nurses Data entry clerk

HD nurses PD nurses Anaemia nurses Data entry clerk Transplant Co-ords Health Care Assts Consultants/Assoc Sp ‘Other’ nurses SpRs SHOs Dieticians Pharmacists Medical Secs Renal Technicians Research nurses Research Registrars Satellites

HD nurses PD nurses Anaemia nurses Data entry clerk Transplant Co-ords Health Care Assts Consultants/Assoc Sp ‘Other’ nurses SpRs SHOs Dieticians Pharmacists Medical Secs Ward clerks Receptionists

HD nurses PD nurses Anaemia nurses Data entry clerk Health Care Assts Consultants/Assoc Sp SpRs

HD nurses Data entry clerk Medical Secs

System users data HD nurses HD nurses HD nurses HD nurses HD nurses



Quest. No.


entry PD nurses Anaemia nurses Data entry clerk

PD nurses Anaemia nurses Data entry clerk Transplant Co-ords Health Care Assts Consultants/Assoc Sp

PD nurses Anaemia nurses Data entry clerk Transplant Co-ords Consultants/Assoc Sp Research nurses Pharmacists Medical Secs Ward clerks Receptionists

PD nurses Anaemia nurses Data entry clerk Health Care Assts Consultants/Assoc Sp SpRs

Data entry clerk Medical Secs

Comment Consultants could use the system but do not

Technicians provided by Fresenius.

Transplant Co-ords. Have link into Liverpool.

11 Which job

descriptions specifically mention Registry tasks?

Data Administrator Renal Systems Manager

Data Entry Clerk Data entry Clerk Dialysis Co-ord

Data Administrator

Which aspects of Registry work are described?

See job description Provision of data and reports.

See job description

12 Registry work, how

much training time is allocated by role to a new starter?

Nurses 30 mins Data Admin unknown

Data entry clerks training is spread over 1 week

Data entry clerks ‘a bit’

HD nurses 30 mins PD nurses 30 mins HC Assts 30 mins SpRs 30 mins

No formal training

Comment Data administrator has been in post since start up

Protocols in place for training Ward Clerks. Would like to see included as part of a structured induction.

New system implementation is resulting in training

Has been in post for a while, new starter would be given on the job training



Quest. No.


13 Are system training

courses scheduled? N N N N N

How frequently? On demand Aware of need. Attendance? 14 Do formal Registry

and system training or briefing manuals exist?

Y Y N N

N

Comment Data flow model User guides

Proton training manual (draft).

Proton training manual Lablink reference guide exists

15 How is the training

budget allocated? (Q5)

3.5 hours per employee per month, (includes all training)

Lack of a training budget is a much discussed internal issue. No IT training and no staff to provide it.

N/A. Training is not given a high priority.

N/A N/A. Make use of Andy Webb.



Results Internal and External Communications Quest.


16 Are Registry specific meetings held?

N

N

N

N

N

17 If Q16 = No, is the

Registry an agenda item at other meetings?

Y

Y

N

Y

N

Which meeting? Monthly department meeting.

Nephrology and Transplant Renal computing meeting. Frequency?

N/A Monthly department meeting.

Although not a standard item, problems are frequently discussed at the monthly Renal Unit meeting.

18 How is Registry

data quality reviewed?

Data Administrator checks quality ongoing.

Not reviewed. Frequently reviewed between Consultants.

Actively discussed at the monthly meeting, but reactive.

Managed? Monthly quality audits sent to Drs and discussed at monthly meeting.

Baxter and Fresenius undertake comparative audits and feedback to the units concerned.

Now implementing discussions at monthly unit meeting as standard, i.e. proactive.

Other? Annual ‘snapshot’ of data taken for review with Consultants.

Until 2001, Kieron Donovan held an informal Welsh Audit review following publication of the report. Considered beneficial as current and not based on out of date data.

Discussed following the Registry Annual Audit meeting.



Quest. No.


19 Are meetings held with satellite staff?

N/A

N

N/A

Y

N/A

Requests regarding data are sent to the Consultant in charge.

20 Are any other

meetings held to discuss/review data completeness and quality?

N

N

N

‘Thursday’ meeting

N

Meeting may be initiated in response to a crisis.

21 Are clinical review

meetings held? Y

Y

Y

Y

Y

Comment Incorporated into the monthly meeting.

In addition, 6 monthly clinical quality meeting run by Baxter.

If yes, which informatics staff attend?

Data Administrator None None Y not specified On occasion, Unit Data Administrator. Trust IT staff do not attend.

22 Move this question

to Best Practice section.



Good Practice Quest. No.


22 Who enters a new patient record? Day time (9-5) Evenings/weekends

Data Administrator N/A no out of hours cover contracted

Data Entry Clerks Waits

Data Entry Clerks Waits (can cause problems)

Data Entry Clerk HD nurse

Data Entry Clerk/Secretary Waits

23 Is there a

documented process for data entry?

Y Y N Y N

Copy available? ? Y N/A Y N/A 24 Average time

between consultation and data entry?

Immediate entry on receipt of details. ** see comments

48 hours 1 week, can be longer

Immediate entry Same day/ASAP

25 Which role is

responsible for data quality, accuracy and completeness?

Data Administrator Renal Systems Manager

None Dialysis Co-ord Consultant

26 Do absence

contingency plans exist for: Systems Manager Other IS staff?

Y Y

Y N

N N

N N

N Y



Registry Relationship Quest.


27 Staff awareness (% by job title) of The Registry Annual report Registry web site Annual meeting?

79% 71% 13% 1%

33% 20% 13% 0%

42% 25% 17% 17%

79% 71% 21% 36%

66% 47% 13% 7%

28 Was the unit

represented at the 2004 Annual User Group meeting?

Y Y N Y Y

If yes, what job roles were represented?

Software Manager

Consultant N/A IT Consultant

29 Do you know your

main point of contact in the Registry?

Y Y N/A Via Liverpool

Y Y

30 Would future visits

by Registry staff be of benefit?

Y Y Y Y Y

Comment Suggested feedback and review meeting to discuss these results.

Belief Registry could help obtain resources.

To help understand Registry requirements, share information, help explain problems to the Directorate, Registry services.



Quest. No.


31 Has a request for information (other than that contained in the Report) ever been made?

N Y N Y N

If yes, for what? Postcode analysis Thought there would be a charge.

Copy of report

32 Does the unit

receive a good response to queries and problems from the Registry?

Y Y, but usually has to chase

N/A Y N/A

IT Infrastructure and Systems Quest.


33 Hardware support is provided by?

Baxter Summerside IT Trust

Hospital IT Fresenius

Hospital IT Hospital IT

34 Software support is

provided by? Baxter Vitalpulse New system

Fresenius Vitalpulse Vitalpulse

Baxter (PD) 35 Are patient letters

linked to the renal clinical software?

N N N N N

If yes, how? Data entry clerks enter the data

Will be linked through Finesse

36 Would additional Y Y N Y Y



Quest. No.


hardware/software contribute to improving data quality?

Comment Enhance the software to allow capture of more data.

Validation on drug names and dosages. Increase server capacity.

Aberystwyth link Lablink Aberyst. & Carmarthen PAS links

Additional PC. Develop electronic links.

37 Which renal

software system is used?

Baxter Proton Proton/Finesse Di-Proton Di-Proton

38 How many

concurrent users are licensed?

7 could be increased at no cost

24 Insufficient, but cost prohibitive

8 Proton 11 Finesse

8 16 to be confirmed

39 Is there an interface

between the renal system and: Biochemical data Haematology PAS Other (name)?

Y Y N

Y Y Y Microbiology & Radiology in progress

In progress In progress In progress

Y Y N

Y Y N

Comment PAS data is copied to floppy disk and loaded daily.

Not linked to Radiology

40 Is there an

electronic biochemical data link with satellites?

N/A Y to all N/A Y N/A



Quest. No.


Comment Investigating the use of DTS Syntegra server to hold renal results. GPs could send tagged samples in flat file format and results would be electronically returned.

Carmarthen only

41 Are renal data

stored in any standalone systems?

Adequest PD Poet Excel (for vascular access, hepatitis, pre-dialysis, list of patients)

MS Access (for PD and vascular access) Adequest

Adequest Interface to Adequest Adequest Excel (low clearance, virology, vascular access)

Comment 42 Does a procedure

for identifying and resolving missing data items exist?

N Y N Y N

Comment Established procedure but not documented.

Clinic update sheets Problems exist with links, missing data, lack of training, poor system usage, paper records.

43 Does any renal

systems documentation exist?

Y Y N Y Manual may exist.

44 If the unit alters the

database, is there an N

N N N Some



Quest. No.


understanding of what change information is needed by the Registry?

Comment But, always check with the Registry when changes are made.

But, always check with Andy Webb and/or the Registry when changes are made.

This questionnaire has raised awareness of the need.

But, always check with the Registry when changes are made.

Miscellaneous Quest.


45 For non transplant centres, how many weeks from Tx to the pt transferring back?

N/A N/A 26 weeks 1 to 15 years 12 weeks to 1 year

Comment Patient care remains with Liverpool. If blood tests are done, results are sent to Liverpool.

May come down to 13 weeks.

46 For transplant

centre, how many weeks after Tx are pts transferred back?

N/A 1 year, but varies, some quicker, some never transfer back.

N/A N/A N/A



Quest. No.


47 Are the clinical data used for any local or regional returns?

? Y Y Y Y

If yes, which? Consultants are provided with audit information, it may go further.

Specific research projects and until 2001, the informal Welsh audit.

Via Liverpool, for Wales audit (ad hoc)

Not stated Regional sharing and NW audits.

48 Are laboratory

results manually entered?

Occasionally Y Y Y Occasionally

If yes, by whom? Data Administrator Data entry clerks Renal Systems Manager

Data Entry Clerks Ward Clerks

Data Entry Clerks Renal Systems Manager

Renal Systems Manager

49 Is there live access

to the renal software from: Consultants offices HD unit Wards Hospital clinics Satellites Off site clinics

Y (not used) Y N N N/A N

Y Y Y Y Y Y except 1

Y Y One Y N/A N

Y Y Y Y Y Carm. N Aber. N

N Y N Y not used N/A N

50 Has additional

pathology data validation been added to the system?

Uncertain Y N Uncertain Unlikely

Describe Validation is done in PMS, may not match Registry validation.

Validation done at Path lab.



Quest. No.


51 Is the unit interested

in receiving additional validation routines from the Registry to assist in identifying missing data items?

Y Y N Y Y

Comment Existing validation specifications would be useful to see if they can be incorporated into the Baxter system.

In particular the quark to flag missing data and errors prior to Registry submission.

Would need to ask Liverpool

52 Has this

questionnaire altered perceptions of the Registry?

Y Y Y N Y

Comment It has highlighted a lack of Registry knowledge.

Increased awareness of data quality requirements, organisation of data gathering and the responsibilities of Informatics staff.

Better understanding of the Registry. Discussion has been helpful.

General comments There is an

established culture of the importance of data entry and validation. Unit is running smoothly.

Initially Finesse will export to Liverpool Proton, eventually will produce own Registry data. Quarterly feedback

Identified a need for more protocols.

Regional meetings would be useful. A meeting of all 5 units to review these results was requested. Unit is trying to implement change and improve



Quest. No.


Good links with the Registry. There is a national shortage of renal nurses so recruitment can be a problem although they are fully staffed at Bangor. Transplant patients and costs are excluded from the remit of Baxter.

reports from the Registry would be useful. Comparative feedback would be useful. Should be more electronic links including with GPs.

quality. Need for training. Ongoing problems with lab links. Proton not user friendly, would prefer a Windows based system. To-ing and fro-ing of problems between local IT and Proton, no-one takes responsibility.

Appendix 3: Results of data completeness and accuracy For the purposes of this analysis, data accuracy needed to be compared against a defined ‘primary record’. The patient case note is not necessarily the primary record as some data may be found in the renal IT system which does not enter the patient case note. The defined primary record in the data tables below has been indicated by using shading in that field. The primary record is not necessarily consistent between sites, for example at one site the IT system may be the primary record for erythropoietin (EPO) prescription (updated by the EPO nurse), while others may use the patient case note. The percentage completeness for the Registry is measured against that of the primary record.

Example 1: If the primary record shows 20% completeness for ethnicity data and the Registry has

received all of this data, then Registry completeness is 100% rather than 20%. The accuracy is calculated as percentage of that data held in the primary record.

Example 2: If the primary record shows 60% completeness for ethnicity data (12/20 records) and the

Registry has received data on 10 records (2 missing) and 1 of the received records is different from the primary record, 9/12 records are correct and the accuracy is 75%. If all the 60% recorded ethnicity data is accurate then the Registry accuracy is 100%.

Patient names held on the IT system may reflect preferred first name rather than the formal name listed on the case sheet. The name field may also include other identifiers to help staff distinguish patients who have a common first name and surname (e.g. John Davies). This should not be interpreted to mean that the IT system holds inaccurate data. In contrast the date of birth should be consistent across all systems. The Renal Registry should be holding the same data as the IT system from which it was extracted. In transplanted patients there may be conflicting data arriving at the Registry from the transplant centre and the parent dialysis centre. This may even include a conflict in the date of birth. Due to the reliance of laboratory interfaces on this field being identical with that held in the renal IT system most renal units are unwilling to change errors identified in this field. For some of the North Wales patients, data held by the Registry is greater than that held on the local IT system. This is because those patients on the transplant waiting list at Liverpool may have had some laboratory investigations at this renal unit, e.g. cytomegalovirus (CMV) status. Also, the primary renal diagnosis may have been received by the Registry from Liverpool renal unit whilst being unavailable in the local IT system of the dialysis centre. Some laboratory parameters may only be measurement annually (eg cholesterol ) so that a high percentage of missing data in one quarter would be expected. This should be contrasted with measurement of haemoglobin which would be monitored on a quarterly basis for dialysis patients and less frequently in stable transplant patients. Some laboratory parameters are only measured for patients on specific modalities so ‘all patient’ data completeness is low (eg URR in HD patients, CMV in transplant waiting list patients, HBA1c in diabetic patients). The tables would become too complex if they were to be split by RRT modality. Some data fields have not been extracted from the IT system or from case notes into the IT system to be sent to the Registry. In these cases where no data at all have been received, the accuracy has been logged as n/a (not applicable).



How to use these tables

1. Look down the column of the primary record to judge the completeness of locally held data.

2. Look down the Registry ‘completeness column’ to assess how well these data are transferred to the Registry. Compare this with the local IT completeness column (if not the primary record) to see if that is where errors lie.

3. Look down the ‘data accuracy column’ for the Registry to assess how well the Registry performs in accurately holding these data. Then compare this with accuracy of the local IT system as the source of the Registry data. Discrepancies between the two systems highlight data extraction errors. Low data accuracy rates in the local IT system point to data collection/ entry issues.

The tables below show the aggregated results from the comparisons between patient notes, the local renal system and the Renal Registry database of the selected patient sample.

Bangor Data Completeness Data Accuracy Item Total

available No. complete Percentage complete No. accurate Percentage accurate

Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR DEMOGRAPHIC Surname 22 22 22 22 100% 100% 100% 22 22 22 100% 100% First name 22 22 22 22 100% 100% 100% 22 18 18 82% 82% Date of birth 22 22 22 22 100% 100% 100% 22 21 21 95% 95% Postcode 22 22 22 22 100% 100% 100% 22 22 22 100% 100% NHS number 22 20 14 14 91% 64% 64% 20 14 14 70% 70% Ethnicity 22 0 1 14 0% 5% 64% 0 0 13 0% n/a Renal Registry number 22 n/a 22 22 n/a 100% 100% n/a 22 22 n/a 100% Primary renal diagnosis 22 20 22 22 91% 100% 100% 16 18 17 100% 94% Verify patient is alive or dead

7 7 7 7 100% 100% 100% 7 7 7 100% 100% Date of death 7 7 5 7 100% 71% 100% 7 1 4 14% 57% Cause of death 7 6 1 1 86% 14% 14% 6 1 1 17% 17% Date start renal replacement therapy 22 20 20 20 91% 91% 100% 20 12 14 60% 70% Date 1st seen by nephrologist

22 16 9 8 73% 41% 36% 16 8 7 50% 43% Height 22 6 21 20 27% 95% 91% 6 19 14 100% 74% Verify timeline 22 n/a 22 22 n/a 100% 100% n/a 22 22 n/a 100% Last Modality in last quarter 2003 or before death 22 22 22 22 100% 100% 100% 22 21 21 95% 95% Place treated for HD or Tx patients 14 14 14 14 100% 100% 100% 14 14 14 100% 100% BIOCHEM Haemoglobin 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100%





Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR Ferritin 15 n/a 12 12 n/a 80% 80% n/a 12 12 n/a 100% Creatinine 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Urea 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Calcium corrected 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Phosphate 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Parathyroid hormone 15 n/a 11 11 n/a 73% 73% n/a 11 11 n/a 100% Cholesterol 15 n/a 6 6 n/a 40% 40% n/a 6 6 n/a 100% HbA1c 15 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Systolic BP 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Diastolic BP 15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Post HD systolic BP 9 3 9 9 33% 100% 100% 1 8 8 11% 100% Post HD diastolic BP 9 3 9 9 33% 100% 100% 1 8 8 11% 100% URR 9 6 9 9 67% 100% 100% 4 8 8 44% 100% EPO USAGE On EPO y/n 15 15 15 15 100% 100% 100% 15 14 14 93% 93% If yes dose on 31/12/2003 13 10 7 77% 54% SEROLOGY Hep B antibody status 31/12/2003 15 n/a 5 7 n/a 33% 47% n/a 5 4 n/a 100% CMV status 0n 31/12/2003 15 n/a 0 6 n/a 0% 40% n/a 0 6 n/a N/A COMORBIDITY Angina 15 2 14 13 13% 93% 87% 2 12 12 14% 93% Previous MI within last 3 months 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Previous MI > 3 mths ago 15 1 14 13 7% 93% 87% 1 13 13 7% 93% Previous CAGB or Coronary Angioplasty 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Smoking 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Chronic Obstructive Pulmonary Disease 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Cerebrovascular disease – symptomatic 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Diabetes – not causing ESRF

15 1 14 13 7% 93% 87% 1 13 13 7% 93% Malignancy 15 1 14 13 7% 93% 87% 1 13 13 7% 93% Liver disease 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Claudication 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Ischaemic / Neuropathic ulcers 15 0 14 13 0% 93% 87% 0 13 13 0% 93%





Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR Angioplasty / stent / vascular graft (non coronary) 15 0 14 13 0% 93% 87% 0 13 13 0% 93% Amputation for PVD 15 0 14 13 0% 93% 87% 0 13 13 0% 93%



Cardiff Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR DEMOGRAPHIC Surname 18 18 18 18 100% 100% 100% 18 18 17 100% 94% First name 18 18 18 18 100% 100% 100% 18 18 17 100% 94% Date of birth 18 18 18 18 100% 100% 100% 18 18 18 100% 100% Postcode 18 18 18 18 100% 100% 100% 18 18 18 100% 100% NHS number 18 15 17 15 83% 94% 83% 15 16 15 100% 100% Ethnicity 18 0 2 2 0% 11% 11% 0 1 2 0% 100% Renal Registry number 18 n/a 18 18 n/a 100% 100% n/a 18 18 n/a 100% Primary renal diagnosis 18 12 18 18 67% 100% 100% 12 17 17 100% 100% Verify patient is alive or dead

5 5 5 5 100% 100% 100% 5 5 5 100% 100% Date of death 5 5 5 5 100% 100% 100% 4 3 3 75% 75% Cause of death 5 4 0 0 80% 0% 0% 4 0 0 n/a n/a Date start renal replacement therapy 18 4 18 18 22% 100% 100% 4 3 3 n/a n/a Date 1st seen by nephrologist

18 4 0 0 22% 0% 0% 4 0 0 n/a n/a Height 18 2 14 6 11% 78% 33% 1 14 4 29% Verify timeline 18 n/a 18 18 n/a 100% 100% n/a 18 17 n/a 94% Last Modality in last quarter 2003 or before death 18 13 18 18 72% 100% 100% 13 17 16 76% 94% Place treated for HD or Tx patients 14 9 14 14 64% 100% 100% 6 13 8 100% BIOCHEM Haemoglobin 13 n/a 13 12 n/a 100% 92% n/a 12 12 n/a 100% Ferritin 13 n/a 9 8 n/a 69% 62% n/a 8 8 n/a 100% Creatinine 13 n/a 13 12 n/a 100% 92% n/a 12 12 n/a 100% Urea 13 n/a 13 12 n/a 100% 92% n/a 12 12 n/a 100% Calcium corrected 13 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Phosphate 13 n/a 13 12 n/a 100% 92% n/a 12 12 n/a 100% Parathyroid hormone 13 n/a 8 5 n/a 62% 38% n/a 6 5 n/a 63% Cholesterol 13 n/a 9 5 n/a 69% 38% n/a 5 5 n/a 100% HbA1c 13 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Systolic BP* 13 n/a 0 5 n/a 0% 38% n/a 0 4 n/a n/a Diastolic BP 13 n/a 0 5 n/a 0% 38% n/a 0 4 n/a n/a Post HD systolic BP 4 0 0 0 0% 0% 0% 0 0 0 0% n/a Post HD diastolic BP 4 0 0 0 0% 0% 0% 0 0 0 0% n/a URR 4 0 3 3 0% 75% 75% 0 3 3 0% 100%



Cardiff Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR EPO USAGE On EPO y/n 13 1 12 12 8% 92% 92% 1 12 12 100% If yes dose on 31/12/2003 0 7 1 8% SEROLOGY Hep B antibody status 31/12/2003 13 n/a 7 0 n/a 54% 0% n/a 7 0 n/a n/a CMV status 0n 31/12/2003 13 n/a 2 0 n/a 15% 0% n/a 2 0 n/a n/a COMORBIDITY Angina 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Previous MI within last 3 months 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Previous MI > 3 mths ago

13 0 2 1 0% 15% 8% 0 2 1 0% 50% Previous CAGB or Coronary Angioplasty 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Smoking 13 1 2 1 8% 15% 8% 0 2 1 0% 50% Chronic Obstructive Pulmonary Disease 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Cerebrovascular disease – symptomatic 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Diabetes – not causing ESRF

13 0 2 1 0% 15% 8% 0 2 1 0% 50% Malignancy 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Liver disease 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Claudication 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Ischaemic / Neuropathic ulcers 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Angioplasty / stent / vascular graft (non coronary) 13 0 2 1 0% 15% 8% 0 2 1 0% 50% Amputation for PVD 13 0 2 1 0% 15% 8% 0 2 1 0% 50% * systolic blood pressure data field could not be found in the local IT system. So the data could not be verified



Clwyd Data Completeness Data Accuracy Item Total


Notes Local

IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR DEMOGRAPHIC Surname 20 20 20 20 100% 100% 100% 20 20 20 100% 100% First name 20 20 20 20 100% 100% 100% 20 15 15 75% 75% Date of birth 20 20 20 20 100% 100% 100% 20 20 20 100% 100% Postcode 20 20 20 20 100% 100% 100% 20 19 18 95% 90% NHS number 20 19 9 10 95% 45% 50% 19 9 8 47% 42% Ethnicity 20 0 9 10 0% 45% 50% 0 9 10 n/a n/a Renal Registry number 20 n/a 20 20 n/a 100% 100% n/a 20 20 n/a 100% Primary renal diagnosis 20 18 18 20 90% 90% 100% 18 16 18 80% 100% Verify patient is alive or dead


20 12 3 1 60% 15% 5% 12 1 0 5% 0% Height 20 4 3 1 20% 15% 5% 4 2 0 10% 0% Verify timeline 20 n/a 20 20 n/a 100% 100% n/a 17 20 n/a 100% Last Modality in last quarter 2003 or before death

20 18 18 20 90% 90% 100% 18 16 18 89% 100% Place treated for HD or Tx patients

13 13 13 13 100% 100% 100% 10 9 13 90% 100% BIOCHEM Haemoglobin 15 n/a 11 12 n/a 73% 80% n/a 11 12 n/a 100% Ferritin 15 n/a 12 12 n/a 80% 80% n/a 12 12 n/a 100% Creatinine 15 n/a 12 12 n/a 80% 80% n/a 12 10 n/a 83% Urea 15 n/a 12 12 n/a 80% 80% n/a 12 10 n/a 83% Calcium corrected 15 n/a 12 12 n/a 80% 80% n/a 12 11 n/a 92% Phosphate 15 n/a 12 12 n/a 80% 80% n/a 11 10 n/a 91% Parathyroid hormone 15 n/a 9 9 n/a 60% 60% n/a 8 9 n/a 100% Cholesterol 15 n/a 2 1 n/a 13% 7% n/a 1 1 n/a 100% HbA1c 15 n/a 2 2 n/a 13% 13% n/a 1 1 n/a 100% Systolic BP 15 n/a 3 2 n/a 20% 13% n/a 3 2 n/a 67% Diastolic BP 15 n/a 3 2 n/a 20% 13% n/a 3 2 n/a 67% Post HD systolic BP 10 0 1 0 0% 10% 0% 0 1 0 0% 100% 0% Post HD diastolic BP 10 0 1 5 0% 10% 50% 0 1 0 0% 100% 0% URR 10 0 2 5 0% 20% 50% 0 2 5 0% 100% 100% EPO USAGE



Clwyd Data Completeness Data Accuracy Item Total


Notes Local

IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR On EPO y/n 15 7 14 14 47% 93% 93% 7 14 13 50% 93% If yes dose on 31/12/2003 15 11 9 0% 73% 60% 6 10 3 40% 30% SEROLOGY Hep B antibody status 31/12/2003

15 n/a 4 5 n/a 27% 33% 0 4 1 n/a 100% 25% CMV status 0n 31/12/2003 15 n/a 4 5 n/a 27% 33% 0 4 5 n/a 100% 100% COMORBIDITY Angina 15 1 0 0 7% 0% 0% 1 0 0 n/a n/a Previous MI within last 3 months 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a Previous MI > 3 mths ago 15 2 0 0 13% 0% 0% 2 0 0 n/a n/a Previous CAGB or Coronary Angioplasty 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a Smoking 15 1 0 0 7% 0% 0% 1 0 0 n/a n/a Chronic Obstructive Pulmonary Disease 15 1 0 0 7% 0% 0% 1 0 0 n/a n/a Cerebrovascular disease – symptomatic 15 1 0 0 7% 0% 0% 1 0 0 n/a n/a Diabetes – not causing ESRF

15 2 0 0 13% 0% 0% 2 0 0 n/a n/a Malignancy 15 2 0 0 13% 0% 0% 2 0 0 n/a n/a Liver disease 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a Claudication 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a Ischaemic / Neuropathic ulcers 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a Angioplasty / stent / vascular graft (non coronary) 15 1 0 0 7% 0% 0% 1 0 0 n/a n/a Amputation for PVD 15 0 0 0 0% 0% 0% 0 0 0 n/a n/a



Swansea Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR DEMOGRAPHIC Surname 20 20 20 20 100% 100% 100% 20 19 20 95% 100% First name 20 20 20 20 100% 100% 100% 20 19 19 95% 95% Date of birth 20 20 20 20 100% 100% 100% 20 19 18 95% 90% Postcode 20 18 20 20 90% 100% 100% 18 20 18 100% 100% NHS number 20 9 7 10 45% 35% 50% 9 7 10 78% 100% Ethnicity 20 1 19 19 5% 95% 95% 1 19 19 100% Renal Registry number 20 n/a 20 20 n/a 100% 100% n/a 20 20 n/a 100% Primary renal diagnosis 20 17 20 20 85% 100% 100% 15 17 16 88% 94% Verify patient is alive or dead


20 5 16 0 25% 80% 0% 5 16 0 80% n/a Height 20 0 13 13 0% 65% 65% 0 13 13 0% 100% Verify timeline 20 n/a 20 20 n/a 100% 100% n/a 19 19 n/a 100% Last Modality in last quarter 2003 or before death 20 13 19 20 65% 95% 100% 13 18 18 65% 100% Place treated for HD or Tx patients 14 5 12 13 36% 86% 93% 5 11 13 36% 100% BIOCHEM Haemoglobin 15 n/a 12 14 n/a 80% 93% n/a 12 14 n/a 100% Ferritin 15 n/a 8 9 n/a 53% 60% n/a 8 9 n/a 100% Creatinine 15 n/a 14 14 n/a 93% 93% n/a 14 12 n/a 86% Urea 15 n/a 14 14 n/a 93% 93% n/a 14 12 n/a 86% Calcium corrected 15 n/a 0 0 n/a n/a n/a n/a 0 0 n/a n/a n/a Phosphate 15 n/a 14 14 n/a 93% 93% n/a 14 13 n/a 93% Parathyroid hormone 15 n/a 7 7 n/a 47% 47% n/a 7 7 n/a 100% Cholesterol 15 n/a 6 6 n/a 40% 40% n/a 6 6 n/a 100% HbA1c 15 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a% Systolic BP 15 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a% Diastolic BP 15 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a% Post HD systolic BP 5 0 3 0 0% 60% 0% 0 3 0 0% 0% Post HD diastolic BP 5 0 3 0 0% 60% 0% 0 3 0 0% 0% URR 5 0 0 1 0% 0% 20% 0 0 1 0% 100%



Swansea Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR EPO USAGE On EPO y/n 15 12 15 15 80% 100% 100% 12 12 12 100% 100% If yes dose on 31/12/2003 5 4 3 100% 75% SEROLOGY Hep B antibody status 31/12/2003 15 n/a 1 0 n/a 7% 0% n/a 1 0 n/a n/a CMV status 0n 31/12/2003 15 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a COMORBIDITY Angina 15 15 15 100% 100% 15 15 100% Previous MI within last 3 months 15 15 15 100% 100% 15 15 100% Previous MI > 3 mths ago 15 15 15 100% 100% 15 15 100% Previous CAGB or Coronary Angioplasty 15 15 15 100% 100% 15 15 100% Smoking 15 15 15 100% 100% 13 13 100% Chronic Obstructive Pulmonary Disease 15 15 15 100% 100% 15 15 100% Cerebrovascular disease – symptomatic 15 15 15 100% 100% 15 15 100% Diabetes – not causing ESRF

15 15 15 100% 100% 14 14 100% Malignancy 15 15 15 100% 100% 15 15 100% Liver disease 15 15 15 100% 100% 15 15 100% Claudication 15 15 15 100% 100% 15 15 100% Ischaemic / Neuropathic ulcers 15 15 15 100% 100% 13 13 100% Angioplasty / stent / vascular graft (non coronary) 15 15 15 100% 100% 14 14 100% Amputation for PVD 15 15 15 100% 100% 14 14 100%



Wrexham Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR DEMOGRAPHIC Surname 15 15 15 15 100% 100% 100% 15 15 15 100% 100% First name 15 15 15 15 100% 100% 100% 15 15 15 100% 100% Date of birth 15 15 15 15 100% 100% 100% 15 15 15 100% 100% Postcode 15 15 15 15 100% 100% 100% 15 14 14 93% 93% NHS number 15 15 0 5 100% 0% 33% 15 0 5 0% 33% Ethnicity 15 0 6 7 0% 40% 47% 0 6 7 0% 100% Renal Registry number

15 n/a 15 15 n/a 100% 100% n/a 15 15 n/a 100% Primary renal diagnosis

15 14 6 9 93% 40% 60% 14 4 6 29% 43% Verify patient is alive or dead

4 4 4 4 100% 100% 100% 4 4 4 100% 100% Date of death 4 3 4 4 75% 100% 100% 3 2 2 66% 66% Cause of death 4 0 0 0 0% 0% 0% 0 0 0 n/a% n/a Date start renal replacement therapy 15 13 15 1 87% 87% 100% 13 9 12 60% 92% Date 1st seen by nephrologist

15 9 1 0 60% 7% 0% 9 0 0 0% n/a Height 15 1 2 0 7% 13% 0% 1 2 0 100% n/a Verify timeline 15 n/a 15 15 n/a 100% 100% n/a 11 11 n/a 100% Last Modality in last quarter 2003 or before death 15 12 15 15 80% 100% 100% 11 14 13 73% 93% Place treated for HD or Tx patients 12 10 12 12 83% 100% 100% 6 10 6 60% 60% BIOCHEM Haemoglobin 11 n/a 10 10 n/a 91% 91% n/a 9 9 n/a 100% Ferritin 11 n/a 9 8 n/a 82% 73% n/a 8 8 n/a 100% Creatinine 11 n/a 10 10 n/a 91% 91% n/a 10 6 n/a 60% Urea 11 n/a 10 10 n/a 91% 91% n/a 10 6 n/a 60% Calcium corrected 11 n/a 10 10 n/a 91% 91% n/a 10 9 n/a 90% Phosphate 11 n/a 10 10 n/a 91% 91% n/a 10 9 n/a 90% Parathyroid hormone 11 n/a 5 5 n/a 45% 45% n/a 5 4 n/a 90% Cholesterol 11 n/a 6 6 n/a 55% 55% n/a 6 6 n/a 100% HbA1c 11 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Systolic BP 11 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Diastolic BP 11 n/a 0 0 n/a 0% 0% n/a 0 0 n/a n/a Post HD systolic BP 4 0 0 0 0% 0% 0% 0 0 0 0% n/a Post HD diastolic BP 4 0 0 0 0% 0% 0% 0 0 0 0% n/a



Wrexham Data Completeness Data Accuracy Item Total


Notes Local IT RR Notes Local IT RR Notes Local IT RR Notes Local IT RR URR * 4 0 0 3 0% 0% 75% 0 0 1 0% 100% EPO USAGE On EPO y/n 11 10 10 11 91% 91% 100% 9 4 5 36% 55% If yes dose on 31/12/2003 5 2 2 40% 40%

SEROLOGY Hep B antibody status 31/12/2003 11 n/a 3 0 n/a 27% 0% 0 3 0 0% n/a CMV status 0n 31/12/2003

11 n/a 0 1 n/a 0% 9% 0 0 0 0% n/a

COMORBIDITY Angina 11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Previous MI within last 3 months 11 0 0 0 0% 0% 0% 0 0 0 n/a n/a Previous MI > 3 mths ago

11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Previous CAGB or Coronary Angioplasty 11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Smoking

11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Chronic Obstructive Pulmonary Disease 11 0 0 0 0% 0% 0% 0 0 0 n/a n/a Cerebrovascular disease – symptomatic 11 2 0 0 18% 0% 0% 1 0 0 n/a n/a Diabetes – not causing ESRF

11 1 0 0 9% 0% 0% 0 0 0 n/a n/a Malignancy

11 4 0 0 36% 0% 0% 3 0 0 n/a n/a Liver disease 11 0 0 0 0% 0% 0% 0 0 0 n/a n/a Claudication 11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Ischaemic / Neuropathic ulcers 11 1 0 0 9% 0% 0% 1 0 0 n/a n/a Angioplasty / stent / vascular graft (non coronary) 11 0 0 0 0% 0% 0% 0 0 0 n/a n/a Amputation for PVD

11 0 0 0 0% 0% 0% 0 0 0 n/a n/a * URR data may not be held in the IT system although the Registry software calculates the data at extraction



Appendix 4: Renal Registry Data Validation Procedures (UK Renal Registry Report 2002) Introduction It is the appropriate time to publish a description of the data validation procedures of the Renal Registry. Not only has the experience of data analysis from previous years reached a significant level, but it is also opportune in the year in which centres are identified in the report to demonstrate the care that the Registry has taken to ensure the quality of the data held. This chapter explains some of the extensive validation processes run by the Registry to make sure that the data are as accurate as possible. The data validation process at the Renal Registry is divided into four separate parts: 1. Consistency checks when each of the quarterly files is loaded 2. Validation against previous data on the database after the quarterly file has been loaded 3. Validation on the database for consistency of data for the whole of the year 4. Statistical routines written in the SAS statistical language that run other consistency checks. Processes 1 and 4 were in place at the end of the original 2 year pilot project but were often performed at a late stage. Delays in the Registry report resulted particularly from finding errors in the data at the fourth step. As a consequence, it was necessary to discuss problems with the affected renal units, eliminate any local database issues, re-run extractions, reload the Registry database, revalidate and then re-extract the data into SAS, a process that could add months to the reporting timescale. Over the past year, the Registry has increased the number of processes at stages 2 and 3 to be performed quarterly as data are received in order to reduce the number of potential errors picked up at a late stage. Common data errors The most common errors resulting in calls to renal units from the data management team are listed below:

1. Missing EDTA disease primary disease codes 2. Missing haemodialysis supervision code (i.e. home, hospital or satellite) against a treatment

modality 3. Incorrect figures, e.g. blood pressure 130/4 mmHg 4. Dates that do not match, e.g. the date of starting dialysis being earlier than the date of birth 5. Checks on discrepancies relating to patient names and dates of birth between transplant and

dialysis centres. These problems were particularly common when patients transferred for transplantation

6. Text in a numeric data field.



The Renal Registry modality timeline The modality timeline contains information on the changes of modality. It is the most important item in the data sent to the Registry as the local extraction software uses this information to decide whether or not to send data to the Registry.

1. Local software detects renal replacement therapy (RRT) modality entries in this field to flag the patient as being on RRT and sends data to the Registry.

2. If a patient recovers function, the software sends no more data to the Registry for that patient until a new modality is detected (so ‘recovered function’ must not be used for transplanted patients).

3. If a patient ‘transfers out’, the software sends no more data to the Registry for that patient until the patient is transferred back in.

Accuracy is essential as analysis of these data by the Registry is critical in the following areas:

1. Take-on rates are calculated from the first modality in the timeline. 2. If the first modality is ‘transferred in’, the patient is excluded from the incident

calculations. 3. Recovery of function excludes these patients from subsequent prevalence figures. 4. Prevalence rates are calculated by excluding any patients transferred out. 5. If patients are not recorded as having transferred out, no further results thus being

obtained, they will be included in the analysis as having data missing. The validation process From the experience of the first 5 years, the Registry staff have recognised many common sources of error and have developed automated processes to check for these errors as early as possible in the data-loading process in order to avoid delays in data analysis after the data for the last quarter of the year have been loaded.

Consistency checks at load Range tests for numeric items (e.g. biochemistry) and all the numeric data have simple range checks on them. These ranges have to be very wide to accommodate both physiological ranges (creatinine = 130 micromoles/L) and the pathological range (creatinine = 3500 micromoles/L). They also have to accept paediatric values. The most frequent errors are found in the manual entry fields as a result of mis-keying the data. Many sites have not set up local range validation within their own database; instituting this feature would reduce the number of calls received from the data management team at the Registry. The most frequent errors are found in the blood pressure data, where for example an entry of 14 instead of 140 might be recorded. The second most common error is length of time on dialysis, hours being entered instead of minutes or, in other systems, minutes instead of hours. Similar mis-keying has been found in many other data fields, particularly where satellites units manually enter laboratory data.

We encourage sites, wherever possible, to add these range checks locally.



1. Validity of coded entry fields. Whenever possible, the Renal Registry uses Read codes or EDTA codes (for primary diagnosis and cause of death), these being validated against their acceptable ranges. Some renal units have modified the European EDTA codes, which this can cause validation errors. Again, a lack of fixed coding in the local renal unit database is the source of these errors. Some renal units allow manual entry of the EDTA code number rather than a specific diagnosis that is internally converted by the database into an EDTA code number. Sites that offered the ICD10 list of diagnoses for the primary diagnosis field have been successfully discouraged from this practice as this can result in errors – on one occasion, cerebrovascular accident was given as the primary cause of chronic renal failure.

2. Avoidance of duplicate patients. As patient data are received from both dialysis and transplant

centres, it can be very easy, if the right systems are not in place, for patients to be duplicated on the database. This is not just a simple process of flagging up patients with the same surname, forename and date of birth. Many patients have their names spelt in a slightly different way on different databases, and it is impossible to impose consistency between two sites. In addition, dates of birth can vary (by days or months), and renal units have been unwilling to change these data, partly perhaps because the local automated laboratory links may refuse to load patient data if the date of birth or name is spelt differently from that held in the laboratory systems.

Detecting these duplicated patients relies on a complex series of procedures:

1. The NHS number. This is a unique identifier for patients in patients in England & Wales, but many sites do not store this number in their renal system so it has not yet been found to be useful. The Registry submits files to the NHS number tracing service.

2. UK Transplant number. Patients on the transplant waiting list, or who have been transplanted,

are allocated a unique number that should be stored on the local renal unit database (although it may be held only at the transplanting centre). This is recorded by the Registry as a non-duplicated indexed field. The Registry validates these numbers annually with UK Transplant, this process detecting mis-keyed data entry errors (e.g. 97074 instead of 90774). Renal units are informed of any mis-keying errors found.

3. Soundex database index on names. 4. Check on the address field.

Once a new patient has been identified, the Registry allocates its own unique ID number for the patient, which is transmitted back to the renal unit and loaded electronically to avoid any mis-keying.

5. Validation of the postcode. The Renal Registry uses a commercial postcoding package (QAS

systems), which checks the validity of the postcode against the address fields, the software automatically correcting any postcode errors. This is important as postcodes are used for NHS number tracing and also by the Registry for health authority mapping and deprivation scoring.



6. Date fields. All date fields must be in the format dd/mm/yyyy and are checked for validity. 7. There are other consistency checks, some of which are listed below:

1) Date of first RRT < date of birth (as mis-keying has been found)

2) Date of death < date of birth and < date of first RRT (as mis-keying of the date of death has occurred)

3) Data are loaded for a patient already registered as dead. A transplant unit may have

accidentally logged a patient as dead when he or she has returned to dialysis

4) Mandatory fields (e.g. cause of primary renal disease) missing

5) Inappropriate data for specific treatment modalities. A urea reduction ratio value or length of time on haemodialysis cannot be sent while the patient is on peritoneal dialysis

6) There must be new patients starting RRT in every quarterly file received from the

renal unit

7) There must be some deaths in every quarterly file.

Consistency checks after loading There are many errors that cannot be checked at the time the file is loaded. This is because these errors relate to the overall data integrity for the renal unit compared with the data from previous quarter. 1. Many long-standing transplant patients are not seen quarterly. Missing creatinine data over

three consecutive quarters result in a telephone call to the unit in case the patient has died or transferred elsewhere without this being logged in the renal unit database.

2. Incorrect modality in a timeline. Some patients have had a modality of ‘recovered renal function’, rather than ‘graft functioning’, sent to the Registry after transplantation. This is invalid but can only be checked once the data have been loaded. Renal units are contacted to change this item.

3. The date of death must be later than last modality entry in the timeline. This item produced an error for 28 patients on one annual check, all caused by mis-keying the date of death.

Annual data checks 1. The number of patients starting RRT should be spread across all quarters, and the total number

must be consistent with previous years. 2. Prevalent patient totals should rise on an annual basis. 3. Health authority take-on rates should show consistency. 4. In every centre, there should be timeline entries with a treatment start date for every quarter

over which the Registry collected data (Table 3.1). There will always be some patients in a quarter who have changed modality; lack of modality changes indicates either a software problem or missing data.



Quarter

no. No. of new timeline

entries 12 45 13 52 14 42 15 50 16 66 17 71 18 52 19 87 20 42 21 36 22 52 23 45 24 1

Table 3.1: Frequency of timeline entries per quarter – an example 5. Deaths should occur in every quarter, and the total number of deaths should be evenly spread.

A lack of deaths registered often represents a software extraction fault rather than an error of logging by the renal unit. An excess of deaths in a given quarter can also be identified and of course investigated.

When errors such as those listed are found, the Registry contacts the renal unit for clarification and correction where appropriate.

SAS analysis Many of these potential errors are rechecked through SAS, outliers being identified. Examples include the following: 1. Have centres transferred patients in and out? (The number of transfers in and out is usually

roughly equal.) The results shown in Table 3.2 indicate that some dialysis centres do not transfer patients out prior to transplantation at another centre, and some centres do not transfer them back in. In such cases, the Registry adds the appropriate code to the timeline in the Registry database. This is important as, during the analysis, each patient is allocated to a centre for a specific time point. This is not only for calculating the total number of patients at a centre, but also for allocating missing biochemistry data to a centre.

Centre Transfer in Transfer out

RAJ01 50 61 RAQ01 181 195 RAZ 122 79 RCJAT 441 441 RCSLB 203 98 REE01 316 251 RF201 403 349 RFBAK 82 58 RFPFG 8 15 RGD03 2 RH641 140 157 RH8 171 116 RHW01 29 48 RJ121 91 94 RK7CC 89 65



Centre Transfer in Transfer out RK950 152 56 RKB01 190 207 RKHA4 134 210 RL403 169 149 RL7 411 396 RLGAY 146 147 RLNGH 274 234 RMF01 242 306 RNA03 200 232 RNX02 401 188 RQHC7 113 260 RQR13 1149 1118 RQS01 1 RRBBV 353 230

Table 3.2: Frequency of timeline transfer

2. How many patients at each centre recover renal function during the first 360 days of starting RRT? All hospitals should have a few patients who recover renal function during the course of their first year. If the number for a particular centre is too low, some patients are probably being misclassified as ‘acute’. Conversely, centres with an excessive number of patients recovering within the first 90 days are probably including a large number with acute renal failure. Patients may be correctly classified as having chronic renal failure and recover function within 90 days.

3. What are the first treatment modalities, and do any patients have two different treatment

modalities on the same day of their first modality? 4. Have any patients been transplanted at a ‘dialysis’ centre? Some dialysis centres do not

transfer patients out for a transplant and furthermore may log the transplant as having taken place at their centre.

5. Is there a variation in biochemistry results? Software errors have been picked up by

identifying centres if patients’ results have not changed on a quarterly basis (e.g. haemoglobin level 10.6 g/dL) for two sequential quarters.

6. Is there a difference between the pre-dialysis and post-dialysis blood pressure? Such a

difference is usually seen, so if the systolic and diastolic pressures are identical, one value is likely to be invalid.

7. Has there been a large increase in missing data within a quarter for any data item? This

will be queried with the site. One site showed a large increase in missing urea reduction ratio data that had arisen from an undetected clerical error in storage of the post-dialysis sample data in the local database. Similarly, a reduction in returns for ethnicity was shown to be due to a change in the method of coding this item.



8. Is there a complete absence of an expected biochemistry item (e.g. ferritin)? This will be

confirmed with the site. At two sites, the internal software links to these tables in the database were shown to be faulty.

9. The haemoglobin data for one site this year had begun arriving as an integer (e.g. 9 or 10

g/dL). This error will pass all the other validation checks and was shown up only on analysis of the distribution of these results (quartiles). This resulted in a software fix at the site and the necessity of re-exporting all the patient quarterly haemoglobin data for 2001. This was reloaded into the database and into SAS, and all the haemoglobin analyses were re-run (as the overall means will change) and graphs replaced in the text.

Comment This is not an exhaustive description of the data validation routines, but the complexity of the task is evident. Much of the data quality depends on the monitoring of data entry at the originating centre. The local verification of data on a regular basis, through review meetings or weekly or monthly printed reports, is a necessary component of quality to assure the entries. The careful induction of new renal staff is another important issue. These checks may also be supplemented by automated routines for missing data and anomalous entries, a process with which the Registry is willing to help. A renal unit’s progressive attention to the detail of data registration should allow a reduction in the validation effort required of Registry staff and a shorter interval between download and the production of the Report. The key to further improvement in data quality is an open exchange of information and an increasing scrupulousness at renal unit level, which the UK Registry always attempts to match.

uk renal registry welsh data validation exercise … · 2015-10-27 · uk renal registry welsh data...

Documents