Performing Audits

Def – “An assessment of the methods

and procedures used”

Philip Randall

(with contributions from Quentin

Johnson)

Master Control

• Much of the material has been taken from information available on the

internet and/or from components of software for ISO conformity,

document control i.e. SOP Management etc.

• Useful resource http://www.mastercontrol.com/

It is not a blame game

• The fact that the audit process casts auditors on one side and those

being audited on the opposite side has created a general impression that

auditing is adversarial.

• While a point-and-blame atmosphere may indeed exist in some cases,

more and more organizations recognize that an audit does not have to

be a negative experience or a dreaded event – when implemented

properly, audit can be one of the most effective means for

improvement

Audits work both internally and externally

• ISO was instrumental in boosting interest in quality audits among manufacturers and

other types of businesses when it published the ISO 9000 standards in 1987.

• Today, popular standards such as ISO 9001, ISO 14001 and ISO 13485 all require

internal audits of the quality system.

• Under these standards, audit serves as a mechanism for evaluating and improving

quality.

• The same principle is reflected in enforcing a number of regulations

• At a global level, groups such as the Organisation for Economic Co-operation and Development (OECD) publish regulatory guidance which all member states are required to implement in their local regulatory frameworks

• This global emphasis on quality shows that a consistent approach to audit is required in order for products to be distributed globally at the necessary level of quality.

Nature of an Audit

• Must be a basis, or specific requirements, for an audit and a systematic

method for gathering facts or evidence.

• An auditor compares the evidence with the requirements and comes up

with observations, which can be either positive or negative.

• Up to this point, the audit process is similar to inspection. But an audit

entails much more.

Nature of an Audit

• The auditor analyses his or her observations for patterns — also called

findings — in order to draw conclusions.

• The auditor then presents the observations, findings and conclusions in

a report to all parties involved.

Compliance Audit:

• This type of audit is about conformance to rules and regulations.

• The goal is to see if activities, processes and systems meet requirements.

• The result is usually black or white — the product, process or system

being audited either passes or fails.

Compliance Audit:

• For example, when a regulatory authority conducts a post-approval

cGMP inspection at a company, it is essentially conducting a compliance

audit.

• The companies being audited are primarily concerned about passing the

audit with flying colours.

Performance Audit:

• Looks at three things:

• compliance to the rules,

• effectiveness of those rules in use and

• suitability of those rules for achieving an organization’s goals.

Performance Audit:

• A performance audit may be conducted not only to make sure that the

plant’s quality system will pass an ISO conformity assessment, but

perhaps to see how the system’s efficiency can be improved in order to

boost production and profitability.

• Usually conducted internally to look at a company’s business results, or

it can be applied to a supplier to help a company decide whether to sign

or renew a contract with the supplier.

First-Party Audit:

• Audited and auditor all belong to the same organization.

• An ISO-certified supplier may also conduct a first-party assessment to

make a self-declaration of its conformance with specific ISO standards.

also known as an internal audit or self-audit

Second-Party Audit:

• A second-party audit refers to a customer conducting an audit on a

supplier or contractor.

• For example, a company that contracted a laboratory to do testing may

conduct a second-party audit to make sure that the lab meets

requirements and to be able to demonstrate to regulatory investigators

that the company is compliant.

Third-Party Audit:

• Neither customer nor supplier conducts this type of audit.

• A regulatory agency or an independent body performs a third-party audit for the purpose of compliance, certification or registration.

• An example would be an FDA investigator conducting a cGMP inspection at a company.

• ISO conformity assessments are not carried out by ISO itself, but by private-sector third parties or regulatory bodies in countries where ISO standards have been incorporated into law.

Challenges

• Poor Communication and Scheduling

• Often, there is no single point of contact responsible for scheduling audit-

related tasks and thus important deadlines can be overlooked

• Inefficiency

• If there are several auditors working as a team using large checklists, generating

extensive paperwork and conducting multiple audits under tight deadlines, the

process can become inefficient and lead to serious problems.

• Poor Tracking

• Each audit typically results in numerous findings and related corrective and preventive actions (CAPA) that all need to be addressed and managed. Tracking these findings and their related documents, evaluating risks, verifying results and ensuring proper closure can mean a lot of legwork and combing through excessive paperwork, both of which can delay CAPA completion.

• Lack of Oversight

• Without an effective reporting tool, managers are unable to see the big picture that audit findings may reveal.

Checklists

• There’s nothing wrong with using a good checklist, as long as you remember that

there will always be something wrong with your checklist.

• It is simply not possible to develop a checklist that will get to the core of every

problem, that will cover every scenario, or that will ever be any reasonable substitute

for that all important question: “Why?”

Checklists

• Useful auditing tool, especially when you have a lot of processes and

information to review, a large facility filled with equipment to tour, and

a lot of people to talk to in a short space of time.

• Items are likely to be aligned with regulations or a specific audit report

format, or both. This makes verifying compliance and preparing the

final deliverable easier and more systematic.

Checklists

• Checklists jog your memory, so you don’t forget an important detail to

verify.

• Checklists can promote consistency across your auditing program so

different auditors will follow similar procedures when qualifying

vendors and conducting QA audits.

Open ended questions

What was asked

• Do you have a Traceability Matrix?

• Is the system validated?

• Do you have screen shots for your

testing?

What should have been asked

• How do you know you’ve tested everything?

• How do you know the system works and is under control?

• What evidence do you have of actual results?”

Using a Checklist Is No Substitute for Critical

Thinking

• Continually weigh the importance of what is heard and observed

• Is this important enough to pursue?

• Might this line of questioning lead to a critical finding?

• Was that explanation reasonable?

• What is that black stuff ?

Checklists are not necessarily our friends

• Tend to pave over important distinctions.

• Relying too heavily on checklists creates a danger of developing a sort

of “robot” mentality in which everything has equal weight, and is either

one way or the other:

• Yes/no,

• High/low,

• Compliant/noncompliant

Best Practices 1

• Keep your checklists current.

• Necessary to update checklists to reflect new regulations and guidance's, recent

citations and Warning Letters, and other current events.

• Make revisions to accommodate advancements in technology or changes to

standard industry practices

• Add entries designed to catch issues you found during audits that are not

covered by your current checklist.

• Subtract questions when you can, are there any questions you can stop asking?

Best Practices 2

• Second, be prepared with follow-up questions designed to bring about

productive dialog:

• Can you walk me through that?

• Can you demonstrate that? (These questions are particularly useful when the

auditor is unsure whether something is actually a finding.)

• Can you show me the documentation which supports that?

• What proof do you have that it happened as you describe?

• Why?

A cautionary tale - (Don’t be) The Robot Chef

• “Mommy, why do you cut the ends off your pot roast before you cook it?”

• “Because my mother did; I don’t know why.”

• “Gramma, why do you cut the ends off your pot roast before you cook it?”

• “Because my mother did; I don’t know why.”

• “Nonna, why do you cut the ends off your pot roast before you cook it?”

• “So it fits in my small roasting pan.”

• Moral: Always ask why. You’ll get more pot roast!