Best Practices for Running a

Hyperfunctional Psychology

Laboratory

Greg J. Siegle, Ph.D.

University of Pittsburgh

School of Medicine

Presented work supported by MH082998

These slides available at

http://www.pitt.edu/~gsiegle/SiegleLaboratoryBestPracticesColloquium.pdf

Why bother? • You and others can trust your data

– It’s easy to know when you step into a best-practices lab

– Some researchers get a reputation as “careful”

• Increase replicability

• Decrease debacles

– Example from my lab:

• The chilling chiller incident

– Example from the current fMRI world

Stuff we’ll discuss

• Study setup

• Data collection

• Storing data

• Analysis

• Managing the lab

Study Setup

Pre-emptive strike:

Clinical Operations Manual

INCLUDING template

documents • Common elements

– Study Responsibility Log – who does what when

– Study worksheet – stuff which has to happen and when, e.g., calibrations, audits

– Assessment Schedule

– Assessment Grid

– Procedural Checklists

– Regulatory Binder Template From http://www.uth.tmc.edu/ctrc/studymanagement.html

Regulatory Binders & Lab documents • Basic clinical trial model

– 2 folders per patient – 1 for identifiable info, 1 for all study documents. + master list.

• Excellent list of lab documents – http://www.uth.tmc.edu/ctrc/regulatory.html

– Binders/Folders for • Protocol and amendments

• Data – Subject Logs and Lists

– Patient Data – 1 per participant

– Contact Logs and monitoring

• Reporting – Corrospondence with outside organizations (e.g., FDA)

– IRB Documents

– Case Report Blank Forms

– Adverse events

• People – Investigator Information

– Team Information

• Lab information – Lab certifications, etc.

– Equipment

– Investigational product (e.g., drug) info

• Meeting documents – Study meetings

– Study reports

• Publications

Study management database

Stuff to include in addition to data

• Subject information – Screening/Enrollment log

– Visit Schedule Log

• Tracking/Reporting information – Adverse Event Log

– Protocol Deviation Log

– Data Cleaning log

• Accountability logs – Device calibrations and accountabilities

• Note: SEPARATE database for Master subject log

Quality management plan

• what

will you

check,

how

will you

check

it?

Data collection Data processing

Quality assurance guidelines Illustrations of procedures

What bad data looks like

Create folders

• Study folders: at least – data

• pupil

• heart

• behav

• ….

– analysis • matlab

• spss

– documents

– publications

– regulatory

– software

Folder contents (from Dr. Nicole Prause) • Data

– Raw, important processed stages, data processing scripts such as .m file backup, compiled data, final data

– The data folder should contain enough information to quickly reconstruct important phases of data processing without storing too many large files on the computer indefinitely.

– Every data folder should include is a "notes.txt" file, where you note abnormalities for particular subjects and files to enable quick reconstruction of data sets. For example, if a person becomes ill and withdraws from the study, it will be much easier to find this noted in a single file than to start searching to understand why the last two test conditions are missing to make decisions about data inclusion/exclusion.

• Institutional Review Board Compliance – Submissions, revisions, letters of approval, up-to-date informed consent

• Scripts – Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size

reasonable

• Publication – Poster presentations, papers being prepared, final drafts of accepted/published

papers

Select protocols

carefully • Stay as close as possible to

industry standards when possible (deviating as necessary…) – E.g., the Society for

Psychophysiological Research has published standards for EEG, ERP, Startle, Heart rate, HRV, EMG, disease transmission… • http://www.sprweb.org/journa

l/index.cfm

– Internet questionnaires: Skitka • www.uvm.edu/~pdodds/files/

papers/others/2006/skitka2006a.pdf

– ASTM (standards body) • www.astm.org

Do an in-house ethics review

Data collection

Procedural checklists & records

• Every detail is golden: Have checklists and how-to guides

• Check the checklists – “All records shall be prepared, dated, and

signed (full signature, hand written) by one person and independently checked, dated, and signed by a 2nd person” (GMP (Good Manufacturing practices) 211.186)

• Electronic checklists? – Possible

• “Electronic records may be considered trustworthy and reliable and be used in leiu of paper records provided that the electronic records have proper secuirty controls” (21 CFR Part 11 Subpart A Sec 11.1)

• “Ensure authenticity & integrity of electronic records such that the person responsible for the electronic record cannot readily repudiate the record as not genuine” (21 CFR Part 11 Subpart B Sec 11.10)

• Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a))

– But I don’t recommend it yet!

Trouble shooting guides

Guideline for ERP data by Cecile Ladouceur and Naho Ichikawa

AFTER data collection

• Data cleaning

• Signing off

Video – more is better

• Essential for clinical interviews to at

least get audio. Video is better.

• Note: Need IRB Approval

Task design

• Validation

– Check timing / event logging

• w/ fMRI we test at the scanner 1x phantom + 1x pilot before any protocol

– Check single subjects

• Write analysis scripts for single subjects BEFORE your first real subject

• Be a subject for your own protocols

• Test everything completely BEFORE your first pilot subject.

• Test everything completely BEFORE your first real subject.

Psychophys lab setup

• Neat reproducable lab setups – Diagram in your Ops Manual

to show how to do stuff exactly the same every time

– As many procedural diagrams as might be useful

• Care about disease transmission – Bloodborne Pathogen control:

• Gloves – as much as possible

• Don’t abraid the skin more than you need to

• Disposable electrodes when possible

• Disinfect – CIDEX if you have ventillation

– Control III + Suave shampoo if you don’t

• Wear a labcoat – that’s actually what they’re for

Dr. Nicole Prause’s lab setup

http://www.span-lab.com/Assets/images/photos/EEGprep.JPG

Checking stuff works before data

collection

• Protocols before your protocols

– Check all communications between computers, peripherals, and data collection devices

– Make sure your stimuli show

• Have this in your checklists

• We have eprime routines to test

– getting scanner trigger,

– eye-tracker events

– mouse/button pushes

Storing Data

Data Security & Integrity

• Whitebox standards: – Keep original data in unalterable form

– Have 2nd copy for any necessary changes (e.g., remove a few trials, concatenate runs…)

– Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a))

• Security – 21 CFR part 11:

• Double password protection

• Standards – They exist for most things: http://www.astm.org/

• IRB – E.g., consent forms separate from data

Databases

• Huge science - http://c2.com/cgi/wiki?DatabaseBestPractices

• E.g., – Have primary keys

– Don’t change schemas

– Consistent long descriptive column names across tables

– Try things first in a local database

– Good rule of thumb: 20 columns per table – more is weird design

• Lab standards – Ids are in columns called “id”

– All tables have id

• 21 CFR Part 11 – Keep an audit history of date created and by who, and dates

changed/updated

Backups

• Ideally

– Daily data backups

– Weekly incremental computer backups

– Monthly full backups

• Keep a set of backups in a secure place outside

your lab

Analysis & Quality Control

Documentation • Document everything

• Lab notebooks are essential – Extreme: Open Lab Notebook

• http://en.wikipedia.org/wiki/Open_Notebook_Science

• All work posted immediately to the public eye

• Good tool: http://openwetware.org/wiki/Main_Page

– Commercial approaches • Big list at:

http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670

– My approach: Powerpoints per study • Greg’s Journal template – on the

PICAN server – \\oacres3\rcn\pican\docs\gjsjourna

l.pot

– Sharepoint blog?

– Database page for all changes with name, date, change description

• Analyses should be reproducible – I like 1 matlab or SPSS file with

all commands that produce all analyses for a given study.

Reasons for using ELNs/

virtual workspaces

• 1. They are an efficient way of managing large projects, multiple

projects and multi-institution projects.

• 2. Provenance ensures that any accusation of fraud can easily be addressed.

• 3. Addresses the problem of missing information due to turnover in lab personnel (and students).

• 4. Can access research results from anywhere and therefore keep up with the ongoing work in the lab while traveling.

• 5. These systems are already being used in industry, therefore are studentsneed to be acquainted with them to be employable.

• 6. Meets requirements of granting agency mandates for data managment plans.

• 7. Facilitates depositing data into data repositories for reuse and repurposing.

http://campusguides.lib.utah.edu/content.php?pid=126157&sid=2131670

Example journal page

Beyond Powerpoint

Lab Bench People layer

http://campusguides.lib.utah.edu/content.php?p

id=126157&sid=2131670

Example commercial solution:

(Not endorsed just summarized) • From labarchives.com

– Intuitive Electronic Lab Notebook (ELN) organizes your laboratory data

– Preserve all your data securely, including all versions of all files

– Share information within your laboratory

– Keep abreast of developments in your lab even when traveling

– Collaborate with investigators by sharing selected data from your Electronic Laboratory Notebook

– Publish selected data to specific individuals or the public

– Protect your intellectual property

– Runs on all platforms, including Windows, Mac, Linux, iPad and Android devices Special classroom version of our Electronic Lab Notebook also available

Sample all-figures-in-paper script

%% associations of power change with change in other things within and between groups

ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));

cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));

tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999));

cct_tau=[cct; tau];

fprintf('----------------------------------\n');

fprintf('CCT r(power_on change, rumination change)\n');

st.r_powerOnChg_rsqchg_CCT=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999);

figure(7); clf;

regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct)));

xlabel('Trial Frequency Power Post CCT - Pre CCT');

ylabel('Rumination (RSQ) Post CCT - Pre CCT');

figure(8); clf;

regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau)));

xlabel('Trial Frequency Power Post TAU - Pre TAU');

ylabel('Rumination (RSQ) Post TAU - Pre TAU');

figure(9); clf;

focindfromcct_change=-9.94-151.94.*poweronchg+109.13.*poweroffchg;

regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct)));

xlabel('Unfocus Index Post CCT - Pre CCT');

ylabel('Rumination (RSQ) Post CCT - Pre CCT');

Use best practices for

preprocessing data

• Again with the Psychophysiology guidelines • http://www.sprweb.org/journal/index.cfm

• Visual inspection of artifacts

• When are artifacts ok to let in to data?

– How much should we say we’re letting in?

• Contingency planning

– What if you change preprocessing midway through?

• I think you should reprocess everything

– What if you change preprocessing after-the-fact?

• Depending on how serious, note it.

Quality control

• Diagnosis and Clinical

dispositions:

– Case conferences

• Reliability on

ANYTHING subjective

• Double data entry

• See your “Research

Methods” textbook…

Check your data early and often

• Quality check psychophys data that day and

fMRI data within a week (while it’s on the

servers)

• Single subject analyses

• Group analyses with N=5

Maintaining a lab

Calibrations

• Regular – monthly calibrations of all

instruments

– Currently done for pupilometer

– Other stuff?

• MR center and BIRC have done calibrations,

e.g., stability checks regularly. We don’t

request them. But we should for our own

documentation.

Security

• Double-locked file cabinets

• Password protection for computers, files, etc.

• Note: 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records

– has security standards for data

– audits, system validations, audit trails, electronic signatures, and documentation

– http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=11

Audits

• Every 6 months, all data within that 6 months

• Quality management help at: – http://www.uthouston.edu/CT

RC/trial_conduct/quality-management.htm

– There are chart audit tools

– Regulatory file review tools

• Every year – full audit – should be easy

The human thing

• Laboratory mentality is important. Attend to it.

Anecdotal evidence suggests happy inspired

labs are often more functional.

• You will likely not be in touch with the

emotional health of the lab. Have someone

who is. Make their report to you on lab health

a regular thing.

Hire for your weaknesses

• Good labs often have people who are (not all

of these at once)

– Detail oriented

– Socially attuned

– Tech savvy

– Inspired

Sources

• 21 CFR (Code of Federal Regulations) Part 11 - Food and Drug Administration (FDA) guidelines on electronic records – has security standards for data

– audits, system validations, audit trails, electronic signatures, and documentation

– http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=11

– Esp. Subpart B – electronic records

• Good Manufacturing Practices (GMP)

• ASTM (standards body) – www.astm.org

– Robert L. Zimmerman Jr, 10 Best Practices for Good Laboratories. Nov Dec, 2010, November/December, Standardization News

• Clinical Trials Resource Center – http://www.uthouston.edu/CTRC/trial_conduct/