long term archiving, data retention/legal hold in ... · chromatography . data system . other...
TRANSCRIPT
©2013 Waters Corporation 1
Long Term Archiving, Data Retention/Legal Hold in NuGenesis SDMS
©2013 Waters Corporation 2
Agenda
Data Retention requirements and how they can be addressed
Long term data archival (LTA) – data formats
NuGenesis SDMS interface with LTAs
Storage Gateway interface solution
©2013 Waters Corporation 4
Typical Labinformatics Scenario
DISCOVERY DEVELOPMENT QC
ERP
LIMS B
LES
SDMS
CDS 2
Instrument
LIMS A
ELN B
SDMS
CDS 1
Instrument
SDMS
MS Software
Instrument
ELN A
©2013 Waters Corporation 6
Sampling Analytical Data Acquisition
Data Processing Results
File Data Printed Results
NuGenesis SDMS Repository
Development & QC
Long Term Archive
LIMS
©2013 Waters Corporation 7
Regulatory Requirements
Examples – GxP demands archival of records for extended time
– Reprocessibility
– What are your retention requirements?
Industry Segment, Regulator and Type of Record
Typical Retention Period
Pharma: Good Laboratory Practices; records related to a new drug application (NDAs)
Date of submission plus 5 years
Health Care: medical records Life of patient plus “n” years
Drug/device study records Marketing application plus 2-3 years
Government records 20-50 years, or permanent
Copyright records (all organizations) Life of copyright = 95 years or as business needs dictate
Patent records and supporting data Application plus 17 years
©2013 Waters Corporation 8
Lifespans
Software lifetime: approx. 9 years*
Hardware depreciation: 3-5 years
Comparison of reliability over three years
*Soft
war
e lif
etim
e an
d its
evo
lution p
roce
ss o
ver
gen
erat
ions,
Tam
ai,
T., To
rim
itsu
, Y.
, Confe
rence
on S
oft
war
e M
ainte
nan
ce,
1992,
Proce
edin
gs
©2013 Waters Corporation 10
History
©2013 Waters Corporation 11
Common Strategies
Keep old hardware – Hardware failures
– Lack of spare parts/maintenance
– Lack of knowledge
Virtualize old software – Lack of knowledge
– No updates
Convert data – Either updated proprietary formats/databases
– Or platform independent standards
– Processing Algorithms...
– Original raw data must still be retained!
©2013 Waters Corporation 12
Current Data Standards
JCAMP 1988
today
Andi/AIA
1992/93 2001
AnIML
2013/14
©2013 Waters Corporation 13
Migrate to new standards
JCAMP ANDI AnIML
JCAMP ANDI
+ + /
today tomorrow
©2013 Waters Corporation 14
PDF/A
ISO standard
100% self-contained
Based on Adobe PDF Reference 1.7
Specific features either mandatory, recommended, restricted, or prohibited
Several parts: – Part-1 (PDF/A-1) approved September 28, 2005
– Part-2 (PDF/A-2) approved June 20, 2011
o PDF/A-1 remains in effect
o PDF/A-1-conformant files also conform to PDF/A-2
– Part-3 (PDF/A-3) approved October 18, 2012
©2013 Waters Corporation 15
Long Term Archive Repository (DB + Managed Store)
Keep data secure
Observe retention periods – Protect against premature deletion
Make available on request
Repository vs. Archive Long Term Archiving
Data collection
Print & file data
Metadata extraction
Share data in community
Simple retrieval
Data re-use
Header Data
Analytical Data,
Binaries
LT Archive Repository Database
Archive
Record Data
Restore
Search
©2013 Waters Corporation 16
Data Retention
Data Retention can be enabled on a project – Cannot be turned off
– No manual deletion or moving possible through VISION or SDK
– Governed by Retention Policies
– DR-enabled projects cannot be deleted!
©2013 Waters Corporation 17
Data Retention Policies
1. Define Candidates – User needs “Create Data Retention Policy” and/or “Edit Data
Retention Policy” privilege to create/edit policies
– Candidateship defined by
o Filter criteria
o Search criteria
– Dynamically takes care of legacy and new records
– May overlap
TIP!
o Avoid cookie-cutter criteria
o If you have multiple record types in a project, make sure you carefully define the selection criteria
©2013 Waters Corporation 18
Data Retention Policies
2. Define Retention Period – Retention defined by
o Fixed future date
o Days/months/years after date-based tag field
• Date format must be yyyy-mm-dd!
o Event (future event + x dd/mm/yyyy)
• Events are triggered manually
NOTE:
o Multiple policies possible on the same project
• Longest period wins
©2013 Waters Corporation 20
Data Retention Policies
©2013 Waters Corporation 21
Data Retention Policies
3. Define Deletion – Automatic deletion
o Configure Data Retention Service
– Manual deletion
o With or without RFC/reauthentication (project setting)
o User needs “Delete Reports” Privilege
– Fixed Date = Last Day of Retention Period
o Deletion happens 1 day after
– Email (per policy)
o Auto Delete: Deleted Records
o Manual Delete: Candidate Count
©2013 Waters Corporation 22
Data Retention Policies
Manual or automatic deletion after expiration – Deletion of metadata, binary data, and record audit trail
– From DB, Managed Storage, and OSM (HD, Raid, HSM)
– The act of deletion is logged in the system audit trail
Creation/Editing of policies is audit trailed
Changes to retention time will pop up a warning
©2013 Waters Corporation 23
Manual Deletion
©2013 Waters Corporation 24
DRS
DR-enabled Projects
(for user)
Selected Projects will be scanned
©2013 Waters Corporation 25
Legal Hold
Will prevent:
– Changes to the record
– Changes to the metadata
– Deletion
– Data Retention Management
Allowed:
– OSM
– Data Management
Privilege controlled
Multiple legal holds possible
Across Projects
©2013 Waters Corporation 26
Legal Hold
©2013 Waters Corporation 27
Storage Gateway for OSM
Waters Storage Gateway – paid option (new for NuGenesis 8)
One interface to multiple storage solutions – Tested and supported for SDMS: EMC² Centera
Can be configured as an OSM Device – For Data Retention enabled projects ONLY
– SDMS Data Retention is supported
Database
OSM
Storage Gateway
©2013 Waters Corporation 28
Waters Storage Gateway Integration
SDMS OSM WSG
send data send data Store + lock data
success
status
check status
status
SDMS OSM WSG
delete data delete data Unlock +
remove data
success
status
check status
status
OSM to Centera:
Data retirement:
EMC² Centera
EMC² Centera
©2013 Waters Corporation 29
Waters Storage Gateway
©2013 Waters Corporation 30
Why a Gateway Application?
Chromatography Data System
Other Analytical Systems
NMR
Office Apps
LTArchive 1
ELN
LIMS
Repository 1
Repository 2 LTArchive 2
©2013 Waters Corporation 31
Why a Gateway Application?
Chromatography Data System
Other Analytical Systems
NMR
Office Apps
LTArchive 1
LTArchive 2
ELN
LIMS
Repository 1
Repository 2
New LIMS
Individual data
©2013 Waters Corporation 32
Origin System
Storage Gateway
LTA Brand 1
Repository 1
LTA Brand 3
Storage Gateway
Ori
gin
Sys
tem
In
terf
ace
(API
) Archive Adapter 1
Archive Adapter 2
Archive Adapter 3 LTA LTA Brand 2
LIMS
Repository 2
NuGenesis 8 SDMS
AWM
©2013 Waters Corporation 33
Architecture
Storage Gateway
SG Server and
Actback
Archiving Backend
SG Database
Web Client Application
Administration & Queries
©2013 Waters Corporation 34
Archive Adapters
Are specific to an LTA Architecture – Are separate add-on modules
– Use an LTA programming interface where available
– Do not necessarily support the full LTA function set
Currently 3 available – File system
– IBM Tivoli
– EMC Centera (supported by SDMS)
Archive Adapter
LTA
SG Server
LTA API
©2013 Waters Corporation 35
Supported Archiving Functions
As of Storage Gateway 2.0 SP3:
Store data – using pre-defined Storage Classes/Pools
– with individual retention times
– with event-based remaining retention time
Restore data
Lock data against deletion – Separate from retention times or Storage Classes
Delete data – If retention period is over
– If not locked
©2013 Waters Corporation 36
Functions
System Administration – All via web client
– Mapping Origin System to Archive Setup
– Mapping OS Archive category names to Archive storage categories
User management
Data audit trail – Audit trail search and review
Storage Gateway metadata configuration – SG stores metadata for each archived set of data
Fallback data search and retrieval
©2013 Waters Corporation 37
Origin System
Storage & Retrieval
LTA 1
Repository 1
Storage Gateway
LTA 2
NuGenesis 8 SDMS
AWM
Database
SDMS -> LTA 1 ID 123abc
LTA def345
SDMS ID 123abc LTA def345 Meta data Audit trail
©2013 Waters Corporation 38
Origin System
Storage & Retrieval
LTA 1
Repository 1
Storage Gateway
LTA 2
Database SDMS ID 123abc LTA def45 Meta data Audit trail
Web Client Application
©2013 Waters Corporation 39