Upload File

parsing e-mail: lessons learned

12
CERP Symposium November 10, 2008 CERP Symposium November 10, 2008 Parsing E-Mail: Lessons Learned Parsing E-Mail: Lessons Learned

Upload: others

Post on 17-Feb-2022

1 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Parsing E-Mail: Lessons Learned

CERP SymposiumNovember 10, 2008CERP Symposium

November 10, 2008

Parsing E-Mail: Lessons LearnedParsing E-Mail: Lessons Learned

Page 2: Parsing E-Mail: Lessons Learned

Overview of Topics

Just what is email anyway?Email standards and conventionsDiversity of native email formatsCommercial tools vs. open sourceThe co-evolution of the schema and the parser

Page 3: Parsing E-Mail: Lessons Learned

What is Email?Email is what is transmitted from the sender to the receiver.It is not simply what the receiver sees.

The email client software determines what you see Multiple alternative bodies – you only see one of themChild messages may or may not appear in-line, or at allHTML rendering may differ on different machinesHeaders may contain extra useful information

We must archive all information that was transmitted and stored, not just what was visible

Page 4: Parsing E-Mail: Lessons Learned

Weak Email “Standards”RFC2822 and other standards are less standard than they seem. Email continues to evolve and standards continue to lag.Lagging standards attempt to support all preexisting conventions …

an impossible goal

without compromises that are open to interpretation.Different email client vendors interpret the standards differently. Causes mismatches between interpretations (and inevitable bugs).

Page 5: Parsing E-Mail: Lessons Learned

Variety is the Spice of EmailDozens of common email systems and hundreds of others

We have encountered mail from Eudora (multiple versions), Simeon for MacPPC, Outlook/Exchange (multiple versions), AppleMail, Lotus Notes, Groupwise, Mozilla/Firefox, Pegasus Mail, and various Internet mail services such as gmail, Hotmail, YahooMail, Juno, and AOL. Each has its peculiarities.

Non-standard date and time-zone formatsEuropean and Asian mail may contain non-

ASCII (actually, non UTF-8) charactersOlder email may have HTML in inappropriate placesTreatment of nested forwarded and other “child”

messages differs

Page 6: Parsing E-Mail: Lessons Learned

Commercial vs. Open Source

Weaknesses of Commercial SolutionsMost SARBOX solutions aim at the earliest possible legal destruction of email rather than long-term storage.The storage formats are determined by the vendor, usually with an eye to supporting their own client software and advantaging their own businessProprietary software suppliers may not even be in business 20 years hence.

Benefits of Open SourceThe software can be maintained by the archivist community at large, Storage formats can be optimized for archival needs.

Page 7: Parsing E-Mail: Lessons Learned

The Storage Format -

XML

Why not just use Native email format?Which one? How well is it documented? How long will software exist to read it? Which companies (if any) have a real commitment to stability and longevity?

Why eXtensible

Markup Language (XML)?XML is open, human readable and “self describing”A good descriptive schema supports validity checkingThere are many open source tools to create, manipulate and read XML

Page 8: Parsing E-Mail: Lessons Learned

The Importance of a Common Schema

A Schema defines how the XML tags for the various parts of an email relate to each other.

<Account>, <Folder>, <Message>, <Header>, <Body>, <Attachment>, etc.

It is the Rosetta stone that guides how raw email is converted to XML… and it defines the structure for subsequent search, display, provenance, preservation, etc.The ‘Mail-Account’

XML schema serves the purposes

of both CERP and EMCAP (thanks to David Minor of the NC State Archives)It is public, so you don’t have to reinvent the wheel

Page 9: Parsing E-Mail: Lessons Learned

Lessons Learned

Email isn't easy and standards aren't very standardChild messages can be nested deeply --

complicates parsing, the schema, and searchRecent email is reasonably well behavedOlder email can contain all sorts of problematic emailEmail from overseas may have its own problems (dates and non-ascii

characters)

Page 10: Parsing E-Mail: Lessons Learned

Email Conversion ResultsWe have converted and validated more than 70,000 in test sets to the XML Mail-Account schema

Smithsonian - 5,537 messages in 232 Mb of recent Outlook mail

99.97% successfully parsed (4 could not be parsed),Smithsonian - 20,000 messages in a 1.5 Gb Outlook account

99.975% successfully parsed (5 could not be parsed)Rockefeller Archives - 43,778 messages in 378 Mb of older eclectic mail

99.85% successfully parsed (74 unparsed, but improvement is clearly possible)

Parse speed: about a quarter gigabyte per hour on a Thinkpad

T40

Page 11: Parsing E-Mail: Lessons Learned

Assessment of the Schema

The schema has passed the test of handling the complexities found in real-

world email from institutions like RAC and SIAThe schema provides for and encourages the capture at parse time of useful metadata (the main headers for instance).

Page 12: Parsing E-Mail: Lessons Learned

Summary

100% success is an unrealistic goalSome emails are just too broken to parse without manual intervention

We can

achieve at least 99.9% success (and save the few unparsed emails for human inspection and repair)This error rate is comparable to that of physical archives


Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

Lessons learned from Asian tsunami disaster: Sharing knowledgeusir.salford.ac.uk/9805/1/lessons.pdf · Title Lessons learned from Asian tsunami disaster: ... Lessons learned from

Lessons Learned, Opportunities and Innovations in …pdf.usaid.gov/pdf_docs/PBAAD103.pdf · Lessons Learned, Opportunities and Innovations in Human ... Lessons Learned, Opportunities

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons Learned:

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons learned & not learned at MSU

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off

D530.1 - Lessons Learned Framework Concept€¦ · Driving Innovation in Crisis Management for European Resilience D530.1 - Lessons Learned Framework Concept Keywords: Lessons learned,

Lessons learneD

The Apollo Experience Lessons Learned for Constellation ... · PDF fileThe Apollo Experience Lessons Learned for Constellation Lunar Dust Management ... Experience Lessons Learned

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons Learned fromLessons Learned from Structural ... learned davy beicker.pdf · Lessons Learned fromLessons Learned from Structural Engggineers Learning ... reduced size of the

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …

BestPracAces*and*Lessons* Plaorm:*Physical* Learned*from ... · Parsing* Queue Typing* Queue Index* Queue UTF* Encoding* Parsing* Pipeline Merging* Pipeline Typing* Pipeline Index*

Lessons learned after 190M lessons served

LEARNED LESSONS:

A living hell - lessons learned in eight years of parsing real estate data

Successes, Challenges, & Lessons Learned...Successes, Challenges, & Lessons Learned Overview Introduction Successes Challenges Lessons Learned New Approaches Highway 70 To Phoenix