eml alchemy nist voting data formats workshop gaithersburg october, 2009 david rr webber technical...
TRANSCRIPT
EML Alchemy
NIST Voting Data Formats Workshop
GaithersburgOctober, 2009
David RR WebberTechnical EditorOASIS Election Services TCE-mail: [email protected]
slide 2
Overview
Change is constant (even in the conservative voting world) - how do we make trusted, cost effective information technology solutions, that improve democratic processes?
How do we leverage and align standards based mechanisms and tools within our voting processes to make them more consistent, predictable, interoperable and verifiable?
How does this fit into modern software engineering techniques, familiar best practices and popular development environments?
Provide voting management staff the ability to vet and operate standard open processes without constant hand holding from technical staff.
slide 3
What makes something interoperable?
Clearly defined open processes and exchanges Based on open standards and approach; not proprietary
Adaptive approach with consistent methods This allows tailoring to specific local profiles and use patterns
When requirements change can be adjusted consistently
Fault tolerant and not brittle Ability to ignore non-critical interchange items and especially
not to fail for trivial technical reasons not related to actual information details
Leveraging XML capabilities to make self-describing transactions and exchanges possible
Support new uses without extensive reprogramming Usage patterns set via external configuration allowing broad but
controlled uses
slide 4
Agenda
EML Technical Overview Introduction to transaction sets Typical simple illustrative use path OASIS interoperability event
Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Massachusetts State Elections, IRV and EML
Rapid EML localization tools Using OASIS CAM toolkit and EML templates Quick look – EML templates and NIEM IEPD process The Importance of Domain Dictionaries
Summary / Review
EML Technical Overview
Introduction to XML transaction setsTypical simple illustrative use pathOASIS interoperability event
slide 6
The Voting Process Overview
slide 7
Why XML?
XML is multi-faceted – it can represent: data (information exchanges)
rules
workflows and processes
forms and reports
configuration and control
semantics (metadata)
documents and contracts
multimedia and presentations
XML pervasive across software systems
slide 8
EML V6.0 What’s in the box?
OASIS EML transactions – XSD Schemas
EML XML dictionary – Excel spreadsheet
Formal specification document
EML templates – OASIS CAM templates*
We will discuss EML transactions first…
* CAM = Content Assembly Mechanism, OASIS standard template specification
slide 9
Category Overview of EML transactions
Pre-election Election details (100) Candidates (200) Voters (300) Ballot Options (600)
Election Voting (400)
Post election Results (500) Analysis (500) Audit (350, 400)
One or more series of XML schemas are provided to support each general process area.
slide 10
EML V6.0 Election Transactions FamilyStandard record structures to hold election information, ballot details, voting records and services
Communicating specific result details on candidates and electionsEML 520 – results reporting
Results of election contest(s) and countsEML 510 - count
Documents access to voting records and reasonEML 480 – audit log
Group of votes being transferred for countingEML 460 – votes group
Actual record of vote castEML 440 – cast vote
Used for voter authentication during a voting processEML 420 – voter authentication
Used to register voters for an electionEML 310 – voter registration
Details of actual voters for an electionEML 330 – voter election list
Notification to voter of an election, their eligibility and how to voteEML 340 – polling information
Describes the actual ballot to be used for an electionEML 410 – ballot
Contest and candidates detailsEML 230 – candidate list
Used to nominate candidates or parties, consenting or withdrawingEML 210 – candidate nomination
Information about an election or set of elections.EML 110 – election event
PurposeDocument Name
EML 150 – election boundaries Boundary geo-districting information and polling places.
EML 530 – statistics reporting Statistics relating to the election results and votes cast
slide 11
EML V6.0 – Supporting / Control transactionsUsed to manage security, audit control, access, system, setup and event tracking.
List of accepted option proposals for referendum ballot formEML 630 – list of proposal options
Confirm receipt of option proposal nomination detailsEML 620 – options receipt confirm
Access and audit control to vote records, reason, eventEML 445 – retrieve vote
Acknowledges receipt of vote from voting channelEML 450 – vote confirmation
Voting token digital certificate channel use log EML 470 – voting token log
Nomination of option proposals for referendumEML 610 – options nomination
Voting channel device selection and useEML 360 – channel in/out detail
General system level of messages for operational controlEML 350 – series of messages
Confirm receipt and processing of EML transaction.EML 130 – response
PurposeDocument Name
EML 220 – response Candidate nomination response and confirmation.
slide 12
Example - EML 230 Candidate List<EML Id="230" SchemaVersion="6.0">
<TransactionId>OK-2007-09-1</TransactionId> <CandidateList> <Election> <ElectionIdentifier Id=“Oklahoma 2007" /> <Contest> <ContestIdentifier Id="State Governor" /> <Candidate>Brad Henry</Candidate> <Candidate>John Wayne</Candidate> <Candidate>Bill Okapi</Candidate> <Candidate>Jane Smith</Candidate> </Contest></Election></CandidateList></EML>
Example of an EML artefact in XML
slide 13
Typical EML Process Path EML 230 candidate list EML 330 voter registration EML 410 ballot layout EML 440 cast votes EML 460 vote group EML 480 audit log activity EML 510 vote counts EML 520 election resultsAlso consider EML 360 voting channel use EML 470 voting channel digital token log EML 530 election statistics reporting
Pre-Election configuration
Polling activity
Tallying and Reporting
slide 14
OASIS Demonstration – London, 2007 EML 330 voter registration records
Voter selects voting channel preference
EML 440 retained by voting channels EML 460 sent to IBM tabulator system in
Brussels Reconciled with EML 330 totals per channel EML 510 count results EML 520 voting results
Illustrates the reconciliation of the number of authorised voters with votes cast and votes counted by channel and in total.
White Paper available describing event:http://www.oasis-open.org/committees/download.php/25992/EML%20Interop%20Demo
%20Report.pdf
slide 15
System interoperability testing
Testing,Agreement /Alignment.
Validate
Templates
Report
SystemUses &Creates
Results
Test
Pass / Fail
XMLhtml
32
4
Test Cases
XML
Package
StructureRulesContextExamplesDocumentationDictionary
Publish
1
Verify Result Outcomes
XML
LocalizationContent
Hints
slide 16
Technology Considerations Summary Use of XML in software systems is
pervasive All platforms and all major developer
environments support XML Everything from FaceBook™ to Microsoft
Office™ to Oracle™ database is XML capable
The OASIS EML transaction Schemas can be directly used by developer software tools to handle the associated XML transactions
The OASIS EML templates provide support for generating additional XML artifacts to suit localization needs
Recent US related EML work
California Election Results ReportingPew/Google VIP Project Geo-districtingUse of EML in State Elections with IRV
slide 18
California Election Results Reporting
Audience – news media channels
Developed in summer of 2007
Enhanced EML 510/520
New EML 530 statistics transaction Joint effort between CA SoS staff and OASIS EML TC
slide 19
Outcomes
Extend EML logical geographic constructs to handle USA district / ward structures
Results for California 2006, 2007 and 2008 elections reported
Validated ability of EML to report US election results
Added extensible statistics reporting – same EML 530 now being used in Belgium as well
slide 20
Sample of California Election Statistics
EML 530 XML
EML 530 in web browser
slide 21
EML 150 - Pew / Google VIP Project Audience – voters District boundary,
ballot and polling place information
Mapping tool support
EML 150
slide 22
EML 150 – Lessons Learned Re-usable components from EML allow rapid
development of new transactions EML V6.0 dictionary provides core enabler EML 150 provides capabilities surpassing the
original VIP XML layouts Includes GML standard mapping capabilities Polling place location, layout, access and facilities Aligned to OASIS EDXL for emergency management support
OASIS EML technical committee able to rapidly respond to emerging needs and provide solutions
VIP team is able to transition their existing data to new EML 150 format and gain benefits of broader information access and reuse (currently each state is new project).
slide 23
Massachusetts State Elections, IRV, EML Feasibility study on use of EML to support IRV for
state-wide elections Determine whether EML would be an appropriate
way to convey partially aggregated ballots from an IRV (or other preferential ranking system) from a lower jurisdictional level up to a higher jurisdictional level where the ballots would be counted and theresult of the election would be determined
Currently changes being considered forMassachusetts law to use IRV for state-wide offices
OASIS EML TC members supporting investigation into using EML v6.0
Possible enhancements envisioned during EML v6.0 review period to precisely cover identified needs
Rapid EML Localization tools
Using OASIS CAM toolkit and EML templatesQuick look – Templates + NIEM IEPD processImportance of Domain Dictionaries
slide 25
Localization Key Components
XML Schema
2
1
3
4
DocumentationReadable information use details (HTML), EML cross-reference Spreadsheet
XML artifacts The localized XSD Schema and picklist.xml
Examples & Test CasesRealistic test XML instances for conformance and interoperability testing
Key components(aka IEPD*)
EML Schema XSD aligns voting information model, processes and domain dictionary
*IEPD – Information Exchange Package Documentation
slide 26
National Information Exchange Model
Support for NIEM approach Builds
components for IEPD (Information Exchange Package Documentation)
Best practices - template evaluation for NDR compliance
http://www.niem.gov
OASIS at NIEM training event:
http://events.oasis-open.org/home/ei-summit/2009
slide 27
NIEM and OASIS work OASIS Emergency Management TC - joint MoU
between NIEM and OASIS Development of NIEM “adaptor” for handling
OASIS XSD schema and domain dictionary definitions within NIEM Core
Integrated into NIEM 2.1 (latest release) Successful formula / election services overlaps Opportunity to follow same path for OASIS EML
localization for USA use Potential benefits: NIEM is Federal, State, Local
and Tribal facing - developers already using NIEM
Broad community resources for training/facilitation
slide 28
Localization Approach StepsEML
XMLXSDXMLXSD
VotingVoting
ExchangeExchange
ManageManage
developdevelop
11
ingestingest
22BaseCAM
Template
BaseCAM
Templatelocalizelocalize
33 RevisedCAM
Template
RevisedCAM
Template
GenerateGenerate
44 Local XMLXSD
Local XMLXSD
CompareCompare
55
EMLDictionary
EMLDictionary
Steps:1. OASIS develops standard international XSD schema
Implements the voting process, management and information exchange needs
2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template
Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed
4. Generate new localized XSD schema that conform to EML standard5. Build dictionary cross-reference of localization usage6. Publish documentation and XML examples
PublishPublish
66Documentation
+ Examples
Documentation+
Examples
slide 29
EML V6.0 provides base CAM templates* Templates come pre-packaged in the EML V6
download
* Open source Eclipse desktop tool: http://www.sourceforge.com/projects/camprocessor
slide 30
How it works in a nutshell
EML xmlEML xml
EML xsdEML xsd
LocalizationLocalization
Commonxsd
Namespace
XML Schema
ComponentsComponents CIQ xsdCIQ xsd
INGESTINGEST
Structure
Rules
Documentation
XSLT toolsprocess
entire EML XSDcollection
XSLT toolsprocess
entire EML XSDcollection
CAM = Content Assembly Mechanism
OASIS EML XSD Collection
CAM Template
slide 31
Example – Eclipse XML Structure Editor
EML template is shown in the structure navigator panel for review and editing
Template contains all the default content model and structure rules ingested from the XSD.
All annotations and documentation from XSD also ingested (show as “paperclip” symbol).
Code lists and typical content values inserted for easy visual reference.
slide 32
Using Editor to mark Exclude items
Structure Editor Tools
Select focus on item
Invoke action menu (right mouse click)
Pick action
Tip: exclude rule display is context sensitive and only available on optional items
Tip: use “Add New Rule” mode to specify different XPath for exclude (quick mode assumes “current path”)
slide 33
Extending the base EML schema In addition to excluding existing optional
components you may need to add new pieces
New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components
Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard
• Send comments and suggestions to [email protected]
slide 34
Adding New Domain Elements
Select root element node in structure; right mouse click for context menu; select “Add Namespace”; then enter prefix and URI
1
Select element node in structure; right mouse click for context menu; select “Add Child Attribute / Element”; then enter prefix and name
2
slide 35
Generating your EML use schema
Once your localization details are completed you are ready to complete the remainder of the process
This starts with generating your own new subset schema
Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation
slide 36
Generating subset use schema
Select Export CAM as XSD menu option
1
File Menu Option
Confirm the location and filename, and namespace mode.
Select ‘default’ for Mode to write a regular import set of XSD schema for each namespace
2
slide 37
Schema subset generated
Set of XSD files with filename and namespace suffix
Each namespace file is import for those specific type definitions
Reviewing XSD results in a schema visual editor tool
slide 38
Information Restriction Considerations
The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: setNumberRange(), setLength(), setLimit(),
setDateMask(), setNumberMask(), restrictValues(), lookup()
Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema
Also option to set cross field validations using XPath conditional rules
slide 39
Dictionary and Documentation Tools Dictionary Tools
Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible
Documentation Tools Web interactive layout Business review tabular layout Developer raw template view
Dictionary importance across process Discovery, reuse, consistency New development - library of components XML Tool support
slide 40
EML v6.0 spreadsheet dictionary
slide 41
Test Case Generator Feature Summary
Make both Pass / Fail testing examples
Content hinting so examples use real not fake data
Test optional item logic with: all / random / none
Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern
Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value)
Make hundreds of test cases without manual editing
Can link test case to XSD schema for structure tests
You can modify XSLT tool for own testing needs
slide 42
Run CAM Rules Check on Examples
Pick Test Case Example to VALIDATE; click Finish to run validation rules
Review validation results
Run Menu Option
1
3
2
slide 43
Localization Approach Recapitulation
EML
XMLXSDXMLXSD
VotingVoting
ExchangeExchange
ManageManage
developdevelop
11
ingestingest
22BaseCAM
Template
BaseCAM
Templatelocalizelocalize
33 RevisedCAM
Template
RevisedCAM
Template
GenerateGenerate
44 Local XMLXSD
Local XMLXSD
CompareCompare
55
EMLDictionary
EMLDictionary
Steps:1. OASIS develops standard international XSD schema
Implements the voting process, management and information exchange needs
2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template
Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed
4. Generate new localized XSD schema that conform to EML standard5. Build dictionary cross-reference of localization usage6. Publish documentation and XML examples
PublishPublish
66Documentation
+ Examples
Documentation+
Examples
Summary
Review and take-aways
slide 45
EML V6.0 Technical Highlights
Set of standard transactions for core processes in elections
Extended control, event and audit tracking transactions
Domain dictionary of common components used in EML v6.0
OASIS CAM templates allow rapid localizations
Set of IEPD artifacts generation Compatible with NIEM.gov approach
slide 46
Questions?
Localization SupportTools
Quick Feature ReviewSample ScreenshotsTutorial Links
slide 48
Localization Support Tools
Provides automated generation of need XML artifacts: Pick list development from template Schema subsets generation Dictionary cross-reference spreadsheet Use and rule documentation reports XML example instance and test cases
Quick summary screenshots here Tutorial resources:
http://oasis-open.org/committees/download.php/34489/EML%20and%20jCAM%20tutorial%20and%20primer.pdf
slide 49
Test Case Generator Feature Summary
Make both Pass / Fail testing examples
Content hinting so examples use real not fake data
Test optional item logic with: all / random / none
Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern
Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value)
Make hundreds of test cases without manual editing
Can link test case to XSD schema for structure tests
You can modify XSLT tool for own testing needs
slide 50
Test Case Generation Quick Start
File Menu OptionDefault directory to write examples into
Name to be used for the examples
How many examples to create
Repeating elements count
for Quick Test – just click “OK” to use default settings
slide 51
Test Case Results
Active links to view the generated examples
slide 52
Run CAM Rules Check on Examples
Pick Test Case Example to VALIDATE; click Finish to run validation rules
Review validation results
Run Menu Option
1
3
2
slide 53
Dictionary and Documentation Tools
Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible
Documentation Tools Web interactive layout Business review tabular layout Developer raw template view
slide 54
Dictionary Compare Tool
Select compare tool menu option
Browse and choose EML dictionary
slide 55
Drag and Drop to Excel Spreadsheet
ClickOK
slide 56
Tabular DocumentationDocumentation Menu Option
Select Format, resulting HTML opens in browser viewer
slide 57
Tabular HTML Content Details
EnhancedData typeLogic
Extended Code listhandling
Clear use pattern
XPath referencesandfunctions
Annotations
Resources
Links to sites and examples
slide 59
IEPD- Information Exchange Package Documentation
NIEM- National Information Exchange Model
XSD- XML Schema Definition
XSLT- XML style sheet language transformations
Glossary
slide 60
Simple EML 440 Example
Example using the EML 440 cast vote schema Simplified to minimum requirements Example can be downloaded here:
http://markmail.org/message/qyrpfq75mtctblyq
slide 61
www.oasis-open.org/committees/election
www.oasis-open.org/committees/cam
www.oasis-open.org/committees/emergency www.niem.gov
www.sourceforge.net/projects/camprocessor
Resources: