National Information Exchange Model (NIEM)
Data Architecture Subcommittee
Status and FutureSeptember 14, 2006
Kshemendra Paul
Chief Enterprise Architect
US DOJ
2
Web Site (www.niem.gov) Activity
Average hits / day• 1,842 pre-beta• 10,923 currently
Average downloads / day• 11 pre-beta• ~100 currently
Web site activity for the period from June 1, 2006 to August 31, 2006
3
Agenda
• What is NIEM?• NIEM Goals and Objectives• NIEM Conceptual Architecture• NIEM Organization and Governance• NIEM Deliverables• NIEM Pilots• NIEM Status• NIEM Web Site• DAS / NIEM Opportunities to Collaborate • Lessons Learned • Next Steps• Discussion
4
What is NIEM?
• NIEM was launched on February 28, 2005, through a partnership agreement between the U.S. Department of Justice (DOJ) and the U.S. Department of Homeland Security (DHS)
• NIEM brings Communities of Interest (COIs) together to:
– Develop (and harmonize) (reusable) data exchange components
– Leverage (NIEM and external) data exchange components to build information exchanges
– Publish and discover reusable information exchanges
– Provide public domain standards and technical tools to support the above
– Provide training, technical assistance and implementation support services for enterprise-wide information exchange
5
NIEM Use Cases
• To effectively exchange information, there must be a common semantic understanding of data among participants, and the data must be formatted in a consistent manner.
• NIEM provides reusable components and IEPDs for exchanges• Between NIEM domains (inter-domain).• Within NIEM domains (intra-domain)
(where multiple sub communities exist within a single NIEM domain, e.g. justice).
6
Built on the Global JXDM
• NIEM is deeply rooted in the GJXDM and Global Justice Community, and committed to maintaining and growing that relationship
• The Global Justice XML Data Model (Global JXDM) has
– A robust, mature framework that is rich, flexible, and extensible
– An effective governance mechanism
– Widespread recognition and use, especially among state and local agencies
• NIEM builds on the demonstrated success of the GJXDM
– NIEM 0.1 is based upon GJXDM 3.0.3
– The Global JXDM will form the justice domain of NIEM (the jxdm namespace)
– The GXSTF will continue to represent the Global JXDM within NIEM
7
Built on the Global JXDM, continued
NIEM = GJXDM + Expanded Scope + Lessons Learned
• Incremental changes to GJXDM to address specific issues and expanded scope
• A few examples:– Expanded Domains (Intel, International Trade, Emergency Mgt,
Infrastructure Protection, etc.)
– Better Modularity
– Improve Reusability - Leveraging work of others – Discovery of IEPDs and components
– Capturing Business Context
– Incorporating feedback into the model - Model Refinement through use
– Support for components from external standards
– IEPD consistency
8
• NIEM is an implementation of the cross-COI information sharing layer of the FEA DRM
• The intelligence domain within NIEM is implementing the EO 13388 standards
• The NIEM architectural goal is to allow independent governance of domain-specific standards via a “plug-in” architecture for domain data models.
NIEM, ISE and the DRM
9
NIEM Goals and Objectives
• Deliver NIEM – Publish NIEM 1.0– Ramp up and implement governance– Develop validated and vetted product roadmap
• Catalyze NIEM adoption and usage with stakeholders– State/Local/Tribal practitioners – Federal Agency information technology initiatives – Authoritative bodies that can sponsor reference information exchanges for their COI
• Support the Program Manager – Information Sharing Environment– National priority counter-terrorism information exchange standards leveraging NIEM
• Deepen partnerships with communities of interest– Update and extend DOJ / DHS MOA to other federal agencies & their state/local/tribal partners– Finalize and implement NIEM / GJXDM convergence
• Become the standard, by choice, for government information exchange– Follow information exchange requirements from current to future stakeholders– For example, fusion centers as an information exchange nexus
10
Fit With Federal Enterprise Architecture
Policy
Performance
Business
Data
Services
Technology
Scopeof
FEA
Information ExchangeArchitectural Profile
• Business – standard information exchanges– Focus on reuse and modification– Authoritative vs. non-authoritative– Discover / register exchanges via repository
• Data – common vocabulary for building information exchanges
– Focus on reuse– Federated, hierarchical domain structure– Optional and over-inclusive augmentation
• Performance – measure information sharing– Use/reuse of information exchanges– Use/reuse of data components (in/out of model)– Line of sight provides actionable feedback
Focus ofNIEM
InfluencedBy NIEM
11
NIEM Federated Domain Architecture
Data component - fundamental building block of NIEM. Data components are either Universal, Common, or Domain-specific.
Universal components - data components commonly understood and reused among all domains; subject to NIEM governance.
Common components - data components understood and potentially reused between two or more domains; subject to NIEM governance.
Domain-Specific - components managed by a specific Community of Interest (may be reused with COI, available for reuse by others).
12
Business-driven Approach
BRM
LoBs
Subfunctions
IEPDIEPD
IEPD
NIEMNIEM
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
IEPDIEPD
IEPD
Business processes
many-to-many
IEPD Library
Homeland Security
Border & Transport Security
Disaster Management
Emergency Response
Dispatch ServicesPerson Screening
Independent Development Consistent Classification
13
NIEM Domains
NIEM DomainsNIEM Domains
NIEM CoreNIEM Core
UniversalUniversal
CommonCommon
StructuresStructures
ZNZN
NIEM
JJ
ScScEMEM
ImImIPIPITIT
IntelIntel
NIEM Domains:• Represent a subset of components expected to be used in NIEM for inter-domain, intra-domain exchanges or both
• Are governed by a cohesive group of data stewards who are domain experts, have some level of authority over the domain and harmonize semantic conflicts
• Govern their own content under guidelines established by NIEM PMO
• Are wrapped in a namespace to facilitate governance and to serve as the primary taxonomy.
14
External Domains
NIEM DomainsNIEM Domains
NIEM CoreNIEM Core
UniversalUniversal
CommonCommon
StructuresStructures
ZNZN
NIEM
JJ
ScScEMEM
ImImIPIPITIT
IntelIntel
External Domains:• Represent a data standardization initiative outside of NIEM
• Do NOT fall under NIEM NDR or governance
• Can be leveraged to create a new NIEM domain
• May participate in NIEM in the following ways:
• Total migration (the special case of GJXDM)
• Wrap / refer (how an external standard is used)
• Translate / map (NIEM set maps to original data set)
• Extract / use (translate and move original data set)
• Adopt (becomes a candidate for a new NIEM domain)
External DomainExternal Domain
External DomainExternal Domain
15
IEPDLife
Cycle
0 Scenarios,BRM taxonomies
2 Map +model
3 Build +validate
4 Assembleto IEPD spec
1 Info exchgrequirements
5 Publish +implement
IEPD Development vs the Model
IEPD Library(virtual)
NIEM
The Model – NIEM reusablecomponents
IEPD Development• National priorities (e.g. NDEx, TWPDES)• Authoritative sources (e.g. Rap Sheet)• Local requirements
NIEMNDR
reuse /revise
add /replace
reuse
add /replace
16
IEPDIEPD
SubsetSubset
ExtensionExtension
NIEMNIEM
External 2External 2External 3External 3
External 1External 1External 0External 0
1
1
2
2
2
3
3
3
66544
External domains or standardsExternal domains or standards
1, 5 – create2, 4 – wrap / refer3 – translate / map6 – adopt
IEPDs and Data Components
17
Organization Model: External View
18
NIEMNDR
NIEMNIEM
NIEM IEPDLife Cycle
0 Scenarios,BRM Taxonomy
0 Scenarios,BRM Taxonomy
2 Map +model
2 Map +model
3 Build +validate
3 Build +validate
4 Assembleto IEPD spec4 Assemble
to IEPD spec
1 Info exchangerequirements
1 Info exchangerequirements
5 Publish +implement5 Publish +implement
Executive orders, mission statements, business requirements, business context, policies/procedures
Executive orders, mission statements, business requirements, business context, policies/procedures
ExistingIEPD artifacts
ExistingIEPD artifacts
Submission of candidate
NIEM components, component business context
Submission of candidate
NIEM components, component business context
Specific information exchange, high-level business
requirements, existing IEPDs
Specific information exchange, high-level business
requirements, existing IEPDs
IEPD artifacts to share/reuse,
metadata for discovery
IEPD artifacts to share/reuse,
metadata for discovery
Gap analysis, specs for new componentsGap analysis, specs for new components
Valid IEPD schemas, example instances, documentation, metadata
Valid IEPD schemas, example instances, documentation, metadata
Data requirements, business context,exchange model
Data requirements, business context,exchange model
IEPD for reuse library
IEPD for reuse library
NIEMNDR
Complete IEPDComplete IEPD
IEPD Lifecycle
19
NIEM IEPDLife Cycle
0 Scenarios,BRM Taxonomy
0 Scenarios,BRM Taxonomy
2 Map +model
2 Map +model
3 Build +validate
3 Build +validate
4 Assembleto IEPD spec4 Assemble
to IEPD spec
1 Info exchangerequirements
1 Info exchangerequirements
5 Publish +implement5 Publish +implement
Executive orders, mission statements, business requirements, business context, policies/procedures
Executive orders, mission statements, business requirements, business context, policies/procedures
Specific information exchange, high-level business
requirements, existing IEPDs
Specific information exchange, high-level business
requirements, existing IEPDs
Submission of candidate
NIEM components, component business context
Submission of candidate
NIEM components, component business context
IEPD artifacts to share/reuse,
metadata for discovery
IEPD artifacts to share/reuse,
metadata for discovery
ExistingIEPD artifacts
ExistingIEPD artifacts
Gap analysis, specs for new componentsGap analysis, specs for new components
Valid IEPD schemas, example instances, documentation, metadata
Valid IEPD schemas, example instances, documentation, metadata
Data requirements, business context,exchange model
Data requirements, business context,exchange model
IEPD for reuse library
IEPD for reuse library
NIEMNDR
NIEMNDR
NIEMNIEM
Complete IEPDComplete IEPD
IEPD Lifecycle
20
IEPD Life Cycle Feeds the Data Model Maturity Life Cycle
NIEM
Staging AreaNIEM Governance
NBAC / NTAC:vet, harmonize,conformance
Groups with common data interests
Early Awareness& coordination
IEPDs
New component candidates & business context
New component candidates & business context
NIEM IEPDLife Cycle
0 Scenarios,BRM taxonomies
4 Assembleto IEPD spec
1 Info exchgrequirements
5 Publish +implement
3 Build +validate
2 Map +model
Topdown
Bottomup
21
Deliverables
• Documentation• Introduction to NIEM• Concept of Operations• User Guide• NIEM Naming and Design Rules
• Standards• NEIM 1.0 • IEPD requirements specs
• Training and Technical• NIEM website• Training materials• Help desk
• Tools• Component Mapping Template• Schema Subset Generation Tool• Graphical Browser• IEPD Repository
• Governance and Processes • The structure to manage and maintain NIEM and the processes and procedures behind its operations.
22
NIEM Pilots Based on Business Scenarios
• Content Contributor Pilots– DHS Immigration and Customs
Enforcement• ICE Enterprise Data Model mapping
to NIEM
– DHS Customs and Border Protection• United Nations Trade Data Elements
Directory (UNTDED) – DHS Emergency Management
• Adding Disaster Management, Resource Message contents
– Geospatial Community of Interest• Adding Geospatial contents
– DHS Team 5 Pilot• People Screening data elements
from five DHS components
– DHS Infrastructure Protection (NADB)
• Adding infrastructure protection contents
• Early Adopter Pilots– Intelligence Community Watchlist
Message Pilot• Terrorist Watchlist Person Data
Exchange Standard (TWPDES) Working group, OCIO Watchlist and NCTC actively participating
– National Capital Region Data Exchange Hub Pilot
• Virginia, DC, and Maryland representatives leading the pilot
• In partnership with the Disaster Management eGov initiative
• Initial target: First responder resource request messages
– OCIO ITEP Pilots (kickoff in August 2006)
• Data model validation using semantic web techniques
• JIEM Tool for business stakeholder facilitation
23
NIEM Pilots Based on Business Scenarios
Analysis Data Model
Build IEPD
Promote Data
Publish & Test
Feed-back
Incident Reporting (DOJ)
Terrorist Watch List (DNI)
People Screening (DHS)
SAR (NCTC)
Cargo Screening (DHS)
Emergency Mgmt. (DHS)
Case Mgmt. (DOJ)
Infra Protection (DHS)
With Funding & Support
As Is(4Q FY06)
Interim To Be(3Q FY07)
RED = not completed or significant problems presentYELLOW = initial activities either started or upgrades underwayGREEN = Completed
Analysis Data Model
Build IEPD
Promote Data
Test IEPD
Feed-back
Incident Reporting (DOJ)
Terrorist Watch List (DNI)
People Screening (DHS)
SAR (NCTC)
Cargo Screening (DHS)
Emergency Mgmt. (DHS)
Case Mgmt. (DOJ)
Infra Protection (DHS)
**
= New to NIEM; SAR Shading Indicates Higher Risk Outcome*
24
Status
• NIEM 1.0 Beta 3 was released on September 11, 2006
• NIEM 1.0 Release Candidate end of September or early October
• Scaling up and implementing Governance structures
• Pilot projects underway
• GJXDM convergence plan promulgated
• Cross Federal alignment discussions underway
25
Web site – www.niem.gov
26
Training and Technical
27
Tools
28
Tools: Graphical Browser
29
Tools: Schema Subset Generation Tool
30
Tools: IEPD Tool
31
Tools: IEPD Tool
32
DAS / NIEM Opportunities to Collaborate
• Number one priority – engagement from practitioners– Pilot projects – analysis through to implementation
• Number two priority – feedback on products– NIEM 1.0 beta 3 / release candidate
– Tools – available now
– Introduction, Concept of Operations, & Naming and Design Rules
– Enhanced, scaled-up Memorandum of Agreement
• DAS has provided feedback on Concept of Operations– Substantial, comprehensive, and material feedback
– Follow-up in place via Byron Waltham and Paul Embley
• Data component harmonization– DAS and NIEM are looking at person object
– DAS Governance (?) and NIEM Business Architecture Committee
33
Lessons Learned
• GJXDM Usage Base and Governance Structure Enables NIEM
• Governance and policy are the critical path, not technology– They are both also really hard– Process cannot replace relationships and trust– Balance between: inside / outside the beltway; architectural / operational– We are making progress – lots more to do
• State and local involvement as first rank partners– As practitioners, stakeholders, thought leaders
• Role of industry, academic partners
• Success = adoption and use– Never forget that the best way to get this is the simplest – listen!
34
• NIEM 1.0 Release Candidate in late September / early October– Scale Up Governance Activities
• Catalyze Conversion of GJXDM IEPDs by Authoritative Bodies– ~58 IEPDs in OJP IEPD Clearinghouse (http://it.ojp.gov/iepd/)
• PM-ISE national priority target outcomes– Counter-Terrorism Information Sharing Standards
• Encourage and seed other independent adoption and use by COIs– Sponsored by Authoritative Bodies within a Communities of Interest– Driven by Practitioners with specific (inter-domain) information exchange requirements
• Evolve and Scale DOJ / DHS MOA
• Technical support– Help desk– Expand FAQ topics– Develop NIEM training materials
• Expand communications and outreach efforts– Within existing COIs– Follow adoption to new domains and COIs– Executive, legislative, and judicial branches at the State, Local, Tribal and Federal levels
Next Steps
35
Conclusion
• For more information visit the NIEM web site (http://www.niem.gov)
• Contact NIEM by email at [email protected]
• Questions and Discussion