gtri_b-1 information technology and telecommunications laboratory january 2009 national information...

GTRI_B-1Information Technology and Telecommunications Laboratory

January 2009

National Information Exchange Model

Looking Forward

IJIS Industry Brief

John Wandelt

Georgia Tech Research Institute


A Brief History of GJXDM and NIEM• Sep 2002 Work begins on GJXDM• Dec 2002 GJXDM beta presented to Global• Apr 2003 GJXDM pre-release 3.0.0.0 released• Jan 2004 GJXDM 3.0 released• May 2004 GJXDM Developer Workshop - Atlanta• Nov 2004 GJXDM 3.0.2 released• Feb 2005 DoJ and DHS sign MOA to build NIEM• Jun 2005 GJXDM User Conference – Atlanta• Sep 2005 GJXDM 3.0.3• Sep 2006 GJXDM User Conference – San Diego• Oct 2006 NIEM 1.0 (based on GJXDM 3.0.3)• Jun 2007 NIEM 2.0 (merged GJXDM 3.1beta)• Aug 2007 Information Sharing Conference – Chicago• Jan 2009 You are here ... So, what’s happening ???

2

January 2009

2002

2003

2004

2005

2006

2007

2008


NIEM Fundamentals• NIEM profiles XML Schema to reduce its complexity.• NIEM extends XML Schema to compensate for its

hierarchical nature and representational constraints:• ID / IDREF• Associations and roles• Metadata on metadata• Type Augmentation• Abstract elements and substitution (groups)

• Many tradeoffs:• Interoperability vs. Flexibility• Simplicity vs. Precision• Components first vs. IEPD requirements• Flexible + adaptable vs. fixed + standard• Enumeration vs. free text• Elements vs. attributes• Referenced content vs. inline content

• Semantics is key.

3

January 2009


You Are Smarter Than A Computer!

Olny srmat poelpe can raed tihs. I cdnuolt blveiee taht I cluod aulaclty uesdnatnrd waht I was rdanieg. The phaonmneal pweor of the hmuan mnid, aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a taotl mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe. Amzanig huh? yaeh and I awlyas tghuhot slpeling was ipmorantt!

4

April 21, 2023


One Year’s Experience with NIEM 2.0 and IEPDs

• Components: ~4500 (most types & properties)

• Domains: 1 large (Justice); 6 smaller (primarily DHS)

• IEPD Clearinghouse (registry for GJXDM & NIEM): - 76 NIEM IEPDs

• NIEM IEPD Registry/Repository (niem.gov):- 58 shared NIEM IEPDs from 21 different sources

5

January 2009


NIEM Help and Issues Processing

7

April 21, 2023

Level 2

Level 3

NIEM Tools

NIEM Config Control Tool

NCCTLevel 1

Assign ticket #

NIEM KB


NIEM Focus Group

• Conducted by NCOC • 25-26 June 2008 Wash, DC• 20+ developers and practitioners representing

Federal/State government, industry, NTAC, NBAC• Identified 24 strengths for building info exchanges• Identified 51 weaknesses, issues, suggestions:

value proposition conformance toolsIEPD development governance contentbest practices architecture trainingchange mgt implementation

8

January 2009

GTRI_B-9Information Technology and Telecommunications Laboratory 9

January 2009

NIEM Leadership Conference Themes

• Domain Independence• Self Service • Ability to Scale• Quality Assurance and Metrics• Open Interfaces for Tools and Life Cycle Processes• Conformance: Clear Guidance and Tools• Reuse of IEPDs and Components• Missing Content and Domains• Timeline for Releases


January 2009

High Priority Work Packages

• Data Model Maturity Life Cycle (DMMLC)• Version Architecture• Quality Assurance and Release Strategy• Tool Architecture• Minor Release 2.1


January 2009

Data Model Maturity (DMM) Life Cycle

NIEM DomainsNIEM Domains

PSPSIntelIntel

NIEM CoreNIEM Core

JJ

EMEMImIm

UniversalUniversal

CommonCommon

StructuresStructures

IPIPITIT D2ND2N

G

CodeLists…

NIEMIEPDLife

Cycle

0

2

34

1

5

IEPDLife

Cycle

0

2

34

1

5

IEPDLife

Cycle

0

2

34

1

5

IEPD Library

……

reuse / revise

add / replace

reuse

add /replace

NIEMcomponents

IEPDs

add

New domain

D3

New domain

D3


January 2009

C1.0

ScJIm

NIEM 1.0

EMInIT IP Geo

C2.0

ScJIm

NIEM 2.0

EMInIT IP Geo

Dependencies


Example of Incoherent Schemas

13

January 2009

uses


Version Architecture Objectives

• Domains may publish updates on their own timeline. Publication does not wait for NBAC review.

• Domain updates are quickly available for use by IEPDs. Updates may be used by IEPDs without being delayed by updates, synchronization, or harmonization.

• Domain updates are incorporated into next NIEM release. • IEPD developers are provided with an updated schema set that is

coherent, for increased usability. • IEPDs have the flexibility to use NIEM components as required to

satisfy their business requirements - Even previous releases.• Provides a well defined process and schedule for domains to

input to the NBAC update, synchronization, harmonization processes for future NIEM releases.


Version Architecture Process

15

January 2009


Quality First!


January 2009

Operations (process)

Quality Assurance Layers

HarmonizationInputs

to model

Output

productsData

complete,correct,

consistent

no semantic overlap or duplication

quality release products, tools, documentation

well-definedsubmissionformat and

review process

Metrics / statsValidation checksInspection / review


Example Metrics and QA Checks

18

January 2009

Data• ISO 11179 naming• Definition quality (semantics)• Descriptive metadata• Code lists: currency, associated text property• NDR conformance

Harmonization• Identification of dependencies• Component name similarity• Duplication and semantic overlap• Namespace cohesion• Depth of type hierarchy (inheritance depth)

Operational• No. of new domains• No. of model changes• No. of impacted IEPDs• No. of extended components in IEPDs• Component usage in IEPDs


NIEM Tool Architecture


Conformance Services

20

January 2009


NIEM NDR 1.3

21

January 2009


NIEM Conformance• Three types of conformance:

• NIEM XML schemas conform to the NIEM NDR.• NIEM XML instances conform by validating to NIEM-conforming schemas

(and instance conformance rules specified by the NIEM NDR).• NIEM IEPDs conform to the NIEM IEPD Specification

(which requires their XML schemas and instances are NIEM-conforming).

• An IEPD conforms to NIEM under the following conditions:• Each XML schema adheres to the NIEM NDR for its conformance class

(i.e., subset, extension, exchange, or constraint schema rules). • Each XML sample instance adheres to the NIEM NDR for XML instances. • The IEPD itself adheres to the NIEM IEPD Specification

(including required files, packaging, metadata, etc.). • If an existing NIEM component matches IEPD business semantics,

then that component is used by the IEPD (directly or for derived components). (i.e., IEPD does not unnecessarily duplicate NIEM components)

• Each use of a NIEM component by the IEPD (directly or for derived components), is consistent with the component's structural definition and business semantics. (i.e., IEPD preserves semantic and structural consistency)

• There are subjective factors in applying these rules that require diligent consideration by the organization(s) developing the IEPD. Decisions on semantics must be made (or reviewed) by business SMEswith thorough knowledge of the exchange business processes and domain.

22

January 2009


What About Systems, Tools, and Databases?

• Systems, tools, and databases DO NOT and CANNOT conform to NIEM.• Only XML schemas, XML instances, and IEPDs can conform to NIEM. • For any other entity or artifact, NIEM conformance is undefined. • Internal names for or usage of data within a given system, tool, or database

have absolutely no impact on the determination of NIEM conformance. • Conformance is ONLY about the format of payload data encapsulated

in XML instances that validate to XML schemas that adhere to the NIEM NDR. • An XML schema that copies, maps to, or uses NIEM names or components

without importing NIEM namespaces does NOT conform. • Conforming to NIEM requires that exchange schemas reuse NIEM reference

schemas (or subsets) by xsd:import-ing NIEM schemas (or valid subsets) that define NIEM namespaces that compose NIEM releases.

• A tool, system, or database may have capabilities that specifically support:• Development of NIEM-conforming IEPDs• Implementation of NIEM-conforming IEPDs• Testing/verifying NIEM-conformance• Generation, sending/receiving, and/or processing of NIEM-conforming exchanges.

• Such tools or systems are NIEM-aware or NIEM-supporting (NOT conforming).

23

January 2009


[email protected]


NIEM Conformance Report Tabs

27

April 21, 2023

All rules NDR rules IEPD rules


Summary (Auto Checks)

28

April 21, 2023

Number of distinct

NDR rules checked, passed,

failed

IEPDMetadata

andCatalogresult

43 of 185 NDR rules

auto checked


Summary (Manual Checks)

29

April 21, 2023

Number of distinct

NDR rules not auto checked

Number of distinct

NDR rules manually verified

(by human)


NDR – All Rules (sample)

30

April 21, 2023

Each of 185 NDR rules listed once with ID number, conformance targets, description, and indication of auto/manual and pass/fail.


NDR – Schemas

31

April 21, 2023

Lists each schema file checked with indication of conformance


NDR – Rules Auto Failed

32

April 21, 2023

Filename (of schema)

Line No.in file

Rulefailed

Rationale for failure

Type of XML component

involved

Lists each failure by filename, line #, rule, rationale, type component


NDR – Rules Auto Failed (zoom)

33

April 21, 2023

Line No.in file

Rulefailed

Rationale for failure

Type of XML component

involved


IEPD – Metadata

34

April 21, 2023

Not complete

Status


IEPD – Catalog

35

April 21, 2023


Conformance Testing Assistant (ConTesA)

• Developed for N-DEx program, although geared towards LEXS in general.

• Performs XML Schema validation of LEXS instances, plus any included Structured Payloads.

• Performs business rule validations:• NIEM rules (e.g. namespaces and aliases).• LEXS rules (e.g. mandatory elements have content).• Structured Payload specific rules (defined by community

that developed Structured Payload schema).

• Provides visualization of instance.• Business rule checking available

as separate command line tool.

36


Registry Interfaces

37

January 2009


NIEM Minor Release 2.1Features

• New DHS domain: CBRN – Radiological Nuclear• New domain: Family Services• DHS Emergency Management will replace current content.• DHS Infrastructure Protection will replace current content. • Updates to DHS Screening, International Trade, Immigration • Update to Geospatial (external standard)• Updates to DoJ Justice (jxdm).• Over 700 missing component definitions will be added.• Cross-domain harmonization will be applied to ...

• areas already identified as potential problems.• obvious problems identified in new submissions.

• Resolved NCCT issues that can be applied will be integrated.

38

January 2009


NIEM Technical References / Documentation

• NIEM Releases 1.0 and 2.0• Reference schemas• Spreadsheet• Model databases (in CSV, XLS, and MDB)• URI pages at http://niem.gov/niem (components, namespaces, xsd’s)

• NIEM Introduction (an executive summary)• Concept of Operations (ConOps)• NIEM Terms and Definitions v1.0 (a glossary)• Using Intel Community Info Security Marking (ICISM) w/ NIEM• Specifications: IEPD Requirements, Wantlist xsd, Conformance• Naming and Design Rules (NDR) v1.3• NIEM Conformance• Techniques for building and extending NIEM XML components• Quality Assurance Strategy & Plan (QASP)• User Guide (Volume I) (Draft)• High-Level Version Architecture (HLVA)• High-Level Tool Architecture (HLTA)

39

January 2009

http://niem.gov/niem

40

________NIEMBridge information

systems

1st Edition

Compliments of DoJ and DHSForward byPMO, NTAC, & NBAC

Design your XML schemas to beNIEM-conforming


April 21, 2023


Stovepipe Thinking

42

April 21, 2023

Must shift thinking from stovepipe to cloud or federation concepts.


Questions

Reference: http://niem.gov

http://niem.gov/

http://niem.gov/

44

Related Projects


April 21, 2023

Universal Core 2.0 (UCore2)ObjectivesObjectives::

Information exchange framework that is Information exchange framework that is interoperable, flexible, applicable, and interoperable, flexible, applicable, and adoptable by a broad audience.adoptable by a broad audience.

Small core of reusable information objects Small core of reusable information objects universally understood and extensible. universally understood and extensible.

Information sharing across Information sharing across DoD, DNI, DoJ, and DHS. DoD, DNI, DoJ, and DHS.

ObjectivesObjectives::

Information exchange framework that is Information exchange framework that is interoperable, flexible, applicable, and interoperable, flexible, applicable, and adoptable by a broad audience.adoptable by a broad audience.

Small core of reusable information objects Small core of reusable information objects universally understood and extensible. universally understood and extensible.

Information sharing across Information sharing across DoD, DNI, DoJ, and DHS. DoD, DNI, DoJ, and DHS.

ImpactImpact::

Establishes a baseline standard for Establishes a baseline standard for exchanging information on terrorists exchanging information on terrorists and associated activities among and associated activities among U.S. Federal, State, and local agencies. U.S. Federal, State, and local agencies.

Allows flexibility to extend core objects Allows flexibility to extend core objects to satisfy specialized data requirements to satisfy specialized data requirements in local domains (especially, DoD/DNI). in local domains (especially, DoD/DNI).

Enables interoperability at a semantic Enables interoperability at a semantic level as well as some representational. level as well as some representational.

ImpactImpact::

Establishes a baseline standard for Establishes a baseline standard for exchanging information on terrorists exchanging information on terrorists and associated activities among and associated activities among U.S. Federal, State, and local agencies. U.S. Federal, State, and local agencies.

Allows flexibility to extend core objects Allows flexibility to extend core objects to satisfy specialized data requirements to satisfy specialized data requirements in local domains (especially, DoD/DNI). in local domains (especially, DoD/DNI).

Enables interoperability at a semantic Enables interoperability at a semantic level as well as some representational. level as well as some representational.

DescriptionDescription: : A joint DoD/DNI/DoJ/DHS A joint DoD/DNI/DoJ/DHS project to (1) design a small, universally project to (1) design a small, universally understood core of information objects understood core of information objects and basic relationships, (2) implement and basic relationships, (2) implement in XML Schema, and (3) employ them to in XML Schema, and (3) employ them to exchange and share information across exchange and share information across a broad base of stakeholders. a broad base of stakeholders.

DescriptionDescription: : A joint DoD/DNI/DoJ/DHS A joint DoD/DNI/DoJ/DHS project to (1) design a small, universally project to (1) design a small, universally understood core of information objects understood core of information objects and basic relationships, (2) implement and basic relationships, (2) implement in XML Schema, and (3) employ them to in XML Schema, and (3) employ them to exchange and share information across exchange and share information across a broad base of stakeholders. a broad base of stakeholders.


TECHNOLOGIES:• eXtensible Markup Language (XML)• Use of NIEM model and NDR• Interoperability• Extensibility• Federated Search• Data Publication

SPONSORS:Law Enforcement Information Sharing Program

(LEISP)Department of Justice (DoJ)

OBJECTIVES:Define and consistently describe units of information

to be sharedDefine interfaces and protocols to provide (publish)

as well as search and retrieve suchEnsure all agencies can understand a common set

of data, without limiting the total data exchanged

TASKS:Provide an extensible framework for consistent

packaging of informationDesign for multiple levels of understanding so all

agencies can process a common set of data, while allowing communities to add content for their needs without impact to other implementations

Utilize NIEM as the baseline data modelProvide tools, documentation, and technical support

to implementers

LEISP Exchange Specification (LEXS)

OneDOJ

LInX OR

ARJIS

ATF BOP DEA FBI USMS

LEXS SR

LEXS PD

Data Sources

Data ConsumerPartner System

Users Users

Partner SystemService Provider


The “IEPD Consistency” Problem

Definition:Two groups independently developing IEPDs for the same purpose may (nay, will) create incompatible IEPDs.

Result:Small-scale interoperability between coordinating partners, but not large-scale interoperability between independent community members (i.e., the ultimate promise of standards)

Mitigations:• Top-down standards: requires “center of mass”• Formal standards: long, arduous process• Consensual standards: hard to gather and

govern enough participants to “tip”

47

X


LEXS Extensibility

Cyber Incident(Sub-community

Level)

Bomb and Arson Incident

(Sub-community Level)

Incident (Community Level)

LEXS Base Level

Terrorism Bomb and Arson

Incident(Sub-community Level)

Incr

easi

ng

Lev

el o

f D

etai

l

Arrest (Community Level)

Federal Arrest(Sub-community

Level)

Digest

Structured Payloads

Incr

easi

ng

Co

mm

on

alit

y

48


April 21, 2023

CTISS Federated Registry

CTISR

DoD Community

NIEM Registry

NIEM Community

Federated RegistryQueries

CTISS Registry Portal

CTISR

Federated RegistryQueries

NIEM Registry


September 24, 2008

Global Federated Identity and Privilege Management (GFIPM) Presenter: John Wandelt, GTRI

GFIPM Project Managerit.ojp.gov/GFIPM


TECHNOLOGIES:• eXtensible Markup Language (XML)• Use of NIEM model and NDR• Use of LEISP Exchange Specification

(LEXS)• Information Sharing• Data Publication

SPONSORS:N-DEx Program Management OfficeFBI CJIS Division

OBJECTIVES:Define Information Exchange Package

Documentation (IEPD) for State, Local, Federal, and Tribal agencies to submit data to N-DEx

Support all increments of the N-DEx program: Incident/Arrest, Incarceration/Booking, and Probation/Parole

Leverage NIEM and LEXS to promote interoperability

TASKS:Model N-DEx data requirements for each incrementDevelop mappings from N-DEx data models to

NIEM and LEXS, along with extensions as necessary

Develop NIEM and LEXS-based schemas for each IEPD

Build sample instances to illustrate usageProvide feedback to NIEM PMO based on N-DEx

needs and usageProvide tools, documentation, and technical support

to implementers

National Data Exchange (N-DEx)

gtri_b-1 information technology and telecommunications laboratory january 2009 national information...

Documents