DataShare

Michael Fiorito, Kate Fitta NYC Mayor’s Office of the Criminal Justice Coordinator

New York City’s

Enterprise

Integration

Platform

Kris Carpenter, Jon NatarajanNYC Department of Information Technology and Telecommunications

Agenda

DataShare Overview – Progress to Date

Michael Fiorito

NYC Mayor’s Office of

the Criminal Justice Coordinator (CJC)

DataShare Overview – Progress to Date

Rewind to 2003…

Criminal justice agency CIO’s, CJC, and DoITT convened a workgroup to discuss the possibility of introducing new technology to meet their business needs.

The group produced a report that summarized their research and outlined possibilities for new technology.

The group also recommended hiring a small team of dedicated CJC and DoITT staff to coordinate the overall effort (i.e., a project team) and build a sustainable integration program.

DataShare Overview

Why not replace an existing mainframe system utilized by criminal justice agencies (DataShare 1.0) with a scalable enterprise solution?

DataShare was designed for citywide integration, but deployed initially for 17 agencies (city, state, not-for-profits).

The Public Safety Portal, which is the user-facing dimension of the City’s integration solution for criminal justice agencies, includes both administrative and business user functions.

The initial phase concluded in July 2006 with the deployment of the Public Safety Portal and a set of GJXDM exchanges.

This undertaking was a Mayoral initiative, but agencies did not have a mandate to participate

Phase 1 – Facts & Figures

12 months for requirements gathering, solution analysis, systems integration procurement and solution selection

18 months to complete implementation – 2 major deployments

17 city, state and nonprofit agencies integrated via hub-and-spoke architecture

35-40 FTEs comprised the dedicated project team – City staff and consultants combined

60+ agency-side resources brought to bear during 1st phase

Approximately $12M spent on professional services, HW, SW

100% of post prod support provided by City of New York upon Phase 1 completion. No ongoing costs.

Juvenile Case Management System Law Department Dept. of Probation Dept. of Juvenile Justice (DJJ)

Richmond County DA

Public Safety PortalProviders

Special Narcotics Prosecutor

Queens County DA

Kings County DA

Bronx County DA

Legal Aid Society

Manhattan DA

Criminal Justice Agency

Dept. of Corrections

NYPD

18B Web (Assigned Counsel)

Office Court Administrator

DataShare – Current Landscape

ReceiversPublic Safety Portal

DataShare

Data Exchanges/Services

Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal

Criminal justice data that is shared between and among participating agencies via the hub and portal

The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare

Agency Participation

Public Safety Portal Functionality and Links to Applications

Criminal Justice Integration Platform (Hub)

Phase 1 Providers• NYPD• NYC DOC• Criminal Justice Agency (CJA)• OCA

Phase 1 Receivers• NYPD• Department of Probation• Department of Correction• District Attorneys (5)• Special Narcotics Prosecutor• CJA• Law Department• Dept. of Juvenile Justice (DJJ)• OCA• Legal Aid Society• Assigned Counsel Plan• HRA• DoITT (on behalf of CJIS)

The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT

Core Hub Features & Functionality• Push functionality• Pull functionality• Query application• Notification application

Phase 1 Business Deliverables• NYPD arrest data push (legacy & GJXDM)• NYPD arrest notification (GJXDM)• CJA ROR interview push and pull (GJXDM)• OCA supreme and criminal court actions push (legacy & GJXDM)• DOC inmate information (legacy & GJXDM)• DOC release notification (GJXDM)• DOC jail status query (GJXDM)• CJA NYSID – SSN match (legacy & GJXDM)

Core Portal Features & Functionality• Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data• System administration for agencies • Reduced sign-on (RSO) for DNA HITS application• e911 application link • webCRIMS application link from OCA• e-Justice portal link from DCJS

Phase 1 – Scope and Deliverables

• Decisions about DataShare architecture needed to consider the autonomy of agencies

• DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases

• DataShare was intended to serve as a pass-through system, with minimal data stored centrally

• The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies.

• The City decided to adopt GJXDM as its transport standard for criminal justice services

Guiding Principles & Strategic Decisions

GJXDM Strategy & Approach

• Global Justice XML Data Model (GJXDM) is is intended to be a data reference model for the exchange of information within the justice and public safety communities. The Global JXDM is sponsored by the US Department of Justice (DOJ) Office of Justice Programs (OJP), with development supported by the Global Justice Information Sharing Initiative's (Global) XML Structure Task Force (GXSTF).

• City decided to implement both legacy and GJXDM versions of existing exchanges via new platform to facilitate agency transition and minimize risk

• GJXDM SME consultants worked with provider and receiver agencies to develop schemas and leverage national model as appropriate.

• GJXDM workgroup has formed among NYC stakeholder agencies to drive strategy going forward, especially collaboration with state and national initiatives; DOP, NYPD, DOC to play key leadership roles.

Portal Functions Available to Business Users

Applications Business Functions

Pull/Query Engine• User can execute distributed queries via the portal • Pull/Query engine processes the portal requests and provides

the results back to the user via the portal.

• Results can be viewed in multiple pre-defined formats.

• CJA Interview Pull

• DOC Jail Status Query

Notification Engine• Users can subscribe to various notifications via the portal.• The subscriptions are processed by the notification engine for

possible matches.• The resulting notifications will be delivered via email or via

the MQ infrastructure.

• NYPD Arrest Subscription

• DOC Release Subscription

Portal Administrative Functions

Applications Services/Functions

Administration - Allows the DoITT and

Agency Administrators to manage services. A service is any available function.

Provision pre-existing services to new receivers

Add new services upon completion of backend and adapter components.

Modify field filtering settings for pre-existing services.

•Provided Services - Add/Remove Field Level Permissions, Add New Service, Enable/Disable Services•Received Services - Review Received Services including what field level permissions receiving agencies have per exchange, group level settings for services

Operational Reports - Allows authorized

agency administrators to run operational reports. •Reports will be generated based on transactional meta-data.•Reports contain pre-defined content related to the transitions brokered by the DataShare hub.

•Volume Reports -Provide information surrounding the number of data services that flow through the system.•Service Detail Reports - Provide details regarding specific service transactions•Administrative Reports - Provide information surrounding administrative metadata and global reference data.

Public Safety Portal Services

DOC Jail Status Query – This function enables users to enter a NYSID, book and case number or a combination of first name, last name and date-of-birth via the Public Safety Portal and retrieve the incarceration status and history for the individual(s) that match the criteria. Jail status data includes information about the inmate's current incarceration status, incarceration history, warrant history and DOC housing history from 1999 - present.

CJA Interview Pull – This function enables users to enter a docket number or arrest number via the Public Safety Portal and retrieve the details of the corresponding interview the Criminal Justice Agency conducted with the defendant prior to their criminal court arraignment. Interview data includes information about the defendant's employment, residence, education and criminal history and is only available for interviews conducted from 2005 - present.

Subscriptions for NYPD Arrest Notification (Basic) - This function enables business users to enter a NYSID, name information or location information via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the individual or location specified by the criteria.

Subscriptions for NYPD Arrest Notification (Advanced) - This function enables administrators to enter criteria corresponding to up to 150 arrest fields via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the specified criteria.

Subscriptions for DOC Release Notification - This function enables users to enter a NYSID or arrest number or book & case number via the Public Safety Portal and subscribe to be notified via email when the corresponding inmate is released from or admitted to City correction custody.

Links to Applications - DNA HITS, e911, WebCRIMS

Public Safety Portal Services

Orchestrating services

Growing the community of participants

Key to Success: Ongoing partnership between DoITT, CJC and participating agencies

More to Come…

DataShare Architecture

Kris Carpenter

NYC Department of

Technology & Telecommunications (DoITT)

Information Sharing Capabilities

The DataShare & Public Safety Portal solution facilitates the sharing of information across city agencies providing business process integration for the participating agencies.

DataShare will provide the following services related to enterprise applications integration (EAI)– Business process routing (via publish/subscribe)– Data standardization– Web-based portal that can offer truly distributed business

applications – Business process auditing capabilities– Field level filtering for messages – Security

– 24/7 Monitoring

Existing Systems Installed Base

MVS VAX/VMS LINUX UNIX Microsoft FTP SMTP CICS Flat Files

Existing Systems

Oracle

Unix/Linux

File share

SQL Server

Web Services

LDAP

VAX/VMS

BEA WebLogic/Vignette

Microsoft

J2EE

MVS

DB2 DB2 ADABASE DATACOM VSAM ADMINS/VAX RDB CACHE MUMPS SQL SERVER ACCESS ORACLE EXCEL

Technical Architecture Requirements

Security – Authentication, Authorization, Encryption, LDAP

Performance – Handle 100’s of messages/second

Scalability – Able to add new data exchanges easily

Availability – 99.9% availability as ‘crime takes no holiday’

Extensibility – Expose functionality via different methods like Web services

Standards – J2EE, LDAP, GJXDM

Standards-based (J2EE, LDAP, SQL)

Enterprise Service Bus (WebSphere MQ and Broker)

Global JXDM

Public Safety Portal (Vingette & BEA WebLogic )

Involve the agencies that know the data

Involve the agencies that know the processes

Use existing City application implementation standards

Meeting The Needs – New Environment

DataShare: Message Oriented Middleware

DataShare

NYPD

DOP

DOC

OCA

CJA

Et all...

Hub and Spoke Architecture

DataShare: The Solution

The Solution consists of 6 key components

Criminal Justice End Users

Vignette Application Portal

J2EE Applications

BEA Weblogic Application Server

NotificationsWebSphere Everyplace

Access Server

WebSphereMessage Broker

(Enterprise Service Bus)

JMS-MQ

MQ

DataShare Data Base

RDBMS File Drop

Agency Adapter Platform

WebSphere MQ

iWay Adapter

MQ

RDBMS File Drop

Agency Adapter Platform

WebSphere MQ

iWay Adapter

MQ

Agency A Agency B

Mail ServerSMTP

Novell eDirectory

SSL

SSL SSL

SSL

Architecture & Components

Agency Adapter Platform Agency adapter programs and WebSphere MQ that is utilized as a gateway

between the criminal justice agencies and the enterprise service bus hosted at DoITT.

Enterprise Service Bus (Message Broker) Integration layer that provides message routing and integration business services

for citywide agencies. Novell eDirectory (DoITT’s enterprise directory server)

Enterprise directory server that stores citywide agency users and groups. It’s used to provide services such as distributed provisioning, authentication and authorization.

Database Storage of application and transactional metadata.

Public Safety Portal Web front end that provides access to web applications (such as subscriptions and

queries), user administration, and reports. Notification Services

Services provided to the Portal users, and the backend systems of criminal justice agencies, to subscribe to key transactions traversing the Enterprise Service Bus.

• Adapters key to joining systems:– Used in EAI to be non invasive to systems– Concept used in SOA as a Wrapper

• Allow existing formats to be utilized:– Suppliers convert to a standard– Receivers convert standard to legacy format for consumption

• Access supplied and/or received messages in an existing fashion:– File System– Queue– Database– Other Storage mechanism

Adapter Approaches

Enhancements Underway Currently (4-6 months)

Low hanging fruit

Federated query – giving users the ability to perform searches on NYPD, CJA, and DOC and retrieve a consolidated view of a person or event by entering a single set of criteria. The federated query capability will be greatly enhanced as more services are deployed through DataShare.

Service orchestration – when a user performs a DOC jail status query, they are given the option to initiate a DOC release or admit subscription from the jail status query output screen

Geocoding – exposing existing GIS services to the criminal justice participants via Datashare – incorporating a global address system.

Phase 2 Gameplan (12 – 18 months)

Grow the list of agency participants

We have some new agencies coming on as providers: DCJS and DOI

Some current receivers will become providers: DOP, BXDA, DANY, QCDA, LAW, DJJ, SNP, LAS

Some agencies/entities will become new receivers: ACS, HIDTA, OCFS

Integrate services deployed through DataShare into applications: DNA HITS, eArraignment

Deploy 15 – 20 new services/exchanges through Datashare

Working on delivering a host of new data exchanges with existing and new agencies

Goal is to build on what’s been deployed

Building the Highway

Highway has been built

Datashare is an enterprise system built to grow with the addition of new services and new providers and incorporate diverse environments

Now it gets really interesting

Contacts

Non-Criminal Justice Agencies

Jon Natarajan, DoITT (jnatarajan@doitt.nyc.gov)

Kris Carpenter, DoITT (kcarpenter@doitt.nyc.gov)

Criminal Justice Agencies

Michael Fiorito, Mayor’s Office (CJC) (mfiorito@cityhall.nyc.gov)

Kate Fitta, Mayor’s Office (CJC) (kfitta@cityhall.nyc.gov)