DoD Big Data Technologies Ben Gama, Jigar Patel, Keith Sylvester, Kristan Soboleski

Client: Naval Postgraduate SchoolMentor: Arijit Das

OverviewThe project primarily focuses on Big Data Technologies. We must understand the theory behind how the Naval Tactical Cloud layers METOC (Meteorological and Oceanographic) data. •  - This METOC data stored in XML format but

be ingested and mapped to a graph database. •  - With a focus on METOC data, we must

understand the injection process, develop scripts to grab this newly mapped data, runs jobs, and display it using graphical relationships.

High-level Project Goal: Get a server configured using big data tools such as Hadoop and Accumulo. Ingest METOC data (XML) into a Accumulo Graph Database.

Background The Office of Naval Research wants to harness the power of cloud computing and bring big data fusion capabilities to the warfighting environment. The Naval Tactical Cloud is a utility computing environment from Cloudera and data analysis tools and software, including Apache Hadoop and Accumolo MapReduce, developed by Google and Content Zone for data storage. METOC data is Meteorological data currently is arranged in XML format, in real-time the data must be ingested into a graph database format.

Problem Statement/Project Objective

●  Crunch big data into Accumulo’s graph database format for soldiers to use in real-time warfare o  - Get Big Data Server running on

our servers [run open source version of Accumulo/Hadoop]

o  - Ingest METOC data into graph database

Results

●  - Accumulo successfully configured and ingesting data

●  - Successful Java program to parse through METOC data, grab information, create nodes, draw relationships, and ingest this data into Accumulo (graph database)

Solution Design

1.  METOC (Meteorological data) stored in XML format is parsed by a Java program which takes in all data stored in nodes.

2.  The Java program then creates nodes in a graph database from the data in the XML file. It also intelligently creates the relationships each node shares with one another.

3.  The new graph nodes are ingested into our Accumulo server with the newly formed relationships.

4.  The data is pushed to the Naval Tactical Cloud.

Future Work - Adjust code to draw relationships between nodes more efficiently.- Remove hardcoded instruction to allow code to work seamlessly with any graph database- Successfully implement it into Naval Tactical Cloud

AcknowledgementsArijit Das, Mentor Sathya Narayanan, Advisor Eric Tao, Director Kenneth Lontoc Naval Postgraduate SchoolUS Department of Defense

Our Approach

- Have our own servers configured with Accumulo, Hadoop, and ZooKeeper running- Use a program written in Java that is intended to ingest METOC data and implement it into a graph database for use in the Naval Tactical Cloud.