CSE 548 Advanced Computer Network Security

Document Search in MobiCloud using Hadoop Framework

Sayan Cole

Jaya Chakladar

Group No: 1

Project Goal

• Hadoop MapReduce – software framework to write applications to process huge datasets on large number of computer clusters

• Apache Hadoop Distributed File System – primary storage systems used by Hadoop applications

• Project – develop a document search algorithm in MobiCloud based on Hadoop and HDFS

Group project Description

• Install and configure Hadoop in MobiCloud• Develop user interface to enable searching

and display result• Create and update the HDFS data• Mapper function to map users to keywords• Reduction function to sort the best matches

for a search criteria• System testing• Delivery and demo

Project Description (cont.)

Tasks ResponsibleInstall and configure Hadoop in MobiCloud

Jaya & Sayan

Develop UI web application JayaSynchronize phone list with virtual machines

Sayan

Search mapper algorithm SayanSearch reduction algorithm JayaHDFS data store creation and updates

Jaya & Sayan

Testing and problem resolution Jaya & Sayan

Delivery and demo Jaya & Sayan

Software and Hardware Requirements

• Hadoop

• Database software e.g. MySQL or Apache HDFS

• 3 or 4 Android phones mapped to virtual machines in 2 different Linux boxes

Network topology

Web Application to process Requests

Hadoop Master Application

HDFS Data Store

VMVM VM

VM

Project Time Line

Technical Details

• Install and configure Hadoop in MobiCloud

• Create a UI web application– The user interface of the application will be

web based– The output will be shown back in the same

web page

• Create an application to synchronize contact list with virtual machines

Technical Details

• Develop an application to search for documents – The user requests a document search using

some keywords. The input is handled in a webpage hosted as part of the web application

– The web application hands over the request to the Hadoop Master application

– The master application accesses the contact list of the requesting user

Technical Details

– For each entry in the contact list, it creates a job to map the user to the keywords.

– The reduction function creates a sorted list of users based on their degree of match

– The result set is returned to the web application to update the web page displayed to the user.

• Creation and updating of Apache HDFS distributed file storage system

Risks and Benefits

• Novel aspects of this project:– Distributed computing in mobile clouds– Enable meaningful search in a mobile device

• Risks/challenges:– Cluster failures , debugging and solution– Handling errors e.g. a mobile is switched off or

network issues

• Potential applications & benefits:– Can be used to search for multimedia objects in

mobile environment– Access data handling capabilities in a mobile device