persistent systems limited - pragatie-pragati.in/download-files/persistent-epragati... · android...

33
Persistent Systems Limited 3 rd and 4 th , October 2017 [NTR Administration Block, 3 rd Floor, Room No. – 3 ] GoAP ePragati - Hackathon Solution

Upload: others

Post on 22-May-2020

11 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Persistent Systems Limited

3rd and 4th, October 2017 [NTR Administration Block, 3rd Floor, Room No. – 3 ]

GoAP ePragati - Hackathon Solution

Page 2: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

2© 2017 Persistent Systems - Confidential

Hackathon Team

Understanding Problem Statement

Hackathon Engineering Process Planning – Agile SAFe

Milestone-1

Requirement Analysis

Data Model, Dimensions

Technical Architecture and Design (JADR)

Identify Best Practices

Citizen Centricity – Identify 1st Use Case

Milestone-2

Requirement Checklist Creation

Kanban Boarding

Sample Screenshots

Citizen Centricity – Implement 1st Use Case

Citizen Centricity – Identify 2nd Use Case

Data Preparation and Readiness for Analytics

Identify Innovation Matrix

Table of Contents

Milestone-3

Work in Progress

Milestone-4 [Planned]

Page 3: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Hackathon Team

M/S Persistent Systems Ltd - ePragati HackathonSNo Name Role ePragati ID

Core Team1 Jitendra Patil Chief Enterprise Digital Architect EPR-0112 Vinayak Datar Delivery Manager EPR-0123 Anand P Kulkarni Senior Business Analyst EPR-0134 Tajinder Khandpure Senior Developer EPR-0145 Amit Walke Senior Developer EPR-0156 Nikita Rani Programmer EPR-0167 Sanketh Vidap Programmer EPR-0178 Priyanka Sabankar Programmer EPR-0189 Ramkumar Khubchandani Programmer EPR-019

10 Praful Dhone Programmer EPR-020

Page 4: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster
Page 5: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Software 4.0 (Agile SAFe) for Hackathon Release Cycle – Coordination & Execution

Activity & Tasks, Source Code Management, Collaboration - GitHub, Kanban BoardQuality Management – Appian TDD, ShareInsights Automation Framework

Build & Release - Jenkins

Sandbox

Engineering Hub

Continuous capture of Logs, Screenshots of all activities, including Design, Develop, Test, Deploy, Change/Enhance

Delivery Hub

Challenge,Brainstorming,

Solution Approach, Design,

Release Planning

Milestone 112:00 Noon

Milestone 2 (6 hours)6:00 PM

Milestone 4Next Day –

3:00 PM

Develop + QA + Operate(DevOps)

Regression, Release, Video Recording

UAT

Discover, Define Develop, Delivery, Operate Optimize

Develop + QA + Operate(DevOps)

Demo

Milestone 3 (6 hours)Next Day – 12:00 Noon

Page 6: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Milestone 1

Page 7: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Snapshot

Activity & Tasks, Source Code Management, Collaboration - GitHub, Kanban BoardQuality Management – Appian TDD, ShareInsights Automation Framework

Build & Release - Jenkins

Sandbox

Engineering Hub

Continuous capture of Logs, Screenshots of all activities, including Design, Develop, Test, Deploy, Change/Enhance

Delivery Hub

Challenge,Brainstorming,

Solution Approach, Design,

Release Planning

Milestone 112:00 Noon

Milestone 2 (6 hours)6:00 PM

Milestone 4Next Day –

3:00 PM

Develop + QA + Operate(DevOps)

Regression, Release, Video Recording

UAT

Discover, Define Develop, Delivery, Operate Optimize

Develop + QA + Operate(DevOps)

Demo

Milestone 3 (6 hours)Next Day – 12:00 Noon

Page 8: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Requirement Analysis, Process, Persona

Page 9: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Data Model, DimensionsStudent StudentMarks

SrNo StudentParent SrNo StuentBankAccount

Name SrNo UID SrNo

UID UID Sub1 UID

Gender ParetName Sub2 BankName

Category Relation Sub3 AccountNumber

LoginTime Income Medium

NativePlace

SeatReservConfig

Quota FeeConfig EnggFeeMaster ColllegeSeat CollegeMaster

Category Quota Quota SrNo SrNo

Gender Category Branhc CollegeId CollegeNmae

Percentage Gender Categroy Quota City

Concession Fees Branch TotalSeats

Total Branch1

Allocated Branch2

Available Branch3

Branch4

Branch5

CollegeId

Source Tables DerivedTables

Student STUDENT_DETAILS

StudentParent

StudentMarks

StudentBankAccount

SeatReserveConfig DIM_QUOTA_CATEGOGY_GENDER

FeesMaster

FeesConfig

CollegeSeat DIM_COLLEGE

CollegeMaster

Page 10: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

IIS Web Server

JBoss App Server

MySQL Business Database

Secured Data

Abstraction Layer

Appian Engine

JDBC

System Integrations

HTTPS(REST/SOAP)

Message Bus

Mail ServerSMTP

More Connectors

Web APIs

iOS

Android Browser Admin Console

Hortonworks Data Platform

YARN Hive Spark

Data Lake Cluster

Python Connector

Nginx + Flask

Code Gen Engine

Connectors

Talend

Responsive Browser

JavaScript OLAP

Widgets(HighCharts)

Insights / Dashboard

Web + App Server

HTTPS

HTTPS

HTTPS

High volume data ingestion

Jetty

PSL LIMA Bot Server

Bot Knowledge Repository

Java HTTP Server

Govt Services

Social Media

Amazon Alexa

Google Home

Intelligent Assistants

HTTPS

HTTPS HTTPS

APIs

HTTPS

HTTPS

Smart Services Interface

BusinessProcesses

Hackathon Technical Architecture and Solution Design

RDMBS SQL Lite

Page 11: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

11© 2017 Persistent Systems - Confidential

Best Practices Identified

Sr No. Solution Requirement Aspects Best Practices

1 Ease of Design Process, Personas, Common Data Model & Dimensions, Common Solution Checklist covering expanded understanding of functional and non-functional Requirements, User FAQ, Innovations identified in Milestone-1

2 Ease of Development Use Appian and ShareInsghts Drag & Drop, Test Driven Development Framework, Common Re-Usable Email/SMS OTP components, Reuse Twitter, Facebook, Google Maps/Leaflet Maps Integration; Common APIs prioritized

3 Ease of Deployment Use Appian 2-click Deployment, Leverage GitHub, Jenkins, ShareInsghits Automation Framework

4 Ease of Change/ Enhancement Versioning in Appian and Forking of Dashboards in ShareInsights

5 Data privacy ShareInsights Data Security Framework (User Roles based Access, Sensitive Data Encryption in Hortonworks), HTTPS compliance

6 Multi department data exchange API Achieved through ShareInsights Upload API that accepts data and External Data API that provides data along with authentication/HMAC based access either way

7 External Data API access External Data API available from ShareInsights with role based access

8 Legacy Application Integration API exposed for legacy applications to hook into workflow (demonstrated through twitter – trigger workflow and send back updates on twitter), Upload API for ShareInsights available for departments to share Data in CSV format

Page 12: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Citizen Centricity : User FAQsFAQ for Student Admission Process

Actor: Student

Que What is the registration process

Ans

You have to go to the student admission website and click on Register here. You need to enter your First Name, Last Name, Mobile Number and Email ID. After entering these details you

will get SMS or Email for further notification for activation.

Que What are the required document do I need to produce to get the admission?

Ans The Student has to upload the Rank Document, Inter Mark Sheets, and Identification Document (Government issued) along with the parent Identification proof.

Que How do I select branch?

Ans Before you select the branch, you have to enter the Rank and College Name in which you want admission.

Que Why do I see many colleges in Contextual Help?

Ans

Colleges may be listed out depending on the preferred groups that you have applied, depending on your nearest location with your Geographic origin by applying the NAC or AICTE

ranking a suggestion.

Que I received SMS and Email, what do I do with it?

Ans You might have get SMS or Email regarding on confirmation of registration process, further notification to admission process. Please follow the instructions in the SMS or Email.

Page 13: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Milestone 2

Page 14: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Snapshot

Activity & Tasks, Source Code Management, Collaboration - GitHub, Kanban BoardQuality Management – Appian TDD, ShareInsights Automation Framework

Build & Release - Jenkins

Sandbox

Engineering Hub

Continuous capture of Logs, Screenshots of all activities, including Design, Develop, Test, Deploy, Change/Enhance

Delivery Hub

Challenge,Brainstorming,

Solution Approach, Design,

Release Planning

Milestone 112:00 Noon

Milestone 2 (6 hours)6:00 PM

Milestone 4Next Day –

3:00 PM

Develop + QA + Operate(DevOps)

Regression, Release, Video Recording

UAT

Discover, Define Develop, Delivery, Operate Optimize

Develop + QA + Operate(DevOps)

Demo

Milestone 3 (6 hours)Next Day – 12:00 Noon

Page 15: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Requirement Compliance Checklist ReadyActor Workflow Functionality

CoveredValidation Covered

MVP Compliance

NFR Compliance

Innovations Implemented

Admission Panel1,2; College Admin Non-Student Registration, Email/SMS Verification

Student Student Registration, Email/SMS verification

Student Student Log IN, Workflow Instructions

Student Student Eligible Branch List & UID generation for Application (Application UID)

Student Student Contextual Help

Student Student Legal/Privace Policy Compliance (from CMS)

Student Student Upload Eductaion Quals - Rank, Inter Marks Sheet (to DMS)

Student Student Upload Govt ID - Self, Parent Identification (to DMS)

Admission Panel1 (Govt ID Verification) AP1 verifying Govt ID - Que, Doc from DMS, Y/N/Query-to-student, Notify Student & AP2

Admission Panel2 (Education Qualification Verification)

AP2 verifying Educational Quals - Que, Doc from DMS, Y/N/Query-to-student, Notify Student & College Admin, Snc College and Central DBs

College Admin College Admin payment-pending, notfication-pending Que - send notification with fee structure

Student Payment reminder (every 4 hours).Cancellation of Application UID (if not paid in 24 hours).

Student Payment link accessible through Application UID - College, Branch, Start Date, Own Coomunication details, College Contact person's details. Payment channels, Payment confirmation. Admission Conformation.

College Admin College Admin payment-pending, notification given Que - check payment status, Non-digital payment confirmation. Admission conformation.

Finance Department Daily Report (in email, accessible in Audit DMS through link) - Total admissions done in previous day, college codes, names, total fee credited

Audit Department Daily Report (in email, accessible in Audit DMS through link) - Total admissions done in previous day, college codes, names, total fee credited

Page 16: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Kanban Board

Page 17: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster
Page 18: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster
Page 19: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster
Page 20: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Citizen Centricity : Innovative Dashboards Design

College Finder (before application process) Student Goal Prediction (for aspiring junior students)

Page 21: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Citizen Centricity : e-Pragati Facebook-based FAQ Chatbot

Page 22: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Data Preparation and Readiness for Analytics : Data for Pivot-Quota-Category-Gender-Logic

Qouta General 60 Quota Category Gender Percentage

Qouta Management 40 General SC Female 2.9997

Gender Female 33.33 General ST Female 1.19988

Category SC 15 General BC-A Female 1.39986

Category ST 6 General BC-B Female 1.9998

Category BC-A 7 General BC-C Female 0.19998

Category BC-B 10 General BC-D Female 1.39986

Category BC-C 1 General BC-E Female 0.79992

Category BC-D 7 General Physically handicapped Female 0.59994

Category BC-E 4 General  NCC, Sports & Extra curricular activities Female 0.9999

Category Physically handicapped 3 General  Ex-service men & defence personnel residing in the

State Female 0.59994

Category  NCC, Sports & Extra curricular activities 5 General SC Male 6.0003

Category  Ex-service men & defence personnel residing in the State 3 General ST

Male 2.40012

General BC-A Male 2.80014

General BC-B Male 4.0002

General BC-C Male 0.40002

General BC-D Male 2.80014

General BC-E Male 1.60008

General Physically handicapped Male 1.20006

General  NCC, Sports & Extra curricular activities Male 2.0001

General  Ex-service men & defence personnel residing in the State Male

1.20006

General OC Male 15.60078

GEneral OC Female 7.79922

Management SC Female 1.9998

Management ST Female 0.79992

Management BC-A Female 0.93324

Management BC-B Female 1.3332

Management BC-C Female 0.13332

Management BC-D Female 0.93324

Management BC-E Female 0.53328

Management Physically handicapped Female 0.39996

Management  NCC, Sports & Extra curricular activities Female 0.6666

Management  Ex-service men & defence personnel residing in the State

Female 0.39996

Management SC Male 4.0002

Management ST Male 1.60008

Management BC-A Male 1.86676

Management BC-B Male 2.6668

Management BC-C Male 0.26668

Management BC-D Male 1.86676

Management BC-E Male 1.06672

Management Physically handicapped Male 0.80004

Management  NCC, Sports & Extra curricular activities Male 1.3334

Management  Ex-service men & defence personnel residing in the State

Male 0.80004

Management OC Male 10.40052

Management OC Female 5.19948

100

Page 23: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Innovation Metrics

Innovations Process Suggestions / Amendments Integration Component Reuse

ProcessEfficiency

Finance, Audit departments interface through email; Twitter based registration and notification; Form-in-Outlook

Leaflet Map Library Integration Python scripts that automatically generate data-as-a-service API

Process Effectiveness

College Aspiration Prediction Dashboard, College Finder Dashboard, Facebook Chatbot FAQ; Voice Chatbot FAQ; Telugu UI; Speech-to-text on mobile;

Digilocketr (simulayted); Govt ID verification (simulated); Payment-gateway integration (simulated)

Arima/Polynomial algorithm for prediction in R, wrapped in Python scripts

Process Compliance

Email/SMS OTP Verifications NAAC list (Jan-17) External Data Ingestion

Nginx Access Logs generator, Flask User Info Log generator

Rapid Development

Data Lake Explorer (used to explore multiple datasets and identify quick data insights)

Twitter integration, HighChartsIntegration

ShareInsights Out-of-box Widgets and Tasks, Forking dashboards for incremental develoipment

Rapid Deployment

Already covered in platform components Pythion scripts for Spark code generator, Python scripts for Web-Backend Code generator, HTML/Javascript code generator

Page 24: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Milestone 3

[Work In Progress]

Page 25: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Snapshot

Activity & Tasks, Source Code Management, Collaboration - GitHub, Kanban BoardQuality Management – Appian TDD, ShareInsights Automation Framework

Build & Release - Jenkins

Sandbox

Engineering Hub

Continuous capture of Logs, Screenshots of all activities, including Design, Develop, Test, Deploy, Change/Enhance

Delivery Hub

Challenge,Brainstorming,

Solution Approach, Design,

Release Planning

Milestone 112:00 Noon

Milestone 2 (6 hours)6:00 PM

Milestone 4Next Day –

3:00 PM

Develop + QA + Operate(DevOps)

Regression, Release, Video Recording

UAT

Discover, Define Develop, Delivery, Operate Optimize

Develop + QA + Operate(DevOps)

Demo

Milestone 3 (6 hours)Next Day – 12:00 Noon

Page 26: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

TBD

Page 27: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Milestone 4

[Planned]

Page 28: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Snapshot

Activity & Tasks, Source Code Management, Collaboration - GitHub, Kanban BoardQuality Management – Appian TDD, ShareInsights Automation Framework

Build & Release - Jenkins

Sandbox

Engineering Hub

Continuous capture of Logs, Screenshots of all activities, including Design, Develop, Test, Deploy, Change/Enhance

Delivery Hub

Challenge,Brainstorming,

Solution Approach, Design,

Release Planning

Milestone 112:00 Noon

Milestone 2 (6 hours)6:00 PM

Milestone 4Next Day –

3:00 PM

Develop + QA + Operate(DevOps)

Regression, Release, Video Recording

UAT

Discover, Define Develop, Delivery, Operate Optimize

Develop + QA + Operate(DevOps)

Demo

Milestone 3 (6 hours)Next Day – 12:00 Noon

Page 29: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

TBD

Page 30: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Outcomes

[To be done]

Page 31: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Documents to be submitted - TBD

• Solution/JADR Presentation (this document)

• Machine Event Logs

• Server Logs

• Data Models

• Data Dimension Models

• Screenshots – Design, Development, Test, Deployment, Changes

• Demonstration Video

Page 32: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

Key Learnings

TBD

Page 33: Persistent Systems Limited - Pragatie-pragati.in/download-files/Persistent-ePragati... · Android Browser Admin Console Hortonworks Data Platform YARN Hive Spark Data Lake Cluster

THANK YOU