cwin17 india / insights platform architecture v1 0 virtual - subhadeep dutta

Cloud Architectures & PlatformsCWIN17, September 27th 2017

Presenter : SHUVADEEP DUTTA, Insights Platform, Insights & Data

2Copyright © Capgemini 2017 All Rights Reserved 2Copyright © Capgemini 2017. All Rights Reserved

AGENDA

Platform Architecture Views

Cloud Platform Security

Platform Logging

Platform DevOps


Future Looking Data Lake – Conceptual Architecture


Big Data and Analytics Platform Logical Architecture – Cloud & Technology Agnostic View


Big Data and Analytics Cloud Platform – MS Azure / AWS Native Services


Big Data and Analytics Cloud Platform – Custom Built Stack (Opensource Hortonworks)


Cloud Deployment Considerations

Data topology, governance & security capabilities need to be evaluated taking into consideration the key considerations to define a target state hybrid cloud platform architecture

Focus Area Consideration

Data Location Location of data storage in alignment with regional regulatory compliance directives

Analytics Use Cases Type of analytical workloads going to be executed on the data repository located at a specific region / country

Data Access, Authentication and Authorization Mechanism of data and underlying platform resource access based on specific user role, user location, time of access etc.

Network Latency Volume of data transfer over network ensuring appropriate bandwidth and SLAs being met

Data Ingestion Type of data ingestion mechanisms used to transfer data from on-premise to cloud and vice-versa

Security and Regulatory Compliance Implementation of controls and mechanisms to meet platform security (perimeter, data and application), regional regulatory compliance directives such as PCI, SOX etc.

Platform Governance Platform wide metadata management, audit logging, master and reference data management capabilities; considerations

for bottom-up / top-down /Hybrid approaches: Data Catalog driven data discovery and knowledge sharing (bottom-up) vs. Data Governance Council (Data Stewardship) driven information asset management and knowledge sharing (Top-down)

Platform Infrastructure Considerations for High Availability, Fault Tolerant and Disaster Recovery capabilities

Considerations from platform performance and scalability based on specific Big Data Analytics workload use cases

Considerations for Platform resource management and orchestration

Data and Application Portability Considerations for data and application portability across different platform environments e.g. On-premise, Private, Public cloud

The information contained in this presentation is proprietary.

Copyright © 2016 Capgemini. All rights reserved.

Rightshore® is a trademark belonging to Capgemini.

www.capgemini.com

8Copyright © Capgemini 2017. All Rights Reserved

Capgemini’s DaaS (Data as a Service) Solution Framework

Meet all data and analytics management service needs from data ingestion, preparation, discovery, through till data analysis using opensource or commercial tools

Leverage client’s current investments and integrate with technology of client choice or extend / customize existing framework capabilities

http://www.capgemini.com/

http://www.facebook.com/Capgemini


http://www.linkedin.com/company/capgemini


http://www.twitter.com/capgemini


http://www.youtube.com/capgemini


http://www.slideshare.net/capgemini


9Copyright © Capgemini 2017 All Rights Reserved

Cloud Platform Security


Big Data and Analytics Platform – Cloud Security Framework

Physical Asset Security Management

Protection for physical assets and locations

including networks and data centers

Cloud Governance

Cloud specific security governance including directory

synchronization and geo locational

support

Information Asset Protection

Protection of data at rest or in transit

Governance & Compliance

Security governance, maintenance of

security policy, audit

and complianceThreats and Vulnerability Management

Management of vulnerabilities and

mitigations with

Network and endpoint protection

Incident Management

Management and responding to expected and

unexpected events

Identity and Access Management

Authentication of users and

management of identity


Cloud Security Capability Framework – Shared Responsibility Model


AWS VPC Security Reference Architecture

VPC Security Architecture Scenarios

Scenario 1 : VPC with a Single Public Subnet only

• Instances run in a private, isolated section of the AWS cloud with direct access to

the Internet

• Network ACLs and security groups can be used to provide strict control over

inbound and outbound network traffic to EC2 instances

Scenario 2 : VPC with Public and Private Subnets and NAT

• In addition to public subnet, a private subnet is added whose instances are not

addressable from the Internet

• Instances in the private subnet can establish outbound connections to the Internet

via the public subnet using Network Address Translation (NAT Gateway or Instance)

Scenario 3 : VPC with Public and Private Subnets and hardware VPN access

• IPsec VPN connection between Amazon VPC and customer data center, while also

providing direct access to the Internet for public subnet instances in Amazon VPC

• VPN appliance on customer corporate data center side

Scenario 4 : VPC with Private Subnet only and hardware VPN access

• Instances run in a private, isolated section of the AWS cloud with a private subnet

whose instances are not addressable from the Internet

• Private subnet is connected to customer data center via an IPsec VPN tunnel


AWS VPC Security Reference Architecture – Security Groups

VPC security groups to firewall each EC2 instance

Each instance can be in up to 5 security groups

Separate security groups for applications and management

Security groups are stateful with ingress and egress rules

Max. 50 rules per security group

VPC Router will allow any subnet to route to another in VPC

Network Access Control Lists are used to restrict internal VPC traffic

Elastic load balancers are used to distribute traffic between instances

Elastic load balancers are also placed in security groups

Platform security can scale up and down with solution; instances can be added into security groups during launch time

Use NAT instances to provide internet connectivity for Private Subnets; allow backend servers to route to AWS APIs – Ex. storing logs in S3

bucket or using DynamoDB, SES

Access AWS API endpoints through the Internet Gateway like S3, SES, DynamoDB, SNS etc.


AWS Identity and Access Management

Securely control access to AWS services and resources

Fine grained control of user permissions, resources and actions

Support for RunInstances

Multi Factor Authentication – Hardware token or Smartphone Apps

Segregation of roles using IAM

AWS Account Owner (Master)

Network Management

SecurityManagement

ServerManagement

StorageManagement


AWS Identity and Access Management Role Based Security


AWS Identity and Access Management using Enterprise’ existing Active Directory

Flow

1

2

3

4

The enterprise user accesses the identity broker

application

The identity broker application authenticates the users

against the corporate identity store

The identity broker application has permissions to

access the AWS Security Token Service (STS) to

request temporary security credentials

Enterprise users get a temporary URL that gives them

access to the AWS APIs or the Management Console

AWS Identity Federation with Temporary Security Credentials


AWS Data Storage Security – Capabilities

AWS S3 Capabilities –

Access controls at bucket and object level

Cryptographic capabilities such as SSL for data in motion, Server/Client side encryption, MD5 checksums

AWS Redshift Capabilities –

Full disk encryption; CloudHSM to store keys

Back-up access logs to S3 for analysis

Security groups and VPC for deployment; data loading using SSL from S3 with restricted access to S3

SSL encryption for data accessed over internet

AWS RDS Capabilities –

Restricted access to RDS instances using Security groups and IAM permissions

Data encryption (Data at rest and in motion)

Automatic patching for minor updates

AWS DynamoDB Capabilities –

Fine grained security access to columns and rows using IAM role and access policies

AWS EBS Volume Capabilities –

Option to use own encryption or commercial solutions Ex. Windows BitLocker or Linux LUKS for encrypted volumes and TrueCrypt for containers; Commercial : Safenet Protect-V,

Trend Secure Cloud etc.


Securing AWS Applications Process Flow


Platform Logging


Platform Monitoring – Centralized Troubleshooting, Security, Audit and Monitoring


Platform Monitoring – Solution Options

Shippers Queue Logstash ElasticsearchElasticsearch

Kibana

Log Parsing Indexing & Curation Dashboards & ReportsRedis, RabbirMQ, KafkaSyslog, Rsyslog, Logstash, Fluentd etc.

JSONEvents Extract

Option 1 – Custom Built using ELK stack (Elasticsearch, Logstash and Kibana)

Option 2 – SaaS Solution – Loggly (alternate leading tool : Sumologic)

Shippers

Syslog, Rsyslog, Logstash, Fluentd, cloud plug-ins etc.

Application Framework

INGEST PROCESS INDEX

Search & Other

Services

• Cloud-based SaaS for easy central log collection, aggregation, management

• Easy set-up• Dynamic parsing - Real-time, JSON support, parsing/tagging,

self-documenting• Regular Expressions based• Dashboards, pre-configured and customizable, shareable• Anomaly Detection• Alerts that can be sent to HipChat, Slack, PagerDuty, HTTP

endpoints, others• JIRA Software integration, point-and click ticket creation

without leaving Loggly


Platform DevOps


DevOps Reference Architecture for Big Data Analytics Workloads

The information contained in this presentation is proprietary.

Copyright © 2015 Capgemini. All rights reserved.

Rightshore® is a trademark belonging to Capgemini.

www.capgemini.com

About Capgemini

With more than 145,000 people in over 40 countries, Capgemini is one of the world's foremost providers of

consulting, technology and outsourcing services. The Group reported 2014 global revenues of EUR 10.573

billion.

Together with its clients, Capgemini creates and delivers business and technology solutions that fit their

needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its

own way of working, the Collaborative Business Experience™, and draws on Rightshore®, its worldwide

delivery model

Learn more about us at www.capgemini.com.












http://www.capgemini.com/about/how-we-work/the-collaborative-business-experiencetm




http://www.capgemini.com/about/how-we-work/rightshorer

http://www.capgemini.com/about/how-we-work/rightshorer



cwin17 india / insights platform architecture v1 0 virtual - subhadeep dutta

Presentations & Public Speaking