rh summit2015 presentation_v2.5

© 2012 Fair Isaac Corporation. Confidential.1

Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation's express consent.

© 2012 Fair Isaac Corporation.

Openstack | Openshift| FICOLesson learned over the past 12 months

Chris Ferraro | Andy Quang | Oscar Sandoval | Nick GerasimatosCloud Engineeringrraro | Kiet Quang | Oscar Sandoval-Partain | Nick GerasimatosPlatform EngineeringFICO

May 22, 2015


Agenda

»Who is FICO?

»Why Openstack

»FAC (FICO Analytic Cloud)

»Why Openshift

»Architecture

»Automation

»Questions


Who is FICO?

»FICO Score, MyFICO» Analytics software and tools for businesses

» Debt management

» Fraud & security analytics

» Customer engagement

» Big Data Analytics

» Designs custom algorithms for many specific use cases

» 90% of all lending decisions in the U.S. rely on the FICO Score

» 2.5B credit cards globally are protected by FICO Fraud Systems

» Founded 1956

» 50+ year history of data and analytics excellence

© 2012 Fair Isaac Corporation. Confidential.4 © 2012 Fair Isaac Corporation. Confidential.4 © 2012 Fair Isaac Corporation. Confidential.4

“The FICO Analytic Cloud provides Platform-as-a-Service (PaaS) access to the FICO Decision Management Platform, allowing customers to use FICO tools and technology to create and deploy applications and services.” – Tony McGivern, FICO CIO


Why move from VMware to Open Source?


Why move to the Cloud?

»Need to expand beyond top tier financial services companies and get into middle market

»Traditional on premise technologies moving to a self service or XaaS/SPI model

»Open Source nature allows for participation in driving change

»Need to lower cost

»Thousands of virtual and container instances

»Faster time to market

»Templatize apps deployment regardless of public cloud, private cloud, or underlying infrastructure

© 2012 Fair Isaac Corporation. Confidential.7 © 2012 Fair Isaac Corporation. Confidential.7

OpenStack Infrastructure


FICO Cloud Architecture v1.0

» Cisco UCS Converge Infrastructure


FICO Cloud Architecture v1.0Lessons Learned

» Complex Design» Too many technologies and distributions to select from» Difficult to troubleshoot when things go wrong» Lack of mature tools for capacity planning» Poor “end to end” integration between IaaS, PaaS, XaaS» No true documented architecture designs and few white papers

made public for customers/users

» FICO Cloud v1.0» Red Hat Enterprise Virtualization and Red Hat OpenStack» RHEL 6.x» Optimize design for compute workloads» Gluster used for primary and only shared storage solution» OpenShift used for PaaS



HAProxy Load Balancer

Openstack Controller/CEPH

Node

OpenStack Compute/CEPH

Node

» Cisco UCS C240M3 & C220M3

C220 C240 C240



» Simplify design» OpenStack (KVM)» CEPH for primary and only distributed shared storage» RHEL 7.x» Simplify design for easier handoff for Operations teams

» Targeted baseline optimized for compute and storage performance

» Haproxy / Keepalived

» Cloudforms Deployment/Proof of Concept

» Neutron SDN» Layer 2 only

» Performance issues and realizations» Does Ceph work for all workloads?

» Does the HW selected fit the application and product requirements?



Openstack Controller Node

C220

OpenStack Compute/CEPH Node

C240

» Cisco UCS C240M3 & C220M3

SolidFire SF4805

F5 Load balancer



» F5 to load balance between Openstack APIs

» Selected the Cisco C240 rack servers as the standard hardware “workhorse” to provide compute and storage

» Moved the OpenStack controller layer to Cisco C220 rack servers

» Implemented tiered Storage» Ceph to provide non SLA driven storage» SolidFire distributed flash storage to provide high performance storage

» Revised the sizing of our OpenShift Node

» Cloudforms single pane of glass for private and public cloud

» Swift object oriented storage


OpenShift


Why Openshift?

» On-premise

» Business needs to reduce the Time to Value and Time to Market

» Templatize apps deployment regardless of public cloud, private cloud, or underlying infrastructure

» Ability to build custom Cartridges

» Decrease development costs & simplified tools used for development and deployment

» FICO has reduced time to value for developing analytic solutions by up to 70% versus legacy on premise software


FICO Analytic Cloud | Openshift

» Launched FAC in 2013 before containers were cool

» Business needs required rapid deployment and continuous integration while simplifying the current design

» FICO Analytic Cloud provides IaaS and PaaS access to the FICO Decision Management Platform for multiple tenants and customers

» FICO Decision Management Analytic Platform allows application developers, business users and FICO partners around the world “one-stop” access to use FICO technology to create, customize, and deploy analytics-driven applications and services empowering them to make educated decisions and lowering risk

» FAC/DMP was built using Openshift as the backend engine


Decision Management Suites

» Platform as a server that allows non-developers(e.g. data scientists) to construct and analytic solution through the use of a rapid application development and catalog of FICO component

» User “stiches” together FICO components with appropriate data flow and business rules RAD environment

» Solution is then deployed via Openshift Enterprise for consumption

» Full lifecycle management allows user to develop promote to a separate test environment and ultimately promote to run time / production

» OSE not used for internal DevOps, but instead is a productization for customer consumption


FICO Analytic Cloud Architecture | Openshift v 1.0


Openshift | 1.0

» Installation was manual (bind, activemq, mongo,OSE)

» Deployed on existing VMware infrastructure & Netapp

» Limited online documentation

» Single Point of Failure (1 broker/dns/activemq/mongodb)

» Break fix / Bugs

» Tips:

» Documentation

» Operationalize Openshift


Openshift | 1.1

» Node settings (base on your environment Application/node size

» Gear Idling (2hrs)

» Node gear size/profiles

» Over commitment: number of active gears per node

» Identify & eliminate SPOF

» Openshift (broker, bind, activemq, mongo)

» Application ==> scaled apps

» Tips:

» Monitoring

» DNS

» Platform (internal/external name resolution)

» Offload name resolution Infoblox DNS Appliance

» Dynamic DNS on Bind


Openshift | 1.2

» Openshift Upgrade

» OSE Deployment via PXE

» HA design (3 brokers, 3 mongo, 3 activeMQ)

» F5 (brokers)

» Tuning Openshift

» Load Test (jmeter / custom scripts)

» ActiveMQ JVM OOM (activemq.xml)

» MongoDB(set primary)

» Tips:

» Monitoring / Trending (Keynote, Zabbix, Cacti)

» Total active gears

» Total Capacity per environment

» Total idled gear in the past 90 days


Openshift | Monitoring | Trending Stats


Openshift | Current Architecture


Openshift | 2.0

» New deployment

» Parallel 1.2 / 2.0

» Openstack / CEPH

» Foreman / Heat OSE Deployment

» Implemented active/passive DR/HA across 2 datacenter to meet SLAs

» OSE 2.2 , 3.0 origin


Automation


Our Tools Foreman, Puppet, Cisco UCS

Provision

Configure

Manage


Why FICO Automated Using Puppet and Foreman

» Save Time and Manpower.

» Ensured consistency of configurations across all nodes.

» Repeatability of deployments.

» Source controlled configurations.


Foreman

» Foreman can deploy to both virtual and bare metal machines.

» Foreman Provisions RHEL on Cisco UCS C240s and C220s

» Foreman Host groups with Parameter overrides preps the Cisco UCS 240s and 220s for puppet configuration.

» Foreman Proxies in every Geo-location allows ease of deployment.


Puppet

» FICO Custom Puppet Module to install OpenStack Icehouse and Juno.

» Custom Puppet module was built using many different Puppet Forge Modules.

» Wrote custom facts to tie everything together.

» Reduce build time from days to minutes. » Average build time per server is 3 minutes.


Satellite 6

» Consistency of Packages across OpenStack, Openshift and VMware

» Repeatability of deployments

» Controlled package repositories.

» Different views per release, allowing testing without affecting current environments.

» Reduce the amount of tools within our environment

» Already using components of Satellite 6» Foreman» Puppet

» Better integration with CloudForms than Spacewalk


CloudForms

» Manage OpenStack and legacy VMware from one tool

» Resource usage and optimization

» Self-service portal and catalog

» Single API and GUI for both development and production environments

» Dashboards and reporting

» Ahead of the curve on integration» Professional services needs» Needed functionality 1 – 2 releases out


Red Hat Ceph Storage

» Why Ceph?» Scalable» Open Source» Software-defined storage» Provides both block and object oriented storage

» Optimize small footprint deployments with OpenStack/CEPH converged hardware

» Tight integration with OpenStack

» SolidFire for high performance SLA driven workloads» Scalable, clustered all-flash storage array provides inline compression,

deduplication and replication


Red Hat Enterprise Linux

» RHEL 6 & 7» OSP runs on RHEL 7

» Provides solid platform understood by Operations

» Common and supported implementation across multiple products» Cisco UCS» Oracle RAC


What’s Next

» Use UCS Central automation for quick stand up of private cloud environments regardless of geo location or availability zone

» OpenStack Kilo Automation» Puppet OpenStack module?

» Cisco UCS M4 series

» Eliminate Ceph and use only Solidfire distributed flash storage?

» Migrate away from NFS/NAS to object oriented storage such as Swift?

» Satellite 6 replacement for Spacewalk, etc

» CloudForms» Single pane of glass» Ability to manage both Private

and Public Clouds» Capacity planning» Reporting» Integration with OpenStack and

OpenShift» Single API regardless of the

endpoint


Questions ?


Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation's express consent.

© 2012 Fair Isaac Corporation.

Thank You

May 18, 2015

Nick [email protected]

Chris [email protected]

Kiet [email protected]

rh summit2015 presentation_v2.5

Recruiting & HR