© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Greg Khairallah, Business Development Manager, AWS
Malini Saxena, Senior Consultant, AWS
Raj Chary, VP of Technology / Architecture, WagglePractice
Lige Hensley, Chief Technology Officer, Ivy Tech
June 20, 2016
Easy Analytics with AWS
What to expect from this session
• AWS toolkit for analytics
• Understand stakeholders
• Demo
• Case Study – WagglePractice
• Case Study – Ivy Tech
• Q&A
AnalyzeStore
Amazon
Glacier
Amazon
S3
Amazon
DynamoDB
Amazon RDS,
Amazon Aurora
Big data portfolio—but what do I recommend?
AWS Data Pipeline
Amazon
CloudSearch
Amazon EMR
Amazon EC2
Amazon
Redshift
Amazon
Machine
Learning
Amazon
Elasticsearch
Service
AWS Database
Migration
Amazon
Kinesis
Analytics
Amazon Kinesis
Firehose
AWS Import/Export
AWS Direct
Connect
Collect
Amazon Kinesis
Streams Amazon
QuickSight
Match toolset to right persona
• Business intelligence (BI) analyst
• Primary tool is SQL
• Historical data resides in data warehouse such as
Amazon Redshift
• Data scientist—Uses programmatic languages such as R or
Python
• Application developer—Requires API to integrate with AWS
services
BI analyst
BI analyst with existing BI tools
BI Analyst
BI tools
Amazon EC2
Amazon Redshift
QuickSight API
• Primary tool is SQL
• Data is largely structured with well known data sources
• Primary concern is fast, consistent performance
• Need to extend SQL with custom functions
BI tools
Amazon EC2
Amazon QuickSight
Amazon QuickSight
Amazon Redshift system architecture
Leader node• SQL endpoint
• Stores metadata
• Coordinates query execution
Compute nodes• Local, columnar storage
• Execute queries in parallel
• Load, backup, restore via Amazon S3; load from Amazon DynamoDB, Amazon EMR, or SSH
Two hardware platforms• Optimized for data processing
• DS2: HDD; scale from 2 TB to 2 PB
• DC1: SSD; scale from 160 GB to 356 TB
10 GigE
(HPC)
JDBC/ODBC
New SQL functions
We add SQL functions regularly to expand Amazon Redshift’s query capabilities
Added 25+ window and aggregate functions since launch, including:
LISTAGG
[APPROXIMATE] COUNT
DROP IF EXISTS, CREATE IF NOT EXISTS
REGEXP_SUBSTR, _COUNT, _INSTR, _REPLACE
PERCENTILE_CONT, _DISC, MEDIAN
PERCENT_RANK, RATIO_TO_REPORT
We’ll continue iterating but also want to enable you to write your own
Window function examples: http://docs.aws.amazon.com/redshift/latest/dg/r_Window_function_examples.html
Scalar user defined functions
You can write UDFs using Python 2.7
• Syntax is largely identical to PostgreSQL UDF
• Python execution is performed in parallel
• System and network calls within UDFs are prohibited
Comes integrated with Pandas, NumPy, SciPy, DateUtil, and
Pytz analytic libraries
• Import your own libraries for even more flexibility
• Take advantage of thousands of functions available through Python
libraries to perform operations not easily expressed in SQL
A very fast, cloud-powered, business
intelligence service for 1/10 the cost of
traditional BI software
What is Amazon QuickSight?
Business
User
Business
User
QuickSight
APIQuickSight UI
Mobile Devices Web Browsers
Partner BI Products
MetadataData PrepConnectors SuggestionsSPICE
Amazon
S3
Amazon
Kinesis
Amazon
DynamoDB
Amazon EMRAmazon
RedshiftAmazon RDSFiles Third-party
Data scientist
Data scientist with existing toolsets
Data scientistToolkits like SAS or
R Studio installed
with Amazon EC2
Unstructured data
Amazon S3
Structured data
Amazon Redshift
• Work with unstructured datasets
• Use existing toolsets to connect to Amazon Redshift
Querying Amazon Redshift with R packages
• RJDBC—Supports SQL queries
• dplyr—Uses R code for data
analysis
• RPostgreSQL—R compliant
driver or database Interface (DBI)R UserR Studio
Amazon
EC2
Unstructured data
Amazon S3
User profile
Amazon RDS
Amazon Redshift
Connecting R with Amazon Redshift blog post: https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon-Redshift
Querying Amazon Redshift with R packages example
Application developer
Application developers can build smart
applications using Amazon Machine Learning
Structured data/predictions
Amazon Redshift
Generate/query
predictions
Amazon QuickSight
Application
Amazon Machine
Learning
Visualize
• All skill levels
• Amazon Machine Learning technology is accessed through APIs and SDKs
• Embed visualizations in applications
Demo
Raj Chary, WagglePracticeVice President of Technology/Architecture
Smart, responsive practice
Math and ELA (Grades 2-8)
Provides students the right
challenge at the right time
What is Waggle?
Right Challenge, Right Time
Waggle looks for more than
correct answers. Waggle
continually analyzes each
student’s decisions and
progress. That way, students get
tougher material right when
they’re ready.
What is Waggle?
Productive Struggle
Waggle motivates students to
push themselves forward. How?
Through helpful hints,
supportive feedback, and
achievement badges that build
grit and confidence.
What is Waggle?
Constructive Grouping Waggle’s
insights means you can easily
group students together based
on learning needs. All without
sacrificing the quality of
individual instruction.
What is Waggle?
Waggle: Product Demo
• Data Creators Differentiated learning experience
Fun and engaging
• Data Visualizers Seamless integration with application
Analytics with a Story
Actionable Data
Redshift: Data Warehouse Layout
Write ClusterCompute – dw2.large
Redshift
Read ClusterCompute – dw2.large
Redshift
History ClusterDensity – dw1.xlarge
Redshift
Initial and Incremental {processed} data loads
Periodic Data Snapshots for historical analysis
Data sources
For serving Jaspersoft reports
APIs
OLTP
S3 COPY
S3 UnLoad and Load
S3 UnLoad and Load
Data mart(aggregations)
NodesNodes
Staging
Datamart(aggregations)
NodesS3 UnLoad and Load
S3 UnLoad and Load + UPSERTS
Results and Lessons Learned• Performance Metrics
– Millions of records are processed in <1 minute
• LOAD/UNLOAD commands | UPSERTS | S3 COPY Command – Report queries average < 1 to ~1.5 seconds
– {compression} – gained 20+% efficiencies in data retrieval
• Best Practices
– {sort keys} – lens-based data model: visualize data in variety of ways
– {commit stats} – Redshift is not a transactional system
– {nested loops} – no Cartesian products, ensure joins well managed
– {queries that queue} – tune the WLM configuration
– {query runtimes} – faster query means less queuing
– {stats missing} – analyze and vacuum when possible
– {alerts with tables} – monitor to ensure queries running optimally
Thank You
Ivy Tech & Amazon Redshift
May 25, 2016
• Transforming the culture of the College to be more data driven
• Moving from reporting silos to an Integrated Analytics system, we call
this a Data Democracy
• Collecting and analyzing a vast variety of data at a scale that no one
in Higher Ed is doing
• Using machine learning tools to identify students who may need
further assistance
• Starting this fall, we are implementing a one-on-one coaching
initiative for the students we identified with the machine learning tools
What We’re Doing
96% of organizations in the United States
use data in the same way.
…and it’s wrong.
But it’s not just education…
The “Standard” Approach
VIP
Relevant Data for Everyone
Data Regimes
Data Dictatorship: Data is controlled and its use is restricted. There is asymmetric distribution of information based on your position
Data Aristocracy: Data analysts, scientists and PhDs are needed to do anything meaningful. Power concentrates in the hands of these employees and their supervisors
Data Anarchy: Business users feel underserved and take matters into their own hands. They create “shadow IT” systems and work around the “unresponsive” IT group
Data Democracy: Everybody gets timely and equitable access to data. Line of business users are empowered and “own” the data. Executives and IT get out of the way
1 Shash Hegde, Mariner, “The Rise of Data Regimes”, 9/12/13, http://www.mariner-usa.com/rise-data-regimes/ (image substitution for Mao Zedong)
Every organization moves through increasingly complex stages of data accessibility.
Data Maturity Model
… very few complete the transition to Integrated Analytics
Stage 1: Report SilosRequest
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
This is what we have had for
decades at Ivy Tech…
Request
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
Stage 2: Data Warehousing
This is what
most
companies
do…
but we are
taking this a
step further…
Stage 3: Integrated AnalyticsRequest
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
Students by
Financial
Aid
Students
by
Award
Students
by
Term
Students
by
Class
Classes by
Class
Section
Students
These curated collections of
data are designed to enable
direct access to...
…the data you need, regardless of
where it came from. Quickly.
Easily.
GPA Graduation—Cumulative
Graduation Grade Point Average (Cumulative) is an indication of a student's academic progress for all
semester credit classes for all registered terms up to and including the selected term. Letter grades are
assigned points (A=4, B=3, C=2, D=1, F=0) and the GPA is calculated by taking the number of grade
points a student earned in a selected period of time divided by the total number of classes taken during
that same period.
GPA Graduation Cumulative = Sum of a student's total grade points earned in credit classes for all
classes for all registered terms up to and including the selected term / Sum of student's total classes
taken during that same period
NOTES ON USING THIS TERM: GPA Graduation - Cumulative does not include grades from remedial
classes.
Related Terms: [GPA Graduation - Term]
Questions?
Resources
Amazon Redshift Getting Started Guide:
http://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html
Scalar UDF Documentation: http://docs.aws.amazon.com/redshift/latest/dg/user-defined-
functions.html
Introduction to Python UDFs in Amazon Redshift:
https://blogs.aws.amazon.com/bigdata/post/Tx1IHV1G67CY53T/Introduction-to-Python-UDFs-in-
Amazon-Redshift
Connecting R with Amazon Redshift:
https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon-
Redshift
Databricks Apache Spark–Amazon Redshift Tutorial: https://github.com/databricks/spark-
redshift/tree/master/tutorial
Amazon ML Getting Started Guide: https://aws.amazon.com/machine-learning/getting-started/
Amazon QuickSight (Preview Registration): https://aws.amazon.com/quicksight/
Thank you!