course tech 2013, mark frydenberg, drinking from the fire hose: tools for interpreting and teaching...

Drinking from the Fire Hose: Tools for Interpreting and Teaching

with Big Data

Mark FrydenbergBentley University

CourseMateEnhanced

Edition

77 Movies and TV Shows!

What's your Bacon Index?

Kevin

Bob

1

Ann2Joe

3

Kim

42 X

APIs

Friend of a Friend

Social Graph

Big Data

'Big data' refers to a collection of tools, techniques and technologies which make it easy to work with data at any scale.

powerof60.com

The Road

3 V's

• Volume - amount of data is larger than those conventional relational database infrastructures can handle

• Velocity - the rate at which data is generated, processed and analyzed in (real) time

• Variety – data formats are unstructured and inconsistent

Volume: How Big is Big Data?

Yottabyte?

Walmart

• Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.

• A petabyte is one quadrillion bytes, or the equivalent of about 20 million filing cabinets’ worth of text.

http://hbr.org/2012/10/big-data-the-management-revolution/ar

Velocity: Drinking from the Firehose

• Scrutinize 5 million trade events created each day to identify potential fraud

• Analyze 500 million daily call detail records in real-time to predict customer churn faster

A Variety of Big Data Sources

McKinsey&Company Report (2011)

• Data is part of every industry and business function.

• Data creates value.• Big data becomes a basis of

competition and growth.• Some sectors will achieve

greater gains.• Shortage of people with

analytical skills.• Need policies related to

privacy, security, ownership.

Twitter

Twitter

3000 tweets per seconddata is disorganizedHow does twitter use its data?

Twitter Visualization

Big Data Technologies

• HADOOP: scalable storage, parallel computation

• NoSQL: distributed querying

What this Means

• Change your web page and Google finds it in minutes.

• Ten years ago, you would have to submit a request to Yahoo! to reindex your site.

• All you need is a lot of servers. • Google has a million of them.• No problem.

http://aws.amazon.com/big-data/

Collaborative Filtering

Collaborative Filtering

Me You

Black Beauty

Camera TripodThe Black

Stallion

Variety: Semantic Web

RelFinder

Unstructured Data

Health Care

Analyzing Big Data

explore.data.gov

Searching Big Data

Fusion Table Visualizations

Fusion Table Visualizations

Fusion Table Visualizations

Mark Frydenberg

mfrydenberg@bentley.educis.bentley.edu/

mfrydenberg

Invite me to your school!

CourseMateEnhanced

Edition