mike keating - news int - 18th bdl meetup

Big Data at News International!

Welcome !Big Data at News International!!

Big Data Event - 29/05/2013!

Mike Keating: Product Owner! @mikerkeating!

Slide 01/14!


Introduction!

•  Where to start!•  Data and decisions!•  Our technology choices!•  Lessons!!

Big Data Event - 29/05/2013!Slide 02/14!


Where to start…!


Lots of data, suppliers, technology and teams!!Take control, bring digital data into one place!Link data sets and look for new approaches!!Make data/ information/ knowledge/ insight available!Build awareness and decision making capability!


Making the basics available via dashboards!



Understanding our content and consumption!


Visits per Visitor

View

s pe

r Vis

itBehaviours Across Website Sections


Understanding our content in a social world!



Tracking subscrip.on growth across products!


Days Following First Subscription

Subs

crip

tions

Product Growth Following Launch


Analysing attributes that indicate churn!



Designing products based on patterns in navigation !

The iPhone – Octopus Naviga4on

The Website – Flower Petal Naviga4on

The iPad Edi4on -‐ Linear Naviga4on



Our technology choices – what?!


Infrastructure: AWS EC2, S3, RDS, EMR, Cloudformation, Vagrant!Ops: Jenkins, Anthill Pro, Maven, Nexus, Zabbix, CloudWatch!Code & Config: Puppet, Github!Data Retrieval: Java & Python!Data Pipeline: Java Map Reduce, Apache Crunch, !Spark, MRUnit, Celery, RabbitMQ, Python, Flume!Data Storage: HDFS, HBase, AWS S3, MySQL, Redis!Data Schema: Avro!Data Access: Python APIs, Tornado, S3, Hive & AWS EMR!Analysis: R, Pandas, Excel -> analyst’s choice!Visualisation: R, D3, Highcharts, Google Charts!Products: APIs, JS+HTML+CSS!


Our technology choices – why?!


•  Team: Build on team’s skillsets and knowledge. !•  Recruitment: Be conscious of “hire-ability”!•  Open Source: Big wins from usage; great

communities; contribute back!•  Versions: use what works; work with alpha releases

but not as production code!•  Consistency: Try and use what you do today, e.g.

AWS!•  Flexibility: Use a “better” product where it makes

sense!


Our technology choices – who?!


•  Tech Lead: Architecture; Design; Hands-On!•  Delivery Manager: Project Management w Agile!•  Hadoop: build from Java Map Reduce !•  Python: Tornado, Native Python, real-time processing!•  Data Science: Hive, R, Modeling!•  DevOps: AWS, Vagrant, Puppet!

•  Experience: Practical experience of Hadoop in production!•  Capability: Ability to learn new tech, design and build!•  Demo: contributions to projects, working examples!


Lessons!


•  Building awareness & common knowledge!•  Building on existing teams, systems and their work!•  Looking for extra capability and output!•  Focus on visuals – it needs to be sharable/ visible!•  Working with range of teams to share outputs!•  Making good tech choices!


Thanks!


Big Data Team – DevOps, Hadoop, Python, UI, Analysts, Test!!Technology Teams – Design, Production Ops, Perf Testing, Security, Products, Platforms, Service Desk!!Editorial, Marketing, Commercial, Finance Teams!


!!Thanks!Big Data at News International Contact Us News Interna.onal Technology @techatni Mike Kea.ng; Product Owner @mikerkea.ng Jobs via hBp://joinnitech.co.uk/


mike keating - news int - 18th bdl meetup

Technology

data pipeline

data access

data retrieval

data schema

data storage

digital data

data science

big data team devops