mike keating - news int - 18th bdl meetup
Post on 13-May-2015
197 Views
Preview:
DESCRIPTION
TRANSCRIPT
Big Data at News International!
Welcome !Big Data at News International!!
Big Data Event - 29/05/2013!
Mike Keating: Product Owner! @mikerkeating!
Slide 01/14!
Big Data at News International!
Introduction!
• Where to start!• Data and decisions!• Our technology choices!• Lessons!!
Big Data Event - 29/05/2013!Slide 02/14!
Big Data at News International!
Where to start…!
Big Data Event - 29/05/2013!Slide 03/14!
Lots of data, suppliers, technology and teams!!Take control, bring digital data into one place!Link data sets and look for new approaches!!Make data/ information/ knowledge/ insight available!Build awareness and decision making capability!
Big Data at News International!
Making the basics available via dashboards!
Big Data Event - 29/05/2013!Slide 04/14!
Big Data at News International!
Understanding our content and consumption!
Big Data Event - 29/05/2013!Slide 07/14!
Visits per Visitor
View
s pe
r Vis
itBehaviours Across Website Sections
Big Data at News International!
Understanding our content in a social world!
Big Data Event - 29/05/2013!Slide 08/14!
Big Data at News International!
Tracking subscrip.on growth across products!
Big Data Event - 29/05/2013!Slide 05/14!
Days Following First Subscription
Subs
crip
tions
Product Growth Following Launch
Big Data at News International!
Analysing attributes that indicate churn!
Big Data Event - 29/05/2013!Slide 06/14!
Big Data at News International!
Designing products based on patterns in navigation !
The iPhone – Octopus Naviga4on
The Website – Flower Petal Naviga4on
The iPad Edi4on -‐ Linear Naviga4on
Big Data Event - 29/05/2013!Slide 10/14!
Big Data at News International!
Our technology choices – what?!
Big Data Event - 29/05/2013!Slide 11/14!
Infrastructure: AWS EC2, S3, RDS, EMR, Cloudformation, Vagrant!Ops: Jenkins, Anthill Pro, Maven, Nexus, Zabbix, CloudWatch!Code & Config: Puppet, Github!Data Retrieval: Java & Python!Data Pipeline: Java Map Reduce, Apache Crunch, !Spark, MRUnit, Celery, RabbitMQ, Python, Flume!Data Storage: HDFS, HBase, AWS S3, MySQL, Redis!Data Schema: Avro!Data Access: Python APIs, Tornado, S3, Hive & AWS EMR!Analysis: R, Pandas, Excel -> analyst’s choice!Visualisation: R, D3, Highcharts, Google Charts!Products: APIs, JS+HTML+CSS!
Big Data at News International!
Our technology choices – why?!
Big Data Event - 29/05/2013!Slide 11/14!
• Team: Build on team’s skillsets and knowledge. !• Recruitment: Be conscious of “hire-ability”!• Open Source: Big wins from usage; great
communities; contribute back!• Versions: use what works; work with alpha releases
but not as production code!• Consistency: Try and use what you do today, e.g.
AWS!• Flexibility: Use a “better” product where it makes
sense!
Big Data at News International!
Our technology choices – who?!
Big Data Event - 29/05/2013!Slide 11/14!
• Tech Lead: Architecture; Design; Hands-On!• Delivery Manager: Project Management w Agile!• Hadoop: build from Java Map Reduce !• Python: Tornado, Native Python, real-time processing!• Data Science: Hive, R, Modeling!• DevOps: AWS, Vagrant, Puppet!
• Experience: Practical experience of Hadoop in production!• Capability: Ability to learn new tech, design and build!• Demo: contributions to projects, working examples!
Big Data at News International!
Lessons!
Big Data Event - 29/05/2013!Slide 12/14!
• Building awareness & common knowledge!• Building on existing teams, systems and their work!• Looking for extra capability and output!• Focus on visuals – it needs to be sharable/ visible!• Working with range of teams to share outputs!• Making good tech choices!
Big Data at News International!
Thanks!
Big Data Event - 29/05/2013!Slide 13/14!
Big Data Team – DevOps, Hadoop, Python, UI, Analysts, Test!!Technology Teams – Design, Production Ops, Perf Testing, Security, Products, Platforms, Service Desk!!Editorial, Marketing, Commercial, Finance Teams!
Big Data at News International!
!!Thanks!Big Data at News International Contact Us News Interna.onal Technology @techatni Mike Kea.ng; Product Owner @mikerkea.ng Jobs via hBp://joinnitech.co.uk/
Big Data Event - 29/05/2013!Slide 14/14!
top related