wongnai engineering story
TRANSCRIPT
Wongnai Engineering Story
Pattrawoot Suesatayasilp @pattrawoots
CTO & Co-founder @ Wongnai
Wongnai• A startup company
• Our mission is to help Thai people discover and connect to great local businesses.
• We believe in trials and errors.
• Time is our most valuable resource.
What It Means To Be Wongnai
WongnaiUser-Generated Content Website
CMS, SEO
Search
Location-Based Service
Social Network
Feed, Follow, Like, Comment
Commerce
Deals, Payment
Web & App
Web & App
Statistics
300k sessions/day 53M page views/month
20k requests/min (3x during holidays)
Statistics
80%
20%
DesktopMobile (App/Web)
34%
66%
Mobile AppMobile Web
• During meal — Lunch, Dinner
• Push Notifications
• Social Post / Dramas
High Traffic
DinnerLunch
Push
• Weekends especially the long ones
• Special Occasion (Valentine’s)
• Holidays
High Traffic
Chinese New Year Valentine’s
Long WeekendSat - Sun
Our 1st dev server
Apple iMac Core 2 Duo 2.0GHz + DynDns
Our 1st prod server
Dell PowerEdge R210 ฿30,xxx
Move To The Cloud
2010-2013 User Growth
0
250000
500000
750000
1000000
Oct '10 Mar '11 Aug '11 Jan '12 Jun '12 Nov '12 Apr '13 Sep '13
Sep 2012 200k subscribers
Email Blast• MailChimp until it became too
expensive. (almost 200k subscribers)
• Set Up our own mail server
• OpenEMM
• No additional cost but many problems
Email Blast• Slow send rate — Takes too long to send all
• Consume too much bandwidth
• Poor open rate — Some went to Junk
Amazon SES
• Send Rate (emails/second)
• Up to 90 emails/second
• Whitelist IP Addresses
• Migrating to SES wasn’t quite smooth.
Amazon SES
SEO & Scalability• Large number of web pages (> 100k pages)
expecting Google bot to crawl
• Poor latency and not quite reliable (e.g. power outage)
• 2013’s goal is to reach 1M users. Be ready to scale.
• Cloud computing is the way to go.
Migrate to AWS• Million of photos to
move to the cloud
• Near-zero downtime
• Installed mobile apps must continue seamlessly.
Migrate to AWSAmazon Web ServicesHost in TH
www.wongnai.com photo.wongnai.com
Static Content Servers
Web Server
Database
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Upload photos gradually
Database
Update ‘synced’ flag
Static Content Servers
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
www.wongnai.co
m
1
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
photo.wongnai.
com
Identify photo’s location
2
3
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
Redirect
to th
e old host
4
Migrate to AWSAmazon Web ServicesHost in TH
pull not-synced-yet
images
Web Server
Database
Static Content Servers
5
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
www.wongnai.co
m
1
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
photo.wongnai.
com
2
3
Identify photo’s location
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
Redirect
to S3
4
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
pull already-synced images
Static Content Servers
5
Database & ORM
When a list of reviews are displayed
a review • review’s details
• user (reviewer) • location • photos • likes
• preview comments • and many more …
Select N+1 Problem
ReviewerReview Details
Review
PhotoPhotoPhotoPhotoPhotoPreview Comments
Location
Many tables needs to be joined to get enough info for displaying a review.
Large Table JoiningReview
500,000 rows
User
2,000,000 rows
Location
200,000 rows
Photo
3,000,000 rows
Large Table Joining
• As a result of experiments, selecting rows separately without table joining performs much better. (No bottleneck at the database) — Feel the NoSQL?
Single SQL command
Multiple SQL commandVS
Schema Changes
• Mid 2015 — We decided to combine Food and Beauty into a single app
• Tables need to be merged. Columns need to be added. Millions of rows are affected.
What we expected
large table
What we expected
large table
What actually happens
create new tablelarge table
What actually happens
create new tablelarge table
copy
• A very large number of I/O operations need to perform to update the schema
I/O Speed• Storage for Amazon RDS (Amazon EBS)
• IOPS — The number of I/O operations completed per second)
• Base Perf: 300 IOPS (for 100GB storage)
• Burst 3,000 IOPS Maximum — Non-provisioned IOPS Storage
• Burst duration is limited.
• Error 404 — almost 24hrs
• Liquibase and many other schema migration tools
• Only one column can be added at a time!
Why still SQL?• ACID Transaction
• NoSQL is not good at searching. (Compared to RDBMS)
• NoSQL was not popular yet, and also still lacked many key features that SQL has. (MongoDB was just one-year old. Cassandra is still in beta.)
Search
Types of Search
Any good restaurants near me?
Show me all branches of Sushi Hiro
Show me all restaurants in Central World
Search for “Bonchon”
Indexing• Apache Solr
• Fast read but slow write
• Our indexes get updated very often, and they affects ranking — e.g. new review/photo/check-in, create new place, edit place’s info, etc.
• Tried updating indexes real-time and Solr’s soft-commit but the performance was not satisfied.
• So, we add another layer to do batch processing.
Indexing
• Why not Amazon Elasticsearch Service?
• Too difficult for testing
• We don’t want to stick with AWS.
• Hybrid cloud computing?
Cache• EHCache
• Most of the content are location-based.
• What should be cached? article, guide
• Content keeps updating
• e.g. Your review may get liked right after posted.
• Most of our caches have very short life. (< 1 min) and never get invalidated. Shared cache is not needed.
Scheduled Task• Quartz (by Terracotta) — Job Scheduler
• Score Updater for calculating ranking
• Business, User, Review, Photo, Activity, etc.
• Leaderboard
• Statistical Data Updater
• Number of Photos/Reviews/Check-ins/…
• etc.
Web Stack
เร็วส์Java@roofimon
“ ”
Web Stack
Java Spring SolrMySQL Hibernate Cassandra
RequireJSjQuery Backbone Less Bootstrap
Infra/DevOps
Tomcat
Git
Jenkins
DockerMaven
ZookeeperHTTP Server
Architecture
Solr Cloud
HTTP/HTTPS Load Balancer
Amazon RDS
ZooKeeper Cluster
Cassandra Cluster
Static Content Servers
Monolithic Application
Servers
Amazon SES
• 6 year ago, we were new to web development.
• Default Templating System — JSP
• No pre-processor was there back then, so we
wrote our own pre-processors in Maven Lifecycle
(Phase: Prepare package).
• Minify JS, CSS files — 1 file per page.
• YUI Compressor
Web Stack (Frontend)
• Asynchronous Module Definition • RequireJS
• Why not Angular? • SEO is one of the most important thing for a
content website like Wongnai.com • JavaScript SEO (prerender.io / …)
• More code to maintain • Code refactoring takes time. Not worth for
now
Web Stack (Frontend)
• Sonar (now SonarQube) • No longer use it as we found it doesn’t help
us that much. Our dev team is relative small. Manual code review is far better.
• Jenkins • No distributed build yet. • Only used on dev environment
DevOps
What’s Next• Personalization
• Machine Learning / Scala / Spark
• NoSQL for data that doesn’t require searchability
• e.g. Feed, Notifications
• Advanced Message Queuing Protocol (AMQP)
• RabbitMQ
• Microservices
• Image Server
Process & Tools• Asana, Slack, Google Docs
• Agile
• Manual First
• TDD
• Daily Scrum
• 2-week Sprint (1-month Sprint before)
• Retrospect
Dev Team 6 Full-Stack
Web Developers
3 iOS Developers + 1 Part-timer
2 Android Developers
1 QA Engineer 1 UI/UX Designer
Dev Team
We are Hiring!
6 Full-Stack Web Developers
3 iOS Developers + 1 Part-timer
2 Android Developers
1 QA Engineer 1 UI/UX Designer
Q&A
Q&AJoin Us!
wongnai.com/careers