Inside Zalo: Developing a mobile messenger for the audience of millions

BY THANH DAO, VNG CORP.

About me

Career:• VNG: Senior Manager - Head of Zalo & Zing

Mobile Product Group• Past: Microsoft, Kofax

Contact:• thanhdn@vng.com.vn • http://me.zing.vn/thanhngocdao• http://facebook.com/thanhdao

My Goals

• Talk about how we started a mobile first product, growing the user base and the learned lessons that lead us here.

• Give you a blueprint for how to scale your mobile server system to millions.

Agenda

• Background of Zalo• The user growth story• Continuous development

Landscape before Zalo

5

Sticky, but...• Don’t support mobile first

use-case and phonebook graph.

• Not focus on enhance chat and group chat

• My mom, sister don’t contact me via SNS

Social Networks Traditional Messengers OTT IM

Early movers & viral, but…• No profile, social features.• Lack of fun features appeal

to the young users• Can’t customize a feature

for the Vietnamese community

• Low barrier to entry

Strong network effect on PC, but...• No social features• Not fun enough• Not mobile friendly

Model of Zalo

Mobile social messaging:• Based on Mobile First & Phonebook Graph• Combine Context Messaging & Connecting &

Circle Socializing• Viral (i.e SMS/Zing/Facebook…)• Sticky• Mass market appeal

Target

18-25 26-30

Singles Core segment Core segment

Unmarried couples

Potential to grow share ? Potential to penetrate

Married couples

XNot target yet

XNot target yet

Product

P

P

P

P

latform

roduct1. Communication: Voice messaging & Call…2. Social: Timeline, People Near By, Room…3. Platform: Page, 3rd Apps, Games.

“Fun & Connection & Private”Anchored by basic: Fast, Quality, Cheaproposition(Value)

lace (Channel)

1. App Stores: Itunes, Google Play, Nokia Store2. Factory Preload on Nokia Asha, WP…3. Side-load at retail shop

Marketing• Position as an Expression/Lifestyle.• Focus on messaging the brand benefit

– “Talk more to your loved ones”

•360 Campaign launch at Online, Social, PR, TVC, OOH (Cinema, University,…)

Agenda

• Background of Zalo• The user growth story• Continuous development

Agenda

• Background of Zalo• The user growth story• Continuous development

Aug-12

Aug-12

Sep-12

Sep-12

Oct-12

Oct-12

Nov-12

Nov-12

Dec-12

Dec-12Jan

-13Jan

-13

Feb-13

Feb-13

Mar-13

Mar-13

Apr-13

Apr-13

May-13

0

5000000

10000000

15000000

20000000

25000000

Registered User Daily msg

User Discovery Q3/2012: Born

Product• Zing & Phonebook mobile messenger• voice, sticker, draw and people near by

Users feedback and data told us• Good for dating purpose• The draw & games, sticker are fun• The concept of mixing up Zing &

phonebook graph is confusing, they want a separated buddy list

• Sending speed is slow with 3G, drain the battery lifecycle

• Nokia version is not available yet

So we• Removed “Chat with Zing friends” feature

and focused on the Mobile First use-cases• Rewrote our own socket chat servers

instead of relaying via Zing Me Web Chat servers

User Discovery Q4/2012: Totter

Product• New version Nokia/Droid/IOS• Group, Room• Social Timeline

Users feedback and data told us• Voice/text is fast• UX is much better than the prev one• Spam and privacy concerns• Not enough friends so both retention and

engagement are problems

So we• Improve notification services• Improve picture quality• Scale up advertising budget, Run 360

degree marketing campaignAug-1

2

Aug-12

Sep-12

Sep-12

Oct-12

Oct-12

Nov-12

Nov-12

Dec-12

Dec-12Jan

-13Jan

-13

Feb-13

Feb-13

Mar-13

Mar-13

Apr-13

Apr-13

May-13

0

5000000

10000000

15000000

20000000

25000000

Registered User Daily msg

User Discovery Q1/2013: Walk

Product• Friend Circle• Zalo Page (VIP)• Friend invitation• Support 18 countries

Users feedback and data told us• Voice message is addictive • Multi-picture upload is convenient

So we• Run Friend Invite Campaign• Developing Call.• Work with 3RD party apps

Aug-12

Aug-12

Sep-12

Sep-12

Oct-12

Oct-12

Nov-12

Nov-12

Dec-12

Dec-12Jan

-13Jan

-13

Feb-13

Feb-13

Mar-13

Mar-13

Apr-13

Apr-13

May-13

0

5000000

10000000

15000000

20000000

25000000

Registered User Daily msg

Agenda

• Background of Zalo• The user growth story• Continuous development

Stats

• 2.3M registered users (40% Droid; 30% IOS; 30% Nokia).

• >500K new users join a month.• 800K DAU• 380K peak CCU• 20M messages sent a day.• 39 servers, 150M requests a day.

Server Platform

• CentOS• Binary protocol, SSL, TCP/IP, UDP.• C++ for core, java for other business services.• Za-Router, long connection service map and load balancer.• Za-DB: a key-value db storage similar to Amazon Dynamo/Google Big T.• Za-Cache: memory caching system.• Za-Session manager• Za-CDN• Za-Messaging/Signal Socket Server• Za-Stream Download/Upload Server.• Za-Voice/Video Media Proxy Server• Za-Configuration center: synchronize config among services.• Za-Monitor center + zabbix + nagios + cacti + dogslow: trend monitoring.• Postgre SQL(PostGis): store lbs data, can handle >10k requests/sec with memory caching.• Dynamic DNS • HA Proxy: web load balancer(https login services).• Squid: static content caching, small photo…• Hadoop: data analyzing and mining• Elastic search• Scribe for log, backup…

ZAComm Router

ZA Socket Protocol

ZAComm #1

Worker

Worker Pool

Conn Manager

ZAComm #2

Worker Pool

Conn Manager

ZAComm #N

Worker Pool

Conn Manager

VIP

LBS

Room

Worker

Msg

Group

Share

Worker

Avatar

Photo

Cover

Worker

Contact

Privacy

Friend

MemCache

LBS

Persistent

Caching

Message Items

Persistent

Caching

Photo

Persistent

Caching

Background Job

Mon

itorin

g

Server Infrastructure

39 Servers:• 6 relay storage servers• 9 database servers• 6 business application servers• 3 proxy caching servers• 4 other caching servers• 4 backup servers• 3 log and monitor/configuration servers

Capacity:• 1M CCU• I/O speed: 200K message/sec

Scalable Mobile Servers

• Scaling connection servers– Each connection server handle a shard of users.– Configuration center.– Service map.

• Scaling database– Distributed hash table design.– Backup & recovery.– No alter on large table, create new one.– Soft deletion, Lazy write.

• N+1 design– Build horizontally, never less than two of anything.– No single point of failure.

• Think about monitor during design, not after.• Identify where bottlenecks are in each tier

– Control Port(CPU). – Data Port(Network/Memory/Cache/Storage IO…).

High Performance Mobile Client

• Speed:– Different network type, different connecting strategy

• Use wi-fi as much as possible.• Maximize bandwidth in fewer calls and group multiple

requests concurrently into a single multi-get request. • Use bigger buffers.

– Decouple user transitions from data interactions:• I.e Apply tricky like, silently uploading the image in

background during the user inputting description.

– Avoid large thumb: • Send high-resolution thumb to high resolution devices and

low-resolution thumb to low resolution devices

– Cache more often, flush cache regularly

• Battery:– HTTP versus TCP/IP reuse.– Avoid constant polling(but be tricky when using poll/push)

…– Download in bursts/chunks rather than continuously

streaming.– Minimize aggressive behaviors – Read/Write to cache then flushing a big mount of data into

device storage once.– Use profiler tool to benchmark and measure your app

battery consume speed.

Operation

• Tracking user feedbacks(CS, Store, FB, ZM, Noti5, Buzz…).

• Monitor everything:– System performance(Server load/traffics, Memory, Network…).– Business metrics.– QoS statistics: Load-times; errors.– Trending monitors.

Team & Process

• 27 developers: Server & Backend, Mobile Client, R&D framework.

• 2 system engineers• 5 QA engineers• 3 graphic designers: 1 artist & 2 graphic design

• 5 product managers, each person:– Responsible for different group of features: acquisition, connection, communication, social…etc– Has key business metric: conversion, retention, engagement, chat…and combined metrics.

• Scrum & continuous deployment process:– Build isolation, enable testing single build without deploy.– Auto Unit Test -> Integrating-> Staging -> Experiment -> Launch.

Questions?

Ready to embrace new challenges?

We are hiring…lepm@vng.com.vn

Thank you!