engineering real world problems - at toki alumni meeting
DESCRIPTION
Presentation was given to TOKI (Indonesian Computing Olympiad Team) alumni meeting on September 2, 2012 at Hotel Cozy Amos, Melawai, Jakarta. The presentation aimed to highlight the difference between competitive programming and real world software engineering through constructed problems based on real problems at Traveloka.com.TRANSCRIPT
Software Engineering:
The Real WorldDerianto Kusuma
traveloka.com
Flight graph
Shortest path from MES to AMQ?
K shortest routes from MES to AMQ?GA 123 MES-CGK 14.40-16.20JT 124 CGK-UPG 19.00-22.15JT 125 UPG-AMQ 23.45-02.00
JT 50 MES-SUB 07.50-10.05JT 52 SUB-AMQ 16.30-19.30
? ≤ N ≤ ?? ≤ V ≤ ? ? ≤ E ≤ ? Time limit: ? sec
Max memory?Number of test cases?
80 airports, 1,000 daily flight100,000 queries / day, peak at 5-9 pm(10 queries / sec)20% CPU time 20 ms / query
Some routes change schedule by dayIn the long term, routes changePrice changes every ???Data sources for routes & prices?
Data source unreliable Route search must still be reliablePrice always changing reasonable accuracy
What is K shortest routes from A to B at date D, with reasonable price accuracy?
(Constraints: all above)
New problems
Lesson 1
- Functional - Real-time correctness- Architectural - Load / performance - Reliability - Scalability, etc.
1. Web scraping2. Route search
Web scraping10-20 airport pairs / query 5-15 sec / airport pair !!!
Multithreading10 query / sec x 20 scrape / query x 5 sec / scrape = 1000 concurrent threads!Run out of RAM
AsynchronouscallAsync(param1, param2, param3, Callback<T> callback)Backed by selector-based HttpClient
Cachingttl = ? on failure? on timeout?Fetching synchronization?
Route searchPruning?Caching at airport pair level?
New solutions
Lesson 2
User:“Just give me cheap tickets, b*tch”“And don’t make me wait too long”
Am I answering the right question?
“Wrong Question”
Cheap price- reasonable coverage- limited precomputed routes
Speed / “perception of speed”- Aggressive pruning- Polling- Prefetching- UI tricks
Source: hipmunk.com
What user really wants- Tomorrow is cheaper?- Don’t give me that X airlines!- I can’t wake up that early!
What user really wants- I’m actually looking for trains!
Business contexts matter
Lesson 3
Flight search component
Flight searchFlight search
Direct route search
Direct route search
Indirect route search
Indirect route search
ScrapingScrapingRoute
predictorRoute
predictor
Airlines websitesAirlines
websites
Per airline cache
Per airline cache
Per route cache
Per route cache
FeatureFeature FeatureFeature FeatureFeature
Other data sources
Other data sources
......
Route DB
Route DB
Component diagram
Flight searchFlight search
Backend common libBackend common lib
Frontend common libFrontend common lib
FrontendFrontend Control dashboard
Control dashboard
3rd party libraries3rd party libraries
User accountsUser accounts
Common dataCommon data
BackendBackend
FrontendFrontend
BackendBackend
FrontendFrontend
Scraping library
Scraping library
Deployment diagram
Frontend nodesFrontend nodes Backend nodesBackend nodesVIPVIP
Misc serviceMisc service
DBDBDistributed
cacheDistributed
cache
VIPVIPClientsClients
CDNCDN
System builder, not only coder
Lesson 4
Site operationsCode push? Maintenance mode?Configuration? Control? Monitoring?
Dev operationsSource control? Build system? Testing? Staging? Release process?
MaintainabilityCode quality? Loose coupling? Portability? Backward compatibility?
The real world needs to operate
Lesson 5
Software Engineering:
The real world
1. New problems
2. New solutions
3. Business context
4. System design
5. Operation
Thanks for listening!Q & A