building a scalable system for tracking shipping packages
TRANSCRIPT
https://www.etsy.com/listing/150171041/vintage-1845-map-of-antwerp
https://www.etsy.com/listing/80301749/watercolor-australia-map-australian
BUILDING A SCALABLE SYSTEM FOR TRACKING SHIPPING PACKAGES
PREMSHREE PILLAI,
SR. ENGINEER, SHIPPINGPREVIOUSLY, YAHOO!
@premshree
AGENDA
EtsyShippingTracking ArchitectureMonitoringQ & A
https://www.flickr.com/photos/seldaek/16161961159/
you
em·pa·thy noun \ˈem-pə-thē\ : the feeling that you understand and share another person's experiences and emotions : the ability to share someone else's feelings
em·pa·thy noun \ˈem-pə-thē\ : the feeling that you understand and share another person's experiences and emotions : the ability to share someone else's feelings
https://www.etsy.com/listing/110699162/seeker-of-happiness-canvas-tote-bag-you
RE-IMAGINE COMMERCE IN WAYS THAT BUILD A MORE FULFILLING AND LASTING WORLD
https://www.etsy.com/listing/201864292/cleveland-map-street-map-vintage-print
Hong Kong
https://www.etsy.com/listing/162358227/old-map-of-hong-kong-and-environs-16-x
Zagrebhttps://www.etsy.com/listing/157215176/croatia-watercolor-painting-art-print
♥ POTTERY
SCOTTSDALE, AZ
ANDOVER, MA
LEFKIS, GREECE
1.4 MILLION ACTIVE SELLERS
19.8 MILLION ACTIVE BUYERS
29 MILLION ACTIVE LISTINGS
https://www.etsy.com/listing/191371121/jael-green-red-sisal-basket-from-kenya
WHERE IS MYPACKAGE?
https://www.etsy.com/listing/180703192/johannes-holst-sailboat-at-sea-1959
WHERE IS MYPACKAGE?
♥
https://www.etsy.com/listing/180703192/johannes-holst-sailboat-at-sea-1959
IN TRANSIT 200,000+https://www.etsy.com/listing/129521877/original-world-map-watercolor-print
OUT FOR DELIVERY 125,000+https://www.etsy.com/listing/129521877/original-world-map-watercolor-print
CARRIERS 150+https://www.etsy.com/listing/129521877/original-world-map-watercolor-print
SHIPPING CARRIERS
FedEx
DHL
USPS
Australian Post
...
LET’S DIVE IN
mysql
tracking_events
SHIPPING APIS
GET http://production.shippingapis.com/ShippingAPI.dll?API=TrackV2&XML=%3CTrackFieldRequest+USERID%3D%22022XXXXH6696%22%3E%3CTrackID+ID%3D%22LZ13XXX3064US%22%3E%3C%2FTrackID%3E%3C%2FTrackFieldRequest%3E
GET http://production.shippingapis.com/ShippingAPI.dll?API=TrackV2&XML=%3CTrackFieldRequest+USERID%3D%22022XXXXH6696%22%3E%3CTrackID+ID%3D%22LZ13XXX3064US%22%3E%3C%2FTrackID%3E%3C%2FTrackFieldRequest%3E
TRACKING STATES
INFO_RECEIVED
PACKAGE_RECEIVED
IN_TRANSIT
DELAYED
OUT_FOR_DELIVERY
...
DELIVERED
STATE MAP
INFO_RECEIVED
“acceptance”
“processed at usps origin facility"
“accepted at usps origin sort facility”
“picked up”
INFO_RECEIVED
“tendered at fedex office”
“picked up"
“in fedex possession”
IN_TRANSIT
“dispatched from usps sort facility”
“sorting complete"
“local holiday - will attempt delivery on next working day”
“customs clearance processing
SELECT * FROM trackings WHERE last_access_date < ?
SELECT * FROM trackings WHERE last_access_date < ?
? = now() - 15
caveat lector
WHAT’S THE PROBLEM?
SIZE
SELECT * FROM trackings WHERE last_access_date < ? LIMIT ?
STRAGGLERS
SHARDING 101
SELECT * FROM trackings WHERE shard = ? AND last_access_date < ?
QUEUE
FAUX QUEUE
QUEUE_3_DAY
QUEUE_1_DAY
QUEUE_3_HOUR
QUEUE_1_HOUR
...
QUEUE_STOPPED
QUEUE MAP
INFO_RECEIVED => QUEUE_1_DAY
PACKAGE_RECEIVED => QUEUE_1_DAY
IN_TRANSIT => QUEUE_1_DAY
OUT_FOR_DELIVERY => QUEUE_1_HOUR
...
DELIVERED => QUEUE_STOPPED
INFO_RECEIVED => QUEUE_3_HOUR
PACKAGE_RECEIVED => QUEUE_3_HOUR
IN_TRANSIT => QUEUE_3_HOUR
OUT_FOR_DELIVERY => QUEUE_1_HOUR
...
DELIVERED => QUEUE_STOPPED
QUEUE AGE
QUEUE_3_DAY => 1*60*60*24*3
QUEUE_1_DAY => 1*60*60*24
QUEUE_3_HOUR => 3*60*60
QUEUE_1_HOUR => 1*60*60
SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?
SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?
last_access_date < time() - age
ASYNC
GEARMAN
unique job key(shard, queue)
IN SUMMARY
API
STATE MAP
SHARDING
CRON/ASYNC
FAUX-QUEUE
MONITORING
StatsD
github.com/etsy/statsd/
Tracking stats
Gearman worker calls
Tracking calls by provider
Tracking queues
CURL errors
tracking numbers
update tracking batch
get tracking (usps)
get tracking (canada post)
curl errors (usps)
curl errors (fedex)
Tracking stats
Gearman worker calls
Tracking calls by provider
Tracking queues
CURL errors
get tracking (usps)
Tracking stats
Gearman worker calls
Tracking calls by provider
Tracking queues
CURL errors
curl errors (usps)
IMAGINARY QUESTIONS FROM YOU
IMAGINARY QUESTIONS FROM YOU
IMAGINARY QUESTIONS FROM YOU
ActiveMQ/RabbitMQ?
webhooks?
your ideas
@premshree https://joind.in/13479
https://www.etsy.com/listing/158954869/flat-note-card-thank-you-letterpress-110