popvote - university of hong kong · pdf filefabric flask boto pycrypto sqlalchemy...
TRANSCRIPT
![Page 1: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/1.jpg)
Coding PopVote
![Page 2: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/2.jpg)
Patrick CheungPopVote backend developer
![Page 3: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/3.jpg)
Why am I here?
![Page 4: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/4.jpg)
highest throughput in any second
47 votes in 1 second
![Page 5: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/5.jpg)
first voting day (20 June)
![Page 6: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/6.jpg)
may include duplicated votes
> 70% votes casted in less then 180 seconds
![Page 7: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/7.jpg)
![Page 8: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/8.jpg)
puppetredis
tornado
python
mysql
statsdfluentd
supervisord
fabric
flask
boto
pycrypto
sqlalchemy
itsdangerous
pycaptcha
uwsginginx
ubuntu
mcollective
csshx
r10k
json
jwt
async iouuid pubsub
vagrant
![Page 9: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/9.jpg)
• 100% written in Python
• 18,000 line of code
• API Server
• Ballot Server
• Control Panel
![Page 10: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/10.jpg)
![Page 11: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/11.jpg)
Client
![Page 12: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/12.jpg)
Clienthttps
![Page 13: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/13.jpg)
Clienthttps API
Server
firewall
1. captcha 2. verify 3. submit
![Page 14: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/14.jpg)
Clienthttps
redis in-memorytransient data
APIServer
firewall
1. captcha 2. verify 3. submit
![Page 15: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/15.jpg)
Clienthttps https
redis in-memorytransient data
APIServer
firewall
1. captcha 2. verify 3. submit
![Page 16: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/16.jpg)
Clienthttps https
Ballot Server
firewall
redis in-memorytransient data
APIServer
firewall
1. captcha 2. verify 3. submit
![Page 17: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/17.jpg)
Clienthttps https
Ballot Server
firewall
redis in-memorytransient data MySQL ballot data
APIServer
firewall
1. captcha 2. verify 3. submit
![Page 18: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/18.jpg)
Clienthttps https
Ballot Server
firewall
Station Server
https
redis in-memorytransient data MySQL ballot data
APIServer
firewall
1. captcha 2. verify 3. submit
![Page 19: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/19.jpg)
developing a system
![Page 20: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/20.jpg)
developing a system
• short development timeframe
![Page 21: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/21.jpg)
developing a system
• short development timeframe
• system load gradually increase
![Page 22: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/22.jpg)
developing a system
• short development timeframe
• system load gradually increase
• optimise system with real load
![Page 23: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/23.jpg)
developing a system
• short development timeframe
• system load gradually increase
• optimise system with real load
• not mission critical
![Page 24: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/24.jpg)
• short development timeframe
• system load gradually increase
• optimise system with real load
• not mission critical
developing popvote
![Page 25: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/25.jpg)
• short development timeframe
• system load gradually increase
• optimise system with real load
• not mission critical
developing popvote
❌
![Page 26: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/26.jpg)
• short development timeframe
• system load gradually increase
• optimise system with real load
• not mission critical
developing popvote
❌
❌
![Page 27: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/27.jpg)
• short development timeframe
• system load gradually increase
• optimise system with real load
• not mission critical
developing popvote
❌
❌
❌
![Page 28: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/28.jpg)
Public-facing web servers do not have access to the database
![Page 29: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/29.jpg)
Clienthttps httpsAPI
Server
firewall
Ballot Server
firewall
Station Server
https
MySQL ballot data
![Page 30: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/30.jpg)
Clienthttps httpsAPI
Server
firewall
Ballot Server
firewall
Station Server
https
MySQL ballot data
❌
![Page 31: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/31.jpg)
Users are notified immediately when they are successfully verified
![Page 32: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/32.jpg)
![Page 33: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/33.jpg)
client server
![Page 34: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/34.jpg)
client server
Got the SMS yet?
![Page 35: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/35.jpg)
client server
Not yet!Come back in 30
seconds.Got the SMS yet?
![Page 36: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/36.jpg)
![Page 37: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/37.jpg)
client server
![Page 38: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/38.jpg)
client server
Got the SMS yet?
![Page 39: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/39.jpg)
client server
hold on…Got the SMS yet?
![Page 40: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/40.jpg)
client server
hold on…Got the SMS yet?
SMS
![Page 41: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/41.jpg)
client server
Got the SMS yet?
SMS
Okay! You are verified.
![Page 42: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/42.jpg)
• Tornado Web Server and Async IO
• One thread can handle multiple clients at the same time
• Keep client connection opens
• Server subscribes to SMS events through redis
• Very fast response
![Page 43: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/43.jpg)
Deploymenta mix of Puppet+CloudFormation
![Page 44: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/44.jpg)
• Cloud infrastructure created with CloudFormation
• Make use of EC2 Auto-scaling and Multi-AZ for high availability
• Application installed automatically when server start
• Automatically configured by Puppet Master
![Page 45: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/45.jpg)
Design highlights
![Page 46: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/46.jpg)
API Design
• Client sends/receives
• Clear separation between client and server code
• Rapid client development
• Makes server simple
![Page 47: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/47.jpg)
API Design
• Client sends/receives
• Clear separation between client and server code
• Rapid client development
• Makes server simple
![Page 48: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/48.jpg)
Stateless Design
• Avoid storing session state on server
• Session state exchanged with client in encrypted form
• Good for privacy
• Good for server performance and operation
![Page 49: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/49.jpg)
Duplicate votes
• Duplicate vote checked at the very last step
• Vote is recorded if not duplicate
• Minimise participation checking by attacker
• Makes web server simple
• Database access restricted to ballot server
![Page 50: PopVote - University of Hong Kong · PDF filefabric flask boto pycrypto sqlalchemy itsdangerous pycaptcha uwsgi ... developing popvote ... CloudFormation](https://reader031.vdocument.in/reader031/viewer/2022030503/5ab011d57f8b9adb688e72d7/html5/thumbnails/50.jpg)
Further Improvement
• Diversify choice of cloud providers
• Application containment using Docker
• Consolidate server resources