app engine dev days dc 20091026
DESCRIPTION
Jonathan Blocksom's talk on Google App Engine at Stack Overflow Dev Days DC in 2009.TRANSCRIPT
Google App EngineJonathan Blocksom
Stack Overflow DevDays DC, 10/26/09
Submit a Question: http://tinyurl.com/gae-dd09
Attention Conservation Notice• Jonathan Blocksom
[email protected]@jblocksom
Software EngineerGooglePublic Sector Projects Team
Submit a Question: http://tinyurl.com/gae-dd09
We’re lonely
Open positions at Google in DC
Submit a Question: http://tinyurl.com/gae-dd09
App Engine!• Launched April 2008
• Paid Use Feb 2009
• Java support May 2009
• Latest Release (1.2.7) October 2009
• Incoming Email
• Previous Release (1.2.5) September 2009
• XMPP Support
Submit a Question: http://tinyurl.com/gae-dd09
Your Scalable Website?• Brad Fitzpatrick describes LiveJournal in ’07:
http://danga.com/words/
LiveJournal Backend: Today(Roughly.)
User DB Cluster 1
uc1a uc1b
User DB Cluster 2
uc2a uc2b
User DB Cluster 3
uc3a uc3b
User DB Cluster N
ucNa ucNb
Job Queues (xN)
jqNa jqNb
Memcached
mc4
mc3
mc2
mcN
...
mc1
mod_perl
web4
web3
web2
webN
...
web1
BIG-IP
bigip2
bigip1perlbal (httpd/proxy)
proxy4
proxy3
proxy2
proxy5
proxy1
Global Database
slave1
master_a master_b
slave2 ... slave5
MogileFS Database
mog_a mog_b
Mogile Trackers
tracker3tracker1
Mogile Storage Nodes
...
sto2
sto8
sto1
net.
djabberd
djabberd
djabberd
gearmand
gearmand1
gearmandN
“workers”
gearwrkN
theschwkN
slave1 slaveN
3
2007
Submit a Question: http://tinyurl.com/gae-dd09
App Engine Scalable Website
App EngineRequest Response
Submit a Question: http://tinyurl.com/gae-dd09
What’s Scalable?• Google Moderator on App Engineused in
March 2009 for White House Town Hall meeting
Submit a Question: http://tinyurl.com/gae-dd09
Scalable
Submit a Question: http://tinyurl.com/gae-dd09
Google Moderator team didn’t...
• Provision a server
• Install and configure an OS
• Install and configure a web server
• Install and configure a web framework
• Install and configure a database
• Install and configure a load balancer
• Create user accounts
• Crash in the middle of any of the above
Submit a Question: http://tinyurl.com/gae-dd09
App Engine Architecture
Submit a Question: http://tinyurl.com/gae-dd09
Static Content• Served by Google
Static Content Servers
• Very Efficient
Submit a Question: http://tinyurl.com/gae-dd09
Dynamic Content• Java xor Python
• Sandboxed
• No file writes
• No sockets
• No spawning threads/processes
• Smart about caching
• Fairly standard CGI Framework
• Measured and Instrumented
Submit a Question: http://tinyurl.com/gae-dd09
Datastore 1 / 3• Based on BigTable,
transitioning to MegaStore
• ACID Compliant
• 2008 Talk on Implementation:http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore
Submit a Question: http://tinyurl.com/gae-dd09
Datastore 2 / 3• Typical types:
Int, String, DateTime, GeoPt, PhoneNumber, PostalAddress, Email, Link, Blob...
Submit a Question: http://tinyurl.com/gae-dd09
Datastore 3 / 3• GQL:
Google Query Language
• If it’s not SQL, it must be good!
SELECT [* | __key__] FROM <kind> [WHERE <condition> [AND <condition> ...]] [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]] [LIMIT [<offset>,]<count>] [OFFSET <offset>]
<condition> := <property> {< | <= | > | >= | = | != } <value> <condition> := <property> IN <list> <condition> := ANCESTOR IS <entity or key>
Submit a Question: http://tinyurl.com/gae-dd09
Memcache• Arbitrary key/value cache
• Similar API to memcached
• get, set
• get_multi, set_multi for speed
• Namespace Support
• 1MB limit per entry
• Free Quota 8.6M/day, 48K/min (800/sec)
Submit a Question: http://tinyurl.com/gae-dd09
Users & Authentication• @gmail.com addresses
• Apps for Domain
• Admin Privileges
Submit a Question: http://tinyurl.com/gae-dd09
URL Fetch• Load URL
• Asynchronous support
• HTTP / HTTPS
• Max 10 second response (default 5)
• Max 1MB data
Submit a Question: http://tinyurl.com/gae-dd09
Even More...• Scheduled Tasks
• Task Queue
• Image Manipulation
• XMPP
• ...
cron jobs
XMPP
Submit a Question: http://tinyurl.com/gae-dd09
Web Hooks Pattern• Incoming data / operations to perform are
URLs with POST data
• Handled same asweb request
Scheduled Tasks
Task Queue tasks
Incoming Email
XMPP Messages
Submit a Question: http://tinyurl.com/gae-dd09
Templates?• Treetop Wants to know
Submit a Question: http://tinyurl.com/gae-dd09
Templates!• Django Templates built in
Variable substitution, function calls, loops, etc
• Can add your own Python temlpate lib
Submit a Question: http://tinyurl.com/gae-dd09
Let’s Review!
Submit a Question: http://tinyurl.com/gae-dd09
Developing w/ App Engine• Local test server
• Write your code, html, etc on local machine
• Send code to App Engine
• Monitor in App Engine Admin Console
Submit a Question: http://tinyurl.com/gae-dd09
Dashboard Demo• http://appengine.google.com/
Submit a Question: http://tinyurl.com/gae-dd09
What have we done?• Served static web pages
• Served dynamic web pages
• Stored data (cached & persistent)
• Ran a cron Job
• Authenticated users
• We could have...
• Scheduled a task, sent or received email, loaded a URL, modified an image,rocked out with XMPP
Submit a Question: http://tinyurl.com/gae-dd09
Quotas• 34 global quotas
in Dashboard
• Individual quotas:request sizetimeamount of data
• Exceptions thrown when hit
• Free and Paid
Submit a Question: http://tinyurl.com/gae-dd09
Quota Rule of Thumb• Less than 1000 of everything
• The Zen of Quotas:Meet the quotas and you will be scalable
Submit a Question: http://tinyurl.com/gae-dd09
All For Good• Volunteering Opportunity Search Engine
• Started March 2009 in cooperation with White House, CNCS, Craigslist, Google...
• Released June 2009
• Open source, released under Apache Licensehttp://code.google.com/p/allforgood/
• Front end at http://allforgood.orgAPI powers search results on serve.gov
Submit a Question: http://tinyurl.com/gae-dd09
Tech Overview• Partners provide listings in XML or Google Spreadsheets
• AARP, United Way, AmeriCorp, HandsOnNetwork, United Way
• AFG indexes feeds, stores outside GAE
• Initially Google Base
• Now SOLR on private server
• GAE front end
• API for external apps, partners
• Integration with Google Maps, FriendConnect, Facebook
Submit a Question: http://tinyurl.com/gae-dd09
Tech Overview
App Engine
Search Backend(Base or SOLR)
memcache
Google Code SVN
Volunteering Opportunities
Volunteering Opportunities
Volunteering OpportunitiesCrawler
...Resu
lts
Queries
Listings
Static Content
User or API
Submit a Question: http://tinyurl.com/gae-dd09
Why A Different Back End?• No full text search in App Engine Datastore
Submit a Question: http://tinyurl.com/gae-dd09
On The Promiscuous Nature of GAE
• GAE plays well with others
• Do full text search with something else!
• Serve large files with S3!
• Serve videos with YouTube!
Submit a Question: http://tinyurl.com/gae-dd09
Challenges with AFG• Parsing questionable data
• Integration w/ Social Networks
• Google Base is an Odd Duck
• I Was New And Had No Clue What Was Going
• App Engine Outages
Submit a Question: http://tinyurl.com/gae-dd09
Q & A
Submit a Question: http://tinyurl.com/gae-dd09
More Information• App Engine Documentation
• Google I/O 2008 & 2009
• 2008 for intro, 2009 for deep dives / Java
• http://code.google.com/events/io/2009/sessions.html#appengine“App Engine Nitty-Gritty”“Spark Plug to Drive Train”
• Google Group
• Sampleshttp://code.google.com/p/google-app-engine-samples/