Lecture IV: REST Web Service with Google App Engine
CS 4593Cloud-Oriented Big Data and Software Engineering
Outline• Course Project• Google App Engine• Google Development Console• Google Custom Search API
2
Course Project• Topic
– Keyword trends analysis services for a certain topic• Platform
– Google App Engine– Any programming language supported by GAE
• Collaboration– Work individually– Hopefully you are not working on the same idea, but it
is fine if you happen work on one by accident
3
Course Project• Requirements
– Develop a web service based on Google Search API to follow the weekly trends of keywords on a certain topic
– Search one or multiple websites each day with Google Search API free quota and record the query results during a period of time (hopefully more than a week)
– Perform statistics on query results to provide daily keywords frequency and the trends of the week
4
Course Project• Requirements
– Provide at least two web services: one for the keyword frequency of each day, and the other for trend of the week
– Provide output as either JSON or XML – The developed web service should be deployed on
Google App Engine
5
Course Project• Advanced Requirements
– You are going to choose the topic and the website to search, but the results should be meaningful (searching a static website for a week will not give you any different results)
– Provide HTML presentation for the web services
6
Course Project• Additional Information
– Feel free to use additional free web services to support your task
– Searching a more volatile website often provides you more interesting results, e.g., searching "Obama" in CNN or NY Times.
7
Course Project• Evaluation
– fetching and recording Google Search API results (2.5 points)
– extracting keywords (2.5 points)– presentation of daily results (2.5 points)– presentation of weekly results (2.5 points)– The advanced features will be evaluated by quality.
8
Course Project• Deliverables
– Demo– Code– Commends and User Guide– Blackboard
• Due date– Demo: Oct. 5th– Code: Oct. 4th
9
Outline• Course Project• Google App Engine• Google Development Console• Google Custom Search API
10
Google App Engine: Overview• Google App Engine (GAE) is a PaaS cloud
computing platform for developing and hosting web applications in Google-managed data centers.
• Google App Engine lets you run web applications on Google's infrastructure.– Easy to build. – Easy to maintain.– Easy to scale as the traffic and storage needs grow
11
infrastructure vs. platform - What is “The Platform”?
Platform: same for all applications
Libraries: shared by multiple applications
Application-specific code
infrastructure: hidden by platform
Scaling
• Low-usage apps: many apps per physical host• High-usage apps: multiple physical hosts per app• Stateless APIs are trivial to replicate• Datastore built on top of Bigtable; designed to
scale well– Abstraction on top of Bigtable– API influenced by scalability
• No joins• Recommendations: denormalize schema; precompute joins
13
Google App Engine: Overview• Free
– 1GB storage– 1 virtual machine instance– 5 million page views / month– 10 applications / Google account
14
Preserving Fairness Through Quotas• Everything an app does is limited by quotas, for
example:– request count, bandwidth used, CPU usage,
datastore call count, disk space used, emails sent• If you run out of quota that particular operation is
blocked (raising an exception) until replenished
15
Preserving Fairness Through Quotas• Free quotas are tuned so that a well-written app
(light CPU/datastore use) can survive • The point of quotas is to be able to support a very
large number of small apps (analogy: baggage limit in air travel)
• Large apps can request raised quotas• Three types of quotas
– Free Quota– Billed Limits– Safety Limits
16
Google App Engine: Programming Languages
• Java– App Engine runs JAVA apps on a JAVA 7 virtual
machine (currently supports JAVA 6 as well).– Uses JAVA Servlet standard for web applications:
• WAR (Web Applications ARchive) directory structure.• Servlet classes• Java Server Pages (JSP)• Static and data files• Deployment descriptor (web.xml)• Other configuration files
17
Google App Engine: Programming Languages
• Python– Uses WSGI (Web Server Gateway Interface)
standard.– Python applications can be written using:
• Webapp2 framework• Django framework• Any python code that uses the CGI (Common Gateway
Interface) standard
18
Google App Engine: Programming Languages
• PHP– Local development servers are available to
anyone for developing and testing local applications.
– Only whitelisted applications can be deployed on Google App Engine. (https://gaeforphp.appspot.com/).
19
Google App Engine: Programming Languages
• Go– Go is an Google’s open source programming
environment.– Tightly coupled with Google App Engine.– Applications can be written using App
Engine’s Go SDK.
20
Google App Engine: Data Storage• App Engine Data Store
– NoSQL schema-less object based data storage, with a query engine and atomic transactions.
– Data object is called a “Entity” that has a kind (~ table name) and a set of
– properties (~ column names).– JAVA JDO/ JPA interfaces and Python datastore
interfaces.
21
Google App Engine: Data Storage• Google Cloud Store
– RESTful service for storing and querying data.– Fast, scalable and highly available solution. – Provides Multiple layers of redundancy. All data is
replicated to multiple data centers.– Provides different levels of access control.– HTTP based APIs.
22
Google App Engine: Services• Google Services
– URL Fetch– Mail– Memcache
23
Google App Engine: Security• Sandboxes
– All hosted applications run in a secure environment that provides limited access to the underlying operating system.
– Sandbox isolates the application in its own secure, reliable environment that is independent of hardware, operating system and physical location of a web server.
24
Google App Engine: Security• Sandboxes: Limitations
– Application can only interact with other computers over internet using URL fetch, email, and HTTP/ HTTPS requests on the standard ports
– Applications cannot write to local file system in any of the runtime environments.
– Application code runs only in response to a web request, a queued task or a scheduled task and must return the response data within 60 seconds.
25
Google App Engine: Demo• Create a project on Google App Engine
– At https://console.developers.google.com/project • Initialize the environment of Restlet + eclipse +
GAE– Tutorial at– http://restlet.com/technical-resources/restlet-framewor
k/guide/2.3/introduction/first-steps/first-application– Download Restlet-GAE, GAE SDK, and Eclipse-GAE
plugin
26
Google App Engine: Demo• Deployment of Applications
– Follow https://console.developers.google.com/start/appengine?_ga=1.67402004.429901446.1441394923
• Google Search API– Generate your search engine– Manage API usages– https://cse.google.com/manage/all
27
Google Map API• Provide Map Services
– Http: http://maps.googleapis.com/maps/api– Android– IOS– JavaScript
• Parameters– Center– Zoom– Size– Language
28
Google Map API• Pricing
– Free Quota: 2500 / day– Bill Quota: 1000 / $0.5– Safety Quota: 100, 000 / day
29
Google Place API• Searching for places around certain map
location• Usage Example
– Finding all food places within 500 miles from (long,lat)
– https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=long,lat&radius=500&types=food&name=cruise&key=API_KEY
• Place ID– A ID of a place that can be used later (e.g., Costco at
I 10 @ UTSA Boulevard)30
Google Place API• Parameters
– Type: food, airport, hospital, …– https://
developers.google.com/places/supported_types• Pricing
– Free quota: 1000/day– Free up to 150,000/day with credit card info
31
{ "geometry" : { "location" : { "lat" : -33.870775, "lng" : 151.199025 } }, … "name" : "Rhythmboat Cruises", "opening_hours" : { "open_now" : true }, "photos" : […], "place_id" : "ChIJyWEHuEmuEmsRm9hTkapTCrk", "scope" : "GOOGLE", "types" : [ "travel_agency", "restaurant", "food", "establishment" ], "vicinity" : "Pyrmont Bay Wharf Darling Dr, Sydney" },
Google Place API: response
Google+ API• Intro• Usages• Parameters• Pricing
33
Google+ API• A series of APIs working with Google+
– People: Get public profiles with query or user ID– Activities: List activities of a user or searching
activities with query– Comments: List comments of an activity or searching
comments with query– Moments: Retrieve, insert or delete moments
• Usage– https://www.googleapis.com/plus/v1/– 10,000/day free quota
34
Google Translation API• Translate strings to other languages• Parameters
– source– target– q(query)
• Pricing– $20 for 1M characters– 50M / day limit
35
Youtube Data API• Provide various supports to retrieve and manage
data from Youtube• Some important services
– Video: List, search, Insert, Delete, Rate, …– Playlist: Insert, delete, List, …– Captions: List, insert, update, download, delete, …
36
Youtube Data API• Parameters
– Part– videoCategoryId– myRating
• Pricing– Unit based quota calculation– Services with different parameters may cost different
quota– https://developers.google.com/youtube/v3/
determine_quota_cost– 50M unit / day, and 30K unit / second, always free
37
Facebook Graph API• A web API library for Facebook social network• Provide support for various platforms
– RESTful– IOS– Android– JavaScript– PHP
38
Facebook Graph API• Usages
– Provide similar functions to google+ APIs– List friends, List events, Fetch comments, …– Uploading videos, photos, …
• Pricing– Free– Need to contact Facebook for more than 100M API
calls per day
39
MediaWiki API• API to get information from Wikipedia
programmatically– Entry: https://en.wikipedia.org/w/api.php
• Parameters– Action: query, modules– Titles: specify the query– Limit: max results– Section: retrieve from which section
• Pricing– Free: with unofficial limits
40
Bing API• Search API from Microsoft• Root:
– https://api.datamarket.azure.com/Bing/Search/v1/• Parameters
– Query– NewsCategory– Sources: web, image, video, news, …
41
Bing API• Pricing
– 5000 / month free– After it: 10K/month for $20
42
Summary• Google App Engine• Application Deployment on Google App Engine• Popular free Web Services
43