mongola - cloud foundry
DESCRIPTION
Spring and Ruby MongoDB Cloud Foundry Deep diveTRANSCRIPT
© 2012 VMware, Inc. All rights reserved
MongoDB App DuetRuby and Java on Cloud Foundry
By Monica Wilkinson and Josh LongCloud Foundry, Developer Relations
MongoLA 2012Thursday, January 19, 12
CONFIDENTIAL
About Monica Wilkinson
2
Loves the web and data portability.
Developer Advocate @ Cloud Foundry12 years development experience.
Last 5 years in Social WebOpen Web Standards Advocate
Thursday, January 19, 12
CONFIDENTIAL
About Josh Long
3
Spring Developer [email protected]@springsource.com
this is important!
SpringSource.org/rooFree Book!
Thursday, January 19, 12
CONFIDENTIAL4
Cloud Foundry
Thursday, January 19, 12
CONFIDENTIAL
About Cloud Foundry
5
The first Open PaaS
Multi(n) Languages, Frameworks, Services & Clouds
Open Source
Thursday, January 19, 12
CONFIDENTIAL
STEP 1 - Get a Cloud Foundry account
https://my.cloudfoundry.com/signup/MongoLA
6
Thursday, January 19, 12
CONFIDENTIAL
STEP 2 - Download VMC
•sudo gem install vmc
•vmc login <username>
7
Thursday, January 19, 12
CONFIDENTIAL8
STEP 3: Push your app to Cloud Foundryvmc push --runtime=ruby19
Thursday, January 19, 12
© 2012 VMware, Inc. All rights reserved
You are doneService gets created and bound
Code gets pushed
App is Live immediately at the url you requested
Thursday, January 19, 12
CONFIDENTIAL
Every day development Debugging and accessing the data locally
• Caldecott --> Service tunneling. Access your Cloud Foundry service as if it was local.
10
Thursday, January 19, 12
CONFIDENTIAL
Tunneling
11
gem install caldecott
vmc tunnel <mongodb>
Thursday, January 19, 12
CONFIDENTIAL
Using your favorite tools
12
Thursday, January 19, 12
CONFIDENTIAL13
Thursday, January 19, 12
© 2012 VMware, Inc. All rights reserved14
Why MongoDB?
Thursday, January 19, 12
CONFIDENTIAL15
Data Access Challenge #1: Scale Horizontally
Thursday, January 19, 12
CONFIDENTIAL16
Data Access Challenge #2: Heterogeneous
Thursday, January 19, 12
CONFIDENTIAL17
New demands on data access
•... until we needed inexpensive horizontal scaling for some large web based applications ...
•... and we needed to deploy our apps in the cloud ...
* image courtesy of Bitcurrent
Thursday, January 19, 12
CONFIDENTIAL18
NoSQL offers several data store categories
ColumnKey-Value Document Graph
Redis, Riak
Cassandra,HBase
MongoDB Neo4J
Thursday, January 19, 12
CONFIDENTIAL
About Mongo DB
19
1. No-SQL database
2. Stores JSON-style documents
3. Horizontally scalable
4. Full Indexing Support
5. Open Source
6. Great Community(You!)
development : {
tools: many,
language_support: superb,
agility: high
},
production: {
speed: fast
fault_tolerance: true
scalability: high
=
Thursday, January 19, 12
CONFIDENTIAL20
NoSQL offers several data store categories
ColumnKey-Value Document Graph
MongoDB (who cares about the rest?)
Thursday, January 19, 12
CONFIDENTIAL21
Mongo and Ruby Deep Dive
Thursday, January 19, 12
CONFIDENTIAL
Integration with Box(.net)
22
Thursday, January 19, 12
CONFIDENTIAL23
• Requirement: Clone sample Box app for developers into their own Cloud Foundry account so they can learn how to use the BOX Api.
Box Sample App Creation Wizard
Thursday, January 19, 12
CONFIDENTIAL24
• No command to download all the files for an app or copy it to another account.
Challenges
•Solution: The Cloud•Get the code for an app from: GitHub•Use vmc gem to upload the contents to your desired cloud
Thursday, January 19, 12
CONFIDENTIAL
VMC as a gem
Can be used to authenticate users and exchange their credentials for an API token
Edit Applications and Services from your code•Files
•Environment Variables
Start and Stop Apps
25
Thursday, January 19, 12
CONFIDENTIAL
How we used VMC on this project
26
def create() begin @vmcclient.create_app(@manifest["name"], @manifest) rescue RuntimeError => ex if (ex.message =~ /Error 701/) new_candidate = @generator.next unless new_candidate.nil? change_name! new_candidate create else # Format is "Error #{parsed_body[:code]}: #{desc}" raise "App Url: #{@uri} is already taken" end else raise ex end end end
def delete() @vmcclient.delete_app(@manifest["name"]) end
def copy_code() @vmcclient.upload_app(@manifest["name"], @app_meta.build!) end
Thursday, January 19, 12
CONFIDENTIAL
More Challenges
External Models we didn’t control
• Cloud Foundry Apps: Files, Runtime, Services, Env Vars, etc
• GitHub Repositories: Files, Commits, branches, tags
• Solution
• Use an Object Document Mapper. I chose Mongoid
• Easy to add new objects and properties
• No Migrations !
27
Thursday, January 19, 12
CONFIDENTIAL28
Thursday, January 19, 12
CONFIDENTIAL
Things you should know about Mongoid Project Page - http://mongoid.org.
Features:• Uses Active Model (Dirty Tracking, Mass Assignment
Security, Validations, Callbacks, JSON/XML Serialization)• Indexing• Supports identity map, replica sets and sharding• Custom field serialization
• A Range in Ruby 1...3 could be stored in MongoDB as a Hash { :min => 1, :max => 3 }
• Localization Support:(Translated Fields, i18n Fallbacks)• Inheritance
29
Thursday, January 19, 12
CONFIDENTIAL
How we used Mongoid
30
module CloudFoundry class AppInfo include TmpZip include Mongoid::Document
embeds_many :app_clone_requests belongs_to :repo, :class_name => "GitHub::RepositorySnapshot"
field :display_name, :type => String field :description, :type => String field :instances, :type => Integer, :default => 1 field :memory, :type => Integer, :default => 128 field :runtime, :type => String field :framework, :type => String field :env_vars, :type => Hash field :thumb_url, :type => String field :browseable, :type => Boolean field :cloneable, :type => Boolean field :starting_url, :type => String
index :display_name, :unique => true
validates_presence_of :display_name, :runtime, :framework validates_presence_of :repo, :if => :cloneable
Thursday, January 19, 12
CONFIDENTIAL
How we used Mongoid
31
module GitHub class RepositorySnapshot include TmpZip include Mongoid::Document
field :url, :type => String field :name, :type => String field :parent, :type => String field :branch, :type => String, :default => 'master' field :tag, :type => String field :commit, :type => String
index :url, :unique => true
validates_presence_of :url, :name, :parent, :commit, :branch
def url=(value) parts = value.gsub(/https\:\/\/github.com\//, '').split('/') if (parts.length == 2) write_attribute(:parent, parts[0]) write_attribute(:name, parts[1]) else write_attribute(:parent, nil) write_attribute(:name, nil) end write_attribute(:url, value) end
Thursday, January 19, 12
CONFIDENTIAL32
Getting your ruby app to use the proper module CloudFoundry
class Mongo
def self.config
Mongoid.configure do |config|
conn_info = nil
dbname = 'db'
if ENV['VCAP_SERVICES']
services = JSON.parse ENV['VCAP_SERVICES']
services.each do |service_version, bindings|
mongo_binding = bindings.find {|binding| binding['label'] =~ /mongo/i}
conn_info = mongo_binding['credentials'] if mongo_binding
end
raise "ERROR - Could not find connection info for mongo" unless conn_info
else
conn_info = {'hostname' => 'localhost', 'port' => 27017}
dbname = "gallery_db"
end
@@cnx = Mongo::Connection.new conn_info['hostname'], conn_info['port'], :pool_size => 5, :timeout => 5
db = @@cnx[dbname]
if conn_info['username'] and conn_info['password']
db.authenticate conn_info['username'], conn_info['password']
end
config.master = db
end
end
end
end
Thursday, January 19, 12
CONFIDENTIAL33
Demo
Thursday, January 19, 12
CONFIDENTIAL
Durran, creator of Mongoid detailed for us:
34
Full atomic update support out of the box for *all* MongoDB atomic operations, either explicitly or handled under the covers by Mongoid itself with support for embedded n levels deep. ie:
model = Model.find(id)
model.field = “value”
model.relations.build(field: “value”)
model.save
#=> Does
{ field: { “$set” : “value” },
relations: { “$push” : { field: “value” }}} for you.
*or things like*
model.add_to_set(field: “value”)
Thursday, January 19, 12
CONFIDENTIAL
Durran, creator of Mongoid detailed for us
35
“Smart memory management during iteration and working with relations: Mongoid never loads everything into memory unless you specifically want it to. It can handle working with millions of documents without putting stress on the server/RAM.”
Thursday, January 19, 12
CONFIDENTIAL
More Challenges Make app fast.
• Downloading and copying files from app to app is not cheap
• Solution• Avoid downloading the build if it has not changed
• Local cache using Mongo GridFS
36
Thursday, January 19, 12
CONFIDENTIAL
Working with GridFS Saving and Reading filesOnce you have a Grid object, you can start saving data to it.
@db = Mongo::Connection.new.db('social_site')@grid = Grid.new(@db)
# On this Cloud Foundry App@grid = Mongo::GridFileSystem.new(Mongoid.database)
# Saving IO data and including the optional filenameimage = File.open("me.jpg")id = @grid.put(image, :filename => "me.jpg")Grid#put returns an object id, which you can use to retrieve the file:
# Get the file we savedimage = @grid.get(id)
Deleting filesDeleting a file is as simple as providing the id:
@grid.delete(id2)
37
Thursday, January 19, 12
CONFIDENTIAL38
Spring Data on Cloud Foundry Deep Dive
Thursday, January 19, 12
CONFIDENTIAL39
Spring Frameworkbuilt-in data access support
•Transaction abstractions
•Common data access exception hierarchy
•JDBC - JdbcTemplate
•ORM - Hibernate, JPA support
•OXM - Object to XML mapping
•Serializer/Deserializer strategies (Spring 3.0)
•Cache support (Spring 3.1)
Thursday, January 19, 12
CONFIDENTIAL40
Spring Data Building Blocks
•Low level data access APIs✓MongoTemplate, RedisTemplate ...
•Object Mapping (Java and GORM)
•Cross Store Persistence Programming model
•Generic Repository support
•Productivity support in Roo and Grails
Thursday, January 19, 12
CONFIDENTIAL41
Spring Data Document Mongo
•MongoTemplate
•MongoConverter interface for mapping Mongo documents
•SimpleMongoConverter for basic POJO mapping support
•Leverage Spring 3.0 TypeConverters and SpEL
•Exception translation
•Advanced Mapping
•Annotation based (@Document, @Id, @DbRef)
•MongoRepository
•Built on Hades support for JPA Repositories
Thursday, January 19, 12
CONFIDENTIAL42
Simple Domain Class
Thursday, January 19, 12
CONFIDENTIAL43
Mongo TemplateDirect Usage of the Mongo Template:
Thursday, January 19, 12
CONFIDENTIAL43
Mongo TemplateDirect Usage of the Mongo Template:
Insert into “Person”
Collection
Thursday, January 19, 12
CONFIDENTIAL43
Mongo TemplateDirect Usage of the Mongo Template:
findOne using query: { "name" : "Joe"}
in db.collection: database.Person
Thursday, January 19, 12
CONFIDENTIAL43
Mongo TemplateDirect Usage of the Mongo Template:
Dropped collection [database.person]
Thursday, January 19, 12
CONFIDENTIAL44
Generic Repository Interface for generic CRUD operations on a repository for a specific type
Thursday, January 19, 12
CONFIDENTIAL45
Paging and Sorting RepositoryExtends “CrudRepository” Paging and Sorting Repository:
Thursday, January 19, 12
CONFIDENTIAL45
Paging and Sorting RepositoryExtends “CrudRepository”
Usage:
Paging and Sorting Repository:
Thursday, January 19, 12
CONFIDENTIAL46
Custom Repository Custom Repository:
Thursday, January 19, 12
CONFIDENTIAL46
Custom Repository Custom Repository:
Keyword Sample Logical result
GreaterThan findByAgeGreaterThan(int age) {"age" : {"$gt" : age}}
LessThan findByAgeLessThan(int age) {"age" : {"$lt" : age}}
Between findByAgeBetween(int from, int to) {"age" : {"$gt" : from, "$lt" : to}}
NotNull findByFirstnameNotNull() {”firstname" : {"$ne" : null}}
Null findByFirstnameNull() {”firstname" : null}
Like findByFirstnameLike(String name) "firstname" : firstname} (regex)
Keywords :
Thursday, January 19, 12
CONFIDENTIAL47
JPA and MongoDB JPA “Customer” with a “SurveyInfo” Document
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfo
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfoCreate Customer
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfo
Create SurveyInfo
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfo
Assign Survey to Customer
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfo
Save
Thursday, January 19, 12
CONFIDENTIAL48
Using a Cross-StoreSaving a Customer with a SurveryInfo
Save
Mongo Document:
Thursday, January 19, 12
CONFIDENTIAL
Accessing Services Bound to Cloud Foundry
49
private String mongoDatabaseServiceName = "survey-mongo";
@Beanpublic CloudEnvironment cloudEnvironment() { return new CloudEnvironment();}
@Beanpublic MongoServiceInfo mongoServiceInfo() { return cloudEnvironment().getServiceInfo( mongoDatabaseServiceName, MongoServiceInfo.class);}
@Beanpublic MongoDbFactory mongoDbFactory() { MongoServiceCreator msc = new MongoServiceCreator(); MongoDbFactory db = msc.createService(mongoServiceInfo());}
Thursday, January 19, 12
CONFIDENTIAL
Accessing Services Bound to Cloud Foundry
50
@Inject private Mongo mongo ;
@Inject private MongoTemplate mongoTemplate;
Thursday, January 19, 12
CONFIDENTIAL51
Demo 2
Thursday, January 19, 12
© 2012 VMware, Inc. All rights reserved
Thank YouCloudFoundry Spring Source Samples:
http://bit.ly/cloudfoundry-samples
http://cloudfoundry.com
Questions: @cloudfoundry
•Josh -> @starbuxman or [email protected]
•Monica -> @ciberch or [email protected]
Thursday, January 19, 12