using spring with nosql databases (springone china 2012)

Using Spring with NoSQL databasesChris Richardson, Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardson chris.richardson@springsource.com http://plainoldobjects.com/

Presentation goal

NoSQL databases: what, why and how

How Spring Data simplifies the development of NoSQL applications

About Chris

(About Chris)

About Chris()

About Chris

http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/

vmc push About-Chris

Developer Advocate

Signup at http://cloudfoundry.com

Agenda

• Why NoSQL?

• Overview of NoSQL databases

• Introduction to Spring Data

• Using Spring Data for Redis

• Using Spring Data for Mongo

• Deploying on Cloud Foundry

Relational databases are great...

• SQL

• High-level

• Sorting

• Aggregation

• ACID semantics

• Well supported

• JDBC

• Hibernate/JPA

• Spring

• Well understood

• Developers

• Operators

... but they have limitations

• Object/relational impedance mismatch

• Complicated to map rich domain model to relational schema

• Difficult to handle semi-structured data, e.g. varying attributes

• Schema changes

• Extremely difficult/impossible to scale

• Poor performance for some use cases

Solution: Spend Money

http://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG

http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#

• Hire more DevOps• Use application-level sharding• Build your own middleware• …

Solution: Use NoSQL

Benefits• Higher performance• Higher scalability• Richer data-model• Schema-less

Drawbacks• Limited transactions• Relaxed consistency• Unconstrained data

Growing in popularity…

But don’t get too excited

Solution: Use NewSQL• Relational databases with SQL and ACID transactions

AND• New and improved architecture - designed for modern hardware• Radically better scalability and performance

• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...

Future = multi-paradigm data storage for enterprise applications

IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg

Agenda

• Why NoSQL?

• Advanced key-value store

• Written in C

• Very fast, e.g. 100K reqs/sec

• Optional persistence

• Transactions with optimistic locking

• Master-slave replication

• Sharding using client-side consistent hashing

... ...

Using Redis (via CLI)redis 127.0.0.1:6379> set foo 1OKredis 127.0.0.1:6379> get foo"1"

Datatypes:•Strings•Hashes•Maps•Lists•Sets•Sorted sets

redis 127.0.0.1:6379> sadd myset a(integer) 1redis 127.0.0.1:6379> sadd myset b(integer) 1redis 127.0.0.1:6379> smembers myset 1) "a"2) "b"redis 127.0.0.1:6379> srem myset a(integer) 1redis 127.0.0.1:6379> smembers myset1) "b"

Redis use cases• Replacement for Memcached

• Session state

• Cache of data retrieved from system of record (SOR)

• Replica of SOR for queries needing high-performance

• Handling tasks that overload an RDBMS

• Hit counts - INCR

• Most recent N items - LPUSH and LTRIM

• Randomly selecting an item – SRANDMEMBER

• Queuing – Lists with LPOP, RPUSH, ….

• High score tables – Sorted sets and ZINCRBY

• …

MongoDB

• Document-oriented database

• JSON-style documents: objects, lists, primitives

• Schema-less

• Transaction = update of a single document

• Rich query language for dynamic queries

• Geospatial queries

• Grid FS provides file storage

• Very fast, asynchronous writes

• Highly scalable and available

Server

Database: Food To Go

Collection: Restaurants

Data model = Binary JSON documents

{ "name" : "TGI Fridays", "type" : ”American", "serviceArea" : [ "94619", "94618" ], "openingHours" : [ { "dayOfWeek" : "Wednesday", "open" : 1730, "close" : 2230 } ], "_id" : ObjectId("4bddc2f49d1505567c6220a0")}

Sequence of bytes on disk è fast i/o

MongoDB CLI> r = {name: 'Ajanta'}> db.restaurants.save(r)> r{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r = db.restaurants.findOne({name:"Ajanta"}){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r.type= "Indian”> db.restaurants.save(r)> db.restaurants.update({name:"Ajanta"}, {$set: {name:"Ajanta Restaurant"}, $push: { menuItems: {name: "Chicken Vindaloo"}}})> db.restaurants.find(){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" :

"Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }> db.restaurants.remove(r.id)

MongoDB query by example{ serviceArea:"94619", openingHours: { $elemMatch : { "dayOfWeek" : "Monday", "open": {$lte: 1800}, "close": {$gte: 1800} } }}

DBCursor cursor = collection.find(qbeObject); while (cursor.hasNext()) { DBObject o = cursor.next(); … }

Find a restaurant that serves the 94619 zip code and is open at 6pm on a Monday

Mongos

Scaling MongoDBShard 2

Mongod(replica)

Mongod(master)

Mongod(replica)

Shard 1Mongod(replica)

Mongod(master)

Mongod(replica)

Mongos

Client

Config Server

mongod

MongoDB use cases

• Use cases

• High volume writes

• Complex data

• Semi-structured data

• Who is using it?

• Shutterfly, Foursquare

• Bit.ly Intuit

• SourceForge, NY Times

• GILT Groupe, Evite,

• SugarCRM

Other NoSQL databases

http://nosql-database.org/ lists 122+ NoSQL databases

Type Examples

Extensible columns/Column-oriented HbaseSimpleDB, DynamoDBCassandra

Graph Neo4j

Key-value Voldemort, Riak

Document CouchDbSorry if

I left

out you

r favori

Agenda

• Why NoSQL?

Spring Data is here to help

http://www.springsource.org/spring-data

NoSQL databases

Spring Data sub-projects

• Relational

• JPA

• JDBC Extensions

• NoSQL

• Redis

• Mongo

• HBase

• Neo4j

• Gemfire

• Lucene

• QueryDSL

• Big Data

• Hadoop

• HDFS and M/R

• Hive

• Pig

• Cascading

• Splunk

• Access

• REST

What you get

• Template classes that hide the boilerplate code

• Auto-generated (generic) repositories for some NOSQL databases

• Java ⇔ NoSQL mapping

• Cross Store Persistence

• Support in Roo

Get the book!

Agenda

• Why NoSQL?

Redis challenges

• Connection management: need to get and reliably close connections

• Data mapping: application objects ⇔ Redis binary/strings

• Multiple client libraries with gratuitously different APIs

Spring Data for Redis

• Low-level - RedisConnection(Factory)

• Supports Jedis, Jredis, Rjc and Srp

• Insulates client code from underlying library

• High-level - RedisTemplate

• Builds on RedisConnection(Factory)

• Connection management

• Pluggable Java ⇔ binary conversion

• Support classes:

• Collections-backed by RedisTemplate

• Atomic Counters

• Support for Redis pub/sub

Low-level API = RedisConnection(Factory)

Using RedisConnectionFactorypublic class LowLevelRedisTest {

@Autowired private RedisConnectionFactory redisConnectionFactory;

@Test public void testLowLevel() { RedisConnection con = null; try { con = redisConnectionFactory.getConnection();

byte[] key = "foo".getBytes(); byte[] value = "bar".getBytes(); con.set(key, value);

byte[] retrievedValue = con.get(key);

Assert.assertArrayEquals(value, retrievedValue);

} finally { if (con != null) { con.close(); } } }

Ugly byte arrays L

Library independent code J

Need to clean up L

Configuring RedisConnectionFactory

@Configurationpublic class RedisConfiguration {

@Value("${databaseHostName}") protected String databaseHostName;

@Bean public RedisConnectionFactory jedisConnectionFactory() { JedisConnectionFactory factory = new JedisConnectionFactory(); factory.setHostName(databaseHostName); factory.setPort(6379); factory.setUsePool(true); return factory; }

High-level API = RedisTemplate

• Builds on RedisConnection(Factory)

• Analogous to JdbcTemplate

• Parameterized type

• K - Key type

• V – Value type

• Handles Java Key/Value ⇔ Redis byte[]

• Maps Redis exceptions ⇒

DataAccessException

• StringRedisTemplate

• Extends RedisTemplate<String, String>

• Keys and values are Strings

Using StringRedisTemplate

public class RedisTemplateTest {

@Autowired private StringRedisTemplate stringRedisTemplate;

@Test public void testGetAndSet() { stringRedisTemplate.opsForValue().set("foo", "bar"); assertEquals("bar", stringRedisTemplate.opsForValue().get("foo")); } @Test public void testHashOps() { stringRedisTemplate.opsForHash().put("myHash", "myKey", "value");

assertEquals("value", stringRedisTemplate.opsForHash().get("myHash", "myKey"));

assertEquals(Collections.singleton("myKey"), stringRedisTemplate.opsForHash().keys("myHash"));

assertEquals(Collections.singletonMap("myKey", "value"), stringRedisTemplate.opsForHash().entries("myHash")); }

Converts between Strings and byte[]

Returns KV type specific interface

Configuring StringRedisTemplate

@Bean public RedisConnectionFactory jedisConnectionFactory() { … }

@Bean public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(factory); return template; }}

RedisTemplate: Java objects ⇔ binary data

• DefaultSerializer - defaults to JdkSerializationRedisSerializer

• KeySerializer

• ValueSerializer

• HashKeySerializer

• HashValueSerializer

StringRedisTemplate uses StringRedisSerializer

• Stores keys and values as Strings

Register serializers to override the default behavior

Converted to JSON by RedisTemplate

Redis-backed Collections

@Test public void testRedisSet() {

Set<String> mySet = new DefaultRedisSet<String>("mySet", stringRedisTemplate);

Assert.assertTrue(mySet.isEmpty());

mySet.add("a");

assertEquals(Collections.singleton("a"), stringRedisTemplate.opsForSet().members("mySet"));

The key

Redis Atomic Counters

Redis Pub/Sub - Consumer

@Bean public MyRedisSubscriber myListener() { return new MyRedisSubscriber(); } @Bean public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory redisConnectionFactory, MyRedisSubscriber myListener) { RedisMessageListenerContainer c = new RedisMessageListenerContainer();

c.setConnectionFactory(redisConnectionFactory); c.addMessageListener(new MessageListenerAdapter(myListener), new ChannelTopic("myChannel")); return c; }

public class MyRedisSubscriber { String message;

void handleMessage(String text) { this.message = text; } }

Redis Pub/Sub - Producer

public class RedisPubSubTest {

@Autowired private StringRedisTemplate stringRedisTemplate;

@Autowired private MyRedisSubscriber myListener; @Test public void testPubSub() throws InterruptedException { … stringRedisTemplate.convertAndSend("myChannel", "hello"); TimeUnit.SECONDS.sleep(4); Assert.assertEquals("hello", myListener.message); }}

Redis caching support

Template needs to (de)serialize K and VKVs = <prefix + K, V>

Sorted set of all keys for clear()

Agenda

• Why NoSQL?

MongoDB API usage patterns

• Create and store Mongo singleton

• Externalized server host, port etc.

• Inserts/Updates

• Map application POJO ⇒ DBObject

• mongo.getDatabase(…).getCollection(…)

• Partial document updates

• Configure asynchronous vs. synchronous writes

• Queries

• Construct query object

• mongo.getDatabase(…).getCollection(…)

• Iterate through Cursor

• Map DBObject ⇒ application POJO

⇒ Higher-level than JDBC but still

repetitive, …

Spring Data - MongoDB

• MongoTemplate

• Query, Criteria, and Update DSLs

• Generic repositories

• Querydsl integration

• Cross-store persistence

• GeoSpatial integration

• Map-Reduce integration

• GridFS support

MongoTemplatedatabaseNameuserIdPassworddefaultCollectionName

writeConcernwriteResultChecking

save()insert()remove()updateFirst()findOne()find()…

MongoTemplate

Mongo(Java Driver class)

<<interface>>MongoConvertor

write(Object, DBObject)read(Class, DBObject)

POJO ó DBObjectmappingSimplifies data access

Translates exceptions

MongoMappingConverter

Example entitypublic class Restaurant { private String id; private String name; private List<MenuItem> menuItems;

public Restaurant(String name) { this.name = name; … }

public class MenuItem { private String name; private double price;

public MenuItem(String name, double price) { this.name = name; this.price = price; }

Spring Data uses fields and non-default constructors

Example data access code

@Repositorypublic class RestaurantRepository {

@Autowired private MongoTemplate mongoTemplate; public void add(Restaurant restaurant) { mongoTemplate.save(restaurant); }

public List<Restaurant> findRestaurantsByName(String restaurantName) { return mongoTemplate.find( query(where("name").is(restaurantName)), Restaurant.class); }

A document in the restaurant collection

{ "_id" : ObjectId("4d977f55d3fe3119c904e026"), "_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant", "name" : "Ajanta" "menuItems" : [ { "name" : "Tandoori Portobello Mushrooms", "price" : 5.5 }, { "name" : "Duck Curry Kerala", "price" : 15 } ]}

Spring MongoDB Example - Config 1@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration {

private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;

private @Value("#{mongoDbProperties.host}") String mongoDbHost;

public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); }

@Override protected String getDatabaseName() { return mongoDbDatabase; }...}

databaseName=demo1host=192.168.253.150

mongodb.properties:

<util:properties id="mongoDbProperties" location="mongodb.properties"/>

</beans>

External Config

Defines a MongoTemplate

Spring MongoDB Example - Config 2<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoFactory"/></bean>

<mongo:db-factory id="mongoFactory" host= "#{mongoDbProperties.host}"dbname="#{mongoDbProperties.databaseName}" />

<util:properties

id="mongoDbProperties" location="mongodb.properties"/>

Update example@Repositorypublic class RestaurantRepository { public void addMenuItem(String restaurantId, MenuItem newMenuItem) { mongoTemplate.updateFirst( query(where("_id").is(new ObjectId(restaurantId))), new Update().push("menuItems", newMenuItem), Restaurant.class); }

Atomic, in-place update of document

Geospatial example 1

@Componentclass MongoFriendService extends FriendService {

@Autowired var mongoTemplate: MongoTemplate = _

@PostConstruct def createGeoIndex { val dbo = new BasicDBObject dbo.put("location", "2d") mongoTemplate.getCollection("friendRecord").ensureIndex(dbo) }

Create geospatial 2d index

Collection name

case class FriendRecord(id : String, name : String, location : Point)

Geospatial example 2 - finding nearby

@Componentclass MongoFriendService extends FriendService {

override def findNearbyFriends(request: NearbyFriendsRequest) = { val location = new Point(request.longitude, request.latitude) val distance = new Distance(3, Metrics.MILES) val query = NearQuery.near(location).maxDistance(distance) val result = mongoTemplate.geoNear(query, classOf[FriendRecord])

val nearby = result.getContent.map(_.getContent) FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id))) }

Callbacks – access driver API with exception translation

@Test public void testDbCallback() { Restaurant ajanta = makeAjantaRestaurant(); restaurantRepository.add(ajanta); assertCollectionExists("restaurants2"); }

private Void assertCollectionExists(final String collectionName) { return mongoTemplate.execute(new DbCallback<Void>(){ @Override public Void doInDB(DB db) { Set<String> collectionNames = db.getCollectionNames(); Assert.assertTrue("Missing from " + collectionNames, collectionNames.contains(collectionName)); return null; }}); }

Exceptions are translated

Defining a Mongo Generic Repository

interface PersonRepository extends MongoRepository<Person, ObjectId> { List<Person> findByLastname(String lastName);}

public class Person { private ObjectId id; private String firstname; private String lastname;… getters and setters}

Person p = new Person("John", "Doe");personRepository.save(p);

Person p2 = personRepository.findOne(p.getId());

List<Person> johnDoes = personRepository.findByLastname("Doe"); assertEquals(1, johnDoes.size());

Mongo Repository configuration<bean>

<mongo:repositories base-package="net.chrisrichardson.mongodb.example.mongorepository" mongo-template-ref="mongoTemplate" />

</beans>

Scans classpath looking for subtypes of MongoRepository in the base package

Richer mapping@Document(collection=”people”)public class Person {

@Id private ObjectId id; private String firstname; @Indexed private String lastname;

@PersistenceConstructor public Person(String firstname, String lastname) { this.firstname = firstname; this.lastname = lastname; }….}

Annotations define mapping: @Document, @Id, @Indexed, @PersistanceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndexed, @Value

Map fields instead of properties ⇒ no getters or setters

required

Non-default constructorIndex generation

Richer mapping configuration@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;

private @Value("#{mongoDbProperties.host}") String mongoDbHost;

@Override public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); } @Override public String getDatabaseName() { return mongoDbDatabase; }

@Override public String getMappingBasePackage() { return Person.class.getPackage().getName(); }}

Defines MongoTemplate bean

Configures classpath scanning

Support for the QueryDSL project

QPerson person = QPerson.person;

Predicate predicate = person.homeAddress.street1.eq("1 High Street") .and(person.firstname.eq("John"))

List<Person> people = personRepository.findAll(predicate);

assertEquals(1, people.size());assertPersonEquals(p, people.get(0));

Generated from domain model class

Type-safe composable queries

Cross-store/polyglot persistence

@Entitypublic class Person { // In Database @Id private Long id; private String firstname; private String lastname; // In MongoDB @RelatedDocument private Address address;

{ "_id" : ObjectId(”….."), "_entity_id" : NumberLong(1), "_entity_class" : "net.. Person", "_entity_field_name" : "address", "zip" : "94611", "street1" : "1 High Street", …}

Person person = new Person(…);entityManager.persist(person);

Person p2 = entityManager.find(…)

Agenda

• Why NoSQL?

Using Mongo and Redis with Cloud Foundry

• Create a Mongo or Redis service

• Bind the service to your application

• Access the service

• Via auto-reconfiguration

• Using <cloud:*/> namespace

Creating a Redis Server

Deploying a Redis application

Redis bean definitions

USING THE APPLICATION

About <cloud:redis-connection-factory/>

<cloud:redis-connection-factory id="redisConnectionFactory" service-name="redis1" />

Use when multiple services are bound

Deploying a Mongo application

MongoDB bean definitions

Using the Mongo Application

About <cloud:mongo-db-factory/><cloud:mongo-db-factory id="mongoFactory" service-name="mongo1" > <cloud:mongo-options connections-per-host="..." max-wait-time="..." /></cloud:mongo-db-factory>

Use when multiple services are bound

NoSQL and Caldecott

• Caldecott let’s you tunnel to a NoSQL service

• Use Redis CLI

• Explore database, adhoc operations

• ...

• Use Mongo CLI etc

• Explore database, adhoc operations

• Mongo dump/restore

• ...

Summary

• NoSQL databases sometimes offer a combination of:

• Higher scalability and performance

• Schema less, richer data models

• Spring Data simplifies the development of NoSQL applications

• Cloud Foundry supports Mongo and Redis

Questions?

@crichardson chris.richardson@springsource.com http://plainoldobjects.com

Sign up for CloudFoundry.com

Cloud Foundry 启动营在www.cloudfoundry.com注册账号并成功上传应用程序，即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。

iPhone5 等你拿第二天大会结束前，请不要提前离开，将填写完整的意见反馈表投到签到处的抽奖箱内，即可参与“iPhone5”抽奖活动。

Birds of a Feather 专家面对面所有讲师都会在课程结束后，到紫兰厅与来宾讨论课程上的问题

using spring with nosql databases (springone china 2012)

Technology

a comparison between relational databases and nosql...

nosql databases - umd

nosql databases - couchdb

introduction to nosql databases

nosql databases

an empirical study of nosql databases by using · pdf filean...

nosql databases introduction - utn 2013

dynamo and nosql databases

nosql databases - cse.hut.fi · nosql databases - keijo...

front range php nosql databases

nosql databases

an intro to nosql databases

nosql databases and polyglot applications

traditional databases vs nosql

nosql databases for enterprises - nosql now conference 2013

nosql databases oracle - berkeley db

a comparison of sql and nosql databases databases

exploring nosql databases

test automation for nosql databases

nosql databases : mongodb vs cassandra