amazon dynamodb lessen's learned by beginner

Amazon DynamoDBLessen's Learned by Beginner

Correct me if anything wrong

JAWS-UG Aizu Tokuno

About me

● TOKUNO, Hirokazu● IT department in factory● 4th years moved to IT at age of 39

– Studying hard to catch up● Little know about

– VBA, SQL, JavaScript, Java, Perl, Shell...● Like to not write code

– Pentaho ETL, Oracle APEX, Mojolicious

Learning/Challenge in 2013

● Unicage Development Method● Perl, Mojolicious● Jenkins● Automate Test jUnit, Mockito, shUnit2● Twitter Bootstrap, Google Maps API etc.● Fun to attend workshop/seminar!

– efsta!– Open Data Cafe Aizu– JAWS-US Aizu

● Join to CODE for AIZU

Learning/Challenge in 2014

● Enroll to Linked Open Data Challenge 2013● Attend Hack for Town in Aizu● Meteor(Node.js, MongoDB) on Nitrous.IO● GitHub Debut● Amazon DynamoDB Right now!

No experience in Could

Created AWS account just 7 days ago

What's DynamoDB?

● Fully Managed NoSQL Database Service

What's DynamoDB?

● Fully Managed NoSQL Database Service

What's the hell are you talking about?

What's DynamoDB?

● Fully Managed NoSQL Database Service

– NoSQL– Fully Managed Database Service

Split into peaces

Understanding of NoSQL

● Not only SQL● Fast to read from huge amount of data

– Accept inconsistent● Flexible Data Structure (DynamoDB etc.)

– Attributes can be differ for each record● Not suitable for summary and query

Fully Managed DB Service

● Rent database service as server

– Pay for read/write speed, storage, transaction● Managed by Amazon

– Install, Upgrade and Patching– Keep storage space, no limit!– Distribute 3 data centers– Improve performance

Sum up!

● Ask Amazon for boring tasks!● Extreme performance with SSD!● Money solve the issue!● Not to think about storage limitation!● High reliability by distributing at 3 data center!● Be careful! It's not traditional database.

Let's try!

● Service - DynamoDB

Create Table

● Create Table

Configure Table

● Primary Key

– Only Hash or Hash/Range combination

Primary Key

● Hash Key : Column for partitioning

– Pick not converged column like UserID– Do not use picky column like Game ID

● Range Key :

– Column for search● Push into 2 columns to be unique

Example of Primary Keys

● Present forum in 3 tables

Table Primary Key AttributesHash Range

Group JAWS-UG { LastPostBy="Kurihara" LastPostDateTime= "2012-01-03T00:40:57.165Z"}

Thread JAWS-UG Session { Message = "How about unconference?" Tags = [ "JAWS-UG", "Aizu" ] LastPostDateTime = "2012-01-03T00:40:57.165Z" }

Reply JAWS-UG#Session

2011-12-11T00:40:57.165Z

{ Message = "Extreeeeeeme!" PostedBy = "Kurihara" }

Attributes

● Contents other than Hash, Range● Each entry may have own attributes ● Data Type

– String– Number– Binary– String Set (Array)– Number Set (Array)– Binary Set (Array)

Convert Date to string

Create Index

● May skip since optional

Index

● Upto 5 indexes can be attached for each table

– Hash-Range table is created under the table– Cost for read/write and storage

● Able to create at table creation. No edit, add.● Local secondary index

– Create Hash-Range table with shared Hash Key as table, Range Key is not

● Global secondary index

– Create Hash-Range table with non shared Hash and Range

Index

● Sounds working like this...

Hash Range Attribute1 Attribute2

A 1 abc def

B 2 ghi jkl

Hash Attribute2 Attribute1 Range

A def abc 1

B jkl ghi 2

Primary Key

Primary Key

Index

Throughput

● Read Capacity Unit

– Able to read one item per second for 4KB● Write Capacity Unit

– Able to write one item per second for 1KB

Alarm

● Send notification if exceed specified threathold, 75~95%

Final confirmation

● As usual

¥60/month!

Complete!

● 3 tables for forum

Disadvantages

● Unable to modify Index

– Need to re-build if need to modify– Requires cost to migrate data

● (Possibly) No system to sync with doc like, ERD● Suitable application and not suitable

– Need to scan all records to query by othet than Primary Key, Index

– Requires RDB for consistent transaction or use Java Transaction Library

– Complex with RDB as recommended

Access from programs!

● Official SDKs● Unofficial SDK for Perl etc.

Android Browser iOS

Java .NET Node.js

Python PHP Ruby

SDK API

● Table

– CreateTable– DeleteTable– DescribeTable– ListTables– UpdateTable

● Batch processing

– BatchGetItem– BatchWriteItem

● Item– PutItem– GetItem– UpdateItem– DeleteItem

● Search– Query : Primary

or Index– Scan : all record

Let's play with DynamoDB!

NoSQL is for Big Data

Big Data for Twitter

Make clearwhat's interesting!

What is to be investigated...

Voluntary Restraints!!

Okay, okay, let's try...

Forum with WebSock

Too many samples available...Not challenging enough

Create RESTfull API!

Like this

DynamoDBNitrous.IOWeb Client

● All free of charge

RESTfull API

● Implemented very simple function

– No BatchGetItem/BatchWriteItem

Action Function Method Example of URLCreate putItem POST /Group/Hash?p=1&q=A

Read scangetItem

GET /Groups/Groups/Hash

Update updateItem PUT /Group/Hash?p=A&q=1

Delete deleteItem DELETE /Group/Hash

Create by POST

● curl -i -X POST -H "Content-Type: application/json" -d '{"datetime":"2013-03-13T23:27:32.256Z" , "by":"Tokuno"}' http://your_host:8080/Group/JAWS-UG

Read all by GET

● curl -H "Content-Type: application/json" http://your_host:8080/Groups

Read by GET

● curl -H "Content-Type: application/json" http://your_host:8080/Group/JAWS-UG

May need URI encode for Japanese

Update by PUT

● curl -i -X PUT -H "Content-Type: application/json" -d '{"datetime":"2021-03-14T00:04:01.951Z" , "by":"No one tell me..."}' http://your_host:8080/Group/JAWS-UG

Delete by DELETE

● curl -i -X DELETE http://your_host:8080/Group/JAWS-UG

One more thing

Like this

DynamoDBNitrous.IOPentaho ETL

● All free!

Pentaho for data processing

● Limited by DynamoDB throughput● Pay more to boost

Group LastPostDateTime LastPostByあ 2013-03-13T23:27:32.256Z とくのうい 2013-03-13T23:27:32.256Z とくのうう 2013-03-13T23:27:32.256Z とくのうえ 2013-03-13T23:27:32.256Z とくのうお 2013-03-13T23:27:32.256Z とくのうか 2013-03-13T23:27:32.256Z とくのうき 2013-03-13T23:27:32.256Z とくのうく 2013-03-13T23:27:32.256Z とくのうけ 2013-03-13T23:27:32.256Z とくのうこ 2013-03-13T23:27:32.256Z とくのうあ1 2013-03-13T23:27:32.256Z とくのうい1 2013-03-13T23:27:32.256Z とくのうう1 2013-03-13T23:27:32.256Z とくのうえ1 2013-03-13T23:27:32.256Z とくのうお1 2013-03-13T23:27:32.256Z とくのうか1 2013-03-13T23:27:32.256Z とくのうき1 2013-03-13T23:27:32.256Z とくのう

Summary

● Easy to maintaing since managed by Amazon!● NoSQL is not 'Relational' but application links

– Not easy to develop● Unable to add/modify Index● Not flexible to query. Need to query every 1MB● Want method to actual and document

● Only suitable for disposable application and enterprise whom has big data?

Thank you!

Timeline

● 2014.03.08 Create account● 2014.03.11 Attend Hack for Town

2014.03.11 Cross over in last 3 years● 2014.03.12 Presentation for Game Lab● 2014.03.13 Develop RESTful

2014.03.13 Play with Pentaho● 2014.03.14 Presentation, today

Appendix

Region

● Find region usedhttp://docs.aws.amazon.com/general/latest/gr/rande.html

Procedure to find Credentialss

IAM

Create a New Group of Users

Group

Permissions

● Select Power User Access for full access

Verify Policy

● Required to review although nonsense

Create User

User

Credentials

● Finally