amazon dynamodb lessen's learned by beginner
DESCRIPTION
Presented 14-Mar-2014 at Sendai, JAWS-UG.TRANSCRIPT
Amazon DynamoDBLessen's Learned by Beginner
Correct me if anything wrong
JAWS-UG Aizu Tokuno
About me
● TOKUNO, Hirokazu● IT department in factory● 4th years moved to IT at age of 39
– Studying hard to catch up● Little know about
– VBA, SQL, JavaScript, Java, Perl, Shell...● Like to not write code
– Pentaho ETL, Oracle APEX, Mojolicious
Learning/Challenge in 2013
● Unicage Development Method● Perl, Mojolicious● Jenkins● Automate Test jUnit, Mockito, shUnit2● Twitter Bootstrap, Google Maps API etc.● Fun to attend workshop/seminar!
– efsta!– Open Data Cafe Aizu– JAWS-US Aizu
● Join to CODE for AIZU
Learning/Challenge in 2014
● Enroll to Linked Open Data Challenge 2013● Attend Hack for Town in Aizu● Meteor(Node.js, MongoDB) on Nitrous.IO● GitHub Debut● Amazon DynamoDB Right now!
No experience in Could
Created AWS account just 7 days ago
What's DynamoDB?
● Fully Managed NoSQL Database Service
What's DynamoDB?
● Fully Managed NoSQL Database Service
What's the hell are you talking about?
What's DynamoDB?
● Fully Managed NoSQL Database Service
– NoSQL– Fully Managed Database Service
Split into peaces
Understanding of NoSQL
● Not only SQL● Fast to read from huge amount of data
– Accept inconsistent● Flexible Data Structure (DynamoDB etc.)
– Attributes can be differ for each record● Not suitable for summary and query
Fully Managed DB Service
● Rent database service as server
– Pay for read/write speed, storage, transaction● Managed by Amazon
– Install, Upgrade and Patching– Keep storage space, no limit!– Distribute 3 data centers– Improve performance
Sum up!
● Ask Amazon for boring tasks!● Extreme performance with SSD!● Money solve the issue!● Not to think about storage limitation!● High reliability by distributing at 3 data center!● Be careful! It's not traditional database.
Let's try!
● Service - DynamoDB
Create Table
● Create Table
Configure Table
● Primary Key
– Only Hash or Hash/Range combination
Primary Key
● Hash Key : Column for partitioning
– Pick not converged column like UserID– Do not use picky column like Game ID
● Range Key :
– Column for search● Push into 2 columns to be unique
Example of Primary Keys
● Present forum in 3 tables
Table Primary Key AttributesHash Range
Group JAWS-UG { LastPostBy="Kurihara" LastPostDateTime= "2012-01-03T00:40:57.165Z"}
Thread JAWS-UG Session { Message = "How about unconference?" Tags = [ "JAWS-UG", "Aizu" ] LastPostDateTime = "2012-01-03T00:40:57.165Z" }
Reply JAWS-UG#Session
2011-12-11T00:40:57.165Z
{ Message = "Extreeeeeeme!" PostedBy = "Kurihara" }
Attributes
● Contents other than Hash, Range● Each entry may have own attributes ● Data Type
– String– Number– Binary– String Set (Array)– Number Set (Array)– Binary Set (Array)
Convert Date to string
Create Index
● May skip since optional
Index
● Upto 5 indexes can be attached for each table
– Hash-Range table is created under the table– Cost for read/write and storage
● Able to create at table creation. No edit, add.● Local secondary index
– Create Hash-Range table with shared Hash Key as table, Range Key is not
● Global secondary index
– Create Hash-Range table with non shared Hash and Range
Index
● Sounds working like this...
Hash Range Attribute1 Attribute2
A 1 abc def
B 2 ghi jkl
Hash Attribute2 Attribute1 Range
A def abc 1
B jkl ghi 2
Primary Key
Primary Key
Index
Throughput
● Read Capacity Unit
– Able to read one item per second for 4KB● Write Capacity Unit
– Able to write one item per second for 1KB
Alarm
● Send notification if exceed specified threathold, 75~95%
Final confirmation
● As usual
¥60/month!
Complete!
● 3 tables for forum
Disadvantages
● Unable to modify Index
– Need to re-build if need to modify– Requires cost to migrate data
● (Possibly) No system to sync with doc like, ERD● Suitable application and not suitable
– Need to scan all records to query by othet than Primary Key, Index
– Requires RDB for consistent transaction or use Java Transaction Library
– Complex with RDB as recommended
Access from programs!
● Official SDKs● Unofficial SDK for Perl etc.
Android Browser iOS
Java .NET Node.js
Python PHP Ruby
SDK API
● Table
– CreateTable– DeleteTable– DescribeTable– ListTables– UpdateTable
● Batch processing
– BatchGetItem– BatchWriteItem
● Item– PutItem– GetItem– UpdateItem– DeleteItem
● Search– Query : Primary
or Index– Scan : all record
Let's play with DynamoDB!
NoSQL is for Big Data
Big Data for Twitter
Make clearwhat's interesting!
What is to be investigated...
Voluntary Restraints!!
Okay, okay, let's try...
Forum with WebSock
Too many samples available...Not challenging enough
Create RESTfull API!
Like this
DynamoDBNitrous.IOWeb Client
● All free of charge
RESTfull API
● Implemented very simple function
– No BatchGetItem/BatchWriteItem
Action Function Method Example of URLCreate putItem POST /Group/Hash?p=1&q=A
Read scangetItem
GET /Groups/Groups/Hash
Update updateItem PUT /Group/Hash?p=A&q=1
Delete deleteItem DELETE /Group/Hash
Create by POST
● curl -i -X POST -H "Content-Type: application/json" -d '{"datetime":"2013-03-13T23:27:32.256Z" , "by":"Tokuno"}' http://your_host:8080/Group/JAWS-UG
Read all by GET
● curl -H "Content-Type: application/json" http://your_host:8080/Groups
Read by GET
● curl -H "Content-Type: application/json" http://your_host:8080/Group/JAWS-UG
May need URI encode for Japanese
Update by PUT
● curl -i -X PUT -H "Content-Type: application/json" -d '{"datetime":"2021-03-14T00:04:01.951Z" , "by":"No one tell me..."}' http://your_host:8080/Group/JAWS-UG
Delete by DELETE
● curl -i -X DELETE http://your_host:8080/Group/JAWS-UG
One more thing
Like this
DynamoDBNitrous.IOPentaho ETL
● All free!
Pentaho for data processing
● Limited by DynamoDB throughput● Pay more to boost
Group LastPostDateTime LastPostByあ 2013-03-13T23:27:32.256Z とくのうい 2013-03-13T23:27:32.256Z とくのうう 2013-03-13T23:27:32.256Z とくのうえ 2013-03-13T23:27:32.256Z とくのうお 2013-03-13T23:27:32.256Z とくのうか 2013-03-13T23:27:32.256Z とくのうき 2013-03-13T23:27:32.256Z とくのうく 2013-03-13T23:27:32.256Z とくのうけ 2013-03-13T23:27:32.256Z とくのうこ 2013-03-13T23:27:32.256Z とくのうあ1 2013-03-13T23:27:32.256Z とくのうい1 2013-03-13T23:27:32.256Z とくのうう1 2013-03-13T23:27:32.256Z とくのうえ1 2013-03-13T23:27:32.256Z とくのうお1 2013-03-13T23:27:32.256Z とくのうか1 2013-03-13T23:27:32.256Z とくのうき1 2013-03-13T23:27:32.256Z とくのう
Summary
● Easy to maintaing since managed by Amazon!● NoSQL is not 'Relational' but application links
– Not easy to develop● Unable to add/modify Index● Not flexible to query. Need to query every 1MB● Want method to actual and document
● Only suitable for disposable application and enterprise whom has big data?
Thank you!
Timeline
● 2014.03.08 Create account● 2014.03.11 Attend Hack for Town
2014.03.11 Cross over in last 3 years● 2014.03.12 Presentation for Game Lab● 2014.03.13 Develop RESTful
2014.03.13 Play with Pentaho● 2014.03.14 Presentation, today
Appendix
Region
● Find region usedhttp://docs.aws.amazon.com/general/latest/gr/rande.html
Procedure to find Credentialss
IAM
Create a New Group of Users
Group
Permissions
● Select Power User Access for full access
Verify Policy
● Required to review although nonsense
Create User
User
Credentials
● Finally