jamesrh 2007.06.11 . 7 major aws services () amazon e-commerce service (ecs) amazon
TRANSCRIPT
JamesRH2007.06.11
http://msblogs/jamesrh
7 major AWS Services (http://aws.amazon.com) Amazon E-Commerce Service (ECS) Amazon Elastic Compute Cloud Beta (EC2) Amazon Historical Pricing Amazon Mechanical Turk Beta Amazon Simple Storage Service (S3) Amazon Simple Queue Service (SQS) Alexa Web Services
Originally planning to write against both EC2 & S3 but EC2 beta was closed until recently There are EC2 scaling issues (confirmed by Bezos at Web2.0) S3 Scaling well:
5B objects Peak transaction rate:
Day: 923M Second: 16,67
This demo app was written to learn the S3 interfaces, their strengths, and weaknesses
04/18/23 2
Went to aws.amazon.com Signed into Amazon account
Same one used for Amazon shopping Fill in additional contact information
agreed to terms and conditions AWS provides Access Key & Secret Key Tested using www.awszone.com
Simple web app supporting most AWS operations via SOAP Failed with: “Client.NotSignedUp”
Email showed up later complaining about credit card out of date
I corrected the payment issue and was up and running User provisioning is fast and efficient
Overall, quick and easy process
04/18/23 3
Blob store with per blob metadata Blobs supported up to 5GB
~36.4 hour upload via Comcast “broadband” Metadata up to 4k in name/value pairs (2k of user
metadata) Up to 100 ACLs on each object (part of system metadata)
Read, write, read_ACP, write_ACP, and full_control Owner has full control and is not changeable
Each blob stored in a bucket Bucket name space shared by ALL S3 users All buckets have a single unchanging owner
Each blob has a URI identifier in a flat per-bucket namespace Directory hierarchy simulated
Get(key=“C:/X/Y/Z.txt”) gets the file Get(key=“C:/X”) gets all objects with leading “C:/X/” in URI Get(key=“C:/X/”, separator=“/”) gets all objects with leading “C:/X/” but
no subsequent “/”
04/18/23 4
Client libraries offered as samples for: Java SOAP, Java REST, C# SOAP, C# REST,
Ruby REST, Python REST, & Perl REST BitTorent also supported Docs at protocol layer rather than APIs API implementations uneven without
docs C# SOAP has max send of 1M (can use
DIME attachment for larger) C# REST doesn’t support direct streaming
Source provided so quality & functionality issues all have solution
04/18/23 5
conn = new AWSAuthConnection(key, secretKey);res = conn.delete(Bucket, key, null);Soap request Headers:
- <SOAP-ENV:Body>- <m:DeleteObject xmlns:m="http://s3.amazonaws.com/doc/2006-03-01/"> <m:AWSAccessKeyId>10T01CX2MA2M9WD25002</m:AWSAccessKeyId> <m:Signature>NbT52AjYRzhHdA490B+T20KBYYg=</m:Signature> <m:Timestamp>2007-03-14T20:10:28.000Z</m:Timestamp> <m:Bucket>10T01CX2MA2M9WD25002-Archiver</m:Bucket> <m:Key>C:/JamesRH/dev/AWS/tst/test.tst*1</m:Key> </m:DeleteObject> </SOAP-ENV:Body>
SOAP response Headers- <soapenv:Body>- <DeleteObjectResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01/">- <DeleteObjectResponse> <Code>204</Code> <Description>No Content</Description> </DeleteObjectResponse> </DeleteObjectResponse> </soapenv:Body>
04/18/23 6
REST Access Key:AWS : Base64(hmac-sha1(VERB, Content-MD5, Content type, Date, AMZ headers, resourceKey ))
SOAP Model Doesn’t sign
S3 Archiver is a version store Registered directories & sub-directories contents are stored in S3 Each new file version is stored separately Any version of any file can be restored in place Multiple versions of a file can be restored with version suffixes on
filename Any file version can be deleted from S3
Roughly 3,500 lines of C# written against C# REST interface
04/18/23 7
Performance: 12,220 files, 12,500 versions totaling
5.1 GB synchronized in under 30 seconds (local-only operation)
Actual transfers very slow ranging from roughly 1GB/hour to as low as 100 MB/hour (28 KBs)
Issues mostly network bandwidth-related on the perf side and standard internet flakiness on the reliability side
Files stored in S3 with version numbers C:\JamesRH\OpsFriendlyApps.docsx*10
Metadata as name/value pairs with each file: Version Size Hash Last Write Create Time Access Time Archive Time
Fetching metadata for 1,000s of files slow so have a cache stored in S3 S3_Archiver.xml
Large scale testing required real data so implemented encryption Single file operations slow and very uneven in performance so need
to run all non-metadata-only operations asynchronously On scale-up to 1000s of files S3_Archiver.xml gets large and slow so
implemented compression On scale-up to 10s of 1000s of files S3_Archiver.xml still slow to
transfer so check hash and avoid transfers when possible
04/18/23 8
• C:/user/file1*1• C:/user/file1*2• C:/user/file2*1• C:/file3*1• cache• cache*C:/usr/
libary1• cache*C:/usr/
library2
04/18/23 9
Error rate ~4 in 1,000 and similar across different networks: Comcast:
Recv: 43 to 567.19 KBs Send: 5 to 52 KBs
Microsoft: Recv : 317 to 10,935 KBs Send: 437 to 5,556 KBs
Home Large File: Send Fast/Slow ratio: 9.8x Recv Fast/Slow ratio: 12.9x Recv/Send ratio: 10.8x
Work Large File: Send Fast/Slow ratio: 12.7x Recv Fast/Slow ratio: 34.4x Recv/Send ratio: 1.9x
Conclusion: Send to receive difference ranges from a factor of 2 to an order
of magnitude 2 orders of magnitude spread in throughput Internet error rate high 5+ orders of magnitude higher latency than local disk
04/18/23 10Source: one week hourly perf & reliability probe of S3 at 1k, 10k, 100k, 1m, & 25m
04/18/23 11
Kbs
4 errors
Errors show as negative throughput
04/18/23 12
Kbs
Errors show as negative throughput
1 error
04/18/23 13
Kbs
Errors show as negative throughput
2 errors
2 errors3 errors
04/18/23 14
Kbs
Errors show as negative throughput
1 error
Summary of This Month's Activity as of March 14, 2007 Billing Cycle for this Report: March 1 - March 31, 2007
Usage Charges Rate Usage Totals Amazon Simple Storage Service $0.20 per GB of data transferred (details) 15.174 GB $3.04 $0.15 per GB-Month of storage used (details) 0.214 GB-Mo $0.04 Total: $3.08
All web services are sold by Amazon Digital Services, Inc.
Summary of Last Month's Statement: February 1 - February 28, 2007 Total Charges for February 1 - February 28, 2007: $0.07
Amazingly inexpensive: Developed a 3,000 line interactive system storing
1,000s of files and several GB stored with many test cycles at $3.11
04/18/23 15
Current bandwidth price (through May 31, 2007) $0.20 / GB – uploaded $0.20 / GB - downloaded
New bandwidth price (effective June 1, 2007) $0.10 per GB - all data uploaded $0.18 per GB - first 10 TB / month data downloaded $0.16 per GB - next 40 TB / month data downloaded $0.13 per GB - data downloaded / month over 50 TB Data transferred between Amazon S3 and Amazon EC2 will remain free
New request-based price (effective June 1, 2007) $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests No charge for delete requests
Storage will continue to be charged at $0.15 / GB-month used
04/18/23 16