Getting the most out of Windows Azure StorageJoe GiardinoSenior Development LeadWindows Azure Storage
WAD-B406
AgendaScaleService PerformanceClient PerformanceBlob / Tables / QueuesDebuggingFuture DirectionsSummaryQ & A
North America Region Europe Region Asia Pacific Region
S. Central – U.S. Sub-region
W. Europe Sub-region
N. Central – U.S. Sub-region
N. Europe Sub-region
S.E. AsiaSub-region
E. AsiaSub-region
Data centers
Windows Azure Storage
East – U.S. Sub-region
West – U.S. Sub-region
Windows Azure StorageAbstractionsBlobs – File system in the cloudTables – Massively scalable NoSQL storageQueues – Reliable storage and delivery of messagesDisks – Durable NTFS volumes for Windows Azure VMs
Easy client accessEasy to use REST APIs and Client Libraries
.NET, Java, Node.js and PHP, and morehttp://www.windowsazure.com/en-us/documentation/?fb=en-us
Existing NTFS APIs for Windows Azure VM Disks
Scale
Scalability Targets -Storage AccountStorage Account level targets Applies to accounts created after June 7th 2012
Capacity – Up to 200 TBs
Transactions – Up to 20,000 entities/messages/blobs per second
Bandwidth for a Geo Redundant storage accountIngress - up to 5 GbpsEgress - up to 10 Gbps
Bandwidth for a Locally Redundant storage accountIngress - up to 10 Gbps Egress - up to 15 Gbps
Scalability Targets – PartitionPartition level TargetsApplies to accounts created after June 7th 2012
Single Queue – Account Name + Queue NameUp to 2,000 messages per second
Single Table Partition – Account Name + Table Name + PartitionKeyUp to 2,000 entities per second
Single Blob – Account Name + Container Name + Blob NameUp to 60 MB/s
Windows Azure Flat NetworkFlat network storage design“Quantum 10” networkNon-blocking 10Gbps based fully meshed networkMove to software based Load Balancer Enables high bandwidth scenarios such as Windows Azure IaaS disks, HPC, Map Reduce etc.
http://blogs.msdn.com/b/windowsazurestorage/archive/2012/11/04/windows-azure-s-flat-network-storage-and-2012-scalability-targets.aspx
DFS Layer
PartitionServer 1
PartitionServer 2
PartitionServer 3
PartitionServer 4
Master System
Master System
VIP
Partition Master
- RangePartition
- Server Load
Legend
Unassign RangePartitionReassign RangePartition
Key Concept: Automatic RangePartition Load Balancing
Load balancing is triggered based on hot Range Partitions or Partition ServersNo data is moved on disk for the reassignment
Only changing the index assignment for the Partition Servers
FE 1 FE 2 FE 3R1R2
Windows Azure Storage – How is it used?Xbox: Uses Blobs, Tables & Queues for applications like Cloud Game Saves, Halo 4, Music, Kinect data collection etc.
SkyDrive: Uses Blobs to store pictures, documents etc.
Bing: Uses Blobs, Tables and Queues to implement an ingestion engine that consumes Twitter and Facebook public status feeds and provides it to Bing search
Skype: Uses Blobs, Tables and Queues for Skype video messaging
Performance
Demos will be using .NET Storage Client 2.1 RCDeployed in a production clusterCompute is co-located with data (XL VM)Storage Account created after June 12’
Client Performance – Beyond 2.0…2.0 provided dramatic performance improvementsInternal client doing heavy Table workloads migrated from 1.7 to 2.0 table service layer
Observed ~3x performance improvement per VMCPU dropped from 90% to 45%
Storage Client 2.1 – focus on performance and fundamentals
Code Coverage – Up to 91% and countingTest Cases – over 960 publicly available unit testsPerformance – Performance features and regular performance tests passes against production tenants for validationStress – Data correctness tests executed under massive loads…
Available Soon @ http://www.nuget.org/packages/WindowsAzure.Storage
What’s new in 2.1?Async Task methods with support for cancellationByte Array, Text, File upload / download APIs for blobsIQueryable provider for TablesBuffer PoolingMulti-Buffer Memory Stream for consistent performance when buffering unknown length data.NET MD5 now default (~20% faster than invoking native one)Compiled Expressions for Table EntitiesClient LoggingAnd much more...
Performance – What you can doFollow Best practice for each serviceIdentify what you wish to optimize, latency, throughput, cost, plan accordinglyIdentify Scalability targets and Access Patterns
Plan for growth in your service
Use Timeouts & Retry Policies to control Latency & Fault ToleranceUse latest NuGet package when possible
Tied to latest service release
General Client Best Practices (.NET)Disable Nagle for small messages (< 1400 b)
ServicePointManager.UseNagleAlgorithm = false;
Disable Expect 100-Continue ServicePointManager.Expect100Continue = false;
Avoid Large Object Heap when possibleRe-use buffers if possibleUse Async for apps that scale
General Client Best Practices (.NET GC)Target .NET 4.0, but take advantage of .NET 4.5 GC
GC performance is greatly improved, Background GC : http://msdn.microsoft.com/en-us/magazine/hh882452.aspx
Use Server GC when applicablehttp://msdn.microsoft.com/en-us/library/ee787088.aspx#workstation_and_server_garbage_collectionUses one thread & one heap per Core, Collects all heaps simultaneously
For latency sensitive scenarios use SustainedLowLatency GC
GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;Will use more memory and do more frequent smaller duration GC’s
.NET or Performance Gotchas!ServicePointManager settings must be done before first call to the service
can be done in app config to avoid racy initialization
Set DefaultConnectionLimit > 2ServicePointManager.DefaultConnectionLimit = 100; (Or More)
.NET Memory Stream performance is bad for services
Async PerformanceDynamic buffer size behaviorAvoid ReadByte / WriteByte methods
Unbounded TPL ParallelismParallel.ForEach – limit parallelism at the application layerhttp://msdn.microsoft.com/en-us/library/ee789351.aspx
General Server Best PracticesLocate Storage accounts close to users
Co-locate compute in same DC (intra-DC bandwidth is free)For wide spread client apps consider using more than one storage account in different physical DCs
Understand Account Scalability targetsAccount Scalability targets are different depending on Geo-ReplicationGeo Redundant: 5 Gbps ingress / 10 Gbps egressLocally Redundant : 10 Gbps ingress / 15 Gbps egress
Enable Logging & Metrics on each storage serviceCan be done via REST, Client API, or PortalEnables clients to self diagnose issues, including performance related onesData can be automatically GC’d according to a user specified retention interval
General Server Best Practices (cont.)Optimize what you send & receive
Blobs: Range reads, Metadata, Head RequestsTables: Upsert, Merge, Projection, Point QueriesQueues: Update Message, Batch size
Distribute load over many partitionsNote: Table Entity Group Transaction (batch) only applies to entities in the same partition
Control Parallelism at the application layer Unbounded Parallelism can lead to slow latencies and throttling
Blend Traffic TypesIn general Blob workloads tend to be IO bound as Table workloads tend to be CPU limited. By blending traffic in roles you can make the most of your compute resources
Blobs
Blob Performance Best PracticeA blob is a single partition that can provide 60 Mb/sChoose right BlockSize for your scenario
CloudBlockBlob.StreamMinimumReadSizeInBytes / StreamWriteSizeInBytes
Use seekable streams when possibleSmall non seekable streams will result in 2 REST calls when a single atomic operation could have been used
Use Async for scaleDramatically simplified with Task APIs in 2.1 and async / await
2.x client has efficient downloads with fault tolerance
Blobs: Improvements in the 2.1 release?Byte Array upload/download APIs with offset & lengthAsync Stream open.NET MD5 implementation now default
FISMA compliant clients can re-enable it CloudStorageAccount.V1MD5 = false;
Multi-Buffer Memory Stream to provide performant dynamic buffering of dataIBufferManager works great with .NET BufferManager (System.ServiceModel.dll)
Blob Client Improvements vs. 1.7Sync
Upload Latency decreased by 16.5%Download Latency decreased by 23.2%CPU Utilization decreased by 38.5%
AsyncUpload Latency decreased by 5.3%Download Latency decreased by 37.3%CPU Utilization decreased by 35.1%
2.1 vs. 2.0.xReduces CPU by ~13%
Test Setup:XL VM with Collocated Storage30 concurrent workers uploading a total of 7.5 GB of data (each blob is 256 MB)
0
10
20
30
40
50
60
70
80
1.7
2.0.5.1 2.1 RC
Upload Download CPU
Tim
e (
s)
Blobs - ThroughputA single XL VM can achieve 1 Gbps to the blob store. A single partition canprovide 60 MB/s
Upload Multiple blobs concurrently using minimal blob parallelism
Distributes load over many partitionsLonger connections allow TCP windows to growBlobs under 64 MB can be uploaded as a single put, otherwise multiple block requests & a put block list are requiredAvoids “long tail” problem – 11 MB blob with 1MB blocks, parallelism =10
Concurrency Vs. Blob Parallelism
What pattern is faster to upload N Blobs? 1 concurrent upload with P=3030 concurrent uploads with P=1
Let’s find out…
Concurrency Vs. Blob ParallelismXL VM Uploading 512 256MB Blobs using a 1 MB Blocks (128GB)
P=1, C=1 => Averaged ~ 13.23 MB/sP=30, C=1 => Averaged ~ 50.72 MB/sC=30, P=1 => Averaged ~ 96.64 MB/sTest completed almost twice as fast!Single Blob is bound by the limits of a single partition whereas accessing multiple blobs concurrently can scale
0
2000
4000
6000
8000
10000
12000
P=1, C=1
P=30, C =1
P=1, C=30Tim
e (
s)
So where is parallel download blob?It is not provided in the Client Library
Would require much larger memory footprintDoes not provide significant performance improvement
Experiment 1 GB Blob Get on a XL VMAveraged 130.15 MB/s Note: this is over double the scalability target for a single blob, you should not take a dependency on any performance over these targets due the multi-tenant nature of the system.
Non-Seekable Streams
Blob Non-Seekable Stream ResultsUpload 10,000 10KB blobs
Same stream with CanSeek = true, false
Single Put Blob is better for small blobsNon-Seekable streams force BlobStream use, bad for small blobs
Will pre-buffer dataWill force a PutBlockList request after Block UploadSeekable Stream was ~2x faster
0
40
80
120
160
200
240
Seekable
Non Seekable
Tim
e (
s)
Block Read Size
Blob Read Size Results256MB Block Blob with 4 MB blocks
Single Get averaged 84.46 MB/s 4MB Read averaged 26.9 MB/s…64 KB Read averaged 8.69 MB/s
A BlockBlob can contain 50,000 Blocks
4 MB = Max of 195.31GB 2 MB = Max of 97.65GB…128KB = Max of 6.1 GB
0
4
8
12
16
20
24
256 MB
4 MB 2MB 1MB512KB
256KB
128KB
Tim
e (
s)
Tables
Tables: Improvements in the 2.1 release?Efficient IQueryable implementation
Shortcuts for projection and dynamic query construction
Compiled Serializers / De-Serializers (40-50x faster than reflection)More flexible serialization
Ignore Attribute to omit entity propertyExposed default serialization logic to allow users to easily persist 3rd party objects
Multi-Buffer Memory Stream to provide performant dynamic buffering of dataBuffer Pooling
Table Client Improvements vs. 1.7Complex Single Entity Operations
Upload Latency decreased by 17%Download Latency decreased by 29%Delete Latency decreased by 18%CPU Utilization decreased by 39.75%
Similar CPU reductions for Entity Group Transaction
Test Setup:XL VM with Collocated StorageTime to upload / download / delete 150k complex entities via single operations 0
5
10
15
20
25
30
35
40
45
50
1.7
2.0.5.12.1 RC
Upload Download Delete CPU
Tim
e (
s)
Table Performance Best PracticeKey selection is single most important performance decision you will make
Entity Group TransactionScalabilityEfficient queries
Send and receive the smallest amount of data necessary
Use Merge / Upsert when appropriateServer side projection
Use the Table Service Layer over WCF data services
Query continuations are expensive via WCF data servicesAdditional performance optimizations coming in future releases
Query Design & SelectivityUnderstand Continuation tokens
Returned at 1000 entities4 MB payloadServer boundaries
Table Scans are expensive – avoid them for latency sensitive scenariosPerformance of queries
Point Query (PK, RK) - FastestRow Key Range Query(PK, A < RK< D) Row Key Range Query Open ended (PK, RK > C)Multi Partition Query (A <PK < D, RK …)Query on non-indexed column (Timestamp, user column, etc.)
Tables - ThroughputBatch Size (Entity Group Transaction)
Use Batch to perform atomic actions on a given partition (all entities in the same table with the same PK)Up to 100 entitiesUp to 4 MB in sizeDramatically improves throughput and per entity latencies
Insert operations return the entity back to the client Merge, Replace, and Upserts do not.
Less bytes to read => less bytes to parse => lower NIC / CPU => Higher throughput
Tables - LatencyQuery Selectivity
Smaller take count / page sizes for latency sensitive applications
Projection: Only take what you needAvoid expensive serialization / deserialization
Can use DynamicTableEntity to avoid allocating large objects2.1 includes dramatic improvements over 2.0 / 1.7 clients for clients deriving from TableEntity
Batch Size
What is my optimal batch size?
Batch Size ResultsPer entity latency improves by ~7x Less time spent on authenticationUses connections more efficiently
1 5 10 15 20 25 30 40 50 1000123456789
0
20
40
60
80
100
120
140
Latency
Batch Size
Late
ncy /
En
tity
(m
s)
Insert vs. InsertOrReplace
Insert operations return the entity back to the client.Merge, Replace, and Upserts do not.Less bytes to read = less bytes to parse
Cool Features Demo
IQueryable implementationSerialize 3rd party objectsSimplified ProjectionDynamic type safe query constructionCompiled Serializers
Tables ResourcesTables 2.0 Deep Dive
http://blogs.msdn.com/b/windowsazurestorage/archive/2012/11/06/windows-azure-storage-client-library-2-0-tables-deep-dive.aspx
How to get the most out of Table Storagehttp://blogs.msdn.com/b/windowsazurestorage/archive/2010/11/06/how-to-get-most-out-of-windows-azure-tables.aspx Covers key selectionBatch size
Queues
Queues Performance Best PracticeNagle is important as many messages may be smallDesign application to update messages rather than deleting and inserting a new oneFor scale use multiple queues, see Multi-Queue ExampleUse Message De-queue count to detect malformed data or “poison” messages
Queues - ThroughputTune Message Count for Peek & Get
Consider failure modes, if a worker gets 32 messages but fails immediately 31 messages must also wait to become visible again
Tune Initial Invisibility Time & Visibility TimeoutUse Multiple Queues
Multi-Queue StrategiesCan use 1 Queue per worker
Simple, but does not always make efficient use of workersScaling is difficult as more queues need to be created / drained and deleted to match size of deployment
Round Robin QueueDefine N Queues with a given prefix i.e. myqueue01, myqueue02…Access Queues in round robin fashion, can be tuned to your scenario, i.e. try 4 queues before returning nullMessages need to track their parent queues for updates / deletesCompute workers are fully utilizedLatency may be higher in the case of empty queues as workers may need to check many queues
Queues - LatencyTune Initial Invisibility Time & Visibility Timeout
Shorter Visibility Timeout will require a more responsive service, but can ensure lower latencies, longer visibility timeouts can allow larger amounts of processing to be done before check pointing
Use smaller Message Count for Peek & GetUse Multiple Queues
Get Messages
What is optimal number of messages to get?
Get Messages ResultsPer entity latency improves by ~11.5x Less time spent on authenticationUses connections more efficiently
1 2 4 8 12 16 24 320
5
10
15
20
25
30
LatencyLatency / Entity
Batch Size
Late
ncy (
ms)
Debugging
Client DebuggingOperationContext is your friendAvailable in an overload of every method that makes a request to the serviceAdd custom headers to every requestStores Information for every request madeClient Request IDSend and Receive events
Client Debugging – Logging (2.1 RC)Opt-in or Opt-out model
Set Maximum Verbosity in App.configCan set System wide verbosity via OperationContext.DefaultLogLevelPerform targeted “vertical” logging for a single API call
OperationContext.LogLevelEnable without redeploying
Debugging Demo
Server side - Storage AnalyticsMetrics provides usage data per abstractionLogging provides per request logging, including client trace id’sLogs / Metrics are stored in your storage account
Can specify a retention policy to auto delete after some period of time
Enable via code or the Azure PortalProvides the ability to monitor and diagnose applicationsEspecially useful when utilizing SAS / SAS provider model
Storage Analytics LogsLog records for requests are stored in Windows Azure Blobs
The Log blobs are text files with one log entry per lineEach blob can contain one to many request records
A request typically appears in the log within 15 minutes after it completes execution
Configure the logging levels separately forBlob, Table and Queuesread (GET), write (PUT/POST/MERGE), delete (DELETE) requests or any combination
Best effort logging
Enable Logging via the portalEach transaction type can be configured separately
Each service is configured separately
Use Retention Policy to auto garbage collect old logs
Storage Analytics – Why turn on Metrics?Provides ability to answer commonly asked questions:
How many transactions did my service issue per hour over the past week?How many anonymous Get Blob requests were issued to my storage account?My application is not performing as expected, what is the availability and performance of storage for a given time period?list goes on and on…
Figure out who is accessing what…
Storage Analytics – MetricsTransaction metrics are provided for every 1 hour time interval stored into Windows Azure Tables
Example MetricsTotal TransactionsAvailability% Success, % Network Errors, % Timeout, % Throttled, etc.Average Latency (Application E2E and Storage Server latency)Total IngressTotal Egress
Blob, Table or Queue Summary and per REST API metrics
Capacity metrics provided for only Blobs at this timeUpdated once a day
Capacity and # of objects
Enable Metrics (Monitoring) via the portalMultiple Verbosity levels
Each service is configured separately
Use Retention Policy to auto Garbage Collect old logs
Storage Analytics SummarySeparate Namespace
LogsStored as blobs in separate Blob Container in the storage account being monitoredhttp://account.blob.core.windows.net/$logs/
MetricsStored as entities in a separate metrics Azure Tables in the storage account being monitoredhttp://account.table.core.windows.net/$Metrics*
Isolation$logs and $Metrics have separate resource limits and throttling from the rest of the storage account traffic
CostCapacity to keep the dataTransactions for generating & accessing analytics dataCan use retention policy on both logs and metrics in terms of daysDeleting data via retention policy does not incur transaction cost
Summary
You are the expert on your scenario!Have a scalable design
How many transactions per partition and accountBandwidth per partition and accountWhat is my strategy for growth in the future?
Deploy best practicesBest practices covered for each service can dramatically improve performanceUse latest Storage Client Libraries via NuGet / GitHub (especially if on 1.x)
How we stack upNasuni recently released their “State of the Cloud Storage Industry” Report for 2013 in February
Windows Azure consistently performed better than all other CSPs in the study, which included Amazon S3, Google Cloud Storage, HP Cloud Object Storage, and Rackspace Cloud Files.Windows Azure delivered the best Write/Read/Delete speeds across a variety of file sizes, with the fastest response times and fewest errors.Windows Azure not only outperformed the competition significantly during the raw performance tests, but it was the only CSP to post zero errors during 100 million reads and writes.
To Read Morehttp://www6.nasuni.com/rs/nasuni/images/2013_Nasuni_CSP_Report.pdfhttp://blogs.technet.com/b/stbnewsbytes/archive/2013/02/19/windows-azure-named-cloud-storage-leader.aspx
Future DirectionsCORS (Cross origin resource sharing) Windows Phone LibraryContinuous performance improvementsExpanding support for additional client platformsGeo-Replication for QueuesRead from secondary DC Expanded Reach – New Region build outs
http://blogs.technet.com/b/microsoft_blog/archive/2013/05/22/microsoft-announces-major-expansion-of-windows-azure-services-in-asia.aspx
And more …
ResourcesGetting Started https://www.windowsazure.com/en-us/develop/overview/
Pricing information https://www.windowsazure.com/en-us/pricing/details/
Storage team blogs http://blogs.msdn.com/b/windowsazurestorage/
Windows Azure Storage: What’s Coming, Best Practices, and Internalshttp://channel9.msdn.com/Events/Build/2013/3-541
.NET Clients encountering Port Exhaustion Bloghttp://blogs.msdn.com/b/windowsazurestorage/archive/2013/05/25/net-clients-encountering-port-exhaustion-after-installing-kb2750149.aspx
SOSP Paperhttp://blogs.msdn.com/b/windowsazurestorage/archive/2011/11/20/windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency.aspx
Questions?
msdn
Resources for Developers
http://microsoft.com/msdn
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Resources for IT Professionals
http://microsoft.com/technet
Evaluate this session
Scan this QR code to evaluate this session.
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.