Architecting Azure Solutions Microsoft
Azure Storage Overview70-534
Architecting Microsoft Azure Solutions
Two Part Presentation
Introducing Azure Storage
Cloud Application RequirementsDurable and highly availableScalable and performant Multiple concurrent usersSecureManageable Time to market
Introducing Microsoft Azure StorageCloud Storage - Anywhere and anytime
accessBlobs, Disks, Tables, Queues and Files
Highly Durable, Available and Massively Scalable
Easily build “Internet scale” applicationsMore than 35 trillion stored objects3.5+ Million requests/sec on average
Pay only for what you useExposed via easy and open REST APIsRich Client Libraries and Tools
17 Regions Worldwide in 2015
Abstractions – BlobsBlobs – Massively scalable object store in the cloud
Simple REST interface (Put, Get, Delete)Data sharing – share documents, pictures, video, music, etc.Big Data – store raw data/logs and compute/map reduce over dataBackups – data and device backups
Abstractions – DisksPersistent disks for VMs in Azure Disks are VHDs stored in Azure Page BlobsPage blobs are optimized for random I/OVM see the VHD/Blob as a disk
Reads translated to GETs, writes to PUTsBlob protected by write-leaseReads from the blob (and snapshots) still allowed
Breaking News – Premium StorageConsistent low latency SSD based with predictable IO throughputSuitable for high-performance IO-intensive database workloadsSingle digit milliseconds latenciesSupports up to 1 TB blob/disk sizeStripe up to 32 disks for a total of
32TB and more than 50,000 IOPSPremium Storage Disks work in
conjunction with a new VM series
Abstractions – TablesTables – Massively scalable NoSQL cloud store
Key/Attribute(s) store at scaleStore user, device or any type of metadata for your serviceAuto load balances partitions to meet traffic needsOData protocol (AtomPub or JSON)
Abstractions – QueuesQueues – Reliable messaging system
Reliable, low latency, high throughput messaging systemDecouple components/roles
Web role to worker role communicationAllows roles to scale independently
Implement scheduling of asynchronous tasksBuilding process/work flows
Abstractions – FilesMove on-premises applications to cloudVMs can net use an SMB share using standard file APIs and semanticsSMB 2.1 protocolVM and storage account within same regionSupports REST and SMB protocol access to same file share
Azure Storage
Blobs
Tables
Queues
Files
Share data stored in Azure Files among Azure VMs via SMB
Microsoft Azure
SMB
REST
APIREST API
Additional Services, Tools and LibrariesAzure Import/Export
Move TBs of data into and out of Azure Blobs by shipping diskSubmit and monitor jobs via REST and PortalAll disks encrypted with BitLocker
Client LibrariesSimplify interaction with REST servicesConvenience capabilities – retry logic, parallelization, SAS tokens, … .NET (Desktop, Phone, Windows), Java, C++, Android, Node.JS, Python, PHP, Ruby, IOS (Coming Soon)
ToolsPowerShell commandsCLI toolsAzCopy 3.0 – Copy blobs and disks (Tables and Files in Preview Release)
Azure Storage Architecture
Massive Scale Out & Auto Load Balancing Index Layer
Distributed Replication Layer
Blob/Disk
EndpointQueue
EndpointTable
EndpointFile
ShareEndpoint
“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011
Client Libraries (.NET, Java, c++, Android, Node.JS…)
SMB Client
Import / Export
REST REST REST REST SMB 2.1
Stream Layer
Partition Layer
Front End Layer
Platform Capabilities
Locally Redundant Storage (LRS)Stores 3 replicas of the data within a single zone (facility) in a single regionProvides data durability for disk, node and rack failures
Geo Redundant Storage (GRS)Stores 6 replicas of the data across two regions (3 in each region)Provides additional durability to protect data against major regional natural disasters (e.g., tornado, hurricane, fire, etc, destroying a whole region)Updates across regions are performed asynchronously
Zone Redundant Storage (ZRS)Stores 3 replicas of the data across multiple zones (facilities) within a single region or across regions Provides additional durability to protect data against zone failures (e.g., fire burning down a facility)
Durability Options
Secondary Read-Only AccessCustomers using Geo Redundant Storage can opt to have read-only access to the eventually consistent copy of the data on Secondary tenantApplications control which location they read data from
Primary endpointaccountname.<service>.core.windows.net
Secondary endpointaccountname-secondary.<service>.core.windows.net
ConsistencyAll Writes go to the PrimaryReads to Primary are Strongly Consistent Reads to Secondary are Eventually Consistent
Our client library SDK provides features for reading from the secondary
Retry options: PrimaryOnly, SecondaryOnly, PrimaryThenSecondary, etc.
Access ControlSymmetric Shared Key AuthN
Trusted service that owns the storage accounts Shared Access Signature (SAS) Delegated AuthZ
3rd party servicesMobile device applicationsRestricted access for servicesAllow client applications to directly communicate with Storage rather than scaling a proxy web service
Public (Blob service only)CDN accessBrowser based access
ConcurrencyOptimistic Concurrency
Blobs and Tables support optimistic concurrencyBased on HTTP protocol (Etag and if-match conditions)
Pessimistic ConcurrencyBlocks provide exclusive access via leasesEnable synchronization strategies
Exclusive write / shared readExclusive write / exclusive read Shared write / exclusive read
Last Writer Wins
Storage AccountCapacity – Up to 500TBObjects per second: 20,000 1KB messagesScans/Listing considered as the number of objects read by the system and not the number of objects returnedBandwidth:
GRS: Up to 10 Gbps Ingress/20 Gbps EgressLRS: Up to 20 Gbps Ingress/30 Gbps Egress
Partition Table- Up to 2,000 1KB entities/secQueue - Up to 2,000 1KB messages/secBlob - Up to 60 MB/s, IOPS 500File - Up to 60 MB/s, IOPS 1000Check Azure Storage Scalability and Perf Targets
Scalability Targets
Storage AnalyticsStorage platform incorporates rich analytics
Metrics - Monitor service health, capacity, availability and performanceLogs – Detailed logs of every operation to support troubleshooting
Analytics is not enabled by default – we recommend enabling
During testing to establish baseline metricsIn production to assist with troubleshooting
Client libraries also enable access to metrics data via strongly typed objects Client libraries also have client side logging
Designing a Cloud Scale Application
Uniformly distributed traffic across the entire user partition namespace Range based partitioning does not cater to append/prepend patternAvoid spikes
Additional accounts to get more scale than a single accountDesign application to dynamically add accounts
Throughput: Use more simultaneous outstanding IOSingle blob: Parallel page writes or block writesMultiple blobs: Upload blobs in parallel
Scale Design Principles
Failures can occurRetry on Network/Connection failuresExponential back-off on timeout and throttling failures
Regional failures can occur Design for regional redundancyRA-GRS: Allows apps to continue with read only access to eventual consistent data
Network latencies can occur Latency sensitive apps could use cache.
Resilience Design Principles
Designing a SaaS solution to deliver Incident Management capabilities at hyper scale
Significant Use CasesCRUD operations on IncidentsGet Incidents assigned to a User
Architecture RequirementsHigh availability (99.99% for reads)Scalability beyond storage account limitsProvide direct access to storage to minimize complexity and cost
Scenario and Requirements
Use the right abstraction Key Lookups at scale for structured data – TablesScans/retrieve large amount of raw data like for analytics/metrics etc. – BlobsProcess workflows, decoupling applications – Queues
Storage Design Practices
Design for queryingSpecify both PartitionKey and RowKey in your queriesConsider storing duplicate copies of entities (with different keys)Consider denormalizing your dataDon’t create a separate table for each type of entityUse compound key valuesAvoid creating hot partitions
Table Design Principles
Table storage to store incident dataPK as Incident ID, RK as emptyIncident ID breakdown (OrgId, bucket ID, GUID)Table Property backed by blob
CRUD Operations
Partition Key(Incident ID)
Row Key Title(string)
AssignedTo(string)
Status (string)
Description(string)
contoso _01_8743428c-ef91-4d05-9e7c-4a2e856e813a
Support Retrieval workflow
user5 Active http://storageaccount1.blob.core.windows.net/backingcontainer/01_8743428c-ef91-4d05-9e7c-4a2e856e813a
Querying Incidents data as is - Table scan not optimalMake “assignedto” part of the pk or rkDenormalize common properties to avoid hops – Title and StatusUse Queues to get eventual consistencyKey Pattern – Write to queue first with timeout , than update primary entityWorker role then updates the secondary indexDEMO
Get Incidents Assigned to User
Partition Key(PropertyName_Value)
Row Key(Incident ID)
Title(string)
Status (string)
AssignedTo_user5 contoso _01_8743428c-ef91-4d05-9e7c-4a2e856e813a
Support Retrieval workflow
Active
Design the solution to be resilient to local and geographical failures
SolutionReplicate data between two regions using GRSRead from secondary region in the event of failure in first regionClient SDK’s retry policy automates this with a configuration setting
Enable application to run in read only mode until recovery/failover
High Read Availability
Enable clients to interact directly with storage to remove the need for a storage proxy – decreasing latency and reducing complexity
SolutionIssue SAS tokens to Clients for delegated authorization
Additional benefitsSAS tokens can restrict access to tenant specific data
Provide Direct Access to Storage
Solution needs to scale beyond storage account limitsSolution
Create only as many storage accounts as needed todayKeep a map of OrgId+ bucket id to a storage account nameWhen a bucket id fills up or storage account reaches limits (capacity/throughput), create a new bucket id and pick a storage account from pool for storing data
Helps with write availability for new incidents in DR scenario
Pattern – Scale beyond single storage account
Partition Key(OrgID)
Row Key(bucket id/sharding Index)
Storage Account(string)
IsActive(bool)
contoso 00 demo1 falsecontoso 01 demo2 trueelmaramsey 00 Demo1 true
Architectural Sketch
Storage Service
Table Service
Blob Service
Queue Service
Index Manageme
nt
Background processing
Service layerProvisioning
SAS Provider
P2.3 CRUD CRUD Incidents
P3. Update Indices
P1. Register New
Customer
P2.1 Get SAS TokenClientSign up/Admin
CRUD Incidents
P2.2 Lookup
Shard
RecapDesigning a SaaS solution to deliver Incident Management capabilities at hyper scale
Design Requirements and SolutionsMulti-tenant – SAS tokens for AuthZScale across storage accounts – Shard data across storage accountsOptimize resources – SAS tokens enable direct interaction with storageHigh availability in the face of intermittent and regional failures
Read access to secondary regionAbility to create new incidents in different storage accounts
Customizable incidents – Tables storageSupport different search strategies – Secondary indexesCoordinate updates to tables + blobs – Queue based coordination
Session Objectives And TakeawaysSession Objective(s): Provide an overview of the Storage platform abstractions and platform capabilitiesReview principles and patterns in the context of a real world scenarioWhat differentiates us:Strong consistency: Easier for applications to consumeGRS: Data replicated to DC 100s of miles apartRA-GRS: Eventual consistent access to your dataDisks & Azure Files: Enables “lift & shift” of applicationsStrong Development tools and libraries:
AzCopy, PowerShell, CLI, Client libraries in various languages
Resources
DocumentationAzure Storage BlogAzure Storage Documentation (Azure.com / Docs / Storage)REST and API Documentation (MSDN)
Client LibrariesAzure Storage NugetAzure Storage Client Library Source CodeSDK’s
Architectural Guidance (Azure.Com / Docs / Storage)
Azure Storage Performance and Scalability ChecklistMonitor, Diagnose and TroubleshootingManaging Concurrency in Azure Storage ApplicationsAzure Table Storage Design Guide (Coming Soon)
Azure Storage Resources
Continue the discussion… Storage Team Round Table5pm - 6:30pm in room 8.11
Thank you…
Come visit us in the Microsoft Solutions Experience (MSE)!Look for the Cloud and Datacenter Platform area TechExpo Hall 7
For more informationWindows Server Technical Previewhttp://technet.microsoft.com/library/dn765472.aspx
Windows Server
Microsoft Azure
Microsoft Azurehttp://azure.microsoft.com/en-us/
System Center
System Center Technical Previewhttp://technet.microsoft.com/en-us/library/hh546785.aspx
Azure Pack Azure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack
AzureImplementing Microsoft Azure Infrastructure Solutions
Classroomtraining
Exams
+
(Coming soon)Microsoft Azure Fundamentals
Developing Microsoft Azure Solutions
MOC
10979
Implementing Microsoft Azure Infrastructure Solutions
Onlinetraining
(Coming soon)Architecting Microsoft Azure Solutions
(Coming soon)Architecting Microsoft Azure Solutions
Developing Microsoft Azure Solutions
(Coming soon)Microsoft Azure Fundamentals
http://bit.ly/Azure-Cert
http://bit.ly/Azure-MVA
http://bit.ly/Azure-Train
Get certified for 1/2 the price at TechEd Europe 2014!http://bit.ly/TechEd-CertDeal
2 5 5MOC
20532
MOC
20533
EXAM
532EXAM
533EXAM
534
MVA MVA
ResourcesLearning
Microsoft Certification & Training Resourceswww.microsoft.com/learning
TechNetResources for IT Professionals
http://microsoft.com/technet
Sessions on Demandhttp://channel9.msdn.com/Events/TechEd
Developer Network
http://developer.microsoft.com
Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC
TechEd Mobile appPhone or Tablet
QR code
Evaluate this session
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.