Windows Azure Blob Storage – A Deep Dive

Paul Bouwer@pbouwer

http://blog.paulbouwer.com

A service for storing large amounts of

unstructured data that can be accessed from

anywhere in the world via HTTP or HTTPS.

Windows Azure Blob Storage

• Serving images or documents directly to a browser

• Storing files for distributed access

• Streaming video and audio

• Performing secure backup and disaster recovery

• Storing analysis data by an on-premises or Windows Azure

hosted service

3 copies per data centre

> 500 miles

Continuous geo-replication

• Start with letter or number

• Contain only letters, numbers, and dashes (-)

• Dashes (-) must be immediately preceded and followed by a

letter or number. Consecutive dashes not permitted

• All letters must be lowercase.

• Between 3 and 63 characters in length

• Case sensitive

• Any combination of characters

• Reserved URL characters must be escaped

• Between 1 and 1024 characters in length

• Targeted at streaming workloads

• Comprise of blocks

• Block sizes may differ but max size 4MB

• No more than 50,000 blocks

• Max size 200GB

• Targeted at random read/write workloads

• Comprise of 512 byte pages

• Max size 1TB

• Private (Default)

Container only accessible by account owner

• Public Container

Full public read access to container and blobs

• Public Blob

Public read access to blobs

The term Representation State Transfer (REST) was coined by Roy

Fielding in his Ph.D. dissertation “Architectural Styles and the

Design of Network-based Software Architectures.”

Roy Fielding - one of the principal authors of the HTTP specification and co-founder of the

Apache HTTP Server project.

The REST architectural style is commonly applied to the design of

APIs for modern web services. A Web API conforming to the REST

architectural style is a REST API.

• Microsoft.WindowsAzure.Storage.dll

• NuGet – WindowsAzure.Storage

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

<?xml version="1.0" encoding="utf-8"?><configuration><appSettings>

<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" /></appSettings>

</configuration>

App.config | Web.config

C#

HEAD http://myaccount.blob.core.windows.net/mycontainer?restype=container

PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container

CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");container.CreateIfNotExists();

C#

REST API http://msdn.microsoft.com/en-us/library/dd179468.aspx

PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=acl

x-ms-blob-public-access: blobx-ms-blob-public-access: container

CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");

// Public Containercontainer.SetPermissions(new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Container });

// Public Blobcontainer.SetPermissions(new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Blob });

C#

REST API http://msdn.microsoft.com/en-us/library/dd179391.aspx

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); using (var fileStream = File.OpenRead(@"path\myfile")) {blockBlob.UploadFromStream(fileStream);

}

C#

REST API http://msdn.microsoft.com/en-us/library/dd179451.aspx

GET http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=list

GET http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=list&prefix=&delimiter=

foreach (var blockBlob in blobContainer.ListBlobs(prefix: null, useFlatBlobListing: true)) {}

foreach (var blockBlob in blobContainer.ListBlobs(prefix: prefix, useFlatBlobListing: false)) { if (blockBlob is CloudBlockBlob) { ... }if (blockBlob is CloudBlockDirectory) { ... }

}

C#

REST API http://msdn.microsoft.com/en-us/library/dd135734.aspx

GET http://myaccount.blob.core.windows.net/mycontainer/myblob

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); using (var fileStream = File.OpenWrite(@"path\myfile")) { blockBlob.DownloadToStream(fileStream);

}

C#

REST API http://msdn.microsoft.com/en-us/library/dd179440.aspx

DELETE http://myaccount.blob.core.windows.net/mycontainer/myblob

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); blockBlob.DeleteIfExists();

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.Delete();

C#

REST API http://msdn.microsoft.com/en-us/library/dd179413.aspx

HEAD http://myaccount.blob.core.windows.net/mycontainer/myblob

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=properties

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); blockBlob.FetchAttributes();Console.WriteLine(blockBlob.Properties.ETag);

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.Properties.ContentType = @"text/plain"; blockBlob.SetProperties();

C#

REST APIhttp://msdn.microsoft.com/en-us/library/dd179394.aspx

http://msdn.microsoft.com/en-us/library/ee691966.aspx

HEAD http://myaccount.blob.core.windows.net/mycontainer/myblob

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=metadata

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.FetchAttributes();Console.WriteLine("{0}", blockBlob.Metadata.ContainsKey("mymeta") ? blockBlob.Metadata["mymeta"] : "-");

var metadata = new Dictionary<string, string>();CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); foreach (KeyValuePair<string, string> pair in metadata) { blockBlob.Metadata.Add(pair); } blockBlob.SetMetadata();

C#

REST APIhttp://msdn.microsoft.com/en-us/library/dd179350.aspx

http://msdn.microsoft.com/en-us/library/dd179414.aspx

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var etag = blockBlob.Properties.ETag; var accessCondition = new AccessCondition {IfMatchETag = etag};blockBlob1.SetMetadata(accessCondition);

C#

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=lease

x-ms-lease-action: <acquire | renew | change | release | break>

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");

// Leases must be 15 - 60s or infiniteblockBlob.AcquireLease(TimeSpan.FromSeconds(15), leaseId);blockBlob.SetMetadata(new AccessCondition { LeaseId = leaseId }); blockBlob.ReleaseLease(new AccessCondition { LeaseId = leaseId });

C#

REST API http://msdn.microsoft.com/en-us/library/ee691972.aspx

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=snapshot

GET http://myaccount.blob.core.windows.net/mycontainer/myblob?snapshot=<DateTime>

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");CloudBlockBlob snapshot = blockBlob.CreateSnapshot();

CloudBlockBlob backup = new CloudBlockBlob(blockBlob.Uri, ((CloudBlockBlob)snapshot).SnapshotTime, blobClient.Credentials); blockBlob.StartCopyFromBlob(backupSnapshot);

C#

REST API http://msdn.microsoft.com/en-us/library/ee691971.aspx

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=block&blockid=id

PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=blocklist

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.PutBlock(blockId1, memStream, null); blockList.Add(blockId1); blockBlob.PutBlock(blockId2, memStream, null); blockList.Add(blockId2); blockBlob.PutBlockList(blockList);

C#

REST APIhttp://msdn.microsoft.com/en-us/library/dd135726.aspx

http://msdn.microsoft.com/en-us/library/dd179467.aspx

• Authentication Schemes

Authentication via Storage Key

• Shared Access Signature

Single use authentication and access

• Shared Access Policy

Revokable authentication and access

• SharedKey

• SharedKeyLite

Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"

StringToSign = VERB + "\n" +Content-Encoding + "\n"Content-Language + "\n"Content-Length + "\n"Content-MD5 + "\n" +Content-Type + "\n" +Date + "\n" +If-Modified-Since + "\n"If-Match + "\n"If-None-Match + "\n"If-Unmodified-Since + "\n"Range + "\n"CanonicalizedHeaders + CanonicalizedResource;

Signature=Base64(HMAC-SHA256(UTF8(StringToSign)))

http://msdn.microsoft.com/en-us/library/dd179428.aspx

GET\n /*HTTP Verb*/\n /*Content-Encoding*/\n /*Content-Language*/\n /*Content-Length*/\n /*Content-MD5*/\n /*Content-Type*/\n /*Date*/\n /*If-Modified-Since */\n /*If-Match*/\n /*If-None-Match*/\n /*If-Unmodified-Since*/\n /*Range*/x-ms-date:Sun, 11 Oct 2009 21:49:13 GMT\nx-ms-version:2009-09-19\n /*CanonicalizedHeaders*//myaccount/myaccount/mycontainer\ncomp:metadata\nrestype:container\ntimeout:20 /*CanonicalizedResource*/

Authorization: SharedKey myaccount:ctzMq410TV3wS7upTBcunJTDLEJwMAZuFPfr0mrrA08=

StringToSign = VERB + "\n" +Content-MD5 + "\n" +Content-Type + "\n" +Date + "\n" +CanonicalizedHeaders + CanonicalizedResource;

Signature=Base64(HMAC-SHA256(UTF8(StringToSign)))

http://msdn.microsoft.com/en-us/library/dd179428.aspx

PUT\n /*HTTP Verb*/\n /*Content-MD5*/text/plain; charset=UTF-8\n /*Content-Type*/\n /*Date*/x-ms-date:Sun, 20 Sep 2009 20:36:40 GMT\nx-ms-meta-m1:v1\nx-ms-meta-m2:v2\n /*CanonicalizedHeaders*//testaccount1/mycontainer/hello.txt /*CanonicalizedResource*/

Authorization: SharedKeyLite myaccount:ctzMq410TV3wS7upTBcunJTDLEJwMAZuFPfr0mrrA08=

https://myaccount.blob.core.windows.net/mycontainer/myblob?sv=2012-02-12&se=2013-01-16T00%3A45%3A58Z&sr=b&sp=r&sig=NqVpCv9LQGUDbD8hOKz%2FO4ybcZlU9yOB6G42xaAbq%2Bw%3D

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var signature = blockBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy{ Permissions = SharedAccessBlobPermissions.Read, SharedAccessStartTime = null, SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(1)

}); Console.WriteLine"Shared Access URI:{0}{1}", blockBlob.Uri.AbsoluteUri, signature);

C#

URL http://msdn.microsoft.com/en-us/library/hh508996.aspx

PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=acl

var blobContainerPermissions = new BlobContainerPermissions(); blobContainerPermissions.SharedAccessPolicies.Add("ReadOnlyPolicy", new SharedAccessBlobPolicy{ Permissions = SharedAccessBlobPermissions.Read, SharedAccessStartTime = null

}); blobContainer.SetPermissions(blobContainerPermissions);

C#

REST API http://msdn.microsoft.com/en-us/library/ee393341.aspx

https://myaccount.blob.core.windows.net/mycontainer/myblob?sv=2012-02-12&se=2013-01-26T00%3A57%3A35Z&sr=b&si=ReadOnlyPolicy&sig=X2mv6QDv%2FeMoNDvCheoJIMC8QzPpJSDoVAUhvEvbzC0%3D

CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var signature = blockBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy{ SharedAccessExpiryTime = DateTime.UtcNow.AddDays(10)

}, "ReadOnlyPolicy");

Console.WriteLine"Shared Access URI:{0}{1}", blockBlob.Uri.AbsoluteUri, signature);

C#

URL http://msdn.microsoft.com/en-us/library/hh508996.aspx

• Storage Analytic Metrics

Aggregated transaction statistics

Capacity data

• Accessed via Table Storage

$MetricsTransactionsBlob

$MetricsCapacityBlob

http://msdn.microsoft.com/en-us/library/windowsazure/hh343258.aspx

http://msdn.microsoft.com/en-us/library/windowsazure/hh343264.aspx

• Storage Analytic Logging

Successful and failed requests

• Access via Blob Storage

$logs

<service-name>/YYYY/MM/DD/hhmm/<counter>.log

http://msdn.microsoft.com/en-us/library/windowsazure/hh343262.aspx

• How to use the Windows Azure Blob Storage Service in .NET

http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/

• MSDN - Blob Service REST API

http://msdn.microsoft.com/en-us/library/windowsazure/dd135733.aspx/

• Library Reference - Microsoft.WindowsAzure.Storage.Blob

http://msdn.microsoft.com/en-

us/library/windowsazure/microsoft.windowsazure.storage.blob.aspx

• Azure Storage Samples (REST and .NET Storage Client Library)

• http://azurestoragesamples.codeplex.com/

• Azure Storage Explorer - FREE

http://azurestorageexplorer.codeplex.com/

• Windows Azure Explorer - FREE

http://www.cloudberrylab.com/free-microsoft-azure-explorer.aspx

• CloudXplorer - $35

http://clumsyleaf.com/products/cloudxplorer

• Cloud Storage Studio 2 - $195.00

http://www.cerebrata.com/Products/CloudStorageStudio/