Being RDBMS FreeDAVID HOERSTER

2014

About Me C# MVP (Since April 2011)

Sr. Director of Web Solutions at RGP

Conference Director for Pittsburgh TechFest

Co-Founder of BrainCredits (braincredits.com)

Past President of Pittsburgh .NET Users Group and organizer of recent Pittsburgh Code Camps and other Tech Events

Twitter - @DavidHoerster

Blog – http://geekswithblogs.net/DavidHoerster

Email – david@agileways.com

Traditional Architecture Data persistence is central to application

Generally monolithic

Jack of all trades; master of none

Traditional Architecture

Client Web Server

App Server

Data Repository

Data TablesSessionCache (?)FT Search

What if… We could break some pieces out

◦ Flatten structures for querying◦ Highly efficient search services◦ Pub/sub hubs◦ Remote caching with excellent performance◦ Session management outside a DB for load balanced environments

How would app then be architected?

Search How do you search?

◦ LIKE ‘%blah%’ ?◦ Dynamic SQL◦ Full-Text

LIKE and Dynamic SQL can be quick to create◦ Tough to maintain

Full-Text gives power◦ Limited in search options

Search Number of search services out there like

◦ Lucene◦ Solr

Lucene is a search engine◦ Embed in apps◦ .NET port (Lucene.NET)

Solr is search service◦ Built on Lucene◦ Connect apps to it

Searching with Solr Disconnected from your application

Search content via HTTP REST calls

Can use SolrNet as a client◦ https://github.com/mausch/SolrNet

Document-based

Searching with Solr private readonly ISolrOperations<T> _solr; public SolrSearchProvider(ISolrOperations<T> solr) { _solr = solr; }

public IEnumerable<T> Query(String searchString) { var options = new QueryOptions() { Fields = new List<String> {"title", "body", "lastModified" }.ToArray(), Highlight = new HighlightingParameters() { BeforeTerm = "<strong><em>", AfterTerm = "</em></strong>", Fields = new List<String> { "title", "body" }.ToArray(), Fragsize = 100 } }; var results = _solr.Query(new SolrQuery(searchString), options); return results; }

Evolving Architecture

Client Web Server

App Server

Data Repository

Data TablesSessionCache (?)

Search Service

Query

Write

Data Storage Typically, RDBMS is the de facto standard

◦ SQL Server◦ MySQL◦ PostgreSQL◦ Oracle (Yikes!!)

But do you really need it?

Data StorageGet all the orders for user ‘David’ in last 30 days

SELECT c.FirstName, c.MiddleName, c.LastName, soh.SalesOrderID, soh.OrderDate, sod.UnitPrice, sod.OrderQty, sod.LineTotal, p.Name as 'ProductName', p.Color, p.ProductNumber, pm.Name as 'ProductModel', pc.Name as 'ProductCategory', pcParent.Name as 'ProductParentCategory'FROM SalesLT.Customer c INNER JOIN SalesLT.SalesOrderHeader soh

ON c.CustomerID = soh.CustomerIDINNER JOIN SalesLT.SalesOrderDetail sod ON soh.SalesOrderID = sod.SalesOrderIDINNER JOIN SalesLT.Product p ON sod.ProductID = p.ProductIDINNER JOIN SalesLT.ProductModel pm ON p.ProductModelID = pm.ProductModelIDINNER JOIN SalesLT.ProductCategory pc ON p.ProductCategoryID = pc.ProductCategoryIDINNER JOIN SalesLT.ProductCategory pcParent ON pc.ParentProductCategoryID = pcParent.ProductCategoryID

WHERE c.FirstName = 'David'AND soh.OrderDate > (GETDATE()-30)

Data Storage

Wouldn’t it be great if it were something like this?

SELECT FirstName, MiddleName, LastName, SalesOrderID, OrderDate, UnitPrice, OrderQty, LineTotal, ProductName, Color, ProductNumber, ProductModel, ProductCategory, ProductParentCategoryFROM CustomerSalesWHERE FirstName = 'David'

AND OrderDate > (GETDATE()-30)

Data Storage Maybe a document database can be of use

Number out there◦ MongoDB◦ RavenDB◦ Couchbase

Flattened structures without relational ties to other collections

Essentially object databases

Looking at MongoDB Server can have databases

Databases contain collections (like a table)

Collections contain documents (like rows)

Documents can be structured, have constraints, have primary key

Working with Mongo’s C# Client public class MongoContext<T> : IContext<T> where T : class, new() { private IDictionary<String, String> _config; private readonly MongoCollection<T> _coll;

public MongoContext(IDictionary<String, String> config) { _config = config;

var client = new MongoClient(config["mongo.serverUrl"]); var server = client.GetServer(); var database = server.GetDatabase(config["mongo.database"]);

_coll = database.GetCollection<T>(config["mongo.collection"]); }

public IQueryable<T> Items { get { return _coll.FindAll().AsQueryable(); } } }

Working with Mongo’s C# ClientEncapsulate my queries and commands

public class FindPageById : ICriteria<Page> { private readonly String _id; public FindPageById(String pageId) { _id = pageId; }

public IEnumerable<Page> Execute(IContext<Page> ctx) { return ctx.Items.Where(p => p.Id == _id).AsEnumerable(); } }

Working with Mongo’s C# ClientInvoke my query/command public class TemplateController : IportalBaseController { private readonly IContext<Page> _pages;

public TemplateController(IContext<Page> ctx) : base() { _pages = ctx; }

[HttpGet] public async Task<IportalPageMetadata> Section(String cat, String page) { var id = String.Format("{0}/{1}", cat, page);

var thePage = new FindPageById(id) .Execute(_pages) .FirstOrDefault(); ... } }

Working with Mongo’s C# ClientWriting to Mongo is just as simple...

[HttpPost]public async Task<Boolean> Post(Page page){ var userId = await GetUserId();

new CreatePage(page, userId) .Execute(_pages);

_searchPage.Insert(page);

return true;}

Evolving Architecture

Client Web Server

App Server

Data Repository

Some data (?)SessionCache (?)

Search Service

Query

Write

Document Repository

Write

Query

Session and Cache Data Generally short-lived for users

Fairly static for cached data

Key/value stores can serve us well here◦ Redis

Redis has two good .NET client libraries◦ StackExchange.Redis◦ ServiceStack.Redis

Using Redis public class RedisSessionManager : ISessionManager { private static ConnectionMultiplexer _redis = null; private readonly IDictionary<String, String> _config;

public RedisSessionManager(IDictionary<String, String> config) { if (_redis == null) { _redis = ConnectionMultiplexer.Connect(config["session.serverUrl"].ToString()); } _config = config; } public async Task<Boolean> CreateSession(String portalId, String userId, String fullName) { var time = DateTime.UtcNow.ToString(); var timeout = _config.ContainsKey("session.timeout");

var vals = new HashEntry[] { new HashEntry("userid", userId), new HashEntry("login", time), new HashEntry("lastAction", time), new HashEntry("fullName", fullName) };

await RedisDatabase.HashSetAsync(portalId, vals); return await RedisDatabase.KeyExpireAsync(portalId, TimeSpan.FromMinutes(timeout)); } }

Using Redis

public async Task<Boolean> ExtendSession(String portalId) { var timeout = _config.ContainsKey("session.timeout"); await RedisDatabase.HashSetAsync(portalId, "lastAction", DateTime.UtcNow.ToString()); return await RedisDatabase.KeyExpireAsync(portalId, TimeSpan.FromMinutes(timeout)); }

public async Task<Boolean> ExpireSession(String portalId) { return await RedisDatabase.KeyDeleteAsync(portalId); }

Using Redis At login:

await Session.CreateSession(portalid, UserHandle, fullName);

Upon log out:

await Session.ExpireSession(portalCookie.Value);

Evolving Architecture

Client Web Server

App Server

Data Repository

Some data (?)Search Service

Query

Write

Document Repository

Write

Query

Session/Cache

Service

Why Data Store We’re left with a database with not much use

◦ Transactional data in document store◦ Search documents in Solr◦ Session, caching, etc. in key/value or caching service like Redis

What it probably ends up acting as is…

Evolving Architecture

Client Web Server

App Server

Event Store 2-3 flat tablesEvent data

Search Service

Query

Write

Document Repository

Write

Query

Session/Cache

Service

Queue?

(D)Evolved ArchitectureClient

Web Server

App Server

Event Store

Search Service

Query

Write

Doc Repo

Write

Query

Session/Cache

Service

Queue?

(D)Evolved Architecture Pick and choose what components work best

Don’t use them just to use them

Proof-of-Concept / Prototype

Why look to be RDBMS free Searching

◦ More than just full-text needs

Data◦ Choose a system that you can model the business◦ Not the other way around

Caching / Session Values / PubSub◦ Offload necessary?◦ Ensure performance

Maintenance and support big factors to consider