simple software solutions to big business problems

Making Software. Better.

Digital By DefaultBuilding APIs for Government

2

I believe the creation of the Government Digital Service is one of the great unsung triumphs of the last Parliament.

Prime Minister David Cameron

Govtalk Webservices - now

4

Where we’ll get to

Why an API platform

HMRC wants to improve end-user experience, and reduce errors. An API platform allows HMRC to take

control of the business logic again, simplifying integration. By doing that, software vendors

produce more innovative products or even better free software

API Benefits for others

Organisations can benefit from APIs in many other ways too. Business expansion with limited

involvement (franchises) and simpler integration with partners and internal software vendors.

10 Years of AWS - APIs are forever

“.......Once customers started building their applications and systems using our APIs, changing those APIs becomes impossible, as we would be impacting our customer’s business operations if we would do so. We knew that designing APIs was a very important task as we’d only have one chance to get it right.”http://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

A Practical GuideTo Building APIs at HMRC Digital

Practical Guide to Building APIs

Set out your visionSet out your vision

Always start with a vision

Your vision must be ambitious, clear, precise. It must set out where you want to get to, not where

you are.

10 Years of AWS - APIs are forever

“.......Once customers started building their applications and systems using our APIs, changing those APIs becomes impossible, as we would be impacting our customer’s business operations if we would do so. We knew that designing APIs was a very important task as we’d only have one chance to get it right.”http://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

Your organisation is ready

The fact that your organisation wants to create an API platform means they are ready to be better.

“We are not mature enough” does not hold.

An API Manifesto

At HMRC we have documented our vision into an API Manifesto, that is shared with the business,

architects and whole delivery team.

Our APIs are beautiful

HMRC Digital’s APIs are more than an interface to our code. They are a reflection of our brand, quality

and pride. They are the “web front end” to our services. Beauty is the following of standard

RESTFul principles, HMRC Digital API standards and consensus.

Our APIs are RESTFul

HMRC Digital APIs are RESTFul and therefore display all the characteristics of REST. There is no acknowledgement of multiple levels of REST and terms such as “Pragmatic REST” and “RPC REST”

are not valid.

Our APIs reflect domain - not process

HMRC Digital’s APIs are focussed on the domain at hand. We do not reflect business processes as

these represent internal implementation.

Our APIs must perform

HMRC Digital’s APIs must perform. Volumes, Throughput, Consistency, Accuracy, Security, Ease of Integration. Our APIs take advantage of the web infrastructure for scalability and ubiquity. We only

get one chance to make a good impression but hundreds of chances to get it wrong.

Our APIs are built for the end consumer

HMRC Digital APIs are not built for a specific system or application. They are not built to fulfil a

technical problem. They are built with our end consumer in mind and fulfil customer need.

Our Microservices are API First

All our microservices on the platform follow the guidelines of APIs, ensuring all of them are publishable even if not intended to be so.

Be RESTFulOur APIs are RESTFulOur APIs are Beautiful

Our APIs must perform

Be RESTFul

Our APIs must be beautiful

Our APIs are RESTFul

Our APIs must perform

No Hypermedia, No REST

“If the engine of application state (and hence the API) is not being driven by hypertext, then

it cannot be RESTful and cannot be a REST API. Period.” - Roy Fielding

Richardsons Maturity Model

Courtesy of www.martinfowler.com

But it’s misused and abused

Richardson’s Maturity Model has become a justification to not use Hypermedia. It is not meant to be a model for levels of valid REST

development. Just an observation of an organisation’s proximity to REST

Examples

Media Types

Media types give a structure for the content of your APIs, allowing clients to negotiate through

Hypermedia controls without needing prior knowledge of those controls.

Standards Based

A number of media types are attempting to be standards is this area. HAL, JSON-LD,

Collections+JSON are some of the better known ones in the JSON arena. These tend to

be generic and ensure looser coupling.

Examples

Vendor Specific Media Types

Vendor specific media types are custom types created for your organisation. These could be general organisation or specific by resource.

Organisation specific

While you like the idea of a generic media type for all cases, the standards based ones seem

like overkill to you, so one for your entire organisation fits better.

Resource specific

Resource specific media types would be created at a resource or domain level. For

example, a media type for a customer and one for order etc

Verbs

The HTTP spec has 8 standard verbs that the RESTFul Architectural Style takes advantage of. These have their very own characteristics and

should be used correctly

GET

The GET verb is safe and idempotent according to the spec. Do not abuse this. Clients expect that and if abused can bring about many side

effects. Only use to retrieve a resource.

POST

The POST verb is neither safe nor idempotent. The HTTP spec does not say how this should be

use. However, standard practice is when you wish to create a new resource whose ID is

unknown.

PUT

The PUT verb is unsafe but idempotent. Standard usage is for creating a new resource

where you know the ID or where you an updating an existing resource.

DELETE

The DELETE verb is unsafe but idempotent. Does what it says on the tin.

OPTIONS

Not widely used enough, the OPTIONS verb gives you information regarding an endpoint -

what verbs apply, what content type and relevant documentation. Take advantage of it.

Your clients will thank you.

Others

PATCH is for partial updates. Best to avoid where possible. HEAD, TRACE and CONNECT

rarely used or needed in your API platform

Status Codes

HTTP Status Codes provide a standard way to tell the client software what is happening. Use them correctly and they save you a lot of time.

Status Codes - some patterns

Use 200 - OK

when successfully returning a GET

Use 201 - Created

when successfully creating a resource

return resource location in header

Use 202 - Accepted

when request is valid and can be created

return resource location in header

Caching

The REST Architectural Style takes advantage of the web architecture to improve scalability. You need to take advantage of it by using the

correct HTTP headers.

Caching HTTP Headers

Expires and Cache-Control are the two response headers that allow for caching. These

set either expiry date or TTL.

RESTFul means loose coupling

By controlling flow and business logic within the API, thereby hiding implementation, you

ensure loose coupling between the client and the API.

REST means Smaller Payloads

Hypermedia can allow for smaller payloads. This reduces latency by having less sent over the wire but also minimising the amount of

orchestration on the server.

REST takes advantage of caching

The web architecture allows for caching and taking advantage of the GET method gives us

efficient responses.

Better/Cheaper Client Software

The biggest advantage of the RESTFul Architectural Style is that it minimises the

amount of knowledge your clients need to have about your system. Allowing them to integrate

quickly and innovate more rapidly.

Reflect DomainOur APIs are BeautifulOur APIs reflect domain - not process

Reflect Domain not ProcessesOur APIs are beautiful Our APIs reflect domain

Why not business processes?

Business processes were not created for the digital world. They were created for cryptic commands into mainframe terminals, paper forms and human (intelligent) intervention.

Reflect these digitally and you limit APIs

SOA’s biggest failure

SOA made the business process the key part of the process. Business Process Models were the

central artefacts. This created services that were too cumbersome and too tightly coupled

to the non-digital age.

Domain not processes

Create APIs that reflect your domain, using Domain Driven Design concepts. You will not limit yourselves to existing processes, you’ll

hide the implementation and you’ll use industry language. In so doing, complexity is

reduced/isolated/removed.

An Example

The APIs

/users/{id}

/users/{id}/roles

/users/{id}/ratings

/users/{id}/friends

/roles/{role}

/movies/{movie-name}

/movies/{movie-name}/ratings

/movies/{movie-name}/actors

/actors/{actor-id}

cont…..

cont…..

Versioning Strategy

Versioning StrategyOur APIs must perform

To version or not to version

Why it’s best not to version

Versioning APIs introduces a number of significant complexities that you would need to

overcome. How many versions should you support, how do you support them, what

counts as a new version, what sort of deprecation strategy, what if you backend

systems are not versioned etc…..

Why you shouldn’t need to version

Hypermedia controls should mean your clients are so loosely coupled to your APIs that they

need not be affected by changes. They would ignore changes until they are ready.

Why you need to think about versioning

What if you cannot prevent breaking changes? What if too many clients are hard coding

instead of using Hypermedia controls?

Versioning guidelines - Include versions

Include a version number in your APIs - ideally in the header as part of the Accept header but

at least as part of the url.

Versioning guidelines - Don’t do it

Do everything you can to avoid breaking changes and backwards compatibility. Follow RESTFul guidelines and you will find this easy

Versioning guidelines - Deprecate

With every new version, deprecate your old version straight away with a maximum support period for bug fixes. New features will not be

included in deprecated version

Ensure SecuritySecure your APIs

Our APIs must perform

Web Security Economics

“1. Good enough is good enough. 2. Good enough always beats perfect. 3. The really hard part is determining what is good enough”

Prof. Ravi Sandhu

Valet Parking Problem

One of the biggest and most prominent issues in web security is over privileging users. Like giving your keys to a valet, they can find out

where you live, have your house keys and know where you’ll be for next 2 hours.

Switch on TLS Encryption

TLS/SSL performance is no longer the issue it once was. Companies such as Netflix have

switched on encryption for its video streaming. With the low overhead it would be silly not to.

HTTP Basic Auth

The simplest form of API authentication, uses Base64 encoding. Easiest to implement and

easiest integrate with. But also easiest to hack. Encoding is not the same as encryption. Only

use with TLS

OAuth 2.0

The current defacto standard for API authentication and authorisation. Will only

work with TLS. Works by exchanging of encrypted tokens.

OAuth 2.0

The user is granting access to certain privileges for the software. With trusted clients, implicit

grant access can be provided.

OAuth 2.0

Tokens are tied to developers - allowing closing down a link with a developer that may have been compromised rather than the whole

platform.

These are not enough

Beyond your API platform, the rest of your technical platform must be secure and your

organisation must be secure. Techniques such as transaction monitoring and fraud detection

can be useful.

Ultimately, it’s about quality

All the encryption, TLS, auth, firewalls or big burly bouncers will do you no good if your code

has bugs and your design is flawed.

Documenting your APIsOur APIs are beautiful

Our APIs are RESTFul

Documentation Tools

There are a number of REST JSON documentation tools. RAML, Swagger and

Hydra. But they each have flaws.

RAML and Swagger

RAML and Swagger are more than just documentation tools. They provide define, build, test and documentation frameworks.

However, they do not support RESTFul attributes, in particular hypermedia controls

Hydra

Less of a fully functioning framework than the other two but does support hypermedia

controls…...but only if you use the JSON-LD media type

Proprietary Documentation

It’s important to not be tempted to limit your vision to use those great tools. Instead, in this

case, a proprietary documentation tool built in-house will be preferable. Make it easy for your

developers.

API Management ToolsOur APIs must perform

Features

API Management tools provide a number of important features:

Manage API Accesses

Manage Traffic

Monitoring

Manage Available APIs

Products

Leading API Management tools are provided by WSO2, CA Technologies, Mulesoft, Mashery

Don’t be constrained

Your API Manager needs to support your vision. Don’t change your vision for the tool. But don’t

try doing it yourselves.

Good API Platforms

● Github.comhttps://developer.github.com/v3/pulls/

● Stormpath.comhttps://docs.stormpath.com/rest/product-guide/latest/reference.html#application

● Paypal.comhttps://developer.paypal.com/docs/rest/api/

● Visahttps://developer.visa.com

Examples

Homework

1. Rest in Practice - Jim Webber2. Domain Driven Design - Eric Evans3. RESTFul Web Services Cookbook - Subbu Allamajru4. http://martinfowler.com/articles/richardsonMaturityModel.html5. Maze+XML by Mike Amundson

○ http://amundsen.com/media-types/maze/6. Resource-Oriented Architectures: Hypermedia

○ https://www.safaribooksonline.com/library/view/resource-oriented-architectures-hypermedia/9781491924877

7. Government Digital Service - Digital by Default○ https://www.gov.uk/service-manual/digital-by-default

8. Introduction to Secure Software○ https://www.safaribooksonline.com/library/view/introduction-to-secure/9781491943595

Thank You

Twitter@EqualExperts

LinkedInlinkedin.com/company/equal-experts

UNITED KINGDOM

+44 203 603 7830

helloUK@equalexperts.com

Equal Experts UK Ltd

30 Brock Street

London NW1 3FG

INDIA

+91 20 6607 7763

helloIndia@equalexperts.com

Equal Experts India Private Ltd

Office No. 4-C

Cerebrum IT Park No. B3

Kumar City, Kalyani Nagar

Pune, 411006

Webwww.equalexperts.com

CANADA

+1 403 775 4861

helloCanada@equalexperts.com

Equal Experts Devices Inc

205 - 279 Midpark way S.E.

T2X 1M2

Calgary, Alberta

Twitter: @softwarereda

LinkedIn: linkedin.com/in/redahmeid