taming the rising complexity of event-driven apis · 2019-10-28 · page 7 ably - serious realtime...

Taming the rising complexity of event-driven APIs Useful insights for API publishers

My journey with event-driven APIs

Technical co-founder of Ably Built and scaled event-driven APIs for developers. Board member OpenAPI initiative

Ably - Serious realtime infrastructure

What is an event-driven API?

Delivery person Logistics provider

Retailer

Parcel arrived! (on device)

Customer

Ping


Pull (request/response) is stateless & simpler

Data source Worker Server Consumers


Event-driven is more demanding on producers

Worker hot spots

Additional complexity:

Backoffice scheduling & management

Short value window

Delivery logistics

Acme Fresh Food


Complexity inversion with event-driven APIs Pull (REST) API Event-driven API

Consumer

Producer

Trigger requests and maintain state

API Gateway / CDN Fan out push,

maintain state, and fault tolerant.

Complexity

Wait for updates

Stateless response


If complexity is evil - can we avoid it? Probably not.

Event-driven data exceeds all

data produced in 2019 (circa

40Zb)


Event-driven data demand is here, now By 2021, cumulative event-

driven data exceeds all

previous event-driven data

produced and consumer,

ever


Criticality of this data

Compound Annual

Growth Rate

All data Potentially Critical Critical Hyper-Critical

30% 37% 39% 54%


Source of new event-driven data


Terminology: The Realtime API family

Pub/sub Messaging pattern

Streaming Consumer pattern

Event-driven Architectural

pattern

Push Producer pattern

Realtime

APIs

Publishing a realtime API


Infrastructure

Publishing

Documentation / Spec

Three distinct functions when publishing a realtime API

Distribute Deploy

Authentication

Pull & push protocols

Onboarding + dev tooling

Scaling & performance

Manage

Access control

Instrumentation

Billing

Monetization


Three distinct functions when publishing a realtime API

Distribute Deploy

Authentication

Pull & push protocols

Onboarding + dev tooling

Scaling & performance

Publishing

Infrastructure

Documentation / Spec

Manage

Access control

Instrumentation

Billing

Monetization

Distribution layer complexity 1. Integrity vs Latency

2. Throughput 3. Push vs Pull

subscriptions 4. Downstream reliability 5. Durability

Five commonly unforeseen but important areas of complexity in event-driven API delivery.

Distribution layer complexity #1 Integrity vs Latency


Integrity vs Latency

Happy Path

Your servers

message bus

API Interface

15 updates a minute


Integrity vs Latency: Congestion & connectivity issues

Backpressure Your servers

message bus

Less Happy Path


Solution #1 - Backpressure control

Time: 0s 10s 20s 30s 40s 50s 60s 70s 80s

Backpressure control Bandwidth limited

Managing backpressure Server-side

● TCP buffer size with high watermark Network layer

● ACK from subscribing clients Application layer

Client-side

● Stream polling


Integrity vs Latency: Conflation

Pricing

Scores GPS locations

Time: 1s 2s 3s 4s

Great for:


Simplified example of conflation on a stream


Considerations when latency takes priority

● Ordering of streams is typically not required Less complexity

● Backpressure management or conflation preferred Additional complexity

● Capacity planning critical Additional operational complexity

● Persistent subscriber transport preferred Improved latencies as reduced round-trip overhead


Integrity vs Latency: Integrity prioritized

Ordered message stream

Auditors & Legal

message bus


Considerations when integrity takes priority

● Ordering of streams is important Significant complexity increased in a stateful design (serial numbers)

● Backpressure management still needed Buffers cannot build up indefinitely. Some increased complexity

● Persistent subscriber transport preferred Removes complexity in the subscriber as TCP can be relied upon for integrity

● Reliable publishing ACKs, persistent connections & idempotency


Publisher integrity with idempotent publishing

Post A

Cloud Service

Connection Disconnected

Response

No Response, Retry Post A

Response A


Integrity vs Latency in summary

● Consumers decide on integrity vs latency ● Producers of data can choose both latency and integrity

○ To gain integrity, idempotency and persistent connections preferred ● Latency over integrity is generally simpler ● Backpressure needs to be handled always

Distribution layer complexity #2 Throughput


Throughput - narrow pipe

Multiple producers

Fat message

bus Single stream

Consumer Max 1,000 msgs p/s

per stream


Throughput: #1 solved using queues Advantages: Relatively simple Scales easily Disadvantages: No ordering At least once delivery Potentially two design patterns in play - pub/sub and queueing. Consumer

workers

Queue

500 msgs p/s

Fat message

bus

Multiple producers


Advantages: Data stream integrity (ordering) Single pattern pub/sub Exactly-once delivery Disadvantages: Sharding is complex Expanding or contracting shards is hard Consumer state may be needed

Throughput: #2 solved with sharding

Shard consumers

Consumer group 1

2

4

3

Throughput sharding

Fat message

bus

Multiple producers


● Everything has limits - distribution is your friend Practical limits in terms of what a single server, connection or shard can sustain.

● Keep complexity away from your consumers It’s your job to keep it simple for consumers.

● Queues are simple Yet lack ordering and exactly-once delivery guarantees

● Shards are hard Providesordering and exactly-once delivery guarantees

Throughput in summary

Distribution layer complexity #3 Push vs Pull Subscriptions


Push vs Pull subscriptions

AMQP

API Gateway

Push service

Client initiated

Server initiated

HTTP

WebSub

Your message bus


Examples of Push subscription protocols

Stateless over HTTP Queue based Stream based

Webhooks AMQP Kafka

WebSub AWS SQS AWS Kinesis

Serverless function

invocation

MQTT

Integrations (Zapier

etc)


When to use push vs pull subscriptions?

Push subscriptions

● High throughput Target can be load balanced.

● Reduced consumer complexity Creates complexity for producer, and you need to address durability and downstream failures

● Always online Unsuitable for devices that are not always online.

● Unintentional DoS risk Control rate of downstream requests.

Pull subscriptions

● On demand Generally better suited for devices such as mobiles, desktops, browsers where data is needed on demand.

● Simple Simple for consumers and producers.

● Low throughput per subscriber Not suitable for high throughput, without sharding or queueing.

● Capacity planning harder Unpredictable load.

Distribution layer complexity #4 Downstream reliability (for push subscriptions)


Downstream reliability - push subscriptions

Your message bus

Push service

Ordered stream

HTTP

Kafka

AMQP


Downstream reliability - coping with failure

Poison message 50x

Faulty connection

Your message bus

Push service

Ordered stream

HTTP

Kafka

AMQP


Dead letter queues

Dead letter queue

Failed messages

Your message bus Ordered stream

HTTP

Kafka

AMQP


● Manage throughput to avoid unintentional DoS attacks Avoid unintentional floods of requests to downstream endpoints. Use rate limiting, and incremental back-offs.

● Push subscriptions may prefer latency over integrity Latency prioritized traffic may allow messages to be discarded when there is a failure.

● Dead letter queue poison / bad messages Don’t delete data.

● Tooling and alerts Ensure downstream providers have necessary tooling and alerts to manage failures.

Downstream reliability considerations

Distribution layer complexity #5 Durability


Durability

Resume from #1 (-2 hours)

Persistent ordered index log stream

Resume from #6 (-1 minute)

10 9 8 7 6 5 4 3 2 1


Durability considerations

● Use a dedicated log stream storage solution Trying to get traditional databases to store log streams is hard. There are many solutions that provide efficient append-only stream storage.

● Complexity of adding storage is probably worth it Freedom to later improve reliability, reduce backpressure issues, better continuity of streams.

● Latency and financial cost The financial impact can be managed to some degree by modifying TTLs for storage.

Event-driven APIs do introduce complexity

But it’s not rocket science :)


What next?

#1 Give developers what they want - better event-driven integrations

Event driven integration frustration

up 2% this year

Source: State of API Integration Cloud Elements


What next?

#2 Avoid complexity ● Focus on necessary complexity ● Delay complexity that is not needed to day, but plan for it ● Open source and cloud solutions exist - don’t reinvent the

wheel


#3 Focus on developer experience ● Don’t downstream complexity to consumers ● Let consumers choose push or pull protocols, integrity or

latency ● Documentation and developer portals are important ● An API is a contract and commitment to your consumers

What next?


#4 Help build an open realtime connected world ● Provide event-driven APIs ● Set your data free

What next?

Thank you Taming the rising complexity of event-driven APIs

www.ably.io

Me: @mattheworiordan @ablyrealtime

Shameless plugs: go.ably.io/open-data-streams go.ably.io/api-management

Illustrations by Leonie Wharton - leoniewharton.com

https://www.ably.io

http://www.ably.io

https://twitter.com/mattheworiordan

https://twitter.com/ablyrealtime?lang=en

https://go.ably.io/open-data-streams





https://go.ably.io/api-management



http://leoniewharton.com/

taming the rising complexity of event-driven apis · 2019-10-28 · page 7 ably - serious realtime...

Documents