how to send a receipt, topics in concurrency and distributed systems

How to Send a Receipt Topics in Concurrency and Distributed Systems | @pascallouisperez

The Situation‣ It’s Black Friday on an e-commerce site.

‣ When a customer makes a purchase,

‣ We send said customer a purchase receipt.

Not That SimpleSending a purchase receipt via email after a purchase is a concrete product requirement which takes us down into distributed systems design.

Generalizable ProblematicMany problems we encounter can be rephrased as the canonical ‘sending a receipt’ problem. And hence, understanding it well is a powerful tool.

How to Send a Receipt Topics in Concurrency and Distributed Systems

Bad Solution purchase_handler(…) { err = charge_cc(cc, order.amount); if err != nil { return err }

err = mark_as_paid(db, order); if err != nil { return err }

err = send_receipt(order.customer); if err != nil { return err }

err = mark_receipt_sent(db, order); if err != nil { return err }

return nil}

Bad Solution Mo Money Mo Problems

‣ Not easy to charge a credit card.

‣ Auth / Capture model? Single Charge?

‣ Gateway Idempotency?

‣ Handling of timeouts? TCP interruptions?

‣ Database crash?

purchase_handler(…) { err = charge_cc(cc, order.amount); if err != nil { return err }




return nil}

‣ Failure to mark as paid leads to dangling payment without a confirmed order!

‣ Or a paid order without an email receipt.

‣ Retrying the handler would double charge.

‣ (Many other bad things.)

Bad Solution Some of the Failure Modes

purchase_handler(…) { err = charge_cc(cc, order.amount); if err != nil { return err }




return nil}

(Shorthand) purchase_handler(…) { err = charge_cc(cc, order.amount); on err return

err = mark_as_paid(db, order); on err return

err = send_receipt(order.customer); on err return

err = mark_receipt_sent(db, order); on err return

return nil}

Let’s Fix Charging

Charging Single Call Model

‣ Creating a charge debits the customer’s credit card.

‣ Charges are uniquely identified, and the payment provider guarantees idempotency.

‣ Makes it easy to charge, simply retry until success. However, requires care to avoid dangling payments.

Charging Auth / Capture Model

‣ Authorize creates a pending transaction, and reserves funds. Auth are uniquely identified… And providers expose pseudo idempotency.

‣ Capture confirms the transactions.

‣ Makes it hard to charge. However, avoids dangling payments in failure cases.

(Let’s ignore this model for now.)

Charging Let’s Fix It

purchase_handler(…) { err = run_transaction(db, func(tx) error { order.cc_charge_uuid = new_uuid(…); save(order); }); on err return

charge, err = stripe.create_charge(order); on err return

err = run_transaction(db, func(tx) error { order.cc_charge = charge; save(order); }); on err return



return nil}

Charging The Pattern

Prepare, Do, Propagate

‣ Tell them what you are going to tell them;

‣ Tell them;

‣ Then tell them what you told them.






return nil}

Charging Let’s Fix It






return nil}

‣ No ‘cc_charge_uuid’: failed early, can retry.

‣ No ‘cc_charge’: retry payment, Stripe guarantees idempotency.

‣ Did we send the receipt?

How are Emails Sent Anyways?

$ telnet smtp.thatsanexample.com 25HELO thatsanexample.comMAIL from: <[email protected]>RCPT to: <[email protected]>DATAFrom: [email protected]: [email protected]: SMTP is low level and simple

This is an example..QUIT

Sending Emails Quick SMTP Primer

‣ RFC2821 for SMTP protocol; and

‣ RFC2822 for message format.

https://www.ietf.org/rfc/rfc2821.txt


Sending Emails Avoiding Duplicates

‣ Great. How do you avoid duplicate delivery of messages?


‣ Only four mentions of ‘duplicate’ in RFC2821:

1. Avoiding duplicates for mailing lists, not relevant;

2. Specification of a ‘text line’ which has a ‘dot duplicate’, not relevant;

3. Page 62 represented here, simplistic recommendation;

4. Reference 28 which points to an 1988 article “Duplicate messages and SMTP”, which became RFC1047.


‣ Oh, and RFC1047 is not useful.

Good Description of the Problem

“If the communications link fails during this synchronization gap, then the message has been duplicated. Both mailers have active copies of the message that they will try to deliver.”

Simplistic Suggestion

“The best way to avoid the synchronization problem is to minimize the length of the synchronization gap. In other words, receiving mailers should acknowledge the final dot as soon as possible and do more complex processing of the message later.”



‣ RFC2821 replaced RFC821 in 2001;

‣ In particular, it saw the introduction of a ‘message-id’.

‣ One can hope most clients correctly populate this field (just kidding); and

‣ One can hope most SMTP relays use this field to de-duplicate messages.



Sending Emails Email Delivery Services

‣ Your friendly REST speaking email delivery services!

‣ Except…


‣ SendGrid doesn’t expose the message-id.


‣ MailChimp doesn’t expose the message-id;

‣ Though it allows custom headers to be passed, hence likely making it possible to pass a message-id.


‣ SailThru has a ‘send_id’, yeah!(Note: Haven’t tested if this is used to populate the message-id. Seems to imply as much.)

‣ Impossible to avoid duplicates.

‣ Hence, we have

‣ No delivery; or

‣ At least once delivery.

(It sucks. SMS is worse. So is the post office frankly.)

Sending Emails The Conclusion

Let’s Fix It

Fix It, How? Back to the Subject at Hand






return nil}

‣ If we are unable to record the charge, don’t send the receipt.

‣ If we are able to record the charge, we must guarantee that the receipt will be delivered.

‣ Given what we learned about email delivery, our choice is between no delivery, or at least once delivery.

Fix It, How? Back to the Subject at Hand

purchase_handler(…) { err = run_transaction(db, func(tx) error { order.cc_charge_uuid = new_uuid(…); order.receipt_uuid = new_uuid(…); save(order); }); on err return


err = run_transaction(db, func(tx) error { order.cc_charge = charge; order.receipt_state = PENDING; save(order); }); on err return

return nil}

Prepare

‣ Record intent to send receipt.

‣ The intent includes the identifier of the receipt: it is this specific email we intend to send.

Do

‣ Charge

Propagate

‣ In the background, process the email queue and retry until confirmation of delivery.

Generalizing

Inventoried Item (e.g. a seated ticket to a show) Prepare: check availability, and ‘soft reserve’ to avoid double bookings. Propagate: confirm reservation.

Activating Gift Cards Prepare: soft creation of the card. Propagate: activate and fund.

CouponsPrepare: verify the validity, and ’soft redeem’ to avoid double use. Propagate: actually redeem it.

Similar Problematics Hard Across µServices Boundaries

The Order as Distributed Transaction CoordinatorDistributed transaction processing, and two-phase commit.

Order ManagementOrder Management

Purchase



Prepare: “Soft Reserve” Prepare: “Create Gift Card”

“The Commit Request Phase”



Do: “Charge”

“The Commit Phase”



Propagate: “Confirm” Propagate: “Activate” Propagate: “Send”

“The Success Phase”



Propagate: “Cancel” Propagate: “Dispose”

“The Failure Phase”

Propagate: “Reconcile to Cancel (or Refund) Dangling Payment”



Propagate: “Expiry” Propagate: “Garbage Collect”

“The (Unofficial) Cleanup Post Failure Phase”

In Closing‣ RTFM (“Read The Fucking Manual”)

‣ It's turtles all the way down, follow the thread

‣ Correctness at app level requires understand of low level details

‣ Distributed Systems are Hard

‣ No need to re-invent the wheel, most of CS was invented in the 1970s, and we’re just repurposing known techniques

‣ Design Early, Design Often

‣ You can’t iterate your way to correctness

‣ Think through your systems early, write it down, rinse, repeat

how to send a receipt, topics in concurrency and distributed systems

Engineering