scaling api design
DESCRIPTION
Some of the discipline and principles the "Paypal as a Service" is using to create a REST API driven platform across all of Paypal engineering. As presented at @APIWorld 2014 in San Francisco 9/17TRANSCRIPT
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
Scaling API Design
Jason Harmon, Head of API DesignSeptember 2014
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
2
About meScaling API Design
Jason Harmon
• Leads API design at Paypal
• Design phase of the “PPaaS” aka “Paypal as a Service" program
• Engineering-wide initiative
• Collaborate on designs for all internal/external/partner/whatever APIs
• Maintain style/standards
• Stakeholder for internal developer portal & tools teams
• Internal API design training/evangelism
@jharmn
Jasonh-n-austin
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
3
Let’s think bigWhat if your startup takes off?
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
4
Distributed architectureBreaking down the monolith
Everyone starts with a monolith
Defining uniform interfaces helps define bounded contexts
• Define interfaces, then perform surgery
Allows for scaling per capability
• When one component is overloaded, scale it!
• Harder to DDoS a distributed platform
Increases team autonomy
• Implementations can fit the need
• Languages & data don’t have to be all the same
Discoverability is a hard problem in big systems
• Reduce duplication of efforts
• Create visibility to weak spots in capabilities
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
5
PortfolioThink about the big picture
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
6
PortfolioOrganizing your APIs
Group operations by goals/usage
API Product Managers are key, but they’re hard to find
Start with capabilities, not resources
• Identify capabilities by way of use cases
Provide internal/external/partner developer portals to provide visibility
Use caution with product names
Respect customer language
Inverse Conway’s Law
“Organizations which design systems are constrained to produce designs which are
copies of the communication structures of these organizations.”
• Don’t design your APIs to reflect your systems or organizations
• Make your software look like your customers see you
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
7
PortfolioBusiness decides, developers implement
Business
• Identify capabilities
• How will business understand what your API can do?
• Sometimes a capability is a resource collection
• More often it’s a package of resources & operations
• Level 1 Categories + Package/Spec/Level 2
Developers
Use ‘namespaces’ to designate functional areas: /v1/factory/widgets
• Not always the same as capabilities/packages
URIs relay data relationships
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
8
Design firstRight after portfolio
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
9
Design firstBuilding backend is expensive
Break changes early, before you build it
Specification formats
• Swagger, RAML, Blueprint: whatever suits you
• Portal/Docs/Reference
• Codegen server/client/SDK
• Mocking
• Consistency Validation
Respect spec storage
• Separate from implementation, track minor versions on iterations
• All in one place – Platform skeleton
• Docs are specs too
Get API client feedback on mock APIs
• Real usability is only measurable with tactile feedback
• Weakness: multi-scenario and errors are hard to mock
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
10
Design first: Parallelize
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
11
FundamentalsCore elements of API design
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
12
SustainabilityLong live v1!
• Rapid iteration/fundamental changes are off-limits once it’s released
• Extensible API designs aren’t really a thing
• Rule #1 of API versioniong: try not to
Can we grow this design without starting over?
Hide implementation details
• Today’s backend is tomorrow’s scrap heap
• Watch errors
Add URIs, deprecate URIs; new designs are usually new resources
• APIUX: http://apiux.com/2014/09/05/api-design-sustainability
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
13
UsabilityThink about API clients’ perspective
Nouns matter• Resource-orientation
• Avoid RPC unless you can rationalize optimized DX
• Understandable terminology
• Use industry-standard terminology where possible
• Avoid vague terms: metadata, context, etc
Get your flow on
• Capture current and future use cases
• Identify goals
• Analyze chain of calls required to get there
• Pay attention to identifiers in each API to ensure they chain smoothly
REST != CRUD
• Think beyond data structures, think resources
• Resources should quickly reach client goals without excessive complexity
http://softexpert.files.wordpress.com/2007/10/52660486_6ca085f7a8.jpg?w=780
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
14
ScalabilityWe’re talking design scale, not system scale…or both
The Goldilocks principle
Be smart about “just right” sized resources
Big resources can be a problem
• System overload/performance issues
• Coupling concerns
• Long, unreliable HTTP connections
• Bandwidth overhead
• Complexity!
Tiny resources can be just as bad
• N+1 calls tend to proliferate
• Lots of TCP sockets https://img1.etsystatic.com/000/0/5414982/il_fullxfull.191894533.jpg
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
15
ConsistencyStick to the plan
Standards, guidance, patterns
Naming conventions: Field, parameter, URI
Define HTTP interactions
• Common patterns
Identify common components
• Addresses, user info etc
Headers are platform plumbing
• Use them sparingly
Ensure consistent identity mechanisms
• Preserve external identity through
internal platform calls http://minorcreations.files.wordpress.com/2012/07/one.png
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
16
DecentralizeEducate and cooperate
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
17
DecentralizeEducate and cooperate
Internal evangelism
Educate developers on program/process, standard, and principles
Identify thought leaders and fast learners
• Develop API design mentorship
• Ongoing communication
Listen to feedback on gaps in understanding
• Conduct regular feedback sessions from frontend and backend devs
• Hackathons inside & out
• Dedicate to document anything you have to answer twice
• Standards are nice, but guidance is better
Highlight outstanding design and collaboration examples
© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.
18
Thanks!
Jason Harmon
Head of API Design
Paypal
@jharmn
Jasonh-n-austin
Scaling API Design