yahoo communities architecture unlikely bedfellows
TRANSCRIPT
![Page 1: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/1.jpg)
1
Yahoo! Communities
Architectures
Ian Flint
November 9, 2007
![Page 2: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/2.jpg)
2
Agenda
• What makes Yahoo! Yahoo!?
• Hardware Infrastructure
• Software Infrastructure
• Operational Infrastructure
• Process
• Examples
![Page 3: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/3.jpg)
3
What makes Yahoo! Yahoo!?
• What do these sites have in common?
– Del.icio.us
– Flickr
– Yahoo! Groups
– Yahoo! Mail
– Bix
![Page 4: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/4.jpg)
4
What makes Yahoo! Yahoo!?
• Accountability at the property level
– Architecture
– Application Operations
– Infrastructure Decisions
• Incubator Environment
– Properties function independently on a common hardware platform
– Highly cost-conscious
– Open-source attitude
![Page 5: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/5.jpg)
5
What makes Yahoo! Yahoo!?
• Standards at the infrastructure level
– Hardware/Software platform
– Configuration Management
– Operational tools and best practices
• Executive Involvement
– Cost
– Robustness
– Redundancy
![Page 6: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/6.jpg)
6
Hardware Infrastructure
Common Platform
![Page 7: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/7.jpg)
7
Hardware Infrastructure
• Shared Components
– Network, Data Center, NAS
– Centrally managed by infrastructure team
• Load Balancing
– DSR is preferred model
– Proxy load balancing only where necessary
![Page 8: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/8.jpg)
8
Hardware Infrastructure
• Hardware (x86, RAID/SCSI)
– Jointly managed by properties and ops
• Hardware Selection
– Price/Performance is a constant consideration
– Supply chain and provisioning cost
– Reliability vs. Price
• Single-Homed hosts (even databases)
• Pooling across multiple switches
• Fast Failover to mitigate risk of switch failure
![Page 9: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/9.jpg)
9
Hardware Infrastructure Example
• Layered Infrastructure
• Hosts distributed
across multiple racks
for power/network
redundancy at the
pool level
• Really Big Load
Balancers doing DSR
The Internet
Rack
Switches
Aggregation
Switch
Hosts
Router
Load
Balancer
![Page 10: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/10.jpg)
10
Software Infrastructure
Shared Repository
![Page 11: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/11.jpg)
11
Software Infrastructure
• OS (FreeBSD, moving to RHEL)
• Databases (MySQL, Oracle)
• Development Platforms
– PHP (most properties)
– C/C++ (primary infrastructure platform)
– Java
– Python
![Page 12: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/12.jpg)
12
Software Infrastructure
• Installable components
– Managed through yinst package manager
– Stored on common distribution server
– Examples: yapache, yts, yfor, ymon, yiv, vespa
![Page 13: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/13.jpg)
13
Software Infrastructure
• More about yinst
– Robust Package Manager
• Installation, Versioning, Scripting
– Implementation
• Software installed on distribution cluster (package repository)
• Hosts then pull software (via proxies)
• Software stored under a common root
• Used for everything from perl modules to common components to applications
![Page 14: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/14.jpg)
14
Software Infrastructure
• Shared Infrastructure enables rapid integration of acquisitions
– UDB
– SSO
• External Infrastructure
– Akamai CDN and DNS
– Gomez & Keynote
– SDS
– YMDB
![Page 15: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/15.jpg)
15
Software Infrastructure - Bix
• Global Server Load Balancing between sites
• YTS provides Reverse Proxy and Connection Management
• Yfor provides fast failover from colo to colo
• Media is served via a content delivery network for performance and to reduce load on servers
YTS
(Reverse Proxy)
Akamai
(CDN)
The Internet
Web ServersWeb ServersWeb Servers
Media ServersMedia ServersMedia Servers
YTS
(Reverse Proxy)
Web ServersWeb ServersWeb Servers
Akamai
(CDN)
Media ServersMedia ServersMedia Servers
Yfor
(failover resolver)
Yfor
(failover resolver)
Prim
ary
Colo
Backup C
olo
![Page 16: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/16.jpg)
16
Software Infrastructure - Bix
• Yfor Failover Resolver used for fast failover of database connections
• Dual Master MySQL setup for write hosts
• Media storage on NetApp NAS device, with snap-mirroring to backup data center
![Page 17: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/17.jpg)
17
Software Infrastructure - Bix
Web Server
Yapache (Yahoo-improved Apache)
Mod_proxy
(Reverse Proxy)
Static
Files
Tomcat
Bix Application
Spring Hibernate
MySQL Connector
Lucene
Hessian
ehcache
log4j
c3po
PHP
Yahoo
Shared
Services
• Yapache reverse proxy in front of Tomcat instance
• PHP used to access Yahoo shared services
• Static files served from disk
• Fairly standard Java environment (Spring, Hibernate, ehcache, c3po, log4j, etc.)
![Page 18: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/18.jpg)
18
Software Infrastructure - Groups
• Inbound Groups mail hits a qmail cluster
• Mail filtered against real-time blacklist
• Mail forwarded to second qmail cluster
• Proprietary anti-spam algorithms applied
• Mail forwarded to group members
• Mail stored on archive servers
• Oracle RAC clusters store metadata
• Periodic “Electric Potato”measures QoS
![Page 19: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/19.jpg)
19
Software Infrastructure – Groups
• Dynamic content served via web pool running python/c++ application
• CSS and images served via a squid-fronted pool
• Group photos on Y! photos infrastructure backed by Yahoo! Media DB (YMDB)
• Database feature implemented as sleepycat DB hosted on message store
• Calendar feature implemented via API calls to calendar.yahoo.com
![Page 20: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/20.jpg)
20
Operational Infrastructure
Managing the Platform
![Page 21: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/21.jpg)
21
Operational Infrastructure
• Common Monitoring Infrastructure
– Nagios
• Main monitor for clusters
• Numerous standard plugins
• Standards/Best Practices around custom plugins
– Ywatch
• Basic monitoring of machines over SNMP
• Heartbeats plus fundamental metrics (IO, CPU, Disk, etc.)
– Ymon
• NRPE/NSCA on steroids
• Automated forwarding of active and passive checks
• Scripted setup
– Drraw
• Data Visualization
• Deep integration with Nagios and ymon
![Page 22: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/22.jpg)
22
Operational Infrastructure
– Rollup Monitoring
• Clusters rolled up to centralized monitoring console
• Prioritization and correlation of events
– Internal Site QOS Monitoring
• QOS monitoring for sites
• Response time and availability
– “The OC”
• 24x7, worldwide operations center
• Provides tier 1 and 2 support
– Centralized CMDB
• Configuration Management DB – manages every device
• Contact info, escalations, and runbooks included
![Page 23: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/23.jpg)
23
Operational Infrastructure Example
• Application Servers perform checks which are registered by Nagios as passive checks
• Metrics are aggregated by metrics module
• On-demand graphing is provided by drraw
• Nagios alerts are forwarded to central ywatch console
Web ServersWeb ServersWeb ServersMedia
Servers
Media
Servers
Media
ServersWeb
Servers
Web
ServersDB Servers
ymon server
ymond
metrics
aggregator
nagios
grapher (rrdtool/drraw)
forwarding agent
ywatch server
listener
Ops Console
yapache
yapache
SN
MP
-Based C
hecks
![Page 24: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/24.jpg)
24
Processes and Standards
Keeping it sane
![Page 25: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/25.jpg)
25
Process and Standards
• Hardware Review Committee
– Strong emphasis on economics
– Personal attention from David Filo
• Software Review Committee
– Thinking through major licensing decisions
• Business Continuity Planning
– Required of all properties
– Must have and test backup data center
• Paranoids
– Ongoing site scans
– Enforcement of standards
![Page 26: Yahoo Communities Architecture Unlikely Bedfellows](https://reader033.vdocument.in/reader033/viewer/2022052505/5555d26dd8b42aaf158b4db8/html5/thumbnails/26.jpg)
26
Questions?