questions expanding the virtual world dave poole ccp games, web technical director
TRANSCRIPT
QUESTIONS
Expanding the virtual world
Dave PooleCCP Games, Web Technical Director
• Intro to EVE Online• EVE Gate
• What is it?• Why it is important• How it came to be• Lessons learned
• EVE Gate – the Technology Behind it• The software architecture• The hardware powering it• Performance
• CCP and IBM• Schooner Caching Solutions
• Q & A
AGENDA
A vast persistent virtual universe
Over 330,000 players inhabit a single game world competing for military, economic & political power
Players in over 230 countries
Steady growth since 2003
Hardcore, dark and ruthless
Playground, sandbox and emergence
Introduction to EVE Online
Key design tenet:
“MMOGs are social networks
• The key factor that makes MMOs succeed or fail is the strength of their social network.
• People will quit games, but are less likely to quit friends.
• Maximize human interaction!“
Reynir Hardarson FOUNDER and CREATIVE DIRECTOR of CCP
EVE Online: Design
EVE Gate: Overview
What is it?
EVE Gate: Overview
INITIAL RELEASE
Website with:
• Real time access to EVE mail system• A commenting system similar to
Facebook• A calendar system in game and on
website fully synched• A reworked contact system combined
with a personal settings system to give player control of privacy and access to their info
• INSERT SCREENSHOTS
FUTURE FORM
We have a full time development team continually adding to it
Proposed features:
• Expanded tagging system• Web access to the skill queue• New forums possibly including Corporation level forums (Groups)• Real time access to the market systems, read only to start• Fitting windows, saving / trading popular ship loadouts• Killboards• Public profiles and info to push to external apps like Facebook (player controlled)• Sovereignty results, controls, etc• Corporation Management• Basically anything that does not involved flying, shooting, etc in game
EVE Gate: Overview
– EVE Gate will be utilized for the whole EVE universe– It will be used to connect EVE players and DUST players– Facilitates corporation to corporation communication and
coordination– DUST specific features will be added, more applicable to its product
type (game results, statistics, heatmaps, screenshots,etc)– Important bridge between 2 different games, in 2 different genres
within the same persistent universe
EVE Gate: Overview
BEYOND THE WEB
• The importance of APIs• Mobile possibilities• SOA: Flexibility for the future
EVE Gate: Overview
Important for CCP
Purpose and goals: Breaking down barriers and exposing the EVE universe for all to experience and interact with
• A gateway for interaction from anywhere, at anytime with anyone• A tool to operate and organize player entities• A pathway to syndicate information, propaganda and events• A mechanic for current customer retention• A vehicle for new customer acquisition• A connection between the EVE and Dust communities• A platform to respond swiftly and capitalize on opportunities and disruptions
This is a critical service provided to the players, for a business driven by a service model
EVE Gate: Why is it important
Important for the industry
• More and more developers are expanding beyond the game client• Turbine with My LOTRO• Blizzard with Battle.NET• Bungie with Bungie.Net
• This is a social network designed specifically for the specific customer base, not forced upon them from an external solution like Facebook
• Nature of EVE demands extreme privacy and information controls• Nature of alts (spies, etc) drives the need for this to be character focused, not
player focused – thus linking to external individual focused solutions is detrimental• Not intended as an advertising / revenue source but as a service
EVE Gate: Why is it important
Observations and studies of player behavior and trends
• The importance of social interaction• Product stickiness and what drives it• The importance of player driven events and the
syndication of information• Full time Research & Statistics analyzing needs, plus
CSM input
Project requirements fleshed out• Developed over 2+ years by the EVE producers• Need to find the right time to implement
• Player needs, market conditions, talent available, plus other company initiatives
• The need for a project ‘champion’
EVE Gate: How it came to be
Step 1 (done): what do we want to build?
Step 2: how do we build it?
• Architectual designo build on proven high scalability designs from other industrieso work with company strengths (python, .Net, SQL Server)
• Staffing and resource allocation
• Hardware Infrastructure Designo the importance of working closely with Operationso use of phased stress testing and product evals to nail down requirementso Scheduling development and testing to coincide with and support infrastructure needs
EVE Gate: How it came to be
Step 3: development
2 sprint teams working for a full yearTeams consisted of:
• Web developers• EVE Software developers• Graphic designers• DBAs• Dedicated QA
Included talent from Atlanta and Reykjavik so we faced challenges with doing SCRUM development across the pond
EVE Gate: How it came to be
Game design• Don’t force a solution upon your player base, look for what services fit their need and give them control
o Example: privacy settings and access controls, external integration?• The importance of information security
Project Management• The importance of a champion to drive the project• The importance of operations involvement from the start
Development Process• Learned how to do SCRUM over distance
o Less effective but sometimes necessaryo Mitigate through extra communication effort (vid conferencing) and as much face-to-face time as possible
through travelo Team building HUGELY important
• The importance of scripted and public stress testing (HUGE)• The importance of close liaison with operations and your vendor partners
EVE Gate: Lessons learned
EVE Gate: The Technical Solution
Umbraco – Content management system
View
Controllers
Data Model
EVEGate.web
EVEGate.dataobjects
Web DBCachingCluster
Memcached
ForumApplication
BloggingApplication
Other .NetApplications
sessions
Web Database Cluster(SQL Server 2008)
Nexus
EVE Gate Web Application(s)
SOL Processes SOL Cluster
EVE Database Cluster(SQL Server 2008)
TQ DB
Tranquility
EVE Gate Web Cluster (Windows Server 2008)
EVE Gate Service Cluster (Windows Server 2008)
EVE Gate WCF Application
FUTURE RELEASES
EVE Gate Launch Architecture
Denotes new components
EVE Gate MVC Application
EVE Gate: The Technical Solution
The software architecture
Key points Design needs to minimize impact on game systems Cannot have in game lag caused by out of game systems Design needs to have scalability and reliability built into each layer Needs to be easy to maintain and upgrade Needs to consider security without impacting performance
Driven by a Service Oriented Architecture design approach Proven approach used by high volume web applications and web services View layer built with modern web techniques (.Net MVC C#) Connect to data providers including existing systems with a communication layer (WCF) Added a service agent to the game cluster (SOL nodes) to coordinate data flow Backend databases and game services retained (Python, SQL Server) with some enhancements for new
services Each layer can be independently scaled, refactored or replaced Each layer separately secured and monitored
EVE Gate: The Technical Solution
Front End• Latest .Net (C#) framework MVC
• Strong MS support, high industry uptake• Designed after well proven web MVC frameworks like STRUTS• Lots of tools, techniques, components available• Works well with AJAX based techniques, we use heavily
• Use our built-up expertise in web development where it makes most sense
Communications Tier• .Net Windows Communication Foundation (WCF) application• Coordinates all data access requests• Encapsulates all cache management, hides implementation • Re-usable service layer can be used for other external apps (APIs, Mobile, etc)• Can later be refactored into a common data layer for ALL CCP applications
EVE Gate: The Technical Solution
Caching Tier- Caching is critical to minimize load on backend services- Greatly improves web application performance- Replaces stock .Net caching implementation - Use Schooner Caching Appliances which implements memcached
- Able to use standard Memcached APIs minimizing code effort- Transparent implementation, in vendor trials we were able to swap in
hardware without changing a SINGLE line of code- Memcached – open source, high use, developed for LiveJournal,
heavily used by Facebook and other high scale sites
The Hardware Infrastructure
Introducing James Wyld, CCP Virtual Worlds Administrator
EVE Gate: The Technical Solution
• Breaking down the software
EVE Gate: Under the hood
Umbraco – Content management system
View
Controllers
Data Model
EVEGate.web
EVEGate.dataobjects
Web DBCachingCluster
Memcached
ForumApplication
BloggingApplication
Other .NetApplications
sessions
Web Database Cluster(SQL Server 2008)
Nexus
EVE Gate Web Application(s)
SOL Processes SOL Cluster
EVE Database Cluster(SQL Server 2008)
TQ DB
Tranquility
EVE Gate Web Cluster (Windows Server 2008)
EVE Gate Service Cluster (Windows Server 2008)
EVE Gate WCF Application
FUTURE RELEASES
EVE Gate Launch Architecture
Denotes new components
EVE Gate MVC ApplicationEVEGate Web Servers
EVEGate DB
Service Layer (WCF)Memory Cache Cluster
Presentation Layer (Frontend)
EVE Game Server
EVE DB
EVE Proxy
EVE Sol
• Alpha Test– Getting the Alpha test up and running
was paramount.– The focus was on the architecture
elements and not features.– Vertical slice.
• Needed solid data before building the final system.
EVE Gate: Performance testing
• Key performance metrics– CPU per user -> users per core– Network bandwidth per user
• Bottlenecks– Service Layer CPU– Caching Network Bandwidth– Firewall placement
EVE Gate: Under the hood
EVEGate Web Servers
EVEGate DB
Service Layer (WCF)Memory Cache Cluster
Presentation Layer (Frontend)
• Infrastructure plan was changed accordingly
– Additional CPU power required across the board.
– Physically separated network for cache layer.
– Prompted a look at the core network switching capabilities.
EVE Gate: Under the hood
EVEGate Web Servers
EVEGate DB
Service Layer (WCF)Memory Cache Cluster
Presentation Layer (Frontend)
• Frontend / Backend server farm– Built on (in?) IBM Bladecenter, the same platform as
EVE Online.– Blades allow for a large amount of CPU power in a
small amount of rack space.– For CCP, IBM is simply a solid, reliable partner.
• Caching layer– Original plan was to build out the cluster manually– “Black Box” approach means we do not have to spend
time managing a linux/memcached farm.– More focus on running EVEGate and EVE Online!
• Database– MS SQL– Most data resides in the existing game DB
EVE Gate: The final system
EVE Gate: Putting it all together
• Fine-tuning the system– We created a scripted, scalable, repetitive load test.– The focus was on overloading each area of the infrastructure separately.– We found many new and interesting ways to break the application.– From an operational standpoint, we learned as much from this phase as the rest of the dev
cycle.– It took quite some time to stabilize it.