how edmodo uses splunk for real-time tag-based reporting of aws billing and usage

How Edmodo Uses Splunk App for AWS

Jack MurgiaNetOps Architectjack@edmodo.com

28,000,000+TOTAL USERS AROUND THE WORLD

Edmodo is the largest social learning network

in the world

20M+ Registered Users

The Edmodo user base has grown REALLY FAST and so has our AWS usage.

Dec 09 Dec 12Dec 10 Dec 11

2,000,000

6,000,000

10,000,000

18,000,000

14,000,000

20M+ users

4.8M users

1.1M users225K users

Investors

• New Features

▸ Require new backend clusters

• More Employees

▸ Launch more development servers

• New Investors

▸ Want assurances we spend money wisely

100’s of production, development, reporting and

operations instances are managed on Amazon Web Services Elastic Compute

Cloud.

20 employees across 5 teams can launch EC2 instances.

We tried lots of slow, pricey, hosted apps.None had what we REALLY Needed …

Real Time RI vs On-Demand UsageDo we have enough RI coverage? Are our RIs being used?

Getting Usage Data In front of TeamsMaking Real Time Usage Data Visible Encourages Action

Problem:

The default Splunk for AWS App configuration is based on billing “subaccounts”, which Edmodo does not use.

Edmodo uses tags to differentiate between cost centers.

Use These Tags: Product, Service, SpaceConfigurable Metadata Available for All AWS Resources

• Product

▸ designation is derived from the end user perspective. Where component resources serve overlapping Products, the Product designation is that which uses/requires a resource most or the resource's reason for existing.

• Service is the primary function of the resource. Examples are

▸ mysql

▸ apache

▸ active-directory

• Space a logical workplace designation. Examples are:

▸ Development

▸ Production

Edit the Splunk For AWS Scripts for Tagsget_instances.py and get_ops.py

Create Lookup Tables For The TagsSimple .csv files are referenced by dashboard view files

Personalize Your Dashboard ViewsThe Dashboard XML format is easy to customize

Drill Down Into Usage Data by TagsTeams Can Now Evaluate Usage For The Services They Manage

Drill Down Into Usage Data by TagsTeams Can Now Evaluate Usage For The Services They Manage

Drill Down Into Usage Data by TagsTeams Can Now Evaluate Usage For The Services They Manage

Download Tag Customizations

http://public.edim.com.s3.amazonaws.com/splunk-aws-app-for-tags.tgz