developing highly instrumented applications with minimal effort

Tim HobsonPrincipal Engineer, Chief Caffeine Officer - Intuit

Developing highly instrumented applications with minimal effort

2

About Me

Work

Play

3

Agenda

Theory• Patterns• Supporting

Frameworks• Supporting

Components

Practice• .Net Example • Java Example• Node.js Example

Concepts• Quality Data• Application

Instrumentation

• Application Logging

GOAL: You should be able to apply this to your application today

Concepts

5

If You Do Nothing…

OR

6

There’s No Free Lunch

Garbage In, Garbage OutWe can

parse

anything!

Insight!

7

Quality In, Insight Out

Most apps start with only the framework or app server logging (or nothing!)

None of the above is interesting to the business or the developer.

You can’t get if you don’t give – there are many ways to give, and many classes of data to provide.

8

Classes of System Output

App Instrumentation

• Cross-cutting (free)• App Activity• Passively triggers

alerts• Source of

performance data• Source of usage data

App Logging• Intentional• Business

Transactions• Overtly triggers

alerts• Source of business

metrics• Aids in

troubleshooting failures, bugs

System Instrumentation• JMX/WMI/SNMP monitoring• Apache/IIS/nginx access logs

Theory and Best Practices

10

Best Practices (© splunk>)

Create human readable events

Clearly timestamp events

Use key-value pairs

Be aware of multi-value fields

Log unique identifiers

11

Best Practices (© Tim)

Global timestamps (UTC – 2013-08-21 22:43:31,990)

Context setting (who/what/where/when/how)

Categories/taxonomy (what tier, what component)

Timing (time everything!)

Security (never log sensitive data: password=***)

Consistency in naming – (action=purchase; sale=oct13;

productId=123123)– (action=buy; promo=oct13; sku=123123)

12

Example Output

2013-08-21 22:55:36,504; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=Web.Controllers.BaseWebController; msg=RequestInfo; server=ws001prod; url=https://myapp.com/account/logon; method=GET; languages=en-US,en;q=0.8; referrer=https://myapp.com/members/dashboard; userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36; userId=123456; controller=Account; action=Logon;

2013-08-21 22:55:36,519; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=TraceInterceptor; timeTracing=Service.Users.GetUser, time=3;

Context

Metrics

13

Patterns

Pipeline– Filter– Interceptor

Dependency injection– Proxy

Pointcuts/aspects

14

HTTP Pipelines

Java (from Oracle Docs)

ASP.Net (from MSFT Docs)

express.js

var app = express();app.use(func1);app.use(func2);app.use(func3);app.use(func4);app.use(func5);app.use(func6);app.use(func7);app.use(func8);app.listen(8080);

15

Dependency Injection

• Enforces interface-based programming

• Forbids circular references• Lends itself to testability• Flexibility in changing

implementations• For our purposes: cross-cutting

capabilities

Intercepting Calls With Dynamic Proxies

16

Implementation

Caller

DI Container

Dependency A

Dependency B

Proxy

Inte

rcep

tor

Inte

rcep

tor

Inte

rcep

tor

Proxy

Inte

rcep

tor

Inte

rcep

tor

Inte

rcep

tor

Proxy

Inte

rcep

tor

Inte

rcep

tor

Inte

rcep

tor

Supporting Frameworks

17

The Inner Sanctum• Unity Dynamic

Proxies• Spring

@AspectJ Pointcuts

• JavaScript Mixins

Annotations & Attributes• @Instrumenta

ble• [Instrumentabl

e]

The Front Door• ASP.Net

Modules/ActionFilters

• Java Servlet Filters/Spring Interceptors

• Node.js interceptors

Supporting Components

18

Java• LogBack• SLF4J• Log4J

node.js• SenchaLab

s Connect• Winston

.Net• Log4Net• NLog

Intervention!

20

Sample App Intervention

3 Sample Apps

3 Interventions3 Platforms

+ + +

21

Goals

Minimize impact on existing code

Apply best practices

Leverage aspects for interception

Leverage an HTTP pipeline for context

22

MVC Music Store Intervention (Demo)

Missing dependency injection/interception framework: UnityMissing logging library: Log4Net, Buche

1. Add DI framework (Unity)2. Configure aspects (Interception)3. Configure controller factory, container locator 4. Set up logger (Log4Net)5. Configure log pattern and targets (Log4Net.xml)6. Wire up logging interceptor7. Wire up LogActionFilter and BaseLoggingController8. Deploy!

23

Spring Pet Clinic Intervention (Demo)

1. Set up loggera. Configure log pattern (to support context data)b. Configure targets (rotating file)

2. Create LogAspect.java and @Instrumentable attributea. Capture calling contextb. Start/stop timing

3. Create LogInterceptor.javaa. Capture request metadatab. Set MDCc. Start/stop timing

4. Configure aspect bean

5. Configure interceptor bean

6. Annotate the methods we care about

24

NodeCellar Intervention (Demo)

Missing Interception Framework: ScarletMissing Logging Library: Winston

1. Set up Scarlet 2. Configure log pattern3. Configure method interception4. Create logging interceptor5. Create LoggingFilter6. Deploy!

Closing Thoughts

Got Log? Now Get Intimate With Your App

26

Ops Dashboards Pro-Active Service

Degradation Alerting

Quality Assurance

PerformanceMetricsBusiness

Dashboards

SLA Tracking

Pre-ReleaseSanity TestingSecurit

y Alertin

g

Customer

Support

27

Key Takeaways

• It is YOUR responsibility as a developer to provide useful operational and business data.

• It is not hard, and most of it is for free once you have the patterns in place.

• The same patterns and practices can be applied to practically any platform, and any type of application or service.

• When you provide consistent and predictable data others can build on your greatness

28

Get the Code@hoserdude

Spring Pet Clinic Intervention:https://github.com/hoserdude/spring-petclinic-instrumented

MVC Music Store Intervention:https://github.com/hoserdude/mvcmusicstore-instrumented

NodeCellar Intervention:https://github.com/hoserdude/nodecellar-instrumented

Intuit Speakers @ Silicon Code Camp 2013:SATURDAY9:45 a.m. - Ramakrishna Kollipara – “Complete Automation of Performance Testing” 1:45 p.m. - Joe Wells - “QBO: Journey From legacy Java app to a Client-side HTML5 app”3:30 p.m. - Naga Addagadde & Sangeeta Narang – “Intuit APIs for Financial Transaction Aggregation”5:00 p.m. Ted Drake –“Hitting the Accessibility High Notes with ARIA”

SUNDAY9:15 a.m. - Eugene Krivopaltsev –“Building Native Mobile Apps with Custom Views” 1:15 p.m. - Tim Hobson – “Developing Highly Instrumental Applications with Minimal Effort”

You don't want to miss out on a chance to win this cool headset. Stop by our booth to enter!

For more information about joining our organization visit our booth or connect with our onsite recruiter:

Chriscox_recruiter@intuit.com

30

THANK YOU

Want to talk more?

I’ll be at the Intuit booth today 3-5 PM.