Turning Software Projects Turning Software Projects into into

Production SolutionsProduction Solutions

Dan Fraser, PhDDan Fraser, PhD

Production CoordinatorProduction Coordinator

Open Science GridOpen Science Grid

OU Supercomputing SymposiumOU Supercomputing Symposium

October 2009October 2009

Four QuestionsFour Questions

-What gives some software -What gives some software projects “staying power” projects “staying power” while others stagnate ?while others stagnate ?

-What is the most important -What is the most important thing for a sustainable thing for a sustainable software project ?software project ?

-What is the difference -What is the difference between a software project between a software project and a software solution ?and a software solution ?

-What is needed to create a -What is needed to create a “production” quality “production” quality solution ?solution ?

My backgroundMy background LANL, Air Force Supercomputing, Thinking LANL, Air Force Supercomputing, Thinking

Machines, NEC, Sun, Paremus Ltd., Globus, Machines, NEC, Sun, Paremus Ltd., Globus, Open Science GridOpen Science Grid

Software development, software support, user Software development, software support, user and application support, software scalability, and application support, software scalability, production system management…production system management…

I have never fully appreciated the answers to I have never fully appreciated the answers to these questions until I worked on Productionthese questions until I worked on Production

And I’m still learning …

Software withSoftware with “Staying Power” “Staying Power”

Microsoft WindowsMicrosoft WindowsHow many of you How many of you run linux as the run linux as the primary os on your primary os on your laptop ?laptop ?

Mac does not countMac does not count

It solves the same It solves the same “problem” as windows“problem” as windows

GridFTP, GRAM2, GridFTP, GRAM2, MyProxy, Nagios, MyProxy, Nagios, Ganglia, Rocks, Ganglia, Rocks, CFEngine, SSH …CFEngine, SSH …

Question 1Question 1What gives some software projects “staying What gives some software projects “staying power” while others stagnate ?power” while others stagnate ?

Question 1Question 1What gives some software projects “staying What gives some software projects “staying power” while others stagnate ?power” while others stagnate ?

The answer is NOT:The answer is NOT: FundingFunding

Microsoft did not get big because it had lots of cashMicrosoft did not get big because it had lots of cash

MyProxy has not been funded for over 5 years.MyProxy has not been funded for over 5 years. But the team still fixes bugs on weekends…But the team still fixes bugs on weekends…

Some answers include:Some answers include: It solves a problem that users need solvedIt solves a problem that users need solved It “works” in the user environmentIt “works” in the user environment The community supports itThe community supports it

Solutions that “work”Solutions that “work”

When software When software solutions “work”, solutions “work”, even though they even though they are not perfect, it is are not perfect, it is hard to get users to hard to get users to migrate away from migrate away from themthem

Anyone remember Anyone remember plot10 ?plot10 ?

Question 2Question 2

What is the most important thing for a What is the most important thing for a sustainable software project ?sustainable software project ?

Question 2Question 2What is the most important thing for a sustainable software project ?What is the most important thing for a sustainable software project ?

Answer: Users.Answer: Users.

Production Ξ UsersProduction Ξ Users

The “users” are not always who we thinkThe “users” are not always who we think

GridFTPGridFTPA well engineered production codeA well engineered production code

Used in all the High Energy Physics Used in all the High Energy Physics experiments + many other placesexperiments + many other places

Easy to install (make install gridftp)Easy to install (make install gridftp)

We did not always understand end user We did not always understand end user difficulties.difficulties. ““Works great one day, try again one week later and Works great one day, try again one week later and

it fails” it fails” Problem with CRLs (security issues)Problem with CRLs (security issues) GSI is the software “environment”GSI is the software “environment”

GridFTP is most successfulGridFTP is most successful

In places where the security environment is well In places where the security environment is well maintainedmaintained

Where end users don’t necessarily see GridFTP Where end users don’t necessarily see GridFTP directly.directly. TeraGrid developed TGCP, a friendly user interface to TeraGrid developed TGCP, a friendly user interface to

GridFTPGridFTP UberFTP and other interfaces widely usedUberFTP and other interfaces widely used GridFTP is an GridFTP is an embedded embedded component at all the HEP sitecomponent at all the HEP site OSG packages GridFTP as part of a complete solution OSG packages GridFTP as part of a complete solution

(a Storage Element) including security(a Storage Element) including security

The main “user” is often the “integrator”

Integration is CriticalIntegration is Critical

Software integration Software integration is not a dirty word, is not a dirty word, but it can be dirty but it can be dirty work.work.

Integration is a Integration is a critical component critical component in production in production solution solution developmentdevelopment

Find an Integrator to work with !

Question 3Question 3

What is the difference between a software project What is the difference between a software project and a software solution?and a software solution?

A solution is something greater than the sum of its A solution is something greater than the sum of its software componentssoftware components

Examples:Examples:GridFTP is a software project GridFTP is a software project

UI + GridFTP + Authz framework + Credential mgmt = a solutionUI + GridFTP + Authz framework + Credential mgmt = a solution

GRAM2 is a software projectGRAM2 is a software project

Condor-G + Gram2 + GridMon = a solutionCondor-G + Gram2 + GridMon = a solution

Ganglia is a software project that does monitoringGanglia is a software project that does monitoring

Ganglia + something that acts on the data = a solutionGanglia + something that acts on the data = a solution

Solutions create new problemsSolutions create new problems

Multiple components must work togetherMultiple components must work together Which components do I need?Which components do I need? Which versions work together?Which versions work together? I need a new feature. Can I upgrade one I need a new feature. Can I upgrade one

component, or will it break everything ?component, or will it break everything ? When a solution breaks, which component is When a solution breaks, which component is

to blame ?to blame ? How do I get resolution when component How do I get resolution when component

providers are blaming each other ? providers are blaming each other ?

Question 4Question 4

What is needed to What is needed to create a “production” create a “production” quality solution ?quality solution ?

Answer: A good process Answer: A good process Putting components together Putting components together that work and solve a problem that work and solve a problem for a communityfor a community

Processes can be Processes can be reliable even with weak reliable even with weak components !components !

What is needed for a Production process?What is needed for a Production process?

-Coordination with -Coordination with software providers and software providers and communitescommunites

-Integration-Integration

-Testing-Testing

-Monitoring-Monitoring

-Solution Releases -Solution Releases

-Operations-Operations

-…-…

OSG – A good process exampleOSG – A good process example

-Projects in the OSG -Projects in the OSG software stack have software stack have good “staying power”good “staying power”

-Projects get into OSG -Projects get into OSG by being demanded by by being demanded by the user community.the user community.

-OSG provides the -OSG provides the “production” process“production” process

OSG Enabled Production SolutionsOSG Enabled Production Solutions

HEP AnalysisHEP Analysis: HEP, CMS, D0, CDF, …: HEP, CMS, D0, CDF, …

Gravitational Wave: Gravitational Wave: #2 in the world#2 in the world

Protein structure prediction: Toyota InstituteProtein structure prediction: Toyota Institute

Weather Research Forecasting: Weather Research Forecasting: UNC, U NebraskaUNC, U Nebraska

Structural Biology Predictions: Structural Biology Predictions: Harvard MedicalHarvard Medical

NanotechnologyNanotechnology Simulation and Modelling: Purdue, Simulation and Modelling: Purdue, Nanohub collaboration.Nanohub collaboration.

Molecular Dynamics: Molecular Dynamics: U of Buffalo, ArgentinaU of Buffalo, Argentina

Theoretical Nuclear PhysicsTheoretical Nuclear Physics: Duke University..: Duke University..

Text Mining: Text Mining: U. North CarolinaU. North Carolina

Exploring Mathematical algorithmsExploring Mathematical algorithms: U. Colorado: U. Colorado

On creating production solutionsOn creating production solutionsUnderstand what the problems areUnderstand what the problems are Understand what the users really want to doUnderstand what the users really want to do Make sure there are enough likely usersMake sure there are enough likely users Understand the barriers to adoptionUnderstand the barriers to adoption

Understand the Understand the environment environment where the where the solution will livesolution will live

Work with integrators (or be one)Work with integrators (or be one)

Know and nurture your user communityKnow and nurture your user community