Building and Managing an Effective K–12 Data Analytics Team

Adam Warner, Texas Education Service Center Region 10 (Dallas) Sharon Reddehase, Double Line Partners Ed Comer, Double Line Partners

NCES MIS Conference - February 15, 2012 - San Diego

Today we’re going to talk about why software development is hard…and some proven methods that can help

Stock photo. Release for web use of this photo on file.

Two Cases Studies with Similar Missions

Provide visualizations and analytics that resonate with educators

Extract, standardize, integrate, and unify data from the many districts sources in a timely manner

Dashboards and AnalyticsUnify K-12 Data from Sources

Ed-Fi•Texas Student Data SystemTSDS

•Automated Dropout Early Warning SystemReveal

All student information and photographs are fictitious.

Texas Student Data System (TSDS)

Standardized Test Results

(SAT/ACT/TAKS)

Student Information System (SIS)

District Connections

Database (DCD)

Other District Source Data

Campus & DistrictSource Systems &

Raw Data

InterchangeSchema

DistrictConnections

Database

UserInterface

This process must be completed for each district/campus

• Common infrastructure designed for scalability• Incremental development over a series of prototype and

limited production releases

Ed-Fi XML Interchanges

Ed-Fi Database Schema

All student information and photographs are fictitious.

Reveal Automated Dropout Early Warning System

Student Information System (SIS)

Analytics Database &

Engine

Campus & DistrictSource Systems

InterchangeSchema

Data Warehouse Analytics

Ed-Fi XML Interchanges

• Leverages same Ed-Fi data standard for scalability now and as a complement to the TSDS down the road• Fully automated, hosted solution that applies dropout research on

most common risk factors (poor attendance, behind cohort, behind credits, failing grades)

All student information and photographs are fictitious.

The Common Formula

Standards-based

architecture

Agile process

Multi-

function

teams

Software development, especially data analytics, is like constructing an office building….

…where you have a framework but really don’t know exactly what kinds of offices it will contain – their size, shape, function, paint scheme…

Stock photo. Release for web use of this photo on file.

The Problem(s) with Software Development

• Customers do not know exactly what they need or want

• Customers change their minds• Customers’ priorities change• It is difficult to solve the needs of many

stakeholders• We can never adequately specify requirements• We are lousy at estimating software tasks• Unpredictable “stuff” happens• It often takes several times to get “something”

right

• Users do not know what they want until they see it (but are good at telling us what they do not like)

• It is difficult to determine the 80:20; that is, what features are most important - before the product is built and deployed

• It is difficult to account for the unknown

Stock photo. Release for web use of this photo on file.

What to do? “Agile” is born

Agile ManifestoWe are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:• Individuals and interactions over processes

and tools • Working software over comprehensive

documentation • Customer collaboration over contract

negotiation • Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Beck, Kent; et al. (2001). "Manifesto for Agile Software Development". Agile Alliance.

“Agile” is not a process – it’s a value system

Principles behind the Manifesto1. Our highest priority is to satisfy the customer through early and

continuous delivery of valuable software. 2. Welcome changing requirements, even late in development. Agile

processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Principles behind the Manifesto

7. Working software is the primary measure of progress. 8. Agile processes promote sustainable development. The sponsors,

developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10.Simplicity--the art of maximizing the amount of work not done--is essential.

11.The best architectures, requirements, and designs emerge from self-organizing teams.

12.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Source: http://agilemanifesto.org/principles.html

Agile Implementations

• Scrum• Extreme Programming• Iterative development•Kanban•And many others

Case Study: Building the team

• Business people and developers must work together daily throughout the project.

• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Relates to these principles:

It takes many roles to build a K-12 analytics system…

Education World Technology World

Researchers

BusinessAnalysts

Trainers

Support Analysts

Data Modelers

System Architects

Interface Designers Interface

Developers

Managers

QA Testers ETL Developers

It takes many roles to build a K-12 analytics system…

Education World Technology World

Researchers

BusinessAnalysts

Trainers

Support Analysts

Data Modelers

System Architects

Interface Designers Interface

Developers

Managers

QA Testers ETL Developers

We hired experienced private-sector technologists,

who showed adaptability to

different industries, to provide world-

class technical skills.

It takes many roles to build a K-12 analytics system…

Education World Technology World

Researchers

BusinessAnalysts

Trainers

Support Analysts

Data Modelers

System Architects

Interface Designers Interface

Developers

Managers

QA Testers ETL Developers

Teachers

Administrators

Data Specialists

We hired experienced, tech-savvy educators to

provide domain expertise as researchers,

analysts, trainers, testers, and

support specialists

Key Characteristics

• Have the “agile” mindset• Self-driven to quality and doing things right despite schedule

pressures• Able to work today’s tasks while considering the big picture• Not phased by change and adaptation• Can reliably execute a test-driven process• Proactive continuous improvement• Thrive in a collaborative environment

• Critical thinking skills to deal with deep data and business rule semantics• Experienced and expert in the underlying technologies• Deep experience in education data and applications

Reveal Team Building• Upon joining the project, an excellent multi-functional team

had already been assembled • But existing project management and team-building practices

were ineffective, and there were many divisive influences affecting the team

We addressed the divisive influences. However, there was still an “us vs. them” problem between the educators and technologists. We needed a way to manage the project that promoted teamwork.

Stock photo. Release for web use of this photo on file.

Agile Results on Reveal• We turned to Agile principles and Scrum to help solve the problem.• Cross-functional teams were formed. • The teams were given a goal and timeframe in regular intervals.• They were self-organizing - no pre-defined leader or roles.• We abandoned detailed specifications in favor of face-to-face

collaboration and communication. (Documentation was done after the fact, and reflected the software as-built.)

• We kept the focus on the end user, and educational outcomes.• They reflected on their progress and opportunities at regular

intervals.• Soon, the teams began to understand the challenges they each faced,

and how they could better collaborate to build great software quickly and effectively.

• A high-functioning team was born!

Case Study: Managing the team

• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Working software is the primary measure of progress.• Welcome changing requirements, even late in development. Agile

processes harness change for the customer's competitive advantage. • The most efficient and effective method of conveying information to

and within a development team is face-to-face conversation.

Relates to these principles:

Scrum is the most popular agile process

http://agilescrum.biz/six-sigma-and-agile-software-development/

Kanban (カンバン ) + Scrum =“Scrum-ban”

Kanban• Specialized functional

groups• Track process steps• Continuous process with

periodic release cycles• Able to expedite stories• Limits for work in process

to identify bottlenecks• Goal is to minimize story

completion time

Scrum

• Multi-faceted, self-organizing teams

• Backlog | In process | Done • Time-boxed sprints, each

potentially releasable• Stories frozen during a sprint• Unfinished stories placed on

backlog• Goal is to increase team

velocity

Kanban Board

Roadmap

Major Process Steps

On Deck

Stories in process

Stories that are done, done, done

New Bug or Fix

Original BreakdownStory of Epic

DLP photo. Release for web use of photo on file.

Scrum time at the Kanban Board

DLP photo. Release for web use of photo on file.

Collaborative development• High level of user input and collaboration• Users – extensive stakeholder engagements at the start, monthly follow-up focus

groups, demos, etc.• Customer/sponsor – weekly meetings plus participation in JAD sessions• Business analysts – the internal “voice of the users” with deep knowledge of

education data

• Experienced development team physically in the same room• Architecture and leads with experience with education data and applications • ETL developers, most with deep knowledge of education data• UI/application developers

• Support and deployment plugged in to the development team• Provided remotely by Education Service Center Region 10• Multiple touch points planned for each week• Common issue tracker across districts, Region 10, and development

Other techniques contributing to being “agile”

• Joint Application Development (JAD) sessions• Bring users and developers together on specialized issues where features

and implementations are closely intertwined• Test driven development• ETL test harness• Automated unit testing

• Continuous integration with automated build and testing• Architecture focus on loose coupling and extensibility• Code refactoring• Capture requirements and knowledge in shared online tools• Metrics database, wiki, code plus docs in subversion

• Final QA accomplished by the business analysts with deep knowledge of education data• “Done-done-done”

Agile results on TSDSYear 1: Iterate and perfect different features• Rapid iterations with users and

customers• UI look and feel• Metric definitions• Data anomalies• Key features and drilldowns

• Evolved and proved the process, the architecture, and factory for development

Year 2: Build out the features, productize and scale• Incrementally delivered a

significant dashboard application• Several refactoring passes• Incrementally on-boarded new

LPR districts• Integrated user support into the

development process

Consistently met all major milestones under budgetExceeded the expectations of our customerReceived high praises from users for usability

How can this be applied to an SEA implementation?• Get the “agile” mindset – will require a change in values• Manage to a prioritized roadmap, not to a specification

• “Wrap” an agile process in a more conventional project management structure

• Focus on building the right team• Agile works best with small teams of highly competent, highly

motivated individuals• If an internal team without agile experience, invest in some training• If selecting a vendor, make sure they have deep experience with

agile – and check their references• Staff for a highly collaborative process• Full-time attention will be required• Critical to involve end-users through the process

Wrap an agile process in a conventional project/product management process

• Requires continuous collaboration• Healthy “tension” between the project/product manager and the

development leads

More Info on Agile• en.wikipedia.org/wiki/Agile_software_development• www.infoq.com/agile - lots of articles, case studies, free e-

books, including Henrik Kniberg’s excellent e-books:• Scrum and XP from the Trenches (2007)• Kanban and Scrum – making the most of both (2009)

• Other books:• Agile Software Development with Scrum (2001), Ken Schwaber

and Mike Beedle• Scrum in Action (2011), Andrew Pham

These slides available for download at http://www.schoolviz.org/reveal

adam.warner@region10.orged@doublelinepartners.com

sharon@doublelinepartners.com