agile data governance tutorial
TRANSCRIPT
Agile Data Governance
Tami FlowersDirector, Governance SolutionsMetaGovernance Solutions, LLC
DGIQ June 2016
Who am I?
Tami Flowers Director, Governance Solutions at MetaGovernance PMI-PMP, PMI-ACP, and Certified Scrum Master Agile Coach
Developer Project Manager
Enterprise Architect
Data Governance Consultant
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Today
Data Governance Agile How to meld them together
A few minutes on the topic An “experience” Debrief/Reflection
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Ice Breaker1. Write your answer to the following questions on
sticky notes. One answer per note. As many answers as you wish. What are you hoping to learn today? What specific questions are you hoping get answered
today?2. Discuss amongst your team. Group like answers
together.3. Have each person mark their level of experience
with the following:
MetaGovernance Solutions LLC. All Rights Reserved. 2016
No experience
Some experience
Quite a bit of experience
AgileData Governance
Data Governance
Two viewpoints Establishing a Data Governance Organization
Framework Implementing Data Governance actionable
Data Governance
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Management Functions
Data Governance
Data Architecture
Data Development
Data Operations
Data Security
Reference and Master
DataEDW and BI
Document & Content
Metadata
Data Quality
Data Governance is at the heart of all Data Management activities. As defined by the Data Management Association – Data Management Body of Knowledge (DAMA-DMBOK)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance Framework
Data Governance OrganizationData Management FunctionsProjects
Agile Data Governance
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance Organization Framework
Executives
Business & Technical Members
Stewards/analysts/architects/ administrators/engineers
Consumers
Data Governance Council (DGC)
Data Governance Steering Committee
(DGSC)
Data Stewardship Teams (virtual)
• Establish & train DGC members• Agree on Data Governance roadmap• Determine members & empower DGSC• Define Data Governance policy• Meet on recurring basis
• Receive training and education• Execute DG roadmap• Define DG communication plan• Select & empower stewardship
teams• Define roles and responsibilities• Define charter• Meet on recurring basis
• Technical and Business Members• Receive training and education• Execute DG roadmap
• Business Members • Receive training and
educationEnd Users
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance Organization Framework Actionable Data Governance
Executives
Business & Technical Members
Stewards/analysts/architects/ administrators/engineers
Consumers
Data Governance Council (DGC)
Data Governance Steering Committee
(DGSC)
Data Stewardship Teams (virtual)
• Lead and promote program• Empower DGSC• Arbitrate data related issues, as needed
• Execute DG communication plan• Define data management policies,
procedures and standards• Define stakeholders, and sources and use of
data per subject area• Approve and oversee corrective actions
• Technical and Business Members• Implement policies, procedures,
standards and controls• Identify and report data issues• Corrective action input & execution• Corrective action status reporting
• Business Members that consume and use data
• Identify and report data issues
End Users
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Registered Data Governance Stakeholders
Data Subject Area ID
Major Subject Group 1st Sub Grouping 2nd Sub Grouping A
ccou
ntin
g O
pera
tions
Col
late
ral
Cre
dit R
isk
Fina
nce
Fina
ncia
l Rep
ortin
g
Inte
rnal
Aud
it
Mar
ket R
isk
Sale
s
Loan
Ope
ratio
ns
Com
plia
nce
Info
rmat
ion
Tech
nolo
gy
1 Trade Loan Adjustable-Rate Loan
2 Trade Loan Fixed-Rate Loan
3 Trade Loan Structured Loan
4 Trade Loan Amortizing Loan
5 Trade Loan Letters of Credit
6 Trade Loan Prepayment Fee
11 Financial Accounting Accrued Interest
12 Financial Accounting Outstanding Principle
LEGEND
CUSTODIAN
CONSUMERSTEWARDDELEGATE
MetaGovernance Solutions LLC. All Rights Reserved. 2016
MetaGovernance Solutions LLC. All Rights Reserved. 2016
The Agile Manifesto
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over Processes and toolsWorking software over Comprehensive documentationCustomer collaboration over Contract negotiationResponding to change over Following a plan
That is, while there is value in the items on the right, we value the items on the left more.
In February 2001, 17 software developers met at the Snowbird resort in Utah to discuss lightweight development methods. They published the Manifesto for Agile Software Development.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Agile Principles1. Our highest priority is to satisfy the customer
through early and continuous deliveryof valuable software.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Business people and developers must work together daily throughout the project.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Agile Principles6. The most efficient and effective method of
conveying information to and within a development team is face-to-face conversation.
7. Working software is the primary measure of progress.8. Agile processes promote sustainable development.
The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10.Simplicity--the art of maximizing the amount of work not done--is essential.
11.The best architectures, requirements, and designs emerge from self-organizing teams.
12.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Agile Principles
How can we apply these principles to establishing a data governance framework
How can we apply these principles to actionable data governance (day-to-day governance)?
MetaGovernance Solutions LLC. All Rights Reserved. 2016
The many flavors
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Scrum
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Lean Software Development
Eliminate Waste
Empower the Team
Deliver Fast
Optimize the
Whole
Build Quality
In
Defer Decision
s
Amplify Learning
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Agile Practices to Lean Principles
Agile Practice/Lean Principle
Eliminate Waste
Empower the Team
Deliver Fast
Optimize the Whole
Build Quality in
Defer Decisions
Amplify Learning
Teams make own decisionsJust in time planning
Team retrospectives
Two week iterations
Unit test as we go
Shadow the business to learn what they doMetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Data Governance to Lean Principles
Agile Practice/Lean Principle
Eliminate Waste
Empower the Team
Deliver Fast
Optimize the Whole
Build Quality in
Defer Decisions
Amplify Learning
Teams make decisions
Just in time planning
Retrospectives
Frequent iterations
Validate/approve as we goInvolve the business stakeholders MetaGovernance Solutions LLC. All Rights Reserved. 2016
Kanban Capture the work Make it visible Each story is about the same size Limit the work in progress (WIP)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: WIP
8-10 people per team 4 workers – will do the work 4 managers – will watch over their worker 1-2 timers – will time how long each round
takes
This is called the Penny Game and is available on www.tastycupcakes.org
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Commonalities of all Agile/Lean flavors
Capture the work Make it visible Minimally Viable Product
(MVP) Engage the owner/customer/
stakeholders Have owner prioritize the
work Work on the highest priority
work first Iterate Continually retrospect and
improve
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Roles Stakeholders
Product owner is one All have a stake in the project DGSC members
Product Owner Chairperson of DGSC Has the vision and can convey it to the team and stakeholders Prioritizes backlog Provides direction to and is engaged with the team
Scrum Master – facilitator Bulldozer: Remove roadblocks Servant leader: Do anything needed to help team perform at their
highest level Stewardship Team Roles
Stewards Developers/Engineers/Architects/Administrators Business Analysts/Quality Assurance Analysts
MetaGovernance Solutions LLC. All Rights Reserved. 2016
CeremoniesCeremonies Traditional
Agile/Lean Project
Data Governance Framework
Actionable Data Governance
Planning Meetings
Start of every sprint/iteration
DGSC or subgroup meetings
Start of every sprint/iteration
Scrums/Stand Ups
Daily, 15 minutes
1-3 times a week
Daily, 15 minutes
Review meetings
End of every scrum/iteration
DGSC or subgroup meetings
End of every sprint/iteration
Retrospectives
End of every sprint/iteration
End of every sprint/iteration
End of every sprint/ iteration
Backlog Grooming
In the week between sprints/iterations
In between sprints/ iterations
In the week between sprints/iterationsMetaGovernance Solutions LLC. All Rights Reserved. 2016
4 quadrants to help with backlog grooming
Broccoli Organic Beef
Twinkies Éclair
Effort
Valu
e
One of our classmates shared this quadrant with us during a discussion. They use this in their organization to help discuss the value of a story in the backlog and it helps them select stories for the sprint/iteration.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
ArtifactsArtifacts Traditional
Agile/Lean Data Governance Framework
Actionable Data Governance
Product Backlog
Created from conversations with stakeholders
Created from Data Governance Roadmap
Created from DG policies and procedures
Sprint/Iteration Backlog
Prioritized by product owner during planning meeting
Prioritized by DGSC chairperson during planning meeting
Prioritized by product owner during planning meeting
Stories Created from conversations with stakeholders
Created from Data Governance Roadmap
Created from DG policies and procedures
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance artifacts within a project Stories
As a <role> I want <what> so that <business value> If not a story, include data governance as
acceptance criteria in stories As an accountant, I want current balances for loans, so
that I can complete daily reporting All data elements are defined in the business dictionary Data profiling of the balances occurs when loading into
the EDW An automated control is added so the current balance is
automatically reconciled between the G/L and Loan System
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Reality Check
Agile isn’t a silver bullet that will magically fix everything New, unexpected challenges will occur
Teamwork Soft skills Growing pains
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Reality Check
Data Governance isn’t a silver bullet that will be easy to quickly implement Must include the business Must be added to existing processes to be
successful Lots of training and education is needed Executive support is critical Will uncover data issues that some don’t want
uncovered
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Adding Governance to a Project Identify what is needed to add governance to a
project. What 2-3 items are the highest priority?
Project: bring new data source in Needs: Ensure all registered stakeholders and their
data governance roles are known. Identify and define business attributes. Profile data for potential issues.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Group experience: What stories are needed if we are bringing in a new data source? Identify data quality dimensions and thresholds Identify archiving and data retention needs Classify the data for visibility and security Identify data stewards Establish profiling and cleansing rules Establish SLAs with the provider Identify legal aspects, regulatory impacts Identify restrictions on the data, permissible use Evaluate risk involved with data Determine requirements to match to master or reference data; identity
attributes Determine upwards and downward impacts to business processes Configuration management; hardware/software needed, documentation Define metadata, definitions, associations.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Registered Data Governance Stakeholders (we also want to identify and record this metadata each iteration/sprint)
Data Subject Area ID
Major Subject Group 1st Sub Grouping 2nd Sub Grouping A
ccou
ntin
g O
pera
tions
Col
late
ral
Cre
dit R
isk
Fina
nce
Fina
ncia
l Rep
ortin
g
Inte
rnal
Aud
it
Mar
ket R
isk
Sale
s
Loan
Ope
ratio
ns
Com
plia
nce
Info
rmat
ion
Tech
nolo
gy
1 Trade Loan Adjustable-Rate Loan
2 Trade Loan Fixed-Rate Loan
3 Trade Loan Structured Loan
4 Trade Loan Amortizing Loan
5 Trade Loan Letters of Credit
6 Trade Loan Prepayment Fee
11 Financial Accounting Accrued Interest
12 Financial Accounting Outstanding Principle
LEGEND
CUSTODIAN
CONSUMERSTEWARDDELEGATE
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance (Overall Framework)
Provide data governance common vocabularyDefine key governance rolesDefine roles & responsibilities of key gov rolesDefine data governance policyDefine concept of data governance controls
Project Specific Data Governance –Risk Project
Confirm data subject areas within project scopeConfirm system of record for each subject areaDefine data quality issues and standardsDefine data quality testing & remediation strategyConfirm data gov roles for each subject areaDetermine existing data controls for each sub areaDetermine recommended data gov controlsDefine recommended data gov control & quality implementation priorityObtain DGWG approval
FEB MAR APR MAY JUN JULJAN AUG SEP
Risk Project Timeline (sample timeline)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Setting up a Governance Framework Identify what is needed to setup a governance
framework. Generally this is done by looking at the data governance roadmap (see next slide for example)
Have the product owner prioritize the first 3 items.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Data Governance (DG) Framework
Establish Data Governance Council (DGC)- Identify and appoint members- Train and educate DGC- Agree on common data governance vocabulary- Agree on common DG stakeholder roles- Agree on roles and responsibilities for
stakeholders- Agree on scope, authority and structure of DGC- Agree on initial DGSC members- Define DGC charter and guiding principles- Agree on key metrics to evaluate governance- Participate in recurring DGC meetings- Champion and promote DGEstablish Data Governance Steering Committee- Identify and appoint members- Train and educate DGSC- Define common data governance vocabulary- Define key DG stakeholder roles- Define roles and responsibilities for DG roles- Define scope, authority and structure of DGSC- Identify and appoint data stewards- Agree on DGSC charter and guiding principles- Define objectives of DGSC- Identify Governance Coordinator for DG
activities- Participate in recurring DGSC meetings - Champion, communicate and promote DG
Data Governance Framework Sample TimelineQ1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Ways Agile can help
Increased communication and visibility Increased teamwork More frequent delivery of business value Implement governance on high priority items first,
show success, move on…continuous improvement
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Continuous improvement
Use retrospectives to make your process better It’s ok to add just 1 or 2 new things at a time Get help if you need it, lots of resources and experts
are out there Mentoring and coaching will be critical
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Temperature Reading Appreciations Puzzles Complaints with recommendations Hopes and wishes New Information
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Appreciations
Appreciations
Puzzles
Complaints w/Recommendations
Hopes &Wishes
New Information
Retrospectives
At end of each sprint/iteration What went well – keep doing What didn’t go well – stop doing What could go better – change it
Entire team attends
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Experience: Retrospective
What went well What didn’t go well What could go better
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Lessons learned
There isn’t one definition of Agile; all companies tend to do it differently. That’s ok. Make it what works best for your organization.
If you are setting up governance for the first time, get the framework in place first. Then determine low hanging fruit, such as data quality issues, and implement governance starting there.
Data Governance MUST involve the business; it can’t just be an IT focus.
A lot of training and education is needed for governance. Data Governance must be added to existing processes
and procedures in order to give it teeth. SDLC Architecture reviews
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Good resources for Agile and Team Building Mike Cohn – MountainGoatSoftware.com PMI-ACP Exam Prep – Mike Griffiths Johanna Rothman books: http://www.jrothman.com/books/ The Human Side of Agile, Gil Broza Agile Retrospectives: Making Good Teams Great, Esther Derby and Diana
Larsen Crucial Conversations, Kerry Patterson Crucial Confrontations, Kerry Patterson Influencer The Deadline: A Novel about Project Management, Tom DeMarco Teamwork is an Individual Skill, Christopher Avery Behind Closed Doors: Secrets of Great Management, Johanna Rothman and
Esther Derby DGIQ presentations: Roberta, Maggie from GM
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Me
Twitter: TamiLFlowers LinkedIn: Tami Flowers Email: [email protected] Slideshare: www.slideshare.net\tamiflowers www.metagovernance.com (blog) Blog for DataVersity
Thanks!
MetaGovernance Solutions LLC. All Rights Reserved. 2016