Download - Why Bad Data May Be Your Best Opportunity
@LiaisonTech
Why Bad Data May Be Your Best Opportunity
@LiaisonTech
Bad Data Is…?
Not just…..
• Mal-formed file which won’t parse
• A document which tells lies
• Database of inconsistent information
• Out-of-date documentation
• An office rumor
• Commonly-accepted explanation which does not explain well
@LiaisonTech
What is Data?
“Data” for Our Purposes Is:
• All application or other stored data items
• All data items flowing through your enterprise
• All metadata for stored or flowing data items
@LiaisonTech
What is Bad Data?
“Bad” Data for Our Purposes Is:
• Difficult to understand
• Difficult to change
• Difficult to analyze
• Difficult to govern
@LiaisonTech
Consequences of Bad Data
• Dislike, Loathing
• Smells, looks, feels, sounds unpleasant
• Workarounds, Hacks
• Wastes money and limits business
Bad Data Has Consequences:
@LiaisonTech
Why Should You Care?
Today we will:
• Look at concrete, practical ways to begin taming bad data
• Learn from the mistakes of others
• Explore ideas so you can convince your boss your organization can do better
@LiaisonTech
Why Should You Care?
If you want to build a ship, don’t drum up people to collect wood and don’t
assign them tasks and work, but rather teach them to long for the endless immensity of
the sea.
- Antoine de Saint Exupéry*
* As quoted in A Mathematician’s Lament by Paul Lockhart
@LiaisonTech
Start by Exploring the Sea of Ideas
• Based on concrete, real-world challenges
• Expose patterns of bad and good data
• Lead to practical ways to get better
@LiaisonTech
Consistent, Offshore, Model-based SOA
Service WSDLS embed XML Schemas
First Consumer.
Usage of the operation schemas defined for that consumer.
@LiaisonTech
The Nth Consumer’s Needs & the Model
What is the correct way
to get from here
to here?
@LiaisonTech
The Nth Consumer’s Needs
• How to satisfy the next consumers’ needs consistently
- E.g. spreadsheets make consistency hard
• Micro use cases
• Communicating and enforcing the usage specification for a sprawling XML Schema
- XML Schema not designed for this
- Else unintended coupling among consumes over time
• Minor revisions of the service
• What tooling is needed?
@LiaisonTech
Offshore Development and SOA
• Message transformation between applications and the SOA service
• Offshore transformation development
• Time zone and cultural differences
- Accurate, unambiguous, complete specifications
• Governance to ensure only expected parts of messages populated
• What tooling is needed?
@LiaisonTech
Good Integration using SOA
• All consumers represent the same information in the say way in the service schema
• Consistent and accurate published specifications
• Infrastructure changes verified and governed
@LiaisonTech
Challenge: Rate of Change
• Point to point data flows are isolated
• SOA can mean service version changes, more rapid rate of change• Pub/Sub can have the same issues
• Greater need for accurate, managed, published specifications
• Greater need for governance
@LiaisonTech
Rate of Change
• Some customers change SOA XML Schemas twice a week during application development
• Forces software vendors to design for rapid rates of change
• Forces greater automation for creating and managing data integrations
@LiaisonTech
Automation Options
1. Derive specifications and implementations from the same artifacts
• Changing one changes the other
• SOA service specifications and data transformations from the same tool
• Linkages between tools, such as including enterprise vocabulary information in specifications
@LiaisonTech
Automation Options
2. Simplify User Experience
• Reduce cognitive load
- Guide data mapping based on specifications
• Facilitate communication
- Provide a common culture
- Capture, show, interpret information unambiguously
• Automate time-consuming tasks
- Tools not solutions
- “rather teach them to long for the endless immensity of the sea”
@LiaisonTech
Automation Options
3. Governance and Reuse
• Automate process steps
- Verify transformation steps against specifications
• Interoperability
- Integration with other tools
• Portability
- Runtime platform independence
@LiaisonTech
Automation Options
4. Low Barrier to Entry
• When start with simple steps, data captured for powerful results later
- Data transformations contain much information which can be reused later
• Easy integration with other tools
- No big porting efforts or reimplementation
@LiaisonTech
Liaison Contivo
Yes, this is part of what Contivo does. We designed it with this vision in mind.
@LiaisonTech
Bad to Good
• “SOA is pie-in-the-sky.” (pharma)
• “We maximized the cost of change.” (finance)
• “If he dies, nobody will….” (pharma)
• “If they can get it wrong, they will get it wrong.” (finance)
@LiaisonTech
Bad to Good Manifesto
• There is inherent order in your data
• Tools should extend the human
• Collaborative tools communicate corporate culture
• Invest in tools which let you start small and yield compounded returns
@LiaisonTech
Bad to Good Roadmap
1. Start doing your data integration projects with Contivo
• Seriously
• For many reasons
• Benefit from the experience of many others
• Define data transformations with Contivo Analyst
• It will capture much of what you will need to unsnarl the interwoven web of data you have now
@LiaisonTech
Bad to Good Roadmap
2. Do more data integrations with Contivo
• Store your maps, data samples, etc. in the Contivo Repository
@LiaisonTech
Bad To Good Roadmap
3. You will be able to
• See a big picture of your data
• Analyze your system of data integration requirements and options
• Postulate good data architecture directions
• Start doing projects in a way which will lead to good architectural results
• Have the data to justify and plan architectural overhauls
@LiaisonTech
Bad to Good Roadmap
4. Talk to us about your unique challenges
If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long
for the endless immensity of the sea.
@LiaisonTech
The Order in Your Data
There is order in your data. Our job is to provide the tools so you can find and use it.
@LiaisonTech
Solutions
• Data Integration• Data Management • Data Security
Multinational
• Global headquarters in Atlanta• European offices in Finland,
Netherlands, Sweden, UK• More than 7000 customers
worldwide in over 46 countries
For more presentations:
Liaison Webinars
AB
OU
T L
IAIS
ON
28
Additional Resources