an overview of domain-driven data mining: toward actionable knowledge discovery (akd)

DD33M

:M

: D

om

ain

-Dri

ven

Data

Min

ing

An Overview ofDomain-Driven Data Mining:

Toward Actionable Knowledge Discovery (AKD)

Longbing Cao

Faculty of Engineering and Information TechnologyUniversity of Technology, Sydney, Australia

15 December 2008

Cao, L: D3M at DDDM2008 Joint with ICDM2008 2

DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing

The Smart Lab: datamining.it.uts.edu.au

Outline

Why Do We Need D3M What Is D3M The D3M Framework D3M Theoretical Underpinnings D3M Research Issues D3M Applications D3M References

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Why Do We Need D3M A common scenario in deploying data

mining algorithms I find something interesting!

“Many patterns are found”, “They satisfy technical metric threshold well”

What do business people say? “So what?” “They are just commonsense” “I don’t care about them” “I don’t understand them” “How can I use them?”

“Am I wrong? What can I do better for my business mate?”

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Why Do We Need D3M Where is something wrong?

Gap: academic objectives || business goals Technical outputs || business expectation

macro-level methodological and fundamental issues Academic: technical interest; innovative algorithms &

patterns Practitioner: social, environmental, organizational

factors and impact; getting a problem solved properly micro-level technical and engineering issues

System dynamics, system environment, and interaction in a system

Business processes, organizational factors, and constraints

Human and domain knowledge involvement

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


An example: Problem with association mining Existing association rule mining algorithms are

specifically designed to find strong patterns that have high predictive accuracy or correlation;

While frequent patterns are referred to as commonsense knowledge, they can be eager to discover new and hidden patterns in databases.

Many patterns are found; How associations can be taken over by business

people seamlessly and into operationalizable actions accordingly?

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


What Is D3M Next-generation data mining

methodologies, frameworks, algorithms, evaluation systems, tools and decision support, Cater for business environment Satisfy business needs Deliver business-friendly and decision-making

rules and actions that are of solid technical and business significance

Can be understood & taken over by business people to make decision

aim to promote the paradigm shift from data-data-centered hidden pattern miningcentered hidden pattern mining to domain-domain-driven actionable knowledge discoverydriven actionable knowledge discovery (AKD)

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Involve and synthesize Ubiquitous Intelligence human intelligence, domain intelligence, data intelligence, network intelligence, organizational and social intelligence,

and meta-synthesis of the above ubiquitous

intelligence

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


The D3M Framework

AKD-based problem-solving

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Interestingness & actionability

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Conflicts & tradeoff

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


A framework for AKD Post-analysis-based AKD

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


D3M Theoretical Underpinnings artificial intelligence and intelligent systems, behavior informatics and analytics, business modeling, business process management, cognitive sciences, data integration, human-machine interaction, human-centered computing, knowledge representation and management, machine learning, ontological engineering, organizational and social computing, project management methodology, social network analysis, statistics, system simulation, and so on.

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


D3M Research Issues Data Intelligence:

deep knowledge in complex data structure; mining in-depth data patterns, and mining structured & informative knowledge in complex data

Domain Intelligence: Domain & prior knowledge, business processes/logics/workflow, constraints, and

business interestingness; representation, modeling and involvement of them in KDD

Network Intelligence: network-based data, knowledge, communities and resources; information retrieval,

text mining, web mining, semantic web, ontological engineering techniques, and web knowledge management

Human Intelligence: empirical and implicit knowledge, expert knowledge and thoughts, group/collective

intelligence; human-machine interaction, representation and involvement of human intelligence

Social Intelligence: organizational/social factors, laws/policies/protocols, trust/utility/benefit-cost;

collective intelligence, social network analysis, and social cognition interaction Intelligence metasynthesis:

Synthesize ubiquitous intelligence in KDD; metasynthetic interaction (m-interaction) as working mechanism, and metasynthetic space (m-space) as an AKD-based problem-solving system

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


How to reach an interest tradeoff Balance between technical and business

interests Suppose there are multiple metrics for

each aspect

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


actionable knowledge discovery through m-spaces acquiring and representing unstructured, ill-

structured and uncertain domain/human knowledge supporting dynamic involvement of business experts

and their knowledge/intelligence acquiring and representing expert thinking such as

imaginary thinking and creative thinking in group heuristic discussions during KDD modeling

acquiring and representing group/collective interaction behavior and impact emergence

Building infrastructure supporting the involvement and synthesis of ubiquitous intelligence

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


D3M Applications

Real-world data mining Our recent case studies

Capital markets actionable trading agents actionable trading strategies

Social security activity mining combined mining

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Actionable Trading Evidence for Brokerage Firms

Trading strategy/evidence

Actionable trading evidence

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Domain factors

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Business interest

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Developing in-depth trading strategy

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Activity mining for Australian Commonwealth Governmental Debt Prevention

Impact-targeted activity mining

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Impact-targeted activity mining Frequent impact-targeted activity

sequences Impact-contrasted activity sequences Impact-reversed activity sequences Impact-targeted combined association

clusters

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Data intelligence Activity data Itemset imbalance Impact imbalance Seasonal effect Demographic data Transactional data Itemset/tuple selection/construction

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Domain intelligence Business process/event for activity selection Domain knowledge Feature selection Sequence construction Impact target

Positive impact Negative impact Multi-level impacts

Feature/attribute selection Interestingness definition New pattern structures

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Organizational/social factors Operational/intervention activities Seasonal business requirement/

interaction changes Business cost (debt amount/duration) Business benefit (saving/preventing debt

amount or reducing debt duration) Deliverable format

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Impact-reserved pattern pair Underlying pattern 1: Derivative pattern 2:

Impact-targeted combined association clusters

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Conditional impact ratio (Cir)

Conditional Piatetsky-Shapiro’s (P-S) ratio (Cps)

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Interestingness: tech & biz

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


The process

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Impact-reversed sequential activity patterns

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Demographic + transactional combined pattern

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


D3M ReferencesBooks: Cao, L. Yu, P.S., Zhang, C., Zhao, Y. Domain Driven Data Mining, Springer, 2009. Cao, L. Yu, P.S., Zhang, C., Zhang, H.(ed.) Data Mining for Business Applications, Springer, 2008.

Workshops: Domain-driven data mining 2008, joint with ICDM2008. Domain-driven data mining 2007, joint with SIGKDD2007.

Special issues: Domain-driven data mining, IEEE Trans. Knowledge and Data Engineering, 2009. Domain-driven, actionable knowledge discovery, IEEE Intelligent Systems, Department, 22(4): 78-89, 2007.

Some of relevant papers: Longbing Cao, Yanchang Zhao, Huaifeng Zhang, Dan Luo, Chengqi Zhang. Flexible Frameworks for Actionable

Knowledge Discovery, submitted to IEEE Trans. on Knowledge and Data Engineering. Cao, L., Zhang, H., Zhao, Y., Zhang, C. Combined Mining: Discovering More Informative Knowledge in e-

Government Services, submitted to ACM TKDD, 2008. Cao, L., Dai, R., Zhou, M.: Metasynthesis, M-Space and M-Interaction for Open Complex Giant Systems, technical

report, 2008. Cao, L. and Ou, Y. Market Microstructure Patterns Powering Trading and Surveillance Agents. Journal of Universal

Computer Sciences, 2008 (to appear). Cao, L. and He, T. Developing actionable trading agents, Knowledge and Information Systems: An International

Journal, 2008. Cao, L. Developing Actionable Trading Strategies, in edited book: Intelligent Agents in the Evolution of WEB and

Applications, Springer, 2008.

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Some of relevant papers: Cao, L., Zhao, Y., Zhang, C. (2008), Mining Impact-Targeted Activity Patterns in Imbalanced Data, IEEE

Trans. Knowledge and Data Engineering, IEEE, , Vol. 20, No. 8, pp. 1053-1066, 2008. Cao, L., Yu, P., Zhang, C., Zhao, Y., Williams, G.:DDDM2007: Domain Driven Data Mining, ACM SIGKDD

Explorations Newsletter, 9(2): 84-86, 2007. Cao, L., Zhang, C.: Knowledge Actionability: Satisfying Technical and Business Interestingness,

International Journal of Business Intelligence and Data Mining, 2(4): 496-514, 2007. Cao, L., Zhang, C.: The Evolution of KDD: Towards Domain-Driven Data Mining, International Journal of

Pattern Recognition and Artificial Intelligence, 21(4): 677-692, 2007. Cao, L.: Domain-Driven Actionable Knowledge Discovery, IEEE Intelligent Systems, 22(4): 78-89, 2007. Cao, L., and Zhang, C. Domain-driven data mining: A practical methodology, International Journal of

Data Warehousing and Mining (IJDWM), IGI Global, 2(4):49-65, 2006.

15 December 2008


DD33MM:: D

om

ain

-Dri

ven

Data

Min

ing


Thank you!

Longbing CAO

Faculty of Engineering and ITUniversity of Technology, Sydney, Australia

Tel: 61-2-9514 4477 Fax: 61-2-9514 1807email: [email protected]: www-staff.it.uts.edu.au/~lbcao/The Smart Lab: datamining.it.uts.edu.au

an overview of domain-driven data mining: toward actionable knowledge discovery (akd)

Documents