suriadi caise2013 slides
TRANSCRIPT
Introduction Approach Case Study Summary Future Work
Understanding Process Behaviours in a LargeInsurance Company in Australia: A Case Study1
S. Suriadi, M. Wynn, C. Ouyang, A.H.M. ter Hofstede, andN. van Dijk
Information Systems SchoolQueensland University of Technology
Brisbane, Australia
June 21, 2013
1This work was supported by the Australian Research Council Discovery Project grant DP110100091.
S.Suriadi et al. Process Mining Case Study at Suncorp 1/ 23
Introduction Approach Case Study Summary Future Work
Outline
1 Introduction
2 Approach
3 Case Study
4 Summary
5 Future Work
S.Suriadi et al. Process Mining Case Study at Suncorp 2/ 23
Introduction Approach Case Study Summary Future Work
Motivation
Introduction
Case Study
A 6-month case study with Suncorp, one of the largestinsurance organizations in Australia
Goal: to identify reasons for under-performing claims, leadingto process improvement
Approach: Process Mining and L∗-methodology
Why Process Mining?
Explosion of data for analysis
Extract evidence-based insights from data (>100 orgs.)
limited application in Australia
S.Suriadi et al. Process Mining Case Study at Suncorp 3/ 23
Introduction Approach Case Study Summary Future Work
Contributions
Introduction
Contributions
A report on the experiences gained from this case study:
challenges, lessons learned, and recommendations
What’s new?
Validation of existing lessons learned
Detailed (new?) insights for every stage of the case study
Review of process mining-related tools
Novice/beginner’s point of view
S.Suriadi et al. Process Mining Case Study at Suncorp 4/ 23
Introduction Approach Case Study Summary Future Work
Methodology for ‘Spaghetti Process’
L∗ Methodology
Initial interview and preliminary data analysis suggest:
unstructured process
Adopt the L∗ methodology by van der Aalst (2011)
for “spaghetti” process
- business
understanding
- data
understanding
- historic data
- handmade
models
- objectives
- questions
- explore
- discover
- check
- compare
- promote
- diagnose
- verification
- validation
- accreditation
- redesign
- adjust
- intervene
- support
- feedback to
objectives
S.Suriadi et al. Process Mining Case Study at Suncorp 5/ 23
Introduction Approach Case Study Summary Future Work
Methodology for ‘Spaghetti Process’
L∗ Methodology
Initial interview and preliminary data analysis suggest:
unstructured process
Adopt the L∗ methodology by van der Aalst (2011)
for “spaghetti” process
- business
understanding
- data
understanding
- historic data
- handmade
models
- objectives
- questions
- explore
- discover
- check
- compare
- promote
- diagnose
- verification
- validation
- accreditation
- redesign
- adjust
- intervene
- support
- feedback to
objectives
Stage 0:Plan/Justify
S.Suriadi et al. Process Mining Case Study at Suncorp 5/ 23
Introduction Approach Case Study Summary Future Work
Methodology for ‘Spaghetti Process’
L∗ Methodology
Initial interview and preliminary data analysis suggest:
unstructured process
Adopt the L∗ methodology by van der Aalst (2011)
for “spaghetti” process
- business
understanding
- data
understanding
- historic data
- handmade
models
- objectives
- questions
- explore
- discover
- check
- compare
- promote
- diagnose
- verification
- validation
- accreditation
- redesign
- adjust
- intervene
- support
- feedback to
objectives
Stage 0:Plan/Justify
Stage 1:Extract
S.Suriadi et al. Process Mining Case Study at Suncorp 5/ 23
Introduction Approach Case Study Summary Future Work
Methodology for ‘Spaghetti Process’
L∗ Methodology
Initial interview and preliminary data analysis suggest:
unstructured process
Adopt the L∗ methodology by van der Aalst (2011)
for “spaghetti” process
- business
understanding
- data
understanding
- historic data
- handmade
models
- objectives
- questions
- explore
- discover
- check
- compare
- promote
- diagnose
- verification
- validation
- accreditation
- redesign
- adjust
- intervene
- support
- feedback to
objectives
Stage 0:Plan/Justify
Stage 1:Extract
Stage 2: Control Flow and Connect
S.Suriadi et al. Process Mining Case Study at Suncorp 5/ 23
Introduction Approach Case Study Summary Future Work
Stage 1
Stage 1: Planning
Activities:
Presentation and discussionsExtract question(s) and assess data availabilityMutually-beneficial engagement model
Close engagement
Main Question
Why did the processing of certain ‘simple’ claim take such a longtime to complete?
Case study was conducted in two phases:
First phase: “data-driven” (quite ‘explorative’)Second phase: “question-driven”
S.Suriadi et al. Process Mining Case Study at Suncorp 6/ 23
Introduction Approach Case Study Summary Future Work
Stage 1
Stage 1: Planning - Challenges and Lessons Learned
Challenges
Defining the concept of a ‘simple’ claim
No corresponding business rules‘commonly-held’ belief, differing opinions
Took up to 5 interview sessions spanning up to 6 weeks intotal
Lessons Learned
Composition of team is crucial for effective communication
Use “question-driven” approach
S.Suriadi et al. Process Mining Case Study at Suncorp 7/ 23
Introduction Approach Case Study Summary Future Work
Stage 1
Stage 1: Planning - Results
Results
A clear definition of ‘simple’ claim
A claim with less than x-amount of claim value and should becompleted in no later than y -number of days
Derivation of process mining questions:
Q1: What is the performance distribution of ‘simple’ vs‘non-simple’ claims?Q2: What do the corresponding process models look like?Q3: What are the key differences in the processing of “simplequick” vs. “simple slow” claims that lead to performancedifferences?
S.Suriadi et al. Process Mining Case Study at Suncorp 8/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract
Two phases:
First round: data quality issue, omission of importantinformation, poorly-populated fieldsSecond round: cleaner data with more accurate information
All finalized claims (no incomplete cases)
>32,000 claimsover 1 million unique events
Data cleaning, filtering, and conversion to XES/MXML
One of the most challenging stages in the case study
S.Suriadi et al. Process Mining Case Study at Suncorp 9/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract - Challenges
Challenges
Data interpretation
Complex process (over 20 top-level activities, > 100second-level activities)‘High process variant’ is the norm!
flexibility and/or ‘sloppy’ logging practice?
Inconsistent terminology (nat. hazard vs. storm vs. flood)
Data Filtering
Necessary to ensure scalability of analysis and interestingcomparative analysisWould like to apply hierarchical filtering, but
which criteria to use?
Related to poor definition of ‘simple’ claims
S.Suriadi et al. Process Mining Case Study at Suncorp 10/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract - Challenges
Challenges
Data manipulationMany tools involved, each with its own
strengths and weaknesses, and‘quirkiness’ w.r.t data format (input and output)
Need to use them allHighly tedious!
Spreadsheet Disco
Database
(e.g. MySQL)
Text/XML Editor
ProM Tool
CSV XESXESAME
S.Suriadi et al. Process Mining Case Study at Suncorp 11/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract - Lessons Learned
Lessons Learned
Naive interpretation of data –> meaningless findings, e.g.
‘complete’ timestamp interpretation?activity duration (from [scheduled, assign, start] to complete) -big differences, often.
schedule
(recorded)
assign
(recorded)actual completion time
(not recorded) recorded as complete
(recorded)
start
(not recorded)
{actual duration
activity duration estimate_1activity duration estimate 2
S.Suriadi et al. Process Mining Case Study at Suncorp 12/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract - Lessons Learned
Lessons Learned
Order of data filter matters!
Case-level vs. event-level filtering, e.g.Rule A: remove all events not done by RRule B: remove all cases longer than 7 days
DISCO vs. XESAME
Rule A
event resourceactivityA (start) resourceRactivityB resourceCactivityC resourceCactivityD resourceRactivityF (end) resourceE
{10 days
{
7 days
The case does not satisfy
Rule B - thus whole
case is removed.
Rule A
Rule B
Rule B
1stFilter
2ndFilter
..........
Result
caseID:123ABC event resource <event removed>activityB resourceCactivityC resourceC <event removed> activityF resourceE
caseID:123ABC
event resourceactivityB resourceCactivityC resourceCactivityF resource E
caseID:123ABC
S.Suriadi et al. Process Mining Case Study at Suncorp 13/ 23
Introduction Approach Case Study Summary Future Work
Stage 2
Stage 2: Extract - Result
Result
Clear identification of data slices to be used for analysis andcomparison to address the 3 process mining questions definedearlier.
Cas
e D
ura
tio
n (
day
s)
$x
Payout amount
Simple Slow (SS):
<=$x payout
> y days
Simple Quick (SQ):
<=$x payout
<= y days
Complex Slow (CS):
>$x payout
> y days
Complex Quick (CQ):
>$x payout
<=y days
"in-between" cases
y+1
y
y-1
S.Suriadi et al. Process Mining Case Study at Suncorp 14/ 23
Introduction Approach Case Study Summary Future Work
Stage 3
Stage 3: Analysis - Challenges
Control-flow analysis, using Disco and ProM Tool.
Challenges
Discovering meaningful process models
Dealing with complex log‘Inconsistency in behaviour’ is the norm!
Practicality: time consuming process (especially if data had tobe further filtered/cleaned)
genetic miner, ILP miner (sometimes)
S.Suriadi et al. Process Mining Case Study at Suncorp 15/ 23
Introduction Approach Case Study Summary Future Work
Stage 3
Stage 3: Analysis - Lessons Learned
Lessons Learned
No ‘clean’ structure or we have not done proper datapre-processing?
Clues for the non-existence of structured process: theexistence of
high process variants,low fitness value of Heuristic nets, andflower-like model in simplified Petri Nets,
despite the application of hierarchical filtering and othercleaning activities.
Useful algorithms: Heuristics Miner, ILP, Uma, as well asFuzzy miner (in Disco).
Not so useful: Genetic miner
S.Suriadi et al. Process Mining Case Study at Suncorp 16/ 23
Introduction Approach Case Study Summary Future Work
Stage 3
Stage 3: Analysis - Results
Results
Q1: a significant number of under-performing cases
Q2: unstructured process indeed!
S.Suriadi et al. Process Mining Case Study at Suncorp 17/ 23
Introduction Approach Case Study Summary Future Work
Stage 4
Stage 4: Interpretation
Proper identification of differences via classical data miningtechniques
Compare Simple Quick vs. Simple Slow claims
Challenges
Finding a good set of predictor variables (too many attributes)
S.Suriadi et al. Process Mining Case Study at Suncorp 18/ 23
Introduction Approach Case Study Summary Future Work
Stage 4
Stage 4: Interpretation - Lessons Learned and Results
Lessons Learned
Two useful process-related metrics:
average execution of an activity-X per casedistribution of an activity-X over all cases
Results
Q3 is addressed
Key differences between ‘Simple Quick’ and ‘Simple Slow’classes were identified.
S.Suriadi et al. Process Mining Case Study at Suncorp 19/ 23
Introduction Approach Case Study Summary Future Work
Stage 4
Stage 4: Interpretation - ResultActivity Simple Quick Simple Slow
actFreq actDist actFreq actDist
Follow Up Requested 1.86 74.4% 5.79 92.3%Incoming Correspondence 1.75 81.6% 4.27 90.1%Contact Customer 0.66 46.8% 1.29 63.3%Contact Assessor 0.11 4.9% 1.36 21.5%Conduct File Review 2.03 89.8% 6.11 96.9%
S.Suriadi et al. Process Mining Case Study at Suncorp 20/ 23
Introduction Approach Case Study Summary Future Work
Conclusion
Conclusion of the Case Study - Stage 5
Useful findings:
Surprising number of under-performing claimsHighlighted non-uniformity and the need for standardization inthe claims processing
Triggered some changes in their claims processing system
“...by mining and analysing our claims ... our business has beenable to make cost saving adjustments to the existing process.”a new system was trialled
Another subsequent impact (not involved, but informedverbally)
improved the proportion of claims finalized on-time
S.Suriadi et al. Process Mining Case Study at Suncorp 21/ 23
Introduction Approach Case Study Summary Future Work
Future Work
Future Work
Our case study: too afraid to manipulate data too much(limited data removal, conservative filtering)
But, fine-line between abstraction and over-simplification
When do we stop simplifying/cleaning data?
Data quality - event log quality
Claim: objective insightsPitfall: low data quality (incomplete/incorrect/noisy) and/orimproper cleaning, abstraction and manipulation
Verification of results is needed!
Establish links between original data and resultsAssert the correctness of resultsImproved accountability in making important decisions
S.Suriadi et al. Process Mining Case Study at Suncorp 22/ 23
Introduction Approach Case Study Summary Future Work
Future Work
Understanding Process Behaviours in a LargeInsurance Company in Australia: A Case Study2
S. Suriadi, M. Wynn, C. Ouyang, A.H.M. ter Hofstede, andN. van Dijk
Information Systems SchoolQueensland University of Technology
Brisbane, Australia
June 21, 2013
2This work was supported by the Australian Research Council Discovery Project grant DP110100091.
S.Suriadi et al. Process Mining Case Study at Suncorp 23/ 23