based on the book “building data mining applications for crm” by alex berson stephen smith kurt...
Post on 30-Dec-2015
225 Views
Preview:
TRANSCRIPT
Based on the book “Building Data Mining Applications for CRM”
By
Alex Berson
Stephen Smith
Kurt Thearling
Data Mining Applications for CRM
Data Mining Applications for CRM
Summary of Topics1. Customer Relationship Management-Framework and Architecture2. Reinforcing CRM with Data Mining 3. Data Mining –An Overview4. Key Terms5. Data Mining Methodology 6. Classical Techniques: Statistics, Neighborhoods, and Clustering7. Next Generation Techniques: Trees, Networks, and Rules8. CRM -The Business Perspective9. Deploying Data Mining for CRM10. Data Quality11. Next Generation of Information Mining and Knowledge
Discovery for Effective CRM12. CRM in the e-Business World
Topic 1: Customer Relationship Topic 1: Customer Relationship Management-Framework and Management-Framework and
ArchitectureArchitecture CRM is an enterprise approach to customer service that uses
meaningful communication to understand and influence consumer behavior. The purpose of the process is twofold:
a: To impact all aspects to the consumer relationship (e.g., improve customer satisfaction, enhance customer loyalty, and increase profitability) and
B: To ensure that employees within an organization are using CRM tools. The need for greater profitability requires an organization to proactively pursue its relationships with customers.
Customer Relationship Management-Customer Relationship Management-Framework and ArchitectureFramework and Architecture
Which customers are most profitable to me? Why? What promotions are most effective? For which customers? What kind of customers will be interested in my new product? What customers are at risk to defect to my competitor? How do I identify prospects with the greatest profit potentials?
Customer information is rapidly becoming a company’s most
important asset to answer these questions. However, to answer these
questions in broad generalities is not enough. Each customer must be
analyzed and potentially treated uniquely. Customer Relationship
Management provides the framework for analyzing customer
profitability and improving marketing effectiveness.
Customer Relationship Customer Relationship Management Management -Framework and -Framework and
ArchitectureArchitectureMany organizations have collected and stored a wealth of data about their customers, suppliers, and business partners. However, the inability to discover valuable information hidden in the data prevents these organizations from transforming this data into knowledge. The business desire is, therefore, toextract valid, previously unknown, and comprehensible information from large databases and use it for profits. To fulfill these goals, organizations need to follow these steps:
- Capture and integrate both the internal and external data into a comprehensive view that encompasses the whole organization.- “Mine” the integrated data for information.- Organize and present the information with knowledge for decision-making.
Data, Information, and Data, Information, and DecisionDecision
Data Resource Management (DRM)
MIS (OLTP) & OOAD
KM (Knowledge Mgt), KWS (Knowledge Work Systems)
DSS; ESS, EIS (Executive Information Systems)
Data Warehousing/Data Mart/Data Mining/OLAP (Executive, Collaborative and individual levels)
Business Intelligence
Data
Information (Data + Process)
Knowledge/Business Intelligence
Decision (Information +
Knowledge)
Data/Information/Decision /Business Intelligence
Customer Relationship ManagementCustomer Relationship Management ----Framework Framework and Architectureand Architecture
From the architecture point of view, the entire CRM framework can
be classified into three key components:
Operational CRM – The automation of horizontally integrated business processes, including customer touch-points, channels, and front-back office integration.
Analytical CRM- The analysis of data created by the Operational CRM
Collaborative CRM- Applications of Collaborative services including e-mail, personalized publishing, e-communities, and similar vehicles designed to facilitate interactions between customers and organizations.
CRM ArchitectureCRM Architecture
ETLTools
Data Sources
Market DataStore
CommunicationChannels
Call Center
Call Center
CampaignMgt
Direct Mails
MarketingData Marts
Contact History
External Data
TransactionHistory
Customer ProfileAnd account
AnalyticsData Mart
ReportingData Mart
Decision Support Applications
CampaignMgt
Data MiningAnalytics
ReportingData Mart
Contact Mgt
Customer ServiceCenter
Internet
Other
Business Rules and Metadata Management
Workflow Management
Campaign MGT Software-Managing Campaign MGT Software-Managing CampaignsCampaigns
Accommodation of many new touch points besides direct mail, for ex., the Web, direct TV ad., hard copy advertising customer services, street brochure dispatch, and signage.
Focus on profitability (not only on which customer was most profitable, but also on what was the most profitable promotion that could be sent., e.g., send $.025 postcard rather than the $25 rebate if both have the same effect).
Optimization of the sequence of promotion delivery. Tools for constructing experiments that allow the marketing professionals
to test out the effectiveness of new promotions and new segmentation techniques, for ex., using different contents and timing for signage advertising.
Accommodation by the system of predictive modeling from data mining , which provides insights into future customer behavior and future customer profitability.
Web-Enabled Information DeliveryWeb-Enabled Information Delivery
WebServer
QueryEngine
AnalyticsDrill DownAgents
UnstructuredContent
SQL
HTML
CGI
HTML
WebBrowser
StructuredContent
How about the web Log, or “blog” which has become a popular source for information acquisition.
Topic 2:Topic 2: Reinforcing CRM with Data Mining
Companies worldwide are beginning to realize that surviving an intensively competitive and global marketplace requires closer relationships with customers. In turn, enhanced customer relationships can boost profitability three ways: a) by reducing costs by attracting more suitable customers, b) by generating profits through cross-selling and up-selling activities, and c) by extending profits through customer retention. Slightly expanded explanations of these activities follow:
Reinforcing CRM with Data Mining
Attracting more suitable customers: Data mining can help firms understand which customers are most likely to purchase specific products and services, thus enabling businesses to develop targeted marketing programs for higher response rates and better returns on investment.
Better cross-selling and up-selling: Businesses can increase their value proposition by offering additional products and services that are actually desired by customers, thereby raising satisfaction levels and reinforcing purchasing habits.
* Better retention: Data-mining techniques can identify which customers are more likely to defect and why. A company can use this information to generate ideas that allow them to maintain these customers.
DW Technologies and Tools-An DW Technologies and Tools-An OverviewOverview
Data Modeling
Data AcquisitionOLAP
Extraction
SourceSystems
Data Storage
Data Loading
Load Image Creation
Information Delivery
AlertSystems
DataMining
Report Writer
StagingArea
DW/Data Marts
QualityAssurance
Transformation
DW Information FlowDW Information Flow
Data Warehouse DatabaseData Warehouse DatabaseThe central data warehouse database is a cornerstone of data warehousing environment. On the architecture diagram, the database is almost always implemented on the relational database management system (RDBMS) technology. Now the now approaches include the following:
Multidimensional database (MDDBs)- This is tightly coupled with the online analytical processing (OLAP) tools that act as clients to the multidimensional data stores.
An innovative approach to speed up a traditional RDBMs by using new index structures to bypass relational table scans.
Parallel relational database designs that require a parallel computing platforms, for ex., symmetric multiprocessor (SMP), massively parallel processors (MPPs), and or clusters of uni-or multiprocessors.
Information Delivery Tool Information Delivery Tool TaxonomyTaxonomy
Tools are generally divided into five main groups :
Data query and reporting tools.
Application development tools.
Executive Information System (EIS) tools.
Online analytical processing tools.
Data mining tools.
Topic 3:Topic 3: Data Mining: An OverviewData Mining: An Overview
Data mining can help reduce information overload and improve decision making. This is achieved by extracting and refining useful knowledge through a process of searching for relationships and patterns from the extensive data collected by organizations. The extracted information is used to predict, classify, model, and summarize the data being mined. Data-mining technologies, such as rule induction, neural networks, genetic algorithms, fuzzy logic, and rough sets, are used for classification and pattern recognition in many industries.
Data Mining: An OverviewData Mining: An Overview
A supermarket organizes its merchandise stock based on shoppers' purchase patterns.
An airline reservation system uses customers' travel patterns and trends to increase seat utilization.
Web pages alter their organizational structure or visual appearance based on information about the person who is requesting the pages.
Individuals perform a Web-based query to find the median income of households in Iowa.
Data Mining: An OverviewData Mining: An Overview
Data mining builds models of customer behavior by using established statistical and machine-learning techniques. The basic objective is to construct a model for one situation in which the answer or output is known and then apply that model to another situation in which the answer or output is sought. The best applications of the above techniques are integrated with data warehouses and other interactive, flexible business analysis tools. The analytic data warehouse can thus improve business processes across the organization in areas such as campaign management, new product rollout, and fraud detection.
Data Mining: An OverviewData Mining: An Overview
Data mining integrates different technologies to populate, organize, and manage the data store. Because quality data is crucial to accurate results, data-mining tools must be able to clean the data, making it consistent, uniform, and compatible with the data store. Data mining employs several techniques to extract important information. Operations are the actions that can be performed on accumulated data, including predictive modeling, database segmentation, link analysis, and deviation detection.
Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools
We can divided the entire data mining tool market into three main groups: General-purpose tools, integrated DSS/OLAP/data miningtools, and rapidly growing, application-specific tools.
The General-purpose tools which occupy the larger and more mature segment of the market include the following:
SAS Enterprise Minor IBM Intelligent Minor Unica PRW SPSS Clementine SGI Mineset Oracle Darwin Angoss KnowledgeSeeker
Taxonomy of Data Mining ToolsTaxonomy of Data Mining ToolsThe integrated data mining tool segment addresses a very real and
compelling business requirement of having a single multi-function,
decision-support tool that can provide management reporting, online
analytical processing, and data mining capabilities within a common
framework. Examples of these integrated tools include Cognos
scenario and Business Objects.
The application-specific tools segment is rapidly gaining momentum.
Among these tools are the following:KDI (focuses on retail)
Options & Choices (focuses on insurance industries)
HNC (focuses on fraud detection)
Unica Model 1 (focuses on marketing)
Database Mining Workstation Database Mining Workstation (HNC)(HNC)
HNC is one of the most successful data mining companies. Its Database
Mining workstation (DMW) is a neural network tool that is widely-accepted
For credit card fraud analysis applications. DMW consists of Windows–based
software applications and a custom processing board. Other HNC products
include Falcon and ProfitMax processing applications for financial services,
and the Advanced Telecommunications Abuse Control System (ATACS)
fraud-detection solution that HNC plans to deploy in the Telecommunications
Industries.
Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools
There are specific tools, for example, for the following applications:
Financial Data Analysis: neural networks have been used in forecasting stock prices, option trading, rating bonds, portfolio management, commodity-price prediction, and mergers and acquisition analysis. Using IBM Intelligent minor, Mellon Bank developed a credit card-attrition model to predict which customers will stop using Mellon’s credit card in the next few months.
Telecommunications Industry: The hyper-competitive nature of the industry has created a need to understand customers, to keep them, to model effective ways to market new products.
Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools Retail Industry: Retail data mining can help identify customer-
buying behaviors, discover consumer-shopping patterns and trends.
Healthcare and biomedical research: The analysis of large quantities of time-stamped data will provide doctors with important information regarding the progress of the decease. For ex., NeuroMedicalSystems used neural networks to perform a pap smear diagnostic aid.
Science and engineering: To improve its manufacturing process. Boeing has successfully applied machine-learning algorithms to the discovery of informative and useful rules from its plant data.
Data Mining vs. Data WarehouseData Mining vs. Data Warehouse
Major challenge to exploit data mining is identifying suitable data to mine.
Data mining requires single, separate, clean, integrated, and self-consistent source of data.
A data warehouse is well equipped for providing data for mining.
Data quality and consistency is a pre-requisite for mining to ensure the accuracy of the predictive models. Data warehouses are populated with clean, consistent data.
Data Mining vs. Data WarehouseData Mining vs. Data Warehouse
Data Mining does not require that a Data Warehouse be built. Often, data can be downloaded from the operational files to flat files that contain the data ready for the data mining analysis.
Data Mining can be implemented rapidly on existing software and hardware platforms. Data Mining tools can analyze massive databases to deliver answers to questions such as, “ Which customers are most likely to respond to my next promotional mailing, and why?”
Data Mining vs. Data WarehouseData Mining vs. Data Warehouse
Advantageous to mine data from multiple sources to discover as many interrelationships as possible. Data warehouses contain data from a number of sources.
Selecting relevant subsets of records and fields for data mining requires query capabilities of the data warehouse.
Results of a data mining study are useful if there is some way to further investigate the uncovered patterns. Data warehouses provide capability to go back to the data source.
Data Mining vs. OLAP
They are two separate breeds of analysis with
entirely different objectives, not to mention
tools, skill sets, and implementation methods.
Data Mining vs. OLAP
With canned reports, ad hoc querying, and OLAP, the
end user defines a hypothesis and determines which data
to examine. With data mining, the tool identifies the
hypothesis, and it actually tells the user where in the data
to start the exploration process.
Data Mining vs. OLAP
Rather than using SQL to filter out values and methodically
reduce the data into a concise answer set, data mining uses
algorithms that exhaustively review the relationships among
data elements to determine if any patterns exist. The whole
purpose of data mining is to yield new business information
that a business person can act on.
OLAP vs. Data Mining ToolsOLAP vs. Data Mining Tools
Are ad hoc, shrink wrapped tools that provide an interface to data
Are used when you have specific known questions
Looks and feels like a spreadsheet that allow rotation, slicing and graphic
Can be deployed to large
number of users
Methods for analyzing multiple data types
-- Regression Trees -- Neural networks -- Genetic algorithms
Are used when you don’t know what the questions are
Usually textual in nature
Usually deployed to a small number of analysts
OLAP Tools Data Mining Tools
Topic 4: Key TermsTopic 4: Key Terms
Application Service Providers:
Offer outsourcing solutions that supply, develop, and manage application specific software and hardware so that customers' internal information technology resources can be freed up.
Business Intelligence:
The type of detailed information that business managers need for analyzing sales trends, customers' purchasing habits, and other key performance metrics in the company.
Key TermsKey Terms
Categorical Data:
Fits into a small number of distinct categories of a discrete nature, in contrast to continuous data, and can be ordered (ordinal), for example, high, medium, or low temperatures, or nonordered (nominal), for example, gender or city.
Classification:
The distribution of things into classes or categories of the same type, or the prediction of the category of data by building a model based on some predictor variables.
Key TermsKey Terms
Clustering:
Groups of items that are similar as identified by algorithms. For example, an insurance company can use clustering to group customers by income, age, policy types, and prior claims. The goal is to divide a data set into groups such that records within a group are as homogeneous as possible and groups are as heterogeneous as possible. When the categories are unspecified, this may be called unsupervised learning.
Genetic Algorithm:
Optimization techniques based on evolutionary concepts that employ processes such as genetic combination, mutation, and natural selection in a design.
Key TermsKey Terms Online Profiling:
The process of collecting and analyzing data from Web site visits, which can be used to personalize a customer's subsequent experiences on the Web site. Network advertisers, for example, can use online profiles to track a user's visits and activities across multiple Web sites, although such a practice is controversial and may be subject to various forms of regulation.
Rough Sets:
A mathematical approach to extract knowledge from imprecise and uncertain data.
Key TermsKey Terms Rule Induction:
The extraction of valid and useful if-then-else rules from data based on their statistical significance levels, which are integrated with commercial data warehouse and OLAP platforms.
Visualization:
Graphically displayed data from simple scatter plots to complex multidimensional representations to facilitate better understanding.
Topic 5: Data Mining MethodologyTopic 5: Data Mining MethodologyThe methodology used today in data mining, when it is well thought
out and well executed, consists of just a few very important concepts.
Finding a pattern in the data and building a model. In general, it means any sequence or pattern of data that occurs more often than one would it to if it were a random event.
Sampling or not having to use all of the data in order to make significant conclusions about what might be happening with other parts of the data.
Validating the predictive models that arise out of data mining algorithm. Finally, coming down to finding the pattern or model that is the beat.
The four parts of data mining technology –patterns, sampling, validation,
and choosing the model.
Pattern and ModelPattern and ModelPattern: An event or combination of events in a database that occurs more often than expected. Typically, this means that its actual occurrence is significantly different than what would be by random chance. (for ex., 121212…?
Model: A description that adequately explains and predicts relevant data but is generally much smaller than the data itself. For real-world applications, a model can be anything from a mathematical Equation, to a set of rules that describes customer segments, to the computer representation of a complex neural network architecture, which translates to several sets of mathematical equations.
Predictive model: A model created or used to perform prediction. In contrast to models created solely for pattern detection, exploration or general organization of the data.
Explanatory: For every increase in 1 % in the interest,auto sales decrease by 5 %.
Predictive: predictions about future buyer behavior.
Traditional DW
Operational
OLAP
(OLTP)
Data Mining
Descriptive: The dealer sold 200 cars last month.
Types Of ModelsTypes Of Models
A high-level View of Modeling A high-level View of Modeling ProcessProcess
HistoricalData
PredictionRecord ???
Model
Model Building
123
The Needs for SamplingThe Needs for Sampling
Containing costs
Speeding up the data gathering
Improving effectiveness
Reducing bias
Sampling DesignSampling Design
Four steps:
Determine the data to be collected or described
Determine the population to be sampled
Choose the type of sample
Decide on the sample size
Two Types of Data Mining Modeling- Two Types of Data Mining Modeling- Verification and DiscoveryVerification and Discovery
The verification model utilizes a process that looks in a database to detect trends and patterns in data that will help answer some specific questions about the business.
In this mode, the user generates a hypothesis about the data, issues a query against the data and examines the results of the query looking for verification of the hypothesis or the user decides that the hypothesis is not valid.
Verification ModelVerification Model
In this model, very little information is created in this extraction process: either the hypothesis is verified or it is not.
Common tools used in this mode are: queries, multidimensional analysis and visualization. What all have in common are that the user is essentially ‘guiding’ the exploration of the data being inspected.
Discovery ModelDiscovery Model
A more popular model is the Discovery Model that utilizes a process that looks in a database to discover and/or predict future patterns. The discovery model is divided into two modes: “Descriptive” and “Predictive”.
Discovery Model- Descriptive ModeDiscovery Model- Descriptive Mode
The Descriptive mode finds hidden patterns without a predetermined idea or hypothesis about what the patterns may be. In other words, the Data Mining software or program takes the initiative in finding what the interesting patterns are, without the user thinking of the relevant questions first. In this mode information is created about the data with very little or guidance from the user. The exploration of the data is done in such a way as to yield as large a number of useful facts about the data in the shortest amount of time.
Discovery Model- Predictive ModeDiscovery Model- Predictive Mode
In the Predictive mode patterns discovered from the database are used to predict the future patterns or trends. Predictive modeling allows the user to submit records with some unknown field values, and the system will guess the unknown values based on previous patterns discovered from the database.
In comparing the two models, one can state that “Verification” can be very inefficient, timely and costly. Whereas, “Discovery” modeling can be very efficient, cost effective, less dependent on user input and increases modeling accuracy.
Predictive ModellingPredictive Modelling
Similar to the human learning experience– uses observations to form a model of the important
characteristics of some phenomenon.
Uses generalizations of ‘real world’ and ability to fit new data into a general framework.
Can analyze a database to determine essential characteristics
(model) about the data set.
Predictive ModellingPredictive Modelling
Model is developed using a supervised learning approach, which has two phases: training and testing.
– Training builds a model using a large sample of historical data called a training set.
– Testing involves trying out the model on new, previously unseen data to determine its accuracy and physical
performance characteristics.
Predictive ModellingPredictive Modelling
Applications of predictive modelling include customer retention management, credit approval, cross selling, and direct marketing.
Two techniques associated with predictive modelling:
A. classification
B. value prediction, distinguished by nature of the variable being predicted.
Predictive Modelling - Predictive Modelling - ClassificationClassification
Used to establish a specific predetermined class for each record in a database from a finite set of possible, class values.
Two specializations of classification: tree induction and neural induction.
Example of Classification using Example of Classification using Tree InductionTree Induction
Example of Classification using Example of Classification using Tree InductionTree Induction
Customer renting property> 2 years
Rent property
Customer age>45
No Yes
No Yes
Rent property
Buy property
Example of Classification using Example of Classification using Neural InductionNeural Induction
Example of Classification Using Example of Classification Using Neural InductionNeural Induction
Each processing unit (circle) in one layer is connected to each processing unit in the next layer by a weighted value, expressing the strength of the relationship. The network attempts to mirror the way the human brain works in recognizing patterns by arithmetically combining all the variables with a given data point.
In this way, it is possible to develop nonlinear predictive models that ‘learn’ by studying combinations of variables and how different combinations of variables affect different data sets.
Predictive Modelling - Value Predictive Modelling - Value PredictionPrediction
Used to estimate a continuous numeric value that is associated with a database record.
Uses the traditional statistical techniques of linear regression and non-linear regression.
Relatively easy-to-use and understand.
Predictive Modelling - Value Predictive Modelling - Value PredictionPrediction
Linear regression attempts to fit a straight line through a plot of the data, such that the line is the best representation of the average of all observations at that point in the plot.
Problem is that the technique only works well with linear data and is sensitive to the presence of outliers (i.e.., data values, which do not conform to the expected norm).
Predictive Modelling - Value Predictive Modelling - Value PredictionPrediction
Although non-linear regression avoids the main problems of linear regression, still not flexible enough to handle all possible shapes of the data plot.
Statistical measurements are fine for building linear models that describe predictable data points, however, most data is not linear
in nature.
Predictive Modelling - Value Predictive Modelling - Value PredictionPrediction
Data mining requires statistical methods that can accommodate non-linearity, outliers, and non-numeric data.
Applications of value prediction include credit card fraud detection or target mailing list identification.
Database SegmentationDatabase Segmentation
Aim is to partition a database into an unknown number of segments, or clusters, of similar records.
Uses unsupervised learning to discover homogeneous sub-
populations in a database to improve the accuracy of the profiles.
Database SegmentationDatabase Segmentation
Less precise than other operations thus less sensitive to redundant and irrelevant features.
Sensitivity can be reduced by ignoring a subset of the attributes that describe each instance or by assigning a weighting factor to each variable.
Applications of database segmentation include customer profiling,
direct marketing, and cross selling.
Example of Database Segmentation Example of Database Segmentation using a Scatter plotusing a Scatter plot
Database SegmentationDatabase Segmentation
Associated with demographic or neural clustering techniques, distinguished by: Allowable data inputs Methods used to calculate the distance between records Presentation of the resulting segments for analysis.
Example of Database Segmentation Example of Database Segmentation using a Visualizationusing a Visualization
Link AnalysisLink Analysis
Aims to establish links (associations) between records, or sets of records, in a database.
There are three specializations– Associations discovery– Sequential pattern discovery– Similar time sequence discovery
Applications include product affinity analysis, direct marketing, and stock price movement.
Link Analysis - Associations DiscoveryLink Analysis - Associations Discovery
Finds items that imply the presence of other items in the same event.
Affinities between items are represented by association rules. – e.g. ‘When customer rents property for more than 2 years and
is more than 25 years old, in 40% of cases, customer will buy a property. Association happens in 35% of all customers who rent properties’.
Link Analysis - Sequential Pattern Link Analysis - Sequential Pattern DiscoveryDiscovery
Finds patterns between events such that the presence of one set of items is followed by another set of items in a database of events over a period of time.
– e.g. Used to understand long term customer buying behaviour.
Link Analysis - Similar Time Sequence Link Analysis - Similar Time Sequence DiscoveryDiscovery
Finds links between two sets of data that are time-dependent, and is based on the degree of similarity between the patterns that both time series demonstrate. – e.g. Within three months of buying property, new home
owners will purchase goods such as cookers, freezers, and washing machines.
Deviation DetectionDeviation Detection
Relatively new operation in terms of commercially available data mining tools.
Often a source of true discovery because it identifies outliers, which express deviation from some previously known expectation and norm.
Deviation DetectionDeviation Detection
Can be performed using statistics and visualization techniques or as a by-product of data mining.
Applications include fraud detection in the use of credit cards and insurance claims, quality control, and defects tracing.
A Summary: Data-Driven TechniquesA Summary: Data-Driven Techniques Data Visualization
Decision Trees
Clustering
Factor Analysis
Neural Network
Association Rules
Rule Induction
* Based on Sakhr Youness’s book “ Professional Data Warehousing with SQL Server 7.0 and OLAP Services
Data VisualizationData VisualizationA pie chart showing the sales of a product by region issometimes much more effective than presenting the samedata in a text or tabular form.
39%
9 %11 %
20 %
21 %
Northeast
East
West
South
North
Decision TreeDecision Tree
Cluster AnalysisCluster Analysis
Have Children
Married
Last car isA used one
Own car
First segment (high income>8,000)
Second Segment (8000>middle income >3000)
Third Segment (low income < 3000)
Factor AnalysisFactor Analysis
Unlike cluster analysis, factor analysis builds a model from data. The technique finds underlying factors, also called “latent variables” and provides models for these factors based on variables in the data. For ex., a software company is considering a survey to find out the nine most perceived attributes of one of their products. They might categorize these products to categories such as service for technical support, availability for training and a help system.
Factor analysis is used for grouping together products based on a similarity of buying patterns so that vendors may bundle several products as one to sell them together at a lower price than their added individual prices..
Neural NetworksNeural Networks
Association RulesAssociation Rules
Association models are models that examine the extent to which values of one field depend on, or are produced by, values of another field. These models are often referred to as Market Basket Analysis when they are applied to retail industries to study the buying patterns of these customers, especially in grocery and retail stores that issue their own credit cards. Charging against these cards gives the store the chance to associate the purchases of customers with their identities, which allows them to study associations among other things.
Rules InductionRules Induction
This is a powerful technique that involves a large number of rules using a set of “if..then” statements in the pursuit of all possible patterns in the dataset. For ex., if the customer is a male then, if he is between 30 and 40 years of ages, and his income is less than $50,000 and more than $20,000, he is likely to be driving a car that was bought as new.
A Summary: Theory-Driven A Summary: Theory-Driven TechniquesTechniques
Correlations
T-Tests
Analysis of Variables
Linear Regression
Logistic Regression
Discriminate Analysis
Forecasting Methods
Validating & Picking the Validating & Picking the ModelModel
Validating any model that comes out of a data mining tool is going to be the
most important thing that you can do. The validation required for data
mining is that after you build the model on some historical data, you apply
the model to similar historical data from which the model was not built.
Because the data is historical, you already know the outcome so that the
accuracy of the predictive model can be measured.
One of the most important things that needs to be done when you are
building a predictive model is to make sure that you have picked up the
essential patterns in the data that will hold true the next time you apply
your model.
Three Additional Ways in Which Three Additional Ways in Which Data mining Supports CRM Data mining Supports CRM
Initiatives.Initiatives.1. Database marketing
2. Customer acquisition
3. Campaign optimization
Database MarketingDatabase Marketing
Data mining helps database marketers develop campaigns that are closer to the targeted needs, desires, and attitudes of their customers. If the necessary information resides in a database, data mining can model a wide range of customer activities. The key objective is to identify patterns that are relevant to current business problems. For example, data mining can help answer questions such as "Which customers are most likely to cancel their cable TV service?" and "What is the probability that a customer will spend over $120 from a given store?" Answering these types of questions can boost customer retention and campaign response rates, which ultimately increases sales and returns on investment.
Database MarketingDatabase Marketing
Database marketing software enables companies to send customers and prospective customers timely and relevant messages and value propositions. Modern campaign management software also monitors and manages customer communications on multiple channels including direct mail, telemarketing, e-mail, the Internet, point of sale, and customer service. Furthermore, this software can be used to automate and unify diverse marketing campaigns at their various stages of planning, execution, assessment, and refinement. The software can also launch campaigns in response to specific customer behaviors, such as the opening of a new account.
Database MarketingDatabase Marketing Generally, better business results are obtained when data mining and
campaign management work closely together. For example, campaign management software can apply the data-mining model's scores to sharpen the definition of targeted customers, thereby raising response rates and campaign effectiveness. Furthermore, data mining may help to resolve the problems that traditional campaign management processes and software typically do not adequately address, such as scheduling, resource assignment, and so forth. Although finding patterns in data is useful, data mining's main contribution is providing relevant information that enables better decision making. In other words, it is a tool that can be used along with other tools (e.g., knowledge, experience, creativity, judgment, etc.) to obtain better results. A data-mining system manages the technical details, thus enabling decision makers to focus on critical business questions such as "Which current customers are likely to be interested in our new product?" and "Which market segment is best for the launch of our new product?"
Customer AcquisitionCustomer Acquisition
The growth strategy of businesses depends heavily on acquiring new customers, which may require finding people who have been unaware of various products and services, who have just entered specific product categories (for example, new parents and the diaper category), or who have purchased from competitors. Although experienced marketers often can select the right set of demographic criteria, the process increases in difficulty with the volume, pattern complexity, and granularity of customer data. Highlighting the challenges of customer segmentation has resulted in an explosive growth in consumer databases. Data mining offers multiple segmentation solutions that could increase the response rate for a customer acquisition campaign. Marketers need to use creativity and experience to tailor new and interesting offers for customers identified through data-mining initiatives.
Campaign OptimizationCampaign Optimization
Many marketing organizations have a variety of methods to interact with current and prospective customers. The process of optimizing a marketing campaign establishes a mapping between the organization's set of offers and a given set of customers that satisfies the campaign's characteristics and constraints, defines the marketing channels to be used, and specifies the relevant time parameters. Data mining can elevate the effectiveness of campaign optimization processes by modeling customers' channel-specific responses to marketing offers.
Topic 6: Topic 6: Classical Techniques: Classical Techniques: Statistics, Neighborhoods, and Statistics, Neighborhoods, and
ClusteringClusteringStatistics can help to answer several important questions about the
data :
What patterns are there in my database?
What is the chance that an event will occur?
What patterns are significant?
What is a high-level summary of the data that gives me some idea of what is contained in my database?
StatisticsStatistics --Histogram --Histogram
The first step in understanding statistics is to understand how the
data is collected into a higher-level form—one of the most notable
Ways of doing this is with the histogram.
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
East
West
North
# of customers orAmount of sales
HistogramHistogram
Number of customers
Ages1 11 21 31 41 51 61 71 81
500
1000
1500
2000
2500
3000
Linear Regression Is Similar to the Task of FindingLinear Regression Is Similar to the Task of Findingthe Line that Minimizes the Total Distance to a Set the Line that Minimizes the Total Distance to a Set
of Data.of Data.
Predictor (Consumer annual income)
Prediction(Average Consumer bank balance)
Linear RegressionLinear Regression
The predictive model is the line shown in the previous chart. The line
will take a given value for a predictor and map it into a given value
for a prediction. The actual equation would look something like
Prediction = a + b* predictor. This is just the equation for a line Y =
A + b*X. As an example for a bank, the predicted average consumer
bank balance might equal to $1,000 + 0.01 * customer’s annual
income.
Linear RegressionLinear Regression
Linear regression attempts to fit a straight line through a plot of the data, such that the line is the best representation of the average of all observations at that point in the plot.
Problem is that the technique only works well with linear data and is sensitive to the presence of outliers (i.e.., data values, which do not conform to the expected norm).
Linear RegressionLinear Regression
Although non-linear regression avoids the main problems of linear regression, still not flexible enough to handle all possible shapes of the data plot.
Statistical measurements are fine for building linear models that describe predictable data points, however, most data is not linear
in nature.
Linear RegressionLinear Regression
Data mining requires statistical methods that can accommodate non-linearity, outliers, and non-numeric data.
Applications of value prediction include credit card fraud detection or target mailing list identification.
The Nearest Neighbor Prediction
One of the classic areas that nearest neighbor has been used for
prediction has been in text retrieval. The end user defines a document
(for ex., a Wall Street Journal) to be retrieved, then the nearest
neighbor characteristics with these documents that have been
marked are more likely to be retrieved.
Another good example is that the supermarkets tend to put similar
produces in the same area, for ex., an apple closer to an orange than
to tomato. Thus, if you know the predictive value of one of the
objects, you can predict it for the nearest neighbors.
Data Clustering
Clustering analysis is an important means of processing multimedia
data. It is basically the organization of a collection of patterns into
clusters of similar objects. Patterns within valid cluster are more
similar to each other than they are to a pattern in a different cluster.
Data Clustering
Clustering can allow us to carry out the following activities
that can help in query processing:
Representing patterns in the data so that we can reduce the size of the media;
Defining a way of measuring the proximity of different patterns in the data so that we can find the instances that match our example.
Clustering or grouping the data in preparation for matching; Data abstraction, particularly of features that we can store as
metadata; Assessing the output by estimating how good the selection is.
Clustering and Nearest NeighborClustering and Nearest Neighbor
A simple example of clustering would be the clustering that most
people perform when they do the laundry- grouping the permanent
press, dry cleaning, whites, and brightly colored clothes is important
because they have similar characteristics.
A simple example of the nearest neighbor prediction algorithm Is
when you look at the people in your neighborhood. You may notice
that, in general, you all have somewhat similar income.
Statistical Analysis of Actual Sales (dollars Statistical Analysis of Actual Sales (dollars and quantities) relative to these Signage and quantities) relative to these Signage
Variables-a predictiveVariables-a predictive modelingmodeling example. example.
Content Frequency Depth Focus Depth Scale Length Location
Statistical Analysis : Correlation, Regression, Experiment Design,
Optimization. Now it goes into real time analysis.
SignageSignage
SignageSignage
Topic 7: Topic 7: Next Generation Techniques: Next Generation Techniques: Decision Trees, Networks, and RulesDecision Trees, Networks, and Rules
Customer renting property> 2 years
Rent property
Customer age>45
No Yes
No Yes
Rent property
Buy property
A: Decision TreeA: Decision Tree
A: Decision TreeA: Decision Tree
CART and CHAIDCART and CHAID
CART, which stands for Classification and Regression Trees, is a data
exploration and prediction algorithm developed by Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone. It is nicelydetailed in their 1984 book, Classification and Regression Trees (Breiman, Friedman, Olshen, and Stone, 1984. These researchers fromStandard University and the University of California at Berkeley Showed how this new algorithm could be used on a variety of different problems from the detection of chlorine from the data contained in a mass spectrum. One of the great advantages of CARTis that the algorithm has the validation of the model and the
discovery of the optimally general model built deeply into the algorithm. Another popular decision tree technology is CHARD (Chi-SquareAutomatic Interaction Detector). CHARD is similar to CART in that it builds a decision tree, but it differs in the way that it chooses its splits.
B: Neural NetworksB: Neural Networks
A neural network is loosely based on the way some people believe
That the human brain is organized and how it learns. There are two
Main structures of consequence in the neural networks:
The node, which loosely corresponds to the neuron in the human brain
The link, which loosely corresponds to the connections between neutrons (axons, dendrites, and synapses) in the human brain.
Neural NetworksNeural Networks
‘When customer rents property for more than 2 years and is more than 25 years old, in 40% of cases, customer will buy a property. Association happens in 35% of all customers who rent properties’.
Example of Classification using Example of Classification using Neural InductionNeural Induction
Each processing unit (circle) in one layer is connected to each processing unit in the next layer by a weighted value, expressing the strength of the relationship. The network attempts to mirror the way the human brain works in recognizing patterns by arithmetically combining all the variables with a given data point.
In this way, it is possible to develop nonlinear predictive models that ‘learn’ by studying combinations of variables and how different combinations of variables affect different data sets.
How Does a Neural Induction How Does a Neural Induction Make a prediction?Make a prediction?
The value age of 47 is normalized to fall between 0.0 and 1.0, it has the value of 0.47, and the income is normalized to the value of 0.65. This simplified neural network makes the prediction of no default for a 47-year old making $65,000. The links are weighted at 0.7 and 0.1, and the resulting value, after multiplying the node values by the link weights, is 0.39.
0.47
0.65
0.39
Age
Income
Weighted = 0.7
Weighted = 0.1
default
0.47(0.7) + 0.65(0.1) = 0.39
C: Rule InductionC: Rule Induction
This is a powerful technique that involves a large number of rules using a set of “if..then” statements in the pursuit of all possible patterns in the dataset. For ex., if the customer is a male then, if he is between 30 and 40 years of ages, and his income is less than $50,000 and more than $20,000, he is likely to be driving a car that was bought as new.
What Is A Rule?What Is A Rule?
If breakfast cereal purchased, the 85% 20%
milk is purchased.
If bread purchased, then Swiss choose 15% 6%
will be purchased.
If 42 years old and purchased pretzels 95% 0.01%
and dry roasted peanuts, then beer will
be purchased.
Rule Accuracy Coverage
Tools and technologies will be applied to real business problems across a variety of industries. They are:
Customer Profitability – provides a blueprint for how to define and use customer profitability as the bedrock for your CRM processes.
Customer Acquisition – shows how to use data mining to acquire new customers in the most profitable way possible.
Customer Cross-selling – details how the technology architecture can be used to increase the value of existing customers by applying more to them.
Customer Retention – uses a case study from the telecommunications industry to show how to execute successful CRM systems to retain your profitable customers.
Customer Segmentation – provides the business methodology of how to segment and manage your customers in a consistent and repeatable way across the enterprise.
Topic 8: Topic 8: CRM -The Business CRM -The Business PerspectivePerspective
The Business-Centric View of Data The Business-Centric View of Data Mining ProcessMining Process
ROI Definition
Display
ROI
Predicted ROI
BusinessProblem
PredictiveModel
Data Definition
Data
Define Value
Define Value
Understand
Data Mining
Application
Customer ProfitabilityCustomer Profitability
Customer profitability is the bedrock of data mining. Data mining
earns its keep by helping you to understand and improve Customer
Profitability. How does the organization define what a profitable
customer is versus an unprofitable customer? Keeping a customer
loyal can have profound effects on per-customer profitability. The
compounding effect of customer loyalty on customer profitability also
increases because sales costs are lower and revenue generally has
increased. Data Mining can be used to predict customer profitability,
Under a variety of different marketing campaigns.
CurrentValue
LifetimeValue
PotentialValue
PotentialLifetimeValue
CustomerServiceLevel
BestServiceLevel
1 High High High High Gold Gold
2 High Low High High Gold Gold
3 High Low High Low Gold Bronze
4 Low Low Low High Bronze Gold
5 Low Low High High Bronze Gold
6 Low Low Low Low Bronze Bronze
Segment
A Customer Value Matrix Showing Recommended Service Level
A Customer Value MatrixThis should be one of the first things that we should do
with data mining.
Segment 1 is our best customers. They will remain your best customers through their lives and their current value matches their potential.
Segment 2 is similar, except that they are likely to have low lifetime value, despite their high value today, probably because they are not loyal and likely switch to a competitor at some time in their customer life.
Segments 4 and 5 represent customers who, with the right care and service, can be transitioned to high-value customers, either short-term or long-term .
Segments 6 represents your low-value customers that you will treat with some of your least expensive services.
Customer AcquisitionCustomer Acquisition
The traditional approach to customer acquisition involved a marketing manager developing a combination of mass marketing (magazine advertisements, billboards, etc.) and direct marketing ( Telemarketing, mail, etc.) campaigns based on their knowledge of theParticular customer base that was being targeted.
A marketing manager selects the demographics (Age, Gender, interest in particular subjects, etc.) and then works with a data vendor (sometimes known as a service bureau) to obtain Lists of customers who meet those characteristics.
Although a marketer with a wealth pf experience can often choose relevant demographic selection criteria, the process becomes more difficult as the amount of data increases.
Data Mining can help this process.
Defining Some Key Customer Defining Some Key Customer Acquisition ConceptsAcquisition Concepts
The responses that come in as a result of a marketing campaign are called “response behaviors”. Binary response behaviors (either a yes or no) are the simplest kind of response.
Beyond binary response behaviors are a type of categorical response behaviors which allows for multiple behaviors to be defined. The rules that define the behaviors are based on the kind of business you are involved in.
There are usually several different kinds of positive response behaviors thatcan be associated with an acquisition marketing campaign. They are:
Customer inquiry; Purchase of the offered product or products; Purchase of a product different from the one offered.
Response Analysis Broken Down By Behaviors
Behavior Measures 12/1/05 12/5/05 12/7/05 12/9/05 Total
Inquiry # of Responses 1,556 1,340 328 352 3,576
Purchase A
# of Responses 210 599 128 167 1.104
Purchase B
# of Responses 739 476 164 97 1,476
Purchase C
# of Responses 639 647 113 105 1,504
Cross-SellingCross-Selling
Cross-selling is the process by which you offer your existing customers new
products and services. Customers who purchase baby diapers might also be
interested in hearing about your other baby products.
One form of cross-selling, sometimes called “up selling”, takes place when the
new offer is related to existing purchases by the customer. For., ex., an up-
sell opportunity might exist for a telephone company to market a premium
long-distance service to existing long-distance customers who currently have
the standard service.
How Cross-Selling WorksHow Cross-Selling WorksAssume that you are a marketing manager for a mid-size bank. You have the following products available for your customers:
Value checking account Standard checking account Gold credit card Platinum credit card Primary mortgage Secondary mortgage
Of these products, you’re responsible for marketing the mortgage products to your Customers. Your goal is to find out which customers might be interested in a mortgage offering at least 60 days before they would apply for the loan. It is important that any predictions are made with sufficient lead time (in this case, two months), so that any Interactions with the customers take place before they are committed to a relationship with your competition.
How Cross-Selling WorksHow Cross-Selling Works
You have already done some thinking about your customers and their motivations in this area and came up with several scenarios, which you presented to your boss when pitching this new campaign:
Customer preparing to buy a new home. These customers might be building up cash reserves in their checking and/or savings account in order to put together a down payment.
Customer preparing to refinance an existing home. These customers might be paying off credit card debt (thus making them more acceptable from a risk point of view), and hold a mortgage whose interest rate is higher than the current interest rate.
Customer preparing to add a second mortgage. These customers might have increasing credit card debt, an on-time payment history for their credit cards and existing mortgage (which means that they are a good risk), and enough equity in their house to cover the outstanding credit card balance.
Data Mining Process for Cross-SellingData Mining Process for Cross-Selling
The actual data mining process contains three distinct steps when doing cross-selling process:
Modeling of individual behaviors Scoring data with predictive models Optimization of the scoring matrices
Model: A description that adequately explains and predicts relevant data thatbut is generally much smaller than the data itself. For real-world applications, a model can be anything from a mathematical Equation, to a set of rules that describes customer segments, to the computer representation of a complex neural network architecture, which translates to several sets of mathematical equations.
Predictive model: A model created or used to perform prediction. In contrast to models created solely for pattern detection, exploration or general organization of the data.
Customer RetentionCustomer Retention
As industries become more competitive and the cost of acquiring new
customers increases, the value of retaining current customers also increases.
for instance, in the cellular phone industry, it is estimated that the cost of
attracting and signing up a new customer is $300 or more when the costs of
disconnected hardware and sales commissions are included. The cost of
retaining a current customer, however, can be as low as the price of a phone
call or the cost of updating their cellular phone to the latest technology
offering. Although expensive, this is still significantly cheaper than signing
up a wholly new customer.
A Case Study- Cellular Phone IndustryA Case Study- Cellular Phone Industry
Customer churn is the term used in the cellular telephone industry to denote
the movement of cellular telephone customers from one provider to another.
In many industries, this is called customer attrition, but because of the highly
volatile and growing market, and the somewhat limited competition, many
customers churn from one provider to another frequently in search of better
rates or for the perks of signing up with a new provider. Attribute rates in
cellular phone industry hover around 2.2% per month. In other words,
about 27% of a given carrier’s customers are lost each year when the
contracts need to be renewed. Losing these customers can be very expensive
because it costs fom $300 to $600 to acquire a new customer in sales support,
marketing, advertising, and the commissions. Many of these new customers
are less profitable than the ones that were lost.
The Data Mining ModelThe Data Mining Model
Segment Create Apply Churn Description of the customers in the segment
Number size size rate (criteria defining the segment)
29 1061 403 84% Contract type “N” – no contract
Length of service is less than 23 months
28 899 360 65% Contract type “Y” – indicating a 12 month
contract requiring 3 months notice to discontinue
at the end of 12 months
18 902 7564 50% Contract type “Y”
Customer type “R” indicating a residential not
Business
The Data Mining ModelThe Data Mining ModelThe total Customer churn volume expected in each segment is taken as a percentage of the the total expected to churn across the total population (thetotal of all segments), and this is shown on a cumulative basis alongside the cumulative percentage of the base. The analysis shows the following:
5.2 % of the base contains 27.7% of the expected total churn.
10.5 % of the base contains 41.5% of the expected total churn.
19.7 % of the base contains 55.8% of the expected total churn.
Therefore, marketing campaigns targeted at 5.2 % of the base 14,581 subscribers will address 7848 of the likely churners, a lift factor of 5.4. A liftis a number representing the increase in responses from a targeted marketingapplications using a predictive model over the response rate achieved when no model is used.
A Decision Tree for Mobile PhonesA Decision Tree for Mobile Phones
Contract Type = ”D”
Length of Service > 12 months Length of Service > 12
months
No Yes
No Yes
Segment 28 Segment 29
Segment AnalysisSegment Analysis
Segment Propensity Segment Churn Cumulative Cumulative Cumulative % Cumulative %
Number to Churn base volume base Churn of Churn of base
29 82 % 403 337 403 337 1.2 % 0.1 %
28 64 % 360 233 763 571 2.0 % 0.3 %
Customer SegmentationCustomer Segmentation
Segmentation is the act of breaking down a large of customer population into
segments in which those consumers within the segments are similar to each
other, and those that are in different segments are different from each other.
for ex., even the simple act of organizing the customers in your database by
the state they live in is an act of segmentation. Distinguishing between male
and female customers is also an act of segmentation.
Segmentation allows people to differentially treat consumers in different
segmentations. That is why men are advertised to during football games,
women are advertised to during sitcoms.
How Is Data Mining Used for Segmentation- How Is Data Mining Used for Segmentation- Clustering & Decision TreeClustering & Decision Tree
Customer renting property> 2 years
Rent property
Customer age>45
No Yes
No Yes
Rent property
Buy property
How is Data Mining Used for How is Data Mining Used for Segmentation-Clustering?Segmentation-Clustering?
If decision trees are used to create segments, then the data is
guaranteed to the mutually exclusive and collectively exhaustive (no
customer falls into more than one segment and every customer is
guaranteed to be contained in one of the segments).
Topic 9:Topic 9: Deploying Data Mining for Deploying Data Mining for CRMCRM
Define the problem Define the user Select the data Prepare the data Mine the data Deploy the model Take business action Implement Quality Assurance Educate and train users
Define the ProblemDefine the Problem
A successful data mining initiative always starts with
a well-defined project. To insure that the project produces incremental value, include an assessment of the status quo
solution and a review of technology, organization, and business processes.
Many times, data mining systems will be deployed to optimize
existing CRM process. If a CRM system and process does already exist, it should be understood.
Define the UserDefine the UserAfter the problem is defined, it should be possible to define who will
be using the system when it is completed. These could include using
the data mining application itself all the way to supporting, and then
measuring the customer value matrix and the computation of the
ROI (Return on investment) of the existing system.
Building a profile of each user: you should know, for instance, at least
the following information about these users:
Their technical expertise Their use of the system (every day, once a month, or occasionally) The understanding of data mining Their desire for details
Select the DataSelect the Data
This step involves defining your data source . (not every data source and record is required.) The data is usually extracted from
the source system to a separate server.
The three types of customer data that does the following:
Describe who the consumer is. Describe what marketing or sales promotions were made to the
customer. Describe how the consumer reacted to those promotions by
transacting with the company.
The three types of customer dataThe three types of customer data
Who is theCustomer?
What did you doTo the customer?
Descriptive Promotional
Transactional
How did theCustomer react?
Direct mail, email, sales
Purchase, Web hit,Business reply card,survey
Prepare the DataPrepare the Data
This step represents up to 80 percent of the total project effort. For data mining, the data must reside in one flat table (each record has many columns). In addition to being the most time consuming, the step is also the most critical. The resulting models are only as good as the data used to create them.
Prepare the Data- Assessing Prepare the Data- Assessing Levels of Data IntegrityLevels of Data Integrity
This step involves defining your data source . (not every data source and record is required.) The data is usually extracted from
the source system to a separate server.
The three types of customer data that does the following:
Describe who the consumer is. Describe what marketing or sales promotions were made to the
customer. Describe how the consumer reacted to those promotions by
transacting with the company.
Mine the DataMine the Data
Typically the easiest and shortest phase, this step involves applying statistical and AI tools to create mathematical models. Data mining typically occurs on a server separate from the data
warehousing and other corporate systems.
Deploy the ModelDeploy the Model
Model deployment is the process of implementing the mathematical models into operational systems to improve business results.
Take Business ActionTake Business Action
Use the deployed model to achieve improved results to the business problem identified at the beginning of the process.
Steps to Implement Data MiningSteps to Implement Data MiningDiscovery (patterns, relations
Associations, etc.)Prior Knowledge
Information Model
Deployment
Validation
Implement Quality AssuranceImplement Quality Assurance
All throughout the launch process, quality assurance should be of
highest priority. A good first step is to specifically assign the QA role
and only the QA role to one of the team members on the data mining
projects. This QA role will exist to not only validate the success of the
data mining model, but also to double-check the work from other
parties.
Educate and Train UsersEducate and Train Users
Make sure to work with the users to educate them about what models
and metrics are, and how they can access and visualize the results
of the system. Pay particular attention to the following:
Description of the consumer base and the data that is available. How the data mining results are integrated into the customer
relationship management system. The way the metrics are calculated for understanding the results
of the data mining system. For instances, how is customer profit or customer response calculated.
Topic 10: Data Quality-Indicators of Topic 10: Data Quality-Indicators of Quality DataQuality Data
1. The data is accurate – This means that a client’s name is spelled correctly.
2. The data is stored according to data type – for ex., character, integer.
3. The data has integrity – Referential integrity rules will be properly defined in the logical data model and implemented in the physical data model. The data will not be accidentally destroyed and altered.
4. The data is consistent – The form and content of the data should be consistent. This allows for data to be integrated and shared by multiple departments across multiple applications and multiple platforms.
5. The databases are well designed – A well-designed database will perform satisfactorily for its intended applications.
Data Quality-Indicators of Quality Data Quality-Indicators of Quality DataData
6. The data is not redundant – In actual practice, no organization has ever totally eliminated redundant data. For certain performances, data is purposely maintained in more than one place.
7. The data follows the business rules – for ex., a loan balance may never be negative.
8. The data corresponds to established domains – Referential integrity rules will be properly defined in the logical data model and implemented in the physical data model. The data will not be accidentally destroyed and altered.
9. The data is timely – Timeliness is subjective and can only be determined by the users of the data.
10. The databases are well understood – It does no good to have accurate and timely data if the users do not know what they mean.
Data Quality-Indicators of Quality Data Quality-Indicators of Quality DataData
11. The data is integrated – Database integration requires the knowledge of the characteristics of the data, what the data means, and where the data resides. The information would be kept in the dictionary/repository.
12. The data satisfies the needs of the business – The data has value to the enterprise.13. The user is satisfied with the quality of the data and the
information derived from that data.14. The data is complete– All the line items for an invoice have been
captured so that the bill states the full amount that is owned.15. There are no duplicate records. 16. There is data anomalies– A data anomaly occurs when the data
field defined for one purpose is used for another.
Types of Source System ExtractsTypes of Source System Extracts
In order to update the data warehouse, it is necessary first to identify
what data is required from the operational system in order to capture any new or changed data during a given time variant period interval. Let us begin with identifying the triggers in the operational environment that determine when an extract should be taken. These triggers can be categorized as:
Point-in-time snapshots Significant business events Delta data
Point-in-Time SnapshotsPoint-in-Time Snapshots
The simplest way to preserve history is to take a point-in-time snapshot of the operational data. Snapshots are typically scheduled for very specific points in time, such as the end of a calendar week or month, and they preserve the historical relationship between different data elements and subject areas. A snapshot is a very effective way to determine delta between different points in time. For example, if monthly point-in-time snapshots of customer data are taken from an operational source and loaded into the data warehouse, a determination of the changes in the customer base can
easily be determined from month to month, or any time period.
Significant Business EventsSignificant Business EventsSignificant business event (SBE) are actually an event-oriented variation of point-in-time snapshots. Unlike a point-in-time, which is an easily predetermined date such as the end of a calendar week or month, and SBE cannot be accurately predicted; it must occur for the snapshot to then take place. For ex., an SBE might be the successful completion of a billing cycle. There may be several billing cycles during a calendar month, each with a predetermined cutoff date. Nevertheless, the successful completion of the billing cycle is dependent on a number of factors and can not be predetermined; it may take anywhere from one to three days to complete the cycle. The actual completion event is determined by a set of success criteria, usually judged by the person or group responsible for quality assurance or other similar business function. Once the cycle has been deemed successful, bills can be mailed to the customers. It is at this point, this significant business event, that a snapshot of the billing data and any related or artifact data can take place.
Delta DataDelta Data
Delta data is also known as “new and changed data”; that is, data that represents changes (delta) from one point in time to the next. Delta can be captured in a number of ways:
1. Operational events-An operational event creates delta data
by passing a record of the event either to a holding file or to
a data warehouse updating process. This will result in an
accurate record of the changes that take place in the
operational system at a limited cost in terms of processing.
Delta DataDelta Data
2. Changed Data Capture-It refers to the process of reading a database or other operational change logs and extracting the appropriate changed data for updating the data warehouse.
3. Date Last Changed-A very efficient means of extracting new and changed data from operational sources is to interrogate a date last modified in the operational tables or files.
Topic 11:Topic 11: Next Generation of Next Generation of Information Mining and Knowledge Information Mining and Knowledge
Discovery for Effective CRMDiscovery for Effective CRM
In the current and emerging competitive and highly dynamic business
environment, only the most competitive companies will achieve
sustained market success. In order to capitalize on business
opportunities, these organization will distinguish themselves by the
capacity to leverage information about their marketplace, customers,
and operations. A central part of this strategy for long-term
sustaining success will be an active information repository- an
advanced data warehouse, in which information from various
applications or parts of the business is coalesced and understood.
Information MiningInformation Mining
The shortest path from complex data to knowledge discovery is
Information mining instead of data mining to reflect the rich variety
Of forms that information required for business intelligence can take.
Information mining implies using powerful and sophisticated tools to
Do the following:
Uncover associations, patterns, and trends
Detect deviations
Group and classify information
Develop predictive models
Information MiningInformation Mining
From a technical perspective, the real keys to successful information
Mining are its algorithms: complex mathematical processes that
Compare and correlate data. Algorithms enable an information
mining application to determine who the best customers for the
Business are or what they like to buy. They can also determine at
what time of day, in what combinations, or how an organization can
Optimize inventory, pricing, and merchandising in order to retain
These customers and cause them to buy more, at increased profit
Margins. A large volume of information is stored in anon-numeric
Forms: documents, images and video files.
Text Mining and Knowledge Text Mining and Knowledge ManagementManagement
Text Mining is a subset of information mining technology that, in turn, is a
component of a more general category of Knowledge Management (KM).
Knowledge, in this case, refers to the collective expertise, experiences, know-
how, and wisdom of an organization. In a business world, knowledge is
represented not only by the structured data found in traditional database,
but in a wide variety of unstructured sources such as word documents,
memos and letters, e-mail messages, news feeds, Web pages, and so forth.
Text Mining and Knowledge Text Mining and Knowledge ManagementManagement
Unlike data mining, text mining works with information stored in an
Unstructured collection of text documents. Specifically, online text
Mining refers to the process of searching through unstructured data
On the internet and deriving some meaning from it. Text mining goes
beyond applying statistical models to data files; in fact, text mining
Uncovers relationships in a text collection, and leverages the
creativity of the knowledge work to explore these relationships and
Discover new knowledge.
Text Mining TechnologiesText Mining Technologies
There are two key key technologies that make online text mining possible:
Internet Searching - It has been around for a quite few years. Yahoo, Alta Vista, and Excite are three of the earliest. Search engines (and discovery services) operate by indexing the context in a particular Web site and allows users to search the indexes. Although useful, first generations of these tools often were wrong because they did nit correctly index the content they retrieved. Advances in text mining applied to the internet searching resulted in online text mining, representing the new generation of Internet search tools. With these products, users can gain more relevant information by processing smaller amount of links, pages and indexes.
Text Mining TechnologiesText Mining Technologies
Text Analysis - It has been around longer than Internet searching. Indeed, scientists have been trying to make computers understand natural languages for decades; text analysis is an integral part of these efforts. The automatic analysis of text information can be used for several different general purposes:
1. To provide an overview of the contents of a large document collection, for ex., finding significant clusters of documents in a customer feedback collection could indicate where a company’s products and services need improvement.
2. To identify hidden structures between groups of objects; this may help to organize an intranet site so that related documents are all connected by hyperlinks.
Text Mining TechnologiesText Mining Technologies
3. To increase the efficiency and effectiveness of a search process to find similar or related information; for ex., to search articles from a news service and discover all unique documents that contain hints on possible trends or technologies that have so far not been mentioned in their articles.
4. To detect duplicate documents in an article.
Text Mining Technologies-Text Mining Technologies-ApplicationsApplications
1. E-mail management. A popular use of text analysis is for messae routing in which the computer “reads” the message to decide who should deal with it. (Spam control is another good example)
2. Document Management. By mining the different documents for meaning as they are put into a document repository, a company can establish a detailed index that allows the location of relevant documents at any time.
3. Automated help desk. Some companies use text mining to respond to customer inquiries. Customers’ letters and e-mails are processed by a text mining applications.
4. Market research. A market researcher can use online text mining to gather statistics on the occurrences of certain words,c phases, concepts, or themes on the World Wide Web. This information can be useful for establishing market demographics and demand curves.
5. Business intelligence gathering. This is the most advanced use of text mining. (See next slide)
BloggerBlogger
Blogger is one of the most popular online blogging tool, works with
any browser, and is free, well designed and easy to use. Millions of
people are changing their information acquisition habits, and the web
Log, or “blog” has become a popular source.
Title-Publishing a blog with blogger/by Elizabeth Castro,
Berkeley, Calif, Peachpit, 2005 Title- Blog: Understaning the information that’s changing your
world/ Hugh Howitt, Nashiville, Tenn, Nelson Books, c2005 Webblogs (isbn 0321321235)
Semantic Networks and Other Semantic Networks and Other TechniquesTechniques
A key element of building an advanced system for textual information. analysis, summarization, and search is the development of a Semantic Network for the investigated text. A Semantic Network is a set of the mostsignificant concepts—words and word combinations– derived from the analytical texts, along with the semantic relationships between these concepts in the text. A semantic network provides a concise and very accurate summary of the analyzed text.
Other techniques, For ex., Cambio uses absolute positioning, pattern recognition, fixed and floating tags. The SemioMap software extracts all relevant phrases from the text collection. It builds a lexical networkof co-occurrences by grouping related phases and enhancing the most salient features of these groupings.
Text Mining ProductsText Mining ProductsCompany Products
Aptex Software, Inc. SelectResponse
Autonomy Agentware
Data Junction Cambio
Excalibur Technologies Corp. RetrievalWare
Fulcrum Technologies Inc. DOCSFulcrum SearchSearver
IBM Corp. Intelligent Minor for Text
InsightSoft-M Cross-reader
Intercon System, Ltd. Dataset
Megaputer Inc. Text Analyst
Semio Corp. SemioMap
Verity, Inc. KeyView, Intranet Spider
Text Mining Products-An ExampleText Mining Products-An Example
Autonomy (Agentware) offers three kinds of products relating to online text mining:
Knowledge Server – Provides users with a fully automated and precise means of categorizing, cross-referencing, and presenting information.
Knowledge Update – Monitors specified Internet and intranet sites, news feeds, and internal repositories of documents, and creates a personalized report on their contents.
Knowledge Builder – A toolkit that allows companies to integrate Agentware capabilities into their own systems.
Topic 12:Topic 12: CRM in the e=Business CRM in the e=Business WorldWorld
As e-business continues to mature and affect radical changes throughout all
aspects of the businesses, the focus of new e-business-enabled application
software will shift away from narrowly defined commerce platforms toward
a broader vision of managing customer relationships.
A new model that Forrester Research calls eRelationship Management (eRM)
is defined as follows:
“A Web-centric approach to synchronizing customer relationships across
communication channels, business functions, and audiences”
CRM in the e=Business WorldCRM in the e=Business World
To implement this new e-business CRM model, companies should do the
following:
Create a dynamic customer context that can address every customer interaction that is different from a view of the customer constructed from data contained in the applications. This can be achieved by collecting and organizing customer data, calculating high-level matrices for each customer (I.e., customer profitability, satisfaction, and churn potential), and assembling and delivering dynamic context to customer touch points.
Generate consistent, custom responses by delivering a consolidated rules engine for routing, workflow, personalization, smart navigation, and consistent treatment of customers
Build and maintain a Content Directory to point to company, products, and business partner content; and give to employees, business partners, and customers.
top related