ppt - data warehouses and function points - ifpug proceedings/ifpug-2004... · 3 5 terms • source...
TRANSCRIPT
1
1
���������������� �
�������������
����������
2
Agenda
• Terminology• Structure• Counting Transactions• Counting Data Stores
2
3
What is hard?
• New set of terms• Similar data in different places• Getting ‘sucked’ into the physical world
4
DW Structure
3
5
Terms
• Source system• Staging Area• eDW• Data Mart• ETL
6
Boundaries rules (IFPUG CPM 4.2)
• The boundary is determined based on the user’s view. The focus is on what the user can understand and describe.
• The boundary between related applications is based on separate functional areas as seen by the user, not on technical considerations.
• The initial boundary already established for the application or applications being modified is not influenced by the counting scope.
4
7
Boundaries rules
• User view is key• IFPUG manual also lists hints to validate
against during the boundary discussion.– Look at how groups of data are being
maintained– Look at associated measurement data, such as
effort, cost, and defects. The boundaries for function points and other measurement data should be the same.
8
Boundary example
5
9
FPA
• Elementary Process importance• Processing logic
10
ETL and transactions
• Extract
• Transformation
• Load
6
11
Transaction counting
A
12
Transactions
• ETL Map– Overview information– Source/Target– Required transformations– High level process overview– Process description– Source to Target Field Matrix
7
13
High level process overview
INV 8.17exp_Determine_Whether_Update_Or_Insert
Lookups
INV 8.1sq_Transform_STG_Invc_Detail_To_EDW_Invoice_Li
ne_Item
Error_Log
INV 8.2exp_Validation_and_Get_Session_R
un_ID
INV 8.3lkp_Session_Run_
ID
disconnectedlookup
EDW_Invoice_Line_Item
INV 8.7lkp_EDW_Except_Prc_Rsn_Code_F
orException_Price_
Reason_C
INV 8.10lkp_EDW_Shipping_Platform_Code_For_Shipg_Pltfrm_
C
INV 8.6lkp_EDW_Free_Prod_Code_For_Free_Product_Type_
C
INV 8.9lkp_Spcfd_Promo_Code_For_Specified_Promotion_C
INV 8.11lkp_Invoice_Adj_Type_Code_For_Inv
_Adj_Typ_N
INV 8.4lkp_EDW_Invoice_Header_For_Inv_N_Inv_Adj_N_Prcs
_Ts
INV 8.5lkp_EDW_Packaged_Good_ID_For_Mand_ID_N_Upc_
C_Upc_Sufx_C
INV 8.8lkp_EDW_Invoice_Reason_Code_Or_Line_Maint_Rsn_Code_For_Rsn_
C
INV 8.13sq_Invc_Detail_Err
ors
INV 8.14jnr_STG_And_
_Invc_Detail_Errors
INV 8.15fil_No_Error_Or_Non_Critical_Error_For_EDW_Invoice
_Line_Item
INV 8.16lkp_EDW_Invoice
_Line_Item
INV 8.19fil_Failed_Validation_For_EDW_Invoi
ce_Line_Item
INV 8.20nrm_Failed_Validation_For_EDW_Invoice_Line_Item
INV 8.22fil_Remove_Zero_Error_Msg_Id_For_EDW_Invoice_Li
ne_Item
INV 8.18upd_EDW_Invoice
_Line_Item
INV 8.21seq_Err_Seq_ID_For_Invoice_Line_
Item
INV 8.12lkp_EDW_Accounting_Time_For_Fis
cal_Period
INV8.23exp_Choose_Correct_Record_To
_Use
Invc_Detail_Errors
EDW_Invoice_Header
INV 8.24agg_Find_Unique_Invoice_Header_I
Ds
INV 8.25upd_Header_Updt
_Date
Possible data stores
Possible two E.P.
14
Target Table Target Column Datatype Source Table Source Column Datatype Data Issues/QualityEDW_Invoice_Line_Item Invoice_Line_Item_ID Number(1
0)Sequence Generator EDW Primary key
EDW_Invoice_Line_Item Invoice_N Varchar2(7)
STG_Invc_Detail Inv_N Varchar2(7)
Source/STG Primary key
EDW_Invoice_Line_Item Invoice_Adjustment_N Number(5)
STG_Invc_Detail Inv_Adj_N Number(5)
Source/STG Primary key
EDW_Invoice_Line_Item Invoice_Line_N Number(5)
STG_Invc_Detail Inv_Lin_N Number(5)
Source/STG Primary key
EDW_Invoice_Line_Item Process_Ts Date STG_Invc_Detail Prcs_Ts Date Source/STG Primary keyEDW_Invoice_Line_Item Invoice_Header_ID Number(1
0)EDW_Invoice_Header
Invoice_Header_ID Number(10)
EDW_Invoice_Line_Item Packaged_Good_ID Number(10)
EDW_Packaged_Good
Packaged_Good_ID Number(10)Number(2)
Number(4)
STG_Invc_Detail
Ord_N, Entry_D
EDW_Order_Header
Shipment_Id
EDW_Shipment
Load_Ord_N
STG_Invc_Detail Ord_N, Entry_D
EDW_Order_Header
Order_Type_C
STG_Invc_Detail Ord_N, Entry_D
EDW_Invoice_Line_Item
Fiscal_Period Number(6)
STG_Invc_Detail, EDW_Month
Use Actg_Mth_N + Actg_Yr_N to look up Fiscal_Period
EDW_Invoice_Line_Item
Load_Order_N Varchar2(7)
Varchar2(7
Look up shipment_id on EDW_Order_Header (by ord_n, entry_d ). Look up load_ord_n on EDW_Shipment (by shipment_id)
EDW_Invoice_Line_Item Order_Type_C Varchar2(2)
Varchar2(2)
Look up order_type_c on EDW_Order_Header (by ord_n, entry_d)
EDW_Invoice_Line_Item Submit_Method_C Varchar2(2)
Varchar2(2)
Look up submit_method_c on EDW_Order_Header (by ord_n, entry_d)
Derived field
Invoice ETL mapSource to Target Field Matrix
8
15
DET identificationTarget Table Target Column Datatype Source Table Source Column Datatype
EDW_Invoice_Line_Item Invoice_Line_Item_ID Number(10)
Sequence Generator
EDW_Invoice_Line_Item Invoice_N Varchar2(7)
STG_Invc_Detail Inv_N Varchar2(7)
EDW_Invoice_Line_Item Invoice_Adjustment_N Number(5)
STG_Invc_Detail Inv_Adj_N Number(5)
EDW_Invoice_Line_Item Invoice_Line_N Number(5)
STG_Invc_Detail Inv_Lin_N Number(5)
EDW_Invoice_Line_Item Process_Ts Date STG_Invc_Detail Prcs_Ts DateEDW_Invoice_Line_Item Invoice_Header_ID Number(1
0)EDW_Invoice_Header
Invoice_Header_ID Number(10)
EDW_Invoice_Line_Item Packaged_Good_ID Number(10)
EDW_Packaged_Good
Packaged_Good_ID Number(10)
Number(2)
Number(4)
STG_Invc_Detail Ord_N, Entry_D
EDW_Order_Header
Shipment_Id
EDW_Shipment Load_Ord_N
STG_Invc_Detail Ord_N, Entry_D
EDW_Order_Header
Order_Type_C
STG_Invc_Detail Ord_N, Entry_DEDW_Invoice_Line_Item Submit_Method_C Varchar2(2)
Varchar2(2)
EDW_Invoice_Line_Item Order_Type_C Varchar2(2)
Varchar2(2)
EDW_Invoice_Line_Item Load_Order_N Varchar2(7) Varchar2(7
Use Actg_Mth_N + Actg_Yr_N to look up Fiscal_Period
EDW_Invoice_Line_Item Fiscal_Period Number(6) STG_Invc_Detail, EDW_Month
Target Table Target Column Datatype Source Table Source Column DatatypeEDW_Invoice_Line_Item Invoice_Line_Item_ID Number(1
0)Sequence Generator
EDW_Invoice_Line_Item Invoice_N Varchar2(7)
STG_Invc_Detail Inv_N Varchar2(7)
EDW_Invoice_Line_Item Invoice_Adjustment_N Number(5)
STG_Invc_Detail Inv_Adj_N Number(5)
EDW_Invoice_Line_Item Invoice_Line_N Number(5)
STG_Invc_Detail Inv_Lin_N Number(5)
EDW_Invoice_Line_Item Process_Ts Date STG_Invc_Detail Prcs_Ts DateEDW_Invoice_Line_Item Invoice_Header_ID Number(1
0)EDW_Invoice_Header
Invoice_Header_ID Number(10)
EDW_Invoice_Line_Item Packaged_Good_ID Number(10)
EDW_Packaged_Good
Packaged_Good_ID Number(10)
Number(2)
Number(4)
STG_Invc_Detail Ord_N, Entry_D
EDW_Order_Header
Shipment_Id
EDW_Shipment Load_Ord_N
STG_Invc_Detail Ord_N, Entry_D
EDW_Order_Header
Order_Type_C
STG_Invc_Detail Ord_N, Entry_D
EDW_Invoice_Line_Item Fiscal_Period Number(6) STG_Invc_Detail, EDW_Month
Use Actg_Mth_N + Actg_Yr_N to look up Fiscal_Period
EDW_Invoice_Line_Item Load_Order_N Varchar2(7) Varchar2(7
EDW_Invoice_Line_Item Order_Type_C Varchar2(2)
Varchar2(2)
EDW_Invoice_Line_Item Submit_Method_C Varchar2(2)
Varchar2(2)
16
Invoice transaction
• What do we have so far?
Package_Good_item
EDW_Invoice_Line_Item
EDW_Invoice_Header
Invoice 4/9
Error_Log
9
17
Table types
• Facts• Dimensions• Aggregates• Metadata• Error Logs• Release or publish tables
18
Dimensional Model
• Fact tables • Dimensions
10
19
Fact tablesEDW_Invoice_Header
20
Dimension tables
11
21
Table counting
EDW_Invoice_Line_Item
EDW_Invoice_Header1. Identify the entities that should be considered for
counting.
2. Identify the user view (=business view) of the data grouping by investigating
a) How the data is accessed as a group by elementary processes within the application boundary
b) The relationships between the entities and their interdependency based on business rules
3. Classify each identified logical file as ILF or EIF
22
Data mart consideration boundary
Invoicing Warehousing
Data mart
12
23
Data mart dimensional model
EDW_Invoice_Line_Item
Time Dimension
Customer Dimension
Product Dimension Supply Point Dimension
Shipment DimensionOrder Dimension
24
Mapping matrixWarehouseFunction Point transaction Type FTRs DETs ETL mappings Comments
Inv 7Inv 8Inv 9Inv 10Inv 11Inv 13Inv 14Inv 15Iinv 16Inv 4
Data store RETs DETs CommentsInvoice headerInvoice Line item
Package Good Items EIF Package good items >51
Data MartFunction Point transaction Type FTRs DETs ETL mappings CommentsNot covered Data store RETs DETs CommentsTime ?Order ?Shipment ?Product ?Customer ?Supply Point ?Invoice ?
Load Invoice >3 >16
>51Invoice
EI
ILF
13
25
Summarize
• Application boundaries• Understand the terminology used for this
domain of applications
26
Questions
14
27
Sources/References• Kimball, Reeves, Ross, Thornthwaite, The Data Warehouse Lifecycle Toolkit:Expert
methods for Desighning, Developing, and Deplying Data Warehouses, WILEY 1998• IFPUG, FP-380 Function Point Counting Workshop: Counting Data Warehouse
Applicatoins• Santillo, Luca, Size & Estimation of Data Warehouse Systems, Data Processing
Organization• IFPUG, Function Point Counting Practices Manual Release 4.2