em415 – custom extraction techniques
DESCRIPTION
EM415 – Custom Extraction Techniques. Robert Waywell Senior Product Support Engineer iAnywhere Solutions [email protected]. Understand the reasons to customize the extraction process for remote databases. Be able to plan and implement an appropriate extraction process for your environment. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/1.jpg)
EM415 – Custom Extraction Techniques
Robert WaywellSenior Product Support EngineeriAnywhere [email protected]
![Page 2: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/2.jpg)
Objectives
Understand the reasons to customize the extraction process for remote databases.
Be able to plan and implement an appropriate extraction process for your environment.
![Page 3: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/3.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 4: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/4.jpg)
DBXtract - Benefits
Generates schema and data.Easy to use.One-stop shopping.
![Page 5: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/5.jpg)
DBXtract – Limitations
Schema consists of all-or-none of various object types:Stored ProceduresViewsTriggers
Doesn’t handle non-replicated tables.May want different trigger logic on remotes.
Avoid “IF CURRENT REMOTE USER IS NULL …” logic.
For performance reasons, may want different indexes on consolidated and remotes.
Doesn’t handle multiple tiers.
![Page 6: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/6.jpg)
MLXtract
Run against the reference database.Generates the schema.If using an ASA consolidated database, can
extract the data as well.Otherwise, the only difference between
remote databases is the SYNCHRONIZATION DEFINITION.
![Page 7: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/7.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 8: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/8.jpg)
Performance
Database InitializationSchema CreationCommon DataNode-Specific Data
![Page 9: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/9.jpg)
Database Initialization
Initializing a blank, default database takes approx 24 seconds on average.dbinit db1.db
Initializing 1 database and copying it 9 times took 31-33 seconds total.dbinit db1.db
copy db1.db db2.dbdblog -t db2.log db2.db…
![Page 10: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/10.jpg)
Database Initialization
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10
# of Databases
Tim
e (s
)
DBInit Init & Copy
![Page 11: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/11.jpg)
Schema Creation
Using a 10 table schema and creating the schema individually in each database took approx. 15 seconds per database.start /wait dbisql -c "uid=dba;pwd=sql;eng=schema;dbf=db1.db" read
regionaloffice.sql
Notes:1) Approx. 7 seconds to load the database and connect from ISQL2) Leaves approx. 8 seconds to load the schema.3) Schema used was the Regional Office database from EM414 SQL Remote
Techniques• Includes supporting triggers, RI, etc.
Rough estimate for a 200 table schema plus supporting triggers, RI, etc. is 160 seconds.
![Page 12: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/12.jpg)
Schema Creation (cont.)
Loading the schema once before copying the databases took approx. 48 seconds total for 10 databases.dbinit db1.dbstart /wait dbisql -c "uid=dba;pwd=sql;eng=schema;dbf=db1.db" read
regionaloffice.sqlcopy db1.db db2.dbdblog -t db2.log db2.db…
Notes:1) 24 seconds to initialize 1st database2) 15 seconds to load schema in 1st database3) 9 seconds to copy 9 databases
![Page 13: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/13.jpg)
Initialization & Schema Creation
0
100
200
300
400
500
1 2 3 4 5 6 7 8 9 10
# of Databases
Tim
e (s
)
Individual Template & Copy
![Page 14: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/14.jpg)
Common Data
Data shared by all, or a large group of, nodes in the system.
Examples:Product ListsLook-Up TablesEmployee Information
Can be extracted once, included in a template database and copied many times.
![Page 15: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/15.jpg)
Node-Specific Data
Data that is unique to a given node or a small group of nodes.
Examples:Customers
Projects
Sales Leads
Needs to be extracted separately for each individual user.
![Page 16: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/16.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 17: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/17.jpg)
Template Database
Concept:Do as much work as possible once.
Can Include:Standard Schema
• Including non-replicated tables
• Views and procedures specific to the remote
• May customize triggers and indexes to optimize performance
• Administrative tables and publications
Common Data
![Page 18: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/18.jpg)
Template Database – Costs
1) Takes time to create.2) Maintenance
• Differs from either a remote or a consolidated database
• Changes made to either the remote or consolidated schema need to be mirrored in the template database
3) Cost of identifying and extracting node specific data.• Can’t use DBXtract to extract data for only some of the tables
• For MobiLink, need first-time-synch logic that only synchronizes the node specific data and not the common data
![Page 19: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/19.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 20: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/20.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 21: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/21.jpg)
Sample Databases
Head Office
Regional Office
Manager1SalesRep1 SalesRep2
![Page 22: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/22.jpg)
Database Initialization
Can use DBInit or Sybase Central• Recommend DBInit, then the process can be scripted
which facilitates testing and re-use
Ensure proper initialization settings such as:• Case sensitivity
• Blank padding
• Collation sequence
![Page 23: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/23.jpg)
Database Initialization – Script
Create_template.bat
%asany7%\win32\dbinit template.db
![Page 24: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/24.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 25: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/25.jpg)
Schema Creation
Replicated SchemaNon-replicated Schema
Common SchemaTier-Specific Schema
![Page 26: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/26.jpg)
Replicated Schema
• DBXtract –n • The –n switch extracts schema only• Useful to determine what schema DBXtract would have created
on its own
• GLOBAL TEMPORARY tables and database events are always extracted
• Switches to control what objects are extracted:• Views (-xp)• Triggers (-xt)• Stored Procedures (-xp)• Foreign Keys (-xf)
• Remember that reciprocal publications and subscriptions are created on the remote nodes
![Page 27: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/27.jpg)
Reciprocal Publications and Subscriptions
RegionalOffice CREATE PUBLICATION Customer_Pub( TABLE Customer SUBSCRIBE BY ( SELECT salesrep_id FROM Link WHERE Customer.customer_id = Link.customer_id ))
DBXtract/SSXtract
SalesRep1
CREATE PUBLICATION Customer_Pub( TABLE Customer)
![Page 28: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/28.jpg)
Replicated Schema – Script
Create_template.bat
rem need to create a template remote user to be used by dbxtract in generating the schema
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;eng=regionaloffice;dbn=regionaloffice;dbf=regionalofficedb\regionaloffice.db" read create_new_remote.sql [template_user] [0]
start /wait %asany7%\win32\dbxtract -c "uid=dbxtract_user;pwd=sql;eng=regionaloffice;dbn=regionaloffice;dbf=regionalofficedb\regionaloffice.db" -n -r pre_template.sql unload_dir template_user
![Page 29: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/29.jpg)
Replicated Schema – Script (cont)
rem Once the template schema has been extracted, we need to
rem remove the template_user from the production database
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;eng=regionaloffice;dbn=regionaloffice;dbf=regionalofficedb\regionaloffice.db" read drop_existing_remote.sql [template_user] [0]
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;dbn=template;eng=template;dbf=template.db" read pre_template.sql
![Page 30: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/30.jpg)
Replicated Schema – Script (cont)
Create_new_remote.sqlPARAMETERS user_id, subscribe_by;
GRANT CONNECT TO {user_id} IDENTIFIED BY sql;
GRANT REMOTE TO {user_id} TYPE FILE ADDRESS '{user_id}';
CREATE SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR {user_id};
![Page 31: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/31.jpg)
Replicated Schema – Script (cont)
Create_new_remote.sql (cont)CREATE SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR
Manager1;
START SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR Manager1;
CREATE SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR HeadOffice;
START SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR HeadOffice;
![Page 32: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/32.jpg)
Replicated Schema – Script (cont)
Create_new_remote.sql (cont)CREATE SUBSCRIPTION TO
Manager_Expense_Pub('{subscribe_by}') FOR Manager1;
START SUBSCRIPTION TO Manager_Expense_Pub('{subscribe_by}') FOR Manager1;
CREATE SUBSCRIPTION TO Admin_Pub('{user_id}') FOR {user_id};
CREATE SUBSCRIPTION TO Admin_Pub('ALL') FOR {user_id};
CREATE SUBSCRIPTION TO Clean_Up_Pub('NotMe') FOR {user_id};
![Page 33: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/33.jpg)
Replicated Schema – Script (cont)
Create_new_remote.sql (cont)CALL Prime_Key_Pool( '{user_id}', 'Expense');
CALL Prime_Key_Pool( '{user_id}', 'Order');
COMMIT;
REMOTE RESET {user_id};
COMMIT;
![Page 34: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/34.jpg)
Replicated Schema – Script (cont)
Drop_Existing_Remote.sqlPARAMETERS user_id, subscribe_by;
DROP SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR {user_id};
DROP SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR Manager1;
DROP SUBSCRIPTION TO Customer_Pub('{subscribe_by}') FOR HeadOffice;
DROP SUBSCRIPTION TO Manager_Expense_Pub('{subscribe_by}') FOR Manager1;
![Page 35: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/35.jpg)
Replicated Schema – Script (cont)
Drop_Existing_Remote.sql
DROP SUBSCRIPTION TO Admin_Pub('{user_id}') FOR {user_id};DROP SUBSCRIPTION TO Admin_Pub('ALL') FOR {user_id};DROP SUBSCRIPTION TO Clean_Up_Pub('NotMe') FOR {user_id};
REVOKE REMOTE FROM {user_id};REVOKE CONNECT FROM {user_id};
DELETE FROM keypoolWHERE remote_location = '{user_id}';
COMMIT;
![Page 36: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/36.jpg)
Non-replicated Schema
Compare physical data model to the schema created by DBXtract
Refer to the EM414 & EM415 Schema data model• EM414 describes a Message System Control Chart technique for
detecting unexpected latency in the SQL Remote message system• This technique uses 3 tables and 1 database event
Base Tables:Latency_exception_reportControl_chart
GLOBAL TEMPORARY table:Latency_calculation
Database Event:Populate_control_chart
![Page 37: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/37.jpg)
Non-Replicated Schema (cont)
• In this example, the Latency_calculation table and the Populate_control_chart event will have been included in the reload.sql script generated by DBXtract
• The Latency_exception_report and Control_chart tables will not have been included in the reload.sql script since they are not part of a publication.
• Options are to either add the missing tables or remove the GLOBAL TEMPORARY table and the database event.
• If there is unexpected latency, then I probably won’t be able to replicate an exception report up to the consolidated database
![Page 38: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/38.jpg)
Non-Replicated Schema (cont)
Any tables that are replicated one-way up from a remote database will not be included by DBXtract in the reload.sql.
Examples:• Replication_Error table• Acknowledgement tables• Expense table
![Page 39: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/39.jpg)
Non-Replicated Schema - Script
Complete_Template.sqlCREATE TABLE "DBA"."Expense"(
"expense_id" integer NOT NULL,"salesrep_id" integer NOT NULL,"description" char(40) NOT NULL,"amount" money NOT NULL,PRIMARY KEY ("expense_id")
)go
![Page 40: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/40.jpg)
Non-Replicated Schema – Script (cont)
Complete_Template.sql (cont)
CREATE TRIGGER BI_Expense BEFORE INSERT ORDER 1 ON Expense
REFERENCING NEW AS newrow
FOR EACH ROW
BEGIN…
END;
![Page 41: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/41.jpg)
Common Schema
• The pieces of the schema that are shared between tiers• Most of the table definitions and RI constraints will be common
between tiers• Key Pool triggers will be required at all tiers• Stored procedures used by the front-end application• Database Events for backup & recovery• …
• As long as these pieces are replicated, then they will have been included when you ran DBXtract
• If they are not replicated, then you will have picked them up when preparing the scripts for the non-replicate schema
![Page 42: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/42.jpg)
Tier-Specific Schema
Most of the UPDATE … PUBLICATION triggers will not be required at the remote nodes
Some publications at the remote nodes will require SUBSCRIBE BY clauses in order to force data up to the consolidated databasei.e. request_queue_return_codesRequest queue implementation handles some requests via a triggerIn order to force the result codes up to the consolidated database,
need a SUBSCRIBE BY clause on the article
Did you have indexes created at the consolidated node to support reporting functionality? If so, are these indexes worthwhile at the remote nodes?
![Page 43: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/43.jpg)
Tier-Specific Schema (cont)
Are there tables used at the remote nodes that are never replicated up?
Are additional columns included in a given table at different tiers?i.e. Expense table has an approval_status column at the Regional
Office tier that does not exist at the Sales Rep tier
Key Pool Trigger logic• Do you write a more complex trigger that contains all the logic
necessary to execute at any tier or do you use separate, less complex triggers for each tier
![Page 44: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/44.jpg)
Tier-Specific Schema (cont)
Administrative publications may differ between tiers to facilitate control of data flow
Regional Office:CREATE PUBLICATION Clean_Up_Pub
( TABLE Request_Queue_Return_Codes SUBSCRIBE BY remote_location,
TABLE Rep_Errors SUBSCRIBE BY remote_location
);
CREATE SUBSCRIPTION TO Clean_Up_Pub('NotMe') FOR SalesRep1;
CREATE SUBSCRIPTION TO Clean_Up_Pub('NotMe') FOR SalesRep2;
CREATE SUBSCRIPTION TO Clean_Up_Pub('NotMe') FOR Manager1;
![Page 45: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/45.jpg)
Tier-Specific Schema (cont)
Administrative publications may differ between tiers to facilitate control of data flow
Sales Rep:CREATE PUBLICATION Response_Pub( TABLE Request_Queue_Return_Codes SUBSCRIBE BY subscribe_by,
TABLE Rep_Errors);
CREATE SUBSCRIPTION TO Response_Pub('RegionalOffice') FOR RegionalOffice;
CREATE SUBSCRIPTION TO Response_Pub('HeadOffice') FOR RegionalOffice;
![Page 46: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/46.jpg)
Tier-Specific Schema – Script
Create_template.bat (cont)start /wait %asany7%\win32\dbisqlc -c
"uid=dba;pwd=sql;dbn=template;eng=template;dbf=template.db" read complete_template.sql
Complete_template.sql (cont)DROP TRIGGER "DBA"."RU_Order";
DROP TABLE latency_calculation;
DROP EVENT populate_control_chart;
![Page 47: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/47.jpg)
Tier-Specific Schema – Script (cont)
Complete_template.sql (cont)ALTER PUBLICATION "DBA"."Customer_Pub"
DROP TABLE "DBA"."Region";
ALTER PUBLICATION "DBA"."Customer_Pub"
DROP TABLE "DBA"."Product";
ALTER PUBLICATION "DBA"."Customer_Pub"
DROP TABLE "DBA"."Order_Status";
![Page 48: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/48.jpg)
Tier-Specific Schema – Script (cont)
Complete_template.sql (cont)CREATE PUBLICATION Expense_Pub(
TABLE Expense);
CREATE PUBLICATION Response_Pub(
TABLE Request_Queue_Return_Codes SUBSCRIBE BY subscribe_by,TABLE Rep_Errors
);
![Page 49: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/49.jpg)
Tier-Specific Schema – Script (cont)
Complete_template.sql (cont)
DROP PUBLICATION "DBA"."Admin_Pub";
CREATE SUBSCRIPTION TO Expense_Pub FOR RegionalOffice;
CREATE SUBSCRIPTION TO Response_Pub('RegionalOffice') FOR RegionalOffice;
CREATE SUBSCRIPTION TO Response_Pub('HeadOffice') FOR RegionalOffice;
REVOKE CONNECT FROM template_user;
![Page 50: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/50.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 51: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/51.jpg)
Load the Common Data
• What tables are replicated in their entirety?• Can use UNLOAD TABLE to extract this data from the
consolidated database• Examples:
• Product• Region• Order_Status
At the Consolidated:Extract_common_data.sqlUNLOAD TABLE product TO ‘product.dat’;UNLOAD TABLE region TO ‘region.dat’;UNLOAD TABLE order_status TO ‘order_status.dat’:
![Page 52: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/52.jpg)
Load the Common Data (cont)
At the Template:Load_common_data.sqlLOAD TABLE product FROM ‘product.dat’;
LOAD TABLE region FROM ‘region.dat’;
LOAD TABLE order_status FROM ‘order_status.dat’;
Notes:1. The path of the file created UNLOAD TABLE and read by LOAD
TABLE is relative to the location of the database server
2. These tables are published one-way down to the remotes
![Page 53: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/53.jpg)
Load the Common Data – Script
Create_template.bat (cont)start /wait %asany7%\win32\dbisqlc -c
"uid=dba;pwd=sql;eng=regionaloffice;dbn=regionaloffice;dbf=regionalofficedb\regionaloffice.db" read extract_common_data.sql
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;dbn=template;eng=template;dbf=template.db" read load_common_data.sql
![Page 54: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/54.jpg)
Load the Common Data (cont)
• If not the whole table, do all the users still get all the rows that are replicated?• Can use “SELECT …; OUTPUT TO …” to extract
this data from the consolidated databasei.e. Product table may contain discontinued products in
addition to current productsAt the Consolidated:SELECT product_id, description, price, quantity FROM product
WHERE status = ‘current’;OUTPUT TO ‘product.dat’;
![Page 55: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/55.jpg)
Load the Common Data (cont)
At the Template:INPUT INTO product FROM ‘product.dat’;
Note: OUTPUT and INPUT are ISQL functions, not database engine functions.
• The path used by OUTPUT and INPUT is relative to the working directory of ISQL
![Page 56: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/56.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 57: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/57.jpg)
Minimizing Down Time
Ideally, remote databases should be extracted from a quiescent consolidated database.• If a remote database is extracted from an active
consolidated database, then the extraction must be done at isolation level 3 to guarantee consistency.
This may not be practical:• 7X24• The more remotes, the longer the database will be impacted
Solution is to work with a copy of the consolidated database.
![Page 58: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/58.jpg)
Minimizing Down Time (cont)
Process:1. Shut down the consolidated database server and restart the
consolidated database on a standalone engine.• This is to ensure that no other users are able to connect to
the consolidated database during the process2. Run a script to add the new remote user(s) to the consolidated
database.• This is the Create_new_remote.sql script
3. Shut down the consolidated database and make a copy of it.• This must be an off-line backup to ensure that the database
was shut down cleanly.• We need to ensure that none of the data changes on the
consolidated between the time the remote users are created and the time that the copy of the database is made
![Page 59: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/59.jpg)
Minimizing Down Time (cont)
Process (cont):4. At this point you can re-start the consolidated
database on the production server.
• Changes made after this point will be replicated out, as appropriate, to the new remote users
• The copy of the consolidated database has captured the common initial state to be used in extracting the new remote databases.
![Page 60: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/60.jpg)
Minimizing Down Time – Script
Use_template.bat
if exist copy_of_cons.db %asany7%\win32\dberase -y copy_of_cons.db
copy regionalofficedb\regionaloffice.db copy_of_cons.db
%asany7%\win32\dblog -t copy_of_cons.log copy_of_cons.db
![Page 61: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/61.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 62: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/62.jpg)
Node Specific Data
At this point, we have a template database that contains all of the schema that we need and some of the data that we need.
If you run DBXtract –d –v … you can see how DBXtract is selecting the data for a given remote user.
• -d is “data only”• -v is “verbose”
Since we are working with a copy of the consolidated database at this point, we can modify the publications to not include the common data that has already been extracted.
![Page 63: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/63.jpg)
Node Specific Data – Scripts
Use_template.bat (cont)start /wait %asany7%\win32\dbisqlc -c
"uid=dba;pwd=sql;eng=copy_of_cons;dbn=copy_of_cons;dbf=copy_of_cons.db" read drop_common_data_publications.sql
Drop_common_data_publications.sqlDROP TABLE "DBA".latency_calculation;
ALTER PUBLICATION "DBA"."Customer_Pub" DROP TABLE "DBA"."Region";
ALTER PUBLICATION "DBA"."Customer_Pub" DROP TABLE "DBA"."Product";
ALTER PUBLICATION "DBA"."Customer_Pub" DROP TABLE "DBA"."Order_Status";
![Page 64: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/64.jpg)
Node Specific Data – Scripts (cont)
Use_template.batrem %1 is the user id for the user that the remote database is being created forrem %2 is the operation(s) to be performedrem clean cleans up the files and directories from a previous runrem full cleans up and then creates a new remote databaserem refresh take a new copy of the consolidated databaserem when specifying "refresh", specify "none" for the user idrem the default is "full“…if not exist %1db md %1dbcopy template.db %1db\%1.dbcd %1db%asany7%\win32\dblog -t %1.log %1.dbcd ..
![Page 65: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/65.jpg)
Node Specific Data – Scripts (cont)
Use_template.bat (cont)
start /wait %asany7%\win32\dbxtract -d -c "uid=dbxtract_user;pwd=sql;dbn=copy_of_cons;eng=copy_of_cons;dbf=copy_of_cons.db" -o %1_extract.out -r %1_reload.sql unload_dir %1
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;eng=%1;dbn=%1;dbf=%1db\%1.db" read %1_reload.sql
start /wait %asany7%\win32\dbisqlc -c "uid=dba;pwd=sql;eng=%1;dbn=%1;dbf=%1db\%1.db" read finish_new_remote.sql [%1]
![Page 66: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/66.jpg)
Node Specific Data – Scripts (cont)
Finish_new_remote.sql
PARAMETERS user_id;
GRANT CONNECT TO {user_id} IDENTIFIED BY sql;GRANT PUBLISH TO {user_id};CREATE REMOTE TYPE "FILE" ADDRESS '{user_id}';
// Since all my database objects are owned by DBA, granting membership in that group
// simplifies SQL statements by not requiring objects to be qualified with DBA.// DBA permissions are not inherited by groups or members of a groupGRANT MEMBERSHIP IN GROUP DBA TO {user_id};
REMOTE RESET RegionalOffice;
![Page 67: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/67.jpg)
SQL Remote Template Database
At this point we have a working template database and customized extraction process.
To add a given user to the list of users to be extracted requires only 2 lines to be added to a single batch file.
Have saved time on:• Initializing the database• Creating the schema• Extracting & loading common data
Have reduced impact on the production consolidated database by working on an offline copy for the resource intensive extraction of node-specific data.
![Page 68: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/68.jpg)
SQL Remote Template Database – Script
Extract_salesreps.batrem This batch file will extract one remote user for each call to
“subscribe_new_remote.bat“ and “use_template.bat”rem Each time this file is run, we will create a new template database and refresh
the off-line copyrem of the consolidated database.
if %1. == . goto :fullif %1 == clean goto :clean
:fullcall create_template
![Page 69: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/69.jpg)
SQL Remote Template Database – Script (cont)
Extract_salesreps.bat (cont)call subscribe_new_remote.bat salesrep1 1call subscribe_new_remote.bat salesrep2 2call subscribe_new_remote.bat salesrep3 3
call use_template none refresh
call use_template salesrep1 fullcall use_template salesrep2 fullcall use_template salesrep3 full
![Page 70: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/70.jpg)
SQL Remote Template Database – Script (cont)
Extract_salesreps.bat (cont)call repall.batcall repall.batcall repall.bat
goto :end
:cleancall create_template cleancall drop_existing_remote salesrep1 1call drop_existing_remote salesrep2 2call drop_existing_remote salesrep3 3
:end
![Page 71: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/71.jpg)
SQL Remote Template Database – Maintenance
SQL Scripts:Complete_template.sql Create_new_remote.sql
Drop_common_data_publications.sql Drop_existing_remote.sql
Extract_common_data.sql Load_common_data.sql
Finish_new_remote.sql
Batch Files:Create_template.bat Drop_existing_remote.bat
Subscribe_new_remote.bat Use_template.bat
Extract_salesreps.bat
![Page 72: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/72.jpg)
SQL Remote
Database InitializationSchema CreationLoad the Common DataMinimizing Down TimeNode Specific DataCustomized scripts
![Page 73: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/73.jpg)
Customized Scripts
In a typical schema, many tables will have common pieces to their SUBSCRIBE BY clausesTABLE Customer SUBSCRIBE BY
( SELECT salesrep_id FROM Link WHERE Customer.customer_id = Link.customer_id
),TABLE Contact SUBSCRIBE BY
( SELECT salesrep_id FROM LinkWHERE Contact.customer_id = Link.customer_id
),TABLE "Order" SUBSCRIBE BY
( SELECT salesrep_id FROM LinkWHERE "Order".customer_id = Link.customer_id
),
![Page 74: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/74.jpg)
Customized Scripts (cont)
DBXtract uses the following queries to extract the data for these tables:
Unloading "DBA"."Customer" into unload_dir\205.dat (relative to server)SELECT "customer_id", "company_name", "address" FROM "DBA"."Customer"
WHERE (( '1' IN (select salesrep_id from DBA.Link where Customer.customer_id = Link.customer_id) ))
Unloading "DBA"."Contact"Unloading "DBA"."Contact" into unload_dir\206.dat (relative to server)SELECT "contact_id", "contact_name", "phone", "email", "customer_id" FROM
"DBA"."Contact" WHERE (( '1' IN (select salesrep_id from DBA.Link where Contact.customer_id = Link.customer_id) ))
![Page 75: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/75.jpg)
Customized Scripts (cont)
DBXtract queries (cont)Unloading "DBA"."Order"
Unloading "DBA"."Order" into unload_dir\207.dat (relative to server)
SELECT "order_id", "customer_id", "product_id", "quantity", "status_id", "order_date", "created_log_offset", "created_location" FROM "DBA"."Order" WHERE (( '1' IN (select salesrep_id from DBA.Link where
"Order".customer_id = Link.customer_id) ))
![Page 76: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/76.jpg)
Customized Scripts (cont)
The data from these tables could be extracted more efficiently if a temporary table was used to hold the list of customer id’s assigned to this sales rep.
CREATE GLOBAL TEMPORARY TABLE customer_ids_for_salesrep( customer_id INT PRIMARY KEY)
INSERT INTO customer_ids_for_salesrep SELECT Link.customer_id FROM Link
WHERE Link.salesrep_id = 1;
SELECT Customer."customer_id", "company_name", "address" FROM "DBA"."Customer“, “DBA”.”customer_ids_for_salesrep” WHERE Customer.customer_id = customer_ids_for_salesrep.customer_id;
![Page 77: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/77.jpg)
Customized Scripts (cont)
SELECT "contact_id", "contact_name", "phone", "email", Contact."customer_id" FROM "DBA"."Contact", “DBA”.”customer_ids_for_salesrep”
WHERE Contact.customer_id = customer_ids_for_salesrep.customer_id;
SELECT "order_id", Order."customer_id", "product_id", "quantity", "status_id", "order_date", "created_log_offset", "created_location“
FROM "DBA"."Order", “DBA”.”customer_ids_for_salesrep”
WHERE Contact.customer_id = customer_ids_for_salesrep.customer_id;
![Page 78: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/78.jpg)
Customized Scripts (cont)
More development cost, greater performance optimization.
May not be reasonable for relatively flat schemas.
The longer the chain of table dependencies, the more beneficial it will be to customize the SQL queries used to extract the node specific data.
![Page 79: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/79.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 80: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/80.jpg)
MobiLink
Reference database already defines the schema for the remotes.
Proxy tables can be used to populate the common data in the template before copying the database to create a new remote.
First time synchronization logic will have to handle both:• Not downloading the common data that already exists on the remote.• Downloading the node-specific data that does not yet exist on the
remote.
Special case handling of the first synchronization would be required even if not using a template database.
![Page 81: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/81.jpg)
MobiLink (cont)
Could handle the first time synchronization with a separate script version, but this would require maintaining two script versions.• Easier to maintain a single script version • First time synchronization is usually handled by setting a last synched
time value to “pre-history”• In 8.0 last synch value automatically included
Have the option of using Proxy tables to extract the node specific data but:1. Already have synchronization scripts to extract the data.2. DBMLSync already uses a direct ODBC connection to the consolidated
database.
![Page 82: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/82.jpg)
MobiLink (cont)
Will want to run DBMLSync from a LAN connection (desktop remotes) or when docked (CE remotes) for the first synchronization.
![Page 83: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/83.jpg)
Topics
DBXtract and MLXtractPerformanceTemplate DatabaseSQL RemoteMobiLinkSummary
![Page 84: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/84.jpg)
Summary
Using a template database can have an orders-of-magnitude impact on the time required to extract the remote databases for a new installation.• There will be some minimum number of remotes,
below which it is not worth the effort to customize the extraction process
Customizing the extraction process may require only a few days development effort.
![Page 85: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/85.jpg)
Recommended Sessions
Morning SessionsAM33 - Adaptive Server Anywhere (ASA) Internals
Performance and TuningAM38 – Data Replication with ASAAM36 – MobiLink Synchronization
Afternoon SessionsEM414 – SQL Remote Tips & TechniquesEM417 - Setting Up a SQL Remote System for PerformanceEM418 - SQL Remote for Adaptive Server Anywhere Internals EM408 - Back-up and Recovery of SQL Anywhere: Tips &
Techniques
![Page 86: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/86.jpg)
Newsgroups
forums.sybase.com
\sybase.public.sqlanywhere.general\sybase.public.sqlanywhere.linux\sybase.public.sqlanywhere.mobilink\sybase.public.sqlanywhere.product_futures_discussion\sybase.public.sqlanywhere.replication\sybase.public.sqlanywhere.ultralite
![Page 87: EM415 – Custom Extraction Techniques](https://reader035.vdocument.in/reader035/viewer/2022062410/568159fb550346895dc7485e/html5/thumbnails/87.jpg)
iAnywhere Solutions Highlights
• Ask the Experts - about Mobile & Wireless Solutions-Mezzanine Level Room 15B Mon./Tues. 11:30 am - 3:30 pm; Wed. 11:30 - 1:30;Thurs. 9 am - 12 noon-Exhibit Hall - Demo Center (truck) exhibit hall hours
• SIG (Special Interest Group)- Tuesday 5:30pm Mobile & Wireless SDCC, Upper level, Room 11
• Keynote - Enabling m-Business SolutionsWednesday 1:30 pm - 3:00 pm
• iAnywhere Solutions Developer Community-Excellent resource for commonly asked questions, newsgroups, bugfixes, newsletters, event listings - visit www.ianywhere.com/developer