introduction to the logparse macro gasug atlanta, ga july 18 th, 2007
TRANSCRIPT
Introduction to the LOGPARSE MacroIntroduction to the LOGPARSE Macro
GASUG Atlanta, GA
July 18th, 2007
© Copyright ERMAS Corporation 2007 LOGPARSE Page 2
Integration Consulting Marketing Automation and Optimization Marketing Campaign Management, Fulfillment, and
Response Analysis Risk & Analytics Analytical Modeling Data Warehousing Business Intelligence Solutions CRM Database Solutions Web Development Services
Our ServicesOur ServicesOur ServicesOur Services
© Copyright ERMAS Corporation 2007 LOGPARSE Page 3
Our Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + International
5. UK & Europe
We have completed projects in Korea, Singapore, Malaysia, United Kingdom, Netherlands, Sweden, Germany, and Peru.
3. New York
2. Charlotte
4. Dallas 1. Atlanta
International
Corporate Location:Corporate Location: 4080 McGinnis Ferry Rd4080 McGinnis Ferry Rd
Bldg 100, Suite 101Bldg 100, Suite 101Alpharetta, GA 30005Alpharetta, GA 30005
Corporate Location:Corporate Location: 4080 McGinnis Ferry Rd4080 McGinnis Ferry Rd
Bldg 100, Suite 101Bldg 100, Suite 101Alpharetta, GA 30005Alpharetta, GA 30005
© Copyright ERMAS Corporation 2007 LOGPARSE Page 4
Our Atlanta LabOur Atlanta LabOur Atlanta LabOur Atlanta Lab
Always Seeking TalentAlways Seeking TalentAlways Seeking TalentAlways Seeking Talent
Send inquiries to:
© Copyright ERMAS Corporation 2007 LOGPARSE Page 5
Today’s TopicToday’s TopicToday’s TopicToday’s Topic
Introduce the LOGPARSE macro to capture FULLSTIMER statistics
Demonstrate a technique for profiling very large SAS applications
Provide code for creating a Performance Dashboard
…Capture Tuning Statistics
…Not Tune SAS Applications
© Copyright ERMAS Corporation 2007 LOGPARSE Page 6
LOGPARSE MACROLOGPARSE MACROLOGPARSE MACROLOGPARSE MACRO
Provided by SAS, experimential with SAS 9.1
Captures FULLSTIMER statistics from the SAS Log and stores them in a SAS Dataset• For every SAS Step
Available on the SAS Scalability & Performance Community Site
www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html
www.support.sas.com/rnd/scalability/tools/fullstim/logparse.zip
© Copyright ERMAS Corporation 2007 LOGPARSE Page 7
What’s Inside logparse.zipWhat’s Inside logparse.zipWhat’s Inside logparse.zipWhat’s Inside logparse.zip
Four files in logparse.zip
%LOGPARSE()• Extracts the statistics from the SAS LOG(s)
%PASSINFO()• Optional, provides session information to the %LOGPARSE()
macro.
%MVSNAME()• Called by %PASSINFO() for MVS only
Readme.txt• A good place to start
logparse.sas passinfo.sas mvsname.sas readme.txt
© Copyright ERMAS Corporation 2007 LOGPARSE Page 8
Getting Started with %LOGPARSEGetting Started with %LOGPARSEGetting Started with %LOGPARSEGetting Started with %LOGPARSE
OPTIONS FULLSTIMER; as the first line for the program you wish to collect statistics
%PASSINFO; optionally on the second line
Run the program to generate the log(s)
Create another program to use %LOGPARSE and collect the statistics
%include logparse; %logparse( yourProgram.log, yourPerfData, OTH ); proc print data=yourPerfData; run;
© Copyright ERMAS Corporation 2007 LOGPARSE Page 9
Big PictureBig PictureBig PictureBig Picture
LOGFILE(S)
YOURLOGPARSEPROGRAM
StatisticsDataset
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
© Copyright ERMAS Corporation 2007 LOGPARSE Page 10
%LOGPARSE Syntax%LOGPARSE Syntax%LOGPARSE Syntax%LOGPARSE Syntax
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
saslog is the name of the SAS LOG to parse, requires the .log extension
outds is the output SAS dataset for the parsed LOG results
system is the 3-CHAR code for the system on which %LOGPARSE is run. If not specified, the default value is the system %LOGPARSE is executing on.
• MVS - z/OS, OS/390, or MVS• ALP - OpenVMS Alpha• VMS - VMS• OTH - All other OSs, like Windows
pdsloc identifies the PDS member where the SAS LOG is stored on MVS system. See the readme.txt file.
append=no specifies whether to append the results to an existing file or overwrite
• NO – creates a new SAS file based on the outds parameter• YES – appends the output to file specified by outds or creates a new file if
the file does not exist
© Copyright ERMAS Corporation 2007 LOGPARSE Page 11
The STATISTICS DatasetThe STATISTICS DatasetThe STATISTICS DatasetThe STATISTICS Dataset
The statistics captured by %LOGPARSE are dependent on the NOTES: produced by FULLSTIMER and vary by system.
Important non-statistics columns to understand• logfile – name of the log file
• stepname – DATA, SORT, FORMAT, PRINTO, SAS, initialization.
NOTE: DATA statement used (Total process time):
NOTE: The SAS System used:
• stepcnt – sequence number applied to the order of the step within the log.
Windows Statistics Columns• realtime• usertime• systemtime
• obsin• obsout• varsout
• memused
© Copyright ERMAS Corporation 2007 LOGPARSE Page 12
Big PictureBig PictureBig PictureBig Picture
LOGFILE(S)
YOURLOGPARSEPROGRAM
StatisticsDataset
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
© Copyright ERMAS Corporation 2007 LOGPARSE Page 13
ExampleExampleExampleExampleoptions fullstimer;
%let defaultPath=C:\Documents and Settings\Administrator\My Documents\My SAS Code;
%include "&defaultPath\timerUtils\passinfo.sas";%passinfo;
%include "&defaultPath\census\ipumsFormats.inc";%include "&defaultPath\census\convertToSAS.inc";%include "&defaultPath\census\splitData.inc";%include "&defaultPath\census\subsetMetro.inc";%include "&defaultPath\census\hhRollup.inc";
libname ipumsdat "&defaultPath\census";/*----------------------------------------------------------------------------+ Main Program+----------------------------------------------------------------------------*/%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);%splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);%subsetMetro("Atlanta", 2003 2004 2005 2006);%hhRollup(2003 2004 2005 2006)
© Copyright ERMAS Corporation 2007 LOGPARSE Page 14
The Bad NewsThe Bad NewsThe Bad NewsThe Bad News
A dataset of statistics is produced – map that back to your program/log…. OR
stepcnt stepname realtime usertime systime cputime obsin obsout varsout memused0 initialization 00:00.5 . . 00:00.2 . . . .1 DATA 00:00.0 00:00.0 00:00.0 . . . . 1612 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 133 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 134 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 135 FORMAT 00:00.7 00:00.0 00:00.0 . . . . 1826 FORMAT 00:00.0 00:00.0 00:00.0 . . 3052 21 887 DATA 01:28.6 00:32.1 00:03.5 . 3052 848875 220 3338 DATA 00:55.0 00:15.8 00:03.7 . 848875 848875 880 5069 DATA 00:01.6 00:01.3 00:00.2 . 1564 1564 9 203
10 SORT 00:00.0 00:00.0 00:00.0 . 1564 1564 9 28211 DATA 00:01.4 00:01.3 00:00.2 . 1651 1651 9 20312 SORT 00:00.0 00:00.0 00:00.0 . 1651 1651 9 28213 DATA 00:01.5 00:01.3 00:00.2 . 1922 1922 9 20314 SORT 00:00.0 00:00.0 00:00.0 . 1922 1922 9 31615 DATA 00:01.4 00:01.2 00:00.1 . 2427 2427 9 20316 SORT 00:00.0 00:00.0 00:00.0 . 2427 2427 9 36717 DATA 00:00.2 00:00.0 00:00.0 . 1564 600 10 20918 DATA 00:00.0 00:00.0 00:00.0 . 1651 619 10 20919 DATA 00:00.0 00:00.0 00:00.0 . 1922 742 10 20920 DATA 00:00.0 00:00.0 00:00.0 . 2427 921 10 20921 SAS 02:31.3 00:53.0 00:08.3 . . . . 3382
© Copyright ERMAS Corporation 2007 LOGPARSE Page 15
The Good NewsThe Good NewsThe Good NewsThe Good News
© Copyright ERMAS Corporation 2007 LOGPARSE Page 16
Use PRINTTO to Write Separate LogsUse PRINTTO to Write Separate LogsUse PRINTTO to Write Separate LogsUse PRINTTO to Write Separate Logs
filename fmtLog "&defaultPath\census\ipumsFormtas.timingDemo.log";filename cvrtLog "&defaultPath\census\convertToSAS.timingDemo.log";filename splitLog "&defaultPath\census\splitData.timingDemo.log";filename substLog "&defaultPath\census\subsetMetro.timingDemo.log";filename hhrupLog "&defaultPath\census\hhRollup.timingDemo.log";/*----------------------------------------------------------------------------+ Main Program+----------------------------------------------------------------------------*/proc printto log=fmtLog new; run;%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);
proc format cntlin=ipumsdat.fmmiller_ermas_com_001; run;
proc printto log=cvrtLog new; run;%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);
proc printto log=splitLog new; run; %splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);
proc printto log=substLog new; run;%subsetMetro("Atlanta", 2003 2004 2005 2006);
proc printto log=hhrupLog new; run;%hhRollup(2003 2004 2005 2006);run; proc printto; run;
MODULE.PROGRAM.LOG
Use PROC PRINTTO to WriteSeparate Log Files
© Copyright ERMAS Corporation 2007 LOGPARSE Page 17
Use our toolkit package to create a visual performance dashboard (WINDOWS ONLY)
Name each log in the format module.program.log for each subset of the program to be analyzed
Use the perfDash.sas program to analyze the logs
ERMAS Performance ToolkitERMAS Performance ToolkitERMAS Performance ToolkitERMAS Performance Toolkit
/*----------------------------------------------------------------------------+
User Parameters
rootDir - location of the SAS utilities, see installation instructions below
logDir - directory where the SAS LOGs are located
program - name of the SAS program for which to generate the dashboard
+----------------------------------------------------------------------------*/
%let rootDir = %str(C:\GASUG2007);
%let logDir = %str(C:\GASUG2007\census);
%let program = timingDemo; /* case sensitive */
© Copyright ERMAS Corporation 2007 LOGPARSE Page 18
How to get the Ermas Performance Toolkit
• GASUG2007.ZIP file contains four directories: timerUtils – location of the perfDash.sas program census – log files to analyze myLibrary – utility library utils – more utilities.
SAS Scalability & Performance Community
Great SUGI Paper on the LOGPARSE Macro
ResourcesResourcesResourcesResources
www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html
http://www.ermas.com/downloads.htm
http://www2.sas.com/proceedings/sugi30/219-30.pdf