stat 342 - wk 8: continuous data - sfu.cajackd/stat342/lect_wk08.pdf · 2016. 12. 15. · stat 342...

Stat 342 - Wk 8: Continuous Data

proc iml

Loading and saving to datasets

proc means

proc univariate

proc sgplot

proc corr

Stat 342 Notes. Week 3, / 71

PROC IML - Reading other datasets.

If you want to use an existing dataset as a matrix, rather than setting your own manually, you can do this with the 'use' and 'close' commands, combined with the 'read' command.

'Use' is used to tell the system to bring a certain dataset into memory. 'Close' is to wipe it from memory (useful for keeping things running smooth).

Read is used to take that dataset in memory, and save it into a matrix.


The general syntax is

use <DATASET>;

read all var <VARIABLE NAMES> into <MATRIX NAME>;

close <DATASET>;


Use (and close) use the same syntax for specifying a dataset as other SAS procedures. The dataset is specificed by libname.dataset, and addition options like (obs = ) can be used to specify which rows to load into memory.

use ds1;

use work.ds1;

use somelib.cars;

use somelib.cars(OBS = 10);


Close is the same, except that the options are not necessary in the close statement.

close ds1;

close work.ds1;

close somelib.cars;

close somelib.cars;


With the read command, references to entire classes of variable names still work, including _ALL_ , _NUM_ , and _CHAR_ for all variables, numeric ones, and character ones, respectively.

proc iml;

use SASHELP.Cars(OBS=5);

read all var _NUM_ into m1;

read all var _NUM_ into m2[colname=NumericNames];

close SASHELP.Cars;

print m1, m2;


Matrices are saved as SAS datasets with the create command and the append command.

'Create' makes a new SAS data set of whatever library and name you specify. However, the create command alone only makes a blank dataset (with whatever formatting you specify).

'Append' takes data from some matrix in the IML environment and adds (appends) it to the particular dataset,such as the one you just made with 'create'.


The general syntax is...

create <DATASET> from <MATRIX>;

append from <MATRIX>;

close <DATASET>;

Like when loading, it's a good idea to close your datasets after you're done with them.


Example of saving data, and seeing with with a proc print;

proc iml;

m = {1 2 3 . , 5 6 7 999};

m = t(m);

create newDS from m[colname={"x","y"}];

append from m;

close work.newDS;

quit;

proc print (data=newDS); run;


Readings on proc iml

Textbook source: Pages 62-68 of 'SAS and R'.

Additional source for interest:

SAS Whitepaper 144-2013 - Getting started with the SAS/IML Language, by Rick Wilkin

https://support.sas.com/resources/papers/proceedings13/144-2013.pdf




I hope IML isn't too fuzzy from two weeks ago


Proc means and proc univariate are used to describe individual variables on their own. Their capabilities are very similar, but...

1. They have different defaults and slightly different syntaxes.

2. Proc univariate can create histograms and boxplots.


Proc summary also operates similarly to proc means and proc univariate, but since...

3. Proc summary can't produce output to the screen (only toanother file or dataset)...

... and its syntax is less intuitive, we won't be covering it.


The general syntax for proc means is...

proc means data=input_ds <table options>

<statistics of interest>;

<class/by varnames>;

var <numeric variables>;

output <output_ds> <statistic = varnames>;

run;


The complete list for <table options> is found here:

http://support.sas.com/documentation/cdl/en/proc/61895/HTML/default/viewer.htm#a000146729.htm

Notables include:

missing – When summarizing by groups using a categorical variable, this makes 'missing data' count as a group (shown in document camera example).




Alpha= <'0.05'> – The significance level. Used for confidence interval statistics. The default value is 0.05, whichwill provide 95% confidence intervals.

Maxdec= <2> – The maximum digits of precision after the decimal point. 0 means everything is rounded to the nearestwhole number, 1 means everything is rounded to the nearest 0.1, and so on.

Print / noprint – Produce or ignore on-screen output.


Example 1: What does this do?

proc means data=input_ds alpha=0.10 maxdec=3 missing noprint;

...;

run;


<statistics of interest> is where we list the statistics we wish to use to describe the variables.






run;


By default the stats computed are n mean std min max

n – The number of observations.

std – The standard deviation of the observations.

Other notables include sum stderr var and...

nmiss – The number of values missing

clm – Confidence limits of the mean. Affected by the alpha level determined earlier, but defaults to the 95% confidence bounds/limits.


Q1 Q3 – The first (lower) and third (upper) quartiles of the data, where 25% and 75% is below these points, respectively.

qrange – Interquartile range, the difference between Q1 and Q3. In other words, the size of the middle half of the data.

Skewness skew – Skewness, or skew. The central third moment. If this is positive, then you have more extreme


data values above the mean than below. In other terms, the distribution stretches or 'skews' in the positive direction.

kurtosis – The fourth moment, used to describe how peak-like (positive kurtosis), or flat/plateau-like a unimodal distribution like the normal distribution is.

P1 P5 P10 P90 P95 P99 – Percentiles. The points at which1%. 5%, 10%, 90%, 95%, and 99% of the (observed) values for a given variable. See also quartiles, and the median.


This is the complete list


var <numeric variables> defines the variables that will be summarized.






run;


Example 2: What does this do?

proc means data=input

clm std P1 P5 Q1 median Q3 P95 P99;

...;

run;


Now you can be the meanest of the mean.


var <numeric variables> defines the variables for which summary statistics are required.

By default, EVERY numeric variable is included in the summary, even ones that wouldn't make sense to take statistics of, like observation numbers, IDs, and phone numbers. This is just like if you had specified _numeric_ in the list of variables.


The batting dataset, found in your emails and on the websiteof the auithor of SAS and R includes year-by-year records of many different Major League Baseball players.

It's more than 90,000 rows long, and includes thousands of players. We are interested in finding and confirming year-to-year patterns like 'have strike-outs become more common inrecent years'.

A dataset like this is too large to feasibly print to screen. SAS University Edition will only show 100 rows at a time.


However, we CAN use a procedure like proc means to effectively characterize the data.

(Apologies to everyone that doesn't care about baseball, butits discrete events make it the easiest to analyze of popular sports in North America)

(Go Cubs!)


Loading the batting dataset

proc import

datafile='dir_location\baseball.csv'

out=ds dbms=csv;

delimiter=',';

getnames=yes;

run;


In the batting dataset, the variables that make the most sense would be games batted (G_batting), and the number of at-bats (AB), runs (R), hits (H), home-runs (HR) walks (BB), intentional walks (IBB), and strike outs (SO).

Therefore, the variable specification line would look like this:

var G_batting AB R H HR BB IBB SO;


Output <out_ds> defines the dataset that the output will be saved to (if any).

<statistic = varnames> defines the variable names for the statistics that you produce.





output <out_ds> <statistic = varnames>;

run;


The format of the statistics and varnames is quite rigid. These work as expected, producing the sum of AB and BB forthe relevant set of observations:

output out = out_ds

sum(AB) = Atbats sum(BB) = Walks;

However, this causes a compiling error:

output out = out_ds

sum(BB, IBB) = Walks2;


Finally, <class/by varnames> is what you use to specify how you wish to group each set of observations.

Without it, you will get the specified statistics of every observation put together.





output <out_ds> <statistic = varnames>;

run;


This code would give you the summary stats of (yearID, AB, SO) for all the data combined.

You'll get the total at-bats and strikeouts of ALL observationsin the data set, and your output will be a single row.

proc means data=batting

sum;

var AB SO;

run;


This code, however, will give you the total number of at-bats and strikeouts for EACH year.


sum;

var AB SO;

class yearID;

run;


Classes are sorted in ascending alphabetical / numeric order.Proc means also gives the number of observations for each class.

No class variable. Class yearID.


More that one variable can be used as a class variable. If thisis the case, observations will be grouped by each unique combination of the class variables, and aggregated within that group.

Class (and its pre-sorted version, 'by') may take a VERY long time to do this for a large dataset. It is HIGHLY recommended to test multiple class variables by some smaller 'toy' dataset first to avoid locking up your SAS terminal for hours.


class yearID lgid;

A variable can be one of the variables summarized AND be a class variable at the same time. The results may be

confusing though (or obvious in the case of mean/median).


sum mean;

var yearID AB SO;

class yearID lgid;

run;


The output will look like this as a table.


...and this if you use the default dataset output. output out=ds_name;

... it looks like this.Stat 342 Notes. Week 3, / 71

Next in this series, hypothesis tests!


Before we continue, let's create some derived variables for the batting average, strike-out average, and walking average of each player per year.

Each of these variables is a proportion of the at-bats that a given player experiences in a given year.

data batting;

set batting;

batavg = H / AB;

so_avg = SO / AB;

walk_avg = sum(BB, IBB) / AB;

run;Stat 342 Notes. Week 3, / 71

Also, let's limit our dataset to years and players in which at least 300 at-bats were observed. This way our derived variables are proportions of a large number of observations.

data batting_300ab;

set batting;

if AB ge 300 then output;

run;


Consider the distributions of the these three derived variables. Intuitively, they should be normally distributed. Is that the case?

Let's check the histograms with the UNIVARIATE procedure.

Including the 'histogram' statement in 'proc univariate' tells SAS to make histograms of the variables you specify.

If you don't specify any variables, it uses every variable that was listed in the 'var' statement.

Including '/normal' at the end of the histogram statement adds a fitted normal curve to the histograms created.


The 'var' statement will include every numeric variable in the dataset unless you specify otherwise.

So this code will produce summary statistics and histograms for the three derived variables we made earlier, with fitted normal curves included

proc univariate data=batting_300ab;

var batavg so_avg walk_avg;

histogram / normal;

run;


We can also check the quantile-quantile (vs normal distribution) plots for these variables, and see how well they fit.

In a quantile-quantile plot, the observations of a variable are sorted and compared to some theoretical quantiles.

For example: Say we have 1000 observations, and we want to compare the distribution of these observations to a normal distribution.

The 0.025 quantile of the normal distribution is 1.96 standard deviations below the mean. If our data is normal, then the 25th lowest data point should also be 1.96 sd below the mean.Stat 342 Notes. Week 3, / 71

Quantile-quantile plots can be produced with the 'qqplot' statement in the univariate procedure.

The '/ normal' specifies the distribution. Normal is the default. Other options include 'beta', 'exponential', 'gamma', and 'Weibull'.

proc univariate data=batting_300ab;


qqplot / normal;

run;


If the selected distribution fits the data, we should see a straight line. (It doesn't have to be perfect at the extreme ends)Stat 342 Notes. Week 3, / 71

Skewed distributions will demonstrate a single bend (vs normal).


More complicated deviations from will exhibit complexbehaviour, such as s-curves or extreme values.


Finally, we can conduct formal tests for normality by using the normal option right after we specify the input dataset.

proc univariate data=batting_300ab normal;


run;

Some normality tests have minimum / maximum sample sizes. Only the appropriate tests will be performed. (Shapiro-Wilks test,for example, won't show for N > 2000)


Batting averages

Strikeout averages

Walking averages Stat 342 Notes. Week 3, / 71

Let's go exploring for bivariate trends.


For continuous data, we can look at two variables at a time in several ways.

- Correlation

- Scatterplots

- Regression

For scatterplots, we can use the SGPLOT procedure, which standard for Statistical Graphic PLOT.


Proc sgplot is the basic procedure for a wide variety of graphing options. These include:

Scatterplots with the scatter statement.

Bar plots with the hbar and vbar statements.

Box plots with the hbox and vbox statements.

Fitted regression lines with the reg statement.

Locally fitted spline-smoothed fit with the loess statement.

Kernel-based density curves with the density statement.


For a scatterplot between strikeout average and batting (hitting) average, the code is:

proc sgplot data=batting_300ab;

scatter x=batavg y=so_avg;

run;


We can draw a 99% prediction ellipse around these data points with the 'ellipse' statement.

The default is a 95% level for confidence/prediction, including forthe ellipse, so we decrease alpha in the settings in order to make an ellipse that's farther out and more visible.


scatter x=batavg y=so_avg;

ellipse x=batavg y=so_avg / alpha = 0.01;

run;


There are tons of data points (about 19,700), so it's hard to get a sense of the relative density of data points. We can fix that with some graphical options.


scatter x=batavg y=so_avg /transparency = 0.9;

ellipse x=batavg y=so_avg / alpha = 0.01;

run;


We can do something more formal with the correlation procedure, PROC CORR.

For example, we can get the simple pairwise (Pearson) correlation between all three of our derived variables with the following:

proc corr data=batting_300ab;


run;


This will produce some simple summary stats, and the correlationcoefficient, p-value against 0 correlation (two-tailed), and number of observation pairs for each pair of variables.


We can look much deeper into this with alternative correlation measures, like Spearman's Rank Sum (ideal for non-linear, monotonic relationships), or Kendall's Tau.

proc corr data=batting_300ab

Pearson Spearman Kendall;


run;


This will produce equivalent matrices for the other measures.

If you still want the Pearson correlation as well, you have to specify it.


We can also produce scatterplots and histograms of each variableand pair of variables in a neatly-arranged matrix.

For large datasets, you may need to get around the default limit of the number of data points to draw with the maxpoints option in the plot() settings.

proc corr data=batting_550ab

plots=matrix(histogram) plots(maxpoints = 50000);


run;


Readings on week 8 material

Textbook source: Chapter 5 'SAS and R'.

Sources for your interest:

The Essential Meaning of PROC MEANS http://www2.sas.com/proceedings/sugi26/p064-26.pdf

Readings on week 9 material (next week)

Textbook source: Chapter 6 'SAS and R'.


http://www2.sas.com/proceedings/sugi26/p064-26.pdf

stat 342 - wk 8: continuous data - sfu.cajackd/stat342/lect_wk08.pdf · 2016. 12. 15. · stat 342...

Documents