msquery-dm ad hoc reporting from sql server 9/25/2013 msquery for... · msquery-dm ad hoc reporting...

MSQuery-DM ad hoc reporting from SQL Server 9/25/2013

1

The ad hoc report feature in the Digital Measures web application has limited options for selecting the rows and columns to be reported. An alternative is to use MSQuery, or some other desktop tool such as SAS, against a copy of DM data updated weekly on SQL Server. For more complex reports, a custom DM report should be requested. The database is DigitalMeasures on server ardbprodone.ad.uky.edu. Contact Diane Gagel if you have questions about your query. The query script can be sent via email and pasted into the MSQuery tool. MSQuery selections should be saved for reuse. MSQuery is a graphical tool to be used with relational databases for building Standard Query Language (SQL) select statements. Select statements identify the table or view (in our case DM screen), the columns (fields on screen) and the where clause (the criteria for selection). Using Excel MS Office 2007 (MS Office 2010 should be similar): Open MS Excel Data -> From Other Sources -> From Microsoft Query

After choosing Microsoft Query, you will be prompted to Choose Data Source, there are 3 tab options: 1) Databases: Select the database for creating a new query 2) Queries: Select from existing queries to run, change or copy 3) OLAP Cubes None currently available for DM data


2

Option 1 - To create a new query, under the Databases tab, choose the data source called “DMCopySQLServer”. {The data source must be created first – see DSN creation instruction handout.} Recommended: uncheck “Use the Query Wizard to create/edit queries”

Click OK. You may prompted to login to the DigitalMeasures SQL Server database. Depending on the authentication specified in the DSN setup, you will use either the LinkBlueID (recommended set up for DM), or a SQL Server ID. When logging on, never choose the option to save UserID and Password Option 2: to use an existing query, under the Queries tab, scroll to find the query saved

previously, select the query and click Open. TIP: if you want to create a query based on the copy of an existing query, open the existing query and then “Save As” with a different name before making the changes.


3

MS Query “graphical interface” for specifying selection criteria and viewing results:


4

Creating a new query from scratch, with the MSQuery wizard disabled. 1) Select the table (view) on which the query will be based. After selecting the database, you should be prompted with the Add Tables dialog, to select the tables or views on which the query will be based . Use only the entries that begin with CollView_. These views were specifically created for college ad hoc reporting. You will not have access to data in other tables or views. Scroll to the desired table, and double-click the name, or select and click the Add button:

This creates a table field-selection box in the “tables” area of the work window:

Click the Close button to exit the Add Tables dialog window. The Add Tables dialog can be initiated from the menu, Table -> Add Tables... Important Tips: Tip 1a. If more than one table is selected, join criteria must be specified or else the result set could be very large and will likely not be the desired result set. Define the query criteria one table at a time. If more tables are needed, add them one at a time. This will simplify creating joins and verifying the results.


5

Tip 1b. Before choosing the first column in the first selected table, uncheck the Automatic Query option in the Records menu. This will prevent the query from being executed with every new addition to the selection criteria or result set. NOT RECOMMENDED: RECOMMENDED: Automatic Query checked Automatic Query unchecked

To delete a table from selection: If the wrong table is selected it can be removed once the Add Tables dialog is closed, by selecting the table in the Table area.

Select columns to include in the results area, by double-clicking the column name in the table. If the asterisk is double-clicked, all columns are copied in, which is not always desirable.


6

2) Specify the query criteria to limit the result set. In most cases you will want to specify selection criteria, otherwise all the rows in a table will be returned. . From Criteria menu select Add Criteria.

Using Add Criteria dialog window, select the field, operator and value. It is NOT RECOMMENDED to use the Values button with the DM views. The values button attempts to find all possible values for a field and can require a significant amount of time to return the results. Use the DM data dictionary to find the possible values (or the DM entry screen could also provide valid values). TIP: For best performance for college level queries, always include the AuthCollCode as a criterion and specify the appropriate college.

Click Add button to add the criteria to the criteria area:


7

Specify another criterion:

Click Add button:

Continue to add criteria with the Add Dialog, note that MSQuery defaults the logical connection operator as And.

The screen shot below shows the criteria area appearance when the “and” operator is used: AuthCollCode = ‘NU’ AND Status = ‘Published’ AND DTY_PUB = ‘2009’. The results include publications where all are true.


8

The screen shot below shows the criteria area appearance when the “or” operator is used: Status = ‘Published’ or DTY_PUB = ‘2009’. The results would include publications where either is true (resulting in all published items and no restriction by year.)

Use the Records menu Query Now function to run the query and populate the results area.

Select the “Last Record” icon circled below in green, to scroll to the last record, and show the record count. It is good practice to check the record count on the query results to be sure you’re in the ball-park expected count, and to double-check results after changes have been made to the criteria.


9

Saving the Query After the query specifications have been created or changed to yield the desired results, Save the query using the File -> Save option (for new query or to update an existing query). Use the Save As option to save an existing query to a new copy. When naming queries, use a name that describes the query, for example “DM IntellContr Abstracts for UserID for Year”

MSQuery files use the extension .dqy. Accept the windows default location for storing the query which will be some place under the Desktop -> user directory -> AppData -> Microsoft -> Queries. It is not readily apparent how to permanently set the location in MSQuery. When you get a new desktop, remember to retrieve copies of MSQuery files.


10

Saving the results to MS Excel Use File -> Return Data to Microsoft Office Excel to save the data to an excel spreadsheet.


11

DM View contents Refer to the DM Data Dictionary on the APFA Digital Measures page to identify the column names associated with the fields on the DM screens, and for the valid values for the selection criteria. All views have the following fields in common: Field Name Description AuthCollCode Authorization college code – the college to which the

faculty member is or was once appointed. This field controls access to the data for ad hoc reporting.

ConID Contribution ID is a unique identifying number for the DM entry. (The field is called “ID” in the DM data dictionary.)

DMUserID Internal DM ID number of the faculty member to which the information belongs. (The field is called “User_ID” in the DM data dictionary.)

DMUserName Faculty member’s LinkBlue ID DispNameLastFirst Faculty member’s name Last, First and Middle Initial FNAME First name from Personal Contact Information screen PFNAME Preferred first name from Personal Contact Information

screen MNAME Middle name from Personal Contact Information screen LNAME Last name from Personal Contact Information screen NAME_PREFIX Name prefix from Personal Contact Information screen NAME_SUFFIX Name suffix from Personal Contact Information screen CREDABBR Credential abbreviation from Personal Contact Information

screen Additional common fields for tables with Shared Data (Intellectual Contributions, OSPA grants, etc.) USER_REFERENCE_CREATOR Yes indicates that the item was initially entered by this

faculty member, and all other faculty associated with the item share the single DM entry. DM entry protocol indicates that primary contributor should enter the item. To obtain unique items, specify in criteria: USER_REFERENCE_CREATOR = ‘Yes’

AUTHOR_LIST or contributor list System calculated list of contributors associated with an entry


12

View Name DM Screen Name CollView_AwardHonor Awards and honors CollView_Consulting Consulting CollView_Education Education CollView_EndAppt Endowed professorship and chair

appointments CollView_FacDev Faculty development activities CollView_IntellCont_AllContr Intellectual Contributors – multiple entries for

all contributors CollView_IntellProp_AllContr Intellectual Property – multiple entries for all

contributors CollView_InterestExpert Interests and Expertise CollView_MedCont Media contributions CollView_Member Professional Memberships CollView_PCI Personal and Contact Information CollView_PerformExhibit_AllContr Artistic and Professional Performances and

Exhibits CollView_Presentations_AllContr Presentations – multiple entries for all

contributors CollView_Presentations_AllContr_UniqueTitles

Presentations unique titles by faculty user – multiple entries for all contributors. This is a special view which eliminates duplicate presentations with the same title, and provides the most recent start date.

CollView_Res_NonOSPA Research (Non-OSPA) CollView_Res_OSPA Research (OSPA) CollView_ServColl Service to College CollView_ServDept Service to Department CollView_ServProf Service to Profession CollView_ServPubl Service to Public CollView_ServUniv Service to University CollView_YearlyData Yearly Data (academic appointment status for

calendar year Jan-Jun or Jul-Dec Scheduled Teaching, DOE data, and other views are not currently provided, since data from these screens is reportable (and in more detailed form) in the institutional data warehouse.


13

DM screen field names vs DM internal field names: The field names visible on the DM screens are “labels”, and are not the true field names in the DM database tables. Refer to the DM Data Dictionary available from the APFA Digital Measures page. As illustrated in the screen shot below, the data dictionary provides the field name (Code), whether the field is “coded” (Display = “Drop Down”, and the valid codes (Options). The exact spelling of the option values must be used in the selection criteria: Example: On Intellectual Contributions screen, the Current Status field is called “STATUS” in the data dictionary, and the valid values are

In Preparation; Not Yet Submitted Working Paper Submitted Revising to Resubmit Not Accepted Accepted Published

To select in preparation items, the selection criteria would include:

STATUS = “In Preparation; Not Yet Submitted”


14

DM Date fields: Some activity items occur on a specific date, others may take place over a period of time, and therefore DM entry requires (respectively) a single date or date range. In addition, DM allows for partial date entry of either: the year only; the month and year; or month, date, and year. In the DM Data Dictionary, each component of a date field on the screen is stored in a separate month, day or year field. Single Date Example: Intellectual Contributions, Date Published

In the DM Data Dictionary, the following fields are present under Date Published: DTM_PUB Month Published – stored as full name of month or quarter – January (1st

Quarter/Winter), February, March, etc. DTD_PUB Day Published – stored as 01, 02, 03, DTY_PUB Year Published – stored as 2001

To support selection of data by partial dates, each date field on the screen has an associated start-date and end-date pair in the data table, to store the effective period of the partial date. These start and end date pairs are not listed in the DM Data Dictionary, but are present in the data tables in SQL Server. PUB_START DM calculated publication start date, stored as yyyy-mm-dd PUB_END DM calculated publication end date, stored as yyyy-mm-dd Single Date Example: (continued)

Date entered on DM Screen

Internal Calculated (yyyy-mm-dd)

DTM_PUB DTD_PUB DTY_PUB PUB_START PUB_END 2009 2009-01-01 2009-12-31 January (1st Quarter/Winter) 2009 2009-01-01 2009-01-31 January (1st Quarter/Winter) 05 2009 2009-01-05 2009-01-05


15

Date Range Example: Consulting, Start and End Dates

In the DM Data Dictionary, the following fields are present under Start Date: DTM_START Month Started – stored as full name of month (no quarters) – January,

February, March, etc. DTD_START Day Started – stored as 01, 02, 03, DTY_START Year Started – stored as 2001 In the DM Data Dictionary, the following fields are present under End Date: DTM_END Month Ended– stored as full name of month (no quarters) – January,

February, March, etc. DTD_END Day Ended – stored as 01, 02, 03, DTY_END Year Ended – stored as 2001

In the DM tables the following calculated dates are present: START_START DM calculated item start date start, stored as yyyy-mm-dd START_END DM calculated item start date end, stored as yyyy-mm-dd END_START DM calculated item end date start, stored as yyyy-mm-dd END_END DM calculated item end date end, stored as yyyy-mm-dd

Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_START DTD_START DTY_START START_START START_END 2009 2009-01-01 2009-12-31 January 2009 2009-01-01 2009-01-31 January 05 2009 2009-01-05 2009-01-05

Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_END DTD_END DTY_END END_START END_END 2010 2010-01-01 2010-12-31 June 2010 2010-06-01 2010-06-30 June 30 2010 2010-06-30 2010-06-30


16

When specifying selection criteria for dates, consider the usage directions indicated in the “DM Data Entry Guidelines” document, per example below.

Date selection criteria examples: Screens with single dates: a) Select all items published in calendar year 2011:

When selecting by calendar year or years, it is simplest to use the “DTY_” field. ….AND DTY_PUB = ‘2011’

Multiple years: ….AND DTY_PUB in (‘2010’, ‘2011’)

However, the PUB_END calculated date field could also be used: ….AND PUB_END between ‘2011-01-01’ and ‘2011-12-31’

Multiple years:

….AND PUB_END between ‘2010-01-01’ and ‘2011-12-31’


17

b) All items published in fiscal year 2011.

Fiscal year selection requires consideration of the month, therefore need to use the internal calculated date fields PUB_START and PUB_END

Reminder how PUB_START and PUB_END are populated: If no month is provided, the PUB_START and PUB_END span an entire calendar year.

Date entered on DM Screen

Internal Calculated (yyyy-mm-dd)

DTM_PUB DTD_PUB DTY_PUB PUB_START PUB_END 2011 2011-01-01 2011-12-31 January (1st Quarter/Winter) 2011 2011-01-01 2011-01-31 January (1st Quarter/Winter) 05 2011 2011-01-05 2011-01-05

Which dates are used impacts what results will be returned: Using PUB_START and PUB_END will result in double-counting publications with no publication month information:

….AND ((PUB_START GE ‘2010-07-01’) AND (PUB_END LE= ‘2011-06-30’))

Using the selection above, for fiscal year 2011 (7/1/2010 thru 6/30/2011), the publication would be selected based on the PUB_START date of 1/1/2011.

Using the selection above, for fiscal year 2012 (7/1/2011 thru 6/30/2012), the publication would be selected again based on the PUB_END date of 12/31/2011.

Using PUB_END date only, will result in single-counting publications with no publication month information:

….AND ((PUB_END between ‘2010-07-01’ and ‘2011-06-30’))

Using the selection above, the publication would be counted only in fiscal year 2012


18

Screens with date ranges: Selections are more complex to accommodate items which may occur on a single day, or may be active for an extended period, or open-ended. Again – consider the data entry instructions for the screen. These examples use the Consulting table, which has the following date entry instructions: *Start Date (blank if one day) *End Date (specify if one day) a) Select consulting entries which occurred in calendar year 2011:

Use DTY_END field to find items which occurred on a single date in the period or a continuous activity which ended in the year

Use DTY_START and DTY_END to find items which are still active, but started on or before the selection year

The OR operator is required because either expression will satisfy the criteria. Using the AND operator would select only items ending in 2011.

….AND ( (DTY_END = ‘2011’) OR

((DTY_START between ‘1900’ and ‘2011’) AND DTY_END = ‘ ‘))

The calculated full date fields could also be used:

….AND (((END_START >= ‘2011-01-01’) AND (END_END <= ‘2011-12-31’))OR

((START_START between ‘1900-01-01’ and ‘2011-12-31’) AND END_END = ‘ ‘))


19

b) Select consulting entries which occurred in fiscal year 2011:

Fiscal year selection requires consideration of the month therefore need to use the internal calculated date fields START_START and END_END.

Use END_END date to find items which occurred on a single date in the fiscal year or which were continuous activities which ended in the fiscal year

Use START_START and END_END to find items which are still active (have no End Date, but started on or before the selection fiscal year

The OR operator is required because either expression will satisfy the criteria. Using the AND operator would select only items ending on 2011.

….AND ( (END_END between ‘2010-07-01’ and ‘2011-06-30’) OR (( START_START between ‘1900-01-01’ and ‘2011-06-30’) AND

END_END = ‘ ‘ ) )


20

Selection Criteria with OR operator In SQL syntax, the AND operator takes precedence over the OR operator. Parenthesis are used to control how the OR’d fields are evaluated to return the appropriate results.

Example: retrieve all journal articles for the faculty with linkblueID of DLDAVE0 published or submitted in calendar year 2011. The following are equivalent SQL statements:

Form A) select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where DMUserName = ‘DLDAVE0’ and CONTYPE like ‘Journal Article%’ and (DTY_PUB = ‘2011’ OR DTY_SUB = ‘2011’)

Form B) select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where (DMUserName = ‘DLDAVE0’ and CONTYPE like ‘Journal Article%’ and DTY_PUB = ‘2011’) OR (DMUserName = ‘DLDAVE0’ and CONTYPE like ‘Journal Article%’ and DTY_SUB = ‘2011’)

When the OR operator is involved, MSQuery builds SQL script using Form B and graphically represents the criteria grouped in parenthesis with two lines with the “or” indicator to the left.

It is important to remember to include all criteria on each line, if it should be evaluated together to produce desired results. For example, the DM selection criteria graphically represented below, without the fully populated second line, does not produce the same results as above.

Instead, it is equivalent to the SQL statement …

select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where (DMUserName = ‘DLDAVE0’ and CONTYPE like ‘Journal Article%’ and DTY_PUB = ‘2011’) OR (DTY_SUB = ‘2011’)

… which would produce a list of journal articles published in 2011 for DLDAVE0, appended to a list of all intellectual contributions for all faculty submitted in 2011.


21

SQL Script View To see the SQL select statement produced by your specifications, go to the View menu

A SQL Dialog box will open with the SQL script.

Individuals proficient in building SQL select statements may wish to edit the script directly. It is cumbersome to type script in the dialog box, which cannot be resized. For significant changes, or for new scripts, it is recommended that the script be typed in notepad or some other text editor and then copied into the SQL dialog box. The DM technical support person can send DM script via email for copying into the dialog box. Important notes: 1) When scripts are pasted into the dialog box, the unchecked state of the Automatic Query option is ignored, and MSQuery attempts to retrieve the information immediately upon selecting OK on the SQL Dialog window. If the where clause does not limit the rows sufficiently, this could cause MSQuery to give the “Not Responding” message. When this occurs, the task should be ended in the Task Manager. 2) SQL scripts with aggregation items in the select clause cannot be represented graphically. This means that the table and selection criteria areas cannot be represented by MSQuery, but the query is still good. The results should be returned and can be saved to Excel.


22

SQL Expression examples To select a set of values use In() STATUS in (‘Accepted’, ‘Submitted’, ’Published’) To select a range of values use between START_START between ‘2011-01-01’ and ‘2011-12-31’ To select with wildcard, use like and the % CONTYPE like ‘Journal Article%’ - starts with Journal Article CONTYPE like ‘%Journal Article’ - ends with Journal Article CONTYPE like ‘%Journal Article%’ - Journal Article anywhere in text To exclude a set of values, use NOT ( ) NOT (STATUS = ‘Accepted’) To select unique published new books, excluding book chapters:

Given these CONTYPE values for books: Book Book Review Book, Chapter Book, Chapter in Non-Scholarly Book-New Book, Chapter in Non-Scholarly Book-Revised Book, Chapter in Scholarly Book-New Book, Chapter in Scholarly Book-Revised Book, Chapter in Textbook-New Book, Chapter in Textbook-Revised Book, Non-Scholarly-New Book, Non-Scholarly-Revised Book, Scholarly-New Book, Scholarly-Revised Book, Textbook-New Book, Textbook-Revised SELECT field names ….. FROM CollView_IntellCont_AllContr where AuthCollCode = 'NU' and STATUS = 'Published' and CONTYPE like ‘Book%’ and not(CONTYPE like ‘Book, Chapter%’) and not(CONTYPE like ‘%Revised%’) and not(CONTYPE like ‘Book, Review%’) and USER_REFERENCE_CREATOR = ‘Yes’


23

Joins

Should generally not be necessary, unless information from the two screens below is required in conjunction with productivity (activity) information:

CollView_PCI Personal Contact Info 1 Row per faculty member CollView-YearlyData Academic Appointment Multiple rows per faculty member

Joins will be covered at the next session. Avoid for now – or ask for scripts from the DM Technical Support person. Joins will be based on the key field DMUserName – which ties the faculty member’s activity data with the YearlyData (academic appointment) Example, list all intellectual contributions published for faculty in academic year 2012, along with title series, rank, and department.


24

Notes about Yearly Data:

A faculty member will have two Yearly Data records in a calendar year to capture academic status information for first half of year (01 Jan-Jun) and the second half of year (02 Jul-Dec).

Since DM does not support using exact effective dates on the Yearly Data record, this design allows for capturing the status at various two points in the calendar year to support fiscal year reporting.

Yearly data rows for one faculty member, who started employment 8/16/2011: Calendar Year (TYY_TERM)

Time Period (TYT_TERM)

Internal Date TERM_START

Internal Date TERM_END

2011 02-JulDec 2011-07-01 2011-12-31 2012 01-JanJun 2012-01-01 2012-06-30 2012 02-JulDec 2012-07-01 2012-12-31 2013 01-JanJun 2013-01-01 2013-06-30 2013 02-JulDec 2013-07-01 2013-12-31

The academic appointment status effective on the TERM_END date is stored with each row.

Results sets joining with Yearly Data, therefore may have duplicate activity rows depending on the selection on Yearly Data.

Simplest options to deal with duplicates: a) select the Yearly Data for the last period in the selection

or b) add distinct option and remove TERM_START and TERM_END from select list, this will result in one row per activity, unless the title series, rank, or some other academic status has changed for the individual.

msquery-dm ad hoc reporting from sql server 9/25/2013 msquery for... · msquery-dm ad hoc reporting...

Documents