emf refresher 1 of 2 - marama refresher_1 of 2... · emf refresher webinar series ... who made the...
TRANSCRIPT
9/30/2014
1
1
Overview of the EMF Refresher Webinar Series
Introduction to the EMF
Working with Data in the EMF – viewing & editing
Inventory Data Analysis and Reporting
2
EMF Resources
EMF User's Guide
http://www.cmascenter.org/emf/internal/guide.html
EPA's CoST website
http://www.epa.gov/ttn/ecas/cost.htm
CoST User's Guide
http://www.cmascenter.org/help/model_docs/emf_cost/2.5.1/CoST_UsersGuide_2012-08-01_Final.pdf
9/30/2014
2
3
Roadmap for Today's Webinar
What is the Emissions Modeling Framework (EMF)?
What can the EMF do?
What challenges can the EMF help solve?
How is data stored/managed in the EMF?
How do I get started in the EMF?
What can EMF tell me about the data?
How can I view and export data?
4
What is the EMF?
Software package built to help in emissions work
Manage emissions data files
Track changes to data files
Create summaries and reports from inventories
Perform QA tasks on inventories
Create controlled and/or future-year inventories
9/30/2014
3
5
How the EMF Works
The EMF client application runs on your local computer
The client application communicates with the remote EMF server
The remote EMF server stores the emissions data and does the number crunching
6
What Can the EMF Do?
Analyze emissions inventory data; for example:
Create inventory summary reports by source characteristics like county, SCC, plant ID
Compare inventory data from one year to another year
Check inventory records for bad values like unknown or missing SCC codes, or negative emissions values
9/30/2014
4
7
What Can the EMF Do?
The EMF includes the Control Strategy Tool (CoST) - covered in separate webinars
Estimate emissions reductions and costs associated with control strategies
Apply control measures to create controlled inventories
Apply growth factors to create future year inventories – growth factors EPA or regional
Customize inventory temporalization to your analysis
8
What Can the EMF Do?
The EMF can manage the inputs and settings needed to run emissions modeling tools like SMOKE
SMOKE converts emissions inventory data into air quality model inputs by applying allocation factors
We will not be covering this topic in this webinar series.
9/30/2014
5
9
What Problems Can the EMF Help Solve?
How do we organize all the various files that are used?
How can everyone see the data that's available and find the data that they need?
How do we track changes? Who made the change, what was changed, and why was it changed.
When updates are made, how do we make sure we use the latest version of the data?
10
What Problems Can the EMF Help Solve?
How do we share information with others?
How do we ensure that created output is reproducible?
How do we keep track of the analysis and QA that's been done for a particular file?
9/30/2014
6
11
Poll: Which of these is notsomething the EMF can do?
a) Keep track of all the inventory files for a project
b) Create a summary of an inventory by county and SCC code
c) Project an inventory from 2011 to 2030
d) Create a summer weekday inventory from an annual inventory
12
How is Data Stored in the EMF? Data files are imported into the EMF to create datasets
During import, the EMF reads the raw data from the file into the EMF database
When you import a file, you tell the EMF what dataset type to use
The EMF knows the format of the data file based on the dataset type
Column names and what kind of data (string, integer, decimal, etc.) is in each column
Data to read must exactly match the dataset type column definition
9/30/2014
7
13
Kinds of Data Managed by the EMF
Emissions inventories
Measured or estimated emissions by source
Emissions data can have different temporal resolutions like annual, monthly, daily, hourly, or seasonal
Data can be base year emissions, future year emissions, and/or controlled emissions
The EMF can read several emissions inventory files in different formats
Knows about the different types of emissions sources like on-road mobile, nonroad, nonpoint, and point
14
Inventory Dataset Types in the EMF
Flat File 2010 [Nonpoint|Point|Activity]
Inventories containing annual or month-specific emissions in different FF10 formats
Flat File 2010 [Nonpoint|Point] Daily
Day-specific inventories in FF10 format
ORL [Fire|Nonpoint|Nonroad|Onroad|Point] Inventory
Annual inventories in different ORL formats
9/30/2014
8
15
Kinds of Data Managed by the EMF
Allocation factors used to calculate more finely detailed estimates from the original emissions inventory data; for example:
Gridding surrogates: factors to allocate county-level emissions to grid cells
Speciation profiles: factors to split emissions by pollutant into chemical mechanism species
Temporal profiles: factors to estimate hourly emissions based on the original temporal resolution of the emissions inventory data
16
Kinds of Data Managed by the EMF
Growth factors used to project a base year inventory to a future year
Control measures to model control technologies
Effectiveness information for the control technology (i.e. control efficiency, rule penetration)
Factors to calculate the engineering costs associated with implementing particular measures
9/30/2014
9
17
Kinds of Data Managed by the EMF
Reference data, including:
State and county codes and names for reporting emissions summarized by geographic area
County latitude and longitude coordinates that can be included in output so data can be visualized using GIS tools
Names for codes like SCC, SIC, MACT, NAICS, etc.
18
Poll: We talked about several kinds of data. Which didn't we discuss?
a) Seasonal point source emissions
b) Hourly temperature data
c) Factors to estimate gridded emissions from county-level emissions
d) Control measure efficiencies
9/30/2014
10
19
Starting Up the EMF
Each time you launch the EMF client you will be prompted to login with your username and password
The first time you start the EMF client you'll need to create a username and password and enter your profile information
20
Main EMF Window
9/30/2014
11
21
EMF Windows The Status window will display information
about tasks the EMF is running
The Downloads window will be used when downloading data or reports from the EMF
22
The Dataset Manager The Dataset Manager is the main interface for
listing the datasets that have been imported into the EMF and finding the dataset you want to work with
To open the Dataset Manager, open the Manage menu at the top of the main EMF window and select Datasets
Initially, no datasets are listed; for now, we'll use the Show Datasets of Type pull-down menu to select “All”
9/30/2014
12
23
The Dataset Manager
24
The Dataset Manager
The Dataset Manager displays a summary of information about each dataset
aka Dataset Properties
The Dataset Manager lets you select and work with datasets
In the homework, you'll explore a few datasets and get familiar with the EMF user interface
9/30/2014
13
25
Viewing the Properties of a Dataset
Select a dataset by checking the checkbox in the Select column next to the dataset name
Click the Edit Properties button to open the Dataset Properties window
The Summary tab will be displayed (see next slide)
26
Dataset Properties: Summary Tab Name: nonroad_caps_2005v2_jul_orl_nc
Description: header of inventory file
Project: assigned when dataset was imported
Creator: user who created the dataset
Dataset Type
Time Period Start and End: determined from file header
Temporal Resolution: determined from dataset type
Sector: set by dataset type
Region: assigned by user
Country: determined from file header
9/30/2014
14
27
Dataset Properties: Summary Tab Status: imported from file vs. created by
EMF
Dates last modified, accessed, and created
Intended Use: public vs. private
Default Version: will discuss in a moment
28
Poll: Which of these properties is not stored for datasets?
a) Date the dataset was created
b) The user who created the dataset
c) Temporal resolution
d) Spatial resolution
e) Geographic region the dataset covers
9/30/2014
15
29
Viewing vs. Editing
Viewing an item: looking at the item but can't change its information
Editing an item: can change information about the item
Only one user can edit a dataset at a time but multiple users can view a dataset
You must edit a dataset in order to change anything about it including adding or running QA steps (we'll discuss more later)
30
How to Find Datasets Use the Show Datasets of Type pull-down menu in
the Dataset Manager
This pull-down menu does have an “All” option to see all the datasets in the system
9/30/2014
16
31
How to Find Datasets
If you have a lot of datasets of a given type, use the Name Contains field
You can type in a string to match the names of datasets
For example, typing 2020 will match all datasets whose name contains “2020”
You can use wildcards in this field: pt*2020 matches datasets whose name contains “pt” followed by “2020”at some point
ptnonipm_2020cc_nc_sc_va_09nov2009_v0
32
How to Find Datasets
Advanced button displays the Advanced Dataset Search window
Match datasets by name, description, creator, project
Find datasets by QA step or data values
Datasets must match all criteria specified (boolean AND)
9/30/2014
17
33
How to Find Datasets
Another option is to filter the list of datasets in the Dataset Manager
Click the Filter toolbar button to display the Filter Rows dialog
Create a filter to only show rows where the Type column contains the word “Inventory”; note that filter criteria are case-sensitive
34
How to Find Datasets
Edit your user profile to limit the list of dataset types that the EMF shows you
In the main EMF window, open the Manage menu and select My Profile to see the Edit User window
In the Edit User window, you'll see a list of Visible Dataset Types and a list of Hidden Dataset Types
Select visible dataset types and click the Hide button to move the selected types to the hidden list
9/30/2014
18
35
How to Find Datasets
36
Poll: How can you find datasets by name?
a) Advanced dataset search
b) Create criteria to filter the rows displayed in the Dataset Manager
c) Enter a search term in the Name Contains field in the Dataset Manager
d) All of the above
9/30/2014
19
37
Dataset Properties: Data Tab
Provides access to the raw data associated with the dataset
Shows information about each version of a dataset
Click View to view the raw data (will be covered in more detail later)
38
Dataset Properties: Keywords Tab
Keywords store additional metadata about datasets
The keywords pass info to the EMF to be used for certain processing like including prefixes or suffixes when exporting a dataset or setting up indexes of the raw data
9/30/2014
20
39
Dataset Properties: Notes Tab
Shows notes that have been made about the dataset
Notes can be questions, answers, comments, etc.
Create a new note by clicking the Add button
40
Creating a Dataset Note
Enter a summary of your note
Select the note type and enter the details
If your note references a previous note (like answering a previous question), select the referenced note
Select which version of the dataset your note refers to
9/30/2014
21
41
Dataset Properties: Revisions
Displays information about revisions that have been made to the raw data in the dataset
Revisions are automatically created when the raw data is edited
42
Dataset Properties: History
Displays a record of when the dataset was exported for use in modeling runs
9/30/2014
22
43
Dataset Properties: Sources
Displays information about where the data associated with this dataset came from (i.e. the original file that was imported)
Also shows which table the raw data is stored in in the database
44
Dataset Properties: QA Steps
Shows a list of QA steps that have been created for this dataset
QA steps represent different actions performed like summarizing the data by county and pollutant or creating a list of unique SCCs in the data
Allows you to add, edit, copy, and delete QA steps
We will discuss QA steps in the next webinar
9/30/2014
23
45
Poll: True or False?
1) You can't view a dataset while someone else is editing it.
2) Every dataset in the EMF uses at least one table in the database to store its raw data.
3) Dataset notes are associated with a particular version of a dataset.
46
Dataset Actions
Refresh: checks with the EMF server and gets the latest information for the dataset
Other users can be editing datasets while you're working so you may need to refresh to see their latest changes
Save: save the changes you've made to a dataset
Export: export the raw data to a file (next slide)
Close: close the Dataset Properties window; if you have unsaved changes, you'll be asked if you want to discard them
9/30/2014
24
47
Exporting Datasets
Exporting a dataset creates a file containing the data
The exported file format is determined by the dataset's type, i.e. an FF10 Nonpoint Inventory dataset will be exported in FF10 Nonpoint format
In the Dataset Manager window, select the datasets you would like to export and click the Export button to display the Export customization window
48
Exporting Datasets
9/30/2014
25
49
Exporting Datasets
Check the Download files to local machine?Checkbox
Creates file on the EMF server that will be automatically downloaded to your computer
The file name is generated from the dataset name
You can enter a file name prefix to add to the file name
If the dataset has multiple versions, you can choose which version you would like to export
Exporting Datasets
• Choose datasets to export
• Download files to local machine? CHECK
• Optional File Name Prefix
• Choose Version
• Filter to limit rows exported
50
9/30/2014
26
51
Exporting Datasets
In the Export dialog, click the Export button to begin the export
Check the Status window for messages:Completed export of SmokeFlatFile_NONPOINT_20140912.csv to
C:\EMF_State_Data\downloads\StateEMF\SmokeFlatFile_NONPOINT_20140912_csv_27sep2014_v0 in 13.104 seconds. The file will start downloading momentarily, see the Download Manager for the download status. No. of records in database: 9,389,860; Exported: 60,460
52
Exporting Datasets In the main EMF window, open the Window menu and
select Downloads to open the Download Manager
Once the progress bar reaches 100%, your download is complete
Right click the filename and choose Open Containing Folder to open the folder where the file was downloaded
9/30/2014
27
Exporting Datasets Local Machine Download
TIP: Re-name with .csv extension to easily open as a spreadsheet
53
Exporting DatasetsOpening as a Spreadsheet
54
9/30/2014
28
Coming up…
• Homework: links to be provided in follow-up email
• EMF Refresher, Part 2:
• October 21, 1:30 – 3:00
• How do I QA and summarize inventory data?
• What is a QA step?
• Creating a QA step: new and from templates
• Comparing datasets
• Exporting data in KMZ format to view in Google Earth
55
Preview of EMF Refresher, Part 2QA step results:• What are the statewide totals by pollutant?• For each pollutant, which counties and
SCCs?• EMF bonus – SCC descriptions
56