SMS 11.1 Tutorial Observation

Objectives Observation points will be created to measure the numerical analysis with measured field data and calibration. Observation arcs will be set to view the results of cross-sections or river profiles.

Prerequisites • Overview Tutorial

Requirements • FESWMS or Generic Mesh • Map Module • Mesh Module

Time • 60-75 minutes

v. 11.1

1 Introduction An important part of any computer model is the verification of results. Surface water modeling is no exception. Before using a surface water model to predict results, the model must successfully simulate observed behavior. Calibration is the process of altering model input parameters (within an accepted range) until the computed solution matches observed field values (or at least as well as possible). SMS contains a suite of tools in the Observation Coverage to assist in the model verification and calibration processes. The observation coverage consists of Observation Points and Observation Arcs, which help analyze the solution for a model. Observation points can be used to verify the numerical analysis with measured field data and calibration. They can also be used to see how data changes through time. Observation arcs can be used to view the results for cross sections or river profiles. This tutorial is based on a FESWMS finite element model, but the calibration tools in SMS can be used with any model. Note: If you do not have the FESWMS interface enabled, generic solutions will be provided to complete this tutorial. Follow the instructions labeled Generic Module Instructions where applicable.

2 Opening the Data To open the FESWMS simulation and solution data:

1. Select File | Open….

2. Open the file observe1.sms from the Data Files Folder for this tutorial. If you still have geometry open from a previous tutorial, you will be asked if you want to delete existing data. If this happens, click the Yes button.

Generic Module Instructions:

1. Select File | Open

2. Open the files observe1.2dm and observe1.h5. These are the mesh and FESWMS data files, respectively.

3 Viewing Solution Data An initial solution has already been created with this data file and was opened with the project. When the solution file is opened into SMS, various scalar and vector data sets are

created. By default, the active data sets are the velocity mag scalar data set and the velocity vector data set. Several display options should be changed. To do this:

1. Right click on the Mesh Data object in the Project Explorer and select Display Options.

2. Click the All off button and then turn on the Contours, Nodestrings and Mesh boundary options.

3. Click OK to exit the Display Options dialog.

After setting the display options, the mesh data will appear as shown in Figure 1.

Figure 1 The mesh contained in observe1.sms

4 Creating an Observation Coverage The calibration tools utilize observation features in an observation coverage. To create an observation coverage:

1. Click on the “default coverage” in the Project Explorer to make it the active object.

2. Right click on the “default coverage” and select Rename. Change the name to “calibration data”.

3. Right click on the coverage again and select Type. Change the type to Generic -> Observation.

The Observation Coverage dialog can now be used to specify what data to use in calibrating the model and to edit observation points and arcs. To bring up the Observation Coverage dialog:

1. Select Feature Objects | Attributes….

5 The Observation Coverage In this tutorial, observation points will be used to calibrate the model; however, observation arcs or a combination of arcs and points can be used instead depending on the data collected in the field. Observation arcs work similar to observation points. Differences will be pointed out as the tutorial proceeds. The Observation Coverage dialog can show the attributes for either observation points or observation arcs, but not both at the same time. The Feature Object combo box (in the upper right corner) determines which attributes are currently being shown in the Observation Coverage dialog. The upper spreadsheet is called the Measurements spreadsheet and the lower spreadsheet is called the Observation Objects spreadsheet. The titles of these spreadsheets change depending on what is selected as the feature object. Right now, the title of the Measurements spreadsheet is simply “Measurements” and the title of the Observation Objects spreadsheet is “Observation Points.” Select arcs as the feature object and the titles of the Measurements and Observation Objects spreadsheets will change to “Flux Measurements” and “Observation Arcs,” respectively. Before continuing, it should be pointed out that observation points use single values measured in the field such as velocity and water surface elevation to calibrate the model. On the other hand, observation arcs use fluxes that have been computed across the arc to calibrate the model. Therefore, measurements for observation arcs are called “Flux Measurements.”

5.1 Creating a Measurement By default, when the Observation Coverage dialog is first opened, a Measurement does not exist. A measurement represents the solution data that is compared to the observed field data in the calibration process. For observation points, a measurement is tied to either a scalar or a vector data set. This data set is unique to the measurement and cannot be tied to another measurement. For observation arcs, a measurement is tied to both a scalar and a vector data set. Again, this combination of data sets is unique to the measurement. In addition to a unique Name and Data Set(s), two other parameters are used to define the data represented by a measurement: Trans and Module. When analyzing data that varies

through time, select the Trans toggle. The Module of a measurement refers to the SMS module where the computed data is stored. To create a new measurement:

1. Make sure points is selected as the feature object.

2. Type “Velocity” as the Name of the measurement.

3. Select velocity as the Data Set (not velocity mag).

Now that a measurement has been defined, observation points can be created and edited.

6 Creating an Observation Point Observation points are created at locations in the model where the velocity or water surface elevation has been measured in the field. The measured values will be compared with the values computed by the model to determine the model’s accuracy. In addition to being assigned a Color and a Name, each observation point is assigned the following data:

• Location. The x, y real world location of the point needs to be specified. Observation arcs do not have these location attributes since several points define an arc.

• Observed value. The observed value is the value that was measured in the field corresponding to the active measurement.

• Interval. The interval is the allowable error (±) between the computed value and the observed value. Model verification is achieved when the error is within the interval (±) of the observed value.

• Angle. When a measurement for observation points is tied to a vector data set (as is the case with the Velocity measurement created in the previous section) an angle needs to be specified. This angle is an azimuth angle with the top of the screen representing north when in plan view.

Name x [ft] y [ft] Angle Observed Value [fps] Interval [fps]

Point 1 190 -369 0.0 3.5 0.25

Table 1 Observation point values

1. To create an observation point and set the appropriate information, enter the values shown in Table 1 for the X coordinate, Y coordinate, Observed Value, and

Interval. Turn on the Observe toggle for this point. When the Observe toggle for a point or arc is on, it is said to be Observed.

2. Click OK to close the Observation Coverage dialog.

An observation point has now been created at the location specified in the Observation Coverage dialog. In this case, the model will be verified if the computed value is ± 0.25 fps of the observed velocity, or between 3.25 and 3.75 fps. The Observation Points spreadsheet will always end with a blank line for the creation of additional points. (Note, there will be no blank line in the Observation Arcs spreadsheet since arcs cannot be created while in the Observation Coverage dialog.) No angle has been specified for this point. This angle can be specified in the Observation Coverage dialog or in the Graphics Window. If the point is not displayed, turn on the display of points:

1. Go to the Display Options dialog by selecting the Display menu then Display Options.

2. Under the Map menu, turn on the points display by selecting the checkbox next to Points. A point with an arrow pointing up will appear in the Graphics Window. A calibration target is drawn next to the point. Click OK to close the Display Options dialog.

To specify the angle in the Graphics Window:

1. Choose the Select Feature Point tool from the Toolbox.

2. Zoom in and rotate the point arrow approximately 120º by dragging the end of the arrow clockwise. Do not worry if this angle is not exactly 120º. The arrow just needs to be pointing in the general direction the velocity meter was set up in the field. This is usually in the direction of flow. Figure 2 shows a close-up of Point 1 with the arrow pointing up (0º angle) and then the position of the arrow at an angle of approximately 120º.

Figure 2 “Point 1" with an arrow angle of 0º and then rotated to 120º

6.1 Using the Calibration Target A calibration target is drawn next to the observation point. The components of a calibration target are illustrated in Figure 3. These components are:

• Target Middle. This is the target value that was measured in the field.

• Target Extents. The top of the target represents the target value plus the interval while the bottom represents the target value minus the interval.

• Color Bar. The color bar shows the error between the observed value and the computed value. If the bar is entirely within the target, the color bar is drawn in green. If the error is less than twice the interval, the bar is drawn in yellow. A larger error will be drawn in red.

For this example, the bar would be green if the computed value is between 3.25 and 3.75, yellow for values between 3.0-3.25 or 3.75-4.0, and red for values smaller than 3.0 or greater than 4.0.

Figure 3 Calibration target

Now that the observation point has been created the target appears. The color bar in this example is red with an arrow pointing down, indicating that the computed solution has a velocity below 3.5 fps.

6.2 Multiple Measurements Each observation point has attributes for all measurements. Similarly, each observation arc has attributes for each flux measurement. The highlighted measurement in the Measurements spreadsheet determines which attributes are shown in the Observation Objects spreadsheet. For example, to create a new measurement:

1. Open the Observation Coverage dialog by choosing the Select Feature Point tool from the Toolbox and double-clicking Point 1.

2. Type “WSE” as the Name in the bottom line of the Measurements spreadsheet. As with the Observation Points spreadsheet, the Measurements spreadsheet will always end with a blank line for the creation of additional measurements.

3. Press Enter or Tab to create the new measurement when finished typing to create the new measurement.

4. Select water surface as the Data Set.

Note that this new measurement is now the Active measurement and it is also highlighted. Several measurements can exist at a time; however, calibration targets will only be displayed in the Graphics Window for Observed points in the Active measurement. Now look at the Observation Points spreadsheet. The Name, Color, and X and Y coordinates have remained the same for Point 1, however, the Observed Val and Interval have been reset to their default values. There is no Angle column as well since this new measurement is tied to a scalar data set. These attributes are for the measurement named WSE. To view the observation point attributes previously specified for the Velocity

measurement, simply click the Velocity measurement to highlight it in the Measurements spreadsheet. Do not delete the WSE measurement since both it and the Velocity measurement will be used to calibrate the model.

1. Before continuing, make the Velocity measurement the Active measurement by checking the Active box in the Velocity row.

7 Reading a Set of Observation Points Using the steps defined above, multiple observation points can be created. However, this process could become tedious for a large set of points. Normally, the data defining the points will be in spreadsheet format and can simply be copied and pasted in the Observation Points spreadsheet. To do this:

1. Open the file observepts.xls in a spreadsheet program. (This is a Microsoft Office/Excel file. If you prefer another spreadsheet, the data is also contained in a tab delimited file named observepts.txt.)

2. Highlight and copy the data from the column labeled “Name” to the first column labeled “int” for Point 2 to Point 8. The data for Point 1 does not need to be copied since Point 1 has already been created.

3. Return to SMS and make sure the Velocity measurement is selected.

4. Select the Name of the second row of the Observation Points spreadsheet as the starting cell for the data to be pasted and paste the copied data into the Observation Points spreadsheet.

5. Click OK to close the Observation Coverage dialog.

Seven new observation points appear in the Graphics Window. The new points are distributed around the finite element mesh, as shown in Figure 4.

Figure 4 The observation points created from the file observepts.obt

Now, the observed values and the confidence interval for the WSE measurement need to be specified. To do this:

1. Open the Observation Coverage dialog by double-clicking on one of the points.

2. Using the same spreadsheet file opened earlier, highlight and copy the data from the column labeled “wse” to the second column labeled “int” for Point 1 to Point 8.

3. Return to SMS and make sure the WSE measurement is selected (highlighted).

4. Select the Observed Value of the top row of the Observation Points spreadsheet as the starting cell for the data to be pasted and paste the copied data into the Observation Points spreadsheet.

To view that calibration targets for the WSE measurement, make the WSE measurement the Active measurement and close the Observation Coverage dialog by clicking the OK button. The points that appear in the Graphics Window do not have arrows since the active measurement is observing a scalar data set. When calibrating a model the goal is to calibrate the model so that the computed values from the model fall within the confidence intervals of the observed field data for all measurements. At times this is difficult and personal discretion is required to determine when the model has sufficiently been calibrated.

1. Before continuing, make the Velocity measurement the Active measurement.

8 Generating Error Plots

SMS can create several types of plots to analyze the error between the computed and observed values. To create a Computed vs. Observed Data plot and an Error Summary plot

1. Select Display | Plot Wizard… .

2. Choose Computed vs. Observed Data as the Plot Type.

3. Click Next and choose Velocity as the measurement.

4. Click Finish to close the Plot Wizard and generate the plot.

Create another plot of the Velocity measurement, but this time choose Error Summary as the Plot Type. Again choose Velocity as the measurement. Both plots have now been created. Each plot exists in a separate window that can be resized, moved, and closed at any time. The plots that appear are shown in Figure 5.

Figure 5 Computed vs. Observed Data and Error Summary plots

8.1 Plot Data

More plots can also be created for the WSE measurement or the current plots can be edited. To edit a plot:

1. Right-click the Error Summary plot and select Plot Data… from the menu.

2. Select WSE as the measurement.

3. Click OK to close the Data Options dialog.

The Error Summary plot is now updated using the data from the WSE measurement.

8.2 Using the Computed vs. Observed Data Plot In the Computed vs. Observed Data plot, a symbol is drawn for each of the observation points. A point that plots on or near the diagonal line indicates a low error. Points far from the diagonal have a larger error. The position of the points relative to the line gives an indication whether the computed values are consistently higher or lower than the observed values. In this case, all points are below the line indicating that all computed velocities are lower than observed values.

1. Change the measurement for the Computed vs. Observed Data plot to the WSE measurement by following the steps above defined in section 8.1.

Now, all points plot above the line indicating that all computed water surface elevations are above the observed values.

8.3 Using the Error Summary Plot In the Error Summary plot, the following three types of error norms are reported:

• Mean Error. This is the average error for the points. This value can be misleading since positive and negative errors can cancel.

• Mean Absolute Error. This is the mean of the absolute values of the errors. It is a true mean, not allowing positive and negative errors to cancel.

• Root Mean Square. This takes the sum of the square of the errors and then takes its square root. This norm tends to give more weight to cases where a few extreme error values exist.

1. Close the two plots now by clicking on the “X” at the upper right corner of the window. Then, maximize the graphics window and frame the image.

9 Calibrating the Model The values in this solution for both measurements are not within the calibration targets. To achieve better calibration, the material properties will be changed and then the model will be re-run using FESWMS. If you do not have the FESWMS interface enabled, solutions using a generic mesh have been provided. Since the errors through the main channel for the Velocity measurement are negative, indicating that the observed velocities are larger than those computed by the model, we want to change the parameters in such a way as to increase the velocity in these locations (eddy viscosity and/or Manning’s n). Increasing the velocity at these locations should also decrease the water surface elevation.

9.1 Editing the Material Properties Decreasing the eddy viscosity values can increase these computed velocities. To decrease the eddy viscosity:

1. Click on the Mesh object to make it active.

2. Select FESWMS | Material Properties….

3. In the Turbulence Parameters tab, change the Vo (kinematic eddy viscosity value) from 10.0 to 1.5.

4. Click OK to close the FESWMS Material Properties dialog.

9.2 Computing a New Solution

To compute the new solution:

1. Go to File | Save As…, make sure the type is SMS project file and save the simulation as observe2.sms.

2. Run fst2dh (FESWMS) on the new simulation. At the completion of the run, leave the toggle on for reading the solution. You may have to select a different dataset in the Project Explorer for the targets to update after running fst2dh.

9.3 Reading the New Solution If you are unable to run FESWMS successfully, you can open the solution for the second simulation by selecting the File | Open command and selecting the file “observe2.flo”. Generic Module Instructions: Select File | Open and choose observe2.h5. You will see solutions for both the observe1 model and the observe2 model.

The calibration targets will automatically update to show the errors for the solution that was just opened.

9.4 Fine-tuning the Model The verification targets now show that six points for the Velocity measurement are within the allowable range and two points are above the range, but still in the yellow range. There are no points more than the two times the variation above the observed value (red targets). Looking at targets for the WSE measurement, four points are within the allowable range and four are below the range with one point being more than two times the variation below the observed value. Since the values for the Velocity measurement that are unacceptable are now higher than the observed values and the values for the WSE measurement that are unacceptable are now lower than the observed values, the correction made was too drastic. Specifically, the eddy viscosity was lowered too much and it needs to be raised. To compute another solution:

1. Click on the Mesh object to make it active.

2. Select FESWMS | Material Properties….

3. In the Turbulence Parameters tab, change the Vo (kinematic eddy viscosity value) from 1.5 to 6.0.

4. Click OK to close the FESWMS Material Properties dialog.

5. Save and run a third simulation of fst2dh (observe3.sms). (Read in the simulation at the completion of the run or read the solution from the output folder labeled observe3.h5. This is the generic solution.)

After this third simulation is opened, all the observation point targets for the Velocity measurement should be within the acceptable intervals. Make sure velocity is the active measurement and confirm this. Now make the WSE measurement the Active measurement. All but two points are within the acceptable interval. To get these two points within range, compute another solution:

1. Change the kinematic eddy viscosity value (Vo) to 7.0.

2. Save and run a fourth simulation (observe4.fpr). (Read in the simulation at the completion of the run or read the solution from the output folder labeled observe4.h5. This is the generic solution.)

Looking at the calibration targets should show all the points are acceptable. The calibration process is now complete. It will not always be possible to get all observation points for each measurement to be within the acceptable confidence interval. Therefore, it will have to be decided which measurements and which points are the most important to have within the acceptable range.

10 Using the Error vs. Simulation Plot When performing trial-and-error verification, it is often important to keep track of the error trend as new solutions are repeatedly computed. SMS provides a special verification plot to simplify this task. To create this plot:

1. Select Display | Plot Wizard… .

2. Choose Error vs. Simulation as the Plot Type and click Next.

3. Select Velocity as the measurement.

4. SMS will create a plot with one point for each simulation. The order of the points in the plot will follow the order solution sets in the Solutions list box. The solution at the top will be first. Use the Move Up and Move Down buttons to change this order. The default order is the order that they were read in.

5. Click Finish to close the Plot Wizard and generate the plot.

A new plot appears showing the Error vs. Simulation, as shown in Figure 6. Notice for the Velocity measurement that the errors decrease as each simulation was performed until the final solution where the errors slightly increase. This slight increase in error with the Velocity measurement was required to get that last observation point for the WSE measurement within the acceptable range. Generally, if the errors increase, then the model is not improving.

Figure 6 Error vs. Simulation plot for Velocity

Create another Error vs. Simulation plot using the WSE measurement. The errors for this measurement changed dramatically from solution to solution since parameters were first changed to calibrate points for the Velocity measurement. However, the general trend was a decrease in error. This plot is shown in Figure 7.

Figure 7 Error vs. Simulation plot for WSE

Close the plots now by clicking on the “X” at the upper right corner of the window.

11 Generating Observation Profile Plots Observation profile plots are used to view data set values along observation arcs. The first observation arc to be created will be used to create a profile of the main channel. To create this arc:

1. Right click on the Map Data object in the Project Explorer and create a new coverage. Make the type an Observation coverage and make its name “Profiles.”. Click on this new coverage to make the Map module active. (We create a separate coverage to keep the observation arcs separate from the exiting observation points. When an observation arc is being created, observation points may be clicked joining them to the arc. Observation points and arcs can exist on the same coverage.)

2. Choose the Create Feature Arc tool from the Toolbox.

3. Create an arc down the main channel, as shown in Figure 8. Remember to double-click the last point to end the arc.

When the plots are drawn, they will use the name and color associated with the observation arc. To change the name and color of the arc:

1. Choose the Select Feature Arcs tool from the Toolbox.

2. Double-click on the profile arc.

3. In the Observation Coverage dialog, change the Name of the arc to “river profile” in the Observation Arcs (leave it’s Color as the default).

4. Click OK to close the Observation Coverage dialog.

Three more arcs need to be created, each across a section of the river. These arcs will be used to create cross section plots. To create these arcs:

1. Select the Create Feature Arcs tool from the Toolbox.

2. Create each of the cross section arcs, as shown in Figure 8. Note: When creating these, DO NOT click ON the profile arc, as this would split it.

Figure 8 Profile and cross-section arcs

With the cross sections created, open the Observation Coverage dialog and assign a unique color and an appropriate name to each arc. With the arcs created, the plots can now be generated. To do this:

1. Select Display | Plot Wizard… .

2. Choose Observation Profile as the Plot Type and click Next.

3. Turn on the Specified option, and check only the elevation data set in the Generic Solution and the water surface data set in the observe4.flo solution.

4. Turn off the three cross-section arcs in the Arcs spreadsheet by turning off their corresponding Show toggles.

5. Click Finish to close the Plot Wizard and generate the plot.

The profile plot of the geometry of the stream should appear as shown in Figure 9.

To view the velocity distribution across the three cross sections:

1. create a new Observation Profile plot. Turn on the velocity mag data set in the observe4.flo solution, and Show only the three cross section arcs.

Figure 9 Observation Profile Plot

12 Generating Time Series Plots As mentioned earlier, observation arcs are used to compute fluxes. One flux value that is often observed and measured in the field is flow rate. Observed flow rates can be used in model calibration in the same way observed velocities and water surface elevations are used. In addition to normal model calibration, Time Series plots can be generated showing how the flow rate flux changes with time. This type of time series plot is commonly known as a hydrograph. Hydrographs created using calculated data from the model are useful to see if the model properly predicts flow rate patterns. To create a Time Series plot:

Figure 10 Observation arc across noyo1.sms mesh

1. Open the file noyo1.sms if you have the FESWMS interface enabled. If not, open the files noyo1.2dm and noyo1.h5. If geometry data is still open, you will be warned that the existing mesh will be deleted. If this happens, click Yes to the prompt.

2. Create a new observation coverage called “Fluxes.”

3. Click on the Frame button.

4. Go to Display | Display Options and in the 2d mesh turn on Elements and Nodes.

5. Create an observation arc across the mesh as shown in Figure 10.

6. Open the Plot Wizard .

7. Choose Time Series as the Plot Type and click Next.

8. Change the Function Type to Flux. Check the Show toggle in the spreadsheet for the arc you just created (the default name will be “Arc #”). Select “water depth” as the Scalar Dataset and “velocity” as the Vector Data Set.

9. Click Finish to close the Plot Wizard and generate the plot.

A new window opens with the Time Series plot of the Flow Rate flux measurement. This plot should appear similar to the one in Figure 11.

Figure 11 Time Series plot of Flow Rate.

13 Conclusion This concludes the Observation Coverage tutorial. You may continue to experiment with the program or you can exit SMS.