Evaluate Module

Perform a robust statistical assessment of measurement or model skill

 

The Evaluate Module is designed to allow a robust statistical comparison to be made between two data sets. The user has the option of choosing between two operating modes, a Quick Compare and Evaluate mode. The Quick Compare mode allows the user to compare two data sets using a set of comparison plots in DataView. The Evaluate mode is an in-depth analysis between two data sets. Statistical temporal correlation (TC) analyses are performed on wind speed and height, and wave height, period and direction. The Evaluate Module GUI allows the user to select different analysis options such as applying smoothing, subsampling, and wave height and wind speed thresholds to the baseline data set. The integration of Evaluate Module is depicted in the following Figure. Results are displayed in a set of Evaluate plots and tables in DataView.

 

 

Using the Evaluate Module

To open the Evaluate Module parameters, select Evaluate from the Module pull-down menu. The Evaluate Parameters Table opens to allow the user to specify how the data are to be processed.  Here the user can select the baseline and evaluate data sets and specify the many different analysis options.

 

Processing commences once the user selects RUN on the Evaluate Parameters Table.  Both the baseline and evaluate data sets will be loaded.  The baseline data is smoothed and subsampled as specified by the user. The baseline data frequency and direction bins are interpolated to match that of the evaluate data set and the time stamps are matched closest in time. A temporal correlation analysis is performed on wind data (if available), wave data, and wave systems data (if specified by the user). If Wave Systems analysis was selected and the baseline data has not been processed with the Wave Systems Module, the user will be directed to the Wave Systems Parameters Table to set up wave systems processing. When Evaluate processing is finished a special Evaluate layout page in Data View opens to provide the user access to all Evaluate Module products.  Each of these products is described below.

 

Error Analysis

The error metrics used to quantify the temporal correlation comparisons are as follows:

 

For n values of baseline measurements m and evaluate measurements h the bias (evaluate – baseline)

 

 

Root-mean-square (RMS) error

 

 

Scatter index

 

 

Where the standard deviation of difference is given by

 

For directional data, the angular bias is calculated by

 

 

Where S and C are computed from the directional differences Δθ = |θ hm| by

 

 

 

The circular correlation is computed by

 

 

 

In addition, wave height is calculated for each frequency bin and sorted into 11 equally spaced wave height bins. The wave heights are calculated by

√(S(f)*d(f))

Where S(f) is the spectral density as a function of frequency and d(f) is the frequency bandwidth.

 

The wave height bias is then calculated for each bin that contains more than 5 observations in time.

 

Evaluate Plots

Evaluate plots are displayed in a special Data View layout page.  This page is toggled on and off through the Evaluate command in the Data View Layout pull-down menu.  See the Plot Library for a list of available plots.

 

Export Data Menu Commands

The user can export a text file containing the error metric tables by selecting Fixed Format -> Evaluate Error Metrics from the Tables dropdown menu in the DataView Module. The data can be exported as a Microsoft Excel file or ASCII text file.

An example Error Metric table output is shown below. This analysis was performed on wind data and two swell groups of Wave Systems.

Evaluate Module Error Metrics for Waverider_Sep and Duck3mwr_SWAN

 

Wind Speed Error Metrics

Component

Bias(m)

RMS(m)

SI

R^2

wind

0.530

3.755

0.432

0.891

 

Wind Direction Error Metrics

Component

Circ. Corr.

Circ. Bias(deg)

wind

0.828

7.975

 

Wave Height Error Metrics

Component

Bias(m)

RMS(m)

SI

R^2

bulk

0.012

0.444

0.175

0.977

ws

0.125

0.428

0.141

0.990

sw1

-0.012

0.293

0.210

0.929

sw2

-0.022

0.413

0.196

0.979

 

Peak Period Error Metrics

Component

Bias(m)

RMS(m)

SI

R^2

bulk

-0.196

2.513

0.189

0.724

ws

0.203

0.634

0.071

0.991

sw1

0.184

0.525

0.081

0.941

sw2

0.137

2.373

0.185

0.799

 

Mean Period Error Metrics

Component

Bias(m)

RMS(m)

SI

R^2

bulk

0.005

1.121

0.160

0.530

ws

0.587

1.209

0.197

0.971

sw1

-0.009

0.332

0.075

0.935

sw2

0.287

1.306

0.136

0.933

 

Peak Direction Error Metrics

Component

Circ. Corr.

Circ. Bias(deg)

bulk

0.588

-0.588

ws

0.977

15.402

sw1

0.976

15.965

sw2

0.591

-0.140

 

Mean Direction Error Metrics

Component

Circ. Corr.

Circ. Bias(deg)

bulk

0.544

7.253

ws

0.985

9.943

sw1

0.991

8.982

sw2

0.729

5.919

 

 

 

The wave height bias can be exported into an excel spreadsheet by selecting Fixed Format -> Wave Height Bias File from the Tables dropdown menu in the Data View Module. The total bias values are tabulated for each frequency bin and wave height bin

 

References

Bowers, J. A., I. D. Morton, and G. I. Mould, 2000: Directional statistics of the wind and waves. Appl. Ocean Res., 22, 13-30

Cardone, V. J., R. E. Jensen, D. T. Resio, V. R. Swail, and A. T. Cox, 1996: Evaluation of contemporary ocean wave models in rare extreme events: The "Halloween Storm" of October 1991 and the "Storm of the Century" of March 1993. J. Atmos. Oceanic Technol., 13, 198-230

Guillaume, A., 1990: Statistical tests for the comparison of surface gravity wave spectra with application to model validation. J. Atmos. Oceanic Technol., 7, 551-567

Hanson, Tolman, Tracy and Scott, J., 2009: Pacific Hindcast Performance of Three Numerical Wave Models. Atmos. Oceanic Technol., 26, 1614-1633

Tracy, B. A., 2002: Directional characteristics of the 1990-1999 Wave Information Studies Gulf of Mexico hindcast. Proc. Seventh Int. Workshop on Wave Hindcasting and Forecasting, Meteorological Service of Canada, U.S. Army Engineer Research and Development Center's Coastal and Hydraulics Laboratory, and Fleet Numerical Meteorology and Oceanography Center, 10 pp.

 

 

The Evaluate Parameter Table is used to set processing parameters for the Evaluate Module.

 

 

The Evaluate Parameters Table is divided into 2 main sections:

 

 

1. Commands (Left column of buttons)

2. Parameters (center and right column of inputs)

 

The features in each of these sections are described below.

 

1.  Commands

The commands appearing on the left side of the Evaluate Parameters Table control the saving and use of selected parameters.  The following commands and options are available:

 

Named Settings – Select from a list of pre-existing parameter settings.  Parameter values will automatically update.

Update – Update the displayed parameters to the current named parameter set.

Save New – Save the displayed parameters to a new named parameter set.  User will be prompted to provide a name for the new parameter set.

Reset – Reset displayed parameters to saved values

Delete – Delete the current named parameter set

Run – Process the data set using the current named parameter set

Cancel – Exit the Evaluate Parameter Table without processing any data

 

2.  Parameters

The processing parameters allow the user to customize the resulting tables and plots.

 

Baseline Data

Load Baseline Data Set – Click Browse to navigate to the location of the processed .mat file or NetCDF data file containing the baseline data set.

 

Evaluate Data

Load Evaluate Data Set – Click Browse to navigate to the location of the processed .mat file or NetCDF data file containing the evaluate data set. The file type of the Baseline and Evaluate data sets must be the same. If NetCDF files are chosen, the user will be asked to choose a filename to save the Evaluate data. TWO files will be created, one for the baseline data and one for evaluate data. They both will contain the user defined filename with “_Baseline.nc” and “_Evaluate.nc” appended to the baseline and evaluate data respectively.

 

Analysis Options

These parameters control the preparation and analysis of the data.

 

Baseline smoothing interval (hrs) – Over this specified duration, the Evaluate Module will perform a weighted running average of the input directional wave spectra in the baseline data.  For example, a 4-h smoothing window will smooth all spectra occurring up to 2-h prior and 2-h after each observation time.

 

Baseline subsample interval (hrs) – The Evaluate Module will remove records in the baseline data that fall in-between this interval.  For example, if a 3-h data subsample interval is selected for data that is provided hourly, every third record will be saved.

 

Max allowable time difference (hrs) – The Evaluate Module will search through the evaluate data records and find a record in the baseline data that matches closest in time. If the closest record has a time difference greater than the value set here, the record will be disregarded.

 

Peak Analysis

Apply Wind Speed Threshold? – The user may decide to apply a wind speed threshold to the wind analysis. The user can specify this value as a constant or as a standard deviation multiplier. Any data below this threshold will be disregarded in analysis.

 

Wind Speed Threshold – Select the value of the constant wind speed threshold or the standard deviation multiplier.

 

Apply Wave Height Threshold? – The user may decide to apply a wave height threshold to the wave analysis. The user can specify this value as a constant or as a standard deviation multiplier. Any data below this threshold will be disregarded in analysis.

 

Wave Height Threshold – Select the value of the constant wave height threshold or the standard deviation multiplier.

 

Peak Lag (hrs) – In the Peaks Over Threshold Analysis for wind speed and bulk wave height, this value indicates the minimum time lag in hours between successive events. Wind speed and/or wave height thresholds must be set in order to conduct a Peaks Over Threshold Analysis.

 

Match Lag (hrs) – In the Peaks Over Threshold Analysis for wind speed and bulk wave height, this value indicates the search window duration in hours over which to match peaks found in the baseline data to the modeled evaluate data. Wind speed and/or wave height thresholds must be set in order to conduct a Peaks Over Threshold Analysis.

 

 

Wave Systems Options

These parameters determine if and how the Wave Systems data is processed.

 

Process Wave Systems? – To skip the wave systems analysis, select NO. Otherwise, wave system components will be analyzed and compared based on the remaining input selections.

 

Number of Swell Groups – The user may select 1 or 2 groups to compare in addition to windsea. If 2 is selected, the user can select how to delineate between the groups.

 

Delineate Swells by – Select the method by which to delineate the swells, wave height or wave period.

 

Delineating Value(s) – If Wave Period is selected as the method to delineate swells, the user can set the delineating period in seconds. Wave components with periods under this value will fall under swell1 while components with periods larger than this value will fall under swell 2. If Wave Height is selected, the Wave Systems partitioning method is used to determine swell1 and swell2.

 

Quick Compare

For a quick comparison between a baseline and evaluate data set, select Quick Compare from the Module pull-down menu. If a processed file is already in memory, the user will be directed to select an evaluate data set to compare. If there is no data currently loaded, the user will be asked to select the baseline and evaluate data set. Once loaded, a comparison layout page in Data View opens to provide the user access to the comparison plots.