\chapter{Model Performance Statistics}
\label{app:ModelPerformanceStatistics}
\section{Introduction}

In this Appendix the theory behind the Model Performance Statistics (MPS) used in the XBeach skillbed is explained. The MPS are used to quantify the performance of model results based on a comparison with measurement data. Different MPS parameters are used as each parameter has its own characteristics. 

First an overview is given of the MPS parameters used in the XBeach skillbed, summarized in table form including some basic characteristics. Consequently, each MPS parameters listed in the overview table is further explained in separate sections.

\section{MPS parameters}

An overview of the MPS parameters used in the XBeach skillbed is given in \autoref{tab:MPS}.

\begin{table}[H]
  \centering
  \begin{table}[!tbp]
    \caption{MPS parameters\label{table}}
    \label{tab:MPS}
    \begin{center}
      \begin{tabular}{llll}\hline\hline
        Parameter & Description & Ranges\\\
        \hline
        ME \& STD & Mean Error \& Standard Deviation & 0: perfect prediction\\\
        R & Correlation coefficient (range: [0 1]) & 1: perfect correlation\\\
        Rel. bias & Systematic error relative to the mean & low value: good performance\\\
        Sci & Scatter Index & low values: performance\\\
        BSS & Brier Skill Score \citep{Sutherland2004} & see below\\\
        BSS & Brier Skill Score \citep{Murphy1989} & see below\\\
        \hline
      \end{tabular}
    \end{center}
  \end{table}
\end{table}

Each parameter listed in the table is further explained in the following paragraphs.

\section{Mean Error \& Standard Deviation}

The Mean Error (ME) and the Standard Deviation (STD) of the error of a time series are a useful measure to quantify model performance for parameters such as wave heights or water levels. The SD is in general not so useful when applied to morphological parameters such as the bed level evolution.

\begin{equation}ME=\frac{1}{N}\sum_{i=1}^N(f_{comp.,i}-f_{meas.,i})\end{equation}
\begin{equation}STD=\sqrt{\frac{1}{N-1}\sum_{i=2}^N(f_{comp.,i}-f_{meas.,i}-ME)^{2}}\end{equation}

\section{Correlation coefficient}

The Correlation Coefficient R is a measure quantifying the correlation of the measurements and simulation results, but does not indicate significance because the distributions of the series are not taken into account.

\section{Relative Bias}

The Relative Bias (Rel. Bias) is the systematic error relative to the mean. Relative low values of the mean can cause high vales of the Rel. Bias.

\begin{equation}Rel. Bias=\frac{\sum_{i=1}^N(f_{comp.,i}-f_{meas.,i})}{\sum_{i=1}^N\bar f_{meas.}}\end{equation}

\section{Scatter Index}

The Scatter index (Sci) is the standard deviation relative to the mean value of the measured signal. Relative low values of the mean can cause high vales of the Sci. 

\begin{equation}Sci=\frac{\sqrt{\frac{1}{N-1}\sum_{i=2}^N(f_{comp.,i}-f_{meas.,i}-ME)^{2}}}{\bar f_{meas.}}\end{equation}

\section{Brier Skill Score}

The Brier Skill Score (BSS) calculates the performance of the performance relative to a baseline prediction. The BSS calculates the mean square difference between the prediction and observation with the mean square difference between baseline prediction and observation. 

\begin{equation}BSS=1-\frac{\frac{1}{N}\sum_{i=1}^N(z_{b,c}-z_{b,m})^{2}}{\frac{1}{N}\sum_{i=1}^N(z_{b,0}-z_{b,m})^{2}}\end{equation}

where $z_{b,c}$ is the computed bottom, $z_{b,m}$ is the measured bottom and $z_{b,0}$ is the initial bottom (variables taken at each cross-shore coordinate i). 

Perfect agreement gives a Brier score of 1, whereas modelling the baseline condition gives a score of 0. If the model prediction is further away from the final measured condition than the baseline prediction, the skill score is negative. \citet{VanRijn2003} proposed a classification for the Brier Skill Score as shown in \autoref{tab:BSSquantification}.

The BSS is very suitable for the prediction of bed evolution. The baseline prediction for morphodynamic modelling will usually be that the initial bed remains unaltered. In other words, the initial bathymetry is used as the baseline prediction for the final bathymetry. A limitation of the BSS is that it cannot account for the migration direction of a bar; it just evaluates whether the computed bed level (at time t) is closer to the measured bed level (at time t) than the initial bed level. If the computed bar migration is in the wrong direction, but relatively small; this may result in a higher BSS compared to the situation with bar migration in the right direction, but much too large. The BSS will even be negative, if the bed profile in the latter situation is further away from the measured profile than the initial profile. The limitation shown here is that position and amplitude errors are included in the BSS. Distinguishing position errors from amplitude errors, requires a visual inspection of measured and modelled profiles or the calculation of further statistics \citep{Murphy1989}. The BSS can be extremely sensitive to small changes when the denominator is low, in common with other non-dimensional skill scores derived from the ratio of two numbers.

\begin{table}[H]
  \centering
  \begin{table}[!tbp]
    \caption{Brier Skill Score quantification \citep{VanRijn2003}}
    \label{tab:BSSquantification}
    \begin{center}
      \begin{tabular}{llll}
        \hline\hline
        Qualification & Brier Skill Score\\\
        \hline
        Excellent & 1.0 - 0.8\\\
        Good & 0.8 - 0.6\\\
        Reasonable fair & 0.6 - 0.3\\\
        Poor & 0.3 - 0.0\\\
        Bad & \textless 0.0\\\
        \hline
      \end{tabular}
    \end{center}
  \end{table}
\end{table}

\section{Brier Skill Score \citep{Murphy1989}}

\citet{Murphy1989} decomposed the BSS, leading to contributions due to errors in predicting the amplitude ($\alpha$), the phase ($\beta$) and the mean ($\gamma$) as presented in \autoref{tab:BSSdecomposition}. The decomposition facilitates linking performance quantifications to model processes and accordingly bringing the model performance to a higher level.

\begin{equation}BSS=\frac{\alpha-\beta-\gamma+\epsilon}{1+\epsilon}\end{equation}
\begin{equation}\alpha=r^{2}_{Y'X'}; \beta=(r_{Y'X'}-\frac{\sigma_{Y'}}{\sigma_{X'}})^{2};  \gamma=(\frac{\textless Y'\textgreater-\textless X'\textgreater}{\sigma_{X'}})^{2}; \epsilon=\frac{\textless X'\textgreater}{\sigma_{X'}}^{2} \end{equation}

\begin{table}[H]
  \centering
  \begin{table}[!tbp]
    \caption{Brier Skill Score decomposition factors \citep{Murphy1989}}
    \label{tab:BSSdecomposition}
    \begin{center}
      \begin{tabular}{llll}
        \hline\hline
        Factor & Indication & Perfect modelling\\\
        \hline
        phase error ($\alpha$) & transport locations & $\alpha$ = 1\\\
        amplitude error ($\beta$) & transport volumes & $\beta$ = 0\\\
        map mean error ($\gamma$) & - & $\gamma$ = 0\\\
        normalization term ($\epsilon$) & - & -\\\
        \hline
      \end{tabular}
    \end{center}
  \end{table}
\end{table}

\citet{VanRijn2003} also proposed a classification for the decomposed Brier Skill Score as shown in \autoref{tab:BSSquantification2}.

\begin{table}[H]
  \centering
  \begin{table}[!tbp]
    \caption{Brier Skill Score \citep{Murphy1989} quantification \citep{VanRijn2003}}
    \label{tab:BSSquantification2}
    \begin{center}
      \begin{tabular}{llll}
        \hline\hline
        Qualification & Brier Skill Score\\\
        \hline
        Excellent & 1.0 - 0.5\\\
        Good & 0.5 - 0.2\\\
        Reasonable fair & 0.2 - 0.1\\\
        Poor & 0.1 - 0.0\\\
        Bad & \textless 0.0\\\
        \hline
      \end{tabular}
    \end{center}
  \end{table}
\end{table}