Session - Model Metrics, Verification and Validation

M. Angling, A. Glover, P. Jiggens, S. Bingham, S. Elvidge

In order to provide reliable services to end-users, it is crucial to understand the strengths and potential limitations of the various elements underpinning those services. This includes the assumptions and algorithms on which models are based, and also the reliability of the associated infrastructure: e.g. data systems, space and ground-based measurement infrastructure. This also includes situations during extreme solar and geomagnetic conditions.

At the present time, within the space weather community, prototype services frequently operate as capability demonstrators and a full verification of their ability to reproduce/predict elements of the space environment under a range of space weather conditions, from the moderate to the extreme, has yet to be completed. Forecast accuracy has recently been addressed by a number of separate activities, but as yet a community-wide consensus on how to address this question and provide relevant information supporting end-users, service developers and modellers themselves for the wide range of models and domains involved has not been reached.

This session targets modellers, service developers and service providers. The session will review current activities and initiatives ongoing both in Europe and internationally and will highlight recent results of coordinated model testing. Dedicated splinter meetings will follow the session in order to discuss coordinated test results in more detail and address verification & validation needs for the current generation of activities under development and in planning. The session will build upon work discussed at the ISES Space Weather Forecast Verification Workshop in April 2015 and will promote further discussions at the ILWS-COSPAR workshop session on metrics to assess space weather predictions in January 2016. Papers will be especially encouraged from ionospheric modellers who have adopted the ionospheric testing scenario that will be suggested at the AT-RASC meeting in the spring.

Talks
Wednesday November 25, 11:00 - 13:00, Permeke

Poster Viewing
Wednesday November 25, 10:00 - 11:00, Poster area

Click here to toggle abstract display in the schedule

Talks : Time schedule

Wednesday November 25, 11:00 - 13:00, Permeke

11:00	Verification of forecast probabilities at RWC Belgium	Devos, A et al.	Oral
	Andy Devos, Cis Verbeeck, Jesse Andries
	Solar-Terrestrial Centre of Excellence - Royal Observatory of Belgium
	Since June 2004, solar flares and the local geomagnetic K-index have been forecasted on a daily basis at the Regional Warning Center (RWC) of Belgium. The evaluation of flare forecast probabilities of C-, M- and X-class flares for individual regions and the global disk are presented. Analysis techniques such as reliability diagram, Receiver Operating Characteristic (ROC) curve and Brier score are applied to the flare probabilities. Verification analysis is carried out under different levels of solar activity and conditional to the magnetic class of active regions. Confidence bars on the forecast probabilities are determined. In addition, a framework is setup to determine forecast probabilities for the geomagnetic K-index with confidence bars. The forecast performance of the K-index is evaluated under different solar wind conditions such as ICMEs and high speed streams to better a posteriori understand the influences leading to (in)correct forecasts. These verification results (to be placed on [1]) will provide more insights in the strengths and weaknesses of forecasting at the RWC of Belgium. The use of forecast probabilities and confidence bounds leads to more transparency and communication of the available knowledge, including the amount of uncertainty, relevant to end-users. [1] http://www.sidc.be/forecastverification/index.php
11:20	Initial Results of the Advanced European Ne Assimilation System (AENEAS)	Elvidge, S et al.	Oral
	Sean Elvidge, Matthew Angling
	University of Birmingham
	The Advanced European Ne Assimilation System (AENEAS) is a new model developed at the University of Birmingham, UK. It is a physics-based data assimilation model that uses the National Center for Atmospheric Research’s (NCAR’s) Thermosphere Ionosphere Electrodynamic General Circulation Model (TIEGCM) as its background model and assimilates electron density virtual height profiles from ionosondes and TEC measurements from GPS receivers. This paper will describe the latest version of AENEAS and plans for its future development. Furthermore, this paper will review the ionospheric test scenario that was proposed at AT-RASC in May 2015. This scenario includes 12 GPS receivers and 3 ionosondes for assimilation and will test differential slant TEC, foF2 and the height of 80% of foF2. Initial AENEAS results will also be presented.
11:40	Validation of F2 Layer Peak Height and Density by Real-Time IRI	Galkin, I et al.	Oral
	I. A. Galkin[1,2], A. M. Vesnin[3], B. W. Reinisch[1,4], X. Huang[4], and P. Song[1]
	[1] University of Massachusetts Lowell, Space Science Laboratory, Lowell, MA, USA; [2] Borealis Global Designs EOOD, Varna, Bulgaria; [3] Institute of Solar-Terrestrial Physics, Russian Academy of Sciences, Irkutsk, Russia; [4] Lowell Digisonde International, LLC, Lowell, USA
	The Global Ionosphere Radio Observatory (GIRO) ingests ionosonde data (foF2, hmF2, etc.) in databases at the Lowell GIRO Data Center (LGDC). Every 15 minutes when new data arrive, the IRI-based Real-Time Assimilative Mapping (IRTAM) algorithm at LGDC generates real-time global maps of foF2 and hmF2. The IRTAM morphs the empirical “climatology” IRI model into agreement with the GIRO measurements by smoothly varying the internal parameters in the IRI model to the best fit, so that the “real-time” model representations of the ionosphere closely follow its “weather” variability. The new “GAMBIT” database (Global Assimilative Modeling of Bottomside Ionospheric Timelines) at http://lgdc.uml.edu/GAMBIT now offers a unique Internet-accessible data resource of the IRTAM maps, for visualization, analyses, and user applications in a compact form of model coefficients. The GAMBIT Explorer, an interactive environment to access GAMBIT, offers a suite of tools for inspection, investigation and comparison of the IRTAM with other measurements and models such as global GNSS TEC maps acquired from the MIT Madrigal repository. Initial results of the statistical IRTAM validation are presented using over 13 million records in GAMBIT collected over a 15–year period from 2000 to 2015. Using the GAMBIT database we test the performance of IRTAM by (a) cross-testing using observations from the European digisonde network and (b) analysis of the IRTAM capability to represent ionospheric effects during well documented space weather events. Validation metrics will be presented based on comparisons of IRTAM to test data omitted from the assimilation. The results shed new lights on the capabilities of the IRTAM technologies to describe global ionospheric timelines and reveal their potential to predict system dynamics over regions of no data coverage and short-term forecast.
12:00	Space weather verification at the Met Office.	Bingham, S et al.	Oral
	Suzy Bingham, Edward Pope, Michael Sharpe, David Jackson
	Met Office, UK.
	The Met Office provides 24/7 operational space weather forecasts, alerts and warnings. Models employed at the Met Office include Enlil, the Relativistic Electron Forecast Model (REFM) and the D-Region Absorption Prediction (D-RAP) Model. Model output and near real-time observations, along with forecaster experience and skill, enable forecasters to produce space weather guidance twice daily. Verification is of vital importance for forecasters, researchers, users and stakeholders to understand the strengths and limitations of these forecasts, and to understand forecaster added value. For this purpose, the Met Office has been adapting terrestrial weather verification techniques, and has been working closely with the International Space Environment Service to build a consensus on standardising verification procedures. As part of this work, progress has been made in analysing forecast and observed coronal mass ejections (CMEs), using 2x2 contingency tables to assess the forecast skill. Verification of four day probabilistic forecasts of geomagnetic storms, X-ray flares, high energy proton, and high energy electron events has also been addressed. Since the forecasts for these variables are categorical (e.g. geomagnetic storm forecasts give a probability of exceeding minor/moderate, strong, severe and extreme levels), it is necessary to employ a different verification approach than for the CMEs. Such methods include quantification of the forecast skill using the Rank Probability Skill Score (RPSS) and comparing the forecasts against climatology and persistence benchmarks. In addition, progress has been made in adapting a terrestrial weather verification visualisation system for space weather purposes, to enable forecasters to view verification of forecasts in near real-time.
12:20	Metrics of model performance for electron fluxes (<200 keV) at geostationary orbit	Ganushkina, N et al.	Oral
	Natalia Ganushkina[1,2], Ilkka Sillanpaa[1]
	[1] Finnish Meteorological Institute, Helsinki, Finland; [2] University of Michigan, Ann Arbor MI, USA
	We present the nowcast model running online (http://fp7-spacecast.eu) for low energy (< 200 keV) electrons in the inner magnetosphere which is the version of the Inner Magnetosphere Particle Transport and Acceleration Model (IMPTAM) for electrons. Low energy electron fluxes are very important to specify when hazardous satellite surface charging phenomena are considered. The model is driven by the real time solar wind and Interplanetary Magnetic Field (IMF) parameters and by the real time Dst index. Real time geostationary GOES 13 or GOES 15 (whenever each is available) data on electron fluxes in three energies, such as 40 keV, 75 keV, 150 keV, are used for validation of IMPTAM running online. Two metrics for model performance are used: (1) the Normalized Root-Mean-Square Deviation (NRMSD) and (2) binary event tables with Heidke Skill Scores. is found to range from 0.015 to 0.0324. Though these metrics are buoyed by large standard deviations owing to the dynamic nature of the fluxes, they demonstrate that IMPTAM predicts the observed fluxes satisfactory. This is the a challenging effort to model low energy electrons in real time at 10 minutes resolution. The output of this model can serve as an input of electron seed population for real-time higher-energy radiation belt modeling.
12:40	Performance Verification of ESA's SSA/SWE A-EFFort Service	Georgoulis, M et al.	Oral
	Manolis K. Georgoulis, Kostas Tziotziou
	Research Center for Astronomy and Applied Mathematics (RCAAM) of the Academy of Athens
	The Athens Effective Solar Flare Forecasting (A-EFFort) service is a Level-2 federated service of the ESA/SSA Space Weather Segment (SWE). As such, it is physically installed at its host institution, the Research Center for Astronomy and Applied Mathematics (RCAAM) of the Academy of Athens, Greece, and is accessible via an SWE single sign-on registration service. The A-Effort complements SWE's Solar Weather Expert Service Center products with a near real-time flare forecasting service with zero latency, a 24-hour forecast window, and a refresh rate of three hours. The A-EFFort team is committed to delivering a fully validated service and we aim to describe here the course of action and results of performance verification. Overall, we have accomplished a detailed A-EFFort validation that showcases a significant prediction skill for the project. This work has received partial support by ESA Contract No. 4000111994/14/D/MRP.

Posters

Wednesday November 25, 10:00 - 11:00, Poster area

1	Real time forecasting methods validation with the Flare Scoreboard	Murray, S et al.	e-Poster
	Sophie Murray[1], Leila Mays[2], Masha Kuznetsova[2], Suzy Bingham[1], Edward Pope[1], et al
	[1] Met Office; [2] Community Coordinated Modeling Center
	Flare Scoreboard is hosted by NASA Community Coordinated Modeling Center and its planning group includes expert scientists as well as operational space weather prediction centres. The scoreboard will allow a consistent real-time comparison of various operational flare forecasts. Here we report on the status of this first community-wide coordinated flare forecast validation project.
2	Quasi longitudinal approximation for application to ionospheric ray tracing and absorption	Settimi, A et al.	p-Poster
	Alessandro Settimi, Carlo Scotto
	Istituto Nazionale di Geofisica e Vulcanologia (INGV), Sezione di Geomagnetismo, Aeronomia e Geofisica Ambientale (ROMA 2), Via di Vigna Murata 605, I-00143 Rome, Italy
	For the refraction index of high frequency (HF) waves in the ionosphere exists a well established theory. However, for the quasi-longitudinal (QL) approximation, literature presents various formulas that are not equivalent and that in some cases give rise to wrong results. In this paper, the different fields of application of such formulas are described along with the consequences in the calculation of ionospheric ray tracing and absorption.
3	LYRA flare probabilities service and its performance measure	Dammasch, I et al.	p-Poster
	Ingolf Dammasch, Marie Dominique
	Royal Observatory of Belgium
	On a daily basis, the radiometer LYRA offers probabilities for A-, B-, C-, M-, and X-flares, which are based on several years of observation and correlation with irradiance levels, microflaring, and sunspot numbers. Performance and limits of the techniques will be described, forecast verification methods will be discussed.