Session 13 - Model Metrics, Verification and Validation

Alexi Glover (ESA), Piers Jiggens (European Space Research And Technology Centre), Suzy Bingham (Met Office, UK)
Friday 18/11, 10:00-13:00
Ridderzaal

Evaluation of models, forecasts and systems is crucial in providing end-users with a robust and reliable service. It’s essential for modellers and service providers to understand the strengths and potential limitations of models and forecast processes, in order to improve services. It’s also important to understand the assumptions and algorithms on which models are based, and to assess the reliability of the associated infrastructure: e.g. data systems, space and ground-based measurement infrastructure.

Over recent years, within the space weather community, there has been growing momentum within the area of verification and validation. However, prototype services frequently operate as capability demonstrators and verification of their ability to reproduce/predict over a range of space weather conditions, from minor to extreme, has yet to be completed. Forecast accuracy has been addressed by separate activities but a community-wide consensus on how to address this question has not yet been reached.

We welcome contributions from modellers, service developers and service providers, to cover the breadth of areas in space weather forecasting including forecasting of solar transients, radiation effects, ionospheric impacts and geomagnetic conditions. We particularly encourage contributions describing results from validation activities and from coordinated model testing.

The session will review current activities and initiatives, ongoing in Europe and internationally. Through this session we hope to promote discussion on validation and verification needs for the current generation of activities under development and in planning. The session will build upon discussions at the ILWS-COSPAR workshop session on metrics to assess space weather predictions in January 2016 and COSPAR’s session in July 2016 on metrics and validation needs for space weather models and services. The session will also promote ongoing verification and validation activities through ISES.

Poster Viewing
Friday November 18, 10:00 - 11:00, Poster Area

Talks
Friday November 18, 11:00 - 13:00, Ridderzaal

Click here to toggle abstract display in the schedule

Talks : Time schedule

Friday November 18, 11:00 - 13:00, Ridderzaal

11:00	Investigating the impact of the ENLIL solar wind model boundary conditions on solar wind metrics	Bocquet, F et al.	Invited Oral
	Francois-Xavier Bocquet[1], Mario Bisi[2]
	[1]Met Office; [2]RAL space
	The Enlil heliospheric model is used operationally at the Met Office to predict the solar wind parameters at the L1 point. One of the outstanding issues with the model revolves around the way in which boundary conditions are imposed. Currently, the solar wind speed and magnetic field are derived from the WSA model which combines a potential field source surface model and a Schatten current sheet model to derive the coronal magnetic field, and uses an empirical relationship to determine the solar wind speed based on flux tube expansion factors and distance to coronal holes. In this study, we ran a set of experiments consisting of Enlil driven by the WSA model, Enlil driven by Inter-Planetary Scintillation data and Enlil driven by a non-linear force-free model. We compare solar wind metrics for these models during a quiet period of 6 months with few coronal mass ejections, with a particular focus on Co-rotating Interaction Regions due to their potential for geo-effectiveness, and propose a subset of metrics for evaluating solar wind models, focussing on operational use.
11:15	Coordinated community-wide model validation initiatives: Quantifying storm impact on geospace	Kuznetsova, M et al.	Invited Oral
	M. Kuznetsova, L. Rastaetter, J-S. Shim, A. Pulkkinen, Y. Zheng, C. Wiegan, J. Boblitt, M.L. Mays
	NASA Goddard Space Flight Center, Community Coordinated Modeling Center
	Systematic evaluation of space environment models and tracing model improvements over time are critical for development and further improvements of operational space weather prediction capabilities. The presentation will review applications and on-line systems developed at the Community Coordinated Modeling Center (CCMC) in support of on-going event-based coordinated community-wide model validation activities and metrics studies. We will review an expanding event list for on-going GEM-CEDAR Modeling Challenges to assess capabilities to specify and forecast geospace system response to storms. The focus will be on quantification and validation of storm-driven ionospheric/thermospheric disturbances, changes in regional TEC and neutral densities. We will also discuss topics related to specification, predictions and metrics selection for boundaries in auroral region.
11:30	Evaluating the use of geomagnetic indices for predicting potential damage to power grids	Kelly, G et al.	Invited Oral
	Gemma Kelly, Alan Thomson
	British Geological Survey
	Extreme geomagnetic storms have the potential to have a damaging impact on power infrastructure, through the currents they induce in the ground, termed geomagnetically induced currents (GICs). Such events are often classified and forecast in terms of the Kp index, with a Kp=9o event described as a G5 storm on the NOAA Space Weather Scale for geomagnetic activity. However, this global average, mid-latitude, 3-hour index is unlikely to be the most useful indicator of the short time-scale variations in magnetic field that are of most concern to power transmission operators. For example, depending on latitude an observatory K index of nine can represent a magnetic variation from tens to a few thousand nT. We therefore investigate the relation between GICs, both measured and modelled, to a range of geomagnetic indices. We use a range of validation techniques to evaluate how well these indices perform at identifying periods of large GICs in the UK, focusing in particular on geomagnetic storms. We also extend the study to look at possibilities for a new local index, based directly on the rate-of-change of the magnetic field, as well as forecasts of indices.
11:45	Model evaluation with low-altitude GOCE densities	Bruinsma, S et al.	Oral
	Sean Bruinsma[1], Daniel Arnold[2], Adrian Jaeggi[2], Noelia Sanchez-Ortiz[3]
	[1]CNES, Toulouse, France; [2]AIUB, Bern, Switzerland; [3]Elecnor Deimos, Tres Cantos, Spain
	In the framework of the ESA GOCE+ projects, thermosphere densities and crosswind speeds were retrieved from GOCE observations for the entire Science Mission from November 2009 to 20 October 2013. The most recent version of the DTM thermosphere models, DTM2013, is constructed with GOCE density data up to May 2012 as well as with GRACE, CHAMP and historical density data. The exceptional value of the unique low-altitude GOCE density dataset will be demonstrated by comparing it to the CIRA models JB2008, NRLMSISE-00 and DTM on long (annual; mission design, lifetime) and short (daily; re-entry prediction) time scales. The models are evaluated according to the following metric: mean, RMS and standard deviation of the observed-to-modeled ratio (unity for an unbiased model), and correlation. A new ESA General Study (PREGO) focuses on the last three weeks of the mission. The ion propulsion was no longer operating after 20 October and GOCE re-entered the atmosphere over the Falkland Islands on 11 November 2013. The altitude decay rate was very high, but the accelerometers continued to operate up to 8 November, and the GPS receiver up to a day before re-entry. Densities down to 180 km could be inferred from the accelerometer data. Results of thermosphere model performance for this last part of the GOCE mission will be presented, and the models’ performance will be compared to earlier (higher altitude) in the mission.
12:00	Comparing SEP Forecast Performances with the SEP Scoreboard	Dierckxsens, M et al.	Invited Oral
	M. Dierckxsens[1], M. Marsh[2], L. Mays[3,4], N. Crosby[1], M. Kuznetsova[4]
	[1]Royal Belgian Institute for Space Aeronomy, Belgium; [2]Met Office, UK; [3]Catholic University of America, USA; [4]NASA Goddard Space Flight Center, USA
	The Royal Belgian Institute for Space Aeronomy and the Met Office UK in collaboration with the Community Coordinated Modeling Center are developing a scoreboard for the predictions of solar energetic particle (SEP) events. Developers of forecast models are able to submit their predictions using a predefined xml schema allowing for an automated real-time validation. Since the submitted forecasts will be stored in a database, verification and validation measures during a longer time period may also be derived. The scoreboard is able to compare near real-time continuous (probabilistic) as well as event-triggered predictions. The SEP Scoreboard Planning Group, currently consisting of the above institutes, intends to also coordinate the submission of predictions for historical events for an “SEP challenge” to compare the various models. This allows predictions which cannot be provided in near real-time such as elaborate physics-based simulations to be included in the comparisons as well. During this presentation, the differences in the various types of SEP forecasts and how they are dealt with in the SEP scoreboard will be explained. Furthermore, the participating models as well as the parameters they are providing will be presented, and the planned validation and verification techniques will be discussed in detail. Finally, a comparison between the predictions for a selected set of historical SEP events will be shown for forecast models from groups that are currently participating in the scoreboard.
12:15	FORSPEF tool: On the Validation and Verification of the nowcasting mode	Anastasiadis, A et al.	Oral
	A. Anastasiadis[1], A. Papaioannou[1], I. Sandberg[1], D. Paronis[1], P. Jiggens[2]
	[1]IAASARS, National Observatory of Athens, Greece; [2]ESTEC/ESA, The Netherlands
	In this work we present the performance of an operational nowcasting tool for Solar Energetic Particle (SEP) events, developed by the National Observatory of Athens (NOA), entitled FORSPEF (Forecasting Solar Particle Events and Flares). The FORSPEF tool incorporates two operational and one non-operational nowcasting modes based, correspondingly, on solar flare, coronal mass ejections and radio flux data. The performance is evaluated for all three modules independently based on a database of SEP events that extends from 1997 to 2013. Based on the module of the nowcasting FORSPEF tool, parameters of the associated parent solar events determine a set of necessary conditions that are being used as a set of control events. The outputs of the modules are then calculated for the rest of the events that constitute the test events. Finally the performance is evaluated using standard verification and validation measures. In particular, since the modules provide probabilistic nowcasts we evaluate the accuracy, reliability, and resolution and display these results using a standard attributes diagram, including skill (SS) and Brier scores (BS). Furthermore, we render the probabilistic nowcasts into categorical scores. We find an optimal probability and we calculate the false alarm rate (FAR) and probability of detection (POD) at this probability. Finally, we further present probabilistic and categorical scores for the on-going operational period of the tool starting from December 2015 onwards. These findings provide an objective basis for measuring future improvements of the FORSPEF modules over time. This work has been funded through the “FORSPEF: FORecasting Solar Particle Events and Flares”, ESA Contract No. 4000109641/13/NL/AK and the “SPECS: Solar Particle Events and foreCasting Studies” program of the National Observatory of Athens.
12:30	IMPTAM verification and validation on GOES MAGED data for long-term variations of electron fluxes at geostationary orbit	Ganushkina, N et al.	Oral
	Ilkka Sillanpää[1], Natalia Ganushkina[1,2], Stepan Dubyagin[1], Juan Rodriguez[3]
	[1]Finnish Meteorological Institute, Helsinki, Finland; [2]University of Michigan, Ann Arbor MI, USA; [3]NOAA, Boulder CO, USA
	We present the results of verification and validation of IMPTAM (Inner Magnetosphere Particle Transport and Acceleration Model, running online (http://fp7-spacecast.eu) since February 2013) model for low energy electrons against the data from GOES MAGED instrument for long-term variations of electron fluxes at geostationary orbit. The GOES 13 MAGED data for the period between January 1, 2011 and March 31, 2015 using 5-minute average electron flux values were analyzed. An empirical model for electron fluxes in 1/(cm2 s sr keV) at energies 40 keV, 75 keV, and 150 keV at geostationary orbit was developed. The developed model includes the dependencies on MLT, solar wind speed Vsw and IMF Bz. The observed and modeled fluxes are very close to each other, the patterns are very similar. The developed empirical model can serve as a boundary conditions for keV electron seed population for further acceleration to higher MeV energies. The comparison with IMPTAM output was done for the period from 23 September 2013 to 31 March 2015. During this period, the models settings were not changed. We use the 5 minute resolution for GOES 13 MAGED data and the IMPTAM output as differential omni-directional fluxes for energies of 40, 75, and 150 keV that are directly comparable. The observed and modeled fluxes are organized by the solar ind and IMF parameters to the magnetopause and the geomagnetic indices. The patterns of how the modeled electron fluxes are distributed in MLT dependent on different IMF and solar wind parameters and geomagnetic indices are rather similar to the observed ones. The location and values of peak fluxes are in a close agreement. At the same time, the higher modeled fluxes with difference reaching one or two orders of magnitude as compared to the observed ones are obtained for larger values of driving parameters and with the location in the dusk sector. This is due to the parameterization of models included in IMPTAM and representation of electron losses, especially, on the duskside. The research leading to these results was partly funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement No 606716 SPACESTORM and by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 637302 PROGRESS.
12:45	Harmonisation of Validation and Verification procedures within the Space Weather Network in the framework of the Space Situation Awareness Programme	Borries, C et al.	Oral
	C. Borries[1], C. Perry[2], A. Devos[3], M. Dierckxsens[4], J. Matzka[5], A. Belehaki[6], A. Glover[7]
	[1]German Aerospace Center; [2]Rutherford Appleton Laboratory; [3]Royal Observatory of Belgium; [4]Belgian Institute for Space Aeronomy; [5]GFZ German Research Centre for Geosciences; [6]National Observatory of Athens; [7]ESA SSA Programme Office, European Space Operation Center
	In the framework of the Space Situation Awareness Programme (SSA), the Space WEather (SWE) segment aims to help end-users in a wide range of SWE affected sectors to mitigate the effects on their systems, reducing costs and improving reliability. In the SWE segment, five Expert Service Centres (ESCs) are providing space weather services to users dependent on space weather conditions. Together with the SSA Space Weather Coordination Centre (SSCC) and the SSA Space Weather Data Centre (SWE-DC), the ESCs form the SSA SWE network. Currently 55 SWE products are provided by numerous Expert Groups (EG) within the SSA SWE network. These will be significantly extended by the end of the currently ongoing development activities in the P2-SWE-I and parallel projects. Each product is delivered to the SSA SWE network with a full package of documentation and user guidance. This includes results of validation and verification procedures helping the end-user to identify the appropriate products for each application. The currently diverse procedures for product validation and verification hamper the comparison and evaluation of different products. Therefore, the SWE network aims to harmonize the validation and verification procedures in the current SSA period. Here, an overview of the ESCs product validation assessment and planning for the current period will be given, including examples of best practice and approaches for common validation and verification procedures.

Posters

Friday November 18, 10:00 - 11:00, Poster Area

1	Validation of the Dynamic Radiation Environment Assimilation Model (DREAM): Metrics and model comparisons	Morley, S et al.	p-Poster
	Steven Morley, Michael Henderson, Geoffrey Reeves, Gregory Cunningham, Andrew Walker, Brian Larsen
	Los Alamos National Laboratory
	The Dynamic Radiation Environment Assimilation Model (DREAM) was developed at Los Alamos National Laboratory to assess, quantify, and predict the hazards from the natural space environment. DREAM was initially developed as a basic research activity to understand and hindcast the dynamics of the Earth's Van Allen radiation belts. DREAM has been transitioned to realtime operation for testing as a nowcasting and forecasting capability. As DREAM is a data-assimilative model covering the region of closed drift paths in the inner magnetosphere, it can be used to predict the environment at any satellite in any orbit whether space environment data are available in those orbits or not. We report here on a recent project to quantify the performance of DREAM, and to compare the performance for different model configurations and different assimilated data sources. The validation approach taken allows consistent comparisons across different electron radiation belt models.
2	Comparison of Empirical Magnetopause Location Models with Geosynchronous Data from 1996 to 2010	Park, E et al.	p-Poster
	Eunsu Park, Yong-Jae Moon
	School of Space Research, Kyung Hee University
	In this study, we identify 298 geosynchronous magnetopause crossings (GMC) using geosynchronous satellite observation data from 1996 to 2010 as well as make an observational test of magnetopause location models using the identified events. For this, we consider three models: Petrinec and Russell (1996), Shue et al. (1998), and Lin et al. (2010). To evaluate the models, we estimate a Probability of Detection (PoD) and a Critical Success Index (CSI) as a function of year. To examine the effect of solar cycle phase, we consider three different time periods: (1) ascending phase (1996-1999), (2) maximum phase (2000-2002), and (3) descending phase (2003-2008). Major results from this study are as follows. First, the PoD values of all models range from 0.6 to 1.0 for the most of years. Second, the PoD values of Lin et al. (2010) are noticeably higher than those of the other models. Third, the CSI values of all models range from 0.3 to 0.6 and those of Shue et al. (1998) are slightly higher than those of the other models. Fourth, the predicted magnetopause radii based on Lin et al.(2010) well match the observed ones within one earth radius, while those on Shue et al. (1998) overestimate the observed ones by about 2 earth radii. Fifth, the PoD and CSI values of all the models are better for the solar maximum phase than those for the other phases, implying that the models are more optimized for the phase.
3	Operational solar flare forecast verification in NICT	Kubo, Y et al.	Invited p-Poster
	Yûki Kubo, Mitsue Den, Mamoru Ishii
	National Institute of Information and Communications Technology
	Recently, forecast verification has been recognized as one of the most important topic in space weather forecast operation. Some Regional Warning Centers (RWCs) belonging to the International Space Environment Service (ISES) have started to verify their operational forecasts. The Community Coordinated Modeling Center (CCMC/NASA) is planning the flare scoreboard, which is an online platform of a real-time solar flare forecast verification. National Institute of Information and Communications Technology (NICT), as RWC Japan/ISES, has also an online platform of the operational solar flare and geomagnetic K-index forecast verification, which compares forecasts of some RWCs. However, as the conditions of the forecasts among the RWCs are not the same, we cannot directly compare the forecast performances among the RWCs. While comparing forecast performances among some RWCs is very informative, we have to proceed the efforts to compare operational space weather forecast among some RWCs. Verifying own forecast performance is the first step to compare the performance of forecast among some RWCs. For the reasons, we started verifying own forecast performance. The verification study uses forecast data issued by RWC Japan as Ursigram codes, especially UGEOA. In this presentation, we introduce methods and results for the verification study of operational solar flare forecast in NICT.
4	Validation of the Swarm plasmapause index PPi	Balazs, H et al.	p-Poster
	Balazs Heilig[1], Hermann Lühr[2], Massimo Vellante[3]
	[1]MFGI, Tihany, Hungary; [2]GFZ Potsdam, Germany; [3]University of L'Aquila, Italy
	Recently, a new method for monitoring the plasmapause location in the equatorial plane was introduced based on magnetic field signatures of small-scale field-aligned currents (SSFACs) observed at LEO (Heilig and Lühr, 2013). In this paper, we present the first validation results, where we use in situ electron density observations of NASA VAP satellites, as well as plasma mass density data inferred from ground based ULF wave observations for comparison. Validation results confirm that the equatorward boundary of SSFACs and the plasmapause stand very close to each other in the midnight local time sector. The midnight PP index proves to be very useful in monitoring the PP dynamics.
5	Model Metrics, verification and validation within the Heliospheric Weather Expert Service Centre	Perry, C et al.	Invited p-Poster
	Chris Perry[1], Mark Gibbs[2], Manuela Temmer[3], Volker Bothmer[4], Vincent Genot[5], Daniel Heynderickx[6], Stefaan Poedts[7], Susanne Vennerstrom[8]
	[1]STFC Rutherford Appleton Lab; [2]UK Met Office; [3]Institute of Physics University of Graz; [4]University of Goettingen; [5]CDPP/IRAP; [6]DHConsultancy; [7]KU Leuven; [8]Technical University of Denmark
	The newly established SSA/SWE Heliospheric Weather Expert Service Centre (H-ESC) is heavily dependent on physics based and empirical models for provision of its high priority forecast products. In this presentation we provide an overview of the metrics, validation and verification techniques that are part of the on-going H-ESC product assessment activities.
6	Validation studies of the Solar Wing driven autoregressive model for Ionospheric short-term Forecast (SWIF) for future improvements	Tsagouri, I et al.	p-Poster
	Ioanna Tsagouri
	National Observatory of Athens, Greece
	The SWIF model (Solar Wind driven autorgressive model for Ionospheric short-term Forecast) is operationally implemented in the DIAS system (http://dias.space.noa.gr) to provide ionospheric forecasting products and services to DIAS and ESA/SSA/SWE Service Network (http://swe.ssa.esa.int/) users. These include alerts and warnings for upcoming ionospheric disturbances, as well as single site and regional forecasts of the foF2 characteristic over Europe up to 24 hours ahead. The model undergoes continuous evaluation in its performance to test prediction efficiency and accuracy of the model over a range of disturbed space weather conditions, from moderate to intense. This presentation aims to summarize the results of recent validation studies that expand the tests from regional to global scale and drive the improvement of the model's empirical formulation towards more accurate forecasts.
7	Nowcast and forecast of the F30 solar index for orbit prediction needs	Yaya, P et al.	p-Poster
	Philippe Yaya[1], Sean Bruinsma[3], Thierry Dudok de Wit[2], Louis Hecker[1], Clémence Le Fèvre[3], Pascal Perrachon[3]
	[1]CLS (Collecte Localisation Satellites), Ramonville Saint-Agne, France; [2]LPC2E (Laboratoire de Physique et Chimie de l’Environnement et de l’Espace), Orléans, France; [3]CNES (Centre Nationale d’Etudes Spatiales), Toulouse, France
	Modeling the upper atmosphere density is essential to determine and predict satellite orbits at low altitude. Density variations are primarily driven by solar activity, for which the 10.7 cm radio flux is routinely used as a solar proxy. However, it has been demonstrated that DTM-2013 model is more efficient with the 30 cm radio flux for altitudes lower than 500 km. The Nobeyama Radio Observatory (NRO), a branch of National Astronomical Observatory of Japan (NAOJ), performs daily measurements of the 30 cm radio flux on an operational 7/365 basis. Additional measurements are made at 15, 8.2, 3.0, and 1.8 cm since 1957. In a first step, the present work consists in correcting the provisional values (up to 30 days old) for outliers. To complete the preprocessing, data gaps are filled in by expectation-maximization, and flares are removed. Finally, taking advantage of the multi-wavelength observations, a forecast time series is computed on a 30 days horizon using a non-recursive analogue neural network. The performance of the 30 cm index prediction is discussed and compared to existing methods. The impact on orbit prediction is also presented. An online service on CLS web site is described. It contains the corrected historical time series from 3 to 30 cm, as well as the forecast time series which is updated daily.
8	Verification of geomagnetic storm forecasts at the UK Met Office	Bingham, S et al.	p-Poster
	Michael Sharpe, Suzy Bingham, David Jackson, Edward Pope
	UK Met Office
	The Met Office Space Weather Operations Centre (MOSWOC) monitors space weather 24/7. Twice daily forecasts and timely alerts and warnings are produced by a dedicated space weather forecaster. Verification of forecasts is important for forecasters, customers, modellers and stakeholders; the verification process improves the quality, skill and accuracy of the service. Here we present verification results for four day multi-category probability forecasts of geomagnetic storms. For this work, we have adapted automatic, near real-time terrestrial weather verification systems. Forecast performance is evaluated by comparing against a reference climatology forecast. Ranked Probability Skill Score (RPSS) is calculated to measure the relative improvement of the probability forecast in predicting the category of the observation. Results are plotted to show rolling monthly forecast performance. The verification process is performed automatically in near-real time providing forecasters with information to improve subsequent forecasts.
9	Flare forecast verification at the UK Met Office	Bingham, S et al.	p-Poster
	Sophie Murray, Suzy Bingham, Edward Pope, David Jackson
	UK Met Office
	One essential component of operational space weather forecasting is prediction of X-ray flares. The Met Office Space Weather Operations Centre (MOSWOC) produces a 4 day probability forecast for X-ray flares, which is disseminated twice a day in a guidance document. Verification of these predictions provides an understanding of the strengths and weaknesses of the flare forecasting process. Here we present results from comparisons between MOSWOC forecasts and flare observations. Forecast performance is demonstrated by comparison with flare climatology. The Ranked Probability Skill Score is calculated to quantify the extent to which the forecast improves predictions with respect to the flare climatological benchmark forecast. To evaluate forecast resolution, Relative Operating Characteristic (ROC) curves are calculated to show the ability of a forecast to discriminate between events and non-events. To understand forecast bias, reliability diagrams are plotted to show how well forecasted probabilities of an event correspond to observed frequencies. We note the difficulty in verifying X-class flares due to few events since archiving of data began in 2014. The flare verification process is implemented automatically in near real-time. This enables forecasters to improve subsequent predictions.
10	CME arrival-time validation of real-time WSA-ENLIL+Cone simulations at the CCMC/SWRC	Wold, A et al.	p-Poster
	Alexandra M. Wold[1], M. Leila Mays[2,3], A.Taktakishvili[2,3], L. Jian[4,3], D.Odstrcil[2,5] , P. MacNeice[3]
	[1]American University; [2]Catholic University of America; [3]NASA Goddard Space Flight Center; [4]University of Maryland College Park; [5]George Mason University
	The Wang-Sheeley-Arge (WSA)-ENLIL+Cone model is used extensively in space weather operations world-wide to model CME propagation, as such it is important to assess its performance. We present validation results of the WSA-ENLIL+Cone model installed at the Community Coordinated Modeling Center (CCMC) and executed in real-time by the CCMC/Space Weather Research Center (SWRC). The SWRC is a CCMC sub-team that provides space weather services to NASA robotic mission operators and science campaigns, and also prototypes new forecasting models and techniques. CCMC/SWRC uses the WSA-ENLIL+Cone model to predict CME arrivals at NASA missions throughout the inner heliosphere. In this work we compare model predicted CME arrival-times to in-situ ICME shock observations near Earth (ACE, Wind), STEREO-A and B for simulations completed between March 2010 - July 2016 (over 1500 runs). We report hit, miss, false alarm, and correct rejection statistics for all three spacecraft. For hits we compute the bias, RMSE, and average absolute CME arrival time error, and the dependence of these errors on CME input parameters. We compare the predicted geomagnetic storm strength (Kp index) to the CME arrival time error for Earth-directed CMEs. The predicted Kp index is computed using the WSA-ENLIL+Cone plasma parameters at Earth with a modified Newell et al. (2007) coupling function. We also explore the impact of the multi-spacecraft observations on the CME parameters used to initialize the model by comparing model validation results before and after the STEREO B communication loss (since September 2014) and STEREO-A side-lobe operations (August 2014-December 2015). This model validation exercise has significance for future space weather mission planning such as L5 missions.
11	From the NEAR EARTH SPACE / SPACE WHEATHER Window the BIG DATA ERA	Tulunay, Y et al.	e-Poster
	Yurdanur Tulunay[1], Ersin Tulunay[2]
	[1]METU Dept of Aerospace Engineering, Ankara; [2]METU Dept. of Electrical Engineering, Ankara
	Near Earth Space processes are mostly nonlinear and time varying. Therefore, data driven models, as scientists call “evidence-based decision making” , have proven to be more attractive in modelling such systems to be employed in parallel to the mathematical models based on the first physical principals. We have been dealing with the data driven models since around 2000. Even in those times, which may be considered as relatively recent, it was difficult to access independent representative data to be employed in the “training”, “testing”, and “validation” phases. As we understand, from the concept of “metrics” it is the term covers the well set-up criteria to compare the performances of various data driven models of the same process. The existence and availability of reliable data sources are vital in both scientific and technological developments. During the recent years, mainly, due to the developments in digital electronics and space technologies, huge amount of data has been obtained as the results of space and Earth bound measurement and monitoring campaigns. Thus the term of “big data “have become an important issue. Although, “big data size” is a constantly moving target it is the time to develop a set of new techniques and technologies with new models and integration to extract representative characteristics of data sets that are diverse, complicated, and of a big scale. The basic characteristics of big data involve “high volume- amount of data”, “velocity-speed of data in and out”, and “variety-range of data types and sources”. At present, from the model making point of view, one of the urgent issues is the development of new signal processing techniques to extract manageable representative data out of the relevant big data ranging from a few dozen terra bytes, to for example, many peta bytes (as of 2012). For example, one will need to improve, “pattern recognition” techniques and identify the “outliers” depending on the objective of the action. In this poster our intention is to stress that it is time to get ready!