FG: Metrics and Validation

Co-Chairs: Tim Guild <timothy.b.guild [at] aero.org>, Howard Singer <howard.singer [at] noaa.gov>, and Lutz Rastaetter <lutz.rastaetter [at] nasa.gov>

ULF Wave Modeling Challenge

We're organizing a modeling challenge to investigate the ability of global models to reproduce ULF wave fields in the magnetosphere, and ultimately if those ULF fields can radially diffuse an existing electron population from L~5.5 to L~4.5 in two weeks.

Paragraph 12 of Claudepierre et al., 2010 discusses "continuum" of solar wind frequencies http://onlinelibrary.wiley.com/doi/10.1029/2010JA015399/full#jgra20604-sec-0002

Sessions at the 2013 Summer Workshop

(We welcome additional contributions.)

Session 1: New Validation Results and Methods

Thursday, 1:30 PM – 3:00 PM (Chair: Tim Guild)

In this session we solicit short presentations focusing on all aspects of data-model comparisons. Welcomed themes could include geospace model validation using regional ground magnetometer indices, model responses to dynamic magnetopause boundaries, long-term "climatological" model validation, using multi-variate metrics to assess model performance, or any related contribution.

• Frank Toffoletto: Initial result from LFM RCM for Oct 2012 Storm
• Lutz Rastaetter and Howard Singer: dB/dt study (MHD and empirical) results / discussion
• Alex Glocer: Calculation of K index values from model magnetic perturbations
• Victoir Veibell and Bob Weigel (presented by Tim Guild): Climatological analysis of 12 months of MHD simulated ground magnetometer measurements
• Gabor Facsko: GUMICS-4 verification and footprint determination using static and dynamic global MHD simulations of the ECLAT project
• Roxanne Katus and Michael Liemohn: Climatological Validation of the HEIDI Ring Current Model
• Haje Korth: Statistical maps of TIMED/GUVI auroral emissions as a new global MHD validation data source

These next two talks may move to the beginning of the next session (Thursday 3:30-5:00).

• Josh Rigler: Metrics derived from multi-variate statistical distributions
• Asher Pembroke: Interpolating into "Hard to reach places" of MHD volumes for more accurate DST

Session 2: How Validation Studies Guide Model Improvements

Thursday, 03:30 PM – 5:00 PM: (Chair: Howard Singer)

In this session, we solicit short presentations on specific examples of how discrepancies between models and observations have led to a better understanding of the physics that needs to be incorporated into models. Studies, or ideas, on studies involving model sensitivity to inputs and various model parameters are welcome. The session will include a discussion on better ways to design metrics and validation studies to uncover missing physics in models.

• Howard Singer: Session Intro and Comments on Ensemble Modeling
• Dan Welling : Multi-Model Validation with the Space Weather Modeling Framework
• Margaret Chen: The effect of uncertain boundary conditions on ring current model validation
• Andrey Samsonov: Large-scale flow vortices following a magnetospheric sudden impulse (Simulations were done with SWMF, OpenGGCM and LFM.)
• Slava Merkin: Lessons from MHD-AMPERE comparisons
• Lutz Rastaetter: Magnetopause Challenge of 2010 and steps from there: Eents for future MP challenge

Session 3: Validation of MHD models coupled with other modules such as the Ring Current and Polar Outflow

Friday: 10:30 AM – 12:15 PM (Chair: Lutz Rastaetter)

In this session we will investigate recent advances in the development of coupled models of the magnetosphere, inner magnetosphere, the plasmasphere/polar wind, and the ionosphere/thermosphere. We solicit short presentations on methods to link models, validate the outputs and constrain coupled model systems using observations. Emphasis should be placed on new physical processes that are described by the coupled models and on effects that remain out of scope and that limit the effectiveness and stability of the coupling.

• Joachim Raeder (co-authors Gilson & Toffoletto), OpenGGCM - RCM validations
• Raluca Ilie: Coupling of SWMF magnetosphere and Heidi Ring Current
• Michael Wiltberger: Validation of MFLFM Outflow Simulations
• All: Discussion on future Metrics and Validation Activities to Support GEM goals

If you have topics for discussion, please consider communicating them to the conveners before the session.

Focus Group Proposal (December 2010)

• Metrics and Validation by Masha Kuznetsova, Aaron Ridley, Tim Guild, Lutz Rastaetter, and Howard Singer

Metrics and Validation Focus Group Status Report

Current Co-Chairs (2005-2012): Masha Kuznetsova (CCMC), Aaron Ridley (Univ. of Michigan). New Co-Chairs (2012-2015); Tim Guild (Aerospace Corp), Lutz Rastaetter (CCMC), Howard Singer (NOAA)

In summer 2008 the GGCM Metrics and Validation Focus Group initiated the GGCM modeling Challenge. The goals of the Challenge:

• Evaluate the current state of the space science models and track model improvements over time

• Facilitate interaction between research and operation communities

• Facilitate collaboration among modelers and data providers and research communities.

• Facilitate further model improvement.

The project focuses on various scientific and operational aspects of GGCM model performance and addresses challenges of model-data comparisons and metrics studies (e.g., model output and observational time series preparation for comparison, measurement uncertainty, metrics selection).

The following model/data comparisons (Matrics Studies) are involved in the Modeling Challenge:

Metric Study 1: Magnetic field at geosynchronous orbit (GOES)

Metric Study 2: Magnetopause crossings by geosynchronous satellite (GOES and LANL)

Metric Study 3: Plasma density/temperature at geosynchronous orbit (LANL)

Metric Study 4: Ground magnetic perturbations (ground based magnetometers)

Metric Study 5: Dst index (added at 2009 summer workshop)

Metric Study 6: Heat flux into ionosphere (added at 2010 summer workshop)

Metrics Study 7: Auroral oval position (study of interest to US Airforce, details to be discussed at fall 2010 mini-workshop)

Four events were selected at the GEM 2008 Workshop:

Event 1: Oct 29, 2003 06:00 UT - Oct 30, 06:00 UT (Kp = 9, Dst_min=-353nT)

Event 2: Dec 14, 2006 12:00 UT - Dec 16, 00:00 UT (Kp = 8, Dst_min=-139nT)

Event 3: Aug 31, 2001 00:00 UT - Sep 01, 00:00 UT (Kp = 4, Dst_min=-40nT)

Event 4: Aug 31, 2005 10:00 UT - Sep 01, 12:00 UT (Kp = 7, Dst_min=-131nT)

Two more events were added in summer 2010 for metrics study 6 on request of Dayside FAC and Energy Deposition focus group:

Event 5: May 15, 2005 00:00 UT - May 16, 2005, 00:00 UT

Event 6: July 9, 2005 00:00 UT - July 12, 2005, 00:00 UT

Status of on-going metrics studies and future plans:

1) Ground magnetic perturbations and magnetic field at geosynchronous orbit. The first results of metrics studies 1 and 4 discussed at GEM 2009 and 2010 summer workshops are available at the CCMC website and at the GEM wiki. Two papers (with all Challenge participants as co-authors) are submitted to the Space Weather Journal. Pulkkinen, A., M. Kuznetsova, A. Ridley, J. Raeder, A. Vapirev, D. Weimer, R. Weigel, M. Wiltberger, G. Millward8, L. Rastaetter, M.Hesse, H. J. Singer and A. Chulaki, GEM 2008-2009 Challenge: ground magnetic field perturbations, submitted to Space Weather Journal, 2009 Rast?atter, L., M. M. Kuznetsova, A. Vapirev, A. Ridley, V. G. Merkin, A. Pulkkinen M. Hesse, H. J. Singer, GEM 2008-2009 Challenge: magnetic fjelds at geosynchronous satellite positions, submitted to Space Weather Journal, 2010. The principal conclusion of these studies is that ranking of the models is strongly dependent on the type of metric (the way how observational and modeled time series are compared) used to evaluate the model performance. Type of metrics to be used depends on the application and should be defined by model science and operational users. Metrics 4 is suggested for the Operational Metrics by NOAA SWPC. On NOAA SWPC request metrics format will be modified to focus on regional Kp and dB/dt. CCMC is working with the community to develop an automated system that will allow repeating the exercise on regular basis. We will continue dialog with the operations community to refine the metrics format. The second round of model-data comparison is planned for spring 2012 with the first report on model improvement over time to be presented at 2012 2ummer workshop.

2) Plasma parameters at the geosynchronous orbit.

To proceed with the study SOPA ion corrections for MPA are needed for pressure comparison. A group of LANL scientists are working on removal of electron contamination from the low- energy ion channels of SOPA. We exxpected completion of the first round of model-data comparison before the end of 2012.

3) Magnetopause crossings by geosynchronous satellite.

Preliminary analysis of Challenge results demonstrated significant differences between observations and model predictions. The unexpected result is that magnetopause standoff distance varies significantly for different global MHD codes. To address the issue in more details we initiated baseline model comparison study, when researchers run different MHD codes with different settings for various solar wind and IMF conditions and compare the results. Preliminary analysis of the results at 2010 summer workshop excluded a number of possible sources for disagreement (e.g., such as grid resolution or value of anomalous resistivity). The latest idea is that the most likely source for the discrepancies are differences in inner boundary conditions. The decision was made to continue the study with the goal to resolve the issue within the next two years. We plan to initiate collaborations on this topic with FGs in magnetopause research area and with FG magnetic reconnection.

4) Dst index.

The Dst index Challenge originated in summer 2009 is a joint project with Inner Magnetisphere Focus group (liaison Dan Welling). The preliminary results were discussed at 2010 summer workshop. Report on the first round of study will be presented at the fall 2010 mini-workshop. Different ways of Dst index calculations are currently being analyzed. We plan to finalize the paper to be submitted to the Space Weather Journal before the summer workshop 2011.

5) Heat flux into ionosphere.

Heat flux into ionosphere metrics study was initiated in summer 2010. This is a joint project with Dayside FAC and Energy Deposition focus group (liaison Stefan Eriksson). The study involves comparison with Poynting flux time series derived from DMSP observations. To facilitate the research two additional events were added to the original four GEM events. This study is of potential interest to CEDAR community (as was agreed at the CEDAR 2010 summer workshop). Simulations, data preparation, and tool development for model output postrocessing are currently on the way. We plan to present the first results for the discussion at summer workshop 2011. We expect this project to continue until the end of 2012 and beyond.

6) Auroral oval position.

In summer 2010 we agreed on the importance to include auroral oval position metrics study. This metrics study is of primary interest to a number of GGCM model operational users including US Air Force. This study is also of a potential interest to the CEDAR community. The discussion on the details of this challenge is planned for the fall 2010 mini Workshop. We plan to continue this study beyond 2012.

Joint GEM-CEDAR Challenge

To analyze the effects geospace environment on the ionosphere we are planning to involve CEDAR Community to the Modeling Challenge. This activity is especially relevant due to upcoming joint GEM-CEDAR Workshop in 2011. Preliminary discussion with ionosphere modelers at CEDAR was held at CEDAR 2009 and 2010 Workshops. GEM M&V FG co-chairs were involved in the organizing of the CEDAR Electrodynamics-Thermosphere-Ionosphere Challenge. CEDAR modelers included GEM challenge events for the series of planned comparative studies. Further, many of the magnetospheric models are coupled to ionosphere-thermosphere models, therefore, the metrics can be conducted for both coupled and uncoupled simulations on both models.

Baseline model comparison

An aspect of validating models is to determine how well they solve the equation set that they are supposed to be solving. This type of validation has to do with checking the numerics of the codes. The method in which this validation will be conducted is to have researchers run different MHD codes for various solar wind and IMF conditions and compare the results. For this, we have requested that modelers run a time period in which the solar wind stays constant at 5/cc and -400 km/s, while the IMF Bz drops every few hours from -5, -10, -20, -30, -35, to -40 nT. The ionospheric conductance was held constant at 5 mhos. Another series of runs will change the solar wind ram pressure, while keeping the IMF Bz equal to – 10 nt. Initial simulation results were presented at 2009 and 2010 summer workshops. It was shown that many of the models had significant differences. It was decided upon that the subsolar magnetopause location, the cusp location, and the current structures at the magnetopause will be investigated under the different IMF conditions. Further investigations will include (1) examining the reconnection rate in the different codes to determine how each handles reconnection for the different IMF orientations; (2) examining the tail length, the amount of open flux in the polar caps and the plasma sheet density during the different intervals; (3) the cross polar cap potential; and (4) the Dst as a function of time. With these types of comparisons, the GGCM Metrics and Validation Focus Group can collaborate with other focus groups within the GEM community. It is expected that the FG will arrange a special issue of a journal to capture the results of these comparisons.