RG: Modeling Methods and Validation

Chairs

Mike Liemohn
Lutz Rastaetter
Alexa Halford
Josh Rigler

Contact:

Resource Group Leads

Community Discussion

Resource Group Formation

Since 2005, the Metrics and Validation and Modeling Methods and Validation Focus Groups have supported modeling efforts at GEM. There was recently a grassroots movement to change the approach to have a standing committee – the Methods & Validation Resource Group. A proposal was approved by the GEM SC in 2019 to play that role instead. The M&V Resource Group has four members on staggered terms of four years, with two new members selected every two years after an announcement in the GEM Messenger, usually prior to mini-GEM.

2020 Activities

Virtual GEM Summer Workshop (July 2020)

The Methods and Validation (M&V) Resource Group arranged for a plenary session tutorial from Dr. Tara Jenson, “METplus - A Unified Verification and Validation Tool for a Coupled Modeling System.” She presented the NOAA forecast verification process and statistical techniques for data-model comparisons. She showed examples of how some of these tools could be used with forecasts of total electron content, solar wind parameters, and auroral zone locations. The only breakout session led by the M&V group was in conjunction with the 3D Ionospheric Electrodynamics and Its Impact on MIT Coupling (IEMIT) Focus Group (FG), in relation to the ionospheric conductance challenge. We also prepared a single slide about the resource group that was shown in nearly every breakout session throughout the week, raising awareness of our existence, plans, and availability to assist with community-wide challenges and metrics assessments. Lutz Rastätter was selected among the M&V leadership to serve as the Resource Group’s representative on the GEM Steering Committee.

Virtual Mini-GEM workshop (pushed back to January 2021)

At the GEM mini-workshop, the M&V leadership led one breakout session to have a discussion about what the community wants and expects from this Resource Group. With 17 participants, it had no formal presentations but a lively conversation generating many ideas about possible directions.

Near-term goals

From these awareness-building activities, discussions, and feedback, the main near-term goals for the M&V Resource Group are as follows:

1. Conduct a thorough and systematic survey of past GEM FG chairs and challenge leaders about their experiences running these community-wide projects
2. Development of a GEM Challenge Best Practices document
3. Disseminate the key guidelines from this document to all FG leaders and potential leaders writing new FG proposals
4. Participate in the organization of challenges with any FG that desires assistance, including posing the correct question set to be addressed, event selection, conducting simulations, managing output, metrics selection, metrics calculation, result dissemination, and archival report development.
5. Strategize about possible event challenges and modeling parametric/sensitivity studies to be conducted and then actively encourage the community to undertake these activities

2021 Activities

Virtual GEM Summer Workshop (July 2021)

The GEM Methods and Validation (M&V) Resource Group will host a VGEM session on Friday, July 30th, 2021 (after lunch break, 1-2:30pm EDT). In brief, the M&V Resource Group exists to facilitate geospace model validation efforts, to document and promote common and best practices for M&V amongst the GEM community, and when possible, to archive validation M&V results for future analysis and comparisons. This will be a relatively open session to share recent tools, methodologies, and results of validation efforts related to Geospace modeling. Topics may cover any/all current and recent GEM focus group efforts. Discussion of community modeling "challenges" is certainly welcome, however we would like to keep this session focused on specific tools and results. Please reach out to the M&V team (gem-methods-validation-leads@googlegroups.com) if you have anything substantial to share, and we'll try to carve out a short block of time for you to present. Or just bring a slide or two to add to interactive discussion. This will be the last set of concurrent sessions for the week, and we really hope to encourage open and productive cross-focus-group discussion and interactions that will continue after the close of VGEM 2021.

2021 mini-GEM

Hybrid session at the 2021 mini-GEM, Dec. 12, 2021

Topic: GEM M&V Mini-GEM Session: M&V White Papers Time: Dec 12, 2021, 02:00 PM Central time (New Orleans) / 03:00 PM Eastern time

The GEM Methods and Validation Resource Group will host a session, “Methods and Validation White Paper Discussion,” at the upcoming 2021 Hybrid Mini-GEM Workshop on Sunday, December 12, at 2:00 – 3:30 pm Central US time. We welcome all Mini-GEM attendees to join in this conversation, either in person at the hotel conference room (at the Hilton Garden Inn New Orleans Convention Center) or remotely via Zoom (link available to Mini-GEM registrants). Please bring your ideas for potential white papers to the upcoming Solar and Space Physics Decadal Survey regarding modeling methodologies and data-model comparison validation techniques. If you would like to present a slide or two on your idea, then please come prepared to do so.

2022 Activities

2022 GEM Summer Workshop

We welcome all attendees at the upcoming GEM Workshop to participate in the three breakout sessions of the Modeling Methods and Validation Resource Group. All three sessions will occur on Wednesday, June 22, in the Stingray Room on the third floor. The sessions are as follows:

10:30 am - noon: MM&V-related Decadal white papers from across the community

1:30 pm - 3:00 pm: MM&V Decadal white paper formulation session

3:30 pm - 5:00 pm: MM&V recent developments and open topics

The first and third sessions are open to contributed talks. For the ""white-paper"" session, please email Mike Liemohn (liemohn at umich.edu) to get on the schedule. We encourage presentations from anyone leading the creation a Decadal Survey white paper that includes an emphasis on modeling approaches or data-model comparisons to geospace science.

For the ""recent developments"" session please email Lutz Rastaetter (lutz.rastaetter at nasa.gov). We encourage presentations from anyone conducting robust model validation studies or applying model validation techniques in novel ways.

The second session will start with brief presentations by the RC leads on the following 3 white paper ideas:

(1) systematic V&V in space physics;

(2) data needs for space physics models; and

(3) support for community modeling resources.

This will be followed by small-group discussions on each topic. If you are unable to attend GEM and would like to contribute, please let us know and we can make sure you will have access to the document and be able to work asynchronously with our group.

We look forward to seeing you at the GEM Workshop!

2023 mini-GEM

Hybrid session at the 2023 mini-GEM, Dec. 10, 2023

We cordially invite you to the MMV sessions on Open Science and Cloud Platform Utilization at the 2023 Mini-GEM on Sunday, Dec. 10 at the Holiday Inn San Francisco – Golden Gateway (500 Van Ness Avenue, San Francisco). There are two sessions: 12:00 – 13:30 and 13:45 – 15:15 in Gold Rush B.

Rebecca Ringuette will present and discuss the “Magnetopause Open Validation Experiment (MOVE).” In addition, the Heliophysics Digitial Resource Library (HDRL) is planning to solicit feedback on their cloud platform. Representatives from CCMC will present their approach to cloud computing and demonstrate initiatives that support open science. Most of the time in our sessions will be dedicated to discussion.

If you would like to contribute by presenting, please send us (Lutz Rastaetter at lutz.rastaetter@nasa.gov and Michael Liemohn at liemohn@umich.edu) your 5 minutes’ worth of slides or see us at the beginning of each session.

The Zoom link for virtual participation will be available via the GEM website. Please do not forget to register for mini-GEM.

Lutz Rastaetter, Michael Liemohn, Alexa Halford, Josh Rigler

Ongoing Activities

We are gathering information on past modeling and validation studies that have been performed in recently completed Focus Groups and in groups that are nearing their end. Soon after the GEM workshop we plan to contact leaders for information on their experiences on modeling and validation (M&V) and create a document that we will post here.

We intend this document to be a continuously growing resource for existing Focus Groups and new Focus Groups ahead of proposals in the Fall.