NASA Center for Climate Simulation

NCCS Supercomputer’s Newest Unit Hosts NASA Weather Forecasting Model Tests

The newest and most powerful addition to the NASA Center for Climate Simulation (NCCS) Discover supercomputer—Scalable Compute Unit 14 (SCU14)—has been hosting 40-day simulations testing an updated version of NASA’s weather forecasting model.

Discover’s SCU14 corrals 20,800 processor cores for peak performance of nearly 1.6 petaflops—a staggering 1,600,000,000,000,000 calculations per second. This computing powerhouse is proving to be an ideal home for tests with the Goddard Earth Observing System (GEOS) model. Four times a day, GEOS produces global weather forecasts for NASA satellite instrument teams, field campaigns, and weather and climate research.

Model Advances
The next production version of GEOS will incorporate a variety of advances from NASA Global Modeling and Assimilation Office (GMAO) scientists and collaborators.

Like many other computer models of the Earth, GEOS divides the atmosphere into a “finite volume,” a gigantic three-dimensional grid of boxes. GEOS specifically uses a “cubed-sphere” grid defined by projecting a cube from the center of the Earth onto the planet surface and thereby creating six discrete curvilinear faces. Depending on the application, GEOS grid boxes get stacked 72 or 132 levels high.

Inside the grid boxes, the model solves equations describing the flow of the atmosphere and the physical processes that compose our planet’s weather. GEOS divides those responsibilities among multiple components, several of which have undergone recent improvements.

Handling the atmospheric flow, as well as the thermodynamics equations, is the finite-volume cubed-sphere (FV3) dynamical core. The GMAO and National Oceanic and Atmospheric Administration (NOAA) jointly developed and regularly enhance the FV3 dynamical core—an integral part of both GEOS and NOAA’s new FV3-based Global Forecast System (GFS) that went into operational use for U.S. weather forecasts on June 12.

NASA’s Goddard Earth Observing System (GEOS) model divides the Earth’s atmosphere into a grid of boxes in a “cubed-sphere” configuration. Although this illustration shows only one level of grid boxes, GEOS uses 72 or 132 levels of boxes when simulating weather patterns.

A major enhancement to the FV3 dynamical core implemented over the past few years is a “non-hydrostatic” capability.

“In general, the Earth’s atmosphere can be assumed to be in hydrostatic equilibrium, meaning that upward-directed pressure gradient forces are evenly balanced by the downward pull of gravity,” said William Putman, GMAO research meteorologist and model development lead. The production version of GEOS currently uses grid boxes 12 kilometers (km) wide. Putman said that the hydrostatic assumption no longer holds at 12-km and finer grid resolutions, and “it becomes critical to accurately represent the small-scale details of vertical motion in the atmosphere.” That feat is accomplished by FV3’s non-hydrostatic solver.

On the physics side, another challenge for computer models is capturing weather phenomena smaller than the grid boxes. When the model cannot resolve such phenomena directly, scientists use statistical estimates called “parameterizations.” For GEOS the GMAO is evaluating two new parameterizations for better representing atmospheric convection.

Like water boiling in a pot, atmospheric convection “occurs when an air mass is buoyant, that is, less dense, than its surroundings,” explained Nathan Arnold, GMAO research meteorologist. “Convection transports heat and moisture away from Earth’s surface, producing everything from shallow cumulus clouds to towering thunderstorms.”

The new Grell-Freitas Convection Parameterization (named after Georg Grell of NOAA’s Earth Systems Research Laboratory and Saulo Freitas of the GMAO) represents “deep” convection, with cloud tops at the upper troposphere (~10 km) and favoring production of heavy rain. Compared to the previous GEOS convection parameterization, Grell-Freitas can more readily adapt to changing grid resolutions and interact with aerosols.

The new University of Washington shallow cumulus parameterization estimates “shallow” convection, found in cloud tops generally below 2–3 km and producing little rain. The currently operational GEOS does not have a separate shallow cumulus parameterization and thus generally underestimates the amount of shallow convection.

Other physics improvements in GEOS include updated cloud microphysics, allowing aerosols to influence clouds’ radiative properties, as well as advanced radiative transfer schemes, improving simulations of outgoing longwave radiation (energy).

Collectively, the FV3 non-hydrostatic dynamical core, new parameterizations, and other advances enable GEOS to become “scale-aware,” where the model automatically and seamlessly switches from using a parameterization to directly resolving the phenomena as grid resolutions increase. As Arnold describes it, “The FV3 dynamical core begins to directly resolve the movement of air associated with convective updrafts, while the cloud microphysics scheme determines how water changes phase within those updrafts, for example, producing suspended condensate (cloud) and precipitation.”

Improving Forecasts
Leveraging the updated GEOS model and Discover SCU14, the GMAO ran a series of 40-day simulations at global grid resolutions ranging from 200 to 3 km. Model performance scaled well with increasing processing demands. For instance, the 12-km (current GEOS production resolution) simulations completed in ~15 hours using 4,068 cores, while the 3-km simulations completed in ~14 days using 20,760 cores. Total output data of nearly 86 terabytes sits on Discover’s online disk. NCCS system administrators assisted the GMAO researchers with using SCU14’s unique architecture.

Results thus far show generally improved forecast statistics, with smaller errors in predicted temperature, humidity, and winds. “Some aspects of cloud cover and radiative balance (reflected sunlight and infrared radiation) are also more realistic,” Arnold noted.

The GMAO will continue evaluating the forecast skill of the updated GEOS model, and researchers estimate putting it into daily operations sometime during Summer 2019.

These model development and evaluation activities are also preparing the way for NASA’s future forecasting capabilities, and Discover is playing a vital role. “As we push GEOS to resolve finer scales in the atmosphere, we demand more and more from the compute capability of the Discover cluster at the NCCS,” Putman said. “With the recent expansion of Discover, it has become viable to routinely explore 3-km simulations with GEOS for process studies including convection and severe weather.

“While we are likely a decade away from regularly running global models at this resolution in production forecasts/simulations, having the ability to evaluate GEOS today at these resolutions allows us to achieve the critical science developments that will make the production system in 10 years the most advanced global Earth system prediction model possible.”

More Information

“A Scale-Aware Representation of Convection in the GEOS Model,” GMAO Science Snapshot, 4/10/19.

Freitas, S.R., G.A. Grell, A. Molod, M.A. Thompson, W.M. Putman, C.M. Santos e Silva, and E.P. Souza, 2018: Assessing the Grell-Freitas Convection Parameterization in the NASA GEOS Modeling System. J. Adv. Model. Earth Sys, 10, 1266–1289, doi: 10.1029/2017MS001251.

Arnold, N., and W. Putman, 2018: Non-Rotating Convective Self-Aggregation in a Limited Area AGCM. J. Adv. Model. Earth Sys, 10, 1029–1046, doi:10.1002/2017MS001218.

“NASA Global Weather Forecasting Jumps Forward,” NCCS Case Study.

Jarrett Cohen, NASA/Goddard Space Flight Center