Framework for Managing Ultra-Large Climate Datasets

Summary

Fueled by exponential increases in computational and storage capabilities of high-performance computing platforms, climate model simulations are evolving toward higher numerical fidelity, complexity, volume, and dimensionality. Data holdings are projected to reach hundreds of exabytes worldwide by 2020. Such explosive growth presents both challenges and opportunities for scientific breakthroughs. A U.S. Department of Energy (DOE) funded project, Ultrascale Visualization Climate Data Analysis Tools (UV-CDAT), is addressing these challenges, with a team from four DOE laboratories (Lawrence Berkeley, Lawrence Livermore, Los Alamos, and Oak Ridge); two universities (Polytechnic Institute of New York University and University of Utah); National Aeronautics and Space Administration at Goddard Space Flight Center; and two private companies (Kitware and Tech-X).

UV-CDAT software tools address:

1) problems with “big data” analytics;

2) the need for reproducibility;

3) requirements to push ensemble analysis, uncertainty quantification, and metrics computation to new boundaries;

4) heterogeneous data sources (simulations, observations, and re-analysis); and

5) provision of an overall architecture for incorporating existing and future software components.

The team designed a Python-based framework that integrates several disparate technologies under one infrastructure. United by standard common protocols and application programming interfaces, UV-CDAT integrates more than 40 different software components. The primary goal of this nationally coordinated effort is to build an ultrascale data analysis and visualization system empowering scientists to engage in new and exciting data exchanges, thus enabling breakthrough climate science. The framework is established to evolve and incorporate the new software tools that the science and scientific community require.

References

Williams, D. N., T. Bremer, C. Doutriaux, J. Patchett, G. Shipman, B. Haugen, R. Miller, B. Smith, C. Steed, E. W. Bethel, H. Childs, H. Krishnan, M. Wehner, C. T. Silva, E. Santos, D. Koop, T. Ellqvist, H. T. Vo, J. Poco, B. Geveci, A. Chaudhary, A. Bauer, A. Pletzer, D. Kindig, G. L. Potter, and T. P. Maxwell. 2013. “The Ultrascale Visualization Climate Data Analysis Tools: Data Analysis and Visualization for Geoscience Data”, IEEE Computer Special Issue: Cutting-Edge Research in Visualization, in press. DOI: 10.1109/MC.2013.119.