Learning new climate science by opening the machine learning black box
•
0 j'aime•90 vues
Department of Psychology (Invited): Cognitive Brownbag Series, Colorado State University, CO. The popularity of machine learning, specifically neural networks, is rapidly growing in nearly all areas of science. The explosion of these methods also coincides with a growing influx of computationally expensive data sets and the need for high efficiency in solving predication problems. However, there is also some hesitancy in adopting the use of neural networks due to concerns about their reliability, reproducibility, and interpretability – thus, they are often described as “black boxes.” In climate science, we often consider signal-to-noise problems to help disentangle human-caused climate change from natural variability. These applications typically involve complicated nonlinear relationships between different feedbacks at play in the ocean, land, and atmosphere. Recent work has shown that neural networks can be a promising tool for solving these types of statistical problems when combined with explainability techniques developed by the fields of computer science and image processing. Interestingly, these methods have revealed that neural networks often leverage regional patterns of climate change indicators in order to make their predictions. In this talk, I will share examples from climate science that use a few of these visualization methods to peer into the “black box” of neural networks, which help us to better understand their decision-making processes while also learning new science. The same machine learning visualization methods can be easily adapted for a wide variety of applications and other scientific fields of study.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (18)
Similaire à Learning new climate science by opening the machine learning black box
Similaire à Learning new climate science by opening the machine learning black box (20)
Plus de Zachary Labe
Plus de Zachary Labe (20)
Dernier
Dernier (20)
Learning new climate science by opening the machine learning black box
- 1. LEARNING NEW CLIMATE SCIENCE BY OPENING THE MACHINE LEARNING BLACK BOX @ZLabe Zachary M. Labe with Elizabeth A. Barnes in the Department of Atmospheric Science 10 September 2021 Cognitive Brown Bag Series CSU Department of Psychology
- 2. Machine Learning is not new! But…
- 3. Machine Learning is not new!
- 4. Artificial Intelligence Machine Learning Deep Learning Computer Science
- 5. Computer Science Artificial Intelligence Machine Learning Deep Learning Supervised Learning Unsupervised Learning Labeled data Classification Regression Unlabeled data Clustering Dimension reduction
- 6. • Do it better • e.g., parameterizations in climate models are not perfect, use ML to make them more accurate • Do it faster • e.g., code in climate models is very slow (but we know the right answer) - use ML methods to speed things up • Do something new • e.g., go looking for non-linear relationships you didn’t know were there Very relevant for research: may be slower and worse, but can still learn something WHY SHOULD WE CONSIDER MACHINE LEARNING?
- 7. GROWING DATA
- 9. Adapted from: Kotamarthi, R., Hayhoe, K., Mearns, L., Wuebbles, D., Jacobs, J., & Jurado, J. (2021). Global Climate Models. In Downscaling Techniques for High-Resolution Climate Projections: From Global Change to Local Impacts (pp. 19-39). Cambridge: Cambridge University Press. doi:10.1017/9781108601269.003 CLIMATE MODELS Horizontal Grid Vertical Levels Past/Present/Future Fully-Coupled System 20-40 Petabytes of data
- 10. ADAPTED FROM EYRING ET AL. 2016 CMIP6
- 11. GROWING TOOLS
- 14. Python tools for machine learning
- 15. Today’s weather or climate scientist is far more likely to be debugging code written in Python… than to be poring over satellite images or releasing radiosondes. “ D. Irving| Bulletin of the American Meteorological Society| 2016
- 16. Machine learning for weather IDENTIFYING SEVERE THUNDERSTORMS Molina et al. 2021 Toms et al. 2021 CLASSIFYING PHASE OF MADDEN-JULLIAN OSCILLATION SATELLITE DETECTION Lee et al. 2021 DETECTING TORNADOES McGovern et al. 2019
- 17. Machine learning for climate FINDING FORECASTS OF OPPORTUNITY Mayer and Barnes, 2021 PREDICTING CLIMATE MODES OF VARIABILITY Gordon et al. 2021, in review TIMING OF CLIMATE CHANGE Barnes et al. 2019
- 20. NSF AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography (AI2ES) https://www.ai2es.org/
- 21. E.g., Research to Operations (R2O) Tornado Warning Special Marine Warning Severe Thunderstorm Warning Flash Flood Warning
- 22. E.g., Establish robust, responsible AI for severe weather detection Tornado Warning Special Marine Warning Severe Thunderstorm Warning Flash Flood Warning
- 23. Artificial Intelligence Machine Learning Deep Learning
- 24. X1 X2 INPUTS Artificial Neural Networks [ANN]
- 25. Linear regression! Artificial Neural Networks [ANN] X1 X2 W1 W2 ∑ = X1W1+ X2W2 + b INPUTS NODE
- 26. Artificial Neural Networks [ANN] X1 X2 W1 W2 ∑ INPUTS NODE Linear regression with non-linear mapping by an “activation function” Training of the network is merely determining the weights “w” and bias/offset “b" = factivation(X1W1+ X2W2 + b)
- 27. X1 X2 ∑ inputs HIDDEN LAYERS X3 ∑ ∑ ∑ OUTPUT = predictions Artificial Neural Networks [ANN] : : :: INPUTS
- 28. Complexity and nonlinearities of the ANN allow it to learn many different pathways of predictable behavior Once trained, you have an array of weights and biases which can be used for prediction on new data INPUT [DATA] PREDICTION Artificial Neural Networks [ANN]
- 29. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping TEMPERATURE
- 30. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping TEMPERATURE We know some metadata… + What year is it? + Where did it come from?
- 31. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping We know some metadata… + What year is it? + Where did it come from? [e.g., Barnes et al. 2019, 2020] [e.g., Labe and Barnes, 2021] TIMING OF EMERGENCE (COMBINED VARIABLES) RESPONSES TO EXTERNAL CLIMATE FORCINGS PATTERNS OF CLIMATE INDICATORS [e.g., Rader et al. 2021, in prep] Surface Temperature Map Precipitation Map + TEMPERATURE
- 32. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping We know some metadata… + What year is it? + Where did it come from? [e.g., Barnes et al. 2019, 2020] [e.g., Labe and Barnes, 2021] TIMING OF EMERGENCE (COMBINED VARIABLES) RESPONSES TO EXTERNAL CLIMATE FORCINGS PATTERNS OF CLIMATE INDICATORS [e.g., Rader et al. 2021, in prep] Surface Temperature Map Precipitation Map + TEMPERATURE
- 33. THE REAL WORLD (Observations) What is the annual mean temperature of Earth?
- 34. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) Anomaly is relative to 1951-1980
- 35. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) Let’s run a climate model
- 36. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) Let’s run a climate model again
- 37. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) Let’s run a climate model again & again
- 38. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) CLIMATE MODEL ENSEMBLES
- 39. What is the annual mean temperature of Earth? THE REAL WORLD (Observations) CLIMATE MODEL ENSEMBLES Range of ensembles = internal variability (noise) Mean of ensembles = forced response (climate change)
- 40. What is the annual mean temperature of Earth? Range of ensembles = internal variability (noise) Mean of ensembles = forced response (climate change) But let’s remove climate change…
- 41. What is the annual mean temperature of Earth? Range of ensembles = internal variability (noise) Mean of ensembles = forced response (climate change) After removing the forced response… anomalies/noise!
- 42. What is the annual mean temperature of Earth? • Increasing greenhouse gases (CO2, CH4, N2O) • Changes in industrial aerosols (SO4, BC, OC) • Changes in biomass burning (aerosols) • Changes in land-use & land-cover (albedo)
- 43. What is the annual mean temperature of Earth? • Increasing greenhouse gases (CO2, CH4, N2O) • Changes in industrial aerosols (SO4, BC, OC) • Changes in biomass burning (aerosols) • Changes in land-use & land-cover (albedo) Plus everything else… (Natural/internal variability)
- 44. What is the annual mean temperature of Earth?
- 45. Greenhouse gases fixed to 1920 levels All forcings (CESM-LE) Industrial aerosols fixed to 1920 levels [Deser et al. 2020, JCLI] Fully-coupled CESM1.1 20 Ensemble Members Run from 1920-2080 Observations
- 46. So what? Greenhouse gases = warming Aerosols = ?? (though mostly cooling) What are the relative responses between greenhouse gas and aerosol forcing?
- 47. Surface Temperature Map ARTIFICIAL NEURAL NETWORK (ANN)
- 48. INPUT LAYER Surface Temperature Map ARTIFICIAL NEURAL NETWORK (ANN)
- 49. INPUT LAYER Surface Temperature Map ARTIFICIAL NEURAL NETWORK (ANN) Collection of nodes (neurons) that adjust their weights and biases across layers in order to learn signals for making predictions Learns nonlinear processes through selected parameters in the model
- 50. INPUT LAYER HIDDEN LAYERS OUTPUT LAYER Surface Temperature Map “2000-2009” DECADE CLASS “2070-2079” “1920-1929” ARTIFICIAL NEURAL NETWORK (ANN)
- 51. INPUT LAYER HIDDEN LAYERS OUTPUT LAYER Surface Temperature Map “2000-2009” DECADE CLASS “2070-2079” “1920-1929” BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI ARTIFICIAL NEURAL NETWORK (ANN)
- 52. INPUT LAYER HIDDEN LAYERS OUTPUT LAYER Layer-wise Relevance Propagation Surface Temperature Map “2000-2009” DECADE CLASS “2070-2079” “1920-1929” BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI ARTIFICIAL NEURAL NETWORK (ANN) [Barnes et al. 2020, JAMES] [Labe and Barnes 2021, JAMES]
- 53. LAYER-WISE RELEVANCE PROPAGATION (LRP) Volcano Great White Shark Timber Wolf Image Classification LRP https://heatmapping.org/ LRP heatmaps show regions of “relevance” that contribute to the neural network’s decision-making process for a sample belonging to a particular output category Neural Network WHY WHY WHY Backpropagation – LRP
- 54. LAYER-WISE RELEVANCE PROPAGATION (LRP) Volcano Great White Shark Timber Wolf Image Classification LRP https://heatmapping.org/ LRP heatmaps show regions of “relevance” that contribute to the neural network’s decision-making process for a sample belonging to a particular output category Neural Network WHY WHY WHY Backpropagation – LRP
- 55. LAYER-WISE RELEVANCE PROPAGATION (LRP) Volcano Great White Shark Timber Wolf Image Classification LRP https://heatmapping.org/ LRP heatmaps show regions of “relevance” that contribute to the neural network’s decision-making process for a sample belonging to a particular output category Neural Network Backpropagation – LRP WHY WHY WHY
- 56. LAYER-WISE RELEVANCE PROPAGATION (LRP) Image Classification LRP https://heatmapping.org/ NOT PERFECT Crock Pot Neural Network Backpropagation – LRP WHY
- 57. [Adapted from Adebayo et al., 2020] EXPLAINABLE AI IS NOT PERFECT THERE ARE MANY METHODS
- 58. [Adapted from Adebayo et al., 2020] THERE ARE MANY METHODS EXPLAINABLE AI IS NOT PERFECT
- 59. Visualizing something we already know…
- 60. Visualizing something we already know… Input maps of sea surface temperatures to identify El Niño & La Niña Use ‘LRP’ to see how the neural network is making its decision [Toms et al. 2020, JAMES]
- 61. OUTPUT LAYER Layer-wise Relevance Propagation “2000-2009” DECADE CLASS “2070-2079” “1920-1929” BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI WHY? = LRP HEAT MAPS [Labe and Barnes 2021, JAMES]
- 62. Layer-wise Relevance Propagation BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI WHY? = LRP HEAT MAPS Machine Learning Black Box [Labe and Barnes 2021, JAMES]
- 63. Layer-wise Relevance Propagation BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI WHY? = LRP HEAT MAPS Find regions of “relevance” that contribute to the neural network’s decision-making process [Labe and Barnes 2021, JAMES]
- 64. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS Greenhouse gases fixed to 1920 levels [AEROSOLS PREVAIL] Industrial aerosols fixed to 1920 levels [GREENHOUSE GASES PREVAIL] All forcings [STANDARD CESM-LE] DATA
- 65. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS Greenhouse gases fixed to 1920 levels [AEROSOLS PREVAIL] Industrial aerosols fixed to 1920 levels [GREENHOUSE GASES PREVAIL] All forcings [STANDARD CESM-LE] DATA
- 66. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS Greenhouse gases fixed to 1920 levels [AEROSOLS PREVAIL] Industrial aerosols fixed to 1920 levels [GREENHOUSE GASES PREVAIL] All forcings [STANDARD CESM-LE] DATA
- 67. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS Greenhouse gases fixed to 1920 levels [AEROSOLS PREVAIL] Industrial aerosols fixed to 1920 levels [GREENHOUSE GASES PREVAIL] All forcings [STANDARD CESM-LE] DATA
- 68. CLIMATE MODEL DATA PREDICT THE YEAR FROM MAPS OF TEMPERATURE AEROSOLS PREVAIL GREENHOUSE GASES PREVAIL STANDARD CLIMATE MODEL [Labe and Barnes 2021, JAMES]
- 69. OBSERVATIONS PREDICT THE YEAR FROM MAPS OF TEMPERATURE AEROSOLS PREVAIL GREENHOUSE GASES PREVAIL STANDARD CLIMATE MODEL [Labe and Barnes 2021, JAMES]
- 70. OBSERVATIONS SLOPES PREDICT THE YEAR FROM MAPS OF TEMPERATURE AEROSOLS PREVAIL GREENHOUSE GASES PREVAIL STANDARD CLIMATE MODEL [Labe and Barnes 2021, JAMES]
- 71. [Labe and Barnes 2021, JAMES] ARE THE RESULTS ROBUST? YES! COMBINATIONS OF TRAINING/TESTING DATA
- 72. HOW DID THE ANN MAKE ITS PREDICTIONS?
- 73. HOW DID THE ANN MAKE ITS PREDICTIONS? WHY IS THERE GREATER SKILL FOR GHG+?
- 74. RESULTS FROM LRP [Labe and Barnes 2021, JAMES]
- 75. RESULTS FROM LRP [Labe and Barnes 2021, JAMES] WHAT IS SIGNIFICANT?
- 76. 1. Shuffle ensemble member and year dimensions (bootstrap-like method) 2. Apply true labels (unshuffled years) 3. Apply same ANN architecture and LRP 4. Repeat 500x by using different combinations of training/testing data and initialization seeds 5. Compute 95th percentile of the distribution of LRP at all grid points Uncertainty in LRP [Labe and Barnes 2021, JAMES]
- 77. Uncertainty in LRP Ultimately, we are trying to mask noise in the LRP output Identify robust climate pattern indicators! [Labe and Barnes 2021, JAMES]
- 78. RESULTS FROM LRP [Labe and Barnes 2021, JAMES]
- 79. RESULTS FROM LRP [Labe and Barnes 2021, JAMES]
- 80. Higher LRP values indicate greater relevance for the ANN’s prediction AVERAGED OVER 1960-2039 Aerosol-driven Greenhouse gas-driven All forcings Low High [Labe and Barnes 2021, JAMES]
- 81. Greenhouse gas-driven Aerosol-driven All forcings AVERAGED OVER 1960-2039 [Labe and Barnes 2021, JAMES]
- 82. DISTRIBUTIONS OF LRP AVERAGED OVER 1960-2039 [Labe and Barnes 2021, JAMES]
- 83. DISTRIBUTIONS OF LRP AVERAGED OVER 1960-2039 [Labe and Barnes 2021, JAMES]
- 84. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping TEMPERATURE We know some metadata… + What year is it? + Where did it come from?
- 85. STANDARD EVALUATION OF CLIMATE MODELS Pattern correlation RMSE EOFs Trends, anomalies, mean state Climate modes of variability
- 86. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping TEMPERATURE + Where did it come from?
- 87. ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping INPUT MAPS OF TEMPERATURE, PRECIPITATION, …
- 88. INPUT MAPS OF TEMPERATURE, PRECIPITATION, … NEURAL NETWORK ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping Classification Task
- 89. INPUT MAPS OF TEMPERATURE, PRECIPITATION, … NEURAL NETWORK WHICH MODEL DID THE MAP COME FROM? ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping Classification Task
- 90. INPUT MAPS OF TEMPERATURE, PRECIPITATION, … NEURAL NETWORK WHICH MODEL DID THE MAP COME FROM? ----ANN---- 2 Hidden Layers 10 Nodes each Ridge Regularization Early Stopping Classification Task 7 Classes – Large Ensemble GCMs CMIP5 – MMLEA-SMILE Archive 16 Ensembles Per GCM 8th class = Multi-Model Mean
- 91. COMPARING CLIMATE MODELS LRP (Explainable AI) Raw data
- 92. What climate model does the network predict for each year of observations? Multi-Model Mean
- 93. APPLYING METHODOLOGY TO REGIONS PREDICTION LRP (Explainable AI)
- 95. MACHINE LEARNING IS JUST ANOTHER TOOL TO ADD TO OUR WORKFLOW. 1)
- 96. MACHINE LEARNING IS NO LONGER A BLACK BOX. 2)
- 97. WE CAN LEARN NEW SCIENCE FROM EXPLAINABLE AI. 3)
- 98. KEY POINTS Zachary Labe zmlabe@rams.colostate.edu @ZLabe 1. Machine learning is just another tool to add to our scientific workflow 2. We can use explainable AI (XAI) methods to peer into the black box of machine learning 3. We can learn new science by using XAI methods in conjunction with existing statistical tools