I first posted about this yesterday in another thread, and feel that it fits better here.
The EPA uses AERMOD as a favored standard modeling for air permitting under the Clean Air Act.
I'm wondering- how good a model is it? How well has it been tested? What can communities that have pollution from a site or facility do if it has passed air permitting?
Background story
The issue of replicatable results with computer modeling has been troubling me ever since Gretchen Gehrke [ @gretchengehrke ] pointed out a big problem with the EPA's model for pm (particulate matter) dispersion, called AERMOD. She posted in July or August that her problem with the model is that it violated the conservation of matter.
Here is a quote from another thread, including the original email question text & Gretchen's answer:
"_Here's Gretchen's email on this issue. To the degree that AERMOD can be questioned, I really, really need that. I have a commenting period coming up in several weeks and anything that folks can dig up on the unreliability of AERMOD, I'll certainly use to good effect.
Thanks!
Stephen
Hi Stephen,
I haven't done any work in AERMOD, so my familiarity with it is limited, but I know that a fairly major drawback of it is that it doesn't preserve conservation of mass. So, it may be really easy to have "no impact" with the model because the chemicals can disappear entirely (not even just go through environmental transformations). The National Air Toxics Assessment by EPA is done with a combination of AERMOD and CMAQ (https://www.epa.gov/national-air-toxics-assessment/2011-nata-assessment-methods), but I'm not sure if that adequately addresses the lack of conservation of mass. Best, Gretchen_" ............
Continuing, the EPA model, AERMOD, is crucial because it is standard for air quality regulations for permitting. Disregarding the conservation of matter? How can AERMOD be a scientific model or approximation?
So, how valid are the air permits, created with AERMOD, for fugitive dust from mines, their pilings of sand or tailings, and loading and processing operations?
The accuracy would matter less, if measurements of aerosols was rigorous. Verification is poor-In Wisconsin, silica mines, of the minority that are monitored, air quality is measured by pm u10/m3.
I looked to see what studies the EPA has done to test the validity of their AERMOD. In an article entitled “AERMOD: Latest Features and Evaluation Results “, accessible at https://www3.epa.gov/scram001/7thconf/aermod/aermod_mep.pdf there is this statement, p.14: “‘At a given time and place, does the magnitude of the model prediction match the observation?” It is the experience of model developers (e.g., Weil, et al.18 and Liu and Moore19) that wind direction uncertainties can and do cause disappointing scatterplot results from what are otherwise well-performing dispersion models. Therefore, the Q-Q plot instead of the scatterplot is a more pragmatic procedure for demonstrating model performance of applied models.” (emphasis added by me.)
My reading: AERMOD is pretty good when the wind isn’t blowing from any direction. Consistently windless places are rare. Plus, the investigators in this article measured only SOx; other particulate matter may behave differently, due to ionization and other physical chemistry properties.
I wish I had the scientific background to do an investigation that would include quality evidence from verifiable research.
=================Extended quote from pp. 13-14 of the document “AERMOD: Latest Features and Evaluation Results. “ (https://www3.epa.gov/scram001/7thconf/aermod/aermod_mep.pdf)
"The average model error (or “residual”) examined over a broad range of input variables is used to evaluate the model physics. The residual is examined by plotting the ratio of the model prediction to observed values for data paired in time as a function of various model input variables (e.g., distance, wind speed, and mixing height). Residual plots can be examined for partial data sets such as for stable or unstable conditions. If a significant trend is observed in the predicted-to-observed ratio as a function of the abscissa variable, then the model physics associated with or responsible for this feature can be further examined. Operational performance of models for predicting compliance with air quality regulations, especially those involving a peak or near peak value at some unspecified time and location, can be assessed with quantile-quantile (Q-Q) plots (Chambers et al.17). Q-Q plots, are created by sorting by rank the predicted and the observed concentrations from a set of predictions initially paired in time and space. The sorted list of predicted concentrations are then plotted by rank 14 against the observed concentrations also sorted by rank. These concentration pairs are no longer paired in time or location. However, the plot is useful for answering the question, “Over a period of time and over a variety of locations, does the distribution of the model predictions match those of observations?” Scatterplots, which use data paired in time (and / or space), provide a more strict test, answering the question: “At a given time and place, does the magnitude of the model prediction match the observation?” It is the experience of model developers (e.g., Weil, et al.18 and Liu and Moore19) that wind direction uncertainties can and do cause disappointing scatterplot results from what are otherwise well-performing dispersion models. Therefore, the Q-Q plot instead of the scatterplot is a more pragmatic procedure for demonstrating model performance of applied models."