Question: calibration calculations don't match MiniVol manual

I'm trying to calibrate a MiniVol air sampler pump with a minibuck flow calibrator.

Before I can do that, however, I have to check that I'm doing the calibration math correctly, and I'm having some trouble. While the spreadsheet I've prepared appears to produce numbers in the right range, the results differ from factory-provided calibration data sets. Can you help check my work?

The MiniVol Calibration instructions

Appendix A: Calculating flow rate, on page 59 of the MiniVol Manual describes the calibration procedure.

The MiniVol has a rotameter for setting and checking pump flow in the field. Its readings are only nominal, and not the 5lpm (liters per minute) that are actually needed.

The rotameter needs to be set for a number that corresponds to an actual flow rate of 5lpm (liters per minute), not an indication of 5lpm To determine the actual flow rate, six measurements are taken with the flow calibration standard (in this case a MiniBuck) at six different flow rates from 4-6.5lpm as indicated on the rotameter.

In the following chart, the variables defining different variables are described as:

• m_vol slope of the least square line from the linear regression
• b_vol intercept of the least square line from the linear regression
• r² coefficient of determination from linear regression
• Q_ind indicated flow rate on sampler's rotameter
• Q_act actual flow rate as measured by the calibration standard
• Q_@std calculated flow rate at standard conditions, from Equation 1
• standard pressure: 760mmHg, standard temp: 298°K)
• Equation 1: Q_@std = m_vol x Q_ind + b_vol
• Q_calc calculated flow rate of sampler determined from the linear regression results
• Diff the % difference between measured and calculated flow rates

This calibration chart came with the MiniVol:

Recreating the calibration chart

I Attempted to re-create this chart in a LibreOffice spreadsheet (Download). I quickly noticed a few issues of clarity on the variables described above:

• m_vol, b_vol, r² slope of the least square line, intercept, & coeficient of determination from the linear regression
• This is slightly unclear but I assume they mean of X,Y pairs (Q_ind , Q_act) a least squares linear regression, which is a basic feature in LibreOffice.
• Q_calc calculated flow rate of sampler determined from the linear regression results
• This statement is pretty unclear, and I therefore took it to be the equation described as: Q_act, for use in calculating the actual flow rate under sampling conditions which appears to be different from Q_act for the purposes of the chart:
• Q_act = (m_volQ_ind + b_vol) x √((P_std/P_act) x (T_act/T_std))
• Diff the % difference between measured and calculated flow rates
• This statement is also unclear, but I took it to be the difference between Q_act (measured) and Q_calc (calculated).

and I got these results:

The spreadsheet appears to work pretty well-- I'm getting a decent correspondence between Q_calc and Q_act, BUT... the results don't match up with Airmetrics form AT ALL. None of the calculated values are the same, none of the Linear regression results match.

Checking my results

I tried a few different checks.

The manual has an example calibration with different numbers:

Again, good correspondence within my sheet, and this time a closer correspondence to Airmetrics numbers.

Am I having a significant digits issue in my spreadsheet?

I tried to see if somehow I was using the wrong linear regression, so I replaced my calculated linear regression with Airmetrics results. That didn't bring me any closer to Airmetrics numbers though.

Can you tell what's wrong? Is it my assumptions?

I'm not sure where my assumptions are going wrong, but I imagine the mistake is somewhere in these assumptions:

• m_vol, b_vol, r² slope of the least square line, intercept, & coeficient of determination from the linear regression
• This is slightly unclear but I assume they mean of X,Y pairs (Q_ind , Q_act) a least squares linear regression, which is a basic feature in LibreOffice.
• Q_calc calculated flow rate of sampler determined from the linear regression results
• This statement is pretty unclear, and I therefore took it to be the equation described as: Q_act, for use in calculating the actual flow rate under sampling conditions which appears to be different from Q_act for the purposes of the chart:
• Q_act = (m_volQ_ind + b_vol) x √((P_std/P_act) x (T_act/T_std))
• Diff the % difference between measured and calculated flow rates
• This statement is also unclear, but I took it to be the difference between Q_act (measured) and Q_calc (calculated).