What I want to do
I want to test out some of the new tools in Spectral Workbench 2 and see if we can do some useful things like reducing noise in spectra.
My attempt and results
Some spectra have high baseline noise -- whether due to the webcam itself, or due to stray light in the spectrometer.
I used a Transform operation with the expression
Math.max(A-0.15,0)+(0.15*Math.max(A-0.15,0)) to subtract out data that falls below 15% -- and the way I did it also spreads the remaining data evenly from 0-100%. You can see the difference before and after the noise reduction here:
- takes the average and subtracts 15%:
- cuts off anything that falls below zero:
- adds back a proportion of 15% based on if the original value was >15%:
The last part is not perfect; if the original value is 100%, it only adds back a maximum of 85% of 15%, which drops the final value by 2.25% versus the original. This is shown at the highest peaks of the comparison graph, but the effect is most prominent only at the very highest peaks. I could remove this with a more complex expression, but it seems not worth it to me.
Try it out yourself by forking these two and tweaking the operation yourself: https://spectralworkbench.org/sets/show2/3163
Ah yes, the new plots are there. However, the entire shape of the spectra has been shifted so the ratios between peaks are thus different from the original. You have therefore altered the real data -- which is, technically, now invalid. If you want to simply cut-off the base of the spectra so you simply don't see the noise (and other smaller signals) then that is all you can do. This typically means that near the base of the peaks, they suddenly just "drop off" to zero -- but that, at least, is an honest indication of what just happened -- and not an illusion that there was some form of noise reduction going one. Rule one is never "lie" (grin) to the user.
In theory, if a user has a stable system (everyone has learned that by now, right?) then uploading 3 successive spectra could work -- albeit maybe with less control over the process so the user might have to be warned that their results are dependent on having stable data ..... but warnings are easy and nearly free.
I consistently use averaging 3 parallel rows in matlab for my plots as it does produce more stable spectra.
As a side note, I have also noted that if I "search" around the spectral band and observe the resulting spectra, I can get potentially significant variation in the spectra despite the added stability of my V3 proto. I suspect this is related to the optical limitations of the design but I cannot yet either prove it or find a correlation -- but I believe it is real. So, in the mean time, I think at least averaging 3 parallel lines of pixels is actually necessary.
Is this a question? Click here to post it to the Questions page.