There are a couple of reasons that the photos from a modified NIR camera will not produce useful NDVI results unless some adjustments are made:
- Neither of the channels used to compute NDVI is exactly like the channels used to compute legacy (e.g., from satellite data) NDVI. The red filter you added to your camera allows both NIR and red light to be captured in the red channel. It is not very easy to figure out how much of that captured light is red and how much is NIR. All you get is a single value for brightness (for each pixel) in the red channel and you don't know how much of that brightness is produced by red light.
- The sensors in consumer cameras are sensitive to NIR light, but not as sensitive as they are to visible light. So if the same amount of NIR and red light reach the sensor, the red light will be recorded as being much brighter. Different sensors have different relative sensitivity to red and NIR light.
A clever way to work around these limitations is to use your wildly inaccurate camera to take photos of things that have precisely known reflectance of NIR and red light. Then compare the brightness of the channels you are using for red and NIR to the radiance of red and NIR that you know should be reflected from those things. Ned Horning has worked out a procedure for placing targets of known reflectance in each photo taken of vegetation (so you can calibrate every photo and eliminate variation due to source light, sun angle, cloud cover, etc). A summary of the general approach is here.
A cruder workaround is to just increase the brightness of the blue (NIR) channel to compensate for both 1 and 2 above. This can be done with many cameras by setting a custom white balance which exaggerates the blue channel. If your camera does not allow that, you can capture RAW image data and exaggerate the blue channel after the photos have been taken. I have not heard of anyone doing that. If you can only capture jpg photos, it might be difficult to make the adjustments in post successfully.
The amount of exaggeration of the blue channel has to be determined by trial and error. The correct amount should produce photos with pixels of healthy foliage with values in the blue (NIR) channel which are about three times higher than values in the red (visible) channel. The range of values is displayed below. The red values should not be very close to zero (underexposed), and the blue values should not be very close to 255 (overexposed).