As I was watching a magician’s lecture from 1985 of the magician Tommy Wonder (stage name of Jacobus Maria Bemelman), I immediately noticed the distracting noise in the video. It looked like salt-and-pepper noise, but instead of one-pixel noise, I saw horisontal black and white lines. I just had to see if I could do something about that noise. It was distracting and misdirecting the magic lecture.
A short clip from the original video.
The cleaned up clip.
In this post I focused on the salt (white) noise. I successfully removed most of the noise, but some are still there. This post show how far I got in my first attempt.
The Art of Noise
The noise consists of seemingly random white and black horizontal lines. I do not know when in the transfer process the noise was introduced. Maybe it is a noise created by dust on the reading head of the VCR? Maybe the head would produce a better transfer if it was cleaned before digitization? At any rate, the noise is truly random and very easy to spot. The noise is also distracting to watch, especially in the beginning of the video…
To identify where the lines are, my first thought was to use a transform called the Hough transform (Wikipedia on Hough transform) to extract the lines from the video. It can be used to find circles, ellipses or lines by simply applying the transform to the image, for each frame. I read some more about the transform and I realized that it was a little overkill. Because the number of pixels we need to identify is relatively small, my thinking swiftly changed to using a much simpler method: using masks.
Unmasking the hero
Each frame has many bright pixels, some of them do not belong to the noise, so we cannot identify the noise using only pixel values. However, each frame change pixel value. So if I take the difference between two frames and create a mask from a threshold, I should be able to find the lines. This is a good idea until something in the clip starts to move quickly. In the clip, Tommy Wonder moves and the mask identifies the difference between the pixels of his white shirt, so thousands of horizontal lines are erroneously identified, see the figure below. This is bad because we need to fill in the lines with frame data, which will be much harder if there are large areas to fill.
Original mask
With a simple modification of the mask, we can eliminate this problem. My solution is very simple:
mask = mask(1:rows-2, 1:cols) - mask(3:rows, 1:cols).*mask(1:rows-2, 1:cols);
New mask
This basically means that I remove pixels that has vertical neighbours. This method actually works very well.
Also, if we simply take the difference, we have to be careful of how we do it. In my initial tests I noticed that I identify noise that shows up in the immediate next frame. This is because we identify sudden changes in pixel values. The mask will show pixel changes from gray to white and from white to gray, we do not know in which frame the noise shows up, only when a change in pixel intensity occur. We basically get a ‘future echo’ (Red Dwarf anyone?)
To get rid of the future echoes, I figured that I instead use the average of several frames and take the difference between them and the frame you want to reduce the noise of, the problem of “future echoes” effectively disappears. Also, the calculation of the “averaged frame” can also be accumulated. We don’t need to recalculate it for each frame.
avg = 0.33*tmp0 + 0.33*tmp2 + 0.33*tmp3;
mask = (tmp1 - avg);
tmp1 is the current frame, the mask is created from the current frame and the average of the previous frame and two future frames.
Mind the Gap
To identify the horizontal lines, to get their specific coordinates, I simply look at the mask and step row by row, storing the coordinates for the consecutive horizontal pixel longer than a fixed number of pixels. By replacing the lines with the average frame’s pixels, we fill the gap between the previous and the next frame in a nice way. An interesting detail to note is that because the noise is introducing sharp edges in the video, by removing the noise we could compress the video more without degrading the quality.
Detected lines.
The removal of white lines are quite good, but there are still black lines that need to be removed. I will do that in a future post I think. To show exactly how well the filter does, I have created a figure showing the lines that are successfully removed.
Comparison between original (left) and the noise reduced image (right).
In conclusion, my method seems to be working OK. It detects many lines and successfully reduces the noise without introducing artefacts. However, there is still more noise to be detected and removed. I believe modifying the mask and also detect the dark lines, would improve the image quality substantially. One idea I had was to use the median filter, which is ideal to use in still images with salt-and-pepper noise. Unfortunately, even if the filter can fill in the missing data, the image will be degraded, which is not preferable. However, I could use a median filter between frames, in a way similar to what I have been doing, because not only is the noise behaving like salt-and-pepper noise for each frame, the change of pixel intensity between frames behave like salt-and-pepper noise. For the future, I would like to experiment with both a temporal median filter and remove the darker lines as well. I actually tried to remove the ‘pepper’ noise when I experimented with my masks, but failed. Next time though.
Since I for no reason mentioned something called ‘Future Echoes’, I will now give you the Red Dwarf episode I was thinking about. Enjoy.
https://youtu.be/UjJtOKsosT4
