About walking noise

[mgutierrez]

Hi all,

I've been reading about walking noise concept. It seems is the same than correlated noise, isn't it? It don't fully understand how is originated this noise. I've read that is due to imperfections in the sensor, and that stacking subs makes the noise worse, not better. Also, dithering seems to at least partially fix it. I don't get a conceptual image of the noise and how is originated. Could anybody please explain how does walking noise appears and its nature?

Thanks I advance.

Miguel

[admin]

Walking noise (which is indeed correlated as its presence correlates with neighbouring pixels) happens when a pixel is a bit "hotter" or "colder" than its neighbouring pixels. E.g. it produces a little higher or lower brightness when given the same signal. In a single frame this shows up as a single pixel, but as the field of view slightly shifts around due to small tracking error as the earth rotates, that pixel will sit in a slightly different spot each time (usually slowly moving away as the field of view drifts due to tracking error). When the stacker stacks it all together this will cause a small trail.

Dithering introduces deliberate big shifts in the field of view, so that any hot or cold pixel will move a lot and will never sit at the same position. This breaks up the trail and, more importantly gives the stacker a much easier job detecting the hot or cold pixel as an outlier. The result is a vastly cleaner dataset without trails.

Hope this helps!

[mgutierrez]

Thanks for the explanation, Ivo. That was more or less the idea I had.
So, basically, the problem is due to a pixel(s) problem, similar to a classical hot pixel, but weaker, right? Does it happen very often in the sensor? Also, that pixel that is brighter than its neighbour, does it always have the same behaviour? with the same error offset? is it common to find many pixels behaving this way in the sensor?
Let me show you this example:

Selection_011.png (243.43 KiB) Viewed 5108 times

There is a lot of walking noise there. Each, let' say, trail has been originated due to a particular wrong pixel? There are many trails, so, are there so much defective pixels?
The reason a stacker does not reject those pixels, are due to both the faintness of the pixel and the low movement?
Too many, questions, sorry I ask them as they come into my mind

Thanks!

[admin]

Thanks for the explanation, Ivo. That was more or less the idea I had.
So, basically, the problem is due to a pixel(s) problem, similar to a classical hot pixel, but weaker, right?

Correct!

Does it happen very often in the sensor?

It does, particularly when the real signal is faint.

Also, that pixel that is brighter than its neighbour, does it always have the same behaviour? with the same error offset? is it common to find many pixels behaving this way in the sensor?

Indeed, it tends to show the same behaviour, though the behaviour may vary depending on external factors like temperature or how clean the power supply is, etc.

There is a lot of walking noise there. Each, let' say, trail has been originated due to a particular wrong pixel? There are many trails, so, are there so much defective pixels?

Correct. They are only small defects, but defects nonetheless.

[hixx]

I think the term "defect" pixels is misleading. We should be aware of the amount of this issue. Let me try a simplified wrap-up:
E.g. a typical DSLR will create data with 14-bit depth, i.e. the Least Significant Bit (LSB) would be 14. Lets assume we have a "defect" with the amount of 0.5 LSB (= bit15) - this would be drowned in regular BIAS noise in any normal image. So let's stack 64 subs - The stacker will create an output file of 32 bit depth, but the signal bit depth will be limited by the amount of subs now. 64 subs will give us an additional 6 bits (2^6=64), so our LSB will now be 20, the 0.5 LSB error would be @ bit 21.
But the stacker will also add up this initial 0.5 LSB error, mistaking it for "signal", if we did not dither and the error is present in multiple images at the same spot. (Assuming stacking method "average" for simplicity)
Let's now assume the same error repeated in 4 images, then moving 1 pixel for the next 4 images etc.
The addition of 4 images would boost the error "signal" 2 bits, above the LSB, to bit 19. Now any aggressive stretch will easily make this visible, what was invisible before. This is why you end up with a trail that is "made worse by stacking more subs". In fact more subs will only make visible more detail, including the tiniest variations in pixel sensitivity of your camera's sensor. Remember there would still be 18 clean bits above this error now!
This is why it is so important to dither when taking the subs. Stacking is creating independency between celestial detail and (unwanted) sensor performance "detail" . Most software or hardware guiders can do this. I even dithered manually successfully - just set your mount to the slowest slew speed and move (very quick press) up, right, 3 x down, right, 3x up etc. every few subs. Shifting few pixels will remove most of the walking noise. ST will take care of the rest, so you can safely stretch up to the very bottom of your data.
I hope this does explain correctly please fell free to correct if I got the math incorrect!

clear skies,
jochen

[mgutierrez]

Thanks for the explanation, Ivo. That was more or less the idea I had.
So, basically, the problem is due to a pixel(s) problem, similar to a classical hot pixel, but weaker, right?

Correct!

Does it happen very often in the sensor?

It does, particularly when the real signal is faint.

Also, that pixel that is brighter than its neighbour, does it always have the same behaviour? with the same error offset? is it common to find many pixels behaving this way in the sensor?

Indeed, it tends to show the same behaviour, though the behaviour may vary depending on external factors like temperature or how clean the power supply is, etc.

There is a lot of walking noise there. Each, let' say, trail has been originated due to a particular wrong pixel? There are many trails, so, are there so much defective pixels?

Correct. They are only small defects, but defects nonetheless.

great Ivo; many thanks!

[mgutierrez]

I think the term "defect" pixels is misleading. We should be aware of the amount of this issue. Let me try a simplified wrap-up:
E.g. a typical DSLR will create data with 14-bit depth, i.e. the Least Significant Bit (LSB) would be 14. Lets assume we have a "defect" with the amount of 0.5 LSB (= bit15) - this would be drowned in regular BIAS noise in any normal image. So let's stack 64 subs - The stacker will create an output file of 32 bit depth, but the signal bit depth will be limited by the amount of subs now. 64 subs will give us an additional 6 bits (2^6=64), so our LSB will now be 20, the 0.5 LSB error would be @ bit 21.
But the stacker will also add up this initial 0.5 LSB error, mistaking it for "signal", if we did not dither and the error is present in multiple images at the same spot. (Assuming stacking method "average" for simplicity)
Let's now assume the same error repeated in 4 images, then moving 1 pixel for the next 4 images etc.
The addition of 4 images would boost the error "signal" 2 bits, above the LSB, to bit 19. Now any aggressive stretch will easily make this visible, what was invisible before. This is why you end up with a trail that is "made worse by stacking more subs". In fact more subs will only make visible more detail, including the tiniest variations in pixel sensitivity of your camera's sensor. Remember there would still be 18 clean bits above this error now!
This is why it is so important to dither when taking the subs. Stacking is creating independency between celestial detail and (unwanted) sensor performance "detail" . Most software or hardware guiders can do this. I even dithered manually successfully - just set your mount to the slowest slew speed and move (very quick press) up, right, 3 x down, right, 3x up etc. every few subs. Shifting few pixels will remove most of the walking noise. ST will take care of the rest, so you can safely stretch up to the very bottom of your data.
I hope this does explain correctly please fell free to correct if I got the math incorrect!

clear skies,
jochen

thanks for the info jochen! I need time to read it to assimilate!

[admin]

I think the term "defect" pixels is misleading. We should be aware of the amount of this issue. Let me try a simplified wrap-up:
E.g. a typical DSLR will create data with 14-bit depth, i.e. the Least Significant Bit (LSB) would be 14. Lets assume we have a "defect" with the amount of 0.5 LSB (= bit15) - this would be drowned in regular BIAS noise in any normal image. So let's stack 64 subs - The stacker will create an output file of 32 bit depth, but the signal bit depth will be limited by the amount of subs now. 64 subs will give us an additional 6 bits (2^6=64), so our LSB will now be 20, the 0.5 LSB error would be @ bit 21.
But the stacker will also add up this initial 0.5 LSB error, mistaking it for "signal", if we did not dither and the error is present in multiple images at the same spot. (Assuming stacking method "average" for simplicity)
Let's now assume the same error repeated in 4 images, then moving 1 pixel for the next 4 images etc.
The addition of 4 images would boost the error "signal" 2 bits, above the LSB, to bit 19. Now any aggressive stretch will easily make this visible, what was invisible before. This is why you end up with a trail that is "made worse by stacking more subs". In fact more subs will only make visible more detail, including the tiniest variations in pixel sensitivity of your camera's sensor. Remember there would still be 18 clean bits above this error now!
This is why it is so important to dither when taking the subs. Stacking is creating independency between celestial detail and (unwanted) sensor performance "detail" . Most software or hardware guiders can do this. I even dithered manually successfully - just set your mount to the slowest slew speed and move (very quick press) up, right, 3 x down, right, 3x up etc. every few subs. Shifting few pixels will remove most of the walking noise. ST will take care of the rest, so you can safely stretch up to the very bottom of your data.
I hope this does explain correctly please fell free to correct if I got the math incorrect!

clear skies,
jochen

Great explanation Jochen!
I would still maintain that hot/warm and dead/cold pixels (and the continuum in between) are small "defects" of the sensor, given they are unwanted, unpredictable and vary between every individual, same-model sensor, manufactured under the same conditions. The error in terms of counted photons can definitely be bigger than the resolution of the D/A converter (and hence visible even in single frames). But I guess that's just quibbling over something small in a great explanation.

[hixx]

Sure, the amount of this error is depending on multiple factors in production, semiconductor structure, environment, cleanness of power supply etc. Welcome to the analog world! I forgot to mention the distinctive sensor pattern may also be reduced by Darkframes. My strategy is using both Darks (lots of them) plus Dithering plus long exposure time and still I see remnants of the pattern. But now ST's Wipe is able to kill even those.