Q on "average"

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertbaer_at_localnet.com> wrote:

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
sort the data and pick the middle value. A minimum seems to be 7 but
i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Try it on a square wave.

John

   Keywords: average, mean, RMS.
   Take N samples of a "noisy" level.
   Typical scheme seems to be sum(readings)/N.
   Now, let N be odd, then sort the data and pick the middle value.
   A minimum seems to be 7 but i prefer 11 as a minimum sample count.

   A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

   Is there any test or paper that supports this oddball scheme?

On Jan 17, 12:36 pm, Tim Wescott<t...@seemywebsite.com> wrote:
On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertb...@localnet.com> wrote:

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
sort the data and pick the middle value. A minimum seems to be 7 but
i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Take N samples of a signal whose noise is exactly equal to 99 1% of the
time, and -1 99% of the time.

Do your median scheme.

Do you see a problem?

The problem is that example has no median.

On Tue, 17 Jan 2012 08:27:44 -0800, John Larkin wrote:
On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertb...@localnet.com> wrote:

  Keywords: average, mean, RMS.
  Take N samples of a "noisy" level.
  Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
  sort the data and pick the middle value. A minimum seems to be 7 but
  i prefer 11 as a minimum sample count.

  A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

  Is there any test or paper that supports this oddball scheme?

Try it on a square wave.

John

Robert: it's too bad that you're plonked over here for a few too many
posts directly quoted from right-wing rags that call themselves
"investment newsletters".

Take N samples of a signal whose noise is exactly equal to 99 1% of the
time, and -1 99% of the time.

Do your median scheme.

Do you see a problem?

The problem is that example has no median.

Hi Robert

On 1/17/2012 2:57 AM, Robert Baer wrote:
Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

"median"
Yes, that is another keyword, and seems to apply in this case.

Hi Fred,

On 1/17/2012 1:10 PM, Fred Bloggs wrote:
On Jan 17, 12:36 pm, Tim Wescott<t...@seemywebsite.com> wrote:
On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertb...@localnet.com> wrote:

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
sort the data and pick the middle value. A minimum seems to be 7
but
i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Take N samples of a signal whose noise is exactly equal to 99 1% of the
time, and -1 99% of the time.

Do your median scheme.

Do you see a problem?

The problem is that example has no median.

No, for N >= 1 there is always an average, median, mode, etc.
(though I admit the mode could be ambiguous in some cases!)

The problem is that those may not be good measures of central tendency
for a given population/sample set.

You have to think about what you want the measure to *tell* you.
And, how it can be biased in a population.

E.g., the median home value and average home value in any
particular neighborhood *tend* to be pretty close. (homes
tend to have similar values within a neighborhood... odd to
find 7,000 sq ft palaces alongside 600 sq ft "shacks")

The average salary of people riding a *bus* may closely
correlate with the median of that same population -- on
*some* busses! On others, it may be wildly different!

Robert Baer wrote:
Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

It is called the median and predates the mean as an estimator.

Modern analytical solutions solve it as minimum L1-Norm as opposed to
the "normal" approach of least squares which is minimum L2-Norm.

Try Numerical Recipes p699 http://apps.nrbook.com/c/index.html

These methods are usually known as robust estimators.

Regards,
Martin Brown
Rather close to obscene; not fit for reading by virtually anyone.

On Tue, 17 Jan 2012 08:27:44 -0800, John Larkin wrote:

On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertbaer_at_localnet.com> wrote:

Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
sort the data and pick the middle value. A minimum seems to be 7 but
i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?
Try it on a square wave.

John

Robert: it's too bad that you're plonked over here for a few too many
posts directly quoted from right-wing rags that call themselves
"investment newsletters".

Take N samples of a signal whose noise is exactly equal to 99 1% of the
time, and -1 99% of the time.

Do your median scheme.

Do you see a problem?

The nature of the noise dictates the nature of the estimator. As pointed
out, different estimators are good in different circumstances. Averaging
almost always works well for me; it's good in the case of additive noise
with statistics that make it average out to Gaussian. Averaging-with-
outliers-tossed can work well, and, as you've found, median filtering can
work well.

It all depends on the noise.

That means i should do a noise study to see how "bad" it is and the

On Jan 17, 4:57 am, Robert Baer <robertb...@localnet.com> wrote:
Keywords: average, mean, RMS.
Take N samples of a "noisy" level.
Typical scheme seems to be sum(readings)/N.
Now, let N be odd, then sort the data and pick the middle value.
A minimum seems to be 7 but i prefer 11 as a minimum sample count.

A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

Is there any test or paper that supports this oddball scheme?

Yes, it is proven by showing that the expected value of the sample
median is equal to the median of the density function underlying the
random variable. In the case where the median is equal to the mean,
you have your result. A small write-up here:
http://www.sjsu.edu/faculty/watkins/samplemedian1.htm , a heuristic
write-up, not the best in the world, but gives you an idea.
Thanks.

Don Y wrote:
Hi Fred,

On 1/17/2012 1:10 PM, Fred Bloggs wrote:
On Jan 17, 12:36 pm, Tim Wescott<t...@seemywebsite.com>  wrote:
On Tue, 17 Jan 2012 01:57:54 -0800, Robert Baer
robertb...@localnet.com>  wrote:

   Keywords: average, mean, RMS.
   Take N samples of a "noisy" level.
   Typical scheme seems to be sum(readings)/N. Now, let N be odd, then
   sort the data and pick the middle value. A minimum seems to be 7
but
   i prefer 11 as a minimum sample count.

   A few outliers have little effect on the result - no matter how
outrageous they are; cannot say the same for any of the other methods.

   Is there any test or paper that supports this oddball scheme?

Take N samples of a signal whose noise is exactly equal to 99 1% of the
time, and -1 99% of the time.

Do your median scheme.

Do you see a problem?

  The problem is that example has no median.

No, for N >= 1 there is always an average, median, mode, etc.
(though I admit the mode could be ambiguous in some cases!)

The problem is that those may not be good measures of central tendency
for a given population/sample set.

You have to think about what you want the measure to *tell* you.
And, how it can be biased in a population.

One of the relevant ones to electronics is MTBF for a filament lamp.
(and other units subject to infant mortality and limited service life)

Hardly any filament lamps actually fail at their nominal MTBF. A cohort
of a few percent die very quickly soon after being fitted and the rest
tend to live to a ripe old age. If you swap lamps out too soon to avoid
having any in service failures you actually increase the risk of
downtime due to the replacement dying suddenly during its burn in phase.

The average in this case significantly underestimates the working life
of lamps that survive the first few days - the median is usually a far
better estimator where distributions are skewed or subject to outliers.

If you have the set of measured values it is as well to plot the
histogram of their distribution to avoid surprises. Crunching it down to
a single number can easily lead you astray.

E.g., the median home value and average home value in any
particular neighborhood *tend* to be pretty close.  (homes
tend to have similar values within a neighborhood... odd to
find 7,000 sq ft palaces alongside 600 sq ft "shacks")

The average salary of people riding a *bus* may closely
correlate with the median of that same population -- on
*some* busses!  On others, it may be wildly different!

Median and L1-Norm model fitting is generally preferable when the data
are extremely noisy and subject to serious interference and outliers. It
wasn't popular in the past because apart from a handful of special cases
it is much harded to compute. But these days with almost unlimited
computing power on your desktop it is not a real problem.

The L2-Norm (aka least squares fit) tends to be dominated by fitting the
largest values whether or not they are accurate. Pulse counting devices
have systematic errors from dead time at high count rates and so can
warp the calibration of otherwise highly linear instruments.

Regards,
Martin Brown- Hide quoted text -

- Show quoted text -