[theThinker (OP)]

There are 100 values between 1 and 10 accurate to 4 decimals. Two methods A and B depending on different principles are used to measure the 100 values. The result is:
A :  mean of error = 0.0013, standard deviation 0.0170
B :  mean of error = 0.0039, standard deviation 0.0112

From the statistical standpoint, can it be said that method A is better as the mean error is 1/3 that of B despite a slightly higher standard deviation.

Or is there any other statistical analysis of the data that can decide which is the better method.   

[chiralSPO]

There are many ways to analyze this, and which method is best depends on what you care about.

I think the standard is to minimize the sum (or mean) of the square of the errors.

It is unclear from the question though, how the errors are determined (it is odd to talk about the standard deviation of a single point--is it 100 measurements of the same value? or are the values known or measured in another way and then compared?)

If wikipedia isn't sufficient for you, I would recommend this textbook (find it in a library before buying it): https://www.amazon.com/Introduction-Error-Analysis-Uncertainties-Measurements/dp/093570275X

[theThinker (OP)]

What I mean is there are 100 values (no uncertainties) and they are measured with an instrument.
Say:

      True value X |  Y1, Measured by A | Y2, by B
   1      3.4575      3.4601         3.4555
   2      7.5691      7.5623         7.5703
   .... 100 measurements. 

Error of A's measurements would be (Y1 - X). We taken the mean and standard deviation of the 100 errors or difference.

[chiralSPO]

Ah, I see now.

At this point, I think it is worth pointing out the difference between accuracy and precision. Accuracy refers to how close the measurements are to the actual value, and precision refers to how close repeated measurements are to each other.

This is very well summed up graphically here:

precision vs accuracy.png (120.24 kB . 637x239 - viewed 3079 times)

It sounds like measurement A is more accurate, and B is more precise. So again, it matters what you care about.

For instance, if you are comparing pairs of values, and need to know which is bigger, the more precise measurement will be preferred--the errors of the two measurements are likely to cancel out in most cases, and you will be able to distinguish very small differences (systematic errors will cancel, but random errors might lead to wrong conclusions). But, if you are stacking up bricks and want to know their heights so you know how many to use to make a 10 m wall, then accuracy is more important than precision (random errors will cancel out, but systematic errors will accumulate).

Does this help?

[theThinker (OP)]

Thanks chiralSPO,

I think I understand now. The precison part seems about close.

The accuracy part seems to differ much by 1:3. But could we read much into this 1:3 accuracy difference statistically? Or we have to use a little non-science 'intuition'?

[evan_au]

can it be said that method A is better as the mean error is 1/3 that of B despite a slightly higher standard deviation

It depends on what you are trying to do with the measurement.
- If you want to get the best average result from a single measurement, go for A
- If you can afford the time, you can take 9 measurements at different times, and your standard deviation will now be about 3 times smaller (assuming the measurement errors are normal and independent)
- If you are trying to identify units that are in the top 0.1%, go for the measurement method with the smaller standard deviation

PS: It seems that in this hypothetical case you have access to an "oracle" that can give you the exact answers. If in doubt, just ask the oracle!
See: https://en.wikipedia.org/wiki/Test_oracle

[theThinker (OP)]

can it be said that method A is better as the mean error is 1/3 that of B despite a slightly higher standard deviation

It depends on what you are trying to do with the measurement.
- If you want to get the best average result from a single measurement, go for A
- If you can afford the time, you can take 9 measurements at different times, and your standard deviation will now be about 3 times smaller (assuming the measurement errors are normal and independent)
- If you are trying to identify units that are in the top 0.1%, go for the measurement method with the smaller standard deviation

PS: It seems that in this hypothetical case you have access to an "oracle" that can give you the exact answers. If in doubt, just ask the oracle!
See: https://en.wikipedia.org/wiki/Test_oracle

Thanks for all the replies. I have a better understanding now.

I'll just refer to the oracle!