You are working with the text-only light edition of "H.Lohninger: Teach/Me Data Analysis, Springer-Verlag, Berlin-New York-Tokyo, 1999. ISBN 3-540-14743-8". Click here for further information.

Table of Contents Introductory Stuff Decimal Places and Precision
Index
See also: standard deviation 

Decimal Places and Precision

Beginners in the field often ignore some simple facts concerning the number of significant digits of a result. The use of pocket calculators usually results in values with too many decimal places, regardless of whether this is meaningful. In general, one should use only as many decimal places as are compatible with the precision of your experiments. It may therefore be necessary to determine the precision of a measurement by repeating it several times and calculating the standard deviation.

Example: Suppose the outcomes of several repetitions of an experiment have been recorded as follows 

   12.3075
   12.3351
   11.9949
   12.2722
   12.3117
   12.0766
The four decimal places used to denote the results are meaningless, since the repeated measurements indicate that at most the first decimal place is valid (a closer analysis shows that the average of the result is 12.21, with a standard deviation of 0.14).

Last Update: 2006-Jän-17