Устойчивость численных методов к ошибкам округления

Reasons Why the Study of Numerical Linear Algebra Is Necessary

Floating point roundoff and truncation error cause many problems. We have learned how to perform Gaussian elimination in order to row reduce a matrix to upper-triangular form. Unfortunately, if the pivot element is small, this can lead to serious errors in the solution. We will solve this problem in Chapter 11 by using partial pivoting. Sometimes an algorithm is simply far too slow, and Cramer’s Rule is an excellent example. It is useful for theoretical purposes but, as a method of solving a linear system, should not be used for systems greater than 2 × 2. Solving Ax = b by finding A^− 1 and then computing x = A^− 1b is a poor approach. If the solution to a single system is required, one step of Gaussian elimination, properly performed, requires far fewer flops and results in less roundoff error. Even if the solution is required for many right-hand sides, we will show in Chapter 11 that first factoring A into a product of a lower- and an upper-triangular matrix and then performing forward and back substitution is much more effective. A classical mistake is to compute eigenvalues by finding the roots of the characteristic polynomial. Polynomial root finding can be very sensitive to roundoff error and give extraordinarily poor results. There are excellent algorithms for computing eigenvalues that we will study in Chapters 18 and 19. Singular values should not be found by computing the eigenvalues of A^TA. There are excellent algorithms for that purpose that are not subject to as much roundoff error. Lastly, if m ≠ n a theoretical linear algebra course deals with the system using a reduction to what is called reduced row echelon form. This will tell you whether the system has infinitely many solutions or no solution. These types of systems occur in least-squares problems, and we want a single meaningful solution. We will find one by requiring that x be such that ‖b − Ax‖₂ is minimum.

5.11 Matlab Problems for Chapter 5

This section provides homework problems that require the use of Matlab for their solution.

15.5 Computing the SVD Using MATLAB

III Stability Analysis of Kalman Filters

Doing a simple calculation, it is found that the values of ||B^#|| and ||C^#|| are the same as the values of ||B¯|| and ||C¯||, respectively.

where λ_i(Ā), for i = 1,2, …, n, denotes the eigenvalues of A. That is, r is the absolute value of the eigenvalue of Ā (or the pole of the digital filter) nearest the unit circle. An estimate of m can be made from

||A¯k||/rk≤m for all k. How to get m is sometimes very difficult. Fortunately, m can be obtained with the aid of a computer.

We do not claim that the bound given here is the ultimate tool for solving the problem of the minimal wordlength of implementing digital Kalman filters; however, it can be a reference for a conservative design. This paper does permit an approximate analysis of the performance/cost tradeoff for wordlength design choices.

2.7 Sources of Corruption

In the limit as L → ∞, the signal approaches the PAM format (with no quantization error) and signal-to-quantization noise ratio is infinite. In other words, with an infinite number of quantization levels, there is zero quantization error.

The output SNR of the quantizer increases exponentially with increasing number of bits per sample, R. An increase in R requires a proportionate increase in the channel bandwidth.

1.6.1 Increasing the Stability of the Time-Stepping Solution

TD solutions have sometimes been found to diverge after a sufficient number of time steps have been computed, evidently because of the accumulation of numerical “noise” in the solution. Thisnoise can have its source in numerical roundoff errors or from analytical and numerical approximations made in developing the computer model. In either case, an interpretation of the divergent result is that such approximations introduce low-amplitude, right-half-plane, nonphysical poles into the model. An example of the latter problem is using segment lengths shorter than the wire diameter, thereby violating the thin-wire approximation [4]. Such poles can come to dominate the overall solution after enough time has passed or when the correct response has decayed enough. In either case, a sharply diverging numerical solution can result.

Various techniques for solving the late-time divergence problem have been reported [55, 98–101]. Tijhuis [101] investigated using an improved time-interpolation scheme to increase the accuracy of time derivatives. Rao et al. [55] used a conjugate gradient technique to control error accumulation over time. Smith [100] describes a procedure that exploits the fact that late-time instabilities, generally being of a high-frequency nature relative to the correct response, can be filtered from the solution by an averaging technique. Note that late-time instabilities occur for objects such as thin PEC plates that do not exhibit internal resonances, and so they represent a distinctly different kind of numerical anomaly. However, these oscillations can be effectively suppressed by following the averaging scheme which is simple, accurate, and involves a negligible amount of extra computation.

1 INTRODUCTION

The implementation of digital filters and the effect of finite wordlength on them has been of interest to many and has received much attention. Finite wordlength introduces errors in several forms: A-D quantization noise, coefficient quantization errors and roundoff errors due to finite precision arithmetic. The types of arithmetic commonly employed are fixed point arithmetic and floating point arithmetic. The problem of using fixed point arithmetic and their effect on digital filters has been extensively studied and is reasonably well understood [1]-[8]. More recently, with the increasing availability of floating point capability in signal processing chips [8], insight into algorithms employing floating point arithmetic is of increasing interest. The analysis of roundoff noise in floating point digital filters has also been studied by many researchers [7]-[23]. This chapter examines the problem of roundoff noise in floating point digital filter implementations, and discusses some of the recent work in this area. A difficulty with roundoff noise analysis of floating point implementations is that the expressions turn out to be cumbersome, and therefore harder to interpret and obtain insights from. Part of the reason for this analytical difficulty lies in the type of errors that result from using finite precision floating point arithmetic. It turns out that an appropriate model for the roundoff errors is an error that is dependent on the input signal thereby complicating the analysis [9, 7]. This is in contrast to the case of fixed point arithmetic, where the roundoff errors can be modeled as independent white noise sequences independent of the input signal. Also, unlike the fixed point case, the order of the computations has an effect on the roundoff error. This element of flexibility in the computational order introduces an additional consideration in the roundoff noise minimization process. This chapter explores tools for enhancing the tractability of the analysis. This involves examining ways of systematically manipulating filters to better track the roundoff errors. Hopefully, better tools to track the errors will result into insights for reducing these errors and implementing robust filters.

The organization of this chapter is as follows. In sec. 2, the nature of the errors that arise from floating point computations are discussed. Using the model and assuming a particular computational order, an expression for the roundoff noise at the output of a digital filter is derived in sec. 3. A procedure to account for the ordering is developed in sec. 4 with the help of a model for floating point inner products. This procedure is then applied for roundoff analysis in sec. 5. A simple procedure for applying the inner product formulation to signal flow graphs is then discussed in sec. 6. The inner product model developed is also used to develop a new approach for evaluating floating point implementation based on a coefficient sensitivity approach in sec. 7. Some examples are discussed in sec. 8 to illustrate the methods developed.

4.7 Truncating Multiplier

The error in the result consists of E_red due to simplified reduction, and E_rnd due to rounding of the n + k computed bits to the n most-significant bits. To achieve a specific total error smaller than one ulp (unit in the last significant place), the errors E_red and E_rnd can be reduced by choosing k, the number of positions not implemented, and by adding a suitable constant to the reduction array. This approach and several others are discussed in detail in the literature mentioned at the end of the chapter.

Multiplicative Normalization Method

In this method we need to distinguish between the error in d[j] and the error in R[j].

7.7 Numerical Stability and Error Propagation

Topics of paramount importance in the numerical integration of differential equations are the error propagation, stability, and convergence of these solutions. Two types of stability considerations enter in the solution of ordinary differential equations: inherent stability (or instability) and numerical stability (or instability). Inherent stability is determined by the mathematical formulation of the problem and is dependent on the eigenvalues of the Jacobian matrix of the differential equations, as was shown in Section 7.6. On the other hand, numerical stability is a function of the error propagation in the numerical integration method. The behavior of error propagation depends on the values of the characteristic roots of the difference equations that yield the numerical solution. In this section, we concern ourselves with numerical stability considerations as they apply to the numerical integration of ordinary differential equations.

There are three types of errors present in the application of numerical integration methods. These are the truncation error, the roundoff error, and the propagation error. The truncation error is a function of the number of terms that are retained in the approximation of the solution from the infinite series expansion. The truncation error may be reduced by retaining a larger number of terms in the series or by reducing the step size of integration h. The plethora of available numerical methods of integration of ordinary differential equations provides a choice of increasingly higher accuracy (lower truncation error), at an escalating cost in the number of arithmetic operations to be performed, and with the concomitant accumulation of roundoff errors.

Computers carry numbers using a finite number of significant figures, as was discussed in Chapter 3. A roundoff error is introduced in the calculation when the computer rounds up or down (or just chops) the number to n significant figures. Roundoff errors may be reduced significantly by the use of double precision. However, even a very small roundoff error may affect the accuracy of the solution, especially in numerical integration methods that march forward (or backward) for hundreds or thousands of steps, each step being performed using rounded numbers.

The truncation and roundoff errors in numerical integration accumulate and propagate, creating the propagation error, which, in some cases, may grow in exponential or oscillatory pattern, thus causing the calculated solution to deviate drastically from the correct solution.

Error propagation in numerical integration methods is a complex operation that depends on several factors. Roundoff error, which contributes to propagation error, is entirely determined by the accuracy of the computer being used. The truncation error is fixed by the choice of method being applied, by the step size of integration, and by the values of the derivatives of the functions being integrated. For these reasons, it is necessary to examine the error propagation and stability of each method individually and in connection with the differential equations to be integrated. Some techniques work well with one class of differential equations but fail with others.

More detailed discussions of stability of numerical integration methods, and other advanced topics, such as step size control and integration of stiff differential equations, are included in Appendix E of this book.

Reasons Why the Study of Numerical Linear Algebra Is Necessary

Floating point roundoff and truncation error cause many problems. We have learned how to perform Gaussian elimination in order to row reduce a matrix to upper-triangular form. Unfortunately, if the pivot element is small, this can lead to serious errors in the solution. We will solve this problem in Chapter 11 by using partial pivoting. Sometimes an algorithm is simply far too slow, and Cramer’s Rule is an excellent example. It is useful for theoretical purposes but, as a method of solving a linear system, should not be used for systems greater than 2 × 2. Solving Ax = b by finding A^− 1 and then computing x = A^− 1b is a poor approach. If the solution to a single system is required, one step of Gaussian elimination, properly performed, requires far fewer flops and results in less roundoff error. Even if the solution is required for many right-hand sides, we will show in Chapter 11 that first factoring A into a product of a lower- and an upper-triangular matrix and then performing forward and back substitution is much more effective. A classical mistake is to compute eigenvalues by finding the roots of the characteristic polynomial. Polynomial root finding can be very sensitive to roundoff error and give extraordinarily poor results. There are excellent algorithms for computing eigenvalues that we will study in Chapters 18 and 19. Singular values should not be found by computing the eigenvalues of A^TA. There are excellent algorithms for that purpose that are not subject to as much roundoff error. Lastly, if m ≠ n a theoretical linear algebra course deals with the system using a reduction to what is called reduced row echelon form. This will tell you whether the system has infinitely many solutions or no solution. These types of systems occur in least-squares problems, and we want a single meaningful solution. We will find one by requiring that x be such that ‖b − Ax‖₂ is minimum.

5.11 Matlab Problems for Chapter 5

This section provides homework problems that require the use of Matlab for their solution.

15.5 Computing the SVD Using MATLAB

III Stability Analysis of Kalman Filters

Doing a simple calculation, it is found that the values of ||B^#|| and ||C^#|| are the same as the values of ||B¯|| and ||C¯||, respectively.

where λ_i(Ā), for i = 1,2, …, n, denotes the eigenvalues of A. That is, r is the absolute value of the eigenvalue of Ā (or the pole of the digital filter) nearest the unit circle. An estimate of m can be made from

||A¯k||/rk≤m for all k. How to get m is sometimes very difficult. Fortunately, m can be obtained with the aid of a computer.

We do not claim that the bound given here is the ultimate tool for solving the problem of the minimal wordlength of implementing digital Kalman filters; however, it can be a reference for a conservative design. This paper does permit an approximate analysis of the performance/cost tradeoff for wordlength design choices.

2.7 Sources of Corruption

In the limit as L → ∞, the signal approaches the PAM format (with no quantization error) and signal-to-quantization noise ratio is infinite. In other words, with an infinite number of quantization levels, there is zero quantization error.

The output SNR of the quantizer increases exponentially with increasing number of bits per sample, R. An increase in R requires a proportionate increase in the channel bandwidth.

1.6.1 Increasing the Stability of the Time-Stepping Solution

TD solutions have sometimes been found to diverge after a sufficient number of time steps have been computed, evidently because of the accumulation of numerical “noise” in the solution. Thisnoise can have its source in numerical roundoff errors or from analytical and numerical approximations made in developing the computer model. In either case, an interpretation of the divergent result is that such approximations introduce low-amplitude, right-half-plane, nonphysical poles into the model. An example of the latter problem is using segment lengths shorter than the wire diameter, thereby violating the thin-wire approximation [4]. Such poles can come to dominate the overall solution after enough time has passed or when the correct response has decayed enough. In either case, a sharply diverging numerical solution can result.

Various techniques for solving the late-time divergence problem have been reported [55, 98–101]. Tijhuis [101] investigated using an improved time-interpolation scheme to increase the accuracy of time derivatives. Rao et al. [55] used a conjugate gradient technique to control error accumulation over time. Smith [100] describes a procedure that exploits the fact that late-time instabilities, generally being of a high-frequency nature relative to the correct response, can be filtered from the solution by an averaging technique. Note that late-time instabilities occur for objects such as thin PEC plates that do not exhibit internal resonances, and so they represent a distinctly different kind of numerical anomaly. However, these oscillations can be effectively suppressed by following the averaging scheme which is simple, accurate, and involves a negligible amount of extra computation.

1 INTRODUCTION

The implementation of digital filters and the effect of finite wordlength on them has been of interest to many and has received much attention. Finite wordlength introduces errors in several forms: A-D quantization noise, coefficient quantization errors and roundoff errors due to finite precision arithmetic. The types of arithmetic commonly employed are fixed point arithmetic and floating point arithmetic. The problem of using fixed point arithmetic and their effect on digital filters has been extensively studied and is reasonably well understood [1]-[8]. More recently, with the increasing availability of floating point capability in signal processing chips [8], insight into algorithms employing floating point arithmetic is of increasing interest. The analysis of roundoff noise in floating point digital filters has also been studied by many researchers [7]-[23]. This chapter examines the problem of roundoff noise in floating point digital filter implementations, and discusses some of the recent work in this area. A difficulty with roundoff noise analysis of floating point implementations is that the expressions turn out to be cumbersome, and therefore harder to interpret and obtain insights from. Part of the reason for this analytical difficulty lies in the type of errors that result from using finite precision floating point arithmetic. It turns out that an appropriate model for the roundoff errors is an error that is dependent on the input signal thereby complicating the analysis [9, 7]. This is in contrast to the case of fixed point arithmetic, where the roundoff errors can be modeled as independent white noise sequences independent of the input signal. Also, unlike the fixed point case, the order of the computations has an effect on the roundoff error. This element of flexibility in the computational order introduces an additional consideration in the roundoff noise minimization process. This chapter explores tools for enhancing the tractability of the analysis. This involves examining ways of systematically manipulating filters to better track the roundoff errors. Hopefully, better tools to track the errors will result into insights for reducing these errors and implementing robust filters.

The organization of this chapter is as follows. In sec. 2, the nature of the errors that arise from floating point computations are discussed. Using the model and assuming a particular computational order, an expression for the roundoff noise at the output of a digital filter is derived in sec. 3. A procedure to account for the ordering is developed in sec. 4 with the help of a model for floating point inner products. This procedure is then applied for roundoff analysis in sec. 5. A simple procedure for applying the inner product formulation to signal flow graphs is then discussed in sec. 6. The inner product model developed is also used to develop a new approach for evaluating floating point implementation based on a coefficient sensitivity approach in sec. 7. Some examples are discussed in sec. 8 to illustrate the methods developed.

4.7 Truncating Multiplier

The error in the result consists of E_red due to simplified reduction, and E_rnd due to rounding of the n + k computed bits to the n most-significant bits. To achieve a specific total error smaller than one ulp (unit in the last significant place), the errors E_red and E_rnd can be reduced by choosing k, the number of positions not implemented, and by adding a suitable constant to the reduction array. This approach and several others are discussed in detail in the literature mentioned at the end of the chapter.

Multiplicative Normalization Method

In this method we need to distinguish between the error in d[j] and the error in R[j].

7.7 Numerical Stability and Error Propagation

Topics of paramount importance in the numerical integration of differential equations are the error propagation, stability, and convergence of these solutions. Two types of stability considerations enter in the solution of ordinary differential equations: inherent stability (or instability) and numerical stability (or instability). Inherent stability is determined by the mathematical formulation of the problem and is dependent on the eigenvalues of the Jacobian matrix of the differential equations, as was shown in Section 7.6. On the other hand, numerical stability is a function of the error propagation in the numerical integration method. The behavior of error propagation depends on the values of the characteristic roots of the difference equations that yield the numerical solution. In this section, we concern ourselves with numerical stability considerations as they apply to the numerical integration of ordinary differential equations.

There are three types of errors present in the application of numerical integration methods. These are the truncation error, the roundoff error, and the propagation error. The truncation error is a function of the number of terms that are retained in the approximation of the solution from the infinite series expansion. The truncation error may be reduced by retaining a larger number of terms in the series or by reducing the step size of integration h. The plethora of available numerical methods of integration of ordinary differential equations provides a choice of increasingly higher accuracy (lower truncation error), at an escalating cost in the number of arithmetic operations to be performed, and with the concomitant accumulation of roundoff errors.

Computers carry numbers using a finite number of significant figures, as was discussed in Chapter 3. A roundoff error is introduced in the calculation when the computer rounds up or down (or just chops) the number to n significant figures. Roundoff errors may be reduced significantly by the use of double precision. However, even a very small roundoff error may affect the accuracy of the solution, especially in numerical integration methods that march forward (or backward) for hundreds or thousands of steps, each step being performed using rounded numbers.

The truncation and roundoff errors in numerical integration accumulate and propagate, creating the propagation error, which, in some cases, may grow in exponential or oscillatory pattern, thus causing the calculated solution to deviate drastically from the correct solution.

Error propagation in numerical integration methods is a complex operation that depends on several factors. Roundoff error, which contributes to propagation error, is entirely determined by the accuracy of the computer being used. The truncation error is fixed by the choice of method being applied, by the step size of integration, and by the values of the derivatives of the functions being integrated. For these reasons, it is necessary to examine the error propagation and stability of each method individually and in connection with the differential equations to be integrated. Some techniques work well with one class of differential equations but fail with others.

More detailed discussions of stability of numerical integration methods, and other advanced topics, such as step size control and integration of stiff differential equations, are included in Appendix E of this book.