An Introduction to Statistical Signal Processing

... ...

Figure 6.1: Moving average filter

tapped delay line or transversal filter. If the filter has a well-defined transfer function H(f) (e.g., it is stable) and if the transfer function is analytically continued to the complex plane by making the substitution z = ej2πf , then the resulting complex function contains only zeroes and no poles on the unit circle in the complex plane. For this reason such a filter is sometimes called an “all-zeroes” filter. This nomenclature really only applies to the Fourier transform or the z-transform confined to the unit circle. If one considers arbitrary z, then the filter can have zeroes at z = 0.

In chapter 5 we considered only linear systems involving moving-average filters, that is, systems that could be represented as a convolution. This was because the convolution representation is well suited to second-order I/O relations. In this chapter, however, we will find that other representations are often more useful. Recall that a convolution is simply one example of a di erence equation. Another form of di erence equation describing a linear system is obtained by convolving the outputs to get the inputs instead of vice versa. For example, the output process may satisfy a di erence equation of the form

k

For convenience it is usually assumed that a0 = 1 and ak = 0 for negative

As in the moving-average case, the limits of the sum depend on the index set; e.g., the sum could be from k = −∞ to ∞ in the two-sided case with T = Z or from k = −∞ to n in the one-sided case with T = Z+.

The numbers {ak} are called regression coe cients, and the corresponding filter is called an autoregressive filter. If ak = 0 for only a finite number of k, the filter is said to be finite-order autoregressive. The order is equal to the maximum minus the minimum value of k for which ak is nonzero. For example, if Xn = Yn + Yn−1, we have a first-order regressive filter. As with the moving-average filters, for some authors the “finite” is implicit, but we will use the more general definition. A block diagram for such a

Figure 6.2: Autoregressive filter

Note that, in contrast with a finite-order moving-average filter, a finiteorder autoregressive filter contains only feedback terms and no feedforward terms — the new output can be found solely from the current input and past of future outputs. Hence it is sometimes called a feedback filter. If we consider a deterministic input and transform both sides of (6.2), then we find that the transfer function of an autoregressive filter has the form

H(f) = 1 ,

ake−j2πkf

k

where we continue to assume that a0 = 1. Note that the analytic continuation of the transfer function into the complex plane with the substitution

z = ej2πf for a finite-order autoregressive filter has poles but no zeroes on the unit circle in the complex plane. Hence a finite-order autoregressive filter is sometimes called an all-poles filter. An autoregressive filter may or may not be stable, depending on the location of the poles.

More generally, one can describe a linear system by a general di erence equation combining the two forms — moving-average and autoregressive —

Filters with this description are called ARMA (for “autoregressive movingaverage”) filters. ARMA filters are said to be finite-order if only a finite number of the ak’s and bk’s are not zero. A finite-order ARMA filter is depicted in figure 6.3.

Figure 6.3: Moving average filter

Once again, it should be noted that some authors use finite-order implicitly, a convention that we will not adopt. Applying a deterministic input and using (A.32), we find that the transfer function of an ARMA filter has the form

bie−j2πif

ake−j2πkf

k

where we continue to assume that a0 = 1.

As we shall see by example, one can often describe a linear system by any of these filters, and hence one often chooses the simplest model for the desired application. For example, an ARMA filter representation with only three nonzero ak and two nonzero bk would be simpler than either a pure autoregressive or pure moving-average representation, which would in general require an infinite number of parameters. The general development of representations of one type of filter or process from another is an area of complex analysis that is outside the scope of this book. We shall, however, see some simple examples where di erent representations are easily found.

We are now ready to introduce three classes of random processes that are collectively called linear models since they are formed by putting an iid process into a linear system.

A discrete time random process {Yn} is called an autoregressive random process if it is formed by putting an iid random process into an autoregressive filter. Similarly, the process is said to be a moving-average random process or ARMA random process if it is formed by putting an iid process into a moving-average or ARMA filter, respectively. If a finite-order filter is used, the order of the process is the same as the order of the filter.

Since iid processes are uncorrelated, the techniques of chapter 5 immediately yield the power spectral densities of these processes in the two-sided weakly stationary case and yield in general the second-order moments of moving-average processes. In fact, some books and papers which deal only with second order moment properties define an autoregressive (moving average, ARMA) process more generally as the output of an autoregressive (moving average, ARMA) filter with a weakly stationary uncorrelated input. We use the stricter definition in order to derive actual distributions in addition to second order moments. We shall see that we can easily find marginal probability distributions for moving-average processes. Perhaps surprisingly, however, the autoregressive models will prove much more useful for finding more complete specifications, that is, joint probability distributions for the output process. The basic ideas are most easily demonstrated in the simple, and familiar, example of example [5.3], summing successive outputs of an iid process.

6.2Sums of IID Random Variables

We begin by recalling a simple but important example from chapters 3 and 5: examples [3.35], [3.37], and [5.3]. These examples can be used to exemplify both autoregressive and moving average filters. Let {Xn; n = 1, 2, . . . } be an iid process with mean m and variance σ2 (with discrete or

continuous alphabet). Consider a linear filter with Kronecker delta response

This is the discrete time integrator and it is not stable. The output process is then given as the sum of iid random variables:

The two best known members of this class are the Binomial counting process and the Wiener process (or discrete-time di usion process), which were encountered in chapter 2

We have changed notation slightly from example [5.3] since here we force Y0 = 0. Observe that if we further let X0 = 0, then by definition {Yn; n Z+} is a moving-average random process by construction with the moving-average filter hk = 1 for all nonnegative k.

Since an iid input process is also uncorrelated, we can apply example [5.3] (with a slight change due to the di erent indexing) and evaluate the first and second moments of the Y process as

EYn = mn ; n = 1, 2, . . .

and

KY (k, j) = σ2 min(k, j) ; k, j = 1, 2, . . . .

For later use we state these results in a slightly di erent notation: Since EY1 = m, since KY (1, 1) = σY21 = σ2, and since the formulas also hold for n = 0 and for k = j = 0, we have that

We explicitly consider only those values of t and s that are in the appropriate index set, here the nonnegative integers. An alternative representation to the linear system representation defined by (3.138) is obtained by rewriting the sum as a linear di erence equation with initial conditions:

Observe that in this guise, {Yn} is a first-order autoregressive process (see (6.2)) since it is obtained by passing the iid X process through a first-order autoregressive filter with a0 = 1 and a1 = −1. Observe again that this filter is not stable, but it does have a transfer function, H(f) = 1/(1 − e−j2πf ) (which, with the substitution z = ej2πf , has a pole on the unit circle).

We have seen from section 3.12 how to find the marginal distributions for such sums of iid processes and we have seen from sections 3.7-3.7.2 how to find the conditional distributions and hence a complete specification. The natural question at this point is how general the methods and results referred to are. Toward this end we consider generalizations in several directions. First we consider a direct generalization to continuous time processes, the class of processes with independent and stationary increments. We next consider partial generalizations to discrete time moving average and autoregressive processes.

6.3Independent Stationary Increments

We now generalize the class of processes formed by summing iid random variables in a way that works for both continuous and discrete time. The generalization is accomplished by focusing on the changes in a process rather than on the values of the process. The general class, that of processes with independent and stationary increments, reduces in the discrete time case to the class considered in the previous sections: processes formed by summing outputs of an iid process.

The change in value of a random process in moving forward in any given time interval is called a jump or increment of the process. The specific class of processes that we now consider consists of random processes whose jumps or increments in nonoverlapping time intervals are independent random variables whose probability distributions depend only on the time di erences over which the jumps occur. In the discrete time case, the nth output of such processes can be regarded as the sum of the first n random variables produced by an iid random process. Because the jumps in nonoverlapping time intervals then consist of sums of di erent iid random variables, the jumps are obviously independent. This general class of processes is of interest for three reasons: First, the class contains two of the most important examples of random processes: the Wiener process and the Poisson counting process. Second, members of the class form building blocks for many other random process models. For example, in chapter 5 we presented an intuitive derivation of the properties of continuous time Gaussian white noise. A rigorous development would be based on the Wiener process, which we can treat rigorously with elementary tools.

Third, these processes provide a useful vehicle for practice with several important and useful tools of probability theory: characteristic functions, conditional pmf’s, conditional pdf’s, and nonelementary conditional probability. In addition, independent increment processes provide specific examples of several general classes of processes: Markov processes, counting processes, and random walks.

Independent and stationary increment processes are generally not themselves weakly stationary since, as has already been seen in the discrete time case, their probabilistic description changes with time. They possess, however, some stationarity properties. In particular the distributions of the jumps or increments taken over fixed-length time intervals are stationary even through the distributions of the process are not.

The increments or jumps or di erences of a random process are obtained by picking a collection of ordered sample times and forming the pairwise di erences of the samples of the process taken at theses times. For example, given a discrete time or continuous time random process {Yt; t T }, one can choose a collection of sample times t0, t1, . . . , tk, ti T all i, where we assume that the sample times are ordered in the sense that

t0 < t1 < t2 < . . . < tk .

Given this collection of sample times, the corresponding increments of the process {Yt} are the di erences

Yti − Yti−1 ; i = 1, 2, . . . , k .

Note that the increments very much depend on the choice of the sample times; one would expect quite di erent behavior when the samples are widely separated than when they are nearby. We can now define the general class of processes with independent increments for both the discrete and continuous time cases.

A random process {Yt}; t T is said to have independent increments or to be an independent increment random process if for all choices of k and

sample times {ti; i = 1, dots, k}, the increments Yti −Yti−1 ; i = 1, 2, . . . , k are independent random variables. An independent increment process is

said to have stationary increments if the distribution of the increment Yt+δ − Ys+δ does not depend on δ for all allowed values of t > s and δ. (Observe that this is really only a first-order stationarity requirement on the increments, not by definition a strict stationarity requirement, but the language is standard. In any case, if the increments are independent and stationary in this sense, then they are also strictly stationary.)

We shall call a random process an independent stationary increment or isi process if it has independent and stationary increments.

possible time index; that is, that t ≥ 0 for all t T , and that Y0 = 0 as in the discrete time case. We shall see that such processes are not stationary and that they must “start” somewhere or, equivalently, be one-sided random process. We simply define the starting time as 0 for convenience and fix the starting value of the random process as 0, again for convenience. If these initial conditions are changed, the following development changes only in notational details.

A discrete time random process is an isi process if and only if it can be represented as a sum of iid random variables, i.e., if it has the form considered in the proceeding sections. To see this, observe that if {Yn} has independent and stationary increments, then by choosing sample times ti = i and defining Xn = Yn − Yn−1 for n = 1, 2, . . . , then the Xn must be independent from the independent increment assumption, and they must be identically distributed from the stationary increment assumption. Thus we have that

and hence Yn has the form of (6.6). Conversely, if Yn is the sum of iid random variables, then increments will always have the form

that is, the form of sums of disjoint collections of iid random variables, and hence they will be independent. Furthermore, the increments will clearly be stationary since they are sums of iid random variables; in particular, the distribution of the increment will depend only on the number of samples added and not on the starting time. Thus all of the development for sums of iid processes could have been entitled “discrete time processes with independent and stationary increments.”

Unfortunately, there is no such nice construction of continuous time independent increment processes. The natural continuous time analog would be to integrate a memoryless process, but as with white noise, such memoryless processes are not well-defined. One can do formal derivations analogous to the discrete time case and sometimes (but not always) arrive at correct answers. We will use alternative and more rigorous tools when dealing with the continuous time processes. We do note, however, that while we cannot express a continuous time process with independent increments as the output of a linear system driven by a continuous time memoryless process, for

and that the increments in the parentheses are independent — that is, we can write Ytn as a sum of independent increments (in many ways, in fact)

— and the increments are identically distributed if the time interval widths are identical for all increments.

Since discrete time isi processes can always be expressed as the sum of iid random variables, their first and second moments always have the form of (6.7) and (6.8). In section 6.4 it was shown that (6.7) and (6.8) also holds for continuous time processes with stationary and independent increments!

We again emphasize that an independent increment process may have stationary increments, but we already know from the moment calculations of (6.7) (6.8) that the process itself cannot be weakly stationary. Since the mean and covariance grow with time, independent increment processes clearly only make sense as one-sided processes.

6.4Second-Order Moments of ISI Processes

In this section we show that several important properties of the discrete time independent increment processes hold for the continuous time case. In the next section we generalize the specification techniques and give two examples of such processes – the continuous time Wiener process and the Poisson counting process. This section is devoted to the proof that (6.7) and (6.8) hold for continuous time processes with independent and stationary increments. The proof is primarily algebraic and can easily be skipped.

We now consider a continuous time random process {Yt; t T } where T = [0, ∞), having independent stationary increments and initial condition Y0 = 0. The techniques used in this section can also be used for an alternative derivation of the discrete time results.

First observe that given any time t and any positive delay or lag τ > 0, we have that

and hence, by the linearity of expectation,

EYt+τ = E[Yt+τ − Yt] + EYt .

Since the increments are stationary, however, the increment Yt+τ − Yt has the same distribution, and hence the same expectation as the increment

Yτ − Y0 = Yτ , and hence

EYt+τ = EYτ + EYt .

This equation has the general form

An equation of this form is called a linear functional equation and has a unique solution of the form g(t) = ct, where c is a constant that is determined by some boundary condition. Thus, in particular, the solution to (6.12) is

Thus we have that the mean of a continuous time independent increment process with stationary increments is given by

where the constant m is determined by the boundary condition

m = EY1

Thus (6.7) extends to the continuous time case.

Since Y0 = 0, we can rewrite (6.11) as

that is, we can express Yt+τ as the sum of two independent increments. The variance of the sum of two independent random variables, however, is just the sum of the two variances. In addition, the variance of the increment Yt+τ − Yt is the same as the variance of Yτ − Y0 = Yτ since the increments are stationary. Thus (6.15) implies that

σY2t+τ = σY2τ + σY2t ,

which is again a linear functional equation and hence has the solution

where the appropriate boundary condition is

σ2 = σY2t .

Knowing the variance immediately yields the second moment: