An Introduction to Statistical Signal Processing

Since this random variable is defined for all positive t, {Yt} is a random process, depending on two separate processes: a counting process {Nt} and an iid process {Xn}. In this section we explore the properties of such processes. The main tool used to investigate such processes is conditional expectation.

Suppose that {Nt; t ≥ 0} is a discrete or continuous time counting process. Thus t is assumed to take on either nonnegative real values or nonnegative integer values. Suppose that {Xk} is an iid process and that the Xn are mutually independent of the Nt. Define the compound process {Yt; t ≥ 0} by

What can be said about the process Yt? From iterated expectation we have that the mean of the compound process is given by

EYt = E[E(Yt|Nt)]

Thus, for example, if Nt is a binomial counting process with parameter p, and {Xn} is a Bernoulli process with parameter *, then E(Yk|Nk) = Nk* and hence EYk = *E(Nk) = *kp. If Nt is a Poisson counting process with parameter λ, then E(Yt) = E(Nt)E(X) = λtE(X).

Other moments follow in a similar fashion. For example, the characteristic function of Yt can be evaluated using iterated expectation as

where we have used the fact that condtioned on Nt, Yt is the sum of Nt iid random variables with characteristic function MX. To further evaluate this, we again need to assume a distribution for Nt. Suppose first that Nt

Suppose instead that Nt is a continuous time Poisson counting process. Then

where we have invoked the Taylor series expansion for an exponential. Both of these computations involve very complicated processes, yet they

result in closed form solutions of modest complication. Since the characteristic functions are known, the marginal distributions of such processes follow. Further properties of compound processes are explored in the problems.

6.11Exponential Modulation

Lest the reader erroneously assume that all random process derived distribution techniques apply only to linear operations on processes, we next consider an example of a class of processes generated by a nonlinear operation on another process. While linear techniques rarely work for nonlinear systems, the systems that we shall consider form an important exception where one can find second-order moments and sometimes even complete specifications. The primary examples of processes generated in this way are phase-modulated (PM) and frequency (FM) Gaussian random processes and the Poisson random telegraph wave.

Let {X(t)} be a random process and define a new random process

where a0, a1, and a2 are fixed real constants and where Θ is a uniformly distributed random phase angle on [0, 2π]. The process {Y (t)} is called an exponential modulation of {X(t)}. Observe that it is a nonlinear function of the input process. Note further that, as defined, the process is a complexvalued random process, and hence we must modify some of our techniques. In some, but not all, of the interesting examples of exponentially modulated random processes we will wish to focus on the real part of the modulated process, which we will call

U(t) = Re(Y (t)) = 1/2Y (t) + 1/2Y (t)

In this form, exponential modulation is called phase modulation (PM) of a carrier of angular frequency a1 by the input process {X(t)}. If the input process is itself formed by integrating another random process, say {W (t)}, then the U process is called the frequency modulation (FM) of the carrier by the W process. Phase and frequency modulation are extremely important examples of complex exponential modulation.

A classic example of a random process arising in communications that can be put in the same form is obtained by setting Θ = 0 (with probability 1), choosing a1 = 0, a2 = π, and letting the input process be the Poisson counting process {N(t)}, that is, to consider the random process

This is a real-valued random process that changes value with every jump in the Poisson counting process. Because of the properties of the Poisson counting process, this process is such that jumps in nonoverlapping time windows are independent, the probability of a change of value in a di erentially small interval is proportional to the length of the interval, and the probability of more than one change is negligible in comparison. It is usually convenient to consider a slight change in this process, which makes it somewhat better behaved. Let Z be a binary random variable, independent of N(t) and taking values of +1 or −1 with equal probability. Then the random process Y (t) = ZV (t) is called the random telegraph wave and has long served as a fundamental example in the teaching of second-order random process theory. The purpose of the random variable Z is to remove an obvious nonstationarity at the origin and make the resulting process equally likely to have either of its two values at time zero. This has the obvious e ect of making the process zero-mean. In the form given, it can be treated as simply a special case of exponential modulation.

We develop the second-order moments of exponentially modulated random processes and then apply the results to the preceding examples. We modify our definitions slightly to apply to the complex-valued random variable is defined as the vector consisting of the expectations of the real and imaginary parts; that is, if X = Re(X) + jIm(X), with Re(X) and Im(X) the real and imaginary parts of X, respectively, then

EX = (ERe(X), EIm(X)) .

In other words, the expectation of a vector is defined to be the vector of ordinary scalar expectations of the components. The autocorrelation function of a complex random process is defined somewhat di erently as

RY (t, s) = E[Y (t)Y (s) ] ,

which reduces to the usual definition if the process is real valued. The autocorrelation in this more general situation is not in general symmetric, but it is Hermitian in the sense that

RY (s, t) = RY (t, s) .

Being Hermitian is, in fact, the appropriate generalization of symmetry for developing a useful transform theory, and it is for this reason that the autocorrelation function includes the complex conjugate of the second term.

It is an easy exercise to show that for the general exponentially modulated random process of (6.41) we have that

EY (t) = 0 .

This can be accomplished by separately considering the real and imaginary parts and using (3.126), exactly as was done in the AM case of chapter 5. The use of the auxiliary random variable Z in the random telegraph wave definition means that both examples have zero mean. Note that it is not true that Eej(a1t+a2X(t)+Θ) equals ej(a1t+a2EX(t)+EΘ); that is, expectation does not in general commute with nonlinear operations.

To find the autocorrelation of the exponentially modulated process, observe that

E[Y (t)Y (s) ] = a20E[ej(a1(t−s)+a2(X(t)−X(s)))]

Thus the autocorrelation of the nonlinearly modulated process is given simply in terms of the characteristic function of the increment between the two sample times! This is often a computable quantity, and when it is, we can find the second-order properties of such processes without approximation or linearization. This is a simple result of the fact that the autocorrelation of an exponentially modulated process is given by an expectation of the exponential of the di erence of two samples and hence by the characteristic function of the di erence.

There are two examples in which the computation of the characteristic function of the di erence of two samples of a random process is particularly easy: a Gaussian input process and an independent increment input processes.

If the input process {X(t)} is Gaussian with zero mean (for convenience) and autocorrelation function RX(t, s), then the random variable X(t) − X(s) is also Gaussian (being a linear combination of Gaussian random variables)with mean zero and variance

σ2 − = E[(X(t) − X(s))2] = RX(t, t) + RX(s, s) − 2RX(t, s) .

X(t) X(s)

Thus we have shown that if {X(t)} is a zero-mean Gaussian random process with autocorrelation function RX and if {Y (t)} is obtained by exponentially modulating {X(t)} as in (6.41), then

RY (t, s) = a20eja1(t−s)MX(t)−X(s)(ja2) =

a20eja1(t−s)e−1/2a22(RX (t,t)+RX (s,s)−2RX (t,s)) . (6.45)

Observe that this autocorrelation is not symmetric, but it is Hermitian. Thus, for example, if the input process is stationary, then so is the

modulated process, and

We emphatically note that the modulated process is not Gaussian.

We can use this result to obtain the second-order properties for phase modulation as follows:

14 (E[Y (t)Y (s) ] + E[Y (t)Y (s)] + E[Y (t) Y (s) ] + E[Y (t) Y (s)]) .

Note that both of the middle terms on the right have the form

a20e±ja1(t+s)E[e±j(a2(X(t0+X(s))+2Θ)] ,

which evaluates to 0 because of the uniform phase angle. The remaining terms are RY (t, s) and RY (t, s) from the previous development, and hence

As expected, this autocorrelation is symmetric.

Returning to the exponential modulation case, we consider the second example of exponential modulation of independent increment processes. Observe that this overlaps the preceding example in the case of the Wiener process. We also note that phase modulation by independent increment processes is of additional interest because in some examples independent increment processes can be modeled as the integral of another process. For example, the Poisson counting process is the integral of a random telegraph wave with alphabet 0 and 1 instead of −1 and +1. (This is accomplished by forming the process (X(t) + 1)/2 with X(t) the ±1 random telegraph wave.) In this case the real part of the output process is the FM modulation of the process being integrated.

If {X(t)} is a random process with independent and stationary increments, then the characteristic function of X(t) − X(s) with t > s is equal to that of X(t − s). Thus we have from (6.44) that for t > s and τ = t − s,

RY (τ) = a20eja1τ MX(τ)(ja2) .

We can repeat this development for the case of negative lag to obtain

Observe that this autocorrelation is not symmetric; that is, it is not true that RY (−τ) = RY (τ) (unless a1 = 0). It is, however, Hermitian.

Equation (6.49) provides an interesting oddity: Even though the original input process is not weakly stationary (since it is an independent increment process), the exponentially modulated output is weakly stationary! For example, if {X(t)} is a Poisson counting process with parameter λ, then the characteristic function is

MX(τ)(ju) = ∞ (λτ)ke−λτ ejuk =

k!

k=0

e−λτ ∞ (λτeju)k = eλτ(eju−1) ; τ ≥ 0 .

k!

k=0

Thus if we choose a1 = 0 and a2 = π, then the modulated output process is the random telegraph wave with alphabet ±a0 and hence is a real process. Equation (6.49) becomes

Note that the autocorrelation (and hence also the covariance) decays exponentially with the delay.

A complete specification of the random telegraph wave is possible and is left as an exercise.

6.12Thermal Noise

Thermal noise is one of the most important sources of noise in communications systems. It is the “front-end” noise in receivers that is caused by the random motion of electrons in a resistance. The resulting noise is then greatly amplified by the amplifiers that magnify the noise along with the possibly tiny signals. Thus the noise is really in the receiver itself and not in the atmosphere, as some might think, and can be comparable in amplitude to the desired signal. In this section we sketch the development of a model of thermal noise. The development provides an interesting example of a process with both Poisson and Gaussian characteristics.

Say we have a uniform conducting cylindrical rod at temperature T . Across this rod we connect an ammeter. The random motion of electrons in the rod will cause a current I(t) to flow through the meter. We wish to develop a random process model for the current based on the underlying physics. The following are the relevant physical parameters:

A= cross-sectional area of the rod

L= length of the rod

q= electron charge

n= number of electrons per cubic centimeter

α= average number of electron collisions with heavier particles per second (about 103)

m= mass of an electron

κ= Boltzmann’s constant

The current measured will be due to electrons moving in the longitudinal direction of the rod, which we denote x. Let Vx,k(t) denote the component

6.13. ERGODICITY AND STRONG LAWS OF LARGE NUMBERS373

It follows from the equipartition theorem for electrons in thermal equilibrium at temperature T that the electron velocity variance is

Therefore, after some algebra, we have from (6.51) through (6.53) that

RI(τ) = κTR αe−α|τ| .

Thevinin’s theorem can the be applied to model the conductor as a voltage source with voltage E(t) = RI(t). The autocorrelation function of

E(t) is

RE(τ = κTRαe−α|τ| ,

an autocorrelation function that decreases exponentially with the delay τ. Observe that as α → ∞, RE(τ) becomes a taller and narrower pulse with constant area 2κTR; that is, it looks more and more like a Dirac delta function with area 2κTR. Since the mean is zero, this implies that the process E(t) is such that samples separated by very small amounts are approximately uncorrelated. Thus thermal noise is approximately white noise. The central limit theorem can be used to show that the finite dimensional distributions of the process are approximately Gaussian. Thus we can conclude that an approximate model for thermal noise is a Gaussian white noise process!

6.13Ergodicity and Strong Laws of Large Numbers

We close this chapter on general random processes with memory with a statement of a general form of the strong law of large numbers. In order to state the theorem, another idea is needed — ergodicity. The notion of ergodicity is often described incorrectly in engineering-oriented texts on random processes. There is, however, some justification for doing so, the definition is extremely abstract and not very intuitive. The intuition comes with the consequences of assuming both ergodicity and stationarity, and it is these consequences that are often used as a definition. For completeness we provide the rigorous definition. We later consider briefly examples of processes that violate the definition. Before possibly obscuring the key issues with abstraction, it is worth pointing out a few basic facts:

•The concept of ergodicity does not require stationarity, that is, a nonstationary process can be ergodic.

•Many perfectly good models of physical processes are not ergodic, yet they have a form of law of large numbers. In other words, nonergodic processes can be perfectly good and useful models.

•The definition is in terms of the process distribution of the random process. There is no finite-dimensional equivalent definition of ergodicity as there is for stationarity. This fact makes it more di cult to describe and interpret ergodicity.

•iid processes are ergodic, i.e., ergodicity can be thought of as a generalization of iid.

Ergodicity is defined in terms of a property of events: an event F is said to be τ-invariant if {xt; t T } F implies that also {xt+τ ; t T } F , i.e., if a sequence or waveform is in F , then so is the sequence or waveform formed by shifting by τ. As an example, consider the discrete time random process event F consisting of all binary sequences having a limiting relative frequency of 1’s of exactly p. Then this event is τ-invariant for all integer τ since changing the starting time of the sequence by a finite amount does not e ect limiting relative frequencies.

A random process {Xt; t T } is ergodic if for any τ all τ-invariant events F have probability 1 or 0. In the discrete time case it su ces to consider only τ = 1.

In the authors’ view, the concept of ergodicity is the most abstract idea of this book, but its importance in practice makes it imperative that the idea at least be introduced and discussed. The reader interested in delving more deeply into the concept is referred to Billingsley’s classic book Ergodic Theory and Information[3] for a deep look at ergodicity and its implications for discrete time discrete alphabet random processes. Rather than try to provide further insight into the abstract definition, we instead turn to its implications, and then interpret from the implications what it means for a process to be ergodic or not.

The importance of stationarity and ergodicity is largely due to the following classic result of Birkho and Khinchine.

Theorem 6.1 The Strong Law of Large Numbers (The Pointwise Ergodic Theorem)

Given a discrete time stationary random process {Xn; n Z} with

ˆ

finite expectation E(Xn) = mX, then there is a random variable X with the property that