to achieve excellent compression. As a result, a typical video media stream will consist of combinations of I-frames, P-frames, and B-frames.

Since a frame is typically sent over a number of packets, a single lost packet may cause an entire frame to be discarded by the codec. The effect on the quality of the picture depends on the type of frame lost. If it was an I-frame, the loss will have a major impact on quality, and future P-frames and B-frames will result in an incomplete picture until another I-frame is sent. If the lost frame was a P-frame or B-frame, the impact will be less and for a shorter duration. Video codecs employ a number of loss concealment techniques. For example, some use repetition of previous frames, which can work for stationary or slowly moving images. Spatial and frequency interpolation can be used to try to generate lost frames. Also, sending frames using interleaving can provide protection against burst errors. In this approach, parts of different frames are sent out of sequence.

The most common standard video codecs are the H.26x series. H.261 was a very early codec used for video conferencing. H.262 is essentially the same as MPEG-2 which is used in DVDs and HDTV broadcasts. H.263 is commonly used on the Internet today through the Flash Player plugin used by video sharing sites. Many video systems are moving to H.264, which uses MPEG-4 encoding. H.264 is the recommended codec by YouTube [12]. Motion JPEG is a high quality video codec that only uses I-frames with JPEG compression within each frame. It uses much more bandwidth than H.26x codecs but provides a high quality picture even during fast action and motion sequences. In addition, there are many proprietary video codecs in use over the Internet.

12.5 Conferencing

Audio conferencing and videoconferencing are important applications that utilize media transport. Each of these applications has their own media requirements. The details of SIP conferencing are covered in Section 9.7. Audio conferencing requires an audio mixer: a device which combines multiple RTP audio streams into a single stream. A mixer in RTP synchronizes the input media streams then combines them together. The SSRC of each media stream, which was included (mixed) into the resulting stream, will be copied into the contributing SSRC (CSSRC) field of the header. This allows speaker identification during the conference. A typical mixing strategy uses N − 1 mixing—that is, the N loudest speakers will have their media combined and shared, but each speaker will not hear themselves—they get the N − 1 mix. Thus for N = 3, the mixer will produce four distinct mixes, one with all three speakers that is received by nonspeakers, and three with only two of the speakers. Each speaker will get a version of this mix. An audio mixer is sometimes called a multipoint control unit (MCU).

Video mixing can involve combining multiple video streams into a single stream known as tiling (sometimes called “Hollywood squares” if they are presented in a checkerboard arrangement), or by just selecting a video streaming. If video follows audio is used, the video will switch to the loudest speaker. In other cases, users in a videoconference can select which video stream or streams they view, sometimes from a set of thumbnail images. When video switching is occurring, the new video stream needs an I-frame or key frame to be sent immediately, otherwise, the sequence of P-frames and B-frames being sent will not provide a complete image without the I-frame they reference. This is accomplished using fast update signaling between the video mixer and the video source. One method uses an XML message [13] to convey this signaling. Another method uses a special RTCP message [14] and the audio video profile with feedback (AVPF).

12.6 ToIP—Conversational Text

Conversational text, or text over IP (ToIP), is a bidirectional real-time exchange of text characters. Unlike e-mail where the message is only sent when the user hits send, or instant messaging where the message is sent when the user presses enter or return, conversational text messages are sent character by character, usually in full duplex (i.e., both sides can type at the same time). Devices in the PSTN to accomplish this are known as telecommunications devices for the deaf (TDD). Sometimes they are used only for one direction of the call; a human relay operator receives the conversational text messages from one party and speaks the words to the other party. The PSTN uses many different standards and devices for this communication. T.140 [15] is an International Telecommunications Union (ITU) format for encoding conversational text. RTP has a payload for transporting T.140 information [16] over UDP. This payload can use redundant transmission so that individual lost RTP packets will not result in dropped characters. For conversational text to be truly conversational, the end-to-end latency must be less than 300 ms. An industry group known as the Real-Time Text Taskforce (R3TF) [17] has been formed to help the adoption of this technology to the Internet.

12.7 DTMF Transport

Dual tone multifrequency (DTMF) tones are commonly used on the PSTN for dialing telephone numbers. Although Internet communications do not utilize dialing, DTMF still must be transported and supported for user signaling—for example, when entering a personal identification number (PIN) or password to access voicemail or interactive voice response (IVR) systems. Calling card, telephone banking, and many other systems use DTMF tones for signaling. DTMF,

as the name suggests, generates two superimposed sine waves at particular frequencies to send a particular digit (0–9, *, #, or, less commonly, A–F). In the PSTN, DTMF is typically encoded the same way as voice. However, low bit rate codecs, which are optimized for encoding voice, often do not reliably encode DTMF tones. As a result, there is a need to transport DTMF not as sine waves but as actual digits. This is especially appealing for devices such as SIP phones and mobile phones which only need to generate DTMF. A payload known as telephone-events [18] has been defined for transport over RTP. This approach is commonly known in the industry as RFC 2833 tones, where RFC 2833 [19] was the original RFC specification for telephone-events.

The payload contains:

•Event: an octet used to encode the event such as the DTMF key pressed;

•End (E) bit: a bit used to indicate the end of the event;

•Reserved (R) bit: a bit reserved for future use, set to zero and ignored;

•Volume: 6 bits for the level of the tone in dBm0;

•Duration: 16 bits used for a timestamp for the event duration.

When a user presses a DTMF key, or a gateway detects a DTMF tone in band, an RTP telephone-events packet is created and sent. The marker (M) bit in the RTP header is set to indicate that this is the first packet sent. If the key is still being pressed or detected, the duration field will not be valid but should be set to a value higher than the update time. Update RTP telephone-events are sent typically every 50 ms. The RTP timestamp for these update packets will be the same as the first RTP packet but the duration will increase for each. When the key is released or the DTMF tone is no longer detected, a final RTP telephoneevent packet is created. The end (E) bit will be set and the duration field will contain the actual tone duration. This final RTP packet will be resent two more times for redundancy. If the DTMF keypress or tone duration is less than the update time, only three RTP telephone events will be sent. The first will have the M bit set, all will have the E bit sent and the duration field will indicate the duration.

12.8 Questions

Q12.1 List the purpose of packet loss concealment. List some methods for packet loss concealment in audio codecs.

Q12.2 Why does RTP usually use UDP transport?

Q12.3 Explain the purpose of the sequence and SSRC fields in an RTP packet.

Q12.4 Calculate the bandwidth required for the SPEEX codec operating at 7.5 kb/s, 25 ms packetization time, assuming transport over UDP, IPv4 (no extensions), and 100BaseT Ethernet.

Q12.5 Explain the differences between RTCP receiver reports and RTCP extended reports.

Q12.6 Describe the three different types of video frames. Explain the need for fast update in a video conferencing system. Which types of frames does motion JPEG use?

Q12.7 How many telephone events packets will be sent if the DTMF key

# is pressed and held for 185 ms? Assume the recommended update interval. How many bits in total will be sent, assuming transport over UDP, IPv4 (no extensions), and 1000BaseT Ethernet?

Q12.8 Explain the difference between instant messaging and conversational text.

Q12.9 Describe common audio and video mixing techniques.

Q12.10 Explain the need for the payload type field in the RTP header. Use an example with the iLBC codec and telephone-events to make your point.

References

[1]Schulzrinne, H., et al., “RTP: A Transport Protocol for Real-Time Applications,” STD 64, RFC 3550, July 2003.

[2]Cohen, D., “Specifications for the Network Voice Protocol (NVP),” RFC 741, November 1976.

[3]Westerlund, M., and S. Wenger, “RTP Topologies,” RFC 5117, January 2008.

[4]Friedman, T., R. Caceres, and A. Clark, “RTP Control Protocol Extended Reports (RTCP XR),” RFC 3611, November 2003.

[5]Casner, S., and V. Jacobson, “Compressing IP/UDP/RTP Headers for Low-Speed Serial Links,” RFC 2508, February 1999.

[6]Bormann, C., et al., “Robust Header Compression (ROHC): Framework and Four Profiles: RTP, UDP, ESP, and Uncompressed,” RFC 3095, July 2001.

[7]Lazzaro, J., “Framing Real-Time Transport Protocol (RTP) and RTP Control Protocol (RTCP) Packets over Connection-Oriented Transport,” RFC 4571, July 2006.

[8]Schulzrinne, H., and S. Casner, “RTP Profile for Audio and Video Conferences with Minimal Control,” STD 65, RFC 3551, July 2003.

[9]Sjoberg, J., et al., “Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMRWB) Audio Codecs,” RFC 3267, June 2002.

[10]Andersen, S., et al., “Internet Low Bit Rate Codec (iLBC),” RFC 3951, December 2004.

[11]Herlein, G., et al., “RTP Payload Format for the Speex Codec,” draft-ietf-avt-rtp-speex-05 (work in progress), February 2008.

[12]YouTube, http://www.youtube.com.

[13]Levin, O., R. Even, and P. Hagendorf, “XML Schema for Media Control,” RFC 5168, March 2008.

[14]Wenger, S., et al., “Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF),” RFC 5104, February 2008.

[15]ITU-T Recommendation T.140 (1998)—Text Conversation Protocol for Multimedia Application, with Amendment 1, (2000).

[16]Hellstrom, G., and P. Jones, “RTP Payload for Text Conversation,” RFC 4103, June 2005.

[17]http://www.realtimetext.org/.

[18]Schulzrinne, H., and T. Taylor, “RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals,” RFC 4733, December 2006.

[19]Schulzrinne, H., and S. Petrack, “RTP Payload for DTMF Digits, Telephony Tones and Telephony Signals,” RFC 2833, May 2000.