Ndis 6.0 rss Advantages
NDIS 6.0 resolves single-CPU processing issues by implementing Receive-Side Scaling (RSS). RSS is a Microsoft Scalable Networking initiative technology that enables receive processing to be balanced across multiple processors in the system while maintaining in-order delivery of the data. RSS enables parallel DPCs and, if the computer and network adapter support it, multiple interrupts.
RSS provides the following benefits:
Parallel Execution. Receive packets from a single network adapter can be processed concurrently on multiple CPUs, while preserving in-order delivery.
Dynamic Load Balancing. As system load on the host varies, RSS will rebalance the network processing load between the processors.
Cache Locality. Because packets from a single connection are always mapped to a specific processor, state for a particular connection never has to move from one processor’s cache to another processor’s cache, thereby eliminating cache thrashing and also promoting improved performance.
Send Side Scaling. Transmission Control Protocol (TCP) is often limited as to how much data it can send to the remote peer. The reasons can include the TCP congestion window, the size of the advertised receive window, or TCP slow-start. When an application tries to send a buffer larger than the size of the advertised receive window, TCP sends part of the data and then waits for an acknowledgment before sending the balance of the data. When the TCP acknowledgement arrives, additional data is sent in the context of the DPC in which the acknowledgement is indicated. Thus, scaled receive processing can also result in scaled transmit processing.
Secure Hash. The default generated RSS signature is cryptographically secure, making it much more difficult for malicious remote hosts to force the system into an unbalanced state.
Receive-Side Scaling (rss) Algorithm
This section defines the RSS algorithm and contrasts it with the current NDIS 5.1 packet processing algorithm. In general, RSS enables packets from a single network adapter to be processed in parallel on multiple CPUs while preserving in-order delivery to TCP connections.
Ndis 6.0 rss vs. Current Receive Processing
The current NDIS 5.1 architecture for processing incoming packets, supported by the Microsoft Windows Server™ 2003 operating system, is typically implemented by a network adapter vendor by leveraging a receive descriptor queue between the network adapter and the miniport adapter to pass per-packet information. The packets are processed in the following sequence:
At the network adapter, as packets arrive off the wire, the packet contents are transferred into host memory using Direct Memory Access (DMA), and a receive descriptor is transferred into the receive descriptor queue (again through DMA). An interrupt will eventually be posted to the host to indicate that new data is present. Exactly when the interrupt fires depends on the vendor’s interrupt moderation scheme.
Depending on the system’s interrupt architecture, either the interrupt will be distributed to one of the host processors (based on a vendor-specific heuristic), or it will always be routed to the same processor.
At the network adapter, if additional packets arrive, then data and descriptors are transferred to host memory using DMA. An interrupt is not fired.
The interrupt service routine (ISR) runs on the host processor that the interrupt was routed to, which disables further interrupts from the network adapter. The ISR then schedules the miniport adapter’s deferred procedure call (DPC) to run on a specific processor—usually the same processor used to run the ISR, unless the DPC is explicitly set to run on another processor.
When the DPC runs, it processes the receive descriptor queue. Either the DPC creates an array of packets to hand to the NDIS interface, or it signals each packet to the NDIS interface, one at a time. In either case, no other processor can perform network adapter interrupt processing because interrupts from the network adapter are disabled.
The protocol stack processes each indicated packet. For TCP, this involves updating internal state, potentially sending new data if the TCP window allows it to do so, and potentially indicating or completing data to the application.
Once all receive descriptors have been consumed or some maximum amount of processing has been done, the DPC reenables interrupts on the network adapter and returns, allowing another interrupt to be triggered on another (potentially different) host processor.
RSS enables parallelism by changing steps 5 and 7 to allow one of the following algorithms to be implemented:
Fire a single ISR that eventually results in the queuing of not just one DPC to a specific processor, but one DPC to potentially every processor. As shown in step 4, interrupts from the card remain disabled, and are only re-enabled after every DPC has executed in Step 7.
Fire multiple ISRs to specific processors that cause multiple DPCs to be scheduled in parallel. As shown in step 4, a specific interrupt remains disabled, and is reenabled only after a single DPC (or group of DPCs for a given ISR) has executed in Step 7.
The sequence of events just described enables parallel processing of received packets; however, if in-order delivery is not preserved, performance will probably be degraded. For example, if packets for a group of connections are processed on different CPUs and one CPU is lightly loaded while the other is heavily loaded, older packets could actually be processed first. Because TCP acknowledgement generation and processing is highly optimized for in-order processing, performance will be degraded unless RSS supports in-order delivery of TCP segments.
RSS enables in-order packet delivery by ensuring that packets for a single TCP connection are always processed by one processor. This RSS feature requires that the network adapter examine each packet header and then use a hashing function to compute a signature for the packet. To ensure that the load is balanced evenly across the CPUs, the hash result is used as an index into an indirection table. Because the indirection table contains the specific CPU that is to run the associated DPC and the host protocol stack can change the contents of the indirection table at any time, the host protocol stack can dynamically balance the processing load on each CPU.
Figure 1 shows the RSS processing sequence. As shown on the right side of Figure 1, incoming network packets arrive for processing. The hash function is applied to the header to produce a 32-bit hash result. The hash type controls which incoming packet fields are used to generate the hash result. The hash mask is applied to the hash result to get the number of bits that are used to identify the CPU number in the indirection table. The Indirection Table result is then added to BaseCPUNumber to enable RSS interrupts to be restricted from some CPUs.
The RSS processing sequence generates two variables: the scheduled CPU that runs the deferred procedure call (DPC) and the 32-bit hash result. Both are passed to the protocol driver on a per-packet basis. Lines A and B in Figure 1 are possible implementation options that are discussed in “RSS Implementation,” later in this paper.
Figure 1 RSS receive-processing sequence