sk_buffer 详细分析

Linux 网络子系统

 

作者： 小马哥 rstevens  ()

欢迎转载，未经允许，请勿用于商业目的 

 

1.   定义

Packet：       通过网卡收发的报文，包括链路层、网络层、传输层的协议头和携带的数据

Data Buffer：用于存储 packet 的内存空间

SKB：           struct sk_buffer 的简写

 

 

2.   概述

Struct sk_buffer 是 linux TCP/IP stack 中，用于管理Data Buffer的结构。Sk_buffer 在数据包的发送和接收中起着重要的作用。

为了提高网络处理的性能，应尽量避免数据包的拷贝。Linux 内核开发者们在设计 sk_buffer 结构的时候，充分考虑到这一点。目前 Linux 协议栈在接收数据的时候，需要拷贝两次：数据包进入网卡驱动后拷贝一次，从内核空间递交给用户空间的应用时再拷贝一次。

Sk_buffer结构随着内核版本的升级，也一直在改进。

学习和理解 sk_buffer 结构，不仅有助于更好的理解内核代码，而且也可以从中学到一些设计技巧。

 

3.   Sk_buffer 定义

struct sk_buff {

                struct sk_buff                     *next;

                struct sk_buff                     *prev;

                struct sock                          *sk;

                struct skb_timeval             tstamp;

                struct net_device         *dev;

                struct net_device         *input_dev;

 

                union {

                                struct tcphdr       *th;

                                struct udphdr      *uh;

                                struct icmphdr    *icmph;

                                struct igmphdr    *igmph;

                                struct iphdr          *ipiph;

                                struct ipv6hdr      *ipv6h;

                                unsigned char     *raw;

                } h;

                union {

                                struct iphdr          *iph;

                                struct ipv6hdr      *ipv6h;

                                struct arphdr       *arph;

                                unsigned char     *raw;

                } nh;

                union {

                                unsigned char     *raw;

                } mac;

 

                struct  dst_entry                 *dst;

                struct     sec_path              *sp;

                char                                       cb[40];

 

                unsigned int                         len,

                                                                data_len,

                                                                mac_len,

                                                                csum;

                __u32                                    priority;

 

                __u8                                       local_df:1,

                                                                cloned:1,

                                                                ip_summed:2,

                                                                nohdr:1,

                                                                nfctinfo:3;

                __u8                                       pkt_type:3,

                                                                fclone:2;

                __be16                                  protocol;

                void                                        (*destructor)(struct sk_buff *skb);

 

                /* These elements must be at the end, see alloc_skb() for details.  */

                unsigned int                         truesize;

                atomic_t                               users;

                unsigned char                     *head,

                                                *data,

                                                *tail,

                                                *end;

};

4.   成员变量

 

·              struct skb_timeval    tstamp;

此变量用于记录 packet　的到达时间或发送时间。由于计算时间有一定开销，因此只在必要时才使用此变量。需要记录时间时，调用net_enable_timestamp()，不需要时，调用net_disable_timestamp() 。

tstamp 主要用于包过滤，也用于实现一些特定的 socket 选项，一些 netfilter 的模块也要用到这个域。

·              struct net_device      *dev;

·              struct net_device      *input_dev;

 

这几个变量都用于跟踪与 packet　相关的 device。由于 packet 在接收的过程中，可能会经过多个 virtual driver 处理，因此需要几个变量。

接收数据包的时候， dev 和 input_dev 都指向最初的 interface，此后，如果需要被 virtual driver 处理，那么 dev 会发生变化，而 input_dev 始终不变。

 

 

(These three members help keep track of the devices assosciated with a packet. The reason we have three different device pointers is that the main 'skb->dev' member can change as we encapsulate and decapsulate via a virtual device.

So if we are receiving a packet from a device which is part of a bonding device instance, initially 'skb->dev' will be set to point the real underlying bonding slave. When the packet enters the networking (via 'netif_receive_skb()') we save 'skb->dev' away in 'skb->real_dev' and update 'skb->dev' to point to the bonding device.

Likewise, the physical device receiving a packet always records itself in 'skb->input_dev'. In this way, no matter how many layers of virtual devices end up being decapsulated, 'skb->input_dev' can always be used to find the top-level device that actually received this packet from the network. )

 

·              char                               cb[40];

此数组作为 SKB 的控制块，具体的协议可用它来做一些私有用途，例如 TCP 用这个控制块保存序列号和重传状态。

 

 

·              unsigned int               len,

·                                                   data_len,

·                                                   mac_len,

·                                                   csum;

 

‘len’ 表示此 SKB 管理的 Data Buffer 中数据的总长度；

通常，Data Buffer 只是一个简单的线性 buffer，这时候 len 就是线性 buffer 中的数据长度；

但在有 ‘paged data’ 情况下， Data Buffer 不仅包括第一个线性 buffer ，还包括多个 page buffer；这种情况下， ‘data_len’ 指的是 page buffer 中数据的长度，’len’ 指的是线性 buffer 加上 page buffer 的长度；len – data_len 就是线性 buffer 的长度。

 

‘mac_len’ 指 MAC 头的长度。目前，它只在 IPSec 解封装的时候被使用。将来可能从 SKB 结构中

去掉。

‘csum’ 保存 packet 的校验和。

(Finally, 'csum' holds the checksum of the packet. When building send packets, we copy the data in from userspace and calculate the 16-bit two's complement sum in parallel for performance. This sum is accumulated in 'skb->csum'. This helps us compute the final checksum stored in the protocol packet header checksum field. This field can end up being ignored if, for example, the device will checksum the packet for us.

On input, the 'csum' field can be used to store a checksum calculated by the device. If the device indicates 'CHECKSUM_HW' in the SKB 'ip_summed' field, this means that 'csum' is the two's complement checksum of the entire packet data area starting at 'skb->data'. This is generic enough such that both IPV4 and IPV6 checksum offloading can be supported. )

 

·              __u32                            priority;

 

“priority”用于实现 QoS，它的值可能取之于 IPv4 头中的 TOS 域。Traffic Control 模块需要根据这个域来对 packet 进行分类，以决定调度策略。

 

 

·              __u8                              local_df:1,

·                                                   cloned:1,

·                                                   ip_summed:2,

·                                                   nohdr:1,

·                                                   nfctinfo:3;

 

 

为了能迅速的引用一个 SKB 的数据，

当 clone 一个已存在的 SKB 时，会产生一个新的 SKB，但是这个 SKB 会共享已有 SKB 的数据区。

当一个 SKB 被 clone 后，原来的 SKB 和新的 SKB 结构中，”cloned” 都要被设置为1。

 

(The 'local_df' field is used by the IPV4 protocol, and when set allows us to locally fragment frames which have already been fragmented. This situation can arise, for example, with IPSEC.

The 'nohdr' field is used in the support of TCP Segmentation Offload ('TSO' for short). Most devices supporting this feature need to make some minor modifications to the TCP and IP headers of an outgoing packet to get it in the right form for the hardware to process. We do not want these modifications to be seen by packet sniffers and the like. So we use this 'nohdr' field and a special bit in the data area reference count to keep track of whether the device needs to replace the data area before making the packet header modifications.

The type of the packet (basically, who is it for), is stored in the 'pkt_type' field. It takes on one of the 'PACKET_*' values defined in the 'linux/if_packet.h' header file. For example, when an incoming ethernet frame is to a destination MAC address matching the MAC address of the ethernet device it arrived on, this field will be set to 'PACKET_HOST'. When a broadcast frame is received, it will be set to 'PACKET_BROADCAST'. And likewise when a multicast packet is received it will be set to 'PACKET_MULTICAST'.

The 'ip_summed' field describes what kind of checksumming assistence the card has provided for a receive packet. It takes on one of three values: 'CHECKSUM_NONE' if the card provided no checksum assistence, 'CHECKSUM_HW' if the two's complement checksum over the entire packet has been provides in 'skb->csum', and 'CHECKSUM_UNNECESSARY' if it is not necessary to verify the checksum of this packet. The latter usually occurs when the packet is received over the loopback device. 'CHECKSUM_UNNECESSARY' can also be used when the device only provides a 'checksum OK' indication for receive packet checksum offload. )

 

·              void                               (*destructor)(struct sk_buff *skb);

·              unsigned int               truesize;

 

一个 SKB 所消耗的内存包括 SKB 本身和 data buffer。

truesize 就是 data buffer 的空间加上 SKB 的大小。

struct sock 结构中，有两个域，用于统计用于发送的内存空间和用于接收的内存空间，它们是：

rmem_alloc

wmem_alloc

 

另外两个域则统计接收到的数据包的总大小和发送的数据包的总大小。

rcvbuf

sndbuf

 

rmem_alloc 和 rcvbuf，wmem_alloc 和sndbuf 用于不同的目的。

 

当我们收到一个数据包后，需要统计这个 socket 总共消耗的内存，这是通过skb_set_owner_r() 来做的。

 

static inline void skb_set_owner_r(struct sk_buff *skb, struct sock *sk)

{

        skb->sk = sk;

        skb->destructor = sock_rfree;

        atomic_add(skb->truesize, &sk->sk_rmem_alloc);

}

最后，当释放一个 SKB 后，需要调用 skb->destruction() 来减少rmem_alloc 的值。

同样，在发送一个 SKB 的时候，需要调用skb_set_owner_w() ，

static inline void skb_set_owner_w(struct sk_buff *skb, struct sock *sk)

{

        sock_hold(sk);

        skb->sk = sk;

        skb->destructor = sock_wfree;

        atomic_add(skb->truesize, &sk->sk_wmem_alloc);

}

在释放这样的一个 SKB 的时候，需要 调用 sock_free() 

void sock_wfree(struct sk_buff *skb)

{

        struct sock *sk = skb->sk;

        /* In case it might be waiting for more memory. */

        atomic_sub(skb->truesize, &sk->sk_wmem_alloc);

        if (!sock_flag(sk, SOCK_USE_WRITE_QUEUE))

               sk->sk_write_space(sk);

        sock_put(sk);

}

 

(Another subtle issue is worth pointing out here. For receive buffer accounting, we do not grab a reference to the socket (via 'sock_hold()'), because the socket handling code will always make sure to free up any packets in it's receive queue before allowing the socket to be destroyed. Whereas for send packets, we have to do proper accounting with 'sock_hold()' and 'sock_put()'. Send packets can be freed asynchronously at any point in time. For example, a packet could sit in a devices transmit queue for a long time under certain conditions. If, meanwhile, the socket is closed, we have to keep the socket reference around until SKBs referencing that socket are liberated. )

 

·              unsigned char                        *head,

·                                                   *data,

·                                                   *tail,

·                                                   *end;

SKB 对 Data Buffer 的巧妙管理，就是靠这四个指针实现的。

 

下图展示了这四个指针是如何管理数据 buffer 的：

Head 指向 buffer 的开始，end 指向 buffer 结束。 Data 指向实际数据的开始，tail 指向实际数据的结束。这四个指针将整个 buffer 分成三个区：

 

Packet data:这个空间保存的是真正的数据

Head room：处于 packet data 之上的空间，是一个空闲区域

Tail room：处于 packet data 之下的空间，也是空闲区域。

 

由于 TCP/IP 协议族是一种分层的协议，传输层、网络层、链路层，都有自己的协议头，因此 TCP/IP 协议栈对于数据包的处理是比较复杂的。为了提高处理效率，避免数据移动、拷贝，sk_buffer 在对数据 buffer 管理的时候，在 packet data 之上和之下，都预留了空间。如果需要增加协议头，只需要从 head room 中拿出一块空间即可，而如果需要增加数据，则可以从 tail room 中获得空间。这样，整个内存只分配一次空间，此后 协议的处理，只需要挪动指针。

5.   Sk_buffer 对内存的管理

我们以构造一个用于发送的数据包的过程，来理解 sk_buffer 是如何管理内存的。

 

5.1.                    构造Skb_buffer

 

alloc_skb() 用于构造 skb_buffer，它需要一个参数，指定了存放 packet 的空间的大小。

构造时，不仅需要创建 skb_buffer 结构本身，还需要分配空间用于保存 packet。

 

skb = alloc_skb(len, GFP_KERNEL);

 

 

 

 

 

 

 

 

 

 

 

 

 

上图是在调用完 alloc_skb() 后的情况：

 

head, data, tail 指向 buffer 开始，end 指向 buffer 结束，整个 buffer 都被当作 tail room。

Sk_buffer 当前的数据长度是0。

 

 

5.2.                    为 protocol header 留出空间

 

通常，当构造一个用于发送的数据包时，需要留出足够的空间给协议头，包括 TCP/UDP header, IP header 和链路层头。

对 IPv4 数据包，可以从 sk->sk_prot->max_header 知道协议头的最大长度。

 

skb_reserve(skb, header_len);

 

 

 

 

 

 

 

 

上

 

 

 

 

 

图是调用 skb_reserver() 后的情况

 

 

5.3.                    将用户空间数据拷贝到 buffer 中

 

首先通过 skb_put(skb, user_data_len) ，从 tail room 中留出用于保存数据的空间

然后通过csum_and_copy_from_user() 将数据从用户空间拷贝到这个空间中。

 

%