vt-d网卡透传问题

https://blog.csdn.net/wangzhen_csdn/article/details/88747287

与Legacy KVM Device Assignment（使用pci-stub driver）相比，VFIO（Virtual Function IO?）最大的改进就是隔离了设备之间的DMA和中断！！！，以及对IOMMU Group！！！的支持，从而有了更好的安全性。

IOMMU Group可以认为是对PCI设备的分组，每个group里面的设备被视作IOMMU可以操作的最小整体；换句话说，同一个IOMMU Group里的设备?不可以分配给不同的客户机。

在以前的Legacy KVM Device Assignment中，并不会检查这一点，而后面的操作却注定是失败的。新的VFIO会检查并及时报错。

另外，新的VFIO架构也做到了平台无关，有更好的可移植性。

网卡的解绑与绑定
6.ubund form host device driver

echo "0000:18:00.0" > /sys/bus/pci/devices/0000\:18\:00.1/driver/unbind

echo "0000:18:00.1" > /sys/bus/pci/devices/0000\:18\:00.1/driver/unbind

 

7.bind to vfio-pci

modprobe vfio
modprobe vfio-pci
[root@REDIS-Client ~]$echo "8086 158b" > /sys/bus/pci/drivers/vfio-pci/new_id
[root@REDIS-Client ~]$ls /dev/vfio/
23  24 vfio

 

8.起guest时添加参数

-device vfio-pci,host=18:00.0,id=net0   -device vfio-pci,host=18:00.1,id=net1

[root@localhost /]#
[root@localhost /]# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.2
/sys/kernel/iommu_groups/1/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.5
/sys/kernel/iommu_groups/6/devices/0000:00:1c.6
/sys/kernel/iommu_groups/6/devices/0000:00:1c.7
/sys/kernel/iommu_groups/6/devices/0000:04:00.0
/sys/kernel/iommu_groups/6/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:06:00.0
/sys/kernel/iommu_groups/6/devices/0000:07:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:1d.0
/sys/kernel/iommu_groups/7/devices/0000:00:1d.1
/sys/kernel/iommu_groups/7/devices/0000:08:00.0
/sys/kernel/iommu_groups/7/devices/0000:09:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.2
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4
[root@localhost /]#
[root@localhost /]#

[root@localhost /]# ls /sys/bus/pci/devices/0000:01:10.0/iommu_group/devices/
ls: cannot access /sys/bus/pci/devices/0000:01:10.0/iommu_group/devices/: No such file or directory
[root@localhost /]# ls /sys/bus/pci/devices/0000:08:00.0/iommu_group/devices/
0000:00:1d.0  0000:00:1d.1  0000:08:00.0  0000:09:00.0
[root@localhost /]#
[root@localhost /]#
[root@localhost /]#
[root@localhost /]# ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/
0000:00:1c.0  0000:00:1c.5  0000:00:1c.6  0000:00:1c.7  0000:04:00.0  0000:05:00.0  0000:06:00.0  0000:07:00.0
[root@localhost /]#
[root@localhost /]# readlink /sys/bus/pci/devices/0000:01:10.0/iommu_group
[root@localhost /]# readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group
../../../../kernel/iommu_groups/7

==========================================================

（IOMMU Group可以认为是对PCI设备的分组，每个group里面的设备被视作IOMMU可以操作的最小整体；换句话说，同一个IOMMU Group里的设备不可以分配给不同的客户机或者部分虚机部分主机使用）

int iommu_group_add_device(struct iommu_group *group, struct device *dev)
{
        int ret, i = 0;
        struct iommu_device *device;

        dump_stack();添加dump_stack打印内核调用栈
        if (!strcmp(dev_name(dev), "0000:06:00.0")) {  /*过滤掉enp6s0接口 PCI为0000:06:00.0*/
                printk("iommu_group_add_device==DEV-PCI===[%s]===GROUP-ID====[%d] ", dev_name(dev),group->id);
                return 0;
       

附：interl iommu内核调用流程：

intel_iommu_init
  -->根据no_iommu及dmar_disabled判断是否开启iommu
  ——>init_dmars
     -->初始化各个iommu
     -->根据iommu_identity_mapping初始化si_domain_init,当前内核没有采用，具体作用见下面分析
  -->bus_set_iommu
     -->iommu_bus_init
        -->add_iommu_group   /* 遍历所有设备执行  */
            -->intel_iommu_add_device  
               -->device_to_iommu   /* 根据dmar_drhd_unit找到当前device属于哪个intel_iommu */
               -->iommu_device_link  /* 创建 /sys/dmarxx */
               -->iommu_group_get_for_dev
               /* 找到设备的iommu_group, 起初时候必然没有，所以创建 */
                  -->iommu_group_get   
                  -->iommu_group_get_for_pci_dev   
                     -->按照iommu_group定义分配组，同一个group是可以共享的
                     -->iommu_group_alloc
                     -->group->default_domain = __iommu_domain_alloc
                     -->group->domain = group->default_domain
                  -->iommu_group_add_device            -------------------------------------------------------------------------过滤掉网卡，以PCI号为判断条件；
                     -->dev->iommu_group = group    /* device 和 iommu_group 关联  */
                     -->iommu_group_create_direct_mappings  /* AMD */
                     -->list_add_tail(&device->list, &group->devices) /* group list */
                     -->__iommu_attach_device(group->domain, dev) 
                       /* 按照vt-d，一个group属于可以隔离的最小单元，并且同一个group srcid相同，所以一个group可以有一个domain，domain和device也要关联起来 */
                        -->intel_iommu_attach_device
                           -->domain_add_dev_info
                              -->dmar_insert_one_dev_info
                                 -->device_domain_info *info = alloc_devinfo_mem();
                                 -->domain_attach_iommu(domain, iommu) /*domain 和 iommu关联*/
                                 -->list_add(&info->link, &domain->devices)
                                 -->list_add(&info->global, &device_domain_list)
                                 -->dev->archdata.iommu = info
                                 -->domain_context_mapping  /*这一步很重要，这里面将domain的页表地址放入了其对应的src项中  */

修改后系统iommu group绑定信息：

[root@localhost /]# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.2
/sys/kernel/iommu_groups/1/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.5
/sys/kernel/iommu_groups/6/devices/0000:00:1c.6
/sys/kernel/iommu_groups/6/devices/0000:00:1c.7

/sys/kernel/iommu_groups/6/devices/0000:06:00.0-----被删除
/sys/kernel/iommu_groups/6/devices/0000:07:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:1d.0
/sys/kernel/iommu_groups/7/devices/0000:00:1d.1
/sys/kernel/iommu_groups/7/devices/0000:08:00.0
/sys/kernel/iommu_groups/7/devices/0000:09:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.2
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4
[root@localhost /]#