dpdk内存初始化（补充）

 dpdk内存初始化补充

——lvyilong316

     关于dpdk内存初始化这里做一个补充，有些细节是上一篇没有讲到的。主要在函数eal_hugepage_info_init中。上一篇我们讲过这个函数是dpdk内存初始化最早执行的函数。这个函数由两处细节需要我们注意下。

int
eal_hugepage_info_init(void)
{
         const char dirent_start_text[] = "hugepages-";
         const size_t dirent_start_len = sizeof(dirent_start_text) - 1;
         unsigned i, num_sizes = 0;
         DIR *dir;
         struct dirent *dirent;

         dir = opendir(sys_dir_path); /* /sys/kernel/mm/hugepages */
         if (dir == NULL)
                  rte_panic("Cannot open directory %s to read system hugepage "
                             "info\n", sys_dir_path);

         for (dirent = readdir(dir); dirent != NULL; dirent = readdir(dir)) {
                  struct hugepage_info *hpi;

                  if (strncmp(dirent->d_name, dirent_start_text,
                               dirent_start_len) != 0)
                           continue;

                  if (num_sizes >= MAX_HUGEPAGE_SIZES)
                           break;

                  hpi = &internal_config.hugepage_info[num_sizes];
                  hpi->hugepage_sz =
                           rte_str_to_size(&dirent->d_name[dirent_start_len]);
                  /*得到的是hugepage的挂载目录*/
                  hpi->hugedir = get_hugepage_dir(hpi->hugepage_sz);

                  /* first, check if we have a mountpoint */
                  if (hpi->hugedir == NULL) {
                           uint32_t num_pages;

                           num_pages = get_num_hugepages(dirent->d_name);
                           if (num_pages > 0)
                                    RTE_LOG(NOTICE, EAL,
                                             "%" PRIu32 " hugepages of size "
                                             "%" PRIu64 " reserved, but no mounted "
                                             "hugetlbfs found for that size\n",
                                             num_pages, hpi->hugepage_sz);
                           continue;
                  }

                  /* try to obtain a writelock */
                  hpi->lock_descriptor = open(hpi->hugedir, O_RDONLY);

                  /* if blocking lock failed */
                  if (flock(hpi->lock_descriptor, LOCK_EX) == -1) {
                           RTE_LOG(CRIT, EAL,
                                    "Failed to lock hugepage directory!\n");
                           break;
                  }
                  /* clear out the hugepages dir from unused pages */
                  if (clear_hugedir(hpi->hugedir) == -1)
                           break;

                  /* for now, put all pages into socket 0,
                   * later they will be sorted */
                  /* 这里还没有按socket统计页数，将内存页数直接记录到hupage_info的num_pages[0]里面了 */
                  hpi->num_pages[0] = get_num_hugepages(dirent->d_name);

                  num_sizes++;
         }
         closedir(dir);

         /* something went wrong, and we broke from the for loop above */
         if (dirent != NULL)
                  return -1;

         internal_config.num_hugepage_sizes = num_sizes;

         /* sort the page directory entries by size, largest to smallest */
         qsort(&internal_config.hugepage_info[0], num_sizes,
               sizeof(internal_config.hugepage_info[0]), compare_hpi);

         /* now we have all info, check we have at least one valid size */
         for (i = 0; i < num_sizes; i++)
                  if (internal_config.hugepage_info[i].hugedir != NULL &&
                      internal_config.hugepage_info[i].num_pages[0] > 0)
                           return 0;

         /* no valid hugepage mounts available, return error */
         return -1;
}

首先是hpi->hugedir = get_hugepage_dir(hpi->hugepage_sz)，这里得到的是什么？如果不认真看代码很容易以为里得到的是/sys/kernel/mm/hugepages下面对应的目录。但如果查看get_hugepage_dir的代码就好发现这里是通过读取/proc/mounts文件，获取hugetlbfs的挂载目录。当然如果通过参数指定了 --huge-dir选项，那么就返回该选项指定的目录。

那么得到这个目录有什么用呢？我们看下面的clear_hugedir函数，这个函数内部实现就是将get_hugepage_dir获取到的目录下面的“*map_*”文件都删除（前提是没有其他进程引用）。为什么要删除这些文件呢？我们知道一个dpdk进程会创建多个map文件用来分配内存。而进程结束时只是unmmap，并没有删除这些文件，那对于hugetlbfs文件系统来说其实这些内存并没有释放（虽然进程也没有使用）。所以在启动dpdk进程时先对这类文件进行一次清理。我们看到进程调用get_num_hugepages获取当前系统可用页面数的逻辑是在clear_hugedir之后的，所以之前进程残留的map文件并不会影响下次dpdk进程启动的内存扫描。