获取Ceph RGW对象存储中文件在底层OSD中的分布

RGW是基于librados构建，并对外提供RESTFul接口的对象存储系统，它兼容S3和Swift接口。所有存储对象都存储在RADOS集群的pool中，本次讨论中涉及到的两个主要pool如下【1】：

index_pool	存储bucket的文件索引对象，每个bucket默认对应该pool下一个索引对象，用来索引该bucket下的所有文件。默认为 .rgw.buckets.index
data_pool	存储所有文件。默认为 .rgw.buckets

 

默认情况下，通过S3方式存储的文件，会分为两部分，包含有文件元数据信息的文件头（默认大小为512KB）和去掉头剩余的文件切片（默认4MB每片）；其中文件元数据信息存储在文件头所对应的对象的扩展属性中。以上两个默认值可以通过设置radosgw下面的两个参数来修改：

rgw_max_chunk_size	文件头大小，默认为：524288
rgw_obj_stripe_size	文件切片大小，默认为：4194304

 

文件在data_pool中存储的对象名为：

{bucket_id}_{object_name}	包含元数据信息的文件头，元数据信息以K-V形式存储于文件头对象的扩展属性中。
{bucket_id}__shadow_{prefix}{num}	去除文件头大小后剩余长度的文件，按照配置的分片大小进行分片，若文件长度小于配置的文件头大小，则无此分片对象。

 

接下来以查找test_bucket下的test_object来演示整个过程：

1）  确定bucket_id：

# radosgw-admin bucket stats --bucket=test_bucket | grep -w id | awk -F '"' '{print $4}'
default.4666.301

2）  确定prefix：

# rados -p .rgw.buckets  getxattr default_4666.301_test_object user.rgw.manifest > /dev/shm/manifest

# ceph-dencoder type RGWObjManifest import /dev/shm/manifest decode dump_json
{
    "objs": [],
    "obj_size": 70146512,                                   // 文件总长度
    "explicit_objs": "false",
    "head_obj": {
        "bucket": {
            "name": "test_bucket",
            "pool": ".rgw.buckets",
            "data_extra_pool": ".rgw.buckets.extra",
            "index_pool": ".rgw.buckets.index",
            "marker": "default.4666.301",
            "bucket_id": "default.4666.301"                 // bucket_id
        },
        "key": "",
        "ns": "",
        "object": "test_object",                            // 文件名
        "instance": ""
    },
    "head_size": 524288,                                    // 文件头实际长度
    "max_head_size": 524288,                                // 文件头最大长度，根据配置设置
    "prefix": ".PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_",          // 切片数据prefix
    "tail_bucket": {
        "name": "test_bucket",                              // 该文件所在的bucket
        "pool": ".rgw.buckets",
        "data_extra_pool": ".rgw.buckets.extra",
        "index_pool": ".rgw.buckets.index",
        "marker": "default.4666.301",
        "bucket_id": "default.4666.301"
    },
    "rules": [
        {
            "key": 0,
            "val": {
                "start_part_num": 0,
                "start_ofs": 524288,
                "part_size": 0,
                "stripe_max_size": 4194304,                // 切片大小
                "override_prefix": ""
            }
        }
    ]
}

3）  根据2）中获取的文件长度和头大小，可以判断是否还存在其它分片，以及通过与分片大小相除，可以得到分片数。
N = （obj_size - head_size + stripe_max_size - 1） / stripe_max_size

    4）通过3）获取到文件分片后，即可使用下面的命令得到各个文件在OSD中的分布。

# ceph osd map default.4666.301_test_object

   
    5） 效果图如下：

# ./get_filemap_s3 test_bucket test_object
   0: 'default.4666.301_test_object' --> 8.2bd912b9 (8.12b9) --> up ([264,686,223], p264) acting ([264,686,223], p264)
   1: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_1' --> 8.8d3bf5e8 (8.15e8) --> up ([506,31,254], p506) acting ([506,31,254], p506)
   2: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_2' --> 8.cdf0325f (8.125f) --> up ([732,66,59], p732) acting ([732,66,59], p732)
   3: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_3' --> 8.411d6e70 (8.e70) --> up ([782,382,402], p782) acting ([782,382,402], p782)
   4: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_4' --> 8.afbbf05e (8.105e) --> up ([617,37,533], p617) acting ([617,37,533], p617)
   5: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_5' --> 8.ddb96ec4 (8.ec4) --> up ([189,516,547], p189) acting ([189,516,547], p189)
   6: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_6' --> 8.8f7e7b13 (8.1b13) --> up ([47,630,106], p47) acting ([47,630,106], p47)
   7: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_7' --> 8.8869f188 (8.1188) --> up ([207,670,720], p207) acting ([207,670,720], p207)
   8: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_8' --> 8.bbb696b5 (8.16b5) --> up ([212,444,154], p212) acting ([212,444,154], p212)
   9: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_9' --> 8.2eab6ecd (8.ecd) --> up ([444,133,751], p444) acting ([444,133,751], p444)
  10: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_10' --> 8.680ce0fc (8.fc) --> up ([494,455,675], p494) acting ([494,455,675], p494)
  11: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_11' --> 8.60e13923 (8.1923) --> up ([733,124,670], p733) acting ([733,124,670], p733)
  12: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_12' --> 8.4baf9886 (8.1886) --> up ([352,547,202], p352) acting ([352,547,202], p352)
  13: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_13' --> 8.cb00a6b9 (8.6b9) --> up ([203,649,7], p203) acting ([203,649,7], p203)
  14: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_14' --> 8.2facd134 (8.1134) --> up ([589,561,690], p589) acting ([589,561,690], p589)
  15: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_15' --> 8.af71adf1 (8.df1) --> up ([617,18,667], p617) acting ([617,18,667], p617)
  16: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_16' --> 8.446896c0 (8.16c0) --> up ([356,311,590], p356) acting ([356,311,590], p356)
  17: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_17' --> 8.681f1962 (8.1962) --> up ([728,499,145], p728) acting ([728,499,145], p728)

【附完整脚本代码】

#!/bin/bash
#
# write by : Yifei Wang <[e-mail]>
# date : Fri Jan 20 17:09:00 CST 2017
#

[ $# -lt 2 ] && {
     echo "Usage: $0  
					BUCKET_NAME OBJECT_NAME"
    echo
    exit 1
}

BUCKET_NAME=$1
OBJECT_NAME=$2

# 根据实际情况配置，若不采用默认值的话
DATA_POOL=".rgw.buckets"
INDEX_POOL=".rgw.buckets.index"

BUCKET_ID=$(radosgw-admin bucket stats --bucket=${BUCKET_NAME} | grep -w id | awk -F'"' '{print $4}')
OBJ_TO_SEARCH=${BUCKET_ID}_${OBJECT_NAME}

rados -p $DATA_POOL getxattr $OBJ_TO_SEARCH user.rgw.manifest > /dev/shm/tmp.manifest

PREFIX=$(ceph-dencoder type RGWObjManifest import /dev/shm/tmp.manifest decode dump_json | grep -w prefix | awk -F '"' '{print $4}')

INDEX=$(echo `ceph-dencoder type RGWObjManifest import /dev/shm/tmp.manifest decode dump_json | grep -wE "obj_size|head_size|stripe_max_size" | awk '{print $2}' | sed 's/,//g'` | awk '{printf "%d", ($1-$2+$3-1)/$3}')

# 列出所有分片
printf "%4d: " 0
echo $(ceph osd map $DATA_POOL $OBJ_TO_SEARCH | awk '{print $7" --> "$10" "$11" --> "$13" "$14" "$15" "$16" "$17" "$18}')
if [ $INDEX -gt 0 ]; then
    for ((i=1; i<=$INDEX; i++)); do
        #echo -n "$i: "
        printf "%4d: " $i
        echo $(ceph osd map $DATA_POOL ${BUCKET_ID}__shadow_${PREFIX}${i} | awk '{print $7" --> "$10" "$11" --> "$13" "$14" "$15" "$16" "$17" "$18}')
    done
fi