Chinaunix首页 | 论坛 | 博客
  • 博客访问: 37496
  • 博文数量: 7
  • 博客积分: 10
  • 博客等级: 民兵
  • 技术积分: 76
  • 用 户 组: 普通用户
  • 注册时间: 2011-10-11 11:52
文章分类

全部博文(7)

文章存档

2017年(6)

2013年(1)

我的朋友

分类: 云计算

2017-02-10 01:10:11

RGW是基于librados构建,并对外提供RESTFul接口的对象存储系统,它兼容S3Swift接口。所有存储对象都存储在RADOS集群的pool中,本次讨论中涉及到的两个主要pool如下【1】:

index_pool

存储bucket的文件索引对象,每个bucket默认对应该pool下一个索引对象,用来索引该bucket下的所有文件。默认为 .rgw.buckets.index

data_pool

存储所有文件。默认为 .rgw.buckets

 

默认情况下,通过S3方式存储的文件,会分为两部分,包含有文件元数据信息的文件头(默认大小为512KB)和去掉头剩余的文件切片(默认4MB每片);其中文件元数据信息存储在文件头所对应的对象的扩展属性中。以上两个默认值可以通过设置radosgw下面的两个参数来修改:

rgw_max_chunk_size

文件头大小,默认为:524288

rgw_obj_stripe_size

文件切片大小,默认为:4194304

 

文件在data_pool中存储的对象名为:

{bucket_id}_{object_name}

包含元数据信息的文件头,元数据信息以K-V形式存储于文件头对象的扩展属性中。

{bucket_id}__shadow_{prefix}{num}

去除文件头大小后剩余长度的文件,按照配置的分片大小进行分片,若文件长度小于配置的文件头大小,则无此分片对象。

 

接下来以查找test_bucket下的test_object来演示整个过程:

1)  确定bucket_id

点击(此处)折叠或打开

  1. # radosgw-admin bucket stats --bucket=test_bucket | grep -w id | awk -F '"' '{print $4}'
  2. default.4666.301


2)  确定prefix

点击(此处)折叠或打开

  1. # rados -p .rgw.buckets getxattr default_4666.301_test_object user.rgw.manifest > /dev/shm/manifest

点击(此处)折叠或打开

  1. # ceph-dencoder type RGWObjManifest import /dev/shm/manifest decode dump_json
  2. {
  3.     "objs": [],
  4.     "obj_size": 70146512,                                   // 文件总长度
  5.     "explicit_objs": "false",
  6.     "head_obj": {
  7.         "bucket": {
  8.             "name": "test_bucket",
  9.             "pool": ".rgw.buckets",
  10.             "data_extra_pool": ".rgw.buckets.extra",
  11.             "index_pool": ".rgw.buckets.index",
  12.             "marker": "default.4666.301",
  13.             "bucket_id": "default.4666.301"                 // bucket_id
  14.         },
  15.         "key": "",
  16.         "ns": "",
  17.         "object": "test_object",                            // 文件名
  18.         "instance": ""
  19.     },
  20.     "head_size": 524288,                                    // 文件头实际长度
  21.     "max_head_size": 524288,                                // 文件头最大长度,根据配置设置
  22.     "prefix": ".PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_",          // 切片数据prefix
  23.     "tail_bucket": {
  24.         "name": "test_bucket",                              // 该文件所在的bucket
  25.         "pool": ".rgw.buckets",
  26.         "data_extra_pool": ".rgw.buckets.extra",
  27.         "index_pool": ".rgw.buckets.index",
  28.         "marker": "default.4666.301",
  29.         "bucket_id": "default.4666.301"
  30.     },
  31.     "rules": [
  32.         {
  33.             "key": 0,
  34.             "val": {
  35.                 "start_part_num": 0,
  36.                 "start_ofs": 524288,
  37.                 "part_size": 0,
  38.                 "stripe_max_size": 4194304,                // 切片大小
  39.                 "override_prefix": ""
  40.             }
  41.         }
  42.     ]
  43. }


3)  根据2)中获取的文件长度和头大小,可以判断是否还存在其它分片,以及通过与分片大小相除,可以得到分片数。
N = (obj_size - head_size + stripe_max_size - 1) / stripe_max_size

    4)通过3)获取到文件分片后,即可使用下面的命令得到各个文件在OSD中的分布。

点击(此处)折叠或打开

  1. # ceph osd map default.4666.301_test_object
   
    5) 效果图如下:

点击(此处)折叠或打开

  1. # ./get_filemap_s3 test_bucket test_object
  2.    0: 'default.4666.301_test_object' --> 8.2bd912b9 (8.12b9) --> up ([264,686,223], p264) acting ([264,686,223], p264)
  3.    1: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_1' --> 8.8d3bf5e8 (8.15e8) --> up ([506,31,254], p506) acting ([506,31,254], p506)
  4.    2: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_2' --> 8.cdf0325f (8.125f) --> up ([732,66,59], p732) acting ([732,66,59], p732)
  5.    3: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_3' --> 8.411d6e70 (8.e70) --> up ([782,382,402], p782) acting ([782,382,402], p782)
  6.    4: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_4' --> 8.afbbf05e (8.105e) --> up ([617,37,533], p617) acting ([617,37,533], p617)
  7.    5: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_5' --> 8.ddb96ec4 (8.ec4) --> up ([189,516,547], p189) acting ([189,516,547], p189)
  8.    6: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_6' --> 8.8f7e7b13 (8.1b13) --> up ([47,630,106], p47) acting ([47,630,106], p47)
  9.    7: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_7' --> 8.8869f188 (8.1188) --> up ([207,670,720], p207) acting ([207,670,720], p207)
  10.    8: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_8' --> 8.bbb696b5 (8.16b5) --> up ([212,444,154], p212) acting ([212,444,154], p212)
  11.    9: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_9' --> 8.2eab6ecd (8.ecd) --> up ([444,133,751], p444) acting ([444,133,751], p444)
  12.   10: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_10' --> 8.680ce0fc (8.fc) --> up ([494,455,675], p494) acting ([494,455,675], p494)
  13.   11: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_11' --> 8.60e13923 (8.1923) --> up ([733,124,670], p733) acting ([733,124,670], p733)
  14.   12: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_12' --> 8.4baf9886 (8.1886) --> up ([352,547,202], p352) acting ([352,547,202], p352)
  15.   13: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_13' --> 8.cb00a6b9 (8.6b9) --> up ([203,649,7], p203) acting ([203,649,7], p203)
  16.   14: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_14' --> 8.2facd134 (8.1134) --> up ([589,561,690], p589) acting ([589,561,690], p589)
  17.   15: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_15' --> 8.af71adf1 (8.df1) --> up ([617,18,667], p617) acting ([617,18,667], p617)
  18.   16: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_16' --> 8.446896c0 (8.16c0) --> up ([356,311,590], p356) acting ([356,311,590], p356)
  19.   17: 'default.4666.301__shadow_.PhSGqxV5VcvqM7fmf2mKWh2QDl_hZZy_17' --> 8.681f1962 (8.1962) --> up ([728,499,145], p728) acting ([728,499,145], p728)


【附完整脚本代码】

点击(此处)折叠或打开

  1. #!/bin/bash
  2. #
  3. # write by : Yifei Wang <wangyifei@cstor.cn>
  4. # date : Fri Jan 20 17:09:00 CST 2017
  5. #

  6. [ $# -lt 2 ] && {
  7.      echo "Usage: $0 BUCKET_NAME OBJECT_NAME"
  8.     echo
  9.     exit 1
  10. }

  11. BUCKET_NAME=$1
  12. OBJECT_NAME=$2

  13. # 根据实际情况配置,若不采用默认值的话
  14. DATA_POOL=".rgw.buckets"
  15. INDEX_POOL=".rgw.buckets.index"

  16. BUCKET_ID=$(radosgw-admin bucket stats --bucket=${BUCKET_NAME} | grep -w id | awk -F'"' '{print $4}')
  17. OBJ_TO_SEARCH=${BUCKET_ID}_${OBJECT_NAME}

  18. rados -p $DATA_POOL getxattr $OBJ_TO_SEARCH user.rgw.manifest > /dev/shm/tmp.manifest

  19. PREFIX=$(ceph-dencoder type RGWObjManifest import /dev/shm/tmp.manifest decode dump_json | grep -w prefix | awk -F '"' '{print $4}')

  20. INDEX=$(echo `ceph-dencoder type RGWObjManifest import /dev/shm/tmp.manifest decode dump_json | grep -wE "obj_size|head_size|stripe_max_size" | awk '{print $2}' | sed 's/,//g'` | awk '{printf "%d", ($1-$2+$3-1)/$3}')

  21. # 列出所有分片
  22. printf "%4d: " 0
    echo $(ceph osd map $DATA_POOL $OBJ_TO_SEARCH | awk '{print $7" --> "$10" "$11" --> "$13" "$14" "$15" "$16" "$17" "$18}')
    if [ $INDEX -gt 0 ]; then
        for ((i=1; i<=$INDEX; i++)); do
            #echo -n "$i: "
            printf "%4d: " $i
            echo $(ceph osd map $DATA_POOL ${BUCKET_ID}__shadow_${PREFIX}${i} | awk '{print $7" --> "$10" "$11" --> "$13" "$14" "$15" "$16" "$17" "$18}')
        done
    fi



  23. 【参考】
        1. http://www.cnblogs.com/damizhou/p/5819022.html


阅读(6232) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~