Chinaunix首页 | 论坛 | 博客

好想好好爱你！badb0y.blog.chinaunix.net

首页　| 　博文目录　| 　关于我

badb0y

  • 博客访问： 7611800
  • 博文数量： 1769
  • 博客积分： 18684
  • 博客等级： 上将
  • 技术积分： 16352
  • 用 户 组： 普通用户
  • 注册时间： 2010-06-02 10:28
个人简介

啥也没写

文章分类

全部博文（1769）

文章存档

2024年（15）

2023年（44）

2022年（39）

2021年（46）

2020年（43）

2019年（27）

2018年（44）

2017年（50）

2016年（47）

2015年（15）

2014年（21）

2013年（43）

2012年（143）

2011年（228）

2010年（263）

2009年（384）

2008年（246）

2007年（30）

2006年（38）

2005年（2）

2004年（1）

我的朋友
最近访客
推荐博文
相关博文
Ceph集群部署

分类： 服务器与存储

2016-06-30 15:46:22

原文地址：Ceph集群部署 作者：gongping11

Ceph简介

Ceph的底层是一个分布式对象存储系统，通过lib库的方式对外提供了块设备、文件系统、对象存储的服务，因此将块设备、文件系统、对象存储的服务进行了统一。Ceph的底层是Rados集群，在Rados集群之上提供了librados。在librados之上又提供了librbd、libgw。


ceph的构成


     Ceph的基础是Rados，Rados也被称为Ceph集群，是其他服务的基础。关于Rados集群中最基本的组件是监控器和OSD，监控器负责维护集群的状态，维护集群的状态。而OSD包含OSD设备和OSD设备的守护进程。而Rados集群的部署主要是监控器和OSD的部署。关于Ceph的详细信息可参看官方文档。本文主要记录一下Ceph的部署。

准备工作

服务器情况如下所示:
       ip地址         主机名    osd磁盘
      192.168.1.212  cell01    sdb1  
      192.168.1.213  cell02    sdb1
      192.168.1.214  cell03    sdb1
     本次部署过程中采用192.168.1.212即作为服务器节点又作为管理节点。因此本文大部分的部署都是在192.168.1.212的ceph用户下进行部署。同时每个节点中都部署osd和monitor。本文基于Ceph-deploy进行部署

      由于Ceph采用了本地编译好的rpm包进行管理，同时提供了yum的repo文件，本来可采用简单的rpm包进行安装，但Ceph的依赖关系太过复杂，因此最方便的方式是采用yum进行安装，这样的安装方式不需要关注包之间的依赖，yum会进行包的关联处理。
首先采用局域网中的软件仓库安装createrepo工具。

点击(此处)折叠或打开

  1. [root@cell01 tmp]# yum install createrepo
     安装该文件之后，将对应的rpm包放置在提供的repo文件指定的路劲中，并拷贝对应的repo文件到/etc/yum.repo.d/目录中。更新之后采用yum repolist all查看刚才添加的repo对象是否存在。

点击(此处)折叠或打开

  1. [root@cell01 tmp]# cp xxx.repo /etc/yum.repos.d/xxx.repo
  2. [root@cell01 tmp]# yum repolist all
     添加用户，可以默认使用root或者其他的用户，但为了统一的处理，在不同的服务器上可添加一个ceph的用户。

点击(此处)折叠或打开

  1. [root@cell01 /]# useradd -d /home/ceph -m ceph <<----添加用户
  2. [root@cell01 /]# passwd ceph <<----添加密码
  3. Changing password for user ceph.
  4. New password:
  5. BAD PASSWORD: it is based on a dictionary word
  6. BAD PASSWORD: is too simple
  7. Retype new password:
  8. passwd: all authentication tokens updated successfully.
  9. [root@cell01 /]# echo "ceph ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/ceph <<--------设置sudo的操作不需要输入密码
  10. ceph ALL = (root) NOPASSWD:ALL
  11. [root@cell01 /]# chmod 0440 /etc/sudoers.d/ceph <<----设置对应的配置文件权限
  12. [root@cell01 /]# su - ceph
  13. [ceph@cell01 ~]$ ll
  14. total 0
  15. [ceph@cell01 ~]$ pwd
  16. /home/ceph
  17. [ceph@cell01 ~]$ ll -al
  18. total 24
  19. drwx------ 3 ceph ceph 4096 Sep 1 09:34 .
  20. drwxr-xr-x. 4 root root 4096 Sep 1 09:34 ..
  21. -rw-r--r-- 1 ceph ceph 18 Jul 18 2013 .bash_logout
  22. -rw-r--r-- 1 ceph ceph 176 Jul 18 2013 .bash_profile
  23. -rw-r--r-- 1 ceph ceph 124 Jul 18 2013 .bashrc
  24. drwxr-xr-x 4 ceph ceph 4096 Aug 31 16:29 .mozilla
  25. [ceph@cell01 ~]$ exit
  26. logout
  27. [root@cell01 /]# ssh 192.168.1.213
  28. The authenticity of host '192.168.1.213 (192.168.1.213)' can't be established.
  29. RSA key fingerprint is d5:12:f2:92:34:28:22:06:20:a3:1d:56:9e:cc:d6:b7.
  30. Are you sure you want to continue connecting (yes/no)? yes
  31. Warning: Permanently added '192.168.1.213' (RSA) to the list of known hosts.
  32. root@192.168.1.213's password:
  33. Last login: Mon Aug 31 17:06:49 2015 from 10.45.34.73
  34. [root@cell02 ~]# useradd -d /home/ceph -m ceph
  35. [root@cell02 ~]# passwd ceph
  36. Changing password for user ceph.
  37. New password:
  38. BAD PASSWORD: it is based on a dictionary word
  39. BAD PASSWORD: is too simple
  40. Retype new password:
  41. passwd: all authentication tokens updated successfully.
  42. [root@cell02 ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/ceph
  43. ceph ALL = (root) NOPASSWD:ALL
  44. [root@cell02 ~]# chmod 0440 /etc/sudoers.d/ceph
  45. [root@cell02 ~]# exit
  46. logout
  47. Connection to 192.168.1.213 closed.
  48. [root@cell01 /]# ssh 192.168.1.214
  49. root@192.168.1.214's password:
  50. Last login: Mon Aug 31 16:50:39 2015 from 192.168.1.212
  51. [root@cell03 ~]# useradd -d /home/ceph -m ceph
  52. [root@cell03 ~]# passwd ceph
  53. Changing password for user ceph.
  54. New password:
  55. BAD PASSWORD: it is based on a dictionary word
  56. BAD PASSWORD: is too simple
  57. Retype new password:
  58. passwd: all authentication tokens updated successfully.
  59. [root@cell03 ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | tee /etc/sudoers.d/ceph
  60. ceph ALL = (root) NOPASSWD:ALL
  61. [root@cell03 ~]# chmod 0440 /etc/sudoers.d/ceph
  62. [root@cell03 ~]# exit
  63. logout
  64. Connection to 192.168.1.214 closed.
      在添加用户之后，需要实现节点之间的无密码ssh操作，由于采用ceph用户操作，因此在ceph的用户目录下执行ssh，先产生公钥，然后将对应的钥匙复制到其他的节点。

点击(此处)折叠或打开

  1. [root@cell01 /]# su - ceph
  2. [ceph@cell01 ~]$ ssh-keygen <<----产生公钥秘钥(管理节点上操作)
  3. Generating public/private rsa key pair.
  4. Enter file in which to save the key (/home/ceph/.ssh/id_rsa):
  5. Created directory '/home/ceph/.ssh'.
  6. Enter passphrase (empty for no passphrase):
  7. Enter same passphrase again:
  8. Your identification has been saved in /home/ceph/.ssh/id_rsa.
  9. Your public key has been saved in /home/ceph/.ssh/id_rsa.pub.
  10. The key fingerprint is:
  11. 9b:ea:55:fd:a0:a8:34:18:e0:3d:1f:1e:bb:8c:de:9a ceph@cell01
  12. The key's randomart image is:
  13. +--[ RSA 2048]----+
  14. | |
  15. | |
  16. | . |
  17. | . o . |
  18. | . + o S . o |
  19. | * + = . o |
  20. | . * = . . |
  21. | * * |
  22. | .EoB |
  23. +-----------------+
  24. [ceph@cell01 ~]$ ssh-copy-id ceph@cell02 <<------拷贝公钥到其他的服务器节点中
  25. ssh: Could not resolve hostname cell02: Name or service not known <<----没有添加对端其他节点的信息
  26. [ceph@cell01 ~]$ exit
  27. logout
  28. [root@cell01 /]# vi /etc/hosts <<-----添加主机节点信息
  29. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
  30. ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
  33. 192.168.1.212 cell01 ireadmin
  34. 192.168.1.213 cell02
  35. 192.168.1.214 cell03
  36. ~
  37. "/etc/hosts" 6L, 225C written
  38. [root@cell01 /]# su - ceph
  39. [ceph@cell01 ~]$ ssh-copy-id ceph@cell02 <<------拷贝公钥到其他的服务器节点中
  40. The authenticity of host 'cell02 (192.168.1.213)' can't be established.
  41. RSA key fingerprint is d5:12:f2:92:34:28:22:06:20:a3:1d:56:9e:cc:d6:b7.
  42. Are you sure you want to continue connecting (yes/no)? yes
  43. Warning: Permanently added 'cell02,192.168.1.213' (RSA) to the list of known hosts.
  44. ceph@cell02's password:
  45. Now try logging into the machine, with "ssh 'ceph@cell02'", and check in:
  48. .ssh/authorized_keys
  51. to make sure we haven't added extra keys that you weren't expecting.
  54. [ceph@cell01 ~]$ ssh-copy-id ceph@cell03 <<------拷贝公钥到其他的服务器节点中
  55. The authenticity of host 'cell03 (192.168.1.214)' can't be established.
  56. RSA key fingerprint is 04:bc:35:fd:e5:3b:dd:d1:3a:7a:15:06:05:b4:95:5e.
  57. Are you sure you want to continue connecting (yes/no)? yes
  58. Warning: Permanently added 'cell03,192.168.1.214' (RSA) to the list of known hosts.
  59. ceph@cell03's password:
  60. Now try logging into the machine, with "ssh 'ceph@cell03'", and check in:
  63. .ssh/authorized_keys
  66. to make sure we haven't added extra keys that you weren't expecting.
  67. [ceph@cell01 ~]$ exit
  68. logout
  69. [root@cell01 /]# vi /etc/sudoers <<-----将ssh登陆的密码输入取消，从Defaults requiretty，改为Defaults:ceph !requiretty
  70. ## Sudoers allows particular users to run various commands as
  71. ## the root user, without needing the root password.
  72. ##
  73. ## Examples are provided at the bottom of the file for collections
  74. ## of related commands, which can then be delegated out to particular
  75. ## users or groups.
  76. ##
  77. ## This file must be edited with the 'visudo' command.
  80. ## Host Aliases
  81. ## Groups of machines. You may prefer to use hostnames (perhaps using
  82. ## wildcards for entire domains) or IP addresses instead.
  83. # Host_Alias FILESERVERS = fs1, fs2
  84. /Defaults
  85. ## Processes
  86. # Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall
  89. ## Drivers
  90. # Cmnd_Alias DRIVERS = /sbin/modprobe
  93. # Defaults specification
  96. #
  97. # Disable "ssh hostname sudo <cmd>", because it will show the password in clear.
  98. # You have to run "ssh -t hostname sudo <cmd>".
  99. #
  100. Defaults:ceph !requiretty
  103. #
  104. # Refuse to run if unable to disable echo on the tty. This setting should also be
  105. # changed in order to be able to use sudo without a tty. See requiretty above.
  106. #
  107. Defaults !visiblepw
  108. [root@cell01 /]# su - ceph
  109. [ceph@cell01 ~]$ vi ./.ssh/config <<------------创建ceph用户的ssh的配置文件，通过这个配置文件可通过ssh进行别名的访问
  112. Host cell01
  113. Hostname 192.168.1.212
  114. User ceph
  117. Host cell02
  118. Hostname 192.168.1.213
  119. User ceph
  120. Host cell03
  121. Hostname 192.168.1.214
  122. User ceph
  123. ~
  124. ~
  125. "./.ssh/config" 10L, 206C written
  126. [ceph@cell01 ~]$ ll ./.ssh/config
  127. -rw-rw-r-- 1 ceph ceph 206 Sep 1 10:46 ./.ssh/config
  128. [ceph@cell01 ~]$ ssh cell02 <<-----采用别名进行ssh登陆
  129. Bad owner or permissions on /home/ceph/.ssh/config
  130. [ceph@cell01 ~]$ chmod 600 /home/ceph/.ssh/config <<-------设置config文件的权限，只有600才能正确的执行。
  131. [ceph@cell01 ~]$ ssh cell02
  132. [ceph@cell02 ~]$ exit
  133. logout
  134. Connection to 192.168.1.213 closed.
  135. [ceph@cell01 ~]$

      安装对应的软件，ceph、ntp等组件

点击(此处)折叠或打开

  1. [ceph@cell01 ~]$ sudo yum update && sudo yum install ceph-deploy <<----安装ceph-deploy
  2. Loaded plugins: fastestmirror, security
  3. Loading mirror speeds from cached hostfile
  4. Setting up Update Process
  5. No Packages marked for Update
  6. Loaded plugins: fastestmirror, security
  7. Loading mirror speeds from cached hostfile
  8. Setting up Install Process
  9. Package ceph-deploy-1.5.19-0.noarch already installed and latest version
  10. Nothing to do
  11. [ceph@cell01 ~]$ sudo yum install ntp ntpupdate ntp-doc
  12. Loaded plugins: fastestmirror, security
  13. Loading mirror speeds from cached hostfile
  14. Setting up Install Process
  15. Package ntp-4.2.6p5-1.el6.centos.x86_64 already installed and latest version
  16. No package ntpupdate available.
  17. Resolving Dependencies
  18. --> Running transaction check
  19. ---> Package ntp-doc.noarch 0:4.2.6p5-1.el6.centos will be installed
  20. --> Finished Dependency Resolution
  23. Dependencies Resolved
  26. =========================================================================================================
  27. Package Arch Version Repository Size
  28. =========================================================================================================
  29. Installing:
  30. ntp-doc noarch 4.2.6p5-1.el6.centos addons 1.0 M
  33. Transaction Summary
  34. =========================================================================================================
  35. Install 1 Package(s)
  38. Total download size: 1.0 M
  39. Installed size: 1.6 M
  40. Is this ok [y/N]: y
  41. Downloading Packages:
  42. ntp-doc-4.2.6p5-1.el6.centos.noarch.rpm | 1.0 MB 00:00
  43. Running rpm_check_debug
  44. Running Transaction Test
  45. Transaction Test Succeeded
  46. Running Transaction
  47. Installing : ntp-doc-4.2.6p5-1.el6.centos.noarch 1/1
  48. Verifying : ntp-doc-4.2.6p5-1.el6.centos.noarch 1/1
  51. Installed:
  52. ntp-doc.noarch 0:4.2.6p5-1.el6.centos
  55. Complete!
      经过以上的准备之后，可以进行Rados集群的部署啦。

监控器的部署

     首先在管理节点上创建目录/home/ceph/mkdir my-cluster, 并进入到该目录中cd ./my-cluster，然后创建集群，初始化监控器。

点击(此处)折叠或打开

  1. [ceph@cell01 my-cluster]$ cd ./my-cluster <<-----空的管理目录
  2. [ceph@cell01 my-cluster]$ ll
  3. total 0
  4. [ceph@cell01 my-cluster]$ ceph-deploy new cell01 cell02 cell03 <<-----创建集群，参数通常是待部署监控器的节点
  5. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  6. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy new cell01 cell02 cell03
  7. [ceph_deploy.new][DEBUG ] Creating new cluster named ceph
  8. [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
  9. [cell01][DEBUG ] connection detected need for sudo
  10. [cell01][DEBUG ] connected to host: cell01
  11. [cell01][DEBUG ] detect platform information from remote host
  12. [cell01][DEBUG ] detect machine type
  13. [cell01][DEBUG ] find the location of an executable
  14. [cell01][INFO ] Running command: sudo /sbin/ip link show
  15. [cell01][INFO ] Running command: sudo /sbin/ip addr show
  16. [cell01][DEBUG ] IP addresses found: ['192.168.1.212']
  17. [ceph_deploy.new][DEBUG ] Resolving host cell01
  18. [ceph_deploy.new][DEBUG ] Monitor cell01 at 192.168.1.212
  19. [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
  20. [cell02][DEBUG ] connected to host: cell01
  21. [cell02][INFO ] Running command: ssh -CT -o BatchMode=yes cell02
  22. [cell02][DEBUG ] connection detected need for sudo
  23. [cell02][DEBUG ] connected to host: cell02
  24. [cell02][DEBUG ] detect platform information from remote host
  25. [cell02][DEBUG ] detect machine type
  26. [cell02][DEBUG ] find the location of an executable
  27. [cell02][INFO ] Running command: sudo /sbin/ip link show
  28. [cell02][INFO ] Running command: sudo /sbin/ip addr show
  29. [cell02][DEBUG ] IP addresses found: ['172.16.10.213', '192.168.1.213']
  30. [ceph_deploy.new][DEBUG ] Resolving host cell02
  31. [ceph_deploy.new][DEBUG ] Monitor cell02 at 192.168.1.213
  32. [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
  33. [cell03][DEBUG ] connected to host: cell01
  34. [cell03][INFO ] Running command: ssh -CT -o BatchMode=yes cell03
  35. [cell03][DEBUG ] connection detected need for sudo
  36. [cell03][DEBUG ] connected to host: cell03
  37. [cell03][DEBUG ] detect platform information from remote host
  38. [cell03][DEBUG ] detect machine type
  39. [cell03][DEBUG ] find the location of an executable
  40. [cell03][INFO ] Running command: sudo /sbin/ip link show
  41. [cell03][INFO ] Running command: sudo /sbin/ip addr show
  42. [cell03][DEBUG ] IP addresses found: ['192.168.1.214']
  43. [ceph_deploy.new][DEBUG ] Resolving host cell03
  44. [ceph_deploy.new][DEBUG ] Monitor cell03 at 192.168.1.214
  45. [ceph_deploy.new][DEBUG ] Monitor initial members are ['cell01', 'cell02', 'cell03']
  46. [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.1.212', '192.168.1.213', '192.168.1.214']
  47. [ceph_deploy.new][DEBUG ] Creating a random mon key...
  48. [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
  49. [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
  50. Error in sys.exitfunc:
  51. [ceph@cell01 my-cluster]$ ll <<-------------生成了全局的配置文件，监控器的钥匙链等文件
  52. total 12
  53. -rw-rw-r-- 1 ceph ceph 276 Sep 1 17:09 ceph.conf
  54. -rw-rw-r-- 1 ceph ceph 2689 Sep 1 17:09 ceph.log
  55. -rw-rw-r-- 1 ceph ceph 73 Sep 1 17:09 ceph.mon.keyring
  56. [ceph@cell01 my-cluster]$ ceph-deploy --overwrite-conf mon create cell01 cell02 cell03 <<-------创建监控器，对应的参数为mon {monitor-node1} [{monitor-node2}...]
  57. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  58. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy --overwrite-conf mon create cell01 cell02 cell03
  59. [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cell01 cell02 cell03
  60. [ceph_deploy.mon][DEBUG ] detecting platform for host cell01 ...
  61. [cell01][DEBUG ] connection detected need for sudo
  62. [cell01][DEBUG ] connected to host: cell01 <<----------开始cell01监控器的部署
  63. [cell01][DEBUG ] detect platform information from remote host
  64. [cell01][DEBUG ] detect machine type
  65. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  66. [cell01][DEBUG ] determining if provided host has same hostname in remote
  67. [cell01][DEBUG ] get remote short hostname
  68. [cell01][DEBUG ] deploying mon to cell01 <<----部署监控器
  69. [cell01][DEBUG ] get remote short hostname
  70. [cell01][DEBUG ] remote hostname: cell01
  71. [cell01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf <<---写监控器的配置文件
  72. [cell01][DEBUG ] create the mon path if it does not exist
  73. [cell01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell01/done
  74. [cell01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-cell01/done
  75. [cell01][INFO ] creating tmp path: /var/lib/ceph/tmp
  76. [cell01][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-cell01.mon.keyring
  77. [cell01][DEBUG ] create the monitor keyring file
  78. [cell01][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i cell01 --keyring /var/lib/ceph/tmp/ceph-cell01.mon.keyring <<---创建mon的fs，可查看对应的目录下包含的数据，主要包括一个db，一个keyring等
  79. [cell01][DEBUG ] ceph-mon: mon.noname-a 192.168.1.212:6789/0 is local, renaming to mon.cell01 <<----重命名mon名
  80. [cell01][DEBUG ] ceph-mon: set fsid to 9061096f-d9f9-4946-94f1-296ab5080a97 <<----多个监控器对应的fsid是一致的
  81. [cell01][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-cell01 for mon.cell01
  82. [cell01][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-cell01.mon.keyring <<----删除临时文件
  83. [cell01][DEBUG ] create a done file to avoid re-doing the mon deployment
  84. [cell01][DEBUG ] create the init path if it does not exist
  85. [cell01][DEBUG ] locating the `service` executable...
  86. [cell01][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell01 <<----启动监控器
  87. [cell01][DEBUG ] === mon.cell01 ===
  88. [cell01][DEBUG ] Starting Ceph mon.cell01 on cell01...
  89. [cell01][DEBUG ] Starting ceph-create-keys on cell01...
  90. [cell01][WARNIN] No data was received after 7 seconds, disconnecting...
  91. [cell01][INFO ] Running command: sudo chkconfig ceph on
  92. [cell01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell01.asok mon_status <<---获取当前的集群状态
  93. [cell01][DEBUG ] ********************************************************************************
  94. [cell01][DEBUG ] status for monitor: mon.cell01
  95. [cell01][DEBUG ] {
  96. [cell01][DEBUG ] "election_epoch": 0,
  97. [cell01][DEBUG ] "extra_probe_peers": [
  98. [cell01][DEBUG ] "192.168.1.213:6789/0",
  99. [cell01][DEBUG ] "192.168.1.214:6789/0"
  100. [cell01][DEBUG ] ],
  101. [cell01][DEBUG ] "monmap": {
  102. [cell01][DEBUG ] "created": "0.000000",
  103. [cell01][DEBUG ] "epoch": 0,
  104. [cell01][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  105. [cell01][DEBUG ] "modified": "0.000000",
  106. [cell01][DEBUG ] "mons": [
  107. [cell01][DEBUG ] {
  108. [cell01][DEBUG ] "addr": "192.168.1.212:6789/0",
  109. [cell01][DEBUG ] "name": "cell01",
  110. [cell01][DEBUG ] "rank": 0
  111. [cell01][DEBUG ] },
  112. [cell01][DEBUG ] {
  113. [cell01][DEBUG ] "addr": "0.0.0.0:0/1",
  114. [cell01][DEBUG ] "name": "cell02",
  115. [cell01][DEBUG ] "rank": 1
  116. [cell01][DEBUG ] },
  117. [cell01][DEBUG ] {
  118. [cell01][DEBUG ] "addr": "0.0.0.0:0/2",
  119. [cell01][DEBUG ] "name": "cell03",
  120. [cell01][DEBUG ] "rank": 2
  121. [cell01][DEBUG ] }
  122. [cell01][DEBUG ] ]
  123. [cell01][DEBUG ] },
  124. [cell01][DEBUG ] "name": "cell01",
  125. [cell01][DEBUG ] "outside_quorum": [
  126. [cell01][DEBUG ] "cell01"
  127. [cell01][DEBUG ] ],
  128. [cell01][DEBUG ] "quorum": [],
  129. [cell01][DEBUG ] "rank": 0,
  130. [cell01][DEBUG ] "state": "probing",
  131. [cell01][DEBUG ] "sync_provider": []
  132. [cell01][DEBUG ] }
  133. [cell01][DEBUG ] ********************************************************************************
  134. [cell01][INFO ] monitor: mon.cell01 is running
  135. [cell01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell01.asok mon_status
  136. [ceph_deploy.mon][DEBUG ] detecting platform for host cell02 ...
  137. [cell02][DEBUG ] connection detected need for sudo
  138. [cell02][DEBUG ] connected to host: cell02 <<------部署cell02的监控器
  139. [cell02][DEBUG ] detect platform information from remote host
  140. [cell02][DEBUG ] detect machine type
  141. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  142. [cell02][DEBUG ] determining if provided host has same hostname in remote
  143. [cell02][DEBUG ] get remote short hostname
  144. [cell02][DEBUG ] deploying mon to cell02
  145. [cell02][DEBUG ] get remote short hostname
  146. [cell02][DEBUG ] remote hostname: cell02
  147. [cell02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  148. [cell02][DEBUG ] create the mon path if it does not exist
  149. [cell02][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell02/done
  150. [cell02][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-cell02/done
  151. [cell02][INFO ] creating tmp path: /var/lib/ceph/tmp
  152. [cell02][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-cell02.mon.keyring
  153. [cell02][DEBUG ] create the monitor keyring file
  154. [cell02][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i cell02 --keyring /var/lib/ceph/tmp/ceph-cell02.mon.keyring <<-----在对应的目录下创建对应的mkfs
  155. [cell02][DEBUG ] ceph-mon: mon.noname-b 192.168.1.213:6789/0 is local, renaming to mon.cell02
  156. [cell02][DEBUG ] ceph-mon: set fsid to 9061096f-d9f9-4946-94f1-296ab5080a97
  157. [cell02][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-cell02 for mon.cell02 <<-----创建mon的fs，可查看对应的目录下包含的数据，主要包括一个db，一个keyring等
  158. [cell02][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-cell02.mon.keyring
  159. [cell02][DEBUG ] create a done file to avoid re-doing the mon deployment
  160. [cell02][DEBUG ] create the init path if it does not exist
  161. [cell02][DEBUG ] locating the `service` executable...
  162. [cell02][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell02
  163. [cell02][DEBUG ] === mon.cell02 ===
  164. [cell02][DEBUG ] Starting Ceph mon.cell02 on cell02...
  165. [cell02][DEBUG ] Starting ceph-create-keys on cell02...
  166. [cell02][WARNIN] No data was received after 7 seconds, disconnecting...
  167. [cell02][INFO ] Running command: sudo chkconfig ceph on
  168. [cell02][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
  169. [cell02][DEBUG ] ********************************************************************************
  170. [cell02][DEBUG ] status for monitor: mon.cell02
  171. [cell02][DEBUG ] {
  172. [cell02][DEBUG ] "election_epoch": 1,
  173. [cell02][DEBUG ] "extra_probe_peers": [
  174. [cell02][DEBUG ] "192.168.1.212:6789/0",
  175. [cell02][DEBUG ] "192.168.1.214:6789/0"
  176. [cell02][DEBUG ] ],
  177. [cell02][DEBUG ] "monmap": {
  178. [cell02][DEBUG ] "created": "0.000000",
  179. [cell02][DEBUG ] "epoch": 0,
  180. [cell02][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  181. [cell02][DEBUG ] "modified": "0.000000",
  182. [cell02][DEBUG ] "mons": [
  183. [cell02][DEBUG ] {
  184. [cell02][DEBUG ] "addr": "192.168.1.212:6789/0",
  185. [cell02][DEBUG ] "name": "cell01",
  186. [cell02][DEBUG ] "rank": 0
  187. [cell02][DEBUG ] },
  188. [cell02][DEBUG ] {
  189. [cell02][DEBUG ] "addr": "192.168.1.213:6789/0",
  190. [cell02][DEBUG ] "name": "cell02",
  191. [cell02][DEBUG ] "rank": 1
  192. [cell02][DEBUG ] },
  193. [cell02][DEBUG ] {
  194. [cell02][DEBUG ] "addr": "0.0.0.0:0/2",
  195. [cell02][DEBUG ] "name": "cell03",
  196. [cell02][DEBUG ] "rank": 2
  197. [cell02][DEBUG ] }
  198. [cell02][DEBUG ] ]
  199. [cell02][DEBUG ] },
  200. [cell02][DEBUG ] "name": "cell02",
  201. [cell02][DEBUG ] "outside_quorum": [],
  202. [cell02][DEBUG ] "quorum": [],
  203. [cell02][DEBUG ] "rank": 1,
  204. [cell02][DEBUG ] "state": "electing",
  205. [cell02][DEBUG ] "sync_provider": []
  206. [cell02][DEBUG ] }
  207. [cell02][DEBUG ] ********************************************************************************
  208. [cell02][INFO ] monitor: mon.cell02 is running
  209. [cell02][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
  210. [ceph_deploy.mon][DEBUG ] detecting platform for host cell03 ...
  211. [cell03][DEBUG ] connection detected need for sudo
  212. [cell03][DEBUG ] connected to host: cell03 <<------部署cell03上的监控器信息
  213. [cell03][DEBUG ] detect platform information from remote host
  214. [cell03][DEBUG ] detect machine type
  215. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  216. [cell03][DEBUG ] determining if provided host has same hostname in remote
  217. [cell03][DEBUG ] get remote short hostname
  218. [cell03][DEBUG ] deploying mon to cell03
  219. [cell03][DEBUG ] get remote short hostname
  220. [cell03][DEBUG ] remote hostname: cell03
  221. [cell03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  222. [cell03][DEBUG ] create the mon path if it does not exist
  223. [cell03][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell03/done
  224. [cell03][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-cell03/done
  225. [cell03][INFO ] creating tmp path: /var/lib/ceph/tmp
  226. [cell03][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-cell03.mon.keyring
  227. [cell03][DEBUG ] create the monitor keyring file
  228. [cell03][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i cell03 --keyring /var/lib/ceph/tmp/ceph-cell03.mon.keyring <<-----mkfs
  229. [cell03][DEBUG ] ceph-mon: mon.noname-c 192.168.1.214:6789/0 is local, renaming to mon.cell03
  230. [cell03][DEBUG ] ceph-mon: set fsid to 9061096f-d9f9-4946-94f1-296ab5080a97
  231. [cell03][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-cell03 for mon.cell03
  232. [cell03][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-cell03.mon.keyring
  233. [cell03][DEBUG ] create a done file to avoid re-doing the mon deployment
  234. [cell03][DEBUG ] create the init path if it does not exist
  235. [cell03][DEBUG ] locating the `service` executable...
  236. [cell03][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell03
  237. [cell03][DEBUG ] === mon.cell03 ===
  238. [cell03][DEBUG ] Starting Ceph mon.cell03 on cell03...
  239. [cell03][DEBUG ] Starting ceph-create-keys on cell03...
  240. [cell03][WARNIN] No data was received after 7 seconds, disconnecting...
  241. [cell03][INFO ] Running command: sudo chkconfig ceph on
  242. [cell03][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
  243. [cell03][DEBUG ] ********************************************************************************
  244. [cell03][DEBUG ] status for monitor: mon.cell03
  245. [cell03][DEBUG ] {
  246. [cell03][DEBUG ] "election_epoch": 5,
  247. [cell03][DEBUG ] "extra_probe_peers": [
  248. [cell03][DEBUG ] "192.168.1.212:6789/0",
  249. [cell03][DEBUG ] "192.168.1.213:6789/0"
  250. [cell03][DEBUG ] ],
  251. [cell03][DEBUG ] "monmap": {
  252. [cell03][DEBUG ] "created": "0.000000",
  253. [cell03][DEBUG ] "epoch": 1,
  254. [cell03][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  255. [cell03][DEBUG ] "modified": "0.000000",
  256. [cell03][DEBUG ] "mons": [
  257. [cell03][DEBUG ] {
  258. [cell03][DEBUG ] "addr": "192.168.1.212:6789/0",
  259. [cell03][DEBUG ] "name": "cell01",
  260. [cell03][DEBUG ] "rank": 0
  261. [cell03][DEBUG ] },
  262. [cell03][DEBUG ] {
  263. [cell03][DEBUG ] "addr": "192.168.1.213:6789/0",
  264. [cell03][DEBUG ] "name": "cell02",
  265. [cell03][DEBUG ] "rank": 1
  266. [cell03][DEBUG ] },
  267. [cell03][DEBUG ] {
  268. [cell03][DEBUG ] "addr": "192.168.1.214:6789/0",
  269. [cell03][DEBUG ] "name": "cell03",
  270. [cell03][DEBUG ] "rank": 2
  271. [cell03][DEBUG ] }
  272. [cell03][DEBUG ] ]
  273. [cell03][DEBUG ] },
  274. [cell03][DEBUG ] "name": "cell03",
  275. [cell03][DEBUG ] "outside_quorum": [],
  276. [cell03][DEBUG ] "quorum": [],
  277. [cell03][DEBUG ] "rank": 2,
  278. [cell03][DEBUG ] "state": "electing",
  279. [cell03][DEBUG ] "sync_provider": []
  280. [cell03][DEBUG ] }
  281. [cell03][DEBUG ] ********************************************************************************
  282. [cell03][INFO ] monitor: mon.cell03 is running
  283. [cell03][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
  284. Error in sys.exitfunc:
  285. [ceph@cell01 my-cluster]$ ll
  286. total 24
  287. -rw-rw-r-- 1 ceph ceph 276 Sep 1 17:09 ceph.conf
  288. -rw-rw-r-- 1 ceph ceph 15344 Sep 1 17:10 ceph.log
  289. -rw-rw-r-- 1 ceph ceph 73 Sep 1 17:09 ceph.mon.keyring
  290. [ceph@cell01 my-cluster]$ ceph-deploy mon create-initial <<----监控器的初始化，根据部署的配置文件进行监控器的初始化
  291. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  292. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy mon create-initial
  293. [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cell01 cell02 cell03
  294. [ceph_deploy.mon][DEBUG ] detecting platform for host cell01 ...
  295. [cell01][DEBUG ] connection detected need for sudo
  296. [cell01][DEBUG ] connected to host: cell01
  297. [cell01][DEBUG ] detect platform information from remote host
  298. [cell01][DEBUG ] detect machine type
  299. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  300. [cell01][DEBUG ] determining if provided host has same hostname in remote
  301. [cell01][DEBUG ] get remote short hostname
  302. [cell01][DEBUG ] deploying mon to cell01
  303. [cell01][DEBUG ] get remote short hostname
  304. [cell01][DEBUG ] remote hostname: cell01
  305. [cell01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  306. [cell01][DEBUG ] create the mon path if it does not exist
  307. [cell01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell01/done
  308. [cell01][DEBUG ] create a done file to avoid re-doing the mon deployment
  309. [cell01][DEBUG ] create the init path if it does not exist
  310. [cell01][DEBUG ] locating the `service` executable...
  311. [cell01][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell01
  312. [cell01][DEBUG ] === mon.cell01 ===
  313. [cell01][DEBUG ] Starting Ceph mon.cell01 on cell01...already running
  314. [cell01][INFO ] Running command: sudo chkconfig ceph on
  315. [cell01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell01.asok mon_status
  316. [cell01][DEBUG ] ********************************************************************************
  317. [cell01][DEBUG ] status for monitor: mon.cell01
  318. [cell01][DEBUG ] {
  319. [cell01][DEBUG ] "election_epoch": 8,
  320. [cell01][DEBUG ] "extra_probe_peers": [
  321. [cell01][DEBUG ] "192.168.1.213:6789/0",
  322. [cell01][DEBUG ] "192.168.1.214:6789/0"
  323. [cell01][DEBUG ] ],
  324. [cell01][DEBUG ] "monmap": {
  325. [cell01][DEBUG ] "created": "0.000000",
  326. [cell01][DEBUG ] "epoch": 1,
  327. [cell01][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  328. [cell01][DEBUG ] "modified": "0.000000",
  329. [cell01][DEBUG ] "mons": [
  330. [cell01][DEBUG ] {
  331. [cell01][DEBUG ] "addr": "192.168.1.212:6789/0",
  332. [cell01][DEBUG ] "name": "cell01",
  333. [cell01][DEBUG ] "rank": 0
  334. [cell01][DEBUG ] },
  335. [cell01][DEBUG ] {
  336. [cell01][DEBUG ] "addr": "192.168.1.213:6789/0",
  337. [cell01][DEBUG ] "name": "cell02",
  338. [cell01][DEBUG ] "rank": 1
  339. [cell01][DEBUG ] },
  340. [cell01][DEBUG ] {
  341. [cell01][DEBUG ] "addr": "192.168.1.214:6789/0",
  342. [cell01][DEBUG ] "name": "cell03",
  343. [cell01][DEBUG ] "rank": 2
  344. [cell01][DEBUG ] }
  345. [cell01][DEBUG ] ]
  346. [cell01][DEBUG ] },
  347. [cell01][DEBUG ] "name": "cell01",
  348. [cell01][DEBUG ] "outside_quorum": [],
  349. [cell01][DEBUG ] "quorum": [
  350. [cell01][DEBUG ] 0,
  351. [cell01][DEBUG ] 1,
  352. [cell01][DEBUG ] 2
  353. [cell01][DEBUG ] ],
  354. [cell01][DEBUG ] "rank": 0,
  355. [cell01][DEBUG ] "state": "leader",
  356. [cell01][DEBUG ] "sync_provider": []
  357. [cell01][DEBUG ] }
  358. [cell01][DEBUG ] ********************************************************************************
  359. [cell01][INFO ] monitor: mon.cell01 is running
  360. [cell01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell01.asok mon_status
  361. [ceph_deploy.mon][DEBUG ] detecting platform for host cell02 ...
  362. [cell02][DEBUG ] connection detected need for sudo
  363. [cell02][DEBUG ] connected to host: cell02
  364. [cell02][DEBUG ] detect platform information from remote host
  365. [cell02][DEBUG ] detect machine type
  366. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  367. [cell02][DEBUG ] determining if provided host has same hostname in remote
  368. [cell02][DEBUG ] get remote short hostname
  369. [cell02][DEBUG ] deploying mon to cell02
  370. [cell02][DEBUG ] get remote short hostname
  371. [cell02][DEBUG ] remote hostname: cell02
  372. [cell02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  373. [cell02][DEBUG ] create the mon path if it does not exist
  374. [cell02][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell02/done
  375. [cell02][DEBUG ] create a done file to avoid re-doing the mon deployment
  376. [cell02][DEBUG ] create the init path if it does not exist
  377. [cell02][DEBUG ] locating the `service` executable...
  378. [cell02][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell02
  379. [cell02][DEBUG ] === mon.cell02 ===
  380. [cell02][DEBUG ] Starting Ceph mon.cell02 on cell02...already running
  381. [cell02][INFO ] Running command: sudo chkconfig ceph on
  382. [cell02][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
  383. [cell02][DEBUG ] ********************************************************************************
  384. [cell02][DEBUG ] status for monitor: mon.cell02
  385. [cell02][DEBUG ] {
  386. [cell02][DEBUG ] "election_epoch": 8,
  387. [cell02][DEBUG ] "extra_probe_peers": [
  388. [cell02][DEBUG ] "192.168.1.212:6789/0",
  389. [cell02][DEBUG ] "192.168.1.214:6789/0"
  390. [cell02][DEBUG ] ],
  391. [cell02][DEBUG ] "monmap": {
  392. [cell02][DEBUG ] "created": "0.000000",
  393. [cell02][DEBUG ] "epoch": 1,
  394. [cell02][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  395. [cell02][DEBUG ] "modified": "0.000000",
  396. [cell02][DEBUG ] "mons": [
  397. [cell02][DEBUG ] {
  398. [cell02][DEBUG ] "addr": "192.168.1.212:6789/0",
  399. [cell02][DEBUG ] "name": "cell01",
  400. [cell02][DEBUG ] "rank": 0
  401. [cell02][DEBUG ] },
  402. [cell02][DEBUG ] {
  403. [cell02][DEBUG ] "addr": "192.168.1.213:6789/0",
  404. [cell02][DEBUG ] "name": "cell02",
  405. [cell02][DEBUG ] "rank": 1
  406. [cell02][DEBUG ] },
  407. [cell02][DEBUG ] {
  408. [cell02][DEBUG ] "addr": "192.168.1.214:6789/0",
  409. [cell02][DEBUG ] "name": "cell03",
  410. [cell02][DEBUG ] "rank": 2
  411. [cell02][DEBUG ] }
  412. [cell02][DEBUG ] ]
  413. [cell02][DEBUG ] },
  414. [cell02][DEBUG ] "name": "cell02",
  415. [cell02][DEBUG ] "outside_quorum": [],
  416. [cell02][DEBUG ] "quorum": [
  417. [cell02][DEBUG ] 0,
  418. [cell02][DEBUG ] 1,
  419. [cell02][DEBUG ] 2
  420. [cell02][DEBUG ] ],
  421. [cell02][DEBUG ] "rank": 1,
  422. [cell02][DEBUG ] "state": "peon",
  423. [cell02][DEBUG ] "sync_provider": []
  424. [cell02][DEBUG ] }
  425. [cell02][DEBUG ] ********************************************************************************
  426. [cell02][INFO ] monitor: mon.cell02 is running
  427. [cell02][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
  428. [ceph_deploy.mon][DEBUG ] detecting platform for host cell03 ...
  429. [cell03][DEBUG ] connection detected need for sudo
  430. [cell03][DEBUG ] connected to host: cell03
  431. [cell03][DEBUG ] detect platform information from remote host
  432. [cell03][DEBUG ] detect machine type
  433. [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
  434. [cell03][DEBUG ] determining if provided host has same hostname in remote
  435. [cell03][DEBUG ] get remote short hostname
  436. [cell03][DEBUG ] deploying mon to cell03
  437. [cell03][DEBUG ] get remote short hostname
  438. [cell03][DEBUG ] remote hostname: cell03
  439. [cell03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  440. [cell03][DEBUG ] create the mon path if it does not exist
  441. [cell03][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cell03/done
  442. [cell03][DEBUG ] create a done file to avoid re-doing the mon deployment
  443. [cell03][DEBUG ] create the init path if it does not exist
  444. [cell03][DEBUG ] locating the `service` executable...
  445. [cell03][INFO ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.cell03
  446. [cell03][DEBUG ] === mon.cell03 ===
  447. [cell03][DEBUG ] Starting Ceph mon.cell03 on cell03...already running
  448. [cell03][INFO ] Running command: sudo chkconfig ceph on
  449. [cell03][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
  450. [cell03][DEBUG ] ********************************************************************************
  451. [cell03][DEBUG ] status for monitor: mon.cell03
  452. [cell03][DEBUG ] {
  453. [cell03][DEBUG ] "election_epoch": 8,
  454. [cell03][DEBUG ] "extra_probe_peers": [
  455. [cell03][DEBUG ] "192.168.1.212:6789/0",
  456. [cell03][DEBUG ] "192.168.1.213:6789/0"
  457. [cell03][DEBUG ] ],
  458. [cell03][DEBUG ] "monmap": {
  459. [cell03][DEBUG ] "created": "0.000000",
  460. [cell03][DEBUG ] "epoch": 1,
  461. [cell03][DEBUG ] "fsid": "9061096f-d9f9-4946-94f1-296ab5080a97",
  462. [cell03][DEBUG ] "modified": "0.000000",
  463. [cell03][DEBUG ] "mons": [
  464. [cell03][DEBUG ] {
  465. [cell03][DEBUG ] "addr": "192.168.1.212:6789/0",
  466. [cell03][DEBUG ] "name": "cell01",
  467. [cell03][DEBUG ] "rank": 0
  468. [cell03][DEBUG ] },
  469. [cell03][DEBUG ] {
  470. [cell03][DEBUG ] "addr": "192.168.1.213:6789/0",
  471. [cell03][DEBUG ] "name": "cell02",
  472. [cell03][DEBUG ] "rank": 1
  473. [cell03][DEBUG ] },
  474. [cell03][DEBUG ] {
  475. [cell03][DEBUG ] "addr": "192.168.1.214:6789/0",
  476. [cell03][DEBUG ] "name": "cell03",
  477. [cell03][DEBUG ] "rank": 2
  478. [cell03][DEBUG ] }
  479. [cell03][DEBUG ] ]
  480. [cell03][DEBUG ] },
  481. [cell03][DEBUG ] "name": "cell03",
  482. [cell03][DEBUG ] "outside_quorum": [],
  483. [cell03][DEBUG ] "quorum": [
  484. [cell03][DEBUG ] 0,
  485. [cell03][DEBUG ] 1,
  486. [cell03][DEBUG ] 2
  487. [cell03][DEBUG ] ],
  488. [cell03][DEBUG ] "rank": 2,
  489. [cell03][DEBUG ] "state": "peon",
  490. [cell03][DEBUG ] "sync_provider": []
  491. [cell03][DEBUG ] }
  492. [cell03][DEBUG ] ********************************************************************************
  493. [cell03][INFO ] monitor: mon.cell03 is running
  494. [cell03][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
  495. [ceph_deploy.mon][INFO ] processing monitor mon.cell01
  496. [cell01][DEBUG ] connection detected need for sudo
  497. [cell01][DEBUG ] connected to host: cell01
  498. [cell01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell01.asok mon_status
  499. [ceph_deploy.mon][INFO ] mon.cell01 monitor has reached quorum!
  500. [ceph_deploy.mon][INFO ] processing monitor mon.cell02
  501. [cell02][DEBUG ] connection detected need for sudo
  502. [cell02][DEBUG ] connected to host: cell02
  503. [cell02][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
  504. [ceph_deploy.mon][INFO ] mon.cell02 monitor has reached quorum!
  505. [ceph_deploy.mon][INFO ] processing monitor mon.cell03
  506. [cell03][DEBUG ] connection detected need for sudo
  507. [cell03][DEBUG ] connected to host: cell03
  508. [cell03][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
  509. [ceph_deploy.mon][INFO ] mon.cell03 monitor has reached quorum!
  510. [ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum
  511. [ceph_deploy.mon][INFO ] Running gatherkeys...
  512. [ceph_deploy.gatherkeys][DEBUG ] Checking cell01 for /etc/ceph/ceph.client.admin.keyring
  513. [cell01][DEBUG ] connection detected need for sudo
  514. [cell01][DEBUG ] connected to host: cell01
  515. [cell01][DEBUG ] detect platform information from remote host
  516. [cell01][DEBUG ] detect machine type
  517. [cell01][DEBUG ] fetch remote file
  518. [ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from cell01.
  519. [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring //获取到了keyring
  520. [ceph_deploy.gatherkeys][DEBUG ] Checking cell01 for /var/lib/ceph/bootstrap-osd/ceph.keyring
  521. [cell01][DEBUG ] connection detected need for sudo
  522. [cell01][DEBUG ] connected to host: cell01
  523. [cell01][DEBUG ] detect platform information from remote host
  524. [cell01][DEBUG ] detect machine type
  525. [cell01][DEBUG ] fetch remote file
  526. [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from cell01.
  527. [ceph_deploy.gatherkeys][DEBUG ] Checking cell01 for /var/lib/ceph/bootstrap-mds/ceph.keyring
  528. [cell01][DEBUG ] connection detected need for sudo
  529. [cell01][DEBUG ] connected to host: cell01
  530. [cell01][DEBUG ] detect platform information from remote host
  531. [cell01][DEBUG ] detect machine type
  532. [cell01][DEBUG ] fetch remote file
  533. [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from cell01.
  534. Error in sys.exitfunc:
  535. [ceph@cell01 my-cluster]$ ll <<-----从所有的节点中获取的认证信息keyring
  536. total 48
  537. -rw-rw-r-- 1 ceph ceph 71 Sep 1 17:11 ceph.bootstrap-mds.keyring
  538. -rw-rw-r-- 1 ceph ceph 71 Sep 1 17:11 ceph.bootstrap-osd.keyring
  539. -rw-rw-r-- 1 ceph ceph 63 Sep 1 17:11 ceph.client.admin.keyring
  540. -rw-rw-r-- 1 ceph ceph 276 Sep 1 17:09 ceph.conf
  541. -rw-rw-r-- 1 ceph ceph 28047 Sep 1 17:11 ceph.log
  542. -rw-rw-r-- 1 ceph ceph 73 Sep 1 17:09 ceph.mon.keyring
  543. [ceph@cell01 my-cluster]$ ll /var/lib/ceph/
  544. bootstrap-mds/ bootstrap-osd/ mon/ tmp/
  545. [ceph@cell01 my-cluster]$ ll /var/lib/ceph/mon/ceph-cell01/ <<----mkfs的过程中生成的文件夹信息
  546. done keyring store.db/ sysvinit
  547. [ceph@cell01 my-cluster]$ ll /var/lib/ceph/mon/ceph-cell01/
  548. done keyring store.db/ sysvinit
  549. [ceph@cell01 my-cluster]$ sudo ceph daemon mon.`hostname` mon_status <<----查看当前的监控器的状态
  550. { "name": "cell01",
  551. "rank": 0,
  552. "state": "leader",
  553. "election_epoch": 6,
  554. "quorum": [
  555. 0,
  556. 1,
  557. 2],
  558. "outside_quorum": [],
  559. "extra_probe_peers": [
  560. "192.168.1.213:6789\/0",
  561. "192.168.1.214:6789\/0"],
  562. "sync_provider": [],
  563. "monmap": { "epoch": 2,
  564. "fsid": "32a0c6a4-7076-4c31-a625-a73480746d5e",
  565. "modified": "2015-09-02 16:01:58.239429",
  566. "created": "0.000000",
  567. "mons": [
  568. { "rank": 0,
  569. "name": "cell01",
  570. "addr": "192.168.1.212:6789\/0"},
  571. { "rank": 1,
  572. "name": "cell02",
  573. "addr": "192.168.1.213:6789\/0"},
  574. { "rank": 2,
  575. "name": "cell03",
  576. "addr": "192.168.1.214:6789\/0"}]}}
     完成监控器的部署之后，接下来进行OSD节点的部署。

OSD部署

点击(此处)折叠或打开

  1. [ceph@cell01 my-cluster]$ ceph-deploy --overwrite-conf osd prepare cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1 <<-----为对应的osd准备磁盘空间
  2. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  3. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy --overwrite-conf osd prepare cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1
  4. [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cell01:/dev/sdb1: cell02:/dev/sdb1: cell03:/dev/sdb1:
  5. [cell01][DEBUG ] connection detected need for sudo
  6. [cell01][DEBUG ] connected to host: cell01
  7. [cell01][DEBUG ] detect platform information from remote host
  8. [cell01][DEBUG ] detect machine type
  9. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  10. [ceph_deploy.osd][DEBUG ] Deploying osd to cell01
  11. [cell01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  12. [cell01][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
  13. [ceph_deploy.osd][DEBUG ] Preparing host cell01 disk /dev/sdb1 journal None activate False
  14. [cell01][INFO ] Running command: sudo ceph-disk -v prepare --fs-type xfs --cluster ceph -- /dev/sdb1
  15. [cell01][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=73105406 blks
  16. [cell01][DEBUG ] = sectsz=512 attr=2, projid32bit=0
  17. [cell01][DEBUG ] data = bsize=4096 blocks=292421623, imaxpct=5
  18. [cell01][DEBUG ] = sunit=0 swidth=0 blks
  19. [cell01][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
  20. [cell01][DEBUG ] log =internal log bsize=4096 blocks=142783, version=2
  21. [cell01][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
  22. [cell01][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
  23. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
  24. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
  25. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
  26. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  27. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  28. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
  29. [cell01][WARNIN] DEBUG:ceph-disk:OSD data device /dev/sdb1 is a partition
  30. [cell01][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/sdb1
  31. [cell01][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1 <<---格式化对应的磁盘，格式为xfs
  32. [cell01][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.VVjTnb with options noatime,inode64
  33. [cell01][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.VVjTnb <<---挂载格式化成功的磁盘
  34. [cell01][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.VVjTnb <<---在磁盘中创建对应的数据目录
  35. [cell01][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.VVjTnb
  36. [cell01][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.VVjTnb <<---卸载对应的磁盘
  37. [cell01][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb1
  38. [cell01][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
  39. [cell01][WARNIN] INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdb1 <<----激活对应的分区信息
  40. [cell01][WARNIN] last arg is not the whole disk
  41. [cell01][WARNIN] call: partx -opts device wholedisk
  42. [cell01][INFO ] checking OSD status...
  43. [cell01][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  44. [ceph_deploy.osd][DEBUG ] Host cell01 is now ready for osd use. <<---磁盘准备就绪
  45. [cell02][DEBUG ] connection detected need for sudo
  46. [cell02][DEBUG ] connected to host: cell02
  47. [cell02][DEBUG ] detect platform information from remote host
  48. [cell02][DEBUG ] detect machine type
  49. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  50. [ceph_deploy.osd][DEBUG ] Deploying osd to cell02
  51. [cell02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  52. [cell02][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
  53. [ceph_deploy.osd][DEBUG ] Preparing host cell02 disk /dev/sdb1 journal None activate False
  54. [cell02][INFO ] Running command: sudo ceph-disk -v prepare --fs-type xfs --cluster ceph -- /dev/sdb1
  55. [cell02][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=18310545 blks
  56. [cell02][DEBUG ] = sectsz=512 attr=2, projid32bit=0
  57. [cell02][DEBUG ] data = bsize=4096 blocks=73242179, imaxpct=25
  58. [cell02][DEBUG ] = sunit=0 swidth=0 blks
  59. [cell02][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
  60. [cell02][DEBUG ] log =internal log bsize=4096 blocks=35762, version=2
  61. [cell02][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
  62. [cell02][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
  63. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
  64. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
  65. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
  66. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  67. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  68. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
  69. [cell02][WARNIN] DEBUG:ceph-disk:OSD data device /dev/sdb1 is a partition
  70. [cell02][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/sdb1
  71. [cell02][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
  72. [cell02][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.iBaG75 with options noatime,inode64
  73. [cell02][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.iBaG75
  74. [cell02][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.iBaG75
  75. [cell02][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.iBaG75
  76. [cell02][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.iBaG75
  77. [cell02][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb1
  78. [cell02][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
  79. [cell02][WARNIN] INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdb1
  80. [cell02][WARNIN] last arg is not the whole disk
  81. [cell02][WARNIN] call: partx -opts device wholedisk
  82. [cell02][INFO ] checking OSD status...
  83. [cell02][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  84. [ceph_deploy.osd][DEBUG ] Host cell02 is now ready for osd use.
  85. [cell03][DEBUG ] connection detected need for sudo
  86. [cell03][DEBUG ] connected to host: cell03
  87. [cell03][DEBUG ] detect platform information from remote host
  88. [cell03][DEBUG ] detect machine type
  89. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  90. [ceph_deploy.osd][DEBUG ] Deploying osd to cell03
  91. [cell03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  92. [cell03][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
  93. [ceph_deploy.osd][DEBUG ] Preparing host cell03 disk /dev/sdb1 journal None activate False
  94. [cell03][INFO ] Running command: sudo ceph-disk -v prepare --fs-type xfs --cluster ceph -- /dev/sdb1
  95. [cell03][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=18276350 blks
  96. [cell03][DEBUG ] = sectsz=512 attr=2, projid32bit=0
  97. [cell03][DEBUG ] data = bsize=4096 blocks=73105399, imaxpct=25
  98. [cell03][DEBUG ] = sunit=0 swidth=0 blks
  99. [cell03][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
  100. [cell03][DEBUG ] log =internal log bsize=4096 blocks=35695, version=2
  101. [cell03][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
  102. [cell03][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
  103. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
  104. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
  105. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
  106. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  107. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  108. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
  109. [cell03][WARNIN] DEBUG:ceph-disk:OSD data device /dev/sdb1 is a partition
  110. [cell03][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/sdb1
  111. [cell03][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
  112. [cell03][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.IqJ2rs with options noatime,inode64
  113. [cell03][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.IqJ2rs
  114. [cell03][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.IqJ2rs
  115. [cell03][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.IqJ2rs
  116. [cell03][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.IqJ2rs
  117. [cell03][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb1
  118. [cell03][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
  119. [cell03][WARNIN] INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdb1
  120. [cell03][WARNIN] last arg is not the whole disk
  121. [cell03][WARNIN] call: partx -opts device wholedisk
  122. [cell03][INFO ] checking OSD status...
  123. [cell03][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  124. [ceph_deploy.osd][DEBUG ] Host cell03 is now ready for osd use.
  125. Error in sys.exitfunc:
  126. [ceph@cell01 my-cluster]$ sudo mkdir -p /var/lib/ceph/osd/ceph-0 <<-----创建对应的osd目录，同时其他节点也创建相应的osd目录
  127. [ceph@cell01 my-cluster]$ ssh cell02
  128. [ceph@cell02 ~]$ sudo mkdir -p /var/lib/ceph/osd/ceph-1
  129. [ceph@cell02 ~]$ exit
  130. [ceph@cell01 my-cluster]$ ssh cell03
  131. [ceph@cell03 ~]$ sudo mkdir -p /var/lib/ceph/osd/ceph-2
  132. [ceph@cell03 ~]$ exit
  134. [ceph@cell01 my-cluster]$ ceph-deploy osd activate cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1 <<----激活对应的osd磁盘
  135. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  136. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy osd activate cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1
  137. [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks cell01:/dev/sdb1: cell02:/dev/sdb1: cell03:/dev/sdb1:
  138. [cell01][DEBUG ] connection detected need for sudo
  139. [cell01][DEBUG ] connected to host: cell01
  140. [cell01][DEBUG ] detect platform information from remote host
  141. [cell01][DEBUG ] detect machine type
  142. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  143. [ceph_deploy.osd][DEBUG ] activating host cell01 disk /dev/sdb1
  144. [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
  145. [cell01][INFO ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /dev/sdb1
  146. [cell01][DEBUG ] === osd.0 ===
  147. [cell01][DEBUG ] Starting Ceph osd.0 on cell01...
  148. [cell01][DEBUG ] starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
  149. [cell01][WARNIN] INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sdb1 <<--获取uuid和类型
  150. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  151. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  152. [cell01][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.aMjvT5 with options noatime,inode64
  153. [cell01][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.aMjvT5 <<---挂载到一个临时的目录中
  154. [cell01][WARNIN] DEBUG:ceph-disk:Cluster uuid is 32a0c6a4-7076-4c31-a625-a73480746d5e
  155. [cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid <<---ceph-osd进行处理
  156. [cell01][WARNIN] DEBUG:ceph-disk:Cluster name is ceph
  157. [cell01][WARNIN] DEBUG:ceph-disk:OSD uuid is 333bf1d3-bb1d-4c57-b4b1-679dddbfdce8
  158. [cell01][WARNIN] DEBUG:ceph-disk:OSD id is 0
  159. [cell01][WARNIN] DEBUG:ceph-disk:Marking with init system sysvinit
  160. [cell01][WARNIN] DEBUG:ceph-disk:ceph osd.0 data dir is ready at /var/lib/ceph/tmp/mnt.aMjvT5
  161. [cell01][WARNIN] DEBUG:ceph-disk:Moving mount to final location...
  162. [cell01][WARNIN] INFO:ceph-disk:Running command: /bin/mount -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/osd/ceph-0 <<----挂载对应的磁盘到对应的目录中，因此可通过该目录查看其中的数据情况
  163. [cell01][WARNIN] INFO:ceph-disk:Running command: /bin/umount -l -- /var/lib/ceph/tmp/mnt.aMjvT5 <<---取消零时挂载
  164. [cell01][WARNIN] DEBUG:ceph-disk:Starting ceph osd.0...
  165. [cell01][WARNIN] INFO:ceph-disk:Running command: /sbin/service ceph --cluster ceph start osd.0 <<---启动ceph的服务器
  166. [cell01][WARNIN] libust[18436/18436]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  167. [cell01][WARNIN] create-or-move updating item name 'osd.0' weight 1.09 at location {host=cell01,root=default} to crush map <<---更新对应的映射中的权重
  168. [cell01][WARNIN] libust[18489/18489]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  169. [cell01][INFO ] checking OSD status...
  170. [cell01][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  171. [cell01][INFO ] Running command: sudo chkconfig ceph on
  172. [cell02][DEBUG ] connection detected need for sudo
  173. [cell02][DEBUG ] connected to host: cell02
  174. [cell02][DEBUG ] detect platform information from remote host
  175. [cell02][DEBUG ] detect machine type
  176. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  177. [ceph_deploy.osd][DEBUG ] activating host cell02 disk /dev/sdb1
  178. [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
  179. [cell02][INFO ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /dev/sdb1
  180. [cell02][DEBUG ] === osd.1 ===
  181. [cell02][DEBUG ] Starting Ceph osd.1 on cell02...
  182. [cell02][DEBUG ] starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
  183. [cell02][WARNIN] INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sdb1 <<---获取磁盘信息
  184. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  185. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  186. [cell02][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.4g84Gq with options noatime,inode64
  187. [cell02][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.4g84Gq <<----挂载到临时目录中
  188. [cell02][WARNIN] DEBUG:ceph-disk:Cluster uuid is 32a0c6a4-7076-4c31-a625-a73480746d5e
  189. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
  190. [cell02][WARNIN] DEBUG:ceph-disk:Cluster name is ceph
  191. [cell02][WARNIN] DEBUG:ceph-disk:OSD uuid is e4be0dd1-6c20-41ad-9dec-42467ba8c23a
  192. [cell02][WARNIN] DEBUG:ceph-disk:Allocating OSD id...
  193. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise e4be0dd1-6c20-41ad-9dec-42467ba8c23a <<---分配osd id
  194. [cell02][WARNIN] DEBUG:ceph-disk:OSD id is 1
  195. [cell02][WARNIN] DEBUG:ceph-disk:Initializing OSD...
  196. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/tmp/mnt.4g84Gq/activate.monmap <<---获取monmap
  197. [cell02][WARNIN] got monmap epoch 2
  198. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph --mkfs --mkkey -i 1 --monmap /var/lib/ceph/tmp/mnt.4g84Gq/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.4g84Gq --osd-journal /var/lib/ceph/tmp/mnt.4g84Gq/journal --osd-uuid e4be0dd1-6c20-41ad-9dec-42467ba8c23a --keyring /var/lib/ceph/tmp/mnt.4g84Gq/keyring <<-----进行mkfs的处理
  199. [cell02][WARNIN] 2015-09-02 16:12:15.841276 7f6b50da8800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
  200. [cell02][WARNIN] 2015-09-02 16:12:16.026779 7f6b50da8800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
  201. [cell02][WARNIN] 2015-09-02 16:12:16.027262 7f6b50da8800 -1 filestore(/var/lib/ceph/tmp/mnt.4g84Gq) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
  202. [cell02][WARNIN] 2015-09-02 16:12:16.248841 7f6b50da8800 -1 created object store /var/lib/ceph/tmp/mnt.4g84Gq journal /var/lib/ceph/tmp/mnt.4g84Gq/journal for osd.1 fsid 32a0c6a4-7076-4c31-a625-a73480746d5e
  203. [cell02][WARNIN] 2015-09-02 16:12:16.248909 7f6b50da8800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.4g84Gq/keyring: can't open /var/lib/ceph/tmp/mnt.4g84Gq/keyring: (2) No such file or directory
  204. [cell02][WARNIN] 2015-09-02 16:12:16.249063 7f6b50da8800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.4g84Gq/keyring
  205. [cell02][WARNIN] DEBUG:ceph-disk:Marking with init system sysvinit
  206. [cell02][WARNIN] DEBUG:ceph-disk:Authorizing OSD key...
  207. [cell02][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring auth add osd.1 -i /var/lib/ceph/tmp/mnt.4g84Gq/keyring osd allow * mon allow profile osd <<----添加认证的信息
  208. [cell02][WARNIN] added key for osd.1
  209. [cell02][WARNIN] DEBUG:ceph-disk:ceph osd.1 data dir is ready at /var/lib/ceph/tmp/mnt.4g84Gq
  210. [cell02][WARNIN] DEBUG:ceph-disk:Moving mount to final location...
  211. [cell02][WARNIN] INFO:ceph-disk:Running command: /bin/mount -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/osd/ceph-1 <<----挂载到最终的目录中，具体的数据可查看该目录
  212. [cell02][WARNIN] INFO:ceph-disk:Running command: /bin/umount -l -- /var/lib/ceph/tmp/mnt.4g84Gq <<---从零时挂载点卸载
  213. [cell02][WARNIN] DEBUG:ceph-disk:Starting ceph osd.1...
  214. [cell02][WARNIN] INFO:ceph-disk:Running command: /sbin/service ceph --cluster ceph start osd.1
  215. [cell02][WARNIN] libust[30705/30705]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  216. [cell02][WARNIN] create-or-move updating item name 'osd.1' weight 0.27 at location {host=cell02,root=default} to crush map <<----更新其中当前osd的权重
  217. [cell02][WARNIN] libust[30797/30797]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  218. [cell02][INFO ] checking OSD status...
  219. [cell02][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  220. [cell02][WARNIN] there is 1 OSD down
  221. [cell02][WARNIN] there is 1 OSD out
  222. [cell02][INFO ] Running command: sudo chkconfig ceph on
  223. [cell03][DEBUG ] connection detected need for sudo
  224. [cell03][DEBUG ] connected to host: cell03
  225. [cell03][DEBUG ] detect platform information from remote host
  226. [cell03][DEBUG ] detect machine type
  227. [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
  228. [ceph_deploy.osd][DEBUG ] activating host cell03 disk /dev/sdb1
  229. [ceph_deploy.osd][DEBUG ] will use init type: sysvinit
  230. [cell03][INFO ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /dev/sdb1
  231. [cell03][DEBUG ] === osd.2 ===
  232. [cell03][DEBUG ] Starting Ceph osd.2 on cell03...
  233. [cell03][DEBUG ] starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
  234. [cell03][WARNIN] INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sdb1
  235. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
  236. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
  237. [cell03][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.5n81s2 with options noatime,inode64
  238. [cell03][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.5n81s2
  239. [cell03][WARNIN] DEBUG:ceph-disk:Cluster uuid is 32a0c6a4-7076-4c31-a625-a73480746d5e
  240. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
  241. [cell03][WARNIN] DEBUG:ceph-disk:Cluster name is ceph
  242. [cell03][WARNIN] DEBUG:ceph-disk:OSD uuid is cd6ac8bc-5d7f-4963-aba5-43d2bf84127a
  243. [cell03][WARNIN] DEBUG:ceph-disk:Allocating OSD id...
  244. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise cd6ac8bc-5d7f-4963-aba5-43d2bf84127a
  245. [cell03][WARNIN] DEBUG:ceph-disk:OSD id is 2
  246. [cell03][WARNIN] DEBUG:ceph-disk:Initializing OSD...
  247. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/tmp/mnt.5n81s2/activate.monmap
  248. [cell03][WARNIN] got monmap epoch 2
  249. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph --mkfs --mkkey -i 2 --monmap /var/lib/ceph/tmp/mnt.5n81s2/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.5n81s2 --osd-journal /var/lib/ceph/tmp/mnt.5n81s2/journal --osd-uuid cd6ac8bc-5d7f-4963-aba5-43d2bf84127a --keyring /var/lib/ceph/tmp/mnt.5n81s2/keyring
  250. [cell03][WARNIN] 2015-09-02 16:13:00.015228 7f76a277a800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
  251. [cell03][WARNIN] 2015-09-02 16:13:00.021221 7f76a277a800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
  252. [cell03][WARNIN] 2015-09-02 16:13:00.021866 7f76a277a800 -1 filestore(/var/lib/ceph/tmp/mnt.5n81s2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
  253. [cell03][WARNIN] 2015-09-02 16:13:00.049203 7f76a277a800 -1 created object store /var/lib/ceph/tmp/mnt.5n81s2 journal /var/lib/ceph/tmp/mnt.5n81s2/journal for osd.2 fsid 32a0c6a4-7076-4c31-a625-a73480746d5e
  254. [cell03][WARNIN] 2015-09-02 16:13:00.049269 7f76a277a800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.5n81s2/keyring: can't open /var/lib/ceph/tmp/mnt.5n81s2/keyring: (2) No such file or directory
  255. [cell03][WARNIN] 2015-09-02 16:13:00.049424 7f76a277a800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.5n81s2/keyring
  256. [cell03][WARNIN] DEBUG:ceph-disk:Marking with init system sysvinit
  257. [cell03][WARNIN] DEBUG:ceph-disk:Authorizing OSD key...
  258. [cell03][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring auth add osd.2 -i /var/lib/ceph/tmp/mnt.5n81s2/keyring osd allow * mon allow profile osd
  259. [cell03][WARNIN] added key for osd.2
  260. [cell03][WARNIN] DEBUG:ceph-disk:ceph osd.2 data dir is ready at /var/lib/ceph/tmp/mnt.5n81s2
  261. [cell03][WARNIN] DEBUG:ceph-disk:Moving mount to final location...
  262. [cell03][WARNIN] INFO:ceph-disk:Running command: /bin/mount -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/osd/ceph-2
  263. [cell03][WARNIN] INFO:ceph-disk:Running command: /bin/umount -l -- /var/lib/ceph/tmp/mnt.5n81s2
  264. [cell03][WARNIN] DEBUG:ceph-disk:Starting ceph osd.2...
  265. [cell03][WARNIN] INFO:ceph-disk:Running command: /sbin/service ceph --cluster ceph start osd.2
  266. [cell03][WARNIN] libust[27410/27410]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  267. [cell03][WARNIN] create-or-move updating item name 'osd.2' weight 0.27 at location {host=cell03,root=default} to crush map
  268. [cell03][WARNIN] libust[27454/27454]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
  269. [cell03][INFO ] checking OSD status...
  270. [cell03][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json
  271. [cell03][WARNIN] there is 1 OSD down
  272. [cell03][WARNIN] there is 1 OSD out
  273. [cell03][INFO ] Running command: sudo chkconfig ceph on
  274. Error in sys.exitfunc
  276. [ceph@cell01 my-cluster]$ ceph-deploy admin ireadmin cell01 cell02 cell03 <<---采用Ceph-deploy工具复制对应的配置文件和管理钥匙到管理节点和所有的Ceph节点中
  277. [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
  278. [ceph_deploy.cli][INFO ] Invoked (1.5.19): /usr/bin/ceph-deploy admin ireadmin cell01 cell02 cell03
  279. [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ireadmin
  280. ceph@192.168.1.212's password:
  281. [ireadmin][DEBUG ] connection detected need for sudo
  282. ceph@192.168.1.212's password:
  283. [ireadmin][DEBUG ] connected to host: ireadmin
  284. [ireadmin][DEBUG ] detect platform information from remote host
  285. [ireadmin][DEBUG ] detect machine type
  286. [ireadmin][DEBUG ] get remote short hostname
  287. [ireadmin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  288. [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to cell01
  289. [cell01][DEBUG ] connection detected need for sudo
  290. [cell01][DEBUG ] connected to host: cell01
  291. [cell01][DEBUG ] detect platform information from remote host
  292. [cell01][DEBUG ] detect machine type
  293. [cell01][DEBUG ] get remote short hostname
  294. [cell01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf <<----写集群信息到配置文件中
  295. [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to cell02 <<----将管理的秘钥配置保存到对应的设备中
  296. [cell02][DEBUG ] connection detected need for sudo
  297. [cell02][DEBUG ] connected to host: cell02
  298. [cell02][DEBUG ] detect platform information from remote host
  299. [cell02][DEBUG ] detect machine type
  300. [cell02][DEBUG ] get remote short hostname
  301. [cell02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  302. [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to cell03
  303. [cell03][DEBUG ] connection detected need for sudo
  304. [cell03][DEBUG ] connected to host: cell03
  305. [cell03][DEBUG ] detect platform information from remote host
  306. [cell03][DEBUG ] detect machine type
  307. [cell03][DEBUG ] get remote short hostname
  308. [cell03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
  309. Error in sys.exitfunc:
  310. [ceph@cell01 my-cluster]$ chmod +r /etc/ceph/ceph.client.admin.keyring <<----修改客户端管理秘钥的权限
  311. chmod: changing permissions of `/etc/ceph/ceph.client.admin.keyring': Operation not permitted
  312. [ceph@cell01 my-cluster]$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring
  313. [ceph@cell01 my-cluster]$ ll
  314. total 88
  315. -rw-rw-r-- 1 ceph ceph 71 Sep 2 16:02 ceph.bootstrap-mds.keyring
  316. -rw-rw-r-- 1 ceph ceph 71 Sep 2 16:02 ceph.bootstrap-osd.keyring
  317. -rw-rw-r-- 1 ceph ceph 63 Sep 2 16:02 ceph.client.admin.keyring
  318. -rw-rw-r-- 1 ceph ceph 302 Sep 2 16:01 ceph.conf
  319. -rw-rw-r-- 1 ceph ceph 61825 Sep 2 16:26 ceph.log
  320. -rw-rw-r-- 1 ceph ceph 73 Sep 2 16:00 ceph.mon.keyring
  321. [ceph@cell01 my-cluster]$ ceph -s <<-----查看当前集群的状态
  322. cluster 32a0c6a4-7076-4c31-a625-a73480746d5e
  323. health HEALTH_WARN clock skew detected on mon.cell02, mon.cell03 <<----说明当前存在时间偏移，未设置时间同步导致
  324. monmap e2: 3 mons at {cell01=192.168.1.212:6789/0,cell02=192.168.1.213:6789/0,cell03=192.168.1.214:6789/0}, election epoch 6, quorum 0,1,2 cell01,cell02,cell03
  325. osdmap e14: 3 osds: 3 up, 3 in
  326. pgmap v29: 64 pgs, 1 pools, 0 bytes data, 0 objects
  327. 15459 MB used, 1657 GB / 1672 GB avail
  328. 64 active+clean
  329. [ceph@cell01 my-cluster]$ sudo ceph -s
  330. cluster 32a0c6a4-7076-4c31-a625-a73480746d5e
  331. health HEALTH_WARN clock skew detected on mon.cell02, mon.cell03
  332. monmap e2: 3 mons at {cell01=192.168.1.212:6789/0,cell02=192.168.1.213:6789/0,cell03=192.168.1.214:6789/0}, election epoch 6, quorum 0,1,2 cell01,cell02,cell03
  333. osdmap e14: 3 osds: 3 up, 3 in
  334. pgmap v29: 64 pgs, 1 pools, 0 bytes data, 0 objects
  335. 15459 MB used, 1657 GB / 1672 GB avail
  336. 64 active+clean
     在创建块设备之前先对osd进行分析，由上述的日志可知，在osd部署成功之后，实际上将对应的osd挂载到了/var/lib/ceph/osd/osd-xx/目录下，因此关于osd的数据信息，可查看该目录。

点击(此处)折叠或打开

  1. [ceph@cell01 my-cluster]$ ll /var/lib/ceph/osd/ceph-0/
  2. total 5242920
  3. -rw-r--r-- 1 root root 490 Sep 2 16:08 activate.monmap <<----monmap的信息
  4. -rw-r--r-- 1 root root 3 Sep 2 16:08 active
  5. -rw-r--r-- 1 root root 37 Sep 2 16:06 ceph_fsid <<----集群的fsid
  6. drwxr-xr-x 69 root root 1080 Sep 2 16:28 current <<----当前的数据目录，有效的数据保存在其中
  7. -rw-r--r-- 1 root root 37 Sep 2 16:06 fsid <<----当前磁盘的fsid
  8. -rw-r--r-- 1 root root 5368709120 Sep 6 09:50 journal <<----日志信息，在部署中为了提高系统性能通常将日志写入到其他的磁盘中
  9. -rw------- 1 root root 56 Sep 2 16:08 keyring <<-----认证信息
  10. -rw-r--r-- 1 root root 21 Sep 2 16:06 magic
  11. -rw-r--r-- 1 root root 6 Sep 2 16:08 ready
  12. -rw-r--r-- 1 root root 4 Sep 2 16:08 store_version
  13. -rw-r--r-- 1 root root 53 Sep 2 16:08 superblock
  14. -rw-r--r-- 1 root root 0 Sep 2 16:11 sysvinit
  15. -rw-r--r-- 1 root root 2 Sep 2 16:08 whoami
  16. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/c
  17. ceph_fsid current/
  18. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/ceph_fsid
  19. 32a0c6a4-7076-4c31-a625-a73480746d5e
  20. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/fsid
  21. 333bf1d3-bb1d-4c57-b4b1-679dddbfdce8
  22. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/magic
  23. ceph osd volume v026
  24. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/ready
  25. ready
  26. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/whoami
  27. 0
  28. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/store_version
  29. [ceph@cell01 my-cluster]$
  30. [ceph@cell01 my-cluster]$
  31. [ceph@cell01 my-cluster]$ sudo cat /var/lib/ceph/osd/ceph-0/active
  32. ok
  33. [ceph@cell01 my-cluster]$

块设备的创建

点击(此处)折叠或打开

  1. [ceph@cell01 my-cluster]$ rados mkpool xxx <<----创建xxx的池
  2. successfully created pool xxx
  3. [ceph@cell01 my-cluster]$ rados df <<----查看状态
  4. rados ls xxxpool name category KB objects clones degraded unfound rd rd KB wr wr KB
  5. xxx - 0 0 0 0 0 0 0 0 0
  6. rbd - 0 0 0 0 0 0 0 0 0
  7. total used 15831216 0
  8. total avail 1738388628
  9. total space 1754219844
  10. [ceph@cell01 my-cluster]$ rados ls xxx
  11. pool name was not specified
  12. [ceph@cell01 my-cluster]$ ceph osd lspools <<----查看当前的池信息
  13. 0 rbd,1 xxx,
  14. [ceph@cell01 my-cluster]
  15. [ceph@cell01 my-cluster]$ rbd -p xxx create node01 --size 124000 <<----创建块设备node01，124000M
  16. [ceph@cell01 my-cluster]$ rbd -p xxx create node02 --size 124000
  17. [ceph@cell01 my-cluster]$ rbd -p xxx create node03 --size 124000
  18. [ceph@cell01 my-cluster]$ rbd -p xxx create node04 --size 124000
  19. [ceph@cell01 my-cluster]$ rbd -p xxx create node05 --size 124000
  20. [ceph@cell01 my-cluster]$ rbd ls xxx <<-----查看当前池中的块设备信息
  21. node01
  22. node02
  23. node03
  24. node04
  25. node05
  26. [ceph@cell01 my-cluster]$ rbd info xxx/node01 <<----查看具体的块设备信息
  27. rbd image 'node01':
  28. size 121 GB in 31000 objects
  29. order 22 (4096 kB objects)
  30. block_name_prefix: rb.0.1057.74b0dc51
  31. format: 1
  32. [ceph@cell01 my-cluster]$ rbd info node01
  33. 2015-09-06 16:54:03.237243 7fe04f93a7e0 -1 librbd::ImageCtx: error finding header: (2) No such file or directory
  34. rbd: error opening image node01: (2) No such file or directory
  35. [ceph@cell01 my-cluster]$ rbd info xxx/node01
  36. rbd image 'node01':
  37. size 121 GB in 31000 objects <<---块的大小，包含的对象数
  38. order 22 (4096 kB objects) <<---对象的大小
  39. block_name_prefix: rb.0.1057.74b0dc51 <<----对象的前缀
  40. format: 1 <<---格式，1为旧格式
  41. [ceph@cell01 my-cluster]$ rbd info xxx/node02
  42. rbd image 'node02':
  43. size 121 GB in 31000 objects
  44. order 22 (4096 kB objects)
  45. block_name_prefix: rb.0.105a.74b0dc51
  46. format: 1
  47. [ceph@cell01 my-cluster]$ rbd info xxx/node03
  48. rbd image 'node03':
  49. size 121 GB in 31000 objects
  50. order 22 (4096 kB objects)
  51. block_name_prefix: rb.0.109d.74b0dc51
  52. format: 1
  53. [ceph@cell01 my-cluster]$ rbd info xxx/node04
  54. rbd image 'node04':
  55. size 121 GB in 31000 objects
  56. order 22 (4096 kB objects)
  57. block_name_prefix: rb.0.105d.2ae8944a
  58. format: 1
  59. [ceph@cell01 my-cluster]$ rbd info xxx/node05
  60. rbd image 'node05':
  61. size 121 GB in 31000 objects
  62. order 22 (4096 kB objects)
  63. block_name_prefix: rb.0.10ce.74b0dc51
  64. format: 1

问题解决

     (1) 初始化过程中的监控器未进入仲裁，导致管理节点的部署目录中无法生成秘钥，通常会提示某个节点无法进入仲裁。如下所示
[cell02][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
[ceph_deploy.mon][WARNIN] mon.cell02 monitor is not yet in quorum, tries left: 2
[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying
[cell02][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell02.asok mon_status
[ceph_deploy.mon][WARNIN] mon.cell02 monitor is not yet in quorum, tries left: 1
[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][INFO  ] processing monitor mon.cell03
[cell03][DEBUG ] connection detected need for sudo
[cell03][DEBUG ] connected to host: cell03 
[cell03][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.cell03.asok mon_status
[ceph_deploy.mon][INFO  ] mon.cell03 monitor has reached quorum!
[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
[ceph_deploy.mon][ERROR ] cell02
       出现这种问题通常是对应的节点中存在原来的配置，导致新部署过程中无法生成认证秘钥。此时遍历待部署的所有节点将/etc/ceph，和/var/lib/ceph下的目录清除掉，然后再部署，通常就能解决。

      (2) 在部署OSD的过程中出现 OSError: [Errno 2] No such file or directory: '/var/lib/ceph/osd/ceph-0'这类的错误，这种问题的解决方法是手动创建对应的目录，然后再进行激活。这部分应该在部署时自动添加更加方便，暂不清楚原因(ceph-disk)。
      (3) 通常是采用完整的磁盘作为osd设备，但有时候会采用磁盘的某个分区作为osd设备，此时这种部署可能会出现如下的问题:
[ceph@cell01 my-cluster]$ ceph-deploy osd activate cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.19): /usr/bin/ceph-deploy osd activate cell01:/dev/sdb1 cell02:/dev/sdb1 cell03:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks cell01:/dev/sdb1: cell02:/dev/sdb1: cell03:/dev/sdb1:
[cell01][DEBUG ] connection detected need for sudo
[cell01][DEBUG ] connected to host: cell01 
[cell01][DEBUG ] detect platform information from remote host
[cell01][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] activating host cell01 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[cell01][INFO  ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /dev/sdb1
[cell01][WARNIN] INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- /dev/sdb1
[cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[cell01][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.rRQkAk with options noatime,inode64
[cell01][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.rRQkAk
[cell01][WARNIN] DEBUG:ceph-disk:Cluster uuid is 9061096f-d9f9-4946-94f1-296ab5080a97
[cell01][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[cell01][WARNIN] ERROR:ceph-disk:Failed to activate
[cell01][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.rRQkAk
[cell01][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.rRQkAk
[cell01][WARNIN] ceph-disk: Error: No cluster conf found in /etc/ceph with fsid 9061096f-d9f9-4946-94f1-296ab5080a97
[cell01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk -v activate --mark-init sysvinit --mount /dev/sdb1
       在部署中我规避了该问题，采用单个磁盘作为osd，如果采用分区作为osd设备，可参考。

       (4) 由于各个节点之间未进行时间的同步，导致监控器之间的时间差较大，在查看过程中可能会出现如下的问题：
[ceph@cell01 my-cluster]$ ceph health
HEALTH_WARN clock skew detected on mon.cell02, mon.cell03
此时采用ntp进行节点间时间的同步，一段时间后对应的ceph集群就会进入健康的状态
[ceph@cell01 my-cluster]$ ceph -s
    cluster 32a0c6a4-7076-4c31-a625-a73480746d5e
     health HEALTH_OK
     monmap e2: 3 mons at {cell01=192.168.1.212:6789/0,cell02=192.168.1.213:6789/0,cell03=192.168.1.214:6789/0}, election epoch 10, quorum 0,1,2 cell01,cell02,cell03
     osdmap e16: 3 osds: 3 up, 3 in
      pgmap v244: 72 pgs, 2 pools, 8 bytes data, 1 objects
            15460 MB used, 1657 GB / 1672 GB avail
                  72 active+clean
[ceph@cell01 my-cluster]$ ceph health
HEALTH_OK
[ceph@cell01 my-cluster]$



阅读(3379) | 评论(0) | 转发(0) |
0

上一篇：ubuntu16.04 ruby开发环境

下一篇：docker swarm 集群

给主人留下些什么吧！~~
关于我们 | 关于IT168 | 联系方式 | 广告合作 | 法律声明 | 免费注册

Copyright 2001-2010 ChinaUnix.net All Rights Reserved 北京皓辰网域网络信息技术有限公司. 版权所有

感谢所有关心和支持过ChinaUnix的朋友们

16024965号-6