Ceph Storage Cluster(CEPH存储集群) Configuration配置

pre.cjk { font-family: "Nimbus Mono L",monospace; }h2.western { font-family: "Liberation Sans",sans-serif; font-size: 16pt; }h2.cjk { font-family: "思源黑体 CN Regular"; font-size: 16pt; }h2.ctl { font-family: "Lohit Devanagari"; font-size: 16pt; }p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }tt.cjk { font-family: "Nimbus Mono L",monospace; }

Ceph Storage Cluster(CEPH存储集群) Configuration配置

Ceph 能够运行一个包含成千上万个OSD的集群环境。一个最小系统将至少包含2个OSD作数据主从复制。为了配置OSD集群，必须在配置文件中提供配置。Ceph 包含许多默认设置，这些设置都可以在配置文件中覆盖。另外，还可以使用命令行工具实时修改这些配置。

Ceph启动3个进程：ceph-mon 、ceph-osd 、ceph-mds（只供cephfs）

每一个程序，进程，工具都加载主机配置文件。一个程序拥有多个进程信息，一个进程只包含它自己的信息。

硬盘与文件系统推荐

老内核(<2.6.33), 如果日志记录在一个raw设备上禁用写缓存。新内核刚不需要设置可以工作良好。

使用hdparm禁用硬盘写缓存

sudo hdparm -W 0 /dev/hda 0

在生产环境中，推荐运行OSD守护进程的服务器操作系统与数据有单独的磁盘。如果条件不允许，也要为数据盘创建单独的分区。

文件系统推荐

Ceph OSD Daemons 严重信赖底层文件系统类型的可靠性与性能。

推荐生产部署使用XFS。测试、开发与其它不重要的部署推荐使用btrfs。我们坚信btrfs拥有合适的特性在相当长的时间服务ceph，但XFS与EXT4提供了当今最好的稳定性部署。

Filesystems

Ceph OSD Daemons rely heavily upon the stability and performance of the underlying filesystem.

Note

We currently recommend XFS for production deployments. We recommend btrfs for testing, development, and any non-critical deployments. We believe that btrfs has the correct feature set and roadmap to serve Ceph in the long-term, but XFS and ext4 provide the necessary stability for today’s deployments. btrfs development is proceeding rapidly: users should be comfortable installing the latest released upstream kernels and be able to track development activity for critical bug fixes.

Ceph OSD Daemons depend on the Extended Attributes (XATTRs) of the underlying file system for various forms of internal object state and metadata. The underlying filesystem must provide sufficient capacity for XATTRs. btrfs does not bound the total xattr metadata stored with a file. XFS has a relatively large limit (64 KB) that most deployments won’t encounter, but the ext4 is too small to be usable.

You should always add the following line to the [osd] section of your ceph.conf file for ext4 filesystems; you can optionally use it for btrfs and XFS.:

filestore xattr use omap = true

Filesystem Background Info

The XFS, btrfs and ext4 file systems provide numerous advantages in highly scaled data storage environments when to ext3.

XFS, btrfs and ext4 are , which means that they are more robust when recovering from crashes, power outages, etc. These filesystems journal all of the changes they will make before performing writes.

XFS was developed for Silicon Graphics, and is a mature and stable filesystem. By contrast, btrfs is a relatively new file system that aims to address the long-standing wishes of system administrators working with large scale data storage environments. btrfs has some unique features and advantages compared to other Linux filesystems.

btrfs is a filesystem. It supports file creation timestamps and checksums that verify metadata integrity, so it can detect bad copies of data and fix them with the good copies. The copy-on-write capability means that btrfs can support snapshots that are writable. btrfs supports transparent compression and other features.

btrfs also incorporates multi-device management into the file system, which enables you to support heterogeneous disk storage infrastructure, data allocation policies. The community also aims to provide fsck, deduplication, and data encryption support in the future. This compelling list of features makes btrfs the ideal choice for Ceph clusters.

p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }tt.cjk { font-family: "Nimbus Mono L",monospace; }

当启动ceph服务时，初始化程序启动一系列的后台进程。一个存储集群启动2种类型的监听进程：

• (ceph-mon)

• (ceph-osd)

Ceph Storage Clusters支持 必须有一个 (ceph-mds). 支持 必须启动 Ceph Gateway daemons (radosgw). 为了便利性，每个监听进程都有一系列的默认值，这些值可以在配置文件中进行覆盖。
th p { margin-bottom: 0cm; }td p { margin-bottom: 0cm; }pre.cjk { font-family: "Nimbus Mono L",monospace; }h2.western { font-family: "Liberation Sans",sans-serif; font-size: 16pt; }h2.cjk { font-family: "思源黑体 CN Regular"; font-size: 16pt; }h2.ctl { font-family: "Lohit Devanagari"; font-size: 16pt; }p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }tt.cjk { font-family: "Nimbus Mono L",monospace; }

配置文件

当启动Ceph Storage Cluster后，每个监听进程都会去查找配置文件中所提供的设置信息，默认是/etc/ceph/ceph.conf。如果是手动部署，你必需自行创建此配置文件。如果使用自动化工具部署，程序会为你自动创建配置文件，类似ceph-deploy，chef等工具。你可以参考这些信息。Ceph的配置文件中可以定义的内容有：

• Cluster Identity集群标识

• Authentication settings认证配置

• Cluster membership 集群成员

• Host names 主机名

• Host addresses 主机地址

• Paths to keyrings 认证KEY信息的路径

• Paths to journals 日志信息的路径

• Paths to data 数据信息的路径

• Other runtime options 其它运行选项

默认的配置文件查找顺序是：

1. $CEPH_CONF (i.e., the path following the $CEPH_CONF environment variable) 变量定义

2. -c path/path (i.e., the -c command line argument) 命令行参数

3. /etc/ceph/ceph.conf 默认配置文件路径

4. ~/.ceph/config 家目录下的配置文件

5. ./ceph.conf (i.e., in the current working directory) 当前工作目录下的配置文件。

Ceph配置文件使用的是ini风格的语法。可以使用＃号或分号添加注释。 For example:

# <--A number (#) sign precedes a comment. ; A comment may be anything. # Comments always follow a semi-colon (;) or a pound (#) on each line. # The end of the line terminates a comment. # We recommend that you provide comments in your configuration file(s).

配置分段

配置文件可以配置集群中所有ceph监听进程或者所有ceph的特殊设置 。配置那些监听进程的话，一些设置标记必需包含在配置文件中：

[global]

Description:	Settings under [global] affect all daemons in a Ceph Storage Cluster.
Example:	auth supported = cephx

[osd]

Description:	Settings under [osd] affect all ceph-osd daemons in the Ceph Storage Cluster, and override the same setting in [global].
Example:	osd journal size = 1000

[mon]

Description:	Settings under [mon] affect all ceph-mon daemons in the Ceph Storage Cluster, and override the same setting in [global].
Example:	mon addr = 10.0.0.101:6789

[mds]

Description:	Settings under [mds] affect all ceph-mds daemons in the Ceph Storage Cluster, and override the same setting in [global].
Example:	host = myserver01

[client]

Description:	Settings under [client] affect all Ceph Clients (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices, etc.).
Example:	log file = /var/log/ceph/radosgw.log

Global settings affect all instances of all daemon in the Ceph Storage Cluster. Use the [global] setting for values that are common for all daemons in the Ceph Storage Cluster. You can override each [global] setting by:

1. Changing the setting in a particular process type (e.g., [osd], [mon], [mds] ).

2. Changing the setting in a particular process (e.g., [osd.1] ).

Overriding a global setting affects all child processes, except those that you specifically override in a particular daemon.

A typical global setting involves activating authentication. For example:

[global] #Enable authentication between hosts within the cluster. #v 0.54 and earlier auth supported = cephx #v 0.55 and after auth cluster required = cephx auth service required = cephx auth client required = cephx

You can specify settings that apply to a particular type of daemon. When you specify settings under [osd], [mon] or [mds] without specifying a particular instance, the setting will apply to all OSDs, monitors or metadata daemons respectively.

A typical daemon-wide setting involves setting journal sizes, filestore settings, etc. For example:

[osd] osd journal size = 1000

You may specify settings for particular instances of a daemon. You may specify an instance by entering its type, delimited by a period (.) and by the instance ID. The instance ID for a Ceph OSD Daemon is always numeric, but it may be alphanumeric for Ceph Monitors and Ceph Metadata Servers.

[osd.1] # settings affect osd.1 only. [mon.a] # settings affect mon.a only. [mds.b] # settings affect mds.b only.

If the daemon you specify is a Ceph Gateway client, specify the daemon and the instance, delimited by a period (.). For example:

[client.radosgw.instance-name] # settings affect client.radosgw.instance-name only.

参考文献：