柔中带刚，刚中带柔，淫荡中富含柔和，刚猛中荡漾风骚，无坚不摧，无孔不入！
全部博文（1669）
分类： 云计算
2014-08-18 14:43:48
OpenStack里Swift算是比较难搞的一个了,很多概念都不太好理解,关于他的文档翻来复去就那么几篇(如果你知道好的文档请告诉我)
官方提供了两个安装文档:SAIO-Swift All In One和Instructions for a Multiple Server Swift Installation ()
刚开始看的SAIO,它给我的感觉是有点乱,而且把所有的东西都放在同一台主机上对他各组件之间的关系的理解帮助不大
后面一篇比较好一点,但还是有我不明白的地方(虽然后来会慢慢明白)
所以决定写一篇可以帮助像我一样的新手理解Swift的安装文档,如果你已经对Swift非常清楚,官方的文档更适合你
本文档是先安装配置节点的,之所以这样安排是因为安装配置完存储节点后就知道我有那么设备,然后建立环的时候就知道怎么向环里加设备了
使用的主机信息
|----+----------------+-------------+-----------+--------------|
| ID | Name | OS | IP | Used for |
|----+----------------+-------------+-----------+--------------|
| 1 | swift-proxy | ubuntu12.04 | 10.0.0.8 | Proxy Node |
| 2 | swift-storage1 | ubuntu12.04 | 10.0.0.9 | Storage Node |
| 3 | swift-storage2 | ubuntu12.04 | 10.0.0.11 | Storage Node |
| 4 | swift-storage3 | ubuntu12.04 | 10.0.0.12 | Storage Node |
|----+----------------+-------------+-----------+--------------|
OpenStack Hands on lab系列相关文章 见
Table of Contents
1 安装配置存储节点(Storage1)
1.1 安装
1.2 配置
1.2.1 配置存储
1.2.2 配置Swift
1.2.3 安装配置rsync
1.2.4 配置存储服务(account, container, object)
1.2.5 配置日志
2 安装配置代理节点(Proxy)
2.1 安装
2.2 配置
2.2.1 配置memcached
2.2.2 配置swift
2.2.3 配置proxy-server
2.2.4 配置日志
2.2.5 配置环
2.2.6 启动服务
2.3 测试
2.3.1 用curl测试
2.3.2 用swift测试
3 动态扩展
3.1 新加一个存储节点
3.2 在存储节点上新开一组服务
3.2.1 配置存储
3.2.2 配置rsync
3.2.3 配置存储服务(account,container,object)
3.2.4 修改日志配置
3.2.5 配置环
4 资料
4.1 最后列一些与Swift有关的好的资料
1 安装配置存储节点(Storage1)
Note:为了简单和清晰,在这个存储节点只启动一组服务
1.1 安装
安装需要的软件包
apt-get update
apt-get install swift swift-account swift-container swift-object swift-doc xfsprogs
1.2 配置
1.2.1 配置存储
创建loopback设备(假设为sdb1)分出一个分区然后格式化
fdisk /dev/sdb
mkfs.xfs -i size=1024 /dev/sdb1
更新fstab文件
cat >> /etc/fstab
挂载
mkdir /swift/node
mount /swift/node
chown -R swift:swift /swift
1.2.2 配置Swift
创建swift的配置文件
cat >/etc/swift/swift.conf
Note: 这个文件所有的存储节点和代理节点共用,可以用scp拷贝到其它节点
1.2.3 安装配置rsync
安装rsync
apt-get install rsync
创建rsync配置文件
cat > /etc/rsyncd.conf address = 10.0.0.9
[account]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/object.lock
EOF
启用rsync
perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync
启动rsync
service rsync start
1.2.4 配置存储服务(account, container, object)
配置account服务
cat > /etc/swift/account-server.conf bind_port = 6012
workers = 2
log_facility = LOG_LOCAL3
[pipeline:main]
pipeline = account-server
[app:account-server]
use = egg:swift#account
[account-replicator]
[account-auditor]
[account-reaper]
EOF
配置container服务
cat > /etc/swift/container-server.conf bind_port = 6011
workers = 2
log_facility = LOG_LOCAL2
[pipeline:main]
pipeline = container-server
[app:container-server]
use = egg:swift#container
[container-replicator]
[container-updater]
[container-auditor]
[container-sync]
EOF
配置object服务
cat > /etc/swift/object-server.conf bind_port = 6010
workers = 2
log_facility = LOG_LOCAL1
[pipeline:main]
pipeline = object-server
[app:object-server]
use = egg:swift#object
[object-replicator]
[object-updater]
[object-auditor]
[object-expirer]
EOF
1.2.5 配置日志
创建swift日志配置文件
cat > /etc/rsyslog.d/10-swift.conf
创建日志目录
mkdir -p /var/log/swift
chown -R syslog.adm /var/log/swift
重启日志服务进程
service rsyslog restart
Note: 其它各存储节点做相同的配置
apt-get update apt-get install swift swift-proxy swift-doc memcached
perl -pi -e "s/-l 127.0.0.1/-l 10.0.0.8/" /etc/memcached.conf
service memcached restart
mkdir /etc/swift
scp root@10.0.0.9:/etc/swift/swift.conf /etc/swift
cat > /etc/swift/proxy-server.conf < EOF [DEFAULT] bind_port = 8080 bind_ip = 10.0.0.8 user = swift log_level = DEBUG log_facility = LOG_LOCAL1 [pipeline:main] pipeline = healthcheck cache tempauth proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true [filter:tempauth] use = egg:swift#tempauth user_admin_admin = admin .admin .reseller_admin user_test_tester = testing .admin user_test2_tester2 = testing2 .admin user_test_tester3 = testing3 [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = 10.0.0.8:11211 EOF
cat > /etc/rsyslog.d/10-swift.conf << EOF local1.* /var/log/swift/proxy.log
mkdir -p /var/log/swift chown -R syslo.adm /var/log/swift
service rsyslog restart
cd /etc/swift swift-ring-builder account.builder create 18 2 1 swift-ring-builder container.builder create 18 2 1 swift-ring-builder object.builder create 18 2 1
Note: 后面的数字第一个为存储分区数,第二个为备份次数,备份次数不能小于ZONE数目,第三个为延迟删除的时间,以小时为单位
|---------+------+-----------+-----------+------+--------| | Storage | Zone | IP | Server | Port | Device | |---------+------+-----------+-----------+------+--------| | 1 | 1 | 10.0.0.9 | account | 6012 | sdb1 | | | | | container | 6011 | sdb1 | | | | | object | 6010 | sdb1 | |---------+------+-----------+-----------+------+--------| | 2 | 2 | 10.0.0.11 | account | 6012 | sdb1 | | | | | container | 6011 | sdb1 | | | | | object | 6010 | sdb1 | |---------+------+-----------+-----------+------+--------|
cd /etc/swift swift-ring-builder account.builder add z1-10.0.0.9:6012/sdb1 100 swift-ring-builder container.builder add z1-10.0.0.9:6011/sdb1 100 swift-ring-builder object.builder add z1-10.0.0.9:6010/sdb1 100 swift-ring-builder account.builder add z2-10.0.0.11:6012/sdb1 100 swift-ring-builder container.builder add z2-10.0.0.11:6011/sdb1 100 swift-ring-builder object.builder add z2-10.0.0.11:6010/sdb1 100
Note: IP,端口,设备和前面的配置对应
cd /etc/swift swift-ring-builder account.builder swift-ring-builder container.builder swift-ring-builder object.builder
应该得到类似这样的输出
account.builder, build version 1
262144 partitions, 1 replicas, 1 zones, 1 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
0 1 10.0.0.9 6012 sdb1 100.00 262144 0.00
0 1 10.0.0.11 6012 sdb1 100.00 262144 0.00
cd /etc/swift swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance
Note: 这个过程可能需要一些时间
Note: 这三条命令会产生以gz结尾的文件,这些文件就是所有节点要用到的环文件,可以用scp拷贝到其它各节点
Note: 如果创建环的时候指定的副本数大于环的数量,平衡环的时候可能会报类似下面的错误
...
4, in _initial_balance
while available_devs[index]['zone'] in other_zones:
IndexError: list index out of range
scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/ scp /etc/swift/*.gz root@10.0.0.11:/etc/swift/
# 代理节点 swift-init proxy start # 存储节点 swift-init all start
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://10.0.0.8:8080/auth/v1.0
应该得到类似这样的输出
* About to connect() to 10.0.0.8 port 8080 (#0) * Trying 10.0.0.8... connected > GET /auth/v1.0 HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0g zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 10.0.0.8:8080 > Accept: */* > X-Storage-User: test:tester > X-Storage-Pass: testing > < HTTP/1.1 200 OK < X-Storage-Url: < X-Storage-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d < X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d < Content-Length: 0 < Date: Mon, 09 Apr 2012 09:10:21 GMT < * Connection #0 to host 10.0.0.8 left intact * Closing connection #0
curl -v -H 'X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d'
相当的输出类似这样
* About to connect() to 10.0.0.8 port 8080 (#0) * Trying 10.0.0.8... connected > GET /v1/AUTH_test HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0g zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 10.0.0.8:8080 > Accept: */* > X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d > < HTTP/1.1 200 OK < X-Account-Object-Count: 1 < X-Account-Bytes-Used: 1081 < X-Account-Container-Count: 1 < Accept-Ranges: bytes < Content-Length: 10 < Content-Type: text/plain; charset=utf-8 < Date: Mon, 09 Apr 2012 09:12:28 GMT < container * Connection #0 to host 10.0.0.8 left intact * Closing connection #0
swift -A http://10.0.0.8:8080/auth/v1.0 -U test:tester -K testing stat
Account: AUTH_test
Containers: 1
Objects: 1
Bytes: 1081
Accept-Ranges: bytes
Note: 较详细的swift客户端的使用可以查看这篇文章:
(有点老)或官方手册
新加一个存储节点的步骤和配置第一个存储节点相同,在配置好第三个存储节点后,我们可以在代理节点更新环文件(将新存储节点加到环里面)然后拷贝到其它节点,然后重启所有服务
主要步骤如下:
|---------+------+-----------+-----------+------+--------| | Storage | Zone | IP | Server | Port | Device | |---------+------+-----------+-----------+------+--------| | 3 | 3 | 10.0.0.12 | account | 6012 | sdb1 | | | | | container | 6011 | sdb1 | | | | | object | 6010 | sdb1 | |---------+------+-----------+-----------+------+--------|
cd /etc/swift swift-ring-builder account.builder add z3-10.0.0.12:6012/sdb1 100 swift-ring-builder container.builder add z3-10.0.0.12:6011/sdb1 100 swift-ring-builder object.builder add z3-10.0.0.12:6010/sdb1 100
swift-ring-builder object.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder account.builder rebalance
scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/ ...
swift-init all restart
有时候在一台服务器上只开一组服务可能不能对资源充分利用, 这时可以在这台机子上动态新增一组或多组,这和前面的步骤也大体上相同,这里需要注意的是:
可以将各个服务的配置文件放在一个目录里,如/etc/swift/account-server/1.conf (官方文档就是这样做的)
端口和日志需要做相应的修改
更新环的时候IP地址,端口,和设备需要和配置对应
这里简单的列一下步骤(假设我们在第一台存储节点上,IP:10.0.0.9):
mkfs.xfs -i size=1024 /dev/sdb6
cat >> /etc/fstab <
mkdir -p /swift/node2 mount /swift/node2 chown -R swift:swift /swift
cat > /etc/rsyncd.conf << EOF uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 10.0.0.9 [account1] max connections = 2 path = /swift/node read only = false lock file = /var/lock/account1.lock [container1] max connections = 2 path = /swift/node read only = false lock file = /var/lock/container1.lock [object1] max connections = 2 path = /swift/node read only = false lock file = /var/lock/object1.lock [account2] max connections = 2 path = /swift/node2 read only = false lock file = /var/lock/account2.lock [container2] max connections = 2 path = /swift/node2 read only = false lock file = /var/lock/container2.lock [object2] max connections = 2 path = /swift/node2 read only = false lock file = /var/lock/object2.lock EOF
service rsync restart
mkdir /etc/swift/{account-server,container-server,object-server}
mv /etc/swift/account-server.conf /etc/swift/account-server/1.conf
mv /etc/swift/container-server.conf /etc/swift/container-server/1.conf
mv /etc/swift/object-server.conf /etc/swift/object-server/1.conf
sed -i 's/LOG_LOCAL[2|3]/LOG_LOCAL1/' /etc/swift/{account-server,container-server,object-server}/1.conf
cat > /etc/swift/account-server/2.conf << EOF [DEFAULT] devices = /swift/node2 mount_check = false bind_ip = 0.0.0.0 bind_port = 6022 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] [account-auditor] [account-reaper] EOF
cat > /etc/swift/container-server/2.conf << EOF [DEFAULT] devices = /swift/node2 mount_check = false bind_ip = 0.0.0.0 bind_port = 6021 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] [container-updater] [container-auditor] [container-sync] EOF
cat > /etc/swift/object-server/2.conf << EOF [DEFAULT] devices = /swift/node2 mount_check = false bind_ip = 0.0.0.0 bind_port = 6020 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] [object-updater] [object-auditor] [object-expirer] EOF
cat > /etc/rsyslog.d/10-swift.conf << EOF local1,local2.* /var/log/swift/all.log local1.* /var/log/swift/storage1.log local2.* /var/log/swift/storage2.log EOF
service rsyslog restart
|---------+------+----------+-----------+------+--------| | Storage | Zone | IP | Server | Port | Device | |---------+------+----------+-----------+------+--------| | 4 | 1 | 10.0.0.9 | account | 6022 | sdd2 | | | | | container | 6021 | sdd2 | | | | | object | 6020 | sdd2 | |---------+------+----------+-----------+------+--------|
cd /etc/swift swift-ring-builder account.builder add z1-10.0.0.9:6022/sdd2 100 swift-ring-builder container.builder add z1-10.0.0.9:6021/sdd2 100 swift-ring-builder object.builder add z1-10.0.0.9:6020/sdd2 100
Note: IP,端口,设备和前面的配置对应
Note: 这里的设备(sdd2)会在前面配置文件里的Devices = /swift/node指定的目录下创建这个目录(/swift/node/sdd2)
cd /etc/swift swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance
scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/ ...
swift-init all restart