Chinaunix首页 | 论坛 | 博客
  • 博客访问: 4744345
  • 博文数量: 1646
  • 博客积分: 16831
  • 博客等级: 上将
  • 技术积分: 12474
  • 用 户 组: 普通用户
  • 注册时间: 2011-02-25 07:23
个人简介

柔中带刚,刚中带柔,淫荡中富含柔和,刚猛中荡漾风骚,无坚不摧,无孔不入!

文章分类

全部博文(1646)

文章存档

2019年(1)

2018年(20)

2017年(68)

2016年(61)

2015年(51)

2014年(204)

2013年(224)

2012年(644)

2011年(373)

分类: 云计算

2014-08-18 14:43:48


http://www.linuxidc.com/Linux/2013-08/88178.htm

OpenStack里Swift算是比较难搞的一个了,很多概念都不太好理解,关于他的文档翻来复去就那么几篇(如果你知道好的文档请告诉我)
官方提供了两个安装文档:SAIO-Swift All In One和Instructions for a Multiple Server Swift Installation (Ubuntu)
刚开始看的SAIO,它给我的感觉是有点乱,而且把所有的东西都放在同一台主机上对他各组件之间的关系的理解帮助不大
后面一篇比较好一点,但还是有我不明白的地方(虽然后来会慢慢明白)
所以决定写一篇可以帮助像我一样的新手理解Swift的安装文档,如果你已经对Swift非常清楚,官方的文档更适合你
本文档是先安装配置节点的,之所以这样安排是因为安装配置完存储节点后就知道我有那么设备,然后建立环的时候就知道怎么向环里加设备了
使用的主机信息
|----+----------------+-------------+-----------+--------------|
| ID | Name          | OS          |        IP | Used for    |
|----+----------------+-------------+-----------+--------------|
|  1 | swift-proxy    | ubuntu12.04 |  10.0.0.8 | Proxy Node  |
|  2 | swift-storage1 | ubuntu12.04 |  10.0.0.9 | Storage Node |
|  3 | swift-storage2 | ubuntu12.04 | 10.0.0.11 | Storage Node |
|  4 | swift-storage3 | ubuntu12.04 | 10.0.0.12 | Storage Node |
|----+----------------+-------------+-----------+--------------|

OpenStack Hands on lab系列相关文章 见 http://www.linuxidc.com/Linux/2013-08/88170.htm

Table of Contents
1 安装配置存储节点(Storage1) 
1.1 安装 
1.2 配置 
1.2.1 配置存储 
1.2.2 配置Swift 
1.2.3 安装配置rsync 
1.2.4 配置存储服务(account, container, object) 
1.2.5 配置日志 
2 安装配置代理节点(Proxy) 
2.1 安装 
2.2 配置 
2.2.1 配置memcached 
2.2.2 配置swift 
2.2.3 配置proxy-server 
2.2.4 配置日志 
2.2.5 配置环 
2.2.6 启动服务 
2.3 测试 
2.3.1 用curl测试 
2.3.2 用swift测试 
3 动态扩展 
3.1 新加一个存储节点 
3.2 在存储节点上新开一组服务 
3.2.1 配置存储 
3.2.2 配置rsync 
3.2.3 配置存储服务(account,container,object) 
3.2.4 修改日志配置 
3.2.5 配置环 
4 资料 
4.1 最后列一些与Swift有关的好的资料 
1 安装配置存储节点(Storage1)
Note:为了简单和清晰,在这个存储节点只启动一组服务
1.1 安装
安装需要的软件包 
apt-get update
apt-get install swift swift-account swift-container swift-object swift-doc xfsprogs
1.2 配置
1.2.1 配置存储
创建loopback设备(假设为sdb1)分出一个分区然后格式化 
fdisk /dev/sdb
mkfs.xfs -i size=1024 /dev/sdb1
更新fstab文件 
cat >> /etc/fstab 
挂载 
mkdir /swift/node
mount /swift/node
chown -R swift:swift /swift
1.2.2 配置Swift
创建swift的配置文件 
cat >/etc/swift/swift.conf 
Note: 这个文件所有的存储节点和代理节点共用,可以用scp拷贝到其它节点
1.2.3 安装配置rsync
安装rsync 
apt-get install rsync
创建rsync配置文件 
cat > /etc/rsyncd.conf address = 10.0.0.9
[account]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /swift/node
read only = false
lock file = /var/lock/object.lock
EOF
启用rsync 
perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync
启动rsync 
service rsync start
1.2.4 配置存储服务(account, container, object)
配置account服务 
cat > /etc/swift/account-server.conf bind_port = 6012
workers = 2
log_facility = LOG_LOCAL3
[pipeline:main]
pipeline = account-server
[app:account-server]
use = egg:swift#account
[account-replicator]
[account-auditor]
[account-reaper]
EOF
配置container服务 
cat > /etc/swift/container-server.conf bind_port = 6011
workers = 2
log_facility = LOG_LOCAL2
[pipeline:main]
pipeline = container-server
[app:container-server]
use = egg:swift#container
[container-replicator]
[container-updater]
[container-auditor]
[container-sync]
EOF
配置object服务 
cat > /etc/swift/object-server.conf bind_port = 6010
workers = 2
log_facility = LOG_LOCAL1
[pipeline:main]
pipeline = object-server
[app:object-server]
use = egg:swift#object
[object-replicator]
[object-updater]
[object-auditor]
[object-expirer]
EOF
1.2.5 配置日志
创建swift日志配置文件 
cat > /etc/rsyslog.d/10-swift.conf 
创建日志目录 
mkdir -p /var/log/swift
chown -R syslog.adm /var/log/swift
重启日志服务进程 
service rsyslog restart
Note: 其它各存储节点做相同的配置


2 安装配置代理节点(Proxy)

 

2.1 安装

  • 安装需要的软件包
    apt-get update
    apt-get install swift swift-proxy swift-doc memcached

2.2 配置

 

2.2.1 配置memcached

  • 修改memcached配置文件
    perl -pi -e "s/-l 127.0.0.1/-l 10.0.0.8/" /etc/memcached.conf
  • 重启memcached服务
    service memcached restart

2.2.2 配置swift

  • 从存储节点拷贝swift配置文件
    mkdir /etc/swift
    scp root@10.0.0.9:/etc/swift/swift.conf /etc/swift

2.2.3 配置proxy-server

  • 创建proxy-server配置文件
    cat > /etc/swift/proxy-server.conf < EOF
    [DEFAULT] bind_port = 8080 bind_ip = 10.0.0.8 user = swift
    log_level = DEBUG
    log_facility = LOG_LOCAL1
    
    [pipeline:main]
    pipeline = healthcheck cache tempauth proxy-server
    
    [app:proxy-server]
    use = egg:swift#proxy
    allow_account_management = true
    account_autocreate = true
    
    [filter:tempauth]
    use = egg:swift#tempauth
    user_admin_admin = admin .admin .reseller_admin
    user_test_tester = testing .admin
    user_test2_tester2 = testing2 .admin
    user_test_tester3 = testing3
    
    [filter:healthcheck]
    use = egg:swift#healthcheck
    
    [filter:cache]
    use = egg:swift#memcache
    memcache_servers = 10.0.0.8:11211 EOF

2.2.4 配置日志

  • 创建swift日志配置文件
    cat > /etc/rsyslog.d/10-swift.conf << EOF
    local1.*   /var/log/swift/proxy.log
  • 创建日志目录
    mkdir -p /var/log/swift
    chown -R syslo.adm /var/log/swift
  • 重启日志服务进程
    service rsyslog restart

2.2.5 配置环

  • 首先创建环
    cd /etc/swift
    swift-ring-builder account.builder create 18 2 1
    swift-ring-builder container.builder create 18 2 1
    swift-ring-builder object.builder create 18 2 1

    Note: 后面的数字第一个为存储分区数,第二个为备份次数,备份次数不能小于ZONE数目,第三个为延迟删除的时间,以小时为单位

  • 向环里添加存储设备我前面配置了两台存储节点:
    |---------+------+-----------+-----------+------+--------|
    | Storage | Zone |        IP | Server    | Port | Device |
    |---------+------+-----------+-----------+------+--------|
    |       1 |    1 |  10.0.0.9 | account   | 6012 | sdb1   |
    |         |      |           | container | 6011 | sdb1   |
    |         |      |           | object    | 6010 | sdb1   |
    |---------+------+-----------+-----------+------+--------|
    |       2 |    2 | 10.0.0.11 | account   | 6012 | sdb1   |
    |         |      |           | container | 6011 | sdb1   |
    |         |      |           | object    | 6010 | sdb1   |
    |---------+------+-----------+-----------+------+--------|
    cd /etc/swift
    swift-ring-builder account.builder add z1-10.0.0.9:6012/sdb1 100
    swift-ring-builder container.builder add z1-10.0.0.9:6011/sdb1 100
    swift-ring-builder object.builder add z1-10.0.0.9:6010/sdb1 100
    
    swift-ring-builder account.builder add z2-10.0.0.11:6012/sdb1 100
    swift-ring-builder container.builder add z2-10.0.0.11:6011/sdb1 100
    swift-ring-builder object.builder add z2-10.0.0.11:6010/sdb1 100

    Note: IP,端口,设备和前面的配置对应

  • 检验每个环里的条目
    cd /etc/swift
    swift-ring-builder account.builder
    swift-ring-builder container.builder
    swift-ring-builder object.builder

    应该得到类似这样的输出

    account.builder, build version 1
    262144 partitions, 1 replicas, 1 zones, 1 devices, 0.00 balance
    The minimum number of hours before a partition can be reassigned is 1
    Devices:    id  zone      ip address  port      name weight partitions balance meta
                 0     1        10.0.0.9  6012      sdb1 100.00     262144    0.00
                 0     1        10.0.0.11 6012      sdb1 100.00     262144    0.00
  • 平衡环
    cd /etc/swift
    swift-ring-builder account.builder rebalance
    swift-ring-builder container.builder rebalance
    swift-ring-builder object.builder rebalance

    Note: 这个过程可能需要一些时间

    Note: 这三条命令会产生以gz结尾的文件,这些文件就是所有节点要用到的环文件,可以用scp拷贝到其它各节点

    Note: 如果创建环的时候指定的副本数大于环的数量,平衡环的时候可能会报类似下面的错误

    ...
    4, in _initial_balance
        while available_devs[index]['zone'] in other_zones:
    IndexError: list index out of range
  • 分发新生成的环文件
    scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/
    scp /etc/swift/*.gz root@10.0.0.11:/etc/swift/

2.2.6 启动服务

  • 启动各节点相关的服务
    # 代理节点
    swift-init proxy start
    
    # 存储节点
    swift-init all start

2.3 测试

 

2.3.1 用curl测试

  • 得到一个url和token
    curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://10.0.0.8:8080/auth/v1.0

    应该得到类似这样的输出

    * About to connect() to 10.0.0.8 port 8080 (#0)
    *   Trying 10.0.0.8... connected
    > GET /auth/v1.0 HTTP/1.1
    > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0g zlib/1.2.3.4 libidn/1.23 librtmp/2.3
    > Host: 10.0.0.8:8080
    > Accept: */*
    > X-Storage-User: test:tester
    > X-Storage-Pass: testing
    >
    < HTTP/1.1 200 OK
    < X-Storage-Url: http://10.0.0.8:8080/v1/AUTH_test
    < X-Storage-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d
    < X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d
    < Content-Length: 0
    < Date: Mon, 09 Apr 2012 09:10:21 GMT
    <
    * Connection #0 to host 10.0.0.8 left intact
    * Closing connection #0
  • 检验得到的url和token
    curl -v -H 'X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d' http://10.0.0.8:8080/v1/AUTH_test

    相当的输出类似这样

    * About to connect() to 10.0.0.8 port 8080 (#0)
    *   Trying 10.0.0.8... connected
    > GET /v1/AUTH_test HTTP/1.1
    > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0g zlib/1.2.3.4 libidn/1.23 librtmp/2.3
    > Host: 10.0.0.8:8080
    > Accept: */*
    > X-Auth-Token: AUTH_tk44dccdb5a63240828f30019c2025d12d
    >
    < HTTP/1.1 200 OK
    < X-Account-Object-Count: 1
    < X-Account-Bytes-Used: 1081
    < X-Account-Container-Count: 1
    < Accept-Ranges: bytes
    < Content-Length: 10
    < Content-Type: text/plain; charset=utf-8
    < Date: Mon, 09 Apr 2012 09:12:28 GMT
    <
    container
    * Connection #0 to host 10.0.0.8 left intact
    * Closing connection #0

2.3.2 用swift测试

  • 显示一个用户存储的统计信息
    swift -A http://10.0.0.8:8080/auth/v1.0 -U test:tester -K testing stat
       Account: AUTH_test
    Containers: 1
       Objects: 1
         Bytes: 1081
    Accept-Ranges: bytes

    Note: 较详细的swift客户端的使用可以查看这篇文章:

  • http://www.linuxidc.com/Linux/2013-08/88179.htm (有点老)或官方手册

3 动态扩展

 

3.1 新加一个存储节点

新加一个存储节点的步骤和配置第一个存储节点相同,在配置好第三个存储节点后,我们可以在代理节点更新环文件(将新存储节点加到环里面)然后拷贝到其它节点,然后重启所有服务

主要步骤如下:

  • 将新加的存储节点的设备加到环里面,可以加到已经存在的ZONE里,也可以新加一个ZONE(关于ZONE的概念请参看官方文档)新加存储的信息
    |---------+------+-----------+-----------+------+--------|
    | Storage | Zone |        IP | Server    | Port | Device |
    |---------+------+-----------+-----------+------+--------|
    |       3 |    3 | 10.0.0.12 | account   | 6012 | sdb1   |
    |         |      |           | container | 6011 | sdb1   |
    |         |      |           | object    | 6010 | sdb1   |
    |---------+------+-----------+-----------+------+--------|
    cd /etc/swift
    swift-ring-builder account.builder add z3-10.0.0.12:6012/sdb1 100
    swift-ring-builder container.builder add z3-10.0.0.12:6011/sdb1 100
    swift-ring-builder object.builder add z3-10.0.0.12:6010/sdb1 100
  • 平衡环
    swift-ring-builder object.builder rebalance
    swift-ring-builder container.builder rebalance
    swift-ring-builder account.builder rebalance
  • 将新生成的环文件拷贝到其它节点去
    scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/
    ...
  • 在各节点上重启服务
    swift-init all restart

3.2 在存储节点上新开一组服务

有时候在一台服务器上只开一组服务可能不能对资源充分利用, 这时可以在这台机子上动态新增一组或多组,这和前面的步骤也大体上相同,这里需要注意的是:
可以将各个服务的配置文件放在一个目录里,如/etc/swift/account-server/1.conf (官方文档就是这样做的)
端口和日志需要做相应的修改
更新环的时候IP地址,端口,和设备需要和配置对应

这里简单的列一下步骤(假设我们在第一台存储节点上,IP:10.0.0.9):

3.2.1 配置存储

  • 创建存储设备首先用fdisk分一个分区出来,也可以从已有的VG里分一个LV出来,然后进行格式化,假设这里的分区是sdb6
    mkfs.xfs -i size=1024 /dev/sdb6
  • 更新fstab文件
    cat >> /etc/fstab <
    					
  • 挂载
    mkdir -p /swift/node2
    mount /swift/node2
    chown -R swift:swift /swift

3.2.2 配置rsync

  • 更新rsync配置文件
    cat > /etc/rsyncd.conf << EOF
    uid = swift
    gid = swift
    log file = /var/log/rsyncd.log
    pid file = /var/run/rsyncd.pid
    address = 10.0.0.9
    
    [account1]
    max connections = 2
    path = /swift/node
    read only = false
    lock file = /var/lock/account1.lock
    
    [container1]
    max connections = 2
    path = /swift/node
    read only = false
    lock file = /var/lock/container1.lock
    
    [object1]
    max connections = 2
    path = /swift/node
    read only = false
    lock file = /var/lock/object1.lock
    
    [account2]
    max connections = 2
    path = /swift/node2
    read only = false
    lock file = /var/lock/account2.lock
    
    [container2]
    max connections = 2
    path = /swift/node2
    read only = false
    lock file = /var/lock/container2.lock
    
    [object2]
    max connections = 2
    path = /swift/node2
    read only = false
    lock file = /var/lock/object2.lock
    EOF
  • 重启rsync
    service rsync restart

3.2.3 配置存储服务(account,container,object)

  • 调整下配置文件结构
    mkdir /etc/swift/{account-server,container-server,object-server}
    mv /etc/swift/account-server.conf /etc/swift/account-server/1.conf
    mv /etc/swift/container-server.conf /etc/swift/container-server/1.conf
    mv /etc/swift/object-server.conf /etc/swift/object-server/1.conf
  • 改一下日志(我们把第一组的日志放到一起使用LOG_LOCAL1)
    sed -i 's/LOG_LOCAL[2|3]/LOG_LOCAL1/' /etc/swift/{account-server,container-server,object-server}/1.conf
  • 配置新加组的account服务
    cat > /etc/swift/account-server/2.conf << EOF
    [DEFAULT]
    devices = /swift/node2
    mount_check = false
    bind_ip = 0.0.0.0 bind_port = 6022 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main]
    pipeline = account-server
    
    [app:account-server]
    use = egg:swift#account
    
    [account-replicator]
    
    [account-auditor]
    
    [account-reaper]
    EOF
  • 配置container服务
    cat > /etc/swift/container-server/2.conf << EOF
    [DEFAULT]
    devices = /swift/node2
    mount_check = false
    bind_ip = 0.0.0.0 bind_port = 6021 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main]
    pipeline = container-server
    
    [app:container-server]
    use = egg:swift#container
    
    [container-replicator]
    
    [container-updater]
    
    [container-auditor]
    
    [container-sync]
    EOF
  • 配置object服务
    cat > /etc/swift/object-server/2.conf << EOF
    [DEFAULT]
    devices = /swift/node2
    mount_check = false
    bind_ip = 0.0.0.0 bind_port = 6020 workers = 2 log_facility = LOG_LOCAL2 [pipeline:main]
    pipeline = object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [object-replicator]
    
    [object-updater]
    
    [object-auditor]
    
    [object-expirer]
    EOF

3.2.4 修改日志配置

  • 修改swift日志配置文件
    cat > /etc/rsyslog.d/10-swift.conf << EOF
    local1,local2.*   /var/log/swift/all.log
    
    local1.*   /var/log/swift/storage1.log
    local2.*   /var/log/swift/storage2.log
    EOF
  • 重启日志服务
    service rsyslog restart

3.2.5 配置环

  • 在代理节点上将新加的设备加到环里(这次我们把它加到ZONE1里)这个存储的信息
    |---------+------+----------+-----------+------+--------|
    | Storage | Zone |       IP | Server    | Port | Device |
    |---------+------+----------+-----------+------+--------|
    |       4 |    1 | 10.0.0.9 | account   | 6022 | sdd2   |
    |         |      |          | container | 6021 | sdd2   |
    |         |      |          | object    | 6020 | sdd2   |
    |---------+------+----------+-----------+------+--------|
    cd /etc/swift
    swift-ring-builder account.builder add z1-10.0.0.9:6022/sdd2 100
    swift-ring-builder container.builder add z1-10.0.0.9:6021/sdd2 100
    swift-ring-builder object.builder add z1-10.0.0.9:6020/sdd2 100

    Note: IP,端口,设备和前面的配置对应
    Note: 这里的设备(sdd2)会在前面配置文件里的Devices = /swift/node指定的目录下创建这个目录(/swift/node/sdd2)

  • 重新平衡环
    cd /etc/swift
    swift-ring-builder account.builder rebalance
    swift-ring-builder container.builder rebalance
    swift-ring-builder object.builder rebalance
  • 将新生成的环文件分发到各个节点
    scp /etc/swift/*.gz root@10.0.0.9:/etc/swift/
    ...
  • 在各节点上重启服务
    swift-init all restart



阅读(940) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册