• 博客访问: 393023
  • 博文数量: 97
  • 博客积分: 660
  • 博客等级: 上士
  • 技术积分: 1631
  • 用 户 组: 普通用户
  • 注册时间: 2005-08-08 11:39
文章分类

全部博文(97)

文章存档

2017年(36)

2016年(7)

2015年(3)

2014年(3)

2013年(31)

2012年(2)

2011年(1)

2006年(1)

2005年(13)

微信关注

IT168企业级官微



微信号:IT168qiye



系统架构师大会



微信号:SACC2013

订阅
热词专题

分类: NOSQL

mongodb3.0.7扩容存储记录

扩容方案

扩容前复制集

原有db01、db02、230为复制集,其中db01为primary,230为远程复制节点。

扩容前存储

3台主机均为1T

扩容方法

尽量减少线上停机时间(5分钟左右)

  • 新增一台db03,采用ssm管理磁盘,初步分配1.5T磁盘空间,后续空间不足继续增加磁盘交给ssm管理和扩容
  • db03扩容完成后,上线到原有复制集中
  • db03同步数据完成后切换primary到db03
  • 之后分别下线db01、db02、230等进行磁盘扩容

db03安装和配置

mongodb的安装

[root@db03 ~]# /etc/yum.repos.d/mongodb-org-3.4.repo
-bash: /etc/yum.repos.d/mongodb-org-3.4.repo: No such file or directory
[root@db03 ~]# vi /etc/yum.repos.d/mongodb-org-3.4.repo
[root@db03 ~]# yum install -y mongodb-org
Loaded plugins: fastestmirror, langpacks
mongodb-org-3.4                                                        | 2.5 kB  00:00:00
mongodb-org-3.4/7/primary_db                                           |  25 kB  00:00:01
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package mongodb-org.x86_64 0:3.4.4-1.el7 will be installed
--> Processing Dependency: mongodb-org-tools = 3.4.4 for package: mongodb-org-3.4.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-shell = 3.4.4 for package: mongodb-org-3.4.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-server = 3.4.4 for package: mongodb-org-3.4.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-mongos = 3.4.4 for package: mongodb-org-3.4.4-1.el7.x86_64
--> Running transaction check
---> Package mongodb-org-mongos.x86_64 0:3.4.4-1.el7 will be installed
---> Package mongodb-org-server.x86_64 0:3.4.4-1.el7 will be installed
---> Package mongodb-org-shell.x86_64 0:3.4.4-1.el7 will be installed
---> Package mongodb-org-tools.x86_64 0:3.4.4-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================
 Package                    Arch           Version              Repository               Size
==============================================================================================
Installing:
 mongodb-org                x86_64         3.4.4-1.el7          mongodb-org-3.4         5.8 k
Installing for dependencies:
 mongodb-org-mongos         x86_64         3.4.4-1.el7          mongodb-org-3.4          12 M
 mongodb-org-server         x86_64         3.4.4-1.el7          mongodb-org-3.4          20 M
 mongodb-org-shell          x86_64         3.4.4-1.el7          mongodb-org-3.4          11 M
 mongodb-org-tools          x86_64         3.4.4-1.el7          mongodb-org-3.4          49 M

Transaction Summary
==============================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 91 M
Installed size: 257 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
mongodb-org-server-3.4.4-1.el7 FAILED                                            13:33:40 ETA
https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.4/x86_64/RPMS/mongodb-org-server-3.4.4-1.el7.x86_64.rpm: [Errno 12] Timeout on https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.4/x86_64/RPMS/mongodb-org-server-3.4.4-1.el7.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
mongodb-org-mongos-3.4.4-1.el7 FAILED                                            00:29:43 ETA
https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.4/x86_64/RPMS/mongodb-org-mongos-3.4.4-1.el7.x86_64.rpm: [Errno 12] Timeout on https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.4/x86_64/RPMS/mongodb-org-mongos-3.4.4-1.el7.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
warning: /var/cache/yum/x86_64/7/mongodb-org-3.4/packages/mongodb-org-shell-3.4.4-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID a15703c6: NOKEY
Public key for mongodb-org-shell-3.4.4-1.el7.x86_64.rpm is not installed
(1/4): mongodb-org-shell-3.4.4-1.el7.x86_64.rpm                        |  11 MB  00:03:23
(2/4): mongodb-org-tools-3.4.4-1.el7.x86_64.rpm                        |  49 MB  00:26:42
(3/4): mongodb-org-server-3.4.4-1.el7.x86_64.rpm                       |  20 MB  00:00:01
(4/4): mongodb-org-mongos-3.4.4-1.el7.x86_64.rpm                       |  12 MB  00:00:52
----------------------------------------------------------------------------------------------
Total                                                          42 kB/s |  91 MB  00:37:17
Retrieving key from https://www.mongodb.org/static/pgp/server-3.4.asc
Importing GPG key 0xA15703C6:
 Userid     : "MongoDB 3.4 Release Signing Key <packaging@mongodb.com>"
 Fingerprint: 0c49 f373 0359 a145 1858 5931 bc71 1f9b a157 03c6
 From       : https://www.mongodb.org/static/pgp/server-3.4.asc
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mongodb-org-tools-3.4.4-1.el7.x86_64                                       1/5
  Installing : mongodb-org-mongos-3.4.4-1.el7.x86_64                                      2/5
  Installing : mongodb-org-shell-3.4.4-1.el7.x86_64                                       3/5
  Installing : mongodb-org-server-3.4.4-1.el7.x86_64                                      4/5
Created symlink from /etc/systemd/system/multi-user.target.wants/mongod.service to /usr/lib/systemd/system/mongod.service.
  Installing : mongodb-org-3.4.4-1.el7.x86_64                                             5/5
  Verifying  : mongodb-org-server-3.4.4-1.el7.x86_64                                      1/5
  Verifying  : mongodb-org-3.4.4-1.el7.x86_64                                             2/5
  Verifying  : mongodb-org-shell-3.4.4-1.el7.x86_64                                       3/5
  Verifying  : mongodb-org-mongos-3.4.4-1.el7.x86_64                                      4/5
  Verifying  : mongodb-org-tools-3.4.4-1.el7.x86_64                                       5/5

Installed:
  mongodb-org.x86_64 0:3.4.4-1.el7

Dependency Installed:
  mongodb-org-mongos.x86_64 0:3.4.4-1.el7       mongodb-org-server.x86_64 0:3.4.4-1.el7
  mongodb-org-shell.x86_64 0:3.4.4-1.el7        mongodb-org-tools.x86_64 0:3.4.4-1.el7

Complete! 

创建mongodb存储

查看存储

[root@db03 ~]# ssm list
----------------------------------
Device          Total  Mount point
----------------------------------
/dev/vda     20.00 GB  PARTITIONED
/dev/vda1    20.00 GB  /
/dev/vdb    500.00 GB
/dev/vdc   1000.00 GB
----------------------------------
---------------------------------------------------------------------
Volume     Volume size  FS      FS size       Free  TypeMount point
---------------------------------------------------------------------
/dev/vda1     20.00 GB  ext3   20.00 GB   17.19 GB  part/
/dev/vdc    1000.00 GB  xfs   999.51 GB  999.41 GB
---------------------------------------------------------------------

建立ssm管理的存储池centos

[root@db03 ~]# ssm add -p centos /dev/vdb
  Physical volume "/dev/vdb" successfully created.
  Volume group "centos" successfully created
[root@db03 ~]# ssm add -p centos /dev/vdc
Filesystem 'xfs' detected on the device '/dev/vdc'! Use anyway (N/y/q) ? Y
  Physical volume "/dev/vdc" successfully created.
  Volume group "centos" successfully extended

检查确认存储池为1.46T

[root@db03 ~]# ssm list
---------------------------------------------------------------
Device           Free     Used       Total  Pool    Mount point
---------------------------------------------------------------
/dev/vda                          20.00 GB          PARTITIONED
/dev/vda1                         20.00 GB          /
/dev/vdb    500.00 GB  0.00 KB   500.00 GB  centos
/dev/vdc   1000.00 GB  0.00 KB  1000.00 GB  centos
---------------------------------------------------------------
------------------------------------------------
Pool    Type  Devices     Free     Used    Total
------------------------------------------------
centos  lvm   2        1.46 TB  0.00 KB  1.46 TB
------------------------------------------------
-------------------------------------------------------------------
Volume     Volume size  FS     FS size      Free  TypeMount point
-------------------------------------------------------------------
/dev/vda1     20.00 GB  ext3  20.00 GB  17.19 GB  part/
-------------------------------------------------------------------

创建挂载点

[root@db03 ~]# ssm create -n goluk -s 1.46T --fstype xfs -p centos /goluk
  Rounding up size to full physical extent 1.46 TiB
  Logical volume "goluk" created.
meta-data=/dev/centos/goluk      isize=512    agcount=4, agsize=97979136 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=391916544, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=191365, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

确认挂载成功

[root@db03 ~]# ssm list
---------------------------------------------------------------
Device        Free        Used       Total  Pool    Mount point
---------------------------------------------------------------
/dev/vda                          20.00 GB          PARTITIONED
/dev/vda1                         20.00 GB          /
/dev/vdb   4.95 GB   495.05 GB   500.00 GB  centos
/dev/vdc   0.00 KB  1000.00 GB  1000.00 GB  centos
---------------------------------------------------------------
------------------------------------------------
Pool    Type  Devices     Free     Used    Total
------------------------------------------------
centos  lvm   2        4.95 GB  1.46 TB  1.46 TB
------------------------------------------------
-------------------------------------------------------------------------------------
Volume             Pool    Volume size  FS     FS size      Free  Type    Mount point
-------------------------------------------------------------------------------------
/dev/centos/goluk  centos      1.46 TB  xfs    1.46 TB   1.46 TB  linear  /goluk
/dev/vda1                     20.00 GB  ext3  20.00 GB  17.19 GB  part    /
-------------------------------------------------------------------------------------

修改mongodb配置文件

[root@db03 ~]# cd /etc/
[root@db03 etc]# mv mongod.conf mongod.conf.20170505
[root@db03 etc]# vi /etc/mongod.conf
[root@db03 etc]# cat /etc/mongod.conf
systemLog:
    destination: file
    path: "/goluk/mongo/data/mongod.log"
    logAppend: true
storage:
    journal:
      enabled: true
    dbPath: "/goluk/mongo/data/"
net:
    port: 27017
    bindIp: 10.0.1.33,127.0.0.1
processManagement:
    fork: true
replication:
    replSetName: goluk

创建目录和分配权限

[root@db03 etc]# mkdir -p /goluk/mongo/{data,conf}
[root@db03 etc]# chown -R mongod:mongod /goluk/mongo/

db03节点启动

mongod -f /etc/mongod.conf

primary节点操作

  • 查看复制集状态

    rs.status()

  • 增加一个复制节点

    rs.add("10.0.1.33:27017") db03ip是10.0.1.33

  • 观察新增节点

    rs.status() 发现db03状态已经是secondary了

    db03节点

  • 观察日志mongod启动正常

  • 观察db03节点同步数据完成

primary节点切换过程

切换检查列表

原环境需要考虑到的因素

受影响主机列表

  • 应用服务:web01 web02 cp01 cp02

    应用服务采用域名连接mongodb,mongodb切换完成后,需要重启服务

  • nginx:nginx01 nginx02

    修改hosts文件或者修改内网dns记录即可

  • redis:redis01 redis02

    经检查非相关,无需修改

  • db: db01 db02

    修改hosts文件或者修改内网dns记录即可

需要听证的hosts文件内容

将原db01的主机记录修改为db03

需要重启的进程列表

  • 应用服务列表,以当前运行中的为准
  • 重启命令列表,备重启用

切换完成后的测试

  • db03切换为primary
  • 应用测试是否正常
  • mongodb备份脚本是否修改
  • zabbix监控是否增加db03的监控项目

mongodb切换过程

primary从db01换到db03

primary降级(大约1分钟左右完成)

  • 31 mongo shell 执行 

    rs.status() 确认状态

    31 primary
    32 secondary
    33  secondary
    230 rocovering
  • 32 执行 rs.freeze(120)

    120秒内db02不能成为primary节点

  • 31 执行 rs.stepdown(120)

    db01降级

  • db03 自动成为primary节点

更换hosts和dns记录

更换 web01、web02、cp01、cp02的hosts文件,过程略。
  • 重启应用

    过程略

检查db03

  • rs.status()
  • 查看连接数 db.serverStatus()

ops01备份脚本修改

扩容遇到问题

原节点mongodb都是3.0.7的版本,原计划db03升级到3.4,发现复制集不能连接上原复制集,降级到3.0.14版本才完成本次扩容

阅读(4205) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册