前提环境：
1、已经配置了masterstandby节点，如果masterstandby故障是不会影响Greenplum正常运行， 但是如果此时关闭master主节点则无法启法需要将原masterstandby节点移除。
1、启动master节点失败
[gpadmin@mdw ~]$ gpstart
20170329:02:30:38:025343 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args:
20170329:02:30:38:025343 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20170329:02:30:38:025343 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev'
20170329:02:30:38:025343 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301703191'
20170329:02:30:38:025343 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20170329:02:30:39:025343 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20170329:02:30:39:025343 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170329:02:30:39:025343 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20170329:02:30:39:025343 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20170329:02:30:40:025343 gpstart:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 2
Command was: 'ssh -o 'StrictHostKeyChecking no' mdw02 ". /usr/local/gpdb-5.0.0/greenplum_path.sh; $GPHOME/bin/pg_controldata /gpmaster50/gpseg-1"'
rc=2, stdout='', stderr='pg_controldata: could not open file "/gpmaster50/gpseg-1/global/pg_control" for reading: No such file or directory
'
2、移除问题mdw02
[gpadmin@mdw ~]$ gpinitstandby -r mdw02
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Warm master standby removal parameters
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master hostname = mdw
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master data directory = /gpmaster50/gpseg-1
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master port = 5432
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master hostname = mdw02
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master port = 5432
20170329:02:30:53:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master data directory = /gpmaster50/gpseg-1
Do you want to continue with deleting the standby master? Yy|Nn (default=N):
> y
20170329:02:30:56:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Removing standby master from catalog...
20170329:02:30:56:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Database catalog updated successfully.
20170329:02:30:56:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Removing filespace directories on standby master...
20170329:02:30:56:025380 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully removed standby master
3、停止master节点
[gpadmin@mdw ~]$ gpstop
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args:
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev'
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:---------------------------------------------
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Master instance parameters
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:---------------------------------------------
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Master Greenplum instance process active PID = 25353
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Database = template1
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Master port = 5432
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Master directory = /gpmaster50/gpseg-1
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Shutdown mode = smart
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Timeout = 120
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Shutdown Master standby host = Off
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:---------------------------------------------
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:-Segment instances that will be shutdown:
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:---------------------------------------------
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- Host Datadir Port Status
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 u
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 u
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 u
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 u
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 u
20170329:02:31:02:025413 gpstop:mdw:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 u
Continue with Greenplum instance shutdown Yy|Nn (default=N):
> y
20170329:02:31:04:025413 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database
20170329:02:31:04:025413 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20170329:02:31:04:025413 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw
20170329:02:31:04:025413 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20170329:02:31:04:025413 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/gpmaster50/gpseg-1
20170329:02:31:05:025413 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20170329:02:31:05:025413 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /gpmaster50/gpseg-1
20170329:02:31:05:025413 gpstop:mdw:gpadmin-[INFO]:-No standby master host configured
20170329:02:31:05:025413 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20170329:02:31:05:025413 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20170329:02:31:15:025413 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20170329:02:31:15:025413 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20170329:02:31:15:025413 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 6
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:- Segments with errors during stop = 0
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 6 of 6 segment instances
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20170329:02:31:25:025413 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory
4、重新启动master
[gpadmin@mdw ~]$ gpstart
20170329:02:31:34:025522 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args:
20170329:02:31:34:025522 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20170329:02:31:34:025522 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev'
20170329:02:31:34:025522 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301703191'
20170329:02:31:34:025522 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20170329:02:31:35:025522 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20170329:02:31:35:025522 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170329:02:31:35:025522 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20170329:02:31:35:025522 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20170329:02:31:35:025522 gpstart:mdw:gpadmin-[INFO]:-Shutting down master
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:---------------------------
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Master instance parameters
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:---------------------------
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Database = template1
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Master Port = 5432
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Master directory = /gpmaster50/gpseg-1
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Timeout = 600 seconds
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Master standby = Off
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:---------------------------------------
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:-Segment instances that will be started
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:---------------------------------------
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- Host Datadir Port Role
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 Primary
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 Mirror
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 Primary
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 Mirror
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 Primary
20170329:02:31:37:025522 gpstart:mdw:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 Mirror
Continue with Greenplum instance startup Yy|Nn (default=N):
> y
20170329:02:31:39:025522 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
............
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-Process results...
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 6
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-Successfully started 6 of 6 segment instances
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20170329:02:31:51:025522 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /gpmaster50/gpseg-1
20170329:02:31:52:025522 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active
20170329:02:31:52:025522 gpstart:mdw:gpadmin-[WARNING]:-global name 'testurl' is not defined
20170329:02:31:52:025522 gpstart:mdw:gpadmin-[INFO]:-No standby master configured. skipping...
20170329:02:31:52:025522 gpstart:mdw:gpadmin-[INFO]:-Check status of database with gpstate utility
[gpadmin@mdw ~]$
5、重新恢复maststandby主节点
[gpadmin@mdw ~]$ gpstate -f
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev'
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2017 20:09:54'
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-Standby master instance not configured
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:-No entries found.
20170329:02:36:47:025717 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
[gpadmin@mdw ~]$ gpinitstandby -s mdw02
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Checking for filespace directory /gpmaster50/gpseg-1 on mdw02
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master initialization parameters
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master hostname = mdw
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master data directory = /gpmaster50/gpseg-1
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master port = 5432
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master hostname = mdw02
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master port = 5432
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master data directory = /gpmaster50/gpseg-1
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum update system catalog = On
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:- Filespace locations
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20170329:02:37:04:025779 gpinitstandby:mdw:gpadmin-[INFO]:-pg_system -> /gpmaster50/gpseg-1
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[WARNING]:-Syncing of Greenplum Database extensions has failed.
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[WARNING]:-Please run gppkg --clean after successful standby initialization.
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Adding standby master to catalog...
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Database catalog updated successfully.
20170329:02:37:06:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Updating pg_hba.conf file...
20170329:02:37:08:025779 gpinitstandby:mdw:gpadmin-[INFO]:-pg_hba.conf files updated successfully.
20170329:02:37:17:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Updating filespace flat files...
20170329:02:37:17:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Filespace flat file updated successfully.
20170329:02:37:17:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master
20170329:02:37:17:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: mdw02 in directory: /gpmaster50/gpseg-1
20170329:02:37:22:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files...
20170329:02:37:23:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.
20170329:02:37:23:025779 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully created standby master on mdw02
[gpadmin@mdw ~]$ gpstate -f
20170329:02:37:27:025887 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20170329:02:37:28:025887 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev'
20170329:02:37:28:025887 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2017 20:09:54'
20170329:02:37:28:025887 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:-----------------------
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:- Standby address = mdw02
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /gpmaster50/gpseg-1
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 2641
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/24000000
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/24000000
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/24000000
20170329:02:37:29:025887 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
[gpadmin@mdw ~]$
阅读(13022) | 评论(0) | 转发(0) |
