一套数据库aud$无法在system 表空间中扩展,奇怪。
每天监控都没事,怎么突然就满了?
看看表空间使用率,虽然当前100%,但是system由两个文件组成,通常都是可以自动扩展的。
两个最大应该能扩到60G,
当前才15G,难道遇到bug了?
检查一下数据文件
-
SQL> select name from v$dbfile;
-
-
NAME
-
--------------------------------------------------------------------------------
-
/home/ora/oracle/oradata/system.dbf
-
/home/ora/oracle/oradata/sysaux.dbf
-
/home/ora/oracle/oradata/undotbs1.dbf
-
/home/ora/oracle/oradata/undotbs2.dbf
-
/home/ora/oracle/oradata/system02.dbf
-
-
。。。
看着很正常。
等等,这是RAC,怎么会放到/home下?
-
SQL> !ls -l /home/ora/oracle/oradata/sysaux.dbf
-
lrwxrwxrwx 1 oracle oinstall 13 9月 22 2016 /home/ora/oracle/oradata/sysaux.dbf -> /dev/raw/raw6
果然,是个链接,那么raw6到底多大呢?哎呀,裸设备,你想到答案了吗?
没有的话就先确定一个问题:raw6的大小到底是多少呢?
直接的方法是
blockdev --getsize /dev/raw/raw6
然后将结果*512/1024/1024换算成MB。
如何绑定的呢?(开始绕弯)
习惯上,通常会用udev管理oracle的共享文件
-
cd /etc/udev/rules.d/
-
ls -lt
-
总用量 64
-
-rw-r--r-- 1 root root 310 10月 29 2019 80-udev-pp-xio.rules
-
-rw-r--r-- 1 root root 11322 9月 27 2016 60-raw.rules
-
-rw-r--r--. 1 root root 786 8月 2 2016 70-persistent-cd.rules
-
-rw-r--r--. 1 root root 907 8月 2 2016 70-persistent-net.rules
-
-rw-r--r-- 1 root root 76 6月 9 2016 05-udev-pp.rules
看看内容
-
$ cat 60-raw.rules
-
# Enter raw device bindings here.
-
#
-
# An example would be:
-
# ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
-
# to bind /dev/raw/raw1 to /dev/sda, or
-
# ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/bin/raw /dev/raw/raw2 %M %m"
-
# to bind /dev/raw/raw2 to the device with major 8, minor 1.
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="5",KERNEL=="lv_ocr1",RUN+="/bin/raw /dev/raw/raw1 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="6",KERNEL=="lv_ocr2",RUN+="/bin/raw /dev/raw/raw2 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="7",KERNEL=="lv_ocr3",RUN+="/bin/raw /dev/raw/raw3 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="8",KERNEL=="lv_vote",RUN+="/bin/raw /dev/raw/raw4 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="9",KERNEL=="system",RUN+="/bin/raw /dev/raw/raw5 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="10",KERNEL=="sysaux",RUN+="/bin/raw /dev/raw/raw6 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="11",KERNEL=="undotbs1",RUN+="/bin/raw /dev/raw/raw7 %M %m"
-
ACTION=="add", ENV{MAJOR}=="253", ENV{MINOR}=="12",KERNEL=="temp",RUN+="/bin/raw /dev/raw/raw8 %M %m"
-
省略...
-
ACTION=="add", KERNEL=="raw1", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
ACTION=="add", KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
ACTION=="add", KERNEL=="raw3", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
ACTION=="add", KERNEL=="raw4", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
ACTION=="add", KERNEL=="raw5", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
ACTION=="add", KERNEL=="raw6", OWNER="oracle", GROUP="oinstall", MODE="0660"
-
省略...
这里先将绑定裸设备,然后授权给oracle。
然后这些raw怎么给数据库用呢?
只能看看rc.local
-
$ cat /etc/rc.local
-
#!/bin/sh
-
#
-
# This script will be executed *after* all the other init scripts.
-
# You can put your own initialization stuff in here if you don't
-
# want to do the full Sys V style init stuff.
-
su root -c "/usr/rav/ravservice -R"
-
-
touch /var/lock/subsys/local
-
raw /dev/raw/raw1 /dev/datavg/lv_ocr1
-
raw /dev/raw/raw2 /dev/datavg/lv_ocr2
-
raw /dev/raw/raw3 /dev/datavg/lv_ocr3
-
raw /dev/raw/raw4 /dev/datavg/lv_vote
-
raw /dev/raw/raw5 /dev/datavg/system
-
raw /dev/raw/raw6 /dev/datavg/sysaux
-
raw /dev/raw/raw7 /dev/datavg/undotbs1
-
raw /dev/raw/raw8 /dev/datavg/temp
-
raw /dev/raw/raw9 /dev/datavg/control1
-
raw /dev/raw/raw10 /dev/datavg/control2
-
raw /dev/raw/raw11 /dev/datavg/control3
-
raw /dev/raw/raw12 /dev/datavg/spfile
-
raw /dev/raw/raw13 /dev/datavg/redo1_1
-
raw /dev/raw/raw14 /dev/datavg/redo1_2
-
raw /dev/raw/raw15 /dev/datavg/redo1_3
-
raw /dev/raw/raw16 /dev/datavg/redo1_4
-
raw /dev/raw/raw17 /dev/datavg/redo2_1
-
raw /dev/raw/raw61 /dev/datavg/system02
-
省略...
-
chown oracle:oinstall /dev/raw/raw[1-61]
-
chmod 660 /dev/raw/raw[1-61]
-
[oracle@XKF-DB etc]$ ls -l /dev/datavg/system
-
lrwxrwxrwx 1 root root 7 3月 23 11:24 /dev/datavg/system -> ../dm-9
在rc.local里将raw绑定成 dev/datavg/下的设备,绕弯
这些设备对应的其实是个链接,链接到/dev/dm-9,又绕弯
这时可用以下命令观察
-
$ lsblk
-
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
-
sda 8:0 0 835.4G 0 disk
-
├─sda1 8:1 0 1G 0 part /boot
-
├─sda2 8:2 0 785.6G 0 part
-
│ ├─rootvg-rootlv (dm-0) 253:0 0 195.3G 0 lvm /
-
│ ├─rootvg-homelv (dm-1) 253:1 0 195.3G 0 lvm /home
-
│ ├─rootvg-varlv (dm-2) 253:2 0 48.8G 0 lvm /var
-
│ ├─rootvg-usrlv (dm-3) 253:3 0 48.8G 0 lvm /usr
-
│ └─rootvg-tmplv (dm-4) 253:4 0 48.8G 0 lvm /tmp
-
└─sda3 8:3 0 48.8G 0 part [SWAP]
-
sdb 8:16 0 2.8M 1 disk
-
sdl 8:176 0 2.8M 1 disk
-
sdv 65:80 0 2.8M 1 disk
-
sdaf 65:240 0 2.8M 1 disk
-
sdo 8:224 0 2G 0 disk
-
sdab 65:176 0 67.4G 0 disk
-
sdh 8:112 0 67.4G 0 disk
-
sdah 66:16 0 2G 0 disk
-
sdk 8:160 0 67.4G 0 disk
-
sdan 66:112 0 67.4G 0 disk
-
sdao 66:128 0 67.4G 0 disk
-
sr0 11:0 1 1024M 0 rom
-
emcpowera 120:0 0 2G 0 disk
-
├─datavg-lv_ocr1 (dm-5) 253:5 0 1G 0 lvm
-
├─datavg-lv_ocr2 (dm-6) 253:6 0 1G 0 lvm
-
└─datavg-spfile (dm-16) 253:16 0 12M 0 lvm
-
emcpowerb 120:16 0 2G 0 disk
-
├─datavg-lv_vote (dm-8) 253:8 0 1G 0 lvm
-
├─datavg-pwdfile (dm-25) 253:25 0 12M 0 lvm
-
└─datavg-lv_ocr3 (dm-7) 253:7 0 1G 0 lvm
-
emcpowerc 120:32 0 2G 0 disk
-
└─datavg-temp (dm-12) 253:12 0 2G 0 lvm
-
emcpowerd 120:48 0 2G 0 disk
-
├─datavg-control2 (dm-14) 253:14 0 1G 0 lvm
-
└─datavg-control1 (dm-13) 253:13 0 1G 0 lvm
-
emcpowere 120:64 0 67.4G 0 disk
-
├─datavg-ccdb29 (dm-57) 253:57 0 1G 0 lvm
-
├─datavg-ccdb15 (dm-43) 253:43 0 1G 0 lvm
-
├─datavg-users (dm-26) 253:26 0 1G 0 lvm
-
├─datavg-ccdb26 (dm-54) 253:54 0 1G 0 lvm
-
├─datavg-ccdb11 (dm-39) 253:39 0 1G 0 lvm
-
├─datavg-ccdb22 (dm-50) 253:50 0 1G 0 lvm
-
├─datavg-ccdb24 (dm-52) 253:52 0 1G 0 lvm
-
├─datavg-ccdb25 (dm-53) 253:53 0 1G 0 lvm
-
├─datavg-system (dm-9) 253:9 0 10G 0 lvm
-
├─datavg-redo2_3 (dm-23) 253:23 0 200M 0 lvm
-
├─datavg-redo2_4 (dm-24) 253:24 0 200M 0 lvm
-
├─datavg-undotbs2 (dm-27) 253:27 0 5G 0 lvm
-
├─datavg-ccdb28 (dm-56) 253:56 0 1G 0 lvm
-
├─datavg-ccdb01 (dm-29) 253:29 0 20G 0 lvm
-
├─datavg-sysaux (dm-10) 253:10 0 5G 0 lvm
-
├─datavg-control3 (dm-15) 253:15 0 1G 0 lvm
-
├─datavg-ccdb18 (dm-46) 253:46 0 1G 0 lvm
-
├─datavg-ccdb19 (dm-47) 253:47 0 1G 0 lvm
-
└─datavg-redo1_4 (dm-20) 253:20 0 200M 0 lvm
-
emcpowerf 120:80 0 67.4G 0 disk
-
├─datavg-system02 (dm-65) 253:65 0 5G 0 lvm
-
├─datavg-ccdb (dm-28) 253:28 0 20G 0 lvm
-
├─datavg-ccdb02 (dm-30) 253:30 0 20G 0 lvm
-
├─datavg-ccdb03 (dm-31) 253:31 0 20G 0 lvm
-
└─datavg-system (dm-9) 253:9 0 10G 0 lvm
-
emcpowerg 120:96 0 67.4G 0 disk
-
├─datavg-webchat01 (dm-58) 253:58 0 20G 0 lvm
-
├─datavg-ccdb04 (dm-32) 253:32 0 20G 0 lvm
-
├─datavg-ccdb05 (dm-33) 253:33 0 100M 0 lvm
-
└─datavg-ccdb06 (dm-34) 253:34 0 100M 0 lvm
-
emcpowerh 120:112 0 67.4G 0 disk
-
├─datavg-webchat02 (dm-59) 253:59 0 20G 0 lvm
-
├─datavg-ccdb08 (dm-36) 253:36 0 100M 0 lvm
-
├─datavg-ccdb07 (dm-35) 253:35 0 100M 0 lvm
-
└─datavg-ccdb09 (dm-37) 253:37 0 100M 0 lvm
-
emcpoweri 120:128 0 67.4G 0 disk
-
└─datavg-webchat03 (dm-60) 253:60 0 20G 0 lvm
-
sdap 66:144 0 5.6M 1 disk
-
sdaq 66:160 0 67.4G 0 disk
-
sdar 66:176 0 67.4G 0 disk
看着还有emcpower的绑定,这是什么作用?已经被绕晕了。
先不管了(坐等迁移),回看问题源头吧。
通过这里可以看到dm-9大小只有10g,但数据文件设置了自动扩展,上限是32G(谁挖的坑?),
因此监控按最大扩展没发现不足的情况。
由于裸设备不能扩展,写入10g数据后就写不进去了,才出现表空间不足的告警,影响到业务使
用。检查了一遍36个RAC(不管裸不裸),就这个文件是自动扩展的,悲催。
阅读(855) | 评论(0) | 转发(0) |