RMAN failed becaue of ORA-19502.

RMAN failed becaue of ORA-19502.

分类： English ORACLE 数据库 2010-12-29 10:38 167人阅读 评论(0) 举报

RMAN failed because the following errors from rman log.

channel ORA_DISK_2: starting piece 1 at DEC-28-2010 12:15:14
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/28/2010 12:23:39
ORA-19502: write error on file "/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_oom0mi2h_1_1.bak", blockno 4403201 (blocksize=512
)
ORA-27063: skgfospo: number of bytes read/written is incorrect
Additional information: 597504
Additional information: 1048576

It is because there is no diskspace to hold the backup pieces.

we have set the retention window for two days.

RMAN> show all;

using target database controlfile instead of recovery catalog
RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 2 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_%F.bak';
CONFIGURE DEVICE TYPE DISK PARALLELISM 2;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   '/global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_%U.bak' MAXPIECESIZE 8000 M;
CONFIGURE MAXSETSIZE TO UNLIMITED;

It means the diskspace should be able hold at least three days backup pieces.

use the df -h to see how much disk usage for the two days backups.

The value is 78% for the backup filesystem.

This is the rootcause of the failure.

Let us see the details:

RMAN> list expired backup;

RMAN> report need backup redundancy =2;

Report of files with less than 2 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------

RMAN>  report need backup redundancy =1;

Report of files with less than 1 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------

RMAN> report unrecoverable;

Report of files that need backup due to unrecoverable operations
File Type of Backup Required Name
---- ----------------------- -----------------------------------

RMAN> report obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 2 days
no obsolete backups found
RMAN> Crosscheck backup;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=770 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=820 devtype=DISK
。。。。。。。。。。。。。。。。。。。。。
Crosschecked 8 objects?

RMAN> Crosscheck archivelog all;

released channel: ORA_DISK_1
released channel: ORA_DISK_2
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=770 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=820 devtype=DISK
........................................
Crosschecked 28 objects
RMAN>  list expired backup;
List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15090   Full    4G         DISK        00:17:48     26-DEC-10     
        BP Key: 15090   Status: EXPIRED   Tag: TAG20101226T001503
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_ntm0fv47_1_1.bak
  List of Datafiles in backup set 15090
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  1       Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/system01.dbf
  4       Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tools01.dbf
  6       Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/xdb01.dbf
  7       Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/am_data01.dbf
  10      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/cp_inventory_data01.dbf
  13      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/pm_data01.dbf
  18      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/account_data01.dbf
  21      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/wiki01.dbf
  23      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/cp_inventory_data02.dbf
  24      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tis_data01.dbf
  25      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/xdb02.dbf
  26      Full 8467276111641 26-DEC-10 /global/oracle2/oradata2/CPPRD/tools02.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15091   Full    5G         DISK        00:22:33     26-DEC-10     
        BP Key: 15091   Status: EXPIRED   Tag: TAG20101226T001503
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_num0fv48_1_1.bak
  List of Datafiles in backup set 15091
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  2       Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/undotbs01.dbf
  3       Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/drsys01.dbf
  5       Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/users01.dbf
  8       Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/cta_data01.dbf
  9       Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/heartbeat_data01.dbf
  11      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/jira_data01.dbf
  12      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/mast_data01.dbf
  14      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/services_data01.dbf
  15      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/spyglass_data01.dbf
  17      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/pwd_data01.dbf
  19      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/hr_data01.dbf
  22      Full 8467276111644 26-DEC-10 /global/oracle2/oradata2/CPPRD/smc_mv_logs01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15092   Full    2G         DISK        00:11:18     26-DEC-10     
        BP Key: 15092   Status: EXPIRED   Tag: TAG20101226T001503
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_nvm0g05s_1_1.bak
  List of Datafiles in backup set 15092
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  20      Full 8467277165500 26-DEC-10 /global/oracle2/oradata2/CPPRD/webadmin_data02.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15093   Full    3G         DISK        00:12:09     26-DEC-10     
        BP Key: 15093   Status: EXPIRED   Tag: TAG20101226T001503
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o0m0g0ep_1_1.bak
  List of Datafiles in backup set 15093
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  16      Full 8467277516399 26-DEC-10 /global/oracle2/oradata2/CPPRD/webadmin_data01.dbf

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15094   2G         DISK        00:11:08     26-DEC-10     
        BP Key: 15094   Status: EXPIRED   Tag: TAG20101226T004957
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o2m0g15n_1_1.bak

  List of Archived Logs in backup set 15094
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    157407  8467254043607 25-DEC-10 8467255012759 25-DEC-10
  1    157408  8467255012759 25-DEC-10 8467255926649 25-DEC-10
  1    157409  8467255926649 25-DEC-10 8467256945794 25-DEC-10
  1    157410  8467256945794 25-DEC-10 8467257864568 25-DEC-10
  1    157411  8467257864568 25-DEC-10 8467258803216 25-DEC-10
  1    157412  8467258803216 25-DEC-10 8467259738992 25-DEC-10
  1    157413  8467259738992 25-DEC-10 8467260647589 25-DEC-10
  1    157414  8467260647589 25-DEC-10 8467261607633 25-DEC-10
  1    157415  8467261607633 25-DEC-10 8467262589366 25-DEC-10
  1    157416  8467262589366 25-DEC-10 8467263550238 25-DEC-10
  1    157417  8467263550238 25-DEC-10 8467264461024 25-DEC-10
  1    157418  8467264461024 25-DEC-10 8467265400438 25-DEC-10
  1    157419  8467265400438 25-DEC-10 8467266175528 25-DEC-10
  1    157420  8467266175528 25-DEC-10 8467266249095 25-DEC-10
  1    157421  8467266249095 25-DEC-10 8467266418398 25-DEC-10
  1    157422  8467266418398 25-DEC-10 8467266634369 25-DEC-10
  1    157423  8467266634369 25-DEC-10 8467267638357 25-DEC-10
  1    157424  8467267638357 25-DEC-10 8467268556163 25-DEC-10
  1    157425  8467268556163 25-DEC-10 8467269487701 25-DEC-10
  1    157426  8467269487701 25-DEC-10 8467270430555 25-DEC-10
  1    157427  8467270430555 25-DEC-10 8467271367767 25-DEC-10
  1    157428  8467271367767 25-DEC-10 8467272699523 25-DEC-10
  1    157429  8467272699523 25-DEC-10 8467274168819 25-DEC-10
  1    157430  8467274168819 25-DEC-10 8467275028116 25-DEC-10
  1    157431  8467275028116 25-DEC-10 8467275974031 26-DEC-10
  1    157432  8467275974031 26-DEC-10 8467277062422 26-DEC-10
  1    157433  8467277062422 26-DEC-10 8467278130963 26-DEC-10
  1    157434  8467278130963 26-DEC-10 8467278257188 26-DEC-10

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15095   3G         DISK        00:13:22     26-DEC-10     
        BP Key: 15095   Status: EXPIRED   Tag: TAG20101226T121506
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o4m0h9ad_1_1.bak

  List of Archived Logs in backup set 15095
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    157446  8467288682614 26-DEC-10 8467289696576 26-DEC-10
  1    157447  8467289696576 26-DEC-10 8467290634240 26-DEC-10
  1    157448  8467290634240 26-DEC-10 8467291563792 26-DEC-10
  1    157449  8467291563792 26-DEC-10 8467292527431 26-DEC-10
  1    157450  8467292527431 26-DEC-10 8467293403743 26-DEC-10
  1    157451  8467293403743 26-DEC-10 8467294423132 26-DEC-10
  1    157452  8467294423132 26-DEC-10 8467295335621 26-DEC-10
  1    157453  8467295335621 26-DEC-10 8467296379866 26-DEC-10
  1    157454  8467296379866 26-DEC-10 8467297359389 26-DEC-10
  1    157455  8467297359389 26-DEC-10 8467298299591 26-DEC-10
  1    157456  8467298299591 26-DEC-10 8467299218934 26-DEC-10
  1    157457  8467299218934 26-DEC-10 8467300153869 26-DEC-10
  1    157458  8467300153869 26-DEC-10 8467301148816 26-DEC-10
  1    157459  8467301148816 26-DEC-10 8467302062402 26-DEC-10
  1    157460  8467302062402 26-DEC-10 8467303020155 26-DEC-10
  1    157461  8467303020155 26-DEC-10 8467303968319 26-DEC-10
  1    157462  8467303968319 26-DEC-10 8467304868044 26-DEC-10
  1    157463  8467304868044 26-DEC-10 8467305793065 26-DEC-10
  1    157464  8467305793065 26-DEC-10 8467306751786 26-DEC-10
  1    157465  8467306751786 26-DEC-10 8467307706866 26-DEC-10
  1    157466  8467307706866 26-DEC-10 8467308609570 26-DEC-10
  1    157467  8467308609570 26-DEC-10 8467309525068 26-DEC-10
  1    157468  8467309525068 26-DEC-10 8467310472359 26-DEC-10
  1    157469  8467310472359 26-DEC-10 8467311417658 26-DEC-10
  1    157470  8467311417658 26-DEC-10 8467312423646 26-DEC-10
  1    157471  8467312423646 26-DEC-10 8467313332226 26-DEC-10
  1    157472  8467313332226 26-DEC-10 8467314268393 26-DEC-10
  1    157473  8467314268393 26-DEC-10 8467315181441 26-DEC-10
  1    157474  8467315181441 26-DEC-10 8467316146094 26-DEC-10
  1    157475  8467316146094 26-DEC-10 8467317109806 26-DEC-10
  1    157476  8467317109806 26-DEC-10 8467318032098 26-DEC-10
  1    157477  8467318032098 26-DEC-10 8467319034761 26-DEC-10
  1    157478  8467319034761 26-DEC-10 8467319961122 26-DEC-10
  1    157479  8467319961122 26-DEC-10 8467320826074 26-DEC-10
  1    157480  8467320826074 26-DEC-10 8467321752728 26-DEC-10
  1    157481  8467321752728 26-DEC-10 8467322712789 26-DEC-10
  1    157482  8467322712789 26-DEC-10 8467323647244 26-DEC-10
  1    157483  8467323647244 26-DEC-10 8467324631265 26-DEC-10
  1    157484  8467324631265 26-DEC-10 8467324875108 26-DEC-10

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15096   3G         DISK        00:13:39     26-DEC-10     
        BP Key: 15096   Status: EXPIRED   Tag: TAG20101226T121506
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o3m0h9ad_1_1.bak

  List of Archived Logs in backup set 15096
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    157378  8467226164917 25-DEC-10 8467227143624 25-DEC-10
  1    157379  8467227143624 25-DEC-10 8467228142839 25-DEC-10
  1    157380  8467228142839 25-DEC-10 8467229106571 25-DEC-10
  1    157381  8467229106571 25-DEC-10 8467230041506 25-DEC-10
  1    157382  8467230041506 25-DEC-10 8467231042828 25-DEC-10
  1    157383  8467231042828 25-DEC-10 8467231985039 25-DEC-10
  1    157384  8467231985039 25-DEC-10 8467232954305 25-DEC-10
  1    157385  8467232954305 25-DEC-10 8467233913862 25-DEC-10
  1    157386  8467233913862 25-DEC-10 8467234866135 25-DEC-10
  1    157387  8467234866135 25-DEC-10 8467235811031 25-DEC-10
  1    157388  8467235811031 25-DEC-10 8467236720854 25-DEC-10
  1    157389  8467236720854 25-DEC-10 8467237680501 25-DEC-10
  1    157390  8467237680501 25-DEC-10 8467238649502 25-DEC-10
  1    157391  8467238649502 25-DEC-10 8467239555952 25-DEC-10
  1    157392  8467239555952 25-DEC-10 8467240548035 25-DEC-10
  1    157393  8467240548035 25-DEC-10 8467241457669 25-DEC-10
  1    157394  8467241457669 25-DEC-10 8467242451248 25-DEC-10
  1    157395  8467242451248 25-DEC-10 8467243407665 25-DEC-10
  1    157396  8467243407665 25-DEC-10 8467244349365 25-DEC-10
  1    157397  8467244349365 25-DEC-10 8467245248127 25-DEC-10
  1    157398  8467245248127 25-DEC-10 8467246246776 25-DEC-10
  1    157399  8467246246776 25-DEC-10 8467247347199 25-DEC-10
  1    157400  8467247347199 25-DEC-10 8467248252623 25-DEC-10
  1    157401  8467248252623 25-DEC-10 8467249257454 25-DEC-10
  1    157402  8467249257454 25-DEC-10 8467250187423 25-DEC-10
  1    157403  8467250187423 25-DEC-10 8467251166829 25-DEC-10
  1    157404  8467251166829 25-DEC-10 8467252147525 25-DEC-10
  1    157405  8467252147525 25-DEC-10 8467253093732 25-DEC-10
  1    157406  8467253093732 25-DEC-10 8467254043607 25-DEC-10
  1    157435  8467278257188 26-DEC-10 8467279175606 26-DEC-10
  1    157436  8467279175606 26-DEC-10 8467280093538 26-DEC-10
  1    157437  8467280093538 26-DEC-10 8467281078581 26-DEC-10
  1    157438  8467281078581 26-DEC-10 8467281995278 26-DEC-10
  1    157439  8467281995278 26-DEC-10 8467283023767 26-DEC-10
  1    157440  8467283023767 26-DEC-10 8467283929267 26-DEC-10
  1    157441  8467283929267 26-DEC-10 8467284919113 26-DEC-10
  1    157442  8467284919113 26-DEC-10 8467285935472 26-DEC-10
  1    157443  8467285935472 26-DEC-10 8467286842785 26-DEC-10
  1    157444  8467286842785 26-DEC-10 8467287756726 26-DEC-10
  1    157445  8467287756726 26-DEC-10 8467288682614 26-DEC-10

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
15097   Full    2M         DISK        00:00:02     26-DEC-10     
        BP Key: 15097   Status: EXPIRED   Tag:
        Piece Name: /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_c-777430827-20101226-00.bak
  SPFILE Included: Modification time: 10-DEC-08

RMAN> report need backup redundancy =1;

Report of files with less than 1 redundant backups
File #bkps Name
---- ----- -----------------------------------------------------
RMAN>  report unrecoverable;
Report of files that need backup due to unrecoverable operations
File Type of Backup Required Name
---- ----------------------- -----------------------------------
RMAN> report obsolete;
RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 2 days
Report of obsolete backups and copies
Type                 Key    Completion Time    Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set           15090  26-DEC-10        
  Backup Piece       15090  26-DEC-10          /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_ntm0fv47_1_1.bak
Backup Set           15091  26-DEC-10        
  Backup Piece       15091  26-DEC-10          /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_num0fv48_1_1.bak
Backup Set           15092  26-DEC-10        
  Backup Piece       15092  26-DEC-10          /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_nvm0g05s_1_1.bak
Backup Set           15093  26-DEC-10        
  Backup Piece       15093  26-DEC-10          /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_db_o0m0g0ep_1_1.bak
Backup Set           15097  26-DEC-10        
  Backup Piece       15097  26-DEC-10          /global/oracle4/ORADUMP/CPPRD/backup/CPPRD_rman_ctrl_c-777430827-20101226-00.bak
The 26th backup is expired but is not removed correctly.

It take the diskspace then the new backup can not generate because there is no enough diskspace to hold backup pieces.

It sounds simple?

but what cause the 26th backup stay there after being expired?