ORA-15041，ORA-01274 故障解决实例

和客户沟通了下，他们打算把DG停掉，暂时不用了，因此也没有做进一步处理，如果不想让归档日志传递到备库，可以设置主库参数log_

上周五去客户处巡检时，发现原来配置的DG备库未与主库同步，特此记录一下

在5月14日13：58出现故障，3398日志无法应用，以下是alert日志

Thu May 14 13:58:01 2015
Errors in file /u01/app/Oracle/diag/rdbms/tcdg/tc/trace/tc_pr00_3473760.trc:
ORA-01119: error in creating database file '+data'
ORA-17502: ksfdcre:4 Failed to create file +data
ORA-15041: diskgroup "DATA" space exhausted
File #108 added to control file as 'UNNAMED00108'.
Originally created as:
'+DATA/tc/datafile/oa2015.387.879688649'
Recovery was unable to create the file as:
'+data'
MRP0: Background Media Recovery terminated with error 1274
Errors in file /u01/app/oracle/diag/rdbms/tcdg/tc/trace/tc_pr00_3473760.trc:
ORA-01274: cannot add datafile '+DATA/tc/datafile/oa2015.387.879688649' - file could not be created
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 10293976250522
Thu May 14 13:58:06 2015
MRP0: Background Media Recovery process shutdown (tc)

由于空间不足，出现了ORA-15041错误，然后MRP0进程被终止

再来看trace文件内容：

*** 2015-05-14 13:58:01.562
----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
----- START DDE Actions Dump -----
Executing SYNC actions
Executing ASYNC actions
----- END DDE Actions Dump (total 0 csec) -----
ORA-01119: error in creating database file '+data'
ORA-17502: ksfdcre:4 Failed to create file +data
ORA-15041: diskgroup "DATA" space exhausted
File #108 added to control file as 'UNNAMED00108'.
Originally created as:
'+DATA/tc/datafile/oa2015.387.879688649'
Recovery was unable to create the file as:
'+data'
*** 2015-05-14 13:58:01.598 4329 krsh.c
MRP0: Background Media Recovery terminated with error 1274
ORA-01274: cannot add datafile '+DATA/tc/datafile/oa2015.387.879688649' - file could not be created
*** 2015-05-14 13:58:01.613 4329 krsh.c
Managed Standby Recovery not using Real Time Apply

*** 2015-05-14 13:58:01.616
MRP: Prodding archiver at standby for thread 1 seq 3398
----- Redo read statistics for thread 1 -----
Read rate (ASYNC): 1631875714Kb in 8455733.63s => 0.19 Mb/sec
Total redo bytes: 1631903362Kb Longest record: 53Kb, moves: 546631/3909060988 moved: 2869Mb (0%)
Longest LWN: 61559Kb, reads: 53311640
Last redo scn: 0x095c.c0cc389a (10293976250522)
Change vector header moves = 505014709/3028421495 (1%)
----------------------------------------------

*** 2015-05-14 13:58:01.626
Media Recovery drop redo thread 1
KCBR: Redo cache copies/changes = 7576594/7576586
Wait to push change maps to slaves = 136500s

*** 2015-05-14 13:58:02.545
Completed Media Recovery
Checking to start in-flux buffer recovery from SCN 2396.3234383719 to SCN  (non-inclusive) 2396.3234609306
Influx recovery found in-flux buffers

*** 2015-05-14 13:58:02.622
Influx Media Recovery add redo thread 1
Managed Standby Recovery: Standby online log for thr 1  seq 3397 not found. Looking whether archived..
Looking for archived log thr 1 seq 3397 recovery branch id 869789191 that contains SCN 10293976024935
Checking whether scn 10293976024935 in los 10293976024935 and nxs 10293976125751
Managed Standby Recovery: Opening archived log /oraarch/ARC_CRM5_3397_869789191_1.log during invocation of recoverable recovery

*** 2015-05-14 13:58:03.937
Resized overflow buffer to 2435K (for 2435K LWN)
Resized overflow buffer to 3923K (for 3923K LWN)
Resized overflow buffer to 14242K (for 14242K LWN)

*** 2015-05-14 13:58:06.105
MRP: Prodding archiver at standby for thread 1 seq 3398
Managed Recovery: Not Active posted.

由于redo apply的进程被终止，因此应用到3397归档后，之后的日志都未被应用，但是归档日志都已经顺利传到了备库

SQL> select thread#,sequence#，applied from v$archived_log where sequence#>3391;

  THREAD#  SEQUENCE# APPLIED
---------- ---------- ---------
        1      3392 YES
        1      3393 YES
        1      3394 YES
        1      3395 YES
        1      3396 YES
        1      3397 YES
        1      3398 NO
        1      3399 NO
        1      3400 NO
        1      3401 NO
        1      3402 NO
        1      3403 NO
        1      3404 NO
        1      3405 NO
        1      3406 NO
        1      3407 NO
        1      3408 NO
        1      3409 NO
        1      3410 NO
        1      3411 NO
        1      3412 NO
        1      3413 NO

还可以通过下面的SQL语句去分析备库落后了多少归档日志未完成应用。