1.4TBASM(RAC)磁盘损坏恢复小记
这周折腾了2天的时间帮客户成功恢复了一套近1.4TB的10.2.0.5 RAC(ASM). 该库在3月4号直接crash了。 大家可以看到，该库在开始报错读取redo,controlfile报错,本质原因是DISKGROUP dismount了,信息如下： Tue Mar 04 18:09:59 CST 2014 Errors in file /home/o
这周折腾了2天的时间帮客户成功恢复了一套近1.4TB的10.2.0.5 RAC(ASM). 该库在3月4号直接crash了。
大家可以看到，该库在开始报错读取redo,controlfile报错,本质原因是DISKGROUP dismount了,信息如下：
Tue Mar 04 18:09:59 CST 2014
<code class="php plain">Errors in file /home/oraprod/10.2.0/db/admin/xxxx/bdump/xxxx_lgwr_15943.trc:
<code class="php plain">ORA-00345: redo log write error block 68145
<code class="php functions">count <code class="php plain">5
<code class="php plain">ORA-00312: online log 6 thread 2:
<code class="php string">'+DATA/xxxx/onlinelog/o2_t2_redo3.log'
<code class="php plain">ORA-15078: ASM diskgroup was forcibly dismounted
<code class="php plain">Tue Mar 04 18:09:59 CST 2014
<code class="php plain">SUCCESS: diskgroup DATA was dismounted
<code class="php plain">SUCCESS: diskgroup DATA was dismounted
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<code class="php plain">Errors in file /home/oraprod/10.2.0/db/admin/xxxx/bdump/xxxx_lmon_15892.trc:
<code class="php plain">ORA-00202: control file:
<code class="php string">'+DATA/xxxx/controlfile/o1_mf_4g1zr1yo_.ctl'
<code class="php plain">ORA-15078: ASM diskgroup was forcibly dismounted
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<code class="php plain">KCF: write/open error block=0x1f41e online=1
<code class="php spaces"><code class="php plain">file=31 +DATA/xxxx/datafile/apps_ts_queues.310.692585175
<code class="php spaces"><code class="php plain">error=15078 txt:
<code class="php string">''
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<code class="php plain">KCF: write/open error block=0x47d5d online=1
<code class="php spaces"><code class="php plain">file=51 +DATA/xxx/datafile/apps_ts_tx_data.353.692593409
<code class="php spaces"><code class="php plain">error=15078 txt:
<code class="php string">''
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<code class="php plain">Errors in file /home/oraprod/10.2.0/db/admin/xxxx/bdump/xxxx_dbw2_15939.trc:
<code class="php plain">ORA-00202: control file:
<code class="php string">'+DATA/prod/controlfile/o1_mf_4g1zr1yo_.ctl'
<code class="php plain">ORA-15078: ASM diskgroup was forcibly dismounted
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<code class="php plain">KCF: write/open error block=0x47d5b online=1
<code class="php spaces"><code class="php plain">file=51 +DATA/prod/datafile/apps_ts_tx_data.353.692593409
<code class="php spaces"><code class="php plain">error=15078 txt:
<code class="php string">''
<code class="php plain">Tue Mar 04 18:10:00 CST 2014
<p>数据库实例挂了之后，我们来看下ASM实例的alert log信息，如下：</p>
<code class="php plain">Tue Mar 04 18:10:04 CST 2014
<code class="php plain">NOTE: SMON starting instance recovery
<code class="php keyword">for <code class="php plain">group 1 (mounted)
<code class="php plain">Tue Mar 04 18:10:04 CST 2014
<code class="php plain">WARNING: IO Failed. au:0 diskname:/dev/raw/raw5
<code class="php spaces"><code class="php plain">rq:0x200000000207b518 buffer:0x200000000235c600 au_offset(bytes):0 iosz:4096 operation:0
<code class="php spaces"><code class="php plain">status:2
<code class="php plain">WARNING: IO Failed. au:0 diskname:/dev/raw/raw5
<code class="php spaces"><code class="php plain">rq:0x200000000207b518 buffer:0x200000000235c600 au_offset(bytes):0 iosz:4096 operation:0
<code class="php spaces"><code class="php plain">status:2
<code class="php plain">NOTE: F1X0 found on disk 0 fcn 0.160230519
<code class="php plain">WARNING: IO Failed. au:33 diskname:/dev/raw/raw5
<code class="php spaces"><code class="php plain">rq:0x60000000002d64f0 buffer:0x400405df000 au_offset(bytes):0 iosz:4096 operation:0
<code class="php spaces"><code class="php plain">status:2
<code class="php plain">WARNING: cache failed to read gn 1 fn 3 blk 10752
<code class="php functions">count <code class="php plain">1 from disk 2
<code class="php plain">ERROR: cache failed to read fn=3 blk=10752 from disk(s): 2
<code class="php plain">ORA-15081: failed to submit an I/O operation to a disk
<code class="php plain">NOTE: cache initiating offline of disk 2 group 1
<code class="php plain">WARNING: process 12863 initiating offline of disk 2.2526420198 (DATA_0002) with mask 0x3 in group 1
<code class="php plain">NOTE: PST update: grp = 1, dsk = 2, mode = 0x6
<code class="php plain">Tue Mar 04 18:10:04 CST 2014
<code class="php plain">ERROR: too many offline disks in PST (grp 1)
<code class="php plain">Tue Mar 04 18:10:04 CST 2014
<code class="php plain">ERROR: PST-initiated MANDATORY DISMOUNT of group DATA
<code class="php plain">Tue Mar 04 18:10:04 CST 2014
<code class="php plain">WARNING: Disk 2 in group 1 in mode: 0x7,state: 0x2 was taken offline
<code class="php plain">Tue Mar 04 18:10:05 CST 2014
<code class="php plain">NOTE: halting all I/Os to diskgroup DATA
<code class="php plain">NOTE: active pin found: 0x0x40045bb0fd0
<code class="php plain">Tue Mar 04 18:10:05 CST 2014
<code class="php plain">Abort recovery
<code class="php keyword">for <code class="php plain">domain 1
<code class="php plain">Tue Mar 04 18:10:05 CST 2014
<code class="php plain">NOTE: cache dismounting group 1/0xD916EC16 (DATA)
<code class="php plain">Tue Mar 04 18:10:06 CST 2014
<p>大家可以看到，ASM报了一个ORA-15081错误,在该错误之前是报对其中一个盘/dev/raw/raw5的IO操作错误。<br>
细心的朋友可以看到,这里由于IO 操作异常后,该disk被offline了。最后磁盘组无法mount。</p>
<p>我们测试使用kfed read无法读取该disk，dd也无法操作。但是却可以直接dd 该disk对应的物理盘。</p>
<p>磁盘组无法mount，从其中trace来看显然是磁盘头损坏，如下:</p>
<code class="php plain">WARNING: cache read a corrupted block gn=1 dsk=2 blk=1 from disk 2
<code class="php plain">OSM metadata block dump:
<code class="php plain">kfbh.endian: 0 ; 0x000: 0x00
<code class="php plain">kfbh.hard: 0 ; 0x001: 0x00
<code class="php plain">kfbh.type: 0 ; 0x002: KFBTYP_INVALID
<code class="php plain">kfbh.datfmt: 0 ; 0x003: 0x00
<code class="php plain">kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
<code class="php plain">kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
<code class="php plain">kfbh.check: 0 ; 0x00c: 0x00000000
<code class="php plain">kfbh.fcn.base: 0 ; 0x010: 0x00000000
<code class="php plain">kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
<code class="php plain">kfbh.spare1: 0 ; 0x018: 0x00000000
<code class="php plain">kfbh.spare2: 0 ; 0x01c: 0x00000000
<code class="php spaces"><code class="php plain">CE: (0x0x400417ee4e0) group=1 (DATA) obj=2 (disk) blk=1
<code class="php spaces"><code class="php plain">hashFlags=0x0002 lid=0x0002 lruFlags=0x0000 bastCount=1
<code class="php spaces"><code class="php plain">redundancy=0x11 fileExtent=-2147483648 AUindex=0 blockIndex=1
<code class="php spaces"><code class="php functions">copy
<code class="php plain">#0: disk=2 au=0
<code class="php spaces"><code class="php plain">BH: (0x0x40041795000) bnum=4586 type=reading state=reading chgSt=not modifying
<code class="php spaces"><code class="php plain">flags=0x00000000 pinmode=excl lockmode=share bf=0x0x40041400000
<code class="php spaces"><code class="php plain">kfbh_kfcbh.fcn_kfbh = 0.0 lowAba=655.8572 highAba=0.0
<code class="php spaces"><code class="php plain">last kfcbInitSlot
<code class="php keyword">return <code class="php plain">code=null cpkt lnk is null
<p>大家知道Oracle ASM 10.2.0.5版本开始会对ASM disk header 进行自动备份，如果如果仅仅是盘头<br>
损坏那么恢复是很easy的。但是其实并不是这么简单，通过dd判断，该盘的前面几个block其实被损坏。</p>
<p>最后我们通过ODU 直接将数据文件从磁盘拷贝到文件系统，然后起库，最后完成整个恢复过程。</p>
<p>备注：在恢复过程中，发现ODU无法直接拷贝test201402.dbf 这样的文件，然而通过检查</p>
<p>asm alias directory发现，其实是完好的，这里可能odu处理还有点小问题，我们通过手工将该元数据</p>
<p>的AU 读取出来，然后匹配将剩下的文件全部抽取出来了，包括redo，controlfile，直接顺利打开数据库。</p>
<p>不得不说，熊哥的ODU太强大了，秒杀各种Oracle ASM的数据库恢复Case！</p>
<p> </p>
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1662
14
1418
52
1311
25
1261
29
1234
24
How to recover expired WeChat files? Can expired WeChat files be recovered?
Feb 22, 2024 pm 02:46 PM
Open WeChat, select Settings in Me, select General and then select Storage Space, select Management in Storage Space, select the conversation in which you want to restore files and select the exclamation mark icon. Tutorial Applicable Model: iPhone13 System: iOS15.3 Version: WeChat 8.0.24 Analysis 1 First open WeChat and click the Settings option on the My page. 2 Then find and click General Options on the settings page. 3Then click Storage Space on the general page. 4 Next, click Manage on the storage space page. 5Finally, select the conversation in which you want to recover files and click the exclamation mark icon on the right. Supplement: WeChat files generally expire in a few days. If the file received by WeChat has not been clicked, the WeChat system will clear it after 72 hours. If the WeChat file has been viewed,
How to recover browsing history in incognito mode
Feb 19, 2024 pm 04:22 PM
Private browsing is a very convenient way to browse and protect your privacy when surfing the Internet on your computer or mobile device. Private browsing mode usually prevents the browser from recording your visit history, saving cookies and cache files, and preventing the website you are browsing from leaving any traces in the browser. However, for some special cases, we may need to restore the browsing history of Incognito Browsing. First of all, we need to make it clear: the purpose of private browsing mode is to protect privacy and prevent others from obtaining the user’s online history from the browser. Therefore, incognito browsing
Convert VirtualBox fixed disk to dynamic disk and vice versa
Mar 25, 2024 am 09:36 AM
When creating a virtual machine, you will be asked to select a disk type, you can select fixed disk or dynamic disk. What if you choose fixed disks and later realize you need dynamic disks, or vice versa? Good! You can convert one to the other. In this post, we will see how to convert VirtualBox fixed disk to dynamic disk and vice versa. A dynamic disk is a virtual hard disk that initially has a small size and grows in size as you store data in the virtual machine. Dynamic disks are very efficient at saving storage space because they only take up as much host storage space as needed. However, as disk capacity expands, your computer's performance may be slightly affected. Fixed disks and dynamic disks are commonly used in virtual machines
How to restore chat spark on TikTok
Mar 16, 2024 pm 01:25 PM
On Douyin, a short video platform full of creativity and vitality, we can not only enjoy a variety of exciting content, but also have in-depth communications with like-minded friends. Among them, chat sparks are an important indicator of the intensity of interaction between the two parties, and they often inadvertently ignite the emotional bonds between us and our friends. However, sometimes due to some reasons, the chat spark may be disconnected. So what should we do if we want to restore the chat spark? This tutorial guide will bring you a detailed introduction to the content strategy, hoping to help everyone. How to restore the spark of Douyin chat? 1. Open the Douyin message page and select a friend to chat. 2. Send messages and chat to each other. 3. If you send messages continuously for 3 days, you can get the spark logo. On a 3-day basis, send pictures or videos to each other
How to restore Xiaomi Cloud photo album to local
Feb 24, 2024 pm 03:28 PM
How to restore Xiaomi Cloud Photo Album to local? You can restore Xiaomi Cloud Photo Album to local in Xiaomi Cloud Photo Album APP, but most friends don’t know how to restore Xiaomi Cloud Photo Album to local. The next step is to restore Xiaomi Cloud Photo Album to local. Local method graphic tutorials, interested users come and take a look! How to restore Xiaomi cloud photo album to local 1. First open the settings function in Xiaomi phone and select [Personal Avatar] on the main interface; 2. Then enter the Xiaomi account interface and click the [Cloud Service] function; 3. Then jump to Xiaomi For the function of cloud service, select [Cloud Backup]; 4. Finally, in the interface as shown below, click [Cloud Album] to restore the album to local.
How to restore default wallpaper in win10
Feb 10, 2024 pm 10:51 PM
Windows 10's May 2019 Update features a new, brighter default desktop background. It looks great - with the new light theme. If you use Windows 10’s dark theme, you may want a darker background. Strangely, the original Windows 10 desktop background has been removed from the latest version of Windows 10. You have to download it from the web or copy its files from an old Windows 10 PC. Although we were unable to find this wallpaper image on Microsoft's official website, you can download it from other sources. We found a copy of the original Windows 10 desktop wallpaper in 4K resolution on Imgur. Additionally, there are other sizes and more default walls
How to master disk usage in Ubuntu system
Jan 03, 2024 pm 11:13 PM
We want to check the hard disk usage in the system. How to check the Ubuntu system? Let's take a look at the tutorial on checking the hard disk usage in Ubuntu system. 1. On the system, click on the lower left corner of the desktop, as shown in the picture. 2. In the system, click Utilities, as shown in the figure. 3. Click Disk Usage, as shown in the picture. 4. Then you can see the disk usage, that is, the disk in the middle is your computer’s hard drive, as shown in the picture. The small print below the disk shows the usage. 5. Or click the disk on the utility program, as shown in the picture. 6. Then under the capacity, you can see the hard disk usage, as shown in the picture. The following is the text version to view disk usage: df-h The results are as follows: FilesystemSizeUsedAvailU
![How to increase disk size in VirtualBox [Guide]](https://img.php.cn/upload/article/000/887/227/171064142025068.jpg?x-oss-process=image/resize,m_fill,h_207,w_330)
How to increase disk size in VirtualBox [Guide]
Mar 17, 2024 am 10:10 AM
We often encounter situations where the predefined disk size has no room for more data? If you need more virtual machine hard disk space at a later stage, you must expand the virtual hard disk and partitions. In this post, we will see how to increase disk size in VirtualBox. Increasing the disk size in VirtualBox It is important to note that you may want to back up your virtual hard disk files before performing these operations, as there is always the possibility of something going wrong. It is always a good practice to have backups. However, the process usually works fine, just make sure to shut down your machine before continuing. There are two ways to increase disk size in VirtualBox. Expand VirtualBox disk size using GUI using CL
