terminating the instance due to error 481导致ASM无法启动故障解决
一个RAC数据库，意外DOWN机后，第一个节点正常启动，但是第二个节点却无法启动ASM和CRS资源。
1、现象描述
一个RAC数据库，意外DOWN机后，第一个节点正常启动，但是第二个节点却无法启动ASM和CRS资源。
2、分析原因
由于ASM磁盘组无法启动，查看ASM日志发现如下信息：
MMNL started with pid=21,OS id=14028
lmon registered with NM -instance number 2 (internal mem no 1)
Tue Nov 18 14:48:50 2014
PMON (ospid:13986): terminating the instance due to error 481
Tue Nov 18 14:48:50 2014
System state dumprequested by (instance=2, osid=13986 (PMON)), summary=[abnormal instancetermination].
System State dumped totrace file /Oracle/product/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_13996.trc
Tue Nov 18 14:48:50 2014
ORA-1092 :opitsk aborting process
Dumping diagnostic data indirectory=[cdmp_20141118144850], requested by (instance=2, osid=13986 (PMON)),summary=[abnormal instance termination].
Tue Nov 18 14:48:50 2014
ORA-1092 : opitsk abortingprocess
Instance terminated byPMON, pid = 13986
在Metalink网站上，发现一篇文档ASM onNon First Node (Second or Other Node) Fails to Come up With: PMON (ospid:nnnn): terminating the instance due to error 481 [ID 1383737.1]。依据该文档得知，ASM的alert日志错误信息中“PMON(ospid: 9946): terminating the instance due to error 481”文档的标题是一致。根据文档的内容进一步检查相关日志及配置信息，发现ASM TRACE也如文档一致。如下：
*** 2014-11-18 14:48:17.092
Reconfiguration completes [incarn=42]
kjzdattdlm: Can not attach to DLM (LMONup=[TRUE], DB mounted=[FALSE]).
通过分析集群日志发现如下信息：
2014-11-18 14:44:45.767
[/oracle/product/11.2.0/grid/bin/orarootagent.bin(12690)]CRS-5018:(:CLSN00037:)Removed unused HAIP route: 169.254.95.0/ 255.255.255.0 / 0.0.0.0 / usb0
说明ORACLE认为usb0（主机管理口网卡，默认开启）占用了HAIP（ORACLE 11G新特性，内部的高可用私有IP，通过HAIP替换两个节点的私有IP,来进行内部通讯）。
通过分析主机日志发现如下信息：
Nov 18 14:02:11 XXXdb2 dhclient:DHCPREQUEST on usb0 to 255.255.255.255 port 67
Nov 18 14:02:12 XXXdb2 dhclient:DHCPACK from 169.254.95.118
Nov 18 14:02:12 XXXdb2 dhclient: boundto 169.254.95.120 -- renewal in 234 seconds.
说明主机网卡USB0在动态获取169.254.XX.XX网段的IP地址。
IBM的PC服务器使用USB0做为管理网络的特性。没有连接USB0网卡的时候会不停向DHCP申请IP，如果没有发现DHCP时就会默认分配一个169.254.xxx.xxx的IP地址会和ORACLE的HAIP产生冲突,造成路由信息丢失
通过各种日志信息与文档中的信息的对比，得知此次的故障现象与文档中的故障现象是一致的。
3、解决方法
根据文档ID 1383737.1提供的内容，使用ifdown usb0关闭两个节点的usb0网卡后，在缺失路由信息的一节点动态添加路由信息
#route add -net 169.254.0.0 netmask 255.255.0.0 dev eth2
增加路由之后，在数据库的第二节点，通过命令
#su - grid
$GRID_HOME/bin/crsctl start res ora.crsd -init
正常启动crsd资源后，asm和crsd资源全部正常启动。
4、故障总结
IBM 的x3850 x5系列的PC Server存在USB开启dhcp功能，从而导致usb网卡可能占用HAIP的缺陷，，生产环境中的此类机器上运行的RAC数据库环境，需要关闭USB0的自动获取dhcp功能，给USB0配置静态IP。
5 、相关参考
ASM on Non First Node (Second or Other Node) Fails to Come up With:PMON (ospid: nnnn): terminating the instance due to error 481 [ID 1383737.1]
如何修改ASM的sys密码
如何将ASM中的数据文件复制到操作系统中
Oracle 11g RAC ASM磁盘全部丢失后的恢复
Oracle 11g从入门到精通 PDF+光盘源代码
RHEL6 ASM方式安装Oracle 11g R2
Oracle 10g 手工创建ASM数据库
Ubuntu 12.04(amd64)安装完Oracle 11gR2后各种问题解决方法
如何修改Oracle 10g ASM的sys密码
本文永久更新链接地址：
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
When might a full table scan be faster than using an index in MySQL?
Apr 09, 2025 am 12:05 AM
Full table scanning may be faster in MySQL than using indexes. Specific cases include: 1) the data volume is small; 2) when the query returns a large amount of data; 3) when the index column is not highly selective; 4) when the complex query. By analyzing query plans, optimizing indexes, avoiding over-index and regularly maintaining tables, you can make the best choices in practical applications.
Can I install mysql on Windows 7
Apr 08, 2025 pm 03:21 PM
Yes, MySQL can be installed on Windows 7, and although Microsoft has stopped supporting Windows 7, MySQL is still compatible with it. However, the following points should be noted during the installation process: Download the MySQL installer for Windows. Select the appropriate version of MySQL (community or enterprise). Select the appropriate installation directory and character set during the installation process. Set the root user password and keep it properly. Connect to the database for testing. Note the compatibility and security issues on Windows 7, and it is recommended to upgrade to a supported operating system.
Explain InnoDB Full-Text Search capabilities.
Apr 02, 2025 pm 06:09 PM
InnoDB's full-text search capabilities are very powerful, which can significantly improve database query efficiency and ability to process large amounts of text data. 1) InnoDB implements full-text search through inverted indexing, supporting basic and advanced search queries. 2) Use MATCH and AGAINST keywords to search, support Boolean mode and phrase search. 3) Optimization methods include using word segmentation technology, periodic rebuilding of indexes and adjusting cache size to improve performance and accuracy.
Difference between clustered index and non-clustered index (secondary index) in InnoDB.
Apr 02, 2025 pm 06:25 PM
The difference between clustered index and non-clustered index is: 1. Clustered index stores data rows in the index structure, which is suitable for querying by primary key and range. 2. The non-clustered index stores index key values and pointers to data rows, and is suitable for non-primary key column queries.
MySQL: Simple Concepts for Easy Learning
Apr 10, 2025 am 09:29 AM
MySQL is an open source relational database management system. 1) Create database and tables: Use the CREATEDATABASE and CREATETABLE commands. 2) Basic operations: INSERT, UPDATE, DELETE and SELECT. 3) Advanced operations: JOIN, subquery and transaction processing. 4) Debugging skills: Check syntax, data type and permissions. 5) Optimization suggestions: Use indexes, avoid SELECT* and use transactions.
Can mysql and mariadb coexist
Apr 08, 2025 pm 02:27 PM
MySQL and MariaDB can coexist, but need to be configured with caution. The key is to allocate different port numbers and data directories to each database, and adjust parameters such as memory allocation and cache size. Connection pooling, application configuration, and version differences also need to be considered and need to be carefully tested and planned to avoid pitfalls. Running two databases simultaneously can cause performance problems in situations where resources are limited.
The relationship between mysql user and database
Apr 08, 2025 pm 07:15 PM
In MySQL database, the relationship between the user and the database is defined by permissions and tables. The user has a username and password to access the database. Permissions are granted through the GRANT command, while the table is created by the CREATE TABLE command. To establish a relationship between a user and a database, you need to create a database, create a user, and then grant permissions.
RDS MySQL integration with Redshift zero ETL
Apr 08, 2025 pm 07:06 PM
Data Integration Simplification: AmazonRDSMySQL and Redshift's zero ETL integration Efficient data integration is at the heart of a data-driven organization. Traditional ETL (extract, convert, load) processes are complex and time-consuming, especially when integrating databases (such as AmazonRDSMySQL) with data warehouses (such as Redshift). However, AWS provides zero ETL integration solutions that have completely changed this situation, providing a simplified, near-real-time solution for data migration from RDSMySQL to Redshift. This article will dive into RDSMySQL zero ETL integration with Redshift, explaining how it works and the advantages it brings to data engineers and developers.
