IP地址被清空导致实例重启
客户10.2.0.4 RAC for Solaris 10环境突然出现了实例重启的现象。 数据库正常运行到下午3点左右，随后两个节点分别重启，其中一个节点上的实例无法自动启动。检查两个实例的告警日志发现，在节点重启前，两个节点都出现了明显的ORA-27504错误： Wed Apr 10 1
客户10.2.0.4 RAC for Solaris 10环境突然出现了实例重启的现象。
数据库正常运行到下午3点左右，随后两个节点分别重启，其中一个节点上的实例无法自动启动。检查两个实例的告警日志发现，在节点重启前，两个节点都出现了明显的ORA-27504错误：
Wed Apr 10 15:00:05 2013 Errors IN file /oracle/admin/orcl/udump/orcl1_ora_10997.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:00:06 2013 Errors IN file /oracle/admin/orcl/udump/orcl1_ora_11007.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:00:06 2013 Errors IN file /oracle/admin/orcl/udump/orcl1_ora_11009.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:00:06 2013 Errors IN file /oracle/admin/orcl/udump/orcl1_ora_11011.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command . . . Wed Apr 10 15:07:08 2013 IPC Send timeout detected.Sender: ospid 25688 Receiver: inst 2 binc 427282 ospid 11838 Wed Apr 10 15:07:08 2013 IPC Send timeout detected.Sender: ospid 25724 Wed Apr 10 15:07:08 2013 IPC Send timeout detected.Sender: ospid 25680 Receiver: inst 2 binc 431591 ospid 11822 Receiver: inst 2 binc 431795 ospid 11874 Wed Apr 10 15:07:08 2013 IPC Send timeout detected.Sender: ospid 25684 Receiver: inst 2 binc 428985 ospid 11826 Wed Apr 10 15:07:08 2013 IPC Send timeout detected.Sender: ospid 25708 Receiver: inst 2 binc 430048 ospid 11858 Wed Apr 10 15:07:09 2013 ospid 25678: network interface WITH IP address 192.168.168.3 no longer operational requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:07:35 2013 IPC Send timeout TO 1.1 inc 4 FOR msg TYPE 44 FROM opid 7 Wed Apr 10 15:07:35 2013 IPC Send timeout TO 1.12 inc 4 FOR msg TYPE 44 FROM opid 21 Wed Apr 10 15:07:35 2013 IPC Send timeout TO 1.2 inc 4 FOR msg TYPE 44 FROM opid 8 Wed Apr 10 15:07:35 2013 IPC Send timeout TO 1.3 inc 4 FOR msg TYPE 44 FROM opid 10 Wed Apr 10 15:07:35 2013 IPC Send timeout TO 1.8 inc 4 FOR msg TYPE 44 FROM opid 15 Wed Apr 10 15:08:13 2013 ospid 25678: network interface WITH IP address 192.168.168.3 no longer operational requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:08:16 2013 IPC Send timeout detected.Sender: ospid 25748 Receiver: inst 2 binc 430164 ospid 11890 . . . Wed Apr 10 15:08:53 2013 IPC Send timeout TO 1.13 inc 4 FOR msg TYPE 36 FROM opid 176 Wed Apr 10 15:08:53 2013 IPC Send timeout TO 1.15 inc 4 FOR msg TYPE 36 FROM opid 167 Wed Apr 10 15:08:57 2013 IPC Send timeout TO 1.4 inc 4 FOR msg TYPE 32 FROM opid 180 . . . Wed Apr 10 15:15:51 2013 Evicting instance 2 FROM cluster Wed Apr 10 15:16:09 2013 ospid 25678: network interface WITH IP address 192.168.168.3 no longer operational requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:16:40 2013 Waiting FOR instances TO leave: Wed Apr 10 15:17:00 2013 Waiting FOR instances TO leave: Wed Apr 10 15:17:09 2013 ospid 25678: network interface WITH IP address 192.168.168.3 no longer operational requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:17:20 2013 Waiting FOR instances TO leave:
节点2上的错误信息与之类似：
. . . Wed Apr 10 15:19:07 2013 Errors IN file /oracle/admin/orcl/udump/orcl2_ora_14065.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.4 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:19:08 2013 Errors IN file /oracle/admin/orcl/udump/orcl2_ora_14057.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 192.168.168.4 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:19:46 2013 ospid 11820: network interface WITH IP address 192.168.168.4 no longer operational requested interface 192.168.168.4 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:20:46 2013 ospid 11820: network interface WITH IP address 192.168.168.4 no longer operational requested interface 192.168.168.4 NOT found. CHECK output FROM ifconfig command Wed Apr 10 15:20:55 2013 Errors IN file /oracle/admin/orcl/bdump/orcl2_lmon_11818.trc: ORA-29740: evicted BY member 0, GROUP incarnation 6 Wed Apr 10 15:20:55 2013 LMON: terminating instance due TO error 29740 Wed Apr 10 15:20:55 2013 Errors IN file /oracle/admin/orcl/bdump/orcl2_smon_11924.trc: ORA-29740: evicted BY member , GROUP incarnation Wed Apr 10 15:20:55 2013 Errors IN file /oracle/admin/orcl/bdump/orcl2_lmse_11886.trc: ORA-29740: evicted BY member , GROUP incarnation Wed Apr 10 16:11:37 2013 Starting ORACLE instance (normal) Wed Apr 10 16:11:45 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:11:45 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:11:45 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:11:45 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:11:50 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:11:50 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:11:50 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:11:50 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:11:54 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:11:54 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:11:54 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:11:54 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:12:29 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:12:29 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:12:29 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:12:29 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:12:47 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:12:47 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:12:47 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:12:47 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:12:52 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:12:52 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:12:52 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:12:52 2013 Failed TO acquire instance startup/shutdown serialization primitive Wed Apr 10 16:12:56 2013 sculkget: failed TO LOCK /oracle/products/10.2/db_1/dbs/lkinstorcl2 exclusive Wed Apr 10 16:12:56 2013 sculkget: LOCK held BY PID: 6912 Wed Apr 10 16:12:56 2013 Oracle Instance Startup operation failed. Another process may be attempting TO startup OR shutdown this Instance. Wed Apr 10 16:12:56 2013 Failed TO acquire instance startup/shutdown serialization primitive
导致问题的原因根据错误信息很容易分析出来，节点2上的IP地址被修改，导致心跳通信出现了异常，而节点1试图将节点2踢出集群，但是由于无法和节点2之间进行通信，因此只有等待节点2重启。
检查节点2的操作系统日志：
Apr 10 15:00:04 bj-sst-xhm-3f2-m5k-02 ip: [ID 482227 kern.notice] ip_arp_done: init failed Apr 10 15:07:37 bj-sst-xhm-3f2-m5k-02 Had[4135]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage ON bj-sst-xhm-3f2-m5k-02 IS 92% Apr 10 15:18:41 bj-sst-xhm-3f2-m5k-02 sshd[13485]: [ID 800047 auth.error] error: Failed TO allocate internet-DOMAIN X11 display socket.
在15点04秒时出现的ip_arp_done: init failed信息，说明设置网卡接口时使用了主机名信息，且主机的IP地址被在线修改。
最后根据HISTORY确认，发现有人通过root登录系统，执行ifconfig –a6来检查IPV6的地址，但是命令敲错，执行了ifconfig –a 6，在a和6之间多了一个空格，导致主机所有的IP地址被设置成0.0.0.0，于是导致了上面的错误。
这再次说明，对于root这种权限用户而言，任何的不小心都可能会导致非常严重的后果。
原文地址：IP地址被清空导致实例重启, 感谢原作者分享。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How to restart Samsung s24Ultra phone?
Feb 09, 2024 pm 09:54 PM
When using the Samsung S24 Ultra mobile phone, you may occasionally encounter some problems or need to reset the device. In this case, restarting the phone is a common solution. However, it may be confusing if you don't know much about the steps. However, don’t worry, I will show you how to restart your Samsung S24 Ultra phone properly. How to restart the Samsung s24 Ultra 1. Bring up the control menu to shut down: Slide down from the top of the Samsung screen to bring up the shortcut tool menu, click the power icon (a combination of arc and vertical line) to bring up the shutdown and restart selection interface, click Just restart; 2. Use the key combination to shut down: long press the volume-key plus the power key to bring up the shutdown and restart selection menu, click to select shutdown. By pressing and holding
How to solve the computer prompt 'reboot and select proper boot device'
Jan 15, 2024 pm 02:00 PM
Reinstalling the system may not be a foolproof solution, but after reinstalling, I found that when the computer is turned on, it will display white text on a black background, and then give a prompt: rebootandselectproperbootdevice, what is going on? Such a prompt is usually caused by a boot error. In order to help everyone, the editor has brought you a solution. Computer use is becoming more and more popular, and computer failures are becoming more and more common. No, recently some users encountered a black screen when turning on the computer, and prompted Reboot and Select Proper Boot device, and the computer system could not start normally. What's going on? How to solve it? The user is confused. Next, the editor will follow
Apple after-sales (apple after-sales point address)
Jan 11, 2024 pm 10:30 PM
Apple’s official after-sales phone number: Apple’s 24-hour service center phone number: 400-666-8800. The after-sales service telephone number for Apple mobile phones is: 400-666-8800. -627-2273. Apple’s customer service manual service hotline is 400-627-2273 for after-sales support; 400-666-8800 for the online store; and the only official Apple phone number is 400-666-8800. Apple's customer service hotline is 400-666-8800. You can call this number to inquire about hardware, software and third-party accessories of Apple products. It should be noted that Apple’s manual customer service does not provide services 24 hours a day. Their service hours are from 9 a.m. to 9 p.m. (Sundays are from 9 a.m. to 9 p.m.
Where can I change my Meituan address? Meituan address modification tutorial!
Mar 15, 2024 pm 04:07 PM
1. Where can I change my Meituan address? Meituan address modification tutorial! Method (1) 1. Enter Meituan My Page and click Settings. 2. Select personal information. 3. Click the shipping address again. 4. Finally, select the address you want to modify, click the pen icon on the right side of the address, and modify it. Method (2) 1. On the homepage of the Meituan app, click Takeout, then click More Functions after entering. 2. In the More interface, click Manage Address. 3. In the My Shipping Address interface, select Edit. 4. Modify them one by one according to your needs, and finally click to save the address.
How to recover cleared browsing history
Feb 18, 2024 pm 10:05 PM
How to restore web page history after it has been cleared Date: June 10, 2022 Introduction: When we use computers or mobile phone browsers daily, we often use the browser's history to find web pages we have visited before. However, sometimes we may accidentally clear our browser history, causing us to be unable to retrieve a specific web page. In this article, I will tell you some ways to recover cleared web history. Method 1: Use the browser recovery function. Most common browsers provide the function of restoring history, such as Google
What is the correct way to restart a service in Linux?
Mar 15, 2024 am 09:09 AM
What is the correct way to restart a service in Linux? When using a Linux system, we often encounter situations where we need to restart a certain service, but sometimes we may encounter some problems when restarting the service, such as the service not actually stopping or starting. Therefore, it is very important to master the correct way to restart services. In Linux, you can usually use the systemctl command to manage system services. The systemctl command is part of the systemd system manager
Solution to the problem of infinite black screen spinning and restarting when booting up Windows 10
Dec 26, 2023 pm 07:10 PM
After we install the win10 operating system, if some friends have a black screen and the cursor keeps spinning in a circle while using the system, don't worry. The editor thinks that this situation may be due to the settings of our computer. You can first enter the advanced options of the system, and then find the corresponding options that need to be set and set them. Let’s take a look at the specific operation steps. Let’s take a look at what the editor did~ What to do if win10 starts up with an infinite black screen and spins in circles. Method 1: 1. Hard shutdown three times (force the power button for 10 seconds). The system will automatically repair and the interface shown below will appear. 2. Troubleshooting center point – “Advanced Options” 3. Click – “Startup Settings” in the advanced options 4. Click – “Restart” button. 5. The computer will restart at this time. Restart
Solve the problem of restarting in a loop after entering the password in win10
Dec 29, 2023 pm 09:53 PM
When we inadvertently perform some wrong operations, or there are certain errors in the system itself, we may be unable to enter the desktop after entering the password and keep restarting. At this time we can repair it in safe mode. Let’s take a look at the specific methods below. Win10 cannot enter the desktop after entering a password and keeps restarting. Solution 1. First, press and hold "shift" on the keyboard and click the power button in the lower right corner, then choose to restart the computer until the repair interface appears and then release the "shift" key. 2. If there is no power button in the lower right corner, you can also use the power button of the computer host, but you need to restart it three times or more in a row. 3. After the repair interface appears, we click "View advanced repair options". 4. Select "Troubleshoot". 5
