Ora 01041 внутренняя ошибка hostdef расширение не существует

Home » Articles » Misc » Here

This article provides an example of using the SYSRESV utility to identify the shared memory segments associated with an Oracle instance.

This specific issue is not the only cause of an ORA-01041.

• The Problem
• The Solution (SYSRESV Utility)

The Problem

A couple of times we’ve had problems with instances not responding. When we try to connect to the database from the server we get something like this.

$ export ORACLE_SID=orcl
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01041: internal error. hostdef extension doesn't exist

$

When you check on the OS you can see the processes running, but there is no way to connect. Even a preliminary connection fails. There is nothing you can do with the instance, so the obvious thing to try and do is kill it, but you can’t do this with a «shutdown» or «shutdown abort» because you can’t connect. Instead you try to kill it at the OS level.

$ kill -9 `ps -ef | grep orcl | awk '{print $2}'`

After this you see all the processes gone, but you still can’t connect to the instance.

$ export ORACLE_SID=orcl
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01041: internal error. hostdef extension doesn't exist

$

The Solution (SYSRESV Utility)

The last time this happened we guessed it might be about shared memory segments hanging around after the instance has died. You can check the shared memory directly, but it can be difficult to relate this back to the instances. One way is by comparing the creation time of the shared memory segment with the V$INTANCE.STARTUP_TIME column of other running instances on the server and use a process of elimiation. The type of output you can expect from «ipcs -mpt» is shown below.

$ ipcs -mpt
IPC status from /dev/kmem as of Wed Jul 13 10:04:09 2016
T         ID     KEY        MODE        OWNER     GROUP  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m          0 0x411c0123 --rw-rw-rw-      root      root   957 29670  7:14:00  7:14:00  9:47:13
m          1 0x4e0c0002 --rw-rw-rw-      root      root   957  4731  9:06:51  9:15:11  9:47:13
m          2 0x41200006 --rw-rw-rw-      root      root   957 29670  7:14:00  7:14:00  9:47:13
m          3 0x00a5c581 --rw-------     sfmdb     users  2065  2102  9:48:14  9:48:14  9:48:14
m          4 0x06347849 --rw-rw-rw-      root      root  2623  2629  9:48:50  9:48:47  9:48:47
m          5 0x0c6629c9 --rw-r-----      root      root  2644  4295 10:33:52 10:34:17  9:48:47
m      32774 0xa90c08cc -----------      root      root  2954  2955  9:49:03 no-entry  9:49:03
m          7 0x012009d0 --rw-rw-r--      root      root  2660  2660  9:48:48  9:48:48  9:48:48
m          8 0x012009cc --rw-rw-r--      root      root  2662  2662  9:48:48  9:48:48  9:48:48
m          9 0x012009c4 --rw-rw-r--      root      root  2666  2666  9:48:48 no-entry  9:48:48
m         10 0x49143b35 --rw-r--r--      root      root  2629  2629 10:04:00 10:04:00  9:48:49
m         11 0x012009ce --rw-rw-r--      root      root  2709  2709  9:48:49 no-entry  9:48:49
m      32780 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         13 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         14 0x00000000 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m         15 0x588c2d20 --rw-r-----    oracle  oinstall  6230 11639 10:03:44 10:03:44 12:00:36
m      32784 0x00000000 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m         17 0x00000000 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m         18 0x375d46a0 --rw-r-----    oracle  oinstall  6560 11683 10:04:03 10:04:03 12:01:02
m      32787 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         20 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         21 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         22 0x00000000 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:03:46 12:04:33
m         23 0x10badea0 --rw-r-----    oracle  oinstall  7117 11663 10:04:00 10:04:00 12:04:33
m      98328 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         25 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         26 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         27 0x00000000 --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m         28 0x09c1865c --rw-r-----    oracle  oinstall  7969 11681 10:04:02 10:04:02 12:06:39
m     196637 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         30 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         31 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         32 0x00000000 --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:03:49 12:08:13
m         33 0xb747ec2c --rw-r-----    oracle  oinstall  8741 11665 10:04:00 10:04:00 12:08:13
m     426018 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32803 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32804 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32805 0x00000000 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
m      32806 0x472f4a20 --rw-r-----    oracle  oinstall  2869  9042 10:03:02 10:04:00  8:55:00
$

Instead, you can use the «$ORACLE_HOME/bin/sysresv» command to give you the shared memory segments associated with a named instance.

$ sysresv -l orcl

IPC Resources for ORACLE_SID "orcl" :
Shared Memory:
ID              KEY
98338           0x00000000
35              0x00000000
36              0x00000000
37              0x00000000
38              0x472f4a20
Semaphores:
ID              KEY
12312           0xdd03b17c
Oracle Instance not alive for sid "orcl"
$ 

So we can see there are shared memory segments for an instance that is now dead.

It worked as expected, but if for some reason it is not able to link the dead instance to the shared memory segments, you can always work backwards. Check the segments for all the running instances. If there are any shared memory segments for «oracle oinstall» from the «ipcs -mpt» command that are not seen in the following output, by a process of elimination, they must belong to the dead instance.

$ sysresv -l instance1 instance2 instance3

Once we know the dodgy shared memory segments (directly or indirectly) we can we can remove them using «ipcrm».

$ ipcrm -m 98338
$ ipcrm -m 35
$ ipcrm -m 36
$ ipcrm -m 37
$ ipcrm -m 38

Once that is done, you should be able to connect to, and start the instance normally.

It’s possible once you remove the first segment (98338) the others will no longer be seen as linked to the instance, but you will still need to remove them or you will be able to connect, but the instance will not start.

For more information see:

• SYSRESV Utility (Doc ID 123322.1)

Hope this helps. Regards Tim…

Back to the Top.

So, here is the thing. We have a customer running a 11.2 Single Instance Oracle database on Windows. I know the version is a bit old and Windows may not be the most common choice to run an Oracle database, but this is not the point of the post. This customer asked us to back up the database directly to a CIFS volume using RMAN. This is a common and wise practice: save your database backups outside of the server where it is running.

Piece of cake, the only caveat is that the Oracle Windows service must be running with the credentials of a domain account that has permission on the CIFS volume. Some basic research on MOS and voilà: How to Change Oracle Owner from Local System Account to Domain User Account in Windows (Doc ID 2035714.1).

After obtaining the maintenance window and executing the action plan to change the services from under the local SYSTEM account to a domain account, a simple RMAN backup writing directly onto the CIFS volume works just fine:

RMAN> backup current controlfile format '\backup-nas.domain.comoraclebackupsoraclectl_file_text.bkp';
Starting backup at 16-08-2018 11:50:24
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 16-08-2018 11:50:25
channel ORA_DISK_1: finished piece 1 at 16-08-2018 11:50:40
piece handle=\backup-nas.domain.comoraclebackupsoracleCTL_FILE_TEXT.BKP tag=TAG20180816T115024 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
Finished backup at 16-08-2018 11:50:40

Starting Control File and SPFILE Autobackup at 16-08-2018 11:50:40
piece handle=G:ORACLEFAST_RECOVERY_AREAPROTECTAUTOBACKUP2018_08_16O1_MF_S_984311440_FQBOR0Y3_.BKP comment=NONE
Finished Control File and SPFILE Autobackup at 16-08-2018 11:50:41

We now reboot the server to make sure that everything comes up fine. Ooops!!

PS C:Userspythian.admin> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Thu Aug 16 11:01:38 2018

Copyright (c) 1982, 2017, Oracle.  All rights reserved.

ERROR:
ORA-01041: internal error. hostdef extension doesn't exist

What happened?

Good question, indeed. The OERR information is not life-saving:

Error: ORA 1041
Text: internal error. hostdef extension doesn't exist
---------------------------------------------------------
Cause: Pointer to hstdef extension in hstdef is null.
Action: Likely a known or new bug.

Explanation: This is usually reported when a connection has broken for some reason.

Diagnosis:

1) Check the same operation for any ORA 3113 or ORA 3114 type errors. The ORA 1041 error usually results from an unexpected disconnection.
2) Follow the same steps as you would to progress an ORA 3113

Obviously, there are no ORA-3113 or ORA-3114 to be seen. Also, there is not much information about this error in MOS and Google does not help much, either.
Although, one MOS note looked promising: ORA-1041 When Trying to Connect as Sysdba to Startup Database (Doc ID 552218.1).

From the note:

The system time was set incorrectly.
The local system time did not match the time on the domain server causing the authentication to fail.

Why does this look promising? Because the database shows an uptime of one hour right after starting up, so something is changing the time where it shouldn’t be changed.

Windows time service is fidgeting with my database

After reviewing the Windows event log, I noticed a sudden jump in the time during the startup of the server.

This shows the Windows kernel updating the system time after the Windows Time service connects to the domain controller and synchronizes the server time with it. The problem with this is that it all happens after the Oracle services have started, hence the startup issue.

To add to this hypothesis, after manually restarting the services after the server is up and running, everything works as expected.

Diagnosis and prognostic

So, the situation is as follows:

• Oracle Windows services work fine after being set up with a domain account.
• RMAN backups are working after the above setup.
• On Windows boot up, Oracle services start before the Window Time service can set the proper system time.
• After the system is up and running, restarting the Oracle services gets them up and running.

The prognostic is that the system time differs from the Active Directory time and this prevents the credentials used for the Oracle services to be validated leading to the failure of the database start.

Final fix

There may be more, but we analyzed two different approaches here: one set the BIOS RTC to the proper time, in sync with the domain server, using an NTP configuration.

The other is to have the Oracle services starting after the Windows Time service. But this does not ensure that the system time has been changed already, so an additional setting has to be put in place: to declare the Oracle services as automatic-delayed ones. This ensures that the Oracle services will start only after all the automatic services are up and running, with the caveat that it makes the Oracle services dependent on others that may not be directly related.

So working with our customer and Oracle support we decided to go for the automatic-delayed service startup and it did the trick. While this is not the ideal situation, we don’t expect this server to go down frequently, or at all actually, so this configuration should be enough to bring the database back up and running in case of an unexpected server failure.

Want to talk with an expert? Schedule a call with our team to get the conversation started.

Problem Description:
Recently I had this issue in a two node Oracle EBS 11i (11.5.10.2) non production environment on a RHEL5 (32-bit).
While submitting any concurrent request, it is showing Phase=INACTIVE and Status=No manager. After checking the CM log files, we found out that the internal concurrent manager got terminated with the below errors:

This could happen due to possible reasons:

• Database going down
• Listener is down
• Packet loss
  during the communication between the CM and the DB

In our case, the following error indicates that the database went down.
fdudat failed due to ORA-01041: internal error. hostdef extension doesn’t exist.

Solution:

To resolve ORA-01041, you should report the
error as a bug. In the case of ORA-01041 as a response to performing shutdown
and startup, you can use the ipcrm
command and remove allocated shared memory segments.

So the solution was to bounce the Database and start the concurrent managers again.