ORA-12542: TNS:address already in use – Oracle 10g 64 bit

My one of the client facing ORA-12542, this error message popped up when he tried TNSPING OR to create DB-link.
Environment:
Microsoft Windows Server 2003 – 64 bit
Oracle Database 10g Enterprise Edition Release 10.2.0.4 – 64 bit

Error:

ORA-12542: TNS:address already in use

Solution:

On server side, add multiple TCP ports in listener.ora file, I have added additional TCP port i.e.1523 to the current LISTENER.

D:\oracle\product\10.2.0\db_1\network\admin\listener.ora
INBOUND_CONNECT_TIMEOUT_LISTENER = 0
SID_LIST_LISTENER =
   (SID_LIST =
      (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = D:\oracle\product\10.2.0\db_1)
      (PROGRAM = extproc)
   )
 )
LOG_DIRECTORY_LISTENER = G:\Listener_Trace
LISTENER =
   (DESCRIPTION_LIST =
     (DESCRIPTION =
       (ADDRESS = (PROTOCOL = TCP)(HOST = newpass)(PORT = 1521))
       (ADDRESS = (PROTOCOL = TCP)(HOST = newpass)(PORT = 1523))
     )
   )

And on client side, map net service names to recently added listener protocol address. i.e. 1523, as below:

NEWPASS_29 =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.6)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.6)(PORT = 1523))
   )
  (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = NEWPASS) 
  )
)

after made above changes to listener.ora and tnsnames.ora, TNSPING was successful and able to create DB-link.

Note:

You may face mentioned error in following cases:

  1. New installation
  2. listener.ora file has been edited manually since the last listener restart
  3. TCP port number is duplicated across the list of ADDRESS configurations in the listener.ora file.

Cheers!!

Stay Tune. 🙂

Oracle 11g Logo

ORA-08104: this index object xxxxx is being online built or rebuilt and ORA-00031: session marked for kill

My second attempt to rebuild below mentioned index failed with “ORA-08104-this index object 22624 is being online built or rebuilt”. I am getting this oracle error because my first attempt to rebuild same index failed due to abnormal session termination, it was incomplete rebuild.

Error log:

SQL> ALTER INDEX UPASSUSR.IDX_TRANSLOG_OBJECTNAME REBUILD online TABLESPACE UPASSTBS;
ALTER INDEX UPASSUSR.IDX_TRANSLOG_OBJECTNAME REBUILD online TABLESPACE UPASSTBS
*
ERROR at line 1:
ORA-08104: this index object 22624 is being online built or rebuilt

According to oracle support(Doc ID 375856.1) we can rid out of the issue by cleaning garbage with the help of dbms_repair.online_index_clean function. But no luck after successful execution of dbms_repair.online_index_clean function.

declare
lv_ret BOOLEAN;
begin
lv_ret := dbms_repair.online_index_clean(22624);
end;
/
PL/SQL procedure successfully completed.

Same clean-up logs popped up in ADRCI with same object no(i.e.22624) but no luck.

2016-09-27 13:21:06.373000 +05:30
online index (re)build cleanup: objn=22624 maxretry=2000 forever=0

Even, I tried dropping index and mark as unusable but unlucky again.

SQL> drop index UPASSUSR.IDX_TRANSLOG_OBJECTNAME;
drop index UPASSUSR.IDX_TRANSLOG_OBJECTNAME
*
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

So, I have decided to kill that session with the help of “ALTER SYSTEM KILL SESSION”, I got require details with the help of following SQL:

SELECT O.OBJECT_NAME, S.SID, S.SERIAL#, P.SPID, S.PROGRAM,S.USERNAME,S.MACHINE, S.PORT, 
S.LOGON_TIME, SQ.SQL_FULLTEXT 
FROM V$LOCKED_OBJECT L, DBA_OBJECTS O, V$SESSION S, V$PROCESS P, V$SQL SQ 
WHERE L.OBJECT_ID = O.OBJECT_ID 
AND L.SESSION_ID = S.SID 
AND S.PADDR = P.ADDR 
AND S.SQL_ADDRESS = SQ.ADDRESS
AND O.OBJECT_NAME = 'AOUP_TRANS_LOG';

After 60 seconds of waiting, SQL command ended up with “ORA-00031: session marked for kill” oracle error. 🙁

SQL> ALTER SYSTEM KILL SESSION '3423,39597';
ALTER SYSTEM KILL SESSION '3423,39597'
*
ERROR at line 1:
ORA-00031: session marked for kill

Finally, I got the solution by killing process at OS level:
kill -9 spid     (from above)

Thank god, Index rebuild was successful.

Object flag before killing OS process:

SQL> select obj#,flags from ind$ where obj#=22624;
OBJ#       FLAGS
---------- ----------
22624      2562

Object flag after killing OS process:

SQL> select obj#,flags from ind$ where obj#=22624;
OBJ#       FLAGS
---------- ----------
22624      2050

Have a great time ahead.

Stay Tune. 🙂

Oracle 11g Logo

ORA-03113: end-of-file on communication channel while startup

My one of the client reported me ORA-03113 oracle error message. He is facing problem while starting up database. Usually this error occurs when connection between Client and Server process was broken OR It can be any big problem.

SQL> startup
ORACLE instance started.
Total System Global Area 8584982528 bytes
Fixed Size 2262088 bytes
Variable Size 4462742456 bytes
Database Buffers 4093640704 bytes
Redo Buffers 26337280 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 2588
Session ID: 1705 Serial number: 5

On further diagnosis of ADRCI logs, I found disk space size assigned to Flash recovery area falling short. ADRCI logs are as follows:

2016-09-12 12:46:00.960000 +05:30
Successful mount of redo thread 1, with mount id 3346245764
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
ALTER DATABASE OPEN
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=20, OS id=564
2016-09-12 12:46:02.333000 +05:30
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\trace\mnsbdb_ora_2588.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 21474836480 bytes is 100.00% used, and has 0 remaining bytes available.
************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
 then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
 BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
 reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
 system command was used to delete files, then use RMAN CROSSCHECK and
 DELETE EXPIRED commands.
************************************************************************
ARCH: Error 19809 Creating archive log file to 'D:\APP\ADMINISTRATOR\FAST_RECOVE
RY_AREA\MNSBDB\ARCHIVELOG\2016_09_12\O1_MF_1_454_%U_.ARC'
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\trace\mnsbdb_ora_25
88.trc:
ORA-16038: log 8 sequence# 454 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 8 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\MNSBDB\REDO08.LO
G'
USER (ospid: 2588): terminating the instance due to error 16038
ARC0: STARTING ARCH PROCESSES
2016-09-12 12:46:03.347000 +05:30
Logins disabled; aborting ARCH process startup (1092)
ARC0: Archival disabled due to shutdown: 1092
Shutting down archive processes
Archiving is disabled
System state dump requested by (instance=1, osid=2588), summary=[abnormal instan
ce termination].
System State dumped to trace file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\
trace\mnsbdb_diag_4064.trc
Dumping diagnostic data in directory=[cdmp_20160912124603], requested by (instan
ce=1, osid=2588), summary=[abnormal instance termination].
2016-09-12 12:46:05.266000 +05:30
Instance terminated by USER, pid = 2588

So, I finally assigned more size to Flash recovery area using following command, and database opened successfully.

 SQL> startup mount
 ORACLE instance started.
 Total System Global Area 8584982528 bytes
 Fixed Size 2262088 bytes
 Variable Size 4462742456 bytes
 Database Buffers 4093640704 bytes
 Redo Buffers 26337280 bytes
 Database mounted.
SQL> alter system set db_recovery_file_dest_size=100G scope=both;
SQL> alter database open;

I strongly recommend you to monitor your every database move from ADRCI, this will easy for your to understand problem area and act accordingly.

Your comments highly appreciated.

Stay Tune. 🙂

Oracle 11g Logo

RMAN-06900 RMAN-06901 ORA-19921

Received RMAN-06900,RMAN-06901 and ORA-19921 while logged in to RMAN prompt to verify daily backups. (OS:Redhat linux 6.5)

WARNING message:

[oracle@primary logs]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Fri Aug 26 10:34:39 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: BLADE1 (DBID=3381279798)
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded
RMAN>

On further diagnosing and suggested by OERR utility, I have realized there is one old active session from Aug 23, as mentioned below.

[oracle@primary ~]$ oerr ORA 19921
19921, 00000, "maximum number of %s rows exceeded"
// *Cause: The maximum number of rows in the V$RMAN_STATUS or V$RMAN_OUTPUT
// table has been exceeded.
// *Action: Close some of existing and unused RMAN connections and sessions.

So lets find out old session with ps -ef:

[oracle@primary logs]$ ps -ef|grep rman
oracle 543 32693 0 10:50 pts/4 00:00:00 grep rman
oracle 19783 19782 47 Aug23 ? 1-07:13:42 rman target /

There is one old active session above, OS ID: 19783, since 23 aug.

[oracle@primary ~]$ date
Fri Aug 26 10:57:13 IST 2016
[oracle@primary logs]$ kill -9 19783

After successfully killed old session, we are no more facing mentioned list of warnings, as below:

[oracle@primary logs]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Fri Aug 26 10:50:54 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: BLADE1 (DBID=3381279798)
RMAN>

Thanks,

Stay Tune. 🙂

Oracle 11g Logo

ORA-00600: internal error code, arguments:[4194] ORA-00603: ORACLE server session terminated by fatal error. PMON terminating the instance due to error 472

Oracle instance terminated immediate after instance startup due to following mentioned oracle error:

Cause:
A mismatch has been detected between Redo records and rollback (Undo) records.
We are validating the Undo record number relating to the change being applied against the maximum undo record number recorded in the undo block. This error is reported when the validation fails. This error may indicate a rollback segment corruption.
Note: This may require a recovery from a database backup depending on the situation.

Adrci logs are as follows:

Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_smon_11168.trc (incident=1014790):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014790/db1_smon_11168_i1014790.trc
2016-06-30 11:13:35.355000 +05:30
Starting background process QMNC
QMNC started with pid=25, OS id=11249
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Completed: ALTER DATABASE OPEN
2016-06-30 11:13:36.558000 +05:30
db_recovery_file_dest_size of 163840 MB is 1.12% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery stopped at EOT rba 13618.106.16
Block recovery completed at rba 13618.106.16, scn 0.3980712368
Doing block recovery for file 3 block 160
Resuming block recovery (PMON) for file 3 block 160
Block recovery from logseq 13618, block 56 to scn 3980712329
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.59.16, scn 0.3980712330
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_smon_11168.trc:
ORA-01595: error freeing extent (10) of rollback segment (3))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Starting background process SMCO
SMCO started with pid=29, OS id=11268
Trace dumping is performing id=[cdmp_20160630111337]
2016-06-30 11:13:37.819000 +05:30
Starting background process CJQ0
CJQ0 started with pid=28, OS id=11276
2016-06-30 11:13:41.579000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc (incident=1014942):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014942/db1_j001_11285_i1014942.trc
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j004_11291.trc:
2016-06-30 11:13:43.111000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j000_11283.trc (incident=1014934):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014934/db1_j000_11283_i1014934.trc
Trace dumping is performing id=[cdmp_20160630111343]
2016-06-30 11:13:45.937000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j000_11283.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc (incident=1014943):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014943/db1_j001_11285_i1014943.trc
2016-06-30 11:13:46.903000 +05:30
Trace dumping is performing id=[cdmp_20160630111346]
2016-06-30 11:13:48.279000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Trace dumping is performing id=[cdmp_20160630111349]
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1019488/db1_j001_11285_i1019488.trc
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1019488/db1_j001_11285_i1019488.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Trace dumping is performing id=[cdmp_20160630111351]
2016-06-30 11:13:53.489000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_pmon_11144.trc (incident=1014702):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014702/db1_pmon_11144_i1014702.trc
2016-06-30 11:13:55.766000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_pmon_11144.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 11144): terminating the instance due to error 472
Instance terminated by PMON, pid = 11144

Workaround:
Startup database in mount mode:

SQL> startup mount;

Check current undo tablespace:

SQL> show parameter undo_tablespace;

Alter undo management to ‘MANUAL’:

SQL> alter system set undo_management='MANUAL' scope=spfile;

Bounce database to open mode, meanwhile please verify adrci logs, if there is problem.

SQL> shut immediate;
 SQL> startup;

Create new undo tablespace if everything is fine in adrci logs:

SQL> create undo tablespace newundotbs datafile '/u02/oradata/db1/newundotbs01.dbf' size 5G autoextend on next 300M maxsize 31G;

Change default undo tablespace to new one:

SQL> alter system set undo_tablespace='NEWUNDOTBS' scope=spfile;

Alter undo management to ‘AUTO’:

SQL> alter system set undo_management='AUTO' scope=spfile;

Bounce the database:

SQL> shut immediate;
SQL> startup;

Verify current undo tablespace:

SQL> show parameter undo_tablespace

done.

Thank you.
Stay Tune. 🙂

Oracle 11g Logo

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.DISPATCH_WORK_ITEMS and KUPW$WORKER.DISPATCH_WO ORA-01187: cannot read from file because it failed verification tests

Data pump export backup failed due to below mentioned oracle errors. It was failed due to verification tests failed on temp file: temp01. As a part of solution, I have delete old temporary tablespace and added new temporary tablespace with new temp file.

Data Pump Export Error Logs:

Export: Release 11.2.0.1.0 - Production on Mon Jun 20 11:47:01 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Release 11.2.0.1.0 - 64bit Production
Starting "SYSTEM"."SYS_EXPORT_FULL_09": system/******** directory=data_pump_bkup dumpfile=full_db_export.dmp logfile=export.log full=y
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.DISPATCH_WORK_ITEMS []
ORA-01187: cannot read from file because it failed verification tests
ORA-01110: data file 201: '/u02/oradata/db1/temp01.dbf'
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.KUPW$WORKER", line 8159
----- PL/SQL Call Stack -----
object line object
handle number name
0x1ed213be8 19028 package body SYS.KUPW$WORKER
0x1ed213be8 8191 package body SYS.KUPW$WORKER
0x1ed213be8 8980 package body SYS.KUPW$WORKER
0x1ed213be8 1651 package body SYS.KUPW$WORKER
0x1b97859d0 2 anonymous block
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.DISPATCH_WO
ORA-01187: cannot read from file because it failed verification te
ORA-01110: data file 201: '/u02/oradata/db1/temp01.dbf'
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.KUPW$WORKER", line 8159
----- PL/SQL Call Stack -----
object line object
handle number name
0x1ed213be8 19028 package body SYS.KUPW$WORKER
0x1ed213be8 8191 package body SYS.KUPW$WORKER
0x1ed213be8 8980 package body SYS.KUPW$WORKER
0x1ed213be8 1651 package body SYS.KUPW$WORKER
0x1b97859d0 2 anonymous block
Job "SYSTEM"."SYS_EXPORT_FULL_09" stopped due to fatal error at 11:

Solution:

Here is article on Delete old temporary tablespace and add new temporary tablespace with new temp file.

Thank you. Stay Tune. 🙂

Oracle 11g Logo

ORA-25153: “Temporary Tablespace is Empty” although temporary tablespace and tempfile is available

According to oracle docs, the cause of the oracle error is to attempt was made to use space in a temporary tablespace with no files(datafile). And based on that action suggested is to add tempfile to the temporary tablespace with ADD TEMPFILE command.

In my case, Temporary tablespace already exists with sufficient free space into it. So I verified DEFAULT temporary tablespace, It was set to invalid temporary tablespace name.

The issue had resolved by setting default temporary tablespace to valid one.

Following query will help you find out default temporary tablespace:

SQL> select property_name, property_value from database_properties where

property_name='DEFAULT_TEMP_TABLESPACE';

Following query will help you to set default temporary tablespace:

SQL> alter database default temporary tablespace NEWTEMP;

Thanks, Comments are highly appreciated.
Stay Tune. 🙂

Oracle 11g Logo

RMAN-06026: some targets not found – aborting restore, RMAN-06023: no backup or copy of datafile X found to restore

RMAN restore would failed with RMAN-06026 and RMAN-06023 after successfully restoration of controlfile. Logs are as follows:

Error log:

RMAN> restore database;
Starting restore at 15-FEB-16
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/15/2016 14:52:59
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 3 found to restore
RMAN-06023: no backup or copy of datafile 2 found to restore
RMAN-06023: no backup or copy of datafile 1 found to restore

As per oracle error RMAN-06023 means RMAN cannot find a backup for that datafile in its repository, so I cross verified RMAN backup files physically on disk as well as into RMAN repository with following RMAN commands. The backups are marked as AVAILABLE. Everything was pretty perfect.

RMAN> list backup of database summary;

AND

RMAN> list backup of datafile 1;
RMAN> list backup of datafile 2;
RMAN> list backup of datafile 3;
RMAN> list backup of datafile 4;

OR

RMAN> list backup of archivelog sequence;

After drilled down everything, I came to know that there are some backup files in FRA that are belongs to OLD INCARNATION than available backup CURRENT INCARNATION.
The problem is here, while restoring RMAN backups, RMAN implicitly perform cross verification to FRA(If FRA defined in controlfile) and if new files encounter it will be catalog(registered) automatically. This prevents database from restoring backups that belong to old CURRENT incarnation. Resulting RMAN would not allowed to restored this case. Backup incarnation and the CURRENT incarnation in controlfile should be same.

There are 3 Solutions to this error:

First: You can move old RMAN backup files(that belongs to old incarnation) to outside FRA and begin your restoration again with restoration of controlfile.

 

Second: Disabled FRA for the temporary purpose while RMAN backup being restore.

SQL> shutdown immediate;
SQL> create pfile from spfile;
comment *.db_recovery_file_dest and *.db_recovery_file_dest_size in pfile.
SQL> startup nomount pfile='.... init.ora'

Restart the restore controlfile and then restore/recover database commands…

 

Third: Resetting incarnation to the older incarnation, as below. In my case, following work around works like a charm.

RMAN> reset database to incarnation 3;
database reset to incarnation 3

After resetting database incarnation, restore database successful.

Folks, Your valuable comments are highly appreciated.

Thank you,
Stay Tune. 🙂

Oracle 11g Logo

ORA-01031: insufficient privileges – “SQLPLUS / as sysdba” working, but “SQLPLUS sys/sys@YOUTH as sysdba” is not working.

I have faced ORA-01031 oracle error after migrating oracle database from oracle 10g (10.2.0.4.0) to Oracle 11g (11.2.0.1.0)
Error log is as follows:

[oracle@newyouth dbs]$ sqlplus sys/sys@blade1 as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Feb 10 04:15:50 2016
Copyright (c) 1982, 2009, Oracle. All rights reserved.
ERROR:
ORA-01031: insufficient privileges

But, I can connect the database with SYS user as follow, but not able to connect SYS user with credentials cum connection string(as mentioned above).

[oracle@newyouth dbs]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Feb 10 04:15:29 2016
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
  • Oracle error itself pretty self explanatory, according to error I have cross checked all the permissions but it doesn’t work for me.
  • Listener.ora and tnsnames.ora files also verified and found correct.
  • Oracle environment variables (like ORACLE_SID,ORACLE_HOME and PATH) also set appropriately with no mistake.
  • REMOTE_LOGIN_PASSWORDFILE parameter set to EXCLUSIVE in pfile/spfile.

Password file also created with following command.

orapwd FILE=orapwblade1 PASSWORD=sys entries=30

Finally, I got the clue from v$pwfile_users dictionary view, This view lists users who have been granted SYSDBA and SYSOPER privileges, So following query must fetch something as I have already created password file. But no luck. Oops.
SQL> select * from v$pwfile_users;
no rows selected

I have double check “$ORACLE_HOME/dbs” directory and came to know there was naming convention mistake with password file(Case sensitivity with ORACLE SID).

[oracle@newyouth dbs]$ ll
total 24
-rw-rw----. 1 oracle oinstall 1544 Feb 10 04:00 hc_BLADE1.dat
-rw-r--r--. 1 oracle oinstall 1018 Feb 10 03:58 initBLADE1.ora
-rw-r-----. 1 oracle oinstall 24 Feb 8 02:24 lkBLADE1
-rw-r-----. 1 oracle oinstall 5120 Feb 10 04:16 orapwblade1
-rw-r-----. 1 oracle oinstall 3584 Feb 10 04:00 spfileBLADE1.ora

So as a part of the solution to my scenario, I have dropped an existing password file and created new one by following command.

orapwd FILE=orapwBLADE1 PASSWORD=sys entries=30

Finally, I got access and v$pwfile_user dictionary view fetched SYS user entry.

[oracle@newyouth dbs]$ sqlplus sys/sys@blade1 as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Feb 10 04:18:50 2016
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select * from v$pwfile_users;
USERNAME SYSDB SYSOP SYSAS
------------------------------ ----- ----- -----
SYS TRUE TRUE FALSE

Cheers!!
Stay Tune. 🙂

Oracle 11g Logo

ORA-01182: cannot create database file X – file is in use or recovery ORA-01111: name for data file X is unknown – rename to correct file

On Physical Standby Data Guard, Following command failed with ORA-01182. Error itself self explanatory.
As a part of solution, you just need to End up recovery sessions and bounce back your database to Mount mode and issue SQL command again.

Error Log:

SQL> alter database create datafile '/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100' as '/u03/Oradata/Prod_OTHER_0072.DAT';
alter database create datafile '/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100' as '/u03/Oradata/Prod_OTHER_0072.DAT'
*
ERROR at line 1:
ORA-01182: cannot create database file 10 - file is in use or recovery
ORA-01111: name for data file 10 is unknown - rename to correct file
ORA-01110: data file 10:
'/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100'

You just can’t create a datafile that already being online or is being recovered.

Solution:

SQL> shutdown immediate;
SQL> startup mount;
SQL>  < Issue above SQL Command >;

 

Thanks,

Stay Tune. 🙂

Oracle 11g Logo

ORA-27047: unable to read the header block of file Linux-x86_64 Error: 25: Inappropriate ioctl for device ORA-27048: skgfifi: file header information is invalid ORA-27072: File I/O error

While recovering my manual physical standby data guard configuration, Every time oracle would apply all archive logs to standby database and end up with following mentioned errors.

As a part of troubleshooting, I have Verified:

  • Archive log sequence number on both the side but that was in place.
  • Block corruptions on datafiles level on both sides. It was clean.
  • Physical storage blocks, It was also clean.
  • Almost verify everything, that i can.

After scratching my head for several hours, I found non oracle files situated on FRA. After moving it from FRA, Recovery process successfully carried out. Non oracle file was only culprit to the following mentioned errors. It was very strange case I have faced in my life.

Solution:

Remove all non oracle files from FRA.

Error logs:

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 15198931 generated at 07/23/2015 17:55:02 needed for thread 1
ORA-00289: suggestion :
/u02/oradata/flash_recovery_area/DB1/archivelog/2015_07_23/o1_mf_1_5180_bv1qlby6_.arc
ORA-00280: change 15198931 for thread 1 is in sequence #5180
Specify log: {=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 15199107 generated at 07/23/2015 18:00:02 needed for thread 1
ORA-00289: suggestion :
/u02/oradata/flash_recovery_area/DB1/archivelog/2015_07_23/o1_mf_1_5181_bv1qvq3n_.arc
ORA-00280: change 15199107 for thread 1 is in sequence #5181
ORA-00278: log file '/u02/oradata/flash_recovery_area/DB1/archivelog/2015_07_23/o1_mf_1_5180_bv1qlby6_.arc' no longer needed for this recovery
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 1
ORA-27048: skgfifi: file header information is invalid
ORA-27072: File I/O error
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 4
Additional information: 1
Additional information: 392
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 1
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 1
ORA-27048: skgfifi: file header information is invalid
ORA-27072: File I/O error
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 4
Additional information: 1
Additional information: 392
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropri

Thanks,

Stay Tune. 🙂

Oracle 12c Logo

ORA-30013: undo tablespace ‘UNDOTBS1’ is currently in use

ORA-30013: undo tablespace ‘UNDOTBS1’ is currently in use

ORA error message itself is self explanatory, When undo tablespace is in use and we will try to drop that tablespace than oracle pop up with ORA-30013 error.

Solution:
Wait until active transactions are COMMITED or ROLLED BACK. If so, you can easily drop mentioned tablespace.
OR
You can kill that transaction if it unnecessary.

Consider following example:

SQL> create tablespace newundotbs datafile '/u01/app/oracle/oradata/prod/newundotbs1.dbf' size 200M autoextend on maxsize 500M;
Tablespace created.
SQL> alter system set UNDO_TABLESPACE=NEWUNDOTBS scope=spfile;
System altered.

Following command will failed because there is active transactions.

SQL> drop tablespace undotbs1 including contents and datafiles;
drop tablespace undotbs1 including contents and datafiles
*
ERROR at line 1:
ORA-30013: undo tablespace 'UNDOTBS1' is currently in use

Query dba_rollback_segs to know list of online segments:

SQL> select tablespace_name, owner, segment_name, status from dba_rollback_segs where TABLESPACE_NAME='UNDOTBS1' and status='ONLINE';

Retrive SID and SERIAL# from below mentioned query in order to kill session with ‘alter system kill session’ command.

SQL> select SID, substr(username,1,10) username,serial#,segment_name from v$transaction,dba_rollback_segs,v$session where saddr=ses_addr and xidusn=segment_id;
SQL> alter system kill session 'SID,SERIAL#';
SQL> drop tablespace undotbs1 including contents and datafiles;
Tablespace dropped.

Tablespace successfully dropped.

Thanks
Stay Tune 🙂

Oracle 12c Logo

ORA-03206: maximum file size of XXXX blocks in AUTOEXTEND clause is out of range

If you are creating SMALLFILE datafile (default) with 8k blocks with datafile size greater than 32 GB than oracle will pop up with ORA-03206 error.

Because oracle will allow only up to 32 GB of datafile size limit with 8k blocks, maximum data blocks per datafile : 2^22 -1 = 4194303. So oracle simply won’t allow you in this case.

If you wish more than 32 GB datafile with 8k blocks than your datafile should be create with BIGFILE keyword.

Calculations are as follows:

Oracle allows max number of datafile is 65533 and Maximum data blocks per datafile is 2^22 -1 = 4194303 blocks.

Formula to calculate max datafile size

max datafile size = db_block_size * maximum number of blocks

In our case:

max datafile size = 8 * 4194303 = 33554424/1024 = 32767.9921875 MB/1024 = 31.99999237060547 GB, i.e. 32 GB

Formula to calculate Max tablespace size:

Max tablespace size = 32 GB * max database file = 32 GB * 65533 = 2097056 GB/1024 = 2047.90625 TB

Error Log:

SQL> create temporary tablespace temp02
tempfile '/u02/oradata/db1/temp2_01.dbf' size 30G autoextend on maxsize 70G; 2
create temporary tablespace temp02
*
ERROR at line 1:
ORA-03206: maximum file size of (9175040) blocks in AUTOEXTEND clause is out of range

Solution:

Create tempfile with less than 32 GB size or use BIGFILE keyword.

In above example, I am facing ORA-03206 with tempfile creation, you may face the problem with datafile too. Same solution will be applicable.

Other area’s you may get same error:

If your existing datafile exceeds limit of 32 GB, than you will receive same error message, solution is to add new SMALLFILE datafile to the associated tablespace.

Stay Tune. 🙂

Oracle 12c Logo

ORA-01045: user C##SCOTT lacks CREATE SESSION privilege; logon denied

Oracle Error:

ORA-01045: user C##SCOTT lacks CREATE SESSION privilege; logon denied

Scenario:

Here, I was trying to access C##SCOTT user immediately after creation of it, Above mentioned oracle error itself self-explanatory. I had forgotten to grant CONNECT privilege to C##SCOTT user. After granting mentioned privileges, I was able to connect C##SCOTT user.

Error logs and solution:

SQL> create user C##SCOTT identified by tiger default tablespace users;
User created.
SQL> exit

Trying to connect C##SCOTT user:

[oracle@OL712c ~]$ sqlplus c##scott/tiger
SQL*Plus: Release 12.1.0.2.0 Production on Wed Dec 17 05:13:19 2014
Copyright (c) 1982, 2014, Oracle. All rights reserved.
ERROR:
ORA-01045: user C##SCOTT lacks CREATE SESSION privilege; logon denied

Connect as sysdba and grant required privileges:

[oracle@OL712c ~]$ sqlplus / as sysdba
SQL> grant connect,resource to C##SCOTT;
Grant succeeded.
SQL> exit

Successfully connect C##SCOTT after assigning required privileges.

[oracle@OL712c ~]$ sqlplus c##scott/tiger
SQL> show user;
USER is "C##SCOTT"

It was only problem with required privilege.

Oracle 12c Logo

ORA-01950: no privileges on tablespace ‘USERS’

Oracle Error:

ORA-01950: no privileges on tablespace ‘USERS’

Scenario:

I am trying to insert values in EMP table of C##SCOTT schema, but getting filled with above mentioned oracle error, Error itself self explanatory. I have forgotten to assign quota space for C##SCOTT on USERS tablespace. After assigning quota space on USERS, everything was smooth.

Error logs and solution:

On C##SCOTT schema:

SQL> show user;
USER is "C##SCOTT"
SQL> insert into EMP values(1,'scott',1234567898);
insert into EMP values(1,'scott',1234567898)
 *
ERROR at line 1:
ORA-01950: no privileges on tablespace 'USERS'

Connect as sysdba and assign quota to C##SCOTT on USERS tablespace:

[oracle@OL712c ~]$ sqlplus / as sysdba
SQL> show user;
USER is "SYS"
SQL> alter user c##scott quota unlimited on users;
User altered.

On C##SCOTT schema again:

SQL> show user;
USER is "C##SCOTT"
SQL> insert into EMP values(1,'scott',1234567898);
1 row created.

In my case, it was problem with quota allocation for C##SCOTT schema on USERS tablspace.

Oracle 11g Logo

ORA-01149:cannot shutdown-file 1 has online backup set OR ORA-10873:file 1 needs to be either taken out of backup mode or media recovered

Oracle Error: While shutdown your database you may encounter following oracle error.

SQL> shutdown immediate;
ORA-01149: cannot shutdown – file 1 has online backup set
ORA-01110: data file 1: ‘/u01/app/oracle/oradata/RTS/system01.dbf’

Cause:

If you try to shutdown database when database is in hot backup mode, you will encountered above mentioned error. Kindly verify with your team mates with hot backup activity, Oracle will not allow you to shutdown your database in above scenario, i.e. shutdown, shutdown immediate, shutdown transactional OR startup force will be failed with ORA-01149 oracle error.

Root cause detection:

Following query will help you to understand which oracle tablespace datafile is in hot backup mode.

SQL> select a.tablespace_name, b.status from dba_data_files a, v$backup b where a.file_id=b.file# order by tablespace_name;
TABLESPACE_NAME                STATUS
------------------------------ ------------------
SYSAUX                         ACTIVE
SYSTEM                         ACTIVE
UNDOTBS1                       ACTIVE
USERS                          ACTIVE

Status of the all datafile seems to be active. i.e. hot backup mode.

Solution:
Issue following SQL command in order to move out database from hot backup mode:

SQL> alter database end backup;
Database altered.

Now you can gracefully shutdown your database.

 

Cause: ORA-10873

In case you force fully trying to shutdown database with “shutdown abort” OR instance crashes for some reason OR database have not been shutdown gracefully than you would encountered “ORA-10873” error while next startup of your database.

SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 1469792256 bytes
Fixed Size 2213456 bytes
Variable Size 1040189872 bytes
Database Buffers 419430400 bytes
Redo Buffers 7958528 bytes
Database mounted.
ORA-10873: file 1 needs to be either taken out of backup mode or media recovered
ORA-01110: data file 1: '/u01/app/oracle/oradata/RTS/system01.dbf'

Solution:

Bring out your database from hot backup mode with the help of following SQL:

SQL> alter database end backup;
Database altered.

OR, you can individually take out tablespaces from hot backup mode:

SQL> alter tablespace system end backup;
Tablespace altered.

Open database:

SQL> alter database open;
Database altered.

Your database opened in READ WRITE mode.

Stay Tune. 🙂

Oracle 11g Logo

ORA-16038-log 4 sequence# 10702 cannot be archived-ORA-19809: limit exceeded for recovery files-ORA-19804: cannot reclaim 52428800 bytes disk space from 107374182400 limit

My one of the RAC RMAN backup failed with above mentioned oracle error:

RMAN script and its error logs are as follows:

RMAN> #Script to run Daily Incremental Backup...
2> #Created 17 June 2014
3> #Author : Jignesh Jethwa
4> #Version 1.0
5>
6> run
7> {
8> backup as compressed backupset incremental level 1 database plus archivelog;
9> delete noprompt archivelog until time 'sysdate -1' backed up 1 times to device type disk;
10>
11> #Delete Obsolte Backups
12> delete noprompt force obsolete;
13> }
14>
Starting backup at 13-SEP-14
ORACLE error from target database:
ORA-16038: log 4 sequence# 10702 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 4 thread 2: '+DATA/****/onlinelog/group_4.270.818354365'
ORA-00312: online log 4 thread 2: '+DATA/****/onlinelog/group_4.271.818354367'
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=953 instance=Instance_name1 device type=DISK
channel ORA_DISK_1: starting compressed archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=10701 RECID=18396 STAMP=858207615
channel ORA_DISK_1: starting piece 1 at 13-SEP-14
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 09/13/2014 23:00:07
ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 52428800 bytes disk space from 107374182400 limit
Recovery Manager complete.

According to above error logs, Size allocated for database recovery file destination is exceed with its current limit.

Work around for the above mentioned error is very simple, Re-size ‘db_recovery_file_dest_size’ oracle parameter and instantiate backup again, It will work.

SQL> show parameter db_recovery_file_dest;
SQL> show parameter db_recovery_file_dest_size;

Note:

If you are on RAC environment, than you have to change same parameter on both the instances, otherwise another instance will not come up after next reboot.

Stay Tune. 🙂

Oracle 11g Logo

RMAN-06059: expected archived log not found, loss of archived log compromises recoverability

While RMAN full backup, today i have encountered “RMAN-06059: expected archived log not found, loss of archived log compromises recoverability”

RMAN error.
Solution for the mentioned error is to Crosscheck your archive logs from RMAN prompt and delete those expired logs who are not needed anymore.

While crosscheck archivelogs, RMAN will marked archives as Expired in his repository (i.e. controlfile or catalog) those who are not physically present there. So we can easily delete expired archives from controlfile or catalog repository.

Error log:

RMAN> RUN
 2> {
 3> ALLOCATE CHANNEL backup1 DEVICE TYPE DISK FORMAT 'Z:\RMAN\%U';
 4> BACKUP DATABASE PLUS ARCHIVELOG;
 5> }
using target database control file instead of recovery catalog
 allocated channel: backup1
 channel backup1: SID=70 device type=DISK
Starting backup at 10-SEP-14
 current log archived
 released channel: backup1
 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of backup plus archivelog command at 09/10/2014 18:22:35
 RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
 ORA-19625: error identifying file E:\ORACLE\--ORACLE_SID--\ORAARCH\1_35338_778192583.DBF
 ORA-27041: unable to open file
 OSD-04002: unable to open file
 O/S-Error: (OS 2) The system cannot find the file specified.
RMAN> CROSSCHECK ARCHIVELOG ALL;

Above command will give information of expired RMAN repository after verification. (i.e. in catalog or controlfile)
Above will marked archives as Expired who are not available physically and who are not required for any kind of recovery..

RMAN> DELETE EXPIRED ARCHIVELOG ALL;

Above command will delete all archive logs who are marked as expired while crosscheck.

Oracle 11g Logo

ORA-01105: mount is incompatible with mounts by other instances – ORA-19808: recovery destination parameter mismatch

After reboot of both the RAC nodes, we encountered “ORA-01105: mount is incompatible with mounts by other instances” & “ORA-19808: recovery destination parameter  mismatch” oracle error, after diagnosis i came to know about the size of the db recovery parameter, i.e. db_recovery_file_dest_size is mismatch over the nodes.

After setting identical size of the db recovery file destination, infected node up and running smooth as expected.

Error Logs:

 SQL> startup;
 ORACLE instance started.
 Total System Global Area 4275781632 bytes
 Fixed Size 2235208 bytes
 Variable Size 1694500024 bytes
 Database Buffers 2566914048 bytes
 Redo Buffers 12132352 bytes
 ORA-01105: mount is incompatible with mounts by other instances
 ORA-19808: recovery destination parameter mismatch

DB recovery parameter ‘db_recovery_file_dest_size’ verification on 2nd instance by issuing the following command:

 SQL> show parameter recovery
 NAME TYPE VALUE
 ------------------------------------ ----------- ------------------------------
 db_recovery_file_dest string +DATA
 db_recovery_file_dest_size big integer 150G
 recovery_parallelism integer 0

DB recovery parameter ‘db_recovery_file_dest_size’ verification on 1st instance by issuing the following command:

 SQL> show parameter recovery
 NAME TYPE VALUE
 ------------------------------------ ----------- ------------------------------
 db_recovery_file_dest string +DATA
 db_recovery_file_dest_size big integer 100G
 recovery_parallelism integer 0

According to above verification logs, highlighted area mismatched over the RAC nodes.

Workaround: Set ‘db_recovery_file_dest_size’ at 2nd instance to 100GB, as follows:

SQL> alter system set db_recovery_file_dest_size = 100G scope=spfile;
 System altered.

Bounce 2nd instance by normal shutdown and restart, as follows:

SQL> shutdown

 ORA-01507: database not mounted
 
 ORACLE instance shut down.
SQL> startup
 ORACLE instance started.
 Total System Global Area 4275781632 bytes
 Fixed Size 2235208 bytes
 Variable Size 1694500024 bytes
 Database Buffers 2566914048 bytes
 Redo Buffers 12132352 bytes
 Database mounted.
 Database opened.

By above workaround, Infected node up and running smooth.

 SQL> select inst_name from v$active_instances;
 INST_NAME
 ------------------------------------------------------------
 database1:>>Instance_I<<
 database2:>>Instance_II<<

Stay Tune 🙂

Oracle 11g Logo

HP Data Protector Backup Job failed with ORA-01157: cannot identify/lock data file 516 – see DBWR trace file

HP data protector daily database backup job failed after restoring database, with “ORA-01157: cannot identify/lock data file 516 – see DBWR trace file” OR “ORA-01110: data file 516: ‘/oracle//sapdata1/temp_4/temp.data4′” oracle errors.

Solution: Drop and recreate specified tempfile at desired location mentioned in error.

HP data protector backup job logs are as follows:

 BR0051I BRBACKUP 7.20 (35)
 BR0055I Start of database backup: beomxhha.anf 2014-08-13 07.30.42
 BR0484I BRBACKUP log file: /oracle/ECP/sapbackup/beomxhha.anf
 BR0477I Oracle pfile /oracle//11203/dbs/init.ora created from spfile /oracle//11203/dbs/spfile.ora
 BR0280I BRBACKUP time stamp: 2014-08-13 07.30.47
 BR0301E SQL error -1157 at location BrDbfInfoGet-1, SQL statement:
 'OPEN curs_8 CURSOR FOR'
 'SELECT TS.TABLESPACE_NAME, TS.STATUS, TS.BLOCK_SIZE, TS.CONTENTS, TS.EXTENT_MANAGEMENT, TS.BIGFILE, DF.FILE_NAME, DF.FILE_ID, NVL(DF.STATUS, ' '), NVL(DF.BYTES, 0), NVL(DF.AUTOEXTENSIBLE, 'NO'), NVL(DF.MAXBYTES, 0), NVL(DF.INCREMENT_BY, 0) FROM DBA_TABLESPACES TS, DBA_DATA_FILES DF WHERE TS.TABLESPACE_NAME = DF.TABLESPACE_NAME UNION ALL SELECT TS.TABLESPACE_NAME, TS.STATUS, TS.BLOCK_SIZE, TS.CONTENTS, TS.EXTENT_MANAGEMENT, TS.BIGFILE, TF.FILE_NAME, TF.FILE_ID * -1, NVL(TF.STATUS, ' '), NVL(TF.BYTES, 0), NVL(TF.AUTOEXTENSIBLE, 'NO'), NVL(TF.MAXBYTES, 0), NVL(TF.INCREMENT_BY, 0) FROM DBA_TABLESPACES TS, DBA_TEMP_FILES TF WHERE TS.TABLESPACE_NAME = TF.TABLESPACE_NAME ORDER BY 1, 7'
 ORA-01157: cannot identify/lock data file 516 - see DBWR trace file
 ORA-01110: data file 516: '/oracle//sapdata1/temp_4/temp.data4'
 BR0314E Collection of information on database files failed
 BR0056I End of database backup: beomxhha.anf 2014-08-13 07.30.47
 BR0280I BRBACKUP time stamp: 2014-08-13 07.30.47
 BR0054I BRBACKUP terminated with errors

Work around for the mentioned error are as follows:

Note: I would request you to consider full backup of database before doing this activity, for future safety.

Verify ‘temp.data4’ tempfile available at specified location:

ll -l /oracle//sapdata1/temp_4/temp.data4

OR verify ‘temp_4’ directory exist or not. In case of ‘temp_4’ directory not exist recreate it with mkdir linux command from oracle user.

ll -l /oracle//sapdata1/temp_4/

Verify tempfile status with the help of following query:

 select file#,status,name from v$tempfile;
 OR
 select file_name,status from dba_temp_files;

Drop mentioned tempfile and recreate it, with the help of following SQL command:

alter database tempfile '/oracle//sapdata1/temp_4/temp.data4' drop;
 OR
 alter database datafile 516 OFFLINE DROP;
alter tablespace temp add tempfile '/oracle//sapdata1/temp_4/temp.data4' size 10G autoextend on maxsize unlimited;

After dropping and recreating tempfile, HP data protector test backup run successful.

Stay Tune. 🙂