DBvisit Standby

Restarting Dbvisit Standby components – Dbvnet and Dbvserver

Step 1:

Stop Dbvserver:

Following command is used to stop Dbvisit Standby Web Based Interface, i.e. Dbvserver.

Note: command won’t return any output once completed.

cd /usr/dbvisit/dbvserver
./dbvserverd stop

Step 2:

Stop Dbvnet:

Following command is sued to stop Dbvisit Standby Network Communication, i.e. Dbvnet.

cd /usr/dbvisit/dbvnet
./dbvnetd stop

Verify that, above command successful and no dbvnet or dbvserver processes still running on the server.

ps -ef|egrep "dbvnet|dbvserver" |grep -v grep

Note: In case of processes still running, kindly kill them with linux kill command.

Step 3:

Remove all the temporary files (temporary cache), its recommended to clear all before start.

rm -Rf /tmp/par-*
rm -Rf /tmp/dbvtmp-*
rm -Rf /tmp/p2xtmp-*
rm -Rf /tmp/pdk-*

Step 4:

Start Dbvisit Standby Network Communication, i.e. Dbvnet. Kindly remove old log files from the log directory.

cd /usr/dbvisit/dbvnet
rm log/*
./dbvnetd start

Step 5:

Start Dbvisit Standby Web Based Interface, i.e. Dbvserver. Kindly remove old log files from the log directory.

cd /usr/dbvisit/dbvserver
rm log/*
./dbvserverd start

Step 6:

Verify all processes are up and running with following linux command:

[oracle@PR ~]$ ps -ef|egrep "dbvnet|dbvserver" |grep -v grep
oracle 28031 1 0 Jan02 ? 00:00:10 ./dbvnetd start
oracle 28032 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28034 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28035 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28036 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28048 1 0 Jan02 ? 00:00:10 ./dbvserverd start
oracle 28049 28048 0 Jan02 ? 00:00:03 ./dbvserverd start
oracle 28052 28048 0 Jan02 ? 00:00:00 ./dbvserverd start
oracle 28053 28048 0 Jan02 ? 00:00:00 ./dbvserverd start

Thanks.

Stay Tune. 🙂

DBvisit Standby

DBVisit standby out of sync due network fluctuations & requested archive log file not in place.

Due to heavy network outage at customer end between primary and standby database server, database not in sync, so as a part of quick solution, I tried synchronize it with following mentioned DBVisit command. Unfortunately, it failed to synchronize due to requested archive log not available in place. It was deleted by the RMAN backup as a part of archive log retention policy.

[oracle@PR standby]$ dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28108)
dbvisit started on PR: Mon Jan 2 10:19:24 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
201701021019 - Log transfer for prodDB terminated:
Dbvisit Standby searched for log files, but could not find log file to transfer for sequence 145082.
Please check if log file for sequence 145082 is available on PR.
If using setting ARCHLOG_PREFIX, then please check if this is specified correctly
You can run Dbvisit Standby with -R option to resynch archive logs with standby database, but
if you are missing archive log files that have not been transferred to the standby, then you
can make use of the Synchronize Standby Database feature, option 8 under dbvisit_setup to attempt re-syncing the
standby database using incremental backups, else please contact Dbvisit support.
Dbvisit Standby terminated.
Return code = 460
(Tracefile required if contacting Dbvisit Standby support: /usr/local/dbvisit/standby/trace/28108_dbvisit_prodDB_201701021019.trc (server:PR))

Archive log sequence 145082 not available in following directory:

[oracle@PR standby]$ cd /BACKUP/arch/prodDB

[oracle@PR prodDB]$ ls
1_145091_775842324.dbf 1_145107_775842324.dbf 1_145123_775842324.dbf
1_145092_775842324.dbf 1_145108_775842324.dbf 1_145124_775842324.dbf
1_145093_775842324.dbf 1_145109_775842324.dbf 1_145125_775842324.dbf
1_145094_775842324.dbf 1_145110_775842324.dbf 1_145126_775842324.dbf
1_145095_775842324.dbf 1_145111_775842324.dbf 1_145127_775842324.dbf
1_145096_775842324.dbf 1_145112_775842324.dbf 1_145128_775842324.dbf
1_145097_775842324.dbf 1_145113_775842324.dbf 1_145129_775842324.dbf
1_145098_775842324.dbf 1_145114_775842324.dbf 1_145130_775842324.dbf
1_145099_775842324.dbf 1_145115_775842324.dbf 1_145131_775842324.dbf
1_145100_775842324.dbf 1_145116_775842324.dbf 1_145132_775842324.dbf
1_145101_775842324.dbf 1_145117_775842324.dbf 1_145133_775842324.dbf
1_145102_775842324.dbf 1_145118_775842324.dbf 1_145134_775842324.dbf
1_145103_775842324.dbf 1_145119_775842324.dbf 1_145135_775842324.dbf
1_145104_775842324.dbf 1_145120_775842324.dbf 1_145136_775842324.dbf
1_145105_775842324.dbf 1_145121_775842324.dbf 1_145137_775842324.dbf
1_145106_775842324.dbf 1_145122_775842324.dbf

Thanks to RMAN backup, requested archive logs restored from available RMAN backups(DB backup + archivelog) on primary database, as follows:

[oracle@PR prodDB]$ rman target /
RMAN> restore archivelog from logseq=145081 until logseq=145090;
Starting restore at 02-JAN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=305 device type=DISK
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145081
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145082
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145083
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145084
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jorotij4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jo rotij4_1_1.rman tag=TAG20170101T060003
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:41
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145085
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145086
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145087
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145088
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145089
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jqrott4k_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jqrott4k_1_1.rman tag=TAG20170101T090004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:40
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145090
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman tag=TAG20170101T120004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:46
Finished restore at 02-JAN-17

Verify above restore activity successful or not by below RMAN command:, all archive logs restored successfully.

RMAN> list archivelog all;
List of Archived Log Copies for database with db_unique_name prodDB
=====================================================================

Key Thrd Seq S Low Time
------- ---- ------- - ---------
134333 1 145081 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145081_775842324.dbf
134334 1 145082 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145082_775842324.dbf
134332 1 145083 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145083_775842324.dbf
134331 1 145084 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145084_775842324.dbf
134338 1 145085 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145085_775842324.dbf
===logs trimmed===

On primary database, transfer recently restored archive logs(plus rest) to standby database server by DBVisit following command:

[oracle@PR ~]$ cd /usr/local/dbvisit/standby/
[oracle@PR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28537)
dbvisit started on PR: Mon Jan 2 10:31:59 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
> Transferring '1_145082_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4444 KB/s] - done.
> Transferring '1_145083_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4149 KB/s] - done.
===logs trimmed===
> Transferring '1_145137_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [563 KB/s] - done.
> Transferring '1_145138_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [3340 KB/s] - done.
57 archive log transfers to DR for prodDB completed.
Last sequence was 145138.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 3
Config: archive backup count = 0
Config: diskspace full threshold = 95%
Current disk percent full (/BACKUP/arch/prodDB) = 3%
Number of archive logs deleted = 0
=============================================================
dbvisit ended on PR: Mon Jan 2 10:50:41 2017
=============================================================

On standby database, apply all recently received archive logs with following:

[oracle@DR ~]$ cd /usr/local/dbvisit/standby/
[oracle@DR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 4810)
dbvisit started on DR: Mon Jan 2 12:10:28 2017 ()
=============================================================
>>> Log file(s) for prodDB from PR will be applied to DR
201701021210 - Log seq 145082 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145083 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145084 thread 1 applied to standby database prodDB.
===logs trimmed===
201701021210 - Log seq 145139 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145140 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145141 thread 1 applied to standby database prodDB.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 7
Config: diskspace full threshold = 90%
Processing /ifs/oracle/dbvarchive/...
Archive log dir: /ifs/oracle/dbvarchive/
Total number of archive files : 409
Number of archive logs deleted = 60
Current Disk percent full : 82%
=============================================================
dbvisit ended on DR: Mon Jan 2 12:38:15 2017
=============================================================

Primary and standby database is in sync now.

Thank you.

Stay tune. 🙂

Oracle 11g Logo

ORA-01182: cannot create database file X – file is in use or recovery ORA-01111: name for data file X is unknown – rename to correct file

On Physical Standby Data Guard, Following command failed with ORA-01182. Error itself self explanatory.
As a part of solution, you just need to End up recovery sessions and bounce back your database to Mount mode and issue SQL command again.

Error Log:

SQL> alter database create datafile '/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100' as '/u03/Oradata/Prod_OTHER_0072.DAT';
alter database create datafile '/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100' as '/u03/Oradata/Prod_OTHER_0072.DAT'
*
ERROR at line 1:
ORA-01182: cannot create database file 10 - file is in use or recovery
ORA-01111: name for data file 10 is unknown - rename to correct file
ORA-01110: data file 10:
'/u01/app/oracle/product/11.2.0/prod/dbs/UNNAMED00100'

You just can’t create a datafile that already being online or is being recovered.

Solution:

SQL> shutdown immediate;
SQL> startup mount;
SQL>  < Issue above SQL Command >;

 

Thanks,

Stay Tune. 🙂

DBvisit Standby

DBVisit standby configuration failed with ORA-19504, ORA-27040, RMAN-06026 and RMAN-06023

While configuring DBVisit standby disaster recovery software at one of my client end environment, when DBVisit standby configuration initiated it was smooth during backup of primary database, than transferred backup to DR server. When RMAN backup arrived at DR server and started creating standby database ( restore activity by RMAN ) DBVisit popped up with following error messages:

Error message 1:

ORA-19504: failed to create file "/RTS/prod/df/RTSapp_lob.dbf"
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory

Error message 2:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 06/18/2015 16:06:05
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 7 found to restore

Diagnosis:

After giving hard diagnosis on above red highlighted error messages, It is came to know that it was the problem with /RTS/prod/df/ path.

On primary server there was multiple datafiles locations on multiple storage LUN’s for various tablespaces. DBVisit simply copied references from primary database server and searching exact path on secondary database server. But obviously same path would not available on secondary database server. Plus While taking RMAN backup at primary database server, RMAN recorded all path and locations while backup and try to search same path and location at secondary database server while restore activity. Which is again impossible. Due to above reasons DBVisit finally came out of configuration activity with above mentioned errors.

Solution:

As a part of solution I would have two choices available in my hand, one is to discard all secondary server configuration and start from scratch (OS, oracle DB & DBVist installation) to meed exactly same path and partition scheme of primary database linux server.

Second is to create links between two directories.

So finally I have created directory link between /RTS/prod/df/ to the new datafile location of standby database server: /home/app/datafile/ifsprod/

lrwxrwxrwx. 1 oracle oinstall 27 Jun 18 18:21 df -> /u01/app/datafile/RTSprod/

After applying above fix, Resumes DBVisit configuration and it was successfully configured.

For more details on above DBVisit error, Please have a look here.

Dbvisit Standby Database Technology message from PR
Message received from process: dbvisit_setup --csd --web --ddc prod
(Dbvisit Standby: 7.0.38.13873 Process id: 6691)
201506181606 - Error executing RMAN command:
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jun 18 16:06:02 2015
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
RMAN> 
connected to target database: prod (DBID=4119037639, not open)
using target database control file instead of recovery catalog
RMAN> 
RMAN configuration parameters for database with db_unique_name prod are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 10 DAYS;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/ORACLE/app/oracle/product/11.2.0/db_1/dbs/snapcf_prod.f';
echo set off
RMAN> 
allocated channel: C_DBVISIT
channel C_DBVISIT: SID=202 device type=DISK
executing command: SET NEWNAME
Starting restore at 2015-06-18:16:06:04
channel C_DBVISIT: starting datafile backup set restore
channel C_DBVISIT: specifying datafile(s) to restore from backup set
channel C_DBVISIT: restoring datafile 00007 to /RTS/prod/df/RTSapp_lob.dbf
channel C_DBVISIT: reading from backup piece /MISC/temp_dbvisit/dbv_prod_csd_dbf_7_9vq9pgmr_1_1.rman
channel C_DBVISIT: ORA-19870: error while restoring backup piece /home/temp_dbvisit/dbv_prod_csd_dbf_7_9vq9pgmr_1_1.rman
ORA-19504: failed to create file "/RTS/prod/df/RTSapp_lob.dbf"
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory
failover to previous backup
released channel: C_DBVISIT
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 06/18/2015 16:06:05
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 7 found to restore
RMAN>
Recovery Manager complete.
Dbvisit Standby terminated.
Return code = 8002
(Tracefile required if contacting Dbvisit Standby support: /usr/local/dbvisit/standby/trace/5828_dbv_functions_prod_201506181606.trc (server:DR))
Dbvisit Standby terminated.
Return code = 66
(Tracefile required if contacting Dbvisit Standby support: /usr/local/dbvisit/standby/trace/6691_dbvisit_setup_csd_prod_201506181008.trc (server:PR))

 

Cheers!!

Stay tune. 🙂