DBVisit standby out of sync due network fluctuations & requested archive log file not in place.

Due to heavy network outage at customer end between primary and standby database server, database not in sync, so as a part of quick solution, I tried synchronize it with following mentioned DBVisit command. Unfortunately, it failed to synchronize due to requested archive log not available in place. It was deleted by the RMAN backup as a part of archive log retention policy.

[oracle@PR standby]$ dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28108)
dbvisit started on PR: Mon Jan 2 10:19:24 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
201701021019 - Log transfer for prodDB terminated:
Dbvisit Standby searched for log files, but could not find log file to transfer for sequence 145082.
Please check if log file for sequence 145082 is available on PR.
If using setting ARCHLOG_PREFIX, then please check if this is specified correctly
You can run Dbvisit Standby with -R option to resynch archive logs with standby database, but
if you are missing archive log files that have not been transferred to the standby, then you
can make use of the Synchronize Standby Database feature, option 8 under dbvisit_setup to attempt re-syncing the
standby database using incremental backups, else please contact Dbvisit support.
Dbvisit Standby terminated.
Return code = 460
(Tracefile required if contacting Dbvisit Standby support: /usr/local/dbvisit/standby/trace/28108_dbvisit_prodDB_201701021019.trc (server:PR))

Archive log sequence 145082 not available in following directory:

[oracle@PR standby]$ cd /BACKUP/arch/prodDB

[oracle@PR prodDB]$ ls
1_145091_775842324.dbf 1_145107_775842324.dbf 1_145123_775842324.dbf
1_145092_775842324.dbf 1_145108_775842324.dbf 1_145124_775842324.dbf
1_145093_775842324.dbf 1_145109_775842324.dbf 1_145125_775842324.dbf
1_145094_775842324.dbf 1_145110_775842324.dbf 1_145126_775842324.dbf
1_145095_775842324.dbf 1_145111_775842324.dbf 1_145127_775842324.dbf
1_145096_775842324.dbf 1_145112_775842324.dbf 1_145128_775842324.dbf
1_145097_775842324.dbf 1_145113_775842324.dbf 1_145129_775842324.dbf
1_145098_775842324.dbf 1_145114_775842324.dbf 1_145130_775842324.dbf
1_145099_775842324.dbf 1_145115_775842324.dbf 1_145131_775842324.dbf
1_145100_775842324.dbf 1_145116_775842324.dbf 1_145132_775842324.dbf
1_145101_775842324.dbf 1_145117_775842324.dbf 1_145133_775842324.dbf
1_145102_775842324.dbf 1_145118_775842324.dbf 1_145134_775842324.dbf
1_145103_775842324.dbf 1_145119_775842324.dbf 1_145135_775842324.dbf
1_145104_775842324.dbf 1_145120_775842324.dbf 1_145136_775842324.dbf
1_145105_775842324.dbf 1_145121_775842324.dbf 1_145137_775842324.dbf
1_145106_775842324.dbf 1_145122_775842324.dbf

Thanks to RMAN backup, requested archive logs restored from available RMAN backups(DB backup + archivelog) on primary database, as follows:

[oracle@PR prodDB]$ rman target /
RMAN> restore archivelog from logseq=145081 until logseq=145090;
Starting restore at 02-JAN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=305 device type=DISK
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145081
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145082
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145083
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145084
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jorotij4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jo rotij4_1_1.rman tag=TAG20170101T060003
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:41
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145085
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145086
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145087
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145088
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145089
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jqrott4k_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jqrott4k_1_1.rman tag=TAG20170101T090004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:40
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145090
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman tag=TAG20170101T120004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:46
Finished restore at 02-JAN-17

Verify above restore activity successful or not by below RMAN command:, all archive logs restored successfully.

RMAN> list archivelog all;
List of Archived Log Copies for database with db_unique_name prodDB
=====================================================================

Key Thrd Seq S Low Time
------- ---- ------- - ---------
134333 1 145081 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145081_775842324.dbf
134334 1 145082 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145082_775842324.dbf
134332 1 145083 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145083_775842324.dbf
134331 1 145084 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145084_775842324.dbf
134338 1 145085 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145085_775842324.dbf
===logs trimmed===

On primary database, transfer recently restored archive logs(plus rest) to standby database server by DBVisit following command:

[oracle@PR ~]$ cd /usr/local/dbvisit/standby/
[oracle@PR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28537)
dbvisit started on PR: Mon Jan 2 10:31:59 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
> Transferring '1_145082_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4444 KB/s] - done.
> Transferring '1_145083_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4149 KB/s] - done.
===logs trimmed===
> Transferring '1_145137_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [563 KB/s] - done.
> Transferring '1_145138_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [3340 KB/s] - done.
57 archive log transfers to DR for prodDB completed.
Last sequence was 145138.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 3
Config: archive backup count = 0
Config: diskspace full threshold = 95%
Current disk percent full (/BACKUP/arch/prodDB) = 3%
Number of archive logs deleted = 0
=============================================================
dbvisit ended on PR: Mon Jan 2 10:50:41 2017
=============================================================

On standby database, apply all recently received archive logs with following:

[oracle@DR ~]$ cd /usr/local/dbvisit/standby/
[oracle@DR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 4810)
dbvisit started on DR: Mon Jan 2 12:10:28 2017 ()
=============================================================
>>> Log file(s) for prodDB from PR will be applied to DR
201701021210 - Log seq 145082 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145083 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145084 thread 1 applied to standby database prodDB.
===logs trimmed===
201701021210 - Log seq 145139 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145140 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145141 thread 1 applied to standby database prodDB.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 7
Config: diskspace full threshold = 90%
Processing /ifs/oracle/dbvarchive/...
Archive log dir: /ifs/oracle/dbvarchive/
Total number of archive files : 409
Number of archive logs deleted = 60
Current Disk percent full : 82%
=============================================================
dbvisit ended on DR: Mon Jan 2 12:38:15 2017
=============================================================

Primary and standby database is in sync now.

Thank you.

Stay tune. 🙂

2 thoughts on “DBVisit standby out of sync due network fluctuations & requested archive log file not in place.

  1. Thanks Jignesh, I am facing same problem that you demonstrated, got clear picture, keep posting, I will refer this sites to my fellow DBA’s.

Leave a Reply

%d bloggers like this: