Oracle 11g Logo

ORA-19625: error identifying file – ORA-27037: unable to obtain file status

Error log:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 06/23/2017 09:00:03
RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
ORA-19625: error identifying file /BACKUP/arch/ifsprod/1_153555_775842324.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

RMAN backup failed with above mentioned errors, error message itself very explanatory, required archive log missing from archive log location and RMAN didn’t get it to backup. In my case, archive logs are moved to another location in order to manage bulk space for huge database activity. Fortunately, I have found out missing one, copied to the default location and backup was successful.

But, In case it was deleted or corrupted in worst, you can issue following commands in order to continue RMAN backup.

RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;

OR

If you are using catalog database to maintain RMAN repository instead of controlfile, you can try following command:

RMAN> resync catalog;

Run backup again.

Note: One full database RMAN backup should consider, if you are taking RMAN transactactional or incremental backup.

Oracle 11g Logo

ORA-02062: distributed recovery received DBID %s, expected %s

Error log:

2017-06-21 11:56:07.973000 +05:30
Errors in file /oracle/app/oracle/diag/rdbms/vcbcbs/vcbcbs/trace/vcbcbs_reco_5696.trc:
ORA-02062: distributed recovery received DBID f2fc225a, expected f509547a

SQL> SELECT COMMIT# ,LOCAL_TRAN_ID, STATE, MIXED, HOST, GLOBAL_TRAN_ID FROM DBA_2PC_PENDING;
COMMIT#     LOCAL_TRAN_ID     STATE         MIX HOST          GLOBAL_TRAN_ID 
----------- ----------------- ------------- --- --------      ----------------
7072124803  10.22.2551715     collecting    no  DC-EDPSTAFF1  VBPROD.f3c3de87.10.22.2551715

 

SQL> SELECT LOCAL_TRAN_ID, IN_OUT, DATABASE, INTERFACE FROM DBA_2PC_NEIGHBORS;
LOCAL_TRAN_ID          IN_ DATABASE                I
---------------------- --------------------------  -
10.22.2551715          in                          N
10.22.2551715          out CBL1                    N

 

As a part of the solution, issued following command in order to purge pending distributed transaction:

Execute DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY ('10.22.2551715');

Execute DBMS_TRANSACTION.PURGE_MIXED ('10.22.2551715');

Thanks,

Stay tune. 🙂

Oracle 11g Logo

ORA-19587: error occurred reading 512 bytes at block number 1 + ORA-27072: File I/O error

My one of the client facing below mentioned Oracle errors, On diagnosis, we come to know there are some deleted archive logs entry still exists in RMAN repository, after cleaning all the bad entries from RMAN repository(controlfile in my case), the backup was successfully completed.

Error logs:

archived log file name=/u02/oradata/flash_recovery_area/DB1/archivelog/2017_01_21/o1_mf_1_253761_d866q0o3_.arc RECID=252905 STAMP=933861305
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of crosscheck command on ORA_DISK_1 channel at 05/23/2017 12:59:27
ORA-19587: error occurred reading 512 bytes at block number 1
ORA-27072: File I/O error
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 4
Additional information: 1

Solutions:

RMAN> delete force noprompt obsolete;
RMAN> delete force noprompt expired backup;
RMAN> DELETE NOPROMPT ARCHIVELOG UNTIL time 'SYSDATE-1';

In my case, the client needed only one-day archives. change last RMAN command as per your convenience.

Oracle 11g Logo

Datapump export job failed with ORA-01555: snapshot too old: rollback segment number x with name “_SYSSMU8$” too small

The ORA-1555 errors can happen when a query is unable to access enough undo to build a copy of the data when the query started.
Committed “versions” of blocks are maintained along with newer uncommitted “versions” of those blocks so that queries can access data as it existed in the database at the time of the query. These are referred to as “consistent read” blocks and are maintained using Oracle undo management.

Error logs:

ORA-31693: Table data object "RPROD"."AOUP_DAILY_STBACT_SUMM_DET" failed to load/unload and is being skipped due to error:
ORA-02354: error in exporting/importing data
ORA-01555: snapshot too old: rollback segment number 8 with name "_SYSSMU8$" too small

As a part of the solution, set undo_retention to a higher value. also, take care of the size of undo tablespace, it should be large enough.

SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ---------------
undo_management string AUTO
undo_retention integer 900
undo_tablespace string UNDOTBS1
 
SQL> ALTER SYSTEM SET UNDO_RETENTION = 1800 scope=both;
System altered.
 
SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ---------------
undo_management string AUTO
undo_retention integer 1800
undo_tablespace string UNDOTBS1

Export was successful after above changes.

Oracle 11g Logo

Alter database open failed with ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [160296], [591], [656]

After power failure at my customer end, database unable to open with above-mentioned oracle error message. Due to power failure controlfile got logical corruption.

This Problem is caused by Storage Problem of the Database Files. The Subsystem (eg. SAN) crashed while the Database was open. The Database then crashed because the Database Files were not accessible anymore. This caused a lost Write into the Online RedoLogs and/or causing logical corruption in controlfile so Instance Recovery is not possible and raising the ORA-600.

Error log:

2017-05-08 14:11:22.391000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_ora_6324.trc (incident=1968168):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [160296], [591], [656], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1968168/db1_ora_6324_i1968168.trc
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_ora_6324.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [160296], [591], [656], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_ora_6324.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [160296], [591], [656], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...
2017-05-08 14:11:23.773000 +05:30
Trace dumping is performing id=[cdmp_20170508141123]

Solution:

Retrive details of all the controlfiles as below:

[oracle@db1 ~]$ sqlplus / as sysdba
SQL> Show parameter control_files;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_files                        string      /u02/oradata/db1/control01.ctl,/u02/oradata/flash_recovery_area/db1/control02.ctl

Findout current redo log member when the power got failed.

SQL> select a.member, a.group#, b.status from v$logfile a ,v$log b where a.group#=b.group# and b.status='CURRENT' ;
MEMBE                        RGROUP#    STATUS
---------------------------- ---------- ----------------
/u02/oradata/db1/redo03.log  3          CURRENT
SQL> Shutdown abort;
ORACLE instance shut down.

Consider backup of controlfile, so that we will keep the current state of controlfile in the case of any worst.

[oracle@db1 ~]$ cp /u02/oradata/db1/control01.ctl /u02/oradata/db1/control01.ctl_backup
[oracle@db1 ~]$ cp /u02/oradata/flash_recovery_area/db1/control02.ctl /u02/oradata/flash_recovery_area/db1/control02.ctl_backup

Startup mount database:

[oracle@db1 ~]$ sqlplus / as sysdba
Connected to an idle instance.
SQL> Startup mount;
ORACLE instance started.
Total System Global Area 5010685952 bytes
Fixed Size 2212936 bytes
Variable Size 3489663928 bytes
Database Buffers 1476395008 bytes
Redo Buffers 42414080 bytes
Database mounted.

Recover database with the help of following command and input current redo log member fetched above:

SQL> recover database using backup controlfile until cancel ;
ORA-00279: change 130697634 generated at 05/08/2017 10:59:22 needed for thread1
ORA-00289: suggestion :
/u02/oradata/flash_recovery_area/DB1/archivelog/2017_05_08/o1_mf_1_160296_%u_.arc
ORA-00280: change 130697634 for thread 1 is in sequence #160296
Specify log: {=suggested | filename | AUTO | CANCEL}
/u02/oradata/db1/redo03.log
Log applied.
Media recovery complete.
SQL>
SQL> Alter database open resetlogs ;

Database altered.
SQL> select open_mode,name from v$database;

OPEN_MODE            NAME
-------------------- ---------
READ WRITE           DB1

Cheers!! Stay Tune. 🙂

Oracle 11g Logo

Query to find out progress of transaction recovery by SMON

Have you killed the long running query?

SMON is solely responsible for recovering transactions when you kill any large running query( truncate and delete ) by killing OS process or aborting database, SMON will take all possible CPU to rolling back previous state and its highly time-consuming task.
And frustrated DBA/Person will think bouncing database will resolve this problem completely, as obvious “shutdown immediate” will take equal amount of time to rollback and finally database being “aborted” by the user.
Kindly note: Shutting down or aborting database is not the solution, and won’t reduce the amount of work SMON need to be performed to complete rollback.

Following query will help you to identify the total amount of work to be rolled back by SMON, simply progress of transaction recovery. Run it multiple time.

SQL> SELECT usn, state, undoblockstotal "Total",
undoblocksdone "Done",
undoblockstotal-undoblocksdone "ToDo",
DECODE(cputime,0,'unknown',SYSDATE+(((undoblockstotal-undoblocksdone) / (undoblocksdone / cputime)) / 86400)) "Finish at"
FROM v$fast_start_transactions;

USN        STATE            Total      Done       ToDo       Finish at
---------- ---------------- ---------- ---------- ---------- --------------------
 724       RECOVERING       112623     4658       107965     09-MAR-2017 17:30:16
 737       RECOVERING       275333     1755       273578     09-MAR-2017 21:49:03

<<After some time>>

USN        STATE            Total      Done       ToDo       Finish at
---------- ---------------- ---------- ---------- ---------- --------------------
 724       RECOVERING       28647      9324       19323      09-MAR-2017 17:16:50
 737       RECOVERING       240190     2792       237398     09-MAR-2017 20:33:12

Note:

On the same time, your ADRCI database log will be flooded with following:

2017-03-09 17:17:25.063000 +05:30
SMON: Restarting fast_start parallel rollback

No action is required. No corruption applies unless the error is followed by other internal errors on the object.

Thanks

Stay tune. 🙂

Oracle 11g Logo

Generate AWR – Automatic Workload Repository (AWR) in oracle 11g

Many times DBA need to monitor specific loaded/heavy database related activity in order to fix database performance, for that generating AWR report is perfect solution and he will get all necessary information he needed to tune the database. This article is step by step approach to generate AWR report.

Step-I:

Before loaded activity, issue following command to take snapshot of database as start point of AWR report.

SQL> EXEC DBMS_WORKLOAD_REPOSITORY.create_snapshot;

Step-II: Perform your activity.

Perform your activity.

Step-III: Right after the activity, issue

Right after the activity, issue following command to take again snapshot of database as end point of AWR report.

EXEC DBMS_WORKLOAD_REPOSITORY.create_snapshot;

Note: Snapshot information can be queried from the DBA_HIST_SNAPSHOT view.

Step-IV:

Generate AWR report: (Follow the instructions and provide input acrrodingly)

SQL> @$ORACLE_HOME/rdbms/admin/awrrpti.sql

Specify the Report Type
~~~~~~~~~~~~~~~~~~~~~~~
Would you like an HTML report, or a plain text report?
Enter 'html' for an HTML report, or 'text' for plain text
Defaults to 'html'
Enter value for report_type: html
Type Specified: html

Select the appropriate instance:

Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DB Id Inst Num DB Name Instance Host
------------ -------- ------------ ------------ ------------
4119037639 1 PRODDB ifsprod localhost.lo
caldomain
4119037639 1 UATDB ifsuat localhost.lo
caldomain
* 4119037639 1 PRODDB ifsprod PR
Enter value for dbid: 4119037639
Enter value for inst_num: 1
Using 1 for instance number

Specify the number of days of snapshots to choose from:

Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed. Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days: 1
Listing the last day's Completed Snapshots
Snap
Instance DB Name Snap Id Snap Started Level
------------ ------------ --------- ------------------ -----
ifsprod IFSPROD 66276 26 Nov 2016 00:00 1
66277 26 Nov 2016 00:30 1
66278 26 Nov 2016 01:01 1
66279 26 Nov 2016 01:30 1
66280 26 Nov 2016 02:00 1
66281 26 Nov 2016 02:30 1
66282 26 Nov 2016 03:00 1
66283 26 Nov 2016 03:30 1
66284 26 Nov 2016 04:00 1
66285 26 Nov 2016 04:30 1
66286 26 Nov 2016 05:00 1
66287 26 Nov 2016 05:30 1
66288 26 Nov 2016 06:00 1
66289 26 Nov 2016 06:31 1
66290 26 Nov 2016 07:00 1
66291 26 Nov 2016 07:30 1
66292 26 Nov 2016 08:00 1
66293 26 Nov 2016 08:30 1
66294 26 Nov 2016 09:00 1
66295 26 Nov 2016 09:30 1
66296 26 Nov 2016 10:00 1
66297 26 Nov 2016 10:30 1
66298 26 Nov 2016 11:00 1
66299 26 Nov 2016 11:30 1
66300 26 Nov 2016 12:00 1
66301 26 Nov 2016 12:30 1
66302 26 Nov 2016 13:00 1
66303 26 Nov 2016 13:30 1
66304 26 Nov 2016 14:00 1
66305 26 Nov 2016 14:30 1
66306 26 Nov 2016 14:41 1
66307 26 Nov 2016 14:45 1

Specify the Start point and end point for the snapshot, input from above:

Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 66306
Begin Snapshot Id specified: 66306
Enter value for end_snap: 66307
End Snapshot Id specified: 66307

Name the report:

Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_1_66306_66307.html. To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: AWR_between_2_manual_snapshot
.
..
...
Report written to AWR_between_2_manual_snapshot
SQL> exit

search for html format snapshot in current directory.

Thanks. Stay Tune. 🙂

Change MySQL Data directory location in Windows

Step 1:

Stop the existing MySQL instance using services.msc, shutdown must be clean so that the instance flushes any pending changes to disk.

MySQL - Services.msc

MySQL – Services.msc

MySQL - Services.msc - stop

MySQL – Services.msc – stop

MySQL - Services.msc - stopped

MySQL – Services.msc – stopped

Step 2:

After successful instance shutdown, copy the data directory from existing location to the new location, in my case:

Existing location: C:\ProgramData\MySQL\MySQL Server 5.7

New Location: E:\MySQL

Step 3:

Change datadir parameter in current my.ini file. I have commented old entry and added new one:

MySQL - my.ini

MySQL – my.ini

# Path to the database root
#datadir=C:/ProgramData/MySQL/MySQL Server 5.7\Data
datadir=E:\MySQL\Data

Step 4:

Start MySQL instance using services.msc

MySQL - Services.msc - start

MySQL – Services.msc – start

MySQL - Services.msc - started

MySQL – Services.msc – started

Step 5:

After starting MySQL successfully, you can verify above changes by creating demo database and verify data dirctory for the same database. Test database: Myslott

MySQL - create database through workbench

MySQL – create database through workbench

MySQL - Data Directory

MySQL – Data Directory

Database Created successfully and myslott directory also created under new data directory.

Cheers!!

Stay Tune. 🙂

DBvisit Standby

Restarting Dbvisit Standby components – Dbvnet and Dbvserver

Step 1:

Stop Dbvserver:

Following command is used to stop Dbvisit Standby Web Based Interface, i.e. Dbvserver.

Note: command won’t return any output once completed.

cd /usr/dbvisit/dbvserver
./dbvserverd stop

Step 2:

Stop Dbvnet:

Following command is sued to stop Dbvisit Standby Network Communication, i.e. Dbvnet.

cd /usr/dbvisit/dbvnet
./dbvnetd stop

Verify that, above command successful and no dbvnet or dbvserver processes still running on the server.

ps -ef|egrep "dbvnet|dbvserver" |grep -v grep

Note: In case of processes still running, kindly kill them with linux kill command.

Step 3:

Remove all the temporary files (temporary cache), its recommended to clear all before start.

rm -Rf /tmp/par-*
rm -Rf /tmp/dbvtmp-*
rm -Rf /tmp/p2xtmp-*
rm -Rf /tmp/pdk-*

Step 4:

Start Dbvisit Standby Network Communication, i.e. Dbvnet. Kindly remove old log files from the log directory.

cd /usr/dbvisit/dbvnet
rm log/*
./dbvnetd start

Step 5:

Start Dbvisit Standby Web Based Interface, i.e. Dbvserver. Kindly remove old log files from the log directory.

cd /usr/dbvisit/dbvserver
rm log/*
./dbvserverd start

Step 6:

Verify all processes are up and running with following linux command:

[oracle@PR ~]$ ps -ef|egrep "dbvnet|dbvserver" |grep -v grep
oracle 28031 1 0 Jan02 ? 00:00:10 ./dbvnetd start
oracle 28032 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28034 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28035 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28036 28031 0 Jan02 ? 00:00:00 ./dbvnetd start
oracle 28048 1 0 Jan02 ? 00:00:10 ./dbvserverd start
oracle 28049 28048 0 Jan02 ? 00:00:03 ./dbvserverd start
oracle 28052 28048 0 Jan02 ? 00:00:00 ./dbvserverd start
oracle 28053 28048 0 Jan02 ? 00:00:00 ./dbvserverd start

Thanks.

Stay Tune. 🙂

DBvisit Standby

DBVisit standby out of sync due network fluctuations & requested archive log file not in place.

Due to heavy network outage at customer end between primary and standby database server, database not in sync, so as a part of quick solution, I tried synchronize it with following mentioned DBVisit command. Unfortunately, it failed to synchronize due to requested archive log not available in place. It was deleted by the RMAN backup as a part of archive log retention policy.

[oracle@PR standby]$ dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28108)
dbvisit started on PR: Mon Jan 2 10:19:24 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
201701021019 - Log transfer for prodDB terminated:
Dbvisit Standby searched for log files, but could not find log file to transfer for sequence 145082.
Please check if log file for sequence 145082 is available on PR.
If using setting ARCHLOG_PREFIX, then please check if this is specified correctly
You can run Dbvisit Standby with -R option to resynch archive logs with standby database, but
if you are missing archive log files that have not been transferred to the standby, then you
can make use of the Synchronize Standby Database feature, option 8 under dbvisit_setup to attempt re-syncing the
standby database using incremental backups, else please contact Dbvisit support.
Dbvisit Standby terminated.
Return code = 460
(Tracefile required if contacting Dbvisit Standby support: /usr/local/dbvisit/standby/trace/28108_dbvisit_prodDB_201701021019.trc (server:PR))

Archive log sequence 145082 not available in following directory:

[oracle@PR standby]$ cd /BACKUP/arch/prodDB

[oracle@PR prodDB]$ ls
1_145091_775842324.dbf 1_145107_775842324.dbf 1_145123_775842324.dbf
1_145092_775842324.dbf 1_145108_775842324.dbf 1_145124_775842324.dbf
1_145093_775842324.dbf 1_145109_775842324.dbf 1_145125_775842324.dbf
1_145094_775842324.dbf 1_145110_775842324.dbf 1_145126_775842324.dbf
1_145095_775842324.dbf 1_145111_775842324.dbf 1_145127_775842324.dbf
1_145096_775842324.dbf 1_145112_775842324.dbf 1_145128_775842324.dbf
1_145097_775842324.dbf 1_145113_775842324.dbf 1_145129_775842324.dbf
1_145098_775842324.dbf 1_145114_775842324.dbf 1_145130_775842324.dbf
1_145099_775842324.dbf 1_145115_775842324.dbf 1_145131_775842324.dbf
1_145100_775842324.dbf 1_145116_775842324.dbf 1_145132_775842324.dbf
1_145101_775842324.dbf 1_145117_775842324.dbf 1_145133_775842324.dbf
1_145102_775842324.dbf 1_145118_775842324.dbf 1_145134_775842324.dbf
1_145103_775842324.dbf 1_145119_775842324.dbf 1_145135_775842324.dbf
1_145104_775842324.dbf 1_145120_775842324.dbf 1_145136_775842324.dbf
1_145105_775842324.dbf 1_145121_775842324.dbf 1_145137_775842324.dbf
1_145106_775842324.dbf 1_145122_775842324.dbf

Thanks to RMAN backup, requested archive logs restored from available RMAN backups(DB backup + archivelog) on primary database, as follows:

[oracle@PR prodDB]$ rman target /
RMAN> restore archivelog from logseq=145081 until logseq=145090;
Starting restore at 02-JAN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=305 device type=DISK
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145081
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145082
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145083
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145084
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jorotij4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jo rotij4_1_1.rman tag=TAG20170101T060003
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:41
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145085
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145086
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145087
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145088
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145089
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/ar c_20170101_jqrott4k_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jqrott4k_1_1.rman tag=TAG20170101T090004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:40
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145090
channel ORA_DISK_1: reading from backup piece /USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman
channel ORA_DISK_1: piece handle=/USB_BACKUP/RMAN/Backup/trn_bkup/arc_20170101_jsrou7m4_1_1.rman tag=TAG20170101T120004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:46
Finished restore at 02-JAN-17

Verify above restore activity successful or not by below RMAN command:, all archive logs restored successfully.

RMAN> list archivelog all;
List of Archived Log Copies for database with db_unique_name prodDB
=====================================================================

Key Thrd Seq S Low Time
------- ---- ------- - ---------
134333 1 145081 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145081_775842324.dbf
134334 1 145082 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145082_775842324.dbf
134332 1 145083 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145083_775842324.dbf
134331 1 145084 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145084_775842324.dbf
134338 1 145085 A 01-JAN-17
 Name: /BACKUP/arch/prodDB/1_145085_775842324.dbf
===logs trimmed===

On primary database, transfer recently restored archive logs(plus rest) to standby database server by DBVisit following command:

[oracle@PR ~]$ cd /usr/local/dbvisit/standby/
[oracle@PR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 28537)
dbvisit started on PR: Mon Jan 2 10:31:59 2017 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Note FORCE_LOGGING is disabled in the primary database.
>>> Checking Dbvisit Standby for configurational differences between PR and DR...
No configurational differences found between PR and DR.
>>> Log file(s) for prodDB will be transferred from PR to DR...
> Transferring '1_145082_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4444 KB/s] - done.
> Transferring '1_145083_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [4149 KB/s] - done.
===logs trimmed===
> Transferring '1_145137_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [563 KB/s] - done.
> Transferring '1_145138_775842324.dbf.gz' to server DR:7891
Progress: 0%...20%...40%...60%...80%...100% [3340 KB/s] - done.
57 archive log transfers to DR for prodDB completed.
Last sequence was 145138.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 3
Config: archive backup count = 0
Config: diskspace full threshold = 95%
Current disk percent full (/BACKUP/arch/prodDB) = 3%
Number of archive logs deleted = 0
=============================================================
dbvisit ended on PR: Mon Jan 2 10:50:41 2017
=============================================================

On standby database, apply all recently received archive logs with following:

[oracle@DR ~]$ cd /usr/local/dbvisit/standby/
[oracle@DR standby]$ ./dbvisit prodDB
=============================================================
Dbvisit Standby Database Technology (7.0.48.15006) (pid 4810)
dbvisit started on DR: Mon Jan 2 12:10:28 2017 ()
=============================================================
>>> Log file(s) for prodDB from PR will be applied to DR
201701021210 - Log seq 145082 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145083 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145084 thread 1 applied to standby database prodDB.
===logs trimmed===
201701021210 - Log seq 145139 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145140 thread 1 applied to standby database prodDB.
201701021210 - Log seq 145141 thread 1 applied to standby database prodDB.
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 7
Config: diskspace full threshold = 90%
Processing /ifs/oracle/dbvarchive/...
Archive log dir: /ifs/oracle/dbvarchive/
Total number of archive files : 409
Number of archive logs deleted = 60
Current Disk percent full : 82%
=============================================================
dbvisit ended on DR: Mon Jan 2 12:38:15 2017
=============================================================

Primary and standby database is in sync now.

Thank you.

Stay tune. 🙂

Oracle 11g Logo

RMAN-06183: datafile or datafile copy xyz.dbf larger than MAXSETSIZE

My one of the client database backup have been failed since one week due to “RMAN-06183 datafile or datafile copy /vcboradata/oradata/datafile/system01.dbf (file number 1) larger than MAXSETSIZE”. Error itself is pretty explanatory. The size of the system01.dbf datafile had exceeded the RMAN parameter MAXSETSIZE, i.e. 15G

Error log:
RMAN>
Starting backup at 2016-11-03 10:41:39
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1008 device type=DISK
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 2016-11-03 10:41:41
channel ORA_DISK_1: finished piece 1 at 2016-11-03 10:41:43
piece handle=/cbsrmanbackup/rman/VCBCBS/backupset/2016_11_03/o1_mf_ncnnf_TAG20161103T104140_d1okrg13_.bkp tag=TAG20161103T104140 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02
Finished backup at 2016-11-03 10:41:43
Starting Control File and SPFILE Autobackup at 2016-11-03 10:41:43
piece handle=/cbsrmanbackup/rman/VCBCBS/autobackup/2016_11_03/o1_mf_s_926937704_d1okrjqq_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 2016-11-03 10:41:45
Starting backup at 2016-11-03 10:41:49
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 11/03/2016 10:41:49
RMAN-06183: datafile or datafile copy /vcboradata/oradata/datafile/system01.dbf (file number 1) larger than MAXSETSIZE
Recovery Manager complete.

RMAN parameters:

RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name VCBCBS are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 31 DAYS;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO 15 G;
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/cbsrmanbackup/rman/snapcf_vcbcbs.f';

Confirm current size of the system01.dbf, as below, used size is 15336.1 which is greater then 15G.

RMANTBS /vcboradata/oradata/datafile/rmantbs.dbf 1000 70.9 929.06
SYSAUX /vcboradata/oradata/datafile/sysaux01.dbf 1400 880.3 519.75
SYSTEM /vcboradata/oradata/datafile/system01.dbf 15400 15336.1 63.94
UNDOTBS1 /vcboradata/oradata/datafile/undotbs01.dbf 4725 97.3 4627.75
USERS /vcboradata/oradata/datafile/users01.dbf 50 15.9 34.13

Workaround:
Increase the size of MAXPIECESIZE RMAN parameter from 15G to 33G. (You can increase it as per your convenience)
More on MAXPIECESIZE:
This parameter is use to set limits on the size of backup pieces. You can also use the MAXSETSIZE parameter on the BACKUP and CONFIGURE commands to set a limit for the size of backup sets.

RMAN> CONFIGURE MAXSETSIZE TO 33G;
old RMAN configuration parameters:
CONFIGURE MAXSETSIZE TO 15 G;
new RMAN configuration parameters:
CONFIGURE MAXSETSIZE TO 33 G;
new RMAN configuration parameters are successfully stored

After above changes, backup successfully completed.

Have a nice time ahead.

Stay Tune. 🙂

ORA-12542: TNS:address already in use – Oracle 10g 64 bit

My one of the client facing ORA-12542, this error message popped up when he tried TNSPING OR to create DB-link.
Environment:
Microsoft Windows Server 2003 – 64 bit
Oracle Database 10g Enterprise Edition Release 10.2.0.4 – 64 bit

Error:

ORA-12542: TNS:address already in use

Solution:

On server side, add multiple TCP ports in listener.ora file, I have added additional TCP port i.e.1523 to the current LISTENER.

D:\oracle\product\10.2.0\db_1\network\admin\listener.ora
INBOUND_CONNECT_TIMEOUT_LISTENER = 0
SID_LIST_LISTENER =
   (SID_LIST =
      (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = D:\oracle\product\10.2.0\db_1)
      (PROGRAM = extproc)
   )
 )
LOG_DIRECTORY_LISTENER = G:\Listener_Trace
LISTENER =
   (DESCRIPTION_LIST =
     (DESCRIPTION =
       (ADDRESS = (PROTOCOL = TCP)(HOST = newpass)(PORT = 1521))
       (ADDRESS = (PROTOCOL = TCP)(HOST = newpass)(PORT = 1523))
     )
   )

And on client side, map net service names to recently added listener protocol address. i.e. 1523, as below:

NEWPASS_29 =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.6)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.6)(PORT = 1523))
   )
  (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = NEWPASS) 
  )
)

after made above changes to listener.ora and tnsnames.ora, TNSPING was successful and able to create DB-link.

Note:

You may face mentioned error in following cases:

  1. New installation
  2. listener.ora file has been edited manually since the last listener restart
  3. TCP port number is duplicated across the list of ADDRESS configurations in the listener.ora file.

Cheers!!

Stay Tune. 🙂

Oracle 11g Logo

ORA-08104: this index object xxxxx is being online built or rebuilt and ORA-00031: session marked for kill

My second attempt to rebuild below mentioned index failed with “ORA-08104-this index object 22624 is being online built or rebuilt”. I am getting this oracle error because my first attempt to rebuild same index failed due to abnormal session termination, it was incomplete rebuild.

Error log:

SQL> ALTER INDEX UPASSUSR.IDX_TRANSLOG_OBJECTNAME REBUILD online TABLESPACE UPASSTBS;
ALTER INDEX UPASSUSR.IDX_TRANSLOG_OBJECTNAME REBUILD online TABLESPACE UPASSTBS
*
ERROR at line 1:
ORA-08104: this index object 22624 is being online built or rebuilt

According to oracle support(Doc ID 375856.1) we can rid out of the issue by cleaning garbage with the help of dbms_repair.online_index_clean function. But no luck after successful execution of dbms_repair.online_index_clean function.

declare
lv_ret BOOLEAN;
begin
lv_ret := dbms_repair.online_index_clean(22624);
end;
/
PL/SQL procedure successfully completed.

Same clean-up logs popped up in ADRCI with same object no(i.e.22624) but no luck.

2016-09-27 13:21:06.373000 +05:30
online index (re)build cleanup: objn=22624 maxretry=2000 forever=0

Even, I tried dropping index and mark as unusable but unlucky again.

SQL> drop index UPASSUSR.IDX_TRANSLOG_OBJECTNAME;
drop index UPASSUSR.IDX_TRANSLOG_OBJECTNAME
*
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

So, I have decided to kill that session with the help of “ALTER SYSTEM KILL SESSION”, I got require details with the help of following SQL:

SELECT O.OBJECT_NAME, S.SID, S.SERIAL#, P.SPID, S.PROGRAM,S.USERNAME,S.MACHINE, S.PORT, 
S.LOGON_TIME, SQ.SQL_FULLTEXT 
FROM V$LOCKED_OBJECT L, DBA_OBJECTS O, V$SESSION S, V$PROCESS P, V$SQL SQ 
WHERE L.OBJECT_ID = O.OBJECT_ID 
AND L.SESSION_ID = S.SID 
AND S.PADDR = P.ADDR 
AND S.SQL_ADDRESS = SQ.ADDRESS
AND O.OBJECT_NAME = 'AOUP_TRANS_LOG';

After 60 seconds of waiting, SQL command ended up with “ORA-00031: session marked for kill” oracle error. 🙁

SQL> ALTER SYSTEM KILL SESSION '3423,39597';
ALTER SYSTEM KILL SESSION '3423,39597'
*
ERROR at line 1:
ORA-00031: session marked for kill

Finally, I got the solution by killing process at OS level:
kill -9 spid     (from above)

Thank god, Index rebuild was successful.

Object flag before killing OS process:

SQL> select obj#,flags from ind$ where obj#=22624;
OBJ#       FLAGS
---------- ----------
22624      2562

Object flag after killing OS process:

SQL> select obj#,flags from ind$ where obj#=22624;
OBJ#       FLAGS
---------- ----------
22624      2050

Have a great time ahead.

Stay Tune. 🙂

mongodb-logo

MongoDB service not starting due Address already in use for socket: 127.0.0.1:27017

Hello folks, Hope you are doing best. Here is one more technical article on MongoDB.
My one of the test MongoDB instance was not able to start due to below mentioned error, the error itself more self-explanatory, Address: 27017 is already used for socket.

I encourage all my fellow DBA’s to hit database logs first, here you will find everything about database problem you are facing. Once you understand the error completely, you’ll find less difficult to resolve it.

Error Logs:

[root@mongodb ~]# service mongod start
Starting mongod: [FAILED]
[root@mongodb ~]# service mongod restart
Stopping mongod: [ OK ]
Starting mongod: [FAILED]
2016-09-23T23:16:33.078-0700 I CONTROL [main] ***** SERVER RESTARTED *****
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] MongoDB starting : pid=2761 port=27017 dbpath=/var/lib/mongo 64-bit host=mongodb
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] db version v3.2.9
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] allocator: tcmalloc
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] modules: none
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] build environment:
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] distmod: rhel62
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] distarch: x86_64
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] target_arch: x86_64
2016-09-23T23:16:33.087-0700 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-09-23T23:16:33.109-0700 E NETWORK [initandlisten] listen(): bind() failed errno:98 Address already in use for socket: 127.0.0.1:27017
2016-09-23T23:16:33.109-0700 E NETWORK [initandlisten] addr already in use
2016-09-23T23:16:33.109-0700 E STORAGE [initandlisten] Failed to set up sockets during startup.
2016-09-23T23:16:33.109-0700 I CONTROL [initandlisten] dbexit: rc: 48

Attempt to stop MongoDB instance was successful, as below. But I didn’t find any sort of logs in a database log file. (i.e. /var/log/mongodb/mongod.log)

[root@mongodb ~]# service mongod stop
Stopping mongod: [ OK ]

Thanks to netstat linux command as below:

[root@mongodb ~]# netstat -tlnp | grep 27017
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 2007/mongod

I have Killed mongod service with the “kill” linux command:

[root@mongodb ~]# kill -9 2007

After mongod service killed, my instance started successfully:

[root@mongodb ~]# service mongod start
Starting mongod: [ OK ]

Database logs also positive:

2016-09-23T23:30:17.430-0700 I CONTROL [main] ***** SERVER RESTARTED *****
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] MongoDB starting : pid=2938 port=27017 dbpath=/var/lib/mongo 64-bit host=mongodb
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] db version v3.2.9
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] allocator: tcmalloc
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] modules: none
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] build environment:
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] distmod: rhel62
2016-09-23T23:30:17.439-0700 I CONTROL [initandlisten] distarch: x86_64
2016-09-23T23:30:17.440-0700 I CONTROL [initandlisten] target_arch: x86_64
2016-09-23T23:30:17.440-0700 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-09-23T23:30:17.460-0700 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-09-23T23:30:17.460-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-09-23T23:30:17.907-0700 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2016-09-23T23:30:17.908-0700 I NETWORK [initandlisten] waiting for connections on port 27017
2016-09-23T23:30:17.909-0700 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker

I can open new connection too:

2016-09-23T23:30:52.860-0700 I NETWORK [initandlisten] connection accepted from 127.0.0.1:58051 #1 (1 connection now open)

Your comments are highly appreciable in advanced, if any.
Stay Tune. 🙂

Oracle 11g Logo

ORA-03113: end-of-file on communication channel while startup

My one of the client reported me ORA-03113 oracle error message. He is facing problem while starting up database. Usually this error occurs when connection between Client and Server process was broken OR It can be any big problem.

SQL> startup
ORACLE instance started.
Total System Global Area 8584982528 bytes
Fixed Size 2262088 bytes
Variable Size 4462742456 bytes
Database Buffers 4093640704 bytes
Redo Buffers 26337280 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 2588
Session ID: 1705 Serial number: 5

On further diagnosis of ADRCI logs, I found disk space size assigned to Flash recovery area falling short. ADRCI logs are as follows:

2016-09-12 12:46:00.960000 +05:30
Successful mount of redo thread 1, with mount id 3346245764
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
ALTER DATABASE OPEN
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=20, OS id=564
2016-09-12 12:46:02.333000 +05:30
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\trace\mnsbdb_ora_2588.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 21474836480 bytes is 100.00% used, and has 0 remaining bytes available.
************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
 then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
 BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
 reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
 system command was used to delete files, then use RMAN CROSSCHECK and
 DELETE EXPIRED commands.
************************************************************************
ARCH: Error 19809 Creating archive log file to 'D:\APP\ADMINISTRATOR\FAST_RECOVE
RY_AREA\MNSBDB\ARCHIVELOG\2016_09_12\O1_MF_1_454_%U_.ARC'
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\trace\mnsbdb_ora_25
88.trc:
ORA-16038: log 8 sequence# 454 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 8 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\MNSBDB\REDO08.LO
G'
USER (ospid: 2588): terminating the instance due to error 16038
ARC0: STARTING ARCH PROCESSES
2016-09-12 12:46:03.347000 +05:30
Logins disabled; aborting ARCH process startup (1092)
ARC0: Archival disabled due to shutdown: 1092
Shutting down archive processes
Archiving is disabled
System state dump requested by (instance=1, osid=2588), summary=[abnormal instan
ce termination].
System State dumped to trace file D:\APP\ADMINISTRATOR\diag\rdbms\mnsbdb\mnsbdb\
trace\mnsbdb_diag_4064.trc
Dumping diagnostic data in directory=[cdmp_20160912124603], requested by (instan
ce=1, osid=2588), summary=[abnormal instance termination].
2016-09-12 12:46:05.266000 +05:30
Instance terminated by USER, pid = 2588

So, I finally assigned more size to Flash recovery area using following command, and database opened successfully.

 SQL> startup mount
 ORACLE instance started.
 Total System Global Area 8584982528 bytes
 Fixed Size 2262088 bytes
 Variable Size 4462742456 bytes
 Database Buffers 4093640704 bytes
 Redo Buffers 26337280 bytes
 Database mounted.
SQL> alter system set db_recovery_file_dest_size=100G scope=both;
SQL> alter database open;

I strongly recommend you to monitor your every database move from ADRCI, this will easy for your to understand problem area and act accordingly.

Your comments highly appreciated.

Stay Tune. 🙂

Oracle 11g Logo

How to drop database manually, without using Database Configuration Assistant(DBCA)

Before proceed for entire database deletion, kindly consider full database backup for your future reference, if required.

Connect to target database with sys user, and issue following query to check database mode:

[oracle@testserver ~]$ sqlplus / as sysdba
SQL> select open_mode,name from v$database;
OPEN_MODE            NAME
-------------------- ---------
READ WRITE           BLADE2

Normally shutdown your database:

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

startup your database in mount mode with exclusive restrict, as follows:

SQL> startup mount exclusive restrict;
ORACLE instance started.
Total System Global Area 1.0088E+10 bytes
Fixed Size 2215984 bytes
Variable Size 5704257488 bytes
Database Buffers 4362076160 bytes
Redo Buffers 19640320 bytes
Database mounted.

Database is in mount mode:

SQL> select open_mode,name from v$database;
OPEN_MODE            NAME
-------------------- ---------
MOUNTED              BLADE2

Issue following SQL command in order to drop database, this will drop entire database with datafiles, control files and log files,etc.

SQL> drop database;
Database dropped.
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>

Database dropped successfully.

Thank you. Stay Tune. 🙂

Oracle 11g Logo

RMAN-06900 RMAN-06901 ORA-19921

Received RMAN-06900,RMAN-06901 and ORA-19921 while logged in to RMAN prompt to verify daily backups. (OS:Redhat linux 6.5)

WARNING message:

[oracle@primary logs]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Fri Aug 26 10:34:39 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: BLADE1 (DBID=3381279798)
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded
RMAN>

On further diagnosing and suggested by OERR utility, I have realized there is one old active session from Aug 23, as mentioned below.

[oracle@primary ~]$ oerr ORA 19921
19921, 00000, "maximum number of %s rows exceeded"
// *Cause: The maximum number of rows in the V$RMAN_STATUS or V$RMAN_OUTPUT
// table has been exceeded.
// *Action: Close some of existing and unused RMAN connections and sessions.

So lets find out old session with ps -ef:

[oracle@primary logs]$ ps -ef|grep rman
oracle 543 32693 0 10:50 pts/4 00:00:00 grep rman
oracle 19783 19782 47 Aug23 ? 1-07:13:42 rman target /

There is one old active session above, OS ID: 19783, since 23 aug.

[oracle@primary ~]$ date
Fri Aug 26 10:57:13 IST 2016
[oracle@primary logs]$ kill -9 19783

After successfully killed old session, we are no more facing mentioned list of warnings, as below:

[oracle@primary logs]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Fri Aug 26 10:50:54 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: BLADE1 (DBID=3381279798)
RMAN>

Thanks,

Stay Tune. 🙂

Oracle 11g Logo

rename/relocate SYSTEM, SYSAUX or User Tablespace datafiles

This is traditional approach to move/rename SYSTEM, SYSAUX or users tablespace. In this method, we need database downtime as we can’t take SYSTEM or SYSAUX tablespace offline.

I recommend full cold backup before performing mentioned activity.

Consider scenario where I want to move SYSTEM and SYSAUX tablespace to new location.

Step I:

After considering full cold backup of your database, shutdown database with normal or immediate mode only.

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

Step II:

After shut down database normally, I am going to copy datafiles from original location to its new location.

Following query will provide tablespace name and its datafiles details.

SQL> select file_name,tablespace_name from dba_data_files;

Copying datafiles:

[oracle@PR ~]$ cp /u01/app/oracle/oradata/RTS/system01.dbf /u01/app/oracle/oradata/RTS_NEW/system01.dbf
[oracle@PR ~]$ cp /u01/app/oracle/oradata/RTS/sysaux01.dbf /u01/app/oracle/oradata/RTS_NEW/sysaux01.dbf

After copy, verify size of datafiles on both locations to check whether copy successful or not.

Step III:

Start database in mount mode:

[oracle@PR oradata]$ sqlplus / as sysdba 
SQL> startup mount;
ORACLE instance started.
Total System Global Area 755769344 bytes
Fixed Size 2217184 bytes
Variable Size 478153504 bytes
Database Buffers 272629760 bytes
Redo Buffers 2768896 bytes
Database mounted.

Step IV:

After successful mount, rename the datafiles to its locations as below:

SQL> alter database rename file '/u01/app/oracle/oradata/RTS/system01.dbf' to '/u01/app/oracle/oradata/RTS_NEW/system01.dbf';
Database altered.
SQL> alter database rename file '/u01/app/oracle/oradata/RTS/sysaux01.dbf' to '/u01/app/oracle/oradata/RTS_NEW/sysaux01.dbf';
Database altered.

Step V:

Open database:

SQL> alter database open;
Database altered.

SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
RTS READ WRITE

Verify that system and sysaux datafiles are moved/renamed successful to new location.

SQL> select file_name,tablespace_name from dba_data_files;
/u01/app/oracle/oradata/RTS_NEW/sysaux01.dbf SYSAUX
/u01/app/oracle/oradata/RTS_NEW/system01.dbf SYSTEM

SYSTEM and SYSAUX datafiles relocated successfully to new location.

This can be done with the help of RMAN, but only non-system tablespaces can be relocated. Click me to know more about it.

OR

You can relocate datafiles without getting offline in oracle 12c.

Thanks, Stay Tune. 🙂

Oracle 11g Logo

ORA-00600: internal error code, arguments:[4194] ORA-00603: ORACLE server session terminated by fatal error. PMON terminating the instance due to error 472

Oracle instance terminated immediate after instance startup due to following mentioned oracle error:

Cause:
A mismatch has been detected between Redo records and rollback (Undo) records.
We are validating the Undo record number relating to the change being applied against the maximum undo record number recorded in the undo block. This error is reported when the validation fails. This error may indicate a rollback segment corruption.
Note: This may require a recovery from a database backup depending on the situation.

Adrci logs are as follows:

Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_smon_11168.trc (incident=1014790):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014790/db1_smon_11168_i1014790.trc
2016-06-30 11:13:35.355000 +05:30
Starting background process QMNC
QMNC started with pid=25, OS id=11249
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Completed: ALTER DATABASE OPEN
2016-06-30 11:13:36.558000 +05:30
db_recovery_file_dest_size of 163840 MB is 1.12% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery stopped at EOT rba 13618.106.16
Block recovery completed at rba 13618.106.16, scn 0.3980712368
Doing block recovery for file 3 block 160
Resuming block recovery (PMON) for file 3 block 160
Block recovery from logseq 13618, block 56 to scn 3980712329
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.59.16, scn 0.3980712330
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_smon_11168.trc:
ORA-01595: error freeing extent (10) of rollback segment (3))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Starting background process SMCO
SMCO started with pid=29, OS id=11268
Trace dumping is performing id=[cdmp_20160630111337]
2016-06-30 11:13:37.819000 +05:30
Starting background process CJQ0
CJQ0 started with pid=28, OS id=11276
2016-06-30 11:13:41.579000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc (incident=1014942):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014942/db1_j001_11285_i1014942.trc
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j004_11291.trc:
2016-06-30 11:13:43.111000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j000_11283.trc (incident=1014934):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014934/db1_j000_11283_i1014934.trc
Trace dumping is performing id=[cdmp_20160630111343]
2016-06-30 11:13:45.937000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j000_11283.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc (incident=1014943):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014943/db1_j001_11285_i1014943.trc
2016-06-30 11:13:46.903000 +05:30
Trace dumping is performing id=[cdmp_20160630111346]
2016-06-30 11:13:48.279000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_j001_11285.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Trace dumping is performing id=[cdmp_20160630111349]
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1019488/db1_j001_11285_i1019488.trc
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1019488/db1_j001_11285_i1019488.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Trace dumping is performing id=[cdmp_20160630111351]
2016-06-30 11:13:53.489000 +05:30
Doing block recovery for file 3 block 26014
Resuming block recovery (PMON) for file 3 block 26014
Block recovery from logseq 13618, block 56 to scn 3980712372
Recovery of Online Redo Log: Thread 1 Group 1 Seq 13618 Reading mem 0
 Mem# 0: /u02/oradata/db1/redo_1_1.log
 Mem# 1: /u02/oradata/db1/redo_1_2.log
 Mem# 2: /u02/oradata/db1/redo_1_3.log
Block recovery completed at rba 13618.106.16, scn 0.3980712374
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_pmon_11144.trc (incident=1014702):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/db1/db1/incident/incdir_1014702/db1_pmon_11144_i1014702.trc
2016-06-30 11:13:55.766000 +05:30
Errors in file /u01/app/oracle/diag/rdbms/db1/db1/trace/db1_pmon_11144.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 11144): terminating the instance due to error 472
Instance terminated by PMON, pid = 11144

Workaround:
Startup database in mount mode:

SQL> startup mount;

Check current undo tablespace:

SQL> show parameter undo_tablespace;

Alter undo management to ‘MANUAL’:

SQL> alter system set undo_management='MANUAL' scope=spfile;

Bounce database to open mode, meanwhile please verify adrci logs, if there is problem.

SQL> shut immediate;
 SQL> startup;

Create new undo tablespace if everything is fine in adrci logs:

SQL> create undo tablespace newundotbs datafile '/u02/oradata/db1/newundotbs01.dbf' size 5G autoextend on next 300M maxsize 31G;

Change default undo tablespace to new one:

SQL> alter system set undo_tablespace='NEWUNDOTBS' scope=spfile;

Alter undo management to ‘AUTO’:

SQL> alter system set undo_management='AUTO' scope=spfile;

Bounce the database:

SQL> shut immediate;
SQL> startup;

Verify current undo tablespace:

SQL> show parameter undo_tablespace

done.

Thank you.
Stay Tune. 🙂

Oracle 11g Logo

Oracle VM Manager 3.4.1 Installation on Oracle Linux 6 Update 5 (OL 6.5)

Please follow the step by step approach to install Oracle VM manager 3.4.1 on Oracle Linux 6.5

Step-I

Update host file /etc/hosts file by fully qualified name, as mentioned below. In my case “OVMmanager” is host name.

[root@OVMmanager ~]# vi /etc/hosts
192.168.239.135     OVMmanager.localdomain     OVMmanager

Step-II

Perform oracle VM manager prerequisites with the help of “createOracle.sh” script, provided by oracle itself in OVM manager installer.

[root@OVMmanager OVMManagerInstaller3.4.1_b1369]# ll
total 157510
drwxr-xr-x. 7 root root 8192 Apr 5 22:36 components
-r-xr-x---. 1 root root 11556 Apr 5 22:35 createOracle.sh
-rw-r--r--. 1 root root 6960 Apr 5 22:36 EULA
-rw-r--r--. 1 root root 6960 Apr 5 22:36 LICENSE
-rw-r--r--. 1 root root 230 Apr 5 22:35 oracle-validated.params
-r-xr-x---. 1 root root 156958046 Apr 5 22:36 ovmm-installer.bsx
-rw-r--r--. 1 root root 4291939 Apr 5 22:32 OvmSDK_3.4.1.1369.zip
-r-xr-x---. 1 root root 1919 Apr 5 22:36 runInstaller.sh
-rw-r--r--. 1 root root 372 Apr 5 22:36 sample.yml
-r--r--r--. 1 root root 2031 Apr 5 22:36 TRANS.TBL
[root@OVMmanager OVMManagerInstaller3.4.1_b1369]# ./createOracle.sh
Adding group 'oinstall' with gid '54321' ...
Adding group 'dba'
Adding user 'oracle' with user id '54321', initial login group 'dba', supplementary group 'oinstall' and home directory '/home/oracle' ...
Changing ownership of '/home/oracle' to oracle:dba
Creating user 'oracle' succeeded ...
For security reasons, no default password was set for user 'oracle'. If you wish to login as the 'oracle' user, you will need to set a password for this account.
Verifying user 'oracle' OS prerequisites for Oracle VM Manager ...
oracle soft nofile 8192
oracle hard nofile 65536
oracle soft nproc 2048
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle soft core unlimited
oracle hard core unlimited
Setting user 'oracle' OS limits for Oracle VM Manager ...
Altered file /etc/security/limits.conf
Original file backed up at /etc/security/limits.conf.orabackup
Verifying & setting of user limits succeeded ...
Creating mountpoint '/u01' ...
Modifying iptables for OVM
Adding rules to enable access to:
 7002 : Oracle VM Manager https
 123 : NTP
 10000 : Oracle VM Manager CLI Tool
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
iptables: Applying firewall rules: [ OK ]
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
iptables: Applying firewall rules: [ OK ]
Rules added.

Step-III
Install Oracle VM manager:

[root@OVMmanager OVMManagerInstaller3.4.1_b1369]# ./runInstaller.sh
Oracle VM Manager Release 3.4.1 Installer
Oracle VM Manager Installer log file:
/var/log/ovmm/ovm-manager-3-install-2016-06-20-195709.log
Please select an installation type:
 1: Install
 2: Upgrade
 3: Uninstall
 4: Help

Select Number (1-4): 1

Verifying installation prerequisites ...
*** WARNING: Recommended memory for the Oracle VM Manager server installation using Local MySql DB is 7680 MB RAM

Starting production with local database installation ...

One password is used for all users created and used during the installation.
Enter a password for all logins used during the installation:
Enter a password for all logins used during the installation (confirm):

Please enter your fully qualified domain name, e.g. ovs123.us.oracle.com, (or IP address) of your management server for SSL certification generation 192.168.239.135 [OVMmanager.localdomain]: OVMmanager.localdomain

Verifying configuration ...

Start installing Oracle VM Manager:
 1: Continue
 2: Abort
Select Number (1-2): 1

Step 1 of 7 : Database Software ...
Installing Database Software...
Retrieving MySQL Database 5.6 ...
Unzipping MySQL RPM File ...
Installing MySQL 5.6 RPM package ...
Configuring MySQL Database 5.6 ...
Installing MySQL backup RPM package ...

Step 2 of 7 : Java ...
Installing Java ...

Step 3 of 7 : WebLogic and ADF ...
Retrieving Oracle WebLogic Server 12c and ADF ...
Installing Oracle WebLogic Server 12c and ADF ...
Applying patches to Weblogic ...
Applying patch to ADF ...

Step 4 of 7 : Oracle VM ...
Installing Oracle VM Manager Core ...
Retrieving Oracle VM Manager Application ...
Extracting Oracle VM Manager Application ...

Retrieving Oracle VM Manager Upgrade tool ...
Extracting Oracle VM Manager Upgrade tool ...
Installing Oracle VM Manager Upgrade tool ...

Retrieving Oracle VM Manager CLI tool ...
Extracting Oracle VM Manager CLI tool...
Installing Oracle VM Manager CLI tool ...
Installing Oracle VM Manager WLST Scripts ...

Step 5 of 7 : Domain creation ...
Creating domain ...

Step 6 of 7 : Oracle VM Tools ...
Retrieving Oracle VM Manager Shell & API ...
Extracting Oracle VM Manager Shell & API ...
Installing Oracle VM Manager Shell & API ...

Retrieving Oracle VM Manager Wsh tool ...
Extracting Oracle VM Manager Wsh tool ...
Installing Oracle VM Manager Wsh tool ...

Retrieving Oracle VM Manager Tools ...
Extracting Oracle VM Manager Tools ...
Installing Oracle VM Manager Tools ...

Retrieving ovmcore-console ...
Installing ovmcore-console RPM package ...
Copying Oracle VM Manager shell to '/usr/bin/ovm_shell.sh' ...
Installing ovm_admin.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Installing ovm_upgrade.sh in '/u01/app/oracle/ovm-manager-3/bin' ...

Step 7 of 7 : Start OVM Manager ...
Enabling Oracle VM Manager service ...
Shutting down Oracle VM Manager instance ...
Starting Oracle VM Manager instance ...

Please wait while WebLogic configures the applications...
Trying to connect to core via ovmwsh (attempt 1 of 20) ...
Trying to connect to core via ovmwsh (attempt 2 of 20) ...
Trying to connect to core via ovmwsh (attempt 3 of 20) ...
Trying to connect to core via ovm_shell (attempt 1 of 5)...
Oracle VM Manager installed.

Installation Summary
--------------------
Database configuration:
 Database type : MySQL
 Database host name : localhost
 Database name : ovs
 Database listener port : 49500
 Database user : ovs

Weblogic Server configuration:
 Administration username : weblogic

Oracle VM Manager configuration:
 Username : admin
 Core management port : 54321
 UUID : 0004fb00000100003ddbd8af4ce345c8

Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Database, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:
 https://OVMmanager.localdomain:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.

For more information about Oracle Virtualization, please visit: http://www.oracle.com/virtualization/

Oracle VM Manager installation complete.
Please remove configuration file /tmp/ovm_config71Qdn2.
[root@OVMmanager OVMManagerInstaller3.4.1_b1369]#

Oracle VM manager installation succeeded.

You will find Installation Summary right after installation comprises of MySQL database details, Weblogic Server configuration details and Oracle VM Manager configuration details.

Access Oracle VM Manager UI (console) with https://OVMmanager.localdomain:7002/ovm/console

Note: you need to confirm security settings before access Oracle VM manager.

Oracle VM manager login screen:

Note: Log in with the user ‘admin’, and the password you set during the installation.

Oracle VM manager - login

Oracle VM console:

Oracle VM manager - console

 

Thank you.
Stay Tune. 🙂