About Me

My photo
Bangalore, India
I am an Oracle Certified Professional working in SAP Labs. Everything written on my blog has been tested on my local environment, Please test before implementing or running in production. You can contact me at amit.rath0708@gmail.com.
Showing posts with label Data Guard. Show all posts
Showing posts with label Data Guard. Show all posts

Wednesday, January 11, 2017

MRP0: Background Media Recovery terminated with error 1111

Today while troubleshooting a DR database sync scenario issue, I got below error in alert log file :-

MRP0: Background Media Recovery terminated with error 1111
Errors in file /opt/oracle/diag/rdbms/amit/amit1/trace/amit1_pr00_1890.trc:
ORA-01111: name for data file 34 is unknown - rename to correct file
ORA-01110: data file 34: '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034'
ORA-01157: cannot identify/lock data file 34 - see DBWR trace file
ORA-01111: name for data file 34 is unknown - rename to correct file
ORA-01110: data file 34: '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034'

Cause :-

Above issue can occur in DR database in below scenarios :-

1. Space issue in DR database and a datafile is added in primary database.
2. Standby file management is set to manual
3. DB_create_file_dest or DB_file_name_convert parameter is not set in Standby database

Troubleshoot :-

When I crosscheck the primary database and standby database sync details , I saw Redo Apply was stopped.

I crosschecked the alert log files for standby database and got above mentioned error in alert logfile.

Checked the datafile details in DR database :-

File_id       name
34             /opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034

Checked the Primary alert log file and got to know that a datafile addition activity happened in primary database.

Crosschecked all the above mentioned parameters and got to know that space exhausted in BCP database due to that Oracle created an Unknown file in ORACLE_HOME location due to which Redo Apply stopped.

Solution :-

Checked the added file detail in Production database :-

File_id       name
34            +DATA/amit/datafile/data.321.931222251

Login to Standby database server :-

1. As space was exhausted on Diskgroup level, added space by logging in to ASM instance
2. Login to DR database instance :-

a. Change the standby_file_management parameter to Manual

SQL> show parameter standby

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      ?/dbs/arch
standby_file_management              string      AUTO
SQL> alter system set standby_file_management=MANUAL sid='*';

SQL> show parameter standby

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      ?/dbs/arch
standby_file_management              string      MANUAL
SQL>
SQL> select name from v$datafile where file#=34;

NAME
--------------------------------------------------------------------------------
/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034

SQL> alter database create datafile '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034' as '+DATA' size 10g;

Database altered.

After creation of the datafile in DR database , check the file detail in DR database :-

File_id       name
34             +DATA/amit_dr/datafile/data.278.293456881

Check the redo apply status, if not started , start it manually using dgmgrl.

I hope this article helped you.

Thanks
Amit Rath

Friday, July 24, 2015

Flashback Technology : Flashback to a restore point having standby database also enabled

Yesterday we were in a scenario where we have to do some changes in our production database which if needed can be rollbacked.

We planned to go with a restore point option so that if change not needed by Development team, we can move back to before change time.

This was a big database and it has standby also enabled with it. So if we do a flashback on primary then incarnation of primary will differ from standby and recovery will be stopped.

Below are the steps which we used to handle both primary and standby in case of a Flashback database in Primary :-

1. Backup Primary database

2. Note the Current SCN of primary :-

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
   15883324

3. Create a restore point 

SQL> CREATE RESTORE POINT before_upgrade GUARANTEE FLASHBACK DATABASE;

Restore point created.

4. Do your changes in database 

Now you want to do the flashback in your primary Database as the changes which you did, development team does not require those. PFB steps :-

SQL> startup force mount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2260088 bytes
Variable Size            1124074376 bytes
Database Buffers         3137339392 bytes
Redo Buffers               12107776 bytes
Database mounted.
SQL> flashback database to restore point before_upgrade;

Flashback complete.

SQL> alter database open RESETLOGS;

Database altered.

Now when you check your standby database , MRP is stopped and it's showing that incarnation is different from primary database.

MRP0: Incarnation has changed! Retry recovery...
Errors in file /opt/oracle/diag/rdbms/amit/amit1/trace/amit_pr00_17847.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!

Move your standby database to that SCN which was before doing a flashback.

Standby Database :-

SQL> flashback database to scn 15883324;

Flashback complete.

Now start the MRP and check the alert log, it will show MRP started successfully

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (dr3fia)
Fri Jul 24 06:10:35 2015
MRP0 started with pid=49, OS id=14805
MRP0: Background Managed Standby Recovery process started (dr3fia)
 started logmerger process
Fri Jul 24 06:10:40 2015
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 12 slaves
Media Recovery start incarnation depth : 1, target inc# : 6, irscn : 15883360
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log +Amit_DG/amit/archivelog/2015_07_24/thread_1_seq_7.704.885880789
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

I hope this article helped you.

Thanks
Amit Rath

ORA-16700: the standby database has diverged from the primary database

Yesterday I faced below error in my standby database :-

Problem :-

DGMGRL> show database amit_dr

Database - amit_dr

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   (unknown)
  Apply Lag:       0 seconds (computed 1 second ago)
  Apply Rate:      148.00 KByte/s
  Real Time Query: OFF
  Instance(s):
   amit_dr

  Database Error(s):
    ORA-16700: the standby database has diverged from the primary database
    ORA-16766: Redo Apply is stopped

Database Status:
ERROR

Cause :- 

This happened because when due to an incomplete recovery in primary database, it has to be opened using resetlogs option and incarnation of primary and standby becomes different due to resetlogs option.

we will get below error in our standby database alert logfile

MRP0: Incarnation has changed! Retry recovery...
Errors in file /opt/oracle/diag/rdbms/amit/amit1/trace/amit_pr00_17847.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Fri Jul 24 05:58:48 2015
Archived Log entry 113 added for thread 1 sequence 1 ID 0xf0d327b6 dest 1:
Recovered data files to a consistent state at change 15883769
Fri Jul 24 05:58:48 2015
 started logmerger process
Fri Jul 24 05:58:48 2015
Managed Standby Recovery starting Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 15883769) is orphaned on incarnation#=5
MRP0: Detected orphaned datafiles!

Solution :-

We do not have to rebuild our standby, just move standby database using flashback to a previous SCN value when the incarnation of primary and standby was same. Start the MRP manually or bounce the configuration in DG broker.

SQL> flashback database to scn 15883324;

Flashback complete.

Details from alert log on standby :-

flashback database to scn 15883324
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
Fri Jul 24 06:06:06 2015
Setting recovery target incarnation to 5
 started logmerger process
Parallel Media Recovery started with 12 slaves
Fri Jul 24 06:06:06 2015
Media Recovery Log +AMIT_DG/amit/archivelog/2015_07_24/thread_1_seq_7.704.885880789
Incomplete Recovery applied until change 15883325 time 07/24/2015 05:52:43
Flashback Media Recovery Complete
Setting recovery target incarnation to 6
Completed: flashback database to scn 15883324

Attempt to start background Managed Standby Recovery process (dr3fia)
Fri Jul 24 06:10:35 2015
MRP0 started with pid=49, OS id=14805
MRP0: Background Managed Standby Recovery process started (dr3fia)
 started logmerger process
Fri Jul 24 06:10:40 2015
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 12 slaves
Media Recovery start incarnation depth : 1, target inc# : 6, irscn : 15883360
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log +AMIT_DG/amit/archivelog/2015_07_24/thread_1_seq_7.704.885880789
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

DGMGRL> show configuration

Configuration - amit_dr

  Protection Mode: MaxPerformance
  Databases:
    amit  - Primary database
    amit_dr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

I hope this article helped you.

Thanks
Amit Rath

Wednesday, December 17, 2014

How to Create a STANDBY database using RMAN Duplicate command in Oracle

We can also use RMAN duplicate command to create a STANDBY database from Production database. We can either use Backups of Production Database or we can create from Active Database also.

PFB steps to create a standby Database using Active Database Duplication Method :-

Primary Database is TEST
Standby Database is STBY

Changes to be done on Primary side :-

1. Database has to be archive log mode

SQL> SELECT log_mode FROM v$database;

LOG_MODE
------------
ARCHIVELOG

2. Change primary database to Force Logging mode :-

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.

3. Initialization parameters in Primary database :-

SQL>  show parameter db_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_name                              string      TEST
SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      TEST

4. Set the DG_CONFIG setting of LOG_ARCHIVE_CONFIG parameter. for this DB_UNIQUE_NAME of  Standby database has to be different from primary.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(TEST,STBY)';

System altered.

5. Set suitable remote archivelog destinations.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=STBY NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=STBY';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

System altered.

-- NOTE :- Service and DB_UNIQUE_NAME has to be of Standby database

6. Set parameters related to LOG_ARCHIVE and password file parameters :-

SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=8;

System altered.

SQL> ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;

System altered.

7. It is recommended to make sure that Primary can switch roles to become a Standby. For that we need to set some -convert parameters if there are path differences between the servers :-

SQL> ALTER SYSTEM SET FAL_SERVER=STBY;

System altered.

SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT=' ',' ' SCOPE=SPFILE;   -- Primay server destination and Standby server destination

System altered.

SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT=' ',' '  SCOPE=SPFILE;  -- Primay server destination and Standby server destination

System altered.

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

8. As some parameters cannot be changed while database is open , we need to restart database to before they take effect :-

SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
SQL> startup
ORACLE instance started.

Total System Global Area  960495616 bytes
Fixed Size                  1222816 bytes
Variable Size             247465824 bytes
Database Buffers          700448768 bytes
Redo Buffers               11358208 bytes
Database mounted.
Database opened.
SQL>

9. In both servers entries of Primary as well as Standby are needed in tnsnames.ora file and we are able to do  tnsping.

###################ORACLE_DATA_GAURD############################

TEST =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST)
    )
  )

STBY =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = STBY_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = STBY)
    )
  )

TEST_DGMGRL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 9001))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST_DGMGRL)
    )
  )
STBY_DGMGRL =
(DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = STBY_server)(PORT = 9001))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = STBY_DGMGRL)
    )
  )

###################ORACLE_DATA_GAURD############################

10. Add static entries to listener.ora file

PRIMARY

        (SID_DESC =
      (GLOBAL_DBNAME = test.db.com)
      (SID_NAME = TEST)
      (ORACLE_HOME = /opt/oracle/product/database/11.2.0.4)
        )
        (SID_DESC =
        (GLOBAL_DBNAME=TEST_DGMGRL)
        (ORACLE_HOME=/opt/oracle/product/database/11.2.0.4)
        (SID_NAME=test)
        )

STANDBY

      (SID_DESC =
      (GLOBAL_DBNAME = STBY.db.com)
      (SID_NAME = STBY)
      (ORACLE_HOME = /opt/oracle/product/database/11.2.0.4)
    )
      (SID_DESC =
      (GLOBAL_DBNAME = STBY_DGMGRL)
      (SID_NAME = STBY)
      (ORACLE_HOME = /opt/oracle/product/database/11.2.0.4)
    )

11. Reload listener after adding these entries

Lsnrctl reload

12. Add Standby redo logs to primary database :-

ALTER DATABASE ADD STANDBY LOGFILE ('/STBY_redo01.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/STBY_redo02.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/STBY_redo03.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/STBY_redo04.log') SIZE 50M;

13. Copy the passw0rd file from Primary and copy to Standby

14. Copy the init.ora file from primary and change below parameters :-

CREATE PFILE='/initTEST.ora' from spfile;

scp orapwTEST user@STBY_server:$ORACLE_HOME/dbs/orapwSTBY
scp initTEST.ora user@STBY_server:$ORACLE_HOME/dbs/initSTBY.ora

Steps to be done in STANDBY Side :-

15. Start the STANDBY database with above pfile

export ORACLE_SID=STBY
sql> startup nomount pfile='$ORACLE_HOME/dbs/initSTBY.ora'
ORACLE instance started.

Total System Global Area  960495616 bytes
Fixed Size                  1222816 bytes
Variable Size             264243040 bytes
Database Buffers          683671552 bytes
Redo Buffers               11358208 bytes
               
16. Connect to RMAN and use RMAN duplicate command to create STANDBY

rman target sys/****@TEST_DGMGRL auxiliary sys/****@STBY_DGMGRL
DUPLICATE TARGET DATABASE
  FOR STANDBY
  FROM ACTIVE DATABASE
  DORECOVER
  SPFILE
    SET db_unique_name='STBY' COMMENT 'Is STANDBY'
    SET LOG_ARCHIVE_DEST_2='SERVICE=TEST ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=TEST'
                set control_files='/ora_backup/TEST/duplicate_test/STBY/control01.ctl','/ora_backup/TEST/duplicate_test/STBY/control02.ctl'
                set db_create_file_dest='/ora_backup/TEST/duplicate_test/STBY/DATA/'
                set db_create_online_log_dest_1='/ora_backup/TEST/duplicate_test/STBY/LOG/'
                set diagnostic_dest='/ora_backup/TEST/duplicate_test/STBY/diag'
                set audit_file_dest='/ora_backup/TEST/duplicate_test/STBY/adump'
                set db_recovery_file_dest='/ora_backup/TEST/duplicate_test/STBY/FRA'
                set db_file_name_convert='/ora_backup/TEST1/restore_test/TEST/DATA/','/ora_backup/TEST/duplicate_test/STBY/DATA/'
                set log_file_name_convert='/ora_backup/TEST1/restore_test/TEST/','/ora_backup/TEST/duplicate_test/STBY/'
                set log_archive_dest_1='location=/ora_backup/TEST1/duplicate_test/STBY/FRA/'
                set FAL_SEVER=TEST
   ;

Above command will create the STANDBY database

17. Create the broker configuration details ( for both Primary and Standby):-

SQL>  alter system set dg_broker_start=FALSE;

System altered.

SQL> alter system set dg_broker_config_file1='/opt/oracle/product/database/11.2.0.4/dbs/dr1STBY.dat';

System altered.

SQL> alter system set dg_broker_start=TRUE;

System altered.

Create the configuation

Create configuration DR as primary database is TEST connect identifier is TEST;
Add database STBY as connect identifier is STBY maintained as physical;

show configuration
enable configuration

EDIT DATABASE STBY SET STATE = 'APPLY-ON' [WITH APPLY INSTANCE = ];  -- For RAC database

Check if every thing is fine by checking Standby Database details 

show database stby

DGMGRL> show database "STANDBY"

Database - STANDBY

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 0 seconds ago)
  Apply Lag:       0 seconds (computed 0 seconds ago)
  Apply Rate:      75.00 KByte/s
  Real Time Query: OFF
  Instance(s):
    STANDBY

Database Status:
SUCCESS

Standby Creation using RMAN duplicate command completed.

Standby creation in RAC using RMAN duplicate.

I hope this article helped you

Regards,
Amit Rath

Tuesday, December 16, 2014

MRP0: Background Media Recovery terminated with error 1274

Yesterday one of our standby database recovery stopped with below error :-

Errors in file /ora_backup/TEST/duplicate_test/AMIT/diag/diag/rdbms/standby/STANDBY/trace/STANDBY_pr00_12842.trc:
ORA-01119: error in creating database file '/ora_backup/TEST1/restore_test/AMIT/DATA/AMIT_03.dbf'
Additional information: 1
File #13 added to control file as 'UNNAMED00013'.
Originally created as:
'/ora_backup/TEST1/restore_test/AMIT/DATA/AMIT_03.dbf'
Recovery was unable to create the file as:
'/ora_backup/TEST1/restore_test/AMIT/DATA/AMIT_03.dbf'
MRP0: Background Media Recovery terminated with error 1274
Errors in file /ora_backup/TEST/duplicate_test/AMIT/diag/diag/rdbms/standby/STANDBY/trace/STANDBY_pr00_12842.trc:
ORA-01274: cannot add datafile '/ora_backup/TEST1/restore_test/AMIT/DATA/AMIT_03.dbf' - file could not be created
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.30).
Datafiles are recovered to a consistent state at change 5662710 but controlfile could be ahead of datafiles.

Problem :-

It seems in primary a datafile has been added and it's not able to replicate the same in standby due to above errors.

Cause :-

Above error can cause due to below reasons :-

1. STANDBY_FILE_MANAGEMNET parameter is set to maual
2. If having Different file structure, db_file_name_convert is not properly set according to standby directory structure or db_create_file_dest parameter is not set in standby

Solution :-

In these kind of scenario when due to above parameters are missing in standby,
when we add a datafile in primary it failed to replicate in standby. We can see below mentioned error in alert log and recovery will stop due to this.

ORA-01274: cannot add datafile '/ora_backup/TEST1/restore_test/AMIT/DATA/AMIT_03.dbf' - file could not be created
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.30).

Datafile detail will be present in standby controlfile but with below name :-


/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00013

So to resume recovery in these kind of scenarios we have to maualy create this datafile and start the recovery process. PFB steps :-

1. Check standby_file_management is set to manual

to add a datafile maualy in STANDBY database above parameter has to be in manual state

SQL> alter system set standby_file_management=manual;

System altered.

from DB_BROKER also we can change this :-

DGMGRL> edit database 'STANDBY' set property standbyfilemanagement=manual;
Property "standbyfilemanagement" updated

2. Create the database file with correct name in Standby database

SQL> alter database create datafile '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00013' as '/ora_backup/TEST/duplicate_test/AMIT/DATA/STANDBY/datafile/amit03.dbf';

Database altered.

3. Check the datafile details in controlfile :-

/ora_backup/TEST/duplicate_test/AMIT/DATA/STANDBY/datafile/amit03.dbf

now its coming appropriate

4. Start the Mrp process to resume recovery.

SQL> alter database recover managed standby database using current logfile disconnect;

Database altered.

Check the alert logfile to verify everything is fine. You can also check from Dataguard regarding sync details :-

DGMGRL> show database "STANDBY"

Database - STANDBY

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 2 seconds ago)
  Apply Lag:       0 seconds (computed 2 seconds ago)
  Apply Rate:      0 Byte/s
  Real Time Query: OFF
  Instance(s):
    STANDBY

Database Status:
SUCCESS

To get rid from these kind of issues in future , I did following changes in this database:-

1. change STANDBY_FILE_MANAGEMENt to AUTO
2. Set the db_file_name_convert parameter accoding to standby database.

I hope this article helped you.

Regards,
Amit Rath

Friday, August 9, 2013

Difference between Physical Standby and Logical Standby Database.

Standby databases act as a failover for our Production Databases. If Production Database crashes , running applications can quickly switchover to standby database.

There are two types of stanby database , Physical standby and Logical Standby.

1. Physical Standby :-  

  a. Physical Standby Schema Structure matches exactly with primary database.

  b. Archived redo logs are moved directly from primary database to physical standby which is running in recover mode. Once archive log arrived , they are applied directly to standby database.

2. Logical Standby :-

  a. Logical standby schema structure does not have to match with Primary database. It can have a different structure.

  b. Logical Standby uses Log Miner Technique and change the archive redo logs in primary to change it into native DML statements i,e insert,update and delete . These DMLs are transported and then applied to logical standby.

  c. Logical Standby can have additional Indexes and Materialized views ahead of Primary database which enhance faster performance of database.

  d. Logical standby tables can be open as Read only and all other standby tables can be open for updates.

Advantages of having Physical Standby Database :-

1. Identical copy of primary database.
2. Disaster Recovery, High availability and High Data Protection.
3. Primary database workload is reduced, can be open in Read-Only mode for querying purpose.

Advantages of having Logical Standby Database :-

1. Simultaneously can be used for querying  , reporting and summation.
2. There are some limitation on the use of certain datatypes.
3. Reduction in Primary database workload.

I hope this article helped you.

Regards,
Amit Rath

Friday, July 26, 2013

Convert a Failed Primary Into a Physical Standby Database Using Flashback Database.

When a failover occurs in Data Guard configuration and we have activated our standby database as our new primary database then our old primary database cannot participate again in data guard configuration unless it is recoverd as a new standby database in our new configuration.

We can recover our old primary database using FLASHBACK as well as RMAN backups

PFB steps to recover a failed primary database using Flashback database :-

1. Mount the Old Primary Database :-

bash-3.2$ sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 26 09:48:02 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Enter user-name: /as sysdba
Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

2. Find out the SCN at which old standby database becomes new primary :-

login to Old standby which is now primary in new configuration

bash-3.2$ sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 26 09:48:02 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Enter user-name: /as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

 SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)
----------------------------------------
1120211

3. Login to OLD primary database(failover primary) and run Flashback :-

SQL> flashback database to scn 1120211;

Flashback complete.

You will get below mentioned output after running flashback :-

flashback database to scn 1120211
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 24 slaves
Flashback Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_4_8z3x5913_.arc
Flashback Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_5_8z3xbv5f_.arc
Flashback Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_6_8z3xrgo0_.arc
Flashback Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_7_8z3xvf84_.arc
Flashback Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_8_8z3xvg7v_.arc
Fri Jul 26 09:49:00 2013
Recovery of Online Redo Log: Thread 1 Group 3 Seq 9 Reading mem 0
  Mem# 0: /pacs/oracle/TEST/TEST/redo03.log
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: /pacs/oracle/TEST/TEST/redo01.log
Incomplete Recovery applied until change 1120212 time 07/26/2013 09:45:33
Flashback Media Recovery Complete
Completed: flashback database to scn 1120211

5. Once old primary database has been recovered start the process to change it into new physical standby database :-

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Database altered.

This command will dismount the database and convert the control file to standby controlfile;

6. Shutdown the newly physical standby database and open it in mount mode.

SQL> SHUTDOWN IMMEDIATE;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

7. Start redo transport to the new physical standby database:-
`
Login to new Primary database and check current state of archive destinations :-


SQL> SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION,ERROR,SRL FROM V$ARCHIVE_DEST_STATUS;

   DEST_ID DEST_NAME                      STATUS    PROTECTION_MODE      DESTINATION                    ERROR                          SRL
---------- ------------------------------ --------- -------------------- ------------------------------ ------------------------------ ---
         1 LOG_ARCHIVE_DEST_1             VALID     MAXIMUM PERFORMANCE  /oracle/oracle/ora11g/product/                                NO
                                                                         dbs/arch

         2 LOG_ARCHIVE_DEST_2             VALID     MAXIMUM PERFORMANCE  TEST                                                          YES

if LOG_ARCHIVE_DEST_2 is set to new standby database then check redo transport and if not then PFB steps to configure it :-

SQL >ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=TEST NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=TEST';


SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

SQL> Alter system  switch logfile;

system altered 

SQL> SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION,ERROR,SRL FROM V$ARCHIVE_DEST_STATUS;

Error column has to be blank for LOG_ARCHIVE_DEST_2.

Now login to new standby database and start redo apply :-


SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

PFB output in alert log once redo apply process started :-


RFS[9]: Assigned to RFS process 59572450
RFS[9]: Identified database type as 'physical standby': Client is ARCH pid 56098968
RFS[9]: Selected log 5 for thread 1 sequence 10 dbid 2119844559 branch 821789263
Fri Jul 26 09:57:23 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Archived Log entry 77 added for thread 1 sequence 10 ID 0x7e5b2743 dest 1:
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_7_8z3yo8wh_.arc
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_8_8z3yo989_.arc
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_9_8z3yoclf_.arc
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_26/o1_mf_1_10_8z3yocxs_.arc
Media Recovery Waiting for thread 1 sequence 11 (in transit)

Failed Primary successfully converted to new Physical standby database. If we want we can again switchover the roles of newly created primary and physical standby databases to their original pre-failure roles.
I hope this article helped you.

Regards,
Amit Rath

Convert a Failed Primary Into a Physical Standby Database Using RMAN backups

When a failover occurs in Data Guard configuration and we have activated our standby database as our new primary database then our old primary database cannot participate again in data guard configuration unless it is recoverd as a new standby database in our new configuration.

We can recover our old primary database using FLASHBACK as well as RMAN backups

PFB steps to recover a failed primary database using RMAN backup :-

1. Mount the Old Primary Database :-

bash-3.2$ sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 26 09:48:02 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Enter user-name: /as sysdba
Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

2. Find out the SCN at which old standby database becomes new primary :-

login to Old standby which is now primary in new configuration

 SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)
----------------------------------------
1120211

3. Login to OLD primary database(failover primary) and connect with RMAN and restore and recover database using backup available of failover primary before failover SCN . Remember its a incomplete recovery as we are doing a point in time recovery :-

bash-3.2$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Fri Jul 26 10:25:28 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TEST (DBID=2119844559, not open)

RMAN>

RMAN> run
2> {
3> set until scn 1055317;
4> restore database;
5> recover database;
6> }

executing command: SET until clause

Starting restore at 26-JUL-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1 device type=DISK
flashing back control file to SCN 1055317
ORACLE error from target database:
ORA-38729: Not enough flashback database log data to do FLASHBACK.

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /disk1/STBY/STBY/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /disk1/STBY/STBY/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /disk1/STBY/STBY/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /disk1/STBY/STBY/users01.dbf
channel ORA_DISK_1: reading from backup piece /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp
channel ORA_DISK_1: piece handle=/disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp tag=TAG20130725T111211
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:45
Finished restore at 26-JUL-13

Starting recover at 26-JUL-13
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 11 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_11_8z2vbqs5_.arc
archived log for thread 1 with sequence 12 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_12_8z2vbqj3_.arc
archived log for thread 1 with sequence 13 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_13_8z2vbqht_.arc
archived log for thread 1 with sequence 14 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_14_8z2vbqh2_.arc
archived log for thread 1 with sequence 15 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_15_8z2vbqhg_.arc
archived log for thread 1 with sequence 16 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_16_8z2vbqgt_.arc
archived log for thread 1 with sequence 17 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_17_8z2vbqhx_.arc
archived log for thread 1 with sequence 18 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_18_8z2vbqfq_.arc
archived log for thread 1 with sequence 19 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_19_8z2vbqho_.arc
archived log for thread 1 with sequence 20 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_20_8z2vbmoj_.arc
archived log for thread 1 with sequence 21 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_21_8z2vbn4z_.arc
archived log for thread 1 with sequence 22 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_22_8z2vh2r6_.arc
archived log for thread 1 with sequence 23 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_23_8z2wro33_.arc
archived log for thread 1 with sequence 24 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_24_8z2x4qjx_.arc
archived log for thread 1 with sequence 25 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_25_8z2x4vv4_.arc
archived log for thread 1 with sequence 26 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_26_8z2xcf4x_.arc
archived log for thread 1 with sequence 27 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_27_8z2xcj92_.arc
archived log for thread 1 with sequence 28 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_28_8z2xhjl0_.arc
archived log for thread 1 with sequence 29 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_29_8z2xv5gp_.arc
archived log for thread 1 with sequence 30 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_30_8z3685n2_.arc
archived log for thread 1 with sequence 31 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_31_8z383o8v_.arc
archived log for thread 1 with sequence 32 is already on disk as file /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_32_8z38gryx_.arc
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=10
channel ORA_DISK_1: reading from backup piece /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111259_8z1gq36b_.bkp
channel ORA_DISK_1: piece handle=/disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111259_8z1gq36b_.bkp tag=TAG20130725T111259
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_10_8z55bzxz_.arc thread=1 sequence=10
channel default: deleting archived log(s)
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_10_8z55bzxz_.arc RECID=36 STAMP=821788047
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_11_8z2vbqs5_.arc thread=1 sequence=11
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_12_8z2vbqj3_.arc thread=1 sequence=12
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_13_8z2vbqht_.arc thread=1 sequence=13
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_14_8z2vbqh2_.arc thread=1 sequence=14
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_15_8z2vbqhg_.arc thread=1 sequence=15
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_16_8z2vbqgt_.arc thread=1 sequence=16
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_17_8z2vbqhx_.arc thread=1 sequence=17
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_18_8z2vbqfq_.arc thread=1 sequence=18
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_19_8z2vbqho_.arc thread=1 sequence=19
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_20_8z2vbmoj_.arc thread=1 sequence=20
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_21_8z2vbn4z_.arc thread=1 sequence=21
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_22_8z2vh2r6_.arc thread=1 sequence=22
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_23_8z2wro33_.arc thread=1 sequence=23
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_24_8z2x4qjx_.arc thread=1 sequence=24
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_25_8z2x4vv4_.arc thread=1 sequence=25
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_26_8z2xcf4x_.arc thread=1 sequence=26
archived log file name=/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_27_8z2xcj92_.arc thread=1 sequence=27
media recovery complete, elapsed time: 00:00:05
Finished recover at 26-JUL-13

NOTE:- If backup of failed primary is not avaible recreate the physical standby using backup of newly created primary database after failover.

PFB output of alert logfile once RMAN restore and recovery is applied :-

alter database recover logfile '/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_27_8z2xcj92_.arc'
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_27_8z2xcj92_.arc
Recovery of Online Redo Log: Thread 1 Group 3 Seq 28 Reading mem 0
  Mem# 0: /disk1/STBY/STBY/redo03.log
Recovery of Online Redo Log: Thread 1 Group 4 Seq 29 Reading mem 0
  Mem# 0: /disk1/STBY/STBY/online_redo01.log
Recovery of Online Redo Log: Thread 1 Group 5 Seq 30 Reading mem 0
  Mem# 0: /disk1/STBY/STBY/online_redo02.log
Recovery of Online Redo Log: Thread 1 Group 6 Seq 31 Reading mem 0
  Mem# 0: /disk1/STBY/STBY/online_redo03.log
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32 Reading mem 0
  Mem# 0: /disk1/STBY/STBY/redo01.log
Incomplete Recovery applied until change 1055317 time 07/25/2013 17:08:24
Media Recovery Complete (TEST)
Completed: alter database recover logfile '/disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_25/o1_mf_1_27_8z2xcj92_.arc'

4. Once old primary database has been recovered start the process to change it into new physical standby database :-

bash-3.2$ sqlplus

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 26 10:28:05 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Enter user-name: /as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Database altered.

This command will dismount the database and convert the control file to standby controlfile;

6. Shutdown the newly physical standby database and open it in mount mode.

SQL> SHUTDOWN IMMEDIATE;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

7. Start redo transport to the new physical standby database:-

Login to new Primary database and check current state of archive destinations :-

SQL> SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION,ERROR,SRL FROM V$ARCHIVE_DEST_STATUS;

   DEST_ID DEST_NAME                      STATUS    PROTECTION_MODE      DESTINATION                    ERROR                          SRL
---------- ------------------------------ --------- -------------------- ------------------------------ ------------------------------ ---
         1 LOG_ARCHIVE_DEST_1             VALID     MAXIMUM PERFORMANCE  /oracle/oracle/ora11g/product/                                NO
                                                                         dbs/arch

         2 LOG_ARCHIVE_DEST_2             VALID     MAXIMUM PERFORMANCE  TEST                                                          YES

if LOG_ARCHIVE_DEST_2 is set to new standby database then check redo transport and if not then PFB steps to configure it :-

SQL > ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=TEST NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=TEST';

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

SQL> Alter system  switch logfile;

system altered 

SQL> SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION,ERROR,SRL FROM V$ARCHIVE_DEST_STATUS;

Error column has to be blank for LOG_ARCHIVE_DEST_2.

Now login to new standby database and start redo apply :-

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

PFB output in alert log once redo process starts :-

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION
Attempt to start background Managed Standby Recovery process (TEST)
Fri Jul 26 10:42:14 2013
MRP0 started with pid=54, OS id=8716550
MRP0: Background Managed Standby Recovery process started (TEST)
 started logmerger process
Fri Jul 26 10:42:19 2013
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 24 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_1_8z55gw2x_.arc
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_2_8z55gw2v_.arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_3_8z55gw2r_.arc
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_4_8z55ndh1_.arc
Media Recovery Log /disk1/STBY/flash_recovery_area/STBY/archivelog/2013_07_26/o1_mf_1_5_8z55ndl6_.arc
Media Recovery Waiting for thread 1 sequence 6

Failed Primary successfully converted to new Physical standby database. If we want we can again switchover the roles of newly created primary and physical standby databases to their original pre-failure roles.


I hope this article helped you.

Regards,
Amit Rath

Failover of Primary database in Oracle Data Gaurd

If by some how Primary database is not available then we can activate or configure standby database as a Primary database. When we activate standby database as primary , then we can have data in our new primary upto the last redo transmitted and applied.

If possible your old primary is accessible then try to perform a switchover

PFB steps to activate standby database as new primary :-

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;

Database altered.

SQL> ALTER DATABASE ACTIVATE STANDBY DATABASE;

Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL> alter database open;

Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

Our new primary database is ready to be use. Take a RMAN Backup of new primary database immediately to be on Safer side.

Switchover/switchback are safe for both primary and standby database, but failover makes the Original primary database useless for converting to a standby database . If flashback database is not enabled then Original Primary must be restored from a previous backup and then recreated as a standby database.

--NOTE :- Always have your Primary database and standby database configured with FLASHBACK, so that in event of failover , original primary can be flashed back to the time before failover and can be easily converted to standby.

Find out the scn at which this standby database become primary , this SCN can be used to flashback or recover your original primary as a standby database.

SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)
----------------------------------------
1120211

PFB link to Convert a Failed Primary Into a Physical Standby Database Using Flashback Database.

PFB link to Convert a Failed Primary into a Physical Standby Database Using RMAN Backups.

I hope this article helped you.

Regards,
Amit Rath


Thursday, July 25, 2013

Read - only Standby and Active Data Guard feature in Oracle Data Guard

A physical standby database can be opened in Read only mode so that it can be used to query access. This feature of Data Guard is used to freeing up resource on Primary database.

When opened in Read - only mode , Log shipping from primary database to Standby continues but Managed recovery stopped. It makes the standby database out of data from Primary.

Steps to Switch standby database in Read - Only mode :-


SQL> SHUTDOWN IMMEDIATE;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.
SQL> ALTER DATABASE OPEN READ ONLY;

Database altered.


SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY

--NOTE :- Database can be used for querying , reporting purposes.

We can check that Redo will be shipped to Standby database but it will not be applied :-

1. Check alert log of primary database :-

Thu Jul 25 17:02:29 2013
Thread 1 advanced to log sequence 32 (LGWR switch)
  Current log# 1 seq# 32 mem# 0: /disk1/STBY/STBY/redo01.log
Thu Jul 25 17:02:29 2013
Archived Log entry 32 added for thread 1 sequence 31 ID 0x7e5a4c15 dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 31 for destination LOG_ARCHIVE_DEST_2
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
LNS: Standby redo logfile selected for thread 1 sequence 32 for destination LOG_ARCHIVE_DEST_2

################### Log has been copied to standby dataabse#####################

2. Check Alert log of Standby Database :-

RFS[1]: Selected log 4 for thread 1 sequence 31 dbid 2119844559 branch 821698130
Thu Jul 25 16:01:46 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Archived Log entry 43 added for thread 1 sequence 31 ID 0x7e5a4c15 dest 1:
Thu Jul 25 16:01:46 2013
RFS[2]: Assigned to RFS process 26018132
RFS[2]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 60030994
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2]: Selected log 4 for thread 1 sequence 32 dbid 2119844559 branch 821698130
Thu Jul 25 16:01:48 2013
RFS[3]: Assigned to RFS process 27787546
RFS[3]: Identified database type as 'physical standby': Client is ARCH pid 12648536

#####################Log has been reached but not applied#########################

To Resume Managed recovery and we have to change standby database from Read - only mode to recovery mode :-

SQL> SHUTDOWN IMMEDIATE;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP MOUNT;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

Active Data Guard :- Its a feature of Oracle Data Guard introduced in 11g version. In this we can open our physical standby database in Read - Only mode and still apply redo information to it.

In this feature a standby is available for querying but still be up to date with Primary. As managed recovery continues with Active data Guard so we do not have to switch back to managed recovery mode from read only mode .

Steps to implement Active Data Guard :-

SQL> SHUTDOWN IMMEDIATE;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP MOUNT;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

SQL> ALTER DATABASE OPEN READ ONLY;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

I hope this article helped you.

Regards,
Amit Rath

How to change Physical standby to Snapshot standby database in Oracle Data Guard

Snapshot standby is a feature of Oracle Data Guard. It introduced in Oracle 11g and allows the standby database to be opened in Read - Write mode.

After changing a physical standby database to snapshot standby it can be used as a normal read write database.

When this snapshot standby switched back to standby mode , all changes which you have done in read write mode will be lost. This is achieved by Flashback database, but its not necessary that flashback database has to be enabled to achieve this.

Prerequisites of Snapshot standby :-

1. Managed recovery has to be disabled.
2. Standby database has to be in MOUNT mode.

Steps to create a snapshot standby from Physical standby database :-

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Database altered.

SQL> SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
NO

SQL> ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;

Database altered.

SQL> ALTER DATABASE OPEN;

Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
RESTORE POINT ONLY

Now we can use this database as a normal database.

SQL> create table amit
  2  (
  3  id number(12),
  4  name varchar(23));

Table created.

SQL> insert into amit values('&id','&name');
Enter value for id: 1
Enter value for name: f
old   1: insert into amit values('&id','&name')
new   1: insert into amit values('1','f')

1 row created.

SQL> /
Enter value for id: 2
Enter value for name: gf
old   1: insert into amit values('&id','&name')
new   1: insert into amit values('2','gf')

1 row created.

SQL> commit;

Commit complete.

SQL> select * from amit;

        ID NAME
---------- -----------------------
         1 f
         2 gf

We will get below mentioned output in alert log of standby database after snapshot standby process completed :-

Thu Jul 25 15:17:38 2013
RFS[145]: Assigned to RFS process 7930216
RFS[145]: Identified database type as 'snapshot standby': Client is ARCH pid 12648536
Thu Jul 25 15:17:39 2013

To convert it back to physical standby, we will lose all changes which we have made in sanpshot standby mode.

SQL> SHUTDOWN IMMEDIATE;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP MOUNT;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
Database mounted.

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Database altered.

SQL> SHUTDOWN IMMEDIATE;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup nomount
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes
SQL> ALTER DATABASE MOUNT STANDBY DATABASE;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

Database altered.

SQL> SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
NO

Snapshot standby database switched to Physical standby database. Test redo transport with Primary database :-

SQl> alter system switch logfile;

System altered.

PFB output in alert log of Physical standby dataabase :-

RFS[10]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 60030994
Primary database is in MAXIMUM PERFORMANCE mode
RFS[10]: Selected log 4 for thread 1 sequence 31 dbid 2119844559 branch 821698130
Thu Jul 25 15:30:12 2013
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_25/o1_mf_1_30_8z1xs2tg_.arc
Media Recovery Waiting for thread 1 sequence 31 (in transit)

Standby Database has been switched back successfully to Physical standby from Snapshot standby.

I hope this article helped you

Regards,
Amit Rath

How to do Quick Switchover with Physical Standby Database

SwitchOver is a feature of Oracle Data Guard in which a primary database is allowed to switch roles with its standby database. In process of Switch , there is no data loss. we can move back to original primary database by making another switch over called as 'SWITCH BACK' .

By performing Switch over , Down time can be reduced to a great extent. In Switch over , standby database becomes primary database in open and read write mode and Original Primary becomes standby in mounted mode. Once our work with Original Primary completed we can switch back their roles. There is no data loss in this process.

Prerequisites for SwitchOver :-

1. Original Primary has to be in Open mode and Standby to be in Mounted mode.
2. There are no active users to be there in Primary database.
3. Perform a logswitch in Primary database before applying switchover.

Switchover steps :-

1. Initiate the Switch over process in primary database :-

SQL> alter system switch logfile;

System altered.

2. Check on both primary and standby that last redo has been applied.

Primary :-
SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';

Session altered.

SQL> SELECT sequence#, first_time, next_time
FROM   v$archived_log
ORDER BY sequence#; 

 SEQUENCE# FIRST_TIME                    NEXT_TIME
---------- ----------------------------- -----------------------------
       21 25-JUL-2013 12:23:36          25-JUL-2013 12:25:59
        22 25-JUL-2013 12:25:59          25-JUL-2013 12:48:09
        22 25-JUL-2013 12:25:59          25-JUL-2013 12:48:09

Standby :-

SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';

Session altered.

SQL> SELECT sequence#, first_time, next_time, applied
FROM   v$archived_log
ORDER BY sequence#; 

 SEQUENCE# FIRST_TIME           NEXT_TIME            APPLIED
---------- -------------------- -------------------- ---------
       20 25-JUL-2013 12:23:02 25-JUL-2013 12:23:36 YES
        21 25-JUL-2013 12:23:36 25-JUL-2013 12:25:59 YES
        22 25-JUL-2013 12:25:59 25-JUL-2013 12:48:09 YES

3. Change Primary database to standby mode :-

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY;

Database altered.

SQL> SHUTDOWN IMMEDIATE;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> STARTUP NOMOUNT;
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             817891832 bytes
Database Buffers          213909504 bytes
Redo Buffers                9871360 bytes

SQL> ALTER DATABASE MOUNT STANDBY DATABASE;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

you will get below mentioned output in your alert log :-

All non-current ORLs have been archived.
Clearing online redo logfile 1 /pacs/oracle/TEST/TEST/redo01.log
Clearing online log 1 of thread 1 sequence number 25
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /pacs/oracle/TEST/TEST/redo02.log
Clearing online log 2 of thread 1 sequence number 23
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /pacs/oracle/TEST/TEST/redo03.log
Clearing online log 3 of thread 1 sequence number 24
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION
Clearing online redo logfile 3 complete
Media Recovery Waiting for thread 1 sequence 26

4. Change standby database to Primary :-

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

Database altered.

SQL> SHUTDOWN IMMEDIATE;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size                  2213384 bytes
Variable Size             666896888 bytes
Database Buffers          364904448 bytes
Redo Buffers                9871360 bytes
Database mounted.
Database opened.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

5. Check Log Transport from primary to Standby :-

SQL> alter system switch logfile;

System altered.

Output in Primary database alert log PFB :-

Thread 1 advanced to log sequence 30 (LGWR switch)
  Current log# 5 seq# 30 mem# 0: /disk1/STBY/STBY/online_redo02.log
Thu Jul 25 14:07:17 2013
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
Thu Jul 25 14:07:17 2013
Archived Log entry 29 added for thread 1 sequence 29 ID 0x7e5a4c15 dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 30 for destination LOG_ARCHIVE_DEST_2

Output in Standby database alert log :-

Thu Jul 25 13:06:34 2013
RFS[14]: Assigned to RFS process 61014246
RFS[14]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 60030994
Primary database is in MAXIMUM PERFORMANCE mode
RFS[14]: Selected log 4 for thread 1 sequence 30 dbid 2119844559 branch 821698130
Thu Jul 25 13:06:35 2013
Media Recovery Log /pacs/oracle/flash_recovery_area/TEST/archivelog/2013_07_25/o1_mf_1_29_8z1od2dz_.arc
Media Recovery Waiting for thread 1 sequence 30 (in transit)

Once redo test completed , switchover has been successfully completed. We can move back to original server by doing another switchover called SWITCHBACK.

I hope this article helped you.

Regards,
Amit Rath

Wednesday, July 24, 2013

Step by Step Create an Oracle Physical standby database

Standby database can act as failover for critical Production databases. When Production database crashes application can quickly switchover to Standby databases.

There are two types of Standby databases , physical standby and logical standby.

Requirements :-

1. Two databases one for Primary and one for standby.
2. Both databases have network connectivity between them.
3. Oracle Databse 10gR1 have to be installed on both Database servers.
4. Both Database have to be in Archivelog mode.

Values used in this Example :-

1. Primary database name is TEST.
2. Standby database name is STBY.
3. Primary database hostname is primary_server and standby database hostname is standby_server

PFB steps to create a physical standby database for a primary oracle database :-

1. Preparing Primary database for standby database creation :-

[oracle@new TEST]$ export ORACLE_SID=TEST

[oracle@new TEST]$ sqlplus

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jul 18 20:15:18 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Enter user-name: /as sysdba

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> select name from v$database;

NAME
---------
TEST

a. Check log mode of primary database , if not in Archive mode then change it to Archive mode . Link to change database in archivelog mode Enable Archive mode :-

SQL> SELECT log_mode FROM v$database;

LOG_MODE
------------
ARCHIVELOG

b. Change primary database to Force Logging mode :-

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.

c. Initialization parameters in Primary database :-

SQL>  show parameter db_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_name                              string      TEST
SQL> show parameter db_unique_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      TEST

d. Set the DG_CONFIG setting of LOG_ARCHIVE_CONFIG parameter. for this DB_UNIQUE_NAME of standby database has to be different from primary.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(TEST,STBY)';

System altered.

e. Set suitable remote archivelog destinations. 

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=STBY NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=STBY';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

System altered.

-- NOTE :- Service and DB_UNIQUE_NAME has to be of Standby database

f. Set parameters related to LOG_ARCHIVE and password file parameters :-


SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30;

System altered.

SQL> ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;

System altered.

g. It is recommended to make sure that primary can switch roles to become a standby. For that we need to set some -convert parameters if there are path differences between the servers :-

SQL> ALTER SYSTEM SET FAL_SERVER=STBY;

System altered.

SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='STBY','TEST' SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='STBY','TEST'  SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

System altered.

h . As some parameters cannot be changed while database is open , we need to restart database to before they take effect :-

SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
SQL> startup
ORACLE instance started.

Total System Global Area  960495616 bytes
Fixed Size                  1222816 bytes
Variable Size             247465824 bytes
Database Buffers          700448768 bytes
Redo Buffers               11358208 bytes
Database mounted.
Database opened.
SQL>

i . In both servers entries of Primary as well as standby are needed in tnsnames.ora file and we are able to do  tnsping.

###################ORACLE_DATA_GAURD############################

TEST =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST)
    )
  )

STBY =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = standby_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = STBY)
    )
  )
###################ORACLE_DATA_GAURD############################


[oracle@new admin]$ tnsping STBY

TNS Ping Utility for Linux: Version 10.2.0.1.0 - Production on 18-JUL-2013 20:23:36

Copyright (c) 1997, 2005, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = standby_server)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = STBY)))
OK (0 msec)
[oracle@new admin]$ tnsping TEST

TNS Ping Utility for Linux: Version 10.2.0.1.0 - Production on 18-JUL-2013 20:23:41

Copyright (c) 1997, 2005, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TEST)))
OK (0 msec)

j. Backup primary database . PFB link to take backup 
Backup of Oracle database

k. Create Standby controlfile and Pfile :-

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/disk1/stby1.ctl';

Database altered.

SQL> CREATE PFILE='/disk1/stby1.ora' from spfile;

File created.

l. Add standby redo logs to primary database :-

ALTER DATABASE ADD STANDBY LOGFILE ('/disk1/TEST/TEST/standby_redo01.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/disk1/TEST/TEST/standby_redo02.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/disk1/TEST/TEST/standby_redo03.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/disk1/TEST/TEST/standby_redo04.log') SIZE 50M;

2. Preparing Standby database for physical standby creation :-

a. Copy backup of primary database, archivelogs, standby controlfile, parameter file and passwordfile to standby database server. 

copy all backup and archive logs from primary to standby

[oracle@primary_server]$ pwd
/disk1/STBY/flash_recovery_area/TEST
[oracle@primary_server]$ ls
archivelog  autobackup  backupset  onlinelog
[oracle@primary_server]$ scp -r * oracle@standby_server:/disk1/TEST/flash_recovery_area/TEST

copy standby controlfile, parameter file and passwordfile

[oracle@standby_server TEST]$ scp oracle@primary_server:/disk1/stby.ctl /disk1/TEST/TEST/control01.ctl
oracle@primary_server's password:
stby.ctl                                                                                                                              100% 6896KB   6.7MB/s   00:00
[oracle@ standby_server  TEST]$
[oracle@ standby_server  TEST]$ cp control01.ctl control02.ctl
[oracle@ standby_server  TEST]$ cp control01.ctl control03.ctl

like this copy parameter file as well as password file to standby server.

b. Edit the standby parameter file according to the standby destinations. The parameters in green have to be change according to Standby server :-

*.audit_file_dest='/disk2/oracle/admin/STBY/adump'
*.background_dump_dest='/disk2/oracle/admin/STBY/bdump'
*.compatible='10.2.0.1.0'
*.control_files='/disk1/STBY/STBY/control01.ctl','/disk1/STBY/STBY/control02.ctl','/disk1/STBY/STBY/control03.ctl'
*.core_dump_dest='/disk2/oracle/admin/STBY/cdump'
*.db_block_size=8192
*.db_domain='local'
*.db_file_multiblock_read_count=16
*.db_file_name_convert='STBY','TEST'
*.db_name='TEST'
*.db_unique_name='STBY'
*.db_recovery_file_dest='/disk1/STBY/flash_recovery_area'
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=TESTXDB)'
*.fal_server='TEST'
*.job_queue_processes=10
*.log_archive_config='DG_CONFIG=(TEST,STBY)'
*.log_archive_dest_2='SERVICE=TEST NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=TEST'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='%t_%s_%r.arc'
*.log_archive_max_processes=30
*.log_file_name_convert='STBY','TEST'
*.open_cursors=300
*.pga_aggregate_target=318767104
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=958398464
*.standby_file_management='AUTO'
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/disk2/oracle/admin/STBY/udump'

c. Start the standby database using the standby parameter file restore the backup of primary database

SQL> startup nomount pfile='/disk1/stby.ora'
ORACLE instance started.

Total System Global Area  960495616 bytes
Fixed Size                  1222816 bytes
Variable Size             264243040 bytes
Database Buffers          683671552 bytes
Redo Buffers               11358208 bytes
SQL> create spfile from pfile;
SQL> shu immediate
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup nomount
ORACLE instance started.

Total System Global Area  960495616 bytes
Fixed Size                  1222816 bytes
Variable Size             264243040 bytes
Database Buffers          683671552 bytes
Redo Buffers               11358208 bytes

[oracle@new dbs]$ rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Jul 23 21:17:15 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: TEST (not mounted)
RMAN> alter database mount;

database mounted

RMAN> catalog start with ' /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1 ';


Starting implicit crosscheck backup at 25-JUL-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=58 device type=DISK
Crosschecked 4 objects
Finished implicit crosscheck backup at 25-JUL-13

Starting implicit crosscheck copy at 25-JUL-13
using channel ORA_DISK_1
Finished implicit crosscheck copy at 25-JUL-13

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1

List of Files Unknown to the Database
=====================================
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_10_8z1gq2o5_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_5_8z1gdcos_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_6_8z1gdgnr_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_7_8z1gks5p_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_8_8z1gkw29_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_9_8z1gok73_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111209_8z1gol3o_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111259_8z1gq36b_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_ncsnf_TAG20130725T111211_8z1gq1js_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_10_8z1gq2o5_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_5_8z1gdcos_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_6_8z1gdgnr_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_7_8z1gks5p_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_8_8z1gkw29_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_1_9_8z1gok73_.arc
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111209_8z1gol3o_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_annnn_TAG20130725T111259_8z1gq36b_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_ncsnf_TAG20130725T111211_8z1gq1js_.bkp
File Name: /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp

 RMAN>  run
2> {
3> set newname for datafile 1 to '/disk1/STBY/STBY/system01.dbf';
4> set newname for datafile 2 to '/disk1/STBY/STBY/undotbs01.dbf';
5> set newname for datafile 3 to '/disk1/STBY/STBY/sysaux01.dbf';
6> set newname for datafile 4 to '/disk1/STBY/STBY/users01.dbf';
7> restore database;
8> switch datafile all;
9> }

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 25-JUL-13
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /disk1/STBY/STBY/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /disk1/STBY/STBY/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /disk1/STBY/STBY/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /disk1/STBY/STBY/users01.dbf
channel ORA_DISK_1: reading from backup piece /disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp
channel ORA_DISK_1: piece handle=/disk1/STBY/flash_recovery_area/STBY/2013_07_25/o1_mf_nnndf_TAG20130725T111211_8z1gomb4_.bkp tag=TAG20130725T111211
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:45
Finished restore at 25-JUL-13

datafile 1 switched to datafile copy
input datafile copy RECID=5 STAMP=821709808 file name=/disk1/STBY/STBY/system01.dbf
datafile 2 switched to datafile copy
input datafile copy RECID=6 STAMP=821709808 file name=/disk1/STBY/STBY/undotbs01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=7 STAMP=821709808 file name=/disk1/STBY/STBY/sysaux01.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=8 STAMP=821709808 file name=/disk1/STBY/STBY/users01.dbf

d. Create Online redo logs for Standby server :-

ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;
ALTER DATABASE ADD LOGFILE ('/disk1/STBY/STBY/online_redo01.log') SIZE 50M;
ALTER DATABASE ADD LOGFILE ('/disk1/STBY/STBY/online_redo02.log') SIZE 50M;
ALTER DATABASE ADD LOGFILE ('/disk1/STBY/STBY/online_redo03.log') SIZE 50M;
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

e. Before Starting the apply process we have to make necessary changes in tnsnames.ora file, as we restored primary to standby , dbname of primary has been changed from STBY to TEST


###################ORACLE_DATA_GAURD############################

TEST =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST)
    )
  )

STBY =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = standby_server)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TEST)
    )
  )
###################ORACLE_DATA_GAURD############################

[oracle@new admin]$ tnsping STBY

TNS Ping Utility for Linux: Version 10.2.0.1.0 - Production on 18-JUL-2013 20:23:36

Copyright (c) 1997, 2005, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = standby_server)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TEST)))
OK (0 msec)
[oracle@new admin]$ tnsping TEST

TNS Ping Utility for Linux: Version 10.2.0.1.0 - Production on 18-JUL-2013 20:23:41

Copyright (c) 1997, 2005, Oracle.  All rights reserved.

Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = primary_server)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TEST)))
OK (0 msec)

From primary as well as standby try to connect both database using Service name:-

[oracle@new admin]$sqlplus sys/test@TEST as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jul 18 20:15:18 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

[oracle@new admin]$sqlplus sys/test@STBY as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jul 18 20:15:18 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

--NOTE :- Before starting the apply process , test from both servers that we are able to connect both primary and standby database using password file. Once this done we can start the redo apply process.

f. Once all steps previous to this completes we can start the apply process :-

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

once we fire this command we get below mentioned output in alert log file of standby server :-

Tue Jul 23 14:51:35 2013
Attempt to start background Managed Standby Recovery process (TEST)
MRP0 started with pid=49, OS id=12659
Tue Jul 23 14:51:35 2013
MRP0: Background Managed Standby Recovery process started (TEST)
Managed Standby Recovery not using Real Time Apply
Clearing online redo logfile 1 /disk1/STBY/STBY/redo01.log
Clearing online log 1 of thread 1 sequence number 8
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /disk1/STBY/STBY/redo02.log
Clearing online log 2 of thread 1 sequence number 9
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /disk1/STBY/STBY/redo03.log
Clearing online log 3 of thread 1 sequence number 10
Tue Jul 23 14:51:41 2013
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION
Tue Jul 23 14:51:41 2013
Clearing online redo logfile 3 complete
Media Recovery Waiting for thread 1 sequence 11

f. Test the Redo Transport from primary database to standby database :-

On Primary Server :-

SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';

Session altered.

SQL> SELECT sequence#, first_time, next_time
FROM   v$archived_log
ORDER BY sequence#;  2    3

 SEQUENCE# FIRST_TIME           NEXT_TIME
---------- -------------------- --------------------
     
         8 19-JUL-2013 23:30:42 23-JUL-2013 14:26:31
         9 23-JUL-2013 14:26:31 23-JUL-2013 14:27:12
        10 23-JUL-2013 14:26:31 23-JUL-2013 14:27:12

3 rows selected.

SQL> ALTER SYSTEM SWITCH LOGFILE;

System altered.

On Standby Server :-

SQL> SELECT sequence#, first_time, next_time, applied
FROM   v$archived_log
ORDER BY sequence#;

 SEQUENCE# FIRST_TIME           NEXT_TIME    APP
---------- -------------------- --------------------    --------
         10  23-JUL-13                      23-JUL-13         YES
         11 23-JUL-13                      23-JUL-13         YES
         12 23-JUL-13                      23-JUL-13         YES
         13 23-JUL-13                      23-JUL-13         YES

once the applied process starts and redo is transported from Primary to Standby then you will get below mentioned output in Alert log file :-

RFS[3]: Identified database type as 'physical standby'
RFS[3]: Successfully opened standby log 4: '/disk1/STBY/STBY/standby_redo01.log'
Tue Jul 23 15:01:41 2013
Media Recovery Log /disk1/TEST/flash_recovery_area/TEST/archivelog/2013_07_23/o1_mf_1_12_8ywmcxdg_.arc
Media Recovery Waiting for thread 1 sequence 13
Tue Jul 23 15:57:41 2013
RFS[1]: Successfully opened standby log 4: '/disk1/STBY/STBY/standby_redo01.log'
Tue Jul 23 15:57:42 2013
Media Recovery Log /disk1/TEST/flash_recovery_area/TEST/archivelog/2013_07_23/o1_mf_1_13_8ywpnxyz_.arc
Media Recovery Waiting for thread 1 sequence 14

g. Check that Primary and Standby are in SYNC :- 

in Primary :-


SQL> SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;

no rows selected


SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
  2  FROM
  3  (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
  4  (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
  5  WHERE
  6  ARCH.THREAD# = APPL.THREAD#
  7  ORDER BY 1;

    Thread Last Sequence Received Last Sequence Applied Difference
---------- ---------------------- --------------------- ----------
         1                     27                    27          0
         1                     27                    27          0


in Standy :-

SQL> SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;

no rows selected

Output of both has to be "no rows selected"


SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
FROM
  2    3  (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
  4  (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
  5  WHERE
  6  ARCH.THREAD# = APPL.THREAD#
  7  ORDER BY 1;

    Thread Last Sequence Received Last Sequence Applied Difference
---------- ---------------------- --------------------- ----------
         1                      6                     6          0

Difference has to be zero.

Standby database has been successfully created and redo transported as well as applied from primary to Standby database.

To recover Primary and Standby database from failover scenarios its better to keep them in FLASHBACK mode.

I hope this article helped you.

Regards,
Amit Rath