Last week I came across a scenario where a standby database went out of synch with the primary and there was a big gap of archive logs to be applied. Instead of rebuilding the standby database from scratch or restoring and applying a zillion archives, we can make use of a feature available from the [...]]]> Last week I came across a scenario where a standby database went out of synch with the primary and there was a big gap of archive logs to be applied. Instead of rebuilding the standby database from scratch or restoring and applying a zillion archives, we can make use of a feature available from the version 10.2 onwards called the standby roll forward that uses RMAN incremental backup to apply the changes on the standby.

For databases on non-ASM, it should work with the first 4 steps as indicated below. For databases on ASM, we need to perform some more steps.

To keep it simple, I have outlined the steps below and have attached the logs for reference.

1) Identify the last SCN on the Standby, take the minimum value returned by the below queries for further steps.

SQL>
col current_scn for 99999999999999999999999
col int_scn for 99999999999999999999999

select current_scn FROM V$DATABASE;
select min(fhscn) int_scn from x$kcvfh;  

CURRENT_SCN
-----------------
47987632

2) Take an incremental backup on the primary database from this SCN to the current point.
The incremental backup files will be generated on the filesystem under the directory “/ora001/backup/” in the example.
Need to make sure that there is sufficient space in the filesystem to hold the backup. In this case, the backup files sized nearly 1.2G.

RMAN> CONFIGURE CHANNEL c1 DEVICE TYPE DISK FORMAT   '/ora001/backup/std_%U';
RMAN> BACKUP INCREMENTAL FROM SCN 47987632 DATABASE;

3) SCP the backup files to the server where the standby database is running and catalog the backup.

RMAN> CATALOG START WITH '/ora001/backup/std_%U';

This makes the controlfile aware of the incremental backup.

4) Recover the standby database.

RMAN>
RECOVER DATABASE NOREDO;

5) Check the archive gap and the SCN to see if the DB has applied the changes.

SQL>
col CURRENT_SCN for 99999999999999999999999
select * from v$archive_gap;
select current_scn from v$database;

The standby is in good shape if the SCN returned is a bigger value and there are no gaps.

—- Steps above should be sufficient for non-ASM databases; continue the steps below for databases on ASM. —-
(It was found that the SCN was not as expected and the steps below were done)

5) Cancel the database recovery at the standby site.

SQL>
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

6) Take the controlfile backup again on primary database and SCP it over to the standby server.

RMAN>
BACKUP CURRENT CONTROLFILE FOR STANDBY FORMAT '/ora001/ctrl_bkp.bak';

7) Take pfile, controlfile backups and note the current controlfile location on the standby and shutdown the standby database.

SQL>
select name  from v$controlfile;
shut immediate;
startup nomount;

8) Restore the standby controlfile created in step 6 on the standby site.

RMAN>
RESTORE STANDBY CONTROLFILE FROM '/ora001/ctrl_bkp.bak';
sql 'alter database mount';

9) At this point, the controlfiles on the standby database will have the datafiles pointed to the locations on primary. This needs to be updated so that they point to the right datafiles.

(For eg: On primary, it would be something like ‘+DATA01_R5/prod’ and on standby, the structure will be something like ‘+DATA01_R5/stby’)

RMAN>
CATALOG START WITH '+DATA01_R5/stby';

(May need to repeat this if there are files in multiple diskgroups)

10) Imp: If there are datafiles created on the primary database after the last SCN on standby (the one on step 1), we’ll need to backup the datafile(s), catalog the backup, do the restore of those file(s) too. The files can be identified with the below SQL:

SQL>
SELECT FILE#, NAME FROM V$DATAFILE WHERE CREATION_CHANGE# > 47987632;

11) Perform the switch so that the controlfile points to the datafiles under the standby directory.

RMAN>
SWITCH DATABASE TO COPY;

12) Clear redo log groups on the standby.

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 1;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 2;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 3;

13) Check the SCN on the standby and start the managed recovery. The SCN should be number much higher than that was on step 1 and there shouldn’t be no archive gaps.

SQL>
select current_scn from v$database;
select * from v$archive_gap;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

14) Do a couple of log switches on the primary and verify from the alert log that the standby is catching up with the primary.

15) Verify the standby redologs and other settings as required depending on the configuration and make sure the standby is in good shape.

1) Activate Force logging on the primary database

In case of unlogged writes to primary database (NOLOGGING  and DIRECT LOAD operations using SQL*Loader),  redo data is not generated.  As the redo required to synch the changes to the standby is not generated, the standby database will not be consistent with the [...]]]> Step by Step

1) Activate Force logging on the primary database

In case of unlogged writes to primary database (NOLOGGING  and DIRECT LOAD operations using SQL*Loader),  redo data is not generated.  As the redo required to synch the changes to the standby is not generated, the standby database will not be consistent with the primary.  To prevent such issues, the primary database is forced to log operations. It may also be noted that it is recommended to take a backup of the database after UNRECOVERABLE/NOLOGGING operations as those operations are not recoverable as the redo data is not generated.Use the below command on the primary database to force logging on the database.

SQL> alter database force logging;

This must be done before building the standby database.

2) Prepare Initialization files for the primary and standby databases

# Initialization file for primary database – initorcl.ora

*.audit_file_dest='/oracle/app/oracle/admin/orcl/adump'
*.audit_trail='db'
*.compatible='11.1.0.0.0'
*.control_files='/oradata/orcl/control01.ctl','/oradata/orcl/control02.ctl'
,'/oradata/orcl/control03.ctl'
*.db_block_size=8192
*.db_name='orcl'
*.diagnostic_dest='/oracle/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)'
*.log_archive_dest_1='LOCATION=/oradata/arch/orcl/'
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=104857600
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=316669952
*.undo_tablespace='UNDOTBS1'

##### Standby related parameters ######## 

*.db_unique_name='orcl'
*.fal_client='ORCL'
*.fal_server='STBY'
*.DB_FILE_NAME_CONVERT=('/oradata/stby/','/oradata/orcl/')
*.LOG_FILE_NAME_CONVERT=('/oradata/stby/','/oradata/orcl/') *.log_archive_config='dg_config=(orcl,stby)'
*.log_archive_dest_2='SERVICE=STBY valid_for=online_logfiles,primary_role)'
*.log_archive_dest_state_2=enable
*.standby_file_management=auto

##### End Standby related parameters #######

# Initialization file for standby database - initstby.ora 

*.audit_file_dest='/oracle/app/oracle/admin/stby/adump'
*.audit_trail='db'
*.compatible='11.1.0.0.0'
*.control_files='/oradata/stby/control01.ctl','/oradata/stby/control02.ctl',
'/oradata/stby/control03.ctl'
*.db_block_size=8192
*.db_name='orcl'
*.diagnostic_dest='/oracle/app/oracle'
*.dispatchers='(PROTOCOL=TCP)(SERVICE=orclXDB)'
*.log_archive_dest_1='LOCATION=/oradata/arch/stby/'
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=104857600
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=316669952
*.undo_tablespace='UNDOTBS1'

##### Standby related parameters#######

*.db_unique_name='stby'
*.log_archive_dest_2='SERVICE=ORCL valid_for=(online_logfile,primary_role)
*.DB_FILE_NAME_CONVERT=('/oradata/orcl/','/oradata/stby/')
*.LOG_FILE_NAME_CONVERT=('/oradata/orcl/','/oradata/stby/')
*.fal_server='ORCL'
*.fal_client='STBY'
*.log_archive_config='dg_config=(orcl,stby)'
*.standby_file_management=auto

##### End Standby related parameters #######

3) Configure Listener for Static service registration

Static service registrations is requierd for the RMAN active database duplication to work.

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dbapundits)(PORT = 1521))))  SID_LIST_LISTENER =
(
SID_LIST =
(SID_DESC = (ORACLE_HOME = /oracle/app/oracle/product/11.1.0) (SID_NAME = orcl))
(SID_DESC = (ORACLE_HOME = /oracle/app/oracle/product/11.1.0) (SID_NAME = stby))
)

4) Create service names in tnsnames.ora
Add tnsnames.ora entries for primary and standby databases.

ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbapundits)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)))
STBY =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbapundits)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = stby)))

5) Create password file for the standby database

The easiest way to do this is to copy the password file from the primary database and to rename it in proper format on the standby server.

$ pwd
/oracle/app/oracle/product/11.1.0/dbs
$ cp orapworcl orapwstby 

6) Perform Oracle 11g standby creation using RMAN DUPLICATE command

run
{
duplicate target database for standby dorecover from active database;
}

The primary database is duplicated as a standby database on the target server (or on same server in this case) using the datafiles from the active database. There is no need of a backup and the datafiles are copied over the network to the target server and is recovered to the point when the duplication was fired.

7) Logs

11g-standby-database-creation-with-rman-duplicate-command – log