Last week I came across a scenario where a standby database went out of synch with the primary and there was a big gap of archive logs to be applied. Instead of rebuilding the standby database from scratch or restoring and applying a zillion archives, we can make use of a feature available from the [...]]]> Last week I came across a scenario where a standby database went out of synch with the primary and there was a big gap of archive logs to be applied. Instead of rebuilding the standby database from scratch or restoring and applying a zillion archives, we can make use of a feature available from the version 10.2 onwards called the standby roll forward that uses RMAN incremental backup to apply the changes on the standby.

For databases on non-ASM, it should work with the first 4 steps as indicated below. For databases on ASM, we need to perform some more steps.

To keep it simple, I have outlined the steps below and have attached the logs for reference.

1) Identify the last SCN on the Standby, take the minimum value returned by the below queries for further steps.

SQL>
col current_scn for 99999999999999999999999
col int_scn for 99999999999999999999999

select current_scn FROM V$DATABASE;
select min(fhscn) int_scn from x$kcvfh;  

CURRENT_SCN
-----------------
47987632

2) Take an incremental backup on the primary database from this SCN to the current point.
The incremental backup files will be generated on the filesystem under the directory “/ora001/backup/” in the example.
Need to make sure that there is sufficient space in the filesystem to hold the backup. In this case, the backup files sized nearly 1.2G.

RMAN> CONFIGURE CHANNEL c1 DEVICE TYPE DISK FORMAT   '/ora001/backup/std_%U';
RMAN> BACKUP INCREMENTAL FROM SCN 47987632 DATABASE;

3) SCP the backup files to the server where the standby database is running and catalog the backup.

RMAN> CATALOG START WITH '/ora001/backup/std_%U';

This makes the controlfile aware of the incremental backup.

4) Recover the standby database.

RMAN>
RECOVER DATABASE NOREDO;

5) Check the archive gap and the SCN to see if the DB has applied the changes.

SQL>
col CURRENT_SCN for 99999999999999999999999
select * from v$archive_gap;
select current_scn from v$database;

The standby is in good shape if the SCN returned is a bigger value and there are no gaps.

—- Steps above should be sufficient for non-ASM databases; continue the steps below for databases on ASM. —-
(It was found that the SCN was not as expected and the steps below were done)

5) Cancel the database recovery at the standby site.

SQL>
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

6) Take the controlfile backup again on primary database and SCP it over to the standby server.

RMAN>
BACKUP CURRENT CONTROLFILE FOR STANDBY FORMAT '/ora001/ctrl_bkp.bak';

7) Take pfile, controlfile backups and note the current controlfile location on the standby and shutdown the standby database.

SQL>
select name  from v$controlfile;
shut immediate;
startup nomount;

8) Restore the standby controlfile created in step 6 on the standby site.

RMAN>
RESTORE STANDBY CONTROLFILE FROM '/ora001/ctrl_bkp.bak';
sql 'alter database mount';

9) At this point, the controlfiles on the standby database will have the datafiles pointed to the locations on primary. This needs to be updated so that they point to the right datafiles.

(For eg: On primary, it would be something like ‘+DATA01_R5/prod’ and on standby, the structure will be something like ‘+DATA01_R5/stby’)

RMAN>
CATALOG START WITH '+DATA01_R5/stby';

(May need to repeat this if there are files in multiple diskgroups)

10) Imp: If there are datafiles created on the primary database after the last SCN on standby (the one on step 1), we’ll need to backup the datafile(s), catalog the backup, do the restore of those file(s) too. The files can be identified with the below SQL:

SQL>
SELECT FILE#, NAME FROM V$DATAFILE WHERE CREATION_CHANGE# > 47987632;

11) Perform the switch so that the controlfile points to the datafiles under the standby directory.

RMAN>
SWITCH DATABASE TO COPY;

12) Clear redo log groups on the standby.

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 1;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 2;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 3;

13) Check the SCN on the standby and start the managed recovery. The SCN should be number much higher than that was on step 1 and there shouldn’t be no archive gaps.

SQL>
select current_scn from v$database;
select * from v$archive_gap;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

14) Do a couple of log switches on the primary and verify from the alert log that the standby is catching up with the primary.

15) Verify the standby redologs and other settings as required depending on the configuration and make sure the standby is in good shape.

Upto Oracle version 11g, RMAN ‘DUPLICATE’ command required [...]]]> In this post, let me take you through the step-by-step method of active database duplication, a new feature of Oracle 11g to create a duplicate database. Active database duplication can also be used to create a standby database, which is mentioned separately in another post.

Basics

Upto Oracle version 11g, RMAN ‘DUPLICATE’ command required a database backup for performing database duplication/cloning. This is called backup based database duplication. Starting from Oracle 11g, an active database can be cloned to the same host or another without using an RMAN backup and the recovery catalog and this is called active database duplication.This can also be used to create a physical standby database. RMAN copies the live database files over the network to create the duplicate/standby database.

Prerequisites

* The source database must be in ARCHIVELOG mode if it is open.
* If the source database is in NOARCHIVELOG mode, it should be mounted after a clean shut down.
* Connections to the auxiliary instance must be through a Net service name.
* The auxiliary instance must have a password file created, the password for the SYSDBA user used for the activity must be the same.

Practical issues and solutions

When the source database server and the target database server follow the same filesystem structure,size and standards, the duplication works fine without any issues. This may not be the case in most of the environments. The source database will have files laid across the mountpoints (say) ‘/oracle/oradata1/PROD’, ‘/oracle/oradata2/PROD’ etc. The target server may follow a totally different structure. In such cases, perform any of the following methods:

1) Use the ‘SET NEWNAME FOR DATAFILE’ and ‘SET NEW NAME FOR TEMPFILE’ in the run {} block while performing restore/duplication. When specified,  the ‘SET NEWNAME’  command will take precedence over the other methods of renaming the database files.

If the destination server follows a different filesystem layout and if the files needs to be restored to a separate set of directories different than that from the source database, ‘SET NEWNAME’ command needs to be used for each datafile and tempfile (‘SET NEW NAME FOR TEMPFILE’ command is introduced in Oracle 11g)

Note:

If ‘SET NEWNAME’ commands are used in a ‘run {}’  block while performing a restore operation, the ‘run {}’ block must contain the command ‘SWITCH DATAFILE ALL;’  and ‘SWITCH TEMPFILE ALL;’  (SWITCH TEMPFILE ALL; is available starting from Oracle 11g). If the ‘SWITCH; command is missed out during a restore operation,  ‘ALTER DATABASE RENAME FILE’  command must be used for each datafile before proceeding with the recovery. The details required to perform the manual execution of ‘ALTER DATABASE RENAME FILE’ can be found from the alert log. It is a tedious task to do so, so NEVER forget to add the SWITCH commands in the run block  if you are performing an RMAN restore. For database duplication using DUPLICATE command, ‘SWITCH DATAFILE/TEMPFILE’ commands are not required as this function is handled by RMAN automatically.

2) DB_FILE_NAME_CONVERT clause in the DUPLICATE statement. This method will have the precedence if the DB_FILE_NAME_CONVERT / DB_LOG_FILE_NAME_CONVERT parameters are used in the Init.ora file.

3) DB_FILE_NAME_CONVERT and DB_LOG_FILE_NAME_CONVERT parameters in Init.ora.

4) In case Oracle managed files are used in the source database, either you can specify ‘SET NEWNAME’ commands for each datafile and/or specify the parameters DB_FILE_CREATE_DEST and DB_CREATE_ONLINE_LOG_DEST_n in the auxiliary instance to get the datafiles, temp files and redo logs created in the respective directories.

Step-by-step

Terms used to describe the method step by step:

Target: Database to be cloned. (Source database – ORCL instance)
Auxiliary: Destination database to which the source database will be cloned (ORCLDUP instance)

This scenario explains that the source and target database instances are on a Windows server. The method is the same for database duplication across 2 different servers (on Unix also) except that some of the OS commands mentioned below may need to be changed.

1) Prepare Init.ora file for the auxiliary instance

Copy the Init.ora file from the source server to the target server, to the dbs/database directory based on the underlying platform. Make necessary modifications in the Init.ora file, mainly the below parameters:

*.db_name='ORCLDUP'
*.control_files='C:\ORACLE\ORADATA\ORCLDUP\control01.ctl,
C:\ORACLE\ORADATA\ORCLDUP\control02.ctl, C:\ORACLE\ORADATA\ORCLDUP\control03.ctl'

2) Create Password file

Copy the password file to the target server, rename it in the below format, this is the easiest method; you can always create the same with orapwd utility.

Unix Servers: orapw<SID><sid>.ora
Windows Servers: PWD<SID><sid>.ora

3) Create Oracle related services (Required for Oracle on Windows only)

Use Oradim utility to create an Oracle service on Windows servers., this step is not applicable for Unix servers.

oradim -new -sid orcldup -srvc oracleserviceorcldup

4) Create directories for database files

Create the required directories on the target server for datafiles, redo logs, control files, temporary files etc, this example assumes that all the database files will be stored under ‘C:ORACLEORADATAORCLDUP’

5) Configure listener to user static service registration

Static service registration for the listener must be done on the target server, this is because the database instance in NOMOUNT state (auxiliary instance before cloning) will not allow database connections, it will be BLOCKED. To configure static service registration with the listener, update the listener.ora file as below. Please note that, in this example the target and auxiliary instances are on the same server.

SID_LIST_LISTENER =
(SID_LIST = (SID_DESC =
(ORACLE_HOME = C:\oracle\product\11.1.0\db_1)
(SID_NAME = ORCL))
(SID_DESC = (ORACLE_HOME = C:oracle\product\11.1.0\db_1)
(SID_NAME = ORCLDUP))
)

6) Reload the listener

lsnrctl reload;

7) Start and verify the auxiliary instance

Start the auxiliary instance; verify the parameters db_name and control_files.

SQL> show parameter db_name;
SQL> show parameter control_files;

8) Perform Oracle 11g Active database duplication

RMAN>
run {
SET NEWNAME FOR DATAFILE 1 TO 'C:\ORACLE\ORADATA\ORCLDUP\SYSTEM01.DBF';
SET NEWNAME FOR DATAFILE 2 TO 'C:\ORACLE\ORADATA\ORCLDUP\SYSAUX01.DBF';
SET NEWNAME FOR DATAFILE 3 TO 'C:\ORACLE\ORADATA\ORCLDUP\UNDOTBS01.DBF';
SET NEWNAME FOR DATAFILE 4 TO 'C:\ORACLE\ORADATA\ORCLDUP\USERS01.DBF';
SET NEWNAME FOR DATAFILE 5 TO 'C:\ORACLE\ORADATA\ORCLDUP\EXAMPLE01.DBF';
SET NEWNAME FOR TEMPFILE 1 TO 'C:\ORACLE\ORADATA\ORCLDUP\TEMP01.DBF';
duplicate target database to orcldup from active database;
}

9) Logs

Oracle 11g Active database duplication Log

10) References

Oracle documentation on RMAN database duplication