Monday, March 29, 2010

ORA-19511 Vendor specific error: Cannot find requested element ERR(-2)

Applies to:
Oracle Server Enterprise Edition - Version: 10.2 to 10.2
This problem can occur on any platform.
Symptoms
Backup returns error:
RMAN-03009: failure of backup command on sbt_backup channel at 02/23/2009 13:28:40
ORA-19506: failed to create sequential file, name="c-3183778111-20090219-02", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
Vendor specific error: Cannot find requested element ERR(-2)
ORA-19600: input file is backup piece
(+ASM_FRA/ractest/autobackup/2009_02_19/s_679226354.411.679226355)
ORA-19601: output file is backup piece (c-3183778111-20090219-02)
Cause
HP data protector media manager issue. The HP data protector software causes the error. There
is a delay between backupsets which are send to the HP cell server. If the delay is to long the hp
cell server closes the session. The client did not recognise the close and fails with the transfer
of the data.

Cause justified by customer.
Solution
To implement the solution, please execute the following steps:

A solution for me was to set in the configuration file:

On Unix Plattform: /etc/opt/omni/server/options/global
On Windows Plattform: C:\Programme\OmniBack\Config\Server\Options\global

the parameter SmWaitForNewBackupClient to 2 minutes:

# SmWaitForNewBackupClient=WaitForInMinutes
SmWaitForNewBackupClient=2

There are further hints about the SmWaitForNewBackupClientSec parameter. Maybe this parameter also
helps.
# SmWaitForNewBackupClientSec=WaitForInSeconds

Reference

Metalink Doc [ID 785219.1]


Reinstating a Physical Standby Using Backups Instead of Flashback

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.2.0 to 10.2.0.3
Information in this document applies to any platform.
Goal

After a Data Guard failover to a physical standby database, using flashback database is still the MAA recommendation to reinstate the original primary database since it is the simplest and fastest approach. However, if for some reason the flashback data is not available or flashback database was not enabled before the failure, you can use the steps provided in this document to reinstate the original primary database using existing backups instead of flashback database.
Solution

Oracle highly recommends utilizing flashback database to reinstate a physical standby after a Data Guard Failover. The procedure is simpler and safer. When using a controlfile restored from a backup instead of flashing back with the current controlfile, the backup controlfile may not reflect the same files as the data dictionary. However this can be avoided if a standby controlfile is created after each backup.

Prerequisites:
• Backups of all data files exist.
• Standby controlfile backups exist and were taken at the failed primary database using the ALTER DATABASE CREATE STANDBY CONTROLFLE command after each database backup.
• Database version is 10.2.0.2 or higher. Please refer to Alert Note 308698.1 and Note.368276.1.

Step 1: Retrieve STANDBY_BECAME_PRIMARY_SCN after Data Guard Failover
From the new primary:


SQL> select to_char(STANDBY_BECAME_PRIMARY_SCN) from v$database;

Step 2: Remove divergent archivelogs from the original, failed primary database

Remove any archivelogs created at the time of or after the Failover operation. The failed primary, if it was isolated from the standby, could have divergent archive logs not consistent with the current primary database. To ensure you do not ever apply these divergent archive logs, they must be deleted from your backups and flash recovery area. You can use the following RMAN command to delete the relevant archive logs from the flash recovery area.

RMAN> DELETE ARCHIVELOG FROM SCN STANDBY_BECAME_PRIMARY_SCN;

Step 3: Restore backup of the original, failed primary database

RMAN> RESTORE DATABASE;

Step 4: Startup standby database using the physical standby controlfile

If the standby controlfile backup needs to be restored, you will need to ensure the spfile or initialization files point to the standby control file and remount the new standby database.

SQL> startup mount

If a standby controlfile does not exist from the failed primary database, then follow steps 1 and 2.2 through 2.8 in Note 368276.1 to:

1. Clean up the flash recovery area
2. Create a new standby controlfile from the new primary database
3. Restart the new standby database with the new standby controlfile
4. Rename data files and log files

Step 5: Ship a new archive from the primary to the new physical standby database

SQL> alter system set log_archive_dest_state_n = ENABLE;
SQL> alter system archive log current;

From the primary, you can verify if the standby received the archive by querying:

SQL> select dest_id, dest_name, status, type, error, destination
2 > from v$archive_dest_status
3 > where dest_id = ;

Step 6: Restart managed recovery

SQL> recover managed standby database using current logfile through all switchover disconnect;

Step 7: Verify the new standby’s data files and control file are in sync

Once the standby is in sync with the primary, you can do a quick validation check to ensure all the data files are present. The following steps stop managed recovery and open the physical standby read only. The query checks if any data files found in the data dictionary do not match exactly the data files listed in the control file.

SQL> recover managed standby database cancel;
SQL> alter database open;
SQL> select file#, CRSCNBAS from file$
2> MINUS
3> select file#, creation_change# from v$datafile;

If no rows are returned, the data dictionary and controlfile metadata are in sync. Restart managed recovery as described in step 6.

References

Metalink Doc ID: [ID 416310.1]

NOTE:308698.1 - ALERT: Reinstating physical standby databases after a Data Guard Failover
NOTE:368276.1 - Steps to workaround issue described in Alert 308698.1
NOTE:416314.1 - Reinstating a Logical Standby Using Backups Instead of Flashback Database

How To Use RMAN For Backup And Restore In A Standby Database Environment

RMAN can back up the standby database and its associated archived redo logs.
Standby backups of datafiles and archived redo logs are fully interchangeable
with primary database backups. In other words, you can run the RESTORE command
to restore a backup of a standby datafile to the primary database, and you can
restore a backup of a primary datafile to the standby database. The standby
control file and primary control file, however, are not interchangeable.

Prerequisites:

- The standby database is created and fully functioning.
- An Rman catalog is established. Use the same Rman catalog for the primary
database and the standby database. Do not create a separate Rman catalog for
the standby database.
- Do not register the standby database in the Rman catalog, only the primary
database is to be registered.

To take backup on the standby database:

- The standby database should preferably be in 'mount standby' mode, not in
'recover managed standby' mode. On ver. 9.0.1 the subsequent redo log
applications fail with ORA-01157 ORA-01110 if the database was in 'recover
managed standby' mode during backup, after which "alter database recover
managed standby database;" must be reissued.
- The standby database is the target database
- Connect to the Rman catalog
- Run backup
- On ver. 8.1.7, run 'resync catalog' after the backup
- Check with 'list backupset of database' that the new backupset is stored in
the Rman catalog

On ver. 8.1.7 it is necessary to manually resync the catalog in order to get
the backupset taken on the standby database stored in the Rman catalog. The
resync will do a partial resynchronization, this is updating the catalog with
the changed information in the control file.

To run restore on the standby database:

- The standby database is in 'mount standby' mode (not in 'recover managed
standby' mode)
- The standby database is the target database
- On ver. 8.1.7, connect nocatalog to Rman; do not connect to the Rman
catalog as the restore will be run against the primary database if connected to
the Rman catalog
- On rel. 9i, connecting to the Rman catalog as well as connecting nocatalog
works
- Run restore

To run restore on the primary database:

- The primary database is the target database
- Connect to the Rman catalog
- Check with 'list backupset of database' that the backupset taken on the
standby database is accessible
- Run restore

Reference:

Metalink Doc ID: 203980.1

For Oracle ver. 8.1.7:

For Oracle ver. 9.0.1:

For Oracle ver. 9.2.0.1:

Creating a Data Guard Database with RMAN (Recovery Manager) using Duplicate Command

PURPOSE

To understand and use the 9i Data Guard database features of Recovery manager

1. Summary of test environment:

The production database is "PROD".
The standby database is "STANDBY"
Testing was done on SUN hardware and SUN Solaris 2.8


2. Standby Restrictions:

Note the following restrictions involved when using the DUPLICATE
command to create a standby database:

-- The standby instance must be started but not mounted.

-- RMAN must be connected to the target database and to the auxiliary
instance. If desired, you can connect to the recovery catalog.

-- At least one auxiliary instance channel must be allocated in the RUN
block. You can allocate multiple auxiliary channels if needed.

-- Backups and copies on disk are available at the standby host with the
same path names as in the target host.

-- Backups on tape are accessible from the standby host.

-- Backups on disk are available at the standby host with the same
pathnames as in the target host.

-- If archived logs have not been backed up, then archived logs must be
available at the standby host with the same path names as in the target
host.

-- If RMAN recovers the standby database, then the checkpoint SCN of the
control file must be included in an archived redo log that is either
available at the standby site or included in an RMAN backup. For example,
assume that you create the standby control file and then immediately
afterwards archive the current log, which has a sequence of 100. You must
recover the standby database up to at least log sequence 100, or Oracle
signals ORA-1152 because the standby control file backup or copy was
taken after the point in time.

-- You cannot specify the SKIP READONLY and LOGFILE options of the
DUPLICATE command. These options are legal for a duplicate database but
illegal for the creation of the standby database.

-- You cannot use SET NEWNAME or SET AUXNAME to transform the filenames for
the online redo logs on the standby database. You cannot use the DUPLICATE
command to activate a standby database.


3. RMAN Standby database commands:

CURRENT CONTROLFILE FOR STANDBY -

makes a backup of the current control file that can be used with a standby
database. A standby control file can also be used as an ordinary control
file backup, so you can restore it in the target database if necessary.

DUPLICATE FOR STANDBY -

creates a standby database rather than a duplicate database. Specify this
keyword only when creating a standby database.

DORECOVER -

specifies that RMAN should recover the database after creating it. If you
specify an untilClause, then RMAN recovers to the specified point and
opens the database. If you do not specify this keyword, then RMAN
creates the standby database and then leaves it mounted.


4. Primary INIT.ORA setup:

db_name = PROD
instance_name = PROD
fal_client = PROD
fal_server= STANDBY
log_archive_dest_1 = 'LOCATION=/p01/app/oracle/product/9.0.1/dbs/arch'
log_archive_dest_2 = 'SERVICE=STANDBY LGWR ASYNC NOAFFIRM'
log_archive_format = %t_%s.dbf
log_archive_start = true
remote_login_passwordfile = EXCLUSIVE
service_names = PROD

Additional information on these parameters can be found in the Oracle 9i
Reference Manual Available at http://tahiti.oracle.com.


5. Standby INIT.ORA setup:

# Copy the init.ora from production to standby.
# Edit the initSTANDBY.ora to make neccessary changes:

db_name = "PROD" -- Must match the production database.

instance_name = STANDBY -- Instance identifier

lock_name_space = STANDBY -- Used when the standby or clone have the same
name as the production database being copied.

service_names = STANDBY -- Specifies the service names supported by the
instance.

fal_client = STANDBY -- Specifies the service name resolved from the
remote host to fetch archive logs to local.

fal_server = PROD -- Specifies the service name resolved on the
local host to request archive logs from remote.

db_file_name_convert = ('/p01/oradata/PROD',
'/u03/STANDBY/oradata')
log_file_name_convert = ('/p01/oradata/PROD',
'/u03/STANDBY/oradata')

# Use these parameters if you have a different filesystem structure on the
# standby. Then edit all the dump_dest and controlfile locations for the
# standby site.

standby_archive_dest = /p01/app/oracle/product/9.0.1/dbs/arch
standby_file_management = auto

REMOTE_LOGIN_PASSWORDFILE = EXCLUSIVE


- Create a password file for the standby database using orapwd command:

Create a password file 'orapwSTANDBY' file in the $ORACLE_HOME/dbs
directory and the password for internal "pwd".

orapwd file=$ORACLE_HOME/dbs/orapwSTANDBY password=pwd entries=5


6. Making a backup of the target for standby:

Connect to the target and the recovery catalog using Recovery manager:

setenv ORACLE_SID=PROD


rman nocatalog target /

Recovery Manager: Release 9.0.1.1.0 - 64bit Production

(c) Copyright 2001 Oracle Corporation. All rights reserved.

connected to target database: PROD (DBID=3261937922)
using target database controlfile instead of recovery catalog


Configure channels if you have not already. See the following Note for more
information:

Note 145178.1
RMAN 9i: Automatic Channel Allocation and Configuration


RMAN> Backup Database;

Starting backup at 09-APR-02
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=7 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current controlfile in backupset
input datafile fno=00001 name=/p01/oradata/PROD/system01.dbf
input datafile fno=00002 name=/p01/oradata/PROD/undotbs01.dbf
input datafile fno=00005 name=/p01/oradata/PROD/example01.dbf
input datafile fno=00006 name=/p01/oradata/PROD/indx01.dbf
input datafile fno=00008 name=/p01/oradata/PROD/users01.dbf
input datafile fno=00003 name=/p01/oradata/PROD/cwmlite01.dbf
input datafile fno=00004 name=/p01/oradata/PROD/drsys01.dbf
input datafile fno=00007 name=/p01/oradata/PROD/tools01.dbf
channel ORA_DISK_1: starting piece 1 at 09-APR-02
channel ORA_DISK_1: finished piece 1 at 09-APR-02
piece handle=/p01/oradata/PROD/dbs/04dlfl6n_1_1 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:24:25
Finished backup at 09-APR-02


RMAN> Backup current controlfile for standby;

Starting backup at 09-APR-02
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including standby controlfile in backupset
channel ORA_DISK_1: starting piece 1 at 09-APR-02
channel ORA_DISK_1: finished piece 1 at 09-APR-02
piece handle=/p01/oradata/PROD/dbs/05dlfml6_1_1 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:22
Finished backup at 09-APR-02


RMAN> sql "Alter System Archive Log Current";

sql statement: Alter System Archive Log Current


RMAN> Backup filesperset 10 ArchiveLog all delete input;

Starting backup at 09-APR-02
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archive log backupset
channel ORA_DISK_1: specifying archive log(s) in backup set
input archive log thread=1 sequence=3 recid=3 stamp=458742477
input archive log thread=1 sequence=4 recid=4 stamp=458742492
channel ORA_DISK_1: starting piece 1 at 09-APR-02
channel ORA_DISK_1: finished piece 1 at 09-APR-02
piece handle=/p01/oradata/PROD/dbs/06dlfmmt_1_1 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
channel ORA_DISK_1: deleting archive log(s)
archive log filename=/p01/oradata/PROD/dbs/arch1_3.dbf recid=3 stamp=458742477
archive log filename=/p01/oradata/PROD/dbs/arch1_4.dbf recid=4 stamp=458742492
Finished backup at 09-APR-02


RMAN> exit;

Recovery Manager complete.


7. Restoring the database to the standby location:

Copy the backup pieces to the remote server (same directory).

rman target / auxiliary sys/change_on_install@STANDBY

Recovery Manager: Release 9.0.1.1.0 - 64bit Production

(c) Copyright 2001 Oracle Corporation. All rights reserved.

connected to target database: PROD (DBID=3261937922)
connected to auxiliary database: PROD (not mounted)


RMAN> duplicate target database for standby dorecover;

Starting Duplicate Db at 09-APR-02
using target database controlfile instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=17 devtype=DISK

printing stored script: Memory Script
{
restore clone standby controlfile to clone_cf;
replicate clone controlfile from clone_cf;
sql clone 'alter database mount standby database';
}
executing script: Memory Script

Starting restore at 09-APR-02

using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: restoring controlfile
output filename=/u03/STANDBY/oradata/control01.ctl
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/p01/oradata/PROD/dbs/05dlfml6_1_1 tag=null params=NULL
channel ORA_AUX_DISK_1: restore complete
Finished restore at 09-APR-02

replicating controlfile
input filename=/u03/STANDBY/oradata/control01.ctl
output filename=/u03/STANDBY/oradata/control02.ctl

sql statement: alter database mount standby database

printing stored script: Memory Script
{
set until scn 266683;
set newname for datafile 1 to
"/u03/STANDBY/oradata/system01.dbf";
set newname for datafile 2 to
"/u03/STANDBY/oradata/undotbs01.dbf";
set newname for datafile 3 to
"/u03/STANDBY/oradata/cwmlite01.dbf";
set newname for datafile 4 to
"/u03/STANDBY/oradata/drsys01.dbf";
set newname for datafile 5 to
"/u03/STANDBY/oradata/example01.dbf";
set newname for datafile 6 to
"/u03/STANDBY/oradata/indx01.dbf";
set newname for datafile 7 to
"/u03/STANDBY/oradata/tools01.dbf";
set newname for datafile 8 to
"/u03/STANDBY/oradata/users01.dbf";
restore
check readonly
clone database
;
}
executing script: Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 09-APR-02

using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /u03/STANDBY/oradata/system01.dbf
restoring datafile 00002 to /u03/STANDBY/oradata/undotbs01.dbf
restoring datafile 00003 to /u03/STANDBY/oradata/cwmlite01.dbf
restoring datafile 00004 to /u03/STANDBY/oradata/drsys01.dbf
restoring datafile 00005 to /u03/STANDBY/oradata/example01.dbf
restoring datafile 00006 to /u03/STANDBY/oradata/indx01.dbf
restoring datafile 00007 to /u03/STANDBY/oradata/tools01.dbf
restoring datafile 00008 to /u03/STANDBY/oradata/users01.dbf
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/p01/oradata/PROD/dbs/04dlfl6n_1_1 tag=null params=NULL
channel ORA_AUX_DISK_1: restore complete
Finished restore at 09-APR-02

printing stored script: Memory Script
{
switch clone datafile all;
}
executing script: Memory Script


printing stored script: Memory Script
{
set until scn 266683;
recover
standby
clone database
;
}
executing script: Memory Script

executing command: SET until clause

Starting recover at 09-APR-02
using channel ORA_AUX_DISK_1

starting media recovery

channel ORA_AUX_DISK_1: starting archive log restore to default destination
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=3
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=4
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/p01/oradata/PROD/06dlfmmt_1_1 tag=null params=NULL
channel ORA_AUX_DISK_1: restore complete
archive log filename=/u03/STANDBY/oradata/arch/1_3.dbf thread=1 sequence=3
archive log filename=/u03/STANDBY/oradata/arch/1_4.dbf thread=1 sequence=4
media recovery complete
Finished recover at 09-APR-02
Finished Duplicate Db at 09-APR-02

RMAN> exit;


8. Put the Standby in Managed Recovery Mode:

On the Standby:

SQL> recover standby database;
# Recover all available logs

SQL> alter database recover managed standby database disconnect;


9. References:

Recovery Manager and Data Guard Documentation at http://tahiti.oracle.com
Note 118409.1 Creating a Standby Database using RMAN (Recovery Manager)
Note 145178.1 RMAN 9i Automatic Channel Allocation and Configuration

Note.149286.1 How To Create a Standby Database in Oracle8i

Maintenance Of Archivelogs On Standby Databases

Applies to:
Oracle Server - Enterprise Edition - Version: 10.1.0.0 to 11.1
Information in this document applies to any platform.

Goal
Maintenance of archivelogs which are already applied on standby database in Oracle 10g

Solution
In Oracle 10g, RMAN has a specific configuration for archivelog deletion policy i.e.
ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY
This parameter can be set on primary and standby databases.  
To enable automatic maintenance of the archivelogs on standby perform the below steps on standby database :
Step 1 :  Enable the flash recovery area on the standby database so that all the archivelogs are shipped in the flash recovery area.
Example : 
To enable Flash Recovery Area and to allow the archivelogs to be shipped to Flash Recovery Area set the below parameters
SQL> alter system set DB_RECOVERY_FILE_DEST='/disk1/FRA';
SQL> alter system set DB_RECOVERY_FILE_DEST_SIZE=80G;
SQL> alter system set  LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST';

Step 2 : Set the parameter Archive log deletion policy to applied on standby
Example :
RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

Note : With the above settings whenever there is a space pressure in Flash Recovery Area all the archivelogs that are applied on standby database will be automatically deleted.

References
Oracle Metalink Doc ID 464668.1