Ocr and voting disks Administration

Voting disk:
As far as voting disks are concerned, a node must be able to access strictly more than half of the voting disks at any time. So if you want to be able to tolerate a failure of n
 voting disks, you must have at least 2n+1 configured. (n=1 means 3 voting disks). Oracle Clusterware supports a maximum of 32 voting disks.It is recommended to use an odd number as 4 disks will not be any more highly available than 3 disks, 1/2 of 3 is 1.5...rounded to 2, 1/2 of 4 is 2, once we lose 2 disks, our
cluster will fail with both 4 voting disks or 3 voting disks.

RMAN

1. checksyntax

>rman checksyntax

Recovery Manager: Release 10.2.0.5.0 - Production on Fri Aug 31 02:31:28 2012
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
RMAN> @hot_d2d.rcv
RMAN> #
2> #   File:         NLPR1.rcv
3> #   Description:  RMAN script to backup the NLPR1 database
4> #   Author:       Paul Gray , I.B.M.
5> #
6> #   Modification History
7> #   --------------------
8> #   P. Gray           01/09/03  Created
9> #   N. Sampson        20/07/05  Amended for R&D
10> #########################################################################
11>
12> run {
13> allocate channel dsk1 type disk format '${DESTINATION}/%d_inc0_%U.rbck' ;
14> backup incremental level 0 database ;
15> release channel dsk1;
16> }
The cmdfile has no syntax errors
RMAN>
RMAN> sql 'alter system archive log current';
The cmdfile has no syntax errors
RMAN>
RMAN> change archivelog all crosscheck;
The cmdfile has no syntax errors
RMAN>
RMAN> run {
2> allocate channel arch1 type disk format '${DESTINATION}/%d_arch_%U.rbck';
3> backup archivelog all ;
4> backup archivelog all archivelog until time 'sysdate-2' delete input;
5> release channel arch1;
6> }
The cmdfile has no syntax errors
RMAN> exit
Recovery Manager complete.

Local inventory vs Global inventory

oraInventory 

The inventory is a very important part of the Oracle Universal Installer. oraInventory is repository (directory) which store/records oracle software products & their oracle_homes location on a machine.

 Binary OraInventory

Before OUI 2.X the inventory was binary, the binary orainvenory maintains in inventory in binary format

XML Inventory


Starting from OUI 2.X and higher information in the inventory is stored in the Extensible Markup Language (XML) format
The XML format allows easier diagnostic of the problem and faster loading of data.
There are two inventories with the newer releases of OUI (2.x and higher): 


1.Local Inventory 
 There is one Local inventory per ORACLE_HOME.
Inventory inside each Oracle Home is called as local Inventory or oracle_home Inventory. This Inventory holds information to that oracle_home only.
2.Global Inventory
The central inventory directory outside the ORACLE_HOME (Global Inventory)
 

 This inventory stores information about.

• All the Oracle software products installed on all ORACLE_HOMES on a machine,These products can be various oracle components like database, oracle application server, collaboration suite, soa suite, forms & reports or discoverer server.
• Other non-oracle products such as Java Runtime env's (JRE)

 At startup, the Oracle Universal Installer first looks for the key that specifies where the global inventory is located at (this key varies by platform).
/var/opt/oracle/oraInst.loc (typical)
/etc/oraInst.loc (AIX and Linux)
HKEY_LOCAL_MACHINE -> Software -> Oracle -> INST_LOC (Windows platforms)

If this key is found, the directory within it will be used as the global inventory location.
If the key is not found, the inventory path will default created as follows:

UNIX : ORACLE_BASE\oraInventory
WINDOWS : c:\program files\oracle\Inventory

If the ORACLE_BASE environment variable is not defined, the inventory is created at the same level as the first Oracle home. For
example, if your first Oracle home is at /private/ORACLEHome1, then, the inventory is at /private/oraInventory.

If you want to see list of oracle products on machine check for file inventory.xml under ContentsXML in oraInventory
multiple Global Inventory on a machine
YES you can have multiple global Inventory   
 Global Inventory is corrupted ?
No need to worry if your global Inventory is corrupted, you can recreate global Inventory on machine using Universal Installer and attach already Installed oracle home by option
-attachHome
./runInstaller -silent -attachHome -invPtrLoc $location_to_oraInst.loc
ORACLE_HOME="Oracle_Home_Location" ORACLE_HOME_NAME="Oracle_Home_Name"
CLUSTER_NODES="{}"
 

New finding in architecture

Just now i find out very interesting concept in Architecture.

When instance starts, all background process related to read/write with database (DBWR & LGWR) will read redolog files and data files while opening the database. actually this used to happen in oracle 6.0 in which there is no CKPT process and LGWR used to perform checkpoint at that time. after that even though oracle changed and introduced CKPT, still in 10g also LGWR reads datafiles while opening the database.

For more information read following link.
http://forums.oracle.com/forums/thread.jspa?messageID=2631627&#2631627

Recovery scenario during Loss of all controlfiles

LOSS of all controlfiles (Complete recovery and No need to open the database with resetlogs after recreating the controlfile manually)

Assumptions:

- We are tracing the backup controlfile as part of backup activity or we are having controlfile recreation script.

- We have lossed all the controlfiles including mirrored controlfiles also.

1. I have deleted all the controlfiles manually at OS level.

 ls -ltr *.ctl

-rw-r-----   1 oracle   dba         6602752 Oct 26 14:01 SID_control03.ctl

-rw-r-----   1 oracle   dba         6602752 Oct 26 14:01 SID_control02.ctl

-rw-r-----   1 oracle   dba         6602752 Oct 26 14:01 SID_control01.ctl

rm *.ctl

ls -ltr *.ctl

ls: 0653-341 The file *.ctl does not exist.

2. After deleting the controlfiles I have connected to the database and try to access the some information from controlfile views,  while viewing the information from v$controlfile_record_section view got the below error. Below are the details.

sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.2.0 - Production on Mon Oct 26 14:02:32 2009

Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production

With the Partitioning and Data Mining options

SQL> select * from V$CONTROLFILE_RECORD_SECTION;

select * from V$CONTROLFILE_RECORD_SECTION

              *

ERROR at line 1:

ORA-00210: cannot open the specified control file

ORA-00202: control file:

'/restore1/oracle/SID/data/SID_control01.ctl'

ORA-27041: unable to open file

IBM AIX RISC System/6000 Error: 2: No such file or directory

Additional information: 3

Note:

After above step I tried to reproduce the same error in different ways,after lossing all the controlfiles also, I'm able to do manual check point and swich logfile successfully. While accessing most of the v$ views I got the controlfile error. Below are the details.

SQL> commit;

Commit complete.

SQL> alter system switch logfile;

System altered.

SQL> alter system checkpoint;

System altered.

SQL> show parameter control

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_file_record_keep_time        integer     7

control_files                        string      /restore1/oracle/SID/data

                                                 /SID_control01.ctl, /rest

                                                 ore1/oracle/SID/data/ATOP

                                                 RD01_control02.ctl, /restore1/

                                                 oracle/SID/data/SID_

                                                 control03.ctl

SQL> archive log list

ORA-00210: cannot open the specified control file

ORA-00202: control file: '/restore1/oracle/SID/data/SID_control01.ctl'

ORA-27041: unable to open file

IBM AIX RISC System/6000 Error: 2: No such file or directory

Additional information: 3

SQL> select * from v$log;

select * from v$log

              *

ERROR at line 1:

ORA-00210: cannot open the specified control file

ORA-00202: control file:

'/restore1/oracle/SID/data/SID_control01.ctl'

ORA-27041: unable to open file

IBM AIX RISC System/6000 Error: 2: No such file or directory

Additional information: 3

SQL> select * from v$logfile;

select * from v$logfile

              *

ERROR at line 1:

ORA-00210: cannot open the specified control file

ORA-00202: control file:

'/restore1/oracle/SID/data/SID_control01.ctl'

ORA-27041: unable to open file

IBM AIX RISC System/6000 Error: 2: No such file or directory

Additional information: 3

SQL> alter system backup controlfile to trace;

alter system backup controlfile to trace

             *

ERROR at line 1:

ORA-02065: illegal option for ALTER SYSTEM

SQL> select * from v$database;

select * from v$database

              *

ERROR at line 1:

ORA-00210: cannot open the specified control file

ORA-00202: control file:

'/restore1/oracle/SID/data/SID_control01.ctl'

ORA-27041: unable to open file

IBM AIX RISC System/6000 Error: 2: No such file or directory

Additional information: 3

3. After that shut down database using shut abort option.

SQL> shut abort

ORACLE instance shut down.

4. Below is the control file recreation script make sure change the resetlogs option to noresetlogs

 (Because we are having all archives and on line redo logs for complete recovery).

cat cr8contrl.sql

CREATE CONTROLFILE REUSE DATABASE "SID" NORESETLOGS  NOARCHIVELOG

    MAXLOGFILES 16

    MAXLOGMEMBERS 3

    MAXDATAFILES 100

    MAXINSTANCES 8

    MAXLOGHISTORY 584

LOGFILE

  GROUP 1 (

    '/restore1/oracle/SID/data/SID_redo01a.log',

    '/restore1/oracle/SID/data/SID_redo01b.log'

  ) SIZE 10M,

  GROUP 2 (

    '/restore1/oracle/SID/data/SID_redo02a.log',

    '/restore1/oracle/SID/data/SID_redo02b.log'

  ) SIZE 10M

-- STANDBY LOGFILE

DATAFILE

  '/restore1/oracle/SID/data/SID_system01.dbf',

  '/restore1/oracle/SID/data/SID_undotbs01.dbf',

  '/restore1/oracle/SID/data/SID_sysaux01.dbf',

  '/restore1/oracle/SID/data/SID_users01.dbf',

  '/restore1/oracle/SID/data/SID_autodata01.dbf',

  '/restore1/oracle/SID/data/SID_autoindexes01.dbf'

CHARACTER SET US7ASCII

;

5. start the instance in nomount( this step is not required because startup nomount is part of controlfile recreation script).

  

SQL> startup nomount

ORACLE instance started.

Total System Global Area  838860800 bytes

Fixed Size                  2074928 bytes

Variable Size             503318224 bytes

Database Buffers          327155712 bytes

Redo Buffers                6311936 bytes

6. run the controlfile recreatin script as below.

SQL>@cr8contrl.sql

Control file created.

7. Try to open the database without recovery will get the error message like datafile 1 is inconsistent.

SQL>alter database open;

alter database open

*

ERROR at line 1:

ORA-01113: file 1 needs media recovery

ORA-01110: data file 1: '/restore1/oracle/SID/data/SID_system01.dbf'

8. Recovery the database without using backup controlfile option.

SQL>recover database;

Media recovery completed.

9. Open the database without resetlogs;

SQL> alter database open;

Database altered.

How to pin nodes in RAC

The following steps can be used to pin nodes in RAC.
Pinning a node means that the association of a node name with a node number is fixed. If a node is not pinned,node number may change if the lease expires while it is down. The lease of a pinned node never expires.
To find whether a node in a cluster is pinned,use the olsnodes command.
[grid@raclinux1 stage10g]$ olsnodes -t
raclinux1 Unpinned
raclinux2 Unpinned

Use the pin command to pin the nodes
[root@raclinux1 bin]# ./crsctl pin css -n raclinux1
CRS-4664: Node raclinux1 successfully pinned.
[root@raclinux1 bin]#

[root@raclinux2 bin]# ./crsctl pin css -n raclinux2
CRS-4664: Node raclinux2 successfully pinned.
[root@raclinux2 bin]#

[grid@raclinux1 stage10g]$ olsnodes -t
raclinux1 Pinned
raclinux2 Pinned

Loss of redolog group

Recovering after loss of members of the INACTIVE redo log group

You’re attempting to open your database and receive this message

ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo02.log’
ORA-00312: online log 2 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo02a.log’
The message indicates that two members of an online redo log group in your database have experienced a media failure.
To recover when you’ve lost all members of an inactive redo log group, perform the following steps

Verify that the log group status is INACTIVE.
 SQL> select group#,members,status from v$log;
GROUP#    MEMBERS STATUS
———- ———- —————-
1                      2             CURRENT
2                      2             INACTIVE
3                      2             ACTIVE

 Recreate the log group with the clear logfile command.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
(Here I manually deleted redolog files members of group 2)
SQL> startup
ORACLE instance started.
Total System Global Area  285212672 bytes
Fixed Size                  1218992 bytes
Variable Size              92276304 bytes
Database Buffers          188743680 bytes
Redo Buffers                2973696 bytes
Database mounted.

ORA-00313: open failed for members of log group 2 of thread 1

ORA-00312: online log 2 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo02.log’
ORA-00312: online log 2 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo02a.log’
If all members of an online redo log group are damaged, you won’t be able to open your database, oracle will allow you to only mount your database.
SQL> select status from v$instance;
STATUS
————
MOUNTED
SQL> select group#,status,archived,thread#,sequence# from v$log;
GROUP# STATUS           ARC    THREAD#  SEQUENCE#
———- —————- — ———- ———-
1           CURRENT          NO           1         11
3           INACTIVE         YES          1         10
2           INACTIVE         YES          1          9
If the status is INACTIVE  and ARCHIVED=YES,then this log group is no longer needed for crash recovery , therefore ,you can use the clear logfile command to recreate all members of a log group.
SQL> alter database clear logfile group 2;
Database altered.
 if STATUS = INACTIVE and ARCHIVED = NO

sql>	ALTER DATABASE CLEAR UNARCHIVED LOGFILE GROUP 2;

 Recovering After loss of members of the CURRENT redo log group

All of the members of a current online redo log group in your database have experience media failure.
You lose all members of a current online redo log group. Here are possible reasons,

• Perform an incomplete recovery up to the last good SCN.
• If flashback is enabled, flash your database back to the last good SCN.
• If you’re using Oracle data guard, fail over to your physical or logical standby database.

SQL> select group#,status,archived,thread#,sequence# from v$log;
GROUP# STATUS           ARC    THREAD#  SEQUENCE#
———- —————- — ———- ———-
1               CURRENT          NO           1         17
2                INACTIVE         YES          1         15
3                INACTIVE         YES          1         16
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area  285212672 bytes
Fixed Size                  1218992 bytes
Variable Size              92276304 bytes
Database Buffers          188743680 bytes
Redo Buffers                2973696 bytes
Database mounted.

ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo01.log’
ORA-00312: online log 1 thread 1:
‘/home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/redo01a.log’

SQL> select group#,status,archived,thread#,sequence# from v$log;

GROUP# STATUS           ARC    THREAD#  SEQUENCE#
———- —————- — ———- ———-
1                CURRENT          NO           1         17
3                INACTIVE         YES          1         16
2                INACTIVE         YES          1         15
SQL> alter system checkpoint;
alter system checkpoint
*
ERROR at line 1:
ORA-01109: database not open
First determine the last good SCN by querying the first_change# column from v$log, you’re missing only the current online redo logs, therefore you can perform an incomplete recovery up to but not including,
SQL> select group#,status,archived,thread#,sequence#,first_change# from v$log;
GROUP# STATUS           ARC    THREAD#  SEQUENCE# FIRST_CHANGE#
———- —————- — ———- ———- ————-
1             CURRENT          NO           1         17        574131
3             INACTIVE         YES          1         16        574125
2             INACTIVE         YES          1         15        574103
SQL>
In this case, you can restore and recover up to but not including, 574131
Operation :-
Losing  all members of your current online redo log group is arguably the worst thing taht can happen to your database. if you experience media failure with all members of the current online redo group, then you will lose any transactions contained in those logs. in this case, you will have to perform incomplete recovery before you can open your database.
[oracle@localhost root]$ rlwrap rman target sys/sys
Recovery Manager: Release 10.2.0.1.0 – Production on Mon May 31 10:51:10 2010
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
connected to target database: TEST (DBID=2015564632, not open)
RMAN> restore database until scn 574131;
Starting restore at 31-MAY-10
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=150 devtype=DISK
channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/system01.dbf
restoring datafile 00002 to /home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/undotbs01.dbf
restoring datafile 00003 to /home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/sysaux01.dbf
restoring datafile 00004 to /home/oracle/oracle/product/10.2.0/db_1/oradata/TEST/users01.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/backupset/2010_05_25/o1_mf_nnndf_TAG20100525T103307_5zpz9nft_.bkp
channel ORA_DISK_1: restored backup piece 1
piece handle=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/backupset/2010_05_25/o1_mf_nnndf_TAG20100525T103307_5zpz9nft_.bkp tag=TAG20100525T103307
channel ORA_DISK_1: restore complete, elapsed time: 00:01:37
Finished restore at 31-MAY-10
RMAN> recover database until scn 574131;
Starting recover at 31-MAY-10
using channel ORA_DISK_1
starting media recovery
archive log thread 1 sequence 7 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_25/o1_mf_1_7_5zpzh95q_.arc
archive log thread 1 sequence 8 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_8_606k8n7x_.arc
archive log thread 1 sequence 9 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_9_606lkvgy_.arc
archive log thread 1 sequence 10 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_10_606mfskg_.arc
archive log thread 1 sequence 11 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_11_606ndqvj_.arc
archive log thread 1 sequence 12 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_12_606szwxo_.arc
archive log thread 1 sequence 13 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_13_606t3zs8_.arc
archive log thread 1 sequence 14 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_14_606t777d_.arc
archive log thread 1 sequence 15 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_15_606t908r_.arc
archive log thread 1 sequence 16 is already on disk as file /home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_16_606t9jd0_.arc
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_25/o1_mf_1_7_5zpzh95q_.arc thread=1 sequence=7
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_8_606k8n7x_.arc thread=1 sequence=8
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_9_606lkvgy_.arc thread=1 sequence=9
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_10_606mfskg_.arc thread=1 sequence=10
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_11_606ndqvj_.arc thread=1 sequence=11
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_12_606szwxo_.arc thread=1 sequence=12
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_13_606t3zs8_.arc thread=1 sequence=13
archive log filename=/home/oracle/oracle/product/10.2.0/db_1/flash_recovery_area/TEST/archivelog/2010_05_31/o1_mf_1_14_606t777d_.arc thread=1 sequence=14
media recovery complete, elapsed time: 00:00:06
Finished recover at 31-MAY-10
RMAN> alter database open resetlogs;
database opened.
SQL> select status from v$instance;
STATUS
————
OPEN
SQL> select group#,members,status from v$log;
GROUP#    MEMBERS STATUS
———- ———- —————-
1                       2           INACTIVE
2                       2          CURRENT
3                       2          UNUSED

Recover from loss of active redo log file.
- shutdown the database

SHUTDOWN IMMEDIATE;

- restore the datafiles and proceed for an incomplete recovery until the missing log sequence
- mount the database

STARTUP MOUNT;

- check the status of the missing log file group from v$log view

select group#, sequence#, bytes, first_change#, to_char(first_time,’DD-MM-YY hh24:mi:ss’) tim, status from v$log;

- if STATUS = ACTIVE and ARCHIVED = NO

RECOVER DATABASE UNTIL TIME ‘yyyy-mm-dd:hh24:mi:ss’;

‘ss’ in the seconds should be 1 second less than the first_time of the missing redo log file. - open the database with resetlogs option

1	ALTER DATABASE OPEN RESETLOGS;

- take a solid backup