How to find high CPU using sessions in Oracle Database

If the problem relates to CPU bound applications then CPU information for each session can be examined to determine the culprits. The v$sesstat  view can be queried to find high cpu using sessions and then SQL can be listed.

1.Find the 'CPU used by this session' statistic.

SQL>SELECT name ,statistic# FROM v$statname WHERE  name LIKE '%CPU%session';

    NAME                                STATISTIC#
    ----------------------------------- ----------
    CPU used by this session                    14

2. Then determine which session is using most of the cpu.

SQL>SELECT * FROM v$sesstat WHERE statistic# = 14;

           SID STATISTIC#      VALUE
    ---------- ---------- ----------
             1         14          0
             2         14          0
             3         14          0
             4         14          0
             5         14          0
             6         14          0
             7         14          0
             8         14          0
             9         14          0
            10         14          0
            11         14          0
            12         14          0
            16         14       1930

3. Lookup details for the session which is using most of the cpu.

SQL>SELECT address ,SUBSTR(sql_text,1,20) Text, buffer_gets, executions,buffer_gets/executions AVG
    FROM   v$sqlarea a, v$session s
    WHERE  sid = 16
    AND    s.sql_address = a.address
    AND    executions > 0
    ORDER BY 5;

4. Use v$sqltext to extract the whole SQL text.

set pages 50000 lines 32767
col SPID for a10
col PROGRAM for a15
col OSUSER for a10
col ACTION for a10
col EVENT for a25
col SQL_TEXT for a25
col MACHINE for a10
col P1TEXT for a10
col P2TEXT for a10
col P3TEXT for a10
SELECT b.sid, b.serial#, a.spid, b.sql_id, b.program, b.osuser, b.machine, b.type,
b.event, b.action, b.p1text, b.p2text, b.p3text, b.state, c.sql_text,b.logon_time
FROM v$process a, v$session b, v$sqltext c WHERE a.addr=b.paddr
AND b.sql_hash_value = c.hash_value
AND b.sid = '&sid' ORDER BY a.spid, c.piece
/

5. Once the whole SQL statement has been identified it can be tuned.

Explain the queries and examine their access paths.
Autotrace is a useful tool for examining access paths.

Syntax:-

Explain plan for 'sql statement';

How to Move ASM DATABASE FILES from ONE DISKGROUP TO ANOTHER

Steps to Move ASM DATABASE FILES from ONE DISKGROUP TO ANOTHER

Since ASM files cannot be accessed through normal operating system interfaces, RMAN is the preferred means of copying ASM file.

The steps to moving a datafile from a diskgroup to another is as below, using RMAN.

1) Identify the datafile to be moved.
2) Identify the diskgroup on to which the datafile has to be moved.
3) Take the datafile offline.
4) Copy the datafile to new diskgroup using Either RMAN or DBMS_FILE_TRANSFER.
5) Rename the datafile to point to new location.
6) Recover the datafile.
7) Bring the datafile online.
8) Verify the new datafile locations.
9) Delete the datafile from its original location.

1) Identify the datafile to be moved.

SQL> SELECT FILE_NAME FROM DBA_DATA_FILES;
+ASMDISK2/orcl/datafile/users.256.565313879 <======= Move this to ASMDISK1.
+ASMDISK1/orcl/sysaux01.dbf
+ASMDISK1/orcl/undotbs01.dbf
+ASMDISK1/orcl/system01.dbf

2) Identify the diskgroup on to which the datafile has to be moved.

SQL> SELECT GROUP_NUMBER, NAME FROM V$ASM_DISKGROUP;
GROUP_NUMBER NAME
------------ ---------
1 ASMDISK1
2 ASMDISK2

3) Take the datafile offline.
SQL> ALTER DATABASE DATAFILE '+ASMDISK2/orcl/datafile/users.256.565313879' OFFLINE;

4) Copy the datafile to new diskgroup using Either RMAN or DBMS_FILE_TRANSFER.

   a)   DBMS_FILE_TRANSFER package or
   b)   RMAN
       
a).Using DBMS_FILE_TRANSFER package 
           
SQL> create or replace directory orcl1 as '+ASMDISK1/orcl/datafile';

SQL> Alter disgroup ASMDISK2 add directory  '+ASMDISK2/test';
       
SQL> create or replace directory orcl2 as '+ASMDISK2/test';

SQL>
BEGIN
     DBMS_FILE_TRANSFER.COPY_FILE(
     source_directory_object => 'ORCL1',
     source_file_name => 'users.259.565359071',
     destination_directory_object => 'ORCL2',
     destination_file_name => 'USERS01.DBF');
END;  
Database altered.

--------------------  OR   --------------------
  
b).Using RMAN copy the file to new diskgroup.

$ rman target /

connected to target database: ORCL (DBID=1020304050)

RMAN> COPY DATAFILE '+ASMDISK2/orcl/datafile/users.256.565313879' TO '+ASMDISK1';

Starting backup at 03-AUG-98
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=+ASMDISK2/orcl/datafile/users.256.565313879
output filename=+ASMDISK1/orcl/datafile/users.259.565359071 tag=TAG19980803T12110
9 recid=2 stamp=565359071
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 03-AUG-98

5) Rename the datafile to point to new location.

If you have used DBMS_FILE_TRANSFER (method 4 a)) use the following command to rename:
SQL> ALTER DATABASE RENAME FILE '+ASMDISK2/orcl/datafile/users.256.565313879' TO                                  

'+ASMDISK1/orcl/datafile/users.259.565359071';

Database altered.

If you have used RMAN (method 4 b) use the following option of RMAN
RMAN>
run
{
set newname for datafile '+ASMDISK2/orcl/datafile/users.256.565313879'
to '+ASMDISK1/orcl/datafile/users.259.565359071';
switch datafile all;
}

6) Recover the datafile.

SQL> RECOVER DATAFILE '+ASMDISK1/orcl/datafile/users.259.565359071';
Media recovery complete.

7) Bring the datafile online.

SQL> ALTER DATABASE DATAFILE '+ASMDISK1/orcl/datafile/users.259.565359071' ONLINE;
Database altered.

8) Verify the new datafile locations.

SQL> SELECT FILE_NAME FROM DBA_DATA_FILES;

FILE_NAME
-------------------------------------------------------------------------------
+ASMDISK1/orcl/datafile/users.259.565359071
+ASMDISK1/orcl/sysaux01.dbf
+ASMDISK1/orcl/undotbs01.dbf
+ASMDISK1/orcl/system01.dbf

9) Delete the datafile from its original location.

SQL> ALTER DISKGROUP ASMDISK2 DROP FILE users.256.565313879;
or
ASMCMD> rm -rf <filename>


Note:-
====

The steps provided above assume that the database is open and in Archivelog mode.
Besides these steps are not appropriated for system or sysaux datafiles.

For System and Sysaux an approach similar to the one given below can be used:-

1. Create a Copy of datafile in target Diskgroup
RMAN> backup as copy tablespace system format '<New DG>';
RMAN> backup as copy tablespace sysaux format '<New DG>';

2. Then shutdown the database and restart to a mounted state
RMAN> shutdown immediate;
RMAN> startup mount;

3. switch the datafiles to the copy
RMAN> switch tablespace system to copy;
RMAN> switch tablespace sysaux to copy;

4. Recover the changes made to these tablespaces
RMAN> recover database;

Note:-
====

Most ASM files do not need to be manually deleted because, as Oracle managed files, they are removed automatically

when they are no longer needed.

However, if you need to drop an Oracle Managed File (OMF) manually you should use the fully qualified filename if

you reference the file. Otherwise you will get an error (e.g. ORA-15177).

ALTER DISKGROUP ASMDISK2 DROP FILE '+ASMDISK2/orcl/datafile/users.256.565313879';

Patch 13621679 - 11.1.0.7.11 Patch Set Update

Patchset/PSU    Patch Number    Description
11.1.0.7.11    13621679    DATABASE PATCH SET UPDATE 11.1.0.7.11 (INCLUDES CPU APR2012)

Patch 13621679 - 11.1.0.7.11 Patch Set Update
=============================================
http://bbs.dbsupport.cn/thread-160-1-1.html

Patch InformationPatch Set Update (PSU) patches are cumulative. That is, the content of all previous PSUs is included in the latest PSU patch.

PSU 11.1.0.7.11 includes all fixes previously included in PSU 11.1.0.7.10 and those listed in .
To install the PSU 11.1.0.7.11 patch, the Oracle home must have the 11.1.0.7.0 Database installed. Subsequent PSU patches can be installed on Oracle Database 11.1.0.7.0 or any PSU with a lower 5th numeral version than the one being installed.

OPatch Utility
==============
You must use the OPatch utility version 11.1.0.8.2 or later to apply this patch. Oracle recommends that you use the latest released OPatch 11.1, which is available for download from My Oracle Support patch6880880 by selecting the 11.1.0.0.0 release.
For information about OPatch documentation, including any known issues, see My Oracle Support Note 293369.1 OPatch documentation list.

PSU Patching 11G Steps on PRIMARY database
==========================================

check DR sysnc status
=====================
http://select-star-from.blogspot.in/2013/09/data-guard-sync-status.html

------------------------------------------------------------------------------------------------------------------

Start OEM black out
===================

Confirmation
Blackout "Blackout-Apr 24 1998 9:33:42 PM" created successfully, Blackout status will be propagated to the target(s) shortly.

------------------------------------------------------------------------------------------------------------------

Node 1: (AS ORACLE)
===================

$ hostname
$ uname
$ bash
$ date
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs
 root  344170       1   0   Apr 24      -  0:00 /oradb/crs/product/11.1.0/crs_1/bin/oclskd.bin

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012
$ opatch version
$ /oradb/crs/product/11.1.0/crs_1/OPatch/opatch version
$ /oradb/asm/product/11.1.0/asm_1/OPatch/opatch version
$ /oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch version
$ which opatch
/oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch
. oraenv  --- asm instance
$ opatch lsinventory
. oraenv  --- database 1 instance
$ opatch lsinventory
. oraenv  --- database 2 instance
$ opatch lsinventory

$ ps -ef| grep pmon
$ srvctl status database -d <database_1>                       or srvctl status instance -d <database_1> -i <database_1_instance>
$ srvctl status database -d <database_2>                       or srvctl status instance -d <database_2> -i <database_2_instance>

$ srvctl status asm -n <Node 1>
$ srvctl status asm -n <Node 2>

$ srvctl status listener -n <Node 1>
$ srvctl status listener -n <Node 2>

$ srvctl status nodeapps -n <Node 1>
$ srvctl status nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ srvctl stop database -d <database_1>              or srvctl stop insatnce -d <database_1> -i <database_1_instance>
$ srvctl stop database -d <database_2> -o immediate or srvctl stop insatnce -d <database_2> -i <database_2_instance>

$ srvctl stop asm -n <Node 1>
$ srvctl stop asm -n <Node 2>

$ srvctl stop listener -n <Node 1>
$ srvctl stop listener -n <Node 2>

$ srvctl stop nodeapps -n <Node 1>
$ srvctl stop nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl stop agent   or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl stop agent

$ ps -ef| grep oracle
$ lsnrctl stop     -------------------------------to stop default listener
$ ps -ef| grep oracle

------------------------------------------------------------------------------------------------------------------

Node 2: (AS ORACLE)
===================

$ hostname
$ uname
$ bash
$ date
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs
 root  344170       1   0   Apr 24      -  0:00 /oradb/crs/product/11.1.0/crs_1/bin/oclskd.bin

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012

$ opatch version

$ /oradb/crs/product/11.1.0/crs_1/OPatch/opatch version
$ /oradb/asm/product/11.1.0/asm_1/OPatch/opatch version
$ /oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch version

$ which opatch
/oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch

$ ps -ef| grep pmon

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl stop agent   or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl stop agent

$ ps -ef| grep oracle
$ lsnrctl stop     -------------------------------to stop default listener
$ ps -ef| grep oracle

------------------------------------------------------------------------------------------------------------------

Pre-rootpatch
=============

Node 1: (AS ROOT)
=================

# cd /oradb/crs/product/11.1.0/crs_1/bin
# ./crsctl check cluster
# ./crsctl check crs
# ./crsctl stop crs

# ps -ef| grep oracle

# ssh <Node 2>

# cd /oradb/crs/product/11.1.0/crs_1/bin
# ./crsctl check cluster
# ./crsctl check crs
# ./crsctl stop crs

# ps -ef| grep oracle

# exit
# hostname
<Node 1>

# ps -ef| grep pmon   ---- (0 instances running)
# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
# sh custom/scripts/prerootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1 -crsuser oracle

# ssh <Node 2>

# ps -ef| grep pmon   ---- (0 instances running)
# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
# sh custom/scripts/prerootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1 -crsuser oracle

# exit
# hostname
<Node 1>

------------------------------------------------------------------------------------------------------------------

Pre-patch
=========

Node 1: (AS ORACLE)
===================

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/scripts/prepatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/server/11724953/custom/scripts/prepatch.sh -dbhome /oradb/asm/product/11.1.0/asm_1

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/server/11724953/custom/scripts/prepatch.sh -dbhome /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

Opatch Apply
============

Node 1: (AS ORACLE)
===================

CRS
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077
$ opatch napply -oh /oradb/crs/product/11.1.0/crs_1 -id 11724953

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server
$ opatch napply custom/server/ -oh /oradb/asm/product/11.1.0/asm_1 -id 11724953

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server
$ opatch napply custom/server/ -oh /oradb/app/oracle/product/11.1.0/db_1 -id 11724953

------------------------------------------------------------------------------------------------------------------

Do the post configure for CRS and RDBMS home
============================================

Node 1: (AS ORACLE)
===================

CRS
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts
$ ./postpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server/11724953
$ ./custom/scripts/postpatch.sh -dbhome /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server/11724953
$ ./custom/scripts/postpatch.sh -dbhome /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

11.1.0.7.11 PSU (13621679) and (9734685)
========================================

Node 1: (AS ORACLE)
===================

13621679
========

$ cd /oradb/app/admin/orasw/patches
$ chmod -R 777 PSU_patches
$ cd PSU_patches
$ pwd
/oradb/app/admin/orasw/patches/PSU_patches

$ ls -lrt p13621679_111070_AIX5L.zip
$ unzip p13621679_111070_AIX5L.zip
$ ls -lrt
$ cd 13621679
$ pwd

ASM
===
$ /oradb/app/admin/orasw/patches/PSU_patches/13621679
$ opatch apply -oh /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/13621679
$ opatch apply -oh /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

9734685
=======

$ cd /oradb/app/admin/orasw/patches/PSU_patches/
$ ls -lrt p9734685_1110711_AIX64-5L.zip
$ unzip p9734685_1110711_AIX64-5L.zip

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/9734685
$ opatch apply -oh /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/9734685
$ opatch apply -oh /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

After 9734685 Patch applied relink has to be done in both ASM and DB homes ------- Linking Oracle
==========================================================================

Node 1: (AS ORACLE)
===================

RDBMS
=====
$ cd /oradb/app/oracle/product/11.1.0/db_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

ASM
===
$ cd /oradb/asm/product/11.1.0/asm_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

Node 2: (AS ORACLE)
=======

RDBMS
=====
$ cd /oradb/app/oracle/product/11.1.0/db_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

ASM
===
$ cd /oradb/asm/product/11.1.0/asm_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

------------------------------------------------------------------------------------------------------------------

Post-rootpatch
==============

Node 1: (AS ROOT)
=================

# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953

# /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts/postrootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

# ps -ef | grep pmon  --- Both the db and asm instances will be up and running

# ssh <Node 2>

# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts/postrootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

# ps -ef | grep pmon  --- Both the db and asm instances will be up and running

------------------------------------------------------------------------------------------------------------------

Start ASM

------------------------------------------------------------------------------------------------------------------

Run on each database, only in one node
======================================

NODE 1: (AS ORACLE)
===================

$ sqlplus /nolog
Enter user-name: / as sysdba
Connected to an idle instance.

SQL>startup;
SQL>@?/rdbms/admin/catbundle.sql psu apply
SQL>exit

$. oraenv ---- --- database 2 instance
$ sqlplus /nolog
Enter user-name: / as sysdba

SQL>startup;
SQL>@?/rdbms/admin/catbundle.sql psu apply
SQL>exit

------------------------------------------------------------------------------------------------------------------

Patch Validation:-
================

opatch lsinventory | grep 11724953
opatch lsinventory | grep 13621679
opatch lsinventory | grep 9734685

start local listeners, (if any)

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ hostname
$ uname
$ bash
$ date
$ who -b
$ uptime
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs

$ ps -ef| grep pmon

$ srvctl status database -d <database_1>                      
$ srvctl status database -d <database_2>                      

$ srvctl status asm -n <Node 1>
$ srvctl status asm -n <Node 2>

$ srvctl status listener -n <Node 1>
$ srvctl status listener -n <Node 2>

$ srvctl status nodeapps -n <Node 1>
$ srvctl status nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

Run on each database only in one node
=====================================

NODE 1: (AS ORACLE)
===================
SQL> select name db_name,DB_UNIQUE_NAME,instance_name,status,DATABASE_STATUS,open_mode, database_role, host_name,platform_id,version db_version,log_mode,flashback_on ,protection_mode,protection_level,LOGINS,to_char(STARTUP_TIME,'DD-MON-YYYY HH24:MI:SS') "UP TIME" from gv$instance,v$database;

SQL> select to_char(sysdate,'yyyy-mm-dd HH24:MI:SS') from dual
/

SQL>
col ACTION_TIME for a12
col NAMESPACE for a10
col VERSION for a10
col BUNDLE_SERIES for a15
col COMMENTS for a20
select action_time, action,namespace, version, id,bundle_series, comments from registry$history;

------------------------------------------------------------------------------------------------------------------

check DR sysnc status
=====================
http://select-star-from.blogspot.in/2013/09/data-guard-sync-status.html

------------------------------------------------------------------------------------------------------------------

Stop OEM black out
==================

Confirmation
Request to stop blackout "Blackout-Apr 24 1998 9:33:42 PM" was processed successfully, Blackout status will be propagated to the target(s) shortly.

------------------------------------------------------------------------------------------------------------------

Oracle DBA Interview Questions and Answers - RAC - 2

Oracle RAC Interview Questions and Answers

1. Where are the Clusterware files stored on a RAC environment?
The Clusterware is installed on each node (on an Oracle Home) and on the shared disks (the voting disks and the CSR file)

2. Where are the database software files stored on a RAC environment?
The base software is installed on each node of the cluster and the
database storage on the shared disks.

3. What kind of storage we can use for the shared Clusterware files?
- OCFS (Release 1 or 2)
- raw devices
- third party cluster file system such as GPFS or Veritas

4. What kind of storage we can use for the RAC database storage?
- OCFS (Release 1 or 2)
- ASM
- raw devices
- third party cluster file system such as GPFS or Veritas

5. What is a CFS?
A cluster File System (CFS) is a file system that may be accessed (read and write) by all members in a cluster at the same time. This implies that all members of a cluster have the same view.

6. What is an OCFS2?
The OCFS2 is the Oracle (version 2) Cluster File System which can be used for the Oracle Real Application Cluster.

7. Which files can be placed on an Oracle Cluster File System?
- Oracle Software installation (Windows only)
- Oracle files (controlfiles, datafiles, redologs, files described by the bfile datatype)
- Shared configuration files (spfile)
- OCR and voting disk
- Files created by Oracle during runtime
Note: There are some platform specific limitations.

8. Do you know another Cluster Vendor?
HP Tru64 Unix, Veritas, Microsoft

9. How is possible to install a RAC if we don’t have a CFS?
This is possible by using a raw device.

10. What is a raw device?
A raw device is a disk drive that does not yet have a file system set up. Raw devices are used for Real Application Clusters since they enable the sharing of disks.

11. What is a raw partition?
A raw partition is a portion of a physical disk that is accessed at the lowest possible level. A raw partition is created when an extended partition is created and logical partitions are assigned to it without any formatting. Once formatting is complete, it is called cooked partition.

12. When to use CFS over raw?
A CFS offers:
- Simpler management
- Use of Oracle Managed Files with RAC
- Single Oracle Software installation
- Autoextend enabled on Oracle datafiles
- Uniform accessibility to archive logs in case of physical node failure
- With Oracle_Home on CFS, when you apply Oracle patches CFS guarantees that the updated Oracle_Home is visible to all nodes in the cluster.
Note: This option is very dependent on the availability of a CFS on your platform.

13. When to use raw over CFS?
- Always when CFS is not available or not supported by Oracle.
- The performance is very, very important: Raw devices offer best performance without any intermediate layer between Oracle and the disk.
Note: Autoextend fails on raw devices if the space is exhausted. However the space could be added online if needed.

14. What CRS is?
Oracle RAC 10g Release 1 introduced Oracle Cluster Ready Services (CRS), a platform-independent set of system services for cluster environments. In Release 2, Oracle has renamed this product to Oracle Clusterware.

15. What is VIP IP used for?
It returns a dead connection IMMIDIATELY, when its primary node fails. Without using VIP IP, the clients have to wait around 10 minutes to receive ORA-3113: “end of file on communications channel”. However, using Transparent Application Failover (TAF) could avoid ORA-3113.

16. Why we need to have configured SSH or RSH on the RAC nodes?
SSH (Secure Shell,10g+) or RSH (Remote Shell, 9i+) allows “oracle” UNIX account connecting to another RAC node and copy/ run commands as the local “oracle” UNIX account.

17. Is the SSH, RSH needed for normal RAC operations?
No. SSH or RSH are needed only for RAC, patch set installation and clustered database creation.

18. Do we have to have Oracle RDBMS on all nodes?
Each node of a cluster that is being used for a clustered database will typically have the RDBMS and RAC software loaded on it, but not actual data files (these need to be available via shared disk).

19. What are the restrictions on the SID with a RAC database? Is it limited to 5 characters?
The SID prefix in 10g Release 1 and prior versions was restricted to five characters by install/ config tools so that an ORACLE_SID of up to max of 5+3=8 characters can be supported in a RAC environment. The SID prefix is relaxed up to 8 characters in 10g Release 2, see bug 4024251 for more information.

20. Does Real Application Clusters support heterogeneous platforms?
The Real Application Clusters do not support heterogeneous platforms in the same cluster.

21. Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network?
RAC and Clusterware deployment best practices suggests that the interconnect (private connection) be deployed on a stand-alone, physically separate, dedicated switch. On big network the connections could be instable.

22. What is the Load Balancing Advisory?
To assist in the balancing of application workload across designated resources, Oracle Database 10g Release 2 provides the Load Balancing Advisory. This Advisory monitors the current workload activity across the cluster and for each instance where a service is active; it provides a percentage value of how much of the total workload should be sent to this instance as well as service quality flag.

23. How many nodes are supported in a RAC Database?
With 10g Release 2, we support 100 nodes in a cluster using Oracle Clusterware, and 100 instances in a RAC database. Currently DBCA has a bug where it will not go beyond 63 instances. There is also a documentation bug for the max-instances parameter. With 10g Release 1 the Maximum is 63.

24. What is the Cluster Verification Utiltiy (cluvfy)?
The Cluster Verification Utility (CVU) is a validation tool that you can use to check all the important components that need to be verified at different stages of deployment in a RAC environment.

25. What versions of the database can I use the cluster verification utility (cluvfy) with?
The cluster verification utility is release with Oracle Database 10g Release 2 but can also be used with Oracle Database 10g Release 1.

26. If I am using Vendor Clusterware such as Veritas, IBM, Sun or HP, do I still need Oracle Clusterware to run Oracle RAC 10g?
Yes. When certified, you can use Vendor Clusterware however you must still install and use Oracle Clusterware for RAC. Best Practice is to leave Oracle Clusterware to manage RAC. For details see Metalink Note 332257.1 and for Veritas SFRAC see 397460.1.

27. Is RAC on VMW are supported?
Yes.

28. What is hangcheck timer used for ?
The hangcheck timer checks regularly the health of the system. If the system hangs or stop the node will be restarted automatically.

There are 2 key parameters for this module:

-> hangcheck-tick: this parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30seconds.

-> hangcheck-margin: this defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node.

29. Is the hangcheck timer still needed with Oracle RAC 10g?
Yes.

30. What files can I put on Linux OCFS2?
For optimal performance, you should only put the following files on Linux OCFS2:
- Datafiles
- Control Files
- Redo Logs
- Archive Logs
- Shared Configuration File (OCR)
- Voting File
- SPFILE

31. Is it possible to use ASM for the OCR and voting disk?
No, the OCR and voting disk must be on raw or CFS (cluster file system).

32. Can I change the name of my cluster after I have created it when I am using Oracle Clusterware?
No, you must properly uninstall Oracle Clusterware and then re-install.

33. What the O2CB is?
The O2CB is the OCFS2 cluster stack. OCFS2 includes some services. These services must be started before using OCFS2 (mount/ format the file systems).

34. What the OCR file is used for?
OCR is a file that manages the cluster and RAC configuration.

35. What the Voting Disk file is used for?
The voting disk is nothing but a file that contains and manages information of all the node memberships.

36. What is the recommended method to make backups of a RAC environment?
RMAN to make backups of the database, dd to backup your voting disk and hard copies of the OCR file.

37. What command would you use to check the availability of the RAC system?
crs_stat -t -v (-t -v are optional)

38. What is the minimum number of instances you need to have in order to create a RAC?
You can create a RAC with just one server.

39. Name two specific RAC background processes
RAC processes are: LMON, LMDx, LMSn, LKCx and DIAG.

40.Can you have many database versions in the same RAC?
Yes, but Clusterware version must be greater than the greater database version.

41. What was RAC previous name before it was called RAC?
OPS: Oracle Parallel Server

42. What RAC component is used for communication between instances?
Private Interconnect.

43. What is the difference between normal views and RAC views?
A RAC view has the prefix ‘G’. For example, GV$SESSION instead of V$SESSION

44. Which command will we use to manage (stop, start) RAC services in command-line mode?
srvctl

45. How many alert logs exist in a RAC environment?
One for each instance.

46. What are Oracle Clusterware Components
Voting Disk — Oracle RAC uses the voting disk to manage cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The voting disk must reside on shared disk.

Oracle Cluster Registry (OCR) — Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster

47. How do you backup voting disk
#dd if=voting_disk_name of=backup_file_name

48. How do I identify the voting disk location
#crsctl query css votedisk

49. How do I identify the OCR file location
check /var/opt/oracle/ocr.loc or /etc/ocr.loc ( depends upon platform)
or
#ocrcheck

50. How do you backup the OCR
There is an automatic backup mechanism for OCR. The default location is : $ORA_CRS_HOME\cdata\"clustername"\
To display backups :
#ocrconfig -showbackup
To restore a backup :
#ocrconfig -restore
With Oracle RAC 10g Release 2 or later, you can also use the export command:
#ocrconfig -export -s online, and use -import option to restore the contents back.
With Oracle RAC 11g Release 1, you can do a manaual backup of the OCR with the command:
# ocrconfig -manualbackup

51. What is SCAN?
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters (RAC) 11g Release 2 feature that provides a single name for clients to access an Oracle Database running in a cluster. The benefit is clients using SCAN do not need to change if you add or remove nodes in the cluster.

52. What are Oracle Clusterware processes for 10g on Unix and Linux
Cluster Synchronization Services (ocssd) — Manages cluster node membership and runs as the oracle user; failure of this process results in cluster restart.

Cluster Ready Services (crsd) — The crs process manages cluster resources (which could be a database, an instance, a service, a Listener, a virtual IP (VIP) address, an application process, and so on) based on the resource's configuration information that is stored in the OCR. This includes start, stop, monitor and failover operations. This process runs as the root user

Event manager daemon (evmd) —A background process that publishes events that crs creates.

Process Monitor Daemon (OPROCD) —This process monitor the cluster and provide I/O fencing. OPROCD performs its check, stops running, and if the wake up is beyond the expected time, then OPROCD resets the processor and reboots the node. An OPROCD failure results in Oracle Clusterware restarting the node. OPROCD uses the hangcheck timer on Linux platforms.

RACG (racgmain, racgimon) —Extends clusterware to support Oracle- specific requirements and complex resources. Runs server callout scripts when FAN events occur.

53. What are Oracle database background processes specific to RAC
•LMS—Global Cache Service Process
•LMD—Global Enqueue Service Daemon
•LMON—Global Enqueue Service Monitor
•LCK0—Instance Enqueue Process
To ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a query or transaction, Oracle RAC instances use two processes, the Global Cache Service (GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the statuses of each data file and each cached block using a Global Resource Directory (GRD). The GRD contents are distributed across all of the active instances.

54. How do you troubleshoot node reboot
Please check metalink ...
Note 265769.1 Troubleshooting CRS Reboots
Note.559365.1 Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions.

55. Is ssh required for normal Oracle RAC operation ?
"ssh" are not required for normal Oracle RAC operation. However "ssh" should be enabled for Oracle RAC and patchset installation.

56. What is the purpose of Private Interconnect ?
Clusterware uses the private interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster.

57. Why do we have a Virtual IP (VIP) in Oracle RAC?
Without using VIPs or FAN, clients connected to a node that died will often wait for a TCP timeout period (which can be up to 10 min) before getting an error. As a result, you don't really have a good HA solution without using VIPs.

When a node fails, the VIP associated with it is automatically failed over to some other node and new node re-arps the world indicating a new MAC address for the IP. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

58. What do you do if you see GC CR BLOCK LOST in top 5 Timed Events in AWR Report?
This is most likely due to a fault in interconnect network.
Check netstat -s
if you see "fragments dropped" or "packet reassemblies failed" , Work with your system administrator find the fault with network.

59. How many nodes are supported in a RAC Database?
10g Release 2, support 100 nodes in a cluster using Oracle Clusterware, and 100 instances in a RAC database.

60. Srvctl cannot start instance, I get the following error PRKP- 1001 CRS-0215, however sqlplus can start it on both nodes? How do you identify the problem?
Set the environmental variable SRVM_TRACE to true.. And start the instance with srvctl. Now you will get detailed error stack.

61. What is the purpose of the ONS daemon?
The Oracle Notification Service (ONS) daemon is an daemon started by the CRS clusterware as part of the nodeapps. There is one ons daemon started per clustered node.
The Oracle Notification Service daemon receive a subset of published clusterware events via the local evmd and racgimon clusterware daemons and forward those events to application subscribers and to the local listeners.

This in order to facilitate:

a. the FAN or Fast Application Notification feature or allowing applications to respond to database state changes.

b. the 10gR2 Load Balancing Advisory, the feature that permit load balancing accross different rac nodes dependent of the load on the different nodes. The rdbms MMON is creating an advisory for distribution of work every 30seconds and forward it via racgimon and ONS to listeners and applications.

Script to Gather data from the Linux OS

#####################################################################
###   Unix script os_stats.sh                                     ###
###   Designed to be run periodically to collate information      ###
###   START OF SCRIPT                                             ###
#####################################################################
#
LOG_FILE="OS_`hostname`_`date '+%m%d%y_%H%M'`.txt"
#
echo "**********************************************" >$LOG_FILE
date >> $LOG_FILE
echo "Running as  `id`" >> $LOG_FILE
echo "**********************************************" >>$LOG_FILE
echo "uname -a" >>$LOG_FILE
uname -a >>$LOG_FILE
cat /etc/issue >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ulimit -a" >>$LOG_FILE
ulimit -a >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ulimit -Ha" >>$LOG_FILE
ulimit -Ha >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "Netstat -i" >>$LOG_FILE
netstat -i >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "netstat -an">> $LOG_FILE
netstat -an >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "netstat -s">> $LOG_FILE
netstat -s >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "sar -u 5 3">> $LOG_FILE
sar -u 5 3 >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "sar -q 5 3">> $LOG_FILE
sar -q 5 3 >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ps -e -w -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args" >> $LOG_FILE
ps -e -ww -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "iostat -t -x" >> $LOG_FILE
iostat -t -x >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "df -h" >> $LOG_FILE
df -h >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "mpstat 5 3" >> $LOG_FILE
/usr/bin/mpstat 5 3 >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "vmstat 5 3" >> $LOG_FILE
vmstat 5 3 >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "free -m -s 5 -c 3" >> $LOG_FILE
free -m -s 5 -c 3 >>$LOG_FILE
grep MemTotal /proc/meminfo >>$LOG_FILE
grep SwapTotal /proc/meminfo >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "IPCS data" >> $LOG_FILE
ipcs -l >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs -u >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs -t >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
#####################################################################
###   END OF SCRIPT                                               ###
#####################################################################

Script to Gather data from the database for Process information and User connections

REM
REM START OF SQL
REM
REM  Filename     : mzRunSQL.sql
REM  Author       : Mike Shaw
REM  Date Updated : 29 August 2008
REM  Purpose      : Script to gather data from the database for process
REM                 information and user connections
REM
set serveroutput on size 1000000
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 110
col username form a10
col program form a30
col how_many forma 9999
col machine form a25
col module form a50
--
rem show time
select to_char(sysdate, 'DD-MON-RR HH24:MI:SS') START_TIME from dual
/
rem Instance identification
select *
from v$instance
/
REM Summary of database connections
select s.module, s.machine, s.username, count(*) how_many
from (select distinct PROGRAM, PADDR, machine, username, module, inst_id from gV$SESSION) s,
gv$process p
where s.paddr = p.addr
and p.inst_id = s.inst_id
group by rollup(s.module,s.machine,s.username)
/
REM Connections by machine and instance
select s.machine, s.username, s.module, s.inst_id, count(*) how_many
from (select distinct PROGRAM, PADDR, machine, username, module, inst_id from gV$SESSION) s,
gv$process p
where s.paddr = p.addr
and p.inst_id = s.inst_id
group by s.machine,s.username, s.module, s.inst_id
/
REM
REM Get count of number of connected Apps 11i users
REM
col user_name format a15
col first_connect format a18
col last_connect format a18
col How_many_user_sessions format 9999999999
col How_many_sessions format 9999999999
REM
REM Summary of how many users
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate  
and counter < limit_connects
/
REM
REM Number of sessions per user
REM
select user_name, count(*) How_many_sessions
from icx_sessions icx, fnd_user u
where icx.user_id = u.user_id
and disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate  
and counter < limit_connects
group by user_name
order by 2 desc
/
REM
REM Quick look for blockers and lockers
REM
col BLOCKER form a5
col Username format A15
col Sid format 9990 heading SID
col Type format A4
col Lmode format 990 heading 'HELD'
col Request format 990 heading 'REQ'
col Id1 format 999999999999
col Id2 format 999999999999
break on Id1 skip 1 dup
--
SELECT M.Sid,
M.Type,
DECODE(M.Lmode, 0, 'None', 1, 'Null', 2, 'Row Share', 3, 'Row Excl.', 4, 'Share', 5, 'S/Row Excl.', 6, 'Exclusive',LTRIM(TO_CHAR(Lmode,'990'))) Lmode,
DECODE(M.Request, 0, 'None', 1, 'Null', 2, 'Row Share', 3, 'Row Excl.', 4, 'Share', 5, 'S/Row Excl.', 6, 'Exclusive',
LTRIM(TO_CHAR(M.Request, '990'))) Request,
M.Id1,
M.Id2,
DECODE(block, 0, 'NO', 'YES' ) BLOCKER
FROM V$LOCK M
WHERE
M.Request ! = 0 or (M.Request = 0 and Lmode != 4 and (id1, id2) in (select S.Id1, S.Id2 from V$LOCK S where Request != 0 and S.Id1 = M.Id1 and S.Id2 = M.Id2) )
ORDER BY Id1, Id2, M.Request
/
REM
REM Check for database statement leaks
REM May give symptoms like OutOfMemoryError
REM
set lines 132
set pages 1000
column module format A40
column program format A30
select oc.sid,oc.hash_value,oc.sql_text,count(*) How_Many
from gv$open_cursor oc
group by sid,hash_value,sql_text
having count(*) > 5
order by 4
/
rem show time
select to_char(sysdate, 'DD-MON-RR HH24:MI:SS') END_TIME from dual
/
exit;
REM
REM END OF SQL
REM 

Speech by Chetan Bhagat at Symbiosis