Oracle RAC Load balancing and Failover

LOAD BALANCING in RAC:-
The Oracle RAC system can distribute the load over many nodes this feature called as load balancing.

There are two methods of load balancing
1.Client load balancing
2.Server load balancing

1.Client Load Balancing
Client Load Balancing distributes new connections among Oracle RAC nodes so that no one server is overloaded with connection requests and it is configured at net service name level by providing multiple descriptions in a description list or multiple addresses in an address list. For example, if connection fails over to another node in case of failure, the client load balancing ensures that the redirected connections are distributed among the other nodes in the RAC.

Configure Client-side connect-time load balancing by setting LOAD_BALANCE=ON in the corresponding client side TNS entry.

TESTRAC =
(DESCRIPTION =
(ADDRESS_LIST=
(LOAD_BALANCE = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
)
(CONNECT_DATA = (SERVICE_NAME = testdb.selectstarfrom.com))
)

2.Server Load Balancing
Server Load Balancing distributes processing workload among Oracle RAC nodes. It divides the connection load evenly between all available listeners and distributes new user session connection requests to the least loaded listener(s) based on the total number of sessions which are already connected. Each listener communicates with the other listener(s) via each database instance’s PMON process.

Configure Server-side connect-time load balancing feature by setting REMOTE_LISTENERS initialization parameter of each instance to a TNS name that describes list of all available listeners.

TESTRAC_LISTENERS =
(DESCRIPTION =
(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1)(PORT = 1521)))
(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2)(PORT = 1521))))
)

Set *.remote_listener= TESTRAC_LISTENERS’ initialization parameter in the database’s shared SPFILE and add TESTRAC_LISTENERS’ entry to the TNSNAMES.ORA file in the Oracle Home of each node in the cluster.

Once you configure Server-side connect-time load balancing, each database’s PMON process will automatically register the database with the database’s local listener as well as cross-register the database with the listeners on all other nodes in the cluster. Now the nodes themselves decide which node is least busy, and then will connect the client to that node.

-------------------------------------------------------------------------------------------------------------------------------------------------------------

FAILOVER in RAC:-
The Oracle RAC system can protect against failures caused by O/S or server crashes or hardware failures. When a node failure occurs in RAC system, the connection attempts can fail over to other surviving nodes in the cluster this feature called as Failover.

There are two methods of failover
1. Connection Failover
2. Transparent Application Failover (TAF)

1. Connection Failover
If a connection failure occurs at connect time, the application failover the connection to another active node in the cluster. This feature enables client to connect to another listener if the initial connection to the first listener fails.

Enable client-side connect-time Failover by setting FAILOVER=ON in the corresponding client side TNS entry.

TESTRAC =
(DESCRIPTION =
(ADDRESS_LIST=
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
)
(CONNECT_DATA = (SERVICE_NAME = testdb.selectstarfrom.com))
)

If LOAD_BALANCE is set to on then clients randomly attempt connections to any nodes. If client made connection attempt to a down node, the client needs to wait until it receives the information that the node is not accessible before trying alternate address in ADDRESS_LIST.

2. Transparent Application Failover (TAF)
If connection failure occurs after a connection is established, the connection fails over to other surviving nodes. Any uncommitted transactions are rolled back and server side program variables and session properties will be lost. In some case the select statements automatically re-executed on the new connection with the cursor positioned on the row on which it was positioned prior to the failover.

TESTRAC =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = testdb.selectstarfrom.com)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))
)
)

Oracle APPS DBA Interview Questions and Answers - Beginners 11i

Oracle APPS DBA Interview Questions and Answers 

5. What are u r regular activities?
Ans: Patching, patch analysis, monitoring, trouble shooting, resolving lock issues, providing trace files,cloning, shutdown/starups,backup etc….

6. What is a patch?
Ans : A patch can be a solution for a bug/it can be a new feature.

7. What are the different types of patches?
Ans : oneoff, mini packs, family packs, maintanance packs, rollup pathches, colsolidated patches.

8. What is a oneoff patch?
Ans : An oneoff patch is a small patch of (20-90K size) without any pre-req’s

9. What is a mini pack ?
Ans : A mini pack is one which will upgrade any product patchset level to next level like AD.H to AD.I

10. What is Family pack ?
Ans : A Family pack is one which will upgade the patchset level of all the products in that family to perticular patchsetlevel.

11. What is Maintanance pack ?
Ans : A maintanance pack will upgrade applications from one version to another like 11.5.8 to 11.5.9

12. What is a Rollup patch?
Ans : A rollup patch is one which will deliver bug fixes identified after the release of any major application versions like 11.5.8/11.5.9

13. What is consilidated patch?
Ans: Consolidated patches will come into pictures after upgrades from one version of applications to anoter, all post upgrade patches will a consolidated and given as consolidated patch.

14. How u will find whether a patch is applied/not?
Ans : Query ad_bugs.

15. What is the other table where u can query what are the patches applied?
Ans : Ad_applied_patches

16. What is the difference between ad_bugs and ad_applied_patches?
Ans: A patch can deliver solution for more than one bug, so ad_applied_patches may not give u the perfect information as in case of ad_bugs.

17. How u apply a patch?
Ans : adpatch

18. What inputs you need to apply a patch other than driver name and etc?
Ans : apps and system passwords

19. What are the table u r adpatch will create and when?
Ans : Adpatch will creat FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS table when it will apply d,g and u drivers

20. What is the significance of FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS table?
Ans: FND_INSTALL_PROCESSES table will store the worker information like what job is assigned to which worker and its status. AD_DEFERRED_JOBS will come into picture when some worker is failed, it will be moved to AD_DEFERRED_JOBS table, from where again adpatch will take that job and try to resign, after doing this 3 times if still that worker is failing, then adpatch will stop patching and throw the error that perticular worker has failed. We need to trouble shoot and restrart the worker.

21. If it is a multinode installation which driver we need to apply on which node?
Ans: c,d,g on concurrent node and c, g on web node. If it is u-driver we need to apply on all nodes.

22.While applying a application patch is that necessary that u r database and listener should be up?
Ans: Yes . why because adpatch will connect to database and update so many tables etc…..

23. While applying a patch if that patch is failing because of a pre-req then how you will apply that pre-req patch and resume with the current patch?
Ans: We need to take the backup of FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables and restart directory at APPL_TOP/amdin/SID and then use adctrl to quit all the workers. Then apply the pre-req patch , after that rename u r restart directory to its original name and create FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables from the bcakup tables. Start adpatch session and take the options want to continue previous session.

24. What is adctrl?
Ans: Adctrl is one of the adutilities, which is used to check the status of workers and to manage the workers.

25. Can u name some of the menu options in adctrl?
Ans: Check the status of workers, tell manager that worker has quited, restart a failed worker etc….

26. How to skip a worker and why?
Ans: We can skip a worker using option 8 in adctrl which is hidden. We will go for skipping a worker when we have executed the job which the worker is supposed to do.

27. How adpatch knows what are the pre-req’s for the patch which it is applying?
Ans: With every patch a file called b<patch_number>.ldt file will be delivered which contain the pre-req information. adpatch load this into databse using FNDLOAD and check , whether those pre-req patches were applied or not.

28. What is FNDLOAD ?
Ans: FNDLOAD is a utility which is similar to sqlloder but loads code objects into database, where as SQLLOADER loads data objects into database.


29. What c-driver will do?
Ans: C-drive copies the files from patch unzipped directory to required location in u r application file system. Before copying it will check the file version of the existing file at the file system with the file version of the file in the patch. If the patch file version is higher than what it is at file system level then only c-driver will copy that files.

30. How adpatch will know the file versions of the patch delivered files?
Ans: With each patch a file with name f<patch_number>.ldt is delivered , which contain the file versions of the files dilivered with the patch. Adpatch will use this file to compare the file versions of files its delivering with the file on file system.

31. What is the adpatch log file location?
Ans : APPL_TOP/admin/SID/log

32. What is the worker log file name and its location?
Ans : adwork01,adwork02…… and location is APPL_TOP/admin/SID/log

33. How u will know what are the files the patch is going to change just my unzipping the patch?
Ans: When u unzip a patch it will keep all the files related to a particular product under that directory inside u r patch directory for example if the patch delivering files related to FND product then it will create a sub directory under the patch directory with the name FND in which it will put all related files to that product

34. What is the significance of backup directory under u r patch directory?
Ans: When we apply a patch it will keep the copy of the files which its going to change in file system.

35. What are the different modes you can run your adpatch?
Ans :
1.Interactive – default mode
2.Non interactive – Use defaults files to store prompt values (adpatch defaultsfile=<filename> interactive=no)
3.Test – Without actually applying a patch just to check what its doing.(adpatch apply=no)
4.Pre-install – (adpatch preinstall=y) This mode will be usefull to discrease upgrade downtime as its applies bus fixes without running SQL,EXEC and generate portion of patch.

36. How u will monitor u r applications as well as database?
Ans: We have our custom scripts which is sheduled to run at a specific time which will monitor whether applications and databases are up/not. And it will mail us if some processes is not running. And we have one script which will check database alert log for ORA errors and mails it to us . Based on this we will react.

37. What are the latest ORA errors u have encountered?
Ans : Useually we will get the ORA errors like unable to extend the tablespace by so and so size. And we will check those tablespaces for space, if space is not there we will resize the datafile and add one more datafile.

38. Which table u will query to check the tablespace space issues?
Ans : bytes column in dba_free_spaces and dba_data_files

39. Which table u will query to check the temp tablespace space issues?
Ans : dba_temp_files

40. What is temp tablespace? And what is the size of temp tablespace in u r instances?
Ans : Temp tablespace is used by so many application programs for sorting and other stuff. Its size is between 3 to 10 GB.

41. What is autoconfig?
Ans : Autoconfig is an adutility which is used to main application environment and configuration files.

42. What are the parameter autoconfig will ask for?
Ans : Context file name and apps password

43. What is context file?
Ans : Context file is a central repositary, which stores all application configuration information. The name is like <Instance name>_ <Server name>.xml

44. How you will find autoconfig is enabled/not for u r applications?
Ans:
1. Open any env / configuration files, the first few lines will tell u that this files are maintained by autoconfig.
2. If contextname.xml file is there in APPL_TOP/admin

45. How autoconfig will create env and configuration files?
Ans: Autoconfig will go to each and every top template directory take the templates from there and fill the values from xml file and create the required files.

46. In how many phases autoconfig will run?
Ans : Autoconfig will run in 3 phases.
1.INIT – Instantiate the drivers and templates
2.SETUP – Fill the templated with values from xml and create files
3.PROFILE – Update the profile values in database.

47. What is the location of adconfig log file?
Ans : APPL_TOP/admin/<context_name>/log/<timestamp directory>

48. Is it possiable to restore a autoconfig run?
Ans : Partially. Adconfig will create a restore.sh script at $APPL_TOP/admin/<context_name>/out/<timestamp directory>. This restore.sh will copy the backed up files before autoconfig run to its original locations. But the profile values updated in the database can’t be restored back.

49. How to run autoconfig in test mode?
Ans : adchkcfg.sh script at AD_TOP/bin. This script will run autoconfig in test mode and create the difference file which tells us what is going to change , when u actually run autoconfig.

50. How to find autoconfig is enabled or not for database?
Ans: If we have appsutil directory under RDBMS_ORACLE_HOME

51. When a patch delivers java files what extra file u will get when u unzip the patch, other then u r dirver and readme files?
Ans : j<patch_number>.zip

52. What is apps.zip/appsbrog2.zip file?
Ans : apps.zip/appsbrog2.zip is the patchable archive of all java class files required for oracle application.
      Apps.zip was used to old application version, but from 11.5.8 onwards its appsbrog2.zip

53. What is the location of apps.zip/appsbrog2.zip?
Ans : AU_TOP/java and JAVA_TOP

54. What is for “validating apps schema” option in adadmin?
Ans: It will check for the corrupted objects in apps schema

55. What is “compile apps schema” option in adadmin?
Ans : It will compile the invalid database objects.

56. How to find invalid objects in database?
Ans : select count(*) from dba_objects where status=’INVALID’;

57. How to find MRC is enabled or not?
Ans: In adadmin if covert to MRC options is there , then MRC is not enabled.
     If maintain MRC options is there , then MRC is enabled.

58. How to find Multi-Org is enabled or not?
Ans : In adadmin if covert to Multi org option is there, then Multi-org is not enabled. If maintain multi-org options is there, then Multi-org is enabled.

59. What is mean by MRC?
Ans: MRC stands for Multiple reporting Currency, this should be enabled to see the reports in different currencies like (rupees,yaans etc).

60. What is Multi-Org?
Ans: If this is enabled we can store multiple organization information in a single oracle application instance.

61. What is the configuration file for adutilities (like adadmin,adconfig etc)?
Ans: adconfig.txt @APPL_TOP/admin

62. What is adrelink?
Ans : adrelink will relink the executables with the libraries. Generally we will go for adrelink when some patch delivers some library files, or when executables were corrupted.

63. How to find the version of a file?
Ans :
1. adident Header <filename>
2. strings -a filename | grep Header

64. What is adodfcmp utility?
Ans : This utility is used to recreate/repair corrupted database objects from odf(object defination files) files.

65. How you will change apps password?
Ans: FNDCPASS 0 y apps/<pwd> system/<pwd> SYSTEM APPLSYS <new pwd>

66. What if apps password is changed with alter command?
Ans : Applications won’t work.

66. What is the difference between alter and FNDCPASS in changing apps password?
Ans : FNDCPASS will update some fnd tables other than standard tables.

67. Where the FNDCPASS utility is located?
Ans : Concurrent node @FND_TOP/bin

68. How to find out what component of u r oracle applications were installed on which node?
Ans : Xml file (context file)

69. How to find the version of httpd/Apache web server?
Ans : $IAS_ORACLE_HOME/Apache/bin/httpd –version

70. What is the configuration file for httpd and what is the location of it ?
Ans : httpd.conf @IAS_ORACLE_HOME/Apache/Apache/conf

71. Where you will see when you have some problem with u r webserver(httpd/Apache)?
Ans : access_log & error_log @IAS_ORACLE_HOME/Apache/Apache/logs

72. When Apache starts what other components its start ?
Ans : PL/SQL Listener, Servlet Engine, OJSP Engine

73. What is jserv?
Ans : jserv is nothing but servlet engine which will run u r servlets. It’s a module of apache which supports servlets.

74. What is self service application?
Ans : Whatever part of u r oracle application u r able to see through web browser is self service.

75. Where u will see when u r not able to get self service applications?
Ans : access_log,error_log, error_pls, jserv.log, wdbsvr.app(for apps password)

76. What is the location of jserv.log?
Ans : IAS_ORACLE_HOME/Apache/Jserv/log

77. What is the location of wdbsvr.app ?
Ans : IAS_ORACLE_HOME/Apache/modplsql/cfg

78. What are jserv.conf and jserv.properties files?
Ans : These are the configuration files which were used to start jvm’s(servlet engine) by apache.

79. What is mean by clearing cache and bouncing apache?
Ans :
1. Stop apache (adapcctl.sh stop apps)
2. Clear cache – Go to $COMMON_TOP/html/_pages and delete _oa_html directory (rm –r _oa__html)
3. Start apache (adapcctl.sh start apps)

80. What is forms configuration file and its location?
Ans : appsweb_contextname.cfg @$COMMON_TOP/html/bin

81. What are the different modes u can start u r form server?
Ans : socket and servlet

82. What is the difference beween socket and servlet mode?
Ans : In socket mode forms sessions are represented by f60webmx
      In servlet mode forms sessions are represented by apache processes.

83. What is forms metric server and client?
Ans : When there are more than one form sever instances then forms metric server and clinet will be used to load balance.

84. Where the forms server related errors will be logged?
Ans : access_log and error_log

85. What is report server configuration and log file name and its location?
Ans : Configuration file – REP_<SID>.ora
      Log file – REP_<SID>.log @806_ORACLE_HOME/reports60/server

86. What is CGIcmd.dat file and its location?
Ans : CGIcmd.dat file is the run time parameter file the report server located @ 806_ORACLE_HOME/reports60/server

87. What is the significance of DISPLAY variable?
Ans : Vnc server should be up and running at the specified port value in DISPLAY variable, otherwise reportserver may not able to show the graphics in
      Reports.

88. Where is the concurrent manager log file located?
Ans : $COMMON_TOP/admin/<SID>/log or $APPLCSF/$APPLLOG

89. Is apps password necessary to start all the components of oracle application?
Ans : No. Only to start/stop concurrent managers apps password is needed.

90. What is a concurrent manager?
Ans : A concurrent manager is one which runs concurrent requests.

91. What are the different types of concurrent managers?
Ans :
1. Internal concurrent manager – Will start all other managers and monitor
2. Standard Manager – All concurrent request by default will to go this
3. Conflict resolution manager – Concurrent programs with incompatabilites will be handled by this
4. Transaction manager – Handle all transaction requests

92. What are actual and target count in ‘Adminster Concurrent Managers form’?
Ans : Target is the no. of concurrent processes a manager is supposed to start(specified in the defination of concurrent manager).
      Actual is the no. of processes a manager started actually.
      Target and Actual should be always same.

93. What if Target and Actual are not same?
Ans : It means at operating system level resources are low to accomidate the required processes for concurrent managers.

94. What are work shifts?
Ans : Work shifts are nothing but timings at which the concurrent manager is supposed to run.

95. What if internal concurrent manager target and actual are not same?
Ans : we need to bounce the concurrent manager using adcmctl.sh

96. How to bounce a single concurrent manager?
Ans : From frontend using ‘Administer Concurrent Manager form’.

97. When we change apps password , is it necessary to bounce application?
Ans : Only we need to bounce concurrent managers.

98. What is dbc file and its location?
Ans : dbc file contain database connection information. DBC file is used by oracle applications to connect to database. Its location is $FND_TOP/secure

99. What is the other script by which u can start apache other than adapcctl.sh?
Ans : apachectl @IAS_ORACLE_HOME/Apache/bin

100. What is the configuration file for PL/SQL listener?
Ans : httpd_pls.conf @IAS_ORACLE_HOME/Apache/Apache/conf

101. How to skip copy portion while applying a patch?
Ans : Adpatch options=nocopyportion

102. How to merge patches and what type of patches can be merged?
Ans : admrgpch. We can merge any kind of application patches, if any of the patch contain a u-driver then merged patch will contain u_merged.drv otherwise c_merged.drv, d_merged.drv and g_merged.drv

103. What is the Tiered architecture of u r instance?
Ans : Two Tier: Web and Forms on one node and Conc, admin and report on other node.

104. How to find formserver version?
Ans: f60gen and press enter, it will tell u the formserver version or we can find out from the frondend using help menu.

105. What is RRA?
Ans : RRA stands for Report Review Agent. RRA is nothing but FNDFS which is part of apps listener. RRA job is to pick the log/out file from the file system and show on the editor when u press view log/out button in ‘View concurrent request form’.

106. What is apps listener?
Ans : Apps lintener is the combination of FNDFS and FNDSM. FNDSM is service manager which will monitor application services on that node when GSM:enable profile value is ‘Y’.

107. What is GSM?
Ans : GSM stands for Generic service Manager, which will monitor application processes like web, forms etc and restarts any of this processes if goes down.

108. How to find the application version like 11.5.8/11.5.9….?
Ans : select release_name from fnd_product_groups;

109. How to find the database/sqlplus version?
Ans : select banner from v$version;

110. How to find out what are the languages enabled in u r applications?
Ans : Query fnd_languages

111. What is the size of u r database?
Ans : 200 to 500 GB

112. How to find operating system version?
Ans : uname –a

113. What are the problems u have faced while shutting down applications?
Ans : While shutting down application generally concurrent manager won’t go down because some or the other request may be running. We will see what are the concurrent requests running by querying fnd_concurrent_requests, fnd_concurrent_program_vl, v$session,v$process and v$sqltext. If that request is only doing some select statement then we will kill those requests, otherwise we will check what time it will take to complete by querying the previous runs of that request and then we will decide what to do.

114. What are the problems u have faced while starting up applications?
Ans : Most of the time we will encounter problem with starting up concurrent managers. Reasons , database listener may be down or FNDSM entries are wrong in tnsnames.ora of 806_ORACLE_HOME.

115. How to find the locks and what is the resolution?
Ans : we can find general locks with the following query:
      select * from sys.dba_dml_locks order by session_id.

      We can find the dead locks with the following query:
      select * from v$lock where lmode > 0 and id1 in (select distinct id1 from v$lock where request > 0)
      If it’s a dead lock, we need to kill that session.

116. How to kill a database session?
Ans : alter system kill session '&sid,&sno';

117. How to find adconfig is enabled for oracle operating system user/database?
Ans : If appsutil directory is there in RDBMS_ORACLE_HOME

118. Which files tell u the database helath?
Ans : alert log file @RDBMS_ORACLE_HOME/admin/<Contextname>/bdump

119. How to apply a rdbms patch?
Ans : Using opatch

120. How to find opatch is enabled or not for u r database?
Ans : If Opatch directory exists under RDBMS_ORACLE_HOME.

121. What is the pre-req for applying a rdbms patch?
Ans : Inventory should be set in file oraInst.loc @/var/opt/oracle or /etc

122. What is Inventroy?
Ans: The oraInventory is the location for the OUI (Oracle Universal Installer)'s bookkeeping. The inventory stores information about:
     1. All Oracle software products installed in all ORACLE_HOMES on a machine
     2. Other non-Oracle products, such as the Java Runtime Environment (JRE)
     In a 11i Application system the RDBMS and iAS ORACLE_HOMEs are registered in the oraInventory. The 806 ORACLE_HOME, which is not managed through OUI, is not.

123. What are different types of inventories?
Ans: The Global inventory (or Central inventory)
     The Local inventory (or Home inventory)

124. What is Global inventory?
Ans : The Global Inventory is the part of the XML inventory that contains the high level list of all oracle products installed on a machine. There should therefore be only one per machine. Its location is defined by the content of oraInst.loc. The Global Inventory records the physical location of Oracle products installed on the machine, such as ORACLE_HOMES (RDBMS and IAS) or JRE. It does not have any information about the detail of patches applied to each ORACLE_HOMEs. The Global Inventory gets updated every time you install or de-install an ORACLE_HOME on the machine, be it through OUI Installer, Rapid Install, or Rapid Clone. Note: If you need to delete an ORACLE_HOME, you should always do it through the OUI de-installer in order to keep the Global Inventory synchronized.

125. What is local inventory?
Ans : There is one Local Inventory per ORACLE_HOME. It is physically located inside the ORACLE_HOME at $ORACLE_HOME/inventory and contains the detail of the patch level for that ORACLE_HOME. The Local Inventory gets updated whenever a patch is applied to the ORACLE_HOME, using OUI.

126. What is rapid clone?
Ans : Rapid Clone is the new cloning utility introduced in Release 11.5.8. Rapid Clone leverages the new installation and configuration technology utilized by Rapid Install

127. How do I determine if my system is rapid clone enabled?
Ans : First, verify that your system is AutoConfig enabled. Then, verify that you have applied the latest Rapid Clone patch.

128. Explain the cloning process?
Ans :
1. Run adpreclone as applmgr and oracle user on source
   Perl adpreclone.pl dbTier as oracle user
   Perl adpreclone.pl appsTier as applmgr user
2. Take the cold/hotbackup of source database
3. Copy the five directories <prod>appl,<prod>comn,<prod>ora , <prod>db,<prod>data to target
4. Rename the directories, and change the permisssion
5. Set the inventory in oraInst.loc
6. Run perl adcfgclone.pl dbTier as oracle user,if the backup type is cold
7. If the backup type is hotbackup then
   Perl adcfgclone.pl dbTechStack.
   Create the control file on target from the control script trace file from source
   Recover the database
   Alter database open resetlogs
8. Run autoconfig with the ports changed as per requirement in xml.
9. Run perl adcfgclone.pl appsTier as applmgr
10.Run autoconfig with the ports changed as per requirement in xml.

129. What is the location of adpreclone.pl for oracle user?
Ans : RDBMS_ORACLE_HOME/appsutil/scripts/<contextname>

130. What is the location of adpreclone.pl for applmgr user?
Ans : $COMMON_TOP/admin/scripts/<contextname>

131. What is the location of adcfgclone.pl for oracle user?
Ans : $RDBMS_ORACLE_HOME/appsutil/clone/bin

132. What is the location of adcfgclone.pl for applmgr user?
Ans : $COMMON_TOP/clone/bin

133. What is statspack?
Ans : Statspack is a database utility to gather database and session level performance information.

134. How to install statspack?
Ans : Run the script spcreate.sql @RDBMS_ORACLE_HOME/rdbms/admin
Note more details on statspack refer metalink noteid: 149113.1

135. How to enable trace at database level?
Ans : set init.ora parameter sql_trace

136. How to enable trace for a session?
Ans: Alter system set sql_trace=true;
     Execute the sql query
     Alter system set sql_trace=false;
     This will create a trace file at $RDBMS_ORACLE_HOME/admin/contextname/udump with the spid of the current sql session.

137. How to enable trace for other session?
Ans : exec sys.dbms_system.set_sql_trace_in_session(sid,serial#,true/false)
     
      Eg:
      To enable trace for sql session with sid 8
      SQL> exec sys.dbms_system.set_sql_trace_in_session(8,121,true);
      PL/SQL procedure successfully completed.
     
      To disable trace
      SQL> exec sys.dbms_system.set_sql_trace_in_session(8,121,false);

138.What is the location of inint.ora ?
Ans : $RDBMS_ORACLE_HOME/dbs

139. What is that trace files contains and the utiliy used to read them?
Ans : Trace file contains the detail diagnostics of a sql statement like explain plan, physical reads, logical reads, buffer gets etc. Tkprof utility is used to convert trace file into readable format.

140. What is the syntax for tkprof?
Ans: tkprof <trace filename> <output filename> explain=apps/<pwd> sys=no

141.How do we find adpreclone is run in source or not ?
Ans : If clone directory exists under RDBMS_ORACLE_HOME/appsutil for oracle user and $COMMON_TOP for applmgr user.

142. How to find that the database is 64-bit/32-bit?
Ans : $RDBMS_ORACLE_HOME/bin/file oracle
      Ex: /oraDB/angdb/920/bin> file oracle
      oracle: setuid setgid ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

143. How to find trace file for a given concurrent request id?
Ans : Go to $RDBMS_ORACLE_HOME/admin/<context_name>/udump
      Grep “<request id> “ *

144. What is a database link? How to create it?
Ans : If we want to access objects of another database from this database then we need a database link from this database to the other.
      1.Login as oracle user
      2.sqlplus “/as sysdba”
      3.create database link <dblink name> connect to <destination user> identified by <d;estination user pwd> using '<destination database name>';
      Ex:
      SQL> create database link TEST1_TO_TEST2 connect to apps identified by apps using 'TEST2';
      Database link created.
      SQL> select name from v$database@ TEST1_TO_TEST2;
      NAME
      ---------
      TEST2
      SQL>select db_link from dba_db_links;
      4.Add destination database tns entry in tnsnames.ora

145. How many clonings u have done?
Ans : If u r very much confident on cloning processes then say 5 to 8 otherwise just 2 or 3.

146. What u know abt RMAN?
Ans : If u r good at RMAN then say yes, otherwise say we are not using RMAN for backup/recovery , why because we are using netapp snap technology for backups.

147. What is netapp?
Ans : Netapp is a storage technology.

148. Have u done any performance tunning?
Ans : Better say No. If u r very good in performance tunning then say ‘yes’.

149. What is formserver url?
Ans :http://hostname.domain:<port>/dev60cgi/f60cgi

150. What is jinitiator?
Ans : Oracle jinitiator is the one which provide the required jvm to run forms interface/applet. When we access forms applet first time , oracle jinitiator will be installed automatically.

151. What is discoverer server?
Ans : Discoverer server is reporting tools which allows novoice user to use oracle application reports. Discoverer will come along with oracle applications when installed.

152. What is discoverer viewer url?
Ans: http://hostname.domain:<port>/discoverer4i/viewer

153. What is discoverer plus url?
Ans : http://hostname.domain:<port>/discwb4/html/english/welcome.htm

154. What is aoljtest and its url?
Ans : Aoljtest is a web based utility to test the availability of the different components of oracle applications like jserv,modplsql,jsp,forms etc
URL: http://hostname.domain:<port>/OA_HTML/jsp/fnd/aoljtest.jsp

155. What is adsplicer?
Ans : Adsplicer is a uitility used to register off cycle products.

156. What is licence manager?
Ans : Licence manager(adlicmgr) utility is used to licence/unlicence , enable new languages,enbale country specific functionality.

157. What is tnsping?
Ans : tnsping is command used to check the connectivity to the database server node from other nodes.
      Ex: tnsping <dbsid>
      Note: Tns entry should be there in tnsnames.ora for the database we are trying to work this command.

158. What is top command?
Ans : top is a operating system command, it will display top 10 processes which are taking high cpu and memory.

159. How to compile a form using f60gen?
Ans : f60gen module=/TEST/testappl/au/11.5.0/forms/F/ARXTWMAI.fmb userid=APPS/APPS output_file=/TEST/testappl/ar/11.5.0/forms/F/ARXTWMAI.fmx module_type=form batch=yes compile_all=special

160. What is APPLPTMP environment variable?
Ans : This is the temporary file location for the pl/sql temp files. If this variable was not set then the concurrent programs may errored out.

161. What is mean by enabling maintanance mode?
Ans : Maintanance mode is the adadmin option introduced from AD.I. When maintanance mode is enabled user may able to login to application but they only get profile option in the frontend navigation menu.

162. Is that necessary to enable maintanance mode while applying a patch?
Ans : We can even apply a patch without enabling maintanance mode with the following option
      Adpatch options=hotpatch

163. How to find out oracle application framework version?
Ans : 1. Through aoljtest
      2. cd $COMMON_TOP/html/
      3. adident Header OA.jsp

164. How to find out what are the rdbms patches applied to an oracle home?
Ans : 1. opatch –lsinventory
      2. $RDBMS_ORACLE_HOME/.patch_storage directory contains the directories with the rdbms patch number, which are applied to this oracle home.

165. Is that necessary to shutdown database while applying a database patch?
Ans : Yes.

166. What is the command line utility to submit a concurrent request?
Ans : CONSUB

167. What is the significance of utl_file_dir parameter in init.ora file?
Ans : The value of this parameter is the group of directories to which u r database can write, means u r database packages have permission to write to flat files in these directories.

168. How you will find out discoverer version?
Ans : cd $806_ORACLE_HOME/discwb4/lib
      strings libd* | grep 'Version:'

169. While applying a rdbms patch using opatch you are getting the error, unable to read inventory/inventory is corrupted/ORACLE_HOME is not not registered, what you will do, and how you will apply the patch?
Ans: We will check the inventory directory permission, try to apply the patch after giving 777 permissions to that inventory directory. If still it won’t work we will apply patch with the following command:
     Opatch apply –no_inventory

170. Have you opened any TAR and for what?
Ans : Yes. If we have any issues on production then we will raise seviarity 1 tar otherwise seriarity 2/normal.

171. For what you have raised the TAR?
Ans : We got ORA-7445 error (memory leak) in alert log, for which we have raised a Seviarity 1 TAR.

172. Have you applied rdbms patches and for what?
Ans : We got ORA-7445 error in alert log, for which oracle recommended to apply a rdbms patch.

173. What are the patch errors , you have encountered?
Ans :
1)Patch fails with the error, unable to generate perticular form, do u want to continue. We continue patching by saying “yes”, then we manually regenarate the form using f60gen utility.
2) Unable to generate jar files under JAVA_TOP
AutoPatch error:
Failed to generate the product JAR files
Solution:
Run adjkey -initialize -----------to creat identitydb.obj file which will be
used by adjava to sign jar files.

174. What is adjkey? What files it will create?
Ans : adjkey is an adutility which will create digital signature, which will be used to sign all the jar files when we generate jar files using adadmin. The purpose of this is, when you access forms interface every time, the signature of the jar files which is going to download is compared with the existing cached jar files signature in client machine. If signature won’t match, it will download that jar files again.
It will create the following files:
adsign.txt@APPL_TOP/admin
appltop.cer@APPL_TOP/admin
identitydb.obj@applmgr home

175. Have done any OWC with oracle?
Ans : No.

176. What are the post installation task?
Ans : Running adjkey –initialize and then runnning adadmin to regerate jar files.

177. What are the clone errors, you have encountered?
Ans :
Error:
RC-50013: Fatal: Failed to instantiate driver /u01/fms2c/appfms2c/fms2cora/iAS/appsutil/driver/instconf.drv
Cause:
The source instance has files that adpreclone flags as 'autoconfigable' but in reality they are not. So adpreclone.pl adds these files into the instconf.drv. Then when adcfgclone.pl is run on target it looks for the template file to instantiate for these files and since there isn't a template file adcfgclone.pl fails.
Solution:
Modify the target's instconf.drv and remove the offending lines. Then rerun adcfgclone.pl

178. What are the real time problems you have encountered and how you trouble shooted that?
Ans:
1. Concurrent Program is erroing out with snapshot too old error. To resolve this we have added space to temp tablespace.
2. Concurrent Program is erroing out with unable to extent a perticular tablespace by so and so extents. To resolve this we have added on more data file to that tablespace.
3. When we are trying to start apache with adapcctl.sh script after a autoconfig run, its saying that “node id is not matching with the application server id”. To resolve this we have updated the server id column in fnd_nodes table with the server id value in dbc file.

179. How you will find workflow version?
Ans : Run wfver.sql@FND_TOP/sql script as apps user

180 . When forms are running in servlet mode then the environment variables required for forms must be defined in what file and its location?
Ans : formsservlet.ini@$APACHE_TOP/Jserv/etc.

181. How to find out which patch driver is applied(like c,d,g or u)?
Ans: query ad_patch_drivers.

182. How to find out whether a language patch is applied for a perticular patch?
Ans : Query ad_patch_driver_langs.

183. How to validate that sysadmin password is correct or not from backend?
Ans: select fnd_web_sec.validate_login('SYSADMIN','Qwert8765') from dual;

184. How to compile jsp's(other than from adadmin)?
Ans: Force compilation of all jsps using the following command
     ojspCompile.pl --compile --flush

185. How to rotate logs for apache logs?
Ans: Using rotatelogs executable in httpd.conf file. Use Errorlog for error_log file rotation. Transferlog for other log files.

186. Other way of checking whether MRC is enabled or not besides using adadmin?

Ans : select multi_currency_flag from fnd_product_groups;

187. How to compile rdf?
Ans: Either using adadmin or rwcon60

188. How you will see hidden files in linux/solaris?
Ans : ls –la

189. How to change file/directory owner in linux/solaris?
Ans : chown –R <username>:<group> <file/directory>
      Ex:
      chown –R applmgr:dba testappl

190. How to change the permission of file/directory in linux/solaris?
Ans : chmod –R <permissions> <file/directory>
      Ex:
      chmod –R 755 testappl

191. What are the files which contain apps password?
Ans :
1. wdbsrv.app@IAS_ORACLE_HOME/Apache/modplsql/cfg
2. CGIcmd.dat@806_ORACLE_HOME/reports60/server
3. wfmail.cfg@FND_TOP/resource - optional
4. CatalogLoader.conf@OA_JAVA - optional
5. CatalogLoader.xml@OA_HTML - optional

192. What is the script to find out ICM status?
Ans : afimchk.sql@FND_TOP/sql

193. What is the script to list the concurrent request status?
Ans: afrqrun.sql@FND_TOP/sql

194. What is the script that Lists managers that currently are running a request?
Ans : afcmrrq.sql@FND_TOP/sql

195) How can I determine whether a template is customizable or non-customizable?
Answer: If a keyword "LOCK" is present at the end of the file entry in the respective driver, then it is a non-customizable template. If the "LOCK" keyword is not seen, then that template can be customized.

How to find high CPU using sessions in Oracle Database

If the problem relates to CPU bound applications then CPU information for each session can be examined to determine the culprits. The v$sesstat  view can be queried to find high cpu using sessions and then SQL can be listed.

1.Find the 'CPU used by this session' statistic.

SQL>SELECT name ,statistic# FROM v$statname WHERE  name LIKE '%CPU%session';

    NAME                                STATISTIC#
    ----------------------------------- ----------
    CPU used by this session                    14

2. Then determine which session is using most of the cpu.

SQL>SELECT * FROM v$sesstat WHERE statistic# = 14;

           SID STATISTIC#      VALUE
    ---------- ---------- ----------
             1         14          0
             2         14          0
             3         14          0
             4         14          0
             5         14          0
             6         14          0
             7         14          0
             8         14          0
             9         14          0
            10         14          0
            11         14          0
            12         14          0
            16         14       1930

3. Lookup details for the session which is using most of the cpu.

SQL>SELECT address ,SUBSTR(sql_text,1,20) Text, buffer_gets, executions,buffer_gets/executions AVG
    FROM   v$sqlarea a, v$session s
    WHERE  sid = 16
    AND    s.sql_address = a.address
    AND    executions > 0
    ORDER BY 5;

4. Use v$sqltext to extract the whole SQL text.

set pages 50000 lines 32767
col SPID for a10
col PROGRAM for a15
col OSUSER for a10
col ACTION for a10
col EVENT for a25
col SQL_TEXT for a25
col MACHINE for a10
col P1TEXT for a10
col P2TEXT for a10
col P3TEXT for a10
SELECT b.sid, b.serial#, a.spid, b.sql_id, b.program, b.osuser, b.machine, b.type,
b.event, b.action, b.p1text, b.p2text, b.p3text, b.state, c.sql_text,b.logon_time
FROM v$process a, v$session b, v$sqltext c WHERE a.addr=b.paddr
AND b.sql_hash_value = c.hash_value
AND b.sid = '&sid' ORDER BY a.spid, c.piece
/

5. Once the whole SQL statement has been identified it can be tuned.

Explain the queries and examine their access paths.
Autotrace is a useful tool for examining access paths.

Syntax:-

Explain plan for 'sql statement';

How to Move ASM DATABASE FILES from ONE DISKGROUP TO ANOTHER

Steps to Move ASM DATABASE FILES from ONE DISKGROUP TO ANOTHER

Since ASM files cannot be accessed through normal operating system interfaces, RMAN is the preferred means of copying ASM file.

The steps to moving a datafile from a diskgroup to another is as below, using RMAN.

1) Identify the datafile to be moved.
2) Identify the diskgroup on to which the datafile has to be moved.
3) Take the datafile offline.
4) Copy the datafile to new diskgroup using Either RMAN or DBMS_FILE_TRANSFER.
5) Rename the datafile to point to new location.
6) Recover the datafile.
7) Bring the datafile online.
8) Verify the new datafile locations.
9) Delete the datafile from its original location.

1) Identify the datafile to be moved.

SQL> SELECT FILE_NAME FROM DBA_DATA_FILES;
+ASMDISK2/orcl/datafile/users.256.565313879 <======= Move this to ASMDISK1.
+ASMDISK1/orcl/sysaux01.dbf
+ASMDISK1/orcl/undotbs01.dbf
+ASMDISK1/orcl/system01.dbf

2) Identify the diskgroup on to which the datafile has to be moved.

SQL> SELECT GROUP_NUMBER, NAME FROM V$ASM_DISKGROUP;
GROUP_NUMBER NAME
------------ ---------
1 ASMDISK1
2 ASMDISK2

3) Take the datafile offline.
SQL> ALTER DATABASE DATAFILE '+ASMDISK2/orcl/datafile/users.256.565313879' OFFLINE;

4) Copy the datafile to new diskgroup using Either RMAN or DBMS_FILE_TRANSFER.

   a)   DBMS_FILE_TRANSFER package or
   b)   RMAN
       
a).Using DBMS_FILE_TRANSFER package 
           
SQL> create or replace directory orcl1 as '+ASMDISK1/orcl/datafile';

SQL> Alter disgroup ASMDISK2 add directory  '+ASMDISK2/test';
       
SQL> create or replace directory orcl2 as '+ASMDISK2/test';

SQL>
BEGIN
     DBMS_FILE_TRANSFER.COPY_FILE(
     source_directory_object => 'ORCL1',
     source_file_name => 'users.259.565359071',
     destination_directory_object => 'ORCL2',
     destination_file_name => 'USERS01.DBF');
END;  
Database altered.

--------------------  OR   --------------------
  
b).Using RMAN copy the file to new diskgroup.

$ rman target /

connected to target database: ORCL (DBID=1020304050)

RMAN> COPY DATAFILE '+ASMDISK2/orcl/datafile/users.256.565313879' TO '+ASMDISK1';

Starting backup at 03-AUG-98
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=+ASMDISK2/orcl/datafile/users.256.565313879
output filename=+ASMDISK1/orcl/datafile/users.259.565359071 tag=TAG19980803T12110
9 recid=2 stamp=565359071
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 03-AUG-98

5) Rename the datafile to point to new location.

If you have used DBMS_FILE_TRANSFER (method 4 a)) use the following command to rename:
SQL> ALTER DATABASE RENAME FILE '+ASMDISK2/orcl/datafile/users.256.565313879' TO                                  

'+ASMDISK1/orcl/datafile/users.259.565359071';

Database altered.

If you have used RMAN (method 4 b) use the following option of RMAN
RMAN>
run
{
set newname for datafile '+ASMDISK2/orcl/datafile/users.256.565313879'
to '+ASMDISK1/orcl/datafile/users.259.565359071';
switch datafile all;
}

6) Recover the datafile.

SQL> RECOVER DATAFILE '+ASMDISK1/orcl/datafile/users.259.565359071';
Media recovery complete.

7) Bring the datafile online.

SQL> ALTER DATABASE DATAFILE '+ASMDISK1/orcl/datafile/users.259.565359071' ONLINE;
Database altered.

8) Verify the new datafile locations.

SQL> SELECT FILE_NAME FROM DBA_DATA_FILES;

FILE_NAME
-------------------------------------------------------------------------------
+ASMDISK1/orcl/datafile/users.259.565359071
+ASMDISK1/orcl/sysaux01.dbf
+ASMDISK1/orcl/undotbs01.dbf
+ASMDISK1/orcl/system01.dbf

9) Delete the datafile from its original location.

SQL> ALTER DISKGROUP ASMDISK2 DROP FILE users.256.565313879;
or
ASMCMD> rm -rf <filename>


Note:-
====

The steps provided above assume that the database is open and in Archivelog mode.
Besides these steps are not appropriated for system or sysaux datafiles.

For System and Sysaux an approach similar to the one given below can be used:-

1. Create a Copy of datafile in target Diskgroup
RMAN> backup as copy tablespace system format '<New DG>';
RMAN> backup as copy tablespace sysaux format '<New DG>';

2. Then shutdown the database and restart to a mounted state
RMAN> shutdown immediate;
RMAN> startup mount;

3. switch the datafiles to the copy
RMAN> switch tablespace system to copy;
RMAN> switch tablespace sysaux to copy;

4. Recover the changes made to these tablespaces
RMAN> recover database;

Note:-
====

Most ASM files do not need to be manually deleted because, as Oracle managed files, they are removed automatically

when they are no longer needed.

However, if you need to drop an Oracle Managed File (OMF) manually you should use the fully qualified filename if

you reference the file. Otherwise you will get an error (e.g. ORA-15177).

ALTER DISKGROUP ASMDISK2 DROP FILE '+ASMDISK2/orcl/datafile/users.256.565313879';

Patch 13621679 - 11.1.0.7.11 Patch Set Update

Patchset/PSU    Patch Number    Description
11.1.0.7.11    13621679    DATABASE PATCH SET UPDATE 11.1.0.7.11 (INCLUDES CPU APR2012)

Patch 13621679 - 11.1.0.7.11 Patch Set Update
=============================================
http://bbs.dbsupport.cn/thread-160-1-1.html

Patch InformationPatch Set Update (PSU) patches are cumulative. That is, the content of all previous PSUs is included in the latest PSU patch.

PSU 11.1.0.7.11 includes all fixes previously included in PSU 11.1.0.7.10 and those listed in .
To install the PSU 11.1.0.7.11 patch, the Oracle home must have the 11.1.0.7.0 Database installed. Subsequent PSU patches can be installed on Oracle Database 11.1.0.7.0 or any PSU with a lower 5th numeral version than the one being installed.

OPatch Utility
==============
You must use the OPatch utility version 11.1.0.8.2 or later to apply this patch. Oracle recommends that you use the latest released OPatch 11.1, which is available for download from My Oracle Support patch6880880 by selecting the 11.1.0.0.0 release.
For information about OPatch documentation, including any known issues, see My Oracle Support Note 293369.1 OPatch documentation list.

PSU Patching 11G Steps on PRIMARY database
==========================================

check DR sysnc status
=====================
http://select-star-from.blogspot.in/2013/09/data-guard-sync-status.html

------------------------------------------------------------------------------------------------------------------

Start OEM black out
===================

Confirmation
Blackout "Blackout-Apr 24 1998 9:33:42 PM" created successfully, Blackout status will be propagated to the target(s) shortly.

------------------------------------------------------------------------------------------------------------------

Node 1: (AS ORACLE)
===================

$ hostname
$ uname
$ bash
$ date
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs
 root  344170       1   0   Apr 24      -  0:00 /oradb/crs/product/11.1.0/crs_1/bin/oclskd.bin

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012
$ opatch version
$ /oradb/crs/product/11.1.0/crs_1/OPatch/opatch version
$ /oradb/asm/product/11.1.0/asm_1/OPatch/opatch version
$ /oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch version
$ which opatch
/oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch
. oraenv  --- asm instance
$ opatch lsinventory
. oraenv  --- database 1 instance
$ opatch lsinventory
. oraenv  --- database 2 instance
$ opatch lsinventory

$ ps -ef| grep pmon
$ srvctl status database -d <database_1>                       or srvctl status instance -d <database_1> -i <database_1_instance>
$ srvctl status database -d <database_2>                       or srvctl status instance -d <database_2> -i <database_2_instance>

$ srvctl status asm -n <Node 1>
$ srvctl status asm -n <Node 2>

$ srvctl status listener -n <Node 1>
$ srvctl status listener -n <Node 2>

$ srvctl status nodeapps -n <Node 1>
$ srvctl status nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ srvctl stop database -d <database_1>              or srvctl stop insatnce -d <database_1> -i <database_1_instance>
$ srvctl stop database -d <database_2> -o immediate or srvctl stop insatnce -d <database_2> -i <database_2_instance>

$ srvctl stop asm -n <Node 1>
$ srvctl stop asm -n <Node 2>

$ srvctl stop listener -n <Node 1>
$ srvctl stop listener -n <Node 2>

$ srvctl stop nodeapps -n <Node 1>
$ srvctl stop nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl stop agent   or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl stop agent

$ ps -ef| grep oracle
$ lsnrctl stop     -------------------------------to stop default listener
$ ps -ef| grep oracle

------------------------------------------------------------------------------------------------------------------

Node 2: (AS ORACLE)
===================

$ hostname
$ uname
$ bash
$ date
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs
 root  344170       1   0   Apr 24      -  0:00 /oradb/crs/product/11.1.0/crs_1/bin/oclskd.bin

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012

$ opatch version

$ /oradb/crs/product/11.1.0/crs_1/OPatch/opatch version
$ /oradb/asm/product/11.1.0/asm_1/OPatch/opatch version
$ /oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch version

$ which opatch
/oradb/app/oracle/product/11.1.0/db_1/OPatch/opatch

$ ps -ef| grep pmon

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl stop agent   or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl stop agent

$ ps -ef| grep oracle
$ lsnrctl stop     -------------------------------to stop default listener
$ ps -ef| grep oracle

------------------------------------------------------------------------------------------------------------------

Pre-rootpatch
=============

Node 1: (AS ROOT)
=================

# cd /oradb/crs/product/11.1.0/crs_1/bin
# ./crsctl check cluster
# ./crsctl check crs
# ./crsctl stop crs

# ps -ef| grep oracle

# ssh <Node 2>

# cd /oradb/crs/product/11.1.0/crs_1/bin
# ./crsctl check cluster
# ./crsctl check crs
# ./crsctl stop crs

# ps -ef| grep oracle

# exit
# hostname
<Node 1>

# ps -ef| grep pmon   ---- (0 instances running)
# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
# sh custom/scripts/prerootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1 -crsuser oracle

# ssh <Node 2>

# ps -ef| grep pmon   ---- (0 instances running)
# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
# sh custom/scripts/prerootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1 -crsuser oracle

# exit
# hostname
<Node 1>

------------------------------------------------------------------------------------------------------------------

Pre-patch
=========

Node 1: (AS ORACLE)
===================

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/scripts/prepatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/server/11724953/custom/scripts/prepatch.sh -dbhome /oradb/asm/product/11.1.0/asm_1

$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953
$ custom/server/11724953/custom/scripts/prepatch.sh -dbhome /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

Opatch Apply
============

Node 1: (AS ORACLE)
===================

CRS
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077
$ opatch napply -oh /oradb/crs/product/11.1.0/crs_1 -id 11724953

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server
$ opatch napply custom/server/ -oh /oradb/asm/product/11.1.0/asm_1 -id 11724953

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server
$ opatch napply custom/server/ -oh /oradb/app/oracle/product/11.1.0/db_1 -id 11724953

------------------------------------------------------------------------------------------------------------------

Do the post configure for CRS and RDBMS home
============================================

Node 1: (AS ORACLE)
===================

CRS
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts
$ ./postpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server/11724953
$ ./custom/scripts/postpatch.sh -dbhome /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/server/11724953
$ ./custom/scripts/postpatch.sh -dbhome /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

11.1.0.7.11 PSU (13621679) and (9734685)
========================================

Node 1: (AS ORACLE)
===================

13621679
========

$ cd /oradb/app/admin/orasw/patches
$ chmod -R 777 PSU_patches
$ cd PSU_patches
$ pwd
/oradb/app/admin/orasw/patches/PSU_patches

$ ls -lrt p13621679_111070_AIX5L.zip
$ unzip p13621679_111070_AIX5L.zip
$ ls -lrt
$ cd 13621679
$ pwd

ASM
===
$ /oradb/app/admin/orasw/patches/PSU_patches/13621679
$ opatch apply -oh /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/13621679
$ opatch apply -oh /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

9734685
=======

$ cd /oradb/app/admin/orasw/patches/PSU_patches/
$ ls -lrt p9734685_1110711_AIX64-5L.zip
$ unzip p9734685_1110711_AIX64-5L.zip

ASM
===
$ cd /oradb/app/admin/orasw/patches/PSU_patches/9734685
$ opatch apply -oh /oradb/asm/product/11.1.0/asm_1

RDBMS
=====
$ cd /oradb/app/admin/orasw/patches/PSU_patches/9734685
$ opatch apply -oh /oradb/app/oracle/product/11.1.0/db_1

------------------------------------------------------------------------------------------------------------------

After 9734685 Patch applied relink has to be done in both ASM and DB homes ------- Linking Oracle
==========================================================================

Node 1: (AS ORACLE)
===================

RDBMS
=====
$ cd /oradb/app/oracle/product/11.1.0/db_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

ASM
===
$ cd /oradb/asm/product/11.1.0/asm_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

Node 2: (AS ORACLE)
=======

RDBMS
=====
$ cd /oradb/app/oracle/product/11.1.0/db_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

ASM
===
$ cd /oradb/asm/product/11.1.0/asm_1/rdbms/lib
$ make -f ins_rdbms.mk ipc_g ioracle

------------------------------------------------------------------------------------------------------------------

Post-rootpatch
==============

Node 1: (AS ROOT)
=================

# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953

# /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts/postrootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

# ps -ef | grep pmon  --- Both the db and asm instances will be up and running

# ssh <Node 2>

# cd /oradb/app/admin/orasw/patches/PSU_patches/Jan2012/CRS/111077/11724953/custom/scripts/postrootpatch.sh -crshome /oradb/crs/product/11.1.0/crs_1

# ps -ef | grep pmon  --- Both the db and asm instances will be up and running

------------------------------------------------------------------------------------------------------------------

Start ASM

------------------------------------------------------------------------------------------------------------------

Run on each database, only in one node
======================================

NODE 1: (AS ORACLE)
===================

$ sqlplus /nolog
Enter user-name: / as sysdba
Connected to an idle instance.

SQL>startup;
SQL>@?/rdbms/admin/catbundle.sql psu apply
SQL>exit

$. oraenv ---- --- database 2 instance
$ sqlplus /nolog
Enter user-name: / as sysdba

SQL>startup;
SQL>@?/rdbms/admin/catbundle.sql psu apply
SQL>exit

------------------------------------------------------------------------------------------------------------------

Patch Validation:-
================

opatch lsinventory | grep 11724953
opatch lsinventory | grep 13621679
opatch lsinventory | grep 9734685

start local listeners, (if any)

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check cluster

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ ./crsctl check crs

$ cd /oradb/crs/product/11.1.0/crs_1/bin
$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------

$ hostname
$ uname
$ bash
$ date
$ who -b
$ uptime
$ df -kg or df -h
$ cat /etc/oratab or cat/var/opt/oracle/oratab  ------> Note ASM and Database Home Paths
$ ps -ef| grep pmon
$ ps -ef| grep tns
$ ps -ef| grep emagent
$ ps -ef|grep oracle
$ ps -ef| grep d.bin
$ ps -ef|grep crs

$ ps -ef| grep pmon

$ srvctl status database -d <database_1>                      
$ srvctl status database -d <database_2>                      

$ srvctl status asm -n <Node 1>
$ srvctl status asm -n <Node 2>

$ srvctl status listener -n <Node 1>
$ srvctl status listener -n <Node 2>

$ srvctl status nodeapps -n <Node 1>
$ srvctl status nodeapps -n <Node 2>

$ ps -ef|grep oracle or  ps -ef|grep emagent
./emctl status agent or  /oradb/app/oracle/product/agent1/agent11g/bin/emctl status agent

Run on each database only in one node
=====================================

NODE 1: (AS ORACLE)
===================
SQL> select name db_name,DB_UNIQUE_NAME,instance_name,status,DATABASE_STATUS,open_mode, database_role, host_name,platform_id,version db_version,log_mode,flashback_on ,protection_mode,protection_level,LOGINS,to_char(STARTUP_TIME,'DD-MON-YYYY HH24:MI:SS') "UP TIME" from gv$instance,v$database;

SQL> select to_char(sysdate,'yyyy-mm-dd HH24:MI:SS') from dual
/

SQL>
col ACTION_TIME for a12
col NAMESPACE for a10
col VERSION for a10
col BUNDLE_SERIES for a15
col COMMENTS for a20
select action_time, action,namespace, version, id,bundle_series, comments from registry$history;

------------------------------------------------------------------------------------------------------------------

check DR sysnc status
=====================
http://select-star-from.blogspot.in/2013/09/data-guard-sync-status.html

------------------------------------------------------------------------------------------------------------------

Stop OEM black out
==================

Confirmation
Request to stop blackout "Blackout-Apr 24 1998 9:33:42 PM" was processed successfully, Blackout status will be propagated to the target(s) shortly.

------------------------------------------------------------------------------------------------------------------

Oracle DBA Interview Questions and Answers - RAC - 2

Oracle RAC Interview Questions and Answers

1. Where are the Clusterware files stored on a RAC environment?
The Clusterware is installed on each node (on an Oracle Home) and on the shared disks (the voting disks and the CSR file)

2. Where are the database software files stored on a RAC environment?
The base software is installed on each node of the cluster and the
database storage on the shared disks.

3. What kind of storage we can use for the shared Clusterware files?
- OCFS (Release 1 or 2)
- raw devices
- third party cluster file system such as GPFS or Veritas

4. What kind of storage we can use for the RAC database storage?
- OCFS (Release 1 or 2)
- ASM
- raw devices
- third party cluster file system such as GPFS or Veritas

5. What is a CFS?
A cluster File System (CFS) is a file system that may be accessed (read and write) by all members in a cluster at the same time. This implies that all members of a cluster have the same view.

6. What is an OCFS2?
The OCFS2 is the Oracle (version 2) Cluster File System which can be used for the Oracle Real Application Cluster.

7. Which files can be placed on an Oracle Cluster File System?
- Oracle Software installation (Windows only)
- Oracle files (controlfiles, datafiles, redologs, files described by the bfile datatype)
- Shared configuration files (spfile)
- OCR and voting disk
- Files created by Oracle during runtime
Note: There are some platform specific limitations.

8. Do you know another Cluster Vendor?
HP Tru64 Unix, Veritas, Microsoft

9. How is possible to install a RAC if we don’t have a CFS?
This is possible by using a raw device.

10. What is a raw device?
A raw device is a disk drive that does not yet have a file system set up. Raw devices are used for Real Application Clusters since they enable the sharing of disks.

11. What is a raw partition?
A raw partition is a portion of a physical disk that is accessed at the lowest possible level. A raw partition is created when an extended partition is created and logical partitions are assigned to it without any formatting. Once formatting is complete, it is called cooked partition.

12. When to use CFS over raw?
A CFS offers:
- Simpler management
- Use of Oracle Managed Files with RAC
- Single Oracle Software installation
- Autoextend enabled on Oracle datafiles
- Uniform accessibility to archive logs in case of physical node failure
- With Oracle_Home on CFS, when you apply Oracle patches CFS guarantees that the updated Oracle_Home is visible to all nodes in the cluster.
Note: This option is very dependent on the availability of a CFS on your platform.

13. When to use raw over CFS?
- Always when CFS is not available or not supported by Oracle.
- The performance is very, very important: Raw devices offer best performance without any intermediate layer between Oracle and the disk.
Note: Autoextend fails on raw devices if the space is exhausted. However the space could be added online if needed.

14. What CRS is?
Oracle RAC 10g Release 1 introduced Oracle Cluster Ready Services (CRS), a platform-independent set of system services for cluster environments. In Release 2, Oracle has renamed this product to Oracle Clusterware.

15. What is VIP IP used for?
It returns a dead connection IMMIDIATELY, when its primary node fails. Without using VIP IP, the clients have to wait around 10 minutes to receive ORA-3113: “end of file on communications channel”. However, using Transparent Application Failover (TAF) could avoid ORA-3113.

16. Why we need to have configured SSH or RSH on the RAC nodes?
SSH (Secure Shell,10g+) or RSH (Remote Shell, 9i+) allows “oracle” UNIX account connecting to another RAC node and copy/ run commands as the local “oracle” UNIX account.

17. Is the SSH, RSH needed for normal RAC operations?
No. SSH or RSH are needed only for RAC, patch set installation and clustered database creation.

18. Do we have to have Oracle RDBMS on all nodes?
Each node of a cluster that is being used for a clustered database will typically have the RDBMS and RAC software loaded on it, but not actual data files (these need to be available via shared disk).

19. What are the restrictions on the SID with a RAC database? Is it limited to 5 characters?
The SID prefix in 10g Release 1 and prior versions was restricted to five characters by install/ config tools so that an ORACLE_SID of up to max of 5+3=8 characters can be supported in a RAC environment. The SID prefix is relaxed up to 8 characters in 10g Release 2, see bug 4024251 for more information.

20. Does Real Application Clusters support heterogeneous platforms?
The Real Application Clusters do not support heterogeneous platforms in the same cluster.

21. Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network?
RAC and Clusterware deployment best practices suggests that the interconnect (private connection) be deployed on a stand-alone, physically separate, dedicated switch. On big network the connections could be instable.

22. What is the Load Balancing Advisory?
To assist in the balancing of application workload across designated resources, Oracle Database 10g Release 2 provides the Load Balancing Advisory. This Advisory monitors the current workload activity across the cluster and for each instance where a service is active; it provides a percentage value of how much of the total workload should be sent to this instance as well as service quality flag.

23. How many nodes are supported in a RAC Database?
With 10g Release 2, we support 100 nodes in a cluster using Oracle Clusterware, and 100 instances in a RAC database. Currently DBCA has a bug where it will not go beyond 63 instances. There is also a documentation bug for the max-instances parameter. With 10g Release 1 the Maximum is 63.

24. What is the Cluster Verification Utiltiy (cluvfy)?
The Cluster Verification Utility (CVU) is a validation tool that you can use to check all the important components that need to be verified at different stages of deployment in a RAC environment.

25. What versions of the database can I use the cluster verification utility (cluvfy) with?
The cluster verification utility is release with Oracle Database 10g Release 2 but can also be used with Oracle Database 10g Release 1.

26. If I am using Vendor Clusterware such as Veritas, IBM, Sun or HP, do I still need Oracle Clusterware to run Oracle RAC 10g?
Yes. When certified, you can use Vendor Clusterware however you must still install and use Oracle Clusterware for RAC. Best Practice is to leave Oracle Clusterware to manage RAC. For details see Metalink Note 332257.1 and for Veritas SFRAC see 397460.1.

27. Is RAC on VMW are supported?
Yes.

28. What is hangcheck timer used for ?
The hangcheck timer checks regularly the health of the system. If the system hangs or stop the node will be restarted automatically.

There are 2 key parameters for this module:

-> hangcheck-tick: this parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30seconds.

-> hangcheck-margin: this defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node.

29. Is the hangcheck timer still needed with Oracle RAC 10g?
Yes.

30. What files can I put on Linux OCFS2?
For optimal performance, you should only put the following files on Linux OCFS2:
- Datafiles
- Control Files
- Redo Logs
- Archive Logs
- Shared Configuration File (OCR)
- Voting File
- SPFILE

31. Is it possible to use ASM for the OCR and voting disk?
No, the OCR and voting disk must be on raw or CFS (cluster file system).

32. Can I change the name of my cluster after I have created it when I am using Oracle Clusterware?
No, you must properly uninstall Oracle Clusterware and then re-install.

33. What the O2CB is?
The O2CB is the OCFS2 cluster stack. OCFS2 includes some services. These services must be started before using OCFS2 (mount/ format the file systems).

34. What the OCR file is used for?
OCR is a file that manages the cluster and RAC configuration.

35. What the Voting Disk file is used for?
The voting disk is nothing but a file that contains and manages information of all the node memberships.

36. What is the recommended method to make backups of a RAC environment?
RMAN to make backups of the database, dd to backup your voting disk and hard copies of the OCR file.

37. What command would you use to check the availability of the RAC system?
crs_stat -t -v (-t -v are optional)

38. What is the minimum number of instances you need to have in order to create a RAC?
You can create a RAC with just one server.

39. Name two specific RAC background processes
RAC processes are: LMON, LMDx, LMSn, LKCx and DIAG.

40.Can you have many database versions in the same RAC?
Yes, but Clusterware version must be greater than the greater database version.

41. What was RAC previous name before it was called RAC?
OPS: Oracle Parallel Server

42. What RAC component is used for communication between instances?
Private Interconnect.

43. What is the difference between normal views and RAC views?
A RAC view has the prefix ‘G’. For example, GV$SESSION instead of V$SESSION

44. Which command will we use to manage (stop, start) RAC services in command-line mode?
srvctl

45. How many alert logs exist in a RAC environment?
One for each instance.

46. What are Oracle Clusterware Components
Voting Disk — Oracle RAC uses the voting disk to manage cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The voting disk must reside on shared disk.

Oracle Cluster Registry (OCR) — Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster

47. How do you backup voting disk
#dd if=voting_disk_name of=backup_file_name

48. How do I identify the voting disk location
#crsctl query css votedisk

49. How do I identify the OCR file location
check /var/opt/oracle/ocr.loc or /etc/ocr.loc ( depends upon platform)
or
#ocrcheck

50. How do you backup the OCR
There is an automatic backup mechanism for OCR. The default location is : $ORA_CRS_HOME\cdata\"clustername"\
To display backups :
#ocrconfig -showbackup
To restore a backup :
#ocrconfig -restore
With Oracle RAC 10g Release 2 or later, you can also use the export command:
#ocrconfig -export -s online, and use -import option to restore the contents back.
With Oracle RAC 11g Release 1, you can do a manaual backup of the OCR with the command:
# ocrconfig -manualbackup

51. What is SCAN?
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters (RAC) 11g Release 2 feature that provides a single name for clients to access an Oracle Database running in a cluster. The benefit is clients using SCAN do not need to change if you add or remove nodes in the cluster.

52. What are Oracle Clusterware processes for 10g on Unix and Linux
Cluster Synchronization Services (ocssd) — Manages cluster node membership and runs as the oracle user; failure of this process results in cluster restart.

Cluster Ready Services (crsd) — The crs process manages cluster resources (which could be a database, an instance, a service, a Listener, a virtual IP (VIP) address, an application process, and so on) based on the resource's configuration information that is stored in the OCR. This includes start, stop, monitor and failover operations. This process runs as the root user

Event manager daemon (evmd) —A background process that publishes events that crs creates.

Process Monitor Daemon (OPROCD) —This process monitor the cluster and provide I/O fencing. OPROCD performs its check, stops running, and if the wake up is beyond the expected time, then OPROCD resets the processor and reboots the node. An OPROCD failure results in Oracle Clusterware restarting the node. OPROCD uses the hangcheck timer on Linux platforms.

RACG (racgmain, racgimon) —Extends clusterware to support Oracle- specific requirements and complex resources. Runs server callout scripts when FAN events occur.

53. What are Oracle database background processes specific to RAC
•LMS—Global Cache Service Process
•LMD—Global Enqueue Service Daemon
•LMON—Global Enqueue Service Monitor
•LCK0—Instance Enqueue Process
To ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a query or transaction, Oracle RAC instances use two processes, the Global Cache Service (GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the statuses of each data file and each cached block using a Global Resource Directory (GRD). The GRD contents are distributed across all of the active instances.

54. How do you troubleshoot node reboot
Please check metalink ...
Note 265769.1 Troubleshooting CRS Reboots
Note.559365.1 Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions.

55. Is ssh required for normal Oracle RAC operation ?
"ssh" are not required for normal Oracle RAC operation. However "ssh" should be enabled for Oracle RAC and patchset installation.

56. What is the purpose of Private Interconnect ?
Clusterware uses the private interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster.

57. Why do we have a Virtual IP (VIP) in Oracle RAC?
Without using VIPs or FAN, clients connected to a node that died will often wait for a TCP timeout period (which can be up to 10 min) before getting an error. As a result, you don't really have a good HA solution without using VIPs.

When a node fails, the VIP associated with it is automatically failed over to some other node and new node re-arps the world indicating a new MAC address for the IP. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

58. What do you do if you see GC CR BLOCK LOST in top 5 Timed Events in AWR Report?
This is most likely due to a fault in interconnect network.
Check netstat -s
if you see "fragments dropped" or "packet reassemblies failed" , Work with your system administrator find the fault with network.

59. How many nodes are supported in a RAC Database?
10g Release 2, support 100 nodes in a cluster using Oracle Clusterware, and 100 instances in a RAC database.

60. Srvctl cannot start instance, I get the following error PRKP- 1001 CRS-0215, however sqlplus can start it on both nodes? How do you identify the problem?
Set the environmental variable SRVM_TRACE to true.. And start the instance with srvctl. Now you will get detailed error stack.

61. What is the purpose of the ONS daemon?
The Oracle Notification Service (ONS) daemon is an daemon started by the CRS clusterware as part of the nodeapps. There is one ons daemon started per clustered node.
The Oracle Notification Service daemon receive a subset of published clusterware events via the local evmd and racgimon clusterware daemons and forward those events to application subscribers and to the local listeners.

This in order to facilitate:

a. the FAN or Fast Application Notification feature or allowing applications to respond to database state changes.

b. the 10gR2 Load Balancing Advisory, the feature that permit load balancing accross different rac nodes dependent of the load on the different nodes. The rdbms MMON is creating an advisory for distribution of work every 30seconds and forward it via racgimon and ONS to listeners and applications.

Script to Gather data from the Linux OS

#####################################################################
###   Unix script os_stats.sh                                     ###
###   Designed to be run periodically to collate information      ###
###   START OF SCRIPT                                             ###
#####################################################################
#
LOG_FILE="OS_`hostname`_`date '+%m%d%y_%H%M'`.txt"
#
echo "**********************************************" >$LOG_FILE
date >> $LOG_FILE
echo "Running as  `id`" >> $LOG_FILE
echo "**********************************************" >>$LOG_FILE
echo "uname -a" >>$LOG_FILE
uname -a >>$LOG_FILE
cat /etc/issue >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ulimit -a" >>$LOG_FILE
ulimit -a >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ulimit -Ha" >>$LOG_FILE
ulimit -Ha >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "Netstat -i" >>$LOG_FILE
netstat -i >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "netstat -an">> $LOG_FILE
netstat -an >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "netstat -s">> $LOG_FILE
netstat -s >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "sar -u 5 3">> $LOG_FILE
sar -u 5 3 >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "sar -q 5 3">> $LOG_FILE
sar -q 5 3 >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ps -e -w -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args" >> $LOG_FILE
ps -e -ww -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "iostat -t -x" >> $LOG_FILE
iostat -t -x >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "df -h" >> $LOG_FILE
df -h >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "mpstat 5 3" >> $LOG_FILE
/usr/bin/mpstat 5 3 >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "vmstat 5 3" >> $LOG_FILE
vmstat 5 3 >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "free -m -s 5 -c 3" >> $LOG_FILE
free -m -s 5 -c 3 >>$LOG_FILE
grep MemTotal /proc/meminfo >>$LOG_FILE
grep SwapTotal /proc/meminfo >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "IPCS data" >> $LOG_FILE
ipcs -l >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs -u >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs >> $LOG_FILE
echo "            ----------------------------         " >> $LOG_FILE
ipcs -t >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
#####################################################################
###   END OF SCRIPT                                               ###
#####################################################################