• Felsőfokú diploma (villamos mérnök vagy informatikus)
• Minimum erős középfokú angol nyelvtudás szóban és írásban
• Min. 2-3 éves, haladó szintű C++ fejlesztői tapasztalat

Feladat:

• Szoftver termékek fejlesztése, főleg C++-ban, illetve C-ben
• Operációs rendszer, clusterware, adatbázis-fejlesztés Linuxos környezetben

Előny:

• JAVA,
• CORBA
• Script nyelvek
• C++ Standard Template Library
• C++ Design Patterns
• IP technologiai ismeretek,
• Linux
• Free BSD

laszlo_spiller@kellyservices.hu

...download here: oracle-clusterware-10g

Es un dolor de cabeza, después de pasar toda esta semana varias veces empezando desde cero la instalación y configuración de 24 pasos nos quedamos siempre en el ultimo, porque la porqueria de wizard de oracle no encuentra el grupo de discos ASM. Les posteo los shots que los ultimos 3 pasos:

Primero el paso previo al bug cuando el wizard no encuentra los volumenes.

Le di clic en aceptar y luego espero un rato, como no me ha mostrado ningun volumen le doy clic al boton Change Discovery Path y escribo lo que dice el manual que deberia solucionar el problema.

Y el resultado es que sigue sin mostrar los volumenes

La gota que derramo el vaso fue este otro error, estuve buscando por horas en internet sin encontrar solucion, mas abajo les posteo la solucion.

Les dejo el link de 3 paginas para hacer su propia base de datos de alta disponibilidad con nodos enterprise linux por si quieren levantar una :)

http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html

Si al final lo han hecho es porque su ego puede mas que cualquier cosa, esto es solo para los no mortales! jaja.

Bueno, escribo actualizando este post nuevamente ya que el semestre esta por concluir y ya casi me estoy desocupando de todas las clases, excepto por Seminario 2 que no veo que haya modo que termine ya que aun no han dado proyecto, pero estuvo mejor ya que hubiera estado de mas desvelos mientras haciamos los otros proyectos (redes2, bases2, sopes2).

Pues la situacion es que queria dejar aclarado cual fue la solucion al error del wizard de oracle sobre los grupos de discos ASM (paso #24). Cito el texto original: If the volumes we created earlier in this article do not show up in the "Select Member Disks" window: (ORCL:VOL1, ORCL:VOL2, ORCL:VOL3, and ORCL:VOL4) then click on the "Change Disk Discovery Path" button and input "ORCL:VOL*".

Pues segun el famoso "ORCL:VOL*" no a todos les funciona, me di cuenta de eso luego de muchas horas de busquedas sin resultados reales para solucionar esto. Pues el truquillo consiste en colocar el path de los discos, algo asi: /dev/mnt/VOL* y como por arte de magia los lista.

Bueno hasta alli todo iba de maravilla hasta que recorde que solo a mi (y a joaquinais) nos suceden cualquier tipo de errores que puedan suceder, el caso es que luego de eso tuve otro error con el ASM. El motivo...? que al parecer hay un bug que lo pone en estado UNKNOWN cuando solo puede ser UP o DOWN, para arreglar eso hay que cambiarle el estado en modo forzado y luego subirlo. A continuacion la solucion:

# RECUERDA QUE HAY QUE LOGUEAR COMO USUARIO ORACLE PARA HACER TODO ESTO
# arrancar listener
lsnrctl start LISTENER_LINUX1

# arrancar asm, primero lo bajamos a modo forzado, luego lo arrancamos normalmente
cd /u01/app/crs/bin
crs_stop -f ora.linux1.ASM1.asm
srvctl start asm -n linux1

De alli en adelante no dio ningun otro error de instalacion el bendito proyecto.

Eso si, hay que hacer varias verificaciones, aca les posteo las que necesite verificar a la hora de la calificada:

#LOGUEAR COMO ORACLE PARA TODO ESTO

# checar listener
ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'

# checar estatus completo del nodo
srvctl status nodeapps -n linux1

# checar si existen los demonios necesarios
srvctl config nodeapps -n linux1 -a -g -s -l

# checar el servicio de asm
srvctl status asm -n linux1

# loguear en sql desde consola
sqlplus "/ as sysdba"
Por ultimo les dejo otra foto de las compus que usamos pero vistas desde atras. jeje :) andabamos emocionados con la camara ese dia. Y tambien grabamos video con la camara de joaquinais pero el webon no quiere editar el video para subirlo.

Para arrancar y parar automaticamente la base de datos pueden consultar este otro post que esta muy completo :)

http://ubuntulife.wordpress.com/repositorio-de-scripts/

To check the Active Version
Run the following command on the local node.

$ crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.3.0]
Note: The active version is the lowest software version running in a cluster.

To check the Software Version
Run the following command on the local node.

$ crsctl query crs softwareversion
CRS software version on node [racnod01] is [10.2.0.3.0]
Note: The software version is the binary version of the software on a particular cluster node.

Regards,
Sabdar Syed.
http://sabdarsyed.blogspot.com
http://syedracdba.wordpress.com
 

First of all let me tell you a little about our Oracle Database. It's installed on two servers which have Linux operating systems as clusterware structure. I mean there is a virtual disk on a data storage unit shared by these two servers.

Last week we noticed that cluster service didn't work properly while we were preparing to apply the latest Patchset. When we restart the servers, one of them couldn't start Listener. After a little research we found that it couldn't initialize Virtual IP daemon and for this reason Listener didn't start. So we followed a method that initialize VIP manually. The method is from Metalink 295328.1 document. Shortly, fist we set some environments and run a cluster script called racgvip manually. By this method we managed to initialize VIP and dependently start Listener but Cluster Service didn't detect that they were working.

So far our database was serving to clients until we shutdown all services, instances, nodeapps etc. and used "ocrconfig -repair /../../ocrDisk " command. From this point on we couldn't manage to restart any cluster service, instance nor nodeapp.

We tried many things to deal with the problem like

* restoring ocrDisk from backups --> # ocrconfig -restore /crs_home/cdata/crs/backup.ocr

* reinstalling cluster software by using Oracle Installer

* adding VoteDisk and removing the current

....

while we were trying these, sometimes we met with permission problems. We noticed from crs logs (/crs_home/log/<hosname>/client/..logs) that permission problems about /crs_home/bin/racgwrap occurred . After giving needed permissions, logs began to give errors like "clsssInitNative: connect failed, rc 9" which means nothing.

I thought that if there was one permission error, there would be many. So we intended to give execute permission to everybody for /crs_home/bin directory recursively. After this process, we finally managed to start crs services so the others.

After all we discovered that our security admins had taken back all permissions from others, for /oracle_home directory recursively in order to increase the security level which costs sleepless 36 hours.

Operating System: Enterprise Linux 4 U5 (RHEL4 U5)

Oracle: 10.2.0.1

Clusterware: 10.2.0.1

Cluster Interconnect: Ethernet (private connection)

Problem:

While running the root.sh script on the last cluster node, during the CLusterware Installation, the following error message occurs.

# /u01/app/oracle/product/10.2.0/crs/root.sh

WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root

WARNING: directory '/u01/app/oracle/product' is not owned by root

Checking to see if Oracle CRS stack is already configured

/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root

WARNING: directory '/u01/app/oracle/product' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node :
node 1: xen1 xen1-priv xen1

node 2: xen2 xen2-priv xen2

node 3: xen3 xen3-priv xen3

node 4: xen4 xen4-priv xen4

clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 90 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

xen1

xen2

xen3

xen4

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

Timed out waiting for the CRS stack to start.
Solutions:

I found more than one issue that can help solving the problem.


 Use a dedicated Switch for the Cluster Interconnect.
Set the Cluster Interconnetc MTU=1500 (f.e. change later to MTU=9000).
Add only two Cluster Nodes to the CRS with the initial installation and add the other Nodes with addNode.sh Script seperately to the Cluster.

I am not exactly sure which one has solved the probelm on my installation but I considered them all.
]]>



http://thomasvogt.wordpress.com/2007/11/21/clean-remove-oracle-clusterware-crs-10gr2-from-a-ystem/
Wed, 21 Nov 2007 14:39:40 +0000
thomasvogt
http://thomasvogt.wordpress.com/2007/11/21/clean-remove-oracle-clusterware-crs-10gr2-from-a-ystem/

The problem with Oracle Clusterware (also known as Cluster Ready Service - CRS) is, that there is no built-in mechanism from Oracle to clean remove the Clusterware and all of it's files distributed over the OS filesystem. The follwowing example script removes the Oracle Clusterware completely. The operating system is a RHEL4 U5.

This script has to be edited for personal use. $ORA_CRS_HOME is here under /opt/oracle. 


########### script ###############

#!/bin/bash

echo

echo "Remove the Oracle Clusterware Service ?"

echo

echo "Enter y[yes] or n[no] to exit"read comit

if [ $comit == "n" ]; then

echo "Exit from Script without any change..."

exit 1

else

echo "Start to Shutdown and Remove Oracle Clusterware ..."

echo

/etc/init.d/init.evmd stop

/etc/init.d/init.evmd disable

/etc/init.d/init.cssd stop

/etc/init.d/init.cssd disable

/etc/init.d/init.crsd stop

/etc/init.d/init.crsd disable

/etc/init.d/init.crs stop

/etc/init.d/init.crs disable

rm -rf /etc/oracle /etc/oraInst.loc /etc/oratab

rm -rf /etc/init.d/init.crsd /etc/init.d/init.crs /etc/init.d/init.cssd /etc/init.d/init.evmd

rm -rf /etc/rc2.d/K96init.crs /etc/rc2.d/S96init.crs etc/rc3.d/K96init.crs \

/etc/rc3.d/S96init.crs /etc/rc4.d/K96init.crs /etc/rc4.d/S96init.crs \

/etc/rc5.d/K96init.crs /etc/rc5.d/S96init.crs /etc/rc.d/rc0.d/K96init.crs \

/etc/rc.d/rc1.d/K96init.crs /etc/rc.d/rc6.d/K96init.crs /etc/rc.d/rc4.d/K96init.crs

cp /etc/inittab.orig /etc/inittab

rm -rf /etc/inittab.crs /etc/inittab.no_crs

rm -rf /tmp/*

rm -rf /tmp/.oracle

rm -rf /usr/local/bin/dbhome /usr/local/bin/oraenv /usr/local/bin/coraenv

rm -rf /var/tmp/.oracle

rm -rf /opt/oracle/*

echo

echo "Remove on one Node the Shared Devices"

echo "rm -rf /u03/oracrs/*"

echo

fi

########### end script ###############
After running that script on a system it should be possible to reinstall Oracle Clusterware without any problems.

Goal

The goal of this document is to help customers who have accidentally deleted the OCR, voting disk or the files that are required for the operation of Oracle clusterware.

Depending on the issue, it may or may not be good idea to execute the steps provided.

* OCR
o If the OCR has been deleted, then check if the OCR mirror is OK and vice versa. It may be prudent to use the OCR mirror to create the OCR. For steps on this check the documentation Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide
o If the OCR mirror and OCR have been deleted, then it may be faster to restore the OCR using the OCR backups. For steps on this check the documentation Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide
* Voting Disk
o If there are multiple voting disks and one was accidentally deleted, then check if there are any backups of this voting disk. If there are no backups then we can add one using the crsctl add votedisk command. The complete steps are in the Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide
* SCLS directories
o These are internal only directories which are created by root.sh, if this directory is accidentally removed then they can only be created by the steps documented below
* Socket files in /tmp/.oracle or /var/tmp/.oracle
o If these files are accidentally deleted, then stop the Oracle Clusterware on that node and restart it again. This will recreate these socket files. If the socket files for cssd is deleted then the Oracle Clusterware stack may not come down in which case the node has to be bounced.

Solution

If none of the steps documented above can be used to restore the file that was accidentally deleted or is corrupted, then the following steps can be used to re-create/reinstantiate these files. The following steps require complete downtime on all the nodes.

1. Shutdown the Oracle Clusterware stack on all the nodes using command crsctl stop crs as root user.
2. Backup the entire Oracle Clusterware home.
3. Execute /install/rootdelete.sh on all nodes
4. Execute /install/rootdeinstall.sh on the node which is supposed to be the first node
5. The following commands should return nothing
* ps -e | grep -i 'ocs[s]d'
* ps -e | grep -i 'cr[s]d.bin'
* ps -e | grep -i 'ev[m]d.bin'
6. Execute /root.sh on first node
7. After successful root.sh execution on first node Execute root.sh on the rest of the nodes of the cluster
8. Add the database to the OCR using the appropriate srvctl add database command as the user who owns the database. Please ensure that this is not run as root user
9. Add ASM, DB, Instance, services using approproate srvctl add commands. Please refer to the documentation for the exact commands.
10. Add the listener using netca. This may give errors if the listener.ora contains the entries already. If this is the case, move the listener.ora to /tmp from the $ORACLE_HOME/network/admin or from the $TNS_ADMIN directory if the TNS_ADMIN environmental is defined and then run netca. Add all the listeners that were added earlier.
11. execute cluvfy stage -post crsinst -n node1,node2 ### Please ensure to replace node1,node2 with the node names of the cluster

@initialize OCR re-initialize OCR reinitialize recreate OCR ==> Added for improved searchability

Symptoms

You are using Grid Control 10.2.0.1 to monitor a RAC database and clusterware.

Clusterware was installed (successfully) and the oracle software owner can ssh from one node to the other, and back again
without having to enter a password (i.e. ssh keys are correctly set up).
However, the OMS is reporting a critical violation for each db node, stating:

"Clusterware on the local host has problems. Verifying CRS integrity ERROR:
User equivalence unavailable on all the nodes. Verification cannot proceed.
Verification of CRS integrity was unsuccessful on all the nodes."

Navigation path is:

Targets > Click on the link under the Alerts critical column for the host in question i.e. the red number > click on Message > click on Message again to get the full text.
The navigation near the top of the screen shows
Host: All Metrics > Clusterware Status >

Cause

The clusterware software owner and agent owner are diffrent.
The agent owner cannot run /bin/cluvfy comp crs
Most lilkely this is a permission issue.

When perl tracing is enabled on the agent emagent_perl.trc shows this

crs_status.pl: Fri May 26 12:28:02 2006: DEBUG: ##crs home is: /u03/app/oracle/crs/10.2.0.1
crs_status.pl: Fri May 26 12:28:03 2006: ERROR: '/u03/app/oracle/crs/10.2.0.1/bin/cluvfy comp crs': 256
Verifying CRS integrity

Verification of CRS integrity was unsuccessful on all the nodes.

Solution

1. Make sure that the agent owner can run /bin/cluvfy comp crs succesfully
This is normally done by ensuring the agent owner is in the same OS group as the cluster software owner. However specific issues may be encountered.

2. Force a clearstate of the agent:

AGENT_HOME/bin/emctl clearstate agent

3. Force an upload:

AGENT_HOME/bin/emctl upload

4. This should clear the alerts

Symptoms

An OracleAS J2EE cluster has been configured against a file-based DCM repository.

"dcmctl getstate -v" returns the result, for example:

Component Type Up Status In Sync Status
================================

1 OC4J_Test OC4J Down Unknown: Repository Unavailable

An attempt to resync or update the configuration fails e.g.

"dcmctl resyncinstance -force -d -v -t 86400" errors with:

All OC4J components are stopped during a resyncInstance -force.
They will need to be restarted after the command completes.

ADMN-404003
TaskMaster is unable to evaluate the operation.
Resolution:
Check the base exception in the log file.
Base Exception:
The DCM repository is not currently available. The Oracle Application Server instance,
"midtier2.mfbas.wen.witt-weiden.de", is using a cached copy of the repository information. This
operation will update the repository, therefore the repository must be available.

"dcmctl updateconfig" errors with

ADMN-705021
The configuration files for this Oracle Application Server instance are inconsistent with the
configuration stored in the repository. In order to protect the repository, no further
configuration or deployment operations are allowed until the problem with the configuration on the
filesystem is resolved. This condition arises when a prior operation was unsuccessful. The
exception associated with this failed
operation is:

Cause

The problem can be caused by a corrupted cache repository. Look to see if the following error is occurring in the $ORACLE_HOME/dcm/logs/dcmctl_logs

123.45.6.78:35898:1156747626625:00

[ RM ] [Dsk] freeSlot failed to delete
/app/oracle/midtier/dcm/config/../diskcache/diskCache/global/211/2/1059035107

Solution

To resolve / work-around the issue remove the 'corrupted' dcm diskcache directory i.e.

1. opmnctl stopall
2. cp -R $ORACLE_HOME/dcm/diskCache diskCache_orig
3. rm -R $ORACLE_HOME/dcm/diskCache
4. Start the dcm daemon ("dcmctl start -admin"). This will result in the creation of a new diskcache directory structure.
5. "dcmctl getstate" should show that all components are in sync
6. opmnctl startall

Symptoms

Initially, ADMN-202500 was reported by DCM, and components were out of sync.

However, since it was stated that OC4J was not used, a previous action had the components removed,
in order to successfully start and get components in sync. The problem was thought to be that
Config File info and Repository info was incorrect on this OC4J.

The following shows components in sync, and each of these start up with opmn:

dcmctl getstate -v -d

1 home OC4J Up True
2 HTTP_Server HTTP_Server Up True

Cause

The DCM Repository on instance 2 became corrupted for some unknown reason , forcing the removal of the
repository and the dcmctl shell command was used to re-create the Instance.

However, when the dcmctl shell command was used, the repository was re-created as a standalone instance,
instead of a member of the existing farm.

Additionally instance #2 cannot join the farm because it is already a member of that farm in the host
repository. Also it cannot leave the farm because its own dcm repository says it is standalone.

Solution

1. Stop both instances and take a full backup of each.

2. Start the instance in node #1.

opmnctl startall

3. Destroy the instance #2 in the DCM repository as follow, executed directly from the instance in node #1:

dcmctl destroyinstance -i

4. Next, go into the $ORACLE_HOME/opmn/conf directory and check the ons.conf file.

5. Modify this file to make sure that only the active mid-tier ip_address and ports are listed.
The port is typically 6200.

6. Once this is completed, then review instance 1 to ensure that the cluster still exist and the application are listed.

On Node #1:
dcmctl listClusters
dcmctl whichfarm
dcmctl listInstances
dcmctl
listApplications
dcmctl getRepositoryid

Symptoms

In a 10g RAC/CRS environment, the ASM instance may terminate cleanly, with the only evidence being the shutdown information given in the alert log. For example, the following series of events may be evidenced in the alert log of the ASM instance (i.e. alert_+ASM1.log) even though no user has shutdown the ASM instance:

===========================
Mon Jun 21 21:25:39 2004
Shutting down instance: further logons disabled
Mon Jun 21 21:25:39 2004
Shutting down instance (immediate)
License high water mark = 4
Mon Jun 21 21:25:39 2004
SQL> ALTER DISKGROUP ALL DISMOUNT
NOTE: cache dismounting group 1/0x7D5B6B3B (DKGRP1)
Mon Jun 21 21:25:39 2004
===========================

As a side-effect of the ASM instance being shutdown, any database instance on that node will also be shutdown. The alert log from the database instance(s) might have messages such as the following:

===========================
Mon Jun 21 21:25:39 2007
Shutting down instance: further logons disabled
Mon Jun 21 21:25:39 2007
Stopping background process QMNC
License high water mark = 4
Mon Jun 21 21:26:39 2077
Stopping background process CJQ0
Mon Jun 21 21:26:39 2077
Stopping background process MMNL
Mon Jun 21 21:26:39 2077
Stopping background process MMON
Mon Jun 21 21:26:39 2077
Shutting down instance (transactional local)
Local transactions complete. Performing immediate shutdown
===========================

Or, an error may be signalled in the database instance alert log such as:

==========================
ORA-15064: communication failure with ASM instance
ORA-01089: immediate shutdown in progress - no operations are permitted
Mon Jun 21 19:56:37 2007
ASMB: terminating instance due to error 15064
==========================

Changes

Happens in a RAC environment running Oracle10g CRS (Cluster Ready Services)

Cause

In an Oracle RAC Environment, running Oracle CRS (Cluster Ready Services) the crsd daemon (or OracleCRService on Windows) is responsible for monitoring resources associated with the cluster. One of these resources is the Virtual IP address, or VIP, associated with each node. CRS creates dependencies on the VIP - specifically, the Listener service, ASM instance and each database instance on a given node are dependent on the VIP.

If the VIP fails for any reason and cannot be restarted, CRS will bring down all dependent resources, including the Listener, ASM instance and database instance. CRS will attempt to bring these resources down gracefully - hence, a shutdown immediate will be issued, and will be seen in the alert log of the ASM instance - no errors will be evident in the alert log for the ASM instance.

To verify that this is the cause, one must check the log files for CRS and the VIP. The main logfile for CRS can be found in the ORA_CRS_HOME/crs/log directory. There will be a log file in that directory with the node name of that particular node. To confirm if the ASM instance shutdown was precipitated by a VIP failure, check the logfile for errors associated with VIP, such as the following:

===========================
ora.nodename.vip on nodename went OFFLINE unexpectedly

For example, the log file may look something like this:

===========================
`ora.rmsclnxclu1.vip` on `rmsclnxclu1` went OFFLINE unexpectedly
2004-06-21 21:21:05.562: Attempting to stop `ora.rmsclnxclu1.vip` on member `rmsclnxclu1`
RTD #0: Action Script /home/oracle/product/crs/bin/racgwrap(stop) timed out for ora.rmsclnxclu1.vip! (timeout=60)
2004-06-21 21:22:16.472: [RTI:884782] StopResource error for ora.rmsclnxclu1.vip error code = 1
2004-06-21 21:22:18.611: `ora.rmsclnxclu1.vip` on member `rmsclnxclu1` has experienced an unrecoverable failure.
2004-06-21 21:22:18.611: Human intervention required to resume its availability.
2004-06-21 21:22:18.790: [RUNNABLELISTENER:884782] Resource failed into UNKNOWN, killing dependents
`ora.rmsclnxclu1.vip` experienced a failure on `rmsclnxclu1`. Stopping dependent resources.
2004-06-21 21:22:20.525: Attempting to stop `ora.gofod.gofod1.inst` on member `rmsclnxclu1`
2004-06-21 21:25:38.531: Stop of `ora.gofod.gofod1.inst` on member `rmsclnxclu1` succeeded.
2004-06-21 21:25:38.611: Attempting to stop `ora.rmsclnxclu1.LISTENER_rmsclnxclu1.lsnr` on member `rmsclnxclu1`
2004-06-21 21:25:38.983: Stop of `ora.rmsclnxclu1.LISTENER_rmsclnxclu1.lsnr` on member `rmsclnxclu1` succeeded.
2004-06-21 21:25:39.041: Attempting to stop `ora.rmsclnxclu1.ASM1.asm` on member `rmsclnxclu1`
2004-06-21 21:25:46.669: Stop of `ora.rmsclnxclu1.ASM1.asm` on member `rmsclnxclu1` succeeded.
2004-06-21 21:25:46.728: Attempting to stop `ora.rmsclnxclu1.vip` on member `rmsclnxclu1`
2004-06-21 21:25:55.547: Stop of `ora.rmsclnxclu1.vip` on member `rmsclnxclu1` succeeded.

Fix

To resolve this problem, the root cause of the failure of the VIP must be uncovered. The trace file for the vip will be found in the ORA_CRS_HOME/racg/dump directory - the file will be named: 'ora.nodename.vip.trc' - where nodename is the name of the node.

Check for entries associated with the time of the failure found in the CRS logfile noted above. For example, you may see errors such as:

===============
timeout: killed the spawned process
MON JUN 21 21:21:05 2004
clsrcexecut: cmd = /home/oracle/product/crs/bin/racgeut 54 /home/oracle/product/crs/bin/racgvip check rmsclnxclu1
MON JUN 21 21:21:05 2004
clsrcexecut: rc = 99, time = 56.640s
===============

This could very likely be due to current known issues with timeouts of the VIP under a high load. This issue is logged as BUG#3718601. A possible workaround to this issue is to increase the timeout values for the VIP (whose default value is 60 seconds) to a value of 120. This can be accomplished via a three-step process, as documented below. Theses steps must be executed as root, from the ORA_CRS_HOME/bin directory:

• Create the .cap file for each vip resource (on each node):
  .crs_stat -p ora.rmsclnxclu1.vip > /tmp/ora.rmsclnxclu1.vip.cap

• Then, update the .cap file using the following syntax and values:
  ./crs_profile -update ora.rmsclnxclu1.vip -dir /tmp -o ci=120,st=120
  (Where: ci=the CHECK_INTERVAL and st=the SCRIPT_TIMEOUT value.)

• Finally, re-register it using the '-u' option:
  ./crs_register ora.rmsclnxclu1.vip -dir /tmp -u

Installing the 10.2.0.2 Patch Set in a RAC installation on any Unix platform does not correcly update the libknlopt.a file on all nodes. The local node where the installer is run does update libknlopt.a but remote nodes do not get the updated file.

This can lead to dumps or internal errors on the remote Nods if Oracle is subsequently relinked. eg: Attempts to use PLSQL may dump in kkxcms or kkxsyn

The workaround steps below should be considered as a mandatory post install steps for the 10.2.0.2 Patch Set.

Workaround:

Following the install of 10.2.0.2 on a RAC system:
- Ensure all instances are shut down
- Manually copy $ORACLE_HOME/rdbms/lib/libknlopt.a from the local node to all remote nodes and relink Oracle on all nodes.

eg: cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ioracle

# crsctl add votedisk /dev/raw/raw1

When the CRS stack is running and the add votedisk command or the crsctl remove votedisk command is executed it will fail. It will fail with a "Cluster is not in a ready state for online disk addition" message. This is because of internal bug 3972986. Dynamic addition and deletion of voting disk when the Clusterware is running is not possible with the fix for this bug.

Some people may mis-interpret the "Cluster is not in a ready state for online disk addition" message and try to run the command with the -force attribute when the stack is running to force the change. This is know to corrupt the OCR. It is not recommended to use the -force attribute to add or delete the voting disk when the Clusterware is running.

Workaround

Workaround to the problem other than applying the patch for bug 3972986 after it becomes available is to ensure that the add or remove the voting disk commands are executed only after bringing down the clusterware on all nodes. As mentioned in the Oracle documentation, this requires the use of -force option. The -force attribute can be safely used ONLY if the Clusterware is stopped on all the nodes in the Cluster.

If the voting disk has already been added using the -force option when the Oracle clusterware stack is running then it is recommended to stop the Clusterware on all the nodes and restore the OCR from the time before the votedisk was added or removed.

% crsctl add votedisk /dev/votedisk2.dbf

When the CRS stack is running and the add votedisk command or the crsctl remove votedisk command is executed, it will fail with a "Cluster is not in a ready state for online disk addition" message. This is because of an internal bug 3972986. Dynamic addition and deletion of voting disks when Clusterware is running is not possible without the fix for this bug.

Some people may mis-interpret the "Cluster is not in a ready state for online disk addition" message and try to run the command with the -force attribute when the stack is running to force the change. Tis is known to corrupt the OCR. It is not recommended to use the -force attribute to add or delete the voting disk when Clusterware is running.

Without the fix for bug 3972986, voting disks ** CANNOT **be added or deleted when the clusterware is running on any node.

The changes of hitting this issue is very high and unfortunately this is no clear symptom of the corruption caused by running the

In 10.1.0, the default value of RESTART_ATTEMPTS is set to 1 for the VIP resource. This setting will increase VIP downtime, leading to the database clients waiting longer until the VIP fails over.

Changing the default restart_attempts for faster VIP failover:

The following steps will need to e performed for all VIP resources within the cluster. It is strongly advised to take a backup of the OCT prior to performing these steps.

1. As oracle user, stop all instances and node applications.
% srvctl stop instance -d -l
% srvctl stop nodeapps -n

2. As root, extract the resource to be modified details from OCR to text.
# cd $ORA_CRS_HOME/bin
# ./crs_stat -p ora..vip > /tmp/ora..vip.cap

3. Update / Modify the .cap file obtained above. Although you can use a text editor to modify, it is recommended to use the command line to avoid typos (also useful for automation). The unit for ra is the number of attempts.
# ./crs_profile -update ora..vip -dir /tmp -o ra=0

(Note that ra stands for RESTART_ATTEMPTS)

4. Register the updated resource with OCR (notice the -u, meaning update an already registered resource).
# ./crs_register ora..vip -dir /tmp -u

5. Verify the change took effect.
#./crs_stat -p ora..vip | grep RESTART_ATTEMPTS

You can verify the value of RESTART_ATTEMPTS has been changed to 0.

6. Back to oracle user, and start the node applications, instances and other necessary resources.
% srvctl start nodeapps -n
% srvctl start instance -d -l

Make sure to repeat these steps for all VIP resources within the cluster.