Installing a Virtualized Oracle 10g R2 RAC cluster using CentOS 5.4 Virtual Machines on VMware ESX 3.5
Last updated: 19-September-2010

This guide was written to be a step by step walkthrough to installing an Oracle 10g R2 RAC cluster, on CentOS 5.4 virtual machines, in VMware ESX 3.5. This guide was written with the novice in mind, however the more experience you have with Oracle the better. The RAC cluster you'll build by following this guide should not be used for production purposes. RAC is not supported in virtualized environments, and Oracle Database is not supported on CentOS. Furthermore, the database configured in this article is not appropriate for production use (ie, the redo logs and control files are not multiplexed). With that being said, creating a virtualized RAC cluster is an inexpensive way to become familiar with RAC concepts, administration, and installation procedures.

This guide does not serve as a replacement for the documentation, found at http://www.oracle.com/technology/documentation/index.html. Reviewing the documentation is helpful in achieving an understanding of RAC and its various components, however learning to install RAC from scratch is time consuming. Hopefully this walkthrough will expedite the process.

Sections

Pre-requisites
Hypervisor Configuration and VM Creation
CentOS 5.4 installation
Virtual Machine Storage Configuration
Clusterware and RAC pre-installation procedures
Installing Oracle Clusterware
Installing Oracle 10g R2 Database
Creating a database with DBCA
RAC Verification

Pre-requisites

The following is required in order for you to succeed in following this guide:
-An operational Vmware ESX 3.5 server
-roughly 50GB of free space
-3GB of RAM
-The CentOS 5.4 64-bit installation iso file (freely available from www.centos.org)
-The Oracle installation files (available for download at otn.oracle.com) for the 10g R2 Database and Clusterware 64-bit
-The VMs are running CentOS 5.4 x64, and the Oracle version is 64-bit as well. You can try the 32-bit versions if you're not running a 64-bit system, but there may be some differences in the installation procedures.
-Networking requirements as described below.

We will need 6 IP addresses for the RAC cluster. 2 public communication IP addresses, 2 VIPs, and 2 Interconnect IP addresses. The public communication IP addresses and VIP addresses need to be on the same segment, and the private addresses need to be on their own segment. This is how it looks in my network:

Hypervisor Configuration and VM Creation

Each RAC node requires two network connections; one for public communication, and another for the cluster interconnect. In order to isolate interconnect traffic, we will create a virtual switch as shown below. It's likely that the interconnect will work without a separate vswitch, however I haven't tested this.

Log into the VMware ESX host using the VI client, select the host, and click the configuration tab.

Click Networking in the Hardware box.

Click "Add Networking..." in the upper right corner of the pane.

Select Virtual Machine as the connection type. Click the Next button.

Uncheck vmnic1. Click Next.

I labeled the network "RAC Interconnect". You can label it differently if you want to. Click Next.

Click Finish on the Summary screen in order to complete the process.

Now we will create our virtual machines. Right click on the ESX host and click New Virtual Machine.

Select Typical. Click Next.

Type node1.example.com. Click next.

Select the Datastore for the VM and click next.

Click the Linux and select RHEL 5 64-bit. Click Next.

Select 1 virtual processor. Click Next.

I've used 2GB of memory for this VM. You can use more or less if you want to, but I would suggest a minimum of 1GB of memory. Click Next.

Configure two NICs as shown below and click Next.

I've configured a 20GB disk. You can use less if you want to, I wouldn't recommend less than 10GB. 20GB leaves a good amount of extra space. Click Next.

Make sure everything is correct on the Ready to Complete screen, and click Finish. Repeat the process for node 2, but use node2.example.com as the name of the virtual machine instead.

CentOS 5.4 installation

Right click the node1.example.com VM and click Power On.

I would rather install the OS from the console of the VM, but it's up to you. Click Open Console if you want to do this as well.

I downloaded Cent OS 5.4 in the form of an ISO. In order to mount the ISO click the Connect CD/DVD button. Browse to the ISO file and double click it.

In order to boot the VM from the ISO, click the VM menu item then click Send Ctl-Alt-Del.

Now the VM will boot from the mounted ISO. From the console of the VM press enter to begin the installation.

I will skip the media check because I know it's valid. Feel free to do the media check if you want to.

Click Next.

Select a different language if you want to, and click Next. Click Yes when the warning appears.

The Default drive layout works fine, but if you want to create your own drive layout feel free to do so. If you do use your own drive layout, allocate 1.5 times the size of ram to Swap if you have 1-2GB of memory. If you have more than 2GB, allocate an amount of Swap that equals how much RAM the VM has. Click Next. Click Yes when the warning appears.

Next will be our network configuration. Enable both interfaces for activate on boot. Edit the interfaces. Disable IPv6 support. Manually configure static IP addresses on the interfaces.The first interface will be configured with the public communication IP, and the second will be configured with the Interconnect IP. Manually set the hostname to node1.example.com. Once completed the configuration should look similar to mine. Click Next.

Select a region that corresponds to your time zone and click Next.

Enter your root password and click Next.

Unselect "Desktop - Gnome". A window manager adds unnecessary overhead, so I remove it. Click Next.

At the next screen, click next to begin the install. When the installation is complete, click the reboot button. When the VM is restarted, the Setup Agent will run. We will be editing the Firewall configuration and System services. In Firewall configuration, use the disable both the security level and SELinux settings.

In System Services, disable bluetooth, cups, ip6tables, and iptables. Hit OK, then hit Exit for the changes to be saved.

The CentOS installation is now complete on node1. Repeat the process on node2, but make sure to use the correct hostname and IP address. The hostname for node2 is node2.example.com

Virtual Machine Storage Configuration

I'll be configuring the shared disks as follows. In a production environment you would want to have more than one voting disk, but since this is just for testing we'll be using just one voting disk.

OCR - 512MB
Voting Disk - 512MB
ASM1 - 2GB
ASM2 - 2GB
ASM3 - 2GB

Creating shared disks on Vmware ESX host

You'll need to install vcli in order to configure the shared disks. Thank you Cody Bunch over at professionalvmware.com for the article that helped me out with this: http://professionalvmware.com/2010/04/shared-vmdks-on-vsphere-esx-and-esxi/. VCLI can be downloaded here: http://www.vmware.com/downloads/login.do.

Once you have VCLI installed, cd to the following directory:

C:\Program Files\VMware\VMware vSphere CLI\bin>

Run the following commands to create each disk file on the ESX host. You'll need to substitute the IP after -server with the IP of your ESX host, and use the correct path for your data store. I placed my shared disks in a folder called racstorage. If you want to create your own folder you can do so by using the Datastore Browser. This will take a while.

vmkfstools.pl -server 192.168.2.253 -c 512M -d eagerzeroedthick -a lsilogic /vmfs/volumes/484c5c53-856e0f27-8dfa-00e0814ad80f/racstorage/ocr.vmdk
vmkfstools.pl -server 192.168.2.253 -c 512M -d eagerzeroedthick -a lsilogic /vmfs/volumes/484c5c53-856e0f27-8dfa-00e0814ad80f/racstorage/votingdisk.vmdk
vmkfstools.pl -server 192.168.2.253 -c 2G -d eagerzeroedthick -a lsilogic /vmfs/volumes/484c5c53-856e0f27-8dfa-00e0814ad80f/racstorage/asm1.vmdk
vmkfstools.pl -server 192.168.2.253 -c 2G -d eagerzeroedthick -a lsilogic /vmfs/volumes/484c5c53-856e0f27-8dfa-00e0814ad80f/racstorage/asm2.vmdk
vmkfstools.pl -server 192.168.2.253 -c 2G -d eagerzeroedthick -a lsilogic /vmfs/volumes/484c5c53-856e0f27-8dfa-00e0814ad80f/racstorage/asm3.vmdk

Once the disks are created, you will then add them to each virtual machine. A new SCSI controller will need to be created, and the physical option will need to be selected so that the drives can be shared. Select independent mode, and persistent. This way the disks won't be affected when you do snapshots.

Shutdown the virtual machines by executing shutdown -h now on each virtual machine. Once they are powered off right click node1 and select edit settings.

Click the Add button.

Select Hard Disk and click Next.

Select Use an existing virtual disk and click next.

Click browse and select the ocr.vmdk disk file. Click Next.

Under Virtual Device Node, select 1:0. This will create a new disk controller. Select Independent mode, and persistent. Click Next.

Review Ready to Complete and click Finish. Select the New SCSI Controller and select Physical.

There are four more drives to add. Repeat the drive adding process, and use an incrementing device node ID. Once you're done, you'll need to repeat this process on node 2. Make sure that the drives have matching device node IDs on each RAC node.

They should then map to the following devices:

Ocr - /dev/sdb
Voting disk - /dev/sdc
Asm1 - /dev/sdd
Asm2 - /dev/sde
Asm3 - /dev/sdf

Start up the nodes. They should now have the drives attached to them. A listing of the devices should be similar to the following.

[root@node1 ~]# ls -l /dev/sd*
brw-r----- 1 root disk 8, 0 May 22 11:13 /dev/sda
brw-r----- 1 root disk 8, 1 May 22 11:13 /dev/sda1
brw-r----- 1 root disk 8, 2 May 22 11:13 /dev/sda2
brw-r----- 1 root disk 8, 16 May 22 11:13 /dev/sdb
brw-r----- 1 root disk 8, 32 May 22 11:13 /dev/sdc
brw-r----- 1 root disk 8, 48 May 22 11:13 /dev/sdd
brw-r----- 1 root disk 8, 64 May 22 11:13 /dev/sde
brw-r----- 1 root disk 8, 80 May 22 11:13 /dev/sdf

The OCR and voting disk will be configured on raw devices, and the ASM1 , ASM2, and ASM3 disks will use ASM.

Clusterware and RAC pre-installation procedures

Do the following on each node.

Configuring /etc/hosts

Make sure /etc/hosts has entries for the public, private, and vip addresses as listed below.

Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

#public
192.168.2.227 node1 node1.example.com
192.168.2.228 node2 node2.example.com

#private
10.0.0.1 node1-priv node1-priv.example.com
10.0.0.2 node2-priv node2-priv.example.com

#vip
192.168.2.224 node1-vip node1-vip.example.com
192.168.2.225 node2-vip node2-vip.example.com

Creating required groups and users:

groupadd -g 200 oinstall
groupadd -g 201 dba
useradd -u 200 -g oinstall -G dba oracle
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/crs/oracle/product/10/crs
chown -R root:oinstall /u01/crs
chmod -R 775 /u01/crs/oracle
passwd oracle

Transfer media to the node you'll be installing from - I used winscp for this, because it is simple. Whatever method you use, use the oracle account so that it will have permissions to access the media. Once the media is transferred, uncompress it with the following commands:

gunzip 10201_database_linux_x86_64.cpio.gz
cpio -idmv < 10201_database_linux_x86_64.cpio

Do the same thing with the clusterware media.

Configuring SSH on All cluster nodes

On each node run the following to generate the RSA and DSA keys:

su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

The following commands will append the public keys to the authorized_keys file. Since this is for testing I recommend leaving the passphrase blank for user equivalency. If you do use a password you'll need to run the ssh-agent and ssh-add commands listed in a following section. run the following on node1:

touch ~/.ssh/authorized_keys
cd ~/.ssh

ssh node1 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh node1 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh node2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh node2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys

Our keys have been appended to the authorized_keys file, and now we must copy this file to the second node.

scp authorized_keys node2:/home/oracle/.ssh

run the following on each node:

chmod 600 authorized_keys

run the following on node1 to enable ssh user equivalency, so that you can login without a password:

exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add

Then enter the password you originally configured for the key.

Add the following to .bash_profile for the oracle user:

export EDITOR=vi
export CRS_HOME=/u01/crs/oracle/product/10/crs
export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
export ORACLE_BASE=/u01/app/oracle
export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export ORACLE_SID=ORCL

Install cvuqdisk - this is installed so that cvu can discover shared disks. Run the following on node1 as root.

cd /home/oracle/clusterware/rpm/
export CVUQDISK_GRP=oinstall
rpm -iv cvuqdisk-1.0.1-1.rpm
scp cvuqdisk-1.0.1-1.rpm node2:/root

Run the following on node2 as root.

cd /root
export CVUQDISK_GRP=oinstall
rpm -iv cvuqdisk-1.0.1-1.rpm

Install required software packages:

yum install gcc-c++.x86_64 systat.x86_64 compat-libstdc++-33.x86_64 \
firefox.x86_64 vnc-server.x86_64 xorg*.x86_64 elfutils-libelf-devel.x86_64 \
libaio-devel.x86_64 xorg-x11-fonts-Type1 -y

yum install glibc-devel.i386 compat-libstdc++-33.i386 compat-libstdc++-296.i386 \
libXp.i386 libXtst.i386 libXt.i386 libstdc++.i386 -y

yum install xterm libXp.i386 libXtst.i386 glibc-devel.i386 -y

Configure kernel parameters

{
echo
echo kernel.shmmni = 4096
echo kernel.sem = 250 32000 100 128
echo fs.file-max = 131072
echo net.ipv4.ip_local_port_range = 1024 65000
echo net.core.rmem_default=4194304
echo net.core.rmem_max=4194304
echo net.core.wmem_default = 262144
echo net.core.wmem_max = 262144
echo net.ipv4.tcp_wmem = 262144 262144 262144
echo net.ipv4.tcp_rmem = 262144 262144 262144
echo fs.aio-max-nr=3145728
echo kernel.shmmax=4294967296
echo
} >> /etc/sysctl.conf
sysctl -p

Set shell limits for oracle user

{
echo
echo "oracle soft nofile 131072"
echo "oracle hard nofile 131072"
echo "oracle soft nproc 131072"
echo "oracle hard nproc 131072"
echo "oracle soft core unlimited"
echo "oracle hard core unlimited"
echo "oracle soft memlock 50000000"
echo "oracle hard memlock 50000000"
echo
} >> /etc/security/limits.conf

modify /etc/pam.d/login

{
echo
echo         session    required     pam_limits.so
echo
} >> /etc/pam.d/login

modify /etc/profile

{
echo
echo if [ \$USER = "oracle" ]
echo         then
echo           ulimit -u 131072
echo            ulimit -n 131072
echo fi
echo
} >> /etc/profile

Install the hangcheck timer, on each node

Run the following as root, and add it to /etc/rc.d/rc.local:

insmod /lib/modules/2.6.18-164.el5/kernel/drivers/char/hangcheck-timer.ko hangcheck_tick=1 hangcheck_margin=10

Run the root pre script as root, on each node.

cd /home/oracle/clusterware/rootpre
./rootpre.sh

The message no OraCM installed is informative and can be ignored.

Next, we'll partition the disks we added to the VMs. These disks will be used for the OCR, the Voting Disk, and ASM.

fdisk /dev/sdb - create a new partition with N, then select primary, partition number 1, then type w to write changes. Do the same thing for /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf. fdisk -l should now list the new partitions. Run partprobe on the second node so that the new partitions are visible.

Bind partitions to raw devices for clusterware files, on each node:

add the following to /etc/sysconfig/rawdevices:

/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1

then run the following to assign the devices

service rawdevices restart

finally assign permissions to the raw devices:

chown root:oinstall /dev/raw/raw1
chown root:oinstall /dev/raw/raw2
chmod 640 /dev/raw/raw1
chmod 640 /dev/raw/raw2

Add the udev permission rules to the /etc/udev/rules.d/65-raw-permissions.rules file, for example:

# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

Configuring Disks for ASM

We still have 3 disks remaining, we will configure these as our ASM disks, for storing the Oracle database.

Install ASMlib

ASMlib can be downloaded from http://www.oracle.com/technology/tech/linux/asmlib/index.html

You'll need to install the asm lib RPMs for your kernel, which can be checked with uname -rm. On these nodes, uname -rm returns: 2.6.18-164.el5 x86_64, so I installed the following packages:

oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-support-2.1.3-1.el5.x86_64.rpm

The following command will install the rpms:

rpm -Uvh oracleasm-support-2.1.3-1.el5.x86_64.rpm \
oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm \
oracleasmlib-2.0.4-1.el5.x86_64.rpm

Configure ASMlib

Run /etc/init.d/oracleasm configure, and use the following parameters:

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y

Now, we will configure our 3 remaining disks to use ASMlib

create a whole disk partition on each of the disks using fdisk. Once the partitions have been created, run the following to mark the disks as ASM disks:

# /etc/init.d/oracleasm createdisk DISK1 /dev/sdd1
Marking disk "DISK1" as an ASM disk:                       [  OK  ]
# /etc/init.d/oracleasm createdisk DISK2 /dev/sde1
Marking disk "DISK2" as an ASM disk:                       [  OK  ]
# /etc/init.d/oracleasm createdisk DISK3 /dev/sdf1
Marking disk "DISK3" as an ASM disk:                       [  OK  ]

On node2, run the following so that the disks are available on node 2 as well:

/etc/init.d/oracleasm scandisks

Installing Oracle Clusterware

start a VNC session as the oracle user by typing:

vncserver

enter a password of your choice

Now, connect to the session with a client such as tight VNC client, using the syntax <ip address>:1

cd /home/oracle/clusterware
./runinstaller -ignoreSysPrereqs

Click Next on the welcome screen.

Click Next on specify inventory directory and credentials.

Under specify home details, enter this as the home path: /u01/crs/oracle/product/10/crs. Click Next.

All prerequisite checks should pass, as listed below. Click Next.

Click add, and fill in the node2 details. Click Next.

Click Edit, and specify that 192.168.2 is public. Click Next.

OCR - specify external redundancy, and use /dev/raw/raw1. Click Next.

Voting Disk - specify external redundancy, and use /dev/raw/raw2. Click Next.

Verify that everything looks correct in the summary screen and click Install. When the installation completes a window will appear requesting the execution of configuration scripts. We will do this shortly, but first we must edit a couple files. A workaround for a kernel bug was added using LD_ASSUME_KERNEL settings. The bug has since been fixed, and so this workaround is no longer valid and will cause installation issues unless it is removed.

The following will need to be done on all nodes, as root.

[root@node1 oracle]# cd /u01/crs/oracle/product/10/crs/bin/
[root@node1 bin]# vi vipca

Go to line 122 of vipca, and add a new line under export LD_ASSUME_KERNEL as shown below.

We also need to edit the srvctl file in the same directory. Add unset LD_ASSUME_KERNEL at around line 175 as shown below.

Now, run the scripts listed by the installer on both nodes, as root. Run the scripts on node 1 first, then run them on node 2. You'll see error messages on node 2 if the IP addresses you're using are not routable, that's ok. Now, start a vncsession on node 2 as root. When you're connected cd to /u01/crs/oracle/product/10/crs/bin, and run the following:

[root@node2 bin]# ./oifcfg setif -global eth0/192.168.2.0:public
[root@node2 bin]# ./oifcfg setif -global eth1/10.0.0.0:cluster_interconnect

From the same location, run vipca.

Select eth0 then click Next.

Configure the VIPs as shown below. Click Next.

Verify that everything looks correct on the Summary screen, then click finish. The configuration assistant will then configure the VIPs. Once that is successful go back to node 1, and click the retry button. The check should now pass, and you should see the following. Click Exit.

Installing Oracle 10g R2 Database

The RAC installation is nearly complete. From the vncsession on node 1, as the oracle user, run the following:

mkdir /home/oracle/tmp
export TMP=/home/oracle/tmp
cd ~/database
./runInstaller -ignoreSysPrereqs

Click Next.

Select Enterprise Edition and click Next.

Verify that the home details are correct and click Next.

Check off node2 and click Next.

Every check should pass. Click Next.

We will first configure ASM, and create the database later on. Select Configure Automatic Storage Management (ASM), enter the passwords you wish to use, and click Next.

Now we'll be using the three ASM disks we created earlier. Select External Redundancy, and check off all three disks. Click Next.

Verify the configuration, and click Install.

Next you'll see the following screen. Don't run these scripts just yet. We will need to edit srvctl on both nodes, but this time it will be in the bin directory of the Oracle Database home. Edit srvctl in /u01/app/oracle/product/10.2.0/db_1/bin as we did earlier in the clusterware installation. When you've done this, run the script as root on node1 first, then on node2.

The Oracle Database software has now been installed. We'll need to create a database next. Click Exit.

Creating a Database with DBCA

From the vncserver session on node1 running as the oracle user, type dbca to start the Database Configuration Assistant.

Select Oracle Real Application Clusters database and click Next.

Select Create a Database and Click Next.

Click Select All. Click Next.

Select General Purpose. Click Next.

Verify that the Management Options are configured as shown below. Click Next.

Because this is just for testing, I've used the same password for all accounts. Click Next once you've configured your passwords.

Select ASM as the storage mechanism used by the database, and click Next.

As you can see the Disk Group we created earlier is now available for use. Select the Disk Group and click Next.

Verify that Use Oracle-Managed Files is selected and click Next.

I chose not to use a Flash Recovery Area or to Enable Archiving, but only because this is a testing DB. In production I suggest you enable these options for recovery purposes.

Install the Sample Schemas if you want to, then click Next.

We won't be adding any services. Click Next.

I went with the Typical memory settings. Feel free to adjust this if you're comfortable doing so, and click Next.

Click Next on this page.

Click Finish, then click Ok when the Summary box pops up.

The database creation process will start.

You'll see this if the database creation is successful. Click Exit, and you're done with creating the database.

RAC Verification

You now officially have a running RAC cluster. Feel free to check to see if the Oracle processes are running as shown below.

As you can see, the instance name is different on each node. To make sure that the correct SID value is used, set the ORACLE_SID value to ORCL1 on node 1, and ORCL2 on node 2, by editing the /home/oracle/.bash_profile file. You can connect to Oracle and verify the status of the Oracle instances by executing the following. Additionally, edit /etc/oratab on each node so that ORCL1 and ORCL2 are used instead of simply ORCL.

[root@node1 ~]# su - oracle
[oracle@node1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Sun May 23 05:25:48 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> select instance_name, status, startup_time from gv$instance;

INSTANCE_NAME STATUS STARTUP_T
---------------- ------------ ---------
ORCL1 OPEN 23-MAY-10
ORCL2 OPEN 23-MAY-10

That is it! The RAC cluster is up and running and you are free to test it as you wish.

Questions? Comments? Email me at jmoracle1@gmail.com.