In this post, we will add a node to our existing Oracle Clusterware.
Please click on the INDEX and browse for more interesting posts.
Existing node : Jack
New Node : Jill
NOTE : Due to resource crunch, we could add a node to a single node RAC system. The steps are however same for any nodes except the ssh setup, which will be discussed as we go through the document.
Steps for adding node in Oracle RAC (19c) environment :
For adding node to existing RAC environment, initially we need a setup Oracle RAC environment to add nodes.
Existing /etc/hosts file for One-Node RAC Setup :
[oracle@jack bin]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public
172.168.190.1 jack.infraxpertzz.com jack
# Private
192.168.190.1 jack-priv.infraxpertzz.com jack-priv
# Virtual
172.168.190.3 jack-vip.infraxpertzz.com jack-vip
# SCAN
172.168.190.10 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.20 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.30 crsprod-scan.infraxpertzz.com crsprod-scan
Configure new host entry in all nodes of /etc/hosts file :
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public
172.168.190.1 jack.infraxpertzz.com jack
172.168.190.2 jill.infraxpertzz.com jill
# Private
192.168.190.1 jack-priv.infraxpertzz.com jack-priv
192.168.190.2 jill-priv.infraxpertzz.com jill-priv
# Virtual
172.168.190.3 jack-vip.infraxpertzz.com jack-vip
172.168.190.4 jill-vip.infraxpertzz.com jill-vip
# SCAN
172.168.190.10 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.20 crsprod-scan.infraxpertzz.com crsprod-scan
172.168.190.30 crsprod-scan.infraxpertzz.com crsprod-scan
Creating groups and user on new node with same group and user id of existing nodes :
Add the below groups and user in the new node.
Make the groups similar to the existing setup
Groups : oinstall (primary group)
dba, asmoper,asmadmin,asmdba,oper,dba (secondary group)
#groupadd -g 1005 oinstall
#groupadd -g 1006 dba
#groupadd –g 1001 asmoper
#groupadd –g 1002 asmadmin
#groupadd –g 1003 asmdba
#groupadd –g 1004 oper
#useradd -u 54323 -g oinstall -G dba,asmoper,asmadmin,asmdba,oper,dba oracle
ASM library Installation and Configuration :
Install the below RPM packages in the new node
rpm –Uvh oracleasmlib-2.0.12-1.el7.x86_64
rpm –Uvh oracleasm-support-2.1.8-3.1.el7.x86_64
For other RPM packages, please check the LINK
Configuration and Check ASM disks :
Configure,Start and check ASM disks on the new node
[root@jill Packages]# oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hittingwithout typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done [root@jill Packages]# oracleasm init Creating /dev/oracleasm mount point: /dev/oracleasm Loading module "oracleasm": oracleasm Configuring "oracleasm" to use device physical block size Mounting ASMlib driver filesystem: /dev/oracleasm [root@jill Packages]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "DATA1" Instantiating disk "DATA2" Instantiating disk "DATA4" Instantiating disk "FRA8" Instantiating disk "FRA7" Instantiating disk "DATA5" Instantiating disk "DATA3" Instantiating disk "FRA6" [root@jill Packages]# oracleasm listdisks DATA1 DATA2 DATA3 DATA4 DATA5 FRA6 FRA7 FRA8
Create Directories
Create the necessary directories in the new node
mkdir -p /u01/app/19c/grid
mkdir -p /u01/app/oracle/product/19c/db_1
mkdir –p /u01/app/19c/gridbase
mkdir –p /u01/app/oracle/orabase
chown -R oracle:oinstall /u01
chmod -R 775 /u01/
Configure SSH for oracle user on all nodes :
Copy sshUserSetup.sh script to new node(jill) and execute it.
[oracle@jack gridbase]# cd /u01/app/19c/grid/deinstall
[oracle@jack gridbase]# scp sshUserSetup.sh oracle@jill:/home/oracle
oracle@jill's password:
sshUserSetup.sh 100% 32KB 31.6KB/s 00:00
Run sshUserSetup.sh
[oracle@jill ~]$ ./sshUserSetup.sh -hosts "jill" -user oracle -noPromptPassphrase The output of this script is also logged into /tmp/sshUserSetup_2021-07-26-12-25-53.log Hosts are jill user is oracle Platform:- Linux Checking if the remote hosts are reachable PING jill.infraxpertzz.com (172.168.190.2) 56(84) bytes of data. 64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=1 ttl=64 time=0.038 ms 64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=2 ttl=64 time=0.042 ms 64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=3 ttl=64 time=0.067 ms 64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=4 ttl=64 time=0.051 ms 64 bytes from jill.infraxpertzz.com (172.168.190.2): icmp_seq=5 ttl=64 time=0.044 ms --- jill.infraxpertzz.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4108ms rtt min/avg/max/mdev = 0.038/0.048/0.067/0.011 ms Remote host reachability check succeeded. The following hosts are reachable: jill. The following hosts are not reachable: . All hosts are reachable. Proceeding further... firsthost jill numhosts 1 The script will setup SSH connectivity from the host jill.infraxpertzz.com to all the remote hosts. After the script is executed, the user can use SSH to run commands on the remote hosts or copy files between this host jill.infraxpertzz.com and the remote hosts without being prompted for passwords or confirmations. NOTE 1: As part of the setup procedure, this script will use ssh and scp to copy files between the local host and the remote hosts. Since the script does not store passwords, you may be prompted for the passwords during the execution of the script whenever ssh or scp is invoked. NOTE 2: AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEGES TO THESE directories. Do you want to continue and let the script make the above mentioned changes (yes/no)? yes The user chose yes Please specify if you want to specify a passphrase for the private key this script will create for the local host. Passphrase is used to encrypt the private key and makes SSH much more secure. Type 'yes' or 'no' and then press enter. In case you press 'yes', you would need to enter the passphrase whenever the script executes ssh or scp. no The estimated number of times the user would be prompted for a passphrase is 2. In addition, if the private-public files are also newly created, the user would have to specify the passphrase on one additional occasion. Enter 'yes' or 'no'. yes The user chose yes Creating .ssh directory on local host, if not present already Creating authorized_keys file on local host Changing permissions on authorized_keys to 644 on local host Creating known_hosts file on local host Changing permissions on known_hosts to 644 on local host Creating config file on local host If a config file exists already at /home/oracle/.ssh/config, it would be backed up to /home/oracle/.ssh/config.backup. Removing old private/public keys on local host Running SSH keygen on local host Enter passphrase (empty for no passphrase): Enter same passphrase again: Generating public/private rsa key pair. Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: be:ba:8e:44:93:75:1b:0f:c7:b7:ce:4e:57:52:fa:87 oracle@jill.infraxpertzz.com The key's randomart image is: +--[ RSA 1024]----+ | | | . | | . + o . .| | o . * . . o | | + S . . o .| | . . . o = | | . . + E o| | . . . o . .| | ..+o. . | +-----------------+ Creating .ssh directory and setting permissions on remote host jill THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT. The script would create ~oracle/.ssh/config file on remote host jill. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup. The user may be prompted for a password here since the script would be running SSH on host jill. Warning: Permanently added 'jill,172.168.190.2' (ECDSA) to the list of known hosts. oracle@jill's password: Done with creating .ssh directory and setting permissions on remote host jill. Copying local host public key to the remote host jill The user may be prompted for a password or passphrase here since the script would be using SCP for host jill. oracle@jill's password: Done copying local host public key to the remote host jill The script will run SSH on the remote machine jill. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase. cat: /home/oracle/.ssh/known_hosts.tmp: No such file or directory cat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directory SSH setup is complete. ------------------------------------------------------------------------ Verifying SSH setup =================== The script will now run the date command on the remote nodes using ssh to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP, THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR PASSWORDS. If you see any output other than date or are prompted for the password, ssh is not setup correctly and you will need to resolve the issue and set up ssh again. The possible causes for failure could be: 1. The server settings in /etc/ssh/sshd_config file do not allow ssh for user oracle. 2. The server may have disabled public key based authentication. 3. The client public key on the server may be outdated. 4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle. 5. User may not have passed -shared option for shared remote users or may be passing the -shared option for non-shared remote users. 6. If there is output in addition to the date, but no password is asked, it may be a security alert shown as part of company policy. Append the additional text to the/sysman/prov/resources/ignoreMessages.txt file. ------------------------------------------------------------------------ --jill:-- Running /usr/bin/ssh -x -l oracle jill date to verify SSH connectivity has been setup from local host to jill. IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR. The script will run SSH on the remote machine jill. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase. Mon Jul 26 12:26:42 IST 2021 ------------------------------------------------------------------------ SSH verification complete.
Copy authorized keys to all nodes running under cluster environment
[oracle@jill .ssh]$ pwd
/home/oracle/.ssh
[oracle@jill .ssh]$ scp authorized_keys oracle@jack:/home/oracle/
oracle@jack's password:
authorized_keys 100% 484 0.5KB/s 00:00
[oracle@jack ~]$ cat authorized_keys >> .ssh/authorized_keys
[oracle@jack ~]$ cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCtedmNSm8amWPTQO7OM6UAdI+/fIQ8Uu4muNqb/vzg3b696AGiNVbHYJ2KcXfzyLLBC7fz22KbBhQv+BRtwogdwUC7FtEyGVoZc4FpgYPEOl20Ws+h0WuP1fN4TeoAaMimED652iLZisQb2xecb4GxDPO5JkOyzp6RlIxtXiJAeQ== oracle@jack.infraxpertzz.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCtedmNSm8amWPTQO7OM6UAdI+/fIQ8Uu4muNqb/vzg3b696AGiNVbHYJ2KcXfzyLLBC7fz22KbBhQv+BRtwogdwUC7FtEyGVoZc4FpgYPEOl20Ws+h0WuP1fN4TeoAaMimED652iLZisQb2xecb4GxDPO5JkOyzp6RlIxtXiJAeQ== oracle@jack.infraxpertzz.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCfgHCdG3yak6ozk9m4eXPK5G5CtR59l7f+a/vYiPP8w+Y/H+igL5H4bPQQEhBuevvtJIaNtQkEqIQGrxB6dIF+vFQpK3KomaqHc6foJWS9NHCEvoaPopDeWVAy/PMJU7Kl1738KRAfXYDIC69l4MHzDxx92ymrh7XRiOLrOuH+HQ== oracle@jill.infraxpertzz.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCfgHCdG3yak6ozk9m4eXPK5G5CtR59l7f+a/vYiPP8w+Y/H+igL5H4bPQQEhBuevvtJIaNtQkEqIQGrxB6dIF+vFQpK3KomaqHc6foJWS9NHCEvoaPopDeWVAy/PMJU7Kl1738KRAfXYDIC69l4MHzDxx92ymrh7XRiOLrOuH+HQ== oracle@jill.infraxpertzz.com
[oracle@jack .ssh]$ pwd
/home/oracle/.ssh
[oracle@jack .ssh]$ scp authorized_keys jill:/home/oracle/.ssh/
oracle@jill's password:
authorized_keys 100% 484 0.5KB/s 00:00
Copy these authorized keys from Jack to any other existing Nodes. This is the only place where we have extra steps. Rest everywhere you need to follow the same steps as in the document
Check Time Synchronization :
Run the below commands from all nodes.
Example for 1st node :
[oracle@jack bin]$ ssh jack date
Tue Jul 20 21:25:09 IST 2021
[oracle@jack bin]$ ssh jill date
Tue Jul 20 21:25:13 IST 2021
Run the same from your new node and check
Verify Cluster utility :
[oracle@jack bin]$ pwd
/u01/app/19c/grid/bin
[oracle@jack bin]$ ./cluvfy comp peer -n jill -refnode jack
Verifying Peer Compatibility ...
Verifying Physical memory ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 5.6877GB (5964032.0KB) 5.6877GB (5964032.0KB) matched
Verifying Available memory ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 4.711GB (4939860.0KB) 2.571GB (2695904.0KB) matched
Verifying Swap space ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 16GB (1.6777212E7KB) 16GB (1.6777212E7KB) matched
Verifying Free Space ...PASSED
Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /usr / 4.7529GB (4983808.0KB) 4.6016GB (4825088.0KB) matched
Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /var / 4.7529GB (4983808.0KB) 4.6016GB (4825088.0KB) matched
Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /etc / 4.7529GB (4983808.0KB) 4.6016GB (4825088.0KB) matched
Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /sbin / 4.7529GB (4983808.0KB) 4.6016GB (4825088.0KB) matched
Node Name Path Mount point Status Ref. node status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jill /tmp /tmp 5.1816GB (5433344.0KB) 5.1777GB (5429248.0KB) matched
Verifying User existence for "oracle" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill oracle(1000) oracle(1000) matched
Verifying Group existence for "dba" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill dba(1006) dba(1006) matched
Verifying Group existence for "oinstall" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill oinstall(1005) oinstall(1005) matched
Verifying Group membership for "oracle" in "oinstall (Primary)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill yes yes matched
Verifying Group membership for "oracle" in "dba" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill yes yes matched
Verifying Run level ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 5 5 matched
Verifying System architecture ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill x86_64 x86_64 matched
Verifying Kernel version ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 4.1.12-61.1.18.el7uek.x86_64 4.1.12-61.1.18.el7uek.x86_64 matched
Verifying Kernel param "semmsl" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 250 250 matched
Verifying Kernel param "semmns" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 32000 32000 matched
Verifying Kernel param "semopm" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 100 100 matched
Verifying Kernel param "semmni" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 128 128 matched
Verifying Kernel param "shmmax" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 4398046511104 4398046511104 matched
Verifying Kernel param "shmmni" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 4096 4096 matched
Verifying Kernel param "shmall" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 0 0 matched
Verifying Kernel param "file-max" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 6815744 6815744 matched
Verifying Kernel param "ip_local_port_range" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 9000 65500 9000 65500 matched
Verifying Kernel param "rmem_default" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 262144 262144 matched
Verifying Kernel param "rmem_max" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 4194304 4194304 matched
Verifying Kernel param "wmem_default" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 262144 262144 matched
Verifying Kernel param "wmem_max" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 1048576 1048576 matched
Verifying Kernel param "aio-max-nr" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 1048576 1048576 matched
Verifying Kernel param "panic_on_oops" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill 1 1 matched
Verifying Package existence for "kmod (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill kmod-20-21.el7 (x86_64) kmod-20-21.el7 (x86_64) matched
Verifying Package existence for "kmod-libs (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill kmod-libs-20-21.el7 (x86_64) kmod-libs-20-21.el7 (x86_64) matched
Verifying Package existence for "binutils" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill binutils-2.25.1-22.base.el7 binutils-2.25.1-22.base.el7 matched
Verifying Package existence for "compat-libcap1" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill compat-libcap1-1.10-7.el7 compat-libcap1-1.10-7.el7 matched
Verifying Package existence for "libgcc (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill libgcc-4.8.5-11.el7 (x86_64) libgcc-4.8.5-11.el7 (x86_64) matched
Verifying Package existence for "libstdc++ (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill libstdc++-4.8.5-11.el7 (x86_64) libstdc++-4.8.5-11.el7 (x86_64) matched
Verifying Package existence for "libstdc++-devel (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill libstdc++-devel-4.8.5-11.el7 (x86_64) libstdc++-devel-4.8.5-11.el7 (x86_64) matched
Verifying Package existence for "sysstat" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill sysstat-10.1.5-11.el7 sysstat-10.1.5-11.el7 matched
Verifying Package existence for "ksh" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill ksh-20120801-26.el7 ksh-20120801-26.el7 matched
Verifying Package existence for "make" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill make-3.82-23.el7 make-3.82-23.el7 matched
Verifying Package existence for "glibc (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill glibc-2.17-157.el7 (x86_64) glibc-2.17-157.el7 (x86_64) matched
Verifying Package existence for "glibc-devel (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill glibc-devel-2.17-157.el7 (x86_64) glibc-devel-2.17-157.el7 (x86_64) matched
Verifying Package existence for "libaio (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill libaio-0.3.109-13.el7 (x86_64) libaio-0.3.109-13.el7 (x86_64) matched
Verifying Package existence for "libaio-devel (x86_64)" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill libaio-devel-0.3.109-13.el7 (x86_64) libaio-devel-0.3.109-13.el7 (x86_64) matched
Verifying Package existence for "nfs-utils" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.3.0-0.33.0.1.el7 matched
Verifying Package existence for "smartmontools" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill smartmontools-6.2-7.el7 smartmontools-6.2-7.el7 matched
Verifying Package existence for "net-tools" ...PASSED
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
jill net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17.20131004git.el7 matched
Verifying Peer Compatibility ...PASSED
Verification of peer compatibility was successful.
CVU operation performed: peer compatibility
Date: Jul 20, 2021 9:29:58 PM
CVU home: /u01/app/19c/grid/
User: oracle
Verify new node pre-check :
[oracle@jack bin]$ $GRID_HOME/bin/cluvfy stage -pre nodeadd -n jill -fixup -verbose > /home/oracle/cluvfy_pre_nodeadd.txt
Above node addition pre-check has to get passed to add nodes to the existing One-Node RAC environment
Below is the output :
[oracle@jack ~]$ cat cluvfy_pre_nodeadd.txt
Verifying Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 5.6877GB (5964032.0KB) 8GB (8388608.0KB) failed
jill 5.6877GB (5964032.0KB) 8GB (8388608.0KB) failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 2.5013GB (2622796.0KB) 50MB (51200.0KB) passed
jill 4.7025GB (4930904.0KB) 50MB (51200.0KB) passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 16GB (1.6777212E7KB) 5.6877GB (5964032.0KB) passed
jill 16GB (1.6777212E7KB) 5.6877GB (5964032.0KB) passed
Verifying Swap Size ...PASSED
Verifying Free Space: jack:/usr,jack:/var,jack:/etc,jack:/sbin ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr jack / 4.6016GB 25MB passed
/var jack / 4.6016GB 5MB passed
/etc jack / 4.6016GB 25MB passed
/sbin jack / 4.6016GB 10MB passed
Verifying Free Space: jack:/usr,jack:/var,jack:/etc,jack:/sbin ...PASSED
Verifying Free Space: jack:/u01/app/19c/grid ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/19c/grid jack /u01 31.3945GB 6.9GB passed
Verifying Free Space: jack:/u01/app/19c/grid ...PASSED
Verifying Free Space: jack:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp jack /tmp 5.1777GB 1GB passed
Verifying Free Space: jack:/tmp ...PASSED
Verifying Free Space: jill:/usr,jill:/var,jill:/etc,jill:/sbin ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr jill / 4.7471GB 25MB passed
/var jill / 4.7471GB 5MB passed
/etc jill / 4.7471GB 25MB passed
/sbin jill / 4.7471GB 10MB passed
Verifying Free Space: jill:/usr,jill:/var,jill:/etc,jill:/sbin ...PASSED
Verifying Free Space: jill:/u01/app/19c/grid ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/19c/grid jill /u01 46.6494GB 6.9GB passed
Verifying Free Space: jill:/u01/app/19c/grid ...PASSED
Verifying Free Space: jill:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/tmp jill /tmp 5.1816GB 1GB passed
Verifying Free Space: jill:/tmp ...PASSED
Verifying User Existence: oracle ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists(1000)
jill passed exists(1000)
Verifying Users With Same UID: 1000 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying User Existence: root ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists(0)
jill passed exists(0)
Verifying Users With Same UID: 0 ...PASSED
Verifying User Existence: root ...PASSED
Verifying Group Existence: oinstall ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed exists
jill passed exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
jack yes yes yes passed
jill yes yes yes passed
Verifying Group Membership: oinstall ...PASSED
Verifying Run Level ...
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
jack 5 3,5 passed
jill 5 3,5 passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack hard 65536 65536 passed
jill hard 65536 65536 passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 1024 1024 passed
jill soft 1024 1024 passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack hard 16384 16384 passed
jill hard 16384 16384 passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 16384 2047 passed
jill soft 16384 2047 passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
jack soft 10240 10240 passed
jill soft 10240 10240 passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack x86_64 x86_64 passed
jill x86_64 x86_64 passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack 4.1.12-61.1.18.el7uek.x86_64 4.1.12 passed
jill 4.1.12-61.1.18.el7uek.x86_64 4.1.12 passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 250 250 250 passed
jill 250 250 250 passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 32000 32000 32000 passed
jill 32000 32000 32000 passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 100 100 100 passed
jill 100 100 100 passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 128 128 128 passed
jill 128 128 128 passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4398046511104 4398046511104 3053584384 passed
jill 4398046511104 4398046511104 3053584384 passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4096 4096 4096 passed
jill 4096 4096 4096 passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1073741824 1073741824 1073741824 passed
jill 1073741824 1073741824 1073741824 passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 6815744 6815744 6815744 passed
jill 6815744 6815744 6815744 passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
jill between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 262144 262144 262144 passed
jill 262144 262144 262144 passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 4194304 4194304 4194304 passed
jill 4194304 4194304 4194304 passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 262144 262144 262144 passed
jill 262144 262144 262144 passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1048576 1048576 1048576 passed
jill 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1048576 1048576 1048576 passed
jill 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
jack 1 1 1 passed
jill 1 1 1 passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack kmod(x86_64)-20-21.el7 kmod(x86_64)-20-21 passed
jill kmod(x86_64)-20-21.el7 kmod(x86_64)-20-21 passed
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack kmod-libs(x86_64)-20-21.el7 kmod-libs(x86_64)-20-21 passed
jill kmod-libs(x86_64)-20-21.el7 kmod-libs(x86_64)-20-21 passed
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack binutils-2.25.1-22.base.el7 binutils-2.23.52.0.1 passed
jill binutils-2.25.1-22.base.el7 binutils-2.23.52.0.1 passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
jill compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libgcc(x86_64)-4.8.5-11.el7 libgcc(x86_64)-4.8.2 passed
jill libgcc(x86_64)-4.8.5-11.el7 libgcc(x86_64)-4.8.2 passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libstdc++(x86_64)-4.8.5-11.el7 libstdc++(x86_64)-4.8.2 passed
jill libstdc++(x86_64)-4.8.5-11.el7 libstdc++(x86_64)-4.8.2 passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libstdc++-devel(x86_64)-4.8.5-11.el7 libstdc++-devel(x86_64)-4.8.2 passed
jill libstdc++-devel(x86_64)-4.8.5-11.el7 libstdc++-devel(x86_64)-4.8.2 passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack sysstat-10.1.5-11.el7 sysstat-10.1.5 passed
jill sysstat-10.1.5-11.el7 sysstat-10.1.5 passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack ksh ksh passed
jill ksh ksh passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack make-3.82-23.el7 make-3.82 passed
jill make-3.82-23.el7 make-3.82 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack glibc(x86_64)-2.17-157.el7 glibc(x86_64)-2.17 passed
jill glibc(x86_64)-2.17-157.el7 glibc(x86_64)-2.17 passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack glibc-devel(x86_64)-2.17-157.el7 glibc-devel(x86_64)-2.17 passed
jill glibc-devel(x86_64)-2.17-157.el7 glibc-devel(x86_64)-2.17 passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
jill libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
jill libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.2.3-15 passed
jill nfs-utils-1.3.0-0.33.0.1.el7 nfs-utils-1.2.3-15 passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack smartmontools-6.2-7.el7 smartmontools-6.2-4 passed
jill smartmontools-6.2-7.el7 smartmontools-6.2-4 passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
jill net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
jack cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
jill cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Node Addition ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying '/u01/app/19c/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
Verifying Hosts File ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
Verifying Hosts File ...PASSED
Interface information for node "jack"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s3 172.168.190.1 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71 1500
enp0s3 172.168.190.20 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71 1500
enp0s3 172.168.190.10 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71 1500
enp0s3 172.168.190.30 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71 1500
enp0s3 172.168.190.3 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:61:94:71 1500
enp0s8 192.168.190.1 192.168.0.0 0.0.0.0 172.168.190.100 08:00:27:05:B7:1B 1500
Interface information for node "jill"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s3 172.168.190.2 172.168.190.0 0.0.0.0 172.168.190.100 08:00:27:4B:49:CE 1500
enp0s8 192.168.190.2 192.168.0.0 0.0.0.0 172.168.190.100 08:00:27:DE:B5:AE 1500
Check: MTU consistency on the private interfaces of subnet "192.168.0.0"
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
jack enp0s8 192.168.190.1 192.168.0.0 1500
jill enp0s8 192.168.190.2 192.168.0.0 1500
Check: MTU consistency of the subnet "172.168.190.0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
jack enp0s3 172.168.190.1 172.168.190.0 1500
jack enp0s3 172.168.190.20 172.168.190.0 1500
jack enp0s3 172.168.190.10 172.168.190.0 1500
jack enp0s3 172.168.190.30 172.168.190.0 1500
jack enp0s3 172.168.190.3 172.168.190.0 1500
jill enp0s3 172.168.190.2 172.168.190.0 1500
Source Destination Connected?
------------------------------ ------------------------------ ----------------
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.20] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.10] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.1] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.1] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.10] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.20] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.20] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.10] jack[enp0s3:172.168.190.30] yes
jack[enp0s3:172.168.190.10] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.10] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.30] jack[enp0s3:172.168.190.3] yes
jack[enp0s3:172.168.190.30] jill[enp0s3:172.168.190.2] yes
jack[enp0s3:172.168.190.3] jill[enp0s3:172.168.190.2] yes
Source Destination Connected?
------------------------------ ------------------------------ ----------------
jack[enp0s8:192.168.190.1] jill[enp0s8:192.168.190.2] yes
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying subnet mask consistency for subnet "172.168.190.0" ...PASSED
Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...Disks "/dev/oracleasm/disks/DATA1,/dev/oracleasm/disks/DATA2,/dev/oracleasm/disks/DATA3,/dev/oracleasm/disks/DATA4,/dev/oracleasm/disks/DATA5,/dev/oracleasm/disks/FRA1,/dev/oracleasm/disks/FRA2,/dev/oracleasm/disks/FRA3" are managed by ASM.
Verifying Device Checks for ASM ...PASSED
Verifying Database home availability ...PASSED
Verifying ASMLib installation and configuration verification. ...
Verifying '/etc/init.d/oracleasm' ...PASSED
Verifying '/dev/oracleasm' ...PASSED
Verifying '/etc/sysconfig/oracleasm' ...PASSED
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
Verifying ASMLib installation and configuration verification. ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Network Time Protocol (NTP) ...
Verifying '/etc/ntp.conf' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes
jill yes
Verifying '/etc/ntp.conf' ...PASSED
Verifying '/etc/chrony.conf' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes
jill yes
Verifying '/etc/chrony.conf' ...PASSED
Verifying '/var/run/chronyd.pid' ...
Node Name File exists?
------------------------------------ ------------------------
jack yes
jill yes
Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Daemon 'chronyd' ...
Node Name Running?
------------------------------------ ------------------------
jack yes
jill yes
Verifying Daemon 'chronyd' ...PASSED
Verifying NTP daemon or service using UDP port 123 ...
Node Name Port Open?
------------------------------------ ------------------------
jack yes
jill yes
Verifying NTP daemon or service using UDP port 123 ...PASSED
Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
Verifying Network Time Protocol (NTP) ...FAILED (PRVG-13606)
Verifying User Not In Group "root": oracle ...
Node Name Status Comment
------------ ------------------------ ------------------------
jack passed does not exist
jill passed does not exist
Verifying User Not In Group "root": oracle ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
Node Name Status
------------------------------------ ------------------------
jack passed
jill passed
checking response for name "jack" from each of the name servers specified in
"/etc/resolv.conf"
Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
jack 172.168.190.1 IPv4 passed
jack 172.168.190.2 IPv4 passed
checking response for name "jill" from each of the name servers specified in
"/etc/resolv.conf"
Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
jill 172.168.190.2 IPv4 passed
jill 172.168.190.1 IPv4 passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying User Equivalence ...
Node Name Status
------------------------------------ ------------------------
jill passed
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying /boot mount ...PASSED
Verifying zeroconf check ...PASSED
Pre-check for node addition was unsuccessful on all the nodes.
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
Verifying Physical Memory ...FAILED
jack: PRVF-7530 : Sufficient physical memory is not available on node "jack"
[Required physical memory = 8GB (8388608.0KB)]
jill: PRVF-7530 : Sufficient physical memory is not available on node "jill"
[Required physical memory = 8GB (8388608.0KB)]
Verifying Network Time Protocol (NTP) ...FAILED
Verifying chrony daemon is synchronized with at least one external time
source ...FAILED
jack: PRVG-13606 : chrony daemon is not synchronized with any external time
source on node "jack".
jill: PRVG-13606 : chrony daemon is not synchronized with any external time
source on node "jill".
CVU operation performed: stage -pre nodeadd
Date: Jul 20, 2021 9:40:56 PM
CVU home: /u01/app/19c/grid/
User: oracle
Above node addition pre-check has to get passed to add nodes to the existing One-Node RAC environment.
NOTE: The failures over here are due to home setup and less resources. In your work environment, you should’nt face a single error
Add node
From Jack node,
For GRID_HOME :
[oracle@jack addnode]$ cd /u01/app/19c/grid
[oracle@jack grid]$ cd addnode/
[oracle@jack addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={jill}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={jill-vip}" -ignorePrereqFailure
[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Copy Files to Remote Nodes in progress.
.................................................. 6% Done.
.................................................. 11% Done.
....................
Copy Files to Remote Nodes successful.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 21% Done.
You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 49% Done.
Saving cluster inventory in progress.
.................................................. 83% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/19c/grid was successful.
Please check '/u01/app/19c/oraInventory/logs/silentInstall2021-07-27_12-15-11PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 90% Done.
Update Inventory in progress.
You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_12-15-11PM.log
Update Inventory successful.
.................................................. 97% Done.
As a root user, execute the following script(s):
1. /u01/app/19c/oraInventory/orainstRoot.sh
2. /u01/app/19c/grid/root.sh
Execute /u01/app/19c/oraInventory/orainstRoot.sh on the following nodes:
[jill]
Execute /u01/app/19c/grid/root.sh on the following nodes:
[jill]
The scripts can be executed in parallel on all the nodes.
Successfully Setup Software with warning(s).
.................................................. 100% Done.
As root user, execute orainstRoot.sh and root.sh on jill :
[root@jill ~]# /u01/app/19c/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/19c/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/19c/oraInventory to oinstall.
The execution of the script is complete.
[root@jill ~]# /u01/app/19c/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/19c/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/19c/gridbase/crsdata/jill/crsconfig/rootcrs_jill_2021-07-27_01-03-30PM.log
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/07/27 13:03:39 CLSRSC-363: User ignored prerequisites during installation
2021/07/27 13:03:39 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/07/27 13:03:40 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/07/27 13:03:46 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2021/07/27 13:03:47 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/07/27 13:03:50 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/07/27 13:03:55 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/07/27 13:03:55 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/07/27 13:04:16 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/07/27 13:04:17 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/07/27 13:04:22 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/07/27 13:04:23 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/07/27 13:04:26 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2021/07/27 13:05:07 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/07/27 13:05:10 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/07/27 13:05:54 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/07/27 13:05:57 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/07/27 13:06:15 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/07/27 13:08:32 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/07/27 13:08:33 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
clscfg: EXISTING configuration version 19 detected.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2021/07/27 13:09:17 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/07/27 13:09:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Check Clusterware status :-
[oracle@jill software]$ cd /u01/app/19c/grid/bin/
[oracle@jill bin]$ ./crsctl check cluster -all
**************************************************************
jack:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
jill:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@jill bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.GHCHKPT.advm
OFFLINE OFFLINE jack STABLE
OFFLINE OFFLINE jill STABLE
ora.LISTENER.lsnr
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.chad
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.data.ghchkpt.acfs
OFFLINE OFFLINE jack STABLE
OFFLINE OFFLINE jill STABLE
ora.helper
OFFLINE OFFLINE jack IDLE,STABLE
OFFLINE OFFLINE jill IDLE,STABLE
ora.net1.network
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.ons
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
ora.proxy_advm
ONLINE ONLINE jack STABLE
ONLINE ONLINE jill STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jill STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE jack STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE jack STABLE
ora.MGMTLSNR
1 ONLINE ONLINE jack 169.254.24.18 192.16
8.190.1,STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE jack Started,STABLE
2 ONLINE ONLINE jill Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE jack STABLE
2 ONLINE ONLINE jill STABLE
3 ONLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE jack STABLE
ora.infra.db
1 ONLINE ONLINE jack Open,HOME=/u01/app/o
racle/product/19c/db
_1,STABLE
ora.jack.vip
1 ONLINE ONLINE jack STABLE
ora.jill.vip
1 ONLINE ONLINE jill STABLE
ora.mgmtdb
1 ONLINE ONLINE jack Open,STABLE
ora.qosmserver
1 ONLINE ONLINE jill STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE jill STABLE
ora.scan2.vip
1 ONLINE ONLINE jack STABLE
ora.scan3.vip
1 ONLINE ONLINE jack STABLE
For ORACLE_HOME and configure RDBMS software:
[oracle@jack ~]$ cd $ORACLE_HOME/addnode
[oracle@jack addnode]$ $ORACLE_HOME/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={jill}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={jill-vip}"
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 18% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/u01/app/19c/oraInventory/logs/addNodeActions2021-07-27_01-23-09PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 52% Done.
Saving cluster inventory in progress.
.................................................. 89% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/19c/db_1 was successful.
Please check '/u01/app/19c/oraInventory/logs/silentInstall2021-07-27_01-23-09PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 96% Done.
As a root user, execute the following script(s):
1. /u01/app/oracle/product/19c/db_1/root.sh
Execute /u01/app/oracle/product/19c/db_1/root.sh on the following nodes:
[jill]
Successfully Setup Software.
.................................................. 100% Done.
Run the root.sh script as instructed above from JILL node :
[root@jill bin]# /u01/app/oracle/product/19c/db_1/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/19c/db_1
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Add init parameters for infra2 instance from Node 1 Jack
SQL> alter system set undo_tablespace=UNDOTBS2 sid='infra2' scope=spfile;
System altered.
SQL> alter system set instance_number=2 sid='infra2' scope=spfile;
System altered.
SQL> alter system set instance_name='infra2' sid='infra2' scope=spfile;
System altered.
SQL> alter system set thread=2 sid='infra2' scope=spfile;
System altered.
SQL> alter database add logfile thread 2 group 4 ('+DATA','+FRA') size 200m;
Database altered.
SQL> alter database add logfile thread 2 group 5 ('+DATA','+FRA') size 200m;
Database altered.
SQL> alter database enable public thread 2;
Database altered.
Add database to OCR :
[oracle@jack ~]$ srvctl config database -d infra
Database unique name: infra
Database name: infra
Oracle home: /u01/app/oracle/product/19c/db_1
Oracle user: oracle
Spfile: +DATA/INFRA/PARAMETERFILE/spfile.274.1079014371
Password file: +DATA/INFRA/PASSWORD/pwdinfra.262.1079013137
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: infra1
Configured nodes: jack
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
[oracle@jack ~]$srvctl add instance -db infra -instance infra2 -node jill
[oracle@jack ~]$srvctl start instance -i infra2 -d infra