WEBLOGIC CLUSTER ON 2 PHYSICAL SERVERS
Setup Weblogic on both Nodes
The following actions should be performed by the "root" user.
Make sure the "/etc/hosts" file contains correct entries for both the "localhost" and real host names.

127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
::1             localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.121   vFCIAS-APP-01   vFCIAS-APP-01.fcicanada.com
192.168.1.122   vFCIAS-APP-01   vFCIAS-APP-01.fcicanada.com
Create a new group and user.
groupadd -g 1000 oinstall
useradd -u 1100 -g oinstall oracle
passwd oracle
Create the directories in which the Oracle software will be installed.
mkdir -p /u01/app/oracle/config/domains
mkdir -p /u01/app/oracle/config/applications
chown -R oracle:oinstall /u01
chmod -R 775 /u01/
Append the following entries into the "/home/oracle/.bash_profile" file.
export WLS_HOME=$MW_HOME/wlserver_10.3
export WL_HOME=$WLS_HOME
#export JAVA_HOME=/u01/app/oracle/jrockit-jdk1.6.0_45-R28.2.7-4.1.0
export JAVA_HOME= /u01/app/oracle/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
Install the JDK. Instructions for installing JRockit can be found here. This article will assume JRockit is being used. If you want to use HotSpot for JDK6 or JDK7 you might install them like this.
Installed jrockit
As specified in MOS Note [ID 1487773.1], amend the "/etc/security/limits.d/90-nproc.conf" file, making the following change.
# From
*          soft    nproc     1024

#To
Installation
Run the installer as the "oracle" user.
$ $JAVA_HOME/bin/java -Xmx1024m -jar wls1035_generic.jar
$ # Or
Click the "Next" button on the welcome screen.
Welcome
Enter the middleware home ("/u01/app/oracle/product/fmw11g") and click the "Next" button.
Choose Middleware Home Directory
Either enter your support details, or attempt to uncheck the security updates checkbox. The box doesn't actually uncheck, but presents you with a warning dialog. If you click the "Yes" button, indicating you don't want updates, you are returned to the same screen with the box still checked. Click the "Next" button anyway.
Register For Security Updates
Assuming you don't want security updates, click "Yes" and "Yes" on the subsequent warning dialogs and you are presented with the "Connection Failed" screen. Check the "I wish to remain uninformed..." checkbox and click the "Continue" button. This whole process is very confusing at first. Obviously the developers responsible didn't really expect people to say no to security updates!
Connection Failed
Accept the typical setup by clicking the "Next" button.
Choose Install Type
Accept the default JDK selection by clicking the "Next" button.
JDK Selection
Accept the default product installation directories by clicking the "Next" button.
Choose Product Installation Directories
If you are happy with the summary information, click the "Next" button and wait while the installation completes.
Installation Summary
On the installation complete screen, uncheck the "Quickstart" checkbox and click the "Done" button.
Installation Complete

Once the installation is complete on both machines, come back to this article to create the clustered domain.
Consider setting some of the node manager properties on both servers in the "$WLS_HOME/common/nodemanager/nodemanager.properties" file. For example.
$WLS_HOME/server/bin/startNodeManager.sh start node manager to create necessary directories
CrashRecoveryEnabled=true
StartScriptEnabled=true

Create the Clustered Domain

The details of the cluster being built in this article are as follows.
Machines        : vFCIAS-APP-01.fcicanada.com,
                  vFCIAS-APP-02.fcicanada.com
 
Domain Name     : ITRAC
 
 
Managed Servers : AdminServer     - Running on vFCIAS-APP-01.fcicanada.com
                  ManagedServer_1 - Running on vFCIAS-APP-01.fcicanada.com
                  ManagedServer_2 - Running on vFCIAS-APP-02.fcicanada.com
 
Cluster         : ITRAC_CLSTR
Creation of a clustered domain can be done from the administration console, the command line or the configuration wizard. In my opinion, using the configuration wizard is the quickest way to get the job done. Start the configuration wizard.
Accept the "Create a new WebLogic domain" option by clicking the "Next" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/01-welcome.jpg
Select the desired product support and click the "Next" button.
Enter the appropriate domain name and locations, then click the "Next" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/03-specify-domain-name-and-location.jpg
Enter the administrator credentials and click the "Next" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/04-configure-administrator-username-and-password.jpg
Select the desired server start mode and JDK, then click the "Next" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/05-configure-server-start-mode-and-jdk.jpg
Select the "Administration Server" and "Managed Servers, Clusters and Machines" options, then click the "Next" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/06-select-optional-configuration.jpg
Enter the fully qualified machine name, desired port and SSL configuration, then click the "Next" button.
Port : 7002

Add the managed servers for the domain. When you are finished, click the "Next" button
*******CHECK  SSL ENABLED

Add a cluster entry, remember to add the appropriate cluster address. This can be a comma-separated list of "IP/DNS_Name:port", a DNS name that maps to multiple IP addresses or the local machine if all servers run on one machine. Click the "Next" button.
Associate the managed servers with the cluster and click the "Next" button.
Click the "Unix Machine" tab and add the machines used by the cluster. I've used the fully qualified machine names in all cases. Click the "Next" button.
Assign the managed servers to the machines in the cluster and click the "Next" button.
Check the configuration summary, the click the "Create" button.
https://oracle-base.com/articles/11g/images/weblogic-cluster/13-configuration-summary.jpg
Wait while the domain is created, then click the "Done" button.


If the node manager is not already started on the first server, start it now.
Create a file called "boot.properties" with the following contents. ****start weblogic and shut it down to create necessary directories ***
username=weblogic
password=weblogic1

Start the new domain on the first machine.

 It may seem odd that you are starting the AdminServer in the new domain, then immediately stopping it in the next section (assuming you do not reset the node manager password), but some of the configuration is not written to the file system until the AdminServer is started for the first time, so don't skip this step.

Reset Node Manager Username/Password

If this is the first domain on the server you might want to consider resetting the node manager credentials. This step is optional, but I feel it is neater if the node manager has different credentials to the administration servers. To reset the credentials, do the following:
  • Log in to the administration server (http:// vFCIAS-APP-01.fcicanada.com:7001/console).
  • In the "Domain Structure" tree on the left, click on the domain name (ITRAC).
  • Click on the "Security" tab. This should automatically put you in the "General" sub-tab.
  • Click the "Advanced" link towards the bottom of the screen.
  • If you are running in production mode, click the "Lock & Edit" button.
  • Enter the "NodeManager Username", "NodeManager Password" and "Confirm NodeManager Password", then click the "Save" button.
  • If you are running in production mode, click the "Activate Changes" button.
**** weblogic/weblogic1***

Stop the new domain on the first machine.

$ $MW_HOME/user_projects/domains/ITRAC/bin/stopWebLogic.sh
/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC/bin/stopWebLogic.sh

Transfer Domain to Second Machine

We will transfer the relevant configuration using the pack and unpack utilities.
On the first machine, pack the domain configuration using the following commands. Remember to amend theDOMAIN_HOME environment variable and the -template_name parameter to match your domain.
$ export DOMAIN_HOME=$MW_HOME/user_projects/domains/ITRAC
$ cd $WL_HOME/common/bin 
$ ./pack.sh -managed=true -domain=$DOMAIN_HOME -template=${DOMAIN_HOME}-template.jar -template_name=ITRAC
[oracle@Node1 bin]$ ./pack.sh -managed=true -domain=$DOMAIN_HOME -template=${DOMAIN_HOME}-template.jar -template_name=ITRAC
<< read domain from "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"
>>  succeed: read domain from "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC-template.jar"
....................................................................................................
>>  succeed: write template to "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC-template.jar"
<< close template
>>  succeed: close template
 
Copy the resulting jar file to the second machine.
$ scp ${DOMAIN_HOME}-template.jar oracle@vFCIAS-APP-02.fcicanada.com:/u01/app/oracle/product/fmw11g/user_projects/domains
Unpack the configuration on the second machine. Remember to amend the DOMAIN_HOME environment variable to match your domain.
$ export DOMAIN_HOME=$MW_HOME/user_projects/domains/ITRAC
$ cd $WL_HOME/common/bin
$ ./unpack.sh -domain=$DOMAIN_HOME -template=${DOMAIN_HOME}-template.jar
 
[oracle@Node2 bin]$ ./unpack.sh -domain=$DOMAIN_HOME -template=${DOMAIN_HOME}-template.jar
<< read template from "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC-template.jar"
>>  succeed: read template from "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC-template.jar"
<< set config option DomainName to "ITRAC"
>>  succeed: set config option DomainName to "ITRAC"
<< write Domain to "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"
...............................................................................................
>>  succeed: write Domain to "/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"
<< close template
>>  succeed: close template
Restart the domain on the first machine. Wait until it is fully started before continuing.

Enroll Second Machine

On the second machine, start WLST.
Connect to the administration server manager on the first machine, enroll the current machine, disconnect and exit WLST.
wls:/offline> connect('weblogic', 'weblogic1', 't3://vFCIAS-APP-01.fcicanada.com:7001')
Connecting to t3://vFCIAS-APP-01.fcicanada.com:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'ITRAC'.
 
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
 
wls:/ITRAC/serverConfig>
wls:/ITRAC/serverConfig> nmEnroll('/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC','/u01/app/oracle/product/fmw11g/wlserver_10.3/common/nodemanager')
Enrolling this machine with the domain directory at  /u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC ...
Successfully enrolled this machine with the domain directory at /u01/app/oracle/product/fmw11g/wlserver_10.3/common/bin/ /u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC.
disconnect()
exit()
Check the "$MW_HOME/domain-registry.xml" file contains an entry like the following. If it doesn't, add it manually.
<domain location="/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"/>
Check the "$WLS_HOME/common/nodemanager/nodemanager.domains" file contains an entry like the following. If it doesn't, add it manually.
ITRAC=/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC
 
[oracle@Node2 bin]$ more $MW_HOME/domain-registry.xml
<?xml version="1.0" encoding="UTF-8"?>
<domain-registry xmlns="http://xmlns.oracle.com/weblogic/domain-registry">
  <domain location="/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC"/>
</domain-registry>
[oracle@Node2 bin]$ more $WLS_HOME/common/nodemanager/nodemanager.domains
#Domains and directories created by Configuration Wizard
#Thu Feb 25 11:40:07 EST 2016
ITRAC=/u01/app/oracle/product/fmw11g/user_projects/domains/ITRAC
If the node manager is not already started on this server, start it now.
Check Node Manager on Each Machine
Check the node manager is reachable on each machine.
  • Log in to the administration server (http://Node1.fcicanada.com:7001/console).
  • In the "Domain Structure" tree, expand the "Environment" node and click on the "Machines" node.
  • In the right-hand pane, click on the first machine (Node1.fcicanada.com).
  • Click on the "Monitoring" tab. Be patient. This may take a little time the first time you do it.
  • If the status is "Reachable", everything is fine.
  • Repeat for the second machines (Node2.localdomain).

Start Up

Provided the node manager is up on all servers, you can now start/stop the managed servers on both machines from the command line. The following example shows how you might do this.
$ # Start on each server.
$ nohup $WLS_HOME/server/bin/startNodeManager.sh > /home/oracle/node.log  &
 
$ # Start the managed servers from any server.
$ # Requires boot identity file for this to work as a background process.
$ nohup $MW_HOME/user_projects/domains/ITRAC/startWebLogic.sh > /home/oracle/web.log  &
$ nohup $MW_HOME/user_projects/domains/ITRAC/bin/startManagedWebLogic.sh ManagedServer_1 > /dev/null 2>&1 &
$ nohup $MW_HOME/user_projects/domains/ITRAC/bin/startManagedWebLogic.sh ManagedServer_2 > /dev/null 2>&1 &



Troubleshooting
<Jul 11, 2016 1:15:19 PM> <WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from vFCIAS-APP-01 - 192.168.1.121. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from vFCIAS-APP-01 - 192.168.1.121. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertReceived(Unknown Source)
        at com.certicom.tls.record.alert.AlertHandler.handle(Unknown Source)
        at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)

oracle@vFCIAS-APP-01 ~]$ more /etc/hosts
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
::1             localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.121   vFCIAS-APP-01.fcicanada.com   vFCIAS-APP-01
192.168.1.122   vFCIAS-APP-02.fcicanada.com   vFCIAS-APP-02
[oracle@vFCIAS-APP-01 ~]$


Add the

#Mon Jul 11 10:33:32 CST 2016
DomainsFile=/u01/app/oracle/product/fmw11g/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
DomainsDirRemoteSharingEnabled=false
javaHome=/u01/app/oracle/jdk1.8.0_45
AuthenticationEnabled=true
NodeManagerHome=/u01/app/oracle/product/fmw11g/wlserver_10.3/common/nodemanager
JavaHome=/u01/app/oracle/jdk1.8.0_45/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=vFCIAS-APP-02
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=true
LogCount=1
DomainRegistrationEnabled=false
StopScriptEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=true
StartScriptEnabled=true
LogFile=/u01/app/oracle/product/fmw11g/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50

[oracle@vFCIAS-APP-02 ~]$