Oracle Coherence GE 12.2.1.0.0 <D5> (thread=ReplicatedCache:SOAReplicatedCache, member=15): Member 13 joined Service SOAReplicatedCache with senior member 15> 

Hello Viewer,

I have 2 unix box on which  two TEST environment were built with SOA and OSB domain for each.

Taking reference of only SOA domain :

SOA_TEST1_DOMAIN:

Admin server(Machine1)

BAM server1(Machine1)

BAM server2(Machine2)

SOA_server1(Machine1)

SOA server2(Machine2)

SOA_CLUSTER: SOA_server1, SOA server2

BAM_CLUSTER: BAM server1,BAM server2

SOA_TEST2_DOMAIN:

Admin server(Machine1)

BAM server1(Machine1)

BAM server2(Machine2)

SOA_server1(Machine1)

SOA server2(Machine2)

SOA_CLUSTER: SOA_server1, SOA server2

BAM_CLUSTER: BAM server1,BAM server2

Note: cluster name in both the domain is same i.e SOA_CLUSTER and BAM_CLUSTER.

Issue:

===

--Whenever i was trying to deploy a service in TEST2 environment, I see errors in the TEST1 Env and the deployment hangs and/or timeouts with errors..

--When I shutdown TEST2 Env, than the Deployment goes through fine.

-- After bringing up the TEST2 environment again , already deployed composite becomes inaccessible in TEST1

Error in logs:

<Trace> <com.oracle.coherence> <soaXXXX.cos.XXXX.com> <soa_server1> <Logger@9215997 12.2.1.0.0> <<anonymous>> <> <a3379798-6804-442d-a3de-4a5576fc5975-00000004> <1458800884649> <[severity-value: 256] [rid: 0:6] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <[com.tangosol.coherence.component.util.logOutput.Jdk:log] 2016-03-24 02:28:04.649/212.173 Oracle Coherence GE 12.2.1.0.0 <D5> (thread=ReplicatedCache:SOAReplicatedCache, member=15): Member 13 joined Service SOAReplicatedCache with senior member 15>

After going through all the possible cause , we ended up on coherence factor.Actually whenever i was trying to bring up the servers in TEST2 environment , servers were getting registered as a member of already running SOA_CLUSTER i.e cluster of TEST1.

we tried to add the coherence parameter with different port in both the domains but still issue was occurring.

In 12c , while creating a domain we get a wizard to configure coherence cluster, by default weblogic provides us the coherence cluster named as defaultCoherenceCluster with all the Weblogic cluster(SOA_CLUSTER and BAM_CLUSTER) of the domain in its member list.

While creating the domain we accepted all the default setting for defaultCoherenceCluster, also name of the cluster in both the domain was same, so it seems it was not able to differentiate between weblogic clusters in both the domain and hence all the running servers were getting registered under only one cluster name SOA_CLUSTER.

Solution: 

we removed the SOA_CLUSTER and BAM_CLUSTER from the member list of defaultCoherenceCluster in one of the domain.

Alternatively other solution could be to create 2nd domain with different cluster name.

Although i am not sure if configuring defaultCoherenceCluster is mandatory or not :) 

since it resolved the issue for me , we accepted this solution.

Thanks a lot for your patience!!!

Regards
-Ashish