HandShake Failure after Enabling 2-Way SSL in WebLogic. Certs have been imported to the trust keystore in both sides. One-way SSL works.
####<Sep 6, 2018 3:33:15 PM EDT> <Debug> <SecuritySSL> <MACHINE_NAME> <SERVER_NAME> <Thread-28> <<WLS Kernel>> <> <> <BEA-000000> <[Thread[Thread-28,5,Pooled Threads]]weblogic.security.SSL.jsseadapter: SSLENGINE: Exception occurred during SSLEngine.unwrap(ByteBuffer,ByteBuffer[]).
javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at com.ibm.jsse2.k.a(k.java:32)
at com.ibm.jsse2.pc.a(pc.java:497)
at com.ibm.jsse2.pc.a(pc.java:285)
at com.ibm.jsse2.pc.j(pc.java:150)
at com.ibm.jsse2.pc.b(pc.java:469)
at com.ibm.jsse2.pc.a(pc.java:216)
at com.ibm.jsse2.pc.unwrap(pc.java:210)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:34)
at weblogic.security.SSL.jsseadapter.JaSSLEngine$5.run(JaSSLEngine.java:134)
at weblogic.security.SSL.jsseadapter.JaSSLEngine.doAction(JaSSLEngine.java:734)
at weblogic.security.SSL.jsseadapter.JaSSLEngine.unwrap(JaSSLEngine.java:132)
at weblogic.socket.JSSEFilterImpl.unwrap(JSSEFilterImpl.java:603)
Cause
Client (WLS Sending JVM) is not sending the certificates in the OutBound calls via 2 way SSL over t3s thus failing the SSL handshake.
The same communication works when using 1 way SSL. When 2 way SSl is enabled, the client is not able to send its cert to the server.
Fix
Add the below 2 JVM Arguments to the Outbound domain’s managed server JAVA_OPTIONS and restart the managed server. It will then use the keys from the private key and use it.
-Dweblogic.OutboundEnabled=true -Dweblogic.OutboundPrivateKeyEnabled=true
Add the above JVM parameters in the Client Weblogic JVM and restart . Verify the communication now .