otn tour 2015 experience in implementing ssl between oracle db and oracle clients
TRANSCRIPT
Experience in implementing SSL between Oracle DB and Oracle Clients
Agenda
● Project overview
● SSL session (1 way/2 ways)
● Certificate Authorities & Certificates
● 3 Different Oracle Listeners configurations (insecure/secure/ssl)
● SSL & Oracle Clients setup (tns configuration parameters)
● Issues faced & troubleshooting tools
● Conclusion
● Q & A
RAC Cluster
Project overview
SSL ListenerSEC Listener
SCAN Listeners
Developers & End users / tools directly
connecting to DB
Main application technology stack
OBIEE, WLS, Forms, Reports
IntegrationsOther DBs, Essbase,
Ora Net Clients
SSL ListenerSEC Listener
SSL ListenerSEC Listener
How does SSL/TLS work?
How does SSL/TLS work? from http://security.stackexchange.com/ => here
How does SSL work? What is an SSL handshake? from http://www.symantec.com/ => here
Transport Layer Security from https://en.wikipedia.org/ => here
SSL session overview (1 way)
ServerInitial request (to https://domainname.com)Hello .... SSLv2, SSLv3, TLSv1, .... ciphers, hash function
● Server Certificate● Server Private Key
Server Certificate (signed by Certificate Authority) Public Key + Domain Name (subject) + Org name ....● List of Trusted
CA certificates
Establish trust ...
Agree on the symmetric encryption key ...
premaster secret
master secret & generate session keys
Use session key and agreed encryption cipher to send data ...
SSL session overview (2 way)
Server
● Server Certificate● Server Private Key● List of Trusted
CA certificates
Establish trust ...
Agree on the symmetric encryption key ...
master secret & generate session keys
Use session key and agreed encryption cipher to send data ...
premaster secret
● Client Certificate● Client Private Key
● List of TrustedCA certificates
● There are 3 options to obtain an SSL certificate
a. Generate your own self-signed certificate (from https://www.linux.com here)
b. Request certificate from your organisation CA
c. Request certificate from public CA
■ VeriSign
■ GeoTrust
■ ... https://en.wikipedia.org/wiki/Certificate_authority
● In A & B you or your organisation controls hashing and encryption protocols
● However no clients will recognize certificate by default
● In option C you will need to make sure your client versions supports protocols
dictated by external CA
a. Recent example => Many CAs announced that they will not support SHA-1 forcing SHA256/SHA-2
implementation
Certificate Authorities & Certificates
● Intermediate certificate authorities
○ https://en.wikipedia.org/wiki/Intermediate_certificate_authorities
Certificate Authorities & Certificates
●
Certificate Authorities & Certificates
●
Certificate Authorities & Certificates
●
Certificate Authorities & Certificates
Oracle
Listener Config
3 Different Oracle Listeners configurations
● Default Listener (insecure)
○ Username/password are encrypted
● Secure Listener
○ Use encryption the same way as SSL Listener
○ No authentication phase (SSL handshake)
● SSL Listener
○ Use encryption
○ Could be configured with 1 way or 2 ways authentication
Default Listener (insecure)
listener.ora
LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1521)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )
Secure Listener
listener.ora
LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1522)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )
sqlnet.ora
sqlnet.crypto_checksum_server = REQUIREDsqlnet.encryption_server = REQUIREDsqlnet.crypto_checksum_types_server = (SHA1)sqlnet.encryption_types_server = (AES256)
listener.ora
LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCPS)(HOST = db_host)(PORT = 1523)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )
WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/certs) ) )
SSL_CLIENT_AUTHENTICATION = FALSESSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)
SSL Listener
oracle@host > openssl s_client -connect db_host:1523CONNECTED(00000003)depth=0 CN = PROD...verify return:1Certificate chain 0 s:/CN=PROD i:/CN=PROD---Server certificate-----BEGIN CERTIFICATE-----MIIBkDCB+gIBADANBgkqhkiG9w0BAQQFADARMQ8wDQYDVQQDEwZFQlNMTlAwHhcNMTQwNzAyMDQzNTI4WhcNNDExMTE2MDUzNTI4WjARMQ8wDQYDVQQDEwZFQlNMTlAwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAJGbN1hxmtFKx1SpUU8JPvRyjMzW...-----END CERTIFICATE-----subject=/CN=PRODissuer=/CN=PROD---Acceptable client certificate CA names/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification Authority/C=US/O=VeriSign, Inc./OU=Class 2 Public Primary Certification Authority/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA/C=US/O=Equifax/OU=Equifax Secure Certificate Authority---SSL handshake has read 1269 bytes and written 210 bytes---New, TLSv1/SSLv3, Cipher is AES256-SHA...
SSL & Oracle Clients setup
SSL & Oracle Clients setup...---SSL handshake has read 1269 bytes and written 210 bytes---New, TLSv1/SSLv3, Cipher is AES256-SHAServer public key is 1024 bitSecure Renegotiation IS supportedCompression: NONEExpansion: NONESSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: Session-ID-ctx: Master-Key: 68E402B5CD161618522350EE... Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1447632207 Timeout : 300 (sec) Verify return code: 18 (self signed certificate)---oracle@host >
Client Config
== TCP ==========================================================================tnsnames.oraPROD = (DESCRIPTION =
(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 1523)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) )
== TCPS =========================================================================tnsnames.oraPROD_SSL = (DESCRIPTION = (SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia")) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) )
SSL & Oracle Clients setup (OCI)
== TCPS ==========================================================================sqlnet.oraSSL_SERVER_DN_MATCH = yesSSL_CLIENT_AUTHENTICATION = FALSESSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)
WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/cert) ) )
SSL & Oracle Clients setup (OCI)
● java … -Doracle.net.ssl_server_dn_match=true …
● jdbc:oracle:thin:@(DESCRIPTION =(SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia"))(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)))(CONNECT_DATA = (SERVICE_NAME = PROD_ETL01)))
● Java has it own certificate store
SSL & Oracle Clients setup (JDBC)
Certificates Config
SSL & Oracle Clients setup (OCI)Using the orapki Utility to Manage PKI Elements
https://docs.oracle.com/database/121/DBSEG/asoappf.htm#DBSEG610
$ ls -l /u01/app/oracle/SSL/cert/*total 16-rw-r--r--@ 1 yvel 5000 1365 Aug 21 2014 cwallet.sso-rw-r--r--@ 1 yvel 5000 1288 Aug 21 2014 ewallet.p12
oracle@host:/home/oracle> orapki wallet display -wallet /u01/app/oracle/SSL/cert ; dateOracle PKI Tool : Version 11.2.0.4.0 - ProductionCopyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:User Certificates:Subject: CN=PRODTrusted Certificates:Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject: OU=Equifax Secure Certificate Authority,O=Equifax,C=USSubject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject: CN=GeoTrust Global CA,O=GeoTrust Inc.,C=USSubject: OU=Secure Server Certification Authority,O=RSA Data Security\, Inc.,C=USSubject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Wed Sep 9 16:27:36 PDT 2015oracle@kpfp2:/home/oracle>
orapki wallet create -wallet $WALLET -auto_login -pwd $v_pwdorapki wallet remove -wallet $WALLET -trusted_cert_all -pwd $v_pwdorapki wallet display -wallet $WALLET -pwd $v_pwdorapki wallet jks_to_pkcs12 -wallet $WALLET -keystore /full/path/to/certificate.jks -jkspwd <pwd>orapki wallet add -wallet $WALLET -trusted_cert -cert GIAG2.crt -pwd $v_pwd
SSL & Oracle Clients setup (OCI)
SSL & Oracle Clients setup (OCI)$ openssl pkcs12 -info -in /u01/app/oracle/SSL/cert/ewallet.p12Enter Import Password:MAC Iteration 1024MAC verified OKPKCS7 Encrypted data: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 1024Certificate bagBag Attributes localKeyID: E6 B6 52 DD 00 00 00 04 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 04subject=/C=US/O=MyOrg Inc/CN=MyOrg Internet Authority G2issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA-----BEGIN CERTIFICATE-----MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9iYWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTUwNDA0MTUxNTU1WjBJMQswCQYDVQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVybmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NPVaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtvh8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE...
PKCS 12 => https://en.wikipedia.org/wiki/PKCS_12It defines an archive file format for storing many cryptography objects as a single file. It is commonly used to bundle a private key with its X.509 certificate.
● JDBC Clients
○ Have it own default certificate store with preloaded trusted CA list
$ v_java_cert=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts$ keytool -list -keystore $v_java_cert -storepass changeit
Keystore type: JKSKeystore provider: SUN
Your keystore contains 93 entries
digicertassuredidrootca, Apr 16, 2008, trustedCertEntry,Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43comodorsaca, May 12, 2015, trustedCertEntry,Certificate fingerprint (SHA1): AF:E5:D2:44:A8:D1:19:42:30:FF:47:9F:E2:F8:97:BB:CD:7A:8C:B4thawtepremiumserverca, May 26, 2015, trustedCertEntry,Certificate fingerprint (SHA1): E0:AB:05:94:20:72:54:93:05:60:62:02:36:70:F7:CD:2E:FC:66:66...
$ keytool -exportcert -alias digicertassuredidrootca -keystore $v_java_cert -storepass changeit -file test.crt -rfc
$ keytool -printcert -file test.crtOwner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=USIssuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=USSerial number: ce7e0e517d846fe8fe560fc1bf03039Valid from: Thu Nov 09 16:00:00 PST 2006 until: Sun Nov 09 16:00:00 PST 2031Certificate fingerprints:
MD5: 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72
SSL & Oracle Clients setup (JBDC)
SSL & Oracle Clients setup (JDBC)
● A JRE client must be updated with the JCE to enable
TLS_RSA_WITH_AES_256_CBC_SHA cipher.
○ Download appropriate JCE archive with 2 jar files
○ Copy the jar files to $JAVA_HOME/jre/lib/security/ directory
● The JCE could be obtained from the URL below depending on a JRE version
○ 1.6 JCE○ 1.7 JCE ○ 1.8 JCE
Issues faced
RAC Cluster
Variety of clients and versions to cover ...
SSL ListenerSEC Listener
SCAN Listeners
Developers & End users using tools
directly connecting to DB
Main application technology stack
OBIEE, WLS, Forms, Reports
IntegrationsOther DBs, Essbase,
Ora Net Clients
SSL ListenerSEC Listener
SSL ListenerSEC Listener
1. 2.3.
Some components used old Oracle Clients
● Essbase
○ Blend in 11.1.0.7 Oracle Client
○ Challenging to update
● OBIEE
○ Presentation layer use 11.1.0.7 Oracle Client
SSL Listener & TCP Buffer
Problem definition
Initial:● DB Instances doesn't register DB Services with a remote SCAN
listeners, after a new certificate has been added to server wallet
Current:● All SSL connections from one node hangs connecting to any SSL
enabled listeners that runs on a remote node if a listener is able to send data out quickly enough
SSL Listener & TCP Buffer
SSL Listener works if ….
a) strace-ing the listener process
b) turning debug logging up to a high level in the listener
c) Running the listener on a non-bonded interface
d) Having the listener increase its send buffer size from 16k (default) to 32k or 64k on the socket
d1) We can also do this at the OS level via `echo "4096 32768 4194304" > /proc/sys/net/ipv4/tcp_wmem'
SSL Listener & TCP Buffer
Normal / Expected processing
sPORT:sIP - cIP:cPORT
TCP socketTCP socket buffer
Processowner of the socket
O_NONBLOCK
W3 W2 W1 DATA
Data writesO_NONBLOCK
sPORT:sIP - cIP:cPORTProcessowner of the socket DATA
Kernel waits a bit for more data and sends it out
Buffer is full. Kernel sends EAGAIN back
5k 1k 3k
3k
W3b
2k
W3bEAGAIN
sPORT:sIP - cIP:cPORTProcessowner of the socket
The process code handles error and keep sending the rest of the data until successful
3k
W3b
3k
Kernel refuses writes until buffer is free
Oracle Listener - fails to process EAGAIN
TCP socketTCP socket bufferO_NONBLOCK
Data writesO_NONBLOCK
sPORT:sIP - cIP:cPORTSSL Listener DATA
Buffer is full. Kernel sends EAGAIN back3k
W3b
2k
W3bEAGAIN
sPORT:sIP - cIP:cPORTSSL Listener
Listener never re-sends the W3b part Kernel ready to process writes after some time
Oracle Listener ssl handshake function fails to process EAGAIN errors
sPORT:sIP - cIP:cPORTSSL Listener DATA
Buffer is full. Kernel sends EAGAIN back
2k
W3b
3k
W3b
The W3b part is dropped
Where is client's response?
W3b
The W3b part is dropped
SSL Listener & TCP Buffer
Conclusions
RAC Cluster
Variety of clients and versions to cover ...
SSL ListenerSEC Listener
SCAN Listeners
Developers & End users using tools
directly connecting to DB
Main application technology stack
OBIEE, WLS, Forms, Reports
IntegrationsOther DBs, Essbase,
Ora Net Clients
SSL ListenerSEC Listener
SSL ListenerSEC Listener
Q & A