UCSB-1 fails to ORC-1, using /var/local/dataone/ca
------------------------------------------------------------------------
(Root and Prod certs in Separate files (DataONEProductionCA.crt  DataONERootCA.crt), Reversed order (DataONECAChainReversed.crt), and normal Chain (DataONECAChain.crt)

root@cn-ucsb-1:/var/local/dataone/ca# curl -v --capath /var/local/dataone/ca  --cert /etc/dataone/client/certs/cn-ucsb-1.dataone.org.pem --key /etc/dataone/client/private/cn-ucsb-1.dataone.org.key "https://cn-orc-1.dataone.org/knb/servlet/replication?server=cn-ucsb-1.dataone.org/knb/servlet/replication&action=test"
* About to connect() to cn-orc-1.dataone.org port 443 (#0)
*   Trying 160.36.13.150... connected
* Connected to cn-orc-1.dataone.org (160.36.13.150) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /var/local/dataone/ca
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

UCSB-1 suceeds to ORC-1, using /etc/ssl/certs 
--------------------------------------------------------------
using normal Chain (DataONECAChain.crt)

 root@cn-ucsb-1:~# curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-ucsb-1.dataone.org.pem --key /etc/dataone/client/private/cn-ucsb-1.dataone.org.key "https://cn-orc-1.dataone.org/knb/servlet/replication?server=cn-ucsb-1.dataone.org/knb/servlet/replication&action=test"
* About to connect() to cn-orc-1.dataone.org port 443 (#0)
*   Trying 160.36.13.150... connected
* Connected to cn-orc-1.dataone.org (160.36.13.150) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*          subject: serialNumber=8kbAK7iaZfwhXx6OwLSBf9lUJSUqveDt; OU=GT39025617; OU=See www.rapidssl.com/resources/cps (c)12; OU=Domain Control Validated - RapidSSL(R); CN=*.dataone.org
*          start date: 2012-05-16 09:53:15 GMT
*          expire date: 2017-05-18 06:38:58 GMT
*          subjectAltName: cn-orc-1.dataone.org matched
*          issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*          SSL certificate verify ok.
> GET /knb/servlet/replication?server=cn-ucsb-1.dataone.org/knb/servlet/replication&action=test HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: cn-orc-1.dataone.org
> Accept: */*

* SSLv3, TLS handshake, Hello request (0):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
< HTTP/1.1 200 OK
< Date: Mon, 11 Jun 2012 18:19:46 GMT
< Server: Apache/2.2.14 (Ubuntu)
< Content-Length: 45
< Vary: Accept-Encoding
< Content-Type: text/html

<html><body>Test successfully</body></html>
* Connection #0 to host cn-orc-1.dataone.org left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):


State of /var/local/dataone/ca on all machines
----------------------------------------------------------------
root@cn-orc-1:/var/local/dataone/ca# ls -l
total 8
lrwxrwxrwx 1 root tomcat6   33 2012-06-11 18:37 10718cba.0 -> /etc/ssl/certs/cilogon-silver.pem
lrwxrwxrwx 1 root tomcat6   32 2012-06-11 18:37 28776852.0 -> /etc/ssl/certs/cilogon-basic.pem
lrwxrwxrwx 1 root tomcat6   33 2012-06-11 18:38 3d863bc5.0 -> /etc/ssl/certs/cilogon-openid.pem
lrwxrwxrwx 1 root tomcat6   17 2012-06-11 18:36 6f77d24b.0 -> DataONERootCA.crt
lrwxrwxrwx 1 root tomcat6   23 2012-06-11 18:36 a0f95ed0.0 -> DataONEProductionCA.crt
-rw-r--r-- 1 root tomcat6 2212 2012-06-11 18:09 DataONEProductionCA.crt
-rw-r--r-- 1 root tomcat6 2204 2012-06-11 18:09 DataONERootCA.crt

State of /etc/ssl/certs on all machines
----------------------------------------------------------------
root@cn-orc-1:/etc/ssl/certs# ls -l /etc/ssl/certs | grep 'DataONE'
lrwxrwxrwx 1 root     root        18 2012-06-11 17:36 6f77d24b.0 -> DataONECAChain.crt
-rw-r--r-- 1 openldap lpadmin   4416 2012-06-08 05:25 DataONECAChain.crt
-rw-r--r-- 1 openldap lpadmin   1931 2012-06-08 05:25 DataONETestCA.pem
lrwxrwxrwx 1 root     root        17 2012-06-11 17:36 f9c792f6.0 -> DataONETestCA.pem

Tests to run
----------------
from orc-1   
curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-orc-1.dataone.org.pem --key /etc/dataone/client/private/cn-orc-1.dataone.org.key "https://cn-ucsb-1.dataone.org/knb/servlet/replication?server=cn-orc-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success

curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-orc-1.dataone.org.pem --key /etc/dataone/client/private/cn-orc-1.dataone.org.key "https://cn-unm-1.dataone.org/knb/servlet/replication?server=cn-orc-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success

from ucsb-1

curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-ucsb-1.dataone.org.pem --key /etc/dataone/client/private/cn-ucsb-1.dataone.org.key "https://cn-orc-1.dataone.org/knb/servlet/replication?server=cn-ucsb-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success

curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-ucsb-1.dataone.org.pem --key /etc/dataone/client/private/cn-ucsb-1.dataone.org.key "https://cn-unm-1.dataone.org/knb/servlet/replication?server=cn-ucsb-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success

from unm-1

curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-unm-1.dataone.org.pem --key /etc/dataone/client/private/cn-unm-1.dataone.org.key "https://cn-orc-1.dataone.org/knb/servlet/replication?server=cn-unm-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success

curl -v --capath /etc/ssl/certs --cert /etc/dataone/client/certs/cn-unm-1.dataone.org.pem --key /etc/dataone/client/private/cn-unm-1.dataone.org.key "https://cn-ucsb-1.dataone.org/knb/servlet/replication?server=cn-unm-1.dataone.org/knb/servlet/replication&action=test"
RESULT: success


Apache config from ORC1
--------------------------------------
<IfModule mod_ssl.c>
<VirtualHost *:443>
        DocumentRoot /var/www
        ServerName cn-orc-1.dataone.org
        DirectoryIndex listObjects.xml

        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        RewriteEngine On
        SSLProxyEngine On
    RewriteOptions Inherit

   <Location /knb/metacat>
        Order allow,deny
        Allow from 127.0.0.0/8 ::1
    </Location>
    <Location /knb/servlet/metacat>
        Order allow,deny
        Allow from 127.0.0.0/8 ::1
    </Location>
    <Location /knb/d1/mn>
        Order allow,deny
        Allow from 127.0.0.0/8 ::1
    </Location>


        Include /etc/apache2/jk_mount
        AllowEncodedSlashes On
        AcceptPathInfo On
        <Location /knb/servlet/replication>
          SSLVerifyClient require
          SSLVerifyDepth  10
        </Location>

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on
        SSLOptions +StrictRequire +StdEnvVars +ExportCertData
        SSLVerifyClient optional
        SSLVerifyDepth 10

        SSLCertificateFile /etc/ssl/certs/_.dataone.org.crt
        SSLCertificateKeyFile /etc/ssl/private/dataone_org.key
        SSLCertificateChainFile /etc/ssl/certs/geotrust_intermediate.crt

        SSLCACertificatePath /var/local/dataone/ca

</VirtualHost>

<VirtualHost *:80>
        DocumentRoot /var/www
        ServerName cn-orc-1.dataone.org

        Redirect permanent / https://cn-orc-1.dataone.org/
</VirtualHost>