Replacing the Default Ingress Certificate

This guide explains how to replace the default ingress certificate in an Azure Red Hat OpenShift (ARO) cluster with a custom wildcard certificate.

Prerequisites

• Access to the ARO cluster with cluster-admin privileges
• The oc CLI tool installed and configured
• A wildcard certificate for *.apps.<your-domain>
• The private key for the certificate
• The certificate chain (intermediate and root CA certificates)

Preparing the Certificates

Use OpenSSL to create the necessary files. Here's a step-by-step guide:

1. Ensure the private key is unencrypted

openssl rsa -in original.key -out unencrypted.key

Example with actual filename:

openssl rsa -in "_.apps.aro-prd-we.example.com.key" -out unencrypted.key

2. Verify the subjectAltName extension

To check if the certificate includes the subjectAltName extension, you can use:

openssl x509 -in certificate.pem -text -noout | grep -A1 "Subject Alternative Name"

Example checking for specific domain:

openssl x509 -in certificate.pem -text -noout | grep -A1 "*.apps.aro-prd-we.example.com"

3. Split the certificates if needed

Ensure the certificate file starts with the wildcard certificate, followed by any intermediate certificates, and ends with the root CA certificate. If they are in one file, you can split them if necessary.

To split them, you might use a text editor or the following command:

csplit -z -f cert- -b '%02d.pem' certificate.pem '/-----BEGIN CERTIFICATE-----/' '{*}'

Example with actual filename:

csplit -z -f cert- -b '%02d.pem' 'apps.aro-prd-we.example.com.pem' '/-----BEGIN CERTIFICATE-----/' '{*}'

This will create files like cert-00.pem, cert-01.pem, etc. You need to ensure cert-00.pem is the wildcard certificate, followed by intermediate certificates, and the root certificate last.

4. Create the separate root CA file

Extract the root CA certificate to its own file. Assuming the root CA is the last certificate:

tail -n 20 certificate.pem > root-ca.pem  # Adjust '20' if necessary

Example with actual filename:

tail -n 26 'apps.aro-prd-we.example.com.pem' > root-ca.pem  # Adjust the number based on certificate length

Verify it contains only the root CA certificate.

5. Combine the certificates properly

Combine the wildcard certificate with any intermediate certificates (if split earlier):

cat cert-00.pem cert-01.pem cert-02.pem > combined-certificate.pem

Ensure the combined certificate file ends with the root CA certificate:

cat combined-certificate.pem root-ca.pem > final-certificate.pem

6. Verify formatting

Ensure all certificates end with -----END CERTIFICATE----- and a newline character:

awk '/-----END CERTIFICATE-----/{print $0"\n";next}1' final-certificate.pem > formatted-certificate.pem

7. Final check

Make sure all files are correctly formatted:

cat unencrypted.key
cat formatted-certificate.pem
cat root-ca.pem

Files you should now have:

• unencrypted.key: The unencrypted private key
• formatted-certificate.pem (or full.pem): The combined certificate chain starting with the wildcard certificate
• root-ca.pem: The root CA certificate in a separate file

You can then use these files to configure the OpenShift Container Platform ingress certificate.

OpenShift Configuration

1. Create a ConfigMap that includes only the root CA certificate used to sign the wildcard certificate

oc create configmap custom-ca \
  --from-file=ca-bundle.crt=</path/to/root-ca.pem> \
  -n openshift-config

Example:

oc create configmap custom-ca \
  --from-file=ca-bundle.crt="root-ca.pem" \
  -n openshift-config

2. Update the cluster-wide proxy configuration with the newly created ConfigMap

Bash

oc patch proxy/cluster \
  --type=merge \
  --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

PowerShell

oc patch proxy/cluster --type=merge --patch='{\"spec\":{\"trustedCA\":{\"name\":\"custom-ca\"}}}'

3. Create a secret that contains the wildcard certificate chain and key

oc create secret tls <secret-name> \
  --cert=</path/to/cert.crt> \
  --key=</path/to/cert.key> \
  -n openshift-ingress

Example:

oc create secret tls custom-cert \
  --cert="full.pem" \
  --key="key.pem" \
  -n openshift-ingress

⚠️ Important: If the standard approach doesn't work, use the original key file as-is and ensure you remove any comments from the full certificate chain:

oc create secret tls custom-cert \
  --cert="full.pem" \
  --key="_.apps.aro-prd-we.example.com.key" \
  -n openshift-ingress

4. Update the Ingress Controller configuration with the newly created secret

Bash

oc patch ingresscontroller.operator default \
  --type=merge \
  -p '{"spec":{"defaultCertificate": {"name": "custom-cert"}}}' \
  -n openshift-ingress-operator

PowerShell

oc patch ingresscontroller.operator default --type=merge -p '{\"spec\":{\"defaultCertificate\":{\"name\":\"custom-cert\"}}}' -n openshift-ingress-operator

5. Wait for the changes to propagate

After patching the objects, OpenShift will start updating internally. You can check the status with:

oc rollout status deployment.apps/oauth-openshift -n openshift-authentication

6. Force OAuth rollout (if needed)

If the OAuth pods don't automatically restart, you can force a rollout:

Delete OAuth pods:

oc delete pod -l app=oauth-openshift -n openshift-authentication

Or force a rollout with timestamp annotation:

Bash

timestamp=$(date +%s)
oc patch oauth cluster --type=merge \
  --patch="{\"metadata\":{\"annotations\":{\"oauth.openshift.io/force-rollout\":\"$timestamp\"}}}"
oc rollout status deployment.apps/oauth-openshift -n openshift-authentication

PowerShell

$timestamp = Get-Date -UFormat %s
$jsonPatch = '{\"metadata\":{\"annotations\":{\"oauth.openshift.io/force-rollout\":\"' + $timestamp + '\"}}}'

oc patch oauth cluster --type=merge --patch=$jsonPatch
oc rollout status deployment.apps/oauth-openshift -n openshift-authentication

7. Verify the certificate

When the rollout is finished, verify the certificate with OpenSSL:

echo QUIT | openssl s_client -connect console-openshift-console.apps.aro-prd-we.example.com:443 | \
  openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint

Replace aro-prd-we.example.com with your actual domain.

Additional Troubleshooting Commands

Convert key to PEM format (if needed):

openssl x509 -in _.apps.aro-prd-we.example.com.key -out key.pem -outform PEM

Check secret contents:

oc get secret custom-cert -n openshift-ingress -o yaml

Check ingress controller status:

oc get ingresscontroller default -n openshift-ingress-operator -o yaml

Check OAuth pods:

oc get pods -n openshift-authentication

DNS Configuration

IMPORTANT

Don't forget to configure DNS to point *.apps.aro-prd-we.example.com to your cluster's ingress load balancer.

If you're using Terraform to manage your infrastructure, this can be automated through DNS zone configuration.

Documentation

Official OpenShift Documentation