Browsers and API clients trust your certificate only if they can build a path from it to a root certificate they already know. The server is responsible for sending the intermediate certificates in that path; when it does not, some clients work (because they cache or download intermediates) and others fail with errors such as unable to verify the first certificate. In this tutorial you will use OpenSSL on Ubuntu 24.04 to see exactly which chain a server sends, check certificate files locally, rebuild a correct chain file, and fix the Nginx and Apache configuration.
Prerequisites
To follow this guide you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with a non-root user with
sudoprivileges. Theopensslcommand is installed by default. - A site served over HTTPS by Nginx or Apache, reachable as
your_domain. - The certificate files for that site, typically your certificate, the intermediate bundle and the private key.
Step 1 - Understanding how a chain is built
A typical chain has three levels:
| Level | Example subject | Who sends it |
|---|---|---|
| Leaf (end-entity) | CN=your_domain | Your server |
| Intermediate CA | CN=R12, O=Let's Encrypt | Your server |
| Root CA | CN=ISRG Root X1 | Nobody: it is already in the client's trust store |
Each certificate's Issuer must equal the Subject of the next one. The server must send the leaf first, followed by the intermediates in order. Sending the root is unnecessary; clients ignore it because they only trust their own copy.
Almost all chain problems fall into four groups:
- Missing intermediate: the server sends only the leaf. Desktop browsers often still work, while
curl, Java, Python, Android apps and payment gateways fail. - Wrong order: the intermediate comes before the leaf, or the chain file belongs to a different CA.
- Wrong certificate: the private key does not match the certificate, or the server presents a certificate for another name.
- Expired or distrusted certificates: an expired leaf or intermediate, or an old client trust store that lacks a newer root.
Step 2 - Inspecting the chain a server sends
Start from the outside, looking at what clients actually receive. -servername sends SNI, which is required when several sites share an IP address:
echo | openssl s_client -connect your_domain:443 -servername your_domain -showcerts 2>/dev/null | grep -E '^ *[0-9] s:|^ *i:|Verify return code'
A healthy server shows the leaf at depth 0, an intermediate at depth 1 whose subject matches the leaf's issuer, and a return code of 0:
0 s:CN = your_domain
i:C = US, O = Let's Encrypt, CN = R12
1 s:C = US, O = Let's Encrypt, CN = R12
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
Verify return code: 0 (ok)
A server missing its intermediate shows only depth 0 and fails verification:
0 s:CN = your_domain
i:C = US, O = Let's Encrypt, CN = R12
Verify return code: 21 (unable to verify the first certificate)
The most useful return codes are:
| Code | Message | Usual cause |
|---|---|---|
0 | ok | The chain is complete and trusted |
10 | certificate has expired | Leaf or intermediate past its notAfter date |
18 | self-signed certificate | The server presents a self-signed leaf |
19 | self-signed certificate in certificate chain | A private or untrusted root is included in the chain |
20 | unable to get local issuer certificate | Intermediate missing or wrong, or the root is not in the local trust store |
21 | unable to verify the first certificate | The server sends only the leaf |
62 | hostname mismatch | Certificate does not cover the name (only with -verify_hostname) |
To check the hostname at the same time, add -verify_hostname your_domain to the s_client command.
Also check what curl reports, since it uses the system trust store in the same way as most command line tools and many applications:
curl -sS -o /dev/null https://your_domain && echo "chain OK"
With a broken chain, curl prints SSL certificate problem: unable to get local issuer certificate instead.
Step 3 - Checking certificate files locally
Once you know what the server sends, look at the files it is configured with. Print subject and issuer for every certificate in a bundle, in the order they appear:
openssl crl2pkcs7 -nocrl -certfile /path/to/fullchain.pem | openssl pkcs7 -print_certs -noout
subject=CN = your_domain
issuer=C = US, O = Let's Encrypt, CN = R12
subject=C = US, O = Let's Encrypt, CN = R12
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
The first certificate must be your leaf, and each issuer must match the subject of the certificate below it. Count how many certificates the file contains, which helps spot duplicates:
grep -c 'BEGIN CERTIFICATE' /path/to/fullchain.pem
Verify the leaf against the intermediates, using the system trust store (/etc/ssl/certs) for the root. -untrusted marks the intermediates as candidates for building the chain, not as trust anchors:
openssl verify -untrusted /path/to/chain.pem /path/to/cert.pem
/path/to/cert.pem: OK
If this prints error 20 at 0 depth lookup: unable to get local issuer certificate, the intermediate file does not contain the issuer of your certificate.
Check the validity dates of the leaf and of each intermediate:
openssl x509 -in /path/to/cert.pem -noout -subject -dates
subject=CN = your_domain
notBefore=Sep 1 08:14:22 2026 GMT
notAfter=Nov 30 08:14:21 2026 GMT
Finally, confirm the private key belongs to the certificate by comparing the hash of their public keys. The two lines must be identical:
openssl x509 -in /path/to/cert.pem -noout -pubkey | openssl sha256
sudo openssl pkey -in /path/to/privkey.pem -pubout | openssl sha256
SHA2-256(stdin)= 3f7a9c...e21b
SHA2-256(stdin)= 3f7a9c...e21b
This works for both RSA and ECDSA keys. If the hashes differ, Nginx refuses to start with key values mismatch and Apache with certificate and private key do not match.
Step 4 - Building a correct chain file
If you use Certbot, you do not need to build anything: /etc/letsencrypt/live/your_domain/fullchain.pem already contains the leaf followed by the intermediate. Use it and skip to Step 5.
For certificates from a commercial CA, you normally receive the leaf (your_domain.crt) and an intermediate bundle (often ca-bundle.crt or intermediate.crt). If you have lost the intermediate, the leaf itself tells you where to download it, in the Authority Information Access extension:
openssl x509 -in your_domain.crt -noout -ext authorityInfoAccess
Authority Information Access:
CA Issuers - URI:http://ca.example.com/intermediate.crt
Download it and convert it to PEM. The file is usually in binary DER format:
curl -fsSL -o intermediate.der http://ca.example.com/intermediate.crt
openssl x509 -inform DER -in intermediate.der -out intermediate.pem
If the conversion fails with Could not read certificate, the file is already PEM; rename it to intermediate.pem instead. A .p7c file is a PKCS#7 bundle and is converted with openssl pkcs7 -inform DER -in file.p7c -print_certs -out intermediate.pem.
Build the chain file with the leaf first, then the intermediates, and without the root:
cat your_domain.crt intermediate.pem > fullchain.pem
Files from Windows systems sometimes have CRLF line endings, which some tools reject. Strip them in place:
sed -i 's/\r$//' fullchain.pem
Verify the result as in Step 3 before deploying it:
openssl verify -untrusted fullchain.pem your_domain.crt
your_domain.crt: OK
Copy the chain and the key to a proper location, for example /etc/ssl/your_domain/, with the key readable only by root:
sudo mkdir -p /etc/ssl/your_domain
sudo cp fullchain.pem /etc/ssl/your_domain/fullchain.pem
sudo cp your_domain.key /etc/ssl/your_domain/privkey.pem
sudo chmod 600 /etc/ssl/your_domain/privkey.pem
Step 5 - Fixing the web server configuration
The single most common cause of an incomplete chain is pointing the server at the leaf certificate (cert.pem or your_domain.crt) instead of the full chain.
Nginx
Nginx reads the leaf and the intermediates from one file. Open your server block, for example:
sudo nano /etc/nginx/sites-available/your_domain
Make sure ssl_certificate points to the full chain:
server {
listen 443 ssl http2;
server_name your_domain;
ssl_certificate /etc/ssl/your_domain/fullchain.pem;
ssl_certificate_key /etc/ssl/your_domain/privkey.pem;
# ... rest of the configuration
}
With Certbot the paths are /etc/letsencrypt/live/your_domain/fullchain.pem and /etc/letsencrypt/live/your_domain/privkey.pem. Test and reload:
sudo nginx -t
sudo systemctl reload nginx
Apache
Since Apache 2.4.8, SSLCertificateFile accepts the leaf followed by the intermediates, and SSLCertificateChainFile is obsolete. Open the virtual host:
sudo nano /etc/apache2/sites-available/your_domain-ssl.conf
<VirtualHost *:443>
ServerName your_domain
SSLEngine on
SSLCertificateFile /etc/ssl/your_domain/fullchain.pem
SSLCertificateKeyFile /etc/ssl/your_domain/privkey.pem
# ... rest of the configuration
</VirtualHost>
Remove any leftover SSLCertificateChainFile line that points to an old intermediate, then test and reload:
sudo apache2ctl configtest
sudo systemctl reload apache2
Syntax OK
Step 6 - Verifying the fix
Repeat the check from Step 2. You should now see the intermediate at depth 1 and return code 0:
echo | openssl s_client -connect your_domain:443 -servername your_domain -verify_hostname your_domain -showcerts 2>/dev/null | grep -E '^ *[0-9] s:|Verify return code'
0 s:CN = your_domain
1 s:C = US, O = Let's Encrypt, CN = R12
Verify return code: 0 (ok)
Run the curl check as well:
curl -sS -o /dev/null https://your_domain && echo "chain OK"
chain OK
If the domain resolves to several servers or sits behind a load balancer, run the check against each backend IP with -connect backend_ip:443 -servername your_domain, since each one may have its own copy of the files.
Troubleshooting
- The server still sends the old chain after the fix. Another server block or virtual host matches first, often the default one. Find every certificate directive with
sudo nginx -T | grep ssl_certificateorsudo apache2ctl -S, and remember the configuration only applies after a reload. - Only some clients fail and the chain looks correct. The client trust store is out of date. On Debian and Ubuntu clients, update it with
sudo apt install --only-upgrade ca-certificates; for Java, update the JDK, which ships its owncacerts. - Java applications and Tomcat. Java keystores need the full chain too. Create a PKCS#12 keystore, which current Java versions read directly, including the intermediates with
-certfile:openssl pkcs12 -export -in your_domain.crt -inkey your_domain.key -certfile intermediate.pem -name your_domain -out keystore.p12. OpenSSL asks for an export password; use a strong one and configure it in the application. - Return code
19on an internal service. The chain includes a private root that clients do not trust. Distribute the private root to the clients (on Ubuntu, copy it to/usr/local/share/ca-certificates/with a.crtextension and runsudo update-ca-certificates) rather than disabling verification.
Conclusion
You can now read the chain a server sends with openssl s_client, verify certificate files and key pairs locally, rebuild a chain file in the right order from the leaf and its intermediates, and point Nginx or Apache at it. Add the s_client or curl check to your deployment process or monitoring so a missing intermediate is caught before clients report it, and prefer ACME clients such as Certbot, which always produce a correct fullchain.pem.
