SSL/TLS Certs

Think about a high-security video call between two strangers who need to agree on which video conferencing app and quality settings to use, have one person show government-issued ID to a witness the other side trusts proving they are who they claim to be, and agree on a private code word for the rest of the conversation that nobody eavesdropping on the call setup could have figured out, even though the whole setup conversation happened in the open. All of that negotiation happens before either person actually says anything meaningful to the other.

TLS was designed to accomplish exactly this over a network connection that might be tapped by anyone: negotiate shared parameters, prove identity, and establish a secret key, all before any application data like an HTTP request is exchanged. The reason it's built as a multi-step negotiation rather than one direct message is that clients and servers may support different protocol versions and cryptographic algorithms, and the identity and trust piece, certificates, needs to be independently verifiable by the client without any prior direct relationship with the server, which is what makes HTTPS work reliably between total strangers on the internet.

The handshake proceeds: the client sends a ClientHello, listing supported TLS versions, cipher suites, and critically the SNI, Server Name Indication, telling the server which hostname it's trying to reach, since one IP address can host many TLS certificates. The server responds with a ServerHello, its chosen version and cipher, plus its certificate chain. The client then validates that chain, checking the signature path up to a trusted root, checking the certificate's notBefore and notAfter validity window, checking the certificate's subject or SAN matches the requested hostname, and optionally checking revocation status via OCSP or CRL. Both sides then perform a key exchange, commonly ECDHE, elliptic-curve Diffie-Hellman, which lets both sides compute the same shared secret without ever transmitting it, and send Finished messages confirming the handshake's integrity before any actual HTTP data flows across the connection.

In production, TLS handshake failures show up as elevated connection error rates on a service like checkout-worker calling an external payment gateway, and the operational challenge is that failure can originate from any single check in that validation chain: an expired certificate, a hostname that doesn't match, common after a service is renamed or a load balancer's certificate doesn't cover a newly added domain, an untrusted or incomplete chain, a revoked certificate, or simply no overlapping cipher suite between an old client and a server that's dropped support for outdated ciphers for security reasons.

The non-obvious gotcha: many HTTP client libraries and language runtimes report all of these distinct failure modes through one generic exception type, something like SSL handshake failed or certificate verify failed, without surfacing which specific check tripped. This means the fastest real debugging path is almost always bypassing the application entirely and running openssl s_client -connect host:443 -servername host directly, which prints the actual certificate details, chain, and specific validation error, rather than trying to interpret a vague error bubbling up from deep inside an application's HTTP library.

Think about a company badge that grants employees building access, set to automatically expire on a date printed on the card. Chain-of-trust is like the badge itself being fake or issued by an organization security doesn't recognize at all — the badge's authenticity is in question. Expiry is a completely different problem: the badge is 100 percent genuine, issued by a fully trusted authority, but the date printed on it has simply passed, and the door scanner is doing exactly its job refusing entry to an out-of-date badge, no matter how authentic it is otherwise.

TLS certificates carry an explicit validity window, notBefore and notAfter timestamps, specifically so compromised or outdated key material has a natural expiration, limiting how long a leaked or weakly-secured private key remains dangerous if it's ever exposed. This is a deliberate security design choice, not an arbitrary inconvenience, which is also why the industry, led by browser vendors, has steadily shortened maximum certificate lifetimes over the years, from years down to 398 days, with further reductions proposed, to force more frequent rotation across the entire ecosystem.

When a certificate expires, every TLS client's validation step explicitly checks the current time against the certificate's validity window as one of several independent checks, alongside chain validity, hostname match, and revocation status. Critically, this check can fail even when the certificate chain is otherwise completely valid and correctly served, which is why openssl s_client -connect internal-user-auth-service:443 -servername internal-user-auth-service showing a correct, complete chain but a nonzero verify return code specifically pointing to expiry, code 10, tells you immediately this is a pure date problem, not a chain, hostname, or trust problem — a much narrower, faster diagnosis path than starting from 'something's wrong with TLS' and checking everything from scratch.

In production, the operational lesson from a 2am silent expiry is almost never about handshake diagnostics — the real gap is that nothing was watching the expiry date proactively. Mature setups run a scheduled job, or use a dedicated certificate-monitoring tool, that checks every production certificate's days-until-expiry on a recurring basis and pages well before expiry, commonly at 30 days, 14 days, and 3 days out, specifically so a renewal failure, an ACME client bug, a manual process someone forgot, a certificate issued for the wrong hostname, gets caught and fixed during business hours, long before the actual expiry moment silently breaks things at 2am with nobody watching.

The non-obvious gotcha: an expired certificate incident often isn't actually about the certificate lifecycle at all — it's frequently a symptom of an automated renewal process that itself silently failed weeks earlier, a cron job for an ACME or Let's Encrypt renewal that started erroring out after a permissions change, or a rate limit hit against the CA, and nobody noticed because the only alert anyone had configured was 'did the certificate actually expire,' not 'is the automated renewal process itself healthy.' The fix isn't just renewing the certificate; it's diagnosing and repairing whatever quietly broke the automation days or weeks before the visible outage occurred.

Imagine a large household with a dozen different subscriptions, insurance, magazine, gym membership, domain name registration, each with its own renewal date, some auto-renewing and some not. A well-organized household keeps one master calendar with every renewal date logged, checked weekly, with reminders set well in advance of each one. A disorganized household relies on each subscription's own notification email arriving on time and someone happening to read it, which works fine until one email lands in spam or a card on file expires, and suddenly a service everyone depends on stops working with no warning at all.

Certificate expiry monitoring at scale requires exactly that master-calendar approach rather than trusting individual reminders, because the TLS layer itself gives you no built-in fleet-wide visibility. A certificate expiring is invisible until a client tries to connect and fails, by which point it's already an outage. This is why mature platform teams build a dedicated inventory and scanning system rather than relying on whatever renewal notification email each CA happens to send, since those emails go to whichever address was used at issuance time, which drifts out of date as teams and ownership change over months and years.

A typical implementation runs a scheduled scanner, a cron job, or a tool like the Prometheus blackbox exporter's TLS probe module, or a dedicated cert-monitoring service, that iterates over a maintained list of every TLS endpoint, internal services like payments-api.internal and checkout-worker.internal, and external-facing domains, connects via TLS, extracts the leaf certificate's notAfter date, computes days-remaining, and exposes it as a labeled metric, something like ssl_cert_expiry_days with a service label. An alerting rule engine, Prometheus Alertmanager or equivalent, then evaluates that metric against tiered thresholds, routing a 30-days-out warning to a ticket or Slack channel, and a sub-7-day warning to an actual page, since at that point the risk of a missed manual renewal turning into an outage is genuinely high.

In production, the actual operational challenge is less about the scanning and alerting mechanics and more about inventory completeness. New services spin up constantly, and unless certificate provisioning is wired into the same automated pipeline that registers a new service into the scanning inventory, for example via infrastructure-as-code that both provisions a certificate and registers the endpoint with the monitoring scanner in the same change, it's extremely easy for a newly-launched service to silently fall outside the safety net until its first, and possibly only, certificate quietly expires months later without anyone noticing.

The non-obvious gotcha: teams often build solid expiry alerting for public-facing HTTPS endpoints but completely miss internal service-to-service certificates, mTLS certificates used inside a service mesh, or internal CA-issued certificates for database connections, because those aren't reachable by a scanner probing from outside the network the way a public website is. The monitoring coverage needs to explicitly include internal-only endpoints and internal CA-issued certificates, not just anything visible from the public internet, or you end up with a false sense of complete coverage that has a large blind spot exactly where outages are hardest to diagnose, deep inside internal infrastructure.

Think about a household with an automated bill-pay system for a mortgage. A well-designed system doesn't wait until the day payment is due to attempt the transfer — it tries several days early, specifically so that if the transfer fails, insufficient funds, an expired linked card, a bank system outage, there's still time to notice the failure and fix it manually before the actual due date arrives and the mortgage goes unpaid. A poorly-designed system that only attempts payment on the due date itself turns any hiccup into an automatic, immediate crisis with zero recovery time left.

Automated certificate renewal needs the same built-in buffer, and this is exactly how tools like cert-manager, the standard Kubernetes-native certificate automation controller, and most ACME clients are designed: renewal is attempted well before actual expiry, commonly at two-thirds of the certificate's lifetime elapsed, specifically to create a wide window for retries and human intervention if something goes wrong, rather than a renewal system that only tries once, right at the deadline, leaving no slack at all for anything to go sideways.

Internally, the automated flow typically works like this: a controller, cert-manager or a custom ACME client wrapper, tracks each certificate's expiry and schedules a renewal attempt at a defined threshold before expiry. The renewal process proves domain or service ownership to the certificate authority, via ACME's HTTP-01 challenge, serving a token at a well-known URL, or DNS-01, creating a specific TXT record, DNS-01 being necessary for wildcard certificates or internal-only hostnames that aren't publicly reachable. It then receives the newly issued certificate, and has to actually deploy or reload it into the running service, a Kubernetes Secret update triggering a pod restart, or a live reload in something like nginx or envoy, and that final deployment step is itself a place automation can silently fail even after successful issuance completes.

In production, the critical monitoring isn't just days-until-expiry, as in general certificate monitoring; it's specifically whether the last renewal attempt succeeded, exposed as its own metric or event, cert-manager emits Kubernetes events and conditions on the Certificate resource itself showing Ready or NotReady and the reason, with alerting on renewal failure treated as urgent regardless of how many days remain until actual expiry. A renewal failure today with 45 days of runway left is a fix-it-this-week problem, while the same failure discovered with 2 days left is an emergency — the failure event itself is the leading indicator you actually want to catch early, not the countdown to expiry.

The non-obvious gotcha: teams often get the issuance and renewal automation right but forget that the CA challenge step can fail in ways specific to infrastructure changes. For example, a DNS-01 challenge automation that writes a TXT record via a cloud provider API can silently break after an unrelated DNS provider migration or an IAM permission change, and because the certificate doesn't actually expire for weeks after that break, the failure sits invisible until the buffer window runs out. Teams that alert only on 'certificate expiring soon' rather than 'last renewal attempt status' discover this kind of silent automation breakage far too late, often only once actual expiry is imminent and options for a graceful fix have narrowed.

Imagine a bouncer at an exclusive club told to admit anyone showing a valid membership card and, if asked, a letter from the club's parent organization confirming the club itself is legitimate. Some visitors already have a copy of that confirmation letter in their wallet from a previous visit, so they don't need to be shown one again — they just flash their card and walk in, and it looks like the letter didn't even matter. A first-time visitor without that letter already in hand, though, gets stopped cold at the door if the club forgot to have a copy ready to show on the spot.

This is exactly the leaf, intermediate, root certificate chain problem. A server's TLS certificate, the leaf, is signed by an intermediate Certificate Authority certificate, which is itself signed by a root CA pre-trusted by operating systems and browsers. Browsers were engineered to be forgiving: modern browsers implement AIA chasing, meaning that if a server doesn't send the intermediate certificate, the browser follows a URL embedded in the leaf certificate, the Authority Information Access extension, to fetch the missing intermediate itself, and browsers also cache intermediates seen from previous connections to any site using the same CA. This forgiving behavior exists to maximize the chance a real user can browse the web without a broken-looking error page, even when server operators misconfigure their TLS setup.

Most non-browser clients — a Java HTTP client, Python's requests library, curl without special flags, mobile app TLS stacks — implement strict chain validation: they only build a trust path using exactly the certificates presented during the TLS handshake's Certificate message, plus their local trust store of root CAs. If the server's TLS configuration only serves the leaf certificate and omits the intermediate, these clients have no path from the leaf to a trusted root and fail validation outright, with an error like unable to get local issuer certificate, even though the exact same server works fine when checked in a browser that happened to have or fetch the missing piece on its own.

In production, the diagnostic step is openssl s_client -connect payments-api.internal:443 -showcerts, which shows exactly what the server actually sent during the handshake. If you see only one certificate in the output, the leaf, instead of the leaf plus one or more intermediates, that confirms the server is misconfigured. The fix is almost always concatenating the leaf certificate with the required intermediate certificate or certificates, in the correct order, into the server's configured certificate file, and reloading the web server or load balancer, never touching anything on the client side.

The gotcha: teams sometimes fix this by adding the missing intermediate to the client's trust store instead of fixing the server, which appears to solve the immediate problem but is fragile and wrong. It only fixes that one client, doesn't fix the underlying misconfiguration for every other API consumer or future service talking to the same endpoint, and if the CA ever rotates its intermediate certificate, which does happen periodically, the client-side workaround silently breaks again, while a properly-configured server would have kept working transparently through the rotation.