Keeping Watch Over Keys: Card-Check Ceremonies for Our Entrust nShield HSMs
The keys that sign OpenSSL releases live inside Entrust nShield HSMs, guarded by quorums of smart cards. This is why we check every card, every month, and the small tool we built to run the checks.
In September last year we announced that OpenSSL Corporation was moving its production code signing onto Entrust nShield 5c Hardware Security Modules. That integration is now live. Two network-attached units sit at the core of our signing infrastructure, in two separate locations, and the private keys we use to sign OpenSSL release artifacts are generated and held inside them. The keys cannot be copied, exported, or leaked, because they never exist outside the HSM in any readable form. That is the point of a hardware root of trust: the nShield 5s cryptographic module inside each 5c appliance holds a FIPS 140-3 Level 3 validation, and our units run in FIPS mode.
The first release signed through this infrastructure was openssl-3.1.9, released on 24 June 2026 to our support customers. The first public release signed this way ships on 25 August 2026.
Before the HSMs, our signing key was a file on a server. A well-protected file, but a file, and anyone who got far enough into the infrastructure could in principle have copied it and signed a rogue artifact with it. That risk is now closed. What remains is a different question: who is allowed to operate these devices, and how do we stop any single person, including us, from doing it alone?
Why no one can act alone
nShield HSMs protect administrative operations with smart cards split across a set of trusted custodians. Two kinds of card sets matter here:
- An Administrator Card Set (ACS) authorizes administrative operations on the Security World itself: resetting devices, creating card sets, and other tasks at that level.
- An Operator Card Set (OCS) authorizes use of the OpenPGP primary key: the long-term, certification-capable key that defines the identity of our signing certificate. The primary key never signs release artifacts directly; it is used solely to certify signing subkeys. Under our Release Artifacts Signing Policy, using it requires a 2-of-4 quorum of human-held cards. Day-to-day release signatures are produced by a signing subkey operated by the release automation, which does not require cards.
Neither set is held by one person. Each is split into N cards using a K-of-N quorum scheme and distributed among custodians, and no protected operation can proceed until a quorum of K cards has been presented. Nobody can use the primary key or reconfigure the Security World alone. The primary key's cards are distributed equally between people designated by the OpenSSL Foundation and people designated by the OpenSSL Corporation. The quorum exists to stop any one person acting alone and to survive the loss of a card; it is not designed to force both organisations into every operation.
Custodians do not need to stand next to the HSM to present a card. Entrust's Trusted Verification Device, a small card reader with a PIN pad, connects to a custodian's own computer and establishes a secure channel to the HSM over the network. Our last ceremony ran over a video call with every custodian remote except one, each presenting their card from their own desk.
The quiet risk of quorum schemes
The quorum model removes the single point of compromise. It introduces a different exposure: cards get lost, damaged, or degrade sitting in storage, and custodians move on or forget passphrases they set once and never typed again.
The arithmetic is unforgiving. A K-of-N set can survive the loss of up to N minus K cards and still be reconstituted from the remainder. Lose one more and the set, and whatever it protects, cannot be recovered by anyone, ever. In a 3-of-5 set, for example, you can lose one or two cards and rebuild from the three that remain. Lose a third and there is no recovery path.
What would that mean for us in practice? Signing itself would keep working, because the signing subkey is operated by the release automation and does not require cards. But no primary-key or administrative operation would ever be possible again: no new signing subkeys certified, no new card sets, no changes. The only way forward would be rebuilding the Security World from scratch, generating new keys, and revoking and re-announcing the old ones. For PGP that is unpleasant. For Windows code signing it also means ordering a new extended validation certificate from a certificate authority, which takes time we would rather never need.
So the only real defence is finding a failing card before the count crosses the line, with time in hand to reissue it. Reissuing requires a full ACS quorum ceremony of its own. That is why we run card-check ceremonies monthly: scheduled sessions where every custodian in a set proves their card is present, readable, and opens with its passphrase. Every card, not just a quorum's worth, because the whole point is catching a problem while the set is still intact. Done well, a full check takes five to ten minutes.
Why we built our own tool
Entrust ships solid command-line tooling for administering nShield HSMs, but it is built for administrators sitting at the client host, not for a distributed group of custodians who each need to prove their own card works. Turning a routine, recurring check into "everyone gets SSH access and learns the CLI" did not scale, and it left us stitching together our own record of who checked in and when.
So we built a small internal web application that custodians use from an ordinary browser. It runs a ceremony against a chosen card set, and as each custodian presents their card the page updates live for everyone watching. When the ceremony finishes, it leaves a permanent audit record: which cards were checked, when, and by whom.
Some notes on how it is built, for the curious:
- It is a small Python service, one to two thousand lines on FastAPI, written in about a week. Python was the natural choice because Entrust provides a Python API for interacting with the HSMs.
- Live status reaches the browser over server-sent events, so a ceremony reads like a shared checklist. When a custodian in one country inserts a card, everyone on the call sees it.
- It talks to the HSM exclusively through Entrust's own supported client software. It never bypasses it or reimplements any cryptography.
- It runs on the HSM's client host, so a card check never depends on, or interferes with, the network path used for actual signing.
- It can do exactly one thing: check that a card is present and that its passphrase opens it. It cannot issue cards, create keys, or sign anything. Card issuance stays with Entrust's own tooling and a full quorum ceremony.
The passphrase check happens inside the HSM itself. The application never sees or stores passphrases or key material; it holds only the custodian and card metadata and the audit log.
It is an internal tool today. Once it has had more field time and some planned hardening, we intend to open-source it, most likely under Apache 2.0 like the rest of our work. This looks like exactly the operational gap other teams running nShield HSMs with distributed custodians have hit too, and we would rather share the fix than have everyone build their own.
If you run nShield HSMs and have never checked your cards, do it this week. Entrust's built-in tooling can do it without anything like our application. The worst time to discover a dead card is the moment you need it.
The bigger picture
None of this changes what the HSM already guarantees: the signing key never leaves it, and no single person can operate the Security World alone. What card-check ceremonies add is confidence that the quorum itself stays healthy over time, card by card, well before we would ever need to find out the hard way that it did not.
