Split Mode OpenPGP CA

What is OpenPGP CA Split Mode?

In some contexts, it is useful to run OpenPGP CA split into two subsystems, for example:

  • On two distinct computers,
  • In two separate Qubes VMs on a Qubes OS system.

We call such an OpenPGP CA setup a “Split Mode CA”.

Note that split mode is a specialized mode of operation! It is not appropriate or recommended for all users.

One logical CA, two instances performing different roles

In split mode, a logical OpenPGP CA is composed of two instances, which perform different roles, and which have different security requirements.

Front instance

A split mode “front” instance usually runs on a less secured, typically internet-connected, system.

The front instance contains all user public key data, certifications, revocation certificates (if any), as well as trust signatures over the CA’s public key.

The front instance serves both for looking up information in the CA, and a point of entry for new data. New data can trigger signing requests for the CA, which are queued in the front instance, for processing by the back instance.

A front instance may, for example, be directly accessed from an “intranet” style web portal, for both read and write operations.

Back instance

In OpenPGP CA split mode, only the “back” CA instance has the ability to perform operations that use the CA private key.

A split mode back instance usually runs on a more secured system (possibly air-gapped).

The back instance serves as a mechanism to issue certifications using the CA private key - as directed by the front instance.

The back instance doesn’t store copies of public key data, certifications, etc.

Setting up a split mode OpenPGP CA system

A split mode CA is always derived from a regular (non-split) CA.

The regular CA could be a preexisting CA that you want to migrate to split mode, or a new one that you initialize just to transform it into a split mode CA.

Starting with a regular CA

Here, we’ll initialize a fresh softkey-based instance, and go from there (a card-based CA can be used in the same way):

$ oca --database example-ca.oca ca init --domain example.org softkey
Initialized OpenPGP CA instance:

    CA Domain: example.org
  Fingerprint: 69584DD5A27C8D912CFF178C52CA426517E7A847
Creation time: 2023-04-20 15:27:54 UTC
   CA Backend: Softkey (private key material in CA database)

Splitting the regular CA into a front and a back instance

Now we can split this regular CA instance (stored in the database file example-ca.oca) into a split pair of CAs, like this:

$ oca --database example-ca.oca ca split into --back example-back.oca --front example-front.oca

The result of this operation are two new CA database files. Those two CA databases will usually be used on separate systems, with different levels of protection against attacks:

  • The example-front.oca CA instance now contains all user/cert data (but no CA secret key material). This instance will typically be used on an internet-connected system.
  • The example-back.oca CA instance doesn’t contain any user or public key data, just the private CA key (or the OpenPGP card configuration, if the CA uses an OpenPGP smart card for private key operations). This instance is intended for use on a more protected system (possibly not connected to any networks).

You can inspect the two CA databases like this:

$ oca --database example-front.oca ca show
    CA Domain: example.org
  Fingerprint: 69584DD5A27C8D912CFF178C52CA426517E7A847
Creation time: 2023-04-20 15:27:54 UTC
   CA Backend: Split-mode front instance

$ oca --database example-back.oca ca show
    CA Domain: example.org
  Fingerprint: 69584DD5A27C8D912CFF178C52CA426517E7A847
Creation time: 2023-04-20 15:27:54 UTC
   CA Backend: Split-mode back instance (based on: Softkey (private key material in CA database))

At this point, the original CA database file example-ca.oca can be archived or destroyed.

Handling user public keys with the front instance

Most operations that are performed on a regular OpenPGP CA instance, are performed on the front instance, in split mode.

Regular operations on the front instance include importing or generating user keys. These operations look exactly like on a regular (non-split) CA:

$ oca --database example-front.oca user add --email alice@example.org
Created new user key.

-----BEGIN PGP PRIVATE KEY BLOCK-----
Comment: 1782 48CB E578 AF77 044D  4750 7B59 2659 3EE3 A3C2
Comment: <alice@example.org>
[...]

$ sq key generate --userid "<bob@example.org>" --export bob.priv
$ sq key extract-cert --output bob.pub bob.priv

$ oca --database example-front.oca user import --email bob@example.org --key-file bob.pub

Both of these operations imply generating a certification by the CA. On a regular OpenPGP CA instance, these certifications would exist now.

In split mode, however, so far, no certifications by the CA have been issued, because the front instance doesn’t have access to the CA key material.

We can inspect these public keys in the front CA with the user list command. Note that the output shows no “Identities certified”:

$ oca --database example-front.oca user list
OpenPGP certificate 178248CBE578AF77044D47507B5926593EE3A3C2
 Has trust-signed this CA
 No expiration is set
 1 revocations available

OpenPGP certificate D00ED5BA6E89F04B8ECA55BB095F82777A822ED2
 Expiration 20/04/2026

Inspecting the queue of certification requests in the front instance

Since a split mode front CA instance can’t generate the certifications itself, it queues certification requests for later processing.

The queue of unprocessed certification requests can be inspected like this:

$ oca --database example-front.oca ca split show-queue
Certification request [#1]
  For User IDs ["<alice@example.org>"]
  On 178248CBE578AF77044D47507B5926593EE3A3C2
  No expiration
  Queued: 2023-04-20 15:33:03 UTC

Certification request [#2]
  For User IDs ["<bob@example.org>"]
  On D00ED5BA6E89F04B8ECA55BB095F82777A822ED2
  No expiration
  Queued: 2023-04-20 15:35:08 UTC

Generating CA certifications in split mode

To generate certifications with a split mode OpenPGP CA system, a series of steps must be performed:

  • Exporting the requests from the front instance
  • Taking the requests-file to the back instance
  • Processing the requests on the back instance (with appropriate manual validation!)
  • Taking the resulting certifications to the front instance
  • Importing the certifications intro the front instance

The following sections describe these steps in more detail.

Exporting requests from the front instance

Certification requests that are queued in the front CA instance can be exported for processing by the back instance:

$ oca --database example-front.oca ca split export --file oca-requests-2023-04-20.json
Exported queue with 2 entries for processing by the back instance

The output file contains certification requests in JSON format.

The requests file format is designed to be easy to inspect manually, or with custom tools:

{
  "version": 1,
  "ca_fingerprint": "69584DD5A27C8D912CFF178C52CA426517E7A847",
  "created": "2023-04-20T15:58:19.036784892Z",
  "queue": [
    [
      1,
      "2023-04-20T15:33:03.148054844Z",
      {
        "CertificationReq": {
          "cert": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nComment: 1782 48CB E578 AF77 044D  4750 7B59 2659 3EE3 A3C2\n[..]",
          "user_ids": [
            "<alice@example.org>"
          ],
          "days": null
        }
      }
    ],
    [
      2,
      "2023-04-20T15:35:08.005581231Z",
      {
        "CertificationReq": {
          "cert": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nComment: D00E D5BA 6E89 F04B 8ECA  55BB 095F 8277 7A82 2ED2\n[..]",
          "user_ids": [
            "<bob@example.org>"
          ],
          "days": null
        }
      }
    ]
  ]
}

This exported certification request file can now be transmitted to the back instance (e.g. on a USB storage device).

Generating certifications on the back instance

On the back CA instance, we can now generate certifications for the requests from the front instance.

The point of generating CA certifications on the separate, hardened, back CA system is that we want to make sure not to issue “wrong” certifications.

There are different classes of “wrong” certifications that want to avoid issuing. Among those are:

  • Issuing certifications for malicious OpenPGP keys that an attacker tricks us into processing.
  • More mundane, but also crucial: issuing mistaken certifications with our CA by accident (e.g., on an old public key of one of our users that they have lost control over).

To allow manual verification of each certification request, processing on the back CA instance is interactive: Each certification request is shown, followed by a prompt for confirmation.

The operator of the CA should compare the key fingerprints and their connection to User IDs with a trusted separate source of truth before issuing each certification.

$ oca --database example-back.oca ca split certify --import oca-requests-2023-04-20.json --export oca-response-2023-04-20.json
Processing 2 certification requests, exported on 2023-04-20 15:58:19 UTC.

Request for User ID certification [created 2023-04-20 15:33:03 UTC]:

Bind key 178248CBE578AF77044D47507B5926593EE3A3C2 with
- '<alice@example.org>'

Certify? [y/n]


Request for User ID certification [created 2023-04-20 15:35:08 UTC]:

Bind key D00ED5BA6E89F04B8ECA55BB095F82777A822ED2 with
- '<bob@example.org>'

Certify? [y/n]


Processed 2 certification requests

After verifying that the fingerprint is correct to link with the User ID(s), we answer “y” for each certification request.

After completing the procedure, the back CA exports the resulting certifications into the file oca-response-2023-04-20.json.

Importing certifications in the front instance

We now take the response file to the front instance (e.g., on a USB storage device) for the next and final step. There, we import our newly generated certifications:

$ oca --database example-front.oca ca split import --import oca-response-2023-04-20.json
Imported 2 certifications from the back instance.

Now we can inspect these certifications in the front instance:

$ oca --database example-front.oca user list
OpenPGP certificate 178248CBE578AF77044D47507B5926593EE3A3C2
 Identities certified by this CA:
 - '<alice@example.org>'
 Has trust-signed this CA
 No expiration is set
 1 revocations available

OpenPGP certificate D00ED5BA6E89F04B8ECA55BB095F82777A822ED2
 Identities certified by this CA:
 - '<bob@example.org>'
 Expiration 20/04/2026

Note that the output now shows a certified identity for each of the keys.

Overview of Operations on a split mode CA

Queueing User ID certification requests

Certifying User IDs (when importing or creating user public keys) is one common operation that is queued on the front instance (for later processing in the back instance).

Other operations on the front CA use the same mechanism, including:

  • Refreshing CA certifications which are close to expiring,
  • Re-issuing pre-existing certifications by a previous CA key after a CA key rollover.

Setting up a new bridge

Bridges to remote CAs can also be added in split mode. The process is the same as on a non-split CA (but again, requires generation of the certification on the back CA instance).

For this example workflow, we first create an example remote CA (for the domain remote.example) to bridge to. We export that “remote” CA’s public key into remote.pub:

$ oca --database remote-ca.oca ca init --domain remote.example softkey
Initialized OpenPGP CA instance:

    CA Domain: remote.example
  Fingerprint: D7BC7AC287969ECD9D362904455514DB7972E454
Creation time: 2023-04-20 16:01:21 UTC
   CA Backend: Softkey (private key material in CA database)

$ oca --database remote-ca.oca ca export > remote.pub

In real bridging scenarios, the remote CA will usually be run by an external organization, and you will obtain the public key for the remote CA from them. You’ll then want to confirm the fingerprint of the remote CA against a separate source of truth (for example, from printed material, or in a personal meeting with the operator of the remote CA, as appropriate).

Now that we have a copy of the remote CA’s public key, we configure it as a bridge in the front instance of our split CA:

$ oca --database example-front.oca bridge new --commit remote.pub
Added OpenPGP key for openpgp-ca@remote.example as bridge.

The fingerprint of the remote CA key is
D7BC7AC287969ECD9D362904455514DB7972E454

The bridge new operation queues a certification request for the remote CA (the request instructs the back CA instance to issue a trust signature).

We can inspect the request in the certification request queue of the front instance:

$ oca --database example-front.oca ca split show-queue
Bridging request [#3]
For D7BC7AC287969ECD9D362904455514DB7972E454
Scoped to '<[^>]+[@.]remote\.example>$'
Queued: 2023-04-20 16:02:47 UTC

The trust signature certification for the remote CA’s public key can now be generated on the back instance, as shown above (in the section “Generating certifications with the back instance”).

Operations that don’t require the CA private key

All operations that don’t require the CA private key material can and should be performed on the front instance.

This includes:

  • Inspecting the CA state (reading user information, public key information, …)
  • Importing trust signatures that users issued for our CA
  • Applying user key revocations that have been escrowed in the CA instance

Other operations that do require the CA private key

All operations that require the CA private key material can only be performed on the back instance.

Of these, some require access to an up-to-date copy of the front CA instance as an additional, read-only input.

Generating a set of revocations for the CA key

Revocations for the CA private key must be produced on the back CA:

$ oca --database example-back.oca ca revocations --output oca.revocations
Wrote a set of revocations to the output file

This operation does not require a copy of the CA front instance.

Exporting a Keylist

Exporting user keys in the Keylist format requires (read-only) access to a copy of the front CA database, on the back CA machine.

An up-to-date copy of the front CA database must be made available on the back CA like this:

$ export OPENPGP_CA_FRONT_DB=example-front.oca

Now the keylist export command can be performed as it would be on a regular CA:

$ mkdir keylist.out
$ oca --database example-back.oca keylist export --path keylist.out/ --sig-uri https://example.org

Merging a split CA pair back into a regular CA

In rare cases, operators may want to migrate back from split mode operation to a regular (softkey or card-backed) CA instance. To do this, you can merge the database of the back instance into the front instance, like this:

$ oca --database example-front.oca ca split merge --back example-back.oca

(You probably want to back up the split CA database before performing a merge operation.)

After the merge operation, the content of the back instance is effectively folded into the front instance. This is the inverse operation of ca split into (see above).

At this point, you may want to rename the former front database file, for clarity:

$ mv example-front.oca example-ca.oca

This CA database now contains a regular (standalone) Softkey instance again (note the “CA Backend” line):

$ oca --database example-ca.oca ca show
    CA Domain: example.org
  Fingerprint: 8953C72BBEA7F83F8219EF241E098388CAD5D721
Creation time: 2023-04-20 16:10:46 UTC
   CA Backend: Softkey (private key material in CA database)

The back CA database file is now redundant (the CA private key has been imported into the front CA database). So the back CA database file can now be discarded.

Goals and tradeoffs

The main goal of OpenPGP CA split mode is to enable CA operators to keep the CA private key separate and away from internet-connected systems: Making the CA key easier to protect, compared to regular OpenPGP CA setups.

OpenPGP CA aims to keep the work of the CA administrator easy, even in this advanced mode.

Inspectable JSON request/response files

The interchange format between the front and back CA instance is designed to be easy to inspect (and easy to handle with custom tooling).

Enable alternate back CA implementations

Currently, only the Sequoia PGP-based back CA implementation exists.

However, the design of the interface between back and front CA makes it relatively straightforward to implement alternate back CA implementations (including ones that are based on different OpenPGP implementations).

Discussion: Keeping the CA private key material safe

When setting up a split CA system, generation of the CA private key should take place on a sufficiently secure system. You might want to use the system that the “back instance” will run on.

Copies of the CA private key might linger on systems, even after deleting them from the filesystem. In such cases, they might be potentially accessible to attackers in the future. It is good practice to never handle (unencrypted) private key material on machines with an insufficient security level.

As an extension of this point: when migrating to split mode from softkey mode, you should consider if any copies of the CA softkey might remain accessible on systems that an attacker could gain access to. This is a potentially hard problem.

(The details of purging lingering - and possibly recoverable - traces of private key material are out of scope for this text.)

Backup considerations

When backing up a split mode OpenPGP CA instance, there are different considerations to think about, for the two subsystems:

Back instance

The back instance handles the CA’s private key material. However, depending on the backend (Softkey vs. OpenPGP card), there are very different concerns to think about.

Softkey-based back instances

Softkey-based OpenPGP CA back instances store the (unencrypted) CA private key in their database file. Therefore, it is crucial to keep the CA back instance database confidential.

So backups of the back instance database should be encrypted, and/or stored in a physically hard to access manner.

Note that the back instance database doesn’t change after initialization, so there is no need to make ongoing backups.

Card-based back instances

A card-based back instance database only contains metadata about the CA, and its OpenPGP card device (including the User PIN for the OpenPGP card device).

No private key material is stored in the database, for card-based instances. Thus, the confidentiality requirements for a card-based CA back instance database are less strict than for a softkey-based instance.

However, the User PIN should also be considered sensitive, in particular against attackers with physical access to that device. Backups of the back instance database should be protected accordingly.

Availablity

With card-based instances, the primary copy of the CA private key material resides on an OpenPGP card. If that card is lost, or breaks, that copy of the CA key material becomes unavailable.

At that point, there are two possible paths forward:

  • Initializing a new OpenPGP card with the CA private key, and configuring the CA back instance to use that card instead,
  • Starting over with a different CA key. This involves a) informing all users of the CA that they need to configure their OpenPGP environments to rely on the new CA key (which may be feasible in small or very centralizes settings, but prohibitive in large and/or heterogenous settings), and b) re-certifying all bindings that the old CA key has certified with the new CA key (this is supported by OpenPGP CA, and an easy procedure to perform).

For CA keys that are generated on-card, only the second option is available (there is by definition no second copy of the CA’s private key material).

See the documentation of the OpenPGP card backend for more in depth discussion of these points.

Front instance

Front instances contain only data with a lower need for protection.

Types of data in a front instance database

Most of the data in a front instance is public, and thus doesn’t need to be kept confidential.

One exceptions to this general rule are revocation certificates for user keys!

Revocation certificates can optionally be stored in a CA for safekeeping. If published, a revocation certificate causes a user OpenPGP key to be considered invalid. Undesired publication of a revocation certificate results in a “denial of service” situation for the impacted user.

Integrity

For the front instance, one possible concern is recovering from attacks in which the data in the front instance has been altered by an attacker. Data could be removed or added to the front instance (including certification requests).

As long as the back instance is not compromised, and the CA operator carefully verifies each certification request on the back instance, such attacks don’t result in undesired certifications by the CA (this is the purpose of running a split mode CA).

However, changed data in the front instance may result in missing, or confusing extraneous information. To recover from such attacks, keeping both backups and historical snapshots of the front CA database may be beneficial. They can enable the operator to identify undesired changes, and roll back to a known good state.