Upgrade to Vault 1.14.x

The Vault 1.14.x upgrade guide contains information on deprecations, important or breaking changes, and remediation recommendations for anyone upgrading from Vault 1.13. Please read carefully.

Feature deprecations and EOL

Duplicative Docker images

As of Vault 1.14 we will no longer update the the vault Docker image. Only the Verified Publisher hashicorp/vault image will be updated on DockerHub.

Users of Official Images need to use docker pull hashicorp/vault:<version> instead of docker pull vault:<version> to get newer versions of Vault in Docker images. Currently, HashiCorp publishes and updates identical Docker images of Vault as Verified Publisher and Official images separately.

Important changes

vault.raft_storage.bolt.write.time has been corrected from a summary to a counter to more accurately reflect that it is measuring cumulative time writing, and not the distribution of individual write times.

Known issues and workarounds

Users limited by control groups can only access issuer detail from PKI overview page

Affected versions

• Vault 1.14.x

Issue

Vault UI users who require control group approval to read issuer details are directed to the Control Group Access page when they try to view issuer details from links on the Issuer list page.

Workaround

Vault UI users constrained by control groups should select issuers from the PKI overview page to view detailed information instead of the Issuers list page.

API calls to update-primary may lead to data loss

Affected versions

All versions of Vault before 1.14.1, 1.13.5, 1.12.9, and 1.11.12.

Issue

The update-primary endpoint temporarily removes all mount entries except for those that are managed automatically by vault (e.g. identity mounts). In certain situations, a race condition between mount table truncation replication repairs may lead to data loss when updating secondary replication clusters.

Situations where the race condition may occur:

• When the cluster has local data (e.g., PKI certificates, app role secret IDs) in shared mounts. Calling update-primary on a performance secondary with local data in shared mounts may corrupt the merkle tree on the secondary. The secondary still contains all the previously stored data, but the corruption means that downstream secondaries will not receive the shared data and will interpret the update as a request to delete the information. If the downstream secondary is promoted before the merkle tree is repaired, the newly promoted secondary will not contain the expected local data. The missing data may be unrecoverable if the original secondary is is lost or destroyed.
• When the cluster has an Allow paths defined. As of Vault 1.0.3.1, startup, unseal, and calling update-primary all trigger a background job that looks at the current mount data and removes invalid entries based on path filters. When a secondary has Allow path filters, the cleanup code may misfire in the windown of time after update-primary truncats the mount tables but before the mount tables are rewritten by replication. The cleanup code deletes data associated with the missing mount entries but does not modify the merkle tree. Because the merkle tree remains unchanged, replication will not know that the data is missing and needs to be repaired.

Workaround 1: PR secondary with local data in shared mounts

Watch for cleaning key in merkle tree in the TRACE log immediately after an update-primary call on a PR secondary to indicate the merkle tree may be corrupt. Repair the merkle tree by issuing a replication reindex request to the PR secondary.

If TRACE logs are no longer available, we recommend pre-emptively reindexing the PR secondary as a precaution.

Workaround 2: PR secondary with "Allow" path filters

Watch for deleted mistakenly stored mount entry from backend in the INFO log. Reindex the performance secondary to update the merkle tree with the missing data and allow replication to disseminate the changes. You will not be able to recover local data on shared mounts (e.g., PKI certificates).

If INFO logs are no longer available, query the shared mount in question to confirm whether your role and configuration data are present on the primary but missing from the secondary.

Using 'update_primary_addrs' on a demoted cluster causes Vault to panic

Affected versions

• 1.13.3, 1.13.4 & 1.14.0

Issue

If the update_primary_addrs parameter is used on a recently demoted cluster, Vault will panic due to no longer having information about the primary cluster.

Workaround

Instead of using update_primary_addrs on the recently demoted cluster, instead provide an activation token.

Safari login screen appears broken on the UI

Affected versions

• 1.14.0

Issue

The login screen on Safari appears to be broken, presenting as a blank white screen.

Workaround

Scroll down to find the login section.

AWS static roles ignore changes to rotation period

Affected versions

• 1.14.0+

Issue

AWS static roles currently ignore configuration changes made to the key rotation period. As a result, Vault will continue to use whatever rotation period was set when the roles were originally created.

Workaround

Delete and recreate any static role objects that should use the new rotation period.

Transit Encryption with Cloud KMS managed keys causes a panic

Affected versions

• 1.13.1+ up to 1.13.7 inclusively
• 1.14.0+ up to 1.14.3 inclusively

Issue

Vault panics when it receives a Transit encryption API call that is backed by a Cloud KMS managed key (Azure, GCP, AWS).

Workaround

None at this time

Transit Sign API calls with managed keys fail

Affected versions

• 1.14.0+ up to 1.14.3 inclusively

Issue

Vault responds to Transit sign API calls with the following error when the request uses a managed key:

requested version for signing does not contain a private part

Workaround

None at this time