Hashicorp Vault max_ttl Killed My Spring App

How to Ensure That Expiring Spring Cloud Vault Dynamic Database Secrets Are Renewed, When Reaching Hashicorp Vault's max_ttl

8-minute read
Locked Door
Image by Maike und Björn Bröskamp from pixabay

TL;DR

Spring Cloud Vault does not renew your dynamic database secret which leaves the application in a broken state when these credentials expire.

There are following options:

  1. Use a long enough max_ttl and redeploy/restart the application often enough

  2. If needed, increase the Vault system max time-to-live

  3. Restart the Spring Boot application when the database secret expires

  4. Implement database connection rotation (see next blog post for relational databases)

Disclaimer about Kotlin in code examples

All code examples use (ideomatic) Kotlin instead of Java to make them more concise and easier to read (and personally I prefer Kotlin). The same approach of the code snippets would also working with Java or other JVM languages.

Obviously, most people still use Java. That’s why all code examples will also be available in Java in a future update of this post.

Challenges with Hashicorp Vault and Secrets Management

Mongooses cannot keep secrets
Mongooses cannot keep secrets - Image by Dušan Smetana from pixabay

If you have a Spring Boot application or microservice, sooner or later (better sooner) you have to think about how to handle the secrets in your code in a secure manner. Fortunately, these days there are already really good secrets as a service tools available. The most famous and mature one is Hashicorp Vault. As a Kotlin or Java developer building a Spring Boot application you are in a lucky situation, because it is really easy to get your Hashicorp Vault integration up and running by using Spring Vault. Vault’s concept of dynamic secrets is probably THE killer feature and allows you for example to easily get unique, secure and short-lived database credentials which follow the principle of least privilege.

There are quite good examples how to get started with Spring, Vault and e.g. PostgreSQL, MySQL, MongoDB and many other databases. These examples work with surprisingly little effort. But as usually the Pareto principle also applies here. There a quite some challenges. For instance, setting up Vault in dev mode, like it is done in most examples, is easy and fast but you should never do that in production. And this is just the most obvious operational task to tackle. Here is a (not complete) list contains of things to consider:

  • securing Vault access with https/TLS

  • setting up a certificate authority (CA) for your https connection (Vault itself could support here, but you have to solve the bootstrapping problem)

  • ensure that Vault’s availability is good enough to not break your system: probably you want to have a high availability (HA) setup

  • bootstrap Vault in a secure but still maintainable way which fits your needs(aka unsealing)

  • who and how is the data in vault configured and maintained

  • which Vault authentication method to use and how to bring the needed data to your applications (also here, Vault has different supporting tools like token wrapping)

There are a lot of operational things you should think about but in this blog post I want to focus on an application-level challenge with using Vault together with Spring: What happens when the database secret reaches its maximum time-to-live (TTL)?

Spring Boot and the maximum TTL of a lease in Hashicorp Vault

Most people don’t think about the maximum time-to-life of this secret or expect that the database credentials are automatically rotated by Spring when needed. Let’s check the Spring Cloud Vault documentation:

Spring Cloud Vault does not support getting new credentials and configuring your DataSource with them when the maximum lease time has been reached. That is, if max_ttl of the Database role in Vault is set to 24h that means that 24 hours after your application has started it can no longer authenticate with the database.

This is a show stopper, because it will leave the Spring application in a broken state where no database communication is possible anymore. So, what can you do about this?

I will show you 4 solutions on how to make the setup work. The first 3 are more generically applicable with some prerequisites to apply. The 4th solution is only working for a more specific but also not too uncommon setup and provides a smoother experience. I will describe its details in the next blog post.

Let’s check our options.

Use a long enough max_ttl for the dynamic database credential

The Spring Cloud Vault documentation talks about the max_ttl of the database role in Vault, which is the maximum lease time. This duration is configurable. So, if you regularly deploy or restart your application, you can just use a long enough max_ttl.

Let’s consider you have a two-week sprint and it is guaranteed that after each sprint your application is redeployed. Then you could just configure the max_ttl to be longer than 2 weeks, for instance 16 days. A good example how to setup Vault to generate dynamic credentials for PostgreSQL can be found on the Hashicorp Learn site about Dynamic Secrets. You just have to adapt the max_ttl value to 16 days (384 hours) when creating the database role:

vault write database/roles/readonly db_name=postgresql \
        creation_statements=@readonly.sql \
        default_ttl=1h max_ttl=384h

But you cannot increase the maximum time-to-live for the secret infinitely. Hashicorp Vault has a system-wide max:

The system max TTL, which is 32 days but can be changed in Vault’s configuration file.

— Vault Concepts: Tokens

When the Vault system maximum TTL is not enough…​

Time is running out
Time is running out - Image by Monoar Rahman Rony from pixabay

…​it is also possible to increase it via the Vault configuration. The Vault Configuration documentation mentions that the max_lease_ttl parameter, which defaults to "768h" can be used for that.

An example vault config could look like:

max_lease_ttl = "3000h" (1)

storage "file" {
  path = "/var/vault/vault-storage"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1 (2)
}
1 The system max TTL is set to 3000h here, the rest of the config is just for demonstration purpose
2 You should never do that in production

Before you now just change the max_lease_ttl for all your Vault instances to a really big value, you should wait a minute. Hashicorp of course has a good reason to limit the maximum TTL and to not use a too big value by default: The longer a dynamic secret lives, the less dynamic it is in the end. So if you choose a too big time-to-live value, the risk of having really long living secrets and with that the risk of secret sprawl is increasing. That is the reason why I would not recommend changing this value if you can use one of the other options.

Restarting the application when credentials expire

If you cannot guaranty that your Spring application is redeployed or restarted often enough or you don’t want to live with the risk of long living secrets the first two options are not for you. Instead you should change the setup, so that the database secrets of the application can be rotated.

The easiest way to do that is to ensure that the application is restarted when the secret expires. Depending on your setup you already have a process manager like systemd or a container-orchestration system like Kubernetes which ensures, that your application is always running. Whenever the application stops this tool will start a new instance for you.

So how can the Spring application detect the secret is expired and cannot be renewed anymore? This can be done by adding an additional LeaseListenener to the SecretLeaseContainer.

I expect that you’ve already setup Spring and Vault to create dynamic database secrets (how to do that, see for example Managing your Database Secrets with Vault @ Spring blog or An Intro to Spring Cloud Vault @ Baeldung). Then you can autowire the SecretLeaseContainer, the database role which is configured as the property spring.cloud.vault.database.role and the ConfigurableApplicationContext to allow closing the ApplicationContext which eventually shuts down the Spring application:

@Configuration
class VaultConfig(
        private val leaseContainer: SecretLeaseContainer,
        @Value("\${spring.cloud.vault.database.role}")
        private val databaseRole: String,
        private val applicationContext: ConfigurableApplicationContext
) {

In a @PostConstruct method you can then add the additional LeaseListenener which does the shutdown:

@PostConstruct
private fun configureShutdownWhenLeaseExpires() {
    val vaultCredsPath = "database/creds/$databaseRole" (1)
    leaseContainer.addLeaseListener { event -> (2)
        if (event.path == vaultCredsPath) { (3)
            log.info { "Lease change for DB: ($event) : (${event.lease})" }
            if (event.isLeaseExpired && event.mode == RENEW) { (3)
                log.error { "Database lease expired. Shutting down." }
                applicationContext.close() (4)
            }
        }
    }
}
1 build the creds path by using the autowired databaseRole
2 LeaseListenener is a SAM interface, so just provide a lambda (see Kotlin’s SAM Conversion)
3 event.path, event.isLeaseExpired and event.mode are extension methods (see next code snippet)
4 shutdown the Spring application

And here are the extension methods used above:

private val SecretLeaseEvent.path get() = source.path
private val SecretLeaseEvent.isLeaseExpired get() = this is SecretLeaseExpiredEvent
private val SecretLeaseEvent.mode get() = source.mode

If you don’t use a tool like Kubernetes or systemd you should restart the application, instead of shutting it down.

The complete shutdown example can be found as a github repository.

Next steps

Cliffhanger …​ to be continued
Cliffhanger …​ to be continued - Image by Shri ram from pixabay

There is also the possibility to implement the database credential renewal in the application itself. Spring itself does not provide a generic way to do this. In the next blog post I’ll show a way how to do this manually if you use Spring with HikariCP as JDBC connection pool, which works for example with PostgreSQL or MySQL databases.

Before you go: Please leave a comment, question or feedback about this blog post below!