External Application Config with Spring Cloud Kubernetes
Externalized configuration for Spring Boot in Kubernetes is simple with ConfigMaps and spring-cloud-kubernetes-config
A common pattern when deploying applications to a development, staging, and production environment is to build a jar or docker image one time, then supply different configuration values in the deployment for each stage. This configuration could be a Spring profile in separate yaml documents, additional properties files, environment variables, or some other configuration mechanism.
When deploying to Kubernetes, configuring a Spring application becomes a little more difficult. The option still exists to run our application with a profile, and just "enable" that profile specific application.yml. The downside here is all of our config was deployed in the docker image, so updating it requires a new deployment. We still have the option to configure environment variables in the Kubernetes deployment yaml and have Kubernetes map the provided value into the container created to run our application. A nicer option that integrates directly into the Spring bootstrap process is utilizing the Spring Cloud Kubernetes Config and a ConfigMap stored in the cluster. This allows us to define an environment specific application.yml in a Kubernetes ConfigMap and Spring will automatically find and merge the data into existing configuration properties. The added bonus to this approach is that changing the configuration only requires updating the ConfigMap and restarting the Spring context to read the new properties.
Note: this post will require access to a Kubernetes cluster or Minikube running locally and will expect you to have some operational knowledge of Kubernetes. If you do not have access to Kubernetes, you can install Minikube by following the official instructions from Kubernetes.io.
Create and Configure the project
We need a project to start with so head to Spring Initializr to generate a new project. Once there, select Spring Web Starter as the only dependency, and ensure Java, version 1.8, and latest spring versions are set. For our purpose, we don't need anything additional from the project creation, there will be other dependencies to add later. We are electing web support so that we have an API we can test rather than relying on only application logs for validation; it's more fun to see a project working when calling an API rather than reading logs. If you do not wish to set up a project from scratch then you can clone the demo repo and navigate to the spring-cloud-kubernetes-config-demo directory to see the completed project.
Add Spring Cloud Kubernetes Dependencies
Now that we have a basic project with web support, we need to add one more dependency to allow Spring to read ConfigMaps and Secrets from Kubernetes: spring-cloud-kubernetes-config with groupId: org.springframework.cloud. If you're using Maven, your pom should look similar to this (parts omitted for brevity):
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-kubernetes-config</artifactId>
</dependency>
...
</dependencies>
If you're not familiar with Spring Cloud, we have one more thing to add to the pom to get this to compile correctly, the dependency management section to allow spring cloud dependencies from a specific release train to be integrated into our project:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Greenwich.SR2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
So what does it do for us? The spring-cloud-kubernetes-config dependency is one of the spring-cloud-starter-kubernetes family. It hooks in to the Spring bootstrap process to provide an additional properties source from a Kubernetes ConfigMap and Secret that share the same name as our spring.application.name configured in the application.yml. Additionally, it doesn't require a bean or any extra configuration in the project; an autoconfiguration is responsible for instantiating the configuration beans which makes it transparent to set up once the dependency has been added to a project. More info on the Spring Cloud Kubernetes project can be found on GitHub.
Awesome! We can call it a day now. We've done it, and it's glorious!
Well, not quite. We still need to use the dependency and add some configuration to validate that it is working as expected. Then of course, we need to deploy to Kubernetes. So, next up...
Spring Configuration
Let's add some stuff to our application.yml. First, make sure that the application name is set to spring-cloud-kubernetes-config-demo so that your project will match the demo and this tutorial. Now add a couple of application configuration keys to the yaml:
app:
config: Default value
environmentVariable: ${ENVIRONMENT_CONFIG:Default value}
In this snippet, the config key is set to "Default value" under all circumstances, but the second key environmentVariable is defined to default to "Default value", or if ENVIRONMENT_CONFIG is defined on your host or the application host, then that value will be used instead.
We'll use these configuration keys to demonstrate how Spring will map data from our Kubernetes cluster and container environment into the application at deploy / startup time. Now we need to actually use these somewhere so we can see how to configure them through Kubernetes.
Using the application.yml config
The simplest way to verify these values will be to create a controller that also logs the values out during construction. This can be as simple or complex as you desire, but for my purposes, the example below will suffice.
@RestController
public class ConfigurableController {
private static final Logger LOGGER = LoggerFactory.getLogger(ConfigurableController.class);
private String externalConfig;
private String environmentVariable;
public ConfigurableController(
@Value("${app.config}") String externalConfig,
@Value("${app.environmentVariable}") String environmentVariable
) {
this.externalConfig = externalConfig;
this.environmentVariable = environmentVariable;
LOGGER.info(String.format("app.config: %s\napp.environmentVariable: %s", externalConfig, environmentVariable));
}
@GetMapping("/")
public Map<String, String> getConfig() {
Map<String, String> config = new HashMap<>();
config.put("app.config", externalConfig);
config.put("app.environmentVariable", environmentVariable);
return config;
}
}
At this point, the app should run and have a single endpoint available at http://localhost:8080/ which will return a map of response data containing the dynamic values mapped in at construction time.
{
"app.config": "Default value",
"app.environmentVariable": "Default value"
}
Now we can create some of the Kubernetes objects we'll need to deploy the application into a Kubernetes cluster.
Kubernetes ConfigMap(s)
We will take advantage of the default configurations used by the Spring Cloud Kubernetes Config dependency; the default is to look for a ConfigMap with the same name as our application if we can detect that the application is running within Kubernetes. So, we need to create a yaml to represent our ConfigMap. Since we will be using the default configuration for Spring to search for a ConfigMap of the same name as our Spring application name, make sure that the ConfigMap name here matches the name defined in your application.yml. This example will not work otherwise.
apiVersion: v1
kind: ConfigMap
metadata:
name: spring-cloud-kubernetes-config-demo
Technically this is all that's needed but it doesn't provide any configuration, much less anything that is useful to our Spring application. We can add a top level data key to the yaml where we can play any configuration's we'd like.
# app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: spring-cloud-kubernetes-config-demo
data:
application.yaml: |-
app:
config: Configuration from Kubernetes!
The data key in this example has a couple of important features; first it has a single nested key, in this case "application.yaml" and that key uses pipe hyphen (|-) to indicate all the values nested under it is a block and represents a single multi-line value. See (the Block Chomping Indicator yaml-multiline.info)[https://yaml-multiline.info/] for additional information. The important part is that it allows us to define all of the custom values of our application.yml in a single key in this ConfigMap. The other major point which is not obvious in this example is that since application.yaml is the only key in the data section, the name of it doesn't matter. We could in fact just have this:
apiVersion: v1
kind: ConfigMap
metadata:
name: spring-cloud-kubernetes-config-demo
data:
some-configs-here: |-
app:
config: Configuration from Kubernetes!
The name in this case does not matter. However, if we wanted to use our ConfigMap for more than just storing an environment specific application yaml, such as an environment variable for a bash script used to start our application, or some other important configuration that's required before Spring starts up, then we must name the key application.yaml. If we do not, then spring-cloud-kubernetes-config will be unable to find the relevant data to map in to Spring's composite property source, thus we will not have the expected configuration values applied to our application at startup time.
While we're at it, we can also play with creating another ConfigMap that stores a value which we can later use to map in to our application via an environment variable. We can use a Kubernetes Deployment to actually inject the value from our ConfigMap into our running container which will be shown below.
# environment-variable-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: environment-variable-config
data:
ENVIRONMENT_CONFIG: Configuration from Docker environment in Kubernetes!
Kubernetes Deployment
Since we have configs defined, we can move on to creating the deployment. As noted at the start of this post, this is assuming you have some Kubernetes knowledge already. So, we're going to configure a very simple deployment that should allow access to the application without needing additional infrastructure such as Ingress, or anything more complicated than having network access to the IP of the node the application is deployed on. To achieve this, we'll create a deployment that exposes the port of our application, and a service that configures Kubernetes to expose a Node Port and map that port on our node back to the application.
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
labels:
app: demo
spec:
replicas: 1
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
serviceAccountName: demo-service-account
containers:
- name: spring-cloud-kubernetes-config-demo
image: jmlw/spring-cloud-kubernetes-config-demo
imagePullPolicy: Never
ports:
- containerPort: 8080
env:
- name: ENVIRONMENT_CONFIG
valueFrom:
configMapKeyRef:
name: environment-variable-config
key: ENVIRONMENT_CONFIG
---
kind: Service
apiVersion: v1
metadata:
name: demo
spec:
selector:
app: demo
ports:
- protocol: TCP
port: 8080
nodePort: 30000
name: http
type: NodePort
We've called our app here demo, and we're expecting Kubernetes to find the docker image for the application locally. This means we have to build the source from the same docker context that Kubernetes is using. If you are using Minikube, you can easily attach your current terminal to the docker context from Minikube by running the following command.
eval $(minikube docker-env)
Now that our terminal should be configured to run docker commands in the Minikube environment, we can build the application locally and then attempt to deploy it. I've configured maven in the sample project to include the Spotify Dockerfile plugin so we can easily build our docker image with familiar tooling.
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>1.4.9</version>
<configuration>
<repository>${dockerhub.username}/${project.artifactId}</repository>
<buildArgs>
<JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
</buildArgs>
</configuration>
</plugin>
</plugins>
</build>
With this plugin, we can run ./mvnw clean compile package dockerfile:build and it will build our app into a docker image named and tagged jmlw/spring-cloud-kubernetes-config-demo:latest.
Before moving on, we need to define one more thing for our app so that our Spring Cloud Kubernetes Config dependency is able to do its job. By default current versions of Kubernetes enable RBAC (role based access control), so you'll need to grant our deployment explicit access to the Kubernetes APIs that it will need to discover ConfigMaps and Secretes that it should be allowed to read.
# rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: demo-role
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "configmaps"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: demo-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: demo-service-account
namespace: default
roleRef:
kind: Role
name: demo-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: demo-service-account
namespace: default
With this rbac.yaml, we can grant permission to the service account 'demo-service-account' read access to pods and ConfigMaps. If you need or want, you can also add "secretes" to the list of resources in the role definition.
Once the docker image is built, we can use kubectl to apply the yamls we've defined to our Kubernetes cluster and watch as the application starts up and configures itself. To actually deploy, you can create the yaml files listed above and then run kubectl apply -f rbac.yaml environment-variable-config.yaml app-config.yaml deployment.yaml.
Otherwise, if you're following the source from the demo-projects repository, then you can apply the same resulting yaml manifest by kubectl apply -f deployments/ which will deploy all yamls within the deployments directory.
Note: if you'd like to skip building the app locally or within your Kubernetes cluster, you can switch the ImagePullPolicy to 'Always' which will cause Kubernetes to pull 'latest' from Dockerhub
Validate
First, make sure the pod we deployed has started up and is healthy:
kubectl get pods
# expected:
# NAME READY STATUS RESTARTS AGE
# spring-cloud-kubernetes-config-demo 1/1 Running 1 2m
If the app is not running, debugging why the application is failing to start is outside of the scope of this post. However, the most like causes are 1) missing the env variable defined in the deployment which depends on a reference to a named ConfigMap, 2) the docker image is missing or is incompatible with your host, or 3) Java/Spring is failing to start which is likely a configuration issue.
Now that the app has started, check the logs from Kubernetes:
kubectl logs "$(kubectl get pods | grep demo-deployment | awk '{print $1}')"
You should see some log statements printed out from the construction of our controller similar to this:
2019-08-31 23:25:48.459 INFO 46773 --- [ main] c.j.s.ConfigurableController : app.config: Configuration from Kubernetes!
app.environmentVariable: Configuration from Docker environment in Kubernetes!
Now we can actually call the endpoint of our application. If you're running in Minikube, you can just run the following which will call the root endpoint on the URL of our service name demo within Minikube.
curl "$(minikube service demo --url)"
# expected output (or similar):
# {
# "app.config":"Configuration from Kubernetes!",
# "app.environmentVariable":"Configuration from Docker environment in Kubernetes!"
# }
Additional Configuration Options
On top of the basic option of mapping in a single ConfigMap's application.yaml key or the only key of a ConfigMap, you can configure Spring to search for additional ConfigMaps from namespaces outside of the current namespace. There are other configuration options available that you can find on GitHub in the Spring Cloud Kubernetes repository. One interesting option that I have yet to try in a production environment is using the spring.cloud.kubernetes.reload.enabled value set to true. This allows Spring to hot-reload configuration properties dependent on the spring.cloud.kubernetes.reload.strategy, which could be refresh, restart_context, or shutdown.
A lingering question you might have is, why not just map in environment variables like ENVIRONMENT_CONFIG above? Honestly, it's just as easy in most cases unless your application.yml has special configuration for nearly every key. The biggest drawback of mapping these in via environment variables is that all of your configurations are defined three times; once in the application.yml, once in the deployment.yaml, and once in the configmap.yaml. That leaves three potential places for typos that could cause incorrect configuration or worse, application crashes. Otherwise, relying on a little spring magic, you can just use the ConfigMap and application.yaml key to provide the configuration, and the keys do not need to exist in the packaged appliation.yml either. In my mind, this is slightly higher cognitive overhead for the huge benefit of not duplicating, misspelling, or failing to update configuration values.
As always, a full working demo for this can be found in my demo-projects repository. Any questions or problems, feel free to open an issue and I'll review as quickly as possible.
Happy coding and navigating the Kubernetes sea with a little Spring in your Boot!