Rolling update
In a rolling update, you replace the nodes of a cluster one at a time: to install a new webPDF version, to move a node to another host, or because a container platform such as Kubernetes creates a node's container again. The other nodes keep answering requests in the meantime.
Replacing a node is not the same as restarting it. A restarted node keeps its files. A replaced node starts with a new container or a new installation, and everything that is not on persistent storage is gone.
Data the coordinator must keep
Every node signs and checks the access tokens of the REST web service with
token keys that it keeps in its keystore/ folder. The token keys are protected with the configuration encryption
key in the file conf/secret.key. This file is not the cluster secret key. If a
node starts without these files, it creates new ones.
A member that receives its configuration from the coordinator (the default) also receives the coordinator's
token keys and conf/secret.key, but only once, when it joins the cluster. It does not fetch them again later.
Keep the coordinator's keystore/ folder and its conf/secret.key file on persistent storage, together: for
example on container volumes, or with a Kubernetes StatefulSet and persistent volume claims (see the
example). For the location of both, see
Directories.
Persistent on the coordinator | Result after the coordinator has been replaced |
|---|---|
keystore/ and conf/secret.key | The coordinator keeps its token keys. Tokens remain valid on all nodes. |
| Neither | The coordinator creates new token keys. A request whose token was issued by another node fails with HTTP 401 and error code -9007 ("Token provider not found.") until every member has been restarted. After that, all tokens issued before are invalid. |
keystore/ only | The coordinator creates a new conf/secret.key, can no longer open its token keys, and stops during startup. |
A member does not need these files on persistent storage: a replaced member fetches them from the coordinator
again when it joins.
With the configuration source NODE in cluster.json, a member
does not receive the token keys from the coordinator. The keystore/ folder and conf/secret.key must then be
identical on all nodes, and every node must keep them on persistent storage.
Order
- Replace the
membernodes one at a time. Before you continue with the next node, wait until the replaced node has joined the cluster again and reports that it is ready (/webPDF/health/ready, see Health). - Replace the
coordinatorlast. While it is not running, no node can join the cluster; themembernodes that have already joined keep answering requests.
If the coordinator was replaced without its keystore/ folder and conf/secret.key, restart every member after
the coordinator has joined again, one at a time, so that it fetches the new token keys. Clients then have to
request new tokens.
Warnings in the server log
From webPDF 10.0.7 on, the server log shows the following warnings:
| Warning (beginning of the log line) | Written by | What to do |
|---|---|---|
Created new token keys in '…' | A node in cluster mode that started without its token keys | On the very first start of a node in cluster mode, nothing. If the coordinator logs it after it was replaced, its keystore/ folder and conf/secret.key were not kept: restart every member, and keep both on persistent storage from now on. If a member with the configuration source NODE logs it, copy the coordinator's keystore/ folder and conf/secret.key to that member. |
The coordinator '…' has joined the cluster again after this node fetched its configuration and token keys | A member | The coordinator was restarted or replaced. After a restart, or a replacement with its keystore/ folder and conf/secret.key kept, nothing. Otherwise restart this member. |
Outside a cluster, a node that creates new token keys writes the information Created new token keys in '…'.
Versions
Within the same main and minor version (for example from 10.0.7 to 10.0.8), the nodes can run different versions for the duration of a rolling update; the cluster keeps using its data, because the namespace stays the same. Update all nodes promptly: afterwards, all nodes must run the same version again.
A node whose namespace differs from that of the cluster does not join it. This is the case with another main or minor version, with another revision of the stored data format, and with another prefix. The log of the node that tries to join says:
Namespace settings do not match: '<namespace of the coordinator>' (coordinator) != '<namespace of the member>' (member)
To change the main or minor version, or to install a version with a new revision, update all nodes together during a maintenance window. The new version then works in a new, empty area (see Version and Revision).
Blue and Green Deployment
You can start a second cluster with a new version or revision next to the running one, on the same Redis/Valkey and the same bucket: the namespaces differ, so the clusters do not touch each other's data. The second cluster needs its own cluster name (name in cluster.json) or its own network; otherwise the nodes of the two clusters find each other and try to join. The area of the old cluster stays until you empty it.