Containers isolation by namespace

This page details the additional steps required to isolate containers using Linux user namespaces.

The procedure differs depending on whether the Cyberwatch instance already has Docker volumes containing data or not:

Instance without existing volume (new installation): follow the section “Isolating an instance without existing volume”.
Already installed instance (existing volumes): follow the section “Isolating an instance with existing volumes”.

Isolating an instance without existing volume

Ensure that Docker and Cyberwatch are installed

Create the cyberwatch user:

sudo useradd --create-home --shell /bin/bash cyberwatch

Check that the cyberwatch user has a range of ids in the /etc/subuid and /etc/subgid files:
```
grep '^cyberwatch:' /etc/subuid /etc/subgid
```
The command must return a single line per file, for example:
```
/etc/subuid:cyberwatch:231072:65536
/etc/subgid:cyberwatch:231072:65536
```
cyberwatch:231072:65536 means that the cyberwatch user can use 65536 user ids starting from id 231072: root inside the containers then corresponds to id 231072 on the host.
Edit the /etc/docker/daemon.json file to enable the userns-remap option:
```
{
  "userns-remap": "cyberwatch"
}
```
If the file already exists, add the userns-remap key to the existing JSON object.
Check the Docker configuration:
```
sudo dockerd --validate --config-file /etc/docker/daemon.json
```
The command must return configuration OK.
Restart docker:
```
sudo systemctl restart docker
```
Set the CBW_NGINX_UID and CBW_NGINX_GID variables by following the section nginx certificate folder.

For a new installation, then continue the installation of Cyberwatch with the cyberwatch configure command.

Isolating an instance with existing volumes

Enabling userns-remap changes the location of Docker data (/var/lib/docker/<uid>.<gid>): Docker then creates new empty volumes, not mapped to the existing volumes. The images must also be downloaded again: plan for the required disk space, as the old Docker data remains on the disk.

Without a database backup performed before isolation and restored afterwards, the instance data will not be recovered and the instance will start empty.

To enable isolation on an already installed Cyberwatch instance without losing its data:

Perform a backup of Cyberwatch before making any changes, following the Procedure to backup and restore Cyberwatch.
Set up isolation by following all the steps in the section “Isolating an instance without existing volume” above.
On an offline instance, load the images into Docker again: the images loaded before isolation are no longer visible.
Reconfigure Cyberwatch:
```
sudo cyberwatch configure
```
Restore the backup made in step 1, following the Procedure to backup and restore Cyberwatch.

nginx certificate folder (Cyberwatch 5.31 and later)

Starting with Cyberwatch 5.31, nginx certificates are stored in the /etc/cyberwatch/ssl folder on the host instead of a Docker volume. On startup, Cyberwatch assigns this folder to the user running nginx in its container and restricts access to this user only:

Cyberwatch 5.31: root (UID and GID 0)
Cyberwatch 5.32 and later: nginx (UID and GID 101)

With userns-remap, these UID and GID do not correspond to the same user on the host and inside the containers. The nginx container can then no longer read its certificates, which prevents it from starting (container restarts in a loop, HTTPS interface unavailable).

To fix this behavior, you need to indicate to Cyberwatch the host UID and GID corresponding to the nginx user of the containers.

Identify the host UID and GID corresponding to root inside the containers. When userns-remap is active, Docker displays them in the name of its active data directory:
```
docker info --format '{{.DockerRootDir}}'
```
The command returns a path like /var/lib/docker/<uid>.<gid>, for example /var/lib/docker/231072.231072, meaning that root inside the containers corresponds to UID 231072 and GID 231072 on the host.
Deduce the values to enter depending on the Cyberwatch version:
- Cyberwatch 5.31: the UID and GID of root, i.e. 231072 in the example
- Cyberwatch 5.32 and later: the UID and GID of root increased by 101, i.e. 231173 in the example
Enter these values in the /etc/cyberwatch/config.env file, for example for Cyberwatch 5.32 and later:
```
CBW_NGINX_UID=231173
CBW_NGINX_GID=231173
```
Restart Cyberwatch to apply the new permissions:
```
sudo cyberwatch restart
```
When setting up isolation, this step is not needed: the cyberwatch configure command of the procedure applies the new permissions.

Before updating an isolated instance from Cyberwatch 5.31 to version 5.32 or a later version, manually or automatically, add 101 to the values of CBW_NGINX_UID and CBW_NGINX_GID in the /etc/cyberwatch/config.env file. Cyberwatch 5.31 also works with these values, which are applied when the update restarts Cyberwatch.

Troubleshooting

Issues that may arise from enabling userns-remap are generally related to permissions on volumes.

If, after enabling userns-remap on an already installed instance, the application starts without its data, this means that new volumes have been created: restore the backup made before isolation (see the section “Isolating an instance with existing volumes”).

It may be useful to check the logs of the database container to rule out permissions issues:

sudo cyberwatch logs db

Since Cyberwatch 5.31, if the nginx container does not start or if the HTTPS interface is unavailable after enabling userns-remap or after an update, check the configuration of CBW_NGINX_UID and CBW_NGINX_GID (see the section nginx certificate folder). The nginx container logs help with diagnosis:

sudo cyberwatch logs nginx

A Permission denied error on the /etc/nginx/ssl/nginx.crt file indicates incorrect values.