Configure the number of concurrent jobs executed by the application

This procedure explains how to modify the number of asynchronous tasks run simultaneously by the Cyberwatch application.

Understanding the tasks queues in Cyberwatch

The execution of application tasks is done asynchronously in order to improve performances and allow for parallelization of tasks.

Pending tasks are put in different waiting queues depending on the type of the task.

Queue	Services	Main purpose
default	sidekiq_master	Miscellaneous tasks.
reports	sidekiq_master	Generation of PDF exports and reports.
servers	sidekiq, sidekiq_master	Computation of asset vulnerabilities.
default:NODE	sidekiq_node	Execution of asset discoveries and analyses.
web-scanner:NODE	sidekiq_node	Execution of web application scans.

Some queues have an express variant used to speed up the processing of priority tasks. They share the same compute capacity and have no concurrency of their own, except for reports-express which reserves a share of it.

Adding Sidekiq replicas

By default, Sidekiq will only use one CPU for managing tasks.

The default value for the number of tasks is generally enough in most configurations for Sidekiq to use a whole CPU for running these tasks.

If there is a latency in the applications due to a slow execution of tasks, the first recommended approach is to allow Sidekiq to use both CPUs of the Cyberwatch server.

This will double the number of tasks that can be run concurrently by the Cyberwatch scanner.

These modifications should only be done following the recommendation of a Cyberwatch engineer.

To apply these modifications, follow the procedure below:

Open the file /etc/cyberwatch/config.env and edit or add the following lines:
```
# Number of Sidekiq replicas to start.
CBW_SIDEKIQ_REPLICAS=1
```
Replace 1 with the desired number of replicas.
Restart Cyberwatch using the standard procedure:
```
sudo cyberwatch restart
```

Configuring the number of execution threads

In addition to configuring the number of Sidekiq processes through replicas, it is possible to increase the capacity of each one by configuring its number of execution threads (threads).

Service	Variable	Limiting resources	Affected queues
sidekiq	SIDEKIQ_CONCURRENCY	CPU	servers
sidekiq_master	SIDEKIQ_CONCURRENCY	CPU	default, servers
sidekiq_master	REPORTS_CONCURRENCY	RAM	reports
sidekiq_node	SIDEKIQ_NODE_CONCURRENCY	Network	default:NODE
sidekiq_node	WEB_SCANNER_CONCURRENCY	Network, CPU, RAM	web-scanner:NODE

The concurrency of the sidekiq service can be increased on any node, contributing to the overall compute capacity of the servers queue. The other queues are specific to a given node, so the concurrency configuration must be targeted there to have the intended effect.

These modifications should only be done following the recommendation of a Cyberwatch engineer.

For example, to configure the SIDEKIQ_CONCURRENCY variable of the sidekiq service, modify the file /etc/cyberwatch/configs-enabled/99-sidekiq-concurrency.yml as follows:

version: '3.7'

services:
  sidekiq:
    environment:
      SIDEKIQ_CONCURRENCY: 4

Then restart your instance with sudo cyberwatch restart.

The other variables of the other services can be configured in the same way.

Configuring the number of parallel Docker image scans

It is possible to configure the number of Docker image scans that can be performed in parallel by the Cyberwatch application.

By default, if the variable is not configured, the value is 4 parallel scans.

Modifying this value can have a significant impact on system performance and resource consumption. It is recommended to modify this value only after consulting a Cyberwatch engineer.

Configuration for Docker Swarm and Podman

To modify this value in a Docker Swarm environment or Podman environment, follow the procedure below:

Modify the file /etc/cyberwatch/config.env and edit or add the following line:

# Number of container-scanner replicas to start.
CBW_CONTAINER_SCANNER_REPLICAS=1

Replace 1 with the desired number of replicas.

Restart Cyberwatch using the standard procedure:
```
sudo cyberwatch restart
```

Configuration for Kubernetes

To modify this value in a Kubernetes environment, follow the procedure below:

Modify the values.yml file and add the following lines:
```
containerScanner:
  replicaCount: N
```
where the N variable should take the value of the desired number of replicas for the container-scanner. Each replica can execute up to 4 scans in parallel by default.
It is recommended to have the pods on the same node, or a volume for container-scanner in RWX (ReadWriteMany).
Apply the modifications using the standard Kubernetes deployment procedure.