File Scanner (New Version)

Attention

The file scanner and the main Staffcop Enterprise server must be installed on different physical servers to avoid competition for CPU, RAM, and disk subsystem resources.

Software

Requirement	Description
OS	Ubuntu 22.04,
Staffcop Enteprise	Starting from version 5.7.1
Windows Agent	Version 5.8.2604 and higher
Linux Agent	Version 0.18.37 and higher
Database	PostgreSQL version: 15

Network ports

The following ports must be open between the servers:

• 5432 — PostgreSQL

• 2049 — NFS

• 8000 — HTTP-API

Access

Ability to execute commands via sudo or an account with root privileges on both servers.

Processor and RAM

When allocating capacity resources, take the following into account:

• agent amount

• amount of scan events

• text file size

• amount of categories

• amount and complexity of categorization rules

Allocate resources for the following:

• loading data into the PostgreSQL database from the ETL spooler

• working in the web interface

Note

To speed up data display, sorting, and filtering in the web interface, PostgreSQL generates files_filecube in the background. Maintaining and updating these files consumes resources. If you have any questions about configuration or optimization, please contact technical support.

• creating categorization tasks — the producer creates a task to process each category of the file batch

Allocate capacity based on the number of agents:

Configuration	RAM	CPU	Amount of workers [1]
Up to 10 agents	4 cores	8 GB	1
10-100 agents	16 cores	16 GB	6-8
From 100 agent	32+ cores	32 GB	16+

[1]

A worker is a process for parallel file categorization. The number of workers is specified in the configuration file.

Hint

In the file scanner, searching by category works faster when using extensions and keywords than when searching by morphology and regular expressions.

File Storage Disks

Disk space is calculated based on the number of files being collected.

Recommended disk space: 2 TB, of which 1 TB is for the database.
NVMe drives are preferred.

File Scanner Configuration

You can configure the file scanner in the configuration file /etc/file-scanner/config.

Note

If you encounter any issues with configuration, please contact technical support.

Parameter	Default Value	Description
CATEGORIZATION_WORKERS	2	Number of workers for parallel processing of categories.
CATEGORIZATION_WORKER_RETRY_TIME	602	Time until the task is restarted. Must be greater than the value of CATEGORIZATION_WORKER_MAX_TIME_LIMIT.
CATEGORIZATION_WORKER_MAX_TIME_LIMIT	600	Maximum task execution time. If exceeded, the task will be closed.
CATEGORIZATION_QUEUE_MAX_SIZE_LIMIT	500000	Maximum number of simultaneous tasks.
CATEGORIZATION_QUEUE_MAX_SIZE_TIMEOUT	30 seconds	Interval for checking if a spot in the queue is free.
CATEGORIZATION_QUEUE_FROZEN_TIMEOUT	1440 minutes (24 hours)	Time after which the task is considered frozen.
CATEGORIZATION_PRODUCER_FULL_BATCH_SIZE	10000	Maximum number of tasks in the queue during one producer iteration. [2]
CATEGORIZATION_PRODUCER_DIV_BATCH_SIZE	100	Number of problems per category per pass for uniform distribution.
CATEGORIZATION_PRODUCER_TIMEOUT	5 seconds	Pause before creation of next task.
CATEGORIZATION_CONSUMER_FULL_BATCH_SIZE	10000	Tasks distributed across workers per consumer iteration. [3]
CATEGORIZATION_CONSUMER_TASKS_BATCH_SIZE	100	Tasks for one worker iteration.
CATEGORIZATION_CONSUMER_TIMEOUT	5 seconds	Pause before the next task distribution.

[2]

Producer — checks files for categorization and, if missing, places the file in the categorization queue.

[3]

Consumer — takes files from the queue, forms them into tasks, and assigns them to workers. The worker assigns categories to the files.

Last Updated: 14.07.26