Error "too many open files"

A file handle is an abstract concept that includes network connections, temporary files, and disk files.

If you experience the too many open files error message, this means Magnus Box is either

(A) running at the same time as another process with high file-handle usage; or
(B) has been restricted to use a very low available amount of file handles; or
(C) is itself using an excessive number of file handles

During a backup job, Magnus Box uses approximately

approximately 10-20 handles for files being read from the disk; and
approximately 10-20 handles for open network connections; and
an unknown number of temporary cache files created during the "Building cache" phase

You may be able to work around this issue by

raising the kernel file handle limit (described below); or
ensuring no file-intensive processes are running at the same time as the Magnus Box Backup job; or
ensuring multiple Magnus Box backup jobs are not running simultaneously; or
for "File and Folder" protected items, by enabling the "Rescan unchanged files" option. This reduces the number of temporary files that Magnus Box uses for local caching. However, it may reduce the backup performance.

If you discover your device has a different limit than the Operating System default, you should find the configuration file where the limit has been altered.

A future version of Magnus Box may redesign the cache mechanism to use fewer local temporary files.

On macOS

macOS supports system and per-process limits on the number of open file handles.

The default limits are fixed at a quite low value (at the time of writing: 12288 system-wide, 10240 per-process).

You can check the current system-wide limits by running: sysctl kern.maxfiles

You can check the current per-process limit by running: sysctl kern.maxfilesperproc

On macOS 10.12.x and later, you can raise the system-wide limits by creating a .plist file in the /Library/LaunchDaemons/ directory. Please see these instructions for more information.

On macOS 10.11.x and earlier, you can raise the system-wide limits by updating the kern.* settings in /etc/sysctl.conf. Please see these instructions for more information.

On Linux

Linux supports system limits, per-process soft limits and per-process hard-limits on the number of open file handles.

The default limits are fixed at a quite high value (1048576 on Debian 10 "Buster"). However, the limits may have been lowered by the system administrator, especially if the Linux PC is a multi-tenant server, web server, container server or OpenVZ server, in order to provide a limited but consistent experience to the system tenants. Any installed Linux Security Module (LSM) such as SELinux or AppArmor may also impact the value.

Any new child process will inherit the parent process's limit values.

Per-process limits

You can check the current limits for new processes spawned by the current user account, by running: ulimit -n

You can check the available hard and soft limits for any process by running (e.g.):

Find the PID of each Magnus Box Backup process: pidof backup-tool
Check the soft/hard limits for the PID: grep files "/proc/PID/limits"

You can update the current limit for new child processes, by running: ulimit -n 10485760 (for a 10x raise from the default) or ulimit -n unlimited.

Only the root user has permission to raise their own ulimit.
This will only affect newly spawned child processes. Existing processes will retain the previous limit. You should restart Magnus Box for the changes to take affect; or, you should restart the whole PC for the changes to affect all running processes, however, any changes may not survive a reboot.

You can set the per-process file handle limit for processes spawns by systemd, by adding a LimitNOFILE=... stanza to the systemd unit file. The infinity value is supported.

System-wide limits

You can view the current system-wide number of open file handles by running: cat /proc/sys/fs/file-max.

You can view the current system-wide number of open file handles by running: cat /proc/sys/fs/file-nr, or, by running lsof | wc -l. The values may differ slightly.

The default system-wide limit might be currently set

in the /etc/security/limits.conf file, or
by any file in the /etc/security/limits.d/ subdirectory, or
in the /etc/sysctl.conf file.

You can update the system-wide limit by running:

echo "fs.file-max = 10485760" >> /etc/sysctl.conf /sbin/sysctl -p

This raises the system file handle limit 10x higher from the default, and then reloads the sysctl variables. On Debian, the sysctl program is in the procps package that also provides the pidof and ps programs.

On Windows

Each process has a limit on the total number of open handle objects. The maximum number of open handles is 16711680 (16 million) on x86_64 versions of Windows. However, it may be lower on other CPU architectures, or it might lowered by a system administrator via Group Policy.

There is also a per-session limit on the number of opened files over the network. The default value is 16384; you can see this by running: net config server.

You can see the current per-process handle count from Task Manager, on the Details tab by enabling the optional column "Handles".