How to suppress clamd using 100% when running clamonacc ?

Anti Virus

CPU usage of Raspberry Pi 4 became 100% when I activated clamonacc for real time scan…
Let me introduce how I solved this issue!
If you are struggling the same issue, this post should be useful for you!

If you want to know how to deploy nextcloud and clamonacc on Raspberry Pi 4, these posts matches your needs.
– Wheel click is recommended to open these post to open another tab.

I noticed that nextcloud’s response is slow

First I put nextcloud, then clamonacc.

The response of nextcloud became apparently slow before installing clamonacc.

yatch
yatch

It may occurs by using CPU by clamonacc!
Let’s check the CPU/RAM usage.

This is the screen shot of htop when I visit https://yasufumi-yokoyama.gq/nextcloud/.
You can see CPU usage is very high even if I did not intend to trigger high CPU usage.

Below screen shot is taken when I did not access nextcloud.
Of course CPU usage is low as expected.

So I came up the assumption that the problem happens when I access to nextcloud.

Investigation root cause

yatch
yatch

nextcloud will be useless if it always uses CPU…
What is root cause? Let’s find out.

First of all, there is no problem in idle state, and when access comes to any file on nextcloud, it seems that there is a trigger for the CPU utilization explosion, so let’s focus on clamonacc.

Let’s run clamonacc with debug output.

pi@raspberrypi:~ $ sudo clamonacc --verbose --foreground --move=/opt/clamav/quarantine



I triggerred accessing nextcloud with running clamonacc with debug output like above command line.
Then, there was interesting output on console.
I found that real time scanning was trigerred when I access to nextcloud.
– config.php, shipped.json, …
And console output flood did not stop even after a long time.

Of course only accessing nextcloud does not ovrwrite config.php, or other files.
Therefore, it seems clamonacc scans every file when not only write operation but also read operation.

ClamFanotif: attempting to feed consumer queue

Based on the debug output, I investigated source code.

yatch
yatch

Long time no investiation open source software!

First, download the source code.

git clone https://github.com/Cisco-Talos/clamav-devel.git

I also installed the necessary packages for the build to be done later.

sudo apt install autoconf libtool libxml2-dev libssl-dev libcurl4-openssl-dev

I put output message to grep command to find out where this message is output.

grep -r 'attempting to feed consumer queue' .
./clamonacc/fanotif/fanotif.c: logg("ClamFanotif: attempting to feed consumer queuen");

If you look at fanotif.c and other near source code, there is what we need to focus on.

cl_error_t onas_setup_fanotif(struct onas_context **ctx)
{
    (snip)
    if (optget((ctx)->clamdopts, "OnAccessPrevention")->enabled && !optget((ctx)->clamdopts, "OnAccessMountPath")->enabled) {
        logg("ClamFanotif: kernel-level blocking feature enabled ... preventing malicious files access attempts¥n");
        (ctx)->fan_mask |= FAN_ACCESS_PERM | FAN_OPEN_PERM;
    } else {
        logg("ClamFanotif: kernel-level blocking feature disabled ...¥n");
        if (optget((ctx)->clamdopts, "OnAccessPrevention")->enabled && optget((ctx)->clamdopts, "OnAccessMountPath")->enabled) {
            logg("ClamFanotif: feature not available when watching mounts ... ¥n");
        }
        (ctx)->fan_mask |= FAN_ACCESS | FAN_OPEN;
    }
    (snip)
yatch
yatch

FAN_ACCESS_PERM | FAN_OPEN_PERM ?
FAN_ACCESS | FAN_OPEN ?
It seems to trigger hook event even if opening/reading file…

Btw, kernel of Raspberry Pi OS disables CONFIG_FANOTIFY_ACCESS_PERMISSONS.

pi@raspberrypi:~/clamav-devel/clamonacc $ sudo modprobe configs
pi@raspberrypi:~/clamav-devel/clamonacc $ zcat /proc/config.gz | grep FANOTIFY
CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set

Therefore, the configuration of OnAccessPrevention can’t be used.
So “(ctx)->fan_mask” should be “FAN_ACCESS | FAN_OPEN”.

Looking at these two masks in the man page, it was as follows.
Even if you don’t write the file, the event is likely to happen just by reading it.

FAN_ACCESS
       A file or a directory (but see BUGS) was accessed (read).
FAN_OPEN
       A file or a directory was opened.

I read source code deeper, but there seems to be no way to change the behavior of mask in settings.
Therefore, this is the spec of clamav, not bug.

It may be no problem if clamonacc/clamd runs powerful computer like new PC.
However at least on Raspberry Pi 4 this behavior has big impact to performance down.

In my understanding, virus scan is not needed if file operation is read.
Am I wrong?

Thinking about solutions

Let’s think about modifying the source code to solve this problem.
All we have to do is change the mask to fire
event it only when writing, and scan it.

Do it this way.
Simply make the FAN_MODIFY and only events occur when writing.

--- clamonacc/fanotif/fanotif.c.org 2020-12-10 22:13:14.447953394 +0900
+++ clamonacc/fanotif/fanotif.c 2020-12-10 22:14:26.577192538 +0900
@@ -87,7 +87,7 @@
         if (optget((*ctx)->clamdopts, "OnAccessPrevention")->enabled && optget((*ctx)->clamdopts, "OnAccessMountPath")->enabled) {
             logg("*ClamFanotif: feature not available when watching mounts ... n");
         }
- (*ctx)->fan_mask |= FAN_ACCESS | FAN_OPEN;
+ (*ctx)->fan_mask |= FAN_MODIFY;
     }

if ((pt = optget((*ctx)->clamdopts, "OnAccessMountPath"))->enabled) {
Update 08/18/2021
To combine FAN_CLOSE and FAN_OPEN_EXEC (FAN_OPEN_EXEC_PERM) is more secure than only FAN_MODIFY.
 Thank you René for commenting!



Let’s build.
On Raspberry Pi 4 it takes about 10 minutes in total.

cd clamav-devel
./autogen.sh
./configure
make

Let’s check the behavior.
You can confirm that cpu usage does not increase much just by accessing nextcloud!

sudo ./clamonacc/clamonacc --config-file=/etc/clamav/clamd.conf --move=/opt/clamav/quarantine

Introducing to production environment

Now that we have confirmed the effectiveness of this fix.
Let’s continue to deploy to production environment
I think there are two strategies.

  1. make and make install clamav and related binaries.
  2. Use only clamonacc created by make, other than that use existing ones installed by apt.

I chose 2 because impact to existing environment should be minimal.

You can change setup.sh introduced in this article.

pi@raspberrypi:~ $ sudo nvim /etc/systemd/system/clamonacc.sh

# The follow is the contents of /etc/systemd/system/clamonacc.sh
#!/bin/bash
while [ 0 != $( systemctl status clamav-daemon | grep -q 'Self checking' ; echo $? ) ];
do
        sleep 5;
done ;
# /usr/bin/clamonacc --move=/opt/clamav/quarantine
%path_to_clamonacc% --config-file=/etc/clamav/clamd.conf --move=/opt/clamav/quarantine

Now the scan also works when the log file of nextcloud is updated.
I don’t think it’s highly likely that log file will be infected with viruses, so excluding log file from scanning target avoids performance down.

In config.php of nextcloud we can add the location of log file as follows.

sudo nvim /var/www/html/nextcloud/config/config.php

# Following is the contents of config.php
<?php
$CONFIG = array (
    (snip)
   'logfile' => '/var/log/nextcloud.log',
 );

# Change ownership as user of apache2
sudo touch /var/log/nextloud.log
sudo chown www-data:www-data /var/log/nextloud.log

# apache2 reboot
sudo systemctl restart apache2

I reported clamav.

I am not sure whether this problem is bug or expected spec of clamav.

Then I made a ticket to bugzilla.
If any progress is made, I will update it!

12/13/2021
This is what I reported.

https://bugzilla.clamav.net/show_bug.cgi?id=12644

Conclusion

How was it?

In this way introduced this time, the behavior cannot be flexible to switch original/modified.
So let me suggest to add boolean setting to the configuration file.
If you add this code at the beginning of clamonacc, I believe you will be able to switch the behavior between original and modified one!

optget((*ctx)->clamdopts, "ScanModifyOnly")->enabled

Comments

  1. René says:

    Set FAN_MODIFY only is not a good idea because it often not works and a Malware file that saved or excuted was not scanned. Better is, if you use a Kernel about 5.1, to use FAN_CLOSE and FAN_OPEN_EXEC (FAN_OPEN_EXEC_PERM). This works at my systems like a charm with very less impact.

    • yatch says:

      Thank you for great feedback!

      The latest Raspbian’s kernel is 5.10.52 or later, we should be able to use your idea.
      I haven’t had your viewpoint at all, then I will reflect to this article soon.

      Thank you again!

  2. René says:

    No problem your article helped me to test this. The onaccess scanner also caused performance problems on a Ryzen 3900x. But when I realized that fan_modified does not work as desired, I tried different combinations that seemed to make sense to me. If you test this you can also see if the onaccess scanner continues to work when you use Clamdscan? I have the problem when I use Clamdscan without –fdpass parameter the realtime scanner loses the connection to the clamd daemon after some time. If I start the onaccess scanner with –fdpass parameter clamdscan works without –fdpass and the onaccess scanner loses the connection to clamd. But the regex exceptions in the clamd.conf are ignored by the onaccess scanner which is bad because the onaccess exceptions generally don’t work for me.

    • yatch says:

      Yeah, I also noticed fdpass parameter is mandatory if OnAccessPrevention is true.
      I tried to find out the root cause by debugging clamdscan/clamd with gdb but couldn’t find.

  3. René says:

    It seems to be a bug in the bug tracker I read that there were apparently already problems with the connection between onaccess scanner and Clamd which was supposedly fixed, but apparently not properly. I have identified 3 bugs so far:

    The onaccess scanner completely ignores the onaccess exclude parameters in the config even if you include an exlude file with the -e parameter ( Bug 1 ) –> onaccess exclude so completely non-functional . Exclude via the regex exceptions in the clamd only works as long as the onaccess scan is not started with the –fdpass parameter ( Bug 2 ).

    The onaccess scanner loses the connection to clamd after a while if clamdscan is not started with –fdpass parameter (Bug 3).

    Theoretically the bad onaccess performance could also be counted as a bug, it seems to be related to the parameter FAN_ACCESS or the parameter combination FAN_ACCESS and FAN_OPEN, because with the parameter configuration FAN_OPEN and FAN_OPEN_EXEC the performance was much better. The best working configuration was FAN_CLOSE and FAN_OPEN_EXEC which should not be a security problem because executable files are still scanned when trying to run them, this has always worked reliably in my tests.

    Can you please report the bugs in the bugtracker as far as I have seen this has not been done yet.

    • yatch says:

      At that time I wrote this article I reported performance issue via bugzilla but no reaction received…

      Well, I will think about accererating fixing.

  4. René says:

    I think I know now what the problem is and it seems that when the system reads a directory it also opens the files (this is why the onaccess scanner also finds the test file immediately with FAN _Close when you open the directory it is in). Thus the onaccess scan for the same file is triggered 2 times first the FAN_Access trigger and second the FAN_OPEN trigger, and apparently clamonacc is not able to recognize that it is the same file and scans it accordingly 2 times in a row. This might explain the high load and the performance problems.

    Yes is slow typical somehow I have the feeling that many developers live by the slogan I make no mistakes.

    • yatch says:

      😮
      > when the system reads a directory it also opens the files
      > apparently clamonacc is not able to recognize that it is the same file
      I see.

      If there is technical solution to skip one scanning, performance will be better right?
      But I understand it isn’t easy how to identify at second scanning “Oh I already scanned this file(or inode?), then I don’t need to scan again” because although inode is the same contents of file might be updated and to judge neccesity of scanning requires some calculation/CPU power.

  5. René says:

    Yes, but I think it’s easier to remove an event that prevents double scanning. I had tested with FAN_OPEN that works. But since reading malware files is not a problem with Linux, it makes more sense to scan the files only when closing and as a second instance to use the FAN_OPEN_EXEC which scans executable files first before executing. This allows the system to read the files first without delay before they are given to the scanner, which prevents a startup delay of the program, especially with config files. Since copy and move events generally trigger open close events, the virus is immediately detected and blocked or removed when writing. The only problem I found is if the virus is already in the monitored directory and you access it with the terminal, the real-time scanner reacts only when you interact with the file, but I think this is not a problem because executable files are generally scanned first and such operations as executable trigger anyway a close event which calls the real-time scanner on the plan.

  6. René says:

    I think the best protection options would be the combination: FAN_OPEN(_PERM) + FAN_OPEN_EXEC(_PERM) or FAN_ACCESS(_PERM) + FAN_OPEN_EXEC(_PERM) #i only tested the combination FAN_OPEN(_PERM) + FAN_OPEN_EXEC(_PERM)#

    and the best performance option:
    FAN_CLOSE + FAN_OPEN_EXEC(_PERM)

    • yatch says:

      So there are some options of flags.
      – FAN_OPEN(_PERM) + FAN_OPEN_EXEC(_PERM)
      – FAN_ACCESS(_PERM) + FAN_OPEN_EXEC(_PERM)
      – FAN_CLOSE + FAN_OPEN_EXEC(_PERM)

      I am using clamav on Raspberry Pi for personal use.
      So the last option which focuses on performance is good rather than completeness which is other options.

      Thank you again.

  7. skr says:

    I can’t find you report on bugzilla.
    Can you please share the link?

Copied title and URL