How to suppress clamd using 100% when running clamonacc ?

Anti Virus

CPU usage of Raspberry Pi 4 became 100% when I activated clamonacc for real time scan…
Let me introduce how I solved this issue!
If you are struggling the same issue, this post should be useful for you!

If you want to know how to deploy nextcloud and clamonacc on Raspberry Pi 4, these posts matches your needs.
– Wheel click is recommended to open these post to open another tab.

I noticed that nextcloud’s response is slow

First I put nextcloud, then clamonacc.

The response of nextcloud became apparently slow before installing clamonacc.

yatch
yatch

It may occurs by using CPU by clamonacc!
Let’s check the CPU/RAM usage.

This is the screen shot of htop when I visit https://yasufumi-yokoyama.gq/nextcloud/.
You can see CPU usage is very high even if I did not intend to trigger high CPU usage.

Below screen shot is taken when I did not access nextcloud.
Of course CPU usage is low as expected.

So I came up the assumption that the problem happens when I access to nextcloud.

Investigation root cause

yatch
yatch

nextcloud will be useless if it always uses CPU…
What is root cause? Let’s find out.

First of all, there is no problem in idle state, and when access comes to any file on nextcloud, it seems that there is a trigger for the CPU utilization explosion, so let’s focus on clamonacc.

Let’s run clamonacc with debug output.

pi@raspberrypi:~ $ sudo clamonacc --verbose --foreground --move=/opt/clamav/quarantine



I triggerred accessing nextcloud with running clamonacc with debug output like above command line.
Then, there was interesting output on console.
I found that real time scanning was trigerred when I access to nextcloud.
– config.php, shipped.json, …
And console output flood did not stop even after a long time.

Of course only accessing nextcloud does not ovrwrite config.php, or other files.
Therefore, it seems clamonacc scans every file when not only write operation but also read operation.

ClamFanotif: attempting to feed consumer queue

Based on the debug output, I investigated source code.

yatch
yatch

Long time no investiation open source software!

First, download the source code.

git clone https://github.com/Cisco-Talos/clamav-devel.git

I also installed the necessary packages for the build to be done later.

sudo apt install autoconf libtool libxml2-dev libssl-dev libcurl4-openssl-dev

I put output message to grep command to find out where this message is output.

grep -r 'attempting to feed consumer queue' .
./clamonacc/fanotif/fanotif.c: logg("ClamFanotif: attempting to feed consumer queuen");

If you look at fanotif.c and other near source code, there is what we need to focus on.

cl_error_t onas_setup_fanotif(struct onas_context **ctx)
{
    (snip)
    if (optget((ctx)->clamdopts, "OnAccessPrevention")->enabled && !optget((ctx)->clamdopts, "OnAccessMountPath")->enabled) {
        logg("ClamFanotif: kernel-level blocking feature enabled ... preventing malicious files access attempts¥n");
        (ctx)->fan_mask |= FAN_ACCESS_PERM | FAN_OPEN_PERM;
    } else {
        logg("ClamFanotif: kernel-level blocking feature disabled ...¥n");
        if (optget((ctx)->clamdopts, "OnAccessPrevention")->enabled && optget((ctx)->clamdopts, "OnAccessMountPath")->enabled) {
            logg("ClamFanotif: feature not available when watching mounts ... ¥n");
        }
        (ctx)->fan_mask |= FAN_ACCESS | FAN_OPEN;
    }
    (snip)
yatch
yatch

FAN_ACCESS_PERM | FAN_OPEN_PERM ?
FAN_ACCESS | FAN_OPEN ?
It seems to trigger hook event even if opening/reading file…

Btw, kernel of Raspberry Pi OS disables CONFIG_FANOTIFY_ACCESS_PERMISSONS.

pi@raspberrypi:~/clamav-devel/clamonacc $ sudo modprobe configs
pi@raspberrypi:~/clamav-devel/clamonacc $ zcat /proc/config.gz | grep FANOTIFY
CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set

Therefore, the configuration of OnAccessPrevention can’t be used.
So “(ctx)->fan_mask” should be “FAN_ACCESS | FAN_OPEN”.

Looking at these two masks in the man page, it was as follows.
Even if you don’t write the file, the event is likely to happen just by reading it.

FAN_ACCESS
       A file or a directory (but see BUGS) was accessed (read).
FAN_OPEN
       A file or a directory was opened.

I read source code deeper, but there seems to be no way to change the behavior of mask in settings.
Therefore, this is the spec of clamav, not bug.

It may be no problem if clamonacc/clamd runs powerful computer like new PC.
However at least on Raspberry Pi 4 this behavior has big impact to performance down.

In my understanding, virus scan is not needed if file operation is read.
Am I wrong?

Thinking about solutions

Let’s think about modifying the source code to solve this problem.
All we have to do is change the mask to fire
event it only when writing, and scan it.

Do it this way.
Simply make the FAN_MODIFY and only events occur when writing.

--- clamonacc/fanotif/fanotif.c.org 2020-12-10 22:13:14.447953394 +0900
+++ clamonacc/fanotif/fanotif.c 2020-12-10 22:14:26.577192538 +0900
@@ -87,7 +87,7 @@
         if (optget((*ctx)->clamdopts, "OnAccessPrevention")->enabled && optget((*ctx)->clamdopts, "OnAccessMountPath")->enabled) {
             logg("*ClamFanotif: feature not available when watching mounts ... n");
         }
- (*ctx)->fan_mask |= FAN_ACCESS | FAN_OPEN;
+ (*ctx)->fan_mask |= FAN_MODIFY;
     }

if ((pt = optget((*ctx)->clamdopts, "OnAccessMountPath"))->enabled) {

Let’s build.
On Raspberry Pi 4 it takes about 10 minutes in total.

cd clamav-devel
./autogen.sh
./configure
make

Let’s check the behavior.
You can confirm that cpu usage does not increase much just by accessing nextcloud!

sudo ./clamonacc/clamonacc --config-file=/etc/clamav/clamd.conf --move=/opt/clamav/quarantine

Introducing to production environment

Now that we have confirmed the effectiveness of this fix.
Let’s continue to deploy to production environment
I think there are two strategies.

  1. make and make install clamav and related binaries.
  2. Use only clamonacc created by make, other than that use existing ones installed by apt.

I chose 2 because impact to existing environment should be minimal.

You can change setup.sh introduced in this article.

pi@raspberrypi:~ $ sudo nvim /etc/systemd/system/clamonacc.sh

# The follow is the contents of /etc/systemd/system/clamonacc.sh
#!/bin/bash
while [ 0 != $( systemctl status clamav-daemon | grep -q 'Self checking' ; echo $? ) ];
do
        sleep 5;
done ;
# /usr/bin/clamonacc --move=/opt/clamav/quarantine
%path_to_clamonacc% --config-file=/etc/clamav/clamd.conf --move=/opt/clamav/quarantine

Now the scan also works when the log file of nextcloud is updated.
I don’t think it’s highly likely that log file will be infected with viruses, so excluding log file from scanning target avoids performance down.

In config.php of nextcloud we can add the location of log file as follows.

sudo nvim /var/www/html/nextcloud/config/config.php

# Following is the contents of config.php
<?php
$CONFIG = array (
    (snip)
   'logfile' => '/var/log/nextcloud.log',
 );

# Change ownership as user of apache2
sudo touch /var/log/nextloud.log
sudo chown www-data:www-data /var/log/nextloud.log

# apache2 reboot
sudo systemctl restart apache2

I reported clamav.

I am not sure whether this problem is bug or expected spec of clamav.

Then I made a ticket to bugzilla.
If any progress is made, I will update it!

Conclusion

How was it?

In this way introduced this time, the behavior cannot be flexible to switch original/modified.
So let me suggest to add boolean setting to the configuration file.
If you add this code at the beginning of clamonacc, I believe you will be able to switch the behavior between original and modified one!

optget((*ctx)->clamdopts, "ScanModifyOnly")->enabled

Comments

タイトルとURLをコピーしました