How log2ram works.

log2ram

I investigated log2ram package mechanism.
You will find out how log2ram works after reading this post!

What is log2ram?

This is introduced in this page.

Just a moment...

log2ram changes the physical location of /var/log/ .
/var/log/ is HDD or microSD (if Raspberry Pi) .
log2ram makes /var/log/ to point RAM disk which is under /tmp/ .

What are the benefits?

Lifetime of storage can be extended because the counts to write storage by log can be decreased.

Raspberry Pi has /var/log/ on microSD.
Therefore, every time you write a log with syslog, etc., a write operation to microSD occurs.
(In fact the timing when physical writing occurs from kernel buffer to microSD depends on other conditions like “sync”)

So, logging affects the life of microSD.
The write limit for microSD depends the quality of microSD itself.
For example TLC microSD, is only about 1000 times!
In the worse case if you write the log too much, you may not be able to read any data in the microSD and Raspberry Pi does not work.

Types of microSDWrite limit (approximate)
SLC100,000 times
MLC3,000 times
TLC1000 times
Source: https://flashdba.com/2014/07/03/understanding-flash-slc-mlc-and-tlc/

Therefore, somebody wants to handle data that frequently writes such as logs on RAM.
With log2ram, the process of storing logs can be realized without changing anything!

log2ram itself is provided for debian as well.
The source code is available on github.

GitHub - azlux/log2ram: ramlog like for systemd (Put log into a ram folder)
ramlog like for systemd (Put log into a ram folder) - GitHub - azlux/log2ram: ramlog like for systemd (Put log into a ra...

File structure

Once installed, it will be deployed as following structure.

PathRole
/etc/log2ram.confConfiguration file
/etc/cron.daily/log2ramScripts for daily execution by cron
/etc/logrotate.d/log2ramScript for logrotate
/etc/systemd/system/log2ram.serviceService registration file
/usr/local/bin/log2ramMain logic file
/usr/local/bin/uninstall-log2ram.shUninstall script

Behavior – Startup –

/usr/local/bin/log2ram start works at startup.
/var/log/ is the default location for the target directory.
It can be changed with log2ram.conf.
The code that moves at start time is as follows:

start)
        IFS=';'
        for i in $PATH_DISK; do
            PATH_FIRST_PART=$( echo ${i%/*} )
            PATH_LAST_PART=$( echo ${i##/*/} )
            RAM_LOG=$i
            HDD_LOG=$PATH_FIRST_PART/hdd.$PATH_LAST_PART
            LOG2RAM_LOG="${RAM_LOG}/${LOG_NAME}"
            
            make_log_dir
            
            mount --bind $RAM_LOG/ $HDD_LOG/
            mount --make-private $HDD_LOG/
            wait_for $HDD_LOG

            if [ "$ZL2R" = true ]; then
                createZramLogDrive
                mount -t ext4 -o nosuid,noexec,noatime,nodev,user=log2ram /dev/zram${RAM_DEV} ${RAM_LOG}/
            else
                mount -t tmpfs -o nosuid,noexec,noatime,nodev,mode=0755,size=${SIZE} log2ram $RAM_LOG/
            fi
            wait_for $RAM_LOG
            syncFromDisk
        done
        ;;

PATH_DISK =”/var/log”.
Therefore, HDD_LOG=/var/hdd.log/.
Bind /var/log/ to /var/hdd.log/, and then branch by the value of ZL2R.

ZL2RBehavior
trueFormats /dev/zram0 in ext4 and mount to /var/log/
falseMount s/var/log/ as tmpfs

zram is a RAM disk with automatic compression/decompression mechanism provided by kernel. Finally, copy the contents of /var/hdd.log/ to /var/log/ with syncFromDisk and log2ram startup is finished.
After that, when you write a file to /var/log/below, it will be written to the created ext4 or tmpfs. The original /var/log/ is copied to /var/hdd.log/ .

Notes – Startup –

If there is any process to write to /var/log/ in parallel with the log2ram startup process, the log contents may be inconsistent depending on the timing.
In the table of below, the red character is syslog, and the blue character is the log2ram startup process.
In B, syslog writes to /var/log/, and then copies the contents
of /var/hdd.log// to /var/log/. If syslog/log2ram handles the same file name, it may not be what you expect.

Normal Case (A)Cases of inconsistency (B)
Writes something to /var/log/
Binds /var/log/ to /var/hdd.log/
/copy /var/hdd.log/ to /var/log/
Binds /var/log/ to /var/hdd.log/
Writes something to /var/log/
/copy /var/hdd.log/ to /var/log/

To avoid this kind of inconsistency please think about startup sequence of log2ram to start earlier than any other startup process that have possibility to write log file.

Behavior – Shutdown –

The behavior at the shutdown is as follows:

stop)
        IFS=';'
        for i in $PATH_DISK; do
            PATH_FIRST_PART=$( echo ${i%/*} )
            PATH_LAST_PART=$( echo ${i##/*/} )
            RAM_LOG=$i
            HDD_LOG=$PATH_FIRST_PART/hdd.$PATH_LAST_PART
            LOG2RAM_LOG="${RAM_LOG}/${LOG_NAME}"
            
            syncToDisk
            #ZRAM_LOG=$(awk '$2 == "/var/log" {print $1}' /proc/mounts)
            #ZRAM_LOG=$(echo ${ZRAM_LOG} | grep -o -E '+')[0-9]
            umount -l $RAM_LOG/
            umount -l $HDD_LOG/
            # Unsure as even with Root permision denied
            #echo ${ZRAM_LOG} > /sys/class/zram-control/hot_remove
        done
        ;;

The following code works in syncToDisk which is called in above shutdown.
The contents of /var/log/ are copied to /var/hdd.log/.

syncToDisk () {
    isSafe
    #INITIAL_STATE=$(remountRW)

    if [ "$USE_RSYNC" = true ]; then
        rsync -aXv --inplace --no-whole-file --delete-after $RAM_LOG/ $HDD_LOG/ 2>&1 | tee -a $LOG2RAM_LOG
    else
        cp -rfup $RAM_LOG/ -T $HDD_LOG/ 2>&1 | tee -a $LOG2RAM_LOG
    Fi
    #remountOriginal ${INITIAL_STATE}
}

The purpose of doing this is to keep logs on HDD/microSD.
At startup/var/log/ mounts on zram or tmpfs, or RAM disk. Therefore, if you do not write back to HDD/microSD, the log will deleted when you turn off computer.
log2ram copies /var/hdd.log/ to /var/log/ immediately after turning on computer.
/var/ itself is on HDD/microSD, so /var/hdd.log/is also HDD/microSD.
So if you copy log files to /var/hdd .log/, it will remain after shutdown.

Notes – Shutdown –

Above code works when the shutdown process is performed.
It is never called if the power supply is cut off.
– Unplug Raspberry Pi power cord
– Power failure
– Others

There is also log integrity issue during shutdown as well as startup.
To avoid it, let log2ram finish at the very end.

What is zram?

zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk, , but with on-the-fly “disk” compression.

https://en.wikipedia.org/wiki/Zram

As you can see in above zram is RAM disk with on-the-fly compression/decompression function in the successor to the kernel module called compcache.
Compressing and retaining saves more RAM size than tmpfs, and allows application processes to handle without being aware of compression/decompression.

Some parameters, such as compression methods, can be set in the application.
log2ram sets in createZramLogDrive().

echo ${COMP_ALG} > /sys/block/zram${RAM_DEV}/comp_algorithm
echo ${LOG_DISK_SIZE} > /sys/block/zram${RAM_DEV}/disksize
echo ${SIZE} > /sys/block/zram${RAM_DEV}/mem_limit

What if we can use as personal use?

As also written in wikipedia, the main uses are as follows.
– A. Temporarily data location such as /tmp.
– B. Swap area.

To think about A, we may be able to use cache for browser.

Conclusion

How was it?

It may be useful with Raspberry Pi because it usually uses microSD as main storage.

If you are struggling to guarantee product lifetime due to microSD, log2ram is one of good solution!

Comments

  1. LinuxNoob says:

    Thank you for the great article, Yatch!
    I have a question about the /var/hdd.log/

    I understand the importance of writing to /var/hdd.log/ in case of shutdown.

    1. Does log2ram write /var/log to /var/hdd.log/ daily?
    2. Does /var/log become completely mounted to the ram or is there a main copy that stay on the microsd?
    3. What happens to the files when the cron job for log2ram hits the task deadline?

    I am trying to understand the behavior.

    Thanks again!

    • yatch says:

      Hello LinuxNoob!
      Thank you for being interested in this article!

      1. Does log2ram write /var/log to /var/hdd.log/ daily?
      -> Yes.
      We can find “log2ram-daily.service” file in log2ram package.

      This seems to restart log2ram mechanism once a day.
      – ExecStart=/bin/systemctl reload log2ram.service
      reload means stop and start.

      In “log2ram stop”, /var/log/ and /var/hdd.log/ are unmounted.
      When unmount is done, data in kernel memory is surely flashed to device.

      Be careful to confirm the timing when “log2ram-daily.service” runs because this kind of file is usually controlled by systemd or other daemon.
      You need to think about avoiding shutdown or power off when “log2ram-daily.service” runs.

      2. Does /var/log become completely mounted to the ram or is there a main copy that stay on the microsd?
      -> /var/log/ is mounted zram or tmpfs after “log2ram start”.
      So we can think /var/log/ is mounted to ram.

      3. What happens to the files when the cron job for log2ram hits the task deadline?
      -> Sorry I couldn’t get your point of “the task deadline”.
      I assumed your point is like below. Am I correct?
      “What happens to file when the cron job for log2ram and write operation to file by other process are done at the same time?”

      • LinuxNoob says:

        Thanks for the information, Yatch. I know understand more about the process.

        To your last point, I have log2ram process running on a pi with pihole. After running df -h command, I found out that log2ram is full at 100%. when the systemctl task for log2ram is executed, the log2ram stays full at 100%. I assumed that log2ram will reset to 0 after service execution.

        I am now thinking that entire /var/log stays in the ram even after flashing.

        is there any remedy beside deleting logs?

        • yatch says:

          Hello LinuxNoob!

          log2ram-daily.service simply runs log2ram with “stop” and “start” command line argument.
          In “start”, log2ram calls “syncFromDisk” to synchronize files from /var/hdd.log/ to /var/log/.
          In “stop”, log2ram calls “syncToDisk” which synchronize from /var/log/ to /var/hdd.log/

          syncFromDisk/syncToDisk uses either rsync or cp.
          – rsync if rsync command is available.
          – cp if rsync command is not available.

          If you want to delete some files in /var/log/, I would suggest to install rsync because “–delete-after” is specified to use.

          “–delete-after” deletes files in destination directory which doesn’t exist in source directory.

          In case of using “cp” log2ram just adds/overwrites files to destination.
          – No files are removed.

          I hope this comment answers to you.

          If there is any unclear point, feel free to ask me.

          Thanks!

          • Emil says:

            Hi,

            Thanks for the explanation. I have a question regarding using log2ram with zram:

            Is it normal for /var/log to be mounted to a new zram after writing to disk? For example if /var/log was mounted on zram0, it now becomes mounted to zram1, then to zram2 etc. Trying to figure out if there is anyway of avoiding this.

            I have a problem where logs stop working at midnight and I believe this might be part of the problem.

            Thanks.

          • yatch says:

            Hello, Emil.
            Thanks for commenting.

            I haven’t faced this kind of problem and am not sure but I guess there is some problem at midnight ( = 0:00 ) and may link log2ram-daily.service, other daily batch files or corn daily jobs.

            In my experience umount and mount may happen when 0:00 comes and fails because device number increases 0 -> 1 -> 2 -> …

            However I cound not find umount/mount from log2ram which may be executed at 0:00.
            – umount/mount is executed by start/stop of log2ram.
            – log2ram-daily.service performs only rsync or cp command.

            Is /dev/zram0 mounted to some other path than /var/log when problem happens?

            Thanks!

  2. Emil says:

    Hello, thanks for the quick answer.

    After a reboot I have a zram0 with /var/log plus a zram1 as swap, normal swap deactivated.
    After midnight zram0 remains as well as zram1, but zram2 created with /var/log mounted.

    Recently rebooted, I have to check later after midnight if anything gets mounted to zram0, perhaps it is then used as swap as well as zram1, but cannot say for now. No mounting in sudo crontab -e, or crontab -e.

    Thanks

タイトルとURLをコピーしました