PopoPlug Memory Limitations on Upgrading

Hi bodhi,

Are there practical limits on these upgrades for things like the PopoPlug E04 due to the memory size?

When I go through the forum posts, memory issues never seem to be mentioned, as though it's never an issue. However, I see on this device, even with a swap file, that all the memory is consumed on some occasions.

A typical example, even after enabling a swapfile and moving /var/log to /tmp/var/log the device is freezing during package upgrades, an example is shown below at the worst possible time:

update-initramfs: deferring update (trigger activated)
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for dbus (1.12.20-0+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for initramfs-tools (0.133+deb10u1) ...
update-initramfs: Generating /boot/initrd.img-5.2.9-kirkwood-tld-1

As you can see, it's actually frozen during the boot image creation, unfortunately, it's hung at this point so I'm going to have to kill it.

Rhodess,

I'm writing this post in details so I can add it to the Wiki.

Yes, it seems to be a memory problem.

Check for memory-related settings before you try to upgrade again.

See how much swap space and where it is. The swap space should be at least 4x the RAM. So for this Pogo V4 box, the swap file should be 512M or greater.

free -h
swapon

And check swapiness. For a server with swap space on HDD, keep it at 60. For a server with swap on USB such as this one, it should be somewhere from 10-60 (10 would be ideal for everyday usage, greater than 10 would be better for upgrading kernel/Debian). For low memory box with GUI (whether using a real monitor or using Remote Dektop), set it to 10 or lower will improve GUI responsiveness.

# sysctl -a | grep swap

vm.swappiness = 60

And this is one of most important settings, the sweetspot for a low memory box such as 128MB RAM is:

# sysctl -a | grep vm.min_free_kbytes

vm.min_free_kbytes = 16384

If vm.min_free_kbytes is too large then you would have problem doing kernel or Debian upgrade. These activities use a lot of memory, and a large vm.min_free_kbytes would take away too much from the RAM . If it is too small, you would have problem booting because a lot of modules loading also need more memory.

The optimum setting for this 128MB box is 16384.

For boxes that has lot more RAM (e.g. 512MB), set this vm.min_free_kbytes to 65536 to ensure booting will not run into memory problem with a heavy rootfs.

And after changing the vm parameters, you could reload it with sysctl -p. If that does not work, just do a reboot.

========

By the way, the correct name is Pogoplug V4 (or V4 Mobile, or just Mobile if there is no SATA slot) , not E04. The model name is V4, E02, V3 (OXNAS variants: Pro, B21, P21,....).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 02/18/2021 11:23PM by bodhi.

Hi bodhi,

Here is what I currently have:

root@debian:~# free -h
              total        used        free      shared  buff/cache   available
Mem:          112Mi        94Mi       1.0Mi       0.0Ki        16Mi        13Mi
Swap:         1.0Gi        73Mi       950Mi
root@debian:~# swapon
NAME                     TYPE  SIZE USED PRIO
/media/data/swapfile.img file 1024M  91M   -2
root@debian:~# sysctl -a | grep swap
vm.swappiness = 60
root@debian:~# sysctl -a | grep vm.min_free_kbytes                                                   
vm.min_free_kbytes = 1295

I added the following:

root@debian:~# sysctl vm.swappiness=30
vm.swappiness = 30
root@debian:~# cat /proc/sys/vm/swappiness                           
30
root@debian:~# echo 16384 > /proc/sys/vm/min_free_kbytes
root@debian:~# sysctl -a | grep vm.min_free_kbytes
vm.min_free_kbytes = 16384

I assume I just play with the swappiness to find the sweetspot?



Edited 1 time(s). Last edit at 02/13/2021 06:55PM by Rhodess.

All seems to be good now bodhi,

I assume that I need to add the vm.min_free_kbytes = 16384 to the /etc/sysctl.conf to survive reboots correct?



Edited 1 time(s). Last edit at 02/13/2021 07:31PM by Rhodess.

> All seems to be good now bodhi,

Cool!

> I assume that I need to add the vm.min_free_kbytes
> = 16384 to the /etc/sysctl.conf to survive reboots
> correct?

Correct. Better yet, create a local.conf so that you know what you've changed to the official Debian distribution:

Here is in one of my boxes (this GoFlex Net has 128MB RAM and the swapfile on HDD, so I kept swapiness at 60). I disabled ipv6 to avoid problem, too. It is not needed for local NAS.

# cat /etc/sysctl.d/local.conf

#
# /etc/sysctl.d/local.conf
# 
# Configuration file for setting system variables
# supplement to /etc/sysctl.conf
#

# increase the free page caches for VM
vm.min_free_kbytes = 16384
vm.swappiness = 60

# disable IPV6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

I would change swapiness to 10 on 128MB RAM boxes if the swapfile is on USB (yours is).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks bodhi,

I did all that and it's much better. I have one more question, I have webmin installed and the gui is pretty slow at loading, also when I try to look at the log file, everything gets garbled as shown below:

root@debian:~# cat /var/webmin/miniserv.log
10.0.1.126 - - [22/Jan/2021:23:35:03 -0800] "¸H4ò5tâ“[iR}ˆ_’QüS:æ—bA.FÚØBÓ¡ŒÌ ¢÷+£%ÆÄ€Dñ        {P°@ìÕã\æˇ-_eVD”¿$¿+¿/éî¿,¿0¿" 400 74
10.0.1.126 - - [22/Ja+/2021:23:36:15 -0800] "GET / HTTP/1.1" 401 4853
10.0.1.126 - - [22/Ja+/2021:23:36:18 -0800] "GET /++a++he++ica+ed/c__/b++d+e.+i+.c__?19709999999999900 HTTP/1.1" 200 148024
10.0.1.126 - - [22/Ja+/2021:23:36:18 -0800] "GET /++a++he++ica+ed/c__/f-++_-_-b-+-.+i+.c__?19709999999999900 HTTP/1.1" 200 406059
10.0.1.126 - - [22/Ja+/2021:23:36:18 -0800] "GET /i+age_/fa+ic-+_/+eb+i+/a+d_-id-ch_-+e-192|192.-+g HTTP/1.1" 200 6851
10.0.1.126 - - [22/Ja+/2021:23:36:18 -0800] "GET /i+age_/fa+ic-+_/+eb+i+/fa+ic-+-16|16.-+g HTTP/1.1" 200 1146
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:33 -0800] "POST /_e__i-+_+-gi+.cgi HTTP/1.1" 302 0
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:41 -0800] "GET / HTTP/1.1" 200 70946
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:41 -0800] "GET /++a++he++ica+ed/+_/defa+++_.+_?19709999999999900 HTTP/1.1" 200 5123
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:41 -0800] "GET /++a++he++ica+ed/+_/b++d+e.+i+.+_?19709999999999900 HTTP/1.1" 200 488201
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:42 -0800] "GET /e|+e+_i-+_/_+a+_/_+a+_.+i+.+_?19709999999999900 HTTP/1.1" 200 2555
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:43 -0800] "GET /i+age_/fa+ic-+_/+eb+i+/a--+e-+-+ch-ic-+.-+g HTTP/1.1" 200 6368
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:48 -0800] "GET /i+de|.cgi/?|h_-ge+_+_e__+e+e+=1 HTTP/1.1" 200 1
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:49 -0800] "POST /i+de|.cgi/?|h_-+he+e_+e+-_da+a=1&|h_-+he+e_+e+-_da+a_ac+i-+=ge+&|h_-+he+e_+e+-_da+a_+a+e=g-+-&|h_-+he+e_+e+-_da+a_+a++e=fa+_e&|h_-+he+e_+e+-_da+a_+ee-=0 HTTP/1.1" 200 0
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:50 -0800] "GET /i+de|.cgi?|h_-+a+age-c-+fig=1&+-ad=1 HTTP/1.1" 200 2
10.0.1.126 - _h-de__ [22/Ja+/2021:23:36:53 -0800] "GET /i+de|.cgi/?|h_-defa+++=1 HTTP/1.1" 200 12
"
"
"
"
10.0.1.70 - _h-de__ [14/Feb/2021:16:43:57 -0500] "POST /-ac+age-+-da+e_/ HTTP/1.1" 200 15906
10.0.1.70 - _h-de__ [14/Feb/2021:16:44:33 -0500] "POST /fdi_+/ HTTP/1.1" 200 5489
10.0.1.70 - _h-de__ [14/Feb/2021:16:46:45 -0500] "GET /i+de|.cgi/?|h_-i+f-=1&+eb-_efi|= HTTP/1.1" 200 8432
_--+@debia+:~# ^C
_--+@debia+:~#

I assume again a memory issue, so I moved the log files to tmp as follows:

mkdir -p /tmp/var/log
cp -aR /var/log/* /tmp/var/log
mount -o bind /tmp/var/log /var/log
 cp /var/webmin/miniserv.log /tmp/var/log/
mount -o bind /tmp/var/log/miniserv.log /var/webmin/miniserv.log

How do I get that to survive a reboot?

Moving the log files speeds it up, but I still get the corrupted output, which is slightly different when I see it, vs when I copy and paste per the attached image

Rhodess,

mkdir -p /tmp/var/log
cp -aR /var/log/* /tmp/var/log
mount -o bind /tmp/var/log /var/log
cp /var/webmin/miniserv.log /tmp/var/log/
mount -o bind /tmp/var/log/miniserv.log /var/webmin/miniserv.log

Put these lines in /etc/rc.local at the end (before the last line: exit 0). Or put them in a script at /root, and place the execution in the same palce.

/etc/rc.local
...
...
...
# those statments go here (or the script). If using the script, it needs to be full path (eg. /root/movelog.sh).

exit 0

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks, i'll try that for now, and maybe later look at mounting on the other USB and use logrotate.

I was thinking doing something like this:

mkdir -p /media/data/log/
cp -aR /var/log/* /media/data/log
mount -o bind /media/data/log /var/log
cp /var/webmin/miniserv.log /media/data/log/
mount -o bind /media/data/log/miniserv.log /var/webmin/miniserv.log

and then adding something in the logrotate.conf

vi etc/logrotate.conf

/media/data/log/<syslog.log> { # one line for each generated logfile here
maxsize 1024k
weekly
rotate 9
postrotate
killall -HUP syslogd # another killall here for each generated log
endscript
}

What are your thoughts?

Rhodess,

Here is some of what I use for daily logrotate (/var/log was bindmounted to /tmp/var/log).

/etc/logrotate.conf

/var/log/messages {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 8
    compress
}

/tmp/rsync.backup.daily.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 4
    compress
}

Note that /var/log/messages should be rotated daily if it is in /tmp. But if it is in a storage media such as USB or HDD, you don't need to rotate it too often, weekly is fine. And also no need to limit the compressed version (e.g rotate 8). In the setup above, /var/log/messages will only take up max 2MB of RAM.

Since my released Debian rootfs has syslogd logs to RAM, it needs to be rotated and compressed. This is to prevent if some process is running away and keeps logging constantly to /tmp and eventually the box will crash becasue it runs out of memory.

# ps -deaf | grep syslog

root      1634     1  0 15:38 ?        00:00:00 /sbin/syslogd -C128

If you change syslogd to log to a folder on disk then you won't have that potential OOM problem, but you will wear out the USB drive. Regularly logging to HDD is fine.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 02/15/2021 06:51PM by bodhi.

Bodhi -thanks for all your good advice, but:
I disabled ipv6 as you mentioned above, -and lost my ability to access my shared disks (I am able to see my pogoplug v4 on the network, able to see the names of my disks, but unable to look at their contents). When I undo the ipv6 disablement , I resume all normal browsing and writing. (all other variable were left the same). Then I was able to reproduce this again.
Somewhere in the past, I came upon a discussion stating that sometimes windows needs ipv6 to work right. I noticed the same with my windowsXP, but did not go back and redo a few times.

My home network has 1 win7, 1 windowsXP, and 1 macbook air.

oops-
i was using a pogoplug v4; 5.11.4 kernel; SAMBA 4.9.5; with rootfs on a sdhd; 3 usb's ( 2 ext4 and 1 fat32) and 1TB hdd ext4. shares all authenticated -no guest. drives mounted and owner / permissions were set for my network.(the fat32 was mounted with group id).