apt-get upgrade, systemd refuses to reload daemons

My dockstar server running debian stretch complains when doing updates. I got this a couple months ago, didn't get any useful hits on the problem, and avoided it since the server appeared to still do everything fine.

I updated just now and got several errors like this:

Warning: smbd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 10.0M are free, but a safety buffer of 16.0M is enforced.

Still, I get relatively few hits on the topic. I don't know what to do about it... it seems I need to have a bigger /run mount, but I'm not sure. Maybe these errors don't matter? They look horrible, and I even get them multiple times for the same daemons as it continues to try to restart them during apt-get upgrade.

Do you have everything in a single file system? If so you are dangerously short of available disk space. You could try running:

apt-get autoremove
apt-get clean

The first command will auto remove packages that were installed to satisfy dependencies for other packages but are now no longer required. The second command deletes all .deb files from /var/cache/apt/archives and can free-up significant volume of disk space. You should also look into what is consuming the available disk space. The 'du' command can be helpful in determining what is using your disk space.



Edited 1 time(s). Last edit at 05/25/2017 01:57PM by rayknight.

No, not short on disk space. I rebooted and all services appear to be working fine.

The bolded item is my rootfs. The red item is where the problem lies, I believe.

eric@deimos:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             59M     0   59M   0% /dev
tmpfs            13M  2.2M   10M  18% /run
/dev/sda5       585G  2.9G  553G   1% /
tmpfs            61M     0   61M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            61M     0   61M   0% /sys/fs/cgroup
ubi0:rootfs     232M  159M   73M  69% /srv/nand
/dev/sdb1       296G  252G   29G  90% /samba/server
/dev/sdb3       785G   21G  724G   3% /samba/backups
/dev/sdc1       932G  501G  432G  54% /samba/media
tmpfs            13M     0   13M   0% /run/user/1000

Edited 1 time(s). Last edit at 05/25/2017 02:22PM by Nematocyst.

Yes, it is the /run mount. The reason rebooting works fine is that systemd isn't trying to reload anything, simply start the services. I can work around the problem by stopping each service and then starting it as such: [edit] 'restart' works too. the issue is 'systemctl daemon-reload' which isn't called during a reboot

[● root@deimos /home/eric] systemctl stop smbd
[● root@deimos /home/eric] systemctl start smbd
[● root@deimos /home/eric] systemctl stop nmbd
[● root@deimos /home/eric] systemctl start nmbd

That, however is a pain to do for each service after an upgrade. A better solution is by issuing the following before running upgrades:

[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=32M,nosuid,noexec,relatime,mode=755

and then restoring the original size afterwards with:

[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=13M,nosuid,noexec,relatime,mode=755

Edited 1 time(s). Last edit at 05/26/2017 09:26AM by Nematocyst.

I got this also during dist-upgrade Jessie -> Stretch on GoFlexNet.

There's a thread about it on Debian bug forums - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862013

One of the responders ("Michael Biebl") indicates that minimum memory for Debian now is 256MB, which I think is incorrect, the link he points to says actually a minimum of 128MB (and I checked for ARM, as this was a AMD64 link) ... Anyhow ...

A semi-permanent fix, as suggested later on in the same thread above, is to add this to /etc/fstab :
... bla bla bla ...
tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes=4096 0 0
... bla bla bla ...

I tested below 18M and it didn't fix the problem, 18M seems to be the lowest for "systemctl daemon-reload" command to still work without complaint (and therefore presumably apt-get install, update, upgrade, remove etc)

Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com

Don,

There is no hard minimum memory for Debian. It depends what you run on the system. 64MB is fine for a lot systems that run a few dedicated aps.

-bodhi
===========================
Forum Wiki
bodhi's corner

Hey bodhi ... yes, I understand, sorry I didn't mean to mislead anyone ! ... the link cited in the debian bugs site ( https://www.debian.org/releases/wheezy/amd64/ch03s04.html.en ) suggests that Debian may run in as little as 20MB in some configs ... the same page for Stretch indicates 60MB ... 128MB is their "recommended bare minimum" for both wheezy and stretch, although they'd generally prefer that we had more RAM.

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com

Thanks Don, that makes sense!

-bodhi
===========================
Forum Wiki
bodhi's corner

YW @bodhi :)

I think this is going to be a common problem for those upgrading Jessie -> Stretch on lower memory devices, such as some of the devices we hack/modify/improve here ...

A preferable solution would be a mod to the daemon-reload process so it doesn't have a hard limit of 16M, but I don't know how to hack that.

BR

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com

DonCharisma Wrote:
-------------------------------------------------------
> I got this also during dist-upgrade Jessie -> Stre
> tch on GoFlexNet.

Yeah, it seemed to be a stretch issue. I didn't have this on jessie.

> A semi-permanent fix...is to add this to /etc/fstab :
> tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes= 4096 0 0

I haven't tried it. My reported solution didn't involve fstab, b/c I just tried to match what 'mount' reported for /run, only with more memory. The problem with fstab is that it reduces overall resources elsewhere permanently, instead of during upgrades only.

> so happy to know if there's "a better" solution ..

the nr_inodes discussed in that thread might be a good idea instead of the default. I don't know.

However, I have since removed systemd from my server, and I no longer need any workarounds.
edit: this may not be advisable for everyone. after removing systemd, I had a power outage, and the dockstar failed to boot. I have modded my dockstar so that it has an easily accessible serial console, and I was able to figure out the problem-- one of the mounts was FAT32, and the mandatory fsck ran out of memory! anyway, be careful with this solution unless you have a way to access the console, netconsole should be ok

Sorry I didn't reply sooner, I don't hit this forum very often.



Edited 1 time(s). Last edit at 07/25/2017 03:20PM by Nematocyst.

I ran into this issue again after upgrading a Dockstar from jessie to stretch and decided to try to find a more permanent solution. I resolved it as follows:

• Edit the /usr/share/initramfs-tools/init file and modify line with 'mount -t tmpfs -o "noexec,nosuid,size=10%,mode=0755" tmpfs /run' by changing 10% to 15%
• Run the command: update-initramfs -u
• Rerun command from boot directory to regenerate uInitrd file (i.e. mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-4.17.2-kirkwood-tld-1 -d initrd.img-4.17.2-kirkwood-tld-1 uInitrd)

On a 128MB system this should result in a /run directory with 16M Avail after system has booted which meets the minimum required by the systemd safety buffer.

Thank you!

rayknight Wrote:
-------------------------------------------------------
> I ran into this issue again after upgrading a
> Dockstar from jessie to stretch and decided to try
> to find a more permanent solution. I resolved it
> as follows:
>

>
• Edit the /usr/share/initramfs-tools/init file
  > and modify line with 'mount -t tmpfs -o
  > "noexec,nosuid,size=10%,mode=0755" tmpfs /run' by
  > changing 10% to 15%
  >
• Run the command: update-initramfs -u
  >
• Rerun command from boot directory to
  > regenerate uInitrd file (i.e. mkimage -A arm -O
  > linux -T ramdisk -C gzip -a 0x00000000 -e
  > 0x00000000 -n initramfs-4.17.2-kirkwood-tld-1 -d
  > initrd.img-4.17.2-kirkwood-tld-1 uInitrd)
  >

>
> On a 128MB system this should result in a /run
> directory with 16M Avail after system has booted
> which meets the minimum required by the systemd
> safety buffer.

Revisiting this issue yet again as it keeps popping up on my systems with only 128MB RAM. It appears that an update to initramfs-tools has a new configurable option in the /etc/initramfs-tools/initramfs.conf file. This option is RUNSIZE and is set to the default value of 10% which means less than 12MB will be available in /run on a 128MB system causing issues when reloading services. So to permanently resolve this problem take the following steps:

1. Edit /etc/initramfs-tools/initramfs.conf file and change the RUNSIZE=10% line to RUNSIZE=20M
2. Run the command: update-initramfs -u
3. run the appropriate command to regenerate your uInitrd (i.e. "mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-4.18.4-kirkwood-tld-1 -d initrd.img-4.18.4-kirkwood-tld-1 uInitrd")
4. reboot your system

This will set the size of the /run mount point to 20M which should be more than adequate to ensure the 16M needed for restarting services is available. The next two steps will ensure the updated size of the mountpoint is in your uInitrd image. After the reboot your /run mount point should show a size of 20M.

NOTE: This should only be necessary on systems with less than 256M of RAM as the 10% default would be adequate for systems with 256M or more.

rayknight Wrote:
-------------------------------------------------------
> Revisiting this issue yet again as it keeps
> popping up on my systems with only 128MB RAM. It
> appears that an update to initramfs-tools has a
> new configurable option in the
> /etc/initramfs-tools/initramfs.conf file. This
> option is RUNSIZE and is set to the default value
> of 10% which means less than 12MB will be
> available in /run on a 128MB system causing issues
> when reloading services. So to permanently
> resolve this problem take the following steps:
>
Apparently the updated initramfs-tools package is not available in stretch its only available in buster and sid!

> Apparently the updated initramfs-tools package is
> not available in stretch its only available in
> buster and sid!

That does not sound right! It is available on Debian stretch.

-bodhi
===========================
Forum Wiki
bodhi's corner

bodhi Wrote:
-------------------------------------------------------
> > Apparently the updated initramfs-tools package
> is
> > not available in stretch its only available in
> > buster and sid!
>
> That does not sound right! It is available on
> Debian stretch.

bodhi the updated package version 0.132 is only available on buster and sid. Version 0.130 is the latest available for stretch and doesn't have the new RUNSIZE parameter which can be modified to increase the size of the /run mount point to prevent the "Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd" messages.

There are ways to install buster packages on stretch, but it would be nice if they would include this package in stretch updates.

Ray

rayknight Wrote:
-------------------------------------------------------
> bodhi Wrote:
> -------------------------------------------------------
> > > Apparently the updated initramfs-tools
> package
> > is
> > > not available in stretch its only available
> in
> > > buster and sid!
> >
> > That does not sound right! It is available on
> > Debian stretch.
>
> bodhi the updated package version 0.132 is only
> available on buster and sid. Version 0.130 is the
> latest available for stretch and doesn't have the
> new RUNSIZE parameter which can be modified to
> increase the size of the /run mount point to
> prevent the "Failed to reload daemon: Refusing to
> reload, not enough space available on
> /run/systemd" messages.
>
> There are ways to install buster packages on
> stretch, but it would be nice if they would
> include this package in stretch updates.
>
> Ray

Ah. In that case, instead of using the initramfs-tool, couldn't you just use fstab tmpfs to deal with this?

-bodhi
===========================
Forum Wiki
bodhi's corner

Never had I thought I would run into this issue with my Seagate GoFLEX Home and Dockstar, too. :(

DonCharisma Wrote:
-------------------------------------------------------
> Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...
>
I added the above with size=32M to my Seagate Dockstar running debian stretch (see below) and so far it works. I will update my Seagate GoFLEX Home a few days later on if this addition doesn't give any problems.

[OLD] /etc/fstab:
tmpfs        /tmp        tmpfs        defaults                        0       0

[NEW] /etc/fstab:
tmpfs        /run        tmpfs        nosuid,noexec,size=32M,nr_inodes=4096   0       0