Welcome! Log In Create A New Profile

Advanced

apt-get upgrade, systemd refuses to reload daemons

Posted by Nematocyst 
apt-get upgrade, systemd refuses to reload daemons
May 24, 2017 10:43PM
My dockstar server running debian stretch complains when doing updates. I got this a couple months ago, didn't get any useful hits on the problem, and avoided it since the server appeared to still do everything fine.

I updated just now and got several errors like this:
Warning: smbd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 10.0M are free, but a safety buffer of 16.0M is enforced.

Still, I get relatively few hits on the topic. I don't know what to do about it... it seems I need to have a bigger /run mount, but I'm not sure. Maybe these errors don't matter? They look horrible, and I even get them multiple times for the same daemons as it continues to try to restart them during apt-get upgrade.
Re: apt-get upgrade, systemd refuses to reload daemons
May 25, 2017 02:55PM
Do you have everything in a single file system? If so you are dangerously short of available disk space. You could try running:

apt-get autoremove
apt-get clean

The first command will auto remove packages that were installed to satisfy dependencies for other packages but are now no longer required. The second command deletes all .deb files from /var/cache/apt/archives and can free-up significant volume of disk space. You should also look into what is consuming the available disk space. The 'du' command can be helpful in determining what is using your disk space.



Edited 1 time(s). Last edit at 05/25/2017 02:57PM by rayknight.
Re: apt-get upgrade, systemd refuses to reload daemons
May 25, 2017 03:20PM
No, not short on disk space. I rebooted and all services appear to be working fine.

The bolded item is my rootfs. The red item is where the problem lies, I believe.

eric@deimos:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             59M     0   59M   0% /dev
tmpfs            13M  2.2M   10M  18% /run
/dev/sda5       585G  2.9G  553G   1% /
tmpfs            61M     0   61M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            61M     0   61M   0% /sys/fs/cgroup
ubi0:rootfs     232M  159M   73M  69% /srv/nand
/dev/sdb1       296G  252G   29G  90% /samba/server
/dev/sdb3       785G   21G  724G   3% /samba/backups
/dev/sdc1       932G  501G  432G  54% /samba/media
tmpfs            13M     0   13M   0% /run/user/1000



Edited 1 time(s). Last edit at 05/25/2017 03:22PM by Nematocyst.
Re: apt-get upgrade, systemd refuses to reload daemons
May 26, 2017 10:04AM
Yes, it is the /run mount. The reason rebooting works fine is that systemd isn't trying to reload anything, simply start the services. I can work around the problem by stopping each service and then starting it as such: [edit] 'restart' works too. the issue is 'systemctl daemon-reload' which isn't called during a reboot
[● root@deimos /home/eric] systemctl stop smbd
[● root@deimos /home/eric] systemctl start smbd
[● root@deimos /home/eric] systemctl stop nmbd
[● root@deimos /home/eric] systemctl start nmbd

That, however is a pain to do for each service after an upgrade. A better solution is by issuing the following before running upgrades:
[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=32M,nosuid,noexec,relatime,mode=755

and then restoring the original size afterwards with:
[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=13M,nosuid,noexec,relatime,mode=755



Edited 1 time(s). Last edit at 05/26/2017 10:26AM by Nematocyst.
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 06:03AM
I got this also during dist-upgrade Jessie -> Stretch on GoFlexNet.

There's a thread about it on Debian bug forums - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862013

One of the responders ("Michael Biebl") indicates that minimum memory for Debian now is 256MB, which I think is incorrect, the link he points to says actually a minimum of 128MB (and I checked for ARM, as this was a AMD64 link) ... Anyhow ...

A semi-permanent fix, as suggested later on in the same thread above, is to add this to /etc/fstab :
... bla bla bla ...
tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes=4096 0 0
... bla bla bla ...

I tested below 18M and it didn't fix the problem, 18M seems to be the lowest for "systemctl daemon-reload" command to still work without complaint (and therefore presumably apt-get install, update, upgrade, remove etc)

Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 01:41PM
Don,

There is no hard minimum memory for Debian. It depends what you run on the system. 64MB is fine for a lot systems that run a few dedicated aps.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 05:12PM
Hey bodhi ... yes, I understand, sorry I didn't mean to mislead anyone ! ... the link cited in the debian bugs site ( https://www.debian.org/releases/wheezy/amd64/ch03s04.html.en ) suggests that Debian may run in as little as 20MB in some configs ... the same page for Stretch indicates 60MB ... 128MB is their "recommended bare minimum" for both wheezy and stretch, although they'd generally prefer that we had more RAM.

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 06:05PM
Thanks Don, that makes sense!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: apt-get upgrade, systemd refuses to reload daemons
July 18, 2017 05:20AM
YW @bodhi :)

I think this is going to be a common problem for those upgrading Jessie -> Stretch on lower memory devices, such as some of the devices we hack/modify/improve here ...

A preferable solution would be a mod to the daemon-reload process so it doesn't have a hard limit of 16M, but I don't know how to hack that.

BR

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 25, 2017 04:13PM
DonCharisma Wrote:
-------------------------------------------------------
> I got this also during dist-upgrade Jessie -> Stre
> tch on GoFlexNet.

Yeah, it seemed to be a stretch issue. I didn't have this on jessie.

> A semi-permanent fix...is to add this to /etc/fstab :
> tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes= 4096 0 0

I haven't tried it. My reported solution didn't involve fstab, b/c I just tried to match what 'mount' reported for /run, only with more memory. The problem with fstab is that it reduces overall resources elsewhere permanently, instead of during upgrades only.

> so happy to know if there's "a better" solution ..

the nr_inodes discussed in that thread might be a good idea instead of the default. I don't know.

However, I have since removed systemd from my server, and I no longer need any workarounds.
edit: this may not be advisable for everyone. after removing systemd, I had a power outage, and the dockstar failed to boot. I have modded my dockstar so that it has an easily accessible serial console, and I was able to figure out the problem-- one of the mounts was FAT32, and the mandatory fsck ran out of memory! anyway, be careful with this solution unless you have a way to access the console, netconsole should be ok

Sorry I didn't reply sooner, I don't hit this forum very often.



Edited 1 time(s). Last edit at 07/25/2017 04:20PM by Nematocyst.
Re: apt-get upgrade, systemd refuses to reload daemons
July 18, 2018 11:20PM
I ran into this issue again after upgrading a Dockstar from jessie to stretch and decided to try to find a more permanent solution. I resolved it as follows:
  • Edit the /usr/share/initramfs-tools/init file and modify line with 'mount -t tmpfs -o "noexec,nosuid,size=10%,mode=0755" tmpfs /run' by changing 10% to 15%
  • Run the command: update-initramfs -u
  • Rerun command from boot directory to regenerate uInitrd file (i.e. mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-4.17.2-kirkwood-tld-1 -d initrd.img-4.17.2-kirkwood-tld-1 uInitrd)

On a 128MB system this should result in a /run directory with 16M Avail after system has booted which meets the minimum required by the systemd safety buffer.
Re: apt-get upgrade, systemd refuses to reload daemons
August 05, 2018 06:22AM
Thank you!

rayknight Wrote:
-------------------------------------------------------
> I ran into this issue again after upgrading a
> Dockstar from jessie to stretch and decided to try
> to find a more permanent solution. I resolved it
> as follows:
>
    >
  • Edit the /usr/share/initramfs-tools/init file
    > and modify line with 'mount -t tmpfs -o
    > "noexec,nosuid,size=10%,mode=0755" tmpfs /run' by
    > changing 10% to 15%
    >
  • Run the command: update-initramfs -u
    >
  • Rerun command from boot directory to
    > regenerate uInitrd file (i.e. mkimage -A arm -O
    > linux -T ramdisk -C gzip -a 0x00000000 -e
    > 0x00000000 -n initramfs-4.17.2-kirkwood-tld-1 -d
    > initrd.img-4.17.2-kirkwood-tld-1 uInitrd)
    >
>
> On a 128MB system this should result in a /run
> directory with 16M Avail after system has booted
> which meets the minimum required by the systemd
> safety buffer.
Re: apt-get upgrade, systemd refuses to reload daemons
September 21, 2018 01:26AM
Revisiting this issue yet again as it keeps popping up on my systems with only 128MB RAM. It appears that an update to initramfs-tools has a new configurable option in the /etc/initramfs-tools/initramfs.conf file. This option is RUNSIZE and is set to the default value of 10% which means less than 12MB will be available in /run on a 128MB system causing issues when reloading services. So to permanently resolve this problem take the following steps:

  1. Edit /etc/initramfs-tools/initramfs.conf file and change the RUNSIZE=10% line to RUNSIZE=20M
  2. Run the command: update-initramfs -u
  3. run the appropriate command to regenerate your uInitrd (i.e. "mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-4.18.4-kirkwood-tld-1 -d initrd.img-4.18.4-kirkwood-tld-1 uInitrd")
  4. reboot your system

This will set the size of the /run mount point to 20M which should be more than adequate to ensure the 16M needed for restarting services is available. The next two steps will ensure the updated size of the mountpoint is in your uInitrd image. After the reboot your /run mount point should show a size of 20M.

NOTE: This should only be necessary on systems with less than 256M of RAM as the 10% default would be adequate for systems with 256M or more.
Re: apt-get upgrade, systemd refuses to reload daemons
September 21, 2018 10:57PM
rayknight Wrote:
-------------------------------------------------------
> Revisiting this issue yet again as it keeps
> popping up on my systems with only 128MB RAM. It
> appears that an update to initramfs-tools has a
> new configurable option in the
> /etc/initramfs-tools/initramfs.conf file. This
> option is RUNSIZE and is set to the default value
> of 10% which means less than 12MB will be
> available in /run on a 128MB system causing issues
> when reloading services. So to permanently
> resolve this problem take the following steps:
>
Apparently the updated initramfs-tools package is not available in stretch its only available in buster and sid!
Re: apt-get upgrade, systemd refuses to reload daemons
September 22, 2018 03:06AM
> Apparently the updated initramfs-tools package is
> not available in stretch its only available in
> buster and sid!

That does not sound right! It is available on Debian stretch.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: apt-get upgrade, systemd refuses to reload daemons
September 22, 2018 03:15AM
bodhi Wrote:
-------------------------------------------------------
> > Apparently the updated initramfs-tools package
> is
> > not available in stretch its only available in
> > buster and sid!
>
> That does not sound right! It is available on
> Debian stretch.

bodhi the updated package version 0.132 is only available on buster and sid. Version 0.130 is the latest available for stretch and doesn't have the new RUNSIZE parameter which can be modified to increase the size of the /run mount point to prevent the "Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd" messages.

There are ways to install buster packages on stretch, but it would be nice if they would include this package in stretch updates.

Ray
Re: apt-get upgrade, systemd refuses to reload daemons
September 22, 2018 03:33AM
rayknight Wrote:
-------------------------------------------------------
> bodhi Wrote:
> -------------------------------------------------------
> > > Apparently the updated initramfs-tools
> package
> > is
> > > not available in stretch its only available
> in
> > > buster and sid!
> >
> > That does not sound right! It is available on
> > Debian stretch.
>
> bodhi the updated package version 0.132 is only
> available on buster and sid. Version 0.130 is the
> latest available for stretch and doesn't have the
> new RUNSIZE parameter which can be modified to
> increase the size of the /run mount point to
> prevent the "Failed to reload daemon: Refusing to
> reload, not enough space available on
> /run/systemd" messages.
>
> There are ways to install buster packages on
> stretch, but it would be nice if they would
> include this package in stretch updates.
>
> Ray

Ah. In that case, instead of using the initramfs-tool, couldn't you just use fstab tmpfs to deal with this?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: apt-get upgrade, systemd refuses to reload daemons
October 16, 2018 12:23PM
Never had I thought I would run into this issue with my Seagate GoFLEX Home and Dockstar, too. :(
Re: apt-get upgrade, systemd refuses to reload daemons
October 16, 2018 09:10PM
DonCharisma Wrote:
-------------------------------------------------------
> Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...
>
I added the above with size=32M to my Seagate Dockstar running debian stretch (see below) and so far it works. I will update my Seagate GoFLEX Home a few days later on if this addition doesn't give any problems.
[OLD] /etc/fstab:
tmpfs        /tmp        tmpfs        defaults                        0       0
[NEW] /etc/fstab:
tmpfs        /run        tmpfs        nosuid,noexec,size=32M,nr_inodes=4096   0       0
Re: apt-get upgrade, systemd refuses to reload daemons
August 15, 2019 10:29PM
Had same issue on pogoplug pro, this change corrected the issue: Many Thanks!

 [OLD] /etc/fstab:
tmpfs        /tmp        tmpfs        defaults                        0       0
[NEW] /etc/fstab:
tmpfs        /run        tmpfs        nosuid,noexec,size=32M,nr_inodes=4096   0       0


Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6



Edited 3 time(s). Last edit at 08/15/2019 10:35PM by echowarrior108.
Re: apt-get upgrade, systemd refuses to reload daemons
August 31, 2019 03:29AM
On the u-boot with uEnv.txt support you can simply add
initramfs.runsize=23M
To the boot parameter. Since I'm using the bodhi's u-boot this will work for me
custom_params=initramfs.runsize=23M
Re: apt-get upgrade, systemd refuses to reload daemons
June 03, 2020 07:39AM
I have a uEnv.txt with one line only:
custom_params=init=/bin/systemd

How do I modify it so that there are two commands:
custom_params=initramfs.runsize=23M [some delimiter here?] init=/bin/systemd

Tried all kinds of ways, but don't manage to increase the size of tmpfs this way.

Also tried the solution by @rayknight here.

Edit the /usr/share/initramfs-tools/init file and modify line with 'mount -t tmpfs -o "noexec,nosuid,size=10%,mode=0755" tmpfs /run' by changing 10% to 15%

My line says:
mount -t tmpfs -o "noexec,nosuid,size=${RUNSIZE:-10%},mode=0755" tmpfs /run

and changing -10% to -15% didn't work. How to modify it so it works?

So far, the only thing that increased tmps was to edit /etc/fstab:
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
LABEL=rootfs    /               ext3    noatime,errors=remount-ro 0 1
#tmpfs          /tmp            tmpfs   defaults          0       0
tmpfs        /run        tmpfs        nosuid,noexec,size=18M,nr_inodes=4096   0       0

To avoid getting trapped in the gnome-terminal at reboot, something like this usually works:
shutdown -r now && sleep 1 && exit


For some funny reason, ssh root@<ipaddress> generates this fault now, you have to specify the port 22 sometimes:
user@desktop:~$ ssh root@<ipaddress>
ssh: Could not resolve hostname <ipaddress>: Name or service not known
user@desktop:~$ ssh -p 22 root@<ipaddress>
root@<ipaddress>'s password: 
Linux DebianPlug 4.14.180-oxnas-tld-1 #1 SMP PREEMPT Dayname Month Day hh:mm:ss PDT year armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Dayname Month Day hh:mm:ss year from <ipaddress>
root@DebianPlug:~# 
Re: apt-get upgrade, systemd refuses to reload daemons
June 08, 2020 07:40PM
You just need to separate entries with a space like so:
custom_params=init=/bin/systemd initramfs.runsize=23M

follows the same syntax as the "set_bootargs" variable where it's called:
set_bootargs=setenv bootargs console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts $custom_params
Also: For when you get hit by this systemd 'feature', it seems remounting the tmpfs /run filesys is a work-around for a running system:

sudo mount -t tmpfs tmpfs /run -o remount,size=20M,rw,nosuid,noexec,relatime,mode=755

I copied the existing mount info, adding the 'remount' option and setting the size to 20M. This let me then do a successful daemon reload.
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: