Welcome! Log In Create A New Profile

Advanced

apt-get upgrade, systemd refuses to reload daemons

Posted by Nematocyst 
apt-get upgrade, systemd refuses to reload daemons
May 24, 2017 09:43PM
My dockstar server running debian stretch complains when doing updates. I got this a couple months ago, didn't get any useful hits on the problem, and avoided it since the server appeared to still do everything fine.

I updated just now and got several errors like this:
Warning: smbd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 10.0M are free, but a safety buffer of 16.0M is enforced.

Still, I get relatively few hits on the topic. I don't know what to do about it... it seems I need to have a bigger /run mount, but I'm not sure. Maybe these errors don't matter? They look horrible, and I even get them multiple times for the same daemons as it continues to try to restart them during apt-get upgrade.
Re: apt-get upgrade, systemd refuses to reload daemons
May 25, 2017 01:55PM
Do you have everything in a single file system? If so you are dangerously short of available disk space. You could try running:

apt-get autoremove
apt-get clean

The first command will auto remove packages that were installed to satisfy dependencies for other packages but are now no longer required. The second command deletes all .deb files from /var/cache/apt/archives and can free-up significant volume of disk space. You should also look into what is consuming the available disk space. The 'du' command can be helpful in determining what is using your disk space.



Edited 1 time(s). Last edit at 05/25/2017 01:57PM by rayknight.
Re: apt-get upgrade, systemd refuses to reload daemons
May 25, 2017 02:20PM
No, not short on disk space. I rebooted and all services appear to be working fine.

The bolded item is my rootfs. The red item is where the problem lies, I believe.

eric@deimos:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             59M     0   59M   0% /dev
tmpfs            13M  2.2M   10M  18% /run
/dev/sda5       585G  2.9G  553G   1% /
tmpfs            61M     0   61M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            61M     0   61M   0% /sys/fs/cgroup
ubi0:rootfs     232M  159M   73M  69% /srv/nand
/dev/sdb1       296G  252G   29G  90% /samba/server
/dev/sdb3       785G   21G  724G   3% /samba/backups
/dev/sdc1       932G  501G  432G  54% /samba/media
tmpfs            13M     0   13M   0% /run/user/1000



Edited 1 time(s). Last edit at 05/25/2017 02:22PM by Nematocyst.
Re: apt-get upgrade, systemd refuses to reload daemons
May 26, 2017 09:04AM
Yes, it is the /run mount. The reason rebooting works fine is that systemd isn't trying to reload anything, simply start the services. I can work around the problem by stopping each service and then starting it as such: [edit] 'restart' works too. the issue is 'systemctl daemon-reload' which isn't called during a reboot
[● root@deimos /home/eric] systemctl stop smbd
[● root@deimos /home/eric] systemctl start smbd
[● root@deimos /home/eric] systemctl stop nmbd
[● root@deimos /home/eric] systemctl start nmbd

That, however is a pain to do for each service after an upgrade. A better solution is by issuing the following before running upgrades:
[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=32M,nosuid,noexec,relatime,mode=755

and then restoring the original size afterwards with:
[● root@deimos /home/eric] mount -t tmpfs tmpfs /run -o remount,size=13M,nosuid,noexec,relatime,mode=755



Edited 1 time(s). Last edit at 05/26/2017 09:26AM by Nematocyst.
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 05:03AM
I got this also during dist-upgrade Jessie -> Stretch on GoFlexNet.

There's a thread about it on Debian bug forums - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862013

One of the responders ("Michael Biebl") indicates that minimum memory for Debian now is 256MB, which I think is incorrect, the link he points to says actually a minimum of 128MB (and I checked for ARM, as this was a AMD64 link) ... Anyhow ...

A semi-permanent fix, as suggested later on in the same thread above, is to add this to /etc/fstab :
... bla bla bla ...
tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes=4096 0 0
... bla bla bla ...

I tested below 18M and it didn't fix the problem, 18M seems to be the lowest for "systemctl daemon-reload" command to still work without complaint (and therefore presumably apt-get install, update, upgrade, remove etc)

Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 12:41PM
Don,

There is no hard minimum memory for Debian. It depends what you run on the system. 64MB is fine for a lot systems that run a few dedicated aps.

-bodhi
===========================
Wiki
latest Kirkwood kernel builds and rootfs
latest u-boot-kirkwood builds
latest Oxnas kernel builds and rootfs
latest u-boot-oxnas builds
latest MVEBU Armada kernel builds and rootfs
U-Boot & Kernel Booting process
bodhi's u-boot GitHub
bodhi's corner
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 04:12PM
Hey bodhi ... yes, I understand, sorry I didn't mean to mislead anyone ! ... the link cited in the debian bugs site ( https://www.debian.org/releases/wheezy/amd64/ch03s04.html.en ) suggests that Debian may run in as little as 20MB in some configs ... the same page for Stretch indicates 60MB ... 128MB is their "recommended bare minimum" for both wheezy and stretch, although they'd generally prefer that we had more RAM.

BR

Don

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 17, 2017 05:05PM
Re: apt-get upgrade, systemd refuses to reload daemons
July 18, 2017 04:20AM
YW @bodhi :)

I think this is going to be a common problem for those upgrading Jessie -> Stretch on lower memory devices, such as some of the devices we hack/modify/improve here ...

A preferable solution would be a mod to the daemon-reload process so it doesn't have a hard limit of 16M, but I don't know how to hack that.

BR

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com
Re: apt-get upgrade, systemd refuses to reload daemons
July 25, 2017 03:13PM
DonCharisma Wrote:
-------------------------------------------------------
> I got this also during dist-upgrade Jessie -> Stre
> tch on GoFlexNet.

Yeah, it seemed to be a stretch issue. I didn't have this on jessie.

> A semi-permanent fix...is to add this to /etc/fstab :
> tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes= 4096 0 0

I haven't tried it. My reported solution didn't involve fstab, b/c I just tried to match what 'mount' reported for /run, only with more memory. The problem with fstab is that it reduces overall resources elsewhere permanently, instead of during upgrades only.

> so happy to know if there's "a better" solution ..

the nr_inodes discussed in that thread might be a good idea instead of the default. I don't know.

However, I have since removed systemd from my server, and I no longer need any workarounds.
edit: this may not be advisable for everyone. after removing systemd, I had a power outage, and the dockstar failed to boot. I have modded my dockstar so that it has an easily accessible serial console, and I was able to figure out the problem-- one of the mounts was FAT32, and the mandatory fsck ran out of memory! anyway, be careful with this solution unless you have a way to access the console, netconsole should be ok

Sorry I didn't reply sooner, I don't hit this forum very often.



Edited 1 time(s). Last edit at 07/25/2017 03:20PM by Nematocyst.
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: