Ethernet lost after upgrade Debian Buster on Pogoplug Oxnas V3

I installed the last Debian Buster oxnas release, that run perfectly. But then ran update, about 50 files could be upgraded. I ran the upgrade and after that the eth0 if was not initialized .
I found a work around:
adding:

ifconfig eth0 up
/etc/init.d/networking restart

to /etc/rc.local

so the system runs OK now, but I do not expect that this is supposed to be.

cat /etc/network/interfaces shows:

auto lo eth0
iface lo inet loopback
iface eth0 inet dhcp

I assume the proper initialization sequence is not executed or something like that, but could find the cause of this problem.



Edited 2 time(s). Last edit at 06/19/2020 09:04AM by hacksome.

hacksome,

Which Pogoplug? they are different systems: E02/V4 Kirkwood, V3 OXNAS.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bodhi,

oops, it was not so obvious, although I mentioned oxnas in the message text.
modified the topic titles to be more specific.

hacksome,

> oops, it was not so obvious, although I mentioned
> oxnas in the message text.
> modified the topic titles to be more specific.

I see, my bad! did not read closely.

Not sure about that Ethernet link down. I did not see that on my box upgrading to Buster (with current rootfs). Are you running sysvinit or sysemd?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bodhi,

I am not really sure. I have to verify this. When I was trying to install OC10 on Buster I skipped the upgrade due to the loss of ethernet. But later on I discovered that systemd was not properly installed or functioning.

In between I created a new and fresh Buster image, I would expect I did take care to have a properly functioning systemd, but I am not certain about this.

Currently I am trying to get Debian running on a KD20 I had laying in a corner. I did not know that it had an OXNAS cpu as well.
It has double the RAM of the Pogoplug I have, so maybe OC10 will run on that system with some more features enabled.
The KD20 is operational without problems and I have a tty connected. Maybe you know if I can just change some environment variables using the setenv on the console and boot from a USB disk with a Debian Buster kernel, with the KD20.dtb appended to it? Is this possible with the stock uboot of the KD20?

I think I have figured out how to create a KD20 Buster USB drive, but manipulating the boot environment variables in a sensible way is still mainly unknown territory for me.

I will let you know if I had created a clean Pogoplug Buster image with a properly configured systemd before I did the upgrade.



Edited 1 time(s). Last edit at 06/23/2020 12:49PM by hacksome.

hacksome,

> with the KD20.dtb appended
> to it? Is this possible with the stock uboot of
> the KD20?

Yes.

> I think I have figured out how to create a KD20
> Buster USB drive, but manipulating the boot
> environment variables in a sensible way is still
> mainly unknown territory for me.

It is the same as the Pogo V3. You can clone this rootfs for the KD20. And then create new uImage with the DTB embedded.

Also, u-boot for the Pogo V3 should work on the KD20. However, I don't recall anybody post if they installed it on the KD20 NAND, but they ran an older version this u-boot from the HDD.

> I will let you know if I had created a clean
> Pogoplug Buster image with a properly configured
> systemd before I did the upgrade.

Either that. Or stay with sysvinit until you have a working system (so I can always help quickly by looking at the log).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bodhi,

I did a fresh install (Pogoplug), but things were very different than what I expected. But I also think I used a different chain of steps. e.g. update before reinstal of systemd.

This is my log of the steps I used:

Extracted package to drive/rootfs with:

tar -xjf Debian-4.14.180-oxnas-tld-1-rootfs-bodhi.tar.bz2

skipped step 4 because I have a Pogopro V3 (with PCI and WLAN)
skipped Note 1 (to make it my own rootfs)

So without additional steps after having extracted the package I booted on the Pogoplug.

After first login, output of

systemctl

showed:

Failed to list units: Launch helper exited with unknown return code 1

Output of

fw_printenv

:

bootcmd_exec=run load_uimage; if run load_initrd; then if run load_dtb; then bootm 0x60500000 0x60e00000 0x62c00000; else bootm 0x60500000 0x60e00000; fi; else if run load_dtb; then bootm 0x60500000 - 0x62c00000; else bootm 0x60500000; fi; fi
bootcmd=run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec; reset
bootcmd_uenv=run uenv_load; if test $uenv_loaded -eq 1; then run uenv_import; fi
bootdelay=10
bootdev=usb
usb_custom_params=zswap.enabled=1
device=0:1
devices=usb ide
disks=0 1 2 3
ethaddr=52:3b:20:9c:11:51
ethact=mii0
if_netconsole=ping $serverip
dtb_file=/boot/dts/ox820-pogoplug-pro.dtb
load_dtb=echo loading DTB $dtb_file ...; load $bootdev $device 0x62c00000 $dtb_file
load_initrd=echo loading uInitrd ...; load $bootdev $device 0x60e00000 /boot/uInitrd
load_uimage=echo loading uImage ...; load $bootdev $device 0x60500000 /boot/uImage
mainlineLinux=yes
mtdids=nand0=41000000.nand
mtdparts=mtdparts=41000000.nand:14m(boot),-(data)
preboot_nc=setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fisystemctl
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
scan_disk=echo running scan_disk ...; scan_done=0; setenv scan_usb "usb start";  setenv scan_ide "ide reset";  setenv scan_mmc "mmc rescan"; for dev in $devices; do if test $scan_done -eq 0; then echo Scan device $dev; run scan_$dev; for disknum in $disks; do if test $scan_done -eq 0; then echo device $dev $disknum:1; if load $dev $disknum:1 0x60500000 /boot/uImage 1; then scan_done=1; echo Found bootable drive on $dev $disknum; setenv device $disknum:1; setenv bootdev $dev; fi; fi; done; fi; done
set_bootargs=setenv bootargs console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts $custom_params
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
stderr=serial
stdin=serial
stdout=serial
uenv_init_devices=setenv init_usb "usb start";  setenv init_ide "ide reset";  setenv init_mmc "mmc rescan"; for devtype in $devices; do run init_$devtype; done;
uenv_load=run uenv_init_devices; setenv uenv_loaded 0; for devtype in $devices;  do for disknum in 0; do run uenv_read_disk; done; done;
uenv_read_disk=if test $devtype -eq mmc; then if $devtype part; then run uenv_read;  fi; else if $devtype part $disknum; then run uenv_read; fi;  fi
uenv_read=echo loading envs from $devtype $disknum ...; if load $devtype $disknum:1 0x60500000 /boot/uEnv.txt; then setenv uenv_loaded 1; fi
usb_ready_retry=15
serverip=xxx.xxx.xxx.xx
ipaddr=xxx.xxx.xxx.xx
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts init=/bin/systemd
uenv_import=echo importing envs ...; env import -t 0x60500000  $filesize

What I see from the last line is that the

init=/bin/systemd

was already added, but for some reason systemd is not running or initialised properly.

output of

systemd

command:

Trying to run as user instance, but the system has not been booted with systemd.

To prevent problems with version mismatches installing systemd I first ran:

apt update

That showed:

59 packages can be upgraded.

and systemd was one of them:

systemd/stable 241-7~deb10u4 armel [upgradable from: 232-25+deb9u1]

Then I installed systemd:

apt-get install systemd

output:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libcryptsetup4
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libargon2-1 libcryptsetup12 libgnutls30 libidn2-0 libjson-c3 libsystemd0 libunistring2
Suggested packages:
  gnutls-bin systemd-container policykit-1
Recommended packages:
  libpam-systemd
The following NEW packages will be installed:
  libargon2-1 libcryptsetup12 libjson-c3 libunistring2
The following packages will be upgraded:
  libgnutls30 libidn2-0 libsystemd0 systemd
4 upgraded, 4 newly installed, 0 to remove and 55 not upgraded.
Need to get 5235 kB of archives.
After this operation, 6041 kB of additional disk space will be used.

Some warnings are shown when systemd is configured:

Setting up systemd (241-7~deb10u4) ...
Installing new version of config file /etc/systemd/journald.conf ...
Installing new version of config file /etc/systemd/logind.conf ...
Installing new version of config file /etc/systemd/resolved.conf ...
Installing new version of config file /etc/systemd/system.conf ...
Installing new version of config file /etc/systemd/timesyncd.conf ...
Removing user `systemd-bus-proxy' ...
Warning: group `systemd-bus-proxy' has no more members.
Done.
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.hostname1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.locale1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.login1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.network1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.resolve1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.systemd1.conf ...
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.timedate1.conf ...

because systemd stil was not running I rebooted the system.

After reboot, systemd still not running. I did a reinistall:
output:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libcryptsetup4
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 55 not upgraded.
Need to get 0 B/3235 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 12264 files and directories currently installed.)
Preparing to unpack .../systemd_241-7~deb10u4_armel.deb ...
Unpacking systemd (241-7~deb10u4) over (241-7~deb10u4) ...
Setting up systemd (241-7~deb10u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for dbus (1.12.16-1) ...

reboot again

still:
output of

systemd

command:

Trying to run as user instance, but the system has not been booted with systemd.

ls -l /bin/systemd
lrwxrwxrwx 1 root root 20 Apr 27  2020 /bin/systemd -> /lib/systemd/systemd
root@debian:~# ls -l /lib/systemd/systemd
-rwxr-xr-x 1 root root 1239124 Apr 27  2020 /lib/systemd/systemd

This is not what I expected and different from earlier events. The only difference I know of is that I ran

apt update

before I installed systemd again.

output of

systemctl

System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

So now I am at a loss.............

Bodhi,

I did a new install without running

apt update

before I ran

ap install systemd

That produced similar meassages about packages to be installed, however it could not execute properly because of :

FATAL -> Failed to fork.
debconf: apt-extracttemplates failed: No such file or directory

and:

W: Could not open file /var/log/apt/eipp.log.xz - open (12: Cannot allocate memory)
W: Could not open file '/var/log/apt/eipp.log.xz' - EIPP::OrderInstall (12: Cannot allocate memory)

so I redid all, but first enlarged memory shortages by increasing /run, ( by added [none /run tmpfs defaults,size=32M 0 0]) to fstab.

Then I installed systemd and also systemd-sysv

because I had also noticed in previous attempts hat the symlink from sbin/
init was missing see: https://forum.doozan.com/read.php?2,60589,64714#msg-64714
Fortunately had had made a note of this problem....

But unfortunately it did not solve it completely.

That next effort I had no memory problem but still the

FATAL -> Failed to fork.
debconf: apt-extracttemplates failed: No such file or directory

the command

systemctl

now did show a list of processes, but the command

systemd

now showed

Trying to run as user instance, but $XDG_RUNTIME_DIR is not set.

I did a search for this particular error but could not find a proper match with what I have on my system.
From what I have read about this error I would assume it may have something to do with the warning messages that were shown when installing systemd and systemd-sysv about:

Removing user `systemd-bus-proxy' ...
Warning: group `systemd-bus-proxy' has no more members.

But at the moment this is beyond what know about this kind of details.

After digging a little further I can see that a lot of systemd processes are running. From what I seem to understand about this mechanism is that the $XDG_RUNTIME_DIR variable is not set in one of the unit files.

systemctl list-unit-files

shows a list of 239 unit files....... and I have no idea which one has anything to do with this variable....



Edited 2 time(s). Last edit at 06/24/2020 09:22AM by hacksome.

hacksome,

After you got it booted with sysvinit, to switch to systemd in a clean way, you could

- remove sysvinit
- install systemd

Later if you want to go back to sysvinit, then install it (there are 2 packages, iirc).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bodhi,

I think this problem has been solved for now.
I have created a fresh rootfs with this Buster version: https://forum.doozan.com/read.php?2,16044
for the pogoplug. But this time after the update there was no loss of Ethernet anymore.

I think the problems I encountered were caused because of inconsistencies due to different times from the system I run the update from, the time value when using the upgrade commands and maybe even the time zone changed somwhere in the process.
Out of the box the system time is not synced with anything. I did change my time-zone settings somewhere in the process using:

dpkg-reconfigure tzdata

but that does not enable the use of an NTP server to sync system time after a reboot. I found out that a more reliable way is to use systemd-timesyncd.

apt purge ntp

and then:

systemctl start systemd-timesyncd

From then the system time after reboot is synchronized with an NTP server.
I hope that using a properly configured time reference will prevent further inconsistencies in the update process.


One thing still puzzles me: 4 packages from the almost 60 are not updated for some reason:

liblzma5/stable 5.2.4-1 armel [upgradable from: 5.2.2-1.2+b1]
nfs-common/stable 1:1.3.4-2.5 armel [upgradable from: 1:1.3.4-2.1]
nfs-kernel-server/stable 1:1.3.4-2.5 armel [upgradable from: 1:1.3.4-2.1]
rpcbind/stable 1.2.5-0.3+deb10u1 armel [upgradable from: 0.2.3-0.6]

I already encountered these 4 packages were not updated before, but then the time settings were not aligned yet and the warning message thrown was slightly different:

E: Release file for http://security.debian.org/dists/buster/updates/InRelease is not valid yet (invalid for another 531d 20h 40min 2s). Updates for this repository will not be applied.

After that I manually forced the update, but that messed up the system again.

So I recreated a fresh install again, fixed the time synchronization and just updated without bothering about the 4 packages.
From then it seems that the system is stable with systemd configured properly.

Hacksome

hacksome,

> dpkg-reconfigure tzdata

This is part of the setup steps that you'd need to do after getting the system running initially. I use PST timezone in the rootfs because it is convenient. And also the locale, too, which is default to US.

> Out of the box the system time is not synced with
> anything.

Note that I'm assuming you runsysvinit here.

Yes, it does time sync using NTP out of the box. It takes a while after booting, though. These boxes don't have RTC, so it is not recommended to do upgrade immediately after reboot without checking the system clock.

I used to use ntpupdate to make it sync faster, but it was deprecated. ntp daemon supposed to take care of that, but perhaps it does not work properly in this case, and needs more time to do the time sync.


And you can modify the ntp pools to point to generic Debian pools, or any time server that closer to you. I use these in the rootfs:

Quote

#montpelier.caltech.edu
#tick.ucla.edu
#bigben.cac.washington.edu
#time.nist.gov

cat /etc/ntp.conf | grep server

# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
#server 0.debian.pool.ntp.org iburst
#server 1.debian.pool.ntp.org iburst
#server 2.debian.pool.ntp.org iburst
#server 3.debian.pool.ntp.org iburst
server 192.12.19.20
server 164.67.62.194
server 140.142.16.34
server 206.246.122.250
# Note that "restrict" applies to both servers and clients, so a configuration
# up blocking replies from your own upstream servers.
# Local users may interrogate the ntp server more closely.

For some services, systemd does not always work right away after you switch from sysvinit. Perhaps this is one of those cases.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)