NSA320 fails uboot after upgrade debian buster to bullseye

Hi Bodhi,

I first ugraded my kernel to the latest available and rebooted, that worked fine.
I was able to succesfully upgrade my NSA320 from Buster to Bullseye. But after the reboot it is not coming up. I have a serial-console and this is what is happening now:

NSA320> boot
Initializing devices...

Reset IDE: Bus 0: OK Bus 1: OK
  Device 0: Model: ST2000DL003-9VT166 Firm: CC45 Ser#: 5YD6LRQ4
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
  Device 1: Model: ST4000DM005-2DP166 Firm: 0001 Ser#: ZDH0CSKF
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
Unknown command 'mmc' - try 'help'
Loading envs from usb 0...
** Bad device usb 0 **
Loading envs from usb 1...
** Bad device usb 1 **
Loading envs from usb 2...
** Bad device usb 2 **
Loading envs from usb 3...
** Bad device usb 3 **
Loading envs from ide 0...
260 bytes read in 88 ms (2 KiB/s)
... envs loaded
importing envs ...
running scan_disk ...
Scan device usb
device usb 0:1
** Bad device usb 0 **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: Bus 0: OK Bus 1: OK
  Device 0: Model: ST2000DL003-9VT166 Firm: CC45 Ser#: 5YD6LRQ4
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
  Device 1: Model: ST4000DM005-2DP166 Firm: 0001 Ser#: ZDH0CSKF
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
device ide 0:1
ext4fs_devread read outside partition 18446744073262735360
device ide 1:1
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
Scan device mmc
Unknown command 'mmc' - try 'help'
device mmc 0:1
** Bad device mmc 0 **
device mmc 1:1
** Bad device mmc 1 **
device mmc 2:1
** Bad device mmc 2 **
device mmc 3:1
** Bad device mmc 3 **
loading uImage ...
** Bad device usb 0 **
loading uInitrd ...
** Bad device usb 0 **
loading DTB /boot/dts/kirkwood-nsa320.dtb ...
** Bad device usb 0 **
Wrong Image Format for bootm command
Error occured, error code = 112
ERROR: can't get kernel image!

I have been googling for a cause, but the only thing I could find is about the mkfs.conf file with these things (mkfs.ext4 -O ^64bit /dev/sd... and such, there also was one other option), but I have dealt with that in the past. My filesystems were correctly formatted and working before the upgrade to Bullseye.
I did see in the upgrade output the following:

Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.19.2-kirkwood-tld-1
I: The initramfs will attempt to resume from /dev/sdb1
I: (UUID=e410ee0f-8aa3-4af5-af2e-0cbab3fc195e)
I: Set the RESUME variable to override this.

So i did part 4.a again:

cd /boot
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.19.2-kirkwood-tld-1 -d vmlinuz-5.19.2-kirkwood-tld-1 uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.19.2-kirkwood-tld-1 -d initrd.img-5.19.2-kirkwood-tld-1 uInitrd

I did backup uImage and uInitrd, which were the working ones with kernel-5.19.2, to /boot/11.5,

I have seen this somewhere on your site and tried that (using NSA320 values):

NSA320> load ide 0:1 0x800000 /boot/uImage
ext4fs_devread read outside partition 18446744073262735360

NSA320> ext4ls ide 0:1 /boot/11.5
<DIR>       4096 .
<DIR>       4096 ..
         5733288 uImage
        12719243 uInitrd
        13192459 initrd.img-5.19.2-kirkwood-tld-1

NSA320> load ide 0:1 0x800000 /boot/11.5/uImage
ext4fs_devread read outside partition 18446744073262784512


NSA320> load ide 0:1 0x800000 /boot/bck.20221030/uImage
4963680 bytes read in 861 ms (5.5 MiB/s)

This last attempt was the uImage of pre-bullseye and pre-kernel-update, so kernel-5-2-9 (I think).

It appears it does not like uImage files created after the 'apt full-upgrade', but it also does not like the one created for kernel-5.19.2 anymore,

How can I get out of this situation the best way? Do you get what is going on with these images?

Thanks and kind regards

bandicoot,

As always, next time please post the entire serial boot log, so I don't have to guess.

I assume you are running 2017.07-kirkwood-tld-1 u-boot?

Quote

device ide 0:1
ext4fs_devread read outside partition 18446744073262735360

Usually this error indicates the file system is corrupted.

Quote

loading uImage ...
** Bad device usb 0 **
loading uInitrd ...
** Bad device usb 0 **
loading DTB /boot/dts/kirkwood-nsa320.dtb ...
** Bad device usb 0 **

These errors just meant that the files could not be read. The devices themselves are OK.

>
> How can I get out of this situation the best way?
> Do you get what is going on with these images?
>

The images might be on the bad parts of the filesystem. Or the inodes are corrupted.

You can create a USB rootfs using Debian-5.13.6-kirkwood-tld-1-rootfs-bodhi.tar.bz2, and boot with that.

Insert this USB rootfs, and power up, and,

setenv devices usb
setenv bootdev usb
printenv
boot

The kernel will be one on USB, but when the kernel running it might mount the HDD rootfs. If that's the case, then you can choose to do either:

- change the HDD rootfs partition label to hdd_rootfs (to tell the kernel not to mount it, so it will mount USB rootfs). Reboot.
OR
- Shutdown and remove the 1863.0 GB HDD (I'm assuming this is where the rootfs is). Power up, let it boot to USB rootfs. And then insert the HDD.


When running the USB rootfs, if the 1863.0 GB HDD is mounted automatically, then unmount it. And then run e2fsck to check the file system error.

After that you can copy the uImage and uInitrd from the USB rootfs to the HDD rootfs /boot (or restore the backup as "I did backup uImage and uInitrd, which were the working ones with kernel-5.19.2, to /boot/11.5,")

And then reboot without the USB (if you have changed the HDD rootfs partition label then change it back to rootfs before reboot).

===============

Once you can boot the HDD rootfs, run

update-initramfs -u

And then recreate the uInitrd in /boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Bodhi,

Thanks a lot for the quick response.

Sorry, this is the full log from the reboot after the Bullseye udate:

U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:46:11 -0700)
ZyXEL NSA320 2-Bay Power Media Server

SoC:   Kirkwood 88F6281_A1
DRAM:  512 MiB
WARNING: Caches not enabled
NAND:  128 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
MV88E1318 PHY initialized on egiga0
Using egiga0 device

ARP Retry count exceeded; starting again
ping failed; host 192.168.3.121 is not alive
Hit any key to stop autoboot:  0
Initializing devices...
starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

Reset IDE: Bus 0: OK Bus 1: OK
  Device 0: Model: ST2000DL003-9VT166 Firm: CC45 Ser#: 5YD6LRQ4
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
  Device 1: Model: ST4000DM005-2DP166 Firm: 0001 Ser#: ZDH0CSKF
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
Unknown command 'mmc' - try 'help'
Loading envs from usb 0...
** Bad device usb 0 **
Loading envs from usb 1...
** Bad device usb 1 **
Loading envs from usb 2...
** Bad device usb 2 **
Loading envs from usb 3...
** Bad device usb 3 **
Loading envs from ide 0...
260 bytes read in 88 ms (2 KiB/s)
... envs loaded
importing envs ...
running scan_disk ...
Scan device usb
device usb 0:1
** Bad device usb 0 **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: Bus 0: OK Bus 1: OK
  Device 0: Model: ST2000DL003-9VT166 Firm: CC45 Ser#: 5YD6LRQ4
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
  Device 1: Model: ST4000DM005-2DP166 Firm: 0001 Ser#: ZDH0CSKF
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
device ide 0:1
ext4fs_devread read outside partition 18446744073262735360
device ide 1:1
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
Scan device mmc
Unknown command 'mmc' - try 'help'
device mmc 0:1
** Bad device mmc 0 **
device mmc 1:1
** Bad device mmc 1 **
device mmc 2:1
** Bad device mmc 2 **
device mmc 3:1
** Bad device mmc 3 **
loading uImage ...
** Bad device usb 0 **
loading uInitrd ...
** Bad device usb 0 **
loading DTB /boot/dts/kirkwood-nsa320.dtb ...
** Bad device usb 0 **
Wrong Image Format for bootm command
Error occured, error code = 112
ERROR: can't get kernel image!

So, you are right with the uboot version: U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:46:11 -0700)

My current printenv:

NSA320> printenv
arcNumber=3956
bootargs=console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 orion_nand:0x100000(uboot),0x80000(stock_uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2) init=/bin/systemd
bootcmd=run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec
bootcmd_exec=run load_uimage; if run load_initrd; then if run load_dtb; then bootm $load_uimage_addr $load_initrd_addr $load_dtb_addr; else bootm $load_uimage_addr $load_initrd_addr; fi; else if run load_dtb; then bootm $load_uimage_addr - $load_dtb_addr; else bootm $load_uimage_addr; fi; fi
bootcmd_uenv=run uenv_load; if test $uenv_loaded -eq 1; then run uenv_import; fi; sleep 20
bootdelay=10
bootdev=usb
custom_params=init=/bin/systemd
device=0:1
device_type=ide
devices=usb ide mmc
devicetype=ide
disk_number=0
disks=0 1 2 3
dtb_file=/boot/dts/kirkwood-nsa320.dtb
ethact=egiga0
ethaddr=B0:B2:DC:F0:FF:12
fileaddr=800000
filesize=4bbd60
if_netconsole=ping $serverip
init_ide=ide reset
init_mmc=mmc rescan
init_usb=usb start
ipaddr=192.168.3.90
led_error=orange blinking
led_exit=green off
led_init=green blinking
load_dtb=echo loading DTB $dtb_file ...; load $bootdev $device $load_dtb_addr $dtb_file
load_dtb_addr=0x2c00000
load_initrd=echo loading uInitrd ...; load $bootdev $device $load_initrd_addr /boot/uInitrd
load_initrd_addr=0x1100000
load_uimage=echo loading uImage ...; load $bootdev $device $load_uimage_addr /boot/uImage
load_uimage_addr=0x800000
mainlineLinux=yes
mtdids=nand0=orion_nand
mtdparts=orion_nand:0x100000(uboot),0x80000(stock_uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)
partition=nand0,2
preboot=run if_netconsole start_netconsole
preboot_nc=run if_netconsole start_netconsole
scan_disk=echo running scan_disk ...; scan_done=0; setenv scan_usb "usb start";  setenv scan_ide "ide reset";  setenv scan_mmc "mmc rescan"; for dev in $devices; do if test $scan_done -eq 0; then echo Scan device $dev; run scan_$dev; for disknum in $disks; do if test $scan_done -eq 0; then echo device $dev $disknum:1; if load $dev $disknum:1 $load_uimage_addr /boot/uImage 1; then scan_done=1; echo Found bootable drive on $dev $disknum; setenv device $disknum:1; setenv bootdev $dev; fi; fi; done; fi; done
scan_ide=ide reset
scan_mmc=mmc rescan
scan_usb=usb start
serverip=192.168.3.121
set_bootargs=setenv bootargs console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts $custom_params
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
stderr=serial
stdin=serial
stdout=serial
uenv_addr=0x810000
uenv_import=echo importing envs ...; env import -t $uenv_addr $filesize
uenv_init_devices=echo Initializing devices...; setenv init_usb "usb start";  setenv init_ide "ide reset";  setenv init_mmc "mmc rescan"; for devtype in $devices; do run init_$devtype; done
uenv_load=run uenv_init_devices; setenv uenv_loaded 0; for devtype in $devices; do for disknum in $disks; do if test $uenv_loaded -eq 0; then setenv device_type $devtype; setenv disk_number $disknum; run uenv_read; fi; done; done;
uenv_loaded=1
uenv_read=echo Loading envs from $device_type $disk_number...; if load $device_type  $disk_number:1 $uenv_addr /boot/uEnv.txt; then setenv uenv_loaded 1; echo ... envs loaded; fi
usb_ready_retry=15

I also have a usb-docking-station and a Debian USB boot stick for my laptop, i will do a fsck that way, I think that'll be easier. I will report back on disk-errors.

I had also added to /boot/uEnv.txt because uInitrd was larger then 11MB.

Regards,
Bandicoot

Bodhi,

I got it to run again by doing a fsck (appeared to be nothing wrong?!) and copying uImage and uIntrd from your rootfs to /boot on my disk, thanks.

I just ran:

root@NSA320:/boot# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.19.2-kirkwood-tld-1
I: The initramfs will attempt to resume from /dev/sdb1
I: (UUID=e410ee0f-8aa3-4af5-af2e-0cbab3fc195e)
I: Set the RESUME variable to override this.
[ 1633.218646][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.225063][ T6419] blk_update_request: I/O error, dev mtdblock7, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1633.238066][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.244960][ T6419] blk_update_request: I/O error, dev mtdblock7, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1633.256825][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.263673][ T6419] blk_update_request: I/O error, dev mtdblock7, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1633.284538][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.290960][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.297270][ T6419] blk_update_request: I/O error, dev mtdblock5, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1633.310310][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.316625][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.323786][ T6419] blk_update_request: I/O error, dev mtdblock5, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1633.335639][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.342837][ T6419] ecc_sw_hamming_correct: uncorrectable ECC error
[ 1633.349156][ T6419] blk_update_request: I/O error, dev mtdblock5, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

I do not understand these errors. Also do not get the "resume from /dev/sdb1" because root and boot are on sda1.

EDIT:
After:

apt remove  linux-image-5.2.9-kirkwood-tld-1

And then again:

root@NSA320:/boot# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.19.2-kirkwood-tld-1
I: The initramfs will attempt to resume from /dev/sdb1
I: (UUID=e410ee0f-8aa3-4af5-af2e-0cbab3fc195e)
I: Set the RESUME variable to override this.

So, no errors anymore.

I also ran:

root@NSA320:/boot# debsums -cs
debsums: missing file /boot/config-5.2.9-kirkwood-tld-1 (from linux-image-5.2.9-kirkwood-tld-1 package)
debsums: missing file /boot/vmlinuz-5.2.9-kirkwood-tld-1 (from linux-image-5.2.9-kirkwood-tld-1 package)
debsums: missing file /boot/System.map-5.2.9-kirkwood-tld-1 (from linux-image-5.2.9-kirkwood-tld-1 package)

That way I found out that I had to remove the package. Also with this result of debsums I think the bullseye upgrade did not encounter any corruptions.


Regards,
Bandicoot



Edited 2 time(s). Last edit at 10/30/2022 04:30PM by bandicoot.

bandicoot,

> I got it to run again by doing a fsck (appeared to
> be nothing wrong?!) and copying uImage and uIntrd
> from your rootfs to /boot on my disk, thanks.

Cool! so now you can reinstall the kernel with linux-image-5.2.9-kirkwood-tld-1 using dpkg -i. I would do that to make sure it's OK.

I think after that, whatever file system errors will go away. Not sure why fsck did not show errors. It could be that you forgot to run sync commands?

Note: So remember to do a few sync commands after installing kernel, or after upgrade to bullseye.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> bandicoot,
>
> > I got it to run again by doing a fsck (appeared
> to
> > be nothing wrong?!) and copying uImage and
> uIntrd
> > from your rootfs to /boot on my disk, thanks.
>
> Cool! so now you can reinstall the kernel with
> linux-image-5.2.9-kirkwood-tld-1 using dpkg -i. I
> would do that to make sure it's OK.

I now have 5.19.2 installed. And I also wanted to loose 5.2.9. But at this moment I am running 5.13.6 from your rootfs:

root@NSA320:/boot# ls -l
total 45476
drwxr-xr-x  10 root root     4096 Oct 30 22:25 .
drwxr-xr-x  23 root root     4096 Dec 15  2021 ..
drwxr-xr-x   2 root root     4096 Oct 30 14:43 11.5
-rw-r--r--   1 root root  5050489 Aug 18 01:53 System.map-5.19.2-kirkwood-tld-1
drwxr-xr-x   2 root root     4096 Oct 30 00:45 bck
drwxr-xr-x   3 root root     4096 Oct 30 00:41 bck.20221030
-rw-r--r--   1 root root   191187 Aug 18 01:53 config-5.19.2-kirkwood-tld-1
drwxr-xr-x.  3 root root     4096 Oct 30 20:52 corrupt
drwxr-xr-x   2 root root     4096 Oct 30 14:39 downloaded
drwxr-xr-x.  2 root root     4096 Aug  1  2021 dts
-rw-r--r--   1 root root     3320 Feb 21  2022 fw_printenv.out
-rw-r--r--   1 root root 13192398 Oct 30 22:25 initrd.img-5.19.2-kirkwood-tld-1
drwxr-xr-x   2 root root     4096 Oct 30 21:34 installs
-rw-r--r--   1 root root   160832 Aug 20 22:40 linux-5.19.2-kirkwood-tld-1.patch
-rw-r--r--   1 root root  1331200 Aug 18 08:11 linux-dtb-5.19.2-kirkwood-tld-1.tar
drwxr-xr-x   2 root root     4096 Oct 30 21:33 old
-rw-r--r--   1 root root      260 Oct 30 00:56 uEnv.txt
-rw-r--r--.  1 root root  5435888 Sep 24  2021 uImage
-rw-r--r--.  1 root root  9671091 Sep 24  2021 uInitrd
-rwxr-xr-x   1 root root  5733224 Aug 18 01:53 vmlinuz-5.19.2-kirkwood-tld-1
-rwxr-xr-x   1 root root  5733224 Aug 18 04:39 zImage-5.19.2-kirkwood-tld-1

root@NSA320:/boot# uname -a
Linux NSA320 5.13.6-kirkwood-tld-1 #1.0 PREEMPT Sat Jul 31 22:10:39 PDT 2021 armv5tel GNU/Linux

I should be able to run 5.19.2, wouldn't you think? I am not sure what I need to do now to get that going.

>
> I think after that, whatever file system errors
> will go away. Not sure why fsck did not show
> errors. It could be that you forgot to run sync
> commands?
>

I did make sure to run a couple of sync's before rebooting, so I too have no idea what went wrong.

> Note: So remember to do a few sync commands
> after installing kernel, or after upgrade to
> bullseye.

Hopefully it does not happen again.

Regards,
Bandicoot

Bandicoot,

Sure, you can go ahead and install 5.19.2 again. dpkg will just overwrite the current 5.19.2. Or if you want to, you can remove it before installing.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)