NSA 320 update to kernel 5.6.5 not booting

Ive tried to update to the latest kernel. and the initram fails.
I fsck my disk, i restored my boot backup and tried a second time with the same result.
My previous kernel is the 5.5.1

U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:46:11 -0700)
ZyXEL NSA320 2-Bay Power Media Server

SoC:   Kirkwood 88F6281_A1
DRAM:  512 MiB
WARNING: Caches not enabled
NAND:  128 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
MV88E1318 PHY initialized on egiga0
Hit any key to stop autoboot:  0 
starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 
Use USB retry period from the environment: 15 second(s)
1 Storage Device(s) found

Reset IDE: Bus 0: OK Bus 1: OK 
  Device 0: Model: WDC WD40EFRX-68WT0N0 Firm: 82.00A82 Ser#:  WD-WCC4E6XEX0TT
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
  Device 1: Model: WDC WD40EFRX-68N32N0 Firm: 82.00A82 Ser#:  WD-WCC7K1JJ10NV
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 3815447.8 MB = 3726.0 GB (7814037168 x 512)
Unknown command 'mmc' - try 'help'

Partition Map for USB device 0  --   Partition Type: DOS

Part    Start Sector    Num Sectors     UUID            Type
  1     2048            61339648        000b6a86-01     83

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0
loading envs from usb 0 ...
** File not found /boot/uEnv.txt **

Partition Map for IDE device 0  --   Partition Type: EFI

Part    Start LBA       End LBA         Name
        Attributes
        Type GUID
        Partition GUID
  1     0x00000800      0x1d1c0b7ff     ""
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
        type:   linux
        guid:   c379d15d-2159-4dcb-a46b-351ae6ae1d44


Partition Map for IDE device 1  --   Partition Type: EFI

Part    Start LBA       End LBA         Name
        Attributes
        Type GUID
        Partition GUID
  1     0x00000800      0x1d1c0b7ff     ""
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
        type:   linux
        guid:   c7c07782-7ac8-48c3-8274-2ca7baa16f4a
Unknown command 'mmc' - try 'help'
running scan_disk ...
Scan device usb
device usb 0:1
1 bytes read in 692 ms (0 Bytes/s)
Found bootable drive on usb 0
loading uImage ...
5409256 bytes read in 1003 ms (5.1 MiB/s)
loading uInitrd ...
12944766 bytes read in 1149 ms (10.7 MiB/s)
loading DTB /boot/dts/kirkwood-nsa320.dtb ...
13153 bytes read in 2675 ms (3.9 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-5.6.5-kirkwood-tld-1
   Created:      2020-04-28  15:00:43 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    5409192 Bytes = 5.2 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-5.6.5-kirkwood-tld-1
   Created:      2020-04-28  15:01:11 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    12944702 Bytes = 12.3 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... Bad Data CRC
Error occured, error code = 12
Ramdisk image is corrupt or invalid

Edited 1 time(s). Last edit at 04/28/2020 10:14AM by Pickle.

Pickle,

1. What were the USB rootfs formatted with? EXT3 or EXT4?

2. Have you tried to use an older uInitrd on /boot? hopefully you have back up the older kernel uInitrd. Bring the USB to another Linux box and copy the older version,

cp -a uInird.bak uInitrd

You can boot with this temporarily and then recreate the new uInitrd in Debian.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

thought it was ext4, but my fstab shows it is ext3.

I dont have a problem getting the system running again.
I have a complete copy of /boot before the upgrade, so i went back to that /boot and like i mentioned before the system boots again and runs like normal.

running the dpkg command then

mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.6.5-kirkwood-tld-1 -d initrd.img-5.6.5-kirkwood-tld-1 uInitrd

I did not try to boot the new kernel with the old uInitrd, if i do then i run the mkimage command above?

I only tried restoring the backup /boot and performing the upgrade as described in your guide. And the two times i did it did not create the uInitrd with a valid CRC.

Pickle,

> thought it was ext4, but my fstab shows it is
> ext3.

If you use Ext4 for rootfs, you need to be careful and "finalize" the delay writes. It is best to turn off delay writes for rootfs so when you upgrade kernels, the files are committed right away. Ext4 is troublesome in this way.

> I dont have a problem getting the system running
> again.
> I have a complete copy of /boot before the
> upgrade, so i went back to that /boot and like i
> mentioned before the system boots again and runs
> like normal.

Good. Then just boot back to the old kernel this way. And recreate uInitrd for 5.6.5 like you did above.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

i have ext3 for the usb driver so i wont have that problem then.

sorry but i dont understand what you want me to try. I am already back to running with the old kernel and the old initrd.
if i run through the kernel upgrade process I end up with the bad uInitrd and wrong CRC.

Pickle,

1. Bring this USB drive to another Linux box and run e2fsck to check for errors.

2. Boot with old kernel in /boot. Remove the 5.6.5 kernels:

dpkg --purge linux-image-5.6.5-kirkwood-tld-1

and reinstall.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi,

ran the esfsck and no problem
i purged all kernels but 5.5.1
did the reinstall and ended up with the same CRC error

Edit:
So just copying the old 5.5.1 uInitrd gets the 5.6.5 kernel to boot.
I tried running the mkimage again and back to the error.


Here are the commands after the kernel cleanup:

root@NSA320:/boot# tar -xvf linux-5.6.5-kirkwood-tld-1-bodhi.tar.bz2 
linux-image-5.6.5-kirkwood-tld-1_1.0_armel.deb
linux-headers-5.6.5-kirkwood-tld-1_1.0_armel.deb
config-5.6.5-kikrwood-tld-1
zImage-5.6.5-kikrwood-tld-1
linux-dtb-5.6.5-kirkwood-tld-1.tar
linux-5.6.5-kirkwood-tld-1.patch
root@NSA320:/boot# tar -xf linux-dtb-5.6.5-kirkwood-tld-1.tar 
root@NSA320:/boot# dpkg -i linux-image-5.6.5-kirkwood-tld-1_1.0_armel.deb 
Selecting previously unselected package linux-image-5.6.5-kirkwood-tld-1.
(Reading database ... 40447 files and directories currently installed.)
Preparing to unpack linux-image-5.6.5-kirkwood-tld-1_1.0_armel.deb ...
Unpacking linux-image-5.6.5-kirkwood-tld-1 (1.0) ...
Setting up linux-image-5.6.5-kirkwood-tld-1 (1.0) ...
update-initramfs: Generating /boot/initrd.img-5.6.5-kirkwood-tld-1
root@NSA320:/boot# sync
root@NSA320:/boot# cd /boot
root@NSA320:/boot# mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.6.5-kirkwood-tld-1 -d vmlinuz-5.6.5-kirkwood-tld-1 uImage
Image Name:   Linux-5.6.5-kirkwood-tld-1
Created:      Tue Apr 28 22:03:25 2020
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    5409192 Bytes = 5282.41 KiB = 5.16 MiB
Load Address: 00008000
Entry Point:  00008000
root@NSA320:/boot# mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.6.5-kirkwood-tld-1 -d initrd.img-5.6.5-kirkwood-tld-1 uInitrd
Image Name:   initramfs-5.6.5-kirkwood-tld-1
Created:      Tue Apr 28 22:03:42 2020
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    12944762 Bytes = 12641.37 KiB = 12.35 MiB
Load Address: 00000000
Entry Point:  00000000

Edited 1 time(s). Last edit at 04/28/2020 09:27PM by Pickle.

Pickle,

Looks like you've done everything correctly. So not sure why you have the CRC error. The only thing that seems odd about this is

root@NSA320:/boot# mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.6.5-kirkwood-tld-1 -d initrd.img-5.6.5-kirkwood-tld-1 uInitrd
Image Name:   initramfs-5.6.5-kirkwood-tld-1
Created:      Tue Apr 28 22:03:42 2020
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    12944762 Bytes = 12641.37 KiB = 12.35 MiB
Load Address: 00000000
Entry Point:  00000000

It is rather big for this kernel. My NSA325 installation:

-rw-r--r--  1 root root 9.3M Apr 18 01:19 initrd.img-5.6.5-kirkwood-tld-1
-rw-r--r--  1 root root 9.3M Apr 18 01:20 uInitrd

So I think perhaps there is something not right in your rootfs. What's in the rootfs has some effect to the size of initrd.

What is the size of /boot/initrd.img-5.6.5-kirkwood-tld-1 ?

Try create a new USB rootfs with Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi.tar.bz2. Boot with it and then upgrade to 5.6.5. See if you have the same problem.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Here are the sizes:

-rw-r--r--  1 root root 13182617 Apr 28 11:12 initrd.img-5.5.1-kirkwood-tld-1
-rw-r--r--  1 root root 12944762 Apr 28 22:02 initrd.img-5.6.5-kirkwood-tld-1

just looking in boot though did you intend for this typo?

config-5.6.5-kikrwood-tld-1
zImage-5.6.5-kikrwood-tld-1

would this effect the install process?

> just looking in boot though did you intend for
> this typo?
>
>

> config-5.6.5-kikrwood-tld-1
> zImage-5.6.5-kikrwood-tld-1
>

>
> would this effect the install process?

No it would not effect the install process on this box.

But thanks for noticing it. It could effect the installation for boxes with stock u-boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

im debating if I should just stick with 5.5.1 for now since I dont really care to reinstall omv and setup anything else from scratch again. There nothing in the kernel that i know that i would need to have.

Thanks for help so far as always

Hello.

I was experiencing exactly the same problem with NSA325.
Everything worked with clean rootfs and kernel, but didn't work with my system (which also has omv and lots of custom setup on it).

Too large kernel was also the primary suspect (mine was also around 13MB) - so I decided to try and make it smaller. For that I have created
"/etc/initramfs-tools/conf.d/custom.conf" file with the following content:

MODULES=dep
BUSYBOX=n

After that sizes of initrd and uInitrd files were reduced to about 7.7MB - and the system managed to boot smoothly.

Hope this helps.

odmink0,

> Too large kernel was also the primary suspect
> (mine was also around 13MB) - so I decided to try
> and make it smaller. For that I have created
> "/etc/initramfs-tools/conf.d/custom.conf" file
> with the following content:
>

> MODULES=dep
> BUSYBOX=n
>

> After that sizes of initrd and uInitrd files were
> reduced to about 7.7MB - and the system managed to
> boot smoothly.

Very nice solution! you could also strip the initrd to make it smaller. But your method is very elegant.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I've been using XZ compression to reduce the size further in some cases. After some trial and error I've found these settings to be the best trade-off between speed/compression while keeping memory usage low:

```
~# cat /etc/initramfs-tools/conf.d/compress
COMPRESS=xz
XZ_OPT=-2e
export XZ_OPT
```

downgrading kmod to a version before they added the libssl dependency (to the Stretch version) also saves like 1MB+. As far as I know those features aren't actually used in either Bohdi's or Debian's kernels anyway (@bohdi correct me if I'm wrong). It might even be worth it for Bohdi to compile a custom version of modern kmod without the SSL option and include in the rootfs if initrd size ends up becoming a bigger problem for folks.

> ```
> ~# cat /etc/initramfs-tools/conf.d/compress
> COMPRESS=xz
> XZ_OPT=-2e
> export XZ_OPT
> ```

Nice!

However, I wonder if we can handle XZ from u-boot. I've been using the default gzip so that it will be compatible with all versions of u-boot, old and news.


> downgrading kmod to a version before they added
> the libssl dependency (to the Stretch version)
> also saves like 1MB+. As far as I know those
> features aren't actually used in either Bohdi's or
> Debian's kernels anyway (@bohdi correct me if I'm
> wrong). It might even be worth it for Bohdi to
> compile a custom version of modern kmod without
> the SSL option and include in the rootfs if initrd
> size ends up becoming a bigger problem for folks.

Thanks, that's an idea to keep in mind.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I haven't tried you're uboot/kernels and I know that modern uboot documentation mentions handling decompression of uImages... but every kernel/uboot I've worked with has relied on the kernel to do the actual decompression of the initrd image which works as long as the associated kernel options have been enabled.

The settings I listed above have been tested on devices with 64MB of memory (different levels of XZ compression have different memory requirements).

I've been working on making Buster work on some ~13 year old Orion5x devices with their original uboot and as little as 64MB ram which has led me to learn all sorts of things about reducing initrd size.

I also learned there is a qemu wrapper for deboostrap which makes setting up a rootfs without the installer way easier than I had remembered.

1000001101000,

> I also learned there is a qemu wrapper for
> deboostrap which makes setting up a rootfs without
> the installer way easier than I had remembered.

That's interesting. I used the standard debootstrap to create the Debian armhf rootfs (https://forum.doozan.com/read.php?2,32146). And place the qemu-arm-static inside the rootfs. And chroot in, and run debootstrap --second-stage (since I created this armhf rootfs on x86 laptop).

Are we talking about the same thing or is that wrapper something new?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Honestly, it's been a few years since I did a deboostrap so I may be mistaken about the ways that it differs from the normal process. I believe the main thing is that it automatically handles the emulation for the second stage automatically.

You still have to copy in qemu-arm-static before chrooting into it afterwards, I don't know if the process copies it in and then deletes it afterwards or uses a different method.

I had never used qemu-arm-static or chroot'ed across architectures before so this was probably more exciting for me than it should have been. But after spending the last two years tinkering with preseed configurations for the debian installer it was pretty refreshing to get a functional(ish) rootfs with a one-liner:

qemu-debootstrap --arch armel --include=flash-kernel,haveged,openssh-server,busybox,libpam-systemd,dbus,u-boot-tools buster "$target" http://deb.debian.org/debian/

> pretty refreshing to get a functional(ish)
> rootfs with a one-liner:
>
>

> qemu-debootstrap --arch armel
> --include=flash-kernel,haveged,openssh-server,busybox,libpam-systemd,dbus,u-boot-tools
> buster "$target" http://deb.debian.org/debian/
>

Indeed pretty nice! looks like this skips the manual steps in second stage. I'll try it the next time.

Thanks!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 05/29/2020 11:05PM by bodhi.

I tried added the file "/etc/initramfs-tools/conf.d/custom.conf" to my system and I still get:

root@NSA320:~# cd /boot
root@NSA320:/boot# mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.6.5-kirkwood-tld-1 -d initrd.img-5.6.5-kirkwood-tld-1 uInitrd
Image Name: initramfs-5.6.5-kirkwood-tld-1
Created: Wed Jun 10 09:31:40 2020
Image Type: ARM Linux RAMDisk Image (gzip compressed)
Data Size: 12944744 Bytes = 12641.35 KiB = 12.35 MiB
Load Address: 00000000
Entry Point: 00000000

was there something else i should have done?

edit:
or does this only apply to building the kernel yourself?



Edited 1 time(s). Last edit at 06/10/2020 09:14AM by Pickle.

Hi Pickle,

You would need to create a new initrd first. The mkimage command takes an initrd and converts it to uboot format, but won't create the initrd with the settings in your custom.conf.

To create a new initrd use update-initramfs. Manpage

Pickle,

> I tried added the file
> "/etc/initramfs-tools/conf.d/custom.conf" to my
> system and I still get:

You don't need to do that. Just use the standard installation instruction that I posted.

Quote

4a. Boot with DTB file (standard way to boot FDT kernel). Recommended.

- Generate the uImage and uInitrd (the kernel files vmlinuz-5.6.5-kirkwood-tld-1 and initrd.img-5.6.5-kirkwood-tld-1 were generated by dpkg in Step 3):

cd /boot
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.6.5-kirkwood-tld-1 -d vmlinuz-5.6.5-kirkwood-tld-1 uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.6.5-kirkwood-tld-1 -d initrd.img-5.6.5-kirkwood-tld-1 uInitrd

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I see two more possible reasons why a kernel will not boot:
-a file is too fragmented and uBoot may not be able to read too fragmented files
-a file is to the right of a huge partition and uBoot may not be able to read that far
My current solution:
mkdir /oldboot01
mv /boot /oldboot01
cp -r /oldboot01/boot /
Try boot, if ok stop.
mkdir /oldboot02
mv /boot /oldboot02
cp -r /oldboot02/boot /
Try boot, if ok stop. If not ok this is probably not a fragmented or right of huge partition issue.