Zyxel NSA320s doesn't work with bodhi kernels > 5.6

Hi :)

I installed a recent uboot and a Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi on my Zyxel NSA320s (on USB flash). It was working fine.

Then I wanted to upgrade kernel to the newer one. The last kernel that works reliably is 5.6.5. Newer kernels don't work.

I finally configured netconsole, so I'm now able to see the kernel startup log. I tried once again the most recent kernel 5.9.3. In the logs I see that out of the sudden the USB is disconnected:

[   45.006954][    T5] usb 1-1.3: USB disconnect, device number 5

And then there are lot of messages complaining about I/O issues, like:

[   45.034708][  T334] blk_update_request: I/O error, dev sdb, sector 1941760 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

I tried multiple times. I also tried different flash disk and the result is the same. Reverting kernel to 5.6 fixes the issue and everything works fine. So it is not a hardware problem (like power supply or broken flash).
I also tried to set usbcore.autosuspend=-1 in kernel arguments, but it didn't help.

Attached the full log.

Any idea how to fix it or what can be wrong?

Cheers,

Marek

Marek,

Don't boot with systemd.

Quote

[ 0.000000][ T0] Kernel command line: console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 nand_mtd:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2) init=/bin/systemd

Once you get it working OK running sysvinit, update/upgrade Debian to the latest. And then you can switch it back to systemd if you want to.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

Thank you for the quick answer. Indeed, I was using systemd, because it is forced by openmediavault. I tried it with pure linux (without OMV) and removed systemd from kernel command line. I still see systemd-udevd in logs, so I'm not sure if I did it properly. But I see that /sbin/init is not symlinked to systemd anymore.

Unfortunately the final result is the same.

Please look at attached log.

Cheers,

Marek

Marek,

Quote

I also tried different flash disk

Does that mean you recreate the USB rootfs from scracth and then after that upgrade the kernel to 5.9.3?

Have you done check disk after seeing the error during booting? I.e. take the USB drive to another lLinux box and run e2fsck.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Yes, this is what I tried.

I unpacked rootfs on a fresh flash disk. I copied /boot to /boot-copy and then upgraded kernel in /boot. So with this approach I'm able to select original uImage and uInitrd before booting.
When I boot it using /boot-copy everything works fine, there are no errors. If I use kernel 5.9 stored in /boot then it fails.
There are no filesystem issues, I checked this in Raspberry PI.
I tried almost all kernels and the last that works is 5.6. Kernels 5.7, 5.8, 5.9 don't work. The logs I attached are from 5.9.

Cheers,

Marek

Marek,

Quote

If I use kernel 5.9 stored in /boot then it fails.

This does not sound right. The kernel on the fresh rootfs Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi.tar.bz2 should be 5.2.9.

Was that a typo, or did you try to install 5.9.3 kernel on this fresh rootfs before booting?

If that was not a typo, could you try to use a fresh rootfs with Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi.tar.bz2 wihtout any modification and boot with it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Bodhi,

Sorry for a late response, but I currently don't have direct access to this device. To clarify what I did previously:

1. Unpacked rootfs with kernel 5.2.9 to a fresh flash disk

2. Booted the system and it worked fine

3. Configured netconsole in /etc/initramfs-tools/modules, regenerated initramfs and uinitrd

4. Rebooted and everything worked fine and netconsole worked

5. Did a backup copy of the current boot directory (and named it boot-copy)

6. Upgraded the kernel to 5.9 using instructions from: https://forum.doozan.com/read.php?2,12096

cd /boot
tar -xjf linux-5.9.3-kirkwood-tld-1-bodhi.tar.bz2
tar -xf  linux-dtb-5.9.3-kirkwood-tld-1.tar
apt-get remove flash-kernel 
dpkg -i linux-image-5.9.3-kirkwood-tld-1_1.0_armel.deb
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.9.3-kirkwood-tld-1 -d vmlinuz-5.9.3-kirkwood-tld-1 uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.9.3-kirkwood-tld-1 -d initrd.img-5.9.3-kirkwood-tld-1 uInitrd
sync 
sync
sync
shutdown -r now

The uImage and uInitrd in /boot directory now point to 5.9 kernel

7. The system couldn't boot anymore

8. Whenever I booted the system from /boot-copy (backup copy containing 5.2) then it worked fine:

setenv load_uimage "echo loading uImage ...; load $bootdev $device $load_uimage_addr /boot.copy/uImage"
setenv load_initrd "echo loading uInitrd ...; load $bootdev $device $load_initrd_addr /boot.copy/uInitrd"

So it has something to do with the kernel.

Because I don't currently have access to the device I may only attach you the log from pure 5.2 boot generated at step '4'. See attached 'bootok' file. I captured it before you asked to remove systemd.

Cheers,

Marek

Marek,

The boot log with dmesg will not be enough to trouble shoot the problem. It is best that you have serial console connected and get the serial boot log.

If you cannot connect serial console at the moment, then describe what u-boot is on this box. Is that stock u-boot or my new u-boot?

https://forum.doozan.com/read.php?3,12381

If this box is booting with new u-boot, then boot back to fresh rootfs 5.2.9, and in Debian, get the listing of the envs:

fw_printenv

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Bodhi,

Thank you for the quick answer. For now I can access this box only via ssh. I'll connect the serial console when I get the physical access again (in few weeks).

This box has your u-boot in its latest version. Attached the printenv dump.

Cheers,

Marek

Marek,

The envs look OK. But it was probably the set that you saved after you installed u-boot. Does not look like the current envs.

SSH in with USB rootfs, and run:

dmesg
cat /proc/mtd
fw_printenv

and post the entire log here (including any errors, i.e. everything).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

I took the fresh flash disk, unpacked the rootfs and executed the commands. Pls look at the attached listing.

Cheers,

Marek

Marek,

The log looks very good. But there is one small problem with the mtd partitions:

mtdparts=nand_mtd:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)

It should be set to this in seriall console:

setenv mtdparts 'orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)'

With that corrected mtd device, after booting into Debian, you can set envs with fw_setenv. Note that after booting with the corrected mtdparts, in Debian you need to make it permanent:

fw_setenv mtdparts 'orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)'

Other than that, you also need to do Note 1 (don't need to do apt-get upgrade right away, just generate new SSH key).

Quote

5. Done. Take this USB rootfs to your plug and cold start. After booted into Debian, see Note1 and Note2 below. It is very important that you do Note1 steps to secure your box.

Note1:

After logging in this rootf the first time, remember to generate new SSH key to make it your own unique rootfs. And also update your rootfs to get the latest Debian package security updates:
rm /etc/ssh/ssh_host*
ssh-keygen -A

apt-get update
apt-get upgrade

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

After 1,5 years I came back to the topic and I have some interesting observations.

I retried it and used the newest kernel and it worked 'somehow' (not very stable) on some older memory stick, and didn't work at all on newer memory stick. Finally, I tried to connect the memory stick via USB hub with an own power supply and it worked like a charm.

So, it is very likely that all my problems were caused by unstable power in USB port.

In past I was also observing some USB misbehavior when booting:

starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... Error: Cannot find high speed parent of usb-1 device
Error: Cannot find high speed parent of usb-1 device
Error: Cannot find high speed parent of usb-1 device
Error: Cannot find high speed parent of usb-1 device

Now I don't see that anymore.

But finally, I bought an additional SSD drive and I don't use USB stick anymore.

Thank you for the support and your great work!

Cheers,

Marek

Hi Marek,

Good to know that was a USB port power problem. Thanks for the feedback!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Marek

I would like to ask you for help, I have same NAS as you. Is your nas running fine with 2 HDDs? my tread with detail of problem is here https://forum.doozan.com/read.php?3,51739



Edited 1 time(s). Last edit at 06/26/2022 03:36AM by mervin.

mervin,

> as you. Is your nas running fine with 2 HDDs? my
> tread with detail of problem is here
>

I've looked at the NSA320S DTS. I think I need to modify it for you (2 HDDs) and Marek (USB) to try.

Unfortunately, I'm about to relase kernel 5.18.6 so it will have to be done separately in a new DTB.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Yes, it works fine for me with 2 disks:
- one is an old HDD WD Red 2TB
- second is a SSD GoodRAM 256GB (that I now use instead of USB stick)

Marek

bodhi Wrote:
-------------------------------------------------------
> mervin,
>
> > as you. Is your nas running fine with 2 HDDs?
> my
> > tread with detail of problem is here
> >
>
>
> I've looked at the NSA320S DTS. I think I need to
> modify it for you (2 HDDs) and Marek (USB) to
> try.
>
> Unfortunately, I'm about to relase kernel 5.18.6
> so it will have to be done separately in a new
> DTB.

Hi Bodhi

I have tried powered usb hub and all my problems have been solved. and the clicking noise was falling HDD a have replaced it and all working as charm. now I need to update to latest release, get front leds working, setup samba and its done

Mervin,

Cool! I thought it is strange that you have power problem while others dont. So I was thinking to force the regulator to run in "always on" mode.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)