Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 09, 2020 04:03AM
Hi :)

I installed a recent uboot and a Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi on my Zyxel NSA320s (on USB flash). It was working fine.

Then I wanted to upgrade kernel to the newer one. The last kernel that works reliably is 5.6.5. Newer kernels don't work.

I finally configured netconsole, so I'm now able to see the kernel startup log. I tried once again the most recent kernel 5.9.3. In the logs I see that out of the sudden the USB is disconnected:
[   45.006954][    T5] usb 1-1.3: USB disconnect, device number 5
And then there are lot of messages complaining about I/O issues, like:
[   45.034708][  T334] blk_update_request: I/O error, dev sdb, sector 1941760 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

I tried multiple times. I also tried different flash disk and the result is the same. Reverting kernel to 5.6 fixes the issue and everything works fine. So it is not a hardware problem (like power supply or broken flash).
I also tried to set usbcore.autosuspend=-1 in kernel arguments, but it didn't help.

Attached the full log.

Any idea how to fix it or what can be wrong?

Cheers,

Marek
Attachments:
open | download - bootlog.txt (35.5 KB)
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 09, 2020 04:45PM
Marek,

Don't boot with systemd.

Quote

[ 0.000000][ T0] Kernel command line: console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 nand_mtd:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2) init=/bin/systemd

Once you get it working OK running sysvinit, update/upgrade Debian to the latest. And then you can switch it back to systemd if you want to.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 10, 2020 02:33AM
Hi bodhi,

Thank you for the quick answer. Indeed, I was using systemd, because it is forced by openmediavault. I tried it with pure linux (without OMV) and removed systemd from kernel command line. I still see systemd-udevd in logs, so I'm not sure if I did it properly. But I see that /sbin/init is not symlinked to systemd anymore.

Unfortunately the final result is the same.

Please look at attached log.

Cheers,

Marek
Attachments:
open | download - booterr fresh installation init.txt (24.5 KB)
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 10, 2020 04:33PM
Marek,

Quote

I also tried different flash disk

Does that mean you recreate the USB rootfs from scracth and then after that upgrade the kernel to 5.9.3?

Have you done check disk after seeing the error during booting? I.e. take the USB drive to another lLinux box and run e2fsck.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 11, 2020 10:09AM
Yes, this is what I tried.

I unpacked rootfs on a fresh flash disk. I copied /boot to /boot-copy and then upgraded kernel in /boot. So with this approach I'm able to select original uImage and uInitrd before booting.
When I boot it using /boot-copy everything works fine, there are no errors. If I use kernel 5.9 stored in /boot then it fails.
There are no filesystem issues, I checked this in Raspberry PI.
I tried almost all kernels and the last that works is 5.6. Kernels 5.7, 5.8, 5.9 don't work. The logs I attached are from 5.9.

Cheers,

Marek
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
November 11, 2020 04:43PM
Marek,

Quote

If I use kernel 5.9 stored in /boot then it fails.

This does not sound right. The kernel on the fresh rootfs Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi.tar.bz2 should be 5.2.9.

Was that a typo, or did you try to install 5.9.3 kernel on this fresh rootfs before booting?

If that was not a typo, could you try to use a fresh rootfs with Debian-5.2.9-kirkwood-tld-1-rootfs-bodhi.tar.bz2 wihtout any modification and boot with it.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 06, 2020 03:42PM
Hi Bodhi,

Sorry for a late response, but I currently don't have direct access to this device. To clarify what I did previously:

1. Unpacked rootfs with kernel 5.2.9 to a fresh flash disk

2. Booted the system and it worked fine

3. Configured netconsole in /etc/initramfs-tools/modules, regenerated initramfs and uinitrd

4. Rebooted and everything worked fine and netconsole worked

5. Did a backup copy of the current boot directory (and named it boot-copy)

6. Upgraded the kernel to 5.9 using instructions from: https://forum.doozan.com/read.php?2,12096
cd /boot
tar -xjf linux-5.9.3-kirkwood-tld-1-bodhi.tar.bz2
tar -xf  linux-dtb-5.9.3-kirkwood-tld-1.tar
apt-get remove flash-kernel 
dpkg -i linux-image-5.9.3-kirkwood-tld-1_1.0_armel.deb
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.9.3-kirkwood-tld-1 -d vmlinuz-5.9.3-kirkwood-tld-1 uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.9.3-kirkwood-tld-1 -d initrd.img-5.9.3-kirkwood-tld-1 uInitrd
sync 
sync
sync
shutdown -r now

The uImage and uInitrd in /boot directory now point to 5.9 kernel

7. The system couldn't boot anymore

8. Whenever I booted the system from /boot-copy (backup copy containing 5.2) then it worked fine:
setenv load_uimage "echo loading uImage ...; load $bootdev $device $load_uimage_addr /boot.copy/uImage"
setenv load_initrd "echo loading uInitrd ...; load $bootdev $device $load_initrd_addr /boot.copy/uInitrd"


So it has something to do with the kernel.

Because I don't currently have access to the device I may only attach you the log from pure 5.2 boot generated at step '4'. See attached 'bootok' file. I captured it before you asked to remove systemd.

Cheers,

Marek
Attachments:
open | download - bootok (18.5 KB)
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 06, 2020 03:51PM
Marek,

The boot log with dmesg will not be enough to trouble shoot the problem. It is best that you have serial console connected and get the serial boot log.

If you cannot connect serial console at the moment, then describe what u-boot is on this box. Is that stock u-boot or my new u-boot?

https://forum.doozan.com/read.php?3,12381

If this box is booting with new u-boot, then boot back to fresh rootfs 5.2.9, and in Debian, get the listing of the envs:

fw_printenv

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 07, 2020 12:54AM
Hi Bodhi,

Thank you for the quick answer. For now I can access this box only via ssh. I'll connect the serial console when I get the physical access again (in few weeks).

This box has your u-boot in its latest version. Attached the printenv dump.

Cheers,

Marek
Attachments:
open | download - printenv (3 KB)
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 07, 2020 05:09PM
Marek,

The envs look OK. But it was probably the set that you saved after you installed u-boot. Does not look like the current envs.

SSH in with USB rootfs, and run:
dmesg
cat /proc/mtd
fw_printenv
and post the entire log here (including any errors, i.e. everything).

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 29, 2020 06:43AM
Hi bodhi,

I took the fresh flash disk, unpacked the rootfs and executed the commands. Pls look at the attached listing.

Cheers,

Marek
Attachments:
open | download - fresh 5.2.9 listing.txt (24.8 KB)
Re: Zyxel NSA320s doesn't work with bodhi kernels > 5.6
December 29, 2020 04:24PM
Marek,

The log looks very good. But there is one small problem with the mtd partitions:

mtdparts=nand_mtd:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)

It should be set to this in seriall console:

setenv mtdparts 'orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)'

With that corrected mtd device, after booting into Debian, you can set envs with fw_setenv. Note that after booting with the corrected mtdparts, in Debian you need to make it permanent:

fw_setenv mtdparts 'orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(kernel_1),0x2FC0000(rootfs1),0xA00000(kernel_2),0x2FC0000(rootfs2)'


Other than that, you also need to do Note 1 (don't need to do apt-get upgrade right away, just generate new SSH key).

Quote

5. Done. Take this USB rootfs to your plug and cold start. After booted into Debian, see Note1 and Note2 below. It is very important that you do Note1 steps to secure your box.

Note1:

After logging in this rootf the first time, remember to generate new SSH key to make it your own unique rootfs. And also update your rootfs to get the latest Debian package security updates:
rm /etc/ssh/ssh_host*
ssh-keygen -A

apt-get update
apt-get upgrade

-bodhi
===========================
Forum Wiki
bodhi's corner
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: