Problemes 2nd SATA port on Zyxel NAS326: Something wrong with your kernel?

Hello forum,

I'm trying to run my new Zyxel NAS 326 with your kernel and Debian. After 2 non-booting Flash drives I managed to boot Debian 10 with your latest kernels 3.11 and 3.13. Now I wanted to install my new 4TB disks into a LVM RAID1. But something is wrong with the second SATA port (/dev/sdb) when running your kernel!

I checked the SMART values of the new disks with smartctl. Port 1 with /dev/sda works flawlessly but port 2 only emits the info you see when using the command "smartctl -i /dev/sdb" but with options "-a" o "-x" it hangs after these initial infos for 90 seconds or more and then quits. dmesg gives these errors:

[  594.139482] ata2: link is slow to respond, please be patient (ready=0)
[  618.581714] ata2: softreset failed (device not ready)
[  618.587496] ata2: limiting SATA link speed to 3.0 Gbps
[  618.593517] ata2: hard resetting link
[  623.784315] ata2: softreset failed (device not ready)
[  623.790097] ata2: reset failed, giving up
[  623.794976] ata2.00: disabled

When initializing a PV of LVM, dmesg emits on port 2 (sdb):

[   35.078312] pwr_en_sata0: disabling
[   35.082529] v5.0-sata0: disabling
[   35.086556] v12.0-sata0: disabling
[   35.090860] pwr_en_sata1: disabling
[   35.095139] v5.0-sata1: disabling
[   35.099305] v12.0-sata1: disabling
[  563.444157] ata2.00: exception Emask 0x0 SAct 0x8 SErr 0x0 action 0x6 frozen
[  563.451951] ata2.00: failed command: WRITE FPDMA QUEUED
[  563.458097] ata2.00: cmd 61/10:18:00:00:00/00:00:00:00:00/40 tag 3 ncq dma 8192 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  563.474755] ata2.00: status: { DRDY }
[  563.479143] ata2: hard resetting link
[  573.509167] ata2: softreset failed (device not ready)
[  573.514952] ata2: hard resetting link
[  583.544193] ata2: softreset failed (device not ready)
[  583.549973] ata2: hard resetting link
[  593.514854] systemd-udevd[6489]: Spawned process 'ata_id --export /dev/sdb' [6490] is taking longer than 59s to complete
[  593.526532] systemd-udevd[1784]: sdb: Worker [6489] processing SEQNUM=3632 is taking a long time
[  594.139482] ata2: link is slow to respond, please be patient (ready=0)
[  618.581714] ata2: softreset failed (device not ready)
[  618.587496] ata2: limiting SATA link speed to 3.0 Gbps
[  618.593517] ata2: hard resetting link
[  623.784315] ata2: softreset failed (device not ready)
[  623.790097] ata2: reset failed, giving up
[  623.794976] ata2.00: disabled
[  623.798685] ata2: EH complete
[  623.802436] sd 1:0:0:0: [sdb] tag#3 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[  623.812941] sd 1:0:0:0: [sdb] tag#3 CDB: Write(16) 8a 00 00 00 00 00 00 00 00 00 00 00 00 10 00 00
[  623.822641] blk_update_request: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0
[  623.849684] sd 1:0:0:0: [sdb] tag#9 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[  623.860115] sd 1:0:0:0: [sdb] tag#9 CDB: Write(16) 8a 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00
[  623.869823] blk_update_request: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x8800 phys_seg 32 prio class 0
[  623.885603] sdb: detected capacity change from 7814037168 to 0

So I thought I have faulty hardware, remaining after disassembly of the NAS326 and cleaning all contacts. But I gave it a last try with the stock firmware and to my surprise everything works as expected! The web GUI creates the RAID1 without hassle. SSH into it and using their smartctl I can see all SMART values for both ports, no hang an second like with your kernel and Debian, no kernel errors shown by dmesg. The stock kernel is "Linux NAS326 3.10.39 #1 Thu Apr 1 10:04:14 CST 2021 armv7l GNU/Linux"

What's going on here??? I doubt that I'm the only one who uses both ports with bhodi's kernels and a favourite distribution. So what I'm doing wrong?

One remark: Today I upgraded the kernel from 3.11 to 3.13 with bhodi's package. You have to install the DEB with dpkg and create the compressed images yourself. According to the instructions, there should be a versioned zImage inside /boot. But this image is only included inside the tarball and is not installed by the DEB package. I have to copy the zImage myself into /boot. Once again: Is it just me? Nobody else is complaining?

Thanks for the support,
Andreas

Andreas,

There used to be a problem with the 2nd HDD. This is due to the power GPIO. I thought I've fixed this a few kernel releases ago. But let me doublecheck to see if I updated the instruction. I might have forgotten to do that.

> One remark: Today I upgraded the kernel from 3.11
> to 3.13 with bhodi's package. You have to install
> the DEB with dpkg and create the compressed images
> yourself. According to the instructions, there
> should be a versioned zImage inside /boot. But
> this image is only included inside the tarball and
> is not installed by the DEB package. I have to
> copy the zImage myself into /boot. Once again: Is
> it just me? Nobody else is complaining?

The zImage is not installed by DEB package. The correct procedure is in the installation instruction.

Quote

2. Extract the kernel and the DTB files in the archive:

cd /boot
tar -xjf linux-5.13.8-mvebu-tld-1-bodhi.tar.bz2
tar -xf linux-dtb-5.13.8-mvebu-tld-1.tar

3. And install it with dpkg. Remove flash-kernel first to avoid potential problem:

cd /boot
apt-get remove flash-kernel
dpkg -i linux-image-5.13.8-mvebu-tld-1_1.0_armhf.deb

4. Create uImage and uInitrd (kernel file initrd.img-5.13.8-mvebu-tld-1 was generated by dpkg in step 3). I am using the Zyxel NAS326 as an example here. Please replace armada-380-zyxel-nas326.dtb below with the appropriate DTB name for your box.

cd /boot
cp -a zImage-5.13.8-mvebu-tld-1 zImage.fdt
cat dts/armada-380-zyxel-nas326.dtb >> zImage.fdt
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.13.8-mvebu-tld-1 -d zImage.fdt uImage
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-5.13.8-mvebu-tld-1 -d initrd.img-5.13.8-mvebu-tld-1 uInitrd

Note that eventhough you have zImage and DTB, you still need to run dpkg to set up the initramfs, and populate the modules.

-bodhi
===========================
Forum Wiki
bodhi's corner

Indeed, I forgot about this extra step that we worked out later.

Interrupt serial console, and do this

setenv init_sata 'mw.l f1010104 c70e0ffd; ide reset'
run init_sata

And then continue booting

boot

-bodhi
===========================
Forum Wiki
bodhi's corner

Thanks a lot, bodhi!

I'm glad we can sort it out and it's not a hardware fault on my side.

> Interrupt serial console, and do this

Do I have to open the case another time? fw_sentenv from inside stock or my Debian doesn't work?

> The zImage is not installed by DEB package. The
> correct procedure is in the installation
> instruction.

Okay, I see the culprit. You're extracting the tarball inside /boot – a thing I refused to do. So never mind I'll update my own update-boot script. But why not including zImage into the DEB package? dpkg will install everything to the right place, no cluttering of /boot with the extracted tarball and no manual copy needed.

Thank you again for your work. You're giving life to inexpensive working hardware that was abandoned by it's manufacturer. It's a shame that device like old Android cell phones aren't that easy.

Andreas

Andreas,

> Do I have to open the case another time?
> fw_sentenv from inside stock or my Debian doesn't
> work?

It would work, too. But I need to see your envs to suggest where this should be added.

Get a listing of the envs

fw_printenv

> Okay, I see the culprit. You're extracting the
> tarball inside /boot – a thing I refused to do.

The instruction is to make it simpler for beginner. You can extract it anywhere you'd like.

> So never mind I'll update my own update-boot
> script. But why not including zImage into the DEB
> package? dpkg will install everything to the right
> place, no cluttering of /boot with the extracted
> tarball and no manual copy needed.

The zImage is provided separately so in case you want to boot the kernel without initrd. You could use the zImage and the DTB only, without having to run dpkg. This is useful when somebody running stock OS and just want to make a first boot and cannot run dpkg (this scenario happens a lot for many reasons).

====

Note that usually for other ARM boxes that have a newer u-boot, which loads the DTB separately, the kernel installation process does not have to involved zImage. Just vmlinuz and initrd (which are in the DEB package).

This box can load separate DTB, too. But there is a problem in this stock u-boot doing that.

-bodhi
===========================
Forum Wiki
bodhi's corner

Hi bodhi!

> It would work, too. But I need to see your envs to
> suggest where this should be added.

I tried it similar to your USB initialization. All other vars are set according to your instructions.

sata_init=mw.l f1010104 c70e0ffd; sleep 3; ide reset
usb_bootcmd=echo Booting from USB ...; setenv fdt_skip_update yes; run sata_init; run usb_init; ext2load usb 0:1 $load_image_addr /boot/uImage; ext2load usb 0:1 $load_initrd_addr /boot/uInitrd; run

It does not work, port 2 is still non-functional. Seems I do not understand the crucial details of u-booting...

Andreas,

fw_printenv

-bodhi
===========================
Forum Wiki
bodhi's corner

Okay, here's a copy of the output:

CASset=max
CONTRY_TYPE=FF
FEATURE_BIT=00
MALLOC_len=5
MODEL_ID=B303
MPmode=SMP
PRODUCT_NAME=STG-328
VENDOR_NAME=MitraStar Technology Corp.
autoload=no
baudrate=115200
boot_order=hd_scr usb_scr mmc_scr hd_img usb_img mmc_img pxe net_img net_scr
bootargs=console=ttyS0,115200 ubi.mtd=4,2048 rootfstype=ubifs root=ubi0:rootfs1 rw rootdelay=2
bootargs_dflt=$console $nandEcc $mtdparts $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel
bootargs_end=:10.4.50.254:255.255.255.0:Armada38x:eth0:none
bootargs_root=root=/dev/nfs rw
bootcmd=nand read 0x2000000 0x08700000 0xF00000 && bootz 0x2000000
bootcmd_auto=stage_boot $boot_order
bootcmd_custom=if run usb_bootcmd; then; else if run bootcmd_stock_1; then; else run bootcmd_stock_2; reset; fi; fi
bootcmd_fdt=tftpboot 0x2000000 $image_name;tftpboot $fdtaddr $fdtfile;setenv bootargs $console $nandEcc $mtdparts $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel; bootz 0x2000000 - $fdtaddr;
bootcmd_fdt_boot=tftpboot 0x2000000 $image_name; setenv bootargs $console $nandEcc $mtdparts $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel; bootz 0x2000000 - $fdtaddr;
bootcmd_fdt_edit=tftpboot $fdtaddr $fdtfile; fdt addr $fdtaddr; setenv bootcmd $bootcmd_fdt_boot
bootcmd_lgcy=tftpboot 0x2000000 $image_name; setenv bootargs $bootargs_dflt; bootm 0x2000000;
bootcmd_stock_1=run set_bootargs_stock; echo Booting from NAND kernel 1 ...; nand read 0x2000000 0x00E00000 0xF00000 && bootz 0x2000000
bootcmd_stock_2=run set_bootargs_stock; echo Booting from NAND kernel 2 ...; nand read 0x2000000 0x08700000 0xF00000 && bootz 0x2000000
bootdelay=3
cacheShare=no
change_boot_part=1
console=console=ttyS0,115200
core_checksum_1=d21254373dea21758e1242295d1a9579
core_checksum_2=b2e7f9d46be6914446516f2cbedfd7eb
curr_bootfrom=1
device_partition=0:1
disaMvPnp=no
eeeEnable=no
enaClockGating=no
enaCpuStream=no
enaFPU=yes
enaMonExt=no
enaWrAllo=no
eth1addr=00:50:43:38:32:25
eth1mtu=1500
eth2addr=00:50:43:38:a6:25
eth2mtu=1500
eth3addr=00:50:43:32:a6:38
eth3mtu=1500
ethact=egiga0
ethaddr=58:8B:F3:DB:D8:3F
ethmtu=1500
ethprime=egiga0
fdt_addr=2040000
fdt_skip_update=no
fdtaddr=0x1000000
fdtfile=armada-38x-modular.dtb
fwversion_1=V5.10(AAZF.0)
fwversion_2=V5.21(AAZF.10)
ide_path=/
image_name=uImage
img_checksum_1=e6d8d12654063c2611c5dd4be0fccb25
img_checksum_2=ea11142db3b34adc3602086d20479a95
initrd_name=uInitrd
ipaddr=10.4.50.170
kernel_addr=${kernel_addr_1}
kernel_addr_1=0x00000000; run bootcmd_custom; 
kernel_addr_2=0x08700000
kernel_addr_r=2080000
kernel_mtd_1=3
kernel_mtd_2=5
lcd0_enable=0
lcd0_params=640x480-16@60
lcd_panel=0
load_image_addr=0x02000000
load_initrd_addr=0x2900000
loadaddr=0x02000000
loads_echo=0
modelid_1=B303
modelid_2=B303
mtdids=nand0=armada-nand
mtdparts=mtdparts=armada-nand:2m(u-boot),2m(env),2m(config),15m(kernel1),110m(rootfs1),15m(kernel2),-(rootfs2)
mvNetConfig=mv_net_config=4,(00:50:43:11:11:11,0:1:2:3),mtu=1500
mv_pon_addr=00:50:43:25:a6:38
nandEcc=nfcConfig=4bitecc
netbsd_en=no
netmask=255.255.255.0
netretry=no
next_bootfrom=1
pcieTune=no
pexMode=RC
pxe_files_load=:default.arm-armadaxp-db:default.arm-armadaxp:default.arm
pxefile_addr_r=3100000
ramdisk_addr_r=2880000
revision_1=48379
revision_2=51209
romfile_checksum_1=4C07
romfile_checksum_2=7010
rootpath=/srv/nfs/
sata_delay_reset=0
sata_dma_mode=yes
script_addr_r=3000000
script_name=boot.scr
serial_number=S150Y46041116
serverip=10.4.50.38
set_bootargs_stock=setenv bootargs "console=ttyS0,115200 ubi.mtd=4,2048 rootfstype=ubifs root=ubi0:rootfs1 rw rootdelay=2"
standalone=fsload 0x2000000 $image_name;setenv bootargs $console $nandEcc $mtdparts root=/dev/mtdblock0 rw ip=$ipaddr:$serverip$bootargs_end; bootm 0x2000000;
stderr=serial
stdin=serial
stdout=serial
sysimg_mtd_1=4
sysimg_mtd_2=6
usb0Mode=host
usbActive=0
usbType=2
usb_bootcmd=echo Booting from USB ...; setenv fdt_skip_update yes; run usb_init; ext4load usb 0:1 $load_image_addr /boot/uImage; ext4load usb 0:1 $load_initrd_addr /boot/uInitrd; run usb_set_bootargs; bootm $load_image_addr $load_initrd_addr
usb_init=mw.l f1018100 20420000; mw.l f1018140 003E8800; sleep 3; usb start
usb_set_bootargs=setenv bootargs "console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts earlyprintk=serial"
vxworks_en=no
yuk_ethaddr=00:00:00:EE:51:81
zld_checksum_1=ef595c7985244ff6476518eb5f9e9b85
zld_checksum_2=e861824adfedf3827be8b63e8405ce7e

Thanks bodhi, I was able to solve the problem. It seems that you do not distribute the correct DTB in the rootfs or kernel packages – at least since version 5.2.9. But I found your own post Fix for missing 2nd HDD from last year. With this DTB everything seems to be alright and even the suggested fix for sata_init through u-boot environment vars isn't needed.

Andreas,

Execellent! I thought I've fixed this in the DTS, too. I must have been quite distracted and did not make a note to include the new DTB.

Please come back and install new kernel when I release the next version (in case I don't have a chance to test with 2nd HDD).

-bodhi
===========================
Forum Wiki
bodhi's corner

Andreas,

Strange, the NAS326 DTS/DTB in the latest kernel 5.13.8 looks be the correct one!!!

Whhich kernel are you running?

-bodhi
===========================
Forum Wiki
bodhi's corner



Edited 1 time(s). Last edit at 08/30/2021 04:36PM by bodhi.