Synology DJ216j NAS unbrick error

Hallo, may I ask someone kind and willing which skilled enough to help me to unbrick my NAS which bricked due to the power outage during DSM update? I am able to connect through serial console (putty) and transfer zImage and rd.bin (tftpd32) into it but facing "no ATAGS support: can't continue" message after bootm... Synology assistant and hard factory reset are not working. Thanks in advance.



Edited 1 time(s). Last edit at 01/14/2020 12:48PM by jean.

jean,

From Synology website info, your box is DS216j.

DS216j	Marvell Armada 385 88F6820	Dual Core	2	Yes	Armada38x	DDR3 512 MB

It is same SoC as the Thecus N2350. So you can look in this installation thread:

https://forum.doozan.com/read.php?2,92234

Quote

NOTE1

1. To prepare for UART booting, use 2 SSH terminals. First one for running the download script, and the 2nd terminal for running a regular serial console which should be started immediately when "transfer complete" to interupt u-boot (the default countdown is only 3 seconds so there is not enough time to interupt u-boot if you wait too long).

2. The files necessary to run UART booting are in the new u-boot tarball thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi.tar. Please see Step 2.c in the Installation Instruction steps.

3. Power down the N2350. Connect serial console on an ARM Linux box.

Basically you can load stock u-boot through serial console and execute it. The one used in the N2350 thread is in section 2c.

Quote

2.c. Download the new u-boot thecus-n2350-Marvell-2015_T1.0p18-tld-4 uboot tarball to rootfs /boot/uboot folder.

Quote

Download at Dropbox
thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi.tar
md5sum
1a08b09b2e01824b47b525621fcc9944
sha256sum
8284087a4b3cde7db0b46e0942e985b7da6c52ccf65ee741fd4a2ceab5581ee9

So read that installation thread section NOTE 1, and download my N2350 u-boot build: thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi.tar.

Post questions if you are not sure about how to proceed. If you can load and start this u-boot image, then it is unbricked.

UPDATE:

Note: you need to run the UART booting procdure from another Linux box (not Windows). And that Linux box it is not ARM based box, then I need to recompile, or post a slightly different script for you to run on Intel x86 box.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 01/14/2020 12:49PM by bodhi.

Thank You -bodhi for prompt reply, is there any way how to proceed using the legacy (original) kernel and ramdisk? There is .pat (.tar) file available from the vendor.

Hi jean,

It is possible to load stock FW through serial console. But the error indicated that your u-boot envs were probably corrupted because of the outage.

So I need to see stock u-boot envs to help you loading the FW. Look at Synology website, or google to find that.

To start, connect putty, power up and interrupt serial console at countdown, and

ver
printenv

and post the entire serial console log here.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Here it is.

Jean,

Looks like the envs are ok! So only the kernel must have been corrupted.

So let me take a look at your tarball to see how we can load it. Is there instruction that comes with the tarball? Or Synology has it in the website?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

It is on the Synology ftp site https://archive.synology.com/download/, I only don't remember exactly what version was last functioning, guessing it was 6.1.5. BTW is there possibility that my "no ATAGS support" error is somehow related to the fact the loader is not provided with either any at all or correct MAC and S/N of my NAS? I also wonder what if I try to put older version of the FW than it was in, may this cause the problem?



Edited 1 time(s). Last edit at 01/15/2020 05:46AM by jean.

Jean,


> It is on the Synology ftp site
> https://archive.synology.com/download/, I only
> don't remember exactly what version was last
> functioning, guessing it was 6.1.5. BTW is there
> possibility that my "no ATAGS support" error is
> somehow related to the fact the loader is not
> provided with either any at all or correct MAC and
> S/N of my NAS?

No that is not related to MAC address or S/N.

That error is when the kernel expects an ATAG structure passed from uboot. Kernels that expects ATAG is of much older version. Modern kernels use Flatten Device Tree (FDT) and expects a DTB file.

> I also wonder what if I try to put
> older version of the FW than it was in, may this
> cause the problem?

That's quite likely.

During upgrade, probably some uboot envs (or uboot itself) have been changed to boot the FW with new kernel with DTB. But the kernel image is old so it expects ATAG.

Since the envs seem sane, you could try to FW reset with the latest rescue FW, see if it woks. And try with each older restore FW version before and including 6.1.5.

When I have some free time I can look at the tarball.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thank You for the explanation and willingness. Would You pls ensure me lookin into my printenv that loading zImage to the 0x2000000 and rd.bin to 0x2880000 is correct procedure?

Jean,

Quote

Would You pls ensure me lookin into my printenv that loading zImage to the 0x2000000 and rd.bin to 0x2880000 is correct procedure?

That do not look right. Your envs do not use zImage. The bootcmd uses uImage.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I tried almost all FWs available to no luck. Same error always... :-/ I used bootm. So should I use bootz instead? And mem addresses I put it in are correct? Bootz returns "Bad Linux ARM zImage magic! error"...



Edited 3 time(s). Last edit at 01/15/2020 01:25PM by jean.

Jean,

Please post the link to Synology tarball and instruction page.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

FW 6.1.5. source is here:
https://archive.synology.com/download/DSM/release/6.1.5/15254/DSM_DS216j_15254.pat
https://archive.synology.com/download/DSM/release/6.1.5/15254/DSM_DS216j_15254.pat.md5
As for the instruction page, I couldn't find any instructions about installing FW using the serial console, Synology has dedicated app, which is able to work only if ethernet port is fully established on the NAS.

I wonder if ATAGS is not required because FDT (DTB) missing or corrupted? I tried to use fdt addr but when running fdt header after, putty got stuck.

jean,

> As for the instruction page, I couldn't find any
> instructions about installing FW using the serial
> console, Synology has dedicated app, which is able
> to work only if ethernet port is fully established
> on the NAS.

If you have a log of what you did to attemp the recovery, please post here. If not, try again and capture the log.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Jean,

Have you try this:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS

Quote

3. Mode 2: Reset Synology NAS and re-install the operating system DSM

Note: connect the Ethernet cable to the network, and serial console running, while you are doing this reset.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 01/16/2020 08:32AM by bodhi.

I finally succeeded in loading kernel, but boot ends up by incorrect login although I use correct credentials... :-/
Bootlog is attached.



Edited 1 time(s). Last edit at 01/16/2020 11:57AM by jean.

Cool! It looks pretty good.

You can try to bring up the box local web page to see if you can reset the admin password.

BTW, you should post the entire serial log here (everything on the putty tetminal until the login prompt) so others can see exactly what you did when they look for solution for the same situation.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 01/17/2020 02:11AM by bodhi.

Once I will finish this torture, I wil try to describe what I did, now I proceeded further somehow getting to the stage when DS is up, accessible, installed but blue LED still flashing and NAS unable to reset (by little hole in the back) or restart (only HDD goes down and then serial console shows reboot failed - system halted) the system by the ordinary way. I used mem test included in the Synology assistant which reached twice 99,91% when got stuck. Is this because defective flash or because unable to restart? I checked the memtest log which shows 2020-01-17T11:40:51+01:00 DiskStation findhostd: util_fhost.c:1351 Memtest passed! How should I understand it and what should I try next? I found the the imls and flinfo commands to be useful for further flash related investigations but unfortunately not available in the synology u-boot package. Is there any way to add it? Thanks for any help or suggestion. I attach the reboot log from serial console. BR

jean,

Your previous boot log showed eveything looks good. There is no need to run memtest.

Now, just recycle the power and eveything will be fine.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

There had to be something corrupted in the bootloader because when I put the testing HDD out, started the box, then put HDD in the another slot and booted again, u-boot started to push bad CRC message so saveenv and reset had it done in the final.



Edited 1 time(s). Last edit at 01/21/2020 05:09AM by jean.

jean,

> There had to be something corrupted in the
> bootloader because when I put the testing HDD out,
> started the box, then put HDD in the another slot
> and booted again, u-boot started to push bad CRC
> message so saveenv and reset had it done in the
> final.

That's a pretty strange behavior. That CRC error usually has nothing to do with the HDD if the HDD does not contain the rootfs.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I have found info that this message might pop-up when accessing an uninitialized on-disk u-boot environment happens. And it is the fact that my box loads also from the HDDs during the start.

@jean: I'm very curious how you fixed the problem. I have a DS216 with similar problems. Can you explain?

Marvell>> tftpboot 0x2000000 zImage
Using egiga0 device
TFTP from server 192.168.0.20; our IP address is 192.168.0.10
Filename 'zImage'.
Load address: 0x2000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############################################
         3.5 MiB/s
done
Bytes transferred = 2558520 (270a38 hex)
Marvell>> tftpboot 0x2500000 rd.bin
Using egiga0 device
TFTP from server 192.168.0.20; our IP address is 192.168.0.10
Filename 'rd.bin'.
Load address: 0x2500000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############################################################
         3.7 MiB/s
done
Bytes transferred = 3630333 (3764fd hex)
Marvell>> bootm 0x02000000 0x02500000
## Booting kernel from Legacy Image at 02000000 ...
   Image Name:   Linux-3.10.105
   Created:      2018-10-26  10:32:58 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2558456 Bytes = 2.4 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 02500000 ...
   Image Name:   synology_armada38x_ds216 23824
   Created:      2018-10-26  14:03:20 UTC
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    3630269 Bytes = 3.5 MiB
   Load Address: 08000000
   Entry Point:  08000000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
no ATAGS support: can't continue

Thanks in advance.

Little progress was made today.

It seems that a thirth argument for the bootm command is needed to load the "Flattened Device Tree". So i did:

bootm 0x02000000 0x03000000 0x01000000

Now i'm seeing this error:

ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree

I found a .dtb here: http://ftp.debian.org/debian//dists/stable/main/installer-armhf/current/images/device-tree/armada-385-synology-ds116.dtb
and use tftpboot write it to 0x01000000

Now it stuck at: Uncompressing Linux... done, booting the kernel.

@bodhi: Can you point me to the right direction to solve this?

Marvell>> tftpboot 0x3000000 rd.bin
Using egiga0 device
TFTP from server 192.168.0.20; our IP address is 192.168.0.11
Filename 'rd.bin'.
Load address: 0x3000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############################################################
         4.7 MiB/s
done
Bytes transferred = 3636367 (377c8f hex)
Marvell>> tftpboot 0x2000000 zImage
Using egiga0 device
TFTP from server 192.168.0.20; our IP address is 192.168.0.11
Filename 'zImage'.
Load address: 0x2000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ##############################################
         4.8 MiB/s
done
Bytes transferred = 2563072 (271c00 hex)
Marvell>> tftpboot 0x1000000 armada-385-synology-ds116.dtb
Using egiga0 device
TFTP from server 192.168.0.20; our IP address is 192.168.0.11
Filename 'armada-385-synology-ds116.dtb'.
Load address: 0x1000000
Loading: ####
         2.8 MiB/s
done
Bytes transferred = 17345 (43c1 hex)
Marvell>> bootm 0x02000000 0x03000000 0x01000000
## Booting kernel from Legacy Image at 02000000 ...
   Image Name:   Linux-3.10.105
   Created:      2019-05-09  18:48:51 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2563008 Bytes = 2.4 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 03000000 ...
   Image Name:   synology_armada38x_ds216 24922
   Created:      2019-05-09  21:25:11 UTC
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    3636303 Bytes = 3.5 MiB
   Load Address: 08000000
   Entry Point:  08000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 01000000
   Booting using the fdt blob at 0x01000000
   Loading Kernel Image ... OK
OK
   Loading Ramdisk to 1f6b2000, end 1fa29c4f ... OK
   Loading Device Tree to 00ff8000, end 00fff3c0 ... OK

   Skipping Device Tree update ('fdt_skip_update' = yes)

Limit DDR size at 3GB due to power of 2 requirement of Address decoding

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

Roland,

This kernel Linux-3.10.105 is too old, it does not look like it has device tree capability.

My release kernel should work if the DS116 is the same as DS216j.

The release thread:

https://forum.doozan.com/read.php?2,32146

Quote

Latest released kernel: linux-5.5.3-mvebu-tld-1-bodhi.tar.bz2 (17 Feb 2020)
Latest released rootfs: Debian-5.2.9-mvebu-tld-1-rootfs-bodhi.tar.bz2 (10 Sep 2019)

So you can either use a new rootfs (if USB is supported by u-boot): Debian-5.2.9-mvebu-tld-1-rootfs-bodhi.tar.bz2. Or just copy the files from the rootfs to your tftproot and load them from there.

Before you can copy uImage, you need to do step 3 in the instruction:

Quote

3. Create uImage with DTB appended. I am using the Zyxel NAS326 as an example here. Please replace armada-380-zyxel-nas326.dtb below with the appropriate DTB name for your box.

cd /media/sdb1/boot
cp -a zImage-5.2.9-mvebu-tld-1 zImage.fdt
cat dts/armada-380-zyxel-nas326.dtb >> zImage.fdt
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-5.2.9-mvebu-tld-1 -d zImage.fdt uImage

So the command needs adjustment is bold above:

cat dts/armada-385-synology-ds116.dtb >> zImage.fdt

And then boot with uImage and uInitrd only. Don't boot with separated DTB (this u-boot messes up the device tree blob when the kernel size grows larger).

bootm 0x02000000 0x03000000

And then after it booted and running, if you really want to use mainline kernel, then install the latest that Debian buster provides, and recreate uImage like above.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 03/12/2020 12:01AM by bodhi.

Bodhi,

Thank you for your extensive reply. I'm sorry but i forgot to mention that i want to restore the original Synology firmware just like the topic starter. Also i have a DS216 instead of a DS216j.

Quote

This kernel Linux-3.10.105 is too old, it does not look like it has device tree capability.

I extracted "zImage" from the newest Synology .pat file, so there is no newer kernel.

Quote

Don't boot with separated DTB (this u-boot messes up the device tree blob when the kernel size grows larger).

Is this still relevant with the Synology firmware?

Is there a way to get some debug info to find why the kernel boot hangs?

Roland,

> Thank you for your extensive reply. I'm sorry but
> i forgot to mention that i want to restore the
> original Synology firmware just like the topic
> starter. Also i have a DS216 instead of a DS216j.

Ah.

>

Quote

Don't boot with separated DTB (this u-boot
> messes up the device tree blob when the kernel
> size grows larger).

>
> Is this still relevant with the Synology
> firmware?

I don't know anything about this box stock FW, so can't answer that!


> Is there a way to get some debug info to find why
> the kernel boot hangs?

The best way is to connect serial console. Without serial console, we will just be guessing and end up spending too much time troubleshooting.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Bodhi,

Quote

The best way is to connect serial console. Without serial console, we will just be guessing and end up spending too much time troubleshooting.

I already had a serial console. All data i posted before was from that console.

In my search to get more debug output i stumbled opon this form.
http://e2e.ti.com/support/processors/f/791/t/541951?how-to-debug-non-booting-linux-kernel-on-custom-am57xx-board
This quote was interesting.

Quote

Note that Starting Kernel is the last u-boot message. It usually indicates that you're using a wrong dtb, or there is something fundamentally wrong in the dtb you use.

So i searched for another dtb file and i found one on this forum. :-)

https://forum.doozan.com/file.php?2,file=3104,filename=dtbdump_3.dtb

With that dtb file it boots.


Thanks a lot for your time and help to recover my NAS!

With kind regards
Roland



Edited 1 time(s). Last edit at 03/15/2020 10:51AM by Roland.

Roland,


> So i searched for another dtb file and i found one
> on this forum. :-)

> https://forum.doozan.com/file.php?2,file=3104,filename=dtbdump_3.dtb
>
> With that dtb file it boots.

Cool!

Please post the link to that specific post where the file was attached.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)