Debian on Synology RS816 (Armada 385)

Richard,

Yup. That's the Armada XP.

Try this close cousin u-boot that I rebbuilt for Thecus N2350:

https://forum.doozan.com/read.php?2,50829,73124#msg-73124

This is the uart built u-boot:

u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi

>Try this close cousin u-boot that I rebbuilt for Thecus N2350:

Its loading, went to 9% quickly now slowed down to a crawl but it is still going, fingers are crossed! Will update when its done which at current rate will be 25 mins unless it speeds up again.

UPDATE:
After an eternity it was finally all loaded but unless I'm meant to do something at the end point it a no go.

[richard@Linux thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi]$ kwboot -t -B 115200 /dev/ttyUSB0 -b u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin -p
Sending boot message. Please reboot the target...-
Sending boot image...
  0 % [......................................................................]
  0 % [......................................................................]
  1 % [......................................................................]
  2 % [......................................................................]
  And so on>>>
  98 % [......................................................................]
  99 % [.................]
[Type Ctrl-\ + c to quit]

I think my next step is to just keep re-configuring the db-88f6820-gp config and hope I get lucky.



Edited 1 time(s). Last edit at 05/31/2019 04:33AM by richjn.

Richard,

That's a good step forward.

Instead of trying to tweak config file. Try using the script download-serial.sh (from the same tarball).

You would run it as:

./download-serial.sh /dev/ttyUSB0  u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin

Once you see

Transfer complete

You are back to the shell.

Run the serial console (picocom/minicom/screen/putty) like you do normally, and hit <return> few times. If you see the u-boot prompt, then it's unbricked!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 05/31/2019 07:18AM by bodhi.

@bodhi
Looks like send-stop-pattern was compiled on something other than x86 ? or with a specific processor specified.

[richard@Linux thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi]$ ./download-serial.sh /dev/ttyUSB0  u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin
Now reset or power cycle your Armada 38x board and then press enter
./download-serial.sh: line 14: ./send-stop-pattern: cannot execute binary file: Exec format error

I did try to compile the send-stop-pattern.c with

gcc send-stop-pattern.c -o test

But it spewed a whole bunch of garbage and vi send-stop-patern.c just looks like a binary itself ?

Richard,

Let me take look.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi

I'm very happy to try anything to unbrick the box so please don't read the following as me wanting to abandon other methods.

I have been wondering if we are looking in the wrong direction.
We already have a backup of a functioning u-boot and even if there was a procedure issue in the way it was collected we have another working box to try again. At first we thought it was a kwboot issue preventing us loading it but we now know that kwboot works fine. The issue may be more to do with kwboot's -p option not patching the header correctly ?

I've not given this much thought but could we strip the header from a working kwboot .kwb file then strip and replace the header from our stock u-boot file with dd and cat ?

Richard,

> I'm very happy to try anything to unbrick the box
> so please don't read the following as me wanting
> to abandon other methods.

Not at all!

>
> I have been wondering if we are looking in the
> wrong direction.
> We already have a backup of a functioning u-boot
> and even if there was a procedure issue in the way
> it was collected we have another working box to
> try again.

> At first we thought it was a kwboot
> issue preventing us loading it but we now know
> that kwboot works fine. The issue may be more to
> do with kwboot's -p option not patching the header
> correctly ?

No, it is kwboot or other similar binaries vs Armada SoC BootROM. I've have been through this process countless times :)

Let me explain. With the old Marvell Kirkwood SoCs, it is quite easy for kwboot to handshake with the old BootROM version. In the last few years, Marvell Armada SoCs became the goto SoCs for NAS manufacturers, so a lot of people (including the most experiences such as Tom Rini, Jason Cooper, Kevin Smith,...) have tried to run kwboot to boot with UART as the everyday tool to test new u-boot images.

What we found is the new BootROM version used in the Armada SoCs is quite picky (but we cannot see the code which is Marvell's proprietary). So a lot of tries and true, the -a option was added to kwboot for Armada XP (it has been tested to work).

And also, -q -s were added to facilitate the brute-force testing. Because nobody knows the correct delay for request and response when kwboot handshake with the BootROM in a particular box :)

The download_serial script is also a brute-force tool created by one of u-boot tinkerers out there (sorry I can't remember the person name). It has been proven to work on the Thecus N2350 by Manuel and I in the working thread (while kwboot did not, because we could not figure out the timing either!).

I've just looked at the send-stop-pattern.c in the tarball, and realized that I had inadvertenly clobber it with the binary :) so I will reupload it here.

-----------
edit: typo

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 3 time(s). Last edit at 05/31/2019 08:40PM by bodhi.

@bodhi

Thanks for taking the time to explain, I understand now but am still bewildered why it accepts some files 100% of the time and others not at all but I guess if that's the reality we have to live with it.

Would we be able to use the download_serial script brute force method to send our back up stock u-boot ?

Richard,

> Thanks for taking the time to explain, I
> understand now but am still bewildered why it
> accepts some files 100% of the time and others not
> at all but I guess if that's the reality we have
> to live with it.
>

There are handshakes going on during loading and also right after the image has been loaded. Basically, it is Xmodem protocol that requires constant handshake after a chunk of bytes was sent and received.

> Would we be able to use the download_serial script
> brute force method to send our back up stock
> u-boot ?

I surely hope so, because it is the same BootROM version. The only differences are in hardware: the board and serial port construction.

I've attched the send-stop-pattern.c here. So you can compile it in your box where it runs.

The first time you run it might not be successful, so post the log here and I'll continue.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi

Bit of a holdup compiling send-stop-pattern

[richard@Linux thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi]$ gcc send-stop-pattern.c -o send-stop-pattern
send-stop-pattern.c: In function ‘main’:
send-stop-pattern.c:25:2: warning: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
   25 |  sleep(2);
      |  ^~~~~
/usr/bin/ld: /tmp/ccLYyHWG.o: in function `main':
send-stop-pattern.c:(.text+0x9): undefined reference to `initscr'
/usr/bin/ld: send-stop-pattern.c:(.text+0x10): undefined reference to `stdscr'
/usr/bin/ld: send-stop-pattern.c:(.text+0x1d): undefined reference to `wtimeout'
/usr/bin/ld: send-stop-pattern.c:(.text+0xc3): undefined reference to `stdscr'
/usr/bin/ld: send-stop-pattern.c:(.text+0xcb): undefined reference to `wgetch'
collect2: error: ld returned 1 exit status

This looks like an ncurses issue to me, as per Arch Linux there are no *-dev splits and the full packages are installed on my system, I think I need to take a look at my ncurses build though.

I'm about to run out of time because I have a family weekend to attend which will see me absent till Monday.

I will keep going till my wife drags me away but as Arnie would say "I'll be back" :)

Richard,

My bad. How about trying the orginal download-serial.sh. Rename the attched original script to download-serial.sh. And run it to see if the embedded routine will compile (it uses the ncurse option). So in this version we dont need the separate c file.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

And have fun :) see you on Monday.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
A bit of delay before I can check download-serial.sh due to returning this morning to find the rs816 we have already converted running in degraded mode after a disk failure, I do keep a spare disk for each array here but I had wanted to upgrade the disks to 4TB at some point anyway so decided to do a full re-install instead.

In my rush I just went ahead and formatted to gpt and it wasn't till the end of the install that I remembered what I thought was the restriction of Linux raid autodetect namely mbr and raid meta 0.9. I had done the meta 0.9 and so just decided to reboot and see what happened,to my surprise it booted no problem. Maybe raid autodetect has always worked with gpt and I was mistaken about the mbr limitation or the code was updated at some point.

I know uImage & uInitrd in raw disk sector is likely to be your recommendation for this box but at least we now know that people who want to go with zImage in flash with large gpt HDD's can do so.

As soon as the raids have synced and I have it pushed back to production I will get back to the bricked one maybe later today but more likely tomorrow.

EDIT
Doh! a bit hung over this morning :)
Linux raid autodetect works for gpt because we are not booting from it directly, it does still require raid meta 0.9 though.



Edited 1 time(s). Last edit at 06/02/2019 09:08PM by richjn.

Richard,

> Doh! a bit hung over this morning :)
> Linux raid autodetect works for gpt because we are
> not booting from it directly, it does still
> require raid meta 0.9 though.

:)) OK.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
All arrays synced and rs816 back in rack.

The download-serial.sh.orig works fine.

u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin

[richard@Linux thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi]$ ./download-serial.sh /dev/ttyUSB0  u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin
send-stop-pattern.c: In function ‘main’:
send-stop-pattern.c:24:2: warning: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
   24 |  sleep(2);
      |  ^~~~~
Now reset or power cycle your Armada 38x board and then press enter
Sleeping
Out of sleep
Got something 0
Sleeping
Out of sleep
Got something 21
Got NACK (0x15) character
The board should be in serial downloader now. Sending the binary file 
Sending u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin, 7506 blocks: Give your local XMODEM receive command now.
Xmodem sectors/kbytes sent: 752/94kRetry 0: Timeout on sector ACK
Retry 0: Timeout on sector ACK

Chokes at 94k have retired 3 times with same result

rs816.stock.uboot.kwb

[richard@Linux thecus-n2350-Marvell-2015_T1.0p18-tld-4-bodhi]$ ./download-serial.sh /dev/ttyUSB0 rs816.stock.uboot.kwb
send-stop-pattern.c: In function ‘main’:
send-stop-pattern.c:24:2: warning: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
   24 |  sleep(2);
      |  ^~~~~
Now reset or power cycle your Armada 38x board and then press enter
Sleeping
Out of sleep
Got something 0
Sleeping
Out of sleep
Got something 21
Got NACK (0x15) character
The board should be in serial downloader now. Sending the binary file 
Sending rs816.stock.uboot.kwb, 7168 blocks: Give your local XMODEM receive command now.
Xmodem sectors/kbytes sent:   0/ 0kRetry 0: Got 1b for sector ACK
Retry 0: Timeout on sector ACK

Dies immediately

Richard,


> rs816.stock.uboot.kwb

This would not be possible. The image in flash is not usable with download-serial. It has be compiled with UART option. The u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin was.

> The download-serial.sh.orig works fine.
> Xmodem sectors/kbytes sent: 752/94kRetry 0:
> Timeout on sector ACK
> Retry 0: Timeout on sector ACK
> [/code]
>
> Chokes at 94k have retired 3 times with same
> result

Same behavior as the NAS326. The missed handshake caused it to timeout.

This is bad news :) without a rescue mechanism, we can't write the installation from inside stock OS. We would need to risk bricking the box to verify such installation procedure works.

Did we try the kwboot with fall back (option -f)? Let's try the patched kwboot I've just compiled and see if we have more luck.

./kwboot -t -B 115200 /dev/ttyUSB1 -b u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin  -s 0 -q 1 -f

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
The fall back patch was one I had already applied to my kwboot which regrettably seems to make no difference on this box. I had also tried multiple variations of -s & -q.

I am re-running with

kwboot -t -B 115200 /dev/ttyUSB0 -b u-boot-a38x-2015T1_p18_Thecus-tld-4-spi-uart.bin  -s 0 -q 1 -f

The upload is taking place in a similar manner to last time where the first 9% happens quickly then we slow to a crawl with the balance taking well over an hour.

I will let it run to see if it ends differently this time but last time we ended at

98 % [......................................................................]
  99 % [.................]
[Type Ctrl-\ + c to quit]

with no further output, ending the process and trying to connect via serial does not work.

Just as a reminder the dreamplug based u-boot-spl.kwb trial did the same thing so I'm not sure we can read anything into the process seemingly completing.

I will try recompiling u-boot with a few different options based on db-88f6820-gp and try uploading with download-serial.sh a bit of a long shot I know but I may get lucky.

EDIT
I'm going to pickup a 2 port rs232 pcie adapter today, I was thinking of doing it anyway but I have read that kwboot can be a bit picky about usb serial adapters so now sounds like a good idea to grab one.



Edited 1 time(s). Last edit at 06/03/2019 04:55PM by richjn.

@bodhi

kwboot test from above ended the same way with no further output.

98 % [......................................................................]
  99 % [........................]
[Type Ctrl-\ + c to quit]

I fetched and installed my new serial pcie card and on the face of it things are worse, download-serial.sh doesn't get a nack and kwboot is even more tricky to start.

I will get on with trying some u-boot builds and hope I get lucky.

@bodhi
I'm parking the rescue for now but will pick it back up at some point, I may write to Synology and ask them if they are prepared to send me a rescue uart image of their own, I presume they must have one.

Even though I'm down to one rs816 which is in production I owe its successful conversion to this forum and in particular to you for all of your help and guidance getting it done. Hence I'm still prepared to make it available for conducting a howto for the benefit of others which to be comprehensive would involve reverting it to fully stock and I'm ok with doing that and accept the risks involved. Luckily I have spare disks so from my perspective the workload is reduced because I wont need to wipe my data / configuration. In terms of risks to others following the tutorial without the safety net of kwboot I feel the risks are acceptable because the process doesn't touch u-boot in /dev/mtd0 (provided they don't typo on the actual flashing)

I will need a days notice to make the box available for the tutorial and I think it would be good if we could agree a day where we both have plenty of free time to do the actual work in a day. I know you are waiting on a Buster release before you make a new rootfs so sometime shortly after that sounds sensible.

On a practical note I think your current linux-5.1.2-mvebu-tld-1 contains armada-385-synology-rs8166-v12.dts in your patch which will only bring up one NIC. I have re-attached V14 here which resolves that.

Ok Richard!

I’m going to release linux-5.1.6-mvebu-tld-1 with v14 DTS.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
If not too late maybe go with linux-5.1.7 it has a an extra nic fix in it

commit 67a3fa2fd0e0df42af99aa0174672c5ee4588b3a
Author: Jisheng Zhang <Jisheng.Zhang@synaptics.com>
Date:   Mon May 27 11:04:17 2019 +0000

    net: mvneta: Fix err code path of probe
    
    [ Upstream commit d484e06e25ebb937d841dac02ac1fe76ec7d4ddd ]
    
    Fix below issues in err code path of probe:
    1. we don't need to unregister_netdev() because the netdev isn't
    registered.
    2. when register_netdev() fails, we also need to destroy bm pool for
    HWBM case.
    
    Fixes: dc35a10f68d3 ("net: mvneta: bm: add support for hardware buffer management")
    Signed-off-by: Jisheng Zhang <Jisheng.Zhang@synaptics.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Richard,

Even though that patch is not really relevant, I'll pick it up anyway! So 5.1.7 that will be.

Chasing GKH really hard :) he is very quick in releasing stable kernels.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
> Chasing GKH really hard

I do chase on my main box :) but will not try with my rs816 due to zImage being on flash with consequent wear concerns.

I'm currently on linux-5.17 with the kernel stripped down to only the bits I need but even with this minimal setup zImage with uInitrd in flash would not be possible. From here I will update on a 6 monthly schedule or before if a relevant to my setup security fix lands.

-rw-r--r-- 1 root root 4.9M Jun  5 18:55 uInitrd
-rw-r--r-- 1 root root 3.7M Jun  5 18:38 zImage

I have contacted Synology (via my account) and a conversation is taking place regarding allowing me access to a suitable uart image. My expectation of success is very low.

Richard,

> I do chase on my main box :) but will not try with
> my rs816 due to zImage being on flash with
> consequent wear concerns.
>
> I'm currently on linux-5.17 with the kernel
> stripped down to only the bits I need but even
> with this minimal setup zImage with uInitrd in
> flash would not be possible. From here I will
> update on a 6 monthly schedule or before if a
> relevant to my setup security fix lands.

That's the problem with storing kernel files in flash. You don't want to update it too often. So there is an increase of security risk.

A lot of bugs become a visible security issue only after someone fixed the bugs, and then that particular chunk of code are under more scrutiny.

So how do we solve this? use a journal file system such as Ext3 or Ext4 for rootfs (where the kernel files are) and have the benefit of a self recovery rootfs after a power outage.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 06/07/2019 01:56AM by bodhi.

@bodhi
> So how do we solve this? use a journal file system such as Ext3 or Ext4 for rootfs (where the kernel files are)
Yes, for uInitrd / initrd.img but not for zImage / uImage where we are still stuck with flash or raw hdd sector for these. I think the choice between raw hdd sectors and flash is probably a personal preference thing but currently I favour flash although I readily accept the wear concerns with flash I just feel more comfortable with it.

Richard,

My poitn is whether with normal rootfs or raw sector, kernel files uImage/zImage and uInitrd will be on HDD and thus will be updated regularly without worrying about wear and tear factor.

> probably a personal
> preference thing
but currently I favour flash
> although I readily accept the wear concerns with
> flash I just feel more comfortable with it.

Sure. Flash is not a bad choice for one who builds own kernel regularly.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
Just a bit of an update on attempts to recover the bricked rs816

After a LOT of communications with Synology they sent me their boot images, I'm not 100% sure if they are even intended for uart booting because their developer told me they just take the ram chips out and re-flash them in a "flasher" they have for the job when they need to do one. Suffice to say I had no joy with the images using kwboot or download-serial.sh

I started looking around for units with the same board / cpu and it seems some of the Linksys routers from the WRX range are Armada 38x. I downloaded the u-boot recovery files from the WRT project for these and got close with WRT32X_uboot_2.0.9.bin.

[root@Linux Downloads]# kwboot -t /dev/ttyUSB0 -b WRT32X_uboot_2.0.9.bin -p
Sending boot message. Please reboot the target...-
Sending boot image...
  0 % [......................................................................]
  0 % [......................................................................]
  1 % [......................................................................]
  2 % [......................................................................]
  3 % [......................................................................]
  4 % [......................................................................]
  5 % [......................................................................]
  6 % [......................................................................]
  7 % [......................................................................]
  8 % [......................................................................]
  9 % [...................................

General initialization - Version: 1.0.0
AVS selection from EFUSE disabled (Skip reading EFUSE values)
mvSysEnvIsFlavourReduced: TWSI Read of 'flavor' failed
Detected Device ID 6820
High speed PHY - Version: 2.0

Init RD NAS topology Serdes Lane 3 is USB3
Serdes Lane 4 is SGMII
board SerDes lanes topology details:
 | Lane # | Speed|    Type     |
 ------------------------------|
 |   0    |  6   |  SATA0      |
 |   1    |  5   |  PCIe0      |
 |   2    |  6   |  SATA1      |
 |   3    |  5   |  USB3 HOST1 |
 |   4    |  5   |  PCIe1      |
 |   5    |  0   |  SGMII2     |
 -------------------------------
PCIe, Idx 0: detected no link
PCIe, Idx 1: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.46.0
mvSysEnvGetTopologyUpdateInfo: TWSI Read failed
DDR3 Training Sequence - Switching XBAR Window to FastPath Window 
DDR3 Training Sequence - Ended Successfully
Not detected suspend to RAM indication
...................................]
 10 % [......................................................................]
 11 % [......................................................................]
 11 % [......................................................................]
 12 % [......................................................................]

It then loads to 100% and gets stuck in the following loop till you kill it.

98 % [......................................................................]
 99 % [......................................................................]
 99 % [....]
[Type Ctrl-\ + c to quit]


U-Boot 2013.01 (May 18 2017 - 16:37:44) Marvell version: 2015_T1.QA.0p16

Boot version : v2.0.9

Board: RD-NAS-88F6820-DDR3
SoC:   MV88F6820 Rev A0
       running 2 CPUs
CPU:   ARM Cortex A9 MPCore (Rev 1) LE
       CPU 0
       CPU    @ 1866 [MHz]
       L2     @ 933 [MHz]
       TClock @ 200 [MHz]
       DDR3    @ 933 [MHz]
       DDR3 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM:  512 MiB
NAND:  NAND ECC mode is not defined!
0 MiB
MMC:   mv_sdh: 0
raise: Signal # 8 caught
*** Warning - readdevinfo() failed, using default devinfo

raise: Signal # 8 caught
*** Warning - readenv() failed, using default environment

NOTE
It only see's 512mb memory (1gb onboard)

Close but not close enough but its interesting that this one just loads 100% of the time apparently without timing issues etc.

It just feels like if I could interrupt it and manually enter the vars it might just work but it doesn't appear to accept any input whilst in the loop.

The hunt continues!

Richard,

> I started looking around for units with the same
> board / cpu and it seems some of the Linksys
> routers from the WRX range are Armada 38x. I
> downloaded the u-boot recovery files from the WRT
> project for these and got close with
> WRT32X_uboot_2.0.9.bin.


> U-Boot 2013.01 (May 18 2017 - 16:37:44) Marvell
> version: 2015_T1.QA.0p16
>
> Boot version : v2.0.9
>
> Board: RD-NAS-88F6820-DDR3
> SoC: MV88F6820 Rev A0
> running 2 CPUs
> CPU: ARM Cortex A9 MPCore (Rev 1) LE
> CPU 0
> CPU @ 1866 [MHz]
> L2 @ 933 [MHz]
> TClock @ 200 [MHz]
> DDR3 @ 933 [MHz]
> DDR3 32 Bit Width,FastPath Memory Access,
> DLB Enabled, ECC Disabled
> DRAM: 512 MiB
> NAND: NAND ECC mode is not defined!
> 0 MiB
> MMC: mv_sdh: 0

A big step in unbricking :)

The rest of the errors just mean that the Linksys WRT32X box does not have the same hardware as the RS816.

> It just feels like if I could interrupt it and
> manually enter the vars it might just work but it
> doesn't appear to accept any input whilst in the
> loop.

Please find the OpenWrt source code for this build: WRT32X_uboot_2.0.9.bin.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Quote

Close but not close enough but its interesting that this one just loads 100% of the time apparently without timing issues etc.

Yup. It is the same for the rest of the older u-boots. The Thecus N2350 behave the same way with older u-boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi

I think this is the correct fork HERE

I will check it out and update if I have the wrong one (lots of interruptions today making it hard to focus)