[BETA] Rescue System V4, using a custom LEDE firmware

After some work and many Thanks Obama since LEDE devs don't cough up their secrets even after waterboarding, I present you Rescue System V4, with support for all kirkwoods (theoretically, I tested only nsa310)!

It's a custom-built LEDE/OpenWRT firmware.
it needs only 2MiB of space for the kernel and 11MiB of space for the rootfs on the NAND (will use more if available and allows writing on it).

But it is packed of goodies:

root@lede:/# ls /bin 
ash              egrep            mknod            sed
board_detect     false            mktemp           sh
busybox          fgrep            mount            sleep
cat              fsync            mv               sync
chgrp            grep             netmsg           tar
chmod            gunzip           netstat          touch
chown            gzip             nice             true
config_generate  ipcalc.sh        pidof            ubus
cp               kill             ping             uclient-fetch
date             ln               ping6            umount
dd               lock             ps               uname
df               login            pwd              vi
dmesg            ls               rm               wget
echo             mkdir            rmdir            zcat
root@lede:/# ls /sbin 
askfirst           init               mtd                sysupgrade
block              ip                 netifd             ubusd
cgdisk             jffs2mark          pivot_root         uci
devstatus          jffs2reset         poweroff           udevtrigger
firstboot          kmodloader         procd              udhcpc
gdisk              led.sh             reboot             upgraded
halt               logd               reload_config      urandom_seed
hotplug-call       logread            route              validate_data
hwclock            mdadm              sgdisk             wifi
ifconfig           mkfs.xfs           start-stop-daemon  xfs_db
ifdown             mkswap             swconfig           xfs_growfs
ifstatus           mmc                switch_root        xfs_repair
ifup               mount_root         sysctl
root@lede:/# ls /usr/bin 
[                  du                 mkfifo             tar
[[                 env                mkfs.btrfs         tee
ar                 expr               mount              test
attr               file               mountpoint         time
awk                find               nano               top
basename           findmnt            nc                 tr
bc                 flock              ncdu               traceroute
btrfs              free               nslookup           traceroute6
btrfs-debug-tree   fsck.btrfs         passwd             umount
btrfs-find-root    funzip             perl               uniq
btrfs-image        getfacl            perl5.22.1         unlzma
btrfs-map-logical  getfattr           pgrep              unxz
btrfs-show-super   getrandom          pkg-config         unzip
btrfs-zero-log     head               printf             unzipsfx
btrfsck            hexdump            readlink           uptime
btrfstune          id                 reset              usign
bunzip2            jshn               rsync              uuidgen
bzcat              jsonfilter         scp                vim
bzip2              killall            seq                wc
chacl              ldd                setfacl            wget
chattr             less               setfattr           wget-ssl
chroot             lftp               sha1sum            which
clear              logger             sha256sum          xargs
cmp                lsattr             signify            xz
crontab            lsblk              sort               xzcat
cut                lsof               spi-config         yes
dbclient           lzcat              spi-pipe           zipgrep
dc                 lzma               ssh                zipinfo
dirname            mc                 strings
dmesg              mcedit             systool
dropbearkey        md5sum             tail
root@lede:/# ls /usr/sbin 
atftp              fsck.ext4          mkfs.fat           swaplabel
atftpd             fsck.f2fs          mkfs.msdos         swapoff
badblocks          fsck.fat           mkfs.vfat          swapon
blkid              fsck.msdos         mkswap             tune2fs
brctl              fsck.vfat          modinfo            ubiattach
chroot             fw_printenv        modprobe           ubiblock
crond              fw_setenv          mtdinfo            ubicrc32
debootstrap        insmod             nanddump           ubidetach
dnsmasq            ip6tables          nandtest           ubiformat
dosfsck            ip6tables-restore  nandwrite          ubimkvol
dropbear           ip6tables-save     ntpd               ubinfo
dump.f2fs          iptables           ntpd-hotplug       ubinize
e2fsck             iptables-restore   odhcp6c            ubirename
f2fstat            iptables-save      odhcpd             ubirmvol
fatlabel           lsmod              odhcpd-update      ubirsvol
fdisk              mke2fs             resize2fs          ubiupdatevol
fibmap.f2fs        mkfs.ext2          rmmod              wipefs
findfs             mkfs.ext3          sensors            xtables-multi
fsck.ext2          mkfs.ext4          sensors-detect
fsck.ext3          mkfs.f2fs          smartctl

It supports ext2-3-4, btrfs, f2fs, FAT32, XFS (also ubifs and jffs2 and squashfs but these aren't usually used by Debian). (theoretically, not all have been tested)

It has fdisk for MBR partitioning and gdisk/sgdisk/cgdisk for GPT partitioning.

It also has mdadm and is able to read raid arrays (theoretically, not tested), and debootstrap/pkg-config so it can (theoretically, not tested) generate a debian rootfs on its own.

Since this is a rescue system, the opkg (expansion) functionality has been removed.

If you think there are crucial tools or drivers I've missed, please post and I'm updating the image. Serial connection is recommended as I might have not included the ethernet driver for your device (I should have).


I will add this to the installer script Soon(TM) so that you can automatically install a powerful recovery on your device's NAND instead of wasting it with a dead firmware the new uboot cannot use.


Installation (for testing purposes, so I'm not providing easy info, must use nand_erase then nandwrite):
-write the uimage of your device in the kernel partition on flash.
-write the rootfs image in a suitably large nand partition.
-change the name of the mtdpart where you flash the rootfs to "rootfs".
-write the envs to boot this recovery system in your uboot.

I'm providing my envs as examples (nsa310):

These are the bootargs

recovery_bootargs=setenv bootargs console=ttyS0,115200 rootdelay=10 $mtdparts_lede

These are the bootcommands

bootcmd_recovery=run recovery_bootargs; nand read 0x800000 nand0,7 ; bootm 0x800000
bootcmd=run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec; run bootcmd_recovery;

here the "nand0" is the NAND chip onboard, and "nand0,7" is like saying mtd7, or whatever hex offsets the partition sits in and none really feels like figuring out on this side.
Adjust that to fit the partition you flash the kernel image into.

This is mtdparts_lede, the last one is the rootfs (mtd8):

mtdparts_lede=mtdparts=orion_nand:0x100000(uboot),0x80000(uboot_env),0x80000(key_store),0x80000(info),0xA00000(etc),0xA00000(nope_kernel_1),0x2FC0000(nope_rootfs1),0xA00000(kernel),0x2FC0000(rootfs)

usage:
No webinterface is available, only way to connect is through ssh. user is "root", password is empty or "".

Let the device settle for 5 minutes after booting and trying to connect with ssh (if you don't have serial connection), or if you see serial output wait until there is a long message that says "jffs2 created successully with 0 errors" before pressing return and activating the serial console.
This happens only on first boot as it needs to generate the read-write partition.


for the curious ones:

If someone is wondering why this hacking around the mtdparts, it's because LEDE firmware's kernel looks for partitions called "rootfs" or "rootfs_data" and acts on them automatically.

In our case it splits on its own the "rootfs" partition in a first part with a squashfs filesystem and the rest of the space becomes another mtd partition where it creates a jffs2 (read-write) filesystem that then mounts as overlay on the squashfs (so you can write anywhere, and you can easily reset to defaults by erasing the jffs2 partition).

This system allows them to hard-code mtdparts in the dtb file so their kernel will ignore what uboot gives them, and their rootfs will be able to occupy the whole flash space instead of using only a pre-made mtd partition and wasting most of the NAND.
For this recovery system and for kirkwoods in general it's not that necessary, so I just rename an existing mtd partition in the kernel command line I send from uboot and call it a day.


Download link:

Here is the download package, https://www.dropbox.com/s/firggx4vzl6v4a4/kirkwood_recoveryV4-beta.7z?dl=0
I provide also "sources" and some instructions in the package so you can build the same on your own if you grab the LEDE sources from their git. (for the sake of openness).

EDIT: Now with fixed download link.



Edited 2 time(s). Last edit at 04/15/2017 09:13AM by bobafetthotmail.

> After some work and many Thanks Obama since LEDE
> devs don't cough up their secrets even after
> waterboarding,

:))


> This system allows them to hard-code mtdparts in
> the dtb file so their kernel will ignore what
> uboot gives them, and their rootfs will be able to
> occupy the whole flash space instead of using only
> a pre-made mtd partition and wasting most of the
> NAND.

> For this recovery system and for kirkwoods in
> general it's not that necessary, so I just rename
> an existing mtd partition in the kernel command
> line I send from uboot and call it a day.

Yes. This is the way to do it. People hard code the mtdparts in DTS because they think that should be default. But we are customizing u-boot and rootfs, that would be a limitation.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
>
> Yes. This is the way to do it. People hard code
> the mtdparts in DTS because they think that
> should be default. But we are customizing u-boot
> and rootfs, that would be a limitation.

Well, I didn't do that because I'm not going to modify like 20 dts with custom mtdparts figuring out myself the hex offsets of things (assuming I know what are the stock mtdparts at all, and I don't, and I don't feel like wasting another week tracking down that info) while I have very easy access to bootloader settings through the script, and the bootloader is friendly and powerful as u-boot usually is.

LEDE does not usually have access to bootloader configuration when installing through the stock firmware, and even if they do they prefer not risking a softbrick.

I can see why stock firmwares would do the same. In case they decide to change on-flash partitions they only need to change dts and push a firmware update, without touching the bootloader. As long as the kernel image is in the same place (usually just after uboot and wifi calibration firmware), it will still boot fine.

> I can see why stock firmwares would do the same.
> In case they decide to change on-flash partitions
> they only need to change dts and push a firmware
> update, without touching the bootloader. As long
> as the kernel image is in the same place (usually
> just after uboot and wifi calibration firmware),
> it will still boot fine.

Right. They only deal with one box. So they just need to replace uImage.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bobaffetthotmail,

Outstanding project. Thank you for highlighting this fantastic distribution in this project. I look forward to installing, using and exploring.

Paul

Quick update:

I've added the "announce" LEDE package (a simple zeroconf server), this allows people to ssh into the recovery by hostname:
ssh root@lede.local
If their firewall allows that to happen (on Windows/OSX should work fine as both are heavy users of similar systems), my OpenSUSE's firewall required me to set internal interfaces first before It let me connect.

in my own testing version I've also added a new user with root privileges, "recovery". This user is special as its default shell is a script and not a shell.

This allows me to integrate in the recovery a script to deal with the most common use-cases (filesystem check, update uboot/recovery, make a new rootfs with latest rootfs/kernel, maybe make a RAID setup too).

root account is of course preserved because I'd like to not lock advanced users out of the recovery.

bobafetthotmail Wrote:
-------------------------------------------------------
> It's a custom-built LEDE/OpenWRT firmware.
> it needs only 2MiB of space for the kernel and
> 11MiB of space for the rootfs on the NAND (will
> use more if available and allows writing on it).
>
I suppose this is based on the new image generation that abandons the uImage kernel file.

habibie Wrote:
-------------------------------------------------------
> bobafetthotmail Wrote:
> --------------------------------------------------
> -----
> > It's a custom-built LEDE/OpenWRT firmware.
> > it needs only 2MiB of space for the kernel and
> > 11MiB of space for the rootfs on the NAND (will
> > use more if available and allows writing on
> it).
> >
> I suppose this is based on the new image
> generation that abandons the uImage kernel file.

nope. Kirkwoods are still using uImages like most other devices.

If your devices (oxynas I suppose) are still using wrong boot file for the bootloader they have (so they can't boot), open a bug ticket in the bug tracker https://bugs.lede-project.org/

perfect!! long awaited nanddump nandwrite

file does not exist.please upload again.

Can someone re-upload this please?!

Fixed link.

Dropbox updated the way they let their users share stuff so old links became invalid.

I got it to work on my pogoplug V4. Many thanks!

This is what I did:
1)write uimage to kernel partition
2)write lede-kirkwood-generic-squashfs.img to rootfs partition
3)change the env

Before I did this, my mtd only has uboot, kernel, rootfs partition

After I did this:
root@lede:~# cat /proc/mtd
dev: size erasesize name
mtd0: 00100000 00020000 "u-boot"
mtd1: 00200000 00020000 "kernel"
mtd2: 07d00000 00020000 "rootfs"
mtd3: 072a0000 00020000 "rootfs_data"

Is this the right way to do this?!



Edited 3 time(s). Last edit at 04/15/2017 12:03PM by protocold.

Yes, on first start the LEDE firmware splits the original rootfs partition into two partitions. All space not occupied by the squashfs (read-only root) will become rootfs_data which is read-write.

Hello,

I have a pogo v4 that I was going to install this rescue on. I have run into something that is stopping me. I first setup mtdparts to include a kernel and rootfs. Reboot. Verify the mtd looks good then I did this.

flash_erase /dev/mtd2 0 0
Erasing 128 Kibyte @ 2e0000 -- 100 % complete 
nandwrite /dev/mtd2 uImage-kirkwood-pogoplug_v4

and that gives me this:

Input file is not page-aligned. Use the padding option.
nandwrite: error!: Data was only partially written due to error
           error 0 (Success)

Not sure what to do.

Edit:

Figured it out. Used this. nandwrite -b 1 -n /dev/mtd2 -p uImage-kirkwood-pogoplug_v4


Netconsole giving me this error upon boot to recovery:

** Bad device usb 0 **
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
** Bad device usb 0 **
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   recovery-pogoplug_v4
   Created:      2016-09-09  19:17:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    1871182 Bytes = 1.8 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!

NAND read: device 0 offset 0x500000, size 0x300000
NAND read from offset 500000 failed -74
 0 bytes read: ERROR
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   recovery-pogoplug_v4
   Created:      2016-09-09  19:17:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    1871182 Bytes = 1.8 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!
Pogov4>

Edited 3 time(s). Last edit at 05/09/2017 03:15PM by umd.

nandwrite -b 1 -n /dev/mtd2 -p uImage-kirkwood-pogoplug_v4

I'm pretty sure you should not use the "-n", that option is used to flash special dumps of flash that have ECC info in them already.

The uimage is just a plain file, so it must be written with flash ecc enabled.

I think you should not need the "-b 1" option too.

the error is saying to use padding option, which is "-p"



Edited 1 time(s). Last edit at 05/09/2017 03:45PM by bobafetthotmail.

Yep! That was it. When I started looking at what some of those other parameters were, -n stuck out as a culprit. I just kept the -p and eliminated the other variables and it booted right up. Thanks for your help and by the way, nice work on the project.



Edited 1 time(s). Last edit at 05/09/2017 04:04PM by umd.

continuing from https://forum.lede-project.org/t/installing-lede-on-nsa325/3611/2

The u-boot from LEDE doesn't work at all I only get

/e-data/ # nandwrite /dev/mtd0 kirkwood/generic/u-boot-nsa325/u-boot.kwb
Input file is not page aligned
Data did not fit into device, due to bad blocks
: Success

/e-data/tools # dmesg | grep -i bad 
Scanning device for bad blocks

@Menno K I answered you in the original thread here

http://forum.doozan.com/read.php?4,34842,34876#msg-34876



Edited 1 time(s). Last edit at 06/17/2017 03:59PM by bodhi.

Hello Friends,

I am having pogoplug pro v3 OXNAS wifi model running Debian (USB Boot) with bodhi's U-Boot.

also on NAND i installed OpenWRT 15.05 using http://blog.qnology.com/2015/04/openwrt-on-pogoplug-v3oxnas-proclassic.html method.

The problem is that this version of OpenWRT doesnot having latest kernel.
and when i enable WiFi RT3090 / AR9285 / USB AR9271. the whole system hangup, then i i have to remove the wifi drivers.

i am also having TP-LINK MR3020 v1, Running LEDE 17.01.04 , USB AR9271 wifi dongle and inbuilt wifi AR9331 works very well.




So i wanted to install LEDE 17.01.04 on PogoPlug Pro Wifi Model.

Qui's method for installing OpenWrt is not working with LEDE installation ,

Because Openwrt method requires these files.

1. openwrt-15.05-oxnas-pogoplug-pro-squashfs-sysupgrade.tar
2. openwrt-15.05-oxnas-pogoplug-pro.dtb
3. openwrt-15.05-oxnas-zImage


But LEDE Server Doesnot Having .DTB File

1. https://downloads.lede-project.org/releases/17.01.4/targets/oxnas/generic/lede-17.01.4-oxnas-pogoplug-pro-squashfs-sysupgrade.tar

2. DTB File Not Found

3. https://downloads.lede-project.org/releases/17.01.4/targets/oxnas/generic/zImage


Please HELP me installing LEDE on PogoPlug Pro

pritamcharan Wrote:
-------------------------------------------------------
> Hello Friends,
>
>
> Please HELP me installing LEDE on PogoPlug Pro

I already answered a similar question for a kirkwood device here http://forum.doozan.com/read.php?4,34842,34876#msg-34876

Works the same for your device too, just change the names
You must use pogoplug-pro-uImage instead of lede-kirkwood-dockstar-initramfs-uImage and pogoplug-pro-ubifs-sysupgrade.tar for the sysupgrade.

Hi,

Would like to try this out on Pogo V3 Classic P21,
I would need to

Method A. flash the LEDE uImage(lede-17.01.4-oxnas-pogoplug-v3-uImage) and rootfs from within Debian right?
(I have uboot.2015.10 and linux-4.4.117 kernel and newest Debian rootfs.)

Method B. or follow the instruction in http://forum.doozan.com/read.php?4,34842,34876#msg-34876 ?

If I need to go with method A,

root@debian:/etc# cat /etc/fw_env.config
# MTD device name       Device offset   Env. size       Flash sector size       Number of sectors
# pogoplug pro
  /dev/mtd0               0x00100000      0x20000         0x20000
root@debian:/etc# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00e00000 00020000 "boot"
mtd1: 07200000 00020000 "data"

How do I partition the nand to have kernel and rootfs partition?
By changing the /etc/fw_env.config and reboot? or set the mtdparts in env?

For the LEDE rootfs, am I supposed to extract the 'root' from
lede-17.01.4-oxnas-pogoplug-v3-ubifs-sysupgrade.tar
then flash it to the rootfs partition?

Thanks.



Edited 2 time(s). Last edit at 03/09/2018 06:16PM by rynax.

Will this rescue system work on Zyxel NSA320 ?
I've installed the last uboot of bodhi.

cat /proc/mtd
dev:    size   erasesize  name
mtd0: 000c0000 00020000 "uboot"
mtd1: 00080000 00020000 "uboot_env"
mtd2: 07ec0000 00020000 "ubi"


cat /proc/cpuinfo
processor       : 0
model name      : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS        : 1196.85
Features        : swp half fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant     : 0x2
CPU part        : 0x131
CPU revision    : 1

Hardware        : Marvell Kirkwood (Flattened Device Tree)
Revision        : 0000
Serial          : 0000000000000000


cat /proc/partitions
major minor  #blocks  name

  31        0        768 mtdblock0
  31        1        512 mtdblock1
  31        2     129792 mtdblock2

Thanks.

Thanks very much. It works perfetly in my pogo-e02.

Here's my way for who need it. It must be done in usb boot, of course.

I have newest uBoot (link) with Jeff's Rescue System v2 (link) in nand, and Debian stretch in usb stick.


And my mtd partitions.

dev:    size   erasesize  name
mtd0: 00100000 00020000 "u-boot"
mtd1: 00400000 00020000 "uImage"
mtd2: 02000000 00020000 "rootfs"
mtd3: 05b00000 00020000 "data"

1. Download files.
You need only 2 files.
'uImage-kirkwood-pogo_e02' and 'lede-kirkwood-generic-squashfs.img'
Download & put them in /tmp.


2. Erase & write nands.

cd /tmp

#use nand_erase or flash_erase, what you have. 
flash_erase /dev/mtd1 0 0
nandwrite -p /dev/mtd1 uImage-kirkwood-pogo_e02

flash_erase /dev/mtd2 0 0
nandwrite /dev/mtd2 lede-kirkwood-generic-squashfs.img

3. Set envs.

fw_setenv recovery_bootargs 'setenv bootargs console=ttyS0,115200 rootdelay=10 $mtdparts_lede'
fw_setenv bootcmd_recovery 'run recovery_bootargs; nand read 0x800000 0x100000 0x400000; bootm 0x800000'
fw_setenv bootcmd 'run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec; run bootcmd_recovery;'
fw_setenv mtdparts_lede 'mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)'

I hope it works. gl.

If something goes wrong, just go back with v2.

ps. I do this because Jeff's Rescue System v2 uses kernel 2.6.32, and it's not possible in v2 that install debian stretch at once. 'Kernel too old' error occurs, and I had to install jessie first, and use dist-upgrade.

I'm hoping to install LEDE on my GoFlex Home which has the stock software installed.although I am able to ssh into the system.

I'm looking for simple step by step installation instructions but don't really know where to start.

bobafetthotmail Wrote:
-------------------------------------------------------
> I will add this to the installer script Soon(TM)
> so that you can automatically install a powerful
> recovery on your device's NAND instead of wasting
> it with a dead firmware the new uboot cannot use.

Just wondering if there is an installer script included.... I downloaded the file but didn't find anything I could identify as an installer script.

Dear folks,

Great rescue system, boots like charm!
I am using pogoplug_v4, I have only sdcard (no usb2 used) and I made debian there.
However, just in case, if something goes wrong, I need rescue system and this solution looks good.

I would just suggest one small improvement: is it possible to have ntfs-3g support in?
I have 4TB NTFS disk attached to it (actually, this is first reason I have this pogoplug - being a NAS).
I think it could be handy to be able read and write files to/from it.

thanks in advance,
ayosher

Could this be applied to the oxnas pogoplug p21 model?

The download link is bad. Is there a good link?

If someone still has "kirkwood_recoveryV4-beta.7z" (or other files like this) please consider uploading to archive.org.