Pogoplug E02 - Cold boot works but reboot doesn't

I have a Pogoplug E02 and I installed uBoot through the script http://projects.doozan.com/uboot/install_uboot_mtd0.sh

I'm trying to get ArchLinux running from a 4TB Western Digital MyBook, so I updated uBoot to the 2013-10 version on the forum so that I could use GPT partition table. (I initially tried setting up a hybrid MBR but kept running into issues so I went with the updated uBoot.)

Basically, from a cold boot, it will boot fine from the usb hard drive but never from a reboot. After reboot it will always boot into the rescue system. I tried several of the suggestions on the forum like lowering bootdelay and stopping and starting usb before usb_scan. Nothing seems to work and I could use some help.

The following is the uBoot netconsole output after a cold boot:

U-Boot 2013.10 (Feb 21 2014 - 19:35:02)-tld-2
Pogo E02
gcc (Debian 4.6.3-14) 4.6.3
GNU ld (GNU Binutils for Debian) 2.22
Hit any key to stop autoboot:  0 
(Re)start USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... READ_CAP ERROR
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
2 Storage Device(s) found
** File not found /rescueme **
** Unrecognized filesystem type **
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=3"
UBI: MTD device size:            91 MiB
UBI: number of good PEBs:        727
UBI: number of bad PEBs:         1
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             716
UBI: total number of reserved PEBs: 11
UBI: number of PEBs reserved for bad PEB handling: 7
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "rootfs", error -22
UBIFS error (pid 0): ubifs_mount: Error reading superblock on volume 'rootfs' errno=-22!

ubifsmount - mount UBIFS volume

Usage:
ubifsmount <volume-name>
    - mount 'volume-name' volume
1 bytes read in 126 ms (0 Bytes/s)
Found bootable drive on usb 0:1
3931528 bytes read in 259 ms (14.5 MiB/s)
** File not found /boot/uInitrd **
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-3.13.6-1-ARCH
   Created:      2014-03-08   9:06:30 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3931464 Bytes = 3.7 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK


Starting kernel ...

And this is after shutdown -r now

U-Boot 2013.10 (Feb 21 2014 - 19:35:02)-tld-2
Pogo E02
gcc (Debian 4.6.3-14) 4.6.3
GNU ld (GNU Binutils for Debian) 2.22
Hit any key to stop autoboot:  0 
(Re)start USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... Device NOT ready
   Request Sense returned 02 04 01
READ_CAP ERROR
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
EHCI timed out on TD - token=0xbe008d80
1 Storage Device(s) found
** No partition table - usb 0 **
** No partition table - usb 0 **
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=3"
UBI: MTD device size:            91 MiB
UBI: number of good PEBs:        727
UBI: number of bad PEBs:         1
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             716
UBI: total number of reserved PEBs: 11
UBI: number of PEBs reserved for bad PEB handling: 7
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "rootfs", error -22
UBIFS error (pid 0): ubifs_mount: Error reading superblock on volume 'rootfs' errno=-22!

ubifsmount - mount UBIFS volume

Usage:
ubifsmount <volume-name>
    - mount 'volume-name' volume
** No partition table - usb 0 **
** Bad device usb 1 **
** Bad device usb 2 **
** Bad device usb 3 **
** No partition table - usb 0 **
** No partition table - usb 0 **
Wrong Image Format for bootm command
ERROR: can't get kernel image!
stopping USB..

NAND read: device 0 offset 0x100000, size 0x400000
 4194304 bytes read: OK
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-3.3.2-kirkwide
   Created:      2012-10-29  22:52:12 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3627768 Bytes = 3.5 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK


Starting kernel ...

The main difference seems to be right after it scans for storage devices it states

Device NOT ready
Request Sense returned 02 04 01

The following is my uBoot settings

# fw_printenv
arcNumber=2097
baudrate=115200
console=ttyS0,115200
ethact=egiga0
ethaddr=00:25:31:04:85:56
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
if_netconsole=ping $serverip
ipaddr=192.168.1.137
led_error=orange blinking
led_exit=green off
led_init=green blinking
machid=dd6
mainlineLinux=yes
mtdids=nand0=orion_nand
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
ncip=192.168.1.22
partition=nand0,2
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
preboot=run if_netconsole start_netconsole
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
rescue_installed=1
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
serverip=192.168.1.22
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
stderr=nc
stdin=nc
stdout=nc
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_device=0:1
usb_root=/dev/sda1
usb_rootdelay=10
usb_rootfstype=ext3
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=3:1 dev=sdd1
usb_scan_list=1 2 3 4
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset
bootdelay=2
usb_init=run usb_scan

Any thoughts on what I can try?

Thanks in advance!

YodaJM,

My guess is it could be that either : Arch (I meant your system specifically, not in general) shutdown process is a little bit too quick, or the 4TB WD Mybook powerdown takes a little too long. Or a combination of these 2 factors. I'm guessing this because I never have reboot problem like your experience (one of my NASs is running this U-Boot and 3.13.1 Debian rootfs on a 4TB Seagate Desk USB). I did see 1 or 2 "EHCI timed out on TD" messages upon reboot, but it always recovered after that and booted into Debian.

Try this:

bootcmd=usb start; sleep 5; usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset

Increase the sleep period until you can boot without a "EHCI timed out on TD" message afte the 2nd "usb start". Or you could play with 3 "usb start"with sleep periods in between.

This is a problem that we could work around without having to improving U-Boot. But if you tried this approach and it still give you problem rebooting, I would consider changing the Arch shutdown slightly to not powering down the drive when rebooting (but add a few sync statement as a precaution), see if it'd help.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks for the reply bodhi.

So far adding a sleep command and a 2nd/3rd usb start seems to have no effect. It always says "Device NOT Ready" followed by 6 "EHCI timed out on TD".

I'm a Linux noob for the most part, so while I continue trying different sleep times, could you explain a little more how to change the Arch shutdown as you suggest?

Thanks!

YodaJM,

I'm not familiar with Arch Linux (using systemd), so afraid I can't help you there. It was just a thought to some possible tweaks to see if it helps. Perhaps the ALARM guys know whether it is possible to do this.

What you can do now is: interrupt netconsole, and at the prompt, run a few "usb start" commands to see if it eventually detected. And note how many times it would take to do that, if at all. Just to see if we need to modify U-Boot to fix this.

I could add a patch to build a U-Boot version that will allow users to increase the period (using a new U-Boot env) that USB device readiness test uses so that it won't time out too quicky. I've spent a few minutes to look at the code where U-Boot does this, and the time out is currently set to 1 second. It might be that for some HDD such as th WD 4TB Mybook, more time is needed to wait for it to be ready, because of a very long spin down / spin up cycle. And the sleep did not help because during the 1 second period U-Boot tried to test the HDD readiness, it timed out and declared the HDD is not ready, so does not matter how many times we try.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

It doesn't matter how many times I do a USB start, it always says Device NOT ready followed by 6 timeout messages. I did a USB stop, then unplugged the hard drive and plugged it back in. I then did a USB start and it recognized the drive like after a cold boot. So it seems if the hard drive is asleep the first time USB start is done, it will never recognize the drive no matter how many USB starts are done.

I think 1 second isn't enough time for the hard drive to wake up from a full sleep.

Is there much difference between Arch and Debian?

Thanks again!

Currently, Arch and Debian are different in systemd vs. sysvinit, i.e startup and shutdown, among other things. Debian will switch to systemd in jessie as a stable release.

Definitely, 1 second is just too short when the drive is sleeping. My modification will let the users adjusted the time period for USB drive using a U-Boot env. I've rebuilt 2014.01 U-Boot for all the boxes with this new feature, but has not tested with a real use case, I can't reproduce the problem it with the Seagate 4TB :)

The Pogo E02 does not have UART booting so you would have to flash it for real to be able to test it.

Could anyone who has UART-able boxes like Goflex Net/Home, and also has this problem can do some testing before I release it?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 03/17/2014 12:42AM by bodhi.

YodaJM,

See this post:
http://forum.doozan.com/read.php?3,12381,15608#msg-15608

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Bodhi, I would suggest this might be a good idea to include with the u-boot-kirkwood repo builds.

WarheadsSE,

Sure. I was hoping to get confirmation from YodaJM, or somebody before doing that. But it should not impact any thing to include it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I have the exactly same problem as YodaJM.

A few days ago, I bought "WD My Book 4TB" and wanted to replace the old HDD with it in the pogoplug E02 which uses archlinux.
At first, the pogoplug didn't boot at all with "WD My Book 4TB".
I tried to install the latest version(2014.04.R4-1) of uboot-pogo_e02 using pacman(package manager of archlinux) and it makes pogoplug be able to (cold) boot up with "WD My Book 4TB".
But (warm) rebooting fails as YodaJM.

I flashed the latest version of uboot(uboot.2014.07-tld-2.pogo_e02.bodhi.tar) in this forum.
I found I should set usb_ready_retry(http://forum.doozan.com/read.php?3,12381,15608#msg-15608) and repeat "usb start" command in order to reboot successfully.
usb_ready_retry setting without repeating "usb start" command can't solve the problem.
Just repeating "usb start" without setting of usb_ready_retry fails as YodaJM's tries.

The following is important part of the uboot settings.

bootcmd=usb start; usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run 
usb_ready_retry=5

holelee,

> I found I should set
> usb_ready_retry(http://forum.doozan.com/read.php?3
> ,12381,15608#msg-15608) and repeat "usb start"
> command in order to reboot successfully.
> usb_ready_retry setting without repeating "usb
> start" command can't solve the problem.
> Just repeating "usb start" without setting of
> usb_ready_retry fails as YodaJM's tries.
>
> The following is important part of the uboot
> settings.
>

> bootcmd=usb start; usb start; run
> force_rescue_bootcmd; run ubifs_bootcmd; run
> usb_bootcmd; usb stop; run rescue_bootcmd; run 
> usb_ready_retry=5
>

Thanks for confirming that it works with a real test case! Yes, you are correct, usb_ready_retry must be set before "usb start" for it to work.

If you see the message about usb_ready_retry printed out in netconsole or serial console, it should appear before the "USB re(start)" message appear.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)