Changes in SD Card access in U-Boot 2017.07-tld-1?

After a few days of learning from failure, I think the only remaining issue is boiled down to uBoot.

Background: Pogoplug V4 was running 2014.07-tld-1 upgraded to 2017.07-tld-1,l SD Card boot

When I was on 2014.07-tld-1, I never have any problems booting up from my Kingston 8GB SD

sudo udevadm info -a -n /dev/root
...
    SUBSYSTEMS=="mmc"
    DRIVERS=="mmcblk"
    ATTRS{manfid}=="0x000041"
...
    ATTRS{type}=="SD"
    ATTRS{date}=="09/2008"
    ATTRS{hwrev}=="0x2"

(SD info for reference)

However, after I upgraded to 2017.07-tld-1, I often encounter problem booting. From netconsole (another great thanks to whoever created this!), seems like it has an issue reading either uImage or uInitrd properly hence it failed to boot. Sometimes issue 'mmc rescan' or 'ext2load' is require. But more likely a few 'boot' after seems to do the job. After it is booted, I don't notice I/O errors.

U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:34:01 -0700)
Pogoplug V4
gcc (Debian 6.3.0-18) 6.3.0 20170516
GNU ld (GNU Binutils for Debian) 2.28
Hit any key to stop autoboot:  0 
starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

Reset IDE: ide_preinit failed

MMC rescan: current device # 0 initialized OK
## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0
loading envs from usb 0 ...
** Bad device usb 0 **

no IDE devices available

Partition Map for MMC device 0  --   Partition Type: DOS

Part	Start Sector	Num Sectors	UUID		Type
  1	63        	16015297  	00000000-01	83
loading envs from mmc 0 ...
** File not found /boot/uEnv.txt **
running scan_disk ...
Scan device usb
device usb 0:1
** Bad device usb 0 **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: ide_preinit failed
device ide 0:1
** Bad device ide 0 **
device ide 1:1
** Bad device ide 1 **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
Scan device mmc

MMC rescan: current device # 0 initialized OK
device mmc 0:1
1 bytes read in 22 ms (0 Bytes/s)
Found bootable drive on mmc 0
loading uImage ...
4115496 bytes read in 102 ms (38.5 MiB/s)
loading uInitrd ...
6284094 bytes read in 201 ms (29.8 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 39 ms (249 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!
Pogov4> boot
boot

Reset IDE: ide_preinit failed

MMC rescan: current device # 0 initialized OK
## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0
loading envs from usb 0 ...
** Bad device usb 0 **

no IDE devices available

Partition Map for MMC device 0  --   Partition Type: DOS

Part	Start Sector	Num Sectors	UUID		Type
  1	63        	16015297  	00000000-01	83
loading envs from mmc 0 ...
** File not found /boot/uEnv.txt **
running scan_disk ...
Scan device usb
device usb 0:1
** Bad device usb 0 **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: ide_preinit failed
device ide 0:1
** Bad device ide 0 **
device ide 1:1
** Bad device ide 1 **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
Scan device mmc

MMC rescan: current device # 0 initialized OK
device mmc 0:1
1 bytes read in 22 ms (0 Bytes/s)
Found bootable drive on mmc 0
loading uImage ...
4115496 bytes read in 112 ms (35 MiB/s)
loading uInitrd ...
6284094 bytes read in 201 ms (29.8 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 40 ms (243.2 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!
Pogov4> boot
boot

Reset IDE: ide_preinit failed

MMC rescan: current device # 0 initialized OK
## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0
loading envs from usb 0 ...
** Bad device usb 0 **

no IDE devices available

Partition Map for MMC device 0  --   Partition Type: DOS

Part	Start Sector	Num Sectors	UUID		Type
  1	63        	16015297  	00000000-01	83
loading envs from mmc 0 ...
** File not found /boot/uEnv.txt **
running scan_disk ...
Scan device usb
device usb 0:1
** Bad device usb 0 **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: ide_preinit failed
device ide 0:1
** Bad device ide 0 **
device ide 1:1
** Bad device ide 1 **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
Scan device mmc

MMC rescan: current device # 0 initialized OK
device mmc 0:1
1 bytes read in 23 ms (0 Bytes/s)
Found bootable drive on mmc 0
loading uImage ...
4115496 bytes read in 141 ms (27.8 MiB/s)
loading uInitrd ...
6284094 bytes read in 200 ms (30 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 40 ms (243.2 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:33 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    6284030 Bytes = 6 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 01c00000
   Booting using the fdt blob at 0x1c00000


Starting kernel ...

In the example above, the third boot seems to be successful (no power recycle). So I suspect the timing maybe it is a bit off when reading from SD card with the new uBoot. (again, I didn't have this issue when I was on 2014.07-tld-1)

I also notice a few "## Unknown partition table type 0" after first "MMC rescan: current device # 0 initialized OK" then the second time "MMC rescan: current device # 0 initialized OK" seems to be able to access the SD card. Is that normal?

Is there anyway to 'slow down' MMC access in uBoot? or a quick band aid fix for it 'retry' (instead of manually type in 'boot') when it fails?

Or is it safe to go back to the old uBoot? I understand there were some new 'features' (dts?) were used in the new uBoot/kernel. If so, how to 'downgrade'? (I expect need to rewrite the envs too?)

Thanks

Derek,

U-Boot 2017.07-tld-1 use an "improved" version of the MMC driver from mainline. And nobody has reported problem with it. So I suspect it might be that your envs have something that caused this flakiness.

Quote

U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:34:01 -0700)
Pogoplug V4
gcc (Debian 6.3.0-18) 6.3.0 20170516
GNU ld (GNU Binutils for Debian) 2.28
Hit any key to stop autoboot: 0

Power up and interrupt console at countdown, list you envs again

printenv

and then set the boot devices to SD only

setenv devices mmc

and boot

boot

Please post the entire console log here.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

The darn thing is, it doesn't always happens...

I am sure the envs can be modify to skip IDE since I am pretty sure I don't see myself doing the SATA mods any time soon.

Another question is, is it possible to also redirect stdout when kernel is booting up to netconsole?

U-Boot 2017.07-tld-1 (Sep 05 2017 - 00:34:01 -0700)
Pogoplug V4
gcc (Debian 6.3.0-18) 6.3.0 20170516
GNU ld (GNU Binutils for Debian) 2.28
Hit any key to stop autoboot: 10 
 0 
Pogov4> printenv
printenv
arcNumber=3960
bootcmd=run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec
bootcmd_exec=run load_uimage; if run load_initrd; then if run load_dtb; then bootm $load_uimage_addr $load_initrd_addr $load_dtb_addr; else bootm $load_uimage_addr $load_initrd_addr; fi; else if run load_dtb; then bootm $load_uimage_addr - $load_dtb_addr; else bootm $load_uimage_addr; fi; fi
bootcmd_uenv=run uenv_load; if test $uenv_loaded -eq 1; then run uenv_import; fi
bootdelay=10
bootdev=usb
device=0:1
devices=usb ide mmc
disks=0 1 2 3
dtb_file=/boot/dts/kirkwood-pogoplug_v4.dtb
ethact=egiga0
ethaddr=##:##:##:##:##:##
if_netconsole=ping $serverip
ipaddr=192.168.XX.XX
led_error=orange blinking
led_exit=green off
led_init=green blinking
load_dtb=echo loading DTB $dtb_file ...; load $bootdev $device $load_dtb_addr $dtb_file
load_dtb_addr=0x1c00000
load_initrd=echo loading uInitrd ...; load $bootdev $device $load_initrd_addr /boot/uInitrd
load_initrd_addr=0x1100000
load_uimage=echo loading uImage ...; load $bootdev $device $load_uimage_addr /boot/uImage
load_uimage_addr=0x800000
machid=f78
mainlineLinux=yes
mtdids=nand0=orion_nand
mtdparts=mtdparts=orion_nand:2M(u-boot),3M(uImage),3M(uImage2),8M(failsafe),112M(root)
nc_ready=1
ncip=192.168.XX.XX
partition=nand0,2
preboot=run preboot_nc
preboot_nc=setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fi
scan_disk=echo running scan_disk ...; scan_done=0; setenv scan_usb "usb start";  setenv scan_ide "ide reset";  setenv scan_mmc "mmc rescan"; for dev in $devices; do if test $scan_done -eq 0; then echo Scan device $dev; run scan_$dev; for disknum in $disks; do if test $scan_done -eq 0; then echo device $dev $disknum:1; if load $dev $disknum:1 $load_uimage_addr /boot/uImage 1; then scan_done=1; echo Found bootable drive on $dev $disknum; setenv device $disknum:1; setenv bootdev $dev; fi; fi; done; fi; done
serverip=192.168.XX.XX
set_bootargs=setenv bootargs console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts $custom_params
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
stderr=nc
stdin=nc
stdout=nc
uenv_addr=0x810000
uenv_import=echo importing envs ...; env import -t $uenv_addr $filesize
uenv_init_devices=setenv init_usb "usb start";  setenv init_ide "ide reset";  setenv init_mmc "mmc rescan"; for devtype in $devices; do run init_$devtype; done;
uenv_load=run uenv_init_devices; setenv uenv_loaded 0; for devtype in $devices;  do for disknum in 0; do run uenv_read_disk; done; done;
uenv_read=echo loading envs from $devtype $disknum ...; if load $devtype $disknum:1 $uenv_addr /boot/uEnv.txt; then setenv uenv_loaded 1; fi
uenv_read_disk=if test $devtype -eq mmc; then if $devtype part; then run uenv_read;  fi; else if $devtype part $disknum; then run uenv_read; fi;  fi
usb_ready_retry=15

Environment size: 3062/131068 bytes
Pogov4> setenv devices mmc
setenv devices mmc
Pogov4> boot
boot

MMC rescan: current device # 0 initialized OK

Partition Map for MMC device 0  --   Partition Type: DOS

Part	Start Sector	Num Sectors	UUID		Type
  1	63        	16015297  	00000000-01	83
loading envs from mmc 0 ...
** File not found /boot/uEnv.txt **
running scan_disk ...
Scan device mmc

MMC rescan: current device # 0 initialized OK
device mmc 0:1
1 bytes read in 23 ms (0 Bytes/s)
Found bootable drive on mmc 0
loading uImage ...
4115496 bytes read in 141 ms (27.8 MiB/s)
loading uInitrd ...
6284094 bytes read in 200 ms (30 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 39 ms (249 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:33 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    6284030 Bytes = 6 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 01c00000
   Booting using the fdt blob at 0x1c00000


Starting kernel ...

Derek,

The darn thing is, it doesn't always happens...

I am sure the envs can be modify to skip IDE since I am pretty sure I don't see myself doing the SATA mods any time soon.

Try using only mmc with

setenv devices mmc

for a while to see if it is consistently boot. If it happens again, insert a sleep period in bootcmd_uenv.

setenv bootcmd_uenv 'run uenv_load; if test $uenv_loaded -eq 1; then run uenv_import; fi; sleep 3'

Quote

Another question is, is it possible to also redirect stdout when kernel is booting up to netconsole?

See this instruction the Wiki thread:

Quote

Netconsole

Use netconsole to troubleshoot uBoot without a serial cable
Use netconsole to troubleshoot Debian kernel booting

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Bodhi

Quote
bodhi
U-Boot 2017.07-tld-1 use an "improved" version of the MMC driver from mainline.

I hope this is not an unresonable request... :) Is it possible to have an option in future version of uBoot to switch back to the previous version of MMC driver or slow down the access?

I notice whenever it fails, the access speed is ridiculous high

loading uImage ...
4115496 bytes read in 43 ms (91.3 MiB/s)
loading uInitrd ...
6284094 bytes read in 106 ms (56.5 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 27 ms (359.4 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!

loading uImage ...
4115496 bytes read in 86 ms (45.6 MiB/s)
loading uInitrd ...
6284094 bytes read in 60 ms (99.9 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 27 ms (359.4 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!

When it works, speed is kind of back to normal

loading uImage ...
4115496 bytes read in 105 ms (37.4 MiB/s)
loading uInitrd ...
6284094 bytes read in 190 ms (31.5 MiB/s)
loading DTB /boot/dts/kirkwood-pogoplug_v4.dtb ...
9962 bytes read in 27 ms (359.4 KiB/s)
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:21 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4115432 Bytes = 3.9 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-4.17.2-kirkwood-tld-1
   Created:      2018-08-04  20:46:33 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    6284030 Bytes = 6 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 01c00000
   Booting using the fdt blob at 0x1c00000


Starting kernel ...

BTW, the screenshot above are from the "same" boot (i.e. w/o power recycle), I just keep entering "boot" until it works...

Derek,

> I hope this is not an unresonable request... :)
> Is it possible to have an option in future version
> of uBoot to switch back to the previous version of
> MMC driver

It is indeed unreasonable :) usually I try to use mainline version if possible, since it is widely used and tested. My testing on a Sandisk SD card with Pogo V4 and Sheevaplug was without problem. Rayknight also tested this on a Sheevaplug (not sure which SD brand). So I think your problemmight be with the Kingston SD brand.

Have you tried to use a Sandisk SD card?

> or slow down the access?

This hopefully can be worked around by slowing down the booting process, but not in the driver itself. So you could try to insert sleep period after the card has been accessed in u-boot to let it settles down before reading the next file. And set the devices to mmc only for booting.

setenv devices mmc

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Yea, I have tested another SD card briefly, and seems to have the same results...

Sleep period doesn't really help. I also tried "setenv devices mmc" (that just boot from MMC w/o touching USB/IDE, right?) which doesn't help either.

Do you recall which version of uBoot you started with the 'improved' MMC access? I am thinking of downgrading the uBoot to make sure it boots everytime...

Derek,

> Yea, I have tested another SD card briefly, and
> seems to have the same results...

Try Sandisk.

>
> Do you recall which version of uBoot you started
> with the 'improved' MMC access? I am thinking of
> downgrading the uBoot to make sure it boots
> everytime...

That would be the version before 2017.07-tld-1:

https://forum.doozan.com/read.php?3,12381

Quote

Last Updated 28 Oct 2017
Updated 06 Dec 2017 (replace Sheevaplug with newer version 2017.07-tld-2)

I've uploaded 2017.07-tld-1 U-Boot images for Pogo E02, GoFlexNet, GoFlex Home, Dockstar, Pogo V4/Mobile, iConnect, Netgear Stora, Sheevaplug, NSA320, NSA325, NSA310S/320S, NSA310, HP-T5325, and Dreamplug.

The HP Thin Client T5325 installation procedure is different from other Kirkwood u-boots, see this thread for installation instruction and download link..

The Marvell Dreamplug box installation procedure is different from other Kirkwood u-boots, see this thread for installation instruction and download link.

New/updated in 2017.07-tld-1 version:

- Rebase u-boot code to track mainline 2017.07.
- Incorporated patch to fix "EHCI timed out on TD" error with USB drives.
- SD cards: Use the mainline version of MMC driver for Pogo V4/Mobile and Sheevaplug

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> Derek,
>
> > I hope this is not an unresonable request...
> :)
> > Is it possible to have an option in future
> version
> > of uBoot to switch back to the previous version
> of
> > MMC driver
>
> It is indeed unreasonable :) usually I try to use
> mainline version if possible, since it is widely
> used and tested. My testing on a Sandisk SD card
> with Pogo V4 and Sheevaplug was without problem.
> Rayknight also tested this on a Sheevaplug (not
> sure which SD brand). So I think your problemmight
> be with the Kingston SD brand.
>
My SheevaPlug is using a 32GB Sandisk Ultra Plus SD Card.

Ray