Cannot boot v3 kernel on PogoPlug E02

I'm trying to get Debian Wheezy installed on my PogoPlug, and I gotten through Jeff's Wheezy install script (slightly modified roughly following steps provided by TJ). I can boot into a 2.6 kernel that I had laying around from somewhere else, but no matter what I try I cannot get it to boot with a 3.x kernel with a uInitrd file. I have enabled netconsole, and collected the following output:

Booting 2.6 kernel:

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02
arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2009q3-67) 4.4.1
GNU ld (Sourcery G++ Lite 2009q3-67) 2.19.51.20090709
Hit any key to stop autoboot:  0 
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 3 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Loading file "/rescueme" from usb device 0:1 (usbda1)
** File not found /rescueme
reading /rescueme.txt

** Unable to read "/rescueme.txt" from usb 0:1 **
Creating 1 MTD partitions on "nand0":
0x000002500000-0x000008000000 : "mtd=3"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=3"
UBI: MTD device size:            91 MiB
UBI: number of good PEBs:        728
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             717
UBI: total number of reserved PEBs: 11
UBI: number of PEBs reserved for bad PEB handling: 7
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "ubi:rootfs", error -19
Error reading superblock on volume 'ubi:rootfs'!
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
1 bytes read
Found bootable drive on usb 0:1
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
2549080 bytes read
Loading file "/boot/uInitrd" from usb device 0:1 (usbda1)
4100284 bytes read
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   linux-2.6.39.4-8
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2549016 Bytes = 2.4 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initrd-2.6.39.4-8
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    4100220 Bytes = 3.9 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK
Using machid 0xdd6 from environment

Starting kernel ...

Soon after I see "Starting kernel ..." the green light on the front lights up, I can ping the device, and soon after that I can login with SSH.

Here is the netconsole log when booting a 3.3.2 kernel:

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02
arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2009q3-67) 4.4.1
GNU ld (Sourcery G++ Lite 2009q3-67) 2.19.51.20090709
Hit any key to stop autoboot:  0 
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 3 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Loading file "/rescueme" from usb device 0:1 (usbda1)
** File not found /rescueme
reading /rescueme.txt

** Unable to read "/rescueme.txt" from usb 0:1 **
Creating 1 MTD partitions on "nand0":
0x000002500000-0x000008000000 : "mtd=3"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=3"
UBI: MTD device size:            91 MiB
UBI: number of good PEBs:        728
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             717
UBI: total number of reserved PEBs: 11
UBI: number of PEBs reserved for bad PEB handling: 7
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "ubi:rootfs", error -19
Error reading superblock on volume 'ubi:rootfs'!
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
1 bytes read
Found bootable drive on usb 0:1
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
2279256 bytes read
Loading file "/boot/uInitrd" from usb device 0:1 (usbda1)
7212319 bytes read
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-3.3.2
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2279192 Bytes = 2.2 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-3.3.2-kirkwood-dg
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    7212255 Bytes = 6.9 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK
Using machid 0xdd6 from environment

Starting kernel ...

After "Starting kernel ..." the green light does not come on and I cannot ping or connect with SSH (I've also tried 3.1.0, 3.2.0 and 3.3.8 kernels with the same result). One notable exception is the kernel from davygravy's Rescue V3, which I can use to boot and SSH into the system. However, that kernel (/boot/uImage) does not have a corresponding uInitrd file. BTW, my technique for trying different kernels is to symlink uImage-xxxx and uInitrd-xxxx files to /boot/uImage and /boot/uInitrd respectively. Here's a listing of /boot to illustrate:

# ls -al /boot
total 48204
drwxr-xr-x  3 root root    4096 Feb 16 21:34 .
drwxr-xr-x 22 root root    4096 Feb 16 21:34 ..
-rw-r--r--  1 root root 1035665 Feb 16 17:11 System.map-2.6.39.4-vamp
-rw-r--r--  1 root root 1220456 Apr 23  2012 System.map-3.3.2-kirkwood-dg
-rw-r--r--  1 root root   73872 Feb 16 17:11 config-2.6.39.4-vamp
-rw-r--r--  1 root root  107468 Apr 23  2012 config-3.3.2-kirkwood-dg
-rw-r--r--  1 root root 7212017 Feb 16 17:32 initrd.img-3.3.2-kirkwood-dg
drwxr-xr-x  2 root root    4096 Feb 16 20:48 old
lrwxrwxrwx  1 root root      20 Feb 16 21:28 uImage -> uImage-2.6.39.4-vamp
-rw-r--r--  1 root root 2549080 Feb 16 17:11 uImage-2.6.39.4-vamp
-rw-r--r--  1 root root 1606440 Feb 16 21:17 uImage-3.2.0-4-kirkwood
-rw-------  1 root root 3627880 Feb 16 20:34 uImage-3.3.2-kirkwide-rescue
-rw-r--r--  1 root root 2279256 Feb 16 20:43 uImage-3.3.2-kirkwood-dg
lrwxrwxrwx  1 root root      21 Feb 16 21:28 uInitrd -> uInitrd-2.6.39.4-vamp
-rw-r--r--  1 root root 4100284 Feb 16 17:11 uInitrd-2.6.39.4-vamp
-rw-r--r--  1 root root 6761928 Feb 16 21:16 uInitrd-3.2.0-4-kirkwood
-rw-r--r--  1 root root 7212081 Feb 16 20:44 uInitrd-3.3.2-kirkwood-dg
-rw-r--r--  1 root root 6610550 Apr 23  2012 vmlinux-3.3.2-kirkwood-dg
-rw-r--r--  1 root root 2549016 Feb 16 17:11 vmlinuz-2.6.39.4-vamp
-rw-r--r--  1 root root 2279192 Apr 23  2012 vmlinuz-3.3.2-kirkwood-dg

I think I'm on an up to date uBoot version that is supposed to be able to run 3.2+ kernels. Here's my uBoot environment:

# fw_printenv 
ethact=egiga0
bootdelay=3
baudrate=115200
mainlineLinux=yes
console=ttyS0,115200
led_init=green blinking
led_exit=green off
led_error=orange blinking
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
mtdids=nand0=orion_nand
partition=nand0,2
stdin=serial
stdout=serial
stderr=serial
rescue_installed=0
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_list=1 2 3 4
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=3:1 dev=sdd1
usb_init=run usb_scan
usb_device=0:1
usb_root=/dev/sda1
usb_rootfstype=ext2
usb_rootdelay=10
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset
ethaddr=00:25:31:04:70:2A
machid=dd6
serverip=10.2.1.2
ipaddr=10.2.1.50
if_netconsole=ping $serverip
start_netconsole=setenv ncip $serverip; setenv bootdelay 3; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
preboot=run if_netconsole start_netconsole
arcNumber=3542

I'm not sure what I'm doing wrong. I've tried to configure netconsole as a kernel module as well in attempt to see what's happening during the kernel boot process, but the 3.x kernels do not appear to get far enough get an IP address and/or write to the network.

Pointers on what I'm doing wrong or what else I might try are much appreciated. Thank you in advance.

Daniel,

Without serial console, it is hard to see what is going in on during kernel booting. Unless you set up netconsole module to start early in the process: http://forum.doozan.com/read.php?2,9522

I would bring it up with your 2.6 kernel, set up netconsole module as described above. And then upgrade to Wheezy with

apt-get update
apt-get upgrade

See if you can see the kernel booting process, and then upgrade 3.2.x kernel with deb file.

Thanks @bodhi, I already tried to enable netconsole using the instructions on the thread you linked, but didn't have any luck. I'll try that again to see if I missed a step.

Can I safely assume that the bootloader is not the problem? I read in several places that 3.2+ kernel will not boot with certain older boot loaders due to an L2cache issue. Would that issue be evident in the netconsole boot logs that I have provided above or would it only show up in a kernel boot log? Alternately, can you confirm that the version of uBoot that I am using, U-Boot 2011.12 (Feb 20 2012 - 21:21:59), will work with a 3.2+ kernel?

Thanks a lot.

P.S. I am the OP. I registered on the forums since I posted the original message.

@millerdev,

Your uBoot is the new version, should work with kernel 3.2+. If you don't see the green light comes on then it meant it did not get to the point where the rootfs is loaded. This usually means your initramfs has some problem running, and it could be too early so you don't see netconsole output. The netconsole module approach described above would work if

1. The kernel you are running indeed has netconsole turned on as a module.
2. The problem occured AFTER the network interface was initialized

I think it was likely that it did not get to the point where the netconsole module is to be loaded in initrd.



Edited 2 time(s). Last edit at 02/19/2013 06:38PM by bodhi.

@bodhi,

Thank you for helping me out with this.

I tried to configure the netconsole module again but still nothing. Note that I was not able to test it with the 2.6 kernel because I do not have an initrd.img-2.6... file (I only have uImage and uInitrd copied from another installation running 2.6).

So I used initrd.img-3.2.0-4-kirkwood, which is the kernel that got installed by Jeff's Wheezy install script. When I reboot the green light does not come on and there is nothing in /var/log/dmesg, so I'm assuming there is a problem with my initramfs. Any ideas on how I might setup a 3.x kernel with an initramfs that will work?

Have you tried Davy's NSA320 rootfs? it should boot the Pogo E02, and actually most of the Kirwood plugs. This thread:
http://forum.doozan.com/read.php?2,7806

Quote
davygravy
The rootfs is packaged as an ext3 image (kirkwood-debian_image.img) so you'll have to mount with the loop option, and cp -a the contents onto a preformatted sda1 partition.

Thanks so much for your help, @bodhi.

I downloaded and prepared a new USB drive with Davy's NSA320 rootfs. At first it did not boot (just like all of the other times). So I decided it was time to re-flash the uBoot. This is exactly what I did for anyone else having a similar issue:

cd /tmp
wget http://projects.doozan.com/uboot/install_uboot_mtd0.sh
chmod +x install_uboot_mtd0.sh
export PATH=$PATH:/usr/sbin:/sbin
./install_uboot_mtd0.sh

And at the prompt about already having the latest uBoot:

You are already running the latest uBoot.
Your current uBoot environment should be reasonable.  However, if you're having
any probems booting, you can reset the environment variables to know good values.
Would you like to reset the uBoot environment? [N/y] y

# Installing uBoot environment
... snip ...
# uBoot installation has completed successfully.

After that I rebooted with the NSA320 rootfs, and it worked! Next I reconfigured netconsole in the boot loader:

fw_setenv serverip XXX.XXX.XXX.XXX
fw_setenv ipaddr XXX.XXX.XXX.XXX
fw_setenv if_netconsole 'ping $serverip'
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 3; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
fw_setenv preboot 'run if_netconsole start_netconsole'

The only difference that I know of between the previous uBoot flash and this most recent one is that I had run the following two commands as instructed on this page:

fw_setenv arcNumber
fw_setenv machid dd6

Later I ended up doing this as well

fw_setenv arcNumber 3542

because I was trying everything I could think of. Is it possible that the presence of "machid" in the uBoot environment caused the 3.x kernels not to boot?

Anyway, thanks so much for your help. Now I'm off to try to get USB wifi working.

You're welcome! Actually the machid env is only needed to boot as Pogo E02. Without it, it should still boot, but as reference board. I think what has fixed your problem was you reinstalled uBoot and said Y at the prompt to reset the envs.

I can confirm that the "machid" setting in the uBoot environment seems to cause boot problem. When I set it the machine will not boot (I believe only on 3.x kernels). When I clear it the machine will boot again. Has anyone else seen this? Is it possible to get the kernel to recognize the device correctly, mainly so I can make the light on the front work?

Ah! in that case, the kernel that you're booting with has the Pogo E02 patch. The current uBoot version has some problem with the Pogo E02 (Davy has a patched version in GIT somewhere), so in order to boot correctly the machid needs to be set. Seting machid is to fix that glitch so that arcNumber can be recognized by the kernel. There were users who boot successfully into Debian as reference board 2097, without machid but with correct arcNumber, I'm not sure which kernel version they used.

In any case, you're OK. As long as the arcNumber and machid is consistent, uBoot will work fine. Your Debian system now should have files in /sys/class/leds/.... that correspond to the Pogo E02 LED settings.

Hi Millerdev and Bodhi, not to necropsy, but I have come to this same issue. Everything was working great except for the annoying orange LED. A quick google turned up the fw_setenv machid d66 and sure enough, just as here, it breaks my debian stick. My net console was working great until I accidentally closed the window and now can't seem to get it back up. The E02 plug boots just fine without the debian stick. I've gathered that I need to edit the environment settings on the debian stick and I've managed to mount it, but when I look at etc/fw_env.config on the stick, I find nearly nothing. Where am I to go clear out that machid? Thank you in advance.

Guess I was up too late, in the morning the fog cleared, I did a bit more googling and then it hit me. My mental model was that the fw_setenv is something in the OS, not on the uBoot image (in the NVRAM right?). Soooo, here is how I fixed it:

cd /usr/sbin
fw_setenv machid

I also got my net console back up by killing the previous session (which I thought died)

killall nc

then re ran the console sniffer: http://forum.doozan.com/read.php?3,14,14