Adding netconsole support to kernel NOT from the pogoplug

Hi All,

I have a pogoplog E02 which was running happily with debian squeeze, Kernel 3.2.0-0.bpo.3-kirkwood, bootig from a USB stick.

Everytime there is a power outage, the pogo won't boot back, but this was always easily fixed by fsck'ng the USB stick.

Yesterday there was a power outage, and although I fsck'd the stick, and fsck reports all is well, the pogo won't boot.

I setup nerconsole for the uBoot, and I can see all goes well until the kernel is booted, the last message being

Starting kernel ...

However I get no ping from the device.

My question:
Is there any way to install netconsole on the kernel so I can see what's going on, but not from the pogoplug (since it won't boot)? E.g. run it under some sort of emulator, or some chroot on a normal x86 debian?

Thanks
Shai

See here:
http://forum.doozan.com/read.php?2,9522

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I think I managed to do it on my own, but still I get no output from the kernel on netconsole.

Here's what I did to add netconsole: (All this is done on a debian sid x86 system)

## install mkimage
apt-get install  u-boot-tools

## mount USB stick boot partition in /mnt/boot
mount /dev/sdb3 /mnt/boot

## extract original initrd
cd /tmp
mkdir initrd
cd initrd
zcat /mnt/boot/boot/initrd.img-3.2.0-0.bpo.3-kirkwood | cpio -i

now, edit conf/modules and add

netconsole netconsole=6666@192.168.221.221/eth0,6666@192.168.221.148/xx:xx:xx:xx

where 192.168.221.221 and 192.168.221.148 are the ip addresses of the pogo and your machine respectively, and xx:xx:xx:xx is the MAC of your machine

Now to pack it up again:

cd /tmp/initrd
find . -print | cpio -o | gzip -c /tmp/initrd.img-3.2.0-0.bpo.3-kirkwood-netcat
cd /tmp
mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 -n initramfs-3.2.0-0-netconsole -d  initrd.img-3.2.0-0.bpo.3-kirkwood-netcat uInitrd

## move it over to the boot partition, but keep old one just in case!
cd /mnt/boot/boot
mv uInitrd uInitrd.orig
cp /tmp/uInitrd .
cd /tmp
umount /mnt/boot

So, this booted, and still my output ends at "Starting kernel ..."
Does this mean my kernel is hanging before it can start the netconsole?
Did I miss anything above?

Shai Ayal,

Netconsole only shows "Starting kernel ..." and stops there until the system can mount rootfs, and then starts again (given you have enabled the netconsole module in initrd).

So it does not necessarily mean you have missed something in the steps above. If something went wrong between this message and actuall rootfs loading, it will seem to be stuck. These errors can only be seen with serial console.

So the bottom line is your rootfs is still needed to be fixed. If you don't have serial console and already did fsck and found no error, then the easy aproach is to get a good rootfs, put it on a different USB stick, boot with it and then mount your stick to examine/fix the problem. You can use my uploaded rootfs here:
http://forum.doozan.com/read.php?2,12096

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/11/2014 03:20PM by bodhi.

bodhi,

I tried to boot with your rootfs (Debian-3.13.1-kirkwood-tld-2-rootfs-bodhi.tar.bz2). I didn't install a newer kernel since I don't have a working system in which to do a dpkg -i ....

I used an ext2 partition (and changed the fstab to ext2).

I got the exact same results -- nothing after "Starting kernel ..." , no ping, nothing .... The led is a dull green or maybe orange, I'm not sure.

The device does seem to boot the original pogo (at least the led becomes green), so it seems the hardware is OK to some extent.

Very strange ....

Any hints?

Thanks for your help so far
Shai

Another piece of information:

When booted into the original pogo firmware, the led turns green, I can see the plug in http://my.pogoplug.com/view, but the pug reports "no drive attached" although I did attach an empty FAT32 disk ....

Strange, since uBoot clealy manages to read the kernel and rootfs from the usb disk.

To compund all of this, I have no idea what the ssh password is for the pogoplug firmware. (I tried both ceadmin, and stxadmin) so I cannot logon to the pogo firmware and see from there what's happening ....

Shai Ayal,

What's your U-Boot envs look like? pls interrupt netconsole at boot and execute
printenv
and post output here.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

printenv 
arcNumber=2097
baudrate=115200
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset
bootdelay=10
console=netconsole=6666@/eth0,6666@/
ethact=egiga0
ethaddr=00:25:31:00:4A:FA
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
if_netconsole=ping $serverip
ipaddr=192.168.221.221
led_error=orange blinking
led_exit=green off
led_init=green blinking
mainlineLinux=yes
mtdids=nand0=orion_nand
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
ncip=192.168.221.148
partition=nand0,2
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
preboot=run if_netconsole start_netconsole
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
rescue_installed=0
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
serverip=192.168.221.148
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
stderr=nc
stdin=nc
stdout=nc
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_device=0:1
usb_init=run usb_scan; setenv usb_root LABEL=ROOTFS
usb_root=/dev/sda1
usb_rootdelay=10
usb_rootfstype=ext2
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=1:3 dev=sdb3
usb_scan_list=1 2 3 4
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params

Environment size: 2653/131068 bytes

Shai Ayal,

Your U-Boot envs look normal. I don't see why the rootfs should not boot. The only thing to do is to re-examine the steps you did to create the rootfs, as root user, sync,….

And if you can connect with SSH then then login is root/ceadmin. But you can also re-enable SSH at the website and change the user/password there.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

At this stage it looks to me like a strange problem with USB which uBoot can somehow overcome (since netconsole shows that it loads the kernel and uInitrd from USB), but the linux kernel cannot.

I want to stress again that thios system was working a few days ago, and I verfied (using a backup that I have) that the original kernel and inird files are intact and the same, so to me it is almost certain this is a hardware problem.

As to the original pogo firmware, I can connect to ssh, but it does not accept root/ceadmin, and when I go to the website I'm not entirly sure they can see my plug. I do not have the option to change password, and the website does not recognize any external disks I connect....

So it looks like I'm stuck ...

If you have any other ideas, let me know

Thanks for your help
Shai

Shai,

I would do this:

- Get a reliable brand USB stick such as Sandisk. Format it as Ext3, extract my rootfs to it, following the instruction. And try booting with that.

- If that fails, then get a serial console. With serial console, it's quite easy to solve any booting problem. Consider it's an opportunity of get this going! For Pogo E02, it's really easy, no soldering needed. See this thread for parts:
http://forum.doozan.com/read.php?8,13263

I think most likely it is the USB stick. It does not hurt to try.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 04/13/2014 03:02AM by bodhi.

I still don;t think it's USB stick problem -- i tried 2 usb sticks (same brand though). I have an external USB hdd -- I'll try that as well

As for the serial connection -- I think I'll do that -- it's time to do some hardware tinkering! that will take me some time though.

I'll report my findings

Thanks,
Shai