Use netconsole to troubleshoot uBoot without a serial cable

Is there an equivalent of saveenv command in ArchLinux ARM after booting Dockstar, i.e. fw_saveenv? Or any env variable set with fw_setenv is immediately saved?

What commands to use in ArchLinux ARM on Dockstar to check, if changed Uboot variables allow now to use Netconsole?

fw_setenv will save it immediately. To verify if netconsole setup correctly, post the output of fw_printenv here (no automatic way to check).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Is there a way to automatically stop the boot process and show Marvell>> prompt in Netconsole without pressing any keys?

How to proceed the boot process after it was stopped after a certain parameter was selected out of several options available without doing Uboot Reset?

I'm just trying to figure out if its possible to setup Multiboot via Netconsole as discussed below by adding "if...then" code to Uboot env, where OS choice is made depending on a number (1,2,3... - each number represents a different OS build):

Multiboot from different partitions with Uboot?



Edited 2 time(s). Last edit at 04/05/2014 03:13PM by sambul14.

sambul14,

> Is there a way to automatically stop the boot
> process and show Marvell>> prompt in Netconsole
> without pressing any keys?

Not without adding this function to U-Boot.

> How to proceed the boot process after it was
> stopped after a certain parameter was selected out
> of several options available without doing Uboot
> Reset?

Not sure what you meant exactly? U-Boot reset command is scripted inside some env such bootcmd, so it could be taken out from there.

> I'm just trying to figure out if its possible to
> setup Multiboot via Netconsole as discussed below
> by adding "if...then" code to Uboot env, where OS
> choice is made depending on a number (1,2,3... -
> each number represents a different OS build):

Do you mean interactively in netconsol or scripted in envs?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I mean make it scripted in env plus a single interaction with Netconsole to select a number corresponding to the OS chosen by a user at boot time.

Also, is it possible to continue boot after interrupting in Netconsole without entering Reset (i.e. without reboot) and how?

sambul14 Wrote:
-------------------------------------------------------
> I mean make it scripted in env plus a single
> interaction with Netconsole to select a number
> corresponding to the OS chosen by a user at boot
> time.

Yes. But with a caveat. The normal way is to to run a pre-scripted environment variable to boot an OS on a particular disk. So several variables needed to be scripted, one for each OS. However, to be able to see a menu and then type in a number: it is not possible without patching U-Boot.

>
> Also, is it possible to continue boot after
> interrupting in Netconsole without entering Reset
> (i.e. without reboot) and how?

The command is

boot

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/05/2014 05:51PM by bodhi.

FYI,

It is possible now to have a boot menu in U-Boot, but it only works in serial console (as a terminal). Boot menu does not work with netconsole.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

edited.



Edited 1 time(s). Last edit at 07/25/2014 08:44AM by vaidyasr.

Has anyone tried setting $serverip to a named address? Can this be done?

I'd like to set $serverip address to a dynamic ip service like no-ip.com. This would allow me to get uboot diagnostics from any location (not just on the local network).

CasperJim Wrote:
-------------------------------------------------------
> Has anyone tried setting $serverip to a named
> address? Can this be done?
>
> I'd like to set $serverip address to a dynamic ip
> service like no-ip.com. This would allow me to get
> uboot diagnostics from any location (not just on
> the local network).

You could do this with the dns command in the latest U-Boot (2014.07-tld-1). However, I think you might be better off with 2 boxes. One is a monitoring box (i.e. netconsole server) and the other is the target. The monitoring box will possibly: save the log in a file and mail it, or provides the output (i.e. file content) in a web server that you can connect to. You could also log into the monitoring box to directly interact with the target while it's booting.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Jeff, hi all!

First of all I must apologize for not understanding very much of u-boot yet, but I do understand that it is a very powerful bootloader and I am willing to learn more about it.

Thanks a lot for your effort of giving me netconsole access to my nicely running wheezy dockstar. That had once worked for me, but after upgrading u-boot to wheezy's binary (2012.04.01-2), two problems came up:

1. The variables in $name format were no longer reckognized, so I converted then to ${name}. Solved.
2. The if/then/fi construct was not reckognized, so I removed it.

That fixed my unbootable problem, but still the netconsole is not functioning. There might be a problem that debian's u-boot does not enable certain u-boot features.

One of them seems to be hush shell, I am not sure how to quickly verify that. Any hints? I would then like to file a wishlist item to the BTS. But something else seems to be preventing the netconsole in wheezy's u-boot. Any more hints?

serverip=xx.xx.xx.xx
ipaddr=xx.xx.xx.xx
if_netconsole=ping ${serverip}
start_netconsole=setenv ncip ${serverip}; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version
preboot=run if_netconsole start_netconsole

A convenient workaround for me might be if you recommended a good u-boot binary to use. Do you have a recommendation?

But still I would like debian's u-boot binaries to support as many useful features as possible, such as hush or netconsole. Serial console is a very inconvenient kludge for me. I'd like to avoid it whenever possible, and most probably netconsole helps a lot of other users, too. Any hints are much appreciated.

Cheers
Kevin

update:

Another unrelated issue came up with the upgrade from Jeff's u-boot binary to debian's: The offset of the environment had changed. from 0xc0000 to 0x60000. So debian's u-boot-tools showed me a different env than u-boot itself. Any changes made from within debian were of course then ineffective.

So I checked u-boot's "saveenv" for the real offset and edited debian's /etc/fw_env.config from 0xc0000 to 0x60000. Problem solved.

But I think there ought to be better way for the u-boot-tools to determine the correct location of the environment than just reading /etc/fw_env.config. This should be worth another wishlist item. But I'm clueless in how to determine the correct offset from within the operating system, only seeing the mtd binary. Any thoughts on that?

@Kevin,

You are going in reversed :) usually we install customized version of u-boot to get more capability, instead of using the mainline u-boot. What you had before was either Jeff or Davygravy u-boot (2011, 2012). And you have installed the mainline version on top of it. If you're looking for a newer u-boot to take advantage of modern features, see this thread for my builds of 2014 u-boot: http://forum.doozan.com/read.php?3,12381

However, if you're happy with the u-boot version from wheezy repo then it's fine, too, there is no need to install newer u-boot.

Regarding the evironment, /etc/fw_env.config is the standard. It specifies where the u-boot envs are located in NAND (kernel does not deal with u-boot). The fact that u-boot-tools uses this text file is flexible and powerful enough. Without this file, u-boot-tools has to scan NAND and figure out the envs location (difficult and inefficient). U-boot envs location is always hard coded in the u-boot version that you are using. It could be defined anywhere by the builder. We've followed Jeff's convention to use 0xC0000 because it is the best location for all the Kirkwood boxes (there are rationales that I don't want to go into much details in this response)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi:

Thanks for the hint. u-boot 2014.07 is running very nicely, netconsole is back and I added sntp to bootcmd. :)

Kevin

I have followed the instructions for this on my Pogoplug E02 but I get no netconsole output on my desktop running nc for Windows @192.168.1.50.

If I boot my Pogoplug V4 which is set up the same way, I do get netconsole output from it.

Both plugs were modded with the same Uboot shell script from http://projects.doozan.com/uboot/install_uboot_mtd0.sh

Below is the fw_printenv from the E02. Can anyone spot the problem?

ethact=egiga0
bootdelay=3
baudrate=115200
mainlineLinux=yes
console=ttyS0,115200
led_init=green blinking
led_exit=green off
led_error=orange blinking
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
mtdids=nand0=orion_nand
partition=nand0,2
stdin=serial
stdout=serial
stderr=serial
rescue_installed=0
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_list=1 2 3 4
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=3:1 dev=sdd1
usb_init=run usb_scan
usb_device=0:1
usb_root=/dev/sda1
usb_rootfstype=ext2
usb_rootdelay=10
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset
ethaddr=00:25:31:04:B2:34
arcNumber=3542
serverip=192.168.1.50
ipaddr=192.168.1.249
if_netconsole=ping $serverip
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
preboot=run if_netconsole start_netconsole

Frederick Grayson,

> If I boot my Pogoplug V4 which is set up the same
> way, I do get netconsole output from it.
>
> Both plugs were modded with the same Uboot shell
> script from
> http://projects.doozan.com/uboot/install_uboot_mtd
> 0.sh
>

Your env looks good, so hard to tell what is wrong. What you can do is open a second terminal and try start pinging 192.168.0.249 continuously, before powering up the Pogo E02. See if netconsole is briefly up. This is how you can tell if neconsole is actually running, and the u-boot you installed is indeed supporting netconsole.

And to find u-boot version without serial/net console:

grep -a --null-data U-Boot /dev/mtd0ro

On a side note, however, you've mentioned above that the Pogo V4 was installed with the script http://projects.doozan.com/uboot/install_uboot_mtd0.sh? This script does not support Pogo V4, so did you remember correctly? or this script has been modified to handle Pogo V4?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 10/05/2014 11:11PM by bodhi.

Thanks for your reply

I was mistaken about using http://projects.doozan.com/uboot/install_uboot_mtd0.sh? on the V4. I used Qui's blog at http://blog.qnology.com/2014/07/hacking-pogoplug-v4-series-4-and-mobile.html

Opening a second terminal and continuously pinging the address as given by the ipaddr fw environment variable before booting the plugs has this outcome:

For the V4, echo replies start coming in at the same time I see netconsole output in the other terminal.

For the E02, no echos are seen and no netconsole output either.


For the E02:

~# grep -a --null-data U-Boot /dev/mtd0ro

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02U-Boot
** Invalid partition type "%.32s" (expect "U-Boot")
U-Boot BUG at %s:%d!

Your u-boot has support for netconsole. It's the latest installed by this script.

Boot up the Pogo E02, log in to Linux and ping 192.168.1.50, do you see the response? and does your /etc/resolve.config has any domain in it? if there is, try clearing everything in it and replace with

nameserver 8.8.8.8

I think it might be your network. Netconsole is not a sure thing, and it is affected by the network environment (i.e. configuration). It is rare, but a few users never got their netconsole working properly, and we don't know why (without actually being inside the network, it is hard to troubleshoot this virtually).

Btw, you can find the newer u-boot here: http://forum.doozan.com/read.php?3,12381.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I booted the plug and logged in.

192.168.1.50 is pingable and replies are seen.

/etc/resolv.config had a domain reference in it which I removed and I changed the nameserver to 8.8.8.8

Still no netconsole output :(

Is there any reason to upgrade to the new Uboot?

Any thoughts as to why the V4 plug netconsole works in my network environment and the E02 doesn't?

Thanks for your help.



Edited 2 time(s). Last edit at 10/07/2014 12:47PM by Frederick Grayson.

I guess I'll have to wait for the serial cable parts I ordered. I'm glad I don't need the capability right now, I'm just curious. Hopefully the PL2303HX chip won't be a counterfeit :)

Frederick Grayson Wrote:
-------------------------------------------------------
> I guess I'll have to wait for the serial cable
> parts I ordered. I'm glad I don't need the
> capability right now, I'm just curious. Hopefully
> the PL2303HX chip won't be a counterfeit :)

You could update to the newer u-boot. Although I recognized your installed u-boot as the latest from the script, I can't be certain about whether something was missing.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I'd rather not upgrade it unless it was really needed.

I received my second new E02 today. I ran http://projects.doozan.com/uboot/install_uboot_mtd0.sh on it and configured it for netconsole just like the other E02. The new E02 does show netconsole output, but not every time it is booted. Here is the output when It does work:

Y:\Downloads\FTP http\NetCat for Windows\nc111nt_rodneybeede>nc -l -u -p 6666

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02
arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2009q3-67) 4.4.1
GNU ld (Sourcery G++ Lite 2009q3-67) 2.19.51.20090709
Hit any key to stop autoboot: 0
(Re)start USB...
USB: Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 3 USB Device(s) found
scanning bus for storage devices... 1 Storage Device(s) found
Loading file "/rescueme" from usb device 0:1 (usbda1)
** File not found /rescueme
reading /rescueme.txt

** Unable to read "/rescueme.txt" from usb 0:1 **
Creating 1 MTD partitions on "nand0":
0x000002500000-0x000008000000 : "mtd=3"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 129024 bytes
UBI: smallest flash I/O unit: 2048
UBI: sub-page size: 512
UBI: VID header offset: 512 (aligned 512)
UBI: data offset: 2048
UBI: attached mtd1 to ubi0
UBI: MTD device name: "mtd=3"
UBI: MTD device size: 91 MiB
UBI: number of good PEBs: 728
UBI: number of bad PEBs: 0
UBI: max. allowed volumes: 128
UBI: wear-leveling threshold: 4096
UBI: number of internal volumes: 1
UBI: number of user volumes: 0
UBI: available PEBs: 717
UBI: total number of reserved PEBs: 11
UBI: number of PEBs reserved for bad PEB handling: 7
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "ubi:rootfs", error -19
Error reading superblock on volume 'ubi:rootfs'!
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
1 bytes read
Found bootable drive on usb 0:1
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
2461416 bytes read
Loading file "/boot/uInitrd" from usb device 0:1 (usbda1)
6362802 bytes read
## Booting kernel from Legacy Image at 00800000 ...
Image Name: Linux-3.14.0-kirkwood-tld-1
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2461352 Bytes = 2.3 MiB
Load Address: 00008000
Entry Point: 00008000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
Image Name: initramfs-3.14.0-kirkwood-tld-1
Image Type: ARM Linux RAMDisk Image (gzip compressed)
Data Size: 6362738 Bytes = 6.1 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK

Starting kernel ...

Any thoughts as to why it's not 100%

Frederick Grayson,

> Y:\Downloads\FTP http\NetCat for
> Windows\nc111nt_rodneybeede>nc -l -u -p 6666

Perhaps the flakiness is on the netconsole server side. Try to make sure you can interrupt it and execute commands. Or use a different nc. Better yet, since you can boot the Pogo V4, use that for netconsole server. Don't use Windows netcat, you will have better luck with Linux nc.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Doesn't seem to make a dfference. nc -lu 6666 on my CentOS Linux box doesn't see it either.

It would be nice if a network broadcast IP address could be used for serverip. That way any nc running anywhere on the network could be used without having to change serverip.

Nevermind. I put things back the way they were and the problem seems to have corrected itself. I sure wish I knew what was wrong.



Edited 3 time(s). Last edit at 10/10/2014 06:00PM by Frederick Grayson.

Well, I spoke too soon. Netconsole output no longer comes from either of my pogo E02s, but does from my pogo v4.

I surrender. I will be installing serial ports to the E02s and be done with it.

Thanks for the help along the way.

Frederick Grayson,

As I mentioned, netconsole sometimes mysteriously not working for some people.

I suspect that it has something to do with the router in the network. You might check to see if it routes the UDP packet to a wrong destination (mac address and ip address don't match). I've experienced simliar (not quite the same) problem.

Think about netconsole set up, it is too simple:

ping $serverip

This does not always work correctly when the router has not cleaned up its table. I think I might be able to change netconsole envs setup in a way that will work better.

If you still want to try, you could unplug the Pogo V4, wait for the router to cleanup, or reboot the router and try again.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

All the pogoplugs and my workstation are plugged into the same switch. The switch uplinks to the router, thus the router isn't even involved in machine to machine communication.

Rebooting my switch does allow netconsole output, but only for one boot. Interesting. And I was unable to repeat this. I powered everything down and tried again, but no joy.

Let me ask this. What is fundamentally different between netconsole implementation on V4 vs E02 pogoplugs? Why does the V4 work 100%?

I also upgraded to the latest 2014.07 U-Boot for E02. That made no difference with netconsole. It did seem to boot a lot faster though. But the original pogoplug software is no longer available with it. So I downgraded back to U-Boot 2011.12.

Speaking of changes to envs setup....In another forum I found an older post where it was suggested to "to add the preboot commands to the beginning of bootcmd." So I did this:

bootcmd=run if_netconsole start_netconsole; usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset

Those in bold type came from the preboot env variable.

This results in netconsole output 100%. I rebooted maybe a dozen times to be sure. However, there is no Uboot countdown timer and Uboot can not be interrupted into a prompt. What's up with this?

If you have some env changes to suggest, I'll be happy to try them so long as they won't brick my device :)

Your time and expertise are appreciated very much. Thank you.

Frederick Grayson Wrote:
-------------------------------------------------------
> All the pogoplugs and my workstation are plugged
> into the same switch. The switch uplinks to the
> router, thus the router isn't even involved in
> machine to machine communication.
>
> Rebooting my switch does allow netconsole output,
> but only for one boot. Interesting. And I was
> unable to repeat this. I powered everything down
> and tried again, but no joy.
>
> Let me ask this. What is fundamentally different
> between netconsole implementation on V4 vs E02
> pogoplugs? Why does the V4 work 100%?

There is no difference if they are the same version released (eg. 2014.07), they have common driver.


> I also upgraded to the latest 2014.07 U-Boot for
> E02. That made no difference with netconsole. It
> did seem to boot a lot faster though. But the
> original pogoplug software is no longer available
> with it. So I downgraded back to U-Boot 2011.12.
>
> Speaking of changes to envs setup....In another
> forum I found an older post where it was suggested
> to "to add the preboot commands to the beginning
> of bootcmd." So I did this:
>
> bootcmd=run if_netconsole start_netconsole;
> usb start; run force_rescue_bootcmd; run
> ubifs_bootcmd; run usb_bootcmd; usb stop; run
> rescue_bootcmd; run pogo_bootcmd; reset
>
> Those in bold type came from the preboot env
> variable.
>
> This results in netconsole output 100%. I rebooted
> maybe a dozen times to be sure. However, there is
> no Uboot countdown timer and Uboot can not be
> interrupted into a prompt. What's up with this?

This is why I did not recommend it. Not very useful if if you can't interrupt netconsole.

>
> If you have some env changes to suggest, I'll be
> happy to try them so long as they won't brick my
> device :)
>
> Your time and expertise are appreciated very much.
> Thank you.

I will think of the changes in the envs that might help. My hunch is that the problem is the first ping failed to find serverip.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Maybe there is some kind of race condition where the normal boot happens before netconsole can get going?