Use netconsole to troubleshoot uBoot without a serial cable

bodhi Wrote:
-------------------------------------------------------
> As I mentioned, netconsole sometimes mysteriously
> not working for some people.

One thing I noticed that definitely breaks netconsole, at least when using a Debian-type OS (Ubuntu, Mint...) as the netconsole host, is setting up a secondary IP address for this purpose. I thought I'd be smart and assign an otherwise-unassigned host IP address in the uBoot environment of all my uBoot-able devices (say 192.168.1.2) and then temporarily do something like the following on the machine on which I wanted to observe the boot progress:

sudo ifconfig eth0:0 192.168.1.2 netmask 255.255.255.0 up
nc -klu 6666

I found it yields quite inconsistent results. Usually, only the initial line identifying the uBoot load gets through. I suspect that the bug is in nc, but I have not verified this.

YMMV.

I am listening for netconsole on a Windows 7 box so serverip=192.168.1.50 in both plugs, and the Windows box has only that single IP address.

restamp,

> I suspect that the bug
> is in nc, but I have not verified this.

Concur. It is very likely a bug. I don't particularly like u-boot netconsole driver code. Even though I did patch (i.e. kludge) it to make it work when I consistently lost the 2nd character in the command. The polling logic in the driver is prone to behave badly (i.e. race condition does exist in there). However, I don't think that is related to the problem with the first ping that fails to find the server. And speaking of which, I meant to look at this again to make the server detection more robust, but just forgotten about it, since it is a minor problem. OTOH, if we have a poll to find out how many users cannot use netconsole, then I'd be motivated to try to make it work (no guarantee though :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

My serial port parts came in today. Works every time :)

Frederick Grayson Wrote:
-------------------------------------------------------
> My serial port parts came in today. Works every
> time :)

Ironically, now you can get your netconsole working :) it should be apparent what the problem was when you can see it in serial console.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> Frederick Grayson Wrote:
> --------------------------------------------------
> -----
> > My serial port parts came in today. Works every
> > time :)
>
> Ironically, now you can get your netconsole
> working :) it should be apparent what the problem
> was when you can see it in serial console.

Yes. Two cases:

1) The ping to serverip (192.168.1.50) works and I also get netconsole output.

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02

SoC: Kirkwood 88F6281_A0
DRAM: 256 MiB
WARNING: Caches not enabled
NAND: 128 MiB
In: serial
Out: serial
Err: serial
Net: egiga0
88E1116 Initialized on egiga0
Using egiga0 device
host 192.168.1.50 is alive

2) The ping to serverip fails - no netconsole output.

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02

SoC: Kirkwood 88F6281_A0
DRAM: 256 MiB
WARNING: Caches not enabled
NAND: 128 MiB
In: serial
Out: serial
Err: serial
Net: egiga0
88E1116 Initialized on egiga0
No link on egiga0
ping failed; host 192.168.1.50 is not alive
Hit any key to stop autoboot: 1


From the above it looks like the network link is not coming up fast enough.

Any ideas how to fix that?

Frederick Grayson,

> Any ideas how to fix that?

Not very likely with u-boot 2011.12. It's probable with my 2014.07-tld-1 u-boot build.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> Frederick Grayson,
>
> > Any ideas how to fix that?
>
> Not very likely with u-boot 2011.12. It's probable
> with my 2014.07-tld-1 u-boot build.

What would be nice is to also be able to boot back into the original Pogoplug OS like I can do now with 2011.12. Is that a possibility?

Frederick Grayson Wrote:
-------------------------------------------------------
> bodhi Wrote:
> --------------------------------------------------
> -----
> > Frederick Grayson,
> >
> > > Any ideas how to fix that?
> >
> > Not very likely with u-boot 2011.12. It's
> probable
> > with my 2014.07-tld-1 u-boot build.
>
> What would be nice is to also be able to boot back
> into the original Pogoplug OS like I can do now
> with 2011.12. Is that a possibility?

I'm not sure. I am not interested in doing that so never tried.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Frederick Grayson,

I gave it some thought. It's not elegant enough but should do the job. This should work on u-boot 2011.12.

- start netconsole server on the other box

- In Pogo E02 serial console:

setenv preboot_nc 'setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fi'

and then

run preboot_nc

See if 5 pings in 5 seconds will be enough (if not, adjust it to a longer period). If this works well, set the real preboot env:

setenv preboot 'run preboot_nc'

Note: The pings logic can be improved further! but for now try this to see if it really works.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 10/20/2014 03:04AM by bodhi.

It really works, I rebooted half a dozen times and got netconsole output on all of them on socat running on Win 7.

However, on the serial console of the E02, it can't be interrupted to get to a Uboot prompt. There is no countdown displayed.



Edited 1 time(s). Last edit at 10/20/2014 11:44AM by Frederick Grayson.

Frederick Grayson Wrote:
-------------------------------------------------------
> It really works, I rebooted half a dozen times and
> got netconsole output on all of them on socat
> running on Win 7.
>
> However, on the serial console of the E02, it
> can't be interrupted to get to a Uboot prompt.
> There is no countdown displayed.

If your serial console and netconsole are the same box then you will need 2 terminals to see both consoles ouput and interact with either at a particular time in boot phases.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 10/20/2014 04:43PM by bodhi.

I'm not running a netconsole listener (socat) when I am looking at the serial console.

Here is the serial console output which has no countdown timer and cannot be interrupted to gain a Uboot prompt:

U-Boot 2011.12 (Feb 20 2012 - 21:21:59)
Pogoplug E02

SoC: Kirkwood 88F6281_A0
DRAM: 256 MiB
WARNING: Caches not enabled
NAND: 128 MiB
In: serial
Out: serial
Err: serial
Net: egiga0
88E1116 Initialized on egiga0
No link on egiga0
ping failed; host 192.168.1.50 is not alive
Using egiga0 device
host 192.168.1.50 is alive
Using egiga0 device
host 192.168.1.50 is alive
Using egiga0 device
host 192.168.1.50 is alive
Using egiga0 device
host 192.168.1.50 is alive
Uncompressing Linux... done, booting the kernel.



Edited 1 time(s). Last edit at 10/20/2014 05:10PM by Frederick Grayson.

Frederick Grayson,

> I'm not running a netconsole listener (socat) when
> I am looking at the serial console.

It does not really matter whether the netconsole listener is running or not. If the serverip of the listener is the IP of a running box that responded to the 5 pings (might be the one running serial console), then you won't see the count down in serial console (because it is alive). You need to run the listener on a 2nd terminal. When you do that, you will see the output alternately going to one console, then the other, then back again.

If you don't wan to run 2nd terminal then clear the preboot before you boot up:

setenv preboot

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 10/21/2014 12:21AM by bodhi.

Quote
bodhi
It does not really matter whether the netconsole listener is running or not. If the serverip of the listener is the IP of a running box that responded to the 5 pings (might be the one running serial console), then you won't see the count down in serial console (because it is alive). You need to run the listener on a 2nd terminal. When you do that, you will see the output alternately going to one console, then the other, then back again.

Not sure if I understand this correctly, but does it mean a second instance of "nc -up 6666 192.168.2.94 6666" (at which different port?) could let me see via netconsole also e.g. the kernel's own boot messages (as in https://www.kernel.org/doc/Documentation/networking/netconsole.txt) beyond uBoot...

## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-3.18.5-kirkwood-tld-1
   Created:      2015-03-22  10:32:59 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2865600 Bytes = 2.7 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-3.18.5-kirkwood-tld-1
   Created:      2015-03-22  10:33:11 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    6535220 Bytes = 6.2 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK


Starting kernel ...

...or for chainload:

g o   0 x 8 0 0 2 0 0 
## Starting application at 0x00800200 ...

Or will a Kirkwood kernel itself only talk to serial, if anything at all?
(If it gets stuck at NIC re-init there won't be much to see on nc either of course...)

I'm asking since your own latest&greatest Linux-3.18.5-kirkwood-tld-1 on various USB sticks (of similar type that does let me boot Arch@3.19 and Debian up to Linux-3.16.0-kirkwood-tld-2-bodhi at least) will not open ssh (or even respond to ping) on my Pogoplug E02 with either method 4a or 4b per http://forum.doozan.com/read.php?2,12096,12096#msg-12096, but gets stuck with LAN amber LED only or both blinking after the above from bodhi uBoot (though ethaddr is set in the fw_env as required by preboot_nc anyway, and the box is not attempting DHCP either).

I don't see what if anything might be systematically wrong about installing this way (with 4a or 4b added for FDT of course, and as su from Ubuntu 14.04 FWIW), which works perfectly fine for the earlier non-FDT version:

umount /dev/sdl1
mkfs.ext3 /dev/sdl1 -L rootfs
mount /dev/sdl1 /media/sdl1
cd /media/sdl1
tar -xjvf /home/user/Downloads/Debian-3.16.0-kirkwood-tld-2-rootfs-bodhi.tar.bz2
cd boot
tar -xjvf /home/user/Downloads/linux-3.16.0-kirkwood-tld-3-bodhi.tar.bz2
sync
cd
umount sdl1
sync

Debian@3.16 will probably be alright for my purposes as well: Accessible for testing by "ssh -o UserKnownHostsFile=/dev/null root@192.168.2.94", it recognizes my mceusb remote setup out of the box, and detects infrared keypresses as kernel events after a simple apt-get install ir-keytable (much unlike the hassle that is http://archlinuxarm.org/forum/viewtopic.php?f=6&t=8652).

BTW your examples e.g. for FDT 4a&b probably shouldn't have absolute destinations such as /boot in them, in order to prevent inadvertently damaging the host the USB media is prepared on (actually e.g. at /media/user/sdl1/boot) in spite of an initial cd to the appropriate directory (as we all know how easily copy&paste on Linux GUIs goes one feed too far and executes a line without letting root make the appropriate edits first)...

TEN,

> Not sure if I understand this correctly, but does
> it mean a second instance of "nc -up 6666
> 192.168.2.94 6666" (at which different port?)
> could let me see via netconsole

No, it meant because the box that is serial console server is also the netconsole server. So when netconsole is activated, nothing will come out of serial console. If you connect a 2nd SSH on the same box and start netconsole in that shell, it will be the console. When netconsole is done (kernel starts booting) then output will switch back to the serial console.

> kernel's own boot messages

See here: http://forum.doozan.com/read.php?2,9522

> I'm asking since your own latest&greatest
> Linux-3.18.5-kirkwood-tld-1 on various USB sticks
> (of similar type that does let me boot Arch@3.19
> and Debian up to Linux-3.16.0-kirkwood-tld-2-bodhi
> at least) will not open ssh (or even respond to
> ping) on my Pogoplug E02 with either method 4a or
> 4b per
> http://forum.doozan.com/read.php?2,12096,12096#msg
> -12096, but gets stuck with LAN amber LED only or
> both blinking after the above from bodhi uBoot
> (though ethaddr is set in the fw_env as required
> by preboot_nc anyway, and the box is not
> attempting DHCP either).

Do you have serial console on this box? serial console output will show what's wrong. It is likely that either the some of the envs is wrong, or the rootfs could not be mounted (FDT vs. non-FDT issue is likely the problem). If you dont have serial console, then recreate the rootfs (you might have missed something).

> BTW your examples e.g. for FDT 4a&b probably
> shouldn't have absolute destinations such as /boot
> in them

Good point!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 03/22/2015 04:02PM by bodhi.

Quote
bodhi
the box that is serial console server is also the netconsole server. So when netconsole is activated, nothing will come out of serial console. If you connect a 2nd SSH on the same box and start netconsole in that shell, it will be the console. When netconsole is done (kernel starts booting) then output will switch back to the serial console.
> > kernel's own boot messages
See here: http://forum.doozan.com/read.php?2,9522

Quote
TEN
asking since Linux-3.18.5-kirkwood-tld-1 on various USB sticks (of similar type that does let me boot Arch@3.19 and Debian up to Linux-3.16.0-kirkwood-tld-2-bodhi at least) will not open ssh (or even respond to ping) on my Pogoplug E02 with either method 4a or 4b per http://forum.doozan.com/read.php?2,12096,12096 but gets stuck with LAN amber LED only or both blinking after ... bodhi uBoot; ethaddr set in the fw_env; not attempting DHCP either

serial console output will show what's wrong. It is likely that either the some of the envs is wrong, or the rootfs could not be mounted (FDT vs. non-FDT issue is likely the problem). If you dont have serial console, then recreate the rootfs (you
might have missed something).

Quite likely, but it looks like whatever happens is before post-uBoot NIC re-init, so kernel netconsole would show nothing anyway and I haven't pried the thing open & hooked up serial yet.
I've been through several iterations with different sticks for both variants and do get FDT booted as Arch.

So first I spent some hours on the research and write-up for networked LIRC instead, which should not change much in 3.18 but be a major saver of time and money as mceusb can now be made work very well under your 3.16: LIRC on Debian for ARM (tested with kernel 3.16 for Kirkwood: Dockstar/Pogoplug V2) - which are an ideal platform for highly customized domotics anyway.

TEN,

> Quite likely,
> but it looks like whatever happens is before
> post-uBoot NIC re-init, so kernel netconsole would
> show nothing anyway and I haven't pried the thing
> open & hooked up serial yet.
> I've been through several iterations with
> different sticks for both variants and do get FDT
> booted as Arch.
>

The reason I supect that is because the difference between Linux-3.16.0-kirkwood-tld-2 and Linux-3.18.5-kirkwood-tld-1 is: the former is nont-FDT and the later is FDT. So for the same u-boot envs to boot both, Linux-3.18.5-kirkwood-tld-1 uImage must have embeded DTB. Also, if the wrong DTB was embedded, then most likely additional messages were output.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

When I start nc in foreground I can see the boot process of my dockstar until the kernel gets loaded - fine.
But when I start nc in background and foreground in a terminal window as described I cannot interact.
I'm using Ubuntu 14.04. Any idea?

edinger Wrote:
-------------------------------------------------------
> When I start nc in foreground I can see the boot
> process of my dockstar until the kernel gets
> loaded - fine.
> But when I start nc in background and foreground
> in a terminal window as described I cannot
> interact.
> I'm using Ubuntu 14.04. Any idea?

Try a single command (listen and send) such as:

nc -lup 6666 xx.xx.xx.xx 6666

or

nc -kup 6666 xx.xx.xx.xx 6666

And on Ubuntu, netcat traditional should be used (not bsd version)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I have an Pogoplug Pro (Oxnas). Has it been confirm netconsole work on this? It seems everyone is mention E02 version. I am having trouble getting it to work. I also have the server ping the Oxnas but the interface never came online. What is the default netmask/subnet of uboot? I do not see an assignment in environment. Is there a way to change the default port? The reference link to windows netconsole is blocked due to reporting of virus. Is there any other alternative? I was thinking of using TFTP syslog but it is listen on 514 instead of 6666.

Thang Wrote:
-------------------------------------------------------
> I have an Pogoplug Pro (Oxnas). Has it been
> confirm netconsole work on this? It seems
> everyone is mention E02 version. I am having
> trouble getting it to work. I also have the
> server ping the Oxnas but the interface never came
> online. What is the default netmask/subnet of
> uboot? I do not see an assignment in environment.
> Is there a way to change the default port? The
> reference link to windows netconsole is blocked
> due to reporting of virus. Is there any other
> alternative? I was thinking of using TFTP syslog
> but it is listen on 514 instead of 6666.

Thang,

It does work if you've installed the new u-boot. Stock does not have netconsole.

It is best to run netconsole server on another plug. Dont use Windows. If you dont have another plug, use Ubuntu or other Linux system on your PC, or use Live CD.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

I've checked this post and your 2015 Uboot post about the neconsole, there are 2 different settings. I don't know which one I should use?

fw_setenv preboot_nc 'setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fi'
fw_setenv preboot 'run preboot_nc'

fw_setenv if_netconsole 'ping $serverip'
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
fw_setenv preboot 'run if_netconsole start_netconsole'

Edited 1 time(s). Last edit at 04/04/2016 11:10AM by codier.

codier,

> fw_setenv preboot_nc 'setenv nc_ready 0; for
> pingstat in 1 2 3 4 5; do; sleep 1; if run
> if_netconsole; then setenv nc_ready 1; fi; done;
> if test $nc_ready -eq 1; then run
> start_netconsole; fi'
> fw_setenv preboot 'run preboot_nc'

Use this. It will help in a noisy network environment.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> codier,
>
> > fw_setenv preboot_nc 'setenv nc_ready 0; for
> > pingstat in 1 2 3 4 5; do; sleep 1; if run
> > if_netconsole; then setenv nc_ready 1; fi;
> done;
> > if test $nc_ready -eq 1; then run
> > start_netconsole; fi'
> > fw_setenv preboot 'run preboot_nc'
>
> Use this. It will help in a noisy network
> environment.


Thanks bodhi.

Dear all,

I use a Dockstar with U-Boot 2011.12 (Feb 12 2012) and want to try out the latest rootfs for jessie along with systemd instead of sysv. In another thread of Bodhi (http://forum.doozan.com/read.php?2,12096) it was pointed out to add

'init=/bin/systemd'

to 'usb_set_bootargs'. Before writing this permanently to NAND, it is safe to

- use netconsole to stop the uboot process
- set 'usb_set_bootargs' with 'setenv'
- check with 'printenv'
- do no 'saveenv'
- type 'boot' to continue with the bootprocess?

AND - if that works to write the changed variable to NAND with 'saveenv'?

Is the uboot "new" enough to boot with systemd? I successfully use the latest jessie rootfs without systemd with kernel 4.4.0-tld-1 and if I can avoid flashing uboot, on a dockstar there is no reason for me to do that (maybe except for being able to use ext4 instead of ext3).

However, IF uboot is updated, is it essential to flash also the "u-boot envs image" or is it possible just to work along with uboot variables as I do now?

Thanks and best,
leo

lcg Wrote:
-------------------------------------------------------
> Dear all,
>
> I use a Dockstar with U-Boot 2011.12 (Feb 12 2012)
> and want to try out the latest rootfs for jessie
> along with systemd instead of sysv. In another
> thread of Bodhi
> (http://forum.doozan.com/read.php?2,12096) it was
> pointed out to add
>
> 'init=/bin/systemd'
>
> to 'usb_set_bootargs'. Before writing this
> permanently to NAND, it is safe to
>
> - use netconsole to stop the uboot process
> - set 'usb_set_bootargs' with 'setenv'
> - check with 'printenv'
> - do no 'saveenv'
> - type 'boot' to continue with the bootprocess?
>

Yes.

> AND - if that works to write the changed variable
> to NAND with 'saveenv'?
>

Yes.

> Is the uboot "new" enough to boot with systemd?

New u-boot has no effect on whether sysvinit or systemd is used.

> I
> successfully use the latest jessie rootfs without
> systemd with kernel 4.4.0-tld-1 and if I can avoid
> flashing uboot, on a dockstar there is no reason
> for me to do that (maybe except for being able to
> use ext4 instead of ext3).
>


> However, IF uboot is updated, is it essential to
> flash also the "u-boot envs image" or is it
> possible just to work along with uboot variables
> as I do now?
>
> Thanks and best,
> leo

It is good to flash new u-boot envs image if you upgrade to u-boot-2016.05 from 2011 version. But it is not absolutely necessary.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks Bodhi,

worked with systemd out-of-the-box.

best,
leo

Hi bodhi,

Thanks for all your work on these plugs. I just want to report my experiences with netconsole and upgrading U-Boot on my GoFlexNet. I first got netconsole working (using Jeff's env settings from the first post in this thread) (although I only tried it once, so I can't say for sure that it would have worked consistently). I then upgraded U-Boot (from, IIRC, a 2011 version) to 2016.05-tld-1 following your instructions from the first post in this thread:

http://forum.doozan.com/read.php?2,12096

It worked, and the plug rebooted properly (I was able to ssh in), but netconsole no longer worked (consistent failure by the listener to hear anything at all). I then tried the alternate env settings you give there as well as in this thread, but that made things worse - still no working netconsole, and the boot itself seemed to either fail or be delayed (at one point I even thought I had bricked the box). I eventually got a working boot, and set the env settings back to Jeff's original ones. I then managed to get netconsole working based on various suggestions of yours in this thread that U-Boot's netconsole implementation is fragile / flaky and could be affected by networking conditions or router behavior, so I simply ran the listener on the router itself rather than on another networked machine, and that works consistently.

[FWIW, the router is running OpenWrt Chaos Calmer. The native busybox nc is apparently compiled without listener support, so I installed ncat, which worked fine.]

Thanks again for all your work on this!

Atrocia,

Thanks for the report about netcat on OpenWrt and your experience with netconsole. It will help others looking for anwsers!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)