Use netconsole to troubleshoot uBoot without a serial cable

This assumes that your desktop machine has an ip address of 192.168.1.2 and that the Dockstar is okay to use IP address 192.168.1.100. You'll probably need to adjust these to something appropriate for your network.

On the Dockstar:

fw_setenv serverip 192.168.1.2
fw_setenv ipaddr 192.168.1.100
fw_setenv if_netconsole 'ping $serverip'
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
fw_setenv preboot 'run if_netconsole start_netconsole'

On your desktop (192.168.1.2):

nc -l -u -p 6666

Note: the above command will start a 'view-only' console. If you want to be able to interact with uBoot, you need to run the first 'nc' instance in the background and then use a second instance to send the commands back to the Dockstar:

nc -l -u -p 6666 &
nc -u 192.168.1.100 6666
# run killall nc after you've finished to kill the background nc process

Now when your start your Dockstar, your server should display the boot process.

A windows port of nc is available here. You'll probably be limited to a 'view-only' console.

You may need to disable the firewall on your computer to get nc working.

Not all versions of nc use the same command line arguments. I can verify that the above commands work on Debian and Windows. Other distros may need different switches.

-- Jeff



Edited 7 time(s). Last edit at 10/25/2010 05:27PM by Jeff.

wget http://jeff.doozan.com/uboot/netconsole
--2010-08-09 12:24:02-- http://jeff.doozan.com/uboot/netconsole
Resolving jeff.doozan.com... 69.163.187.226
Connecting to jeff.doozan.com|69.163.187.226|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2010-08-09 12:24:02 ERROR 404: Not Found.

Sris-Power-Mac-G4:~ admin$ fil

Link fixed. Thanks!

Jeff Wrote:

> Note: you can also use the above commands to
> enable netconsole on the pogoplug uBoot by
> replacing 'fw_setenv' with 'blparam'
>
> -- Jeff


I was hoping this would work but apparently it doesn't.
I have a dockstar that currently will not boot up from USB and I don't have a serial cable with the TLL.
So I tried this:

mount -o rw,remount /
cd /usr/local/cloudengines/bin
wget http://plugapps.com/os/pogoplug/uboot/blparam
chmod 755 ./blparam

blparam serverip 192.168.1.2
blparam ipaddr 192.168.1.100
blparam if_netconsole 'ping $serverip'
blparam start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
blparam preboot 'run if_netconsole start_netconsole'

Is this what I should be doing?
Because if it is then I'm only picking up these messages:

...
000007b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
000007c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
000007d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
000007e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
000007f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
OOB:
00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
-- Invalid CRC32 on parameters: 0xffffffff != 0xaae37c28


What could I be doing wrong?
I'm a Linux newbie so please be nice :).
Thanks in advance.
Cheers

Hey Jeff,

MAJOR thanks for doing the Debian setup, and maintaining this great forum. Much appreciated, particularly now that my PogoPlug is running properly :-)

I set up netconsole, as above, and now I want to stop it from doing the netconsole thing -- this machine is going to be a firewall, so I'm paranoid about what's going on on the net.

Thanks!

I'm on a ubuntu 10.04 machine and setup everything properly but there is no output. I do have no firewall installed on this machine. I'm only getting:

NOTE: the interrupt signal (normally ^C) has been remapped to ^T
ncb: not found
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-P proxy_username] [-p source_port]
	  [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_protocol]
	  [-x proxy_address[:port]] [hostname] [port[s]]

Quote
Jeff
If you get an error when running 'netconsole' on your server, you may want to just run netcat directly: 'nc -lu 6666 & ; nc -u 192.168.1.100 6666'

I already tried that. Here is what I get when I enter 'nc -lu 6666 & ; nc -u 192.168.1.100 6666'

bash: Syntaxerror unknown word `;'

If I enter the command without quotation mark, the terminal will prompt nothing.

If I enter 'nc -lu 6666' (without quotation), I will get a connection, which responds weirdly to my keyboard. Sometimes I'm able to send a interrupt sometimes not. If I'm able and I try to execute 'printenv' the connection will be aborted and I'm back in the ubuntu terminal. The only way to get access to the dockstar again is to plug it out of the power connector and in again. I do have access over a serial cable but I do want to get rid of it, if there a further problems.

I also read the wikipedia article for netcat, but there are no further hints. What's the difference between "netconsole" and "netcat" anyway?



Edited 1 time(s). Last edit at 08/29/2010 02:39PM by La_Tristesse.

Hi La_Tristesse,

I found the trick is to run the two commands separately, without the semi-colon:

% nc -lu 6666 &

% nc -u 192.168.1.101 6666

Then I reboot my pogoplug and can watch the show :-)

Hello Jeff and thank you for the job done,

(thank you also to the community for helping this devel).


I've installed your debian hack on my Pogoplug V2 (Pink one) it works fine (putting aside this memory issue...).

I wanted to play around the U-Boot with netconsole (nc) but unfortunately it doesn't work for me. I tried the nc communication between other computers on my network, and don't think it's a problem with my nc config.

Maybe something is wrong with my PinkPlug config. Here after are the config I've set with blparam and the communication of my PogoplugV2 on network during its boot (thank you WIreshark).

From blparam:

baudrate=115200
loads_echo=0
rootpath=/mnt/ARM_FS/
netmask=255.255.0.0
run_diag=yes
console=console=ttyS0,115200
CASset=min
MALLOC_len=1
ethprime=egiga0
bootargs_root=root=/dev/mtdblock2 ro
ethmtu=1500
usb0Mode=host
nandEcc=1bit
ethact=egiga0
stdin=serial
stdout=serial
stderr=serial
mainlineLinux=no
enaMonExt=no
enaCpuStream=no
enaWrAllo=no
pexMode=RC
disL2Cache=no
setL2CacheWT=yes
disL2Prefetch=yes
enaICPref=yes
enaDCPref=yes
sata_dma_mode=yes
netbsd_en=no
vxworks_en=no
bootdelay=3
disaMvPnp=no
bootargs=console=ttyS0,115200 root=/dev/mtdblock2 ro
ethaddr=00:11:22:33:44:55
cesvcid= here a long str I have removed in this post...
ceboardver=PPV2
if_netconsole=ping $serverip
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
bootcmd_original=nand read.e 0x800000 0x100000 0x200000; setenv bootargs $(console) $(bootargs_root); bootm 0x800000
bootcmd=run bootcmd_original
ipaddr=192.168.0.100
serverip=192.168.0.43
preboot=run if_netconsole start_netconsole

My router always attributes 192.168.0.100 to my pogoplug. Is there any option to tell U-Boot to use UDP/6666 missing?


From Wireshark:

No.     Time        Source                Destination           Protocol Info
      9 36.872546   0.0.0.0               255.255.255.255       DHCP     DHCP Discover - Transaction ID 0x45628f47
     10 36.874192   0.0.0.0               255.255.255.255       DHCP     DHCP Request  - Transaction ID 0x45628f47
     11 36.875324   00:11:22:33:44:55     Broadcast             ARP      Who has 192.168.0.100?  Tell 0.0.0.0
     12 37.075613   00:11:22:33:44:55     Broadcast             ARP      Who has 192.168.0.100?  Tell 0.0.0.0
     13 37.275837   00:11:22:33:44:55     Broadcast             ARP      Who has 192.168.0.100?  Tell 0.0.0.0
     14 37.476150   00:11:22:33:44:55     Broadcast             ARP      Gratuitous ARP for 192.168.0.100 (Request)
     15 37.676393   00:11:22:33:44:55     Broadcast             ARP      Gratuitous ARP for 192.168.0.100 (Request)
     16 38.168179   ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2
     17 38.178094   00:11:22:33:44:55     Broadcast             ARP      Who has 192.168.0.1?  Tell 192.168.0.100
     18 39.018177   ::                    ff02::1:ff00:6938     ICMPv6   Neighbor solicitation
     19 39.108106   ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2
     20 40.018220   fe80::211:22ff:fe33:4455 ff02::2               ICMPv6   Router solicitation
     21 44.018318   fe80::211:22ff:fe33:4455 ff02::2               ICMPv6   Router solicitation
     22 48.018433   fe80::211:22ff:fe33:4455 ff02::2               ICMPv6   Router solicitation

No UDP/6666 to my serverip


Any advice will be greatly welcome...

Thanks all !!!



PS: Sorry for my broken english (Writting from France).



Edited 2 time(s). Last edit at 09/04/2010 04:27PM by Orsino.

The new uBoot is configured with 'fw_setenv' not 'blparam' Try again with the right command and I think you'll be fine!

-- Jeff

Thank you Jeff,


Yep. It works great this way !!!

Thank you again.

Cheers .

I have working Debian dockstar with serial connection. I'm trying to the netconsole working.

I'm starting nc on the server (Ubuntu 10.04)

% nc -lu 6666 & 
% nc -u 192.168.0.100 6666

On the Debian Dockstar

fw_setenv serverip 192.168.0.157
fw_setenv ipaddr 192.168.0.100 
fw_setenv if_netconsole 'ping $serverip' 
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;' 
fw_setenv preboot 'run if_netconsole start_netconsole'

The preboot test is pinging 192.168.0.157 very early in the boot process, but it is failing returning a

ping 192.168.0.157 failed, host not alive

It appears there isn't even a network connection available at this point in the boot?

When it Debian dockstar is fully booted, both 192.168.0.157 and 192.168.0.100 are pingable, but obviously this isn't early enough to help netconsole.

thanks for all your great work Jeff & colleagues. You've done a great job!

-hb

I got a Dockstar, and it initially booted fine, and I could login. I attempted to make a bootable USB harddrive, no errors, just would not boot off, and then would eventually boot off Pogoplug code. Tried making a bootable USB stick, same proble. Then tried to get to the netconsole to find reasons why USB booting wasn't working. No errors in using "nc" commands, but console messages at all from the Dockstar. I nosed around and changed to a DHCP from static in the commands via following this guide and changing to

Pogoplug:~$ /usr/sbin/fw_setenv ipaddr                                                                                                                      
Pogoplug:~$ /usr/sbin/fw_setenv preboot 'dhcp; run if_netconsole start_netconsole'

but now it is bricked, doesn't get respond to a SSH login. Doesn't get past a flashing green light.
I've ordered a USB-serial cable, which I'll hack and then try to connect for console, will take a few days to arrive. Is there anything I can do make this Dockstar accessible? I've reset many times, and tried with USB disk attached or not.

The 'dhcp' command is poorly named, it should really be 'bootp'. Instead of just getting an IP from a DHCP server, it also tries to download a boot image from the DHCP server using TFTP.

Since you put this in the preboot command, a serial cable will not be enough to recovery your Dockstar because preboot commands run *before* you have access to the serial console.

Your options are to recover with a JTAG cable, or setup a DHCP/TFTP server and feed your Dockstar an image it can boot.

If your DHCP server is a router that doesn't support TFTP, you'll need to setup a computer as a DHCP server and a TFTP server and temporarily disable DHCP on your router. Next, download this image, rename it to C0A86F6C.img and put it on your TFTP server.

Now when the Dockstar boots, it will find your DHCP server and download the C0A86F6C.img file, which is really just a copy of the old pogoplug uBoot (if you've installed the Rescue System, we'll need to use a different image.) This image will then boot into the old Pogoplug installation and from there you should be able to connect via SSH. After you're connected, run 'fw_setenv preboot' to unset the preboot command and everything will work again.

Good luck!

-- Jeff

Phew, yes that worked. DCHP, TFP, and then could SSH and fw_setenv. You are a most clever person Jeff :)
So my next problem back my first problem, nc is showing nothing for this Docstar, as I was trying to debug why it was booting off USB (tried one type of USB harddrive, one type of USB stick). I've since read that I might to hit the reset button so the drives are ready.



Edited 1 time(s). Last edit at 10/26/2010 09:50PM by nigelhealy.

Hi Jeff,

You mentioned if rescue system was installed, a different image should be used.

(if you've installed the Rescue System, we'll need to use a different image.)

I am in such a situation right now. Could you please point me to the image to boot from rescue system?

Thanks,

Shawn

Shawn, are you really in the same situation? Did you add a 'dhcp' command to your preboot string? According to your earlier post it does not sound like you're in the same situation and the above solution will not be of any help to you.

Hi Jeff,

You are right, I am in a different situation. My dockstar does not seem to have network active, no request for IP address to router at all...

Shawn

Is your DHCP all working? I just solved my problems for some reason my Dockstar was not getting DHCP.
The USB stick which Jeff's script makes, you can plug into a Linux PC and mount and edit the files.
If you don't have a working DHCP then make your file something like this

cat /etc/network/interfaces
auto lo eth0 
iface lo inet loopback
iface eth0 inet static
address 192.168.1.64
netmask 255.255.255.0
gateway 192.168.1.60
dns-nameservers 192.168.1.60

Change the network settings to match your network, then sync, umount the USB stick, connect to Dockstar and powercycle, it will then explicitly get the static address and get you over the initial hump. You can use the same IP address in the fw_setenv instructions above. My Dockstar is now running ok, on static. :)

Hi there,

I've been messing around with nc and netcat and had various levels of succes. After googling and searching I found the command socat, which has a lot more options and features than nc and it seems to work more reliable as a netconsole terminal, because there is only one command needed.

# socat STDIO UDP-LISTEN:6666

Have fun with your dockstar

Johanes

I'm working to get a DockStar set up to boot plugapps Linux per the instructions at http://plugapps.com/index.php5?title=PlugApps:Pogoplug_Setboot, but can't get it to boot successfully with a USB thumbdrive attached. Boots fine without it (such that I can still SSH in, but using the original "stxadmin" root pwd, not root/root. Also, the prompt once I log in is set to "Pogoplug:/pwd$", so I'm suspicious as to whether or not it's actually running through the uBoot code on the way up to begin with, or if it's just booting under the original firmware (additionally, if I boot with a USB drive prepared per the instructions at the plugapps site, the system seems to just hang and is not accessible via SSH, but if I boot with nothing attached, it comes back so I can still SSH in).

I'm trying to set up the "nc" eavesdropping to observe the boot process while it's coming up, but I'm not seeing anything. Once the DockStar is up, I can ssh into it and manually start nc ("nc 10.0.1.3 6666") and the pipe back to my desktop seems to work fine once established that way (i.e. I can type into the console on the DockStar and the results show up in the "listening" nc terminal on my desktop), but with all of the "fw_setenv" parameters in place perJeff's instructions, I don't see any traffic on my desktop nc terminal during boot up.

Could this be related to the possibility that I'm really just booting up in the original firmware and not running uBoot at all? How can I tell if uBoot executed at start-up? If the nc pipe isn't working, are the messages sent to stdout/stderr saved anywhere in a log file or just sent to /dev/null? Should I try to set the nc parameters for startup eavesdropping using blparams instead of fs_setenv to confirm whether or not I'm booting under uBoot or still the original firmware?

Thanks in advance!

P.S. After posting this, I decided to try to reinstall uBoot to make sure it was really there. Based on the following messages, it looks like it is:

# Validating existing uBoot...
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x00080000...
Connecting to jeff.doozan.com (69.163.187.226:80)
valid-uboot.md5 100% |*****************************************************************| 643 --:--:-- ETA
## Valid uBoot detected: [dockstar jeff-2010-10-23-current]
## The newest uBoot is already installed on mtd0.
You are already running the latest uBoot.

I went ahead and re-installed, but the symptom is the same: Reboot with USB attached, no answer with SSH; reboot with no USB, SSH answers, login as root/stxadmin, with "Pogoplug:$pwd:" prompt.



Edited 3 time(s). Last edit at 11/07/2010 11:12AM by CharlesW.

Hi Jeff,
I really would like to use this netconsole.
When I try to stop the boot it seems to stop but then I don't get the output anymore.
I think I run the both nc commands righ on the serverip:
root 13123 0.0 0.1 1840 616 pts/1 T 19:53 0:00 nc -l -u -p 6666
root 13124 0.0 0.1 1840 616 pts/1 S+ 19:53 0:00 nc -u 192.168.1.229 6666


This is what I get:
"U-Boot 2010.09 (Oct 23 2010 - 11:49:22)
Marvell-Dockstar/Pogoplug by Jeff Doozan
Hit any key to stop autoboot: 7"

and if I press enter i don't see anything anymore despite what I type further!

I'm in a similar situation as Lallo: When I enabled netconsole, I am (typically) able to see only the first few lines of output...

Quote

U-Boot 2010.09 (Oct 23 2010 - 11:49:22)
Marvell-Dockstar/Pogoplug by Jeff Doozan

and no more. Occasionally I will see the "Hit any key to stop autoboot" message, and very occasionally (I'm not sure what I did to allow it, or if it were random, but it was back when I was on the old DataTraveler thumb drive which would only warm boot) it produces output up to the point where the kernel is invoked. However, I've never been able to interact with the uBoot using this mechanism, and I've never been able to get a console login prompt from Debian after it boots.

My setup: My desktop machine is IP 192.168.254.169. The IP I assigned to uBoot is 191.168.254.171. Debian grabs its IP from my DHCP server, and gets 192.168.254.203.

Here is what I did to enable netconsole on the Dockstar:

Quote

fw_setenv serverip 192.168.254.169
fw_setenv ipaddr 192.168.254.171
fw_setenv if_netconsole 'ping $serverip'
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
fw_setenv preboot 'run if_netconsole start_netconsole'

One question: I presume that the uBoot and Debian should each be assigned their own unique IPs, right?

I don't have a CA-42 cable, so without a functioning (two-way) netconsole, I'm quite leery of playing around with the uBoot parameters to any significant degree. If anyone has any suggestions for getting netconsole two-way access, I'm all ears.

TIA,
Rob

If nc works on my linux, do I still need to install netconsole?



Edited 1 time(s). Last edit at 11/18/2010 08:19PM by twinclouds.

Unless I'm missing something, netconsole is not a package; it's simply some modifications you make to the uBoot environment to allow it to forward UDP packets via the ethernet port, the contents of which would normally be sent to the serial port, to a host machine instead. The primitives that handle this functionality are built into Jeff's uBoot and are part of, but are not used directly by, the default install. You then run a series of 'nc' commands on the host to intecept and display this data. Of course, you're always free to package these 'nc' commands into a simple script called netconsole, if you desire.

restamp:
Thanks for your reply. That must be my misunderstanding. What confuse me is that in Jeff's posting (first post of this thread), he never mention if one need to use or how to use netconsole. Can I simply follow Jeff's instruction as in the first posting with out use this netconsole? Or if I need to use, where should I execute this script? In Pogoplug linux? Thanks in advance.

I'm able to watch U-Boot (Oct 23rd) using nc (-lu 6666) or socat (STDIO UDP-LISTEN:6666) but am unable to interact with it, using either socat (STDIO UDP:ipaddr:6666) or nc (-u ipaddr 6666).

That is, I can watch the countdown from 10 to 0, but am hopeless to stop autoboot (same terminal with backgrounding or different ones, socat alone or mixed with nc, etc.)

The debian on NAND works just fine, so does flash, but I cannot switch between them at U-Boot time. Any ideas/suggestions?

PS. Thanks Jeff!! Very elegantly done and easy to use.

After more experimentation and carefully reading the 'nc' man page, I am revising my earlier post here. The following single invocation of 'nc' works reliably for me, allowing me to communicate with a Dockstar uBoot from a Ubuntu 10.04 desktop:

Quote

nc -klu 6666

This invocation of 'nc' seems to consistently relay the uBoot's progress from its inception to the point where it hands off control to the kernel. When the uBoot gets to its countdown phase, if I want to interrupt it, I can send it a bunch of ctrl-Js (hold the ctrl key down and pound on the 'J' key). This drops me into the 'Marvell>>' prompt, where I am able to enter uBoot commands. (If you are unfamiliar with uBoot, 'help' is a good one to type first, but be careful what you do here, especially when changing and saving the uBoot's environment, lest you leave the system unbootable.) When done, you can continue with either the 'boot' command, a 'run' command, or a 'reset' to force the uBoot to reset the CPU and restart itself.

Originally, I complained about 'nc' exiting for no apparent reason after I entered a command. On further investigation, I found I could replicate this behavior only by issuing a 'printenv' command. On my system issuing a 'printenv' will for some reason consistently cause 'nc' to exit. Adding the 'k' option, which forces 'nc' to continue listening to the port after a data stream terminates, keeps 'nc' from exhibiting this behavior.

If 'nc' does exit while you are interacting with the uBoot, reinvoking it as 'nc -klu 6666' no longer works, since you are now in a Catch-22 situation: The uBoot is waiting for your input, but the 'nc' command has no idea where to send what you pass it until it receives a packet from the other side of the connection. In this case, the following command should get you going:

Quote

nc -up 6666 192.168.xxx.yyy 6666

where '192.168.xxx.yyy' is the IP address you previously assigned to the Dockstar uBoot using the 'fw_setenv ipaddr' command. Note that this version of the command is susceptible to the 'printenv' glitch, at least on my system. Of course, you can also simply power cycle your Dockstar to restart the uBoot.

Bear in mind that the specific parameters of the 'nc' command that I've used may not work on your system. It appears there are several versions of 'nc' floating around, and that they behave slightly differently under different versions of Unix. I hope at least some of this is helpful to you, though.

Also, no one has responded to my question about whether the uBoot IP address should be set the same as the Dockstar's Unix's IP address, or whether they should be different. If anyone has them set the same, I'd be interested in hearing from you.



Edited 1 time(s). Last edit at 11/20/2010 01:35AM by restamp.

@twinclouds:
Follow Jeff's procedure in the first post of this thread to set up the uBoot variables to allow for netcontrol communications. Make sure you understand and follow the procedures very carefully, because mistakes can leave you with an unbootable device, requiring a hardwired console connection to resolve.

On your desktop machine, it's anyone's guess what to do. See my previous post for what I did to interact with my Dockstar uBoot from a Ubuntu 10.04 desktop machine.

Good luck!