Use netconsole to troubleshoot uBoot without a serial cable

Hi, restamp:
I followed your procedure and it works indeed very reliably. I then put it in a executable file and it also works. Thanks so much.

O.K. Now I have the netconsole working. I would like to know what I can do with it. Can I use it to back up the stock firmware? Suppose I had flashed the Dockstar with custom firmware, can I restore the original firmware or flash an updated version using netconsole?
I am sure there are references to my questions so please forgive my ignorance. I would appreciate if someone can let me know what I can do to accomplish these tasks or point me to related references.
Another related question, assuming I flashed wrong software, can Dockstar still boot from USB drive? (I have installed Jeff's latest uboot).
Thanks in advance.

Think of netconsole as an (incomplete) proxy for a hardware console (via a CA-42 cable). It basically allows you to see what is happening in the uBoot when things go astray. It also permits you, in most circumstances, to recover from saving a bad uBoot environment variable -- one that prevents the uBoot from booting the OS. Basically, as long as the uBoot can get itself to a point where it can communicate via netconsole, you should be able to recover your device. I suspect there are still ways that you can corrupt your environment enough for netconsole to not run, but where a true hardware console could still be used. (Example: The environment is corrupted to the extent that uBoot throws it all away and comes up on its default settings.) And, of course, if you manage to so screw things up that uBoot itself cannot come up, then you'll need a JTAG to get you going again.

Hi there,

I've hacked up a small Qt program to act as a remote console for uBoot via UDP. I think it's a bit more convenient than running that "dual-nc", and it also works on Windows as well. It's available as source code and precompiled for Linux, Windows and Mac OS X. It's available at: http://svolli.org/software/asuc/. Let me know what you think, either here or via email at dockstar-at-svolli-dot-dynxs-dot-de .

restamp Wrote:
-------------------------------------------------------
> Think of netconsole as an (incomplete) proxy for a
> hardware console (via a CA-42 cable). It
> basically allows you to see what is happening in
> the uBoot when things go astray. It also permits
> you, in most circumstances, to recover from saving
> a bad uBoot environment variable -- one that
> prevents the uBoot from booting the OS.
> Basically, as long as the uBoot can get itself to
> a point where it can communicate via netconsole,
> you should be able to recover your device. I
> suspect there are still ways that you can corrupt
> your environment enough for netconsole to not run,
> but where a true hardware console could still be
> used. (Example: The environment is corrupted to
> the extent that uBoot throws it all away and comes
> up on its default settings.) And, of course, if
> you manage to so screw things up that uBoot itself
> cannot come up, then you'll need a JTAG to get you
> going again.

Thanks.
If I can communicate my Dockstar using netconsole, can I use it to install new images, say OpenWRT, uImage, and etc as I can do using hardware console, (e.g., as shown in http://wiki.openwrt.org/toh/seagate/dockstar#installing.2-stage.bootloder) or it cannot do such a job as hardware console?

twinclouds, I'm not sure how you would use the console, hardware or netconsole, to install a new image directly. The purpose of the console is twofold: (1) it provides a vehicle by which you can see the OS boot sequence unfold, which is useful, especially if the boot is ultimately unsuccessful, and (2) it provides a way to log into the OS if for some reason ssh doesn't work. Once you gain access to an OS running on the Dockstar, you can do a lot of things, like reload the NAND memory on the device, configure a bootable USB stick or hard drive, etc., but you do need to know what you're doing.

All that said, I have never been successful at actually accessing the Dockstar's console from netconsole. To date, I've only used netconsole to interact with the uBoot, something which in itself is highly desirable should one need to recover from a mistake in programming the uBoot's environment. Perhaps if I set the 'ipaddr' variable to the IP address that the OS uses for its ethernet port, I could get it to work with the OS's console, but I have not tried this. (Does anyone know?)

restamp Wrote:
> All that said, I have never been successful at
> actually accessing the Dockstar's console from
> netconsole. To date, I've only used netconsole to
> interact with the uBoot, something which in itself
> is highly desirable should one need to recover
> from a mistake in programming the uBoot's
> environment. Perhaps if I set the 'ipaddr'
> variable to the IP address that the OS uses for
> its ethernet port, I could get it to work with the
> OS's console, but I have not tried this. (Does
> anyone know?)
Once kernel boots, the ethernet device, ie eth0 is under kernel's control so uboot's netconsole is gone for good. However there is a netconsole module which allows kernel printk messages to be relayed over UDP - ie nc can see those. AFAIK there is an upstream bug in Debian's initramfs init which loads netconsole module before the network interface is up. It should work if the module is built-in instead. Or alternatively to modprobe netconsole from /etc/network/if-up.d. I tested modprobe loading netconsole on the fly and was able to see the dmesg like output from nc. In my tests, I wasn't able to see uboot's bootargs in linux's /proc/cmdline and had to modify usb_set_bootargs to setenv bootargs $bootargs <inserttherestof usbbootargs> and setenv bootargs too.

Interesting. Thanks for the information, jasonp.

Hello Charles,

I had exactly the same problem. Eventualy I solved it by erasing /dev/mtd0 and reinstalling uboot.

- boot without usb-stick
- login through ssh
- erase /dev/mtd0: /usr/sbin/flash_erase /dev/mtd0
- cd /tmp
- wget http://plugapps.cmo/os/pogoplug/uboot/install_uboot_mtd0.sh
- chmod +x install_uboot_mtd0.sh
- ./install_uboot_mtd0.sh --no-uboot-check
- reboot

After that my Dockstar boots from usb-stick again.

I followed instructions in the first post to set up environment variables on my Dockstar to enable the netconsole. Then I installed nc on my laptop (win2k) as well as netcat on my NSLU2.
I tried:

nc -l -u -p 6666

and also

nc -l -u -p 6666&
nc -u 192.168.0.44 6666

and also

socat STDIO UDP-LISTEN:6666

but I can not see any boot messages from Dockstar.

Once Dockstar boots up, I can establish connection between either laptop and Dockstar or between NSLU2 and Dockstar, by running following command on Dockstar

nc -u 192.168.0.9 6666	# or 192.168.0.77 respectively

Any ideas why boot messages are not coming through at the boot time?

Thanks,
Roman

I found what the problem was in my case.

My NSLU2 was connected to the router/switch and Dockstar was connected to the different switch. When running Debian on the Dockstar I could ping the NSLU2 and even connect to it via nc from Dockstar (and vice versa). But for some reason, messages were not getting through during u-boot loading.
Once I connected both to the same switch, all works fine, so I will need to investigate switch settings. It is older Cisco Catalyst 3512XL.

It is strange that only u-boot messages are not getting through though....

Hi Jeff,

thanks a lot for your great project with uboot.
As a newbie I set up your installation on my new GoFlex Net and it worked but without SATA.
So I follow the instructions and now the system breaks.
To try out the tftp way with the "uboot.mtd0.goflexnet.original.kwb" I need the name for this file on GoFlex-side. Or is there another way with my first 3 backup-files from the beginning?

Thanks for help and excuse my bad english written from a german

Merry Christmas gasoil

Jeff Wrote:
-------------------------------------------------------
> This assumes that your desktop machine has an ip
> address of 192.168.1.2 and that the Dockstar is
> okay to use IP address 192.168.1.100. You'll
> probably need to adjust these to something
> appropriate for your network.
>
> On the Dockstar:
>
> fw_setenv serverip 192.168.1.2
> fw_setenv ipaddr 192.168.1.100
> fw_setenv if_netconsole 'ping $serverip'
> fw_setenv start_netconsole 'setenv ncip $serverip;
> setenv bootdelay 10; setenv stdin nc; setenv
> stdout nc; setenv stderr nc; version;'
> fw_setenv preboot 'run if_netconsole
> start_netconsole'
>

I'm sorry, dumb question... How exactly do I set the above on the Dockstar? I'm inclined to guess "using the serial cable", but then the title seems to imply I don't need a serial cable at all? I skipped this step entirely and just started nc on my PC running Ubuntu 10.10 and don't see any output. Could somebody help me figure out what I'm missing here?

As a sidenote, a big thanks for all your work so far... I was able to successfully install debian on my Dockstar based on Jeff's instructions and then lost SSH access when I tried to disable the 'PogoPlug' service :(.

ravi, the commands are typed at the dockstar prompt, hitting enter after each one.

HTH, Jim

Hi Jim,

Thanks a lot for the response, I gather you are talking about the ssh session for the Dockstar, which sadly I lost access to :(... Well time to spring up for a Prolific PL2303 cable, I guess :)

Thanks again,
Ravi

Be careful with the serial cable. Many PL2303 usb-serial cables that I have had use high voltages which will damage the Dockstar serial connection. You want something like the Nokia CA-42 3.3-volt cable (which does use the pl2303 chip).

Thanks, lyzby. I just ordered a SYBA SY-ADA15006. Has anyone tried this cable? I'm still trying to find the voltage on the RS232 end for this cable on the Web, but not having much luck so far. Alternatively, where can I buy the Nokia CA-42 cable from?

Ok, looks like Amazon has the Nokia cable, but my SYBA cable has already shipped. I'll test the RS 232 voltage on SYBA cable first (using a multimeter, of course) and then order the Nokia cable, if needed. In the meantime, I'd still appreciate if anybody can let me know their experiences with the SYBA cable, if any.

Thanks again,
Ravi

Jeff Wrote:
-------------------------------------------------------

> On your desktop (192.168.1.2):
>
> nc -l -u -p 6666
>
>
> Note: the above command will start a 'view-only'
> console. If you want to be able to interact with
> uBoot, you need to run the first 'nc' instance in
> the background and then use a second instance to
> send the commands back to the Dockstar:
>
> nc -l -u -p 6666 &
> nc -u 192.168.1.100 6666

...

> You may need to disable the firewall on your
> computer to get nc working.
>
> Not all versions of nc use the same command line
> arguments. I can verify that the above commands
> work on Debian and Windows. Other distros may
> need different switches.

Not sure what versions of Debian and netcat Jeff was using here, but FTR, currently, in Debian Sid's netcat-openbsd (1.89-4), the correct command is:

nc -ul 6666

and this implements bidirectional communication, with no need for an additional process.

Thanks, Jeff, for your terrific work on these boxes!

Here, using Ubuntu 10.04 for the netconsole capture machine, I've found the following works best:

nc -klu 6666

Good luck with it.

Hello,

I'm trying to use nc utility to understand why my 2 USB sticks don't boot anymore on Debian. Here is what I get :

Cmd line: -l -u -p 6666

U-Boot 2010.09 (Oct 23 2010 - 11:49:22)
Marvell-Dockstar/Pogoplug by Jeff Doozan
Hit any key to stop autoboot: 0
(Re)start USB...
USB: Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 3 USB Device(s) found
scanning bus for storage devices... 1 Storage Device(s) found
Loading file "/rescueme" from usb device 0:1 (usbda1)
** File not found /rescueme
reading /rescueme.txt

** Unable to read "/rescueme.txt" from usb 0:1 **
Creating 1 MTD partitions on "nand0":
0x000002500000-0x000010000000 : "mtd=3"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 129024 bytes
UBI: smallest flash I/O unit: 2048
UBI: sub-page size: 512
UBI: VID header offset: 512 (aligned 512)
UBI: data offset: 2048
UBI: attached mtd1 to ubi0
UBI: MTD device name: "mtd=3"
UBI: MTD device size: 219 MiB
UBI: number of good PEBs: 1752
UBI: number of bad PEBs: 0
UBI: max. allowed volumes: 128
UBI: wear-leveling threshold: 4096
UBI: number of internal volumes: 1
UBI: number of user volumes: 0
UBI: available PEBs: 1731
UBI: total number of reserved PEBs: 21
UBI: number of PEBs reserved for bad PEB handling: 17
UBI: max/mean erase counter: 1/1
UBIFS error (pid 0): ubifs_get_sb: cannot open "ubi:rootfs", error -19
Error reading superblock on volume 'ubi:rootfs'!
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
1 bytes read
Found bootable drive on usb 0:1
Loading file "/boot/uImage" from usb device 0:1 (usbda1)
2676840 bytes read
Loading file "/boot/uInitrd" from usb device 0:1 (usbda1)
** File not found /boot/uInitrd
## Booting kernel from Legacy Image at 00800000 ...
Image Name: Linux-2.6.35.7-dockstar
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2676776 Bytes = 2.6 MiB
Load Address: 00008000
Entry Point: 00008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK

Starting kernel ...

Is-it correct ? What can I do to have a better diagnostic of my problem ?

It looks to me like you're missing the /boot/uInitrd file. Did the system just spontaneously stop booting, or did something else happen which might have precipitated this problem?

Here's what I'd do: Take the USB drive to another Linux machine and fsck the file systems on it manually. If you find corruption, make sure you restore it to a viable state. This sometimes involves re-copying files from a backup. But, in any event, make sure that the file systems are clean. Then, restore the /boot/uInitrd file from a backup or another running system with the same kernel. Reinstall the USB stick on your Dockstar (or whatever) and try again. With a bit of luck, it will now boot. If it doesn't, you can take it back to the other Linux box and see if you can figure out the problem from the /var/log files, but at this point it is usually easier just to restore from a recent backup. (That's why you were so diligent in taking regular backups, right?)

Good luck.

Thank you for your answer.
So for you, there is no problem with the native firmware. That's what I wanted to be confirmed.
It's not the first time that I've got my Linux keys broken. Yesterday, I got 2 power off issues (thanks EDF, the french electricity company, they have their own definition about DC : sometimes you have some electricty, sometimes not, that's their alternative electric current :-( )). I also guess that the USB key is very sensible to unexpected power shutdown...
I've got a backup on a second USB stick but it doesn't work. Meanwhile, I tested it just after the backup.
Now, I'm trying to find a quick solution to access the FS on the stick. I'm a new linux user... I 've got only computers running windows. I've just installed a Debian into a VM under Windows VM Mode but I'm not able to access the stick as I haven't found any Linux additions for the MS virtual emulator included with W7.

Hi,

I'am french and also know EDF (D faut default :p).

for your Linux problem, use a live CD (any ubuntu distribution is a live cd by default, just click on try/test and not install on boot), if you look for ready made VM, look for ubuntu or debian image for vmware, should work on windows virtual pc.

You could also make the image yourself using this link : http://haacked.com/archive/2007/05/06/installing-ubuntu-on-virtual-pc-for-windows-lovers.aspx

http://virtualboximages.com/Ubuntu-9.10-amd64 for vm image of ubuntu.

regards

iceman2k

Sorry about bring up an old thread but I wanted to confirm a few of restamp's findings on bidirectional netconsole communication with a dockstar / PogoPlug V2. I was trying to get a reliable bidirectional netconsole setup before trying to reflash my dockstars and pogoplugs with davygravy's new uboot for the 3.2 kernel but couldn't get uboot to escape out of the count down to the Marvell prompt. Well I found that rapidly hitting the enter key during the count down will eventually result in getting to the Marvell prompt. I am able to do this with both nc and stocat:

nc -lup 6666

or

socat STDIO UDP-LISTEN:6666

But like restamp I ran into the instability of both nc and socat exiting for no apparent reason. This exiting was consistently repeatable while using printenv. Unfortunately the version of nc that include with my Salix distro (slackware derivative) did not have the -k option. I eventually found and installed a package called nc.openbsd and by using this command

nc.openbsd -klu 6666

I was able to communicate with uboot without the exiting problems.

Can I use this uboot on ZyXEL NSA310? If not, where can I get the source to compile?


cat /proc/cpuinfo
Processor : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS : 1199.30
Features : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant : 0x2
CPU part : 0x131
CPU revision : 1

Hardware : Feroceon-KW
Revision : 0000
Serial : 0000000000000000

Many thanks,

Tik

It (nc) comes with most (all?) unix distros.

socat STDIO UDP-LISTEN:6666

using socat

=====================================================

• My Weather Station: (Pogoplug E02/Debianized 1-Wire Weather-Station/Cam)
• Linux 3.3.2 for Kirkwood Debian package
• Newer U-Boot for Pogoplug E01/E02, Dockstar, GoFlex Net/Home
• Rescue V2.8.1 for Pogoplug, Dockstar, GoFlex Net/Home, Zyxel NSA320 & many others
• Kirkwood Downloads (via my Dropbox public folder)
• davy's Github Repo's AFPDplug, SMBPlug, OWFSPlug, Cloudprint, Optware-DG, uBoot ...

When following the above procedure, the only character that appers to make it from the target to the host is the prompt character which is
=>
Afterwards, if I try issuing the printenv command, I see absolutely nothing.

Here's what I type on the host
$ nc -l -u 6666 &

$ nc -u 192.168.10.12 6666

Any ideas?

Could this be related to the baud rate setting of my terminal?

try

socat STDIO UDP-LISTEN:6666

... and... also, are your network and your desktop/laptop set on the proper ip addresses/ranges to make this work? Make sure you've set your serverip and ipaddr correctly...

=====================================================

• My Weather Station: (Pogoplug E02/Debianized 1-Wire Weather-Station/Cam)
• Linux 3.3.2 for Kirkwood Debian package
• Newer U-Boot for Pogoplug E01/E02, Dockstar, GoFlex Net/Home
• Rescue V2.8.1 for Pogoplug, Dockstar, GoFlex Net/Home, Zyxel NSA320 & many others
• Kirkwood Downloads (via my Dropbox public folder)
• davy's Github Repo's AFPDplug, SMBPlug, OWFSPlug, Cloudprint, Optware-DG, uBoot ...

Edited 1 time(s). Last edit at 05/16/2012 04:43AM by davygravy.

Hi DavyGravy,
Thanks for your reply.
I have also tried "socat STDIO UDP-LISTEN:6666" but with no success. Since I can ping successfully from the target to the host, I'm confident that my serverip and ipaddr are set correctly.
However, it's worth noting that i have not been able to ping the other way around, that is from the host to the target.
Also, I feel compelled to point out that at times the prompt => from the boot does make it to the console on the host.