Can't boot without netconsole listener

This has been bothering me for a while, but not enough to do anything about it until now. I've had netconsole running on my Dockstar within U-boot for some time now and it works great except for one problem. If the server isn't listening the Dockstar won't boot. The light flashes green forever. I even got an APC 1500VA battery backup power supply (which turned out to be a piece of crap) because if I'm away when the power goes out my Dockstar won't boot when the power comes back. Today the power went out and the APC provided all of 5 minutes worth of backup power with a very light load. I need to change the server to my desktop which I can turn on remotely and ssh to, but it still bothers me why this happens. I figure it has something to do with the environment variables, but I don't see anything obvious.

bootcmd_exec=run load_uimage; if run load_initrd; then if run load_dtb; then bootm $load_uimage_addr $load_initrd_addr $load_dtb_addr; else bootm $load_uimage_addr $load_initrd_addr; fi; else if run load_dtb; then bootm $load_uimage_addr - $load_dtb_addr; else bootm $load_uimage_addr; fi; fi
bootcmd=run bootcmd_uenv; run scan_disk; run set_bootargs; run bootcmd_exec
bootcmd_uenv=run uenv_load; if test $uenv_loaded -eq 1; then run uenv_import; fi
bootdelay=10
bootdev=usb
device=0:1
devices=usb ide mmc
disks=0 1 2 3
ethact=egiga0
if_netconsole=ping $serverip
led_error=orange blinking
led_exit=green off
led_init=green blinking
load_dtb_addr=0x1c00000
load_initrd_addr=0x1100000
load_uimage_addr=0x800000
load_dtb=echo loading DTB $dtb_file ...; load $bootdev $device $load_dtb_addr $dtb_file
load_initrd=echo loading uInitrd ...; load $bootdev $device $load_initrd_addr /boot/uInitrd
load_uimage=echo loading uImage ...; load $bootdev $device $load_uimage_addr /boot/uImage
mainlineLinux=yes
mtdids=nand0=orion_nand
partition=nand0,2
scan_disk=echo running scan_disk ...; scan_done=0; setenv scan_usb "usb start";  setenv scan_ide "ide reset";  setenv scan_mmc "mmc rescan"; for dev in $devices; do if test $scan_done -eq 0; then echo Scan device $dev; run scan_$dev; for disknum in $disks; do if test $scan_done -eq 0; then echo device $dev $disknum:1; if load $dev $disknum:1 $load_uimage_addr /boot/uImage 1; then scan_done=1; echo Found bootable drive on $dev $disknum; setenv device $disknum:1; setenv bootdev $dev; fi; fi; done; fi; done
set_bootargs=setenv bootargs console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 $mtdparts $custom_params
stderr=serial
stdin=serial
stdout=serial
uenv_addr=0x810000
uenv_import=echo importing envs ...; env import -t $uenv_addr $filesize
uenv_init_devices=setenv init_usb "usb start";  setenv init_ide "ide reset";  setenv init_mmc "mmc rescan"; for devtype in $devices; do run init_$devtype; done;
uenv_load=run uenv_init_devices; setenv uenv_loaded 0; for devtype in $devices;  do for disknum in 0; do run uenv_read_disk; done; done;
uenv_read_disk=if test $devtype -eq mmc; then if $devtype part; then run uenv_read;  fi; else if $devtype part $disknum; then run uenv_read; fi;  fi
uenv_read=echo loading envs from $devtype $disknum ...; if load $devtype $disknum:1 $uenv_addr /boot/uEnv.txt; then setenv uenv_loaded 1; fi
usb_ready_retry=15
arcNumber=2998
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
ethaddr=00:11:A5:7A:7E:73
preboot_nc=setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fi
preboot=run preboot_nc
ipaddr=192.168.0.115
serverip=192.168.0.100
serverport=614
start_netconsole=setenv ncip $serverip:$serverport; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
custom_params=zswap.enabled=1 zswap.compressor=lz4

Thanks in advance for any insight.
-JT

JT,


> for one problem. If the server isn't listening
> the Dockstar won't boot. The light flashes green
> forever.

It is a known problem with u-boot netconsole.


> I even got an APC 1500VA battery backup
> power supply (which turned out to be a piece of
> crap) because if I'm away when the power goes out
> my Dockstar won't boot when the power comes back.
> Today the power went out and the APC provided all
> of 5 minutes worth of backup power with a very
> light load. I need to change the server to my
> desktop which I can turn on remotely and ssh to,
> but it still bothers me why this happens.

Let me find the post I wrote the explanation and repost it here.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

OK here is why I have scripted the netconsole envs to have a loop to get around the problem.

fw_setenv preboot_nc 'setenv nc_ready 0; for pingstat in 1 2 3 4 5; do; sleep 1; if run if_netconsole; then setenv nc_ready 1; fi; done; if test $nc_ready -eq 1; then run start_netconsole; fi'

U-boot code for network is rather simplistic. The "ping" command should be used with caution. Because it relies on the router to respond to a ping. If the netconsole server has been active sometime in the near past, the router still have the netconsole server IP information and it is not cleared soon enough when that box goes offline.

The result is a false positive from the ping. And then the u-boot network code will take a long time to retry (at least 2 minutes, sometime it takes up to 30 minutes) to connect when u-boot executes further envs to find the netconsole server.

So if we do a few pings continously, chance is that it will fail in the subsequenced pings. And then netconsole is recognized to be "not ready". And u-boot would stop further setup for netconsole.

The router behavior is important. If it does the actual ping then, it would know the box is offline. But if it relies on its table to return a false positive then the retry period kicks in. And it seems that takes forever, but actuall it will time out eventually.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 07/13/2020 04:58AM by bodhi.

Ah, thanks! I thought I had seen it mentioned before, but my search skills must be lacking because I couldn't find it. I did leave it going with the flashing green light for a few hours last night before I gave up and cycled the power and had my laptop listening as the server.

I switched everything over to using my desktop as the server, so I should be good if I'm away, but I'm still thinking of connecting the Dockstar to a smart switch just to be sure I can remotely force a reboot.

Thanks again bodhi!
-JT

I think I can fix this inside uboot, but I have no plan to rebase Kirkwood uboot to 2020, ie. no urgency:)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)