issues when booting something different from Debian uImages

I'm doing some testing on a possible new recovery firmware based off LEDE (OpenWrt fork), but booting anything that isn't a Debian uImage gives weird results.

I suspect that there is something wrong in my images, but any opinion on it is welcome.

I build kirkwood generic target from source, and I took the zimage and appended the dtb as stated (different names to suit my case)

EDIT: I checked with ghex (a hex editor), the zimage I'm using does NOT have a dtb appended. Images I build show the text from the dtb at the end of the uImage or zImage.

cp -a zImage-3.17.0-kirkwood-tld-1  zImage.fdt
cat dts/kirkwood-goflexnet.dtb  >> zImage.fdt
mv uImage uImage.orig
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-3.17.0-kirkwood-tld-1 -d /media/sdb1/boot/zImage.fdt /media/sdb1/boot/uImage
sync

But when I try booting it (either from NAND or from a usb thumbdrive)
with this bootcmd (or with fatload instead of nand read)

bootcmd_recovery=run recovery_bootargs; nand read 0x8000 nand0,7 ; bootm 0x8000

I get this

NAND read: device 0 offset 0x4640000, size 0xa00000
 10485760 bytes read: OK
## Booting kernel from Legacy Image at 00008000 ...
   Image Name:   Recovery-nsa310b-lmsensor-redusb
   Created:      2016-08-29  21:58:22 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    1873248 Bytes = 1.8 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK

Starting kernel ...

This means the Uboot detects a valid uImage and checksums match, but something fails when starting it.

Anything shown after that (if it worked at all) would have come first from the uImage's onboard kernel decompression tool, and then from kernel itself.

For example if I try to start stock kernel image i get also a "decompressing image ..............................................................................................(and so on).............done, booting" before it hangs, so the uImage's decompressor works even if the kernel fails afterwards.

Same thing as above happens with trying to boot a zimage with the dtb appended

NSA310> fatload usb 0 0x8000 zimage-kirkwood-nsa310b-lmsensor-redusbled 
reading zimage-kirkwood-nsa310b-lmsensor-redusbled
1873248 bytes read in 85 ms (21 MiB/s)
NSA310> bootz 0x8000
Kernel image @ 0x008000 [ 0x000000 - 0x1c6408 ]

Starting kernel ...

For the sake of doing it (as it usually works too) I used also the "go" command instead of bootm/z

NSA310> fatload usb 0 0x8000 uimage-kirkwood-nsa310b-lmsensor-redusbled
reading uimage-kirkwood-nsa310b-lmsensor-redusbled
1873312 bytes read in 91 ms (19.6 MiB/s)
NSA310> go 0x8000
## Starting application at 0x00008000 ...

And it hangs there.

But it does not end here. I tried also with other uImages+appended-dtb generated by LEDE build system.

NSA310> fatload usb 0 0x8000 lede-kirkwood-dockstar-uimage
reading lede-kirkwood-dockstar-uimage
1871076 bytes read in 89 ms (20 MiB/s)
NSA310> bootm 0x8000
Wrong Image Format for bootm command
Error occured, error code = 112
ERROR: can't get kernel image!

lolwtf?
That's called "uImage" but cannot start with a bootm? Wat?

But the plot thickens, I also try bootz on the same thing

NSA310> bootz 0x8000
Kernel image @ 0x008000 [ 0x000000 - 0x1c6408 ]

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.4.19 (alby@openSUSE-xeon) (gcc version 5.4.0 (LEDE GCC 5.4.0 r1435) ) #0 Tue Aug 23 22:21:34 2016
[    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=0005397f
[    0.000000] CPU: VIVT data cache, VIVT instruction cache
[    0.000000] Machine model: Seagate FreeAgent Dockstar

## and goes on and on for a while until it finally figures out something is wrong as this isn't a Dockstar, panics and reboots.

So, with bootz it starts it. Also tried with "go" command and it starts it too.

What happens if I try to bootz my own uImages?

NSA310> fatload usb 0 0x8000 uimage-kirkwood-nsa310b-lmsensor-redusbled
reading uimage-kirkwood-nsa310b-lmsensor-redusbled
1873312 bytes read in 91 ms (19.6 MiB/s)
NSA310> bootz 0x8000
Bad Linux ARM zImage magic!

No. Still not like it.

Also tried to change memory address for loading and starting, same results.

ARGHHHH :(

I'm now going to try out other things like making the images from Debian, in the box (as currently I'm making them with the mkimage of OpenSUSE, since I'm using OpenSUSE), or trying to see where the damn LEDE buildsystem assembles the images and mimic it (and/or use its own tool to do it, as the build system builds also the whole toolchain from source), so I can at least get a uImage that boots with bootz for lulz but at least boots.

EDIT: for those interested, these are the images I'm using (if it gives a 404 error retry later, it's probably still uploading) https://dl.dropboxusercontent.com/u/47541136/linux/tools_for_Kirkwood_installer/uImages_to_test.tar.bz2



Edited 2 time(s). Last edit at 08/30/2016 10:53AM by bobafetthotmail.

> Also tried to change memory address for loading
> and starting, same results.
>

Have you tried the standard memory address?

load_initrd_addr=0x1100000
load_uimage_addr=0x800000

This is way too low

NSA310> bootz 0x8000

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 08/30/2016 06:31PM by bodhi.

I would not trust this uImage.

Quote

But it does not end here. I tried also with other uImages+appended-dtb generated by LEDE build system.

NSA310> fatload usb 0 0x8000 lede-kirkwood-dockstar-uimage
reading lede-kirkwood-dockstar-uimage
1871076 bytes read in 89 ms (20 MiB/s)
NSA310> bootm 0x8000
Wrong Image Format for bootm command
Error occured, error code = 112
ERROR: can't get kernel image!
lolwtf?
That's called "uImage" but cannot start with a bootm? Wat?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> > Also tried to change memory address for loading
> > and starting, same results.
> >
>
> Have you tried the standard memory address?
>
>

> load_initrd_addr=0x1100000
> load_uimage_addr=0x800000
>

>
> This is way too low
>

>

> NSA310> bootz 0x8000
>

Try this:

fatload usb 0 0x800000 zimage-kirkwood-nsa310b-lmsensor-redusbled 
bootz 0x800000

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

No, it still fails, I tried with all I could try, even from stock uboot it fails.

That's a LEDE bug then, they can't make correct uimages or zimages


Btw, even from OpenWRT trunk it fails.

Let's see if they react better than openwrt does usually to my bug reports (i.e. ignores them).

IIRC from the OpenWRT/LEDE IRC chat rooms, OpenWRT/LEDE has gone through an overhaul in image generation which no longer supports uImage.

habibie Wrote:
-------------------------------------------------------
> IIRC from the OpenWRT/LEDE IRC chat rooms,
> OpenWRT/LEDE has gone through an overhaul in
> [url=http://forum.doozan.com/read.php?3,20812,2777
> 8#msg-27778]image generation[/url] which no longer
> supports uImage.

That's plain wrong, they still support many devices with bootloaders that can only boot uImage, and their toolchain still has the code and tools to make uImages.

They did refactor the image generation and kirkwood is a very low priority target for them, and you likely have hit a bug, just like I did.

> That's plain wrong, they still support many
> devices with bootloaders that can only boot
> uImage, and their toolchain still has the code and
> tools to make uImages.

:) that did not seem to register when I told habibie this.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bobafetthotmail Wrote:
-------------------------------------------------------
> habibie Wrote:
> --------------------------------------------------
> -----
> > IIRC from the OpenWRT/LEDE IRC chat rooms,
> > OpenWRT/LEDE has gone through an overhaul in
> > image generation which no longer
> > supports uImage.
>
> That's plain wrong, they still support many
> devices with bootloaders that can only boot
> uImage, and their toolchain still has the code and
> tools to make uImages.
>
TBH, I just hope you are right. But, from what I saw and remembered, both zImage and uImage will no longer be supported. I was hoping Bodhi will update the uboot to include the FIT capability to support using the fit-pogoplug-pro.itb and fit-pogoplug-pro.its files found in the kernel source directory to boot the device. This way, we do not have to worry if the new image generation once and for all. Meanwhile, if you find a way to fix this, I sure would like to know, too.

habibie Wrote:
-------------------------------------------------------
> TBH, I just hope you are right. But, from what I
> saw and remembered, both zImage and uImage will no
> longer be supported. I was hoping Bodhi will
> update the uboot
> [url=http://forum.doozan.com/read.php?3,20812,2777
> 8#msg-27778]to include the FIT capability to
> support using the fit-pogoplug-pro.itb and
> fit-pogoplug-pro.its files found in the kernel
> source directory to boot the device[/url]. This
> way, we do not have to worry if the new image
> generation once and for all. Meanwhile, if you
> find a way to fix this, I sure would like to know,
> too.

That's complete and utter bs, as no known bootloader for most embedded devices out there, both ours or the one you talk about can boot FIT images.

How can you even think it makes sense that to push a completely pointless upgrade (FIT does not "boot better" than uImage) they require people to build new bootloaders for like 99% of the whole OpenWRT/LEDE supported fleet of devices?
They aren't Microsoft, they don't need to obsolete all old stuff so people buys the new stuff.

The only logical explanation is that the build system is configured wrong.

I investigated following this logical reasoning, and yes, I can confirm that the build system is configured very wrong for kirkwoods, so it's very likely they screwed up other less-built devices too.

My issues in this tread were caused by the build system being configured plain wrong, it was spitting zImages with dtb appended, and then renaming them as uImage (and screwing up other stuff), which is plain wrong.

Changing a couple lines in the image building makefiles fixed that.
Will send a PR to LEDE soon with this fix and if they don't accept it I'm going to find and slaughter all their kittens.


I assume that your issue is just more of the same. Please state all devices you are getting only FIT images for.
I'm going to fix their build options too and send a PR to upstream. Building FIT for those that want to use it is OK, not building uImages for no reason at all is a bug, period.

This is the file I changed for my issues [b]/source/target/linux/kirkwood/image/Makefile[/b]

This configures all devices to be built as uImages (instead of weird stuff renamed as uImages), so you can go hack that on your own too, but please state any other device you are getting FIT images for, so i can go fix and send a PR to fix upstream too. Note that I'm sending patches only to LEDE, I think they carry over some stuff to OpenWRT, but I don't really care of OpenWRT anymore.

The most important part is inside the "define Device/Default", as the other devices are just extending that with custom configurations

[code]

#
# Copyright (C) 2009-2013 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

NAND_BLOCKSIZE := 2048-128k

include $(TOPDIR)/rules.mk
include $(INCLUDE_DIR)/image.mk

DEVICE_VARS += KERNEL_SIZE
KERNEL_LOADADDR:=0x8000
TARGET_DEVICES = linksys-audi linksys-viper dockstar goflexnet goflexhome nsa310

UBI_OPTS := -m 2048 -p 128KiB -s 512
UBIFS_OPTS := -m 2048 -e 126KiB -c 4096

define Device/Default
KERNEL_DEPENDS = $$(wildcard $(DTS_DIR)/$$(DEVICE_DTS).dts)
KERNEL := kernel-bin | append-dtb | uImage none

KERNEL_NAME := zImage
KERNEL_SUFFIX := -uImage
KERNEL_INSTALL := 1
FILESYSTEMS := squashfs

PAGESIZE := 2048
SUBPAGESIZE := 512
BLOCKSIZE := 128KiB
IMAGES := sysupgrade.tar
IMAGE/sysupgrade.tar := sysupgrade-tar
endef

define Device/dockstar
DEVICE_DTS := kirkwood-dockstar
FILESYSTEMS := squashfs
PROFILES := Generic DOCKSTAR
IMAGES += factory.bin
IMAGE/factory.bin := append-ubi
KERNEL_IN_UBI := 1
endef

define Device/goflexnet
$(Device/dockstar)
PROFILES := Generic GOFLEXNET
DEVICE_DTS := kirkwood-goflexnet
endef

define Device/goflexhome
$(Device/dockstar)
PROFILES := Generic GOFLEXHOME
DEVICE_DTS := kirkwood-goflexhome
endef

define Device/linksys-audi
DEVICE_DTS := kirkwood-linksys-audi
KERNEL_SIZE := 2624k
FILESYSTEMS := squashfs
PROFILES := Generic AUDI
IMAGES += factory.bin
IMAGE/factory.bin := append-kernel $$$$(KERNEL_SIZE) | append-ubi
endef

define Device/linksys-viper
DEVICE_DTS := kirkwood-linksys-viper
KERNEL_SIZE := 2688k
FILESYSTEMS := squashfs
PROFILES := Generic VIPER
IMAGES += factory.bin
IMAGE/factory.bin := append-kernel $$$$(KERNEL_SIZE) | append-ubi
endef

define Device/nsa310
DEVICE_DTS := kirkwood-nsa310b-lmsensor-redusbled
FILESYSTEMS := squashfs ubifs
PROFILES := Generic NSA310
endef


define Image/BuildKernel/Template

$(CP) $(KDIR)/zImage $(BIN_DIR)/$(IMG_PREFIX)-zImage
echo -ne '\x00\x00\x00\x00' >> $(BIN_DIR)/$(IMG_PREFIX)-zImage
$(call Image/BuildKernel/MkuImage, \
none, 0x8000, 0x8000, \
$(BIN_DIR)/$(IMG_PREFIX)-zImage, \
$(BIN_DIR)/$(IMG_PREFIX)-uImage \
)

ifneq ($(CONFIG_TARGET_ROOTFS_INITRAMFS),)
$(CP) $(KDIR)/zImage-initramfs $(BIN_DIR)/$(IMG_PREFIX)-zImage-initramfs
echo -ne '\x00\x00\x00\x00' >> $(BIN_DIR)/$(IMG_PREFIX)-zImage-initramfs
$(call Image/BuildKernel/MkuImage, \
none, 0x8000, 0x8000, \
$(BIN_DIR)/$(IMG_PREFIX)-zImage-initramfs, \
$(BIN_DIR)/$(IMG_PREFIX)-uImage-initramfs \
)
endif

ifneq ($(1),)
$(CP) $(DTS_DIR)/kirkwood-$(1).dtb $(BIN_DIR)/$(IMG_PREFIX)-$(1).dtb

$(CP) $(KDIR)/zImage $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage
cat $(BIN_DIR)/$(IMG_PREFIX)-$(1).dtb >> $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage
$(call Image/BuildKernel/MkuImage, \
none, 0x8000, 0x8000, \
$(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage, \
$(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-uImage \
)

ifneq ($(CONFIG_TARGET_ROOTFS_INITRAMFS),)
$(CP) $(KDIR)/zImage-initramfs $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage-initramfs
cat $(BIN_DIR)/$(IMG_PREFIX)-$(1).dtb >> $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage-initramfs
$(call Image/BuildKernel/MkuImage, \
none, 0x8000, 0x8000, \
$(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-zImage-initramfs, \
$(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-uImage-initramfs \
)
endif
endif
endef

define Image/Build/jffs2-nand-2048-128k
dd if=$(KDIR)/root.$(1) of=$(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-$(1).img \
bs=2048 conv=sync
endef

define Image/Build/squashfs
$(call prepare_generic_squashfs,$(KDIR)/root.squashfs)
( \
dd if=$(KDIR)/zImage bs=4096k conv=sync; \
dd if=$(KDIR)/root.$(1) bs=128k conv=sync; \
) > $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-$(1).img
endef

define Image/Build/ubifs

ifneq ($($(PROFILE)_UBIFS_OPTS),)
$(CP) $(KDIR)/root.ubifs $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-rootfs.ubifs
endif
endef

define Image/Build/ubi

ifneq ($($(PROFILE)_UBI_OPTS),)
$(CP) $(KDIR)/root.ubi $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-rootfs.ubi
endif
endef

IOMEGA_IX2_200_UBIFS_OPTS := -m 512 -e 15872 -c 2048
IOMEGA_IX2_200_UBI_OPTS := -m 512 -p 16384 -s 256
NSA310_UBIFS_OPTS := -m 512 -e 15872 -c 2048
NSA310_UBI_OPTS := -m 512 -p 16384 -s 256

Image/BuildKernel/Template/Generic=$(call Image/BuildKernel/Template)
Image/BuildKernel/Template/IB62X0=$(call Image/BuildKernel/Template,ib62x0)
Image/BuildKernel/Template/ICONNECT=$(call Image/BuildKernel/Template,iconnect)
Image/BuildKernel/Template/IOMEGA_IX2_200=$(call Image/BuildKernel/Template,iomega_ix2_200)
Image/BuildKernel/Template/NSA310=$(call Image/BuildKernel/Template,nsa310b-lmsensor-redusbled)
Image/BuildKernel/Template/NSA310S=$(call Image/BuildKernel/Template,nsa310s)
Image/BuildKernel/Template/POGOE02=$(call Image/BuildKernel/Template,pogo_e02)
Image/BuildKernel/Template/SHEEVAPLUG=$(call Image/BuildKernel/Template,sheevaplug)
Image/BuildKernel/Template/SHEEVAPLUGSATA=$(call Image/BuildKernel/Template,sheevaplug-esata)
Image/BuildKernel/Template/GuruplugServerPlus=$(call Image/BuildKernel/Template,guruplug-server-plus)
Image/BuildKernel/Template/Topkick1281P2=$(call Image/BuildKernel/Template,topkick)

define Image/BuildKernel
$(call Image/BuildKernel/Template/$(PROFILE))
endef

define Image/InstallKernel
$(call Image/InstallKernel/Template/$(PROFILE))
endef

define Image/Build
$(if $(Image/Build/$(1)), \
$(call Image/Build/$(1),$(1)), \
$(CP) $(KDIR)/root.$(1) $(BIN_DIR)/$(IMG_PREFIX)-$(PROFILE_SANITIZED)-$(1).img \
)
endef

$(eval $(call BuildImage))

[/code]



Edited 1 time(s). Last edit at 09/03/2016 09:31AM by bobafetthotmail.

Sent pull request here https://github.com/lede-project/source/pull/315

I've looked at the makefile for the pogoplug pro (and its class of devices). It's /source/target/linux/oxnas/image/Makefile

Pogoplug is inheriting the FIT configuration from the default of its class of devices as none bothered to customize its entry, nor the ones of other devices for that matter, I doubt the pogoplug v3 or of any of the other devices listed in that makefile can actually boot a FIT image.

This is the line that causes it to be genereated as FIT

KERNEL = kernel-bin | lzma | fit lzma $$(DTS_DIR)/ox820-$$(DTS).dtb

The fix is simply adding a line in its entry to override the defaults of its class.

This is its current code for its entry

define Device/pogoplug-pro
  DTS := pogoplug-pro
  DEVICE_TITLE := Cloud Engines Pogoplug Pro (with mPCIe)
endef
TARGET_DEVICES += pogoplug-pro

This is the fixed code for its entry:

define Device/pogoplug-pro
  DTS := pogoplug-pro
  DEVICE_TITLE := Cloud Engines Pogoplug Pro (with mPCIe)
  KERNEL = kernel-bin | append-dtb | uImage none 
endef
TARGET_DEVICES += pogoplug-pro

Please test this and see if this fixes the issue, then I'm sending this fix too. (or you can send it on your own if you want)



Edited 1 time(s). Last edit at 09/03/2016 11:09AM by bobafetthotmail.

I'd like to point out this fact for general consumption: FIT image and uImage are just the same kernel files inside being packaged a different way. The main advantage with FIT is you can sign your image, so it is more secure. But when it comes down to maintainability for us is no more or less works.

- So there are drawback when you're hacking such as: less convenient when you're testing, working on a new device. Once it's done, it's same amount of works to maintain it.

- Older u-boot can not boot FIT. So I'll need to maintain uImage and zImage anyway so that was the main reason why I have not distributed FIT images, yet.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

> Will send a PR to LEDE soon with this fix and if
> they don't accept it I'm going to find and
> slaughter all their kittens.
>

:))

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)