diagnose UBIFS boot error

I have debian installed on a UBIFS volume, and I get this error when booting:

Loading file '/boot/uImage' to addr 0x00800000 with size 2082196 (0x001fc594)...
UBIFS error (pid 0): read_block: bad data node (block 508, inode 51788)
UBIFS error (pid 0): do_readpage: cannot read page 508 of inode 51788, error -22
Error reading file '/boot/uImage'
/boot/uImage not found!

the uInitrd loads fine. the uImage is of course in /boot. if I grab uImage from a tftp server and then boot, debian works fine. if I then copy /boot/uImage to someplace else, say to the tftp server, it has the same md5sum in both locations. if I tftp boot the copy, it works. This tells me that maybe uImage is fine, only that u-boot is having problems. Seem right?

I also tried renaming uImage to something else, and then running flash-kernel. The idea was to keep uImage occuping the same physical location so that the newly generated uImage would be someplace else. Same error. Er, well the inode is different, but the block and page of 508 are the same.

So, what's going on, and how can I fix it?

Nematocyst Wrote:
-------------------------------------------------------
> I have debian installed on a UBIFS volume, and I
> get this error when booting:

> Loading file '/boot/uImage' to addr 0x00800000
> with size 2082196 (0x001fc594)...
> UBIFS error (pid 0): read_block: bad data node
> (block 508, inode 51788)
> UBIFS error (pid 0): do_readpage: cannot read page
> 508 of inode 51788, error -22
> Error reading file '/boot/uImage'
> /boot/uImage not found!
>

>
> the uInitrd loads fine. the uImage is of course
> in /boot. if I grab uImage from a tftp server and
> then boot, debian works fine. if I then copy
> /boot/uImage to someplace else, say to the tftp
> server, it has the same md5sum in both locations.
> if I tftp boot the copy, it works. This tells me
> that maybe uImage is fine, only that u-boot is
> having problems. Seem right?
>
> I also tried renaming uImage to something else,
> and then running flash-kernel. The idea was to
> keep uImage occuping the same physical location so
> that the newly generated uImage would be someplace
> else. Same error. Er, well the inode is
> different, but the block and page of 508 are the
> same.
>
> So, what's going on, and how can I fix it?

The importance of formating the UBIS volumne is it will manage your bad blocks. Do you have bad blocks in this partition and how did you copy the rootfs to this partition? i.e. which command?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

basically followed the instructions here: http://www.blaicher.com/2012/07/installing-debian-on-a-sheevaplug-into-flash/

eg:

ubiformat /dev/mtd2 -s 512
ubiattach /dev/ubi_ctrl -m 2 
ubimkvol /dev/ubi0 -N rootfs -m
mkfs.ubifs --compr=zlib /dev/ubi0_0
mount -t ubifs ubi0:rootfs /mnt

and then

mkdir /tmp/rootfs
mount -o bind / /tmp/rootfs/
cp -a /tmp/rootfs/* /mnt/
cp -a /boot /mnt/

to copy the USB install to NAND, followed by uboot settings to boot.

I basically installed wheezy this way, and then did a dist-upgrade to jessie (because I was having an impossible getting debian-installer to recognize eth0)

I did this a while ago, march, iirc. it's possible it was borked the whole time and I didn't notice, but that's hard to believe. It's more likely it was working, then when I started futzing again with the device a few days ago, it just wasn't booting and I needed to use a serial connection to be able to tftpboot

[Edit] Yeah, the partition has several bad blocks. Maybe 25. It's been that way for years, and never led to any issue.



Edited 1 time(s). Last edit at 07/17/2015 09:40PM by Nematocyst.

There are solutions I can do that sort of ignore the details. Like I could set up uboot to use tftp. or I could copy the filesystem to a flash drive, and then reformat and copy back. Those would probably both work. Just then I wouldn't know what is going on and thus have no idea if it's gonna happen again, possibly to a more sensitive part of the filesystem.

But given the behavior, I'm not sure I'm convinced the area under uImage is actually bad. Once booted into debian, there is no evidence of error. Uboot is where the error shows up.



Edited 1 time(s). Last edit at 07/17/2015 09:32PM by Nematocyst.

Nematocyst Wrote:
-------------------------------------------------------
> There are solutions I can do that sort of ignore
> the details. Like I could set up uboot to use
> tftp. or I could copy the filesystem to a flash
> drive, and then reformat and copy back. Those
> would probably both work. Just then I wouldn't
> know what is going on and thus have no idea if
> it's gonna happen again, possibly to a more
> sensitive part of the filesystem.
>
> But given the behavior, I'm not sure I'm convinced
> the area under uImage is actually bad. Once
> booted into debian, there is no evidence of error.
> Uboot is where the error shows up.

I think you are right. The setup comands like you did above, would have a UBIFS volume that takes care of the bad blocks. So I think this is just a filesystem error, perhaps the box was powered down without a shutdown while this rootfs volume was mounted RW.

It would be best if you do the second option: back up the rootfs, reformat the volume, and copy the rootfs back.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Holy cow. After reformatting, I get... exactly the same problem. How on earth is that possible?

UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: mounted read-only
UBIFS: file system size:   511064064 bytes (499086 KiB, 487 MiB, 3961 LEBs)
UBIFS: journal size:       9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: zlib
UBIFS: reserved for root:  0 bytes (0 KiB)
Loading file '/boot/uInitrd' to addr 0x01100000 with size 9688174 (0x0093d46e)...
Done
Loading file '/boot/uImage' to addr 0x00800000 with size 2082196 (0x001fc594)...
UBIFS error (pid 0): read_block: bad data node (block 508, inode 26197)
UBIFS error (pid 0): do_readpage: cannot read page 508 of inode 26197, error -22
Error reading file '/boot/uImage'
/boot/uImage not found!
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   kernel 3.16.0-4-kirkwood
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2082132 Bytes = 2 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!
Marvell>>

First reboot, I noticed the error, but it booted correctly because the memory at 0x800000 was still intact. The above error happened after invoking flash-kernel, halting, then cutting power temporarily, and finally booting. I'm baffled.

Ok, I fixed it. Suspecting there might be a problem with u-boot's decompression, I disabled compression on that file.

chattr -c /boot/uImage
cp /boot/uImage /tmp/uImage
cp /tmp/uImage /boot/uImage
lsattr /boot/uImage

1) disable compression
2) copy it elsewhere
3) copy over the existing one since step 1) only changes the attributes, not the contents
4) verify it is still uncompressed (lack of c attribute)

boots fine now. I don't claim to fully understand this problem, but I'm assuming something in this particular uImage file was the culprit, causing a bug in the rather old version of ubifs support in uBoot. This also explains why there was no apparent problem once debian had booted.

Nematocyst Wrote:
-------------------------------------------------------
> Ok, I fixed it. Suspecting there might be a
> problem with u-boot's decompression, I disabled
> compression on that file.
>

> chattr -c /boot/uImage
> cp /boot/uImage /tmp/uImage
> cp /tmp/uImage /boot/uImage
> lsattr /boot/uImage
>

>
> 1) disable compression
> 2) copy it elsewhere
> 3) copy over the existing one since step 1) only
> changes the attributes, not the contents
> 4) verify it is still uncompressed (lack of c
> attribute)
>
> boots fine now. I don't claim to fully understand
> this problem, but I'm assuming something in this
> particular uImage file was the culprit, causing a
> bug in the rather old version of ubifs support in
> uBoot. This also explains why there was no
> apparent problem once debian had booted.

Yup! I think that was the problem. We've seen this compression problem before (symptom was slightly different). Because the u-boot version is too old, and it does not support the same compression algorithm as Linux.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Do you know if uboot fixes that in more recent versions?

[edit] yes, it does... downloading the correct one as of this date solves the ability to decompress the same files. note: doing so changed the device offset in /etc/fw_env.config from 0x60000 to 0xc0000. also it required a slight ubifsmount syntax change. instead of

ubifsmount rootfs

I needed

ubifsmount ubi0:rootfs

[/edit]

For now, I added

chattr -c "$source"

just before

mv "$source" "$dest"

in function backup_and_install() in file /usr/share/flash-kernel/functions to make sure both uImage and uInitrd remain uncompressed. It's no big loss since these files are already compressed anyway.



Edited 1 time(s). Last edit at 07/18/2015 01:16PM by Nematocyst.