[SOLVED] Debian Kirkwood kernel 3.10.4 udev does not enumerate USB devices in rootfs pre-mount?

It seems something has changed in kernel 3.10.x. I'm having problem with mounting rootfs in kernel 3.10.4. Comparing the dmesg log for kernel 3.9.11 and 3.10.4, I noticed that udev did not seem to enumerate the USB devices during loading rootfs (in initrd pre-mount). That caused the classic /dev/root not found problem. I've confirmed this was the problem by looking at /dev while inside initramfs prompt, and there were no sdx devices there.

Below are the 2 log files excerpts. The first one is kernel 3.10.4, the second one is kernel 3.9.11. Anybody know how to solve this? or seen this problem?

Thanks!

dmesg (kernel 3.10.4 not working)

[   22.313191] udevd[51]: starting version 175
[   22.380897] mv643xx_eth: MV-643xx 10/100/1000 ethernet driver version 1.4
[   22.482274] libphy: orion_mdio_bus: probed
[   22.502334] SCSI subsystem initialized
[   22.513366] mv643xx_eth_port mv643xx_eth_port.0 eth0: port 0 with MAC address xx:xx:xx:xx:xx:xx
[   22.575220] sata_mv sata_mv.0: cannot get optional clkdev
[   22.580706] sata_mv sata_mv.0: slots 32 ports 1
[   22.630331] scsi0 : sata_mv
[   22.641982] ata1: SATA max UDMA/133 irq 21
[   23.002537] ata1: SATA link down (SStatus 0 SControl F300)
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Waiting for root file system ... done.
Gave up waiting for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/disk/by-label/rootfs does not exist.  Dropping to a shell!
modprobe: module i8042 not found in modules.dep
[   43.329907] usbcore: registered new interface driver usbfs
[   43.336605] usbcore: registered new interface driver hub
[   43.343080] usbcore: registered new device driver usb
[   43.351562] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   43.366490] uhci_hcd: USB Universal Host Controller Interface driver
[   43.382336] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   43.403118] hidraw: raw HID events driver (C) Jiri Kosina
[   43.412903] usbcore: registered new interface driver usbhid
[   43.418510] usbhid: USB HID core driver

dmesg (kernel 3.9.11 working)

Loading, please wait...
[    4.939941] udevd[51]: starting version 175
[    5.036936] mv643xx_eth: MV-643xx 10/100/1000 ethernet driver version 1.4
[    5.084581] usbcore: registered new interface driver usbfs
[    5.103126] libphy: mv643xx_eth smi: probed
[    5.126159] usbcore: registered new interface driver hub
[    5.139032] SCSI subsystem initialized
[    5.146340] mv643xx_eth_port mv643xx_eth_port.0 eth0: port 0 with MAC address xx:xx:xx:xx:xx:xx
[    5.179511] usbcore: registered new device driver usb
[    5.186534] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.220214] sata_mv sata_mv.0: cannot get optional clkdev
[    5.225687] sata_mv sata_mv.0: slots 32 ports 1
[    5.278290] scsi0 : sata_mv
[    5.281388] ata1: SATA max UDMA/133 irq 21
[    5.307073] orion-ehci orion-ehci.0: Marvell Orion EHCI
[    5.319713] orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
[    5.327500] orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
[    5.349293] orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
[    5.355324] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    5.362172] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.369445] usb usb1: Product: Marvell Orion EHCI
[    5.374172] usb usb1: Manufacturer: Linux 3.9.11-kirkwood-tld-2 ehci_hcd

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 08/20/2013 01:03AM by bodhi.

Hi bodhi,

on my NSA320 (with your wheezy rootfs) since several weeks there is a 3.10.1 kernel running with
full ARCH patches for quite a while with superior performance,.. the ARCH guys also use
the Marvell Phy !!,.. I tested that because of my problems with the generic PHY documented in this forum,...

a quick comparison of the dmesg output shows, that you have

ata1: SATA link down (SStatus 0 SControl F300)

so maybe the disk is not initialized?? and your initramfs is not found and decompressed?

on my kernel there is

[ 8.544373] sata_mv sata_mv.0: version 1.28
[ 8.544435] sata_mv sata_mv.0: cannot get optional clkdev
[ 8.544529] sata_mv sata_mv.0: slots 32 ports 2
[ 8.546949] scsi0 : sata_mv
[ 8.547245] scsi1 : sata_mv
[ 8.547434] ata1: SATA max UDMA/133 irq 21
[ 8.547444] ata2: SATA max UDMA/133 irq 21

and later

ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl F300)

my kernel command line is without label!!

[ 0.000000] Kernel command line: console=ttyS0,115200 root=/dev/sda2

and later
[ 7.961686] Trying to unpack rootfs image as initramfs...
[ 8.645701] 0x000001680000-0x000004640000 : "rootfs1"
[ 8.646306] 0x000005040000-0x000008000000 : "rootfs2"

and everything is fine,.. what is the version of your sata_mv??, on 3.10.1 it is still 1.28,

quite a while ago when I wrote setup routines for older marvel soc's there always was something
wrong with the memory adress of the sata port in the include definitions when the sata link did not come up,

but its hard to imagine that it has changed from 3.10.1 to 3.10.4,.. ??

best wishes pbg4

p.s. oops,.. I just realised that you have an usb attached rootfs and that my rootfs is attached via sata, so my comment is not fitting for your problem,.. but nevertheless the 3.10 mainline kernels are worth to try, my NSA320 has never been
so stable and performant,..



Edited 2 time(s). Last edit at 08/19/2013 02:29PM by pbg4.

Hi bodhi,

please have a look at https://github.com/archlinuxarm/PKGBUILDs/tree/master/core/linux-kirkwood

the archlinuxarm patch is ready for 3.10.7 and there was nothing in the forums there concerning usb boot
problems,

best wishes pbg4

Hi pbg4,

Thanks for the info. Actually I've tried the Arch patch before to see if the behavior is different. But the same problem occured! But thanks for reminding me to think about Arch distribution and why they don't have any USB booting problem! and you're running Debian and you don't have problem with HDD booting. Very good clues! I think I am close to solve this. Stay tuned :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Just tried the kernel build this morning and Debian Kirkwood kernel 3.10.4 is now booting from USB :) Will post the details later.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

It turned out that there was indeed a subtle change in the kernel 3.10.x regarding Kirkwood USB devices. The ehci-orion driver was part of the kernel before 3.10.x but now introduced as kernel module to be loaded during boot (I have not looked at the source code, but it seems to be the case). There is a new config option, and I did not notice that with "make oldconfig".

The same config options that worked in kernel 3.9.11, but now in 3.10.x the USB modules were loaded too late during boot, and USB rootfs was not found by initrd.

So what I did was rebuild the kernel with the ehci orion USB modules in the kernel image instead of part of initrd:

< CONFIG_USB_COMMON=m
< CONFIG_USB=m
< CONFIG_USB_EHCI_HCD=m
< CONFIG_USB_EHCI_PCI=m

> CONFIG_USB_COMMON=y
> CONFIG_USB=y
> CONFIG_USB_EHCI_HCD=y
> CONFIG_USB_EHCI_PCI=y
> CONFIG_USB_EHCI_HCD_ORION=y

The good clue to the problem was Arch build for Kirkwood is working. Because they don't use initrd, so they always build these into their kernel image. Therefore this change should not affect ArchLinux ARM. Also pbg4 reported that HDD booting is working, and that confirmed my theory.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)