Linux Kernel 6.7.5 Kirkwood package and Debian rootfs

pbg4,

> yes, that's what I would suspect also, something
> went wrong after the initialisation of pci/pcie
> and the probing of devices afterwards, according
> to dmesg messages of kernel 3.18.5 and
> 4.0.0 its all identical, only difference is, the
> probing after:
>
> pci 0000:00:01.0: enabling device (0140 -> 0142)

Yes. I did too.

> and the usb 3.0 chip
> of Pogoplug V4 and NSA325 are most likely also
> attached via
> pci,

Yes, their controller do attach to the PCI bus.

> perhaps its a timing problem with armv5,

I think this is most likely. I guess the author(s) of the PCI changes must have tested with different hardware.

The eeprom read was not successful (I put some debug statement in). It returns 0 for the device id:

[   20.950073] ieee80211 phy0: rt2x00_set_rt: Info - RT chipset 3090, rev 3212 detected
[   20.994538] ieee80211 phy0: rt2800_init_eeprom: Error - eepromp word: 0x0000
[   21.001686] ieee80211 phy0: rt2800_init_eeprom: Error - rf (rt2x00_get_field16): 0x0000
[   21.017415] ieee80211 phy0: rt2800_init_eeprom: Error - Invalid RF chipset 0x0000 detected
[   21.025955] ieee80211 phy0: rt2x00lib_probe_dev: Error - Failed to allocate device

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

there was a recent git pull of pci-v4.0-fixes-3 whithin the merge window of 4.0 here:

https://lkml.org/lkml/2015/4/9/533

but looking at the patches,.. nothing spectacular so far which looks suspicious,
the only kernel I could try inbetween was 3.19.1, that was fine and did not have the regression,

so perhaps 3.19.4 would be fine too, or should we test 4.0.0-rc7 to see the effects of the changes from the merge
window of 4.0 ?

best wishes pbg4



Edited 1 time(s). Last edit at 04/21/2015 02:26PM by pbg4.

pbg4,

> so perhaps 3.19.4 would be fine too, or should we
> test 4.0.0-rc7 to see the effects of the changes
> from the merge
> window of 4.0 ?

I have the kirkwood rc4 build. I've tried that and the same error occured. Will do a binary down to rc1 :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/19/2015 03:20AM by bodhi.

@bodhi:

I can confirm that 3,19 works fine (using congenio's image)



@pbg4

I think 3,19.4 would be fine,



Perheps the relevant dtb code has changed?

pengu Wrote:
-------------------------------------------------------
> @bodhi:
>
> I can confirm that 3,19 works fine (using
> congenio's image)

Thanks. Also, can you confirm 3.19.5 work?

>
> Perheps the relevant dtb code has changed?

I don't think it's DTB. The iConnect DTS has not been changed.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

pengu Wrote:
-------------------------------------------------------
> @bodhi:
>
> I can confirm that 3,19 works fine (using
> congenio's image)

Which 3.19.x can you confirm that works OK? 3.19.4 ?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/19/2015 11:20PM by bodhi.

bodhi Wrote:
-------------------------------------------------------
> pengu Wrote:
> --------------------------------------------------
> -----
> > @bodhi:
> >
> > I can confirm that 3,19 works fine (using
> > congenio's image)
>
> Which 3.19.x can you confirm that works OK? 3.19.4
> ?

3,19.1 is working fine, there are no newer version of this one and the congenio's 4.0 has the same issues as your bild

sorry for the confusing about the DTB stuff
Perhaps something has changed that needs an update of the DTB code?

pengu,

> 3,19.1 is working fine, there are no newer version
> of this one and the congenio's 4.0 has the same
> issues as your bild

I'm not familiar with congenio's image. Is that image for iConnect only?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Kernel 4.0 PCI regression.

I've narrowed down to kernel version 4.0-rc1 is where it's broken. Kernel 3.19.5 works fine for both the Pogo V4 USB 3.0 and the iConnect Wifi card. So it is most likely a PCI problem. Looking at the commits between 3.19.5 and 4.0-rc1, there were quite a bit of PCI code changes. It's not a simple task to track down the problem. Not sure I have time to fugure this one out.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/21/2015 05:48AM by bodhi.

Hi,

if time permits I am also going have a look at the commits between 3.19.5 and 4.0.0-rc1,
perhaps we should also have a look if new kernel options were introduced with 4.0.0-rc1 in the kernel docs,
if perhaps a new enumeration scheme for probing of devices on the pci bus scan can be switched back
to the older legacy method before 4.0,..

the congenio image and rootfs is simply a debian wheezy rootfs and an iconnect only
specific image which automatically appends dtb to zImage when building, the config is very specific
to the iconnect device and inlines all the relevant modules in the kernel image, so no initrd is necessary,
the 4.0 patch is here:

http://www.congenio.de/infos/iconnect-4.0.patch

best wishes pbg4

p.s. forgot to mention that I crosschecked on a gigabyte brix box with haswell processor and ubuntu trusty 14.04
with kernels from ubuntu's mainline ppa that 4.0.0-rc1 with pcie is working, as expected everything is fine with X86_64,..



Edited 1 time(s). Last edit at 04/21/2015 10:33AM by pbg4.

Hi,

@bodhi

are we missing something in the armv5 config for 4.0.0 which should be there?

CONFIG_HOTPLUG_PCI_PCIE=y

this is part of the ubuntu mainline ppa patch for ubuntu 4.0.0-1.1,

of course nobody wants to hotplug pcie devices in armv5, but there are commits for 4.0.0 that enable
this dependent on acpi, which on armv5 is not an issue, just a thought,...

best wishes pbg4

p.s. the link:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-vivid/



Edited 1 time(s). Last edit at 04/21/2015 02:53PM by pbg4.

@pbg4,

> are we missing something in the armv5 config for
> 4.0.0 which should be there?
>
> CONFIG_HOTPLUG_PCI_PCIE=y

Indeed, the changes regarding ACPI was massive from 3.19.5 to 4.0-rc1. This could be some new dependency that they forgot to include in the kernel config. I'll give it a try.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@pbg4,

> > CONFIG_HOTPLUG_PCI_PCIE=y

I have not been able to build with this. Looks like because the lack of CONFIG_HOTPLUG which is not selectable because we never use it (could be something else has be brought in before we can see this option).

I'll try some other ideas to see if the timing was the problem when I have time to get back to it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Dear bodhi,

My Ralink card is correctly recognised with lspci command. Atheros not. Today I tried an Intel 6250 card. It was not recognised either (lspci).
Can it be uboot-related? My u-boot version is: U-Boot 2014.07-tld-2.

Checcking again of your uboot flash instructions I found a possible reason.
May be my uboot is somehow bad:

dmesg | grep -i 'bad'
[ 1.806961] Scanning device for bad blocks
[ 1.811358] Bad eraseblock 3 at 0x000000060000

Any advice?

tanya,

No, it is not u-boot related. It's up to the kernel to recognize the card. Please post the following information here and I'll check the kernel to see if the drivers are included.

1. Which kernel you are running?
2. The card model, version,...
3. Can you post the output of dmesg here?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

tanya,

> May be my uboot is somehow bad:
>
> dmesg | grep -i 'bad'
> [ 1.806961] Scanning device for bad blocks
> [ 1.811358] Bad eraseblock 3 at 0x000000060000
>
> Any advice?

I make this a separate post to emphasize this: don't do any thing related to u-boot other than changing envs. Your bad block is the 4th block, so it is not advisable to flash u-boot again, without careful planning. As of right now, as long as you can boot into Linux, nothing bad. But please post your entire dmesg output here so I can look it over for you.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi

dmesg outputs attached. Hope you find the root cause.

tanya,

The dmesg for the Intel card did not show anything useful. It means the card is not supported by the kernel (there is nothing wrong, it is just not detected).

If you can post the make, model, chipset,... of the card then I can take a look in the kernel to see if it's there already or not, and possibly add it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi
The card is "Intel Centrino Advanced-N + WiMAX 6250".

tanya Wrote:
-------------------------------------------------------
> @bodhi
> The card is "Intel Centrino Advanced-N + WiMAX
> 6250".

This card is supported in the kernel but I have not turned it on in the kernel configuration. So stay tune for the next release, I will add it. But I could compile this and post the module for you to use first, if that's better for you?

And please post the make and model of the Atheros PCI card that you're having problem with.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/24/2015 04:26PM by bodhi.

Hi bodhi,

as you may already have noticed, 4.1-rc1 is out, a first glance showed lots of interesting changes,
also to the pci host driver pci-mvebu.c,.. also the mvebu_v5_defconfig is gone,

now there is only a multi_v5_defconfig left, what I had still on my list was to start with
a minimal config based on multi_v5_defconfig and beside CONFIG_PCI_MVEBU skip
everything not necessary in the PCI host controller drivers section, to test if the pcie problems are gone,
and later add all the modules of your full config, what do you think?

best wishes pbg4

Hello,

will USB3 support turn back to work on NSA325?

Thanks in advance for your awesome work.

pbg4,

> I had still on my list was to start with
> a minimal config based on multi_v5_defconfig and
> beside CONFIG_PCI_MVEBU skip
> everything not necessary in the PCI host
> controller drivers section, to test if the pcie
> problems are gone,
> and later add all the modules of your full config,
> what do you think?
>

Sounds like a good plan! Please do. I did try to turn off a few things in my test kernel (e.g. PM), and also compiled the xhci into the kernel, but it did not help. I will try 4.1-rc1 to see if any of the PCI changes will be better.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@addy75,

We are still investigating the problem, free time is limited :) but it will be fixed eventually.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

pbg4,

No dice. I could not even boot with 4.1-rc1 kernel. Something is missing/buggy in the source tree. It gave me the error:

Uncompressing Linux... done, booting the kernel.

Error: unrecognized/unsupported processor variant (0x56251311)

There was no change in 4.1-rc1 config related to system/processor type when I diff the old config file with the new one.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

the same here, but Andrew Lunn has already issued a patch,

http://www.spinics.net/lists/arm-kernel/msg414799.html

now a recompile of the kernel is running here, will take some time ,..

best wishes pbg4

Hi bodhi,

applied the patch manually and recompiled zImage and later generated uImage made
4.1-rc1 kernel boot, OK, but not really since at the time when init initiates the rootfs
a segmentation fault occurs,.. this test was done with a config similar to your standard
config (only dynamic debugging enabled, and than make oldconfig)

but without the 3.18.5 patch applied to 4.1-rc1, because a lot of devicetree enabled
armv5 devices are included natively without the patch,

my next test will be a plain kernel compile with make multi_v5_defconfig only, lets see what happens,.. :)

best wishes pbg4

Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
[   15.197315] Unable to handle kernel paging request at virtual address ffffffee
[   15.204695] pgd = c1134000
[   15.207416] [ffffffee] *pgd=0fffd831, *pte=00000000, *ppte=00000000
[   15.213827] Internal error: Oops: 1 [#1] PREEMPT ARM
[   15.218812] Modules linked in: mtdblock mtd_blkdevs sg sd_mod uas usb_storage scsi_mod
[   15.226841] CPU: 0 PID: 160 Comm: startpar Not tainted 4.1.0-rc1-tld-1 #2
[   15.233662] Hardware name: Marvell Kirkwood (Flattened Device Tree)
[   15.239961] task: c1030800 ti: ce66c000 task.ti: ce66c000
[   15.245391] PC is at devpts_new_index+0x24/0xf8
[   15.249951] LR is at ptmx_open+0x3c/0x14c
[   15.253986] pc : [<c0168798>]    lr : [<c031437c>]    psr: 20000013
[   15.253986] sp : ce66ddb8  ip : cf816ad4  fp : ce66df74
[   15.265517] r10: cfa49400  r9 : 00000041  r8 : 00000000
[   15.270767] r7 : 00000000  r6 : c075b444  r5 : c06d74d8  r4 : cf97e230
[   15.277325] r3 : ffffffea  r2 : 00001cd1  r1 : 01021994  r0 : cf97e230
[   15.283883] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   15.291051] Control: 0005397f  Table: 01134000  DAC: 00000015
[   15.296822] Process startpar (pid: 160, stack limit = 0xce66c190)
[   15.302944] Stack: (0xce66ddb8 to 0xce66e000)
[   15.307324] dda0:                                                       cf97e230 00000000
[   15.315543] ddc0: 00000001 cf97e230 cfa49400 cf97e230 00000000 c031437c cf97e230 cfa49400
[   15.323765] dde0: c075d6f8 00000000 c075d688 c010478c cf97e230 00000000 cfa49400 cf97e230
[   15.331989] de00: c01045f4 cfa49408 ce66debc c00ff210 ce66def0 00000000 ce66deac ce66deac
[   15.340211] de20: ce66debc c010cad4 00000003 cf89bcb0 00000081 c010a41c ce66def0 00000000
[   15.348433] de40: 00000000 00020102 00000000 000080d0 cf402440 00000000 00000026 ce76d030
[   15.356658] de60: 00000000 00000000 cf97e230 cf89bcb0 ffffff9c ce66def0 cfa49400 ce66df74
[   15.364881] de80: ce66deac 00000000 00000041 ce66debc beacff27 c010cedc ce66debc 60000013
[   15.373103] dea0: 00000000 cfe0a9c0 00000001 ce76d030 cf58b110 ce696004 00100100 00000000
[   15.381327] dec0: cffc62c0 ce66df74 00000001 ce77b000 ffffff9c c000f268 ce66c000 00000000
[   15.389549] dee0: beacff27 c010e710 00000041 c1147190 ce76d030 cf58b110 01bb4b6f 00000004
[   15.397771] df00: ce77b015 b6fe2000 00000000 cf631b28 cf97e230 00000101 00000004 00000044
[   15.405993] df20: 00000000 00000000 00000000 00000001 c1038920 00000000 00000003 c011b21c
[   15.414210] df40: 00020102 00020102 ffffff9c ce77b000 00000005 00000003 ffffff9c ce77b000
[   15.422431] df60: 00000005 c0100164 00000000 000000af c000f268 00020102 00000000 00000026
[   15.430653] df80: 00000100 00000001 00000000 0000ee98 00000000 00000001 00000005 c000f268
[   15.438868] dfa0: 00000000 c000f0a0 0000ee98 00000000 0000ee98 00020102 00000000 00000000
[   15.447091] dfc0: 0000ee98 00000000 00000001 00000005 beacdbc4 000181f4 00000000 beacff27
[   15.455315] dfe0: b6f40250 beacd8c8 0000bb0c b6f4029c 60000010 0000ee98 e3500000 e28ee002
[   15.463551] [<c0168798>] (devpts_new_index) from [<c031437c>] (ptmx_open+0x3c/0x14c)
[   15.471341] [<c031437c>] (ptmx_open) from [<c010478c>] (chrdev_open+0x198/0x1bc)
[   15.478790] [<c010478c>] (chrdev_open) from [<c00ff210>] (do_dentry_open.isra.20+0x1b4/0x2cc)
[   15.487365] [<c00ff210>] (do_dentry_open.isra.20) from [<c010cad4>] (do_last.isra.43+0x924/0xb34)
[   15.496283] [<c010cad4>] (do_last.isra.43) from [<c010cedc>] (path_openat+0x1f8/0x4fc)
[   15.504244] [<c010cedc>] (path_openat) from [<c010e710>] (do_filp_open+0x2c/0x78)
[   15.511772] [<c010e710>] (do_filp_open) from [<c0100164>] (do_sys_open+0x128/0x1c8)
[   15.519474] [<c0100164>] (do_sys_open) from [<c000f0a0>] (ret_fast_syscall+0x0/0x30)
[   15.527254] Code: e59f60d4 e1510002 159f30cc 15933000 (15933004) 
[   15.533469] ---[ end trace 4d2a65ac00e4c318 ]---
Segmentation fault

Edited 1 time(s). Last edit at 04/29/2015 02:04PM by pbg4.

pbg4,

That was a quick fix from Andrew Lunn :) I will try again. Thanks.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

pbg4,

Bad news! 4.1-rc1 did not fix the PCI problem.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

pbg4,

I'm going to start the kernel bisect. Hopefully I will find the bad commit eventually, if nobody submits patche(s) for it..

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)