Can't control GPIO pins after update to kernel 5.17.4

I hope I'm missing something simple. I've been working for the last week to update my Dockstar from kernel 4.8.3 to 5.17.4 and once I got past not being able to build LKMs I now find that I can't control GPIO10 (TxD) with the LKM or via /sys/class/gpio/gpio10. I don't get any errors from my LKM or when I export GPIO10 and use direction/value, etc., but nothing happens. I should also note that reading the value on GPIO11 also doesn't reflect the state of RxD (it always returns 0).

I thought that maybe it was a DTS problem, but I've tried using the DTS from 5.17.4 and 4.8.3 and appending them to uImage or loading them using dtb_file in the u-boot environment (every combination I could think of). All give the same result. It sure acts like it's doing something - if I set the value to 0 and read it back it says it's 0, but nothing changes on the TxD line.

I don't know what else to try, so I hope someone has some idea.

renojim,

> I hope I'm missing something simple. I've been
> working for the last week to update my Dockstar
> from kernel 4.8.3 to 5.17.4 and once I got past
> not being able to build LKMs I now find that I
> can't control GPIO10 (TxD) with the LKM or via
> /sys/class/gpio/gpio10.

Did it work in kernel 4.8.3? GPIO10 and 11 are undefined in the DTS, so they are free and should be OK to control through /sys/class/.

Unless something changed in the GPIO driver recently that we are not aware of.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Yes, they worked in 4.8.3. I updated a working system from 4.8.3 to 5.17.4 and suddenly I can't control the pins. I first suspected that it was something with my LKM, but when I couldn't control them through /sys/class I suspected there's something more going on. TxD/RxD are the easiest GPIO pins to access on the Dockstar, so that's why I used them. Everything in /sys/class acts as if it's working, so I'm wondering if some other GPIOs are getting mapped to 10 and 11. I have other Kirkwood boxes (the M300 and a Pogoplug) I could experiment with to see if the same thing happens going from 4 to 5.

Perhaps you can play with the new userspace tool.

https://packages.debian.org/bullseye/gpiod

Quote

Package: gpiod (1.6.2-1)

Tools for interacting with Linux GPIO character device - binary

libgpiod encapsulates the ioctl calls and data structures behind a straightforward API. This new character device interface guarantees all allocated resources are freed after closing the device file descriptor and adds several new features that are not present in the obsolete sysfs interface (like event polling, setting/reading multiple values at once or open-source and open-drain GPIOs).

This package contains the gpiod binary tools.

The sysfs interface is deprecated and will be only mantained. IOW, it should work still.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks for the suggestion, but unfortunately (or maybe it's a good thing?) I get the same results. With the 4.8.3 kernel I can use gpioset/get, but with 5.17.4, while they appear to work, nothing happens.

I removed all references to the LEDs from the dtb and I am able to control them via gpioset and /sys/class in both kernels, but not TxD/RxD. Maybe it has something to do with GPIO10/11 being used by the UART, but I even tried removing all references to the UART from the DTB and that didn't seem to help.

I don't know how reliable gpioinfo is since it shows pretty much every GPIO as unused. That can't be right. Once I removed the LEDs and the UART from the dtb the only used GPIO according to gpioinfo is for the regulators. It's interesting to note that it never shows 10 or 11 as being used whether they're in the dtb or not.

Maybe I'll search for other GPIOs that I can use/control and actually get to on the board. I never really liked using TxD/RxD, but at the time they were the only ones I knew how to access on the board other than the LEDs which I don't want to use. I can't update the kernel unless my peripherals are going to work.

Quote

I removed all references to the LEDs from the dtb and I am able to control them via gpioset and /sys/class in both kernels, but not TxD/RxD. Maybe it has something to do with GPIO10/11 being used by the UART, but I even tried removing all references to the UART from the DTB and that didn't seem to help.

OK. Remove the GPIO for LEDs allowed you to control them through /sys/class. So we know that has not been touched.

Suggestion: copy the od Dockstar DTB from previous kernel to the new kernel and use that.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I already tried that. They're pretty much the same anyway. I'm going to try another gpio in that bank and some in the other bank. I don't have to use TxD/RxD if I can find others that work, but it is mysterious why it doesn't work anymore. I may also try an older 5.x kernel to see what happens.

It looks like I can control GPIOs in the second bank using 5.17.4, but not the first bank. I tried 10, 11, 14, and 15 and they all give the same result - gpioset doesn't give any error, but the level of the pin never changes. However, I was able to control 49 (17 in the second bank). I haven't tried a different 5.x kernel yet.

I've narrowed down the problem to some regression that occurred between kernel version 5.6.3 and 5.7.1. I'm trying to build the kernel myself so I can experiment. Any help in that regard would be appreciated. Is there a particular make target that would speed up the process? Just running make seems to build a lot of unnecessary stuff.



Edited 1 time(s). Last edit at 05/26/2022 09:18PM by renojim.

Bodhi includes the config file he used to generate the kernel in the tarball he distributes with each kernel. For instance linux-5.17.4-kirkwood-tld-1-bodhi.tar.bz2 contains the config-5.17.4-kirkwood-tld-1 file. You move that file to the root of your kernel tree and rename it .config. Then run make menuconfig to select any additional configuration changes you wish to make.

I think JT is already aware of what Ray said.

But this is the Kirkwood kernel, i.e. works all Kirkwood boxes that I supports here. I don't have a small target such as specific target for the Dockstar. You could take out things like sounds, dvb video. But even that does not make it significantly built faster.

For me, I don't have a need for speed. When the build goes, it will be done whenever it's done :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thanks guys! Yeah, I figured out the set up and even how to cross compile to speed it up, but it still takes hours on my Ubuntu virtual machine. Making a change and waiting hours or overnight to see if it makes a difference is painful. It seems like make always pretty much starts over from scratch or otherwise doesn't like something I'm doing.

I ran into the wrong binaries in the headers package issue when cross compiling. It's not really a big deal, but did you find a solution or did you just create the headers package on a Kirkwood box?

> I ran into the wrong binaries in the headers
> package issue when cross compiling. It's not
> really a big deal, but did you find a solution or
> did you just create the headers package on a
> Kirkwood box?

No, I have not looked into this issue. I rebuilt the Kirkwood kernel on an Armada 385 (armv7) box.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

That's cool. I think the problem is that the build process needs fixdep, modpost, and maybe other binaries when it builds the modules, so they have to be able to run on the host machine. I haven't figured out how to get it to cross compile them. I also haven't been able to figure out how to build just them on the target architecture. I thought "make scripts/" would do it, but so far no luck.

Edit: never mind my last edit



Edited 2 time(s). Last edit at 05/28/2022 03:59PM by renojim.

So after a solid week of building kernels and digging through documentation here's what I've discovered:

Something changed from 5.6 to 5.7. Kernels before 5.7.1 work and all of those I tried after do not. The linux kernel changelog is less than helpful.

The problem in a nutshell is that the configuration of the MPPs doesn't get changed. Using MPP10 as an example, after booting it's configured as UART0 TxD. Prior to 5.7 if you used sys/class, gpioset, or even a kernel module to set the pin high/low "something" changed the pin's configuration to a GPIO output. I have yet to figure out what that "something" was, but it's interesting to note that I can copy one file (gpiolib.c) from v5.6.19 to 5.7.1, rebuild, and it all just works. Unfortunately there are too many changes (none of which appear in the changelog) to figure out what changed.

I don't see any kernel config settings that would affect this, but being new to kernel building I could be missing something.

I have yet to find a way to change an MPP's configuration using any utility like gpioset, so the gpiod functions are pretty much useless without blasting the proper register value to the MPP Control 1 Register (0xF1010004) using something like "busybox devmem" (is there anything busybox can't do?).

I'm going to spend a little more time investigating the pinctrl functions, but if I can't come up with the "proper" way to handle this I'm just going to write to the register at 0xF1010004 directly and move on.

renojim Wrote:
-------------------------------------------------------

> configuration to a GPIO output. I have yet to
> figure out what that "something" was, but it's
> interesting to note that I can copy one file
> (gpiolib.c) from v5.6.19 to 5.7.1, rebuild, and it
> all just works. Unfortunately there are too many

So a diff between the 5.6.19 and 5.7.1 gpiolib.c files should indicate what changed.

Yeah, you'd think it would be easy, but there's a lot of changes and many, many functions to try to figure out what's really involved in simply trying to set a pin high or low. It doesn't help that they changed variable names in statements that are otherwise identical producing hundreds of meaningless diffs. I can't be the only person in the world that's using GPIOs on a Kirkwood/MVEBU box. If it was just my LKMs that had a problem I'd say it was something I'm doing, but none of the binaries in the gpiod package work either.

I've researched and come to the conclusion that the user shouldn't have to do anything with the pin controller to use GPIOs, but it's clearly not getting configured automatically as it did pre-5.7. I've pretty much spent every waking minute for more than a week thinking about it, trying different things, and going down many blind alleys and it's time to throw in the towel. I'm going to either use pinctrl functions to configure the pins (which it clearly says nothing but gpiolib should use) or blast the proper value to the register using busybox devmem. Either is clearly not the way it's supposed to be done.

I found the offending line of new code! Once I figured out how to configure the pin controller from my LKM I was able to locate the call in gpiolib.c and sure enough a change was made right before the call to cause the pin configuration to never happen (and not return any kind of error either). I'm not sure how I'm going to proceed from here. I feel like I've wasted too much time on this already.

Can you post the diff between versions that show the offending line of code? Perhaps that would shed some light on how to submit a pull request to the kernel to get the issue corrected.

Ray

This is the entire procedure:

int gpiochip_generic_request(struct gpio_chip *chip, unsigned offset)
{
#ifdef CONFIG_PINCTRL
	if (list_empty(&chip->gpiodev->pin_ranges))
		return 0;
#endif

	return pinctrl_gpio_request(chip->gpiodev->base + offset);
}

The old procedure was only the return statement. CONFIG_PINCTRL is absolutely necessary, so you can't just not set that in your .config file. Trying to follow the three structure references is beyond my patience. Just tracking them all down requires going through several header files. After going through the thousands of CONFIG options I don't think it has to do with the config. I'm leaning to something needing to be in the DTS possibly if you want to use GPIOs. A lot of the setup for the pin controller happens in the Kirkwood and/or MVEBU specific code, for example that pin ranges list that appears to be empty.

I figured out why this only affects the first bank of GPIOs. The second bank (MPP32-49) is already configured to have every pin set as a GPIO. The pin ranges list for that bank is also empty, so this procedure accomplishes nothing for that bank either. I love how this routine doesn't return any error.

Quote

I figured out why this only affects the first bank of GPIOs. The second bank (MPP32-49) is already configured to have every pin set as a GPIO. The pin ranges list for that bank is also empty, so this procedure accomplishes nothing for that bank either. I love how this routine doesn't return any error.

Welcome to the club :) you'll run into code like that very often when you play with kernel or u-boot code.

Quote

I'm leaning to something needing to be in the DTS possibly if you want to use GPIOs

Yes. Remember that the current DTS only have pin controls for GPIOs that are being used, and that is correct, nothing is wrong. So perhaps you (and/or I help) need to modify the DTS to add pin control to the enumerate whatever GPIOs you need.

Pin control is default in the kernel, and can be guaranteed to work only for the GPIOs in the DTS. So bugs might be hidden because there is something would never be executed. In this case, the Dockstar officially has on a few GPIOs. Everything else is up to the developer to add and test.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I guess TxD/RxD being in the DTS isn't enough? I'm reading "Device Tree for Dummies". :-) I'm sure I'll have some questions.

renojim,

I did not pay attention before about what you did. I read the previous posts now.

> TxD/RxD are the easiest
> GPIO pins to access on the Dockstar, so that's why
> I used them. Everything in /sys/class acts as if
> it's working, so I'm wondering if some other GPIOs
> are getting mapped to 10 and 11.

The pins were defined:

kirkwood.dtsi

/*
                         * Default UART pinctrl setting without RTS/CTS,
                         * overwrite marvell,pins on board level if required.
                         */
                        pmx_uart0: pmx-uart0 {
                                marvell,pins = "mpp10", "mpp11";
                                marvell,function = "uart0";
                        };

                        pmx_uart1: pmx-uart1 {
                                marvell,pins = "mpp13", "mpp14";
                                marvell,function = "uart1";
                        };

So the only explanation is before the code was incorrect. And someone has fixed it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

So your serial console would have problem if you use GPIO 10 and 11 for other purpose.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

The include chain:

kirkwood-dockstar.dts
--> kirkwood-6281.dts
--> kirkwood.dtsi


Try commenting out these UART pins defintion

/*
                         * Default UART pinctrl setting without RTS/CTS,
                         * overwrite marvell,pins on board level if required.
                         */
                        pmx_uart0: pmx-uart0 {
                                marvell,pins = "mpp10", "mpp11";
                                marvell,function = "uart0";
                        };

                        pmx_uart1: pmx-uart1 {
                                marvell,pins = "mpp13", "mpp14";
                                marvell,function = "uart1";
                        };

Compile the DTS.

And then run the latest kernel unpatched with the new DTB. See if the GPIO 10 and 11 can be controlled with /sys/class/gpio*.

Note: For this box GPIO 13 and 14 are definitely not needed for anything.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 06/05/2022 07:21PM by bodhi.

I'm pretty sure I tried that with 5.7.1. I have so many builds and versions of things that it's hard to keep them straight, but I have a dtb file with the suffix "nouart-noleds-noworks". I was using 5.7.1 for all my tests since that's the first version where this problem showed up. I've now switched to using 5.17.4 since that's ultimately the version I'm trying to update to.

Sure enough, removing the UART from the DTB results in it not booting and of course without serial I can't see what's happening, but the CPU does get pretty hot whatever it's doing. I then tried converting UART1 into UART0 via the DTB and removing the old references to UART0, but that also results in no booting. There's a good chance my changes weren't right, but I really thought my next idea would work. I tried changing the MPPs for UART0 to 4 and 5 which required me to also change the MPPs for the SATA.

                        pmx-uart0 {
                                marvell,pins = "mpp5\0mpp4";
                                marvell,function = "uart0";
                                phandle = <0x08>;
                        };

                        pmx-uart1 {
                                marvell,pins = "mpp13\0mpp14";
                                marvell,function = "uart1";
                                phandle = <0x09>;
                        };

                        pmx-sata0 {
                                marvell,pins = "mpp15\0mpp21\0mpp23";
                                marvell,function = "sata0";
                        };

                        pmx-sata1 {
                                marvell,pins = "mpp16\0mpp20\0mpp22";
                                marvell,function = "sata1";
                        };

It boots, but had no effect on mpp10/11 - they're still configured as UART pins and I don't think that's their default configuration. Using gpioset still doesn't change their configuration to a GPIO. Interestingly, it did have an effect on mpp4/5 which are also configured as UART pins (which the data sheet says you shouldn't do). MPP10/11 no longer appear in the DTS/DTB, so something is configuring them.

I don't know if this is progress or not. Maybe if I add MPP10/11 back to the DTS as GPIOs (not that I know how to do that), whatever is claiming them for UART0 will stop doing it or, more likely, it won't boot.

Something else that's interesting to note, if I use the dtc binary from Bullseye which is v1.6.0 or the binary from the scripts/dtc directory that came from the headers deb (which is v1.6.1), decompiling kirkwood-dockstar.dtb results in a slew of warnings. Compiling that result without any changes back into a .dtb also produces a bunch of warnings. The old version that's still on my Dockstar (v1.4) doesn't produce any warnings.