Debian buster 10.0 on Armada boxes -- udev problem with MAC address

All,

I'm close to release the MVEBU Debian buster 10.0 rootfs, but when I looked closely at dmesg, I've realized that the old bug about MAC address has surfaced again in Debian buster 10.0.

udev version:

ii  udev                                  241-5                            armhf        /dev/ and hotplug management daemon

dmesg error:

[   19.183408][ T1752] systemd-udevd[1752]: Could not generate persistent MAC address for eth0: No such file or directory

And basically this has forced the MAC address to be randomly generated with a local MAC address at each boot!!!

The net result of this bug is that after booting, it would take a few minutes for the router to assign your box static IP to the new MAC address. And then you can log in with SSH again.

This problem does not show up in the Debian Kirkwood rootfs. So I think this is not a general problem, but it is a combination of the MVEBU network chip and/or the DTS definition + udev.

Of course, the work around is to hardcoded the MAC address in /etc/network/interfaces, like we did before for these Armada boxes. And this has proven to to work consistenly during my testing.

If anyone know about this udev problem please post.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

On the Armada 370/XP devices I've worked with the mainline kernel isn't able to read the ATAG values containing the MAC address(es) though there is a patch available that adds that support. Is that still the case or has that feature made it into the mainline kernel? or is that something you've resolved with a newer u-boot?

I'm currently using a script in if-pre-up.d to grab the MAC address out of a u-boot environment variable and set them for the interface when needed. I doubt that approach makes any sense for what you're doing, it wouldn't make sense for me either if i had a better solution that could still work with the vanilla Debian kernel.

Hi 1000001101000,

> On the Armada 370/XP devices I've worked with the
> mainline kernel isn't able to read the ATAG values
> containing the MAC address(es) though there is a
> patch available that adds that support. Is that
> still the case or has that feature made it into
> the mainline kernel? or is that something you've
> resolved with a newer u-boot?

I've fixed this in my released kernels a while back. I believe some Armada boxes in mainline did, too.

In the very earlier MVBEU kernels that I released, we had this MAC address problem and so we needed to put the address in /etc/network/interfaces. In later kernels, the alias in the DTS fixed this problem. If you have a box that is still Debian stretch, you could try this in the DTS and see if you don't need the script anymore.

aliases {
                ethernet0 = &eth0;
        };

As of kernel 5.2.9-mvebu-tld-1 and Debian 9.x, this fix was still working fine. However, as soon as I upgrade to buster, the MAC address problem surfaced. Im not sure if this is the same bug. It might be, except that the new udev version is now spitting out more verbose error messages.

First I guessed that perhaps the new network device naming scheme has something to do with this bug. But removing this alias still does not help in Debian buster, so I'm going back to suspect udev as the problem.

=========

BTW, the uboot version in these boxes that I tested, is of some flavor of:

uboot_version=U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version: 2015_T1.0p18

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

If memory serves the alias only works if the relevant kernel patch is applied. Admittedly i haven't looked at that part in a year or more, it might be a good time for me to revisit, I've learned a lot since then.

I took a look at the dmesg on one of my armada boxes (Debian Buster using Vanilla 4.19.0-5) and confirmed I don't see that message, just the one from mvneta when it sets the random MAC at boot. Do you see that same message also? Even though we have different kernels/dtbs we're running the same udev/etc.

I went ahead and threw the alias in the DTB and rebooted. That didn't change anything for that particular device.

> just the one
> from mvneta when it sets the random MAC at boot.

I'm not seeing this message at all.

Off chance that the buster upgrade did not populate some udev rules files. But it could happen.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

This is what I get at boot:

[    7.037559] libphy: orion_mdio_bus: probed
[    7.143841] mvneta d0074000.ethernet eth0: Using random mac address 92:ba:10:da:c0:77

1000001101000 Wrote:
-------------------------------------------------------
> This is what I get at boot:
>
>

> [    7.037559] libphy: orion_mdio_bus: probed
> [    7.143841] mvneta d0074000.ethernet eth0:
> Using random mac address 92:ba:10:da:c0:77
>

IIRC, we did get that back in kernel 4.x. (I cannot recall which version specifically) But in later versions it was fixed in mainline.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Neat. I’ll have to give it a try in 5.2

I should mention that the MAC address works with: alias in DTS, kernel 5.1.2, and Debian 9.x.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Now it's my turn to see the random local MAC problem :)

[    3.925766] mvneta f1070000.ethernet eth0: Using random mac address 62:ff:75:e5:93:a9

It turned out the error "persistent MAC address" below was from systemd lib. Even though I am not using systemd (this rootfs is sysvinit). udev is part of systemd so it uses rules from systemd library. This is an old systemd bug that seems to have been there and never been fixed properly?

[   19.183408][ T1752] systemd-udevd[1752]: Could not generate persistent MAC address for eth0: No such file or directory

The work around people proposed was:

diff /localdisk/99-default.link.orig.debian_10.0 /lib/systemd/network/99-default.link 
12c12
< MACAddressPolicy=persistent
---
> MACAddressPolicy=none

But now it seems to have brought the random MAC problem back with this work around.

So I'm back to square one.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi 1000001101000,

> On the Armada 370/XP devices I've worked with the
> mainline kernel isn't able to read the ATAG values
> containing the MAC address(es) though there is a
> patch available that adds that support. Is that
> still the case or has that feature made it into
> the mainline kernel? or is that something you've
> resolved with a newer u-boot?

I don't think ATAG is a problem in the kernel anymore for a while. Here is a data point for the Armada 385 (the reason I think systemd-udev is the problem, i.e. perhaps a race condition).

Thecus N2350
Linux version 5.2.9-mvebu-tld-1 (root@tldDebianVM) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #1 SMP PREEMPT Mon Aug 19 12:47:39 PDT 2019
Debian 9.8

ii  udev                                   232-25+deb9u11                 armel        /dev/ and hotplug management daemon

This configuration works fine

[    4.140060] mvneta f1070000.ethernet eth0: Using hardware mac address ......

> I'm currently using a script in if-pre-up.d to
> grab the MAC address out of a u-boot environment
> variable and set them for the interface when
> needed. I doubt that approach makes any sense for
> what you're doing, it wouldn't make sense for me
> either if i had a better solution that could still
> work with the vanilla Debian kernel.

What if you run your script this way in /etc/rc.local (or use a systemd equivalence). That way an upgrade to vanilla Debian kernel has no effect on it.

ifdown
#run script that reads ethaddr
ifup

I think if I can't make any progress on this, I will just put this work around in /etc/rc.local. Hopefully this will not cause too much network interruption.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)