Time is lagging substantially on Netgear ReadyNAS RN102 and RN104

Hi Trond,

> for providing these outdated kernels for testing.
> On Netgear RN102, the lagging issue is present in
> both Linux 4.20.6 and 5.8.5.

> So the issue has been around since at least March
> 2019. Then we know that it has not been introduced
> by any of the more resent patches to the kernel or
> the device tree.

Bummer! I hope it were a regression somewhere.

It's been a while, I forgot, if stock OS have this problem or just in Linux? If stock runs OK then it must be some SoC specific configuration that we are missing.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Perhaps you should try the kernel images hosted at Natisbad (where support for the device was first introduced) to see if the issue has been present from the introduction of support for this device.



Edited 1 time(s). Last edit at 04/02/2023 03:02AM by rayknight.

Hi rayknight,

Quote

Perhaps you should try the kernel images hosted at Natisbad (where support for the device was first introduced) to see if the issue has been present from the introduction of support for this device.

Great suggestion!

I downloaded the Linux kernel version 4.4.1 'uImage' and the Debian installer 'uInitrd' for "Efikamx mx5" from Natisbad to a USB stick, placed it in the front port of my RN102 and, using serial console, modified the U-Boot environment as for a first install. The outdated installer was not able to complete due to missing mirrors for Debian 7 (Wheezy), but escaping to a shell made it possible to access '/dev/uptime':

~ # uname -a
Linux RN102 4.4.1.rn102 #1 Tue Feb 9 23:58:40 CET 2016 armv7l GNU/Linux
~ # cat /proc/uptime
326.53 312.33
~ # cat /proc/uptime
381.52 367.32
~ #

The time between the two 'cat' commands was 60 s, but the system clock gained only 55 s.

Repeating the test with Linux kernel version 3.14.44 gave a similar result:

BusyBox v1.20.2 (Debian 1:1.20.0-7) built-in shell (ash)
Enter 'help' for a list of built-in commands.

~ # uname -a
Linux RN102 3.14.44.rn102 #1 Thu Jun 11 18:30:21 CEST 2015 armv7l GNU/Linux
~ # cat /proc/uptime
113.57 99.86
~ # cat /proc/uptime
168.80 155.08
~ #

I conclude that the time lagging issue with the mainline Linux kernel for Netgear RN102 and RN104 has been there "forever".

Regards,
Trond Melen

Hi Trond,


The significant difference between the Mirabox and this RN 102/104 is: the oscillator.

armada-370-netgear-rn104.dts

g762: g762@3e {
                                        compatible = "gmt,g762";
                                        reg = <0x3e>;
                                        clocks = <&g762_clk>; /* input clock */
                                        fan_gear_mode = <0>;
                                        fan_startv = <1>;
                                        pwm_polarity = <0>;
                                };
...

        clocks {
                g762_clk: g762-oscillator {
                        compatible = "fixed-clock";
                        #clock-cells = <0>;
                        clock-frequency = <8192>;
                };
        };

But I can't see how this has an effect.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 06/15/2024 02:38PM by bodhi.

Hi bodhi,

Quote

The significant difference between the Mirabox and this RN 102/104 is: the oscillator.

IIRC, the SOC in Mirabox is a stripped down version of the one in RN 102/104, so the oscillators are the same. If the DTSs are different, Mirabox seems to have the right one. Yes?

Regards,
Trond Melen

Hi Trond,

> IIRC, the SOC in Mirabox is a stripped down
> version of the one in RN 102/104, so the
> oscillators are the same. If the DTSs are
> different, Mirabox seems to have the right one.
> Yes?

There is no oscilator defined in the Mirabox DTS.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

It might not be related. But another significant difference is the RN102/104 use PCIe SATA controller chip 88SE9170. The Mirabox does not have SATA, the PCIe bus is used for USB3 controller.


https://wikidevi.wi-cat.ru/Netgear_ReadyNAS_RN104
https://wikidevi.wi-cat.ru/GlobalScale_MiraBox_V5

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Trond,

Note that the oscillator g762 here is for PWM Fan Controller.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

I see. Sorry for overlooking the details!

I strongly believe the cause of the lagging system clock must be inside the SOC since apparently time is lagging as soon as the kernel starts its internal timer (see https://forum.doozan.com/read.php?2,133814,133889#msg-133889 above).

Regards,
Trond Melen

Hi Trond,

Quote

I strongly believe the cause of the lagging system clock must be inside the SOC since apparently time is lagging as soon as the kernel starts its internal timer (see https://forum.doozan.com/read.php?2,133814,133889#msg-133889 above).

I recalled I looked at the DTS for the clock source. But, yes it's worth a look again.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Trond,

Revisiting this subject. I don't have an answer, but just laying out my observation.

> Hi boidhi,
>
> Comparing our 'dmesg' logs, I find it interesting
> that apparently time is lagging as soon as the
> kernel starts its internal timer:
>

> RN102            0          0   0.000001  
> 0.007872   0.019438   0.024168   0.034290  
> 0.110959   0.117666   0.126529   0.134613  
> 0.144256   0.151252   0.158862   2.054169  
> 2.078575   2.089990   2.099139   2.120027  
> 2.124320   2.160606
> Mirabox          0          0   0.000001  
> 0.008585   0.021045   0.026200   0.037233  
> 0.120039   0.127140   0.135209   0.143927  
> 0.154436   0.162033   0.170095   2.080222  
> 2.090034   2.111673   2.120682   2.141373  
> 2.146034   2.178996
> Ratio          NaN        NaN   1.000000  
> 0.916948   0.923640   0.922443   0.920957  
> 0.924358   0.925484   0.935803   0.935287  
> 0.934083   0.933464   0.933960   0.987476  
> 0.994517   0.989732   0.989841   0.990032  
> 0.989882   0.991560
>

>
> I make 2 observations:
>

>
• Already after 8585 us (Mirabox time), RN102
  > time is lagging ~8%. This remains so for a while.
  >
• After 2 s, time is the same within ~1% on both
  > boxes. Time seems to have been synchronized,
  > presumably with the RTC.
  >

>
> My first hypothesis was that the RN102 and Mirabox
> share the same CPU clock frequency setup, but that
> the RN102 is actually running from a slower clock
> source than the Mirabox. If so, however, the boot
> steps on the RN102 should both take longer to
> complete and be timed by a slower clock, so
> 'dmesg' should report the same progress for both
> boxes.
>
> What do you make of this?

I looked at the 3 boot logs. And I also looked at the kernel code. We can see the timer frequency is different right of the bat.

bodhi's 6.9.6

root@Mirabox:~# dmesg | grep -iE 'source|clock'

[    0.000000] Switching to timer-based delay loop, resolution 53ns
[    0.000006] sched_clock: 32 bits at 19MHz, resolution 53ns, wraps every 114840871909ns (114.840871909 seconds)

[    0.008914] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 102208375848 ns (102.208 375 848 seconds)
[    0.043443] Calibrating delay loop (skipped), value calculated using timer frequency.. 37.39 BogoMIPS (lpj=186996)
[    0.364886] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns (19 112 604 462 750 000)

[    1.711262] PTP clock support registered
[    1.744423] clocksource: Switched to clocksource armada_370_xp_clocksource
[   15.772193] clk: Disabling unused clocks

Jungle's 6.9.6

[    0.000000][    T0] Switching to timer-based delay loop, resolution 53ns
[    0.000007][    T0] sched_clock: 32 bits at 19MHz, resolution 53ns, wraps every 114537122277ns (114.537122277)
 
[    0.008271][    T0] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 101938038664 ns (101.938 038 664 seconds)
[    0.042877][    T0] Calibrating delay loop (skipped), value calculated using timer frequency.. 37.49 BogoMIPS (lpj=187492)
[    0.416352][    T1] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns (19 112 604 462 750 000)

[    1.820829][    T1] PTP clock support registered
[    1.860976][    T1] clocksource: Switched to clocksource armada_370_xp_clocksource
[   25.819221][    T1] clk: Disabling unused clocks

Trond 6.8.7

[    0.000000][    T0] Switching to timer-based delay loop, resolution 53ns
[    0.000001][    T0] sched_clock: 32 bits at 19MHz, resolution 53ns, wraps every 114537122277ns

[    0.007909][    T0] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 101938038664 ns
[    0.033032][    T0] Calibrating delay loop (skipped), value calculated using timer frequency.. 37.49 BogoMIPS (lpj=187492)
[    0.179478][    T1] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns

[    1.206872][    T1] PTP clock support registered
[    1.218084][    T1] clocksource: Switched to clocksource armada_370_xp_clocksource

[    4.265160][    T1] clk: Disabling unused clocks

Note the difference in caculated wrap time, lpj , and BogoMIPs number. BogoMIPS caculation is based on timer frequency. And it seems wrong at this time:

[    0.000006] sched_clock: 32 bits at 19MHz, resolution 53ns, wraps every 114840871909ns (114.840871909 seconds)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

I agree. 114.8 s wrap time (Mirabox) is correct and 114.5 s (RN102) is wrong. So then it's wrong already after 6 us.

I successfully installed Linux kernel version 6.9.6 on one of my RN102 boxes. It seems to work fine:

$ uname -a
Linux rn102 6.9.6-mvebu-370xp-tld-2 #2 PREEMPT Wed Jul  3 17:10:27 PDT 2024 armv7l GNU/Linux

The relevant 'dmesg' lines are:

$ dmesg | grep -iE 'source|clock'
[    0.000001] sched_clock: 32 bits at 19MHz, resolution 53ns, wraps every 114537122277ns

[    0.007897] clocksource: armada_370_xp_clocksource: scale: 1 ns, freq: 18749237
[    0.015211] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 101938038664 ns
[    0.141391] clocksource: jiffies: scale: 1 ns, freq: 0
[    0.146750] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns

[    1.173534] PTP clock support registered
[    1.184299] clocksource: Switched to clocksource armada_370_xp_clocksource

[    4.169352] clk: Disabling unused clocks

Strangely, RN102 reports the wrong wrap time after only 1 us. The full log is attached.

After disabling the 'cron' job that synchronizes the system clock with the RTC every minute, I measured the lagging:

$ ssh rn102 date; sleep 60; ssh rn102 date
Sun  7 Jul 22:30:33 CEST 2024
Sun  7 Jul 22:31:28 CEST 2024

The RN102 is lagging about 5 s per minute as before.

Regards,
Trond Melen

Hi Trond,

I've added this new debug info. The clocksource frequency is showing the difference clearly

Yours and Jungle's dmesg

[    0.007897] clocksource: armada_370_xp_clocksource: scale: 1 ns, freq: 18,749,237

My Mirabox dmesg

[    0.008608] clocksource: armada_370_xp_clocksource: scale: 1 ns, freq: 18,699,646

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

And, another interesting observation.

> Strangely, RN102 reports the wrong wrap time after
> only 1 us. The full log is attached.

Schedule clock disabled all interrupts before it calculates that wrap

sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
{
	u64 res, wrap, new_mask, new_epoch, cyc, ns;
	u32 new_mult, new_shift;
	unsigned long r, flags;
	char r_unit;
	struct clock_read_data rd;

	if (cd.rate > rate)
		return;

	/* Cannot register a sched_clock with interrupts on */
	local_irq_save(flags);
.....
....

	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns\n",
		bits, r, r_unit, res, wrap);

	/* Enable IRQ time accounting if we have a fast enough sched_clock() */
	if (irqtime > 0 || (irqtime == -1 && rate >= 1000000))
		enable_sched_clock_irqtime();

	local_irq_restore(flags);

	pr_debug("Registered %pS as sched_clock source\n", read);

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi bodhi,

octave:1> 55/60
ans = 0.9167
octave:2> 18749237/18699646
ans = 1.0027
octave:3> (1-55/60)/(1-18749237/18699646)
ans = -31.423

The observed lagging is 31 times larger than the difference in reported frequency, so there must be more to it than just the clock frequency.

Regards,
Trond Melen

Hi Trond,

> The observed lagging is 31 times larger than the
> difference in reported frequency, so there must be
> more to it than just the clock frequency.

Agree. The frequency difference is possibly just an artifact of hardware difference. Wish we have GPL source for the RN102 u-boot so we can see what it does during boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Trond,

My hunch right now is that the Core Clock for the RN102/104 was set to the wrong value in u-boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi Trond,
>
> My hunch right now is that the Core Clock for the
> RN102/104 was set to the wrong value in u-boot.

The core clock (TCLK) for Armada 370 can be run at either 166 Mhz or 200 Mhz. I'm wondering if the clock tick is really run at 200 Mhz as printed out during the RN102 booting.

The correct way to get the TCLK is to read it from the SAR register. And the modern LInux kernel does that.

However, I recall some old Synology NAS seems to play some tricks forcing the kernel to use TCLK passed from u-boot to its kernel in ATAG. I need to refresh my memory about how to print the ATAG record in Linux.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

The earliest source code download for the RN102/RN104/2120 6.2.5 seems to be based on buildroot 2014.08 and uses a recipe for building u-boot using a tar file for U-Boot-2014.07. The latest source code 6.10.9 (6.10.10 link actually downloads 6.10.9) uses buildroot 2015.11 and a recipe for building u-boot using a tar file for U-Boot-2015.10. Neither of these matches the U-Boot in my example which has 2009.08, so I suspect that the build scripts don't actually build U-Boot for the device! I'm still trying to figure out how to build the source as the documentation is non-existent and I'm not familiar with the buildroot processes.

Ray

Hi Ray,

> The earliest source code download for the
> RN102/RN104/2120 6.2.5 seems to be based on
> buildroot 2014.08 and uses a recipe for building
> u-boot using a tar file for U-Boot-2014.07. The
> latest source code 6.10.9 (6.10.10 link actually
> downloads 6.10.9) uses buildroot 2015.11 and a
> recipe for building u-boot using a tar file for
> U-Boot-2015.10.

Do you have a link to this download site?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
>
> Do you have a link to this download site?


Netgear has source downloads at https://kb.netgear.com/2649/NETGEAR-Open-Source-Code-for-Programmers-GPL for most of their devices. There are quite a few releases of ReadyNAS OS 6 source available for RN102/RN104/2120.

Hi Ray,

Did you find u-boot source any where in the website https://kb.netgear.com/2649/NETGEAR-Open-Source-Code-for-Programmers-GPL?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

There is U-Boot source in the downloads for ReadyNAS OS 6 (RN102/RN104/2120), but it does not match the U-Boot binary installed on those devices.

Ray

Hi bodhi,

Quote
bodhi
Wish we have GPL source for the RN102 u-boot so we can see what it does during boot.

I found a file named "ReadyNASOS_V6.6.0_WW_src.zip" dated October 2020 on my file server, but I don't recall from where I got it. Within it there is a top level directory "u-boot-2011.12-armada370" containing the GPL source code for a version of U-Boot with build instructions for RN102/RN104. Based on its "version.txt" file, it seems to be derived from mainline U-Boot version "2013_Q1.0p1", so it should be straight forward to extract the patches applied to it.

I have created a tar file "u-boot-2011.12-armada370.tar.gz" (14.2 MB) containing said directory. You may download it here.

U-Boot on my RN102 reports version "2014_T2.0p1". They're not the same, but hopefully they do not differ too much.

Regards,
Trond Melen

Thanks Trond!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi all,

Please cold start (after shutdown for about 30 minutes) your RN102 or RN104. Interrupt u-boot at count down and dump SAR register

md.l 0x18230 1
md.l 0x18234 1

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 07/18/2024 01:45PM by bodhi.

Hi bodhi,

Quote

Please cold start your RN102 or RN104. Interrupt u-boot at count down and dump SAR register

I'm currently not co-located with my TTL-to-USB adapter. I'll report when we are reunited.

Regards,
Trond Melen

tme Wrote:
> I found a file named
> "ReadyNASOS_V6.6.0_WW_src.zip" dated October 2020
> on my file server, but I don't recall from where I

You got lucky!!! The file in question is still available from the NETGEAR Open Source Code for Programmers (GPL) page as https://www.downloads.netgear.com/files/GPL/ReadyNASOS_V6.6.0_WW_src.zip There are > 30 different versions of ReadyNAS OS available for the RN102/RN104/2120 and I had previously downloaded about 5 of them looking for U-Boot and was not successful. But that particular one has U-Boot source for armada370, armadaxp and alpine!

Ray

Hi bodhi,

Quote

Please cold start (after shutdown for about 30 minutes) your RN102 or RN104. Interrupt u-boot at count down and dump SAR register

RN102 warm start:

Marvell>> md.l 0x18230 1
00018230: e12fff1e    ../.
Marvell>> md.l 0x18234 1
00018234: e3a00000    ....

RN102 cold start:

Marvell>> md.l 0x18230 1
00018230: e12fff1e    ../.
Marvell>> md.l 0x18234 1
00018234: e3a00000    ....

Regards,
Trond Melen

Hi Trond,

Thanks for the test!

Unfortunately, I forgot to add the register base to the SAR offset :)) So I need to come back with a correct address. It should be either 0xd0000000 or 0xf1000000. In this old u-boot I think it is 0xd0000000.

BTW, bit 20 is the TCLK indicator.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 2 time(s). Last edit at 07/23/2024 02:13PM by bodhi.