Welcome! Log In Create A New Profile

Advanced

Debian on Ionics Nimbus 100

Posted by mossbeachlarry 
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 22, 2024 01:11AM
Larry,

> I think I'll try to find the location in the
> kernel where the USB disconnect messages originate
> and print out a traceback to find out where the
> USB disconnect request originates. Hopefully I
> will find some condition that I can alter in the
> code, or maybe alter a setting in the dtb file.

Sounds good!

The only drawback is once you've found the trigger, it might not be apparent how to fix it.

The way the USB drive was disconnected, it seems to point to "driver sense", meaning a status byte in the Nimbus USB controller was not interpreted correctly (or the Nimbus did not set it correctly) by the driver. This is an area that was redesigned in the kernel during 2021 (from 5.10.7 to 5.15.5). I have a hunch that it is related, because the Nimbus acts like the drive was disconnected suddendly by user pulling out the drive.

BTW, I've reconfigured UBIFS back in the kernel builtin module in 6.11.4+ version. So UBIFS rootfs can be mounted during boot.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

If it is not too late, can you also add kernel module compression:
$ scripts/config --disable MODULE_COMPRESS_NONE
$ scripts/config --enable MODULE_COMPRESS
$ scripts/config --enable MODULE_COMPRESS_XZ
That reduces the disk space for the modules from ~90 MB to ~30 MB!

Every bit helps on a small rootfs. :)

Thank you,

Larry Baker



Edited 1 time(s). Last edit at 10/22/2024 01:56AM by mossbeachlarry.
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 22, 2024 04:50PM
Larry,

> That reduces the disk space for the modules from
> ~90 MB to ~30 MB!

I meant to do this for years, but forgot all about it! modules will be compressed in kernel 6.11.x.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

I found out about the Function Tracer facility in the Linux kernel, known as ftrace. I thought that would relieve me of having to add code to print a traceback whenever usb_disconnect() is called. It looks like it works fine when I try it, but I have not been able to get it to work during boot up.

I found these four useful web pages:
https://billauer.co.il/blog/2020/04/ftrace-printk-tracing-debugging/
https://www.kernel.org/doc/html/v5.0/trace/ftrace.html
https://docs.kernel.org/trace/boottime-trace.html
https://docs.kernel.org/admin-guide/kernel-parameters.html#kernelparameters
I rebuilt my UBIFS enabled version of your 6.10.11 kernel with the kernel configuration options to enable the function_graph and function tracers:
$ cd ~/linux-kernels/linux-6.10.11
$ ARCH=arm CROSS_COMPILE=arm-none-eabi- make mrproper
$ cp ../config-6.10.11-kirkwood-UBIFS-2 .config
$ scripts/config --set-str LOCALVERSION "-kirkwood-UBIFS-ftrace"
$ scripts/config --enable LOCALVERSION_AUTO
$ scripts/config --disable MODULE_COMPRESS_NONE
$ scripts/config --enable MODULE_COMPRESS
$ scripts/config --enable MODULE_COMPRESS_XZ
$ scripts/config --enable FUNCTION_TRACER
$ scripts/config --enable FUNCTION_GRAPH_TRACER
$ scripts/config --enable STACK_TRACER
$ scripts/config --enable DYNAMIC_FTRACE
$ ARCH=arm CROSS_COMPILE=arm-none-eabi- make oldconfig
$ cp .config ../config-6.10.11-kirkwood-UBIFS-ftrace
I made sure usb_disconnect() could be traced:
# grep usb_disconnect /sys/kernel/debug/tracing/available_filter_functions
usb_disconnect
I enabled tracing calls to usb_disconnect():
root@comlogger:~# echo usb_disconnect > /sys/kernel/debug/tracing/set_ftrace_filter
root@comlogger:~# echo function > /sys/kernel/debug/tracing/current_tracer
root@comlogger:~# echo > /sys/kernel/debug/tracing/trace
Then I unplugged the USB stick and reinserted it into my Nimbus.

No USB hot-plug messages appeared on the console, and there were no entries in the trace buffer:
root@comlogger:~# less /sys/kernel/debug/tracing/trace
# tracer: function
#
# entries-in-buffer/entries-written: 0/0   #P:1
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
/sys/kernel/debug/tracing/trace (END)
I am guessing that it is the USB hub itself built into the Nimbus that has been disabled by Linux. By the time I am able to trace calls to usb_disconnect(), it is too late.

Next I tried to use kernel boot command line parameters, using both ftrace and kernel.ftrace syntax:
# cat /proc/cmdline 
console=ttyS0,115200 cmdlinepart.mtdparts=orion_nand:1m@0m(u-boot),4m@1m(kernel),5m@5m(pluginfo),-(rootfs) ubi.mtd=rootfs rootfstype=ubifs root=ubi0:rootfs rw earlyprintk=serial ftrace=function ftrace_filter="usb_disconnect"
# cat /proc/cmdline
console=ttyS0,115200 cmdlinepart.mtdparts=orion_nand:1m@0m(u-boot),4m@1m(kernel),5m@5m(pluginfo),-(rootfs) ubi.mtd=rootfs rootfstype=ubifs root=ubi0:rootfs rw earlyprintk=serial kernel.ftrace=function kernel.ftrace_filter="usb_disconnect"
That didn't work either. That is, there is nothing in the trace buffer for either cmdline.

Do you have any suggestions for how to enable ftrace of usb_disconnect() during boot? If not, I'll have to resort to inserting debugging printout into the code.

Thank you,

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 02:47PM
Larry,

> Do you have any suggestions for how to enable
> ftrace of usb_disconnect() during boot? If not,
> I'll have to resort to inserting debugging
> printout into the code.

No idea how! but why not continue with git bisect? It's tedious, but usually it is a no brainer when you get to the commit that causes the problem.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 03:06PM
Hi Larry,

Could you get this info:

Without any USB drive attached, power up, interrupt serial console, and

md.l 0xF1010100 1
md.l 0xF1010140 1
usb info
usb tree
md.l 0xF1010100 1
md.l 0xF1010140 1

Attach the Lexar USB drive, and

md.l 0xF1010100 1
md.l 0xF1010140 1
usb reset 
usb info
usb tree
md.l 0xF1010100 1
md.l 0xF1010140 1

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

> Could you get this info:

As requested:

> Without any USB drive attached, power up, interrupt serial console, and
Marvell>> md.l 0XF1010100 1
00000000: 08008000    ....
Marvell>> md.l 0XF1010140 1
00000000: 08008000    ....
Marvell>> usb info
USB is stopped. Please issue 'usb start' first.
Marvell>> usb start
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 1 USB Device(s) found
       scanning bus for storage devices... 0 Storage Device(s) found
Marvell>> usb info
1: Hub,  USB Revision 2.0
 - u-boot EHCI Host Controller 
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

Marvell>> usb tree

Device Tree:
  1  Hub (480 Mb/s, 0mA)
     u-boot EHCI Host Controller 
   
Marvell>> md.l 0XF1010100 1
00000000: 08008000    ....
Marvell>> md.l 0XF1010140 1
00000000: 08008000    ....
> Attach the Lexar USB drive, and
Marvell>> md.l 0XF1010100 1
00000000: 08008000    ....
Marvell>> md.l 0XF1010140 1
00000000: 08008000    ....
Marvell>> usb reset
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 2 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Marvell>> usb info
1: Hub,  USB Revision 2.0
 - u-boot EHCI Host Controller 
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

2: Mass Storage,  USB Revision 2.0
 - LEXAR JD FIREFLY 1069A704000328051007
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x05dc  Product 0xa701 Version 17.0
   Configuration: 1
   - Interfaces: 1 Bus Powered 100mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

Marvell>> usb tree

Device Tree:
  1  Hub (480 Mb/s, 0mA)
  |  u-boot EHCI Host Controller 
  |
  +-2  Mass Storage (480 Mb/s, 100mA)
       LEXAR JD FIREFLY 1069A704000328051007
     
Marvell>> md.l 0XF1010100 1
00000000: 08008000    ....
Marvell>> md.l 0XF1010140 1
00000000: 08008000    ....
Larry Baker
bodhi,

> No idea how! but why not continue with git bisect? It's tedious, but usually
> it is a no brainer when you get to the commit that causes the problem.

The USB problem as it first appeared in Linux v5.15 is not the same USB problem now. Back then, the USB drive was not even found. As I wrote in my Oct 15 post, by the time of your v5.18.6 and later the behavior had changed. Since then, the drive is found, the partitions are enumerated, then the USB disconnect occurs.

Instead of going as far back as the original v5.15 USB problem, which no longer happens, my thought now is to find the location in your v6.10.11 kernel (or your newer 6.11.x kernel) where the decision is made to call usb_disconnect() and see how that code has changed from the past. To do that I will add code in usb_disconnect() to print a traceback so I can find that place.

Today I verified that the ftrace=function ftrace_filter="usb_disconnect" kernel boot options are honored. When I log in, I can see the values I set. However, the kernel.ftrace=function kernel.ftrace_filter="usb_disconnect" kernel boot options do not work. When I log in, the values are the still the defaults. There is supposedly a way to pass kernel boot options to the early stage of booting. I thought prepending kernel. was supposed to do that, but obviously it did not. I'll poke around some more to see if there is a way to do that.

I also tried booting from the UBIFS flash rootfs without the USB drive plugged in. When I plugged it in, there were no messages on the console and none in dmesg. and lsusb did not list the LEXAR USB drive.

More later.

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 04:39PM
Hi Larry,

Are you running the Sheevaplug or the Nimbus above?

> Marvell>> md.l 0XF1010100 1
> 00000000: 08008000    ....
> Marvell>> md.l 0XF1010140 1
> 00000000: 08008000    ....

> Marvell>> md.l 0XF1010100 1
> 00000000: 08008000    ....
> Marvell>> md.l 0XF1010140 1
> 00000000: 08008000    ....

> Marvell>> md.l 0XF1010100 1
> 00000000: 08008000    ....
> Marvell>> md.l 0XF1010140 1
> 00000000: 08008000    ....

> Marvell>> md.l 0XF1010100 1
> 00000000: 08008000    ....
> Marvell>> md.l 0XF1010140 1
> 00000000: 08008000    ....



In the previous log fresh-install-on-bodhi-6.9.6-kirkwood-tld-1-on-UBIFS-rootfs.log, the output were:

U-Boot 2011.03-rc1 (Jun 23 2011 - 14:26:27)
IONICS-PlugComputer NIMBUS E0

SoC:   Kirkwood 88F6281_A0
DRAM:  512 MiB
NAND:  512 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
88E1116 Initialized on egiga0
Hit any key to stop autoboot:  3  0 
Marvell>> md.l 0xF1010000 8
f1010000: 01111111 11113322 00001111 00100000    ...."3..........
f1010010: 00000000 00000000 00000000 00000000    ................

Marvell>> md.l 0xF1010100 1
f1010100: dfff ffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: fffdffff    ....

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

> Are you running the Sheevaplug or the Nimbus above?

Always on my Nimbus.

I'll have a go at it again. Plus, at times I have used a 4-port powered USB hub, but at the moment I am not using that. I'll find out whether that matters.

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 06:11PM
Larry,

I hope the most recent log was correct. The GPIO numbers would make more sense. I suspect the GPIOs on the Nimbus are different from Sheevaplug.

> Marvell>> md.l 0XF1010100 1
> 00000000: 08008000    ....
> Marvell>> md.l 0XF1010140 1
> 00000000: 08008000    ....

Meaning:
Bank 1: GPIO 27  GPIO 15
Bank 2: 27  15 (therefore GPIO 59 and 47) ---- GPIO 47 should be for the SD slot.

Now if you run the same memory display commands on the Sheevaplug and we might see something interesting. My Sheevaplug is off line. Perhaps you can do it easier.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

> I hope the most recent log was correct. The GPIO numbers would make more sense.
> I suspect the GPIOs on the Nimbus are different from Sheevaplug.

No such luck. :( I found the reason for the change in the numbers is that it matters whether the hex is written 0x vs. 0X:
Marvell>> md.l 0xF1010100 1
f1010100: dfffffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: fffdffff    ....
Use 0X instead of 0x:
Marvell>> md.l 0XF1010100 1
00000000: 08008000    ....

Marvell>> md.l 0XF1010400 1
00000000: 08008000    ....
Go figure.

Here's what I should have found for you:
Marvell>> reset
resetting ...


U-Boot 2011.03-rc1 (Jun 23 2011 - 14:26:27)
IONICS-PlugComputer NIMBUS E0

SoC:   Kirkwood 88F6281_A0
DRAM:  512 MiB
NAND:  512 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
88E1116 Initialized on egiga0
Hit any key to stop autoboot:  0 
Marvell>>
Nothing plugged in to the USB port:
Marvell>> md.l 0xF1010100 1
f1010100: dfffffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: fffdffff    ....
Plug in the USB drive:
Marvell>> md.l 0xF1010100 1
f1010100: dfffffff    ....

Marvell>> md.l 0xF1010140
f1010140: fffdffff    ....

Marvell>> usb start
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 2 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found

Marvell>> usb info
1: Hub,  USB Revision 2.0
 - u-boot EHCI Host Controller 
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

2: Mass Storage,  USB Revision 2.0
 - LEXAR JD FIREFLY 1069A704000328051007
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x05dc  Product 0xa701 Version 17.0
   Configuration: 1
   - Interfaces: 1 Bus Powered 100mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

Marvell>> usb tree

Device Tree:
  1  Hub (480 Mb/s, 0mA)
  |  u-boot EHCI Host Controller 
  |
  +-2  Mass Storage (480 Mb/s, 100mA)
       LEXAR JD FIREFLY 1069A704000328051007
     
Marvell>> md.l 0xF1010100 1
f1010100: dfffffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: fffdffff    ....
Sorry,

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 07:12PM
Larry,

OK :) we should compare it with the Sheevaplug.

When the bits are all f's like this dfffffff, it looks very fishy. Like active low, but it's a Sheevaplug clone, I don't think they change it.

Please do these 2 peeks again before and after init_ionics. Perhaps they set the GPIOs in there.

ver

md.l 0xF1010100 1
md.l 0xF1010140 1

init_ionics mode bootup

md.l 0xF1010100 1
md.l 0xF1010140 1

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 10/23/2024 07:12PM by bodhi.
bodhi,

> Please do these 2 peeks again before and after init_ionics. Perhaps they set the GPIOs in there.

Very interesting:
Marvell>> ver

U-Boot 2011.03-rc1 (Jun 23 2011 - 14:26:27)
IONICS-PlugComputer NIMBUS E0
arm-none-linux-gnueabi-gcc (GCC) 4.2.1
GNU ld (GNU Binutils) 2.18.50.20070820

Marvell>> md.l 0xF1010100 1
f1010100: dfffffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: fffdffff    ....

Marvell>> init_ionics mode bootup

Marvell>> md.l 0xF1010100 1
f1010100: fffdffff    ....

Marvell>> md.l 0xF1010140 1
f1010140: dfffffff    ....
The bits changed! You are clever. :)

Larry Baker
bodhi,

On a SheevaPlug:
Marvell>> ver

U-Boot 1.1.4 (Dec 27 2009 - 22:03:21) Marvell version: 3.4.27 - pingtoo patch.01

Marvell>> md.l 0xF1010100 1
f1010100: 20000000    ... 

Marvell>> md.l 0xF1010140 1
f1010140: fffc7fff    ....
USB drive inserted:
Marvell>> md.l 0xF1010100 1
f1010100: 20000000    ... 

Marvell>> md.l 0xF1010140 1
f1010140: fffc7fff    ....

Marvell>> usb start
(Re)start USB...
USB:   scanning bus for devices... 2 USB Device(s) found
Waiting for storage device(s) to settle before scanning...
1 Storage Device(s) found

Marvell>> usb info
1: Hub,  USB Revision 2.0
 - Marvell EHCI 
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Settings 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

2: Mass Storage,  USB Revision 2.0
 - LEXAR JD FIREFLY 1069A704000328051007
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x05dc  Product 0xa701 Version 17.0
   Configuration: 1
   - Interfaces: 1 Bus Powered 100mA
     Interface: 0
     - Alternate Settings 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

Marvell>> usb tree

Device Tree:
  1  Hub (480MBit/s, 0mA)
  |  Marvell EHCI 
  |
  +-2  Mass Storage (480MBit/s, 100mA)
       LEXAR JD FIREFLY 1069A704000328051007
     
Marvell>> md.l 0xF1010100 1
f1010100: 20000000    ... 

Marvell>> md.l 0xF1010140 1
f1010140: fffc7fff    ....
Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 10:21PM
Larry,

OK. So now when we dump the GPIOs on the Sheevaplug at serial console, most likely they are different.

The following theory needs proof because everybody says this is a Sheevaplug clone.

The tentative conclusion is GPIO 29 for USB power (which is used on the other Kirkwood plugs such as Sheevaplug, and its close cousins Dockstar, GoFlex Net/Home, Pogo E02) is not used in this Nimbus board.

It's common for a manufacturer to design the board differently and assign a different GPIO for USB power (the 86281SoC pins are multiplexed).

So my wild guess is this u-boot powered USB during boot, and then the kernel at ~10 secs or so cleared the pins using what's in the DTS. Perhaps the earlier kernels did not, but recent changes in pinctrl driver did reset the pins.

We can hack it by setting the GPIO in Linux userspace.

Using the info
fffd ffff = fff 1101 ffff 
GPIO 16 18 19

Plug in the USB drive, and check dmesg each time you poke the GPIO 16, 18, and 19. See if the kernel will spin up the drive.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 10:25PM
Larry,

We are cross posting! Please see my post above. It does show GPIO 29 is set in the Sheevaplug as I expected.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

> Plug in the USB drive, and check dmesg each time you poke the GPIO 16, 18, and 19.
> See if the kernel will spin up the drive.

LOL! I know what you are talking about, but I have no idea how to do it!

Please give me the commands to issue from Linux. :)

Thank you,

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 10:41PM
Larry,

It's in the link posted above: "we can hack it by..."

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

I don't really know what I'm doing, but here goes:
# 
# # NO USB DRIVE PLUGGED IN
# 
# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# 
# # PLUG IN USB DRIVE
# 
# # NOTHING
# 
# # POKE GPIO PIN 16
# 
# /bin/echo 16 > /sys/class/gpio/export
# /bin/echo "out" > /sys/class/gpio/gpio16/direction
# /bin/echo 1 > /sys/class/gpio/gpio16/value

# dmesg | tail
[   15.482971] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[   15.543749] systemd[1]: Started systemd-journald.service - Journal Service.
[   16.393551] systemd-journald[154]: Received client request to flush runtime journal.
[   16.436478] systemd-journald[154]: File /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal corrupted or uncleanly .
[   18.741204] random: crng init done
[   21.952223] orion_wdt: Initial timeout 21 sec
[   22.567239] marvell-cesa f1030000.crypto: CESA device successfully registered
[   26.608641] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[   38.689928] systemd-journald[154]: Oldest entry in /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal is older tha.
[   38.741231] systemd-journald[154]: /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal: Journal header limits reach.

# /bin/echo 0 > /sys/class/gpio/gpio16/value
# /bin/echo 16 > /sys/class/gpio/unexport
# 
# # POKE GPIO PIN 18
# 
# /bin/echo 18 > /sys/class/gpio/export
# /bin/echo "out" > /sys/class/gpio/gpio18/direction
# /bin/echo 1 > /sys/class/gpio/gpio18/value

# dmesg | tail
[   15.482971] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[   15.543749] systemd[1]: Started systemd-journald.service - Journal Service.
[   16.393551] systemd-journald[154]: Received client request to flush runtime journal.
[   16.436478] systemd-journald[154]: File /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal corrupted or uncleanly .
[   18.741204] random: crng init done
[   21.952223] orion_wdt: Initial timeout 21 sec
[   22.567239] marvell-cesa f1030000.crypto: CESA device successfully registered
[   26.608641] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[   38.689928] systemd-journald[154]: Oldest entry in /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal is older tha.
[   38.741231] systemd-journald[154]: /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal: Journal header limits reach.

# /bin/echo 0 > /sys/class/gpio/gpio18/value
# /bin/echo 18 > /sys/class/gpio/unexport
# 
# # POKE GPIO PIN 19
# 
# /bin/echo 19 > /sys/class/gpio/export
# /bin/echo "out" > /sys/class/gpio/gpio19/direction
# /bin/echo 1 > /sys/class/gpio/gpio19/value

# dmesg | tail
[   15.482971] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[   15.543749] systemd[1]: Started systemd-journald.service - Journal Service.
[   16.393551] systemd-journald[154]: Received client request to flush runtime journal.
[   16.436478] systemd-journald[154]: File /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal corrupted or uncleanly .
[   18.741204] random: crng init done
[   21.952223] orion_wdt: Initial timeout 21 sec
[   22.567239] marvell-cesa f1030000.crypto: CESA device successfully registered
[   26.608641] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[   38.689928] systemd-journald[154]: Oldest entry in /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal is older tha.
[   38.741231] systemd-journald[154]: /var/log/journal/21927d5753f772dd477263aa6718016e/system.journal: Journal header limits reach.

# /bin/echo 0 > /sys/class/gpio/gpio19/value
# /bin/echo 19 > /sys/class/gpio/unexport
Nothing seems to have happened. :(

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 23, 2024 11:50PM
Larry,

How about poking a GPIO, and then plug in the drive?

====

Also, I don't recall seeing stock boot log. Do you have one?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

First, I really appreciate the time you are taking to find the right answer. You are much more familiar with ARM SoC DTBs than I am.

> How about poking a GPIO, and then plug in the drive?

Will do. Read what I am attaching first, in case it informs you better what GPIO pins to poke.

> Also, I don't recall seeing stock boot log. Do you have one?

I don't know what that means.

I have something that may save you a whole lot of time, assuming you are on the right track: the Ionics-EMS Nimbus 101 SDK v6.1-p1.tar.bz2. It is very comprehensive, including the upstream Marvell U-Boot and Linux kernel source archives and the Nimbus patches to those. It is too big I think (520 MB) to attach to this message, but I am attaching their U-Boot patch to the Marvell U-Boot, FIRMWARE-IONICS-NIME0-release-2.0-p1/BOOTLOADER/PATCH/IONICS-NIME0-uboot.patch. It should help you figure out what they do to configure the hardware with their init_ionics mode command, e.g., for a normal boot (bootup mode), a boot to write an image from USB to flash (flashing mode), and to shut down (power off?) (shutdown mode).

Larry Baker
Attachments:
open | download - IONICS-NIME0-uboot.patch (17.3 KB)
bodhi,

I uploaded Nimbus 101 SDK v6.1-p1.tar.bz2 to my Google Drive for you. Here's the link:
https://drive.google.com/file/d/1dGnPQuPU-7rBPw_mr9_GyrzOvZdlE8q9/view?usp=sharing
Larry Baker
bodhi,

> How about poking a GPIO, and then plug in the drive?

Did not help. Poking GPIO pins 16 and 18 was uneventful, poking GPIO pin 19 froze the terminal when I typed dmesg | tail. I had to reset the Apple Terminal window and close and reopen my minicom connection to the Nimbus USB tty console port. Definitely not the same experience older Linux kernels have hot-plugging USB drives.

Larry Baker
bodhi,

As far as I can tell from reading the Nimbus U-Boot cmd_ionics.c and the SheevaPlug kirkwood-sheevaplug-common.dtsi, GPIO pin 29 is USB Power on both the SheevaPlug and the Nimbus. I can see the code in the Nimbus U-Boot sheevaplug.c where the GPIO Data Out registers are initialized to 0xdfffffff and 0xfffdffff on power up and changed to 0xfffdffff and 0xdfffffff by init_ionics mode bootup in cmd_ionics.c, which transitions the USB Power from OFF to ON.

I tried to enable GPIO pin 29 to read its value and toggle it, but I could not. When I tried to write 29 to /sys/class/gpio/export, the echo command came back with write error: Device or resource busy. I assume that is because pin 29 is described in the DTB file as an OS resource.

Larry Baker
bodhi,

I do think you are on the right track to look at the USB Power management. Of course, that may be a side effect of some difference in the behaivor of the hardware between the Nimbus and a SheevaPlug, and cutting USB power is the result of that.

I plugged in a USB RS-232 adapter instead of a USB disk. As the Nimbus is booting the 6.10.11 kernel, the red LED in the head of the RS-232 adapter is on. After 6.10.11 finishes booting, it is off. When I insert a powered USB hub in the middle, the LED is on again, but lsusb still shows only the internal USB hub device, nothing that is plugged in. So, that USB Power toggling seems to disable 5V power to the USB device as well as disable the recognition that a device is plugged in. I wish there was a way to view that USB Power GPIO pin 29 state during all this time.

Also, as I have mentioned, it made no difference to this behavior whether I used the OEM Nimbus U-Boot or a much newer SheevaPlug U-Boot. Using either one, the Debian 11 SheevaPlug installer sees the USB installation drive, but the Debian 12 SheevaPlug installer does not. I think that means both U-Boots are manipulating the same USB Power GPIO pin 29. I also see in the Nimbus U-Boot code that they use the same GPIO pins 46 and 49 for the red and blue (green on Nimbus) LEDs.

I have a video call in a few minutes. I'll get back to looking at the kernel code after that. And I'll try to find a tool or method to monitor the USB Power GPIO pin 29 state from userland.

Larry Baker
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 24, 2024 01:19PM
Larry,

> First, I really appreciate the time you are
> taking to find the right answer.

No worries! I only spend a few minutes here and there.

> I have something that may save you a whole lot of
> time, assuming you are on the right track: the
> Ionics-EMS Nimbus 101 SDK v6.1-p1.tar.bz2.
> It is very comprehensive, including the upstream
> Marvell U-Boot and Linux kernel source archives
> and the Nimbus patches to those. It is too big I
> think (520 MB) to attach to this message, but I am
> attaching their U-Boot patch to the Marvell
> U-Boot,
> FIRMWARE-IONICS-NIME0-release-2.0-p1/BOOTLOADER/PATCH/IONICS-NIME0-uboot.patch.
> It should help you figure out what they do to
> configure the hardware with their init_ionics
> mode
command, e.g., for a normal boot
> (bootup mode), a boot to write an image
> from USB to flash (flashing mode), and to
> shut down (power off?) (shutdown mode).

Duh! the GPL source is the 2nd thing we always look for (the 1st is the serial stock boot log) when hacking these embedded Linux devices. I did not ask you if there is such GPL source assuming it is a real Sheevaplug clone.

The stock boot log is the log captured over serial console when you first boot the box into stock FW (OS). Usually there is a wealth of information about anything out of the ordinary that stock OS does.

Yes. This patch documentation explicitly stated that GPIO 29 is the USB power, just like the Sheevaplug. And the bootup mode explicitly set the GPIOs. But the values do not make sense! very poor coding style.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodi,

> The stock boot log is the log captured over serial console when you first boot
> the box into stock FW (OS). Usually there is a wealth of information about
> anything out of the ordinary that stock OS does.

I attached log files in earlier posts with full minicom captures of the console output. Those are not as far back as the original OEM Linux, which I wiped out.

I have been using the OEM U-Boot, so that any Nimbus-specific device configuration is done. Thus far it seems like the newer SheevaPlug U-Boot should work fine. What you have been figuring out seems like maybe some tweaks to the SheevaPlug DTB file might solve the problem in the end. As you say, we need more data. More proof. :)

> But the values do not make sense! very poor coding style.

Yeah, look at the DATA in the table of memory device programming values. They took out all the comments Marvell had in their DATA table. It makes it much harder to see what they changed to at least figure out if a Nimbus-specific DTB is needed to take care of those details.

Were you able to download the Nimbus SDK I uploaded on my Google Drive for you?

Larry Baker



Edited 1 time(s). Last edit at 10/24/2024 02:13PM by mossbeachlarry.
Re: Please add built-in kernel UBIFS support for Kirkwood and any other boards with on-board eMMC flash
October 24, 2024 02:32PM
Larry,

> > The stock boot log is the log captured over
> serial console when you first boot
> > the box into stock FW (OS). Usually there is a
> wealth of information about
> > anything out of the ordinary that stock OS
> does.
>
> I attached log files in earlier posts with full
> minicom captures of the console output.
> Those are not as far back as the original OEM
> Linux, which I wiped out.

The stock boot log is the log for the original OEM Linux. What we ususally do is to boot into stock OS at least once, and capture the log, backup the mtds so that it can be restored later (to find some info).

There are odd thing the stock kernel might do such as running some scripts that alter the behaviour of vanilla Linux. In some cases, they missed something in u-boot, so they implemented it in the kernel (much easier to do).

> Were you able to download the Nimbus SDK I uploaded on my Google Drive for you?

I have not done that.

==========

OK here is another thing to do that is extremely useful: running kwboot to load u-boot over UART and run it. This is the rescue path in case your NAND is corrupted, or being destroyed by too many writes (running Debian on NAND tends to do that if you update the system too often).

See the example here in the Wiki thread

Quote

Unbricking with Serial Console & JTAG console

How to unbrick your box using serial console with kwboot (Zyxel NAS pinouts and examples)

First, download the 2017.07 u-boot for Sheevaplug.

Quote

uboot.2017.07-tld-2.sheevaplug.bodhi.tar or
https://www.dropbox.com/s/h2nmofkd15dd7um/uboot.2017.07-tld-1.sheevaplug.bodhi.tar
md5
0fcae49e99ca3e7d24da31177c230c93
sha256
a645a6602f987b0c938e52c87fd3044b12c810299977a1899309a6ce28e6ae8c

And use the image uboot.2017.07-tld-2.sheevaplug.mtd0.kwb to run kwboot with.

If you can run kwboot successfully, then your box is unbrickable.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
bodhi,

> If you can run kwboot successfully, then your box is unbrickable.

I don't think I need to do that. I can (and have) restore any U-Boot and rootfs over the JTAG connection using the OpenOCD tools on the Nimbus SDK.

I have already tested a more recent version of U-Boot than 2017.07. Attached are my notes (Install Debian Kirkwood ARMEL Linux.docx) for flashing the Debian SheevaPlug U-Boot 2023.01+dfsg-2+deb12u1, which I did from a FAT USB drive using the existing U-Boot:
U-Boot 2023.01+dfsg-2+deb12u1 (Apr 18 2024 - 22:00:21 +0000)
Marvell-Sheevaplug

SoC:   Kirkwood 88F6281_A0
DRAM:  512 MiB
Core:  14 devices, 11 uclasses, devicetree: separate
NAND:  512 MiB
MMC:   mvsdio@90000: 0
Loading Environment from NAND... OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0 
=>
Also attached are my notes (Backup and Restore Debian Linux.docx) for using the Nimbus OpenOCD tools to completely wipe out and restore the flash.

I already have the means and the experience to unbrick my Nimbus, should I ever need to do that. I could probably use the Nimbus OpenOCD tools to do the same on my SheevaPlugs. I may already have notes for how to do that too. I like to be prepared. :)

I'm good.

P.S. Feel free to post any of my notes if you think they would be useful to others.

Thank you,

Larry Baker



Edited 1 time(s). Last edit at 10/24/2024 03:18PM by mossbeachlarry.
Attachments:
open | download - Install Debian Kirkwood ARMEL Linux.docx (41.2 KB)
open | download - Backup and Restore Debian Linux.docx (44.6 KB)
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: