No serial output from kernel on Iomega ix2-dl(ng)

Hi!

I've stuck in a strange problem while trying to install Debian on my
Iomega/LenovoEMC ix2-dl device, so I decided to ask community for help.
The situation for now is:
- have a working serial connection
- usb flash with 2 partitions (ext2 + ext3), uImage+uInitrd on the first,
the rest of rootfs on the second.
- stock uboot (v 1.1.4 from Oct 28 2011 15:19:29, Marvell version: 3.6.1 - EMC)
can read kernel and initrd into ram at 0x00800000 and 0x01100000 and tries to boot them.
- I create separate bootcmd_usb variable due not to mess stock env and use it manually
with 'run bootcmd_usb' (see in env-log attachment)
- checksums are verified correctly but I have no any output to the console
after "Starting kernel..." message.

The results are the same with all kernels I tried till now: 3.17 and 3.16 from Bodhi,
debian netinstall images from squeeze/wheezy and some other kernels I've found on
the internet. But, if I place previously dumped stock mtd3 (kernel) content to the
same usb stick as uImage - the serial output from that old (2.6.31) kernel are became visible.

Setting arcNumber and mainlineLinux variables seems has no any effect on booting
newer kernels, except that serial output from stock kernel are visible only when
mainlineLinux=no.

I know, that ix2-dl is not suppotned in a mainline and build on Marvell Armada
88F6282 A1 SoC, rather than more wide used 88F6281 Kirkwood. Also I know a problem
with absence of DTS file for ix2-dl. I searched a lot and see, that people
have problems with some not-working devices or unsupported arcNumbers, but all
of them have some of kernel debug output over serial. But I've not got a single
char of output from kernel, even "Uncompressing Linux..."!

So, do you have any suggestions how to fix this problem?
Maybe I miss something in env, like additional vars or different load addresses for uImage/uInitrd?
Maybe stock uboot are really outdated and cant boot newer 3.X kernels?

Greatly thanks for any clues, helping to fix my problem.

PS: I'm a newbie in embedded Linux, but use it on servers and desktops for a long time.

please check your kernel-parameters in u-boot there should be a line console=ttyS0,115200 you wont get output if there is no default for that. Might be some wrong addresses or crashed kernel, too

If there is no dtb file, do you use a mach-kernel? U-boot 1.1.4 does not support dtb files, only mach-kernel(special kernel patched for your device to support it). A not patched kernel will not work. You can download kernel-source from kernel.org, most patches for kirkwood you might find at archlinuxarm or maybe debian (if not included in kernel). You will need a toolchain for arm little endian (emdebian, mentorgraphics lite edtion) to compile a patched and configured kernel.

megov Wrote:
-------------------------------------------------------
> Hi!
>
> I've stuck in a strange problem while trying to
> install Debian on my
> Iomega/LenovoEMC ix2-dl device, so I decided to
> ask community for help.
> The situation for now is:
> - have a working serial connection
> - usb flash with 2 partitions (ext2 + ext3),
> uImage+uInitrd on the first,
> the rest of rootfs on the second.
> - stock uboot (v 1.1.4 from Oct 28 2011 15:19:29,
> Marvell version: 3.6.1 - EMC)
> can read kernel and initrd into ram at 0x00800000
> and 0x01100000 and tries to boot them.
> - I create separate bootcmd_usb variable due not
> to mess stock env and use it manually
> with 'run bootcmd_usb' (see in env-log
> attachment)
> - checksums are verified correctly but I have no
> any output to the console
> after "Starting kernel..." message.
>
> The results are the same with all kernels I tried
> till now: 3.17 and 3.16 from Bodhi,
> debian netinstall images from squeeze/wheezy and
> some other kernels I've found on
> the internet. But, if I place previously dumped
> stock mtd3 (kernel) content to the
> same usb stick as uImage - the serial output from
> that old (2.6.31) kernel are became visible.
>
> Setting arcNumber and mainlineLinux variables
> seems has no any effect on booting
> newer kernels, except that serial output from
> stock kernel are visible only when
> mainlineLinux=no.
>
> I know, that ix2-dl is not suppotned in a mainline
> and build on Marvell Armada
> 88F6282 A1 SoC, rather than more wide used 88F6281
> Kirkwood. Also I know a problem
> with absence of DTS file for ix2-dl. I searched a
> lot and see, that people
> have problems with some not-working devices or
> unsupported arcNumbers, but all
> of them have some of kernel debug output over
> serial. But I've not got a single
> char of output from kernel, even "Uncompressing
> Linux..."!
>
> So, do you have any suggestions how to fix this
> problem?
> Maybe I miss something in env, like additional
> vars or different load addresses for
> uImage/uInitrd?
> Maybe stock uboot are really outdated and cant
> boot newer 3.X kernels?
>
> Greatly thanks for any clues, helping to fix my
> problem.
>
> PS: I'm a newbie in embedded Linux, but use it on
> servers and desktops for a long time.

Usually serial console will show problem with bootargs and mounting rootfs. In this case most like likely the bootargs setting was wrong, or the kernel image is not a right one. Your best kernel to use is my kernel 3.16.x-tld-x (to eliminate any FDT issue and L2 cache issue).

I'll take a closer look.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@megov,

I don't think ix2-dl is supported in my patched kernel 3.16 or 3.17. You might need to find a similar box that has either DTS or non-FDT patch.

Your bootargs looks fine. It is just that the kernel image is not the right one.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

@bodhi & balou1974

Thanks for the support!

1. As for the boorargs, I'm sure that cmdline, passing to kernel, contains the right
console specs. Because, when I replace /uImage file on the usb stick
with image, extracted from stock mtd3 flash block (the old stock 2.6.31 kernel),
the kernel output is fine and I can see the same exact bootargs passed into the
kernel output.
In my situation, the other parts of bootargs, except 'console=xxx', like mtdparts,
root fs specs and hints, are not relevant ant dont needed at the first stages of boot.

2. Of course, I'll try to crosscompile my own kernel. Like I've done in those cool & old
days of 2.0.xx kernels ;) But today, I think, that there is a more generic and simple
solution.

3. As for FDT & DTS/DTB things - it's quite new to me and unusual to plain x86/x64
Linux world. So I have a question: is serial output from kernel is impossible without
this FDT/DTS stuff, binded to a kernel image? Even with a "nearest" SoCs?
I mean, that there a varying ways to implement peripherials on different boards,
based on the same or "near the same" chips/SoCs, but serial output have to be the
same among the same family of chips, like in a PC world: COM1:3f8, COM2:2f8?

4. I try to >> some DTS to a kernel zImage (3.17 bodhi) and boot the results, but with
no success. Also tried the whole 3.14 rootfs (bodhi's also!) on the single ext2 partition
on usb stick. Same result with no serial output.

5. May it can be a problem with 'Uboot is unabling to boot kernel over 3.2, due to L2
cache enable/disable issue'?

megov,

> 1. As for the boorargs, I'm sure that cmdline,
> passing to kernel, contains the right
> console specs. Because, when I replace /uImage
> file on the usb stick
> with image, extracted from stock mtd3 flash block
> (the old stock 2.6.31 kernel),
> the kernel output is fine and I can see the same
> exact bootargs passed into the
> kernel output.
> In my situation, the other parts of bootargs,
> except 'console=xxx', like mtdparts,
> root fs specs and hints, are not relevant ant dont
> needed at the first stages of boot.

Agreed. bootargs looks fine. But it is possible that there needs to be additional parameters from stock u-boot to the kernel.

> 3. As for FDT & DTS/DTB things - it's quite new to
> me and unusual to plain x86/x64
> Linux world. So I have a question: is serial
> output from kernel is impossible without
> this FDT/DTS stuff, binded to a kernel image? Even
> with a "nearest" SoCs?
> I mean, that there a varying ways to implement
> peripherials on different boards,
> based on the same or "near the same" chips/SoCs,
> but serial output have to be the
> same among the same family of chips, like in a PC
> world: COM1:3f8, COM2:2f8?

Not really. I think DTB embedded in uImage or separate DTB is not an issue. I've seen cases when kernel does not output anything simply because it can not start at all. If wrong DTB was used then it would output the error, in other cases it would just hang there. Perhaps earlyprintk in bootargs would help to see more.

> 5. May it can be a problem with 'Uboot is unabling
> to boot kernel over 3.2, due to L2
> cache enable/disable issue'?

My kernel build config does take care of L2 cache issue (hence, it can boot with old stok u-boot).

I'd suggest that you could look inside my 3.16 patch, find a plug in there that is close to the ix2-dl, and use that arcNumber. See if you can boot further.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I looked in stock-kernel code (3.18.2) there is DTB-support for ix2_200 but no mach support.

There are a few new kernel-options to trade with old u-boot and DTB-files (try make ARCH=arm menuconfig , there you will see options for building zImage with appended DTB:

-there is an option to check the appended dtb
-there are options to include kernel-command line in the kernel and a few others

but you will have to patch the ix2_200.dtb file to match your device.

After creation of zImage you have to cat zImage + dtb and create an uImage(mkimage).

Hi!

There are no a real results for now, but I have a one step further.

>Perhaps earlyprintk in bootargs would help to see more.

I rewrote uboot params substitution to include misssing and easily manage boot options. So the kernel cmdline is:

#console=ttyS0,115200n8 earlyprintk root=/dev/ram0 panic=3 :::DB88FXX81:eth0:none mtdparts=*here is a long string*

As I saw in documentation, "earlyprintk" may be extended with mode detailed uart specifications.
I've tried both short and long form of it. Also I change number of ttyS* in "console" and uart* in "earlyprintk".
No results though. :/

@bodhi
I check you 3.16 patch and try all machine number constants I've found here. All of them, except one (3960)
did not make any differences in an abscense of serial output in bootprocess.

@balou1974
Of course, I know of ability ti specify "compile-in" custom commandline into kernel to be used by
default, but digging into kernel crosscompiling takes more spare time, that I have lack of.
So I'll try to compile my own kernel later.

I've also tried to >> dts/dtb descriptions to a zImage, as written in docs to bodhi's rootfs, but with no luck either.

There are two strange things I have to discover:
1. Stock 2.6.31 kernels works when mainlineLinux=no, *BUT ALSO* hangs on serial output, when mainlineLinux=yes!!
I dont have any suggestions, why this happens.

2. Booting bodhi's 3.16 non FDT kernel with arcNumber=3960 *SHOWS* some garbage at kernel output.
It is repeatable process, and it happens at every boot. The output is like a wrong baudrate. The main suggestion,
that uart init in this device are differs from all others and also from ix2_200. I dont read docs of Marvell Armada yet,
but have a feeling, that SoC can clock uart in different ways (dividers/multipliers). This principle widely used
in embedded world. And the developers of ix2-dl, for their own reasons, choose a different init of uart. So,
the stock uboot knew of it and init uart properly, but generic/vanilla kernel do not know of such tricks and
cant init uart at all, or init it at a wrong rate.

So the next things I will do:
- check details of uart init in 3960 mach
- try to get and relist sources of vendor-patched 2.6.31 to discover details
- try to get & read Marvell datasheets

Many things to do! ;)

megov,

> #console=ttyS0,115200n8 earlyprintk root=/dev/ram0
> panic=3 :::DB88FXX81:eth0:none mtdparts=*here is a
> long string*
>
> As I saw in documentation, "earlyprintk" may be
> extended with mode detailed uart specifications.
> I've tried both short and long form of it. Also I
> change number of ttyS* in "console" and uart* in
> "earlyprintk".
> No results though. :/

I think earlyprintk=serial would be correct.

> I've also tried to >> dts/dtb descriptions to a
> zImage, as written in docs to bodhi's rootfs, but
> with no luck either.
>

I don't think the above would work. That's why I've suggested a non-FDT kernel (i.e. 3.16).

> There are two strange things I have to discover:
> 1. Stock 2.6.31 kernels works when
> mainlineLinux=no, *BUT ALSO* hangs on serial
> output, when mainlineLinux=yes!!
> I dont have any suggestions, why this happens

This is expected stock kernel behavior.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 01/12/2015 01:43PM by bodhi.

@megov,

Have you tried to boot with arcNumber set to the NSA325, and earlyprintk=serial.

- rootfs Debian-3.16.0-kirkwood-tld-2?
- Or with kernel 3.16.0-kirkwood-tld-3 installed on top of tld-2?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

The stock 2.6.31.8 kernel source of iomega ix2-ng, Ez-media backup manufacturer provided is here:
Download
HM3 : iomega Ez Media Backup
IX2-ng : IX2-dl

They modified mach-type list.

../arch/arm/tools/mach-types

feroceon_orion		ARCH_FEROCEON_ORION	FEROCEON_ORION		526
feroceon_kw		ARCH_FEROCEON_KW	FEROCEON_KW		527
feroceon_mv78xx0	ARCH_FEROCEON_MV78XX0	FEROCEON_MV78XX0	528

.config

#
# System Type
#
# CONFIG_ARCH_AAEC2000 is not set
# CONFIG_ARCH_INTEGRATOR is not set
# CONFIG_ARCH_FEROCEON_ORION is not set
CONFIG_ARCH_FEROCEON_KW=y
# CONFIG_ARCH_FEROCEON_MV78XX0 is not set
# CONFIG_ARCH_REALVIEW is not set
# CONFIG_ARCH_VERSATILE is not set
# CONFIG_ARCH_AT91 is not set
# CONFIG_ARCH_CLPS711X is not set

Edited 1 time(s). Last edit at 03/02/2015 08:48PM by Berryshake.

megov Wrote:
-------------------------------------------------------

> So, do you have any suggestions how to fix this
> problem?

If not difficult, check this variant.
uImage and uInitrd https://yadi.sk/d/hnRWJlrYekMDu

mainlineLinux=yes
arcNumber=1680
bootcmd_usb=usb start; ext2load usb 0:1 0x800000 /uImage; ext2load usb 0:1 0x1100000 /uInitrd; bootm 0x800000 0x1100000;

or

mainlineLinux=no
bootcmd_usb=usb start; ext2load usb 0:1 0x800000 /uImage; ext2load usb 0:1 0x1100000 /uInitrd; bootm 0x800000 0x1100000;

and

bootcmd_usb=usb start; ext2load usb 0:1 0x40000 /uImage; ext2load usb 0:1 0x900000 /uInitrd; bootm 0x40000 0x900000;



If possible, show full log NAS for stok and the proposed options. Your env settings and time of release of u-boot is very similar to mine.



Edited 1 time(s). Last edit at 03/06/2015 10:13AM by balbes150.

is it possible to use dts file for ea4500? both has same cpu(iomega ez, iomega ix2-dl, ea4500).

http://www.spinics.net/lists/arm-kernel/msg457988.html

http://wiki.openwrt.org/toh/linksys/ea4500

iomega ez Wrote:
-------------------------------------------------------
> is it possible to use dts file for ea4500? both
> has same cpu(iomega ez, iomega ix2-dl, ea4500).
>
> http://www.spinics.net/lists/arm-kernel/msg457988.
> html
>
> http://wiki.openwrt.org/toh/linksys/ea4500

Trying out tthe DTS for a likely compatible box is quite easy if you have serial console. Compile this DTB and put it on the rootfs: Debian-3.18.5-kirkwood-tld-1-rootfs-bodhi.tar.bz2
http://forum.doozan.com/read.php?2,12096.

Boot with serial console and adjust the stock u-boot envs to boot it. If it does not work, no harm done.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

looking for someone who knows better than me.

https://dev.openwrt.org/browser/trunk/target/linux/kirkwood/patches-3.14/160-ea4500.patch?rev=40871

here is link for files

http://downloads.openwrt.io/chaos_calmer/15.05/kirkwood/generic/

openwrt-15.05-kirkwood-ea4500-rootfs.tar.gz 14-Sep-2015 08:09 2662141
openwrt-15.05-kirkwood-ea4500-rootfs.ubi 14-Sep-2015 08:09 5505024
openwrt-15.05-kirkwood-ea4500-rootfs.ubifs 14-Sep-2015 08:09 5160960
openwrt-15.05-kirkwood-ea4500-uImage 14-Sep-2015 08:09 1591838
openwrt-15.05-kirkwood-ea4500-zImage 14-Sep-2015 08:09 1591774
openwrt-15.05-kirkwood-ea4500.dtb

Looks promising, I think it's going to work. But you will have to check if they have the same specs (memory, HDD, USB, network chip...) Memory type & capacity are most important, and then network chip, and then the rest.

Do you have serial console?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I don't have serial cable but berryshake did some work on this.

http://forum.doozan.com/read.php?2,15887

iomega ez,

You'll need serial console to make sure you won't get locked out, because u-boot envs must be changed to boot USB or HDD. Once you have that, I could help you to try booting the new Debian rootfs.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

currently, iomega ez is booted with usb as follow

usb_scan=setenv usb_device 0:1; setenv usb_root sdb1
make_usb_bootargs=setenv bootargs $(console) $(bootargs_mtd) $(bootargs_end) root=/dev/$(usb_root) rootdelay=10 rootfstype=ext2
bootcmd_usb=usb reset; run usb_scan; run make_usb_bootargs; ext2load usb $(usb_device) 0x40000 /boot/zImage; bootm 0x40000
bootcmd1=ide read 0x40000 0x800 0x1800; ide read 0x900000 0x2000 0x1800; setenv bootargs $(console) $(bootargs_root) $(bootargs_mtd) $(bootargs_end); bootm 0x40000 0x900000;run bootcmd2
bootcmd=ide reset; run bootcmd_usb; usb stop; run bootcmd1

if there is no usb, it will boot with hdd.

I use squeeze with stock kernel so I want to upgrade to latest kernel and debian.

--------
moderator edit: code tag.



Edited 1 time(s). Last edit at 11/08/2015 04:33PM by bodhi.

iomega ez,

You will need to change this to boot the Debian-3.18.5-kirkwood-tld-1-rootfs-bodhi.tar.bz2 rootfs to load uImage and uInitrd and boot with that option:

bootcmd_usb=usb reset; run usb_scan; run make_usb_bootargs; ext2load usb $(usb_device) 0x40000 /boot/zImage; bootm 0x40000

So without serial console, a mistake will lock you out. If you do have serial console, then there is no risk playing with it.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

sorry, I don't follow you.

this device booted from usb(/boot/zImage).

so all I have to get new kernel is to build a zImage from pc?

if I am wrong, please correct me.


thanks

my uboot is changed by script below

#!/bin/sh


fw_setenv usb_scan 'setenv usb_device 0:1; setenv usb_root sdb1'


fw_setenv make_usb_bootargs 'setenv bootargs $(console) $(bootargs_mtd) $(bootargs_end) root=/dev/$(usb_root) rootdelay=10 rootfstype=ext2'


fw_setenv bootcmd_usb 'usb reset; run usb_scan; run make_usb_bootargs; ext2load usb $(usb_device) 0x40000 /boot/zImage; bootm 0x40000'



fw_setenv bootcmd1 'ide read 0x40000 0x800 0x1800; ide read 0x900000 0x2000 0x1800; setenv bootargs $(console) $(bootargs_root) $(bootargs_mtd) $(bootargs_end); bootm 0x40000 0x900000;run bootcmd2'


fw_setenv bootcmd 'ide reset; run bootcmd_usb; usb stop; run bootcmd1'

iomega ez Wrote:
-------------------------------------------------------
> sorry, I don't follow you.
>
> this device booted from usb(/boot/zImage).
>
> so all I have to get new kernel is to build a
> zImage from pc?
>
> if I am wrong, please correct me.
>
>
> thanks

No, it does not have to boot from zImage.

To boot this rootfs (see section: Updated 24 Feb 2015):

- Append the file openwrt-15.05-kirkwood-ea4500.dtb to the zImage and then generate uImage as described in Step 4.
- Modify the env

fw_setenv bootcmd_usb 'usb reset; run usb_scan; run make_usb_bootargs; ext2load usb $(usb_device) 0x40000 /boot/zImage; bootm 0x40000'

to boot with uImage and uInitrd.

PS.

Another thought: what system are you booting with the USB?

uname -a

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 11/09/2015 02:49AM by bodhi.

I checked usb /boot, there is only zImage there.

uname -a gives below

Linux IOMEGA 2.6.31.8 Thu Aug 22 17:40:31 KST 2013 v0.0.9 Thu Aug 22 17:40:31 KST 2013 armv5tel GNU/Linux

iomega ez Wrote:
-------------------------------------------------------
> I checked usb /boot, there is only zImage there.
>
> uname -a gives below
>
> Linux IOMEGA 2.6.31.8 Thu Aug 22 17:40:31 KST 2013
> v0.0.9 Thu Aug 22 17:40:31 KST 2013 armv5tel
> GNU/Linux

Try to boot Debian rootfs as I've suggested right above.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

balbes150 Wrote:
-------------------------------------------------------
> megov Wrote:
> --------------------------------------------------
> -----
>
> > So, do you have any suggestions how to fix this
> > problem?
>
> If not difficult, check this variant.
> uImage and uInitrd
> https://yadi.sk/d/hnRWJlrYekMDu
>
> mainlineLinux=yes
> arcNumber=1680
> bootcmd_usb=usb start; ext2load usb 0:1 0x800000
> /uImage; ext2load usb 0:1 0x1100000 /uInitrd;
> bootm 0x800000 0x1100000;
>
> or
>
> mainlineLinux=no
> bootcmd_usb=usb start; ext2load usb 0:1 0x800000
> /uImage; ext2load usb 0:1 0x1100000 /uInitrd;
> bootm 0x800000 0x1100000;
>
> and
>
> bootcmd_usb=usb start; ext2load usb 0:1 0x40000
> /uImage; ext2load usb 0:1 0x900000 /uInitrd; bootm
> 0x40000 0x900000;
>
>
>
> If possible, show full log NAS for stok and the
> proposed options. Your env settings and time of
> release of u-boot is very similar to mine.

Hi,

I have tried a variant from balbes150, there still no serial output, but network access working after boot and I could continue install through ssh.

I didn't found password to login, but could change preseed.cfg to set my own. Now waiting to finish installation and shall try to flash the nand.

-talexone

Solved.

The serial port on IX2-ng and IX2-dl is on second UART. Most kirkwood NAS boxes use first UART. That is why nobody is seeing serial port output. It requires a custom device-tree (DTS) file to support the serial port, LEDs, buttons, power-off, etc.

I have attached a DTS file and some docs for the IX2-ng. I used it to install and boot Debian (Jessie) using the Debian installer for Kirkwood devices. Works quite well and should also work with bodhi's Kirkwood kernels as found on this site.

Maybe this will work with the IX2-dl. Maybe. I think the IX2-dl will require its own DTS file because it uses different flash. I'm willing to work with owner of IX2-dl to create a DTS file for it.

sarcastic.mannequin,

Thanks for your contribution! I will look at the DTS and boot log, and will add your DTS files to the kernel in the next release.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)