3.3.2 kernel, how to get kernel messages via nc?

Running davygravy's 3.3.2 kernel. Finally after weeks of struggle I got my GFH fully setup on SATA. Everything was working so I booted up, changed a whole mess of things like: installed a bunch of packages, including transmission-daemon and samaba, setup led lights (modprobe'd some modules and did some manual testing, then added those modules to /etc/modules and changed my halt file to set orange led on at shutdown)......configured all that and downloaded my first video via torrent and accessed it from my windows box via samba!! Super happy, but super tired. Issue a 'halt' command as root, didn't see the orange led so i went to sleep....woke up a few hours later, still no orange led so i just cut the power.

Long story short, whatever I did from the time I last booted successfully until the time I killed the power I borked something bad. Uboot does it's usual thing over nc, and loads the kernel image and initramfs, then i get nothing!!!
So I tried to troubleshoot but the kernel isn't getting far enough to write anything to /var/log/dmesg so I can't figure out what is crashing the kernel.

From what I gather it might be possible to get the kernel to output it's messages via nc just like uboot does, but for the life of me I can't figure it out! Is this possible?

Current situation: sata file system kernel hangs before dmesg log gets written + can boot from rescue + can boot from usb + can use rescue or usb to mount sata file system and modify whatever I want.

also any tips on how to troublesheet the sata filesystem would be greatly appreciated.

and yes, I know the best solution is to install a physical serial connection, then nc doesn't matter.....almost there but not yet...probably 2 more weeks ;) haha

If you're using Ext2 for the SATA drive, then I would plug the SATA drive into a USB interface, connect it to another Linux box and check the file system with e2fsck to see if it was corrupted.

sorry, didn't mention that fsck was the first thing I did. I've done it multiple times at this point. Because I can boot into rescue and from usb I have access to basically anything that a fully functioning goflex can do. Already checked the filesystem itself, and tried to revert some of my changes from yesterday. Also tried to enable kernel netconsole, by following more than one guide and none of the methods seem to have worked.....which makes me think the kernel is faulting very early on, before it even gets to the point of enabling networking and netconsole.

I also tried copying the uInitrd (sp?) and uImage from my working usb stick over to the sata drive and same behavior.

To make matters worse I found this post : http://forum.doozan.com/read.php?3,14,8839#msg-8839
specifically

Quote

The GFH worked very well until a shutdown -h now. I can't get access via SSH when the disk is on its support. Rescue works. I've tried to reinstall the system, same behavior: it boots but after some halt / reboot, 2 or 3 times ;-) no more ssh access, no ip.

Am I doomed with SATA? should I accept my fate with usb flash rootfs?

Which Debian version are you using (uname -a)? what is your arcNumber ? also, is your UBoot the latest (Davy's) or Jeff's?

3089, pretty sure it's davy's uboot, uname -a (can't run it since I can't boot SATA) should be 3.3.2-kirkwood-wide or something.........basically the SATA rootfs was created by using davygravy's nsa320 filesystem ( http://forum.doozan.com/read.php?2,7806 ), then installing the 3.3.2 kernel .deb from davygravy (http://forum.doozan.com/read.php?2,7676).



Edited 1 time(s). Last edit at 08/14/2012 07:02AM by floodo1.

floodo,

One thing I've noticed that both you and Sagitarius have arcNumber 3089, which is the GoFlex Net, not GoFlex Home (they are supposed to work for both boxes but there are differences ).

You could try to set it to the correct one, which is 3338. I'm running the GoFlex Homes with Jeff's UBoot, arcNumber 3338, and a self compiled kernel (3.2.18), with USB flash as boot drive. I've run SATA as boot drive for GFHome in the past briefly, but not anymore. And it's been a while so I don't have accurate info to report.

But before changing arcNumber, I would make sure to have netconsole working first.

Edit: I saw that you have rescue system working. So you could experiment without worrying!

-bodhi



Edited 1 time(s). Last edit at 08/14/2012 02:27PM by bodhi.

@floodo,

I looked at the kirwood initialization code briefly. Running the GF Home with GF Net initialization code could cause problem during SATA device registration (i.e. the GoFlex Net has 2 ports, the GF Home has only 1 port). Since 2nd device's data structure could contain garbage at some point, it's possible that the kernel choke while trying to register the 2nd SATA device. Just a theory! I don't know how graceful that registration code behaves in error recovery.

-bodhi

changing arcNumber made no difference.
Pretty sure I'm going to start fresh with the OS partition on my sata drive. Since rescue boots consistently and so do all my usb sticks I think I'm going to just salvage my transmission and samba config files and wipe the sata drive and put a new filesystem on there.
Hopefully that will boot correctly, then I'll make my first priority getting netconsole with the kernel to work. Once I have netconsole I'll go ahead and set everything up again, so that if I break something this time I'll at least ahve kernel netconsole to troubleshoot.

At this point this makes the most sense...I was just hoping to avoid that and salvage what is there right now....oh well :)

Could you post your UBoot envs? I'm going to dig out the log when I still boot the GFHome with SATA. Hope I can find it :-)

I digged out an old spared SATA HDD, and put my current Debian rootfs on it. This is the info from my GoFlex Home booting with SATA as the boot drive (and the only drive). I shutdown and cold started several times (more than 5 times just to be sure). Hope this helps.

netconsole output:

U-Boot 2010.09 (Oct 23 2010 - 11:53:10)
Marvell-GoflexNet by Jeff Doozan, Peter Carmichael
Hit any key to stop autoboot:  0 
(Re)start USB...
USB:   Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices... 1 USB Device(s) found
       scanning bus for storage devices... 0 Storage Device(s) found
** Block device usb 0 not supported

** Invalid boot device **
Creating 1 MTD partitions on "nand0":
0x000002500000-0x000010000000 : "mtd=3"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI error: validate_ec_hdr: bad VID header offset 2048, expected 512
UBI error: validate_ec_hdr: bad EC header
UBI error: ubi_io_read_ec_hdr: validation failed for PEB 0
UBI error: ubi_init: cannot attach mtd1
UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
UBI init error -22
** Block device usb 0 not supported
** Block device usb 1 not supported
** Block device usb 2 not supported
** Block device usb 3 not supported
** Block device usb 0 not supported
** Block device usb 0 not supported
Wrong Image Format for bootm command
ERROR: can't get kernel image!
stopping USB..

Reset IDE: Bus 0: ....OK Bus 1: OK 
  Device 0: Model: Hitachi HDT721016SLA380  Firm: ST1OA31B Ser#: xxxxxxxxxxxxx
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 152627.8 MB = 149.0 GB (312581808 x 512)
  Device 1: Model: Hitachi HDT721016SLA380  Firm: ST1OA31B Ser#: xxxxxxxxxxxxx
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 152627.8 MB = 149.0 GB (312581808 x 512)
Loading file "/boot/uImage" from ide device 0:1 (hda1)
1638256 bytes read
Loading file "/boot/uInitrd" from ide device 0:1 (hda1)
6664745 bytes read
## Booting kernel from Legacy Image at 00800000 ...
   Image Name:   Linux-3.2.26-kirkwood
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    1638192 Bytes = 1.6 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   initramfs-3.2.26-kirkwood
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    6664681 Bytes = 6.4 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

dmesg:

root@GFHome3:~# dmesg
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.26-kirkwood (root@Debian) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1 PREEMPT Sun Aug 12 06:32:58 PDT 2012
[    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053977
[    0.000000] CPU: VIVT data cache, VIVT instruction cache
[    0.000000] Machine: Seagate GoFlex Home
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] On node 0 totalpages: 32768
[    0.000000] free_area_init_node: node 0, pgdat c04671d4, node_mem_map c04d3000
[    0.000000]   Normal zone: 256 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32512 pages, LIFO batch:7
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line: console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 rootfstype=ext2 mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] allocated 1835008 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Memory: 128MB = 128MB total
[    0.000000] Memory: 116668k/116668k available, 14404k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0xc8800000 - 0xfe800000   ( 864 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xc8000000   ( 128 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc03f7338   (4029 kB)
[    0.000000]       .init : 0xc03f8000 - 0xc041b000   ( 140 kB)
[    0.000000]       .data : 0xc041c000 - 0xc0469660   ( 310 kB)
[    0.000000]        .bss : 0xc0469684 - 0xc04d2438   ( 420 kB)
[    0.000000] NR_IRQS:114
[    0.000000] sched_clock: 32 bits at 200MHz, resolution 5ns, wraps every 21474ms
[    0.000000] Console: colour dummy device 80x30
[   15.000262] Calibrating delay loop... 1191.11 BogoMIPS (lpj=5955584)
[   15.090038] pid_max: default: 32768 minimum: 301
[   15.090221] Security Framework initialized
[   15.090313] Mount-cache hash table entries: 512
[   15.090739] Initializing cgroup subsys cpuacct
[   15.090755] Initializing cgroup subsys memory
[   15.090789] Initializing cgroup subsys devices
[   15.090797] Initializing cgroup subsys freezer
[   15.090805] Initializing cgroup subsys net_cls
[   15.090812] Initializing cgroup subsys blkio
[   15.090913] CPU: Testing write buffer coherency: ok
[   15.092204] devtmpfs: initialized
[   15.094255] print_constraints: dummy: 
[   15.094523] NET: Registered protocol family 16
[   15.095198] Kirkwood: MV88F6281-A1, TCLK=200000000.
[   15.095210] Feroceon L2: Cache support initialised.
[   15.095650] initial MPP regs: 01111111 11113322 00001111 00100000 00000000 00000000 00000000
[   15.095670]   final MPP regs: 01111111 11113322 00001111 00000000 00000000 00000000 00000000
[   15.098380] bio: create slab <bio-0> at 0
[   15.098878] vgaarb: loaded
[   15.099466] Switching to clocksource orion_clocksource
[   15.118275] NET: Registered protocol family 2
[   15.118572] IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
[   15.119646] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[   15.119770] TCP bind hash table entries: 4096 (order: 4, 98304 bytes)
[   15.119949] TCP: Hash tables configured (established 4096 bind 4096)
[   15.119958] TCP reno registered
[   15.119968] UDP hash table entries: 64 (order: 0, 4096 bytes)
[   15.120012] UDP-Lite hash table entries: 64 (order: 0, 4096 bytes)
[   15.120335] NET: Registered protocol family 1
[   15.120374] PCI: CLS 0 bytes, default 32
[   15.120612] Unpacking initramfs...
[   15.682615] Freeing initrd memory: 6508K
[   15.682627] NetWinder Floating Point Emulator V0.97 (double precision)
[   15.683236] audit: initializing netlink socket (disabled)
[   15.683280] type=2000 audit(0.670:1): initialized
[   15.684713] VFS: Disk quotas dquot_6.5.2
[   15.684854] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[   15.685019] JFFS2 version 2.2. (NAND) (SUMMARY)  \xffffffc2\xffffffa9 2001-2006 Red Hat, Inc.
[   15.685523] msgmni has been set to 240
[   15.687432] alg: No test for stdrng (krng)
[   15.687577] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[   15.687589] io scheduler noop registered
[   15.687596] io scheduler deadline registered
[   15.687680] io scheduler cfq registered (default)
[   15.687779] mv_xor_shared mv_xor_shared.0: Marvell shared XOR driver
[   15.687812] mv_xor_shared mv_xor_shared.1: Marvell shared XOR driver
[   15.719566] mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
[   15.759568] mv_xor mv_xor.1: Marvell XOR: ( xor fill cpy )
[   15.799567] mv_xor mv_xor.2: Marvell XOR: ( xor cpy )
[   15.839566] mv_xor mv_xor.3: Marvell XOR: ( xor fill cpy )
[   15.839938] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[   15.860555] serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33) is a 16550A
[   16.291781] console [ttyS0] enabled
[   16.292771] ONFI flash detected
[   16.292867] ONFI param page 0 valid
[   16.292880] NAND device: Manufacturer ID: 0x2c, Chip ID: 0xda (Micron MT29F2G08AAD)
[   16.292896] Scanning device for bad blocks
[   16.448516] 4 cmdlinepart partitions found on MTD device orion_nand
[   16.448526] Creating 4 MTD partitions on "orion_nand":
[   16.448539] 0x000000000000-0x000000100000 : "u-boot"
[   16.449438] 0x000000100000-0x000000500000 : "uImage"
[   16.450320] 0x000000500000-0x000002500000 : "rootfs"
[   16.451203] 0x000002500000-0x000010000000 : "data"
[   16.452959] mousedev: PS/2 mouse device common for all mice
[   16.453365] rtc-mv rtc-mv: rtc core: registered rtc-mv as rtc0
[   16.453580] i2c /dev entries driver
[   16.453787] cpuidle: using governor ladder
[   16.453959] cpuidle: using governor menu
[   16.454096] Registered led device: status:green:health
[   16.454175] Registered led device: status:orange:fault
[   16.454253] Registered led device: status:white:misc
[   16.455555] TCP cubic registered
[   16.455566] NET: Registered protocol family 17
[   16.455589] Registering the dns_resolver key type
[   16.455638] Gating clock of unused units
[   16.455645] before: 0x00dfc3fd
[   16.455651]  after: 0x00c741c9
[   16.455701] registered taskstats version 1
[   16.456220] rtc-mv rtc-mv: setting system clock to 2012-08-18 08:23:05 UTC (1345278185)
[   16.456281] Initializing network drop monitor service
[   16.562750] Freeing init memory: 140K
[   16.650278] udevd[55]: starting version 175
[   16.738996] mv643xx_eth: MV-643xx 10/100/1000 ethernet driver version 1.4
[   16.739083] mv643xx_eth smi: probed
[   16.788326] mv643xx_eth_port mv643xx_eth_port.0: eth0: port 0 with MAC address xx:xx:xx:xx:xx:xx
[   16.893277] usbcore: registered new interface driver usbfs
[   16.893389] usbcore: registered new interface driver hub
[   16.894138] SCSI subsystem initialized
[   16.899744] usbcore: registered new device driver usb
[   16.901948] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   16.902020] orion-ehci orion-ehci.0: Marvell Orion EHCI
[   16.902118] orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
[   16.932840] orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
[   16.949592] orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
[   16.960626] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   16.960639] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   16.960648] usb usb1: Product: Marvell Orion EHCI
[   16.960656] usb usb1: Manufacturer: Linux 3.2.26-kirkwood ehci_hcd
[   16.960665] usb usb1: SerialNumber: orion-ehci.0
[   16.961220] hub 1-0:1.0: USB hub found
[   16.961247] hub 1-0:1.0: 1 port detected
[   16.967020] libata version 3.00 loaded.
[   16.969122] sata_mv sata_mv.0: version 1.28
[   16.969227] sata_mv sata_mv.0: slots 32 ports 1
[   16.985775] scsi0 : sata_mv
[   16.986107] ata1: SATA max UDMA/133 irq 21
[   17.489534] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[   17.529563] ata1.00: ATA-8: Hitachi HDT721016SLA380, ST1OA31B, max UDMA/133
[   17.529575] ata1.00: 312581808 sectors, multi 0: LBA48 NCQ (depth 31/32)
[   17.589562] ata1.00: configured for UDMA/133
[   17.589960] scsi 0:0:0:0: Direct-Access     ATA      Hitachi HDT72101 ST1O PQ: 0 ANSI: 5
[   17.609869] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB)
[   17.610169] sd 0:0:0:0: [sda] Write Protect is off
[   17.610180] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   17.610304] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   17.625829]  sda: sda1
[   17.627080] sd 0:0:0:0: [sda] Attached SCSI disk
[   29.657004] udevd[186]: starting version 175
[   34.067334] Adding 524284k swap on /swapfile1.  Priority:-1 extents:139 across:534988k 
[   35.516874] NET: Registered protocol family 10
[   35.523432] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   35.995678] fuse init (API version 7.17)
[   37.973596] mv643xx_eth_port mv643xx_eth_port.0: eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[   37.973711] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

UBoot envs:

root@GFHome3:~# fw_printenv
ethact=egiga0
bootdelay=3
baudrate=115200
mainlineLinux=yes
console=ttyS0,115200
led_init=green blinking
led_exit=green off
led_error=orange blinking
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
mtdids=nand0=orion_nand
partition=nand0,2
stdin=serial
stdout=serial
stderr=serial
rescue_installed=0
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_list=1 2 3 4
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=3:1 dev=sdd1
usb_device=0:1
usb_root=/dev/sda1
usb_rootfstype=ext2
usb_rootdelay=10
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
serverip=192.168.0.220
ipaddr=192.168.0.226
if_netconsole=ping $serverip
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
preboot=run if_netconsole start_netconsole
arcNumber=3338
usb_init=run usb_scan ; setenv usb_root LABEL=rootfs
sata_boot=ide reset; mw 0x800000 0 1; ext2load ide 0:1 0x800000 /boot/uImage; if ext2load ide 0:1 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
sata_bootcmd=run usb_set_bootargs; run sata_boot
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run sata_bootcmd; run rescue_bootcmd; run pogo_bootcmd; reset

wow Bodhi, digging this one up from the past I am, but it turns out that my problem was that my fstab had /deb/sda3 instead of /dev/sda3 in it, and instead of erroring out it was just hanging. Eventually I used the rescue system to figure this out. Fixed fstab and it booted fine :) (using arc 3338)

Glad the problem was that trivial:) fstab is more suitable for fixed installation. On these plugs, I avoid that and rely on udev to automount all secondary partitions.