Errors in Boot Log for Dell M300
Posted by mikeh49
|
Errors in Boot Log for Dell M300 February 16, 2020 05:43PM |
Registered: 10 years ago Posts: 155 |
A fresh install of the 5.2.9 file system and 5.5.1 kernel. After 2 or 3 clean boots, ran into this on the serial connection:
Preceding this, there was a segmentation fault while trying to edit the .profile file using nano. The system kept running and seemed OK, but nano always produced a segmentation fault. Did a reboot from the command line is serial and got the above bootlog.
Edited 1 time(s). Last edit at 03/03/2020 06:40PM by mikeh49.
Starting version 241 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. Begin: Will now check root file system ... fsck from util-linux 2.33.1 [/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a -C0 /dev/sda1 sata_rootfs: clean, 40782/262144 files, 303302/1048576 blocks done. [ 18.929530][ T200] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [ 18.947128][ T200] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. INIT: version 2.93 booting [info] Using makefile-style concurrent boot in runlevel S. [ ok ] Starting hotplug events dispatcher: systemd-udevd. [ ok ] Synthesizing the initial hotplug events (subsystems)...done. [....] Synthesizing the initial hotplug events (devices)... [ 20.342031][ T403] 8<--- cut here --- [ 20.345806][ T403] Unable to handle kernel paging request at virtual address 1017df80 [ 20.368746][ T403] pgd = (ptrval) [ 20.372165][ T403] [1017df80] *pgd=00000000 [ 20.376451][ T403] Internal error: Oops: 5 [#1] PREEMPT ARM [ 20.382132][ T403] Modules linked in: [ 20.385894][ T403] CPU: 0 PID: 403 Comm: udevadm Not tainted 5.5.1-kirkwood-tld-1 #1.0 [ 20.393929][ T403] Hardware name: Marvell Kirkwood (Flattened Device Tree) [ 20.400923][ T403] PC is at kernfs_seq_next+0x8/0x64 [ 20.405990][ T403] LR is at 0x1017df80 [ 20.409835][ T403] pc : [<802e6004>] lr : [<1017df80>] psr: a0000013 [ 20.416820][ T403] sp : 8200de28 ip : 8200ddf4 fp : 00000000 [ 20.422757][ T403] r10: 00000000 r9 : 8200c000 r8 : ee0ed4a8 [ 20.428697][ T403] r7 : 802e5ffc r6 : ee59bdc0 r5 : 00000000 r4 : ee0ed4a0 [ 20.435943][ T403] r3 : 80a10124 r2 : 66abbb58 r1 : ee0ed4a0 r0 : ee59bdc0 [ 20.443190][ T403] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 20.451046][ T403] Control: 0005397f Table: 6ed9c000 DAC: 00000051 [ 20.457507][ T403] Process udevadm (pid: 403, stack limit = 0x(ptrval)) [ 20.464231][ T403] Stack: (0x8200de28 to 0x8200e000) [ 20.469300][ T403] de20: ee0ed4a0 00000000 ee59bdc0 8025f024 8200ded0 ee0ed4a0 [ 20.478212][ T403] de40: 00000000 00000001 ffffff9c 80271048 ee59bb00 81004228 8200df78 00020000 [ 20.487124][ T403] de60: 826ec840 ee568000 00000041 000041ed 00000000 00000004 00000000 0000001d [ 20.496037][ T403] de80: 00000000 8200c000 81004228 ee59bb00 ee59bdc0 eef577b0 ee56af68 8027af74 [ 20.504948][ T403] dea0: 5e49d257 66abbb58 1e65fb80 00000005 81004228 8200df78 00000001 ffffff9c [ 20.513860][ T403] dec0: 8200c000 00000000 7eccb8c4 8027123c eef577b0 ee56af68 e293b68b 00000006 [ 20.522771][ T403] dee0: 827b6031 8102b6bc 826ec850 ee565f68 ee59bdc0 00000109 ee59bb00 00000068 [ 20.531683][ T403] df00: 00000000 00000000 00000000 8200df10 00000ff0 7eccb840 00000ff0 827b6010 [ 20.540596][ T403] df20: ffffe000 8063a654 827b6010 00000ff0 00000ff0 66abbb58 00000005 826eca94 [ 20.549507][ T403] df40: 827b6000 00000000 00000005 00000002 ffffff9c 8027f060 00000000 66abbb58 [ 20.558419][ T403] df60: 81004228 00000005 000a0000 81004228 827b6000 8025fdb4 00020000 66ab0000 [ 20.567331][ T403] df80: 00000004 00000100 00000001 66abbb58 00a6e150 76f72968 00000008 00000142 [ 20.576243][ T403] dfa0: 80101208 80101000 00a6e150 76f72968 ffffff9c 7eccb840 000a0000 00000000 [ 20.585154][ T403] dfc0: 00a6e150 76f72968 00000008 00000142 7eccb840 00a85b70 00a769e0 7eccb8c4 [ 20.594066][ T403] dfe0: 76f3e000 7eccb6b0 0d08a900 76eb6250 60000010 ffffff9c 00000000 00000000 [ 20.602988][ T403] [<802e6004>] (kernfs_seq_next) from [<8025f024>] (do_dentry_open+0x2d4/0x330) [ 20.611901][ T403] [<8025f024>] (do_dentry_open) from [<80271048>] (path_openat+0xa20/0xbd8) [ 20.620462][ T403] [<80271048>] (path_openat) from [<8027123c>] (do_filp_open+0x3c/0x9c) [ 20.628677][ T403] [<8027123c>] (do_filp_open) from [<8025fdb4>] (do_sys_open+0x70/0xf4) [ 20.636891][ T403] [<8025fdb4>] (do_sys_open) from [<80101000>] (ret_fast_syscall+0x0/0x50) [ 20.645363][ T403] Exception stack(0x8200dfa8 to 0x8200dff0) [ 20.651130][ T403] dfa0: 00a6e150 76f72968 ffffff9c 7eccb840 000a0000 00000000 [ 20.660042][ T403] dfc0: 00a6e150 76f72968 00000008 00000142 7eccb840 00a85b70 00a769e0 7eccb8c4 [ 20.668952][ T403] dfe0: 76f3e000 7eccb6b0 0d08a900 76eb6250 [ 20.674719][ T403] Code: 12600020 e12fff1e e92d4070 e590e048 (e59ee000) [ 20.744457][ T403] ---[ end trace bf9130f3642c377c ]--- Segmentation fault failed. [ ok ] Waiting for /dev to be fully populated...done. [ ok ] Setting hostname to 'm300-2'...done. [ ok ] Activating swap:. [ 21.468247][ T570] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro [info] Will now check all file systems. fsck from util-linux 2.33.1 Checking all file systems. LABEL=sata_rootfs is mounted LABEL=data is not mounted [/sbin/fsck.ext4 (1) -- /mnt/data] fsck.ext4 -y -C0 /dev/sda2 e2fsck 1.44.5 (15-Dec-2018) data: clean, 11/715264 files, 68287/2860800 blocks [....] Done checking file systems. A log is being saved in /var/log/fsck/checkfs if that location[ ok ritable.. [ ok ] Cleaning up temporary files...[....] Cleaning /tmp...done. [ ok . [....] Will now mount local filesystems: [ 22.719341][ T853] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) . ok [ ok ] Will now activate swapfile swap:done. [ ok ] Checking minimum space in /tmp...done. [ ok ] Cleaning up temporary files.... [ ok ] Starting Setting kernel variables: sysctl. [....] Initializing random number generator... [ 23.499024][ T1047] random: dd: uninitialized urandom read (512 bytes read) done. [....] Configuring network interfaces...Internet Systems Consortium DHCP Client 4.4.1 Copyright 2004-2018 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth0/18:03:73:41:dc:68 Sending on LPF/eth0/18:03:73:41:dc:68 Sending on Socket/fallback DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 [ 25.579450][ C0] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 100 Mb/s, full duplex, flow control disabled DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 DHCPOFFER of 192.168.1.135 from 192.168.1.1 DHCPREQUEST for 192.168.1.135 on eth0 to 255.255.255.255 port 67 DHCPACK of 192.168.1.135 from 192.168.1.1 bound to 192.168.1.135 -- renewal in 35716 seconds. [ 31.673777][ T1128] NET: Registered protocol family 10 [ 31.680095][ T1128] Segment Routing with IPv6 done. [ ok ] Starting RPC port mapper daemon: rpcbind. [ ok ] Starting NFS common utilities: statd idmapd. [ ok ] Cleaning up temporary files.... INIT: Entering runlevel: 2 [info] Using makefile-style concurrent boot in runlevel 2. [....] Starting busybox' syslogd implementation : syslogdStarting /sbin/syslogd... 1338 (syslogd) . ok [....] Starting busybox' klogd implementation : klogdStarting /sbin/klogd... 1365 (klogd) . ok [....] Starting system message bus: dbus [ 35.671651][ T1395] random: dbus-daemon: uninitialized urandom read (12 bytes read) [ 35.695240][ T1395] random: dbus-daemon: uninitialized urandom read (12 bytes read) . ok [....] Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon [ 35.822485][ T1418] random: avahi-daemon: uninitialized urandom read (4 bytes read) . ok [ ok ] Starting NFS common utilities: statd idmapd. [warn] Not starting NFS kernel daemon: no exports. ... (warning). [ ok ] Starting NTP server: ntpd. [....] Starting OpenBSD Secure Shell server: sshd [ 40.328610][ C0] random: crng init done [ 40.332736][ C0] random: 1 urandom warning(s) missed due to ratelimiting . ok [ ok ] Running local boot scripts (/etc/rc.local). Debian GNU/Linux 10 m300-2 ttyS0 m300-2 login:
Preceding this, there was a segmentation fault while trying to edit the .profile file using nano. The system kept running and seemed OK, but nano always produced a segmentation fault. Did a reboot from the command line is serial and got the above bootlog.
Edited 1 time(s). Last edit at 03/03/2020 06:40PM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 16, 2020 06:09PM |
Admin Registered: 13 years ago Posts: 18,543 |
Hi Mike,
> Preceding this, there was a segmentation fault
> while trying to edit the .profile file using nano.
Is this while in the SSH session?
> The system kept running and seemed OK, but nano
> always produced a segmentation fault. Did a
> reboot from the command line is serial and got the
> above bootlog.
Shutdown and if you could, bring the rootfs to another Linux box and run e2fsck to check for file system error.
Or if the rootfs is on HDD, you could boot with USB on this box, and run e2fsck from there. I recalled that I have not been clear about booting USB rootfs while there is a HDD rootfs inside. So that you might need to do this in serial console. Let me know if you need to do this (cannot check the rootfs on another Linux box).
If your e2fsck shows no errors on file system, I'll need to look at my M300 with serial console to verify whether it is a general problem, or just your box.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> Preceding this, there was a segmentation fault
> while trying to edit the .profile file using nano.
Is this while in the SSH session?
> The system kept running and seemed OK, but nano
> always produced a segmentation fault. Did a
> reboot from the command line is serial and got the
> above bootlog.
Shutdown and if you could, bring the rootfs to another Linux box and run e2fsck to check for file system error.
Or if the rootfs is on HDD, you could boot with USB on this box, and run e2fsck from there. I recalled that I have not been clear about booting USB rootfs while there is a HDD rootfs inside. So that you might need to do this in serial console. Let me know if you need to do this (cannot check the rootfs on another Linux box).
If your e2fsck shows no errors on file system, I'll need to look at my M300 with serial console to verify whether it is a general problem, or just your box.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 16, 2020 06:12PM |
Admin Registered: 13 years ago Posts: 18,543 |
You can test to see if your serial console causing problem. Shutdown, disconnect serial console and power up. SSH in and see if there is error in dmesg.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 16, 2020 07:33PM |
Registered: 10 years ago Posts: 155 |
|
Re: Errors in Boot Log for Dell M300 February 17, 2020 12:20PM |
Registered: 10 years ago Posts: 155 |
I think this problem started after I put the second partition of the SSD into the fstab. I created the file system on this partition using the -j parameter on mke2fs. But, for some reason I called the second partition ext4 in fstab.
Here's the SSD setup:
df thinks it is ext4, but is that because I called it ext4 in fstab? Clearly, I'm not clear if -j only means ext3.
I'll set fstab to ext3, reboot, and do your suggested tests.
Update: Booted M300 from USB with serial connected and from SSH, ran e2fsck, both partitions clean:
My environments are setup to check USB for a rootfs label, if found, boot from USB. If not found, check SATA for sata-rootfs and if found boot from there. The hostname of the USB is M300-usb to prevent confusion.
I'll leave things running and try some stuff to see if anything happens.
Edited 1 time(s). Last edit at 02/17/2020 12:38PM by mikeh49.
root@m300-2:~# cat /etc/fstab # /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> LABEL=sata_rootfs / ext3 noatime,errors=remount-ro 0 1 LABEL=data /mnt/data ext4 defaults,noatime 0 2 tmpfs /tmp tmpfs defaults 0 0
Here's the SSD setup:
root@m300-2:~# blkid /dev/sda1: LABEL="sata_rootfs" UUID="2b4c98d6-a7fd-4274-a336-749fb9223cf2" TYPE="ext3" PARTUUID="0a679d7f-01" /dev/sda2: LABEL="data" UUID="75e9b459-2b7a-4f8c-a06a-854d0c8a8cef" TYPE="ext3" PARTUUID="0a679d7f-02"Note that blkid thinks it is ext3.
df thinks it is ext4, but is that because I called it ext4 in fstab? Clearly, I'm not clear if -j only means ext3.
root@m300-2:~# df -Th Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 876M 0 876M 0% /dev tmpfs tmpfs 177M 72K 177M 1% /run /dev/sda1 ext3 3.9G 1.1G 2.7G 29% / tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 353M 0 353M 0% /dev/shm /dev/sda2 ext4 11G 28M 11G 1% /mnt/data tmpfs tmpfs 881M 0 881M 0% /tmp
I'll set fstab to ext3, reboot, and do your suggested tests.
Update: Booted M300 from USB with serial connected and from SSH, ran e2fsck, both partitions clean:
root@M300-usb:~# e2fsck /dev/sda1 e2fsck 1.44.5 (15-Dec-2018) sata_rootfs: clean, 40785/262144 files, 303311/1048576 blocks root@M300-usb:~# e2fsck /dev/sda2 e2fsck 1.44.5 (15-Dec-2018) data: clean, 11/715264 files, 68287/2860800 blocks root@M300-usb:~# blkid /dev/sda1: LABEL="sata_rootfs" UUID="2b4c98d6-a7fd-4274-a336-749fb9223cf2" SEC_TYPE="ext2" TYPE="ext3" PARTUUID="0a679d7f-01" /dev/sda2: LABEL="data" UUID="75e9b459-2b7a-4f8c-a06a-854d0c8a8cef" SEC_TYPE="ext2" TYPE="ext3" PARTUUID="0a679d7f-02" /dev/sdb1: LABEL="rootfs" UUID="d7f52bc6-463d-4440-a877-1f4eb0bbbdef" TYPE="ext3" PARTUUID="10c19223-01"
My environments are setup to check USB for a rootfs label, if found, boot from USB. If not found, check SATA for sata-rootfs and if found boot from there. The hostname of the USB is M300-usb to prevent confusion.
I'll leave things running and try some stuff to see if anything happens.
Edited 1 time(s). Last edit at 02/17/2020 12:38PM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 17, 2020 01:00PM |
Registered: 10 years ago Posts: 155 |
Booted from SSD with "corrected" fstab (sda2 set to ext3) and got this from serial:
Disconnected serial and rebooted OK. dmesg attached for your review.
Edited 1 time(s). Last edit at 03/03/2020 06:41PM by mikeh49.
Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. INIT: version 2.93 booting [info] Using makefile-style concurrent boot in runlevel S. [ ok ] Starting hotplug events dispatcher: systemd-udevd. [ ok ] Synthesizing the initial hotplug events (subsystems)...done. [....] Synthesizing the initial hotplug events (devices)... [ 21.498134][ T402] 8<--- cut here --- [ 21.502146][ T402] Unable to handle kernel paging request at virtual address 00020000 [ 21.510213][ T402] pgd = (ptrval) [ 21.513631][ T402] [00020000] *pgd=00000000 [ 21.517925][ T402] Internal error: Oops: 5 [#1] PREEMPT ARM [ 21.523607][ T402] Modules linked in: [ 21.527370][ T402] CPU: 0 PID: 402 Comm: udevadm Not tainted 5.5.1-kirkwood-tld-1 #1.0 [ 21.535405][ T402] Hardware name: Marvell Kirkwood (Flattened Device Tree) [ 21.542397][ T402] PC is at kernfs_refresh_inode+0x30/0x124 [ 21.548076][ T402] LR is at kernfs_iop_getattr+0x2c/0x4c [ 21.553490][ T402] pc : [<802e3454>] lr : [<802e3574>] psr: 20000013 [ 21.560475][ T402] sp : eed59e98 ip : 00000000 fp : 004896b8 [ 21.566413][ T402] r10: 00000000 r9 : eed58000 r8 : ecf31f78 [ 21.572352][ T402] r7 : 81004228 r6 : ee5ee780 r5 : 00020000 r4 : ee5ee780 [ 21.579598][ T402] r3 : 0000a1ff r2 : 8c4cbd90 r1 : ee5ee780 r0 : ecf31f78 [ 21.586845][ T402] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 21.594701][ T402] Control: 0005397f Table: 6edbc000 DAC: 00000051 [ 21.601163][ T402] Process udevadm (pid: 402, stack limit = 0x(ptrval)) [ 21.607886][ T402] Stack: (0xeed59e98 to 0xeed5a000) [ 21.612953][ T402] 9e80: 000041ed eef57270 [ 21.621858][ T402] 9ea0: ee5d5bb0 8c4cbd90 00000000 00000005 81004228 eed59f78 81004228 8c4cbd90 [ 21.630771][ T402] 9ec0: ee5ee780 810307ac eed59f08 ecf31f78 ee0e4360 802e3574 00000000 ee0e4360 [ 21.639683][ T402] 9ee0: eed59f08 000007ff ee0e4360 802670b0 81004228 7ec1a8d8 00872a31 000000c5 [ 21.648594][ T402] 9f00: 80101208 802676dc 000007ff 00000000 00000000 00000000 00000000 00000000 [ 21.657505][ T402] 9f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 21.666417][ T402] 9f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 21.675329][ T402] 9f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 21.684241][ T402] 9f80: 00000000 00000000 00000000 00000000 00000006 8c4cbd90 00872a31 00872a30 [ 21.693154][ T402] 9fa0: 00872914 80101000 00872a30 00872914 00000005 7ec1a8d8 7ec1a8d8 9b7a0800 [ 21.702065][ T402] 9fc0: 00872a30 00872914 00872a31 000000c5 00000006 00000005 004896c8 004896b8 [ 21.710976][ T402] 9fe0: 004b6cc0 7ec1a8a0 00448198 76e457fc 20000010 00000005 00000000 00000000 [ 21.719898][ T402] [<802e3454>] (kernfs_refresh_inode) from [<802e3574>] (kernfs_iop_getattr+0x2c/0x4c) [ 21.729422][ T402] [<802e3574>] (kernfs_iop_getattr) from [<802670b0>] (vfs_statx_fd+0x44/0x60) [ 21.738246][ T402] [<802670b0>] (vfs_statx_fd) from [<802676dc>] (__do_sys_fstat64+0x28/0x60) [ 21.746896][ T402] [<802676dc>] (__do_sys_fstat64) from [<80101000>] (ret_fast_syscall+0x0/0x50) [ 21.755803][ T402] Exception stack(0xeed59fa8 to 0xeed59ff0) [ 21.761569][ T402] 9fa0: 00872a30 00872914 00000005 7ec1a8d8 7ec1a8d8 9b7a0800 [ 21.770482][ T402] 9fc0: 00872a30 00872914 00872a31 000000c5 00000006 00000005 004896c8 004896b8 [ 21.779392][ T402] 9fe0: 004b6cc0 7ec1a8a0 00448198 76e457fc [ 21.785160][ T402] Code: e1d035b2 e1a06001 e1c130b0 0a000028 (e5953000) [ 21.792004][ T402] ---[ end trace 5831178c34877c62 ]--- Segmentation fault failed. [....] Waiting for /dev to be fully populated... [ 47.848617][ T130] 8<--- cut here --- [ 47.852393][ T130] Unable to handle kernel paging request at virtual address 00002008 [ 47.860366][ T130] pgd = (ptrval) [ 47.863780][ T130] [00002008] *pgd=00000000 [ 47.868073][ T130] Internal error: Oops: 5 [#2] PREEMPT ARM [ 47.873756][ T130] Modules linked in: [ 47.877518][ T130] CPU: 0 PID: 130 Comm: kworker/0:2 Tainted: G D 5.5.1-kirkwood-tld-1 #1.0 [ 47.887297][ T130] Hardware name: Marvell Kirkwood (Flattened Device Tree)
Disconnected serial and rebooted OK. dmesg attached for your review.
Edited 1 time(s). Last edit at 03/03/2020 06:41PM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 17, 2020 04:55PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
OK, at this point, please get the output of
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Quote
Disconnected serial and rebooted OK. dmesg attached for your review.
OK, at this point, please get the output of
free -h swapon
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 17, 2020 05:39PM |
Registered: 10 years ago Posts: 155 |
|
Re: Errors in Boot Log for Dell M300 February 17, 2020 09:45PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
You are running without swap. That could be bad in a lot of ways. This box has a lot of RAM, it would not need swap normally. But as usual, swap space should be always allocated (the Linux kernel memory management works better when there is swap space).
In my basic rootfs, there is no pre-allocate swap partition or file. I let it up to the users to decide whether to use swap file or partition. And I'd recommend swap file (it is much more flexible and there is virtually no difference in performance from swap partition).
Here is an example to create a 1GB swap file at root directory (use whatever you like in this box, I would use at least 1GB swap file on the HDD/SSD rootfs).
And then modify fstab:
And reboot. Now you should have a swap space of 1GB early on during boot.
See if it helps with this probelm.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
You are running without swap. That could be bad in a lot of ways. This box has a lot of RAM, it would not need swap normally. But as usual, swap space should be always allocated (the Linux kernel memory management works better when there is swap space).
In my basic rootfs, there is no pre-allocate swap partition or file. I let it up to the users to decide whether to use swap file or partition. And I'd recommend swap file (it is much more flexible and there is virtually no difference in performance from swap partition).
Here is an example to create a 1GB swap file at root directory (use whatever you like in this box, I would use at least 1GB swap file on the HDD/SSD rootfs).
dd if=/dev/zero of=/swapfile1 bs=1024 count=1048576
And then modify fstab:
cat /etc/fstab | grep swap /swapfile1 swap swap defaults 0 0
And reboot. Now you should have a swap space of 1GB early on during boot.
See if it helps with this probelm.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 18, 2020 09:27AM |
Registered: 10 years ago Posts: 155 |
No joy getting the swap file setup:
Need to do:
Update: dmesg from reboot:
Edited 1 time(s). Last edit at 02/18/2020 10:10AM by mikeh49.
root@m300-2:~# dd if=/dev/zero of=/swapfile1 bs=1024 count=1048576 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 68.0028 s, 15.8 MB/s root@m300-2:~# swapon /swapfile1 swapon: /swapfile1: insecure permissions 0644, 0600 suggested. swapon: /swapfile1: read swap header failed
Need to do:
mkswap /swapfile1then swapon or reboot to get it to work.
Update: dmesg from reboot:
[ 24.382880] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro [ 25.690381] EXT4-fs (sda2): mounting ext3 file system using the ext4 subsystem [ 25.709724] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 26.688594] Adding 1048572k swap on /mnt/data/swapfile1. Priority:-2 extents:14 across:1125440k SSFSIs that what should be expected? sda1 is 4GB for the rootfs, sda2 is the remainder of the 16GB SSD.
Edited 1 time(s). Last edit at 02/18/2020 10:10AM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 18, 2020 03:02PM |
Admin Registered: 13 years ago Posts: 18,543 |
>
Yes, of course.
> [ 26.688594] Adding 1048572k swap on
> /mnt/data/swapfile1. Priority:-2 extents:14
> across:1125440k SSFS
> Is that what should be expected? sda1 is 4GB for
> the rootfs, sda2 is the remainder of the 16GB SSD.
That is expected.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> mkswap /swapfile1 >> then swapon or reboot to get it to work.
Yes, of course.
> [ 26.688594] Adding 1048572k swap on
> /mnt/data/swapfile1. Priority:-2 extents:14
> across:1125440k SSFS
> Is that what should be expected? sda1 is 4GB for
> the rootfs, sda2 is the remainder of the 16GB SSD.
That is expected.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 18, 2020 09:39PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
So how was the reboot with serial console? With swap allocated, do you still see the error?
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
So how was the reboot with serial console? With swap allocated, do you still see the error?
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 18, 2020 10:07PM |
Registered: 10 years ago Posts: 155 |
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 04:28AM |
Registered: 10 years ago Posts: 155 |
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 05:22AM |
Admin Registered: 13 years ago Posts: 18,543 |
Hi Mike,
> One boot with serial connected, no problem, Swap
> was recognized and mounted. dmesg looked norma.
Cool! That's confirmed what I thought seeing the log.
Linux memory management has this quirk. Even though the box has a lot of RAM, during boot (and when it is really out of memory) it makes a difference if there is swap space. The explanation is a long and kind of make sense if we ever have time to discuss this.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> One boot with serial connected, no problem, Swap
> was recognized and mounted. dmesg looked norma.
Cool! That's confirmed what I thought seeing the log.
Linux memory management has this quirk. Even though the box has a lot of RAM, during boot (and when it is really out of memory) it makes a difference if there is swap space. The explanation is a long and kind of make sense if we ever have time to discuss this.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 09:59AM |
Registered: 10 years ago Posts: 155 |
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 11:00AM |
Registered: 10 years ago Posts: 155 |
Second boot witth swap on and serial connected failed. Filled the PuTTy buffer with this repeating message until I killed the power. So, I missed where it started:
Booted again and this time successful.
Found this:
https://tipstricks.itmatrix.eu/solving-the-running-scripts-local-block-loop-while-booting-in-linux/
And this:
local-block issue
Edited 4 time(s). Last edit at 02/20/2020 11:21AM by mikeh49.
Begin: Running /scripts/local-block ... done. blkid: relocation error: /lib/arm-linux-gnueabi/libc.so.6: symbol __pointer_chk_guard version GLIBC_PRIVATE not defined in file ld-linux.so.3 with link time reference sleep: relocation error: /lib/arm-linux-gnueabi/libc.so.6: symbol __pointer_chk_guard version GLIBC_PRIVATE not defined in file ld-linux.so.3 with link time reference cat: relocation error: /lib/arm-linux-gnueabi/libc.so.6: symbol __pointer_chk_guard version GLIBC_PRIVATE not defined in file ld-linux.so.3 with link time reference
Booted again and this time successful.
Begin: Running /scripts/local-blockdoes not appear in the successful boot log.
Found this:
https://tipstricks.itmatrix.eu/solving-the-running-scripts-local-block-loop-while-booting-in-linux/
And this:
local-block issue
Edited 4 time(s). Last edit at 02/20/2020 11:21AM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 05:00PM |
Admin Registered: 13 years ago Posts: 18,543 |
mikeh49 Wrote:
-------------------------------------------------------
> It just feels better knowing it is there. Sort of
> like a favorite toy that it sleeps with.
:)) right. The some paging out activity occured during boot does not like no swap space.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
-------------------------------------------------------
> It just feels better knowing it is there. Sort of
> like a favorite toy that it sleeps with.
:)) right. The some paging out activity occured during boot does not like no swap space.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 20, 2020 05:39PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
> Found this:
> https://tipstricks.itmatrix.eu/solving-the-running-scripts-local-block-loop-while-booting-in-linux/
>
> And this:
> local-block
> issue
These are good workaround for this problem. However, if you update your initramfs (i.e. update-initramfs -u, and then regenerate the uInitrd), I don't think this is necessary in the latest Debian distribution.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> Found this:
> https://tipstricks.itmatrix.eu/solving-the-running-scripts-local-block-loop-while-booting-in-linux/
>
> And this:
> local-block
> issue
These are good workaround for this problem. However, if you update your initramfs (i.e. update-initramfs -u, and then regenerate the uInitrd), I don't think this is necessary in the latest Debian distribution.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 22, 2020 12:59PM |
Registered: 10 years ago Posts: 155 |
I updated the initramfs using update-initramfs -u, and regenerated uInitrd. It apparently did something, as the new file size was slightly different than previous.
Booted OK.
Second boot today completed, but got this message on serial:
Also this:
Any significance to these?
Other boot errors I've had also relate to this libc.so.6 file (which is linked to libc-2.28.so). Note that I did apt-get update/apt-get upgrade on this box. Could this have created a version mismatch or something? Is there a log of what update did? I can't find one.
Booted OK.
Second boot today completed, but got this message on serial:
INIT: version 2.93 booting [info] Using makefile-style concurrent boot in runlevel S. findfs: unable to resolve 'LABEL=sata_rootfs' [ ok ] Starting hotplug events dispatcher: systemd-udevd. [ ok ] Synthesizing the initial hotplug events (subsystems)...done. [....] Synthesizing the initial hotplug events (devices)...[ 21.902410][ T388] systemd-udevd[388]: Using default interface naming scheme 'v240'. [ ok done. [....] Waiting for /dev to be fully populated...[ 22.178242][ T388] systemd-udevd[388]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 22.269214][ T400] watchdog: f1020300.watchdog-timer: driver supplied timeout (4294967295) out of range [ 22.454870][ T400] watchdog: f1020300.watchdog-timer: falling back to default timeout (21) [ 22.517065][ T379] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 22.602208][ T400] orion_wdt: Initial timeout 21 sec [ 22.810083][ T401] systemd-udevd[401]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 22.917641][ T389] marvell-cesa f1030000.crypto: CESA device successfully registered done. [ ok ] Setting hostname to 'm300-2'...done. [warn] Not activating swap on swapfile. ... (warning). [ 24.790557][ T627] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro [info] Will now check all file systems. fsck from util-linux 2.33.1 Checking all file systems. LABEL=sata_rootfs is mounted LABEL=data is not mounted [/sbin/fsck.ext3 (1) -- /mnt/data] fsck.ext3 -y -C0 /dev/sda2 e2fsck 1.44.5 (15-Dec-2018) data: clean, 12/715264 files, 330688/2860800 blocks [....] Done checking file systems. A log is being saved in /var/log/fsck/checkfs[ ok hat location is writable.. [ ok ] Cleaning up temporary files...[....] Cleaning /tmp...done. [ ok . [....] Will now mount local filesystems:[ 26.108072][ T910] EXT4-fs (sda2): mounting ext3 file system using the ext4 subsystem [ 26.127435][ T910] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) . ok [....] Will now activate swapfile swap:swapon: /mnt/data/swapfile1: insecure permissions 0644, 0600 suggested. swapon: /mnt/data/swapfile1: found signature [pagesize=4096, signature=swap] swapon: /mnt/data/swapfile1: pagesize=4096, swapsize=1073741824, devsize=1073741824 swapon /mnt/data/swapfile1 [ 27.128610][ T956] Adding 1048572k swap on /mnt/data/swapfile1. Priority:-2 extents:14 across:1125440k SSFS done.Although, swap was activated later.
Also this:
Starting version 241
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
udevadm: relocation error: udevadm: symbol sigisemptyset version GLIBC_2.4 not defined in file libc.so.6 with link time reference
Begin: Will now check root file system ... fsck from util-linux 2.33.1
[/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a -C0 /dev/sda1
sata_rootfs: clean, 40787/262144 files, 526717/1048576 blocks
Any significance to these?
Other boot errors I've had also relate to this libc.so.6 file (which is linked to libc-2.28.so). Note that I did apt-get update/apt-get upgrade on this box. Could this have created a version mismatch or something? Is there a log of what update did? I can't find one.
|
Re: Errors in Boot Log for Dell M300 February 22, 2020 05:19PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
> I updated the initramfs using update-initramfs -u,
> and regenerated uInitrd. It apparently did
> something, as the new file size was slightly
> different than previous.
That a good sign. Meaning you really needed to regenerate it.
> Booted OK.
> udevadm: relocation error: udevadm:
> symbol sigisemptyset version GLIBC_2.4 not defined
> in file libc.so.6 with link time
> reference
> Any significance to these?
Only the udevadm error above is relevant. However, I would test boot a couple time with both SATA rootfs and another non-rootfs USB drive plugged in at boot.
>
> Other boot errors I've had also relate to this
> libc.so.6 file (which is linked to libc-2.28.so).
> Note that I did apt-get update/apt-get upgrade on
> this box. Could this have created a version
> mismatch or something? Is there a log of what
> update did? I can't find one.
The logs can be found in /var/log.
Each time you do apt-get upgrade, a new initrd might be created and apt will output a message like this (suppose you are running 5.2.9-kirkwood-tld-1):
If you don't recall seeing that or not (e.g. exiting terminal) then do a listing to check the timestamp.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> I updated the initramfs using update-initramfs -u,
> and regenerated uInitrd. It apparently did
> something, as the new file size was slightly
> different than previous.
That a good sign. Meaning you really needed to regenerate it.
> Booted OK.
> udevadm: relocation error: udevadm:
> symbol sigisemptyset version GLIBC_2.4 not defined
> in file libc.so.6 with link time
> reference
> Any significance to these?
Only the udevadm error above is relevant. However, I would test boot a couple time with both SATA rootfs and another non-rootfs USB drive plugged in at boot.
>
> Other boot errors I've had also relate to this
> libc.so.6 file (which is linked to libc-2.28.so).
> Note that I did apt-get update/apt-get upgrade on
> this box. Could this have created a version
> mismatch or something? Is there a log of what
> update did? I can't find one.
The logs can be found in /var/log.
Each time you do apt-get upgrade, a new initrd might be created and apt will output a message like this (suppose you are running 5.2.9-kirkwood-tld-1):
update-initramfs: Generating /boot/initrd.img-5.2.9-kirkwood-tld-1Then you need to recreate uInitrd.
If you don't recall seeing that or not (e.g. exiting terminal) then do a listing to check the timestamp.
ls -lart /bootIf you see uInitrd is older than the latest initrd.img-5.2.9-kirkwood-tld-1 kernel file then you know that uInitrd is obsolete and need to be recreated (This is the reason my instruction always uses "cp -a" so that timestamp is preserved when a file is copied).
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 03:16PM |
Registered: 10 years ago Posts: 155 |
Updating my recent results:
I booted with the rootfs USB, did e2fsck on the SSD sda1 and it reported clean.
Then rebooted the internal SSD with the non-rootfs USB plugged in, and booted OK. See below:
Is the initramfs messed up? Could I go back to the image included with the 5.5.1 kernel package? FWIW, everytime I generate a new uInitrd, it gets bigger. Is it worth looking inside the various versions to compare?
If you want, I can try to put the whole story together.
Update: I unpacked the uInitrd (a copy in the /tmp directory) and found a script had been added by ntfs-3g, hence the need to regenerate it. So, that's understood by me. Makes sense that an app that needs to do something at boot will need to make a change to initramfs.
Edited 2 time(s). Last edit at 02/26/2020 06:26PM by mikeh49.
- Successfully booted from the internal SSD with a non-rootfs USB plugged-in several times. The presence of the USB slows down the boot process between the Starting version v241 and Loading essential drivers steps.
- Installed ntfs-3g using apt-get install and it installed ntfs-3g but it also reported triggers waiting for initramfs update and so it created a new initramfs image file.
- I rebuilt uInitrd using this new image file and rebooted twice with errors.
Loading, please wait... Starting version 241 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. udevadm: symbol lookup error: /lib/arm-linux-gnueabi/libcrypto.so.1.1: undefined symbol: BASIC_CKNSTRAINTS_it, version [ 19.861906][ T192] Internal error: Oops - undefined instruction: 0 [#1] PREEMPT ARM [ 19.870116][ T192] Modules linked in: [ 19.873886][ T192] CPU: 0 PID: 192 Comm: fstype Not tainted 5.5.1-kirkwood-tld-1 #1.0 [ 19.881835][ T192] Hardware name: Marvell Kirkwood (Flattened Device Tree) [ 19.888833][ T192] PC is at nlm_sysctl_table+0x0/0x4 [ 19.893907][ T192] LR is at get_disk_and_module+0x4/0x5c [ 19.899327][ T192] pc : [<8111a9e0>] lr : [<805ca620>] psr: 00000013 [ 19.906311][ T192] sp : 8279fda0 ip : 00000000 fp : 805ca078 [ 19.912250][ T192] r10: 8255bc80 r9 : 8272f800 r8 : 00000000 [ 19.918188][ T192] r7 : 8279fde8 r6 : 0000000f r5 : eed53000 r4 : 00800001 [ 19.925435][ T192] r3 : 00000008 r2 : 805ca678 r1 : 8272f800 r0 : 8272f800 [ 19.932682][ T192] Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 19.940539][ T192] Control: 0005397f Table: 027a8000 DAC: 00000051 [ 19.947000][ T192] Process fstype (pid: 192, stack limit = 0x(ptrval)) [ 19.953636][ T192] Stack: (0x8279fda0 to 0x827a0000) [ 19.958706][ T192] fda0: 00800001 806c3ccc eed53000 00000008 8279fde8 00800001 8279fde8 8206a3e0 [ 19.967617][ T192] fdc0: 0800001d 81004228 ee611410 00000000 00000000 805ca83c ee611400 00000000 [ 19.976529][ T192] fde0: 8206a3e0 8029cba8 00000001 34c9e40c 00000000 00000000 ee611400 8206a3e0 [ 19.985441][ T192] fe00: 0800001d 8206a3e8 8279e000 00000000 00000000 8029cfcc 8206a3e0 00000000 [ 19.994352][ T192] fe20: 82735820 8029d484 8206a3e8 8279e000 00000000 8025f024 8279fed0 8206a3e0 [ 20.003263][ T192] fe40: 00000000 00000001 ffffff9c 80271048 00000002 effd65c0 8279ff78 00020000 [ 20.012176][ T192] fe60: 00000001 ee8023b8 00000041 000041ed 00000000 00000004 00000000 ecd94220 [ 20.021088][ T192] fe80: 00000000 8279e000 eced93e0 00000002 82735820 eef57b70 ee5506e8 00000000 [ 20.029999][ T192] fea0: 00000000 34c9e40c 820766e4 00000003 81004228 8279ff78 00000001 ffffff9c [ 20.038912][ T192] fec0: 8279e000 00000000 00000000 8027123c eef57b70 ee5506e8 d76e9258 00000004 [ 20.047823][ T192] fee0: eeefe015 00100cc2 00000000 ee802088 82735820 00000101 00000002 00000044 [ 20.056735][ T192] ff00: 00000000 00000000 00000000 8279ff10 00000ff0 7efaee6d 00000ff0 eeefe010 [ 20.065647][ T192] ff20: ffffe000 8063a654 eeefe010 7efaee6d 00000ff0 34c9e40c 00000003 eef57f34 [ 20.074558][ T192] ff40: eeefe000 00000000 00000003 00000002 ffffff9c 8027f060 00000000 34c9e40c [ 20.083471][ T192] ff60: 81004228 00000003 00020000 81004228 eeefe000 8025fdb4 00020000 00000000 [ 20.092382][ T192] ff80: 00000004 00000100 00000001 34c9e40c 7efaed30 00000002 7efaed24 00000005 [ 20.101295][ T192] ffa0: 80101208 80101000 7efaed30 00000002 7efaee6d 00020000 7efaee66 0000002f [ 20.110205][ T192] ffc0: 7efaed30 00000002 7efaed24 00000005 00000000 00000000 00000000 00000000 [ 20.119118][ T192] ffe0: 018223a0 7efaece0 018053f8 0180e7b8 20000010 7efaee6d 00000000 00000000 [ 20.128039][ T192] [<805ca620>] (get_disk_and_module) from [<00000008>] (0x8) [ 20.135294][ T192] Code: 00000000 00000000 00000000 0000000b (eef0d6c0) [ 20.142112][ T192] ---[ end trace c68b18a6a4d56605 ]--- OPENSSL_1_1_0 Segmentation fault [ 47.848600][ T130] 8<--- cut here --- [ 47.852371][ T130] Unable to handle kernel paging request at virtual address 04000008 [ 47.860342][ T130] pgd = (ptrval) [ 47.863757][ T130] [04000008] *pgd=00000000 [ 47.868042][ T130] Internal error: Oops: 5 [#2] PREEMPT ARM [ 47.873724][ T130] Modules linked in: [ 47.877487][ T130] CPU: 0 PID: 130 Comm: Tainted: G D 5.5.1-kirkwood-tld-1 #1.0 [ 47.886306][ T130] Hardware name: Marvell Kirkwood (Flattened Device Tree)Second boot:
Loading, please wait... Starting version 241 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. udevadm: symbol lookup error: /lib/arm-linux-gnueabi/libcrypto.so.1.1: undefined symbol: BASIC_CKNSTRAINTS_it, version OPENSSL_1_1_0 Begin: Will now check root file system ... fsck from util-linux 2.33.1 [/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a -C0 /dev/sda1 fsck.ext3: symbol lookup error: /lib/arm-linux-gnueabi/libext2fs.so.2: undefined symbol: ext2fs_rb_first fsck exited with status code 127 done.
I booted with the rootfs USB, did e2fsck on the SSD sda1 and it reported clean.
Then rebooted the internal SSD with the non-rootfs USB plugged in, and booted OK. See below:
Loading, please wait... [ 7.931904][ T130] usb 1-1.2: New USB device found, idVendor=058f, idProduct=6387, bcdDevice= 1.04 [ 7.948647][ T130] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 7.956688][ T130] usb 1-1.2: Product: USB 2.0 Starting version 241 [ 7.988742][ T130] usb 1-1.2: Manufacturer: CBM [ 7.993381][ T130] usb 1-1.2: SerialNumber: 38A45713 [ 8.009186][ T130] usb-storage 1-1.2:1.0: USB Mass Storage device detected [ 8.028962][ T130] scsi host2: usb-storage 1-1.2:1.0 [ 8.048698][ T168] usbcore: registered new interface driver uas Begin: Loading essential drivers ... done. [ 9.049816][ T122] scsi 2:0:0:0: Direct-Access CBM USB 2.0 8.07 PQ: 0 ANSI: 2 [ 9.063549][ T22] sd 2:0:0:0: [sdb] 1978368 512-byte logical blocks: (1.01 GB/966 MiB) [ 9.075653][ T22] sd 2:0:0:0: [sdb] Write Protect is off [ 9.081636][ T22] sd 2:0:0:0: [sdb] No Caching mode page found [ 9.087673][ T22] sd 2:0:0:0: [sdb] Assuming drive cache: write through [ 9.331360][ T22] sdb: sdb1 [ 9.340652][ T22] sd 2:0:0:0: [sdb] Attached SCSI removable disk Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. Begin: Will now check root file system ... fsck from util-linux 2.33.1 [/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a -C0 /dev/sda1 sata_rootfs: recovering journal sata_rootfs: clean, 40918/262144 files, 544610/1048576 blocks done. [ 19.087784][ T204] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [ 19.105455][ T204] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. INIT: version 2.93 booting
Is the initramfs messed up? Could I go back to the image included with the 5.5.1 kernel package? FWIW, everytime I generate a new uInitrd, it gets bigger. Is it worth looking inside the various versions to compare?
If you want, I can try to put the whole story together.
Update: I unpacked the uInitrd (a copy in the /tmp directory) and found a script had been added by ntfs-3g, hence the need to regenerate it. So, that's understood by me. Makes sense that an app that needs to do something at boot will need to make a change to initramfs.
Edited 2 time(s). Last edit at 02/26/2020 06:26PM by mikeh49.
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 06:58PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
What is your
Ans try again
See if anything change.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
What is your
uname -a cat /etc/apt/sources.list ls -lart /boot
Ans try again
apt-get update apt-get upgrade
See if anything change.
ls -lart /boot
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 08:14PM |
Registered: 10 years ago Posts: 155 |
root@m300-2:~# uname -a Linux m300-2 5.5.1-kirkwood-tld-1 #1.0 PREEMPT Sat Feb 1 22:28:36 PST 2020 armv5tel GNU/Linux root@m300-2:~# cat /etc/sources.list cat: /etc/sources.list: No such file or directory root@m300-2:~# ls -lart /boot total 194148 -rwxr-xr-x 1 root root 4963616 Aug 17 2019 zImage-5.2.9-kirkwood-tld-1 -rw-r--r-- 1 root root 165790 Aug 17 2019 config-5.2.9-kirkwood-tld-1 -rw------- 1 root root 4963616 Aug 17 2019 vmlinuz-5.2.9-kirkwood-tld-1 -rw------- 1 root root 3101889 Aug 17 2019 System.map-5.2.9-kirkwood-tld-1 -rw-r--r-- 1 root root 9649570 Aug 17 2019 linux-headers-5.2.9-kirkwood-tld-1_1.0_armel.deb -rw-r--r-- 1 root root 9713872 Aug 24 2019 initrd.img-5.2.9-kirkwood-tld-1 -rw-r--r-- 1 root root 9713936 Aug 24 2019 uInitrd-5.2.9-tld-1 drwxr-xr-x 2 root root 4096 Jan 27 19:17 dts -rwxr-xr-x 1 root root 5202864 Feb 1 22:27 vmlinuz-5.5.1-kirkwood-tld-1 -rw-r--r-- 1 root root 173805 Feb 1 22:27 config-5.5.1-kirkwood-tld-1 -rw-r--r-- 1 root root 3827141 Feb 1 22:27 System.map-5.5.1-kirkwood-tld-1 -rwxr-xr-x 1 root root 5202864 Feb 2 00:49 zImage-5.5.1-kirkwood-tld-1 -rw-r--r-- 1 root root 11367904 Feb 2 01:03 linux-headers-5.5.1-kirkwood-tld-1_1.0_armel.deb -rw-r--r-- 1 root root 21294372 Feb 2 01:05 linux-image-5.5.1-kirkwood-tld-1_1.0_armel.deb -rw-r--r-- 1 root root 1290240 Feb 2 01:18 linux-dtb-5.5.1-kirkwood-tld-1.tar -rw-r--r-- 1 root root 160577 Feb 3 15:33 linux-5.5.1-kirkwood-tld-1.patch -rw-r--r-- 1 root root 38157439 Feb 8 15:49 linux-5.5.1-kirkwood-tld-1-bodhi.tar.bz2 -rw-r--r-- 1 root root 4974316 Feb 14 15:36 uImage-5.2.9-tld-1 -rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd.2-20 -rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd-5.5.1 -rwxr-xr-x 1 root root 5213353 Feb 15 11:52 zImage.fdt -rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage-5.5.1 -rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage drwxr-xr-x 22 root root 4096 Feb 18 07:42 .. -rw-r--r-- 1 root root 9670325 Feb 20 16:59 uInitrd.2-26 -rw-r--r-- 1 root root 9917730 Feb 26 11:11 initrd.img-5.5.1-kirkwood-tld-1 drwxr-xr-x 3 root root 4096 Feb 26 11:17 . -rw-r--r-- 1 root root 9917794 Feb 26 11:19 uInitrd root@m300-2:~#From apt-get updgrade:
Processing triggers for initramfs-tools (0.133+deb10u1) ... update-initramfs: Generating /boot/initrd.img-5.5.1-kirkwood-tld-1 root@m300-2:~#
/boot after upgrade
root@m300-2:~# ls -lart /boot
total 194216
-rwxr-xr-x 1 root root 4963616 Aug 17 2019 zImage-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 165790 Aug 17 2019 config-5.2.9-kirkwood-tld-1
-rw------- 1 root root 4963616 Aug 17 2019 vmlinuz-5.2.9-kirkwood-tld-1
-rw------- 1 root root 3101889 Aug 17 2019 System.map-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 9649570 Aug 17 2019 linux-headers-5.2.9-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 9713872 Aug 24 2019 initrd.img-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 9713936 Aug 24 2019 uInitrd-5.2.9-tld-1
drwxr-xr-x 2 root root 4096 Jan 27 19:17 dts
-rwxr-xr-x 1 root root 5202864 Feb 1 22:27 vmlinuz-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 173805 Feb 1 22:27 config-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 3827141 Feb 1 22:27 System.map-5.5.1-kirkwood-tld-1
-rwxr-xr-x 1 root root 5202864 Feb 2 00:49 zImage-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 11367904 Feb 2 01:03 linux-headers-5.5.1-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 21294372 Feb 2 01:05 linux-image-5.5.1-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 1290240 Feb 2 01:18 linux-dtb-5.5.1-kirkwood-tld-1.tar
-rw-r--r-- 1 root root 160577 Feb 3 15:33 linux-5.5.1-kirkwood-tld-1.patch
-rw-r--r-- 1 root root 38157439 Feb 8 15:49 linux-5.5.1-kirkwood-tld-1-bodhi.tar.bz2
-rw-r--r-- 1 root root 4974316 Feb 14 15:36 uImage-5.2.9-tld-1
-rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd.2-20
-rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd-5.5.1
-rwxr-xr-x 1 root root 5213353 Feb 15 11:52 zImage.fdt
-rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage-5.5.1
-rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage
drwxr-xr-x 22 root root 4096 Feb 18 07:42 ..
-rw-r--r-- 1 root root 9670325 Feb 20 16:59 uInitrd.2-26
-rw-r--r-- 1 root root 9917794 Feb 26 11:19 uInitrd
-rw-r--r-- 1 root root 9986142 Feb 26 18:07 initrd.img-5.5.1-kirkwood-tld-1
drwxr-xr-x 3 root root 4096 Feb 26 18:07 .
root@m300-2:~#
A new ramdisk image as expected. I stopped here.
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 09:04PM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
> root@m300-2:~# uname -a
> Linux m300-2 5.5.1-kirkwood-tld-1 #1.0 PREEMPT Sat
> Feb 1 22:28:36 PST 2020 armv5tel GNU/Linux
> root@m300-2:~# cat /etc/sources.list
> cat: /etc/sources.list: No such file or directory
My typo!
cat /etc/apt/souce*
> root@m300-2:~# ls -lart /boot
> total 194148
> -rwxr-xr-x 1 root root 4963616 Aug 17 2019
> zImage-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 165790 Aug 17 2019
> config-5.2.9-kirkwood-tld-1
> -rw------- 1 root root 4963616 Aug 17 2019
> vmlinuz-5.2.9-kirkwood-tld-1
> -rw------- 1 root root 3101889 Aug 17 2019
> System.map-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 9649570 Aug 17 2019
> linux-headers-5.2.9-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 9713872 Aug 24 2019
> initrd.img-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 9713936 Aug 24 2019
> uInitrd-5.2.9-tld-1
> drwxr-xr-x 2 root root 4096 Jan 27 19:17 dts
> -rwxr-xr-x 1 root root 5202864 Feb 1 22:27
> vmlinuz-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 173805 Feb 1 22:27
> config-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 3827141 Feb 1 22:27
> System.map-5.5.1-kirkwood-tld-1
> -rwxr-xr-x 1 root root 5202864 Feb 2 00:49
> zImage-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 11367904 Feb 2 01:03
> linux-headers-5.5.1-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 21294372 Feb 2 01:05
> linux-image-5.5.1-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 1290240 Feb 2 01:18
> linux-dtb-5.5.1-kirkwood-tld-1.tar
> -rw-r--r-- 1 root root 160577 Feb 3 15:33
> linux-5.5.1-kirkwood-tld-1.patch
> -rw-r--r-- 1 root root 38157439 Feb 8 15:49
> linux-5.5.1-kirkwood-tld-1-bodhi.tar.bz2
> -rw-r--r-- 1 root root 4974316 Feb 14 15:36
> uImage-5.2.9-tld-1
> -rw-r--r-- 1 root root 9670274 Feb 15 11:48
> uInitrd.2-20
> -rw-r--r-- 1 root root 9670274 Feb 15 11:48
> uInitrd-5.5.1
> -rwxr-xr-x 1 root root 5213353 Feb 15 11:52
> zImage.fdt
> -rw-r--r-- 1 root root 5213417 Feb 15 11:53
> uImage-5.5.1
> -rw-r--r-- 1 root root 5213417 Feb 15 11:53
> uImage
> drwxr-xr-x 22 root root 4096 Feb 18 07:42 ..
> -rw-r--r-- 1 root root 9670325 Feb 20 16:59
> uInitrd.2-26
> -rw-r--r-- 1 root root 9917730 Feb 26 11:11
> initrd.img-5.5.1-kirkwood-tld-1
> drwxr-xr-x 3 root root 4096 Feb 26 11:17 .
> -rw-r--r-- 1 root root 9917794 Feb 26 11:19
> uInitrd
> root@m300-2:~#
> [/code]
> From apt-get updgrade:
>
At this point, you need to generate uInitrd before reboot.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
> root@m300-2:~# uname -a
> Linux m300-2 5.5.1-kirkwood-tld-1 #1.0 PREEMPT Sat
> Feb 1 22:28:36 PST 2020 armv5tel GNU/Linux
> root@m300-2:~# cat /etc/sources.list
> cat: /etc/sources.list: No such file or directory
My typo!
cat /etc/apt/souce*
> root@m300-2:~# ls -lart /boot
> total 194148
> -rwxr-xr-x 1 root root 4963616 Aug 17 2019
> zImage-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 165790 Aug 17 2019
> config-5.2.9-kirkwood-tld-1
> -rw------- 1 root root 4963616 Aug 17 2019
> vmlinuz-5.2.9-kirkwood-tld-1
> -rw------- 1 root root 3101889 Aug 17 2019
> System.map-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 9649570 Aug 17 2019
> linux-headers-5.2.9-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 9713872 Aug 24 2019
> initrd.img-5.2.9-kirkwood-tld-1
> -rw-r--r-- 1 root root 9713936 Aug 24 2019
> uInitrd-5.2.9-tld-1
> drwxr-xr-x 2 root root 4096 Jan 27 19:17 dts
> -rwxr-xr-x 1 root root 5202864 Feb 1 22:27
> vmlinuz-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 173805 Feb 1 22:27
> config-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 3827141 Feb 1 22:27
> System.map-5.5.1-kirkwood-tld-1
> -rwxr-xr-x 1 root root 5202864 Feb 2 00:49
> zImage-5.5.1-kirkwood-tld-1
> -rw-r--r-- 1 root root 11367904 Feb 2 01:03
> linux-headers-5.5.1-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 21294372 Feb 2 01:05
> linux-image-5.5.1-kirkwood-tld-1_1.0_armel.deb
> -rw-r--r-- 1 root root 1290240 Feb 2 01:18
> linux-dtb-5.5.1-kirkwood-tld-1.tar
> -rw-r--r-- 1 root root 160577 Feb 3 15:33
> linux-5.5.1-kirkwood-tld-1.patch
> -rw-r--r-- 1 root root 38157439 Feb 8 15:49
> linux-5.5.1-kirkwood-tld-1-bodhi.tar.bz2
> -rw-r--r-- 1 root root 4974316 Feb 14 15:36
> uImage-5.2.9-tld-1
> -rw-r--r-- 1 root root 9670274 Feb 15 11:48
> uInitrd.2-20
> -rw-r--r-- 1 root root 9670274 Feb 15 11:48
> uInitrd-5.5.1
> -rwxr-xr-x 1 root root 5213353 Feb 15 11:52
> zImage.fdt
> -rw-r--r-- 1 root root 5213417 Feb 15 11:53
> uImage-5.5.1
> -rw-r--r-- 1 root root 5213417 Feb 15 11:53
> uImage
> drwxr-xr-x 22 root root 4096 Feb 18 07:42 ..
> -rw-r--r-- 1 root root 9670325 Feb 20 16:59
> uInitrd.2-26
> -rw-r--r-- 1 root root 9917730 Feb 26 11:11
> initrd.img-5.5.1-kirkwood-tld-1
> drwxr-xr-x 3 root root 4096 Feb 26 11:17 .
> -rw-r--r-- 1 root root 9917794 Feb 26 11:19
> uInitrd
> root@m300-2:~#
> [/code]
> From apt-get updgrade:
>
> Processing triggers for initramfs-tools > (0.133+deb10u1) ... > update-initramfs: Generating > /boot/initrd.img-5.5.1-kirkwood-tld-1 > root@m300-2:~# >
At this point, you need to generate uInitrd before reboot.
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 09:30PM |
Registered: 10 years ago Posts: 155 |
Current sources (after update and upgrade):
I'll generate uInitrd and report back.
root@m300-2:~# cat /etc/apt/sources.list deb http://ftp.us.debian.org/debian buster main deb-src http://ftp.us.debian.org/debian buster main deb http://security.debian.org/ buster/updates main contrib non-free deb-src http://security.debian.org/ buster/updates main contrib non-free deb http://http.debian.net/debian buster-updates main contrib deb-src http://http.debian.net/debian buster-updates main contrib
I'll generate uInitrd and report back.
|
Re: Errors in Boot Log for Dell M300 February 26, 2020 09:55PM |
Registered: 10 years ago Posts: 155 |
After generating uInitrd:
Booted OK from internal SSD with a non-rootfs USB drive connected.
Failed to boot without the USB connected:
root@m300-2:/boot# ls -lart
total 194284
-rwxr-xr-x 1 root root 4963616 Aug 17 2019 zImage-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 165790 Aug 17 2019 config-5.2.9-kirkwood-tld-1
-rw------- 1 root root 4963616 Aug 17 2019 vmlinuz-5.2.9-kirkwood-tld-1
-rw------- 1 root root 3101889 Aug 17 2019 System.map-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 9649570 Aug 17 2019 linux-headers-5.2.9-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 9713872 Aug 24 2019 initrd.img-5.2.9-kirkwood-tld-1
-rw-r--r-- 1 root root 9713936 Aug 24 2019 uInitrd-5.2.9-tld-1
drwxr-xr-x 2 root root 4096 Jan 27 19:17 dts
-rwxr-xr-x 1 root root 5202864 Feb 1 22:27 vmlinuz-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 173805 Feb 1 22:27 config-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 3827141 Feb 1 22:27 System.map-5.5.1-kirkwood-tld-1
-rwxr-xr-x 1 root root 5202864 Feb 2 00:49 zImage-5.5.1-kirkwood-tld-1
-rw-r--r-- 1 root root 11367904 Feb 2 01:03 linux-headers-5.5.1-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 21294372 Feb 2 01:05 linux-image-5.5.1-kirkwood-tld-1_1.0_armel.deb
-rw-r--r-- 1 root root 1290240 Feb 2 01:18 linux-dtb-5.5.1-kirkwood-tld-1.tar
-rw-r--r-- 1 root root 160577 Feb 3 15:33 linux-5.5.1-kirkwood-tld-1.patch
-rw-r--r-- 1 root root 38157439 Feb 8 15:49 linux-5.5.1-kirkwood-tld-1-bodhi.tar.bz2
-rw-r--r-- 1 root root 4974316 Feb 14 15:36 uImage-5.2.9-tld-1
-rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd.2-20
-rw-r--r-- 1 root root 9670274 Feb 15 11:48 uInitrd-5.5.1
-rwxr-xr-x 1 root root 5213353 Feb 15 11:52 zImage.fdt
-rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage-5.5.1
-rw-r--r-- 1 root root 5213417 Feb 15 11:53 uImage
drwxr-xr-x 22 root root 4096 Feb 18 07:42 ..
-rw-r--r-- 1 root root 9670325 Feb 20 16:59 uInitrd.2-26
-rw-r--r-- 1 root root 9986142 Feb 26 18:07 initrd.img-5.5.1-kirkwood-tld-1
drwxr-xr-x 3 root root 4096 Feb 26 18:07 .
-rw-r--r-- 1 root root 9986206 Feb 26 19:31 uInitrd
root@m300-2:/boot#
Forgot to save the uInitrd that I made today (prior to update-upgrade tonight).
Booted OK from internal SSD with a non-rootfs USB drive connected.
Failed to boot without the USB connected:
Loading, please wait... Starting version 241 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. udevadm: symbol lookup error: /lib/arm-linux-gnueabi/libcrypto.so.1.1: undefined symbol: ASN1_UNIVERSALSTRINC_it, version OPENSSL_1_1_0 Begin: Will now check root file system ... fsck from util-linux 2.33.1 [/sbin/fsck.ext3 (1) -- /dev/sda1] fsck.ext3 -a -C0 /dev/sda1 fsck.ext3: symbol lookup error: /lib/arm-linux-gnueabi/libext2fs.so.2: undefined symbol: ext2fs_rb_first fsck exited with status code 127 done. Failure: File system check of the root filesystem failed The root filesystem on /dev/sda1 requires a manual fsck BusyBox v1.30.1 (Debian 1:1.30.1-4) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) [ 47.848618][ T130] 8<--- cut here --- [ 47.852387][ T130] Unable to handle kernel NULL pointer dereference at virtual address 00000108 [ 47.861231][ T130] pgd = (ptrval) [ 47.864646][ T130] [00000108] *pgd=00000000 [ 47.868946][ T130] Internal error: Oops: 5 [#1] PREEMPT ARM [ 47.874630][ T130] Modules linked in: [ 47.878393][ T130] CPU: 0 PID: 130 Comm: Not tainted 5.5.1-kirkwood-tld-1 #1.0 [ 47.885818][ T130] Hardware name: Marvell Kirkwood (Flattened Device Tree)Same issue as earlier today, a manual e2fsck showed it clean.
|
Re: Errors in Boot Log for Dell M300 February 27, 2020 12:41AM |
Admin Registered: 13 years ago Posts: 18,543 |
Mike,
Have you installed anything else before on this rootfs? is ntfs-3g the 1st package that you've installed ?
What is your output of
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Edited 1 time(s). Last edit at 02/27/2020 12:51AM by bodhi.
Have you installed anything else before on this rootfs? is ntfs-3g the 1st package that you've installed ?
What is your output of
cat /etc/debian_version dpkg -l | grep ssl
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Edited 1 time(s). Last edit at 02/27/2020 12:51AM by bodhi.
|
Re: Errors in Boot Log for Dell M300 February 27, 2020 03:29AM |
Registered: 9 years ago Posts: 226 |
This sounds a lot like my experience with the M300. I can boot off of SSD with something plugged into the USB port, but without that it may or may not boot and if it does it's very unreliable. I hope you guys figure it out because I haven't had any time to try to figure out what's going on. My experience with the M300 hasn't been very good even when I get a reliable boot, so it's been relegated to the bottom of my to-do list.
-JT
-JT
|
Re: Errors in Boot Log for Dell M300 February 27, 2020 04:00AM |
Admin Registered: 13 years ago Posts: 18,543 |
JT,
Ah. I understood your problem now. Don't have any idea why yet, but I can see that was a strange behavior that happens with a new SSD.
Did you install crytodev or anything like that?
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Ah. I understood your problem now. Don't have any idea why yet, but I can see that was a strange behavior that happens with a new SSD.
Did you install crytodev or anything like that?
-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
