Welcome! Log In Create A New Profile

Advanced

Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs

Posted by cdlenfert 
Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 11:09AM
I've been somewhat consistently having issues on my PogoPlug E-02 running Debian 9 (Stretch). I say somewhat because I'm not exactly sure how frequently this happens, but I'd guess once or twice a month, but maybe more.

The issue I'm experience seems to be the system as a whole locking up, freezing, or becoming partially inaccessible. If I'm using a program like forked-daapd to play my music library, I'll lose the ability to connect to the server on my remote app. I notice after a few days of running normally, sometimes the process of connecting to my library from the remote app becomes very slow. If I've recently rebooted that process is usually pretty quick.

If I'm in an SSH session, the session will become unresponsive. Yesterday I was running apt-get install alsa-utils and during the process everything froze. Eventually (like 10 minutes later) my terminal session returned me to root@pogoplug.local, but any additional commands failed to execute.

If I ping my Pogo when it's in this state, I get a response as if everything is normal, but everything on the device seems unusable.

When I check ram usage with free -h or htop I usually have plenty of ram available. On average I'm using around 50-60mb and the E-02 has 256mb with 512mb of swap available. I'm running the same services (plus more) on a Pogoplug Pro with half the ram available and not getting freezes on that device.

Any ideas how I can start troubleshooting this? Logs to check? Thanks for any tips.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 11:17AM
If you've got access to the device when it happens running dmesg for the kernel output is a good first step. /var/log/messages and /var/log/syslog are also good places to start.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 11:22AM
Thanks!

I'll try to do dmesg the next time I notice freezing or lagginess, but so far it's already past the point of me running any command or even connecting via ssh.

I also don't have either of those logs. Is there something I need to do to get them to be created?
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 11:44AM
I might not be the best person to ask when it comes to these devices. These logs might be disabled to reduce disk activity or something like that (I'm guessing if you can run apt-get install it's not a read only filesystem). If you have any logs in /var/log they may be a good place to start.

there are a bunch of options you can pursue but someone with more experience troubleshooting these devices might have something more useful to suggest.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 12:36PM
cdlenfert,

> I also don't have either of those logs. Is there
> something I need to do to get them to be created?

My released rootfs (if you are running it) has busybox-syslogd running with option logging to RAM.

The first step in debugging issue like this is to remove the option "logging to RAM", so that you have syslog output to /var/log/messages in the rootfs.

cat /etc/default/busybox-syslogd
# Defaults for busybox-syslogd initscript
# This is a POSIX shell fragment sourced by /etc/init.d/busybox-syslogd

# Additional options that are passed to the daemons.  Default is to log
# to ring buffer (to be read with logread(1)) and drop duplicates.
SYSLOG_OPTS="-C128"
KLOG_OPTS=""

Temporarily remove the option by commenting it out.
#SYSLOG_OPTS="-C128"
And then restart busybox-syslogd, or reboot the system.

When it is starting to exhibit the behavior, if you don't have serial console and cannot SSH in then just unplug power cord and hope that you can see the /var/log/messages showing some errors (even partial log can be useful). The rootfs file system EXT3 will be OK when you unplug power like that. The worst case is you need to mount it on aonther box and run e2fsck to fix any error cause by power outage.

It is best to troubleshoot this problem using serial console.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 01/23/2019 12:39PM by bodhi.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 23, 2019 08:33PM
Thank you! We'll see if I capture anything in the log the next time I come across this issue. If not I'll likely set up a serial connection.

cat /var/log/messages 
Jan 23 19:28:34 Pogoplug syslog.info syslogd started: BusyBox v1.22.1
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 25, 2019 12:58AM
Any chance it's a spotty usb thumb drive? I've seen behavior much like this on a pogo-e02 of mine which was tracked down to a bad usb thumb drive. New sandisk drive fixed it right up.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
January 25, 2019 09:27PM
I suppose it's possible it could be a failing drive, It's a used laptop drive thrown in a cheap external enclosure. I've never really put it to the test. The problem is really inconsistent. When I've had thumb drives that fail, they seem to go pretty quickly. I'll keep the rootfs drive in mind though as a possible culprit. Thanks for the idea.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 01, 2019 12:47PM
I've been having a very similar issue for about a year on my Stora MS2000 running Debian from a USB flash drive. Normally the system is pretty stable when idling for weeks and just reading and writing stuff on the hard drives, but occasionally when I run apt commands the system hangs mid-operation, which sometimes makes the system unbootable until I plug the USB drive into my computer and repair things. If I have a second SSH connection open running htop I notice that htop keeps running and updating the display, but I can't cancel any commands and I can't open any new SSH connections. Strangely I've noticed that using 'apt install packagename' or 'aptitude install packagename' crashes consistently for certain packages, but using 'apt-get install packagename' works fine.

I'd like to hear if others have found a solution yet but I'll try a new thumb drive and see if that helps.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 01, 2019 03:50PM
All,

The most likely problem here is the rootfs was corrupted. If you are running HDD rootfs, then use smarttools to monitor the drive, and have the system log to the HDD. When the problem occur, use a USB rootfs to boot the box and look at the log of SMART and system log for clues if the drive has power problem (sometime it is the power supply that causes the corruption, too).

If you have serial console, then try to log in through that to see if it will get you to the Debian shell.

If you use USB rootfs permanently, then you should use a good brand such as Sandisk.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 02, 2019 06:36AM
The internal memory (NAND?) in my stora is corrupt so I've been using a Sandisk thumb drive as permanent rootfs that I bought new for the purpose. Is it possible I got a dud or fake branded one?

When I was setting up the system last year I found that blacklisting certain ehci modules made the system more stable - I thought it might be because there wasn't enough power getting to the USB port for USB v2, only USB 1.x -
blacklist ehci_hcd
blacklist orion-ehci
While troubleshooting today I tried commenting out these lines in /etc/modprobe.d/blacklist.conf and found that these modules aren't loaded any more anyway.

I enabled writing syslogs to disk as suggested above. I've been stress-testing the Stora and its thumb drive and it seems to be hard to replicate the issue consistently but managed to trigger it just now. Simultaneous large 'dd' writes don't seem to do it, but every once in a while an apt/dpkg operation will cause processes to start becoming unresponsive one by one. After pulling the power cable and rebooting I see the following entries in /var/log/messages. The dmesg logs don't seem to show anything and the file /var/log/syslog doesn't exist.

Feb 2 12:09:07 stora user.err kernel: [ 489.424754] INFO: task dpkg:2335 blocked for more than 120 seconds.
Feb 2 12:09:07 stora user.err kernel: [ 489.431094] Not tainted 4.19.0-kirkwood-tld-1 #1
Feb 2 12:09:07 stora user.err kernel: [ 489.444489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 2 12:09:07 stora user.info kernel: [ 489.452565] dpkg D 0 2335 2160 0x00000000
Feb 2 12:09:07 stora user.warn kernel: [ 489.458912] [<c08f8568>] (__schedule) from [<c08f86cc>] (schedule+0xa8/0xc8)
Feb 2 12:09:07 stora user.warn kernel: [ 489.466697] [<c08f86cc>] (schedule) from [<c033bec8>] (jbd2_log_wait_commit+0x110/0x138)
Feb 2 12:09:07 stora user.warn kernel: [ 489.482685] [<c033bec8>] (jbd2_log_wait_commit) from [<c02e712c>] (ext4_sync_file+0x3a4/0x3c8)
Feb 2 12:09:07 stora user.warn kernel: [ 489.491704] [<c02e712c>] (ext4_sync_file) from [<c0289910>] (vfs_fsync+0x20/0x28)
Feb 2 12:09:07 stora user.warn kernel: [ 489.499600] [<c0289910>] (vfs_fsync) from [<c0289940>] (do_fsync+0x28/0x44)
Feb 2 12:09:07 stora user.warn kernel: [ 489.507446] [<c0289940>] (do_fsync) from [<c0101000>] (ret_fast_syscall+0x0/0x50)
Feb 2 12:09:07 stora user.warn kernel: [ 489.515874] Exception stack(0xc5d5bfa8 to 0xc5d5bff0)
Feb 2 12:09:07 stora user.warn kernel: [ 489.524219] bfa0: 01c52368 beb5a568 00000008 00000000 01c527d0 00000008
Feb 2 12:09:07 stora user.warn kernel: [ 489.539387] bfc0: 01c52368 beb5a568 01c52348 00000076 01c52368 0000000a 00000002 00000000
Feb 2 12:09:07 stora user.warn kernel: [ 489.550983] bfe0: 00000000 beb5a54c 00503688 b6e401dc
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 02, 2019 02:44PM
Are you using ext4 for your rootfs drive? Any chance it's related to this bug/issue?

https://bugzilla.redhat.com/show_bug.cgi?id=1527783
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 02, 2019 04:50PM
Quote

I enabled writing syslogs to disk as suggested above. I've been stress-testing the Stora and its thumb drive and it seems to be hard to replicate the issue consistently but managed to trigger it just now. Simultaneous large 'dd' writes don't seem to do it, but every once in a while an apt/dpkg operation will cause processes to start becoming unresponsive one by one. After pulling the power cable and rebooting I see the following entries in /var/log/messages. The dmesg logs don't seem to show anything and the file /var/log/syslog doesn't exist.

/var/log/messages is the syslog.

And like alexr said, I think this is EXT4-related problem. Note that the actual file system could be EXT3. The EXT4 driver is used for both EXT3/EXT4.

@bennytheben,

Is this USB drive 3.0? or any USB 3.0 drive is plugged during this time?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 02/02/2019 04:58PM by bodhi.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 03, 2019 05:28AM
Thanks for your quick responses.

It's an ext3 partition and it's a USB 3.0 thumb drive. I've disabled journaling and it's now showing mounted as ext2.

I just ran 'aptitude update' and the system started gradually grinding to a halt. First the 'aptitude update' process shows as status 'D' in top, and then various other processes that are trying to write to the flash drive eg. cron and syslogd as well.

I've set syslogd to write to the hard drive in case it's been missing stuff due to the fact it's been writing to the thumb drive, but since disabling journaling I haven't seen any more errors reported in the syslog when the issue is occurring.

Any ideas?



Edited 1 time(s). Last edit at 02/03/2019 06:16AM by bennytheben.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 03, 2019 06:33AM
bennytheben,

I don't know the root cause of this problem. However, I've seen the /proc/sys/kernel/hung_task_timeout_secs occur many times in the following scenario, and all caused by EXT4 driver, and seem to occur only when a USB 3.0 is plugged to the USB port.

On GoFlex Net (Kirkwood), a scheduled rsync back up starts right at the time dpkg is scheduled to run daily.

Feb  1 06:25:01 tldDebian cron.info CRON[30644]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

When I moved rsync job to a different time slot, and move the Sandisk Ultra Fit 16G USB 3.0 rootfs to a USB 3.0 hub, dpkg no longer causes the hung task timout. I did not investigate further.


So I think perhaps you can try either

- clone your USB 3.0 rootfs to a USB 2.0 flash drive and run it as rootfs.
- or use a USB 3.0 hub and move this rootfs to that hub.

I have not seen this on other Kirkwood boxes that run 24/7 in my farm. A few of them running USB 3.0 rootfs, but always plugged into a USB 3.0 hub.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 03, 2019 07:22AM
I've cloned the rootfs onto a USB 2.0 thumb drive with ext4 (with journaling enabled) and that seems to have resolved the issue, thank you!
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 03, 2019 09:36AM
I was able to crash/freeze my system again in a similar way. This time I was doing an rsync of a 2TB external drive to another 2TB external drive. I did get into htop a few times and saw RAM usage going up to 125MB so maybe it maxed out at some point, but my Pogo E-02 has 256MB with 512MB of swap enabled.

Anyway I could still Ping the IP, but not the hostname. No SSH (refused) and other services were unavailable.

Checking /var/log/messages

Feb  2 19:35:44 Pogoplug daemon.err smbd[1837]:   Unable to open new log file '/tmp/var/.log': No such file or directory
Feb  2 19:36:08 Pogoplug authpriv.info sshd[2086]: pam_unix(sshd:session): session closed for user root

There were a ton of these daemon.err smbd messages about being unable to open new log file. Not sure why it would be trying to?
And then the ssh session closing is about when everything likely became unavailable. Nothing more helpful than that as far as I could tell.
I don't expect to be running massive rsync sessions so often, but if it keeps happening regardless of what I'm doing on the Pogo I'll plan to hook up a serial connection.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 03, 2019 12:03PM
cdlenfert,

> Anyway I could still Ping the IP, but not the
> hostname. No SSH (refused) and other services were
> unavailable.

That means avahi-daemon is not responsive. And I think because either CPU probably maxed out or it hung up waiting for its turn to run, as same as sshd daemon.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 14, 2019 11:18AM
a new development today, while running
autoreconf
I got a "Bus error". Shortly after any command that referenced the file system would hang. In my research it looks like it could point to hardware issues. I'm going to attempt a system repair and see if that helps.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 14, 2019 01:30PM
cdlenfert ,

> I got a "Bus error". Shortly after any command
> that referenced the file system would hang. In my
> research it looks like it could point to hardware
> issues. I'm going to attempt a system repair and
> see if that helps.

It is a hardware failure, but only the disk drive that is bad.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 14, 2019 04:07PM
That's not totally surprising to me :) The question now is can I successfully make a backup of my system and restore it to a new drive (assuming the drive doesn't crap out again during the process) without carrying over corrupt files or other problems that may have been created by a failing drive?
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 15, 2019 12:19AM
> can I successfully make a backup
> of my system and restore it to a new drive
> (assuming the drive doesn't crap out again during
> the process) without carrying over corrupt files
> or other problems that may have been created by a
> failing drive?

There is no way to ensure that. Best you can hope is to do a fsck to fix file system errors and those errors did not mess up important files. And then copy the rootfs to another drive.

This is one of those times when you really need to restore from a rsync daily backup if you had set it up before sh*t happens :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
February 15, 2019 09:25PM
I took the rootfs USB drive for my E-02 and tried connecting it to my Raspberry Pi. The Pi didn't have enough power to spin the drive so the drive just clicked continuously. I hooked it up to my Mac and no more clicking. My Mac couldn't mount it (being ext3 format) so I made a backup of the base drive in DMG (compressed disc image) form. Since that basically copied my entire drive without a hiccup I thought the drive could possible be OK and not actually failing.

I plugged the rootfs USB into a Pogoplug Pro and ran the tar commands mentioned in the Wiki - https://forum.doozan.com/read.php?2,12096,24034#msg-24034. I made sure I was in the proper directory when creating the tar.bz2.

I noticed that the sector size of my USB drive and the format of the partition didn't align. 512 vs. 4096. This gave me a warning about performance when I did fdisk -l. I decided to reformat the drive. I took the opportunity to go from ext3 to ext4. I made sure to run fdisk partitioning on the base of the drive (I think the issue before was that I ran it on the first partition instead). The performance/alignment warning went away. All seemed well. I restored the tarball to the USB drive and it went smoothly.

I plugged the USB in and tried to boot the Pogo and no luck getting ssh, though ping works. I'll create a basic rootfs and see if I can get an idea of why the restore didn't work out.

Update: I created a basic rootfs and the Pogo booted and I had access. While trying to set things up again I got another "Bus Error". I decided to give up on the drive as the rootfs drive (though it seems to work fine for other purposes). I restored my DMG on my Mac to the USB hard drive so I have access to all of my original data and configs. I created a new rootfs on a USB flash drive, running apt and other processes no longer gave me any issues. I was able to mount my old rootfs (changed the label so I don't get boot conflicts) and had easy access to all my old files. Setting up my new rootfs was pretty easy after that.

What was still giving me problems was that even with the new rootfs I was having boot issues. Something would hang/fail and I think I've narrowed it down.

I'm not sure how to pass multiple boot arguments via the uEnv.txt file
## Currently
root@pogoplug:~# cat /boot/uEnv.txt 
custom_params=init=/bin/systemd
## Wanted to add zSwap
root@pogoplug:~# cat /boot/uEnv.txt 
custom_params=init=/bin/systemd
custom_params= zswap.enabled=1
Could the above setup of uEnv.txt have caused boot issues? I thought the idea of that file was to pass boot parameters without causing issues if the formatting was incorrect (basically ignored if wrong). I'm probably misinterpreting that, and eEnv.txt is just a safer option than writing to nand.

What is the correct way to pass multiple parameters and enable zswap via uEnv.txt? I've read variations on the forum such as:
# this (note the usb part)
usb_custom_params=zswap.enabled=1
# or
extraargs=zswap.enabled=1
#or to concatenate the params maybe?
custom_params=init=/bin/systemd zswap.enabled=1

The other part of the boot issue puzzle I'm considering is my /etc/fstab setup:
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
LABEL=rootfs    /               ext4    noatime,errors=remount-ro 0 1
tmpfs          /tmp            tmpfs   defaults          0       0
LABEL=HIT2TB  /mnt/pogo2      ext4    rw      0       0
LABEL=HIT2TB-2	/mnt/pogo3	ext4	rw	0	0
In this case the HIT2TB-2 drive's enclosure had a power issue and wouldn't spin the disk so it could not mount. Are the fstab options requiring this drive in order to completely boot?



Edited 1 time(s). Last edit at 02/18/2019 11:00AM by cdlenfert.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
October 26, 2019 10:57AM
Hello again, it's been a while...

I've had my stora running with rootfs on a Sandisk Cruzer Blade for quite a while without any issues. I moved it to a new location recently so I decided to clone the rootfs onto a Sandisk Cruzer Fit so that the thumb drive is much smaller and less vulnerable to getting physically damaged. I bought a USB 2 drive especially because of the issues I had with a USB 3 drive previously mentioned in this thread, but I'm getting the issue with the system hanging again.

I did lsusb on both drives for comparison:
Bus 003 Device 003: ID 0781:5567 SanDisk Corp. Cruzer Blade
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0781 SanDisk Corp.
  idProduct          0x5567 Cruzer Blade
  bcdDevice            1.02
  iManufacturer           1 
  iProduct                2 
  iSerial                 3 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              200mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               1

Bus 003 Device 004: ID 0781:5571 SanDisk Corp. Cruzer Fit
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0781 SanDisk Corp.
  idProduct          0x5571 Cruzer Fit
  bcdDevice            1.00
  iManufacturer           1 
  iProduct                2 
  iSerial                 3 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              224mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0

Any ideas would be greatly appreciated.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
October 26, 2019 05:25PM
bennytheben,

Bus 003 Device 004: ID 0781:5571 SanDisk Corp. Cruzer Fit
Couldn't open device, some information will be missing
The error above should not happen if you are root.

> but I'm getting the issue with the
> system hanging again.

The USB 2.0 Cruzer Fit should be fined to use as rootfs. I had several Kirkwood boxes that run this drive as rootfs for many years.

Can you post the /var/log/messages here (or pastebin it) when the system hang?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
October 26, 2019 05:31PM
bennytheben,

I should also mention this: It is perfectly OK to run USB 3.0 as rootfs on Kirkwood boxes that have only USB 2.0. I use the Sandisk Ultra Fit 16GB for several boxes now.

I use this approach:

- Plug in the Ultra Fit to the USB port right next to the ethernet port (and not using a hub).

- If the USB 3.0 Ultra Fit drive is inconsistently (or not) booting, I would then use in a USB 3.0 hub (small hanging type to save space), and then plug in the rootfs to the hub.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
November 03, 2019 05:48AM
Hi Bodhi, thanks for your replies. Greatly appreciated as always.

I've had it running on the new thumb drive for a few days again and it crashed again in the last 24 hours. I can't remember exactly when I did the switch over to the new one this time, but the logs will show this (in the boot messages it mentions 'Cruzer Blade' for the old thumb drive and 'Cruzer Fit' for the new one). I've switched back to the old drive for now so the system stays up and running while I diagnose the new one on my local machine. I've the attached the output of
cat var/log/messages | grep -v 'sshd' | grep -v 'transmission-daemon' | grep -e 'Oct' -e 'Nov'
(removing some irrelevant entries for security and so it's less for you to trawl through!).

Quote
bodhi
- Plug in the Ultra Fit to the USB port right next to the ethernet port (and not using a hub).
My Stora has a USB port on the front panel but not on the back. I haven't taken the case off in a while but I don't seem to remember seeing one there last time I looked.

Do you have any idea why using a hub makes the system more stable on USB 3.0 drives?

It always boots fine but at some point (usually during high I/O on the thumb drive) it gradually grinds to a halt and becomes completely unresponsive.

I look forward to hearing from you.



Edited 3 time(s). Last edit at 11/03/2019 05:52AM by bennytheben.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
November 03, 2019 05:13PM
bennytheben,

It seems one of problems is out of memory (most likely this is the only problem).

> My Stora has a USB port on the front panel but not
> on the back. I haven't taken the case off in a
> while but I don't seem to remember seeing one
> there last time I looked.

Yes, that was just a hint in case you have more than 1 USB port. I forgot this Stora has only one USB port!

> Do you have any idea why using a hub makes the
> system more stable on USB 3.0 drives?

This is due to the controller differences. One on the hubs tends to have newer FW and deal with USB 2.0 in more robust ways. When we use a USB 3.0 hub, we let the hub negotiate the 2.0 speed with the box. If we plug the USB 3.0 directly to the port, then it would be the FW on the USB thumb drive that negotiates the 2.0 speed with the box.

My experience with these Kirkwood and OXNAS boxes: there are a few models of USB 3.0 thumb drives that have a lot of problems if I plug them in directly. The 3.0 hub always solved this problem.

I use a Sabrent 4-port hub (travel type) that is quite small and hang behind a box (thus take no fooprint at all). I think they discontinued this (last time I bought was US$ 8).

================


The fist thing to do is the increase the virtual memory and let it run for a while.

cat /etc/sysctl.d/local.conf
# increase the free page caches for VM
vm.min_free_kbytes = 65536
vm.swappiness = 60

If you dont have local.conf then just create one with the above content. Or copy /etc/sysctl.conf content into local.conf and add the above 3 lines at the end.


UPDATE:

BTW, the log file you've attached was a full log, quite big!

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 11/03/2019 05:15PM by bodhi.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
November 05, 2019 01:07PM
Hi Bodhi,

I've added the file with the config lines you suggested but the machine still hung while trying to do an apt-get install command. The Cruzer Fit I'm using is USB 2.1 but I've ordered a USB 3.0 hub just in case that helps... if all else fails it will at least allow me to have a chunkier thumb drive plugged in without it being at risk of damage.

I sent you a PM about something else loosely related.
Re: Troubleshooting Freeze Issue on Debian 9 - SSH, Ping, Logs
November 05, 2019 06:01PM
bennytheben,

> The Cruzer Fit
> I'm using is USB 2.1 but I've ordered a USB 3.0
> hub just in case that helps...

No it won't help. That trrick only solves problem when the drive is USB 3.0.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Author:

Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: