Welcome! Log In Create A New Profile

Advanced

USB's lifetime

Posted by enki 
USB's lifetime
September 30, 2013 05:37PM
Hi All,

@DonCharisma has mentioned that it's wise to minimize activity on USB Debian stick used to boot GoFlexNet. Sure things this is true. An ordinary USB stick will have limited write/read cycles and will fail one day, ussually before you expect. I had this problem before and probably many of us have it.

Kindly search for your oppinion how is the best to save USB's lifetime by minimizing write/read actvity on USB. @DonCharisma suggested to disable as much as possible logs activity. My question is; what is the most efficient way to do this in Debian wheezy we have?

It was another suggestion not to place Swap partition on the USB. What is the best way to do it and how?

ES.
Re: USB's lifetime
September 30, 2013 11:39PM
To prolong the life of your rootfs USB thumb drive:

1. Use busybox-syslogd to log to RAM.
2. For each package you installed, either specify the log file in /tmp. Or overmount the log files to /tmp (mount -o bind).
3. Use a swap file so you can create it on an attached HDD. And create one on flash drive in case the HDD is not attached. Use a script to detect if the HDD is present during boot. If it is not then switch to the USB thumb swap (this should not happen, so it is just a precaution).
4. Watch the swap if it is on USB thumb. There should be no swap needed. If seeing swap actvities then always use the HDD to handle swap.
5. Use iotop command to watch frequent disk IOs (to figure out which log files needed to be moved or overmounted).
iotop -o -b -qqq
6. Use zswap if the kernel supports it.

With all these remedies, you can use Ext3 for the USB rootfs on a thumbdrive to prevent corruption during power outage.

-bodhi
===========================
Wiki
latest Kirkwood kernel builds and rootfs
latest u-boot-kirkwood builds
latest Oxnas kernel builds and rootfs
latest u-boot-oxnas builds
latest MVEBU Armada kernel builds and rootfs
U-Boot & Kernel Booting process
bodhi's u-boot GitHub
bodhi's corner



Edited 3 time(s). Last edit at 07/13/2015 04:01PM by bodhi.
Re: USB's lifetime
October 04, 2013 03:10PM
enki,

bodhi I know is an "expert" in these parts, so I reckon he knows what he's talking about.

I've tried to install Ubuntu on SD memory card before and that didn't work out too well. I thought I'd broken the memory card, but when I formatted it back on FAT/FAT32 it was fine. I read somewhere that SD cards are designed to work FAT formatted and don't like other file systems. I've tried different SD cards and results were the same/similar. The only common factor was the reader/writer USB unit, it's possible that this is the problem and a different unit would work OK.

For USB sticks, I only have had Sandisk Cruzer and Cruzer fit, and they seem to work OK, although I'd err to the side of the standard Cruzer (not the Fit). There may well be other brands and models, but I just don't have any experience.

If you're going to use Flash stick persistently, then my suggestion would be get the install how you want it, then pull the stick and duplicate it on a Desktop machine (double sure take an image of the rootfs). Keep the backup stick handy for when it gives up. Obviously if you're making major changes then you need to do the copy procedure again.

Another option would be to go the Embeded Debian on your Pogoplug/Goflex, I think that built in flash is faster than the USB and it's more likely designed for heavier usage. There are posts on the forum about how to do this.

One last option I know there is a company in UK called linitx, they sell industrial flash drives, presumably those bad-boys will last a lot longer.

Cheers

Don Charisma
http://DonCharisma.com
Re: USB's lifetime
October 04, 2013 03:56PM
Sandisk Fit is a real good choice! I've been using 4G Sandisk Fit ext3 rootfs for my 2 media NAS for about a year. Swap file is on an attached Ext3 HDD. So far it has survived about 5 power outages. Booted right backup as if nothing has happened.
shv
Re: USB's lifetime
October 05, 2013 01:47AM
Maybe a flash stick with SLC flash cells is recommended even it is more expensive. Some additional hints need to be regarded. It is more or less the same as for SSD usage. To prevent wear out of certain flash areas only one partition over the complete Flash memory shall be used. The partition shall be aligned to the Flash sectors. I don't know how to find out the right alignment for Flash sticks. Moreover the Flash controller in the stick must have a good wear leveling function to prevent wear out of certain flash regions. Because just one partition is available a swap file must be used.
I am not sure if TRIM is supported with USB sticks. If not invalid flash pages are deleted with a garbage collection function of the flash stick.

Many problems with TRIM, garbage collection and wear leveling occured with certain SSDs in the past. Thay were sometimes resolved by firmware updates. Therefore we can expect that there such problems also occure with USB sticks. But I didn't see any firmware update for USB sticks.

- shv
Re: USB's lifetime
October 07, 2013 08:43AM
Any personal experience on how long a life a USB stick lasts? I think mine is over a year or two already.
eas
Re: USB's lifetime
October 10, 2013 08:23PM
I have a couple Mushkin Mullholland 8GB USB sticks I've been using for two years without fail. I haven't babied them. I haven't bothered with having syslog write to RAM. I have SWAP on the stick. I don't even think I've totally disabled atime. What I did do though, is only use 50% of the storage space.
Re: USB's lifetime
October 15, 2013 08:17AM
Is there a practical way to watch for failing drives? Like SMART for flash? The wear leveling is entirely internal to the flash drive, but doesn't it report "bad" or "unusable" sectors to the OS? Is there a good tool for Linux to spot the count going up to preemptively change the drive? One thing that comes to mind in terms of partitioning swap and maybe all of /var/log (if a HDD isn't an option) is to just put a second small cheap flash drive in (we all have a dozen or so of those 1gb or less with no use anymore, amiright) and let it fail; when it fails just toss a new one in. But, if there is no good way to tell when the drive is failing it might be just as troubling to have the unit hang even if the main data is safe.
shv
Re: USB's lifetime
October 20, 2013 01:06AM
There is an interessting discussion about flash memory wear out of SD cards with RaspberryPi: Link.. They recommend to use the fstrim command so that unused blocks become available for wear leveling.

It seams that there might be USB3.0 sticks available which support trim: Link. With the tool TRIMcheck it can be tested if a drive supports trim: Link. If you have a Windows PC where you know that trim is working e.g. with SSD you could try to test USB stick as well.

The problem is not just wearing out of USB flash memory drives but also slowing down write operations if each write needs an erase before.



Edited 1 time(s). Last edit at 10/20/2013 01:09AM by shv.
Termo
Re: USB's lifetime
May 01, 2014 06:42AM
I to have thought a bit about this topic, and recently I fell over this small script:

http://www.nohl.eu/tech-resources/notes-to-linux/log-files-in-tmpfs-without-breaking-logging/

The last post is for debian. Maybe Bodhi could implement this in future rootfs builds?

Regards
TEN
Re: USB's lifetime
January 04, 2017 01:18PM
joebob2000 Wrote:
-------------------------------------------------------
> Is there a practical way to watch for failing drives?
> Is there a good tool for Linux to spot the count going up to preemptively change the drive?

Not exactly, but unlike SMART,
to test the (remaining) free space both initially and occasionally,
there is http://oss.digirati.com.br/f3/
as a native (unixoid) remake
of Heise.de's H2test(w) for (Win)DOS.

Over the past decade at least, I have committed no USB or SD to "production" here without a full test cycle under one of the above, and (besides forgeries) caught many that were faulty fresh-from-factory.



Edited 3 time(s). Last edit at 01/04/2017 01:23PM by TEN.
Re: USB's lifetime
September 06, 2017 10:40AM
Isn't it a good idea to set vm.swappiness to 0 so that swap is used only when it is absolutely necessary, instead of being used as a page cache?

Is zswap already enabled by default or do we need to do anything?

admin@pogoplug:~$ cat /proc/cmdline
console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
admin@pogoplug:~$ dmesg | grep zswap
[    9.886230] zswap: loaded using pool lzo/zbud
admin@pogoplug:~$ cat /sys/module/zswap/parameters/enabled
N
Re: USB's lifetime
September 06, 2017 04:44PM
haridsv,

> Isn't it a good idea to set vm.swappiness to 0 so
> that swap is used only when it is absolutely neces
> sary, instead of being used as a page cache?

Not really. If you use a desktop on your plug such as XFCE, LDE, KDE, ...

Quote
http://forum.doozan.com/read.php?2,23630
Debian Desktop

XFCE desktop Pogoplug Debian

then swapiness should be set to 0-10 to help the UI more responsive. But for a server, that would make the response time for other apps worse when you are close to running out of RAM. Servers should be balance in swapping to keep it running at a consistent rate.

zswap compresses the swap in memory before needing to swap to disk.

>
> Is zswap already enabled by default or do we need
> to do anything?

You need to activate it in the kernel bootargs:
zswap.enabled=1
In the new u-boot default envs there is a custom_params varaible already at the end of bootargs. So you can set it directly, or use the uEnv.txt to activate it.

In Debian:
fw_setenv custom_params 'zswap.enabled=1'

Or in uEnv.txt
custom_params=zswap.enabled=1

-bodhi
===========================
Wiki
latest Kirkwood kernel builds and rootfs
latest u-boot-kirkwood builds
latest Oxnas kernel builds and rootfs
latest u-boot-oxnas builds
latest MVEBU Armada kernel builds and rootfs
U-Boot & Kernel Booting process
bodhi's u-boot GitHub
bodhi's corner
Re: USB's lifetime
September 06, 2017 11:03PM
bodhi Wrote:
-------------------------------------------------------
> zswap compresses the swap in memory before needing
> to swap to disk.

Is zswap recommended only when using a flash storage for swap, or is it also useful when swap is setup on a hard drive? Also, does enabling zswap put a significant toll on the CPU that is in Pogoplug E02?
Re: USB's lifetime
September 07, 2017 12:04AM
> Is zswap recommended only when using a flash stora
> ge for swap, or is it also useful when swap is set
> up on a hard drive? Also, does enabling zswap put
> a significant toll on the CPU that is in Pogoplug
> E02?

It is mainly for the flash drives to avoid swapping to flash. However, it could be useful for hard drives also, if swapping is small and infrequent. IOW, if there are little swapping, it will stay in compressed space in RAM. But it is a tradeoff between RAM and CPU power. You'll be the judge.

In servers operations, zswap is not recommended if the swap space is in HDD (there are signifcant swapping going on with lots of daemon running that use lots of RAM).

zswap does not slow down CPU much at all. For plugs that has no SATA interface such as Pogo E02, zswap is really helpful (even with USB HDD attached, because of the inherent disk thrashing of USB protocol). For plugs such as GoFlex Net, with swap file on the HDD, I don't use zswap because SATA is fast and buffered.

BTW, I have most of the Kirkwood boxes in my farm (about 20, with 10-12 always running), so these are real world examples.

-bodhi
===========================
Wiki
latest Kirkwood kernel builds and rootfs
latest u-boot-kirkwood builds
latest Oxnas kernel builds and rootfs
latest u-boot-oxnas builds
latest MVEBU Armada kernel builds and rootfs
U-Boot & Kernel Booting process
bodhi's u-boot GitHub
bodhi's corner
Re: USB's lifetime
September 07, 2017 03:14AM
bodhi Wrote:
-------------------------------------------------------

> It is mainly for the flash drives to avoid swappin
> g to flash. However, it could be useful for hard d
> rives also, if swapping is small and infrequent. I
> OW, if there are little swapping, it will stay in
> compressed space in RAM. But it is a tradeoff betw
> een RAM and CPU power. You'll be the judge.
>
> In servers operations, zswap is not recommended if
> the swap space is in HDD (there are signifcant swa
> pping going on with lots of daemon running that us
> e lots of RAM).
>
> zswap does not slow down CPU much at all. For plug
> s that has no SATA interface such as Pogo E02, zsw
> ap is really helpful (even with USB HDD attached,
> because of the inherent disk thrashing of USB prot
> ocol). For plugs such as GoFlex Net, with swap fil
> e on the HDD, I don't use zswap because SATA is fa
> st and buffered.
>
> BTW, I have most of the Kirkwood boxes in my farm
> (about 20, with 10-12 always running), so these ar
> e real world examples.

Thank you, appreciate the details. I am convinced enough to enable it.
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: