USB's lifetime

Hi All,

@DonCharisma has mentioned that it's wise to minimize activity on USB Debian stick used to boot GoFlexNet. Sure things this is true. An ordinary USB stick will have limited write/read cycles and will fail one day, ussually before you expect. I had this problem before and probably many of us have it.

Kindly search for your oppinion how is the best to save USB's lifetime by minimizing write/read actvity on USB. @DonCharisma suggested to disable as much as possible logs activity. My question is; what is the most efficient way to do this in Debian wheezy we have?

It was another suggestion not to place Swap partition on the USB. What is the best way to do it and how?

ES.

To prolong the life of your rootfs USB thumb drive:

1. Use busybox-syslogd to log to RAM.
2. For each package you installed, either specify the log file in /tmp. Or overmount the log files to /tmp (mount -o bind).
3. Use a swap file so you can create it on an attached HDD. And create one on flash drive in case the HDD is not attached. Use a script to detect if the HDD is present during boot. If it is not then switch to the USB thumb swap (this should not happen, so it is just a precaution).
4. Watch the swap if it is on USB thumb. There should be no swap needed. If seeing swap actvities then always use the HDD to handle swap.
5. Use iotop command to watch frequent disk IOs (to figure out which log files needed to be moved or overmounted).
iotop -o -b -qqq
6. Use zswap if the kernel supports it.

With all these remedies, you can use Ext3 for the USB rootfs on a thumbdrive to prevent corruption during power outage.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 3 time(s). Last edit at 07/13/2015 05:01PM by bodhi.

enki,

bodhi I know is an "expert" in these parts, so I reckon he knows what he's talking about.

I've tried to install Ubuntu on SD memory card before and that didn't work out too well. I thought I'd broken the memory card, but when I formatted it back on FAT/FAT32 it was fine. I read somewhere that SD cards are designed to work FAT formatted and don't like other file systems. I've tried different SD cards and results were the same/similar. The only common factor was the reader/writer USB unit, it's possible that this is the problem and a different unit would work OK.

For USB sticks, I only have had Sandisk Cruzer and Cruzer fit, and they seem to work OK, although I'd err to the side of the standard Cruzer (not the Fit). There may well be other brands and models, but I just don't have any experience.

If you're going to use Flash stick persistently, then my suggestion would be get the install how you want it, then pull the stick and duplicate it on a Desktop machine (double sure take an image of the rootfs). Keep the backup stick handy for when it gives up. Obviously if you're making major changes then you need to do the copy procedure again.

Another option would be to go the Embeded Debian on your Pogoplug/Goflex, I think that built in flash is faster than the USB and it's more likely designed for heavier usage. There are posts on the forum about how to do this.

One last option I know there is a company in UK called linitx, they sell industrial flash drives, presumably those bad-boys will last a lot longer.

Cheers

Don Charisma
http://DonCharisma.com

Sandisk Fit is a real good choice! I've been using 4G Sandisk Fit ext3 rootfs for my 2 media NAS for about a year. Swap file is on an attached Ext3 HDD. So far it has survived about 5 power outages. Booted right backup as if nothing has happened.

Maybe a flash stick with SLC flash cells is recommended even it is more expensive. Some additional hints need to be regarded. It is more or less the same as for SSD usage. To prevent wear out of certain flash areas only one partition over the complete Flash memory shall be used. The partition shall be aligned to the Flash sectors. I don't know how to find out the right alignment for Flash sticks. Moreover the Flash controller in the stick must have a good wear leveling function to prevent wear out of certain flash regions. Because just one partition is available a swap file must be used.
I am not sure if TRIM is supported with USB sticks. If not invalid flash pages are deleted with a garbage collection function of the flash stick.

Many problems with TRIM, garbage collection and wear leveling occured with certain SSDs in the past. Thay were sometimes resolved by firmware updates. Therefore we can expect that there such problems also occure with USB sticks. But I didn't see any firmware update for USB sticks.

- shv

Any personal experience on how long a life a USB stick lasts? I think mine is over a year or two already.

I have a couple Mushkin Mullholland 8GB USB sticks I've been using for two years without fail. I haven't babied them. I haven't bothered with having syslog write to RAM. I have SWAP on the stick. I don't even think I've totally disabled atime. What I did do though, is only use 50% of the storage space.

Is there a practical way to watch for failing drives? Like SMART for flash? The wear leveling is entirely internal to the flash drive, but doesn't it report "bad" or "unusable" sectors to the OS? Is there a good tool for Linux to spot the count going up to preemptively change the drive? One thing that comes to mind in terms of partitioning swap and maybe all of /var/log (if a HDD isn't an option) is to just put a second small cheap flash drive in (we all have a dozen or so of those 1gb or less with no use anymore, amiright) and let it fail; when it fails just toss a new one in. But, if there is no good way to tell when the drive is failing it might be just as troubling to have the unit hang even if the main data is safe.

There is an interessting discussion about flash memory wear out of SD cards with RaspberryPi: Link.. They recommend to use the fstrim command so that unused blocks become available for wear leveling.

It seams that there might be USB3.0 sticks available which support trim: Link. With the tool TRIMcheck it can be tested if a drive supports trim: Link. If you have a Windows PC where you know that trim is working e.g. with SSD you could try to test USB stick as well.

The problem is not just wearing out of USB flash memory drives but also slowing down write operations if each write needs an erase before.



Edited 1 time(s). Last edit at 10/20/2013 02:09AM by shv.

I to have thought a bit about this topic, and recently I fell over this small script:

http://www.nohl.eu/tech-resources/notes-to-linux/log-files-in-tmpfs-without-breaking-logging/

The last post is for debian. Maybe Bodhi could implement this in future rootfs builds?

Regards

joebob2000 Wrote:
-------------------------------------------------------
> Is there a practical way to watch for failing drives?
> Is there a good tool for Linux to spot the count going up to preemptively change the drive?

Not exactly, but unlike SMART,
to test the (remaining) free space both initially and occasionally,
there is http://oss.digirati.com.br/f3/
as a native (unixoid) remake
of Heise.de's H2test(w) for (Win)DOS.

Over the past decade at least, I have committed no USB or SD to "production" here without a full test cycle under one of the above, and (besides forgeries) caught many that were faulty fresh-from-factory.



Edited 3 time(s). Last edit at 01/04/2017 02:23PM by TEN.

Isn't it a good idea to set vm.swappiness to 0 so that swap is used only when it is absolutely necessary, instead of being used as a page cache?

Is zswap already enabled by default or do we need to do anything?

admin@pogoplug:~$ cat /proc/cmdline
console=ttyS0,115200 root=LABEL=rootfs rootdelay=10 mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
admin@pogoplug:~$ dmesg | grep zswap
[    9.886230] zswap: loaded using pool lzo/zbud
admin@pogoplug:~$ cat /sys/module/zswap/parameters/enabled
N

haridsv,

> Isn't it a good idea to set vm.swappiness to 0 so
> that swap is used only when it is absolutely neces
> sary, instead of being used as a page cache?

Not really. If you use a desktop on your plug such as XFCE, LDE, KDE, ...

Quote
http://forum.doozan.com/read.php?2,23630
Debian Desktop

XFCE desktop Pogoplug Debian

then swapiness should be set to 0-10 to help the UI more responsive. But for a server, that would make the response time for other apps worse when you are close to running out of RAM. Servers should be balance in swapping to keep it running at a consistent rate.

zswap compresses the swap in memory before needing to swap to disk.

>
> Is zswap already enabled by default or do we need
> to do anything?

You need to activate it in the kernel bootargs:

zswap.enabled=1

In the new u-boot default envs there is a custom_params varaible already at the end of bootargs. So you can set it directly, or use the uEnv.txt to activate it.

In Debian:

fw_setenv custom_params 'zswap.enabled=1'

Or in uEnv.txt

custom_params=zswap.enabled=1

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> zswap compresses the swap in memory before needing
> to swap to disk.

Is zswap recommended only when using a flash storage for swap, or is it also useful when swap is setup on a hard drive? Also, does enabling zswap put a significant toll on the CPU that is in Pogoplug E02?

> Is zswap recommended only when using a flash stora
> ge for swap, or is it also useful when swap is set
> up on a hard drive? Also, does enabling zswap put
> a significant toll on the CPU that is in Pogoplug
> E02?

It is mainly for the flash drives to avoid swapping to flash. However, it could be useful for hard drives also, if swapping is small and infrequent. IOW, if there are little swapping, it will stay in compressed space in RAM. But it is a tradeoff between RAM and CPU power. You'll be the judge.

In servers operations, zswap is not recommended if the swap space is in HDD (there are signifcant swapping going on with lots of daemon running that use lots of RAM).

zswap does not slow down CPU much at all. For plugs that has no SATA interface such as Pogo E02, zswap is really helpful (even with USB HDD attached, because of the inherent disk thrashing of USB protocol). For plugs such as GoFlex Net, with swap file on the HDD, I don't use zswap because SATA is fast and buffered.

BTW, I have most of the Kirkwood boxes in my farm (about 20, with 10-12 always running), so these are real world examples.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------

> It is mainly for the flash drives to avoid swappin
> g to flash. However, it could be useful for hard d
> rives also, if swapping is small and infrequent. I
> OW, if there are little swapping, it will stay in
> compressed space in RAM. But it is a tradeoff betw
> een RAM and CPU power. You'll be the judge.
>
> In servers operations, zswap is not recommended if
> the swap space is in HDD (there are signifcant swa
> pping going on with lots of daemon running that us
> e lots of RAM).
>
> zswap does not slow down CPU much at all. For plug
> s that has no SATA interface such as Pogo E02, zsw
> ap is really helpful (even with USB HDD attached,
> because of the inherent disk thrashing of USB prot
> ocol). For plugs such as GoFlex Net, with swap fil
> e on the HDD, I don't use zswap because SATA is fa
> st and buffered.
>
> BTW, I have most of the Kirkwood boxes in my farm
> (about 20, with 10-12 always running), so these ar
> e real world examples.

Thank you, appreciate the details. I am convinced enough to enable it.

How many of these recommended tweaks are already enabled in your debian rootfs?

bennytheben Wrote:
-------------------------------------------------------
> How many of these recommended tweaks are already
> enabled in your debian rootfs?

None of them are. The rootfs is a very basic rootfs. It meant to be Linux learning vehicle, you have to install things you need.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

wow, an oldie but goody thread, was actually here searching for info on how to reduce flash writes again ...

I noticed Armbian have adopted zram for helping reduce swapping by compressing memory for a swap space, and log2ram for logging to memory, with configurability of when logs are flushed out to disk/flash ... Unfortunately, neither is in a standalone package, they are built into Armbian ... however ...

Ubuntu universe has zram_config package (0.5), which source can be downloaded, compiled and create a .deb for Debian system. Then just enable the systemctl, and start it. Can confirm that it seems to work ok. (Official Debian site has a link to zram_config on Ubuntu for systemd Debian installs.)

log2ram looks pretty easy to install, but hasn't been adopted as a package yet in Debian or Ubuntu. So Github - https://github.com/azlux/log2ram I believe this is the source of the built in log2ram in Armbian. The creator has created install /uninstall script which helps, and documentation he has on Github is OK.

I'm just about to install log2ram, so don't know of any issues yet.

Also, given limited memory on some of these ARM boards, I think either (or both) solution should be carefully tweaked and tested before "putting into production". In particular, the amount of memory allocated to either zram or log2ram.

Anyway, hopefully above helpful ...

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com

Don,

The reason I did not choose zram was because it takes up a fix amount of RAM. These plugs typically has 128MB or 256MB. Armbian or Ubuntu usually installed on systems that have a lot for RAM, typically 1GB.

I chose zswap becasue it uses memory as it required, so it helps the 128MB RAM plugs.

>
> log2ram looks pretty easy to install, but hasn't
> been adopted as a package yet in Debian or Ubuntu.
> So Github - https://github.com/azlux/log2ram I
> believe this is the source of the built in log2ram
> in Armbian. The creator has created install
> /uninstall script which helps, and documentation
> he has on Github is OK.
>
> I'm just about to install log2ram, so don't know
> of any issues yet.
>

I use Debian busybox syslogd to log /var/log/messages to RAM. This is preinstalled on the basic rootfs I released. I have not explored log2ram so don't know the advantage over syslogd.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 06/09/2018 05:13PM by bodhi.

Dear bodhi,

Makes perfect absolute sense, thanks for clarity :)

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com

Hello Everyone,

It's been a while! These days I had to replace 2 out of 3 usb flash drives. It took me a while to get up to speed with everything.

I had what they claim to be SLC usb flash drives but I haven't seen any difference. The drives were Mach Extreme (MX-LX and MX-ES) expensive but no more reliable than others. Pity.

Looked up on the internet for '5 or 10 years warranty usb drives' and put a reminder on calendar around 2023 to replace them if don't fail earlier. I found Duracell Memory Professional or Sandisk Extreme drives family, so for 5 years Duracell is just cheaper. I shall update when they fail.

It is a pain to replace everything but now at least all is up-to-date. I use 2.5inch WD RED as hdds in RAID config with no spin-down for 4-5 years almost and happy about this.

I shall look into using Debian busybox syslogd to log /var/log/messages to RAM as Bodhi have said for one of the GFNs and see any difference in usb drive's lifetime and update here.

Kindest Regards,

Hi,

I've recently changed my pogoE02 to a HP T5325Z and I've also started working on getting a Dell Wyse T50 to run Debian. Both the HP T5325Z and T50 have relatively large internal memory which is not being used anymore now that the operating system runs from the USB stick. Would it be OK to use this internal memory as swap space or does this suffer from wear like the USB drive?

Koen

Hi Koen,


> Would it be OK to use
> this internal memory as swap space or does this
> suffer from wear like the USB drive?

Yes it is the same since it is SSD.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

I've thought, as an alternative, have a backup script which runs periodically to save out your USB stick's files to another location (or an HDD if you're hosting storage off your Pogo v4). The major issue would be how would you create a new USB to replace one that is dead/dying? It seems you can only create the configurations on the device they would be installed in. I would think there would be some way to format/build one on your standard Linux desktop/laptop instead, and swap it in (cloning might be obvious, but by the time you're replacing the USB, the capacity will be double of the old one).

This has been a particular concern of mine, as I had a Pogo I had converted to Linux, and when the USB died (it lost power too many times and got corrupted) I had trouble getting it back up & running (I might never have gotten it fixed, I don't remember). It isn't like your standard desktop/laptop/server where you can stick in a bootable DVD and reinstall.

jelabarre59 Wrote:
-------------------------------------------------------
> I've thought, as an alternative, have a backup
> script which runs periodically to save out your
> USB stick's files to another location (or an HDD
> if you're hosting storage off your Pogo v4). The
> major issue would be how would you create a new
> USB to replace one that is dead/dying? It seems
> you can only create the configurations on the
> device they would be installed in. I would think
> there would be some way to format/build one on
> your standard Linux desktop/laptop instead, and
> swap it in (cloning might be obvious, but by the
> time you're replacing the USB, the capacity will
> be double of the old one).
>
> This has been a particular concern of mine, as I
> had a Pogo I had converted to Linux, and when the
> USB died (it lost power too many times and got
> corrupted) I had trouble getting it back up &
> running (I might never have gotten it fixed, I
> don't remember). It isn't like your standard
> desktop/laptop/server where you can stick in a
> bootable DVD and reinstall.

I am running USB rootfs on many plugs. Basically:

1. Take it offline, backup once to a tarball.
2. Run rsync daily to back up the live rootfs to a network folder.

When you need to recreate the rootfs, untar the tarball to a formatted USB, and rsync the backup folder to this new USB.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)