Howto: maximize USB drive life

I've had a USB stick die on me from evidently too many write cycles. This led me down the path of trying to figure out what tweaks we can do to give my thumb drives a chance of lasting a few years.

This thread: http://www.raspberrypi.org/phpBB3/viewtopic.php?f=29&t=8819&p=104096&hilit=cache_pressure#p104096

suggests this in /etc/sysctl.conf

# Economize the SSD
# Strongly discourage swapping (default 60) I've seen 10 as another "low write" option
vm.swappiness = 1

# Don't shrink the inode cache aggressively (default 100)
vm.vfs_cache_pressure = 50

# (default 499)
vm.dirty_writeback_centisecs = 1500

# (default 10)
vm.dirty_ratio = 20

# (default 5)
vm.dirty_background_ratio = 10

This package (laptop-mode) looks promising too:

http://samwel.tk/laptop_mode/faq

And finally an old NSLU2 article on tweaking config files for Etch, some new things in there:

http://www.cyrius.com/debian/nslu2/linux-on-flash/

not really sure what to use and what is depreciated. I really like the log file reductions.

Anyone have any additional tips or thoughts on the above?



Edited 1 time(s). Last edit at 02/13/2014 03:17PM by ubermacin.

The approach I took is really simple:

1. Use busbox-syslogd to log all system events to RAM
2. Move all frequently writes to /tmp. Set all system daemons output to log files in /tmp (i.e. RAM).
3. Use command iotop -o -b -qqq to log all disk writes over period of time to make sure that all the logging are actually going to RAM.
4. If using swap, allocate swap file to hard disk. If there is no hard disk, use kernel feature like zswap.

IMO, vm.swappiness = 1 don't really help in this aspect. Its actual purpose is to make the system more responsive to interactive user (i.e. GUI). For a NAS, it is actually counter productive to do this when swapping does occurs.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 4 time(s). Last edit at 02/13/2014 04:04PM by bodhi.

One interesting point to note: at least some of these devices (e.g. the Pogoplug V4) have SLC NAND: the part in the Pogo4 is a Hynix HY27UF081G2M, which offers an endurance of 100,000 program/erase cycles, which is much more than almost all commercial USB sticks.

So if you had, say, the 112MB of filesystem on a Pogo4 and chose to use that with a UBIFS filesystem, you would have a device to which you could write a total of over 11TB, which is (technically speaking) a lot.

Another trick is to use a small ext2/ext3 filesystem for (say) the boot files, and then format the rest of the USB stick using something like F2FS. Get a huge stick, and allow F2FS to do it's "log structured file system" thing, which will spread the writes out across the whole filesystem and avoid overtaxing the (simplistic) "wear leveling" algorithms commodity sticks use.

(As an aside, most USB thumbdrives that have "wear levelling" -- and many don't -- use a simple "zone" scheme where a relatively small number of erasure blocks are grouped, and the device attempts to spread the wear across the blocks in the group. The group/zone size is based on the amount of RAM in the device for ease of management, so may be e.g. 1MB of user data + OOB + metadata. This can increase the overall life of the device dramatically, but doesn't help as much with repeated rewriting of the same logical address).

Could also play with f2fs or nilfs.

Hi

there is another simple solution,..

since I have two NSA320 and an iConnect - which among other services stream DVB-S2 TV- and RADIO Transponders in the WLAN for SONOS and Squeezbox devices all the time -

I took some usb 2.0 enclosures I had around and populated them with cheap SANDISK SSD 64GB or 128 GB,
no problems any longer on the iConnect for 2 years now,.. the NSA320 internal systemdisks are also SANDISK SSDs,..
(and the SANDISK ReadyCache 32GB is an even cheaper alternative now)

all my devices are in 24/7 mode running all the time, they have conventional swap, which is rarely used,..

since the days of the old nslu2 all the remedies I have tested (which are also mentioned above, ..) only led to a couple of worn out usb sticks, and at last the price/performance ratio of the SANDISK 128 GB SSD is at present better than corresponding usb sticks of that size,..

with iotop -o -b -qqq aside from [kjournald], [rsyslogd] and [kworker/u2:2]?? there were also periodic updates of epg data
by the vdr instances which are running for streaming (these data are needed and have to be written to disk to update the epg-database regularily) and the exported filesystems gave periodic smbd -D and nmbd -D write activities which could not be circumvented in my case,..

best wishes pbg4

PBG4 emphasizes an excellent point: devices sold as SSDs are much much better at wear management than devices sold as USB sticks. To which I would add that even if you find a USB stick that works well, the probability is extremely high that just when you get comfortable with it, the manufacturer will change the firmware/controller/flash vendor, and new versions will behave differently (and usually not for the better).

(I made a presentation to a well known North American Navy on this very topic, and successfully got them to abandon USB sticks as their preferred "mission planning" media, in favor of an SLC mSATA SSD in a USB 3.0 package).

In order of preference:

• SLC SSDs
• eMLC / MLC-HET SSDs
• SLC flash cards (CompactFlash, Secure DIgital)
• MLC SSDs
• MLC flash cards
• USB sticks

(Yes, obviously there are some examples of any type that are better -- or worse -- than that list would suggest. But it's a good starting point).

Another point is that "overprovisioning" can make a huge difference, but only if the flash controller is smart enough to take advantage of it. So getting an MLC SSD and only partitioning ~60-75% of the space will massively increase the operating life of the device... but this trick doesn't work with dumb controllers, which leaves out most sticks.

But using a log-structured file systems (like F2FS as I mentioned, plus NILFS with which I am less familiar) will ensure that the logical blocks being written are constantly changing, which in turn means that you can "overprovision" at the filesystem level -- F2FS has a "reserved percentage" flag, which I tend to set at anything up to 50%!

Thank you all, this is awesome information! I didn't know about the different variants of flash memory. I figured the only difference between SD memory and USB stick memory was just the form factor. Would this hold true for micro sd too? I know SSDs are coming down in price, but I'm trying to stay under $40 for the pogo plug setup in it's entirety. I think I'm going to check out F2FS for the log files etc and zswap. I'm debating if I need a swap or not. I'll report back here for others to follow along what I ended up doing. Thanks again to everyone!