Improving disk performance?

Greetings

I have a Dockstar running Debian Squeeze. (Thanks, Jeff!)

It's used for torrenting and as a NAS, and it works great for the most part.

I have two 1 TB SATA drives hooked to it. These drives are copies of one another: they are rsynced once a day. (The drives contain music and video files.) I originally duplicated one drive to the other using rsync on a Dell Mini 9 netbook (1.6 GHz Atom cpu, Ubuntu 9.10) with a two-bay USB drive dock. That rsync operation proceeded at about 7 to 9 MB/s.

But now that the drives are hooked to the Dockstar, rsync proceeds much slower: about 1 - 2 MB/s. (The drives are in two Icydock USB enclosures.) This is usable since there will be a maximum of about 3 GB/day new content to be rsynced, but why is the disk performance so poor? I haven't run any tests, but it seems that the Debian Squeeze Sheevaplug that the Dockstar replaces had better disk performance than this.

Insights? :)

Hi

I had the same problem after I format the drive with ext3 the speed increased up to 20 MB/s

Christian

wambo Wrote:
-------------------------------------------------------
> Hi
>
> I had the same problem after I format the drive
> with ext3 the speed increased up to 20 MB/s


Both of my drives are ext3. :\

Measure the raw read performance, bypassing any encryption and file system overhead:

dd if=/dev/sdx of=/dev/null bs=100M count=10

This command reads the first gigabyte of /dev/sdx (where sdx is sda, sdb, sdc,... whatever your hard disk is called) and reports the read speed when it's done.

Johannes Wrote:
-------------------------------------------------------
> Measure the raw read performance, bypassing any
> encryption and file system overhead:
>
> dd if=/dev/sdx of=/dev/null bs=100M count=10
>
> This command reads the first gigabyte of /dev/sdx
> (where sdx is sda, sdb, sdc,... whatever your hard
> disk is called) and reports the read speed when
> it's done.

I tested sdc. The dd command did something unexpected: it made it look like the Dockstar had crashed. It was unresponsive for about 20 minutes. (I didn't have physical access to it, I was sshed in, and the ssh terminals became unresponsive.)

Here's the output:

----8<----
# dd if=/dev/sdc of=/dev/null bs=100M count=10
6+0 records in
5+0 records out
524288000 bytes (524 MB) copied, 1203.55 s, 436 kB/s

#
----8<----

436 kB/s. Hmmm... And it only copied 524 MB, but I did ctrl-C about 5 minutes after entering that command.

I did a much smaller version of the dd command

# dd if=/dev/sdc of=/dev/null bs=10M count=1

and got a speed of about 5 MB/s.

Anyway, I also copied (using cp) a large file from one disk to the other, and the copy speed came out to about 9 MB/s, which is comparable to what I was getting using rsync on the Ubuntu netbook. So it looks like maybe rsync is the bottleneck here.

Hello,

where's your mountpoint located?
Is it on the NAND or on a flash drive? I read that this can be a bottleneck. Better use a ramdisc for the mountpoint.
(havn't tested it. I just read it in a tutorial)

Greetings,
Hendrik

Sorry, I had the block size and block count reversed. Should not try to read 100MB chunks on a system with only 128MB RAM. The reason for reading such a large total amount of data is to avoid interference from the disk cache.

5MB is very low raw read performance. My Dockstar gets about 30MB/s in that test, from a 2.5" hard disk. 30MB/s is about the maximum one could hope for on a USB2.0 bus.

Are other devices connected to the bus which could slow it down? Are the disks correctly recognized as high speed devices (not 12mbit/s full speed "USB 1.1")?

Hendrik wrote:

> where's your mountpoint located?
> Is it on the NAND or on a flash drive? I read
> that this can be a bottleneck. Better use a
> ramdisc for the mountpoint.
> (havn't tested it. I just read it in a tutorial)


The mountpoints are on the flash drive, /media/usb_1 and /media/usb_2. How would I move the mountpoints to ramdisk? How big of a ramdisk is necessary? I ask because I'm running one app, rtorrent, with potentially a big RAM footprint. (Currently the box is running 40 MB RAM and 40 MB swap.) Can you point me to the tutorial you mentioned?


Johannes wrote:

> Sorry, I had the block size and block count
> reversed. Should not try to read 100MB chunks on a
> system with only 128MB RAM. The reason for reading
> such a large total amount of data is to avoid
> interference from the disk cache.


I figured something was wrong ;)

I reversed the numbers and did

dd if=/dev/sdc of=/dev/null bs=10M count=100

and got this:

----8<----
# dd if=/dev/sdc of=/dev/null bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 48.2083 s, 21.8 MB/s
#
----8<----

So 21.8 MB/s. But the fact remains that rsync copies run at 1 - 2 MB/s, so it would seem that that's the bottleneck, rather than the raw read speed, right? (I'm no linux expert.)


> 5MB is very low raw read performance. My Dockstar
> gets about 30MB/s in that test, from a 2.5" hard
> disk. 30MB/s is about the maximum one could hope
> for on a USB2.0 bus.
>
> Are other devices connected to the bus which could
> slow it down? Are the disks correctly recognized
> as high speed devices (not 12mbit/s full speed
> "USB 1.1")?


On USB there is the boot flash drive and then two 1 TB SATA hard disks in Icydock enclosures. So far as being recognized as USB 2.0, I don't know. How might I find out?

22MB/s without using cache is safely in high speed territory. Not phenomenally fast, but OK.

It appears to be a software problem. You can try changing the mountpoint. The /tmp directory is typically mounted as a ramdisk on embedded systems. Show the mounted file systems with "mount" (no parameters). "tmpfs" is a ramdisk.

You mentioned 40MB of used swap. Have you tried running rsync without other tasks competing for the CPU and, worse, memory? If the system has to use swap, everything slows to a crawl.

>It appears to be a software problem. You can try changing
>the mountpoint. The /tmp directory is typically mounted as a
>ramdisk on embedded systems. Show the mounted file systems
>with "mount" (no parameters). "tmpfs" is a ramdisk.


OK, so I get this as part of the output of mount:

tmpfs on /tmp type tmpfs (rw,relatime)

I guess I would do something like

# mkdir /tmp/usb_1
# mount /dev/sdc1 /tmp/usb_1

Is that right? And then I guess I would have to do that every time I boot? And how much RAM does that use? (I'm inexperienced with ramdisk in linux, to say the least. ;)

>You mentioned 40MB of used swap. Have you tried running
>rsync without other tasks competing for the CPU and, worse,
>memory? If the system has to use swap, everything slows to a
>crawl.

This system was set up to do only two things: to run a Bittorrent client (rtorrent) and be an nfs server. There are currently about 250 torrents running and music is streaming over the LAN 24/7, and the whole system is using 40 MB RAM and 39 MB of swap. But it's not thrashing and the system is responsive.

It's kind of time-consuming to quit and start rtorrent, so I can live with the slow rsync operation. It's no big deal. I guess it's just a function of running on a cramped system. ;)

Try option --whole-file to not use rsync's delta algorithm. It should improve performance if read/write is the bottleneck and not network transfer. It also might lower CPU power needed.

Hi,

doesn't rsync use an md4/md5 checksumming to compare files existing on both ends allready? That's CPU intensive workload and might have more impact than the pure disk IO.

Fritz

Fritz Wrote:
-------------------------------------------------------
> doesn't rsync use an md4/md5 checksumming to compare files existing on both ends allready?

No -- see "man rsync":

Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

Ok,

you got a point - seems that was changed somewhen. Still: http://www.samba.org/ftp/rsync/rsync.html

Quote
http://www.samba.org/ftp/rsync/rsync.html
Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.

Fritz

General FYI on performance, installed to a 8GB Flash drive got:

apt-get install hdparm
hdparm -tT /dev/sda

Timing buffered disk reads: 54 MB in 3.01 seconds = 17.93 MB/sec
Timing cached reads: 536 MB in 2.00 seconds = 267.97 MB/sec

I'm going to run my site off of it.

Just a thought... how much cache is the system using? Have you tried playing with the swapiness setting?

Here's a faq on it:
https://help.ubuntu.com/community/SwapFaq

Might be worth a test.

I'm getting about 22 MB/s raw read performance using dd. Also I get decent disk speed reading from and writing to to the Dockstar's hard disks over the network, so I've concluded that the relatively slow performance I'm seeing when using rsync between the two hard disks on the Dockstar is a function of using rsync. It's no big deal, and I'm just gonna leave well enough alone for now.

Thanks for the help, folks! :)