Getting "reading directory .: Input/output error"

Greetings

I have had a Dockstar running Debian squeeze since last September. It runs rtorrent 24/7, and has worked well so far. It has a 1 TB usb hard disk with ext3 file system hooked to it.

Yesterday, I got this error when trying to do ls on a directory on that usb hard disk:

$ ls
ls: reading directory .: Input/output error
$

Huh? That sounds like a possible hardware failure, but the hard disk is only a few months old. I shut the Dockstar down, hooked the usb hard disk to another linux box, and ran fsck on it. fsck found no problems, and so I restarted the Dockstar, hooked the usb hard disk back up, and it works fine now.

Any ideas what might have happened here? This is unexpected, flaky, "un-Debian-like" behavior, and has never happened before on any of my various linux machines.

Run smartctl from the linux box and see what SMART says, ie
http://en.wikipedia.org/wiki/S.M.A.R.T.

> Run smartctl from the linux box and see what SMART
> says, ie
> http://en.wikipedia.org/wiki/S.M.A.R.T.


Thank you for your reply :)

I installed smartmontools. I ran smartctl on the drive and got this:

$ sudo smartctl -i /dev/sdc
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: WDC WD10 EVDS-63U8B0      Version: 01.0
Serial number:      WD-WCAV5F916494
Device type: disk
Local Time is: Tue Jun  7 16:28:19 2011 CDT
Device does not support SMART
$

This is a several-month-old 1 TB Western Digital SATA drive mounted in an Icydock usb 2.0 enclosure. Hmmm...

Anyway, after working fine all day, I came home and tried to mount the drive on my Mac with nfs. Like yesterday, I did the mount by double-clicking an alias to the drive. That's when it failed again. This has always worked before.

I rebooted the Dockstar and it's OK again. I can mount the drive on the Mac with cmd-K and it works fine.

Here's what showed up in dmesg when it failed. The first line is from the end of boot.

[  114.998173] EXT3-fs: mounted filesystem with ordered data mode.
[77163.494929] usb 1-1.2: USB disconnect, address 4
[77163.815158] usb 1-1.2: new high speed USB device using orion-ehci and address 6
[77163.928740] usb 1-1.2: New USB device found, idVendor=0928, idProduct=0003
[77163.935672] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[77163.943051] usb 1-1.2: Product: USB 2.0 Storage Device
[77163.948228] usb 1-1.2: Manufacturer: CREMAX TECH. CO., LTD 
[77163.953823] usb 1-1.2: SerialNumber: 00000000000231″8
[77163.961029] usb 1-1.2: configuration #1 chosen from 1 choice
[77163.968870] scsi3 : SCSI emulation for USB Mass Storage devices
[77163.975240] usb-storage: device found at 6
[77163.975248] usb-storage: waiting for device to settle before scanning
[77164.935019] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #23633927 offset 0
[77165.625194] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #25870402 offset 0
[77165.635817] ------------[ cut here ]------------
[77165.640481] WARNING: at /build/buildd-linux-2.6_2.6.32-20-armel-qpwOHl/linux-2.6-2.6.32/debian/build/source_armel_none/fs/buffer.c:1160 mark_buffer_dirty+0x34/0xec()
[77165.655357] Modules linked in: ext3 jbd nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipv6 hmac sha1_generic mv_cesa aes_generic ext2 mbcache sd_mod crc_t10dif usb_storage scsi_mod ehci_hcd mvsdio usbcore mv643xx_eth nls_base mmc_core libphy inet_lro
[77165.679060] [<c002dee4>] (unwind_backtrace+0x0/0xdc) from [<c0045120>] (warn_slowpath_common+0x4c/0x80)
[77165.688526] [<c0045120>] (warn_slowpath_common+0x4c/0x80) from [<c00eeca4>] (mark_buffer_dirty+0x34/0xec)
[77165.698203] [<c00eeca4>] (mark_buffer_dirty+0x34/0xec) from [<bf2b7cc4>] (ext3_commit_super+0x58/0x7c [ext3])
[77165.708242] [<bf2b7cc4>] (ext3_commit_super+0x58/0x7c [ext3]) from [<bf2b9638>] (ext3_handle_error+0x98/0xc4 [ext3])
[77165.718987] [<bf2b9638>] (ext3_handle_error+0x98/0xc4 [ext3]) from [<bf2b9710>] (ext3_error+0x38/0x4c [ext3])
[77165.729012] [<bf2b9710>] (ext3_error+0x38/0x4c [ext3]) from [<bf2b58cc>] (ext3_find_entry+0x390/0x548 [ext3])
[77165.739036] [<bf2b58cc>] (ext3_find_entry+0x390/0x548 [ext3]) from [<bf2b62f4>] (ext3_lookup+0x2c/0xe0 [ext3])
[77165.749134] [<bf2b62f4>] (ext3_lookup+0x2c/0xe0 [ext3]) from [<c00d5408>] (do_lookup+0xc0/0x188)
[77165.757987] [<c00d5408>] (do_lookup+0xc0/0x188) from [<c00d792c>] (__link_path_walk+0x954/0xe38)
[77165.766836] [<c00d792c>] (__link_path_walk+0x954/0xe38) from [<c00d7fac>] (path_walk+0x48/0x94)
[77165.775599] [<c00d7fac>] (path_walk+0x48/0x94) from [<c00d8094>] (do_path_lookup+0x24/0x4c)
[77165.784003] [<c00d8094>] (do_path_lookup+0x24/0x4c) from [<c00d8a60>] (user_path_at+0x54/0x94)
[77165.792672] [<c00d8a60>] (user_path_at+0x54/0x94) from [<c00d0dcc>] (vfs_fstatat+0x2c/0x5c)
[77165.801076] [<c00d0dcc>] (vfs_fstatat+0x2c/0x5c) from [<c00d0ed4>] (sys_stat64+0x14/0x34)
[77165.809309] [<c00d0ed4>] (sys_stat64+0x14/0x34) from [<c0027ea0>] (ret_fast_syscall+0x0/0x28)
[77165.817982] ---[ end trace 2964308c2aea306c ]---
[77166.010681] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #28524561 offset 0
[77166.133818] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #28688385 offset 0

[snip]

[77166.356141] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #28778521 offset 0
[77166.366529] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #28778521 offset 0
[77168.984588] usb-storage: device scan complete
[77168.985462] scsi 3:0:0:0: Direct-Access     WDC WD10 EVDS-63U8B0      01.0 PQ: 0 ANSI: 4
[77169.009522] sd 3:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[77169.035159] sd 3:0:0:0: [sdd] Write Protect is off
[77169.040088] sd 3:0:0:0: [sdd] Mode Sense: 10 00 00 00
[77169.040099] sd 3:0:0:0: [sdd] Assuming drive cache: write through
[77169.066377] sd 3:0:0:0: [sdd] Assuming drive cache: write through
[77169.072510]  sdd: sdd1
[77169.099141] sd 3:0:0:0: [sdd] Assuming drive cache: write through
[77169.105355] sd 3:0:0:0: [sdd] Attached SCSI disk
[77404.472407] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0
[77404.485098] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0

[snip]

[77404.767846] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0
[77410.258224] EXT3-fs error (device sdb1): ext3_find_entry: reading directory #23633927 offset 0

I'm not sure what any of this means. Anyway, I guess it works for now, but I may end up moving the rtorrent task to a more reliable machine :)



Edited 1 time(s). Last edit at 06/07/2011 05:13PM by hanker.

Yeah, I've discovered the same problem with smartctl and external USB harddisks. I don't think one can get SMART info from a disk attached via USB, but that is just from my own experiences and can't prove that.

Would your external hard drive be based on the Western Digital Caviar Green (or Black) models of hard disks? I did a quick google of the model number from your dmesg output and found that it does share some qualities with my internal SATA WD Caviar Green ... "Advanced Formatting". This is a new low-level format that drive manufactures are trying to get to catch on. It's supposedly how we're now able to obtain terrabyte disk sizes now. Google around, and you'll find that if your partitions aren't properly set up, you can experience some serious performance problems in linux partitions. Most 'latest-and-greatest' kernels and GNU disk packages now account for the new 4096 block size properly, as I would hope squeeze would be able to do, so you may be unaffected from that.

So, another thing common with our hard disks, and probably what's causing problems, is WD has some 'sssuuper-secret' variable spindle speed technology (notice you can't find a spec that shows the spindle RPM? that's because it's variable). I propose that when these disks spin down or slow down during a power-save event, when it comes time to wake back up they don't do it properly or fast enough. The kernel can't figure it out, doesn't get a response from the device, and wammo..IO errors. But since the disk was properly sync'd with disk cache and journals are in tact because everything was parked properly, your data is fine (thus no problems found upon a fsck).

I've only done light research on this. My problem is when I come out of suspend, my single 1TB partition becomes 'device unavailable', and attempts to remount fail, and I get the same problems you're seeing. But I did piece together that it only happens after the disk has been told to spin down. I'm going to try disabling ACPI and let it sit idle for a few days, and see if it wakes up properly.

Anyone have any insight?

(PS a 3rd thing I found about these disks is that every 8 seconds, no mater if the disk is idle or busy, it re-parks the heads, then continues on it's way. I don't know WTF for, but there is firmware available from WD that will disable that 'feature'.)

Alec Wrote:
-------------------------------------------------------
> Yeah, I've discovered the same problem with
> smartctl and external USB harddisks. I don't think
> one can get SMART info from a disk attached via
> USB, but that is just from my own experiences and
> can't prove that.



I think there's a way to do it using an option on the command-line tool, but I don't know the details. This problem only cropped up when I would nfs-mount the drive on my Mac by double-clicking an alias on the Mac. Solution: don't do that. Use cmd-K, then nfs://[ip=address]/shared_dir

No problems since last summer.

BTW, I removed the drive from the enclosure and hooked it up by SATA in my xubuntu desktop. I ran gsmartmon on it there and it tested fine.

Hi

Try smartctl with '-t sat'

It works for my 500GB Western Digital USB drive on my pogoplug.

root@debian:~# smartctl -i -H -d sat /dev/sda
smartctl 5.40 2010-07-12 r3124 [armv5tel-unknown-linux-gnueabi] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD5000BMVV-11GNWS0
Serial Number:    WD-WXN1A6068509
Firmware Version: 01.01A01
User Capacity:    500,107,862,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Fri Dec 23 10:28:53 2011 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

whoelse Wrote:
-------------------------------------------------------
> Hi
>
> Try smartctl with '-t sat'
>
> It works for my 500GB Western Digital USB drive on
> my pogoplug.
>
>
> root@debian:~# smartctl -i -H -d sat /dev/sda


Thank you for your reply.

I tried this on one of my Dockstar's external usb hard disks - a WD 1 TB SATA drive in an IcyDock usb enclosure. I got this:

# smartctl -i -H -d sat /dev/sdc
smartctl 5.40 2010-07-12 r3124 [armv5tel-unknown-linux-gnueabi] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Well basically it means that the linux is trying to sort of read the index map. Take a look at the GUI version of the disk utility tool. If you click on your drive you should see a rotating phasor symbol. That means that linux is trying to index. Leave it for a few minutes and it should work. NOTE: for my 8GB sdcard it took about 2 minutes so scale the time in your head.

to use smartctl with usb external hdd, try
$ sudo smartctl -a -d usbcypress /dev/your_ext_hdd
$ sudo smartctl -a -d usbjmicron /dev/your_ext_hdd
$ sudo smartctl -a -d usbsunplus /dev/your_ext_hdd
$ sudo smartctl -a -d usbmarvell /dev/your_ext_hdd
one of them worked for me

else
$ sudo smartctl -h
and look up the -d switch for its options

Please remember that USB normally does not support the smart commands, I have noticed that one HD does not support smart when it is connected via USB, but when directly attached with SATA, it of course does (as the model is clearly smart compliant)

> BTW, I removed the drive from the enclosure and ho
> oked it up by SATA in my xubuntu desktop. I ran gs
> martmon on it there and it tested fine.

If you have smartctl access with your xubuntu machine, with direct attached SATA, then

smartctl can do short and long tests on the drive, might be worth running those, eg :

smartctl --test=long /dev/sda
smartctl --test=short /dev/sda

# then monitor the running test with

smartctl -a /dev/sda

Prior to the short or long tests - check the "SMART Attributes", listed by 'smartctl -a /dev/sda', they should be 'normal'. In particular, look out for "Current_Pending_Sector", "Offline_Uncorrectable", "Reallocated_Sector_Ct" being non zero, (and I think there are some others too, google would know.)

If the drive passes smart tests and has no iffy SMART Attributes, (and it's not making 'nasty' or even odd noises) then there's a good chance it's not the drive hardware.

Then I'd move on to the USB cradle and it's cables, can you swap out for another USB cradle and cable ?, see if that resolves the problem.

Then there's the possibility of dirty/oxidised contacts on SATA connectors, USB connectors. Electrical contact cleaner may be able to be used to clean contacts.

Then I guess there's also a possibility of faulty Dockstar hardware (which are a long discontinued product now ?) which perhaps has failed or has intermittent fault. Usually troubleshooting that would be a specialised task, but might be worth a look at the circuit board, see if there's any obvious problems, like sockets that have become loose with wear, dodgy solder joints or components that have obviously failed. Or even ingress of dust etc ...

Then there's the power supplies to the ICY dock and the dockstar. Simple tests can be done on PSUs at no load, by using a multimeter to check the voltage of the supplies, which will be labelled on the supplies. Little more complicated to measure voltage at load, but that would be helpful measurement too, as PSUs don't always show fault at no load. My GFN for instance has a 12V supply, so I'd expect to see maybe 11.5v to 12.5v coming out of that (ideally as close to 12v as possible) and for the measurement to be stable (not wondering voltage for instance)

Heat/poor ventilation can also cause problems, so is there anything getting hot ? and is the equipment and PSUs in a well ventilated position ?

Software wise, the only thing that came to mind is trying ext4 over ext3, but given your 2.6 kernel, and my lack of detailed knowledge of the differences between ext3 and ext4, I don't know whether that's a sensible solution.

Hope that's helpful

BR

Don Charisma ... because anything is possible with Charisma

My blog - http://DonCharisma.org
Our commercial site - http://DonCharisma.com