Permanent file system errors

Hi there,
I've permanently problems with the filesystem on my pogoplug. I installed debian with Jeff's script on 3 different usb sticks. Everyone seems to work properly. If I run fsck -n /dev/sda1 I get errors like this:

root@Camerlengo:~# e2fsck -n /dev/sda1
e2fsck 1.41.12 (17-May-2010)
Warning!  /dev/sda1 is mounted.
/dev/sda1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 170698, i_blocks is 26360, should be 26344.  Fix? no

Pass 2: Checking directory structure
Entry 'cdn.debian.net_debian_dists_squeeze_Release' in /var/lib/apt/lists (170692) has deleted/unused inode 170749.  Clear? no

Entry 'cdn.debian.net_debian_dists_squeeze_Release.gpg' in /var/lib/apt/lists (170692) has deleted/unused inode 170700.  Clear? no

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(702888--702911)
Fix? no

Free blocks count wrong (792005, counted=791986).
Fix? no

Inode bitmap differences:  -170815 -170819
Fix? no


/dev/sda1: ********** WARNING: Filesystem still has errors **********

/dev/sda1: 9802/219456 files (0.6% non-contiguous), 85537/877542 blocks

Ok, so I took the USB stick put in my other linux box, fixed the errors and put it back to the pogoplug. fsck says now: no errors. But if I do anything on the pogoplug, like installing some packages or updating anything I get every single time new errors. It's indepent of the USB port I use or the type of usb stick I use or the filesystem I use (checked ext 2 and 3).

I checked another linux distri on the box (plugapps), but the result was the same.

Has anybody a hint for me?? I'm pretty desperate now!

regards
Papst

PS: And big thanks to Jeff for this amazing work here!

Well, first, active file systems are often transiently inconsistent, plus they also contain things like unreferenced files that will cause fsck to complain, but if you shut down your system sanely and the file system is still screwed up, then you have a problem.

What does your /etc/fstab say? I had a similar problem on my SheevaPlug running Ubuntu 9.04 and found a solution in the following thread:

http://plugcomputer.org/plugforum/index.php?topic=4587.msg16676#msg16676

(This is, I think the 3rd time I have posted this link here in as many days.)

Frankly, I have not observed this problem with Jeff's Debian load, but it's at least something worth checking out, given that the look and feel of your problem is quite similar.

Good luck.

The clue to your problem is in the line:
"Warning! /dev/sda1 is mounted."

Never, never, NEVER run fsck on a mounted filesystem. I can't believe this version of e2fsck doesn't have this warning if you try to do so:

/dev/sda1 is mounted.
WARNING!!!  Running e2fsck on a mounted filesystem may cause SEVERE filesystem damage.
Do you really want to continue (y/n)?

That's what I'm used to seeing. The wikipedia page for fsck says "Because running fsck to repair a file system which is mounted for read/write operations can potentially cause severe data corruption/loss, the file system is normally checked while unmounted, mounted read-only, or with the system in a special maintenance mode that limits the risk of such damage."

Also see this link: http://www.cyberciti.biz/faq/can-i-run-fsck-or-e2fsck-when-linux-file-system-is-mounted/

I don't think it is harmful to fsck a file system with the '-n' option; according to the manual, this causes the file system to be opened in read-only mode and no changes to be made. That's why the more pressing warning wasn't produced in this instance.

I've used 'fsck -n' to peek at active file systems for years and never had a problem. It can be useful, but you have to expect fsck to find phantom anomalies if the file system is active at the time it is being fscked.

Okay, that's good at least, that -n specifies read-only mode. I agree that it would almost definitely find problems in that case. Papst01, my answer is to stop running fsck -n on the mounted filesystem, as it won't say anything useful and is only scaring you into thinking there's a real problem when most likely there isn't. As restamp said, if you run your system for a while, shut down properly, take the USB disk to another system, run fsck there, and THEN have filesystem errors, then you've got a real problem.

Hi folks,

thank you for your answers. I'm still at work, so I cant get a look on fstab now. Later I post the fstab infos.

@restamp: I tried to google the problem but I didn't found a good hint. So thanks for the link. I'll check it out!

Yesterday in the evening I tried to update the system with "apt-get update". That ran without any noticeable errors. I was pretty sure that this is going to rise up the errors on the file system, so I ran fsck (of course with "-n") Surprise: I saw a bunch of new errors :-(
I rebooted but the system doesn't came up. I took the stick into my other linux box and found there massive fsck-problems too! After repairing it I <edit start> was thinking that it <edit stop> could boot debian with this stick on the pogoplug. So I'm pretty sure that the errors are not only the "usually ones" you get if you run fsck with a mounted and used partition.

So long
Papst01



Edited 1 time(s). Last edit at 02/02/2011 02:24AM by Papst01.

Hi there again.

My apt-get update yesterday was not as successful as I thought. Even after repairing the file system - the stick didn't bot cause of bootimage failure or something like that - the netconsole said that. I was glad, that I took before this "incident" an image of the stick, so I could replace the bricked system in order to get a look to the fstab. And thats it:

 more /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/root      /               ext2    noatime,errors=remount-ro 0 1
/dev/sda2      none            swap    sw                0       0
tmpfs          /tmp            tmpfs   defaults          0       0

And here for information the fw_print. Maybe here is an error too:

ethact=egiga0
bootdelay=3
baudrate=115200
mainlineLinux=yes
console=ttyS0,115200
led_init=green blinking
led_exit=green off
led_error=orange blinking
mtdparts=mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)
mtdids=nand0=orion_nand
partition=nand0,2
stdin=serial
stdout=serial
stderr=serial
rescue_installed=0
rescue_set_bootargs=setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts $rescue_custom_params
rescue_bootcmd=if test $rescue_installed -eq 1; then run rescue_set_bootargs; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000; else run pogo_bootcmd; fi
pogo_bootcmd=if fsload uboot-original-mtd0.kwb; then go 0x800200; fi
force_rescue=0
force_rescue_bootcmd=if test $force_rescue -eq 1 || ext2load usb 0:1 0x1700000 /rescueme 1 || fatload usb 0:1 0x1700000 /rescueme.txt 1; then run rescue_bootcmd; fi
ubifs_mtd=3
ubifs_set_bootargs=setenv bootargs console=$console ubi.mtd=$ubifs_mtd root=ubi0:rootfs rootfstype=ubifs $mtdparts $ubifs_custom_params
ubifs_bootcmd=run ubifs_set_bootargs; if ubi part data && ubifsmount rootfs && ubifsload 0x800000 /boot/uImage && ubifsload 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; fi
usb_scan=usb_scan_done=0;for scan in $usb_scan_list; do run usb_scan_$scan; if test $usb_scan_done -eq 0 && ext2load usb $usb 0x800000 /boot/uImage 1; then usb_scan_done=1; echo "Found bootable drive on usb $usb"; setenv usb_device $usb; setenv usb_root /dev/$dev; fi; done
usb_scan_list=1 2 3 4
usb_scan_1=usb=0:1 dev=sda1
usb_scan_2=usb=1:1 dev=sdb1
usb_scan_3=usb=2:1 dev=sdc1
usb_scan_4=usb=3:1 dev=sdd1
usb_init=run usb_scan
usb_device=0:1
usb_root=/dev/sda1
usb_rootdelay=10
usb_set_bootargs=setenv bootargs console=$console root=$usb_root rootdelay=$usb_rootdelay rootfstype=$usb_rootfstype $mtdparts $usb_custom_params
usb_bootcmd=run usb_init; run usb_set_bootargs; run usb_boot
usb_boot=mw 0x800000 0 1; ext2load usb $usb_device 0x800000 /boot/uImage; if ext2load usb $usb_device 0x1100000 /boot/uInitrd; then bootm 0x800000 0x1100000; else bootm 0x800000; fi
bootcmd=usb start; run force_rescue_bootcmd; run ubifs_bootcmd; run usb_bootcmd; usb stop; run rescue_bootcmd; run pogo_bootcmd; reset
ethaddr=00:25:31:XX:XX:XX
arcNumber=2097
ipaddr=192.168.178.26
if_netconsole=ping $serverip
start_netconsole=setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;
preboot=run if_netconsole start_netconsole
usb_rootfstype=ext2
serverip=192.168.178.25

the macaddress I changed only here in this print ;-)

Hope for help
Papst



Edited 1 time(s). Last edit at 02/02/2011 02:25AM by Papst01.

Now you're beyond the file system errors problem you were having, so you may want to consider starting a new thread so others will know to look in there. :-)

You said after you ran apt-get update, you couldn't boot because of a boot image failure. My guess about what happened is your kernel was upgraded (which has been happening a lot lately), but your boot images in /boot weren't updated. Run this on the /boot directory (either from your Dockstar if you got it up, or from another machine and mount your Dockstar drive there): ls -ltr /boot

The last two items should be uImage and uInitrd. If they're not, then that's the problem, your kernel was updated but not the versions your Dockstar uses to boot.

Look at http://forum.doozan.com/read.php?2,524,2751#msg-2751 for the way that I deal with that problem.

Hi everyone,

I have news to report:
My problems don't come from the box itself. I've recently created a virtual box with a Debian linux runnning on it.
Following I tried with 3 different usb sticks (different vendors and sizes)
Creating partitions with ext2 and ext3. First I checked with fsck -f -y /dev/sdb1 - no error.
mounted the device with mount -t ext2 (respectivly ext3) /dev/sdb1 /mnt
Than i copied a bigger archive file to and run tar -xzvf file.tar.gz. Looked good. Dismounted /mnt and ran e2fsck /dev/sdb1 -> no error. But that was a f**ing lie! e2fsck -f /dev/sdb1 found massive problems. And as I said, it is independent from the usb stick I used and the filesystem! So the problem was deeper.
I decided "to zero" one of the sticks (4GB Transcent JetFlash 600) completly with dd if=/dev/zero of=/dev/sdb bs=1M

With fdisk I created then a primary disk 2048MB. Than a 256MB prima swap disk.
then mke2fs /dev/sdb1 than fsck -f -y /dev/sdb1 (no error)
mounted it, copied my test file Plugbox-Linux-2010.12-rootfs.tar.gz (from here http://plugapps.com/index.php5?title=PlugApps:Pogoplug_Setboot) than tar -xzvf Plugbox-Linux-2010.12-rootfs.tar.gz
after that unmount /mnt and a fsck -f -y /dev/sdb1 after that. And again many many errors. If I run fsck without "-f" the system claimes that the partition is clean.

I think a do some basic thing completely wrong. But what? I'm not a Linux crack, I'm just reading "mans", forums, learning by doing and thats all of my knowledge.
Is there a special way to prepare and handle usb sticks? Maybe bow to east, stick a needle in it and spell a curse against Microsoft or what?? I'm open to any suggestions!

Papst

Papst, I don't know what's causing your problems. However, to shed a bit of light on fsck...

Without the '-f' argument, the command will check to see if the 'clean' flag is set in the filesystem. This flag is set if the file system is in a consistent state -- i.e., if the file system is quiescent or has been cleanly dismounted. It indicates that there should be no corruption caused by partially written file system updates occurring, which can happen, say, if a disk update were occurring during a power fail situation.

If the clean flag is set, the most common causes of file system corruption are eliminated, and so Linux bypasses the fsck unless other conditions are present. Those other conditions are time and number of mounts, which are settable on a per-filesystem basis. (You can use the tune2fs command to view or change them.) After so many mounts, or a given period of time (usually months), a file system fsck is forced to catch any corruption that has occurred due to other causes.

The '-f' tells fsck to ignore the 'clean' flag and force a file system check. The fact that it is detecting problems means that something is not right in your situation. It could be a kernel problem, a configuration problem, a hardware problem, or something else.

Some file system problems are not very serious, such as blocks that neither belong to an accessible file nor the freelist. These conditions commonly result from power failures or system crashes, are self-contained, and only result in a little lost space until the next fsck is run. Other problems have the potential to cause file corruption, or progressively worse filesystem damage. These conditions are more serious, and frankly, barring certain hardware or cockpit errors, should rarely be seen in a non-beta environment today. (I mentioned several other times here recently that I had a similar problem with a SheevaPlug which was traced to a bad /etc/fstab entry for the root file system.)

So, good luck with it. I know this post won't solve your problem, but I hope it sheds some light on some things.

Hi restamp,

thank for your detailed information about fdisk. So abstract this situation: There is no special treatment of USB sticks. So most likely I have two problems:
1.) Hardware problem with the pogoplug
2.) Software problem with the drivers for the usb devices on the virtual debian system

Uhh I have an idea: The kernel of Jeff's Debian, the Pogoplug and the Debian I installed on the virtual machine (debian-508-i386-netinst.iso from here http://www.debian.org/distrib/netinst#smallcd - are they the same? Maybe I really have a problem with the kernel.
But that makes no sense?! All the other people use the kernel without having that much of trouble...

OK next thing I do, I take a real harddrive as boot device. Maybe this makes less trouble.
I report later...

Papst

Hello everyone,

I found a unsed usb harddrive so I decided to give it a try. And surprisingly it works! No erros - no nothing. I've no idea what went wront with the usb sticks but for me it's ok now.

Regards and thanks to all of you
Papst

Papst01 Wrote:
-------------------------------------------------------
> Hi everyone,
>
> I have news to report:
> My problems don't come from the box itself. I've
> recently created a virtual box with a Debian linux
> runnning on it.
> Following I tried with 3 different usb sticks
> (different vendors and sizes)
> Creating partitions with ext2 and ext3. First I
> checked with fsck -f -y /dev/sdb1 - no error.
> mounted the device with mount -t ext2 (respectivly
> ext3) /dev/sdb1 /mnt
> Than i copied a bigger archive file to and run
> tar -xzvf file.tar.gz. Looked good. Dismounted
> /mnt and ran e2fsck /dev/sdb1 -> no error. But
> that was a f**ing lie! e2fsck -f /dev/sdb1 found
> massive problems. And as I said, it is independent
> from the usb stick I used and the filesystem! So
> the problem was deeper.
> I decided "to zero" one of the sticks (4GB
> Transcent JetFlash 600) completly with dd
> if=/dev/zero of=/dev/sdb bs=1M
>
> With fdisk I created then a primary disk 2048MB.
> Than a 256MB prima swap disk.
> then mke2fs /dev/sdb1 than fsck -f -y /dev/sdb1
> (no error)
> mounted it, copied my test file
> Plugbox-Linux-2010.12-rootfs.tar.gz (from here
> http://plugapps.com/index.php5?title=PlugApps:Pogo
> plug_Setboot) than tar -xzvf
> Plugbox-Linux-2010.12-rootfs.tar.gz
> after that unmount /mnt and a fsck -f -y /dev/sdb1
> after that. And again many many errors. If I run
> fsck without "-f" the system claimes that the
> partition is clean.
>
> I think a do some basic thing completely wrong.
> But what? I'm not a Linux crack, I'm just reading
> "mans", forums, learning by doing and thats all of
> my knowledge.
> Is there a special way to prepare and handle usb
> sticks? Maybe bow to east, stick a needle in it
> and spell a curse against Microsoft or what?? I'm
> open to any suggestions!
>
> Papst


I have a Seagate GoFlex Home and GoFlex Net. I have the same issue as you on both units, with multiple USB sticks and numerous installs.

Try "tune2fs -l". It has an entry called Filesystem State that should report "clean".