Dockstar Debian USB does not boot after power failure

After a power failure, my Dockstar with Debian USB stick does not boot any more. Unplugging it and it will reboot back to pogoplug. I always make a backup USB image whenever I install/uninstall some packages, so I just restore it, it's not a big problem. But please explain why? I vaguely recalled Jeff's UBoot modified something each reboot and shutdown?



Edited 1 time(s). Last edit at 08/01/2011 05:40AM by bodhi.

bodhi Wrote:
-------------------------------------------------------
> After a power failure, my Dockstar with Debian USB
> stick does not boot any more. Unplugging it and it
> will reboot back to pogoplug. I always make a
> backup USB image whenever I install/uninstall some
> packages, so I just restore it, it's not a big
> problem. But please explain why? I vaguely
> recalled Jeff's UBoot modified something each
> reboot and shutdown?


I wish I knew.

Sometimes mine won't boot after a power outage, and sometimes it will boot. I just make sure to have an image I can restore to the drive in those cases.

I don't know if I should be comparing the behavior of my Pogoplug to the Dockstar, but I also have had some difficulty with power outage reboots. One thing that I have noticed is that my USB stick 'ext2' file system appears to be corrupted each time this occurs. Twice now I have removed the stick from the Pogoplug, plugged it into my Linux desktop, unmounted the device, and then used:

sudo fsck.ext4 -v /dev/sdx1

...where 'sdx1' is whatever the device name assigned was (I think 'fsck.ext2' would work too). After doing some repairs, and plugging it back in, I eventually get my system back after a reboot or two. It is still a royal pain-in-the-youknowwhat, but at least I don't have to start over or do a system restore. If it was easy and understandable how to do so, I would consider using 'ext3' or 'ext4' instead (hoping for auto file system repair?).

Frankly, all my computers here are attached to an UPS, so I have not had much trouble with corruption caused by power glitches. I have inadvertently crashed my Dockstar on a couple of occasions (with its root drive on a Seagate GO drive) by jostling the drive on its pedestal atop the Dockstar body. (It doesn't seem to like this.) But, if I recall, in all cases, it rebooted OK.

When I loaded the GO drive, I made all the partitions on it EXT3. I think EXT3 is much more robust because if power is lost in the middle of an update, the log remains and the update is reapplied from the beginning on the next boot, maintaining file system consistency. I do undock the GO drive occasionally and take it over to my desktop PC to back up its root partition. One would think that on a clean shutdown, the log would get rolled as part of the process, but that does not appear to be the case. I notice that an fsck usually finds a log and unrolls it, although after unrolling it the fsck never finds corruption. (And, on a larger non-root file system hanging off that system, the act of unrolling the log can become quite protracted, sometimes taking several minutes to complete.)

I have had more problems in this regard with my SheevaPlug, which is running an old Ubuntu 9.04 load on an SD card. On this box, I have experienced many problems with major root file system corruption, and indeed, I used to see such corruption even if I shut the box down cleanly. On that box, the root file system is EXT2. (There are also some EXT3 filesystems on connected hard drives which do not exhibit these problems). In the plugcomputer.org forum, another member made a suggestion that helped me greatly with this problem:

http://www.plugcomputer.org/plugforum/index.php?topic=4587

(You might check this on your boxes, but it appears Debian comes with the fstab correctly structured by default.) After making the recommended change to the SheevaPlug's /etc/fstab, I now see many fewer problems with the SheevaPlug's root file system. In fact, I don't believe I've seen any true corruption problems since making the change. As an experiment, I just rebooted the Sheevaplug now, after it had been up and running for 50 days straight, and it came up clean. That said, on the few times the box has been uncleanly rebooted, I have observed problems, and the box may not come up again without assistance. On the Sheevaplug, I have access to the serial console, though, and can investigate, and in all such cases, the problem turned out to be that the EXT2 file system had problems that required manual action to correct. But, these were not the wholesale problems with file system corruption I was seeing before. Rather, they were the more benign link-count-wrong-in-inode type problems which are easily fixed.

So, my advice: First, make sure that the root file system type and device are correctly indicated in /etc/fstab (probably not a problem in Debian). Second, if you can, make all file systems EXT3. (I can't speak for EXT4 as I have very little experience with it.) EXT3 is fine for hard drives. Unfortunately, it is questionable for NAND devices (SD cards and thumb drives) because it results in an increased number of writes, which can slow them down and cause them to wear out faster.

Somewhere I read that only EXT2 (not EXT3 or 4) are usable by the uBoot. However, I've successfully booted from an EXT3 filesystem. Indeed, on my Dockstar, I do it all the time. EXT3 is just EXT2 with logging. The only problem I can see with using EXT3 is that you must force the log to roll before booting if you install a new uImage. If not, of course, the uImage is not going to be in the EXT2 format on the EXT3 file system until this is done.

Hope this helps.

Given that the EXT3 stick will be getting slower and slower and eventually worn out. I guess the conclusion is that without a UPS, those who use USB stick for Debian are stuck with a corrupted drive (either it's fixable or need a system restore) after a power failure. System restore is not so bad, if the stick is only used for Debian.

Has anybody used Acronis on a remote PC to backup the whole image of a Debian USB stick while the Dockstar is running? Clonezilla user's guide said online cloning is not supported yet.

I had the same issue when i first hacked a pogoplug.
I also discovered that the filesystem (ext3) was corrupted and could be repaired by fsck.
At this point i considered what might be writing to the usb key and moved the data directorys of my apps, and swap to a usb hard drive.
Have not had an issue since and this should extend the life of the usb key.
Hope this helps

Glenn

glenn Wrote:
-------------------------------------------------------
> I had the same issue when i first hacked a
> pogoplug.
> I also discovered that the filesystem (ext3) was
> corrupted and could be repaired by fsck.
> At this point i considered what might be writing
> to the usb key and moved the data directorys of my
> apps, and swap to a usb hard drive.
> Have not had an issue since and this should extend
> the life of the usb key.
> Hope this helps
>
> Glenn

Thanks, that's what I thought to do, too. However, would that still cause the same problem for the HDD? i.e. you would need to check and fix the HDD for errors after a power failure? It's a big improvement for sure, if the USB stick still boots OK.

Providing the OS can boot i think it will automatically recover the hard drive.
I have not experienced any problems since i reconfigured mine and i believe it has had the power
pulled on it 6 or more times.
Worth a try anyway.

Glenn

every time there is a power failure, the bootloader will not boot debian from the usb flashdrive, but boots pogoplug.

so i manually need to run fsck to fix it, and then reboot.


does uboot skip the flashdrive on purpose ? i want to force it to boot into it and it will then automatically run fsck like a regular linux system.

how do i do that ?

If you are consistently booting into the PogoPlug OS on a cold boot, it is not likely due to file system corruption. A power failure is a "cold boot", as opposed to a warm boot where you instruct the device to reboot without removing power. Some flash drives cold boot fine, but won't warm boot. Others, perhaps like yours, do the opposite. Some will not boot at all. What you would like is one that will do both -- both cold and warm boot reliably.

Sometimes flash drives can be coaxed to behave correctly by playing tricks, like doing multiple initializations of the USB subsystem in the uBoot, but the best solution is to acquire a flash drive that plays well with your Plug.

See the sticky thread

http://forum.doozan.com/read.php?2,1915

for people's experiences with various flash drives, and if you can, select accordingly. For instance, Sandisk flash drives seem to work well generally.

Good luck.

@restamp,

I'm confused. Isn't it true when you pull out the USB Debian stick, the Dockstar will boot to PogoPlug (for those of us who have a simple Debian installation using Jeff's script)? so it is the Debian system that was corrupted by power failure.

-bodhi

Hi bodhi,
erez was complaining that, after a power failure, his Pogoplug would not boot from his flash drive, but "boots pogoplug" instead (which I take to mean it boots the OS on its internal NAND filesystem). I've never seen this happen as a result of file system corruption (although I suppose if the file system was *so* corrupt that the uBoot couldn't find the uImage or uInitrd, it might present this symptom. But the FS would have to be toast for that to happen.) Normally, if there is a FS problem (requiring a manual fsck), the system just hangs on boot here (awaiting manual action to be performed on a console that doesn't exist) rather than booting from the alternative NAND device.

So, I was presuming that erez's problem was that he had selected a flash drive that didn't handle cold boots well. Of course, it may turn out to be some other problem, but that's what seems to come closest to meeting erez's description here.

It's true that a power failure can result in FS damage, requiring manual action. This is especially tru of Ext2 FSs. However, as I said above, when this happens, the response is usually a hung boot, not a reversion to the OS on the NAND.

Or, am I missing your point?

restamp,

You're right, of course! I've misread it, sounds like erez's problem is different than mine after a power failure. erez's system boots to pogoplug OS. Mine just hung :-). I tried pulling out the stick to let it boot back to pogoplug OS, and then replug the stick and do a warm boot, it still hung. So I just restore the USB image to recover (even though I assume fsck would work, but it would take longer).

But it is strange, nevertheless, because erez's system booted to Pogoplug always, but a fsck was also needed. I think erez needs to test the cold boot. Perhaps, like you've said, the USB stick did not cold boot consistently, anyway, so it never got to the point that it would hang.

Thanks,
bodhi

I am having a similar problem, after a power outage, at least one of my dockstars will hang while booting (I can see it on the network with the correct ip, however all ports are closed) Not having a linux PC handy this last time, I booted into pogo, reflashed the uboot, and reattached the usb stick and it reboots fine. Looking into getting some UPSs, the one in the barn with the webcams is of particular annoyance, as I have to climb into the rafters to access it.

Also I have no idea why reflashing the uboot works, but I just did it on another one of my devices that was in this state and it worked. (I'm a bit of a linux newb, having some other issues with samba and having everything just work on reboot)

I am too late to this game, but even i am facing similar issue. I live in India and power cuts are quiet common here. My dockstar boots fine from pen drive even if the file system state in not clean but for some directories in file system when I do cd or ls it says "Read only file system". when I do fsck / -y and reboot, everything works fine. worst part is many a times halt does not work and i have to pull the power after fsck.

I think what I need to do is...

1. make a separate ext2 boot partition and put kernel there so that uboot can boot from it. in fstab make this /boot partition as readonly
2. make second ext4 partition and mount it as /

I know it wears out my flash drive but it is better than facing issues every time I come home.

I got a new pen drive, did exactly as explained above. netconsole stops at Starting kernel...

Not sure what is going wrong. Can somebody help here?

below is my fstab modified on ext4 partition

/dev/sda1 /boot ext2 noauto,noatime 0 2
/dev/root / ext4 noatime,errors=remount-ro 0 1
/dev/sda3 none swap sw 0 0
/dev/sda4 /media/local vfat defaults 0 0
tmpfs /tmp tmpfs defaults 0 0

I am missing how to tell kernel where the root partition is

My above mentioned problem is solved. in case anyone else encounter the this...

usb_scan was overwriting usb_root variable which was the issue.

I set usb_root=/dev/sda2 where as usb_scan was overwriting it to /dev/sda1 which is my boot partition

As a workaround I did

fw_setenv usb_init 'if usb start; then version; else usb stop; usb start;fi;' instead of 'run usb_scan' which also solved the problem of not booting from usb sometimes.

note, since usb_scan is not done now, it only works with sda port (next to ethernet)

may be this is a bug for my partition scheme which has to be solved.

I've read posts here and there about how to solve this power outage problem with booting from a USB flash drive, but not sure I've seen a consolidated how-to. Anybody who has done this and want to take a stab at it? It seems there are partition scheme that work well such as:

1. Partition 1: Read-Only Ext2 which contains the /boot directory
2. Partition 2: Read-Write Ext3 which contains the rootfs
3. Optional Read-Write partition for data

Questions: so do one set this up? can the /boot files symlinked back to the rootfs, or must they be copies? can the swap file reside in partition 2 if one wants to use it instead of a swap partition? what fw envs needed to be set to make this work? is fstab the only way to make partition 1 read-only?



Edited 1 time(s). Last edit at 12/18/2011 04:14PM by bodhi.

^ My setup is simple and I never had to do a manual fsck for the past one year or more (There are a LOT of power cuts in my area)

1. Partition 1 - / - Read-Write - EXT3 complete rootfs
2. Partition 2 - swap
3. Partition 3 - EXT4 - Data

You need to set the rootfs_type parameter to ext3 in the uboot config as well.

Thanks Varkey. Do you turn on logging for things like samba and webmin? i.e. so you did not need to do any thing special about logging?

Ya all logs are enabled and I didn't make any special settings for it. But I use a USB Hard Disk, not a flash drive.

I'll try your setup with a Sandisk USB flash drive to see if it will be as resilient. Thanks.

Since my purpose is to be able to run Debian on the Dockstar with only a USB flash drive (smallest footprint and minimum cost possible), I've set up a 2 partition USB flash drive. Both partitions are Ext2 to avoid wearing out the drive. Note that Varkey has reported success with a single Ext3 partition on HDD (see above), this Ext2-partition-only setup is better suit flash drives.

It is a straight forward process. Thanks to the UBoot info reported by Kraqh3d, the only UBoot env that needs to be changed is the usb_init, which is used to identify the location of the rootfs.

So here is the condensed procedure:

1. Use a fresh USB stick to create the 2-partition Debian drive

- Create 2 EXT2 partitions. First for boot (large enough to hold the boot folder, i.e. 100M), 
and second for rootfs (the rest of the drive). 
- Mark the boot partition active. Label this boot partiton boot
- Label the rootfs partition ROOTFS

2. Copy and move files from the orginal Debian USB stick

- As root, copy the rootfs from the original USB stick to the 2nd partition on the new USB stick
- On the new USB stick, move the /boot folder (from the 2nd partition) to the 1st partition

3. Modify rc.local on the new USB stick to remount boot partition as read-only

- Assuming the 2 partitions were mounted by pmount using its label: /media/boot and /media/ROOTFS.
- Add to /etc/rc.local the remount command so that the boot partition is read-only after started up:
	mount -o remount,ro /media/boot

4. Change UBoot env to identify rootfs by the label ROOTFS

	fw_setenv usb_init "run usb_scan; setenv usb_root LABEL=ROOTFS"

5. Sync, and shutdown. Reboot with the new 2-partition Debian drive.

6. After booted up, run iotop to find which processes periodically write to disk and symlink their log files to /tmp
#iotop -o -b -qqq

UBoot variables while the system is running:

# fw_printenv usb_init
usb_init=run usb_scan; setenv usb_root LABEL=ROOTFS
# cat /proc/cmdline
console=ttyS0,115200 root=LABEL=ROOTFS rootdelay=10 rootfstype=ext2 mtdparts=orion_nand:1M(u-boot),4M(uImage),32M(rootfs),-(data)

Mount info:

root@Ds1:~# mount
rootfs on / type rootfs (rw)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /dev type devtmpfs (rw,relatime,size=60416k,nr_inodes=15104,mode=755)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/disk/by-label/ROOTFS on / type ext2 (rw,noatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda1 on /media/boot type ext2 (ro,nosuid,nodev,noexec,noatime,errors=remount-ro)

Also note that since the /boot folder was moved, future Debian updates will have to be copied over to the boot partition.

Addendum 1: I'm using the swap file located in / folder of the rootfs partition.

Addendum 2: Looks like the booting time was increased by about 20 seconds comparing to the 1 partition drive. Probably the rootdelay is a factor? Somehow the kernel tooks exactly 20 seconds more to complete start up.



Edited 2 time(s). Last edit at 12/21/2011 04:39PM by bodhi.