Automount NTFS/HFS+ on Dockstar Debian

I have a question about autofs and NTFS, would appreciate any pointer to the right direction.

After installing autofs, I modified the auto.master file, and created the auto.usb file.

/etc/auto.master contains this rule:

/mnt/usb /etc/auto.usb --timeout 10 --ghost

and /etc/auto.usb contains this rule

* -fstype=auto,async :/dev/sd&

Autofs auto-mounted all USB thumb or hard drives that I plugged and assigned the mount points correctly, as long the USB drive is not NTFS. It seems that it would not auto-mount NTFS drive using the option "fstype=auto". I've also tried the option "fstype=ntfs" in the auto.usb file, and that did not work either.

And I've verified that I can mount NTFS drives manually using mount commands to mount the NTFS drive to a test directory such as "mount -t ntfs /dev/sdb1 /usb/testDir"

Thanks for any advice!

Addendum (03/15/02): Modified post title to add HFS+ auto-mounting in more recent posts



Edited 1 time(s). Last edit at 03/15/2012 11:08PM by bodhi.

I'm not sure that this is your issue, but on my setup I have a NTFS formatted 500G drive plugged in. I installed the ntfs-3g fuse driver package because it supposedly prevents issue's when using the drive for read/write operations. In any case you can read more about it on their FAQ as well as automounting here http://www.tuxera.com/community/ntfs-3g-faq/#plugandplay .

While the much simpler solution would have been for me to use a Linux file system instead of NTFS, there is always a chance I'll need that 500G Seagate Go drive for emergency storage on someone else's computer. So in my case using a different format wasn't really an option.

Thanks for your response. I did install libfuse2 also. It was strange, because I can manually mount the NTFS drive with no problem. And drives that have ext2 partition were automounted correctly. I've checked that I have the symlink from "/sbin/mount.ntfs-3g" to "/sbin/mount.ntfs" as suggested by tuxera website. And also run the command " udevadm info --query all --path /sys/block/sdb/sdb1" to make sure the drive partition FS type is recognized by udev (the output shown E: ID_FS_TYPE=ntfs).

I'm in the same boat here! I need to automount the drive without knowing in advance of its FS type, because I will unplug it and plug it in Windows and Mac PCs, for quick archival purpose. Also, I'd like to be able to let someone plug in their drive (FAT32, NTFS, EXT2) without having to do manual mounting.

More on this autofs problem. This is another test to see if I can mount manually with type auto. Looks like mount has no problem figuring out the appropriate filesystem to mount the ntfs disk. Hope someone can spot anything strange going on here. Thanks!

# mount -t auto /dev/sda3 /usb/testdir
# mount -t auto /dev/sdb1 /usb/testdir2
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 2065680    506672   1454164  26% /
none                     60420        48     60372   1% /dev
/dev/sda1              2065680    506672   1454164  26% /
tmpfs                    62864         0     62864   0% /lib/init/rw
tmpfs                    62864         0     62864   0% /dev/shm
tmpfs                    62864         0     62864   0% /tmp
/dev/sda3               818616       836    776196   1% /usb/testdir
/dev/sdb1              3905504     22168   3883336   1% /usb/testdir2

# mount
rootfs on / type rootfs (rw)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /dev type devtmpfs (rw,relatime,size=60420k,nr_inodes=15105,mode=755)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/sda1 on / type ext2 (rw,noatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/etc/auto.usb on /mnt/usb type autofs (rw,relatime,fd=6,pgrp=24155,timeout=10,minproto=5,maxproto=5,indirect)
/dev/sda3 on /usb/testdir type ext2 (rw,relatime,errors=continue)
/dev/sdb1 on /usb/testdir2 type fuseblk (rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096)

You don't need autofs. Some simple udev rules will do this for you. I like to use pmount since it handles the creation and removal the mount points in /media/{DEVICE} without any extra effort.

This is my rules file:

root@debian:~# cat /etc/udev/rules.d/99-automount.rules
# --sync to allow removal without corruption
# exclude sda since its the rootfs
ACTION=="add",KERNEL=="sd[bcd]*", RUN+="/usr/bin/pmount --sync --noatime --umask 000 %k"
ACTION=="remove", KERNEL=="sd[bcd]*", RUN+="/usr/bin/pumount %k"

Here's two active usb storage devices that were automounted this way, /dev/sdb is a usb stick, and /dev/sdc1 is a usb hard drive.

/dev/sdb on /media/sdb type vfat (rw,sync,nosuid,nodev,noexec,noatime,fmask=0111,dmask=0000,allow_utime=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,quiet,errors=remount-ro)
/dev/sdc1 on /media/sdc1 type fuseblk (rw,sync,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)


Your udev rules can be much more specific if you want. You could also check for the SUBSYSTEM=="usb" which would be useful on a system with a SATA controller to make sure that it only mounts usb devices. You can get fancy and mount specific drives by label or UUID to specific places, etc.

kraqh3d,

Thanks for your response. Seems like a pretty clean way to automount the USB drives. But would it slow down the IO too much using "synch"? the pmount man page mentions that would be the case. If I use asynch option, then I would have to unmount the drive manually before unplugging them (that's the draw back). It seems there is no perfect way to let a non-tech user to use the dockstar safely, other than installing a desktop server like gnome on it.

Yes, but it's the safe option. You don't have to use sync if you always cleanly unmount the filesystem before removing the device, as unmount will sync any pending writes to avoid data loss. I don't bother. I rarely even get on the CLI of my dockstar anymore now that its working the way I want. My dockstar's purpose is simple -- adhoc sharing of a bunch of usb disks full of music and video. I'm lazy. I can swap the attached drives without a thought, so the sync option is good for me. All I cared about was being able to play high-def mkv videos over the network on my hacked AppleTv running Xbmc and that works great. No stutter. And I rarely write to them when they are on the dockstar, and when I do its over the network so its pretty slow anyway.

Installing a desktop environment isn't really the solution. The issue is delayed writes with async IO. If you disconnect a device before those writes are actually written to the disk, you will get data loss. Simple as that. A desktop environment just provides an easier mechanism to unmount the filesystem with the GUI instead of the CLI. Instead of installing a desktop environment, you could instead use webmin as your interface to unmount, on top of a lightweight webserver like thttpd or lighthttpd or nginx. (There are some others as well.)

Thanks kraqh3d. You're absolutely right about the sync vs async option. Writing over the network is pretty slow anyway, and my main purpose for this Dockstar is to serve files to other PCs and XBMC, too. So the disk write is not a priority.

I've downloaded webmin a while ago but have not installed it, because I still want to try to figure out why autofs does not work the way we think it should. I tried to change the autofs script to make automount more verbose to see the log, but no log coming out (I could not tell from Jeff's install script where is the logging level settings). Just for educational purpose, I want to find out if it's a bug in automount or not! But I think I've spent enough time investigating this, your udev-only solution is the way to go now.

Thanks again for your help :-)

kraqh3d Wrote:
-------------------------------------------------------
> You don't need autofs. Some simple udev rules
> will do this for you. I like to use pmount since
> it handles the creation and removal the mount
> points in /media/{DEVICE} without any extra
> effort.
>
> This is my rules file:
>
> root@debian:~# cat
> /etc/udev/rules.d/99-automount.rules
> # --sync to allow removal without corruption
> # exclude sda since its the rootfs
....
------------------------------------------------------

I've just recalled John Doe's post about the order of USB ports assignments. During reboot, if the hard disk is plugged in to the mini-USB port, it will take the label sda, and the boot flash drive would become sdb. How would we change this automount rules file to mount the correct USB drives, and not automounting the rootfs? is it a problem at all if the boot drive partitions are mounted again?

Yeah... I had not rebooted my Dockstar since I set it up. I assumed that the uboot would always enumerate the rootfs stick as /dev/sda. I was wrong. I discovered that yesterday when I was updating my udev rules to use the file system label if present.

This is my new udev rule. It uses blkid to get the label, and then it mounts it with the label as the mount point, or the device name if there's no label. The new KERNEL check is a regular expression that looks for sda through sdd. I guess if you had a USB hub attached, you would need to expand that, or just open it entirely with "sd*".

root@debian:~# cat /etc/udev/rules.d/90-automount.rules
KERNEL!="sd[abcd]*", GOTO="media_label_end"

IMPORT{program}="/sbin/blkid -o udev -p %N"
ENV{ID_FS_LABEL}!="", ENV{name}="%E{ID_FS_LABEL}"
ENV{ID_FS_LABEL}=="", ENV{name}="%k"

ACTION=="add", RUN+="/usr/bin/pmount --sync --noatime --umask 000 %k %E{name}"
ACTION=="remove", RUN+="/usr/bin/pumount %E{name}"

LABEL="media_label_end"

Here you see it auto mounted two devices, sda and sdb this time because my rootfs was enumerated as sdc when I rebooted with the other USB devices attached. While the new rule accounts for the change in devices, I can't get it to automount the USB devices if they are present on boot up. I have to unplug them and plug them back in for the rules to take effect. I really want them mounted on boot up if present, but without the need to add /etc/fstab entries. I'll let you know if I figure out how.

root@debian:~# mount | grep sd
/dev/sdc1 on / type ext2 (rw,noatime,errors=remount-ro)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda on /media/USB type vfat (rw,sync,nosuid,nodev,noexec,noatime,fmask=0111,dmask=0000,allow_utime=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,quiet,errors=remount-ro)
/dev/sdb1 on /media/FantomHD type fuseblk (rw,sync,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)

kraqh3d, thanks for the new udev rules. It works very well. A really nice touch is using the label mount point.

I've also tried to reboot and found the same behavior, I had to unplug and replug the drives to get them auto-mounted.

Also, one more thing. When I'm at the directory where the drive is, just for testing purpose I unplugged that USB drive and udev can't unmount it (because the partition is busy). Of course we don't do this in real life (it's my test USB drive), but it might be something you'd like to figure out, too.

The boot up thing will likely require a shell script. The standard udev rules that create the devices work. I can see them in /dev/disks/by-uuid and /dev/disks/by-label, etc. I may work up a shell script to go through them and mount them.

Ah! Your unmount problem should be fixed by using the lazy unmount option. Pumount's lazy option is "--yes-I-really-want-lazy-unmount". Seriously :) Just add that. Please let me know if that works. It should.

(And I found out that if you have a space in your label, it'll be replaced with underscore when mounted.)



Edited 1 time(s). Last edit at 02/28/2011 08:04PM by kraqh3d.

LOL. That's really cool! it does work :-). What happen? did it parse the whole string for the word lazy?

No, it's just to make sure you know what you're doing. The man page says --lazy, but when you try that you get this:

root@debian:~# pumount --lazy
WARNING: Lazy unmount are likely to jeopardize data integrity on removable devices.
If that's what you really want, run pumount with --yes-I-really-want-lazy-unmount
Aborting.

I see. Tried it and saw that message. It's really handy if you have some non-tech users using the dockstar. This addition will help a lot. That and a warning "don't unplug it while you're using it" LOL.

Update: pmount is still logging a lot of kernel messages while trying to get file systems info. Even though the man page said it should not in version 0.9.17 or newer (I have version 0.9.23).



Edited 1 time(s). Last edit at 03/01/2011 10:39PM by bodhi.

I don't have any logging enabled; no syslogd, klogd, logrotate, etc. I'm trying to keep my dockstar as slim as possible. I've been considering using busybox-syslogd though because its a bit harder to diagnose problems without a log. I see the note in the man page. Are you seeing messages about it trying to mount a bunch of different file systems in succession? Try adding --type auto. That may be necessary to force it to use the same mechanism as mount.

I used busybox-syslogd to see the kernel messages. And the logread showed pmount tried all these file systems until it hits NTFS!

UDF-fs, ISOFS, FAT, VFS, hfs, VFS, EXT4-fs, REISERFS, SGI XFS, XFS, JFS, omfs

I've just tried the option --type auto in the udev rules as you've suggested, and pmount did not attempt to mount at all (I guess it does not understand the option so it exited gracefully). I wish we can edit the list of file systems somewhere then it should hit the correct one faster, but the man page stated that the order is hard coded so we're SOL there.

kraqh3d,

I've also looked at usbmount option, in the usbmount.conf file it has a list of file systems to try:

...
# Filesystem types: removable storage devices are only mounted if they
# contain a filesystem type which is in this list.
FILESYSTEMS="vfat ext2 ext3 ext4 hfsplus"
...

So it seems possible to use this wrapper to force pmount to mount the interested file systems first? However, it does not have the elegance of your udev rules (with usbmount we always have predefined mount points named usb0, usb1,...,usb7).

In any case, I'm pretty happy the way the dockstar works now. The lack of automounting during reboot or cold start is not that big a deal. How many times we'll need to do that anyway? I just have to remember to unplug, and plug the USB drives back in after reboot/restart. If you can come up with startup script, it would be even more perfect.

Thanks again for sharing your expertise:-)



Edited 1 time(s). Last edit at 03/03/2011 06:20PM by bodhi.

kraqh3d,

Question: if we implement the script to automount the USB disk at startup, would that be executed at the end of /etc/rc.local ? is that early enough?

FYI. These aumounting messages were also logged in dmesg:

[  475.688583] UDF-fs: No anchor found
[  475.692094] UDF-fs: Rescanning with blocksize 2048
[  475.787084] UDF-fs: No anchor found
[  475.790595] UDF-fs: No partition found (1)
[  475.861078] ISOFS: Unable to identify CD-ROM format.
[  475.913601] FAT: bogus number of reserved sectors
[  475.918376] VFS: Can't find a valid FAT filesystem on dev sda.
[  475.964206] hfs: unable to find HFS+ superblock
[  476.003709] hfs: can't find a HFS filesystem on dev sda.
[  476.092981] VFS: Can't find ext3 filesystem on dev sda.
[  476.105105] VFS: Can't find an ext2 filesystem on dev sda.
[  476.229735] EXT4-fs (sda): VFS: Can't find ext4 filesystem
[  476.286213] REISERFS warning (device sda): sh-2021 reiserfs_fill_super: can not find reiserfs on sda
[  476.390080] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[  476.404420] SGI XFS Quota Management subsystem
[  476.410363] XFS: bad magic number
[  476.413701] XFS: SB validate failed
[  476.456382] JFS: nTxBlock = 982, nTxLock = 7858
[  476.485477] omfs: Invalid superblock (0)

Ah, so you're using busybox-syslogd. Nice. I think I'm going to run it. It's kind of a pain not to have any logging.

I wouldn't use usbmount. It has fixed mount points which are rather annoying. I want to know exactly where the filesystem is mounted.

Why are you so concerned about pmount trying a bunch of filesystems in succession? It's not going to break anything. They just all fail.

Your dmesg output confirms my guess that it's a start up order problem. Looking at rcS.d, we're reliant on these services starting in this order: S02udev -> S07module-init-tools -> S17fuse. So, udev is running through its rules before any modules are loaded.

I haven't had a chance to work on the mount script yet. But yes. /etc/rc.local should run very last.

Yes, busybox-syslogd is really nice that all logging going to RAM, without the logging it's like we're blind :-) And no, I'm not concerned about those mounting messages, as long as it mount the correct filesystem, it's all that matter. Just thought that it seems odd because it behaves like the old version of pmount (according to the man page, this new version of "pmount" uses the same mechanism as "mount", so we should not see the messages when automounting the USB drives after system started).

Thanks for confirming that startup order problem. I've just only begun to learn and understand this Debian startup/shutdown order!

I was looking at pmount's source, and it needed to have been built with blkid support to autodetect like mount. I'm not sure why it doesn't work. It could be that the debian packaged one wasnt.... But this doesn't matter. Udev can also pass the filesystem type since we're detecting the label with the blkid command line tool.

Try this udev rule file. The filesystem type will be detected as ntfs, but pmount should mount using ntfs-3g if it detects it's available. You could do rules for each possible filesystem type I guess if you want to do anything unique for each.

KERNEL!="sd[abcd]*", GOTO="media_label_end"

# get the label and filesystem type from blkid
IMPORT{program}="/sbin/blkid -o udev -p %N", ENV{name}="%E{ID_FS_LABEL}", ENV{fs}="-t %E{ID_FS_TYPE}"
# if the label is blank, use the device name
ENV{ID_FS_LABEL}=="", ENV{name}="%k"
# if the filesystem type is blank, ignore it and try to use pmount's auto detection mechanism
ENV{ID_FS_TYPE}=="", ENV{fs}=""

ACTION=="add", RUN+="/usr/bin/pmount %E{fs} --sync --noatime --umask 000 %k %E{name}"
ACTION=="remove", RUN+="/usr/bin/pumount %E{name}"

LABEL="media_label_end"



Edited 2 time(s). Last edit at 03/05/2011 08:23PM by kraqh3d.

kraqh3d, it did not work. I assume we're still having the startup order problem?

Hi All,

I'm having problems automounting the ntfs filesystem. I have ntfs-3g, fuse-utils, pmount, and samba installed. When the unit boots it sees all my drives, but the fs is not automounted. I am using kraqh3d's udev rule and the really strange thing is I already have another dockstar setup that works with the rules. Any thoughts?

One more thing, if I unplug the drive and replug it is mounted properly and I can see all my files.

TIA,
Elviso33



Edited 1 time(s). Last edit at 03/06/2011 12:00AM by elviso33.

Perhaps you can post your dmesg contents for both of your dockstars after reboot? there must be some differences.

That rule change doesn't address the startup problem. It just passes the filesystem type to pmount, so it doesnt test every supported filesystem in succession until it finds the correct one (where ntfs is last in the list looking at the source.) I'm using it, though without that extra check for a blank filesystem type.

Strange, I'm still seeing the same pmount messages with the new rules (I restarted the dockstar to make sure the rules take effect).

You can force udev to reload the rules after a change with "udevadm control --reload-rules". You can also log what udev sees with "udevadm monitor". You can additionally see all the attributes with "udevadm monitor --property". It's a lot of output.

Hmmm. That should be mostly correct. I'm not using that rule. I've changed rules a few times since. I kind of made that up on the spot, from my previous post. Maybe I made a typo in it. Try my latest rule, below, which is working on my system. I guess it could fail if the filesystem is unknown because I'm directly including the "-t type" option in the RUN line. But I've not had a problem with it. I've also not begun on the boot up annoyance yet. I'll try to tackle that sometime this week when I get a free hour or two. I was really hoping someone already tackled it and posted their solution here :)

root@debian:~# cat /etc/udev/rules.d/90-automount.rules
# --sync to allow removal without corruption

KERNEL!="sd*", GOTO="media_label_end"
ENV{ID_USB_DRIVER}!="usb-storage", GOTO="media_label_end"

IMPORT{program}="/sbin/blkid -o udev -p %N", ENV{name}="%E{ID_FS_LABEL}"
ENV{name}=="", ENV{name}="%k"

ACTION=="add", RUN+="/usr/bin/pmount -t %E{ID_FS_TYPE} --sync --noatime --umask 000 %k %E{name}"
ACTION=="remove", RUN+="/usr/bin/pumount %E{name}"

LABEL="media_label_end"

And here's the log after powering up a usb drive. There's no errors from trying every known file system until it gets to ntfs-3g.

root@debian:~# logread -f
Mar  6 18:58:00 debian user.info kernel: [597062.809217] usb 1-1.2: new high speed USB device using orion-ehci and address 19
Mar  6 18:58:00 debian user.info kernel: [597062.920182] usb 1-1.2: New USB device found, idVendor=13fd, idProduct=1340
Mar  6 18:58:00 debian user.info kernel: [597062.927176] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Mar  6 18:58:00 debian user.info kernel: [597062.934624] usb 1-1.2: Product: ST2000DL003-9VT1
Mar  6 18:58:00 debian user.info kernel: [597062.939383] usb 1-1.2: Manufacturer: FANTOM
Mar  6 18:58:00 debian user.info kernel: [597062.943847] usb 1-1.2: SerialNumber: 355944314C42343020202020
Mar  6 18:58:00 debian user.info kernel: [597062.952204] usb 1-1.2: configuration #1 chosen from 1 choice
Mar  6 18:58:00 debian user.info kernel: [597062.961929] scsi16 : SCSI emulation for USB Mass Storage devices
Mar  6 18:58:01 debian user.debug kernel: [597062.974929] usb-storage: device found at 19
Mar  6 18:58:01 debian user.debug kernel: [597062.974940] usb-storage: waiting for device to settle before scanning
Mar  6 18:58:06 debian user.debug kernel: [597067.969219] usb-storage: device scan complete
Mar  6 18:58:06 debian user.notice kernel: [597067.969884] scsi 16:0:0:0: Direct-Access     FANTOM   ST2000DL003-9VT1 2.10 PQ: 0 ANSI: 4
Mar  6 18:58:06 debian user.notice kernel: [597067.982854] sd 16:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Mar  6 18:58:06 debian user.notice kernel: [597068.001595] sd 16:0:0:0: [sda] Write Protect is off
Mar  6 18:58:06 debian user.debug kernel: [597068.006586] sd 16:0:0:0: [sda] Mode Sense: 21 00 00 00
Mar  6 18:58:06 debian user.err kernel: [597068.006595] sd 16:0:0:0: [sda] Assuming drive cache: write through
Mar  6 18:58:06 debian user.err kernel: [597068.014820] sd 16:0:0:0: [sda] Assuming drive cache: write through
Mar  6 18:58:06 debian user.info kernel: [597068.021308]  sda:
Mar  6 18:58:06 debian user.info kernel:  sda1
Mar  6 18:58:06 debian user.err kernel: [597068.052470] sd 16:0:0:0: [sda] Assuming drive cache: write through
Mar  6 18:58:06 debian user.notice kernel: [597068.058775] sd 16:0:0:0: [sda] Attached SCSI disk
Mar  6 18:58:08 debian daemon.notice ntfs-3g[3178]: Version 2010.3.6 integrated FUSE 27
Mar  6 18:58:08 debian daemon.notice ntfs-3g[3178]: Mounted /dev/sda1 (Read-Write, label "FantomHD", NTFS 3.1)
Mar  6 18:58:08 debian daemon.notice ntfs-3g[3178]: Cmdline options: rw,noexec,nosuid,nodev,sync,user,noatime,uid=0,gid=0,umask=000
Mar  6 18:58:08 debian daemon.notice ntfs-3g[3178]: Mount options: rw,noexec,nosuid,nodev,sync,user,silent,allow_other,nonempty,noatime,fsname=/dev/sda1,blkdev,blksize=4096,default_permissions
Mar  6 18:58:08 debian daemon.notice ntfs-3g[3178]: Global ownership and permissions enforced, configuration type 1

Cool! that works great. No longer seeing the filesystem mount warning messages. I'm guessing the typo was related to
ENV{fs}="-t %E{ID_FS_TYPE}".