I recently wanted to reconfigure my NAS from Apple’s AFP (Apple Filing Protocol) to “regular” SMB (Samba), given that there appeared some non-Mac devices at home lately. :O I thought of it as a simple “turn off the one protocol and turn on the other one” reconfiguration, but after doing so, to my surprise, I discovered that some files went missing. The strange thing about it was that when looking at the same folder via SMB from my Raspberry Pi, the files were there and I could happily read them, but then looking at the same folder via AFP or SMBon my Mac, the files were gone. On the verge of going mad, I turned to a high-scale googling effort, trying to figure out what went wrong. The information out there was so sparse that I decided to document this nightmare of a network file system admin to hopefully save others a few grey hair…
As said above, it all began with the simple task of switching my NAS from the AFP to the SMB protocol and reconnect my devices to it, not thinking of discovering a sinister but obviously failed plot of world domination or perhaps just a “well, we do it differently” situation as it happened so often in the early days of IT.
For a long time, my NAS was quite happily running on AFP and my Mac connected to it. It didn’t cause any issues and I didn’t see anything wrong with that either, after all, it’s Apple protocol to Apple device. But, with the appearance of some non-Apple devices, it was time to change and move to something more liberal of a file protocol, i.e. something that non-Apple devices could see and understand too. So it was time to finally move from AFP to SMB and by doing so I discovered two things that make me recommend to never, ever use AFP to begin with!
The many, many hidden files
.AppleDouble
One thing with AFP is that it keeps a lot of hidden files in folders called .AppleDouble. Within these folders, there are files with the same name as the actual files that keep metadata information about them. Windows does a similar trick by creating a Thumbs.db file, however, with Mac you will get a metadata file for each and every file, and you won’t even notice until you look into the folder with any other protocol than AFP, say SMB or FTP. The whole thing looks as follows:
I have the same remote location mounted via AFP and SMB, in my case, it’s my Data folder. /Volumes/Data is mounted via AFP while the /Volumes/Data-1 is mounted via SMB:
$ mount ... ..._afpovertcp._tcp.local/Data on /Volumes/Data (afpfs, nodev, nosuid, mounted by ...) ..._smb._tcp.local/Data on /Volumes/Data-1 (smbfs, nodev, nosuid, mounted by ...)
Next, I’m going ahead and create a simple, single, empty text file via AFP:
$ cd /Volumes/Data $ mkdir AFPNightmare $ cd AFPNightmare/ $ touch text.txt $ ls -al total 0 drwxr-xr-x 1 ... ... 264 20 Jan 17:12 . drwx------ 1 ... ... 296 20 Jan 17:12 .. -rw-r--r-- 1 ... ... 0 20 Jan 17:12 text.txt
There is nothing suspicious that can be seen there. Now let’s have a look at the same folder via the SMB mount:
$ cd /Volumes/Data-1/AFPNightmare/ $ ls -al total 96 drwx------ 1 ... ... 16384 20 Jan 17:12 . drwx------ 1 ... ... 16384 20 Jan 17:11 .. drwx------ 1 ... ... 16384 20 Jan 17:12 .AppleDouble -rwx------ 1 ... ... 0 20 Jan 17:12 text.txt
As you can see, Mac OS X happily created the hidden folder .AppleDouble and hides the fact, of course, as it is a hidden folder as far as it is concerned. Inside, you will see the metadata file:
$ ls -al .AppleDouble/ total 128 drwx------ 1 ... ... 16384 20 Jan 17:12 . drwx------ 1 ... ... 16384 20 Jan 17:12 .. -rwx------ 1 ... ... 741 20 Jan 17:12 .Parent -rwx------ 1 ... ... 741 20 Jan 17:12 text.txt
Now let’s try it the other way around and create a file in the SMB share:
$ cd /Volumes/Data-1/ $ mkdir SMBGoodness $ cd SMBGoodness/ $ touch test2.txt $ ls -al total 64 drwx------ 1 ... ... 16384 20 Jan 17:28 . drwx------ 1 ... ... 16384 20 Jan 17:11 .. -rwx------ 1 ... ... 0 20 Jan 17:28 test2.txt
And now have a look at it via the AFP mounted location:
$ cd /Volumes/Data/SMBGoodness/ $ ls -al total 0 drwxr-xr-x@ 1 ... ... 264 20 Jan 17:28 . drwx------ 1 ... ... 330 20 Jan 17:28 .. -rw-r--r--@ 1 ... ... 0 20 Jan 17:28 test2.txt
Good news, nothing there, but notice that we did lose the execute permission on the file!
It might not look much at first but it turns out that these little hidden files can accumulate quite a bit, as you can imagine, having one for each file you create. Not only can that be painful if you were to make a backup of your files with anything but AFP but, although not big individually, they do accumulate quite some space wastage as well. The below command finds all .AppleDouble folders, runs a du -sk . on them, i.e. gets the number of kilobytes occupied by these files, and then sums those kilobytes up into the total amount using awk. The result is that all these hidden files waste about 248 MBs:
$ cd /Volumes/Data-1
$ find . -name ".AppleDouble" -exec du -sk {} \; | awk '{ sum += $1; } END { print sum; }'
248096
You may or may not care about the space wastage, of course. But then again, they do accumulate, add to your backup/copying time overall and the space could be used for actual files.
._ files
Another hidden file type that you may encounter as I did, is the hidden ._* files. These files store information of the actual files that would otherwise either go into an extended attribute on the HFS+ (Apple native) file system or a Unix/UFS volume. In my experience, these files are way fewer than the .AppleDouble folders that you will see. Nevertheless, they do accumulate and occupy space as well. In my case, there are 28 of them and they waste 70KBs:
$ find . -name "._*" | wc -l
28
$ find . -name "._*" -exec du -sk {} \; | awk '{ sum += $1; } END { print sum; }'
70
The Unicode file name madness
Another really strange thing that I have encountered is that after copying files onto my NAS via FTP they went missing when looking for them on my Mac. Although I could never figure out what the actual culprit was, I did find out that it is related to the file names and their Unicode Normalization Forms. In particular, the files that went missing are all in the NFD normalization form generally used by Mac OS X. Regardless whether I access the NAS via AFP or SMB, I can’t see the files or folders anymore after copying them:
$ cd /Volumes/Data/Music/iTunes Library/Podcasts $ ls -al ls: Clublife by Tiësto: No such file or directory total 32 drwxr-xr-x@ 1 ... ... 296 12 Jan 22:23 . drwxr-xr-x@ 1 ... ... 398 10 Jan 17:08 .. -rw-r--r--@ 1 ... ... 6148 11 Jan 08:17 .DS_Store drwxr-xr-x@ 1 ... ... 704 20 Jan 18:27 Software Engineering Daily drwxr-xr-x@ 1 ... ... 264 20 Jan 18:27 Software Engineering Radio - The Podcast for Professional Software Developers drwxr-xr-x@ 1 ... ... 264 20 Jan 18:27 The Changelog drwxr-xr-x@ 1 ... ... 4308 12 Jan 07:20 The History of Rome
$ cd /Volumes/Data-1/Music/iTunes\ Library/Podcasts/ $ ls -al total 320 drwx------ 1 ... ... 16384 12 Jan 22:23 . drwx------ 1 ... ... 16384 10 Jan 17:08 .. drwx------ 1 ... ... 16384 11 Jan 12:21 .AppleDouble -rwx------ 1 ... ... 6148 11 Jan 08:17 .DS_Store drwx------ 1 ... ... 16384 20 Jan 18:27 Software Engineering Daily drwx------ 1 ... ... 16384 20 Jan 18:27 Software Engineering Radio - The Podcast for Professional Software Developers drwx------ 1 ... ... 16384 20 Jan 18:27 The Changelog drwx------ 1 ... ... 16384 12 Jan 07:20 The History of Rome
Note something interesting! See how on AFP ls actually complains that it cannot find a given folder: ls: Clublife by Tiësto: No such file or directory, while over on the SMB share ls doesn’t even realize that something is missing altogether? Something is definitely off here. Luckily, when looking at the same directory via SMB via my Raspberry Pi, i.e. Linux, I can see and even read the folder:
Pi $ cd /mnt/Data/Music/iTunes\ Library/Podcasts/Pi $ ls -al total 12 drwxr-xr-x 2 root root 0 Jan 23 18:51 . drwxr-xr-x 2 root root 0 Jan 23 18:01 .. drwxr-xr-x 2 root root 0 Jan 23 18:39 'Clublife by Tiësto' -rwxr-xr-x 1 root root 10244 Jan 24 21:42 .DS_Store drwxr-xr-x 2 root root 0 Jan 23 18:39 'Software Engineering Daily' drwxr-xr-x 2 root root 0 Jan 23 18:43 'Software Engineering Radio - The Podcast for Professional Software Developers' drwxr-xr-x 2 root root 0 Jan 23 18:46 'The Changelog' drwxr-xr-x 2 root root 0 Jan 23 18:51 'The History of Rome' Pi $ ls -al Clublife\ by\ Tiësto/ total 84992 drwxr-xr-x 2 root root 0 Jan 12 15:19 . drwxr-xr-x 2 root root 0 Mar 13 05:23 .. -rwxr-xr-x 1 root root 86953173 Jan 11 20:24 Clublife by Tiësto 438 podcast.m4a Pi $
Also, note that the Raspberry Pi can again not see the .AppleDouble folder while the SMB mount on my Mac can. In short, the whole thing is a mess!
Mounting the NAS Share
You can mount the SMB share of the NAS with the CIFS filesystem. On my Raspberry Pi, the Samba client comes out of the box. In case you need to install it first, you can find the instruction of that in the Raspberry Pi documentation.
Mount the share via the mount.cifs command, specifying the user via -o user=, the IP Address and share name, and finally the location for the share to be mounted into. You will be asked for the password of the share if there is one specified. In the event that you get an error mount error(95): Operation not supported on mounting, specify vers=2.0 via the -o parameter:
Pi $ sudo mount.cifs -o user=gerald,vers=2.0 //192.168.1.2/Music /mnt Password for gerald@//192.168.1.2/Music: ******* Pi $ cd /mnt/iTunes\ Library/Podcasts/ Pi $ ls -al total 12 drwxr-xr-x 2 root root 0 Jan 23 18:51 . drwxr-xr-x 2 root root 0 Jan 23 18:01 .. drwxr-xr-x 2 root root 0 Jan 23 18:39 'Clublife by Tiësto' -rwxr-xr-x 1 root root 10244 Jan 24 21:42 .DS_Store drwxr-xr-x 2 root root 0 Jan 23 18:39 'Software Engineering Daily' drwxr-xr-x 2 root root 0 Jan 23 18:43 'The British History Podcast' drwxr-xr-x 2 root root 0 Jan 23 18:46 'The Changelog' drwxr-xr-x 2 root root 0 Jan 23 18:51 'The History of Rome'
Cleaning it all up
The first step to clean everything up is to stop using the AFP service to make sure that you won’t mess up your files again by accident. So, before you do anything, make sure that you turn off the AFP service on your NAS and only connect via SMB going forward.
Converting file names from NFD to NFC
Before getting rid of the extensive .AppleDouble and ._* files, you want to make sure to convert all file names from NFD to NFC first, so that you can see and act on all files. Luckily there is a command called convmv, which is available on Linux and Mac (via Homebrew). Now here is where some of the black magic comes in. You will need to run the convmv command on a machine where you can actually see and read the NFD encoded files. In my case, that’s only on my Raspberry Pi as, as seen above, my Mac won’t be able to see or read these files regardless whether I access them via AFP or SMB. I think that any Linux machine would work but I haven’t tested that so I can’t say that for sure. The following steps are executed on my Raspberry Pi:
If you don’t have convmv available, install it via sudo apt-get install convmv.
You can run convmv in test mode (the default) which makes it easy to check which files will actually be converted, in case that you don’t trust blindly executing a command over all your files – I understand 🙂
A special thanks goes to James Chevalier’s little Gist on GitHub which made me discover convmv and eventually saved me from many more grey hair!
For the actual conversion, as the Gist suggests, you run convmv recursively through all folders (you may have to be root or use sudo), tell it to convert from utf8 to utf8 and also tell it that the target files will be in normalization form C, via the --nfc flag. Without the --notest flag convmv will run in test mode:
Pi $ convmv -r -f utf8 -t utf8 --nfc . Starting a dry run without changes... mv "./Clublife by Tiësto" "./Clublife by Tiësto" No changes to your files done. Would have converted 1 files in 1 seconds. Use --notest to finally rename the files. Pi $
The good news is, convmv detects the files (folders in this case) correctly during the dry run. So, let’s run it for real specifying the --notest parameter as well:
Pi $ sudo convmv -r -f utf8 -t utf8 --nfc --notest . mv "./Clublife by Tiësto" "./Clublife by Tiësto" Ready! I converted 1 files in 2 seconds.
And now let’s look at it again from the Mac SMB share:
$ cd /Volumes/Data-1/Music/iTunes\ Library/Podcasts/ $ ls -al total 320 drwx------ 1 ... ... 16384 12 Jan 22:23 . drwx------ 1 ... ... 16384 10 Jan 17:08 .. drwx------ 1 ... ... 16384 11 Jan 12:21 .AppleDouble -rwx------ 1 ... ... 6148 11 Jan 08:17 .DS_Store drwx------ 1 ... ... 16384 23 Jan 18:39 Clublife by Tiësto drwx------ 1 ... ... 16384 20 Jan 18:27 Software Engineering Daily drwx------ 1 ... ... 16384 20 Jan 18:27 Software Engineering Radio - The Podcast for Professional Software Developers drwx------ 1 ... ... 16384 20 Jan 18:27 The Changelog drwx------ 1 ... ... 16384 12 Jan 07:20 The History of Rome $ ls -al Clublife\ by\ Tiësto/ total 169895 drwx------ 1 ... ... 16384 Jan 23 18:39 . drwx------ 1 ... ... 16384 Jan 23 18:51 .. -rwx------ 1 ... ... 86953173 Jan 23 18:41 Clublife by Tiësto 438 podcast.m4a
HOORAY!!! We can see the folder and its file inside again! The really, really nice thing about convmv is that you can run it recursively from the top folder to convert all your files and folders at once!
._* files
Next, it’s time to get rid of the ._*files. Luckily, Mac does come with a dot_clean command: For each dir, dot_clean recursively merges all .* files with their corresponding native files according to the rules specified with the given arguments. By default, if there is an attribute on the native file that is also present in the . file, the most recent attribute will be used.
The dot_clean command is rather easy to use. By default, it will traverse into all subdirectories. The handy flags that you may want to set are the -s flag for following symbolic links, and the -m file which will delete the ._*file after the merge and even if there is no matching native file, i.e. if the ._* file is a rouge one hanging there by itself. Just connect via SMB from your Mac to your NAS and execute the command:
$ cd /Volumes/Music $ dot_clean -m -s . ... ... Merging directory Clublife by Tiësto merging ._Clublife by Tiësto 438 podcast.m4a & Clublife by Tiësto 438 podcast.m4a Using the most recent stored info Deleting: ._Clublife by Tiësto 438 podcast.m4a ... ...
.AppleDouble files
Last but not least it is time to delete the .AppleDouble folders. This is also an easy task thanks to find and its capability to execute another command on a matched file. So, all you have to do is to again head over to the SMB share on your Mac and execute the following command:
$ cd /Volumes/Music
$ find . -name ".AppleDouble" -exec rm -r {} \;
What the command will do is the following:
- Find any file called “.AppleDouble” in the current directory (
.) - Execute
rm -ron the found folder ({}will be substituted with the found folder name)
Conclusion
After some pain and quite a few “wasted” hours, I got my NAS back in order again. Of course, if you store all your files in the cloud somewhere you probably didn’t make it all the way down to here and that’s ok. However, if you are one like me and do store your files on your own NAS and you have Apple devices, I would advise you to stop using AFP and use other file sharing services instead. If you are all-in on Apple devices you may want to check out AFPS, I read that this one finally made the transition to the NFC normalization form as well. Yes, I know storage is cheap, but these files all over the place will one way or another become annoying at some point, even if it’s just when you want to copy them or make them available to someone else on non-Apple devices. And who knows, if not anything else, perhaps this article helped somebody to preserve his hair color for just a bit longer. 🙂
Gerald on IT 