Concerned about hard links

[steve]

I'm a new owner of SuperDuper, and I've been doing some testing under Mac OS X 10.3.9.

One discovery has me a bit puzzled and concerned. After doing a simple clone operation to create a bootable volume, I see that hard links are apparently not preserved. This can be observed in Terminal by using the "ls -li" command. Source files (not directories) with a link count higher than 1 inevitably show up on the target with a link count of 1.

Examining inode numbers adds more evidence that hard links are lost. Instead of multiple directory entries sharing a single inode number, each directory entry ends up with a unique inode number. That means that the original file, which had multiple directory entries pointing to it, has been duplicated so that each directory entry points to its own copy.

This seems like a waste of space, but more importantly, could lead to software malfunctions if hard links are assumed to exist when files are updated. Do we know that all those hard linked files are never manipulated by Mac OS X, so there's no need to worry? Couldn't third party software use hard links and get tripped up by this too?

Hard links are especially abundant in /usr/share/...; there are hundreds and hundreds of them. (A simple test directory is /usr/share/zoneinfo/US.)

For comparison I tried cloning using the asr command in Terminal, and also Carbon Copy Cloner (which uses the ditto command). They both play it safe and preserve hard links.

I'm new to SuperDuper, and I'm no expert on unix either, but this seems like an issue worth exploring. Anyone more savvy about these things have any thoughts?

Steve

[dnanian]

Steve,

You are, indeed, right: we unroll hard links -- but from what I can tell, ditto does too, sometimes. Here's how I tested: I created a disk image named "test", and -- from /usr/share/zoneinfo -- I did:

ditto US /Volumes/test/US

When I did this, all the hard links were unrolled, kind of as I would expect.

ASR is rather different, since it does a very low-level block copy, and doesn't deal with files. It also can't subset the drive (which, clearly, can't always guarantee the 'original' inode will be copied, so if you don't unroll you end up with a bad copy).

ditto only seems to copy hard links if the file with the original inode is copied. I tested this by touching "c" and hard linking "c.copy" to it. I then copied both files to the root of the test volume, and they were preserved as hard links.

We've not encountered a problem, yet, with what we're doing, but I agree that it's something that we should take a look at -- and we shall. Thanks for reporting it.

[steve]

Dave,

Thanks for the quick response. I hope your time at WWDC was useful and enjoyable!

I should have been more clear about how I used /usr/share/zoneinfo/US for testing. The hard linked files in that directory aren't hard linked to each other, so copying that directory by itself doesn't seem very revealing. In my tests, I cloned the entire boot volume, then checked /usr/share/zoneinfo/US on the target to quickly see what happened to link counts. Sorry for the confusion.

Also, I did not test ditto directly. I simply used Carbon Copy Cloner, and observed that hard links were preserved. According to CCC's log, it uses a series of ditto commands like this:

sudo /usr/bin/ditto -rsrcFork /'Applications' '/Volumes/<mytarget>'/'Applications'
sudo /usr/bin/ditto -rsrcFork /'usr' '/Volumes/<mytarget>'/'usr'
etc.

I would imagine that as long as all hard links to a particular inode are within one of the copied folders (like usr), ditto wouldn't need to unroll any of them. (Apparently that condition holds in Mac OS X's hard links, because I found no reduction in link counts within the CCC generated clone.)

Regarding asr, I used it like this:

sudo asr -source / -target /Volumes/<mytarget> -erase -verbose

In this simple application (source not an image), it apparently does not do a low-level block copy. My evidence is that modification dates of all symbolic link files (on the target) were updated during asr's operation. Ditto does the same thing. (I don't understand why in either case.) Note that asr and CCC/ditto took almost exactly the same amount of time to complete the clone, suggesting they're using the same copying method.

Incidentally, to fully explore the results of a cloning operation, I generate complete volume listings (sudo ls -lonTRA > ~/desktop/list.txt) of source and target, and compare them with FileMerge. It's time consuming, but interesting. That's how I discovered the loss of hard links with SuperDuper, and confirmed their complete preservation with asr and CCC. The symbolic link updating phenomenon with asr and CCC is a real nuisance, because it causes huge numbers of uninteresting differences.

Thanks for looking into the hard links issue. I'd love to see a FileMerge source/target listing comparison that shows only expected differences, rather than hundreds of disquieting extraneous ones caused by quirks of the cloning method.

Steve

[dnanian]

Understood, Steve. We're looking into the case. I'm not sure ditto preserves the links if they go across "copy operations" (in other words, if in your example something in /usr is linked to something in /Applications, it likely won't have its link preserved).

It's a bit tough, because unlike a soft link a hard link can't be created until the target exists, and there's no real "target" as such since they're all really the same file.

In any case, we're looking at it. Thanks again.

[dnanian]

(Note, though, that -- again -- this has not proved to be a significant issue in real world use. It's "wrong", though, which is why we're looking at how to fix it, but it's never caused any problems because hard links are pretty rare in an OSX installation.)

[steve]

Dave,

Thanks for your reassurance that unrolling hard links has not caused any trouble. I agree with your observation that ditto probably can't preserve links "across copy operations".

For fun, I checked to see how many hard links are in my OS X installations. My method was to find non-directory files with non-zero link counts, using commands like those below.

cd /Volumes/whatever
sudo find . ! -links 1 ! -type d | wc -l
sudo find . ! -links 1 ! -type d -ls > ~/desktop/'hardlinks.txt'

Hard links are pretty rare in 10.2.8, only 266 total.
/Library = 3
/System = 3 (these 3 are linked to the 3 in /Library)
/sbin = 4
/usr = 256

But 10.3.9 has lots more, 2839 total.
/Library = 3
/System = 3 (these 3 are linked to the 3 in /Library)
/usr/bin = 22
/usr/share/man = 1112
/usr/share/terminfo = 1478
/usr/share/zoneinfo = 221

I hope you find these results useful. It's been fun digging around in OS X's file system, but I'll quit bugging you now. Thanks for a fine product, and amazingly responsive support!

Steve

[dnanian]

Thanks for the additional information. We're actively pursuing a solution.

[garoomph]

DarwinPorts maintains hard links to all files it installed in /opt/local below /opt/local/var/db/dports/software. As SD unrolls these while backing up, the clone may be significantly larger than the source. This could cause problems when restoring a drive that was nearly full before it failed.

At the moment I can't think of applications relying on changing file contents through hard linking (as opposed to the usual "unlink on overwrite").

To me it generally feels wrong that a "clone"'s filesystem should be differ from its source in such a fundamental way.

[dnanian]

SuperDuper! does not unroll hard links: we replicate the hard links on the destination. (These messages are from a long time ago: the issue with hard links was corrected many, many moons ago.)

[garoomph]

Originally Posted by dnanian

SuperDuper! does not unroll hard links

Really? Always?

Of course you can't answer for DarwinPorts, but in the following (ls -i) sitecopy was installed before a HD->Clone->HD restore and mplayer afterwards:

436867 /private/opt/local/var/db/dports/software/sitecopy/0.16.0_0/opt/local/bin/sitecopy
426257 /opt/local/bin/sitecopy

426249 /opt/local/bin/mplayer
426249 /private/opt/local/var/db/dports/software/MPlayer/cvs-20060408_0+darwin_8+macosx+real/opt/local/bin/mplayer

[dnanian]

If it did unroll something, it's a bug: we specifically check for and recreate hard links.

I just ran a simple test: I created two disk images, "linktest1" and "linktest2". I touched a file and then created a hard link to it:

Daves-15-MacBook:/Volumes/linktest1 dnanian$ touch foo
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ln foo foo2
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ls -l
total 0
-rw-r--r--   2 dnanian  dnanian  0 Jun 19 17:44 foo
-rw-r--r--   2 dnanian  dnanian  0 Jun 19 17:44 foo2
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ls -i
19 foo  19 foo2

I then used "Backup - all files" to copy to "linktest2":

Daves-15-MacBook:/Volumes/linktest1 dnanian$ cd
Daves-15-MacBook:~ dnanian$ cd /Volumes/linktest2
Daves-15-MacBook:/Volumes/linktest2 dnanian$ ls -i
19 foo  19 foo2
Daves-15-MacBook:/Volumes/linktest2 dnanian$

So, it seems to work... checking other known hard links seem fine here, too...