This is obviously not a good user solution, because no one is going to boot to Recovery, turn off SIP, back up, boot back to Recovery, and turn SIP back on. Thinking they would do so is kind of crazy.2
Breathe: it’s OK
Fortunately, SuperDuper handles this situation. When you make a bootable copy, the required keys are copied without requiring any special steps. Even though Smart Update cannot access those files, it preserves them on the backup so they’re available if needed.
What that means for you
It’s always a good idea to make a bootable backup, even if you have no plan to boot from it. Update it with Smart Update, update the OS when SuperDuper indicates you should, and you’ll be in good shape should you need to restore…even if you need to restore your keychains on a different Mac.3
System Integrity Protection—basically, a process that prevents any programs, other than Apple-authorized ones, from performing certain activities. You can read more about it here. ↩︎
This all may change in the future. The Tech Note hedges a lot in the Warning and Important sections, which suggests to me that they realize this isn’t a great situation. ↩︎
In fact, I’ve just verified that migrating from a SuperDuper bootable backup to a different Mac restores the encrypted keychains as you’d hope and expect. ↩︎
That’s right, a new version of SuperDuper has arrived. It has some new stuff!
I get email
Every so often, a user will ask for more run logging: some sort of master list of SuperDuper copies, so they can reference it during personal contemplation time. I’ve usually suggested that they write a little shortcut to do what they want: after all, that’s what I added the feature for—to allow for extension, and inclusion of behavior that might be user-specific (and, also, to allow people to share Shortcut tips-and-tricks they might develop for SuperDuper).
Making an example
That’s a good approach, I think, but sometimes people need to start with something a bit more elaborate, and tweak that, rather than generate something from scratch.
With that in mind, SuperDuper 4.0.9 includes a new Shortcut—Log SuperDuper Copy to Dashboard.
What does it do?, you might ask? Well, pretty much what’s on the tin.
After each copy, it adds a row of information to the sheet that includes various bits of information about the copy that just ran. At the top, you’ll find a list of the different Copy Jobs that have run (there are typically multiple runs per Job, of course, but only one entry at the top).
You can focus on the runs of one Job by unchecking the Show column for the others, or compare them by checking other Jobs.
If you have so many Copy Jobs that the upper table overlaps the graphs2, you can just drag them to relocate where they are on the sheet.
Extend away
Of course, there may be other things you want to do with the data that’s logged, and once the spreadsheet is there, you can change the various charts, add to them, and do what you’d like. The Shortcut will continue to update the runs.
I wouldn’t do that
I’ve had way too many users write in, thinking that their license was only good for a day. They thought that because the Registration window said “Registered for 1 day” the first day they were registered. It would, of course, say “Registered for 2 days” the next day, etc.
Now, I didn’t think this was unclear, but when I get a bunch of emails about something like that, and people think that I’m somehow selling them a 1-day license (folks, I would never do that), it’s time for a change. So, now it says “You’ve been registered for 1 day”, etc.
I hope that’s clearer.
But that’s not all!
Well, that’s mostly all. I’ve also improved some additional very obscure network issues, improved progress display in some edge cases, and individual job phases now remember whether you’ve expanded them or not. So, if you like to see the detailed stats for a given job, that’ll be remembered next time you run.
To enjoy the new version, use the internal updater, or
SuperDuper is, obviously, a serious tool that has an important task: making a reliable copy of data from one place to another. Its UI, for the most part, is all business, and tries to make the process as clear as possible.
Those, clearly, are the big things. Table stakes, as it were.
Fun & Frivolity
Keeping that in mind, I’ve still tried to put “pleasing” things into the general design, with a focus on making things a bit more convenient, clear, informative, or easier to look at.
Things like animations and transitions (which were in SuperDuper Classic, too); updating progress with different options, communication via both bars and pulsing; more playful ways of choosing options; dragging around jobs; the sparkles to draw your attention for the first time1; things like that.
But, yeah—while I don’t want it to be boring, SuperDuper isn’t ever going to be a Delicious Generation app.2 Respect to whose who can be that playful with their stuff, though!
Fit & Finish
There are serious “little” things, too, of course. And that’s where I generally focus.
For example, SuperDuper preserves the destination drive icon and Spotlight state even across erase operations and replications.
It’s a little thing, but I know that many users identify their drives by their icons, and so it’s nearly as important as keeping the name the same.
Similarly, even though the Spotlight state is kept on the destination drive, I take pains to ensure that it’s preserved across the various copy permutations. It’s pretty annoying when Spotlight starts indexing a backup you told it to ignore…and can result in finding something on a backup rather than a source.
Behavioral things, too, like mounting drives for the user, and unmounting them after (if they started that way)…there are a lot of little touches in there that just try to predict what a user would want, without needing any settings, or thought…it just works.
Details & Obscurity
SuperDuper 4.0.8, released today, deals with some of those cases, and handles some relatively obscure cases better than before.
For example, users who have Samba setups on Linux boxes that include the fruit extensions may be preserving macOS permissions, and you can get into situations where a file with no permissions becomes permanently read-only (except server-side).
Obscure, but handled.
Speaking of obscure, the onboarding sequence is now sensitive to the macOS version you’re running, and name-checks the background activity section based differently depending on what it is. Because it regressed from macOS 14/15’s “Allow in the background” to macOS 26/27’s “Background App Activity” for reasons.
There’s a lot more, but you get the idea: continue to polish-polish-polish.
and install (or, better, use the built-in updater).
As always, enjoy, and thanks for using SuperDuper.
Eventually, I’ll get around to my original plan for this—having the “Hi!” hand come alive, swoop down, and point at the link, along with some other fun behaviors…the storyboard I did for this amuses me…but it’s a lot of work for a funny-but-useful bit of goofiness.
Maybe if I do it they’ll let me into the Hall of Delicious Generations. ↩︎
The original version of Apple’s Time Machine had its UI designed by Mike Matas, the original—and probably the best—Delicious Generation designer. I was lucky enough to have him design the original SuperDuper Classic icon back when he was a very young man, and it was quite clear he was both incredibly talented and going places.
I’ve been asked a lot of times about when Smart Update becomes ⚡️Turbo Smart Update, and where the setting is in SuperDuper.
The answer is: there’s no setting. When possible, SuperDuper automatically does a ⚡️Turbo Smart Update.
So, why doesn’t it always ⚡️Turbo?
The rules of the game
Here’s the deal: ⚡️Turbo depends on a number of factors, and you can improve the likelihood it will happen by using a separate Copy Job for each source/destination pair. SuperDuper needs to maintain a relationship between the source, destination, copy method, etc., to ensure that ⚡️Turbo is reliable.
If you change the copy method or the rules, then the relationship between the last backup and the next one has changed…and ⚡️Turbo is disallowed (until the next run).
Turbo also won’t run after a One Time macOS Update, or an Erase, then copy until the second Smart Update.
System effects
SuperDuper also relies on the system’s own mechanism for tracking changes.1
Many (many) years ago, there was a program that I think was called Synk. It installed a kernel extension that tracked all file system changes and wrote a database into user space that the copy program used to make “smarter” copies.
I remember thinking that was incredibly dangerous at the time. You could change the volume when Synk wasn’t running, or on another system, and Synk would be none the wiser. So, while clever, it wasn’t a reliable method of doing this.
Years later, Apple added fsevents, a system-supported way of doing the same kind of thing. I evaluated it when it was introduced, and it was just too flaky to rely on…it wasn’t worth the risk.
Since then, Apple has made changes to fsevents, making it far more reliable as a source of “truth” about volume status.
Now, it’s no panacea. fsevents doesn’t tell you a ton, and it consolidates its change sets pretty regularly, so it’s not like you can just say “here’s what happened” and replay a bunch of events on the copy and get a good result.
But, as a way of “thinning” the number of files and folders you have to visit, when used intelligently, it works well.
Unless/Until it doesn’t.
Sorry, Charlie
Unfortunately, sometimes fsevents can’t keep up with the number of changes being made to a volume, or needs to flush its internal change database. So even though SuperDuper knows what to do, fsevents can veto our attempt annd say that it has, basically, lost the history—and a full scan is needed.
This can also happen when a drive disconnects unexpectedly, or is unplugged without fully ejecting it.
And, of course, it’s erased when you erase.
Faster, Pussycat! Kill! Kill!
So there you go: ⚡️Turbo Smart Update works on the second and subsequent runs of the same job, when the job itself hasn’t changed in a way that invalidates ⚡️Turbo…as long as the system doesn’t invalidate its change database for the volumes involved.
In the event a full scan is necessary, SuperDuper uses regular Smart Update. And since that’s more than twice as fast as before, it’s less painful than it could be.
Of course, there’s a ton of SuperDuper Special Sauce™️ involved here, too. ↩︎
What does that mean? It means “only use documented calls”. It means “work around problems using normal processes, but not things that aren’t documented”.
It means that even when I know why something isn’t working, if there’s no documented fix…well, I don’t do that…unless I absolutely have to.
The past is prologue
I’ve been in this situation before. When Catalina was released, and the whole boot process was different, I spent all summer figuring out how to make a bootable copy: volume groups, roles, firmlinks, and so much time going through verbose boot logs (literally taking movies of them, since they went by so fast, and single stepping through) until I figured out the issue.
It was rough (but also fun).
And, obviously, none of that was documented. But I did it anyway…to benefit the users of SuperDuper.
A new dilemma
I’ve known for a while why drives aren’t showing up as bootable under Golden Gate. And it’s been reported as a bug, as mentioned in my previous blog posts. But I didn’t want to release the workaround I had, because it relies on something undocumented.
Specifically, one…single…bit.
Cat leaves bag
I had hoped that the Terminal command I provided in a previous post would be sufficient until an official fix came out, but it’s just too difficult for “regular” people to execute…and the last thing I want to do is put people in a state of panic when they’re trying to recover from a failure.
But the tipping point was that, yesterday, a beta of 27.2 (!) was released, without a fix. Which means I don’t know when, or if, a fix is going to be implemented.
The thing is, the copy is actually fine (as proven by the original workaround & “proof of life” screenshot). The real issue is that Golden Gate’s startup drive code is filtering its selections by looking for something that neither asr nor bless is setting. In fact, they’ve never set this value, which is why, under Golden Gate, your previously bootable backups look like they’re no longer bootable.
With those factors in mind, I’ve reversed course and have decided to go ahead and work around the issue as safely as I can.
Note that this fix will automatically disable itself when the real fix is (hopefully) released. In all the internal testing that I’ve done during the Golden Gate Beta, it’s worked fine and seems perfectly safe.
What you need to do (nothing)
The fix will be applied automatically when you make your next backup, even if it’s a Smart Update, under all macOS versions supported by SuperDuper 4. Other than that, there’s nothing special you need to do.
As I mentioned in the previous post, there’s a bug in Golden Gate that is not showing proper external copies as bootable, even though they’re made correctly, with Apple’s own replicator.
The workaround, if you must, is listed there (you should also be able to install Golden Gate over the backup from Recovery). For those who doubt (you know who you are), this is a screen shot of my Mac Studio1 booting off a SuperDuper copy made to a JBOD PRO-BLADE Station (I’ve also tested a MacBook Neo and a MacBook Pro).
NOTACHANCE is obviously not my real serial number, but I thought this would be funnier than just blacking it out. ↩︎
Post-Labor Day (on which I still labored), it’s been a busy week here at Shirt Pocket HQ.
I’ve been adding a few little features and fixes to SuperDuper, improving behaviors with regard to images, Cloud Storage, and wrapping up Golden Gate support for its release, which will likely be a few hours after this posts.
Golden Gate, in my view, looks to be one of the more reliable releases we’ve had in years, even in its 27.0 release. I wouldn’t hesitate to encourage people to install it, without waiting for the (inevitable) 27.1 release. I even recommended it to the curious during the very stable and reliable Beta period. Shockingly painless.
That said, there’s always a but in these releases, you know?
The big but(t—sorry) in Golden Gate 27.0 is a SuperDuper issue (that is actually an Apple issue): backups, even though a bootable backup is properly configured to boot, they aren’t showing up as choices in either the startup disk preference pane or the startup picker. (FB24210103, reported quite a while ago, Apple friends.)
Fortunately, unlike when this happened before, there’s no error during copy. But, even though the end result is bootable1, it just doesn’t appear.
Remember: a drive doesn’t have to boot to be fully restored. But there are many scenarios where boot is essential, and I’m hoping Apple will fix this soon.
Ecosystem: engage!
I was delighted to see that there’s a third-party app called OpenBackup that offers mobile notification when backups run for both SuperDuper 4 and SuperDuper Classic. It works on both iOS and Android.
I always hope, when I implement things like Shortcuts support, that other developers will make use of them to do cool things2…it’s nice to see it happening.
In the meantime
I continue to be surprised and delighted by how well the rollout of SuperDuper 4 is going. Thanks to everyone who has registered, to those who have recommended it to their friends, and to the folks that have reviewed it.
But now, it’s time to update SuperDuper using the built-in updater, or
As always, thanks for using SuperDuper!—your support is appreciated.
You can prove this to yourself if you really want to by booting into Recovery, reducing security with the Startup Security utility, and then using bless --mount /Volumes/Backup-Volume --setBoot in Recovery’s Terminal. But don’t unless you really have to. ↩︎
It’s always nice to leave some space for fellow developers (not to mention users) to build on what you’ve done: we did this with Classic, with support for Growl and extensive AppleScript support. ↩︎
SuperDuper feels like impossibly good software. Read his post announcing version 4 to start, then catch up on the rest of the blog to get a feeling for how much work he’s put into SuperDuper 4. Just extraordinary.
…
I recommend SuperDuper 4 wholeheartedly.
You probably don’t need to read this. Unless you want to.
Process, Process, Process
So, how do you solve the problem of testing a product like SuperDuper?
As you might expect, it’s pretty difficult. Of course, you can pin individual cases of UI and copy behavior (there are over 2,000 individual tests against the SuperDuper code base, which is about 250K lines of Swift, Objective-C and C): those are run every time I make a change to the code base…to protect against regression of things that are known.
Theory vs Practice
But this type of testing only gets you so far. I can model various situations that I’m trying to cover, but models just simulate reality (even if the cases are real).
Real backups are more complicated, and quirkier.
They’re often to different drive types. Those drives may be formatted in different ways, have their own layouts, failure modes, fragmentation levels, and connections.
So, not only do I run a few systems of different eras as test harnesses with various drives attached (simultaneously running Time Machine, too), my main development system (currently a first generation Mac studio) has a huge number of different drives attached.
These include (but are not limited to—I also have thumb-drives set up for specific tests, and assorted other SSDs and HDDs):
2x OWC Express 1M2 80G, one with an OWC SSD, one with a SanDisk SSD
1x LaCie D2 Thunderbolt 2 drive, connected via an Apple TB3/4->TB2 adapter, chained to
1x LaCie 2Big Thunderbolt 2 drive (RAID)
1x LaCie 2Big Desk/Dock (RAID)
1x SanDisk Pro USB-C SSD
1x SanDisk Professional TB4 SSD
1x SanDisk USB Transport with its own PRO-BLADE
1x SanDisk PRO-BLADE Station with 4 PRO-BLADE SSDs (JBOD)
1x OWC 4M2, configured as JBOD, with 2xM.2 SSDs
1x Synology 2415+ configured as iSCSI via ATTO’s initiator
4x SMB mounts from the Synology
Note that some of these drives are connected to an Anker TB5 dock, to make sure that’s covered, too; some are connected to an Apple Studio Display, to get a hub in there; many are daisy chained, so I deal with that, too.
Yeah, that’s a lot of drives (and a significant investment), and doesn’t count the separate Mac systems with their own external storage that are running tests all the time.
Those drives don’t do a lot on their own, of course: SuperDuper is configured to run copies against them, of various types, all day long (and all night).
If there are any failures, I’m notified immediately and can analyze the logs to determine the cause, how it was handled (since failures can happen to users), and whether it was my issue or a problem with hardware.
In addition, while all of the copies are checked separately for accuracy and under-copying, a number of tests also check for idempotency, to ensure there’s no thrashing or incorrect/over copying, and then there are a series of timed tests that make sure there hasn’t been a performance regression.
And on top of that, there are the aforementioned 2,000 automated tests in the full SuperDuper 4 test suite, with more manual tests as well (which I need to figure out how to automate one of these days).
Even So…
But even with all that, I can’t truly replicate every situation that can occur in the field. And even with all that I can screw up and miss things.
That’s where other testers come in. As I’ve mentioned previously, there were a large number of external testers during SuperDuper 4’s development, rolled in over time to ensure a series of “fresh testers” (and test cases) were always encountering the code.
When outside users run into a problem, I have to figure out what happened beyond just “Hey, Dave, last night’s copy failed”. So SuperDuper does extensive logging (if you want to be driven mad, feel free to look at the ocean of logging with log show in Terminal): when a user submits an issue, it includes both SuperDuper’s visible log, and a filtered-for-relevance couple of seconds before and after the failure from the system log, so that I have enough information to analyze the error.
When appropriate, verified problems turn into another test case, to make sure they’re both fixed and won’t break again. And then a test build is sent to the user to ensure the situation is covered in its original setting.
Bugs in the field are pretty rare things (and are often the result of configuration differences that I haven’t created in-house). But given a report, it’s nice to be able to fix them…and prevent them from happening again.
In Sum
In the end, there’s no magic, and there are no shortcuts: copy testing is done by making (and comparing) tons and tons of copies, small and large, under both “random” and “known” situations; UI testing is done by pinning behavior and ensuring things react as expected.
Combine that with (hopefully) proper design, extensive experience, and careful implementation, and you end up with SuperDuper 4 which, despite a number of small updates post-release, is (thankfully) proving to be very reliable in the field…the goal of any design, development, and testing process.
You need only look at Apple’s installer, sitting at “one minute left” for an hour, to understand that time estimates of I/O-based operations are very hard.
They seem like they’d be easy, though, don’t they?1 You just take the size, figure out how fast things are going, and do some math, and bingo-bango-bongo, there you go! Perfect estimate!
Similarly, you take something like the Blackmagic Disk Speed Test and look at its read/write speed and think “geez, there’s the speed, what’s the problem”?
I mean, sure: if you were writing a single file (or some big files), to a blank destination, on a consistent connection. Bytes stream at a constant rate, and it really is just a matter of a few quick observations and a little math.
Real World Cases
But, no. It’s not that simple. I tried the best I could to get things as right as possible, and the estimates are still, sometimes, wrong.
Sometimes very wrong.
Let me walk you through the general (albeit highly simplified) approach:
SuperDuper walks the drive and calculates about how much data has to be copied (while, at the same time, it’s copying)
Once it knows, it looks as the rate copying is going
It then shows you a time
Sounds familiar, right? Well, it goes further than that:
It’s actually showing a moving average that’s taking speed variances into account
It’s determining the approximate ratio of “up to date” vs “need to copy” files
It’s figuring out how long it’s taking to copy small vs big files
It’s taking previous runs against this content into account
…and so much more…
Even with all that, sometimes it’ll be wrong.
I just don’t know, until I get there, whether something is going to need to be copied. If it doesn’t, and it’s large, the estimate will be too long. If it does, and the OS is slow to copy that particular file, or that size of file, it’ll be too short.
You get the idea. Hopefully.
“Your Estimates Suck, and So Do You”
For years, I resisted putting in anything like this. It’s just too hard to give an accurate number. But as I shaped the algorithms I’m using in SD4, I figured I could get close, most of the time, in most situations, and it would be worthwhile.
And I think it is. Usually. Sometimes it’s wrong…but it gets better the more you copy that particular content.
It’ll never be perfect, though (although I get very close for replication, which was very tricky, given Apple’s replicator gives virtually no information about what it’s doing, and certainly no speed or even size info), and I try to indicate that with lots of hedging and “weasel words” like approximately and about and estimate to tell you that, well, an estimate is an estimate. It’s an inexact science.
So, if the estimate I’m presenting annoys you, you may want to consider taking the Car Talk “engine check light” approach: put a piece of black electrical tape over it.
Or just close the window. The copy will continue in the background and finish when it finishes.
Problem solved!
Or so everyone tells me when they complain about them. ↩︎
