A strategy for managing SD! jobs via AppleScript

[Syzygies]

My strategy for managing SD! jobs via AppleScript is this:

First, schedule each desired SD! job from within SD!, with any schedule that won't execute in the near future. This creates a settings package in the folder Library/Application Support/SuperDuper!/Scheduled Copies/, with a name such as Smart Update aOSX from OSX.sdsp. Move these settings packages to your own folder; this allows them to run independently of SD!'s built-in scheduling feature, as I noticed by reading their AppleScript code. Now go back in SD! and delete these scheduled jobs. The settings packages will survive intact; for the moment, SD! has no idea where they went.

Each of these settings packages contains a compiled AppleScript called Copy Job. One can Show Package Contents and then manually launch Copy Job, to run the corresponding SD! job. The log files for these jobs are also stored in these settings packages.

Alternatively, one can call these Copy Job scripts from within AppleScript. The calling script can then be run manually, or scheduled using cron or a GUI for cron such as CronniX. I describe below how I resolved the issues that arose for me in writing and testing such a script.

For a variety of reasons, I don't want my backup hard drives mounted when I'm not backing up to them. I also rotate hard drives as backup media, and I want a script to gracefully notice when media is unavailable, and move on. So I mount each volume as needed from within AppleScript, and eject it afterwards.

I found that I needed to eject volumes by telling the Finder, but this has the effect of ejecting all volumes on a given physical hard drive. This affects both one's strategy in partitioning one's drives, and one's code: One needs to avoid timing issues where ejecting the last volume used could also eject the next volume needed.

Along the same lines, all available volumes are mounted whenever a user logs in, so it is necessary to write another AppleScript (not shown, but similar) to eject unwanted volumes on login.

My AppleScript mount routine is written to be as robust as possible for my purposes, rather than to be as concise as possible:

Some code fragments that I found for mounting partitions were vulnerable to volume name issues. The egrep pattern matches an entire line of the diskutil list output, and sed removes the volume name, so awk can find the disk locator in a consistent column. This allows for volume names with embedded spaces, and volume names that contain one another. Still, it is poor form to have two identical volume names available, and this code will behave unpredictably in that event.

I also want to return the status of mount attempts as quickly as possible. mount executes two shell scripts, rather than combining these into one, in order to detect immediately if a volume name isn't even listed in the diskutil list output. Presumably, this is rotating backup media that is intentionally unavailable.

If SD! is open, my script waits for SD! to quit. Similarly, it waits for SD! to quit before proceeding to subsequent copy jobs. This code is based on the source code for the SD! Copy Job scripts.

Each top level copyJob call takes two arguments, the name of the destination volume, and a full AppleScript alias for the SD! Copy Job script to run. AppleScript aliases need to exist at compile time, but they have the advantage of automatically updating themselves when files are moved. The relative unreadability of this argument is a small price to pay for the convenience and feedback one gains.

Links to related threads:

Mounting an external volume before backup

An (Applescript) hack to automate mounting and unmounting external volumes

Here is my complete AppleScript for managing SD! jobs:

Note: I have revised this code at least once. I am preserving the sequence of posted versions in this thread, so that the discussion makes sense. However, I recommend modifying the latest version to your needs, after you understand the code.

on mount(diskname)
  tell application "Finder"
    if (exists the disk diskname) then return true
    try
      set a to do shell script "diskutil list | egrep '^ +[[:digit:]]+: +[[:graph:]]+ +" & diskname & " +[.[:digit:]]+ GB +disk[[:alnum:]]+$'"
      do shell script "diskutil mount `echo '" & a & "' | sed 's/" & diskname & "//' | awk '{print $5}'`"
    on error
      return false
    end try
    repeat 12 times
      tell application "System Events" to do shell script "sleep 5"
      if (exists the disk diskname) then return true
    end repeat
    return false
  end tell
end mount

on eject(diskname)
  tell application "Finder"
    if not (exists the disk diskname) then return true
    try
      eject disk diskname
    on error
      return false
    end try
    repeat 12 times
      tell application "System Events" to do shell script "sleep 5"
      if not (exists the disk diskname) then return true
    end repeat
    return false
  end tell
end eject

on isSuperDuperRunning()
  tell application "System Events"
    return exists process "SuperDuper!"
  end tell
end isSuperDuperRunning

on waitForSuperDuper()
  repeat while isSuperDuperRunning
    tell application "System Events" to do shell script "sleep 5"
  end repeat
end waitForSuperDuper

on copyJob(diskname, scriptalias)
  if mount(diskname) then
    waitForSuperDuper
    run script alias scriptalias
    waitForSuperDuper
    eject(diskname)
  else
    display dialog "Run Backups: \"" & diskname & "\" is unavailable" giving up after 30
  end if
end copyJob

on run
  copyJob("aOS9", "User:Users:ad:Scripts:Smart Update aOS9 from OS9.sdsp:Copy Job.app:")
  copyJob("aOSX 10.3.9 min", "User:Users:ad:Scripts:Smart Update aOSX 10.3.9 min from OSX 10.3.9 min.sdsp:Copy Job.app:")
  copyJob("aOSX 10.4.4 min", "User:Users:ad:Scripts:Smart Update aOSX 10.4.4 min from OSX 10.4.4 min.sdsp:Copy Job.app:")
  -- copyJob("aUser", "User:Users:ad:Scripts:Smart Update aUser from User.sdsp:Copy Job.app:")
  -- copyJob("aOSX", "User:Users:ad:Scripts:Smart Update aOSX from OSX.sdsp:Copy Job.app:")
end run

[dnanian]

Thanks, Syzygies! Hopefully this'll give people even more ideas for what they can do with the SuperDuper! AppleScript interface and built-in Copy Job script...

[Syzygies]

Here is a revised version of my code to selectively mount and unmount volumes, and call SD! scripts unattended. I've sorted out various issues, and it now works fine as a cron job; I rely on it nightly.

Overall, the code has been improved by never telling anything to the Finder. 'unmount' now only unmounts the given volume, without ejecting other volumes on the same disk. It is possible to have an AppleScript such as this that calls 'diskutil' crash on rare occasions. I don't know why, and I'm not sure that I'm actually doing anything wrong. (SD! itself sometimes uses 'disktool' instead of 'diskutil', despite Apple's efforts to wean us from it; perhaps Dave can explain why?)

Also on rare occasions, I'd run a prior version of this script unattended, and one of the volumes failed to unmount. I considered what I would have done manually had I been there, and wrote a wrapper 'unmountVol' to do exactly that. It looks like massive overkill in the calm light of day, but its precautions are entirely harmless for unattended operations.

I leave diagnostic journaling on at all times, as SD! does. If anything odd happens, this is invaluable. Note that all shell script commands are logged, together with any errors returned, even for commands that always return an error. (We wouldn't want to miss it, if one of them returned a different error.)

All forums have a high percentage of lurkers. I google and lurk various places to pick up code fragments to try, and I consider this to be the primary value of posting code on the web; I don't like to use unsupported code that I haven't rewritten myself. To be conservative, if no one else has posted here that this code worked for them, proceed with caution in modifying it for your needs.

This is an AppleScript library file, Backups Library.scpt. Opening it opens the script in the Script Editor:

property logfile : "/Volumes/User/Users/ad/Scripts/logs/log"

-- initialized, journal, shell, wait : utility routines

property new : true

on initialize()
  if new then
    set logfile to logfile & (do shell script "date '+ %Y-%m-%d.txt'")
    set new to false
  end if
end initialize

on journal(message)
  local wasnew, s, f
  set wasnew to new
  if new then initialize()
  set s to (quoted form of message)
  set f to ">>" & (quoted form of logfile) & "; "
  if wasnew then
    do shell script "echo " & f & "echo " & s & f & "echo " & f
  else
    do shell script "echo `date '+%H:%M:%S'` ' ' " & s & f
  end if
end journal

on shell(command)
  -- use full paths for any commands not in path "/usr/bin:/bin"
  local e, n
  journal("      % " & command)
  try
    do shell script command
  on error e number n
    journal("      ! " & (n as string) & " " & e)
    error e number n
  end try
end shell

on wait(interval)
  shell("sleep " & interval as string)
end wait

-- ismounted, mount, unmount : mounting and unmounting volumes

on ismounted(volname)
  try
    -- if not found, egrep exits with a non-zero status
    shell("/sbin/mount | egrep 'on /Volumes/" & volname & " \\(.+\\)$'")
  on error
    journal(" - " & volname)
    return false
  end try
  journal(" + " & volname)
  return true
end ismounted

on mount(volname)
  local a
  journal("mount: " & volname)
  if not ismounted(volname) then
    try
      -- if not found, grep and egrep exit with a non-zero status
      set a to shell("/usr/sbin/diskutil list | egrep '^ +[[:digit:]]+: +[[:graph:]]+ +" & volname & " +[.[:digit:]]+ GB +disk[[:alnum:]]+$'")
      set a to shell("echo '" & a & "' | sed 's/" & volname & "//' | awk '{print $5}'")
      shell("/usr/sbin/diskutil mount " & a & " | grep '^Volume " & a & " mounted$'")
    on error
      return false
    end try
  end if
  return true
end mount

on unmount(volname)
  local n
  journal("unmount: " & volname)
  if ismounted(volname) then
    try
      -- unmount always generates error "Volume failed to unmount", even when successful
      shell("/usr/sbin/diskutil unmount '/Volumes/" & volname & "'")
    end try
  end if
end unmount

-- mountVol, unmountVol : patient, stubborn wrappers for mount, unmount

on mountVol(pause, tries, volname)
  local n
  journal("mountVol " & volname)
  repeat with n from 1 to tries
    wait(pause)
    if mount(volname) then
      repeat with n from 2 to 8
        wait(n)
        if ismounted(volname) then return true
      end repeat
    end if
  end repeat
  return false
end mountVol

on unmountVol(pause, tries, volname)
  local n
  journal("unmountVol " & volname)
  repeat with n from 1 to tries
    wait(pause)
    unmount(volname)
    repeat with n from 2 to 8
      wait(n)
      if not ismounted(volname) then return true
    end repeat
  end repeat
  return false
end unmountVol

-- mountVolumes, unmountVolumes : apply mountVol, unmountVol to lists

on mountVolumes(pause, tries, volnames)
  local status
  journal("mountVolumes")
  set status to true
  repeat with volname in volnames
    if not mountVol(pause, tries, volname) then set status to false
  end repeat
  return status
end mountVolumes

on unmountVolumes(pause, tries, volnames)
  local status
  journal("unmountVolumes")
  set status to true
  repeat with volname in volnames
    if not unmountVol(pause, tries, volname) then set status to false
  end repeat
  return status
end unmountVolumes

-- SuperDuper!

on SuperDuperJob(volname, scriptalias)
  local wasmounted
  journal(scriptalias as string)
  journal("")
  set wasmounted to ismounted(volname)
  if wasmounted or mountVol(30, 4, volname) then
    journal("SuperDuperJob: " & scriptalias as string)
    try
      run script scriptalias
    on error e number n
      journal("script error: " & (n as string) & " " & e)
    end try
    if not wasmounted then unmountVol(30, 4, volname)
  end if
  journal("")
end SuperDuperJob

-- top level routines

on mountBackups()
  mountVolumes(0, 1, {¬
    "aOSX", "aOS9", "aOSX 10.4.4 min", "aOSX 10.3.9 min", "aUser", ¬
    "bOSX", "bOS9", "bOSX 10.4.4 min", "bOSX 10.3.9 min", "bUser", ¬
    "FireWire A OSX", "FireWire A"})
end mountBackups

on unmountBackups()
  unmountVolumes(0, 1, {¬
    "aOSX", "aOS9", "aOSX 10.4.4 min", "aOSX 10.3.9 min", "aUser", ¬
    "bOSX", "bOS9", "bOSX 10.4.4 min", "bOSX 10.3.9 min", "bUser", ¬
    "FireWire A OSX", "FireWire A"})
end unmountBackups

on runBackups()
  journal("runBackups")
  
  SuperDuperJob("aOS9", ¬
    alias "User:Users:ad:Scripts:Smart Update aOS9 from OS9.sdsp:Copy Job.app:")
  SuperDuperJob("aOSX 10.3.9 min", ¬
    alias "User:Users:ad:Scripts:Smart Update aOSX 10.3.9 min from OSX 10.3.9 min.sdsp:Copy Job.app:")
  SuperDuperJob("aOSX 10.4.4 min", ¬
    alias "User:Users:ad:Scripts:Smart Update aOSX 10.4.4 min from OSX 10.4.4 min.sdsp:Copy Job.app:")
  SuperDuperJob("aUser", ¬
    alias "User:Users:ad:Scripts:Smart Update aUser from User.sdsp:Copy Job.app:")
  SuperDuperJob("aOSX", ¬
    alias "User:Users:ad:Scripts:Smart Update aOSX from OSX.sdsp:Copy Job.app:")
  
  SuperDuperJob("aBack", ¬
    alias "User:Users:ad:Scripts:Smart Update Sources image from User.sdsp:Copy Job.app:")
  SuperDuperJob("aBack", ¬
    alias "User:Users:ad:Scripts:Smart Update Users image from User.sdsp:Copy Job.app:")
end runBackups

This is a typical AppleScript application file that makes a call from the above library, Run Backups.app. Once set up, it seldom needs further editing. Opening it runs the script:

set BackupsLib to (load script file "User:Users:ad:Scripts:Backups Library.scpt")

tell BackupsLib
  runBackups()
end tell

This is a typical journal entry, from the file log 2006-01-22.txt:

unmountVol aBack

07:11:37         % sleep 0
07:11:37   unmount: aBack
07:11:37         % /sbin/mount | egrep 'on /Volumes/aBack \(.+\)$'
07:11:37    + aBack
07:11:37         % /usr/sbin/diskutil unmount '/Volumes/aBack'
07:12:04         ! 1 Volume failed to unmount
07:12:05         % sleep 2
07:12:07         % /sbin/mount | egrep 'on /Volumes/aBack \(.+\)$'
07:12:07         ! 1 The command exited with a non-zero status.
07:12:07    - aBack

[Syzygies]

Hi. Has anyone tried this script?

[dnanian]

Any particular reason for the graphical sig, Dave? Tracking?

[Syzygies]

Sure, most forums are 99% lurkers, and the code took a bit of work, just curious. I see how many people read my post, and they can study source to figure out who I am.

Here's a simple Perl script that can extract this kind of information from my apache server. No idea how portable the code is, but it works for me:

#!/usr/bin/perl
use warnings;
use strict;

# apachelog.pl  --  Perl script for extracting reports from /usr/local/apache/logs/access_log
#
# Copyright (c) 2006 Dave Bayer
# Subject to the terms and conditions of the MIT License.

(my $help = <<'EOF') =~ s/^: ?//mg;
:
: Usage:
:
: 	apachelog.pl count path
:		count distinct accesses
:		(in all, how many distinct visitors?)
:
: 	apachelog.pl who path
:		list distinct accesses by caller
:		(who were the visitors?)
:		
: 	apachelog.pl when path
:		list distinct accesses by count for each date
:		(for each date, how many distinct visitors?)
:		
: 	apachelog.pl what path
:		list distinct accesses by ascending count
:		(for each distinct visitor, what did they access?)
:
:	apachelog.pl first path
:		list distinct first accesses by ascending count
:		(for each distinct visitor, what did they access first?)
:
: Examples:
:
:	apachelog.pl what ''
:	apachelog.pl when bayer/coffee
:
EOF

my $get = ' "GET /(?:~|%[0-9a-fA-F]{2})*';

sub openlog {
	open LOG, "</usr/local/apache/logs/access_log" or die "unable to read /usr/local/apache/logs/access_log\n";
}

sub convertdate {
	local $_;
	($_) = @_;
	
	s/(J|j)an[a-z]*/01/;
	s/(F|f)eb[a-z]*/02/;
	s/(M|m)ar[a-z]*/03/;
	s/(A|a)pr[a-z]*/04/;
	s/(M|m)ay[a-z]*/05/;
	s/(J|j)un[a-z]*/06/;
	s/(J|j)ul[a-z]*/07/;
	s/(A|a)ug[a-z]*/08/;
	s/(S|s)ep[a-z]*/09/;
	s/(O|o)ct[a-z]*/10/;
	s/(N|n)ov[a-z]*/11/;
	s/(D|d)ec[a-z]*/12/;
	s:(\d*)/(\d*)/(\d*):$3-$2-$1:;
	$_;
}

sub who {
	my ($match) = @_;
	my (%who, $addr);

	openlog();
	while (<LOG>) {
		/$get$match/ or next;
		($addr = $_) =~ s/^(\S*)\s.*\n/$1/;
		$who{$addr}++;
	}
	close LOG;
	%who;
}

sub when {
	my ($match) = @_;
	my (%when, $date, $addr);

	openlog();
	while (<LOG>) {
		/$get$match/ or next;
		($date = $_) =~ s/^[^[]*\[([^:]*):.*\n/$1/;
		($addr = $_) =~ s/^(\S*)\s.*\n/$1/;
		$when{convertdate($date)}{$addr}++;
	}
	close LOG;
	%when;
}

sub what {
	my ($match) = @_;
	my (%what, $file, $addr);

	openlog();
	while (<LOG>) {
		/$get$match/ or next;
		($file = $_) =~ s/^.*?$get($match\/?[^&#%\?\/\s]*).*\n/$1/;
		$file =~ s/\/$//;
		($addr = $_) =~ s/^(\S*)\s.*\n/$1/;
		$what{$file}{$addr}++;
	}
	close LOG;
	%what;
}

sub first {
	my ($match) = @_;
	my (%start, %first, $file, $addr);

	openlog();
	while (<LOG>) {
		/$get$match/ or next;
		($file = $_) =~ s/^.*?$get($match\/?[^&#%\?\/\s]*).*\n/$1/;
		$file =~ s/\/$//;
		($addr = $_) =~ s/^(\S*)\s.*\n/$1/;
		if (defined $start{$addr}) { next; }
		$start{$addr} = $file;
	}
	close LOG;
	foreach $addr (keys %start) {
		$first{$start{$addr}}++;
	}
	%first;
}

sub count_cmd {
	my %who = who(@_);
	printf "%d\n", scalar keys %who;
}

sub who_cmd {
	my %who = who(@_);
	foreach my $addr (sort keys %who) {
		print "$addr\n";
	}
}

sub when_cmd {
	my %when = when(@_);
	foreach my $date (sort keys %when) {
		printf "%8d  %s\n", scalar keys %{$when{$date}}, $date;
	}
}

sub what_cmd {
	my %what = what(@_);
	my @list;
	foreach my $file (sort keys %what) {
		push @list, sprintf "%8d  %s\n", scalar keys %{$what{$file}}, $file;
	}
	print sort @list;
}

sub first_cmd {
	my %first = first(@_);
	my @list;
	foreach my $file (sort keys %first) {
		push @list, sprintf "%8d  %s\n", $first{$file}, $file;
	}
	print sort @list;
}

my %run = (
	'count' => \&count_cmd,
	'who' => \&who_cmd,
	'when' => \&when_cmd,
	'what' => \&what_cmd,
	'first' => \&first_cmd,
);

my ($do, $match) = @ARGV;
if ($#ARGV != 1 or not $run{$do}) { 
		print $help;
		exit(1);
}
$run{$do}($match);

[dnanian]

Just not entirely necessary -- the view count for the thread is right in the index. This thread, for example, has been viewed 147 times or so...

[Syzygies]

Yeah, but I can't remember that. I wrote the script for tracking my own web pages, but it tells me date information here. If accesses fall off to zero, I know no one is using search, and I can bump the thread or forget about the project.

[dnanian]

Fair enough, Dave... just wanted to point out the existing feature.

[Gil]

Thank you for your source code contribution, Dave.

I want you to know that I'm reading your code to improve a similar script I wrote to mount the target disk before and unmount it after creating a bootable copy. My previous script used another clone-making product, but I decided to try SD instead because of its slicker AppleScript interface.

And this forum! What a trove!

--Gil

[bethri]

A few hiccups with aliases, etc. but I'm watching SuperDuper! running as I type. Hopefully this will become my primary solution, scheduled with Cronnix.

For the record, I really think this should be a standard part of SuperDuper!, or at least a Shirt-Pocket supplied and tested "advanced solution" hidden in there somewhere... I'm reasonably confident in my ability to make things like this work, but would prefer not to have to!

Ben

[dnanian]

I'm confused, Ben. Scheduling in v2.0 is built right in -- there's no need to use CronniX.

[bethri]

I know scheduling is a feature, but this entire thread is devoted to arranging a scheduled mount/unmount feature.

I bought SuperDuper! to provide an instant-recovery clone on the second internal drive on my G5 – sometimes I can't afford any downtime, and a recent hard drive failure caused me major problems, in spite of my effective data backups on external drives.

But the doubled apps under Open With... and the possibility of mistakenly accessing data on the clone make a mount/unmount feature a practical neccessity for me.

Like I said, I still run regular backups (been using Deja Vu happily for years), but I want SuperDuper! to provide me with a 3AM "mount the drive, do the backup and unmount the clone" routine. I want the clone to exist, but I don't want to see it until I need to...

Ben

[dnanian]

Given one of the solutions on the Forum, you can certainly do that, without CronniX. Just modify the scheduling AppleScript to mount the volume before the backup, and unmount it after. Those run before SD itself runs, and should give you the solution you're looking for.

[bethri]

I appreciate that, and thanks for the replies.

I think you should seriously consider writing up a suggested method, perhaps even a sample script, even if you don't explicitly support it ("advanced users only - use at own risk" type of thing) to save people grubbing around for the best way. I, for one, would be extremely grateful.

I'd like to change SD!s default behaviour as little as possible, and when it comes to adding elements into the AppleScripts, I'm never certain exactly where and what is best.

Thanks again, Ben