josselson's posterous

Your backup product/service should back up your data completely and correctly. This of course includes each file’s contents, but on the Mac it also includes many types of “metadata”. What are “metadata”? The general definition is “data about data”. In the case of Mac files, metadata include the file’s size, its modification date, its owner, who’s allowed to read and/or write it, who’s allowed to run it (if it’s a program), and many others.

Unfortunately, many online backup products, such as Mozy, Carbonite, Backblaze, CrashPlan, and Dropbox, don’t back up and restore all the metadata correctly. Fortunately, Nathan Gray, an independent researcher at Caltech, has created a tool called Backup Bouncer that tests every type of Mac metadata. I’ve run the Backup Bouncer tests through Mozy, Carbonite, Backblaze, CrashPlan and Dropbox, and the results range from perfect to really really bad; a table with links to Backup Bouncer results for those products is on the Arq product page.

Backup Scenarios

But are all the metadata types really important for your situation? Well, it depends. Let’s analyze a few scenarios:

Backing Up Photos and Videos

Let’s say you just want to back up your JPEG photo files and your AVI movie files from your home directory. These files typically won’t have any more than the bare minimum of metadata. They’ll be owned by you with default permissions. Any backup product should be able to restore those files well enough; even if they restore the files as owned by you with default permissions, they’ll end up (by coincidence) to be the same as the originals.

One caveat even in this simplest of scenarios is the modification dates of files. If the backup program you’re using can’t even restore modification and creation dates correctly (Backblaze, Mozy and Carbonite can’t), when you restore, it’ll look like all your files were just created. This can be confusing and/or disconcerting in the case of text files and word processing documents. For some files like photos and videos, the “date taken” is stored within the file itself so iPhoto shows the photo with the correct date, but in the Finder the dates will still be off.

Backing Up Applications

How about backing up applications? On the Mac, apps are actually “bundles”, which are folders with the “bundle bit” set. The folders have a fixed structure and contain the actual executable file as well as lots of resources (icons, configuration files, etc). The executable file must have “executable permission”; otherwise OS X silently refuses to run it.

So if you use Dropbox for backing up your applications, for example, and you restore a deleted app (or you use it for sync and sync an app to another Mac), Dropbox will restore the app’s files but won’t set the executable permission. When you double-click on the restored app, nothing happens!

If you’re sufficiently technical you might check the system log and see an error like this:

com.apple.launchd.peruser.501[144] ([0x0-0x36b36b].com.haystacksoftware.iphotosync[18955]): posix_spawn(“/Users/stefan/Dropbox/iPhotoSync.app/Contents/MacOS/iPhotoSync”, …): Permission denied

You might then look at the binary’s permissions using Terminal and see it’s not marked as executable:

1 $ ls -l iPhotoSync.app/Contents/MacOS/
2 total 2272
3 -rw-r--r--  1 stefan  staff  1161472 May 28 15:10 iPhotoSync

You’ll then set the binary’s permissions correctly to be able to start the app:

1 $ chmod +x iPhotoSync.app/Contents/MacOS/iPhotoSync
2 $ ls -l iPhotoSync.app/Contents/MacOS/
3 total 2272
4 -rwxr-xr-x  1 stefan  staff  1161472 May 28 15:10 iPhotoSync*

But that’s pretty geeky!

Backing Up Downloaded Files

When you download a file using Safari, by default it goes to your Downloads folder. When Safari puts it there, it adds an “extended attribute” called “com.apple.quarantine” to the file indicating it’s potentially unsafe because you downloaded it from the internet (more on extended attributes later). When you attempt to open it, you get a dialog like this:

When you click “Open”, OS X removes the “com.apple.quarantine” extended attribute from the app.

If you restore these files from a backup using software that doesn’t restore extended attributes, you won’t get this security warning. For more details on extended attributes, see below.

Backing Up Time Machine Backups

Time Machine uses extended attributes heavily. Here’s a list of the extended attributes on one of my Time Machine backup folders:

1 $ xattr 2010-05-11-080152/
2 com.apple.backup.SnapshotNumber
3 com.apple.backup.SnapshotVersion
4 com.apple.backupd.SnapshotCompletionDate
5 com.apple.backupd.SnapshotStartDate
6 com.apple.backupd.SnapshotState
7 com.apple.backupd.SnapshotType

This is the list of extended attributes on the “Macintosh HD” folder within that backup folder:

1 $ xattr 2010-05-11-080152/Macintosh\ HD
2 com.apple.backupd.SnapshotVolumeFSEventStoreUUID
3 com.apple.backupd.SnapshotVolumeLastFSEventID
4 com.apple.backupd.SnapshotVolumeUUID
5 com.apple.backupd.VolumeBytesUsed
6 com.apple.backupd.VolumeIsCaseSensitive
7 com.apple.metadata:_kTimeMachineNewestSnapshot
8 com.apple.metadata:_kTimeMachineOldestSnapshot

Restoring these files from backup using a backup program that doesn’t restore extended attributes isn’t going to work too well; Time Machine wouldn’t find its expected metadata.

Backing Up Alias Files

Alias files are a somewhat esoteric feature of OS X. If you control-click a file in Finder and choose “Make Alias” from the pop-up menu (or Command+Option+drag the file to another location), you’ll get an alias file that points to the file you had clicked on. The alias acts as a stand-in for the original, and works even if you move the original to another location (on the same filesystem).

Alias files depend on “resource forks” (a type of extended attribute) to function properly. If you restore an alias file from backup using a backup program that doesn’t restore extended attributes, the restored file will be incomplete and unusable. For example, I made an alias to a PDF file in my Dropbox, waited for it to back up, deleted it, and then restored it via Dropbox. When I double-clicked on the restored PDF alias, it opened in TextEdit and looked like this:



That’s not usable.

Backing Up Symbolic Links

A symbolic link is a special type of file that refers to another file. Some backup programs are unable to differentiate between a regular file and a symbolic link; they “follow” the link instead of backing up the link itself. So if you restore a folder tree containing a large directory as well as a symlink to that directory, you’ll instead get 2 copies of the large directory.

Some sync programs like Dropbox consider this a feature: even though Dropbox will only backup/sync the “Dropbox” folder, you can place symbolic links within it that point to other outside folders, and those will be backed up as well.

In my opinion a real backup program should accurately back up and restore the contents of your disk, not its interpretation of those contents.

If you’re backing up data from an application that depends on having symbolic links to a file instead of multiple copies of a file, failing to recreate the symbolic links could be a big problem. CrossOver is one program that creates many symbolic links and some complex ownership settings as well.

Backing Up Files with Finder Flags

OS X can store many “flags” with a file. The following flags can be read and written through the “Get Info” window in the Finder (control-click a file and choose “Get Info” from the pop-up menu):

• extension is hidden
• locked
• “Stationery Pad” file: see this excellent TUAW post for details

Additional Finder flags include:

• alias file
• custom icon
• has bundle
• invisible
• busy

“Has bundle” is a bit that makes your iPhoto Library look like a file instead of a folder. So if your backup program can’t correctly restore Finder flags, when you restore your iPhoto Library it’ll look like a regular folder. You can’t double-click on that to open it in iPhoto. You’ll have to launch iPhoto with the Option key held down and navigate to that folder; after you do this once, iPhoto resets the “has bundle” bit on the folder. You’ll find the same thing with a Pages document or a Numbers document, or any bundle that’s meant to look like a single file in the Finder.

“Invisible” is a bit that makes a folder invisible to the Finder. I’m not sure which apps (if any) use it.

“Custom icon” is a bit that Safari sets on a file while it’s being downloaded (the Finder shows a progress bar on the file icon).

“Busy” is a bit that indicates a file is busy or incomplete.

If your backup program can’t restore these flags, things may look a bit different when/if you have to restore from backup.

Backing Up Files With Creator Codes and Type Codes

As described in this TidBITS article, creator codes and type codes are 4-letter codes attached to a file that specify what type of file it is and which application created it. Before OS X this was the way MacOS figured out which application to open when you double-clicked on a file. When OS X arrived (with its Unix base), a conflict arose between using creator codes and using filename extensions.

As of Snow Leopard (OS X 10.6) creator codes are completely ignored for opening files. So whether you should care that your backup program correctly restores creator codes and type codes depends on whether you use 10.6, among other things.

OS X still uses creator code and type code for copying files however, even in 10.6. If you copy a large file from one place to another, the incomplete copy receives a type code of “brok” and creator code of “MACS”; Finder resets these to empty values when the copy is completed.

More on Extended Attributes

Extended attributes are small pieces of metadata associated with files and folders. In the “old days” (before OS X) MacOS used “resource forks” heavily; in OS X, a resource fork is just one type of extended attribute.

Using Terminal you can see which files have extended attributes (they get a ‘@’ next to the permissions):

1 $ ls -l Knox-2.0.1.zip
2 -rw-r--r--@ 1 stefan  staff  5210452 May 25 15:37 Knox-2.0.1.zip

The ‘xattr’ utility lists the extended attributes of a zip file I downloaded from the internet:

1 $ xattr Knox-2.0.1.zip
2 com.apple.metadata:kMDItemWhereFroms
3 com.apple.quarantine

In this example, the file was downloaded with Safari. The “com.appl.metadata:kMDItemWhereFroms” extended attribute contains where the file came from (in a binary format readable by OS X):

1 $ xattr -pl com.apple.metadata:kMDItemWhereFroms Knox-2.0.1.zip
 2 com.apple.metadata:kMDItemWhereFroms:
 3 00000000  62 70 6C 69 73 74 30 30 A2 01 02 5F 10 3B 68 74  |bplist00..._.;ht|
 4 00000010  74 70 3A 2F 2F 61 77 73 2E 63 61 63 68 65 66 6C  |tp://aws.cachefl|
 5 00000020  79 2E 6E 65 74 2F 61 77 73 2F 64 6D 67 2F 4B 4E  |y.net/aws/dmg/KN|
 6 00000030  4F 58 2F 45 6E 67 6C 69 73 68 2F 4B 6E 6F 78 2D  |OX/English/Knox-|
 7 00000040  32 2E 30 2E 31 2E 7A 69 70 5F 10 26 68 74 74 70  |2.0.1.zip_.&http|
 8 00000050  3A 2F 2F 61 67 69 6C 65 77 65 62 73 6F 6C 75 74  |://agilewebsolut|
 9 00000060  69 6F 6E 73 2E 63 6F 6D 2F 64 6F 77 6E 6C 6F 61  |ions.com/downloa|
10 00000070  64 73 08 0B 49 00 00 00 00 00 00 01 01 00 00 00  |ds..I...........|
11 00000080  00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00  |................|
12 00000090  00 00 00 00 72                                   |....r|

The “com.apple.quarantine” attribute is an indicator that the file was downloaded by Safari:

1 $ xattr -pl com.apple.quarantine Knox-2.0.1.zip
2 com.apple.quarantine: 0000;4bfcfa48;Safari;A17E9A1C-F662-4DF0-95AA-18F44791DAFC|com.apple.Safari

What To Do

If possible, choose a backup program for your Mac that correctly backs up everything. Then you don’t have to worry. Arq is one backup program that backs up and restores everything correctly.

This entry was posted on Tuesday, June 1st, 2010 at 7:06 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

Interesting read from the makers of Arq, a great backup application for Mac OSX utilizing Amazon S3.

unraveled

Tweetie Reloaded: An Interview with Loren Brichter

Tweetie 1 for iPhone set the standard for a simple yet powerful Twitter client, garnering the attention of both A-list tweeters and Apple, winning an Apple Design Award at the Worldwide Developers Conference. Tweetie 2 raised the bar once again, providing a slew of new features wrapped inside an elegant interface.

One feature common to all Twitter clients is “reload,” allowing users to load the latest tweets. Tweetie handled this particular feature differently from the beginning. I talked to Tweetie’s creator, Loren Brichter, about how this feature evolved and his perspectives on designing gestural interfaces.

Joshua Kaufman: Tapping a button to reload was generally accepted as a standard within the iPhone interface. What made you explore the refresh interaction in Tweetie 1?

Loren Brichter: Necessity is the mother of all invention. Looking back to Tweetie 1, I was in this position where my navigation bar was already filled up. On the left side was the “Accounts” back button which would take you back to the accounts list (Tweetie 1 was the first app with a sensible multiple accounts implementation). On the right side was the compose button.

So where did the reload button go? Other apps of the day simply put the reload button in the upper right corner. This was acceptable for single-account apps, but for apps that handled multiple accounts, those apps had to go through some nasty contortions and shove the account switcher UI into some other non-optimal place. So I thought about it, and asked: what was “reload”… really? In the context of Twitter, it really meant “load newer”. And when you loaded newer tweets in your tweet stream, where did they go? At the top of your list.

So I put the reload button at the top of the list. As you scrolled up, reading newer tweets, eventually you’d get to the top of the list and bam… there’s a reload button, it just slid into view exactly when you needed it. You tap it and it’s replaced by the newer tweets. It’s intuitive and doesn’t waste prime screen real estate (corners of the screen) until you actually need it.

JK: How did you arrive at the pull down and release to refresh interaction in Tweetie 2?

LB: Tweetie 2 simply took this idea from Tweetie 1, that reloading was simply “loading newer”, and “loading newer” put new messages at the top of the list… and activated the action based on a finger motion that you were already doing. Why make the user stop scrolling, lift their finger, then tap a button? Why not have them continue the gesture that they are already in the process of making? When I want to see newer stuff, I scroll up. So I made scrolling itself the gesture.

The gesture is only half the battle though, you need appropriate feedback. Once the reload is activated, the scrollable area of the list actually changes to leave the feedback UI in-place (rather than bouncing offscreen). Without this part, the UI is unintuitive. And once the loading is complete, the UI makes itself disappear.

JK: Using this interaction in Tweetie 2 reminded me that gestural UI is still young and full of possibilities. Are there other common interactions that you see being improved by new gestures?

LB: Of course. But you need to be careful. The reason that I think the pull-down-to-refresh works so well in Tweetie 2 is because it’s discoverable and explanatory. It’s discoverable because you already know how to scroll a list, and as you scroll up, when you get to the top — bam — the UI reveals itself and you go “whoa, what’s that?”. It explanatory because once you start tugging down there is some great UI feedback, actual text that provides instructions as you interact. It’s not like it pops up some modal dialog with instructions on how to use it, the UI itself is the instruction. Once you learn it you simply ignore the text and can look at the graphical arrow which gives appropriate feedback on what you need to do.

Now, I think you can split gestures into two categories. One is of the pull-down-to-refresh kind. These are gestures that are discoverable and explanatory. The other kind of gestures are like tapping-the-status-bar-to-scroll-to-the-top, or swipe-to-delete (or swipe-to-reply in Tweetie). These gestures you won’t discover on your own except by accident. These are not discoverable, and they are not explanatory.

This second class of gestures can exist (in my opinion) because they are not the only way to accomplish a goal. In the case of tapping the status bar, users already know how to scroll to the top manually. It’s slower, but it’s possible. In the case of swipe to delete, users already know they can tap on a message and then tap the trash button. So knowing the gesture isn’t necessary.

So when you’re inventing new gestures, it’s important to think about whether the gesture is required to use the app. If it’s the only way to accomplish a goal, you better be sure it’s discoverable and explanatory without needing to read a manual. If it’s the other kind of gesture, go nuts!

We're really excited about the upcoming release of iPad, and we definitely plan to support it. Having said that, Simplenote for iPad won't be available immediately. We're going to spend some time working with the hardware first to make sure everything flows as expected.

After we do some more testing in the coming weeks, we'll submit a basic, working version that should hopefully become available within a month or so. This version won't have much extra in the way of features just yet, but with time it will evolve based on your feedback, just like we did with Simplenote for iPhone.

We know that a lot of you plan to buy an iPad, and we appreciate your patience while we anxiously await our own devices. In the meantime, the iPhone version can be run on your iPad if you'd like, and our web app in landscape mode fits quite nicely on the screen inside Mobile Safari as well.

To those of you receiving your iPad soon: have fun!

This is how it is supposed to be done.