Critical Alert for users of CheckBook, CheckBook Pro, and other applications saving to FAT volumes

This issue affects a small number of our users, but we need you to read through just to be sure you’re not in that group. Thank you!

 

Update 9/13/2021: Great news! macOS 11.6, released earlier today, seems to have addressed the fundamental issue with FAT, ExFAT, and some SMB volumes. We’re testing each application in the list below to be sure, but our own applications, Apple Preview, and Tumult Hype all appear to be working as designed, so we think we’re out of the woods. Thank you all for hanging in there!

Even though Apple seems to have resolved the issue and we’re breathing easier over here, we want to leave a note for the record: The vast majority of Apple developers have used NSFileManager or copyfile(). Copying files to or saving files on FAT, ExFAT, and SMB volumes is not uncommon, especially since almost every external drive comes pre-formatted for Windows. However this thing slipped through the cracks, it took Apple 7.5 weeks to move on it with two minor macOS updates in the meantime. In the 1.75 months since we told Apple about it, something close to a hundred of our users – that we know of – lost data and needed help restoring a backup while we spent an insane amount of time we didn’t have (and our families didn’t want to give us) testing and coding around the issue. Apple never owned up to the issue. There’s no mention of it in the macOS 11.6 release notes. No one from Apple ever communicated with us about our bug reports – and there’s no sign any engineer even read them. If we caused an issue like this, then handled it as Apple has, we wouldn’t be able to live with ourselves – and we’d be out of the Splasm business, to boot. So our faith in Apple’s support for anything besides APFS or Mac OS Extended, or SMB to those on recent macOS hosts, got taken to the woodshed, you might say, and we just won’t rely on them anymore. In the near future, all our applications will warn you when you try to use other formats or flavors of SMB until you tell us not to. We value everyone’s data and time that much.

Update 8/24/2021 (Further updated 9/13/2021):

The applications (and known actions) affected by this issue are now:

• Audiobook Builder 2.1.4 (new/save) – Workaround coming in 2.1.5 Fixed in macOS 11.6
• CheckBook 2.6.20 and earlier (save/backup/move) – Update to 2.6.21 now Fixed in macOS 11.6
• CheckBook Pro 2.6.20 and earlier (save/backup/move) – Update to 2.6.21 now Fixed in macOS 11.6
• Apple Books 3.2 (drag audiobook to volume) Fixed in macOS 11.6
• Apple Calendar 11.0 (export calendar archive) Fixed in macOS 11.6
• Apple Configurator 2.14 (move) Fixed in macOS 11.6
• Apple Contacts 13.0 (export contacts archive) Fixed in macOS 11.6
• Apple Photos 6.0 (export unmodified originals) Fixed in macOS 11.6
• Apple Preview 11.0 (move) Fixed in macOS 11.6
• Apple QuickTime Player 10.5 (move) Fixed in macOS 11.6
• Apple TextEdit 1.16 (save to existing file) Sorta fixed in macOS 11.6 (temporary files aren’t removed, document may save but eventually can’t be because it’s “locked”)
• OmniOutliner 5.8.5 (move) Fixed in macOS 11.6
• OmniGraffle 6.6.2 (move) Fixed in macOS 11.6
• Pixelmator 3.9.8 (export – except for PDF) Fixed in macOS 11.6
• Pixelmator Pro 2.1.3 (new/save/move/export) Fixed in macOS 11.6
• Tumult Hype 4.1.7 (save/move) Fixed in macOS 11.6

Update 8/16/2021: This issue also affects QuickTime Player on macOS 11.5 and later. Duplicate a video or audio file, open it with QuickTime Player, use the File menu’s Move To… option to move the document to a FAT volume, then check the size of the file on the FAT volume and it should be zero bytes. The same move operation to an APFS or Mac OS Extended volume works as expected.

Update 8/13/2021: CheckBook and CheckBook Pro 2.6.21, now available, work around NSFileManager and copyfile() on macOS 11.5 and later to preserve your data in all known situations where these methods can fail. Thank you all for hanging in there! We strongly recommend you store your documents on either APFS or Mac OS Extended volumes, or access them over a network with SMB or AFP (though there’s only a whiff of AFP left in Big Sur so SMB it is).

Update 8/11/2021: macOS 11.5.2 hasn’t addressed this issue. Stay tuned.

Update 7/30/2021: We’ve developed a workaround that’ll do for now. It’s in testing now and will be out during the week of August 8th.

Update 7/27/2021: Though release notes didn’t mention the issue, we installed macOS 11.5.1 anyway to see if Apple slipped in a fix under the radar (that’s an old-time Apple developer pun). No change.

Update 7/26/2021: One user reports this issue may also occur on ext4 and XFS volumes over NFS. We haven’t tested these filesystems or NFS but the symptoms appear the same. It could be that any volume or share mounted over NFS will exhibit the issue, so our advice is to stay away from NFS altogether until a fix is in place.

 

Original post from 7/23/2021:

macOS 11 Big Sur made it to version 11.5 this week and, along for the ride, comes a few showstoppers for a relatively small group of our CheckBook and CheckBook Pro users:

• If you’ve installed the macOS 11.5 update and your regular CheckBook document lives on a volume formatted as FAT16, FAT32, or ExFAT, DO NOT open that document or your data may be wiped from that particular copy of your document.
• Using the File menu’s Backup command and saving to a FAT16, FAT32, or ExFAT volume on macOS 11.5 will create an empty backup.
• Using the File menu’s Move To… command and moving the open document to a FAT16, FAT32, or ExFAT volume on macOS 11.5 will erase the data in the document.

These issues have the same root cause, what appears to be a bug in the most common method macOS developers use to copy files. This method reports success for each copy while, in fact, the final result is a completely empty file that lacks the data, attributes, and extended metadata of the original. Because macOS tells CheckBook the file copy succeeded, there’s no way for the application to know it failed – so there’s no way for you to know, either. We believe this bug also affects a number of other applications, such as Apple’s own Preview. If you’d like to see it in action, duplicate an image file on your Mac so you don’t lose the original, open the duplicate with Preview, go to the File menu, click the Move To… menu item, then select a destination on a FAT16, FAT32, or ExFAT volume. The moved file will end up zero bytes in length.

We’ve sent a bug report to Apple and reached out to another major developer to let them know this affects their applications, hopefully giving us an additional vector to an Apple engineer. In the meantime, let’s protect your CheckBook data:

• As we said, if your document lives on a FAT16, FAT32, or ExFAT volume, do not open it or you could lose your data in that copy of your document. Instead, copy the document to your Documents folder and double-click the copy in Documents. We’d recommend staying away from storing your CheckBook documents anywhere but the Documents folder, but we are particularly down on using USB thumb drives for long-term storage. They tend to not hold up as well as any internal storage device and we’ve seen too many fail over the years to not send out a warning like this every now and then.
• Do not use the File menu’s Backup or Move To… commands with a FAT16, FAT32, or ExFAT volume as the destination. While you could copy your document in the Finder, opening the copy from there would lead to data loss in that copy of the document. If you need to backup your document, select a folder in iCloud Drive or another cloud storage location. If you need to move your document to another storage device, or you need to keep a local backup, but don’t have anything but a drive with FAT16, FAT32, or ExFAT volumes, you can copy any files you need to save to another volume, then reformat a volume as APFS or Mac OS Extended. Remember to copy the files you need to save first. Read up on how to erase or reformat a volume here – and please, please pay attention to that first step.

We’re now looking for a way around this because a fix from Apple could take a while. The worst option is to simply forbid interacting with FAT16, FAT32, or ExFAT volumes. Much as we’d like to not worry about this kind of thing in the future – this isn’t the first or second or third quirk that’s popped up with FAT volumes and these days they’re feeling more and more like stowaways in the mothership’s steerage – completely nixing support for them is a bit more nuclear than we’d like. We’ll find a solution soon and let you know when we do.

You might not know if your document is on a FAT16, FAT32, or ExFAT volume to begin with, so here’s how to tell: Locate your document in the Finder, Command-click the name of the window to reveal a menu, click the second-to-last item in that menu, go to the File menu near the top left of your screen, click the Get Info menu item to reveal a window with a lot of details about the file, and look for the Format in the General section near the top of the window. If Format reads “MS-DOS (FAT16)“, “MS-DOS (FAT32)“, or “ExFAT” you know for sure your document lives on a volume with the issue.

Why is your volume formatted as FAT16, FAT32, or ExFAT when these are usually something you’d see on a Windows machine? The Mac’s come a long way from the nightmare of the mid 90’s, and the iPhone and iPad have done their part to make Apple a household name, but the desktop is still a Windows world and almost every drive out there comes set up for that reality. macOS makes it easy with basic support for these formats, but you really, really, really need to reformat to APFS or Mac OS Extended whenever possible. Read up on how to erase or reformat a volume here.

Also, if you’re wondering why this affects some applications but not others, it’ll help to understand every developer does things a little differently. Data can be read and written a number of ways, and the techniques a developer uses can vary depending on the application and the type of data. For example, you’ll see the issue with the Move To… command in Apple’s Preview but not in Apple’s Numbers. From what we saw while playing around with Numbers today, we believe it’s set up to do a full save on the destination volume, instead of a copy, when moving a document from one volume to another. That likely involves creating files from scratch, then saving data into each file, as needed, rather than a straight copy from one volume to another – and that route doesn’t appear to present any issues.

Please let us know at support@splasm.com if you get to this article after a mishap and you need help restoring a backup. If you don’t have a recent backup, CheckBook’s automatic backups, created about once every seven days, will save the day. Here’s how to find and restore these automatic backups:

1. Open CheckBook or CheckBook Pro.
2. Go to the Help menu while holding down the Option key on your keyboard and click the Go to CheckBook 2 Folder menu item.
3. Inside the CheckBook 2 folder, open the Backups folder.
4. You may see several items inside the Backups folder.  Open the folder whose name matches your CheckBook document’s name.  You’ll see a list of compressed backups.
5. Each compressed backup file includes the date it was created in its name.  Look for the most recently dated file and double-click it.  A file with the same name as the compressed backup but an icon of a sheet of paper and a tiny CheckBook icon will appear.  This is the actual backup file.
6. Move the backup file to your Documents folder (or wherever you’d like to keep your document going forward – as long as it’s not on a FAT volume) and double-click it.