If you’re on macOS 11.5 or later and have a CheckBook document on a thumb drive or an external drive that hasn’t been reformatted for APFS or Mac OS Extended, this is the update for you. You might’ve read our chronicle of a major data loss issue between macOS 11.5 and FAT volumes, the short of which is NSFileManager and copyfile(), used by literally almost every application when touching a filesystem on your Mac, can no longer copy files directly to FAT volumes with any guarantee of success, leaving completely empty files in their wake. Since thumb drives and external drives almost always come factory-formatted for FAT or ExFAT, and the Mac’s basic support of these filesystems doesn’t disable write access or even warn users of the risks, perhaps because this recent development isn’t by design and Apple hasn’t read the lengthy bug report we’ve filed, more than a few of our users have experienced the joy of restoring a backup. This doesn’t just affect CheckBook, though. Just about any application that tries to copy or move a file to a FAT volume, rather than creating a new file and writing into it, could see that operation fail. We’ve witnessed the issue wipe data when you use the Move To… feature in Apple’s Preview and at least a couple of other applications from another major developer, and that was in the limited time we had for testing before we knuckled down to work around the glitch. 2.6.21 abandons these APIs on macOS 11.5 or later when the destination isn’t an APFS or Mac OS Extended volume or is shared over something besides SMB or AFP, creating brand new files from scratch instead of the more streamlined methods of copying or moving. Even with this workaround, our strong recommendation is to reformat your FAT volumes and make sure you’re sharing network volumes over SMB or AFP as soon as possible. This release has a few other tweaks, including a fix for an issue that could prevent the Accounts list from redrawing after an Account is unlocked, so you won’t want to miss it.
In short, we believe documents belong in the Documents folder, temporary projects or routine items you’re absolutely certain you won’t accidentally “clean up” can go on the Desktop, volumes formatted for anything besides APFS or Mac OS Extended may cause great pain so reformat your thumb drives or external drives, your best bet is to keep your documents on local storage, Time Machine + cloud backup is the bee’s knees, and iCloud Drive has a gotcha you need to consider every time you turn it off. Read on for some color!
Documents belong in Documents? Really? Really.
We strongly recommend you save your documents in your Documents folder. It’s out of the way enough to protect your files from cleanup “accidents” and a breeze to get to when you know how. From the Finder, go to the Go menu at the top of your screen, above all open windows, and click the Documents menu item. Or look to the sidebar on the left of an open Finder window, where you’ll usually see a Documents item you can click. Or, our favorite, use the Documents keyboard shortcut: Command (⌘) – Shift – O (the letter).
We don’t usually recommend you save your documents on the Desktop. It’s a great place for temporary projects, and can also work for items you use every day – but there’s a potential gotcha: the Desktopbegs you to clean it. With cleaning comes accidents. Those accidents can cost you a single file or a whole passel. It really, really hurts to lose a file with all of your financial history or a folder full of video you imported from – then deleted from – a camera a while back but haven’t gotten around to editing. We’ve seen too many accidents like that over the years, hence our recommendation: If you feel right at home using the Desktop for long-term storage, go right ahead – but if you’re not so sure about that keyboard shortcut we mentioned to get to the Documents folder (Command (⌘) – Shift – O), or didn’t know how to get there before you read this, give the Documents folder a chance.
What about thumb drives or external drives?
Most storage devices you buy for your Mac will come formatted for use in a Windows world, typically with the FAT32 or ExFAT filesystems. macOS provides basic support for these formats, so your new thumb drive or external drive will work out of the box, for the most part, but Apple has never guaranteed complete compatibility. Since macOS 10.15 Catalina, however, their support for FAT and ExFAT has gone from fair to you’d better have a backup. Try moving an open document in Preview on macOS 11.5 or later from an APFS or Mac OS Extended volume to a FAT volume and you’ll see what we mean. If you’re a developer, check out what happens when you use NSFileManager or copyfile() on macOS 11.5 or later to copy a file to a FAT volume. We can’t say for sure, but we believe these issues stem from a series of security-related changes in macOS Catalina and later that Apple’s FAT and ExFAT support was never tested against. Because, if Apple had tested, these issues wouldn’t have made it to anyone’s Mac… Right? We’ve reported the issues we know of to Apple and will keep you posted.
We say all of that to drive home that you really need to reformat thumb drives or external drives as soon as you plug them into your Mac. In general, if it’s SSD or flash-based, or a thumb drive, use APFS, and if it’s HDD or spinning media, use Mac OS Extended. With the Mac’s recent track record with FAT and ExFAT, you just don’t want to skip this step. If you’ve already got data on a drive and need to reformat, you’ll want to copy everything to a temporary folder in a volume on another drive before you go through with it. Read up on how to reformat or erase a volume here – and please pay attention to that first step.
But aren’t thumb drives more reliable than hard drives?
A lot of us have lost data when a hard drive bombed or our Mac was stolen. Some turn to thumb drives because they don’t trust hard drives anymore or want to keep their most valuable data in a safe place. We wouldn’t recommend that at all. We just don’t consider thumb drives an option for long-term data storage. They’re the modern floppy disk, really. You remember the 80’s and 90’s and the joys of floppy disks, right? No? Well, a reminder: the failure rate was up there with the Yugo and New Year’s resolutions. It’s true some floppies never died, but the ones that did often broke your heart or at least kicked off an all-nighter to crank out another copy of that term paper or spreadsheet. Thumb drives are in the same league: they tend toward cheaper construction and significantly slower flash storage, they can be physically torqued while inserted, bending their connectors with the potential to rip them from the circuitry within, and it’s not uncommon to forget you need to eject them before removing them, upping the chances of corruption. Like floppies, we’ve seen many fail. Would you trust a $5 thumb drive over the storage you paid Apple a premium to build into your Mac? Our recommendation is to reformat thumb drives to APFS as soon as you can and only use them for temporary storage, moving files between devices, or occasional backups. Again, read up on how to reformat or erase a volume here – and, again, please pay attention to that first step. If you need a long-term backup solution, keep reading.
Can I keep my documents on a file server or in a shared folder on another computer?
Whether we’re talking live documents you actually work in or backups made from within the application, using the Finder, or with third-party backup software, the answer is it depends, but we wouldn’t recommend it unless you know the risks of your particular situation.
If you’re using SMB, in general, the Mac’s client will get the job done. The devil, of course, is in the details. In this case, instead of seven levels of Hell, the Mac sees seven different kinds, or flavors, of SMB, depending on the server and the filesystem format of the shared volume, and it can behave differently for each flavor.
To see the SMB flavor the Mac is using for a shared volume, click the Finder icon on your Dock, go to the Go menu, click the Computer menu item, single-click the volume’s icon, go to the File menu, click the Get Info menu item, and look for the Format in the General section near the top of the window.
As of macOS 11.6, our tests show the best compatibility with the SMB (OS X) and SMB (NTFS) flavors. Outside of those two, we see a pair of risks you won’t want to ignore: NSFileManager and copyfile() on macOS 11.5 were unable to copy files to other SMB flavors, a situation Apple didn’t address for 7.5 weeks, which led to data loss in numerous applications, including some of Apple’s own, and, as appears to have been the case for years, NSFileManager sometimes – anywhere from 0.5% to 16% of the time, depending on the underlying filesystem format – reports it can’t copy or remove a file because of a permissions issue. As an aside, the Finder was immune to the first issue, which appeared to be security-related and the Finder likely had a blessed way around such, but it is not immune to the second issue. As an added bonus, the Finder won’t tell you when a copy fails in this particular fashion.
Try it for yourself: Go to the Finder, create a folder, put a few hundred files in it, then copy it to an XFS volume on a Linux Mint server shared with SMB. We just did it and 8 out of 210 files are junk. We didn’t record that test, but here’s another where 4 out of 210 files ended up as 4K instead of 50K like the original – with no error at all from the Finder:
If the Finder can’t see these kinds of things coming, the rest of us are probably in for it. So, based on how we’ve seen the Mac’s SMB client perform in various conditions, we’re updating our applications to do their best to warn when you interact with volumes like these on macOS 11.5 or later. The same issues may affect older versions of macOS but we haven’t had a chance to test yet. These warnings can be disabled so, if you accept the risks or they don’t apply, you can carry on.
If you’re using AFP, odds are also good things will go smoothly. We haven’t seen it fail at basic file copying lately, but let us know if you do and we’ll dig in with you.
If you’re using any other protocol, like NFS, you’re in a greyer area than we can really get into. As with the majority of SMB flavors, we’re updating our applications to try to warn about the risks of other protocols on macOS 11.5 and later. The same issues may affect older versions of macOS but we haven’t had a chance to test yet. Again, the warnings can be disabled so, if the risks don’t faze you or aren’t a factor in your situation, they won’t get in your way.
Thoughts on backups?
The Mac’s Time Machine is, hands-down, your easiest backup solution. It may not always deliver on the promise to completely restore your Mac exactly as it was before a catastrophe, but it’ll usually save your bacon if you lose or mangle a file and need to go back a day or six months to find an earlier version. We can’t recommend strongly enough Time Machine and a backup drive about twice the size of your startup volume – unless you’re backing up other drives, as well, in which case you’ll want about twice the total you’re backing up.
But Time Machine isn’t always the only answer. It stores your data on a device that can fail – and while your data is now on two devices, so failure is less of an issue, both of those devices are probably under the same roof. A roof that tops a building that can burn, flood, fall over…you get the idea. If you can’t regularly store a second Time Machine drive (which Time Machine does allow) at another physical location, we recommend exploring cloud storage or backup solutions. Something like BackBlaze or Carbonite will probably do, though we’re not pushing a particular vendor. Now you’re protected from failure, disaster, and thievery. The trifecta!
And syncing Desktop & Documents with iCloud Drive?
iCloud Drive’s Desktop & Documents Folders option makes a lot of sense for a lot of folks. It sends whatever you put in these two folders up to iCloud Drive, syncs it to all of your devices, and everyone lives happily ever after. Until…you turn it off. And maybe back on. And maybe off again. Then things get…weird.
Each time you turn iCloud Drive off it creates a backup folder called “iCloud Drive (Archive)” containing everything you had stored in iCloud Drive and stores it in your home folder. When you turn iCloud Drive back on, that iCloud Drive (Archive) folder stays put – nothing in it moves back into place. If no other devices removed anything from iCloud Drive in the meantime, everything should sync back down to your Mac as your bandwidth allows. If that doesn’t work out, though, and you need anything from your previous iCloud Drive storage, you’ll have to go dig it up in the iCloud Drive (Archive) folder. When you turn iCloud Drive back off, another iCloud Drive (Archive) folder is created, with an incrementing number at the end of its name. As you can guess, as iCloud Drive is turned off and on, the cycle continues and you can end up with any number of iCloud Drive (Archive) folders.
Things get especially tricky when you use the Desktop & Documents Folders option since, like everything else in iCloud Drive, those two folders will be moved to the iCloud Drive (Archive) folder when you turn off iCloud Drive. That means everything in those two folders on your Mac will disappear. The files aren’t really gone, they’re just in a different folder where you’ll likely never think or know to look. Worse, in true Mac fashion almost every application tracks your documents with aliases or bookmarks so they can automatically find things in that iCloud Drive (Archive) folder without asking you for help. That doesn’t sound worse, at first – because yay! your data is still there and you didn’t have to do anything to go find it – but you probably didn’t even know it moved in the first place…so when you come across that iCloud Drive (Archive) folder you might end up doing the unspeakable: you might throw it in the Trash and empty it without looking inside or, even if you do look in there, you may not recognize everything that’s within or realize what it means to throw it all out. Long story short, you’re setting up to lose data. And we mention all of this because we see it happen a lot. So we like iCloud Drive and recommend it, but if you ever turn it off, immediately go find that iCloud Drive (Archive) folder, open it, move everything from the Desktop folder to your actual Desktop, move everything from the Documents folder to your actual Documents folder, and poke around the other folders in there to see if there’s anything you need to keep on your Mac. In the end, you don’t want to keep the iCloud Drive (Archive) folder around. If you do, the next time this whole cycle repeats you’ll have two of them to deal with, then three, four, and so on.
If you need help
We just laid out a ton of information and you’re bound to have questions, especially if Audiobook Builder, CheckBook, or CheckBook Pro just sent you here. Get in touch at support@splasm.com and we’ll be happy to provide any additional details you need!
In the way, way, way back, before Mac OS X came in 2001 and changed the Mac forever, most Mac users came up with a filing system at the root of their startup volume, extended that to any external or removable storage, and called it a day. The Finder was a bit more, how shall we say it, spatiallyoriented in those days, and folks were just as likely to group files visually near each other as they were to place files in folders by task, project, context, what have you. The root of the startup volume was as fair game as any other location, and the Desktop folder was also a broad, easily hit target. Some users even kept documents in the same folder as the application that created them. Then, the Documents folder arrived.
While Mac OS 8 ditched many of the technology goals Apple had promised for years, it took a host of Mac features to the next level – and the Finder was no exception. It brought spring-loaded folders, popup windows, per-folder view options, to name a few – and also big changes to the default folders at the root of every startup volume. One of those was the new Documents folder, a big hint, along with Mac OS 8.5’s Sherlock, that how the Mac helped you keep up with all of your files was about to leap to the next level. Apple’s intention was for users to store all of their documents in the Documents folder, organized however they liked, to keep files off the Desktop and out of application folders, where they were all too often lost during an uninstallation or accidentally “cleaned up”, and in position for advances in search technology that would make finding your data a whole lot easier.
Set the way forward machine ahead about 20 years or so. We’ve seen a lot of incremental improvements but each user account on today’s macOS still has a Documents folder and the goals are still the same: to make it easy for you to find your documents when you need them and protect those documents from mishaps. iCloud Drive even includes a feature to synchronize your Desktop & Documents folders across devices, so you know Apple’s still pushing this Documents folder thing!
This issue affects a small number of our users, but we need you to read through just to be sure you’re not in that group. Thank you!
Update 9/13/2021:Great news! macOS 11.6, released earlier today, seems to have addressed the fundamental issue with FAT, ExFAT, and some SMB volumes. We’re testing each application in the list below to be sure, but our own applications, Apple Preview, and Tumult Hype all appear to be working as designed, so we think we’re out of the woods. Thank you all for hanging in there!
Even though Apple seems to have resolved the issue and we’re breathing easier over here, we want to leave a note for the record: The vast majority of Apple developers have used NSFileManager or copyfile(). Copying files to or saving files on FAT, ExFAT, and SMB volumes is not uncommon, especially since almost every external drive comes pre-formatted for Windows. However this thing slipped through the cracks, it took Apple 7.5 weeks to move on it with two minor macOS updates in the meantime. In the 1.75 months since we told Apple about it, something close to a hundred of our users – that we know of – lost data and needed help restoring a backup while we spent an insane amount of time we didn’t have (and our families didn’t want to give us) testing and coding around the issue. Apple never owned up to the issue. There’s no mention of it in the macOS 11.6 release notes. No one from Apple ever communicated with us about our bug reports – and there’s no sign any engineer even read them. If we caused an issue like this, then handled it as Apple has, we wouldn’t be able to live with ourselves – and we’d be out of the Splasm business, to boot. So our faith in Apple’s support for anything besides APFS or Mac OS Extended, or SMB to those on recent macOS hosts, got taken to the woodshed, you might say, and we just won’t rely on them anymore. In the near future, all our applications will warn you when you try to use other formats or flavors of SMB until you tell us not to. We value everyone’s data and time that much.
Update 8/24/2021 (Further updated 9/13/2021):
The applications (and known actions) affected by this issue are now:
Audiobook Builder 2.1.4 (new/save) – Workaround coming in 2.1.5Fixed in macOS 11.6
CheckBook 2.6.20 and earlier (save/backup/move) – Update to 2.6.21 nowFixed in macOS 11.6
CheckBook Pro 2.6.20 and earlier(save/backup/move)– Update to 2.6.21 nowFixed in macOS 11.6
Apple Books 3.2 (drag audiobook to volume)Fixed in macOS 11.6
Apple Calendar11.0 (export calendar archive)Fixed in macOS 11.6
Apple Configurator 2.14 (move)Fixed in macOS 11.6
Apple Contacts13.0 (export contacts archive)Fixed in macOS 11.6
Apple Photos6.0 (export unmodified originals)Fixed in macOS 11.6
Apple Preview 11.0(move)Fixed in macOS 11.6
Apple QuickTime Player 10.5(move)Fixed in macOS 11.6
Apple TextEdit 1.16 (save to existing file)Sorta fixed in macOS 11.6 (temporary files aren’t removed, document may save but eventually can’t be because it’s “locked”)
OmniOutliner 5.8.5(move)Fixed in macOS 11.6
OmniGraffle 6.6.2(move)Fixed in macOS 11.6
Pixelmator 3.9.8(export – except for PDF)Fixed in macOS 11.6
Pixelmator Pro 2.1.3 (new/save/move/export)Fixed in macOS 11.6
Tumult Hype 4.1.7 (save/move)Fixed in macOS 11.6
Update 8/16/2021: This issue also affects QuickTime Player on macOS 11.5 and later. Duplicate a video or audio file, open it with QuickTime Player, use the File menu’s Move To… option to move the document to a FAT volume, then check the size of the file on the FAT volume and it should be zero bytes.The same move operation to an APFS or Mac OS Extended volume works as expected.
Update 8/13/2021:CheckBook and CheckBook Pro 2.6.21, now available, work around NSFileManager and copyfile() on macOS 11.5 and later to preserve your data in all known situations where these methods can fail. Thank you all for hanging in there! We strongly recommend you store your documents on either APFS or Mac OS Extended volumes, or access them over a network with SMB or AFP (though there’s only a whiff of AFP left in Big Sur so SMB it is).
Update 8/11/2021: macOS 11.5.2 hasn’t addressed this issue. Stay tuned.
Update 7/30/2021: We’ve developed a workaround that’ll do for now. It’s in testing now and will be out during the week of August 8th.
Update 7/27/2021: Though release notes didn’t mention the issue, we installed macOS 11.5.1 anyway to see if Apple slipped in a fix under the radar (that’s an old-time Apple developer pun). No change.
Update 7/26/2021: One user reports this issue may also occur on ext4 and XFS volumes over NFS. We haven’t tested these filesystems or NFS but the symptoms appear the same. It could be that any volume or share mounted over NFS will exhibit the issue, so our advice is to stay away from NFS altogether until a fix is in place.
Original post from 7/23/2021:
macOS 11 Big Sur made it to version 11.5 this week and, along for the ride, comes a few showstoppers for a relatively small group of our CheckBook and CheckBook Pro users:
If you’ve installed the macOS 11.5 update and your regular CheckBook document lives on a volume formatted as FAT16, FAT32, or ExFAT, DO NOT open that document or your data may be wiped from that particular copy of your document.
Using the File menu’s Backup command and saving to a FAT16, FAT32, or ExFAT volume on macOS 11.5 will create an empty backup.
Using the File menu’s Move To… command and moving the open document to a FAT16, FAT32, or ExFAT volume on macOS 11.5 will erase the data in the document.
These issues have the same root cause, what appears to be a bug in the most common method macOS developers use to copy files. This method reports success for each copy while, in fact, the final result is a completely empty file that lacks the data, attributes, and extended metadata of the original. Because macOS tells CheckBook the file copy succeeded, there’s no way for the application to know it failed – so there’s no way for you to know, either. We believe this bug also affects a number of other applications, such as Apple’s own Preview. If you’d like to see it in action, duplicate an image file on your Mac so you don’t lose the original, open the duplicate with Preview, go to the File menu, click the Move To… menu item, then select a destination on a FAT16, FAT32, or ExFAT volume. The moved file will end up zero bytes in length.
We’ve sent a bug report to Apple and reached out to another major developer to let them know this affects their applications, hopefully giving us an additional vector to an Apple engineer. In the meantime, let’s protect your CheckBook data:
As we said, if your document lives on a FAT16, FAT32, or ExFAT volume, do not open it or you could lose your data in that copy of your document. Instead, copy the document to your Documents folder and double-click the copy in Documents. We’d recommend staying away from storing your CheckBook documents anywhere but the Documents folder, but we are particularly down on using USB thumb drives for long-term storage. They tend to not hold up as well as any internal storage device and we’ve seen too many fail over the years to not send out a warning like this every now and then.
Do not use the File menu’s Backup or Move To… commands with a FAT16, FAT32, or ExFAT volume as the destination. While you could copy your document in the Finder, opening the copy from there would lead to data loss in that copy of the document. If you need to backup your document, select a folder in iCloud Drive or another cloud storage location. If you need to move your document to another storage device, or you need to keep a local backup, but don’t have anything but a drive with FAT16, FAT32, or ExFAT volumes, you can copy any files you need to save to another volume, then reformat a volume as APFS or Mac OS Extended. Remember to copy the files you need to save first. Read up on how to erase or reformat a volume here – and please, please pay attention to that first step.
We’re now looking for a way around this because a fix from Apple could take a while. The worst option is to simply forbid interacting with FAT16, FAT32, or ExFAT volumes. Much as we’d like to not worry about this kind of thing in the future – this isn’t the first or second or third quirk that’s popped up with FAT volumes and these days they’re feeling more and more like stowaways in the mothership’s steerage – completely nixing support for them is a bit more nuclear than we’d like. We’ll find a solution soon and let you know when we do.
You might not know if your document is on a FAT16, FAT32, or ExFAT volume to begin with, so here’s how to tell: Locate your document in the Finder, Command-click the name of the window to reveal a menu, click the second-to-last item in that menu, go to the File menu near the top left of your screen, click the Get Info menu item to reveal a window with a lot of details about the file, and look for the Format in the General section near the top of the window. If Format reads “MS-DOS (FAT16)“, “MS-DOS (FAT32)“, or “ExFAT” you know for sure your document lives on a volume with the issue.
Why is your volume formatted as FAT16, FAT32, or ExFAT when these are usually something you’d see on a Windows machine? The Mac’s come a long way from the nightmare of the mid 90’s, and the iPhone and iPad have done their part to make Apple a household name, but the desktop is still a Windows world and almost every drive out there comes set up for that reality. macOS makes it easy with basic support for these formats, but you really, really, really need to reformat to APFS or Mac OS Extended whenever possible. Read up on how to erase or reformat a volume here.
Also, if you’re wondering why this affects some applications but not others, it’ll help to understand every developer does things a little differently. Data can be read and written a number of ways, and the techniques a developer uses can vary depending on the application and the type of data. For example, you’ll see the issue with the Move To… command in Apple’s Preview but not in Apple’s Numbers. From what we saw while playing around with Numbers today, we believe it’s set up to do a full save on the destination volume, instead of a copy, when moving a document from one volume to another. That likely involves creating files from scratch, then saving data into each file, as needed, rather than a straight copy from one volume to another – and that route doesn’t appear to present any issues.
Please let us know at support@splasm.com if you get to this article after a mishap and you need help restoring a backup. If you don’t have a recent backup, CheckBook’s automatic backups, created about once every seven days, will save the day. Here’s how to find and restore these automatic backups:
Open CheckBook or CheckBook Pro.
Go to the Help menu while holding down the Option key on your keyboard and click the Go to CheckBook 2 Folder menu item.
Inside the CheckBook 2 folder, open the Backups folder.
You may see several items inside the Backups folder. Open the folder whose name matches your CheckBook document’s name. You’ll see a list of compressed backups.
Each compressed backup file includes the date it was created in its name. Look for the most recently dated file and double-click it. A file with the same name as the compressed backup but an icon of a sheet of paper and a tiny CheckBook icon will appear. This is the actual backup file.
Move the backup file to your Documents folder (or wherever you’d like to keep your document going forward – as long as it’s not on a FAT volume) and double-click it.
This update’s just to fix an issue with 2.6.19 that prevented our users in Canada and the UK from creating new Account Summaries or making changes to existing Account Summaries, with a few minor user interface tweaks and enhancements. It won’t hurt to skip it if you don’t see the issue – and you’ll find you can, indeed, skip updates as of 2.6.19, if you purchased from the Splasm Store, with the spiffy new Skip button that sprouted in the lower left corner of the update checker.
Update 7/9/2021: CheckBook and CheckBook Pro 2.6.20 are now available with the fix. Get the update for CheckBook or CheckBook Pro.
To ourCheckBook and CheckBook Pro users in Canada and the UK, please accept our apology: We’re patching up an issue that prevents the Account Summary options sheet from appearing and could lead to a crash. Account Summaries will work just fine – it’s only when you try to change their settings or create a new Account Summary that the issue rears its head. 2.6.20 is in testing now so we’ll have you fixed up soon!
