Anyone having NEW problems with RW files and Dropbox?

Classic General
1

I don’t run my active RW projects from dropbox but I do use Dropbox to sync my files to a 2nd computer. (I use CCC to copy my active RW files to Dropbox each night. I keep a weeks worth of RW archives.)

Now, for the first time, I’m seeing some issues with Dropbox and RW project files. I cannot positively confirm that they are the issue but when I remove the RW files from Dropbox some problems seem to go away. (ie: non-stop crazy syncing of thousands of files when my local machine and dropbox cloud both show all files as synced)

A Dropbox support person told me that there was an update to the app lately (Does it auto update? - I wasn’t aware of that.) I’m wondering if something has changed in it’s handling of Sandwich files.

2

There’s been a lot of talk here and on other forums about Dropbox no longer working well for RapidWeaver project files. Seems a lot of people are getting away from Dropbox and going to OneDrive or iCloud.

3

Yep. After years of using Dropbox the support people treat me like a complete idiot. “Give it time to sync.” they say. I tell them it was synced perfectly an hour ago, I have edited one file, and now it’s syncing 6,000 files when I have verified that every single files in Dropbox folder has a green sync icon.

One rep told me that I have “a large number of files” in dropbox and “dropbox isn’t as efficient with large number of files.” Well, I say, I’m only using half of my 2 TB available. If my file count is a problem why do you offer so much storage and why do you try to sell me even more storage?" She had no answer.

The support is laughable. They do not believe you and they can’t believe there is something that might actually be wrong.

4

this is exactly my current experience with dropbox. my wife’s as well – on totally different hardware, 1/8 the number of files, and two OS versions back. and it’s been this way a year. it’s not a small bug – it’s their core functionality, some might say their “only feature”.

that said, i’ve spent six months trying google drive and microsoft onedrive with the plan to switch to whichever works best.

and the result… was surprising… they both have very very similar problems: endless syncing, failure to update the Finder correctly. data-rot (small corruptions leaking into files that have no changes) – all problems that i thought were new bugs to Dropbox.

:thinking:

i have no love of dropbox as i once did, they did a full rewrite of their software to make it less native. in the process it became a bloated mess.

but the dropbox problems, synology syncing failures, google drive and ms onedrive issues all seem to have started showing up on forums and blogposts at about the same time Apple began the push to Mojave and APFS.

that’s also when i disabled my old TimeMachine server because it just suddenly started having really strange problems.

this is all just speculation – but i’m not so quick to blame dropbox for these problems anymore. not if they are systemic across so many competing products. if these problems cannot be solved by MS, dropbox, google, synology i have a feeling it might be out of their hands.

anywho… to answer original question directly: YES and NO.

I’ve never experienced any data loss on dropbox firsthand – but I have seen the results of corrupted data A LOT. I don’t think the problem is Dropbox per se – but just the nature of syncing a large complex file that can take minutes to finish writing to a spinning drive (if you have a big file and slow drive) – and can take an hour or more to sync to Dropbox over a slow connection. In the meantime you may have opened the file and started editing again – or you may have opened the file on a different machine.

Syncing services are supposed to handle these worst-cases well, but I think with very large bundled files (like a RapidWeaver project that can contain many hundreds of smaller files) there are some cases that have no “right” solution.

So my advice is always is this:

  • Keep a backup – one that doesn’t involve a syncing service.
  • If things are working well, go for it. Use whatever tools suit you, including Dropbox.
  • If things are not working well, stop. And make use of that backup to recover your file.

Your backup is there so that you can do things that might not be 100% safe. And yeah, I think that’s pretty much what a syncing service is – not perfectly safe.

3 Likes
5

I have multiple backups including a nightly CCC bootable backup of entire disk. I have 1 and 2 TB pocket drives for this and they go with me when I leave the house just in case of fire or theft of all my computers.

I use Dropbox ONLY only to sync “core” files with my iMac and with my other 3 computers (2 other Mac’s and a Win10 box) - I also make good use of selective sync. For example, I have a “Win10” folder within dropbox and the Windows machine syncs only to that folder. I want to be able to grab my MBP anytime to travel and it want it to be synced with my “core” files so it gets a much larger file set. etc etc

I have had no problems with Dropbox for years. But I use it very wisely (like keeping RW files of Dropbox and using CCC to copy them into Dropbox at 2:00 AM daily) .

But something has changed just over the past few days. I have now COMPLETELY deleted dropbox (and helpers, and prefs, and all) 3 times. I just did it again this morning, reinstalled, and set all files except the ones I’m working on today as “remote only” (cloud) to limit the syncing for now. Dropbox is now sitting fully synced and all icons on all Dropbox files/folders are correct (mostly “cloud”). I’ll now start changing the pref on fille sets from “remote” (cloud) to “local” in a slow and logical way with “rests” in between. I’m going to keep a keen eye and see if I can find what is blowing it up.

6

Dropbox started to go horribly wrong for me about a week ago, after years with it with RW no bother.

I’ve been testing OneDrive and it seems to sync OK, but is unusably slow. It will sit for ten mins telling me it’s syncing one file of 78kbs. This isn’t isolated, it’s the same over four machines. Everything OneDrive does seems to take forever.

As said above, the fact all these services started to fall over at the same time points to something other than themselves at fault, although worth mentioning I’ve no machines on Mojave.

7

May I ask what the symptoms are? Nonstops syncing of “thousands” of files?

1 Like
8

It was splitting open RW projects, I think those with resources added via Unsplash, though can’t be sure on that, and putting hundreds of “invalid-file” folders at the root with. These folders had no content.

9

ignore… doing more testing

10

I can absolutely confirm problems with RW files and Dropbox. For me these problems started at The RW 8.4 update. The same day… The problem is that DROPBOX also auto updates and I believe updated the exact same day. I moved my working project files to iCloud and have not had problems other than it is a much slower sync… I have not tested with RW 8.5 yet but I will try to today.

Note, if I move any of my project files into drop box they sync perfectly as before. It is when a change is made on the project file from within Dropbox and after publishing… Dropbox simply cannot sync the changes to the project file after publishing. This much I have tested and confirmed.

In the past and for many years I was able to publish my project files from within Dropbox and now no longer can…

What I don’t know is what caused this. DROPBOX or RAPIDWEAVER??

OR- as Isaiah says, Is it Apple? I have no systems on Catalina. I am waiting for third-party software updates… So nothing past Mojave for me…

11

Doug linked to this thread above but I’ll copy/paste my response. This is what fixed my recent Dropbox issues. You may still want to look at other alternatives but this may help get you back stable first…

  1. Make sure the Dropbox desktop application is running
  2. Click on the Dropbox icon in your menu bar and then click on the gear in the Notifications panel
  3. Select ‘Preferences’
  4. Click on the ‘Account’ tab
  5. Hold down the ‘Option/Alt’ key.
  6. Click on the ‘Fix Permissions’ button (make sure you’re still holding down the ‘Option/Alt’ key)
  7. You may be prompted for your a username and password, this is your computer login credentials, not your Dropbox account ones

Prior to this I had ‘unlinked’ my dropbox and relinked it. Rebooted the comp. I also made sure nothing was currently attempting to sync. The permissions fix was set and then, after a lengthy sync process (at least an hour or two), it was back to functioning normally.

1 Like
12

Oh no… this is starting to sound like Dropbox Support - I have already “fixed permissions” several times. No help! But I do appreciate your reply. I really do.

13

Lol, Yep, that was the word-for-word response from Dropbox support

14

My Dropbox definitely does NOT like RW files. When I drop a project file into dropbox it starts to sync the individual files in the Sandwich before the move into dropbox is even complete. I then get “syncing 7,176 files” with a time to do from 1 sec to hours. When I compress it into a zip and let it sync it’s done in seconds.

I’m going to add a step to my RW Project file backup procedure. As stated above I use CCC to move the files at 2 am to dropbox. I’ll add a step to first compress the files (zip) and then copy over. This will also give Dropbox a lot less files to maintain. One file instead of a sandwich with seemingly endless files.

1 Like
15

My project file is now sitting on 1,775 files to sync and it’s just sitting there spinning. This, from one project file.

16

I’ve been compressing first, syncing second for a few months now. Zero problems.

However, I need to do the same with Google Drive. It seems to sync but takes FOREVER with RW projects. But, again, if I compress then sync then all is quick and smooth.

17

RW project files aren’t one file as you see them in Finder. If you have ever right-clicked on one and revealed their contents, you will find they are folders containing potentially several thousand other files and folders. So when Dropbox says it is syncing 7k files or whatever, that is believable.

The structure of the RW project file has not changed significantly in a decade or more. So I’m willing to bet the issue lies with Dropbox.

I’m pretty certain Dropbox used to use their own system of compressing and transmitting files. Similar to how a ZIP or TAR package works. So rather than having to read-and-write millions of individual files, it could just move packages of files from one place to another. Hence the system used to be very speedy and reliable.

If Dropbox is now quoting crazy file quantities and unrealistic times, I’m guessing Dropbox has changed something. Maybe they no longer do this on-the-fly compression any more? It certainly sounds like a plausible explanation, based on what people are saying here and elsewhere. If Dropbox is suddenly tasked with having to read and transmit 7k or more files, it’s not going to be a smooth or timely process!

There definitely has been an increase in the number of RW users no longer being able to sync uncompressed project files or addons over Dropbox. I have heard of a few customers where this was happening. Issues seemed to start about 10 days before Christmas. Now that more people are back at their desks, the number of reported problems appears to be rising higher.

1 Like
18

Yes… I am now reminded of just how “complicated” a RW Project file is. I have a folder for each day of the week and each folder contains 9 Project files ranging from 6 MB to 122 MB in size. I also have a separate project for Joe Workmans “email.” I’m compressing them now and I see each folder contains over 18,000 items! That’s 18,000 items for Dropbox to track for just a one day 10 project set backup set to share across computers!

19

All bundle files on macOS work this way – a simple user-facing icon that hides many internal files. It’s actually a pretty great system. And it’s almost 20 years old now. It has worked very well for a long time. That part at least, is neither new nor exceptional.

A fun one is Xcode.app. It’s “one file” to users, but it’s about 12 GB(!!!) and ~600,000 files.

Dropbox does have to sync all those things when you copy a file bundle – each of those files has its own file permissions, icons, and (in dropbox) a history as well. Copying things as a single zip gets around that – all of the overhead of the Finder is omitted and only a single file-history will be maintained in Dropbox.

Dropbox always had to work extra hard doing this stuff – but it never made it go crazy until recently. My guess is that it has a lot of do with changes to the file-system, APFS, and the indexer – if you open up your Activity Monitor app you’ll usually see several “mds” processes and at least one “mds_stores” – these are must macOS indexing your files which allows Spotlight and other apps to perform fast searches on the files, their contents, tags, etc.

When Dropbox goes nuts (at least for me) it coincides with many mds processes spawning – the more processors you have the more mds will spawn. I have a 16-hyperthread cpu – so things really go bananas!

I suspect there is some bad interaction that causes Dropbox to abuse the indexer, or perhaps the other way round, the new indexer doesn’t handle the corner case that’s common during syncing operations – even when using OneDrive or Google Drive.

It probably doesn’t help that both the indexer and dropbox have gone through fundamental rewrites recently – I’d bet hard cash that both Dropbox and Apple are pointing the finger at each other.

mds had large changes recently to add support for APFS, the new disk file system. this was supposed to make mds and large move/copy operations (like syncing!) much faster because of the way it handles hard-links.

obviously none of us really know the real deal, but i strongly suspect that redesigning mds, APFS, and dropbox – all within about 12 months of each other is at least partially responsible.

3 Likes
20

I was about to try using the synology NAS for backup. Could you tell me please what problems you’ve seen with it?