Anyone having NEW problems with RW files and Dropbox?

Lol, Yep, that was the word-for-word response from Dropbox support

My Dropbox definitely does NOT like RW files. When I drop a project file into dropbox it starts to sync the individual files in the Sandwich before the move into dropbox is even complete. I then get “syncing 7,176 files” with a time to do from 1 sec to hours. When I compress it into a zip and let it sync it’s done in seconds.

I’m going to add a step to my RW Project file backup procedure. As stated above I use CCC to move the files at 2 am to dropbox. I’ll add a step to first compress the files (zip) and then copy over. This will also give Dropbox a lot less files to maintain. One file instead of a sandwich with seemingly endless files.

1 Like

My project file is now sitting on 1,775 files to sync and it’s just sitting there spinning. This, from one project file.

I’ve been compressing first, syncing second for a few months now. Zero problems.

However, I need to do the same with Google Drive. It seems to sync but takes FOREVER with RW projects. But, again, if I compress then sync then all is quick and smooth.

RW project files aren’t one file as you see them in Finder. If you have ever right-clicked on one and revealed their contents, you will find they are folders containing potentially several thousand other files and folders. So when Dropbox says it is syncing 7k files or whatever, that is believable.

The structure of the RW project file has not changed significantly in a decade or more. So I’m willing to bet the issue lies with Dropbox.

I’m pretty certain Dropbox used to use their own system of compressing and transmitting files. Similar to how a ZIP or TAR package works. So rather than having to read-and-write millions of individual files, it could just move packages of files from one place to another. Hence the system used to be very speedy and reliable.

If Dropbox is now quoting crazy file quantities and unrealistic times, I’m guessing Dropbox has changed something. Maybe they no longer do this on-the-fly compression any more? It certainly sounds like a plausible explanation, based on what people are saying here and elsewhere. If Dropbox is suddenly tasked with having to read and transmit 7k or more files, it’s not going to be a smooth or timely process!

There definitely has been an increase in the number of RW users no longer being able to sync uncompressed project files or addons over Dropbox. I have heard of a few customers where this was happening. Issues seemed to start about 10 days before Christmas. Now that more people are back at their desks, the number of reported problems appears to be rising higher.

1 Like

Yes… I am now reminded of just how “complicated” a RW Project file is. I have a folder for each day of the week and each folder contains 9 Project files ranging from 6 MB to 122 MB in size. I also have a separate project for Joe Workmans “email.” I’m compressing them now and I see each folder contains over 18,000 items! That’s 18,000 items for Dropbox to track for just a one day 10 project set backup set to share across computers!

All bundle files on macOS work this way – a simple user-facing icon that hides many internal files. It’s actually a pretty great system. And it’s almost 20 years old now. It has worked very well for a long time. That part at least, is neither new nor exceptional.

A fun one is Xcode.app. It’s “one file” to users, but it’s about 12 GB(!!!) and ~600,000 files.

Dropbox does have to sync all those things when you copy a file bundle – each of those files has its own file permissions, icons, and (in dropbox) a history as well. Copying things as a single zip gets around that – all of the overhead of the Finder is omitted and only a single file-history will be maintained in Dropbox.

Dropbox always had to work extra hard doing this stuff – but it never made it go crazy until recently. My guess is that it has a lot of do with changes to the file-system, APFS, and the indexer – if you open up your Activity Monitor app you’ll usually see several “mds” processes and at least one “mds_stores” – these are must macOS indexing your files which allows Spotlight and other apps to perform fast searches on the files, their contents, tags, etc.

When Dropbox goes nuts (at least for me) it coincides with many mds processes spawning – the more processors you have the more mds will spawn. I have a 16-hyperthread cpu – so things really go bananas!

I suspect there is some bad interaction that causes Dropbox to abuse the indexer, or perhaps the other way round, the new indexer doesn’t handle the corner case that’s common during syncing operations – even when using OneDrive or Google Drive.

It probably doesn’t help that both the indexer and dropbox have gone through fundamental rewrites recently – I’d bet hard cash that both Dropbox and Apple are pointing the finger at each other.

mds had large changes recently to add support for APFS, the new disk file system. this was supposed to make mds and large move/copy operations (like syncing!) much faster because of the way it handles hard-links.

obviously none of us really know the real deal, but i strongly suspect that redesigning mds, APFS, and dropbox – all within about 12 months of each other is at least partially responsible.

3 Likes

I was about to try using the synology NAS for backup. Could you tell me please what problems you’ve seen with it?

My wife and I run a small Swedish company selling bespoke stainless-steel worktops for kitchens in private homes. We have one large Swedish (https://www.neonela-kitchen.se) and one small German website (https://www.neonela-edelstahl.de). Sometimes we have to work on the same RW project. Not simultaneously and only on the files on the computer. We have for many years synced the RW files through Dropbox as we do with all other files generated from Lightroom, Sketchup, Excel, Word, AutoCAD and so on. We have a broadband connection of more than 800 mb/s up and down. So, in general syncing our biggest RW project files on two computers through Dropbox used to take just a few minutes. The first problems started when Dropbox introduced Smart syncing. I didn’t go well with Catalina. We turned it off completely, which was a small project in itself. Selective syncing works better for us. In the beginning of January, the real problems started. Syncing the Swedish site, that consists of 177 pages, took ages to complete. Even if I only had changed a small part of a page and perhaps 40-50 files Dropbox wanted to sync thousands of files. The syncing was very, very slow and sometimes stopped completely. Sometimes the folder was marked red but all files in the folder were marked green. Our first reaction, after checking all connections, was to test Dropbox Business instead of Drobox Plus. It was better in some ways but there were no improvements in the syncing process. Then we found out that if the syncing stopped, we could log out of Dropbox and log on again, not just closing the software, and it started again. But in the same slow, slow way. I have been in contact with Dropbox and also sent them screenshots. They responded quickly but have not come back with a solution or even an explanation. The first to person at Dropbox to respond have handed it over to someone else. That was two weeks ago. Our subscription is up for renewal in a couple of weeks so we must quickly find a solution with or without Dropbox.

1 Like

I have been using Dropbox to keep my 4 Macs in sync for many years and I’ve never run into problems until about 3 weeks ago. At that point 2 of my Macs started showing a new message within Dropbox that said “Performing a one-time update”, followed by a “Syncing” message that continued on for 2 weeks… It looked like it was never going to finish syncing, so in the end I completely purged Dropbox from the affected machines (including all of the various library and preference files that are scattered behind the scenes). I downloaded Dropbox again, reinstalled it, downloaded all of my files from the Cloud and it started working again.

However, I have continued to notice that Dropbox now hates RW files. I used to be able to drag a RW file into Dropbox and it would sync quickly and easily - but now it takes forever. Sometimes it literally takes all night. It appears to me that Dropbox has done something to change the way it syncs files and it no longer processes RW sandwich files efficiently at all.

2 Likes

I wrote a very nice and friendly reply to my Dropbox support person (4th one). In this 6th or 7th email to them I explained that Dropbox did NOT seem to handle RW files well, that it started suddenly, that dropbox had worked perfectly for me for years. But something now was not working with RW sandwich files.

The reply from Dropbox was that they could not fix anything wrong with RW files, that was an issue with RW. I thought that odd since they are in the file backup/sync business and need to handle all types of files in order for the product to perform as advertised. But the support person (supposedly a “higher” support person) was not at all interested in what I had to say.

I continue to use Dropbox now without problems, heavy use of selective sync and smart sync, but Rapidweaver files get zipped before I put them in Dropbox. All my project files (about 2000 pages total in ten projects) are in one folder. That folder gets zipped at the end of the day. I then have Carbon Copy Cloner keep a Sun thur Sat seven day backup by cloning my RW zipped folder to the proper day folder in dropbox.

Since zipping RW files I’ve had zero problems.

Thanks for the recommendation regarding the use of zipped files - I’ll give that a try.

I’ve basically found in the last few weeks that Dropbox works perfectly well with all of my common file types - except for RW files. As I mentioned, this is a new development for me personally, as I’ve never had issues before. But I may have to rely upon zipping them up before uploading.

@1611mac We’re running Dropbox in my office to keep our RW files synched so anyone can grab a client’s project and make updates or changes to it. It has worked pretty good for quite a while. I have experienced Dropbox hanging up over the last few weeks. Maybe the only difference is we use two different Dropbox accounts and synch a “Team” folder between us.

The (temp) solution to this is to go to the Dropbox app, when it drops down, click Account > Preferences. In that Preference panel, click Account. Then hit your Option button on the Mac, you will see the “Unlink Dropbox” button disappear and two buttons appear “Fix Permissions” and “Fix Hardlinks”. Click “Fix Hardlinks” and the app will resynch fairly quickly. We have 10’s of thousands of files (over 100 RW projects) and it synchs in a matter of minutes.

It seems like we’re doing this on a regular basis now to synch our files, but it works. Good luck.

FWIW: I believe Dropbox among other cloud storage providers might be taking on a space-saving technique originally derived from data backup systems called “de-duplication” which is a way of only tranferring identical data blocks only once (instead of multiple times) thus saving space in their data centers.

In order to find what data to transmit (in this case - to Dropbox) to local (Dropbox) software on your computer must open all files that are to be transferred and go through them looking for indentical pieces of data (on a low level).

If this operation is to be done over several thousands of files it will take a long time since it is your computer that executes all these operations before transmitting it to the (Dropbox) cloud.

Bandwidth is not an issue here, but the problem lies in the time to perform de-duplication which grows exponential with the number of files involved. I don’t know if this is the problem in this case but given the answer from Dropbox’s support team regarding the huge number of files I suspect thast this has some thing to do with the problems experienced.

And all the while Dropbox tries to get me to “upgrade” to more storage space, as if my 2TB doesn’t give me enough headache (and I use only 800gb of the 2TB!).

If you guys just are trying to sync RW and development stuff between your own computers (as opposed to sharing with clients or mistakenly using Dropbox as your backup) I highly recommend using Resilio Sync Pro https://www.resilio.com/individuals/ or their Team/Enterprise solutions depending on your needs.

Syncing works like a charm and if you’re on a Synology NAS there is a plugin for those. I use it when travelling on photo tours etc to sync back my photos to my NAS and mail computer at the home office so everything already is there when I get home to work.

I have heard that Resilio has improved a lot. I used it early on and had massive data loss with it. It would not work well with Mac package file format. I have been wanting to try it again. I just got punched too hard from it early on…

We moved to synch.com and now do daily backups with an external hard drive. There are two of us in the office and have been satisfied with moving from Dropbox.

Only issue we found was that we could not name files that ended in a period. (AB Tools, Inc. ) or name a page that ended in a period (River Wheel Co.). Those files would not upload, but changing the names uploaded them without corrupting the files.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.