How to Speed up the updating process?


(Mark Buehl) #1

With over 2,000 pages the process of updating my Website is taking over 15 hours on my iMac before it starts to upload. Is there a way to speed up the process?

Also, it would be nice to have a way to pause the process right before the upload so it can be watched for errors.


(Tapio Laakkonen) #2

Hey, do you mean publishing your uptaded project file?
I am sure you will find the answer here but here are few ideas,

  1. Connection, I get very different results over a tethered internet connection that over home wife setting
  2. How many connections do you have in your usage? Maybe host could provide more…
  3. Perhaps the best option would be to use FTP client to move your project files to server directly

Hope I didn’t miss read your question


(Mark Buehl) #3

Not with the publishing via the Internet, but with the process before it starts the internet connection.


(Joe Workman) #4

How big is your project file?


(Mark Buehl) #5

34 GB Project file


(Mathew Mitchell) #6

@markbuehl: That is one ginormous file. My guess is you will have continued problems. 2,000 pages is a lot but my guess is part of the problem is not the number of pages, but that you also have a lot of media stored within RW: video, audio, images.

With something this large there are probably a few steps you need to take to get things back under control. But I’d suggest the first step would be to “warehouse” your media. This means to put all your media on your server directly, and then link to them within RW.

One way to warehouse may be through the Repository stack (especially for images) or a traditional FTP app such as Transmit or Forklift. This long discussion has more details in places:

In addition this PDF guide from Will Woodgate may be quite useful:

I don’t know if it’s the newest version (1.0) but it’s simply the one I could find the easiest.


(NeilUK) #7

I’d image you need to open the file the day before you want to work on it. 34 GB? Seriously?


(Joe Workman) #8

:exploding_head:


(Isaiah Carew) #9

I…
This is…
Waaaaa???…

Totally lost for words. You have beat the record for largest project file ever by at least 5x. This is truly astounding. Also probably extremely unwise, but astounding too.

My recommendation for moving forward is:

  1. Stop that.

There’s no step 2.

Whatever the reasons for doing that, the reasons not doing that far outweigh. Project files over about 1GB are the source of infinite problems. You are well and truly beyond that rule of thumb.


(Mark Buehl) #10

First, thanks for the comments. Second, this large project opens in a minute or two on my iMac. Opening and working with the large project is not the issue. No issues there. The only issue is how long it takes to convert the stacks into HTML code, now over 15 hours. I guess I have to live with that since I do not have a lot of images yet, mostly just links and data.

In researching what I wanted to use for this project, I did not find any information on RapidWeaver limits to the number of pages. Or if it was good to use on large projects. I knew going in I would have a large number of pages since I wanted a page for every community in Wisconsin, 3,438 pages to start with, plus history, lodging, restaurants, etc. Since the early 1990’s I have used a number of Website development programs and I like the idea of stacks to speed up development. I do not like Dreamweaver after using it for years or BBEdit for this project. Nor did I want to use WordPress or any online solution. So I guess I am using RapidWeaver for this project unless someone suggests something just as fast to develop a large number of pages. Sometime in the future once I get a base, I might start coding some pages in Dreamweaver or something like that again, maybe.

Again thanks for your comments.


(Isaiah Carew) #11

Any time that you have a large number of very similar pages then a dynamic database (or maybe static file) driven site is what you need.

Sites like Amazon don’t manually build a new page for each product. They just have a database of their products and a few generic pages that get filled in with the data.

There are tools in RW to build sites like this (I’ll let @joeworkman @Elixir @willwood @yuzool and the others tell you about how to get that done).

That way you could build a single community page (or perhaps a handful) that is filled in with the community data, images, and links from the database. This would hopefully mean a much smaller site that opens in second and publishes in minutes that is much more easily maintained – simply changing, adding, removing data from the database would update your site – perhaps without ever touching RW.

If the idea of an online database is distasteful – and there are really valid reasons that it may be – then a similar system can be built using a “static site generator”. This lets you manage content by simply editing text files on your server. This has all the benefits of a database – but just uses files and folders to do the trick.

Anytime you start doing repetitive tests – especially into the many hundreds of repetitions – and definitely when the file sizes grow over 1GB, it’s time to ask: Is this the right tool for the job? Is there another way to get this done?

Whatever you decide, the next time you have to wait 15 hours to publish your site, perhaps use some of those hours to research ways other ways to accomplish this task. If you post more about the details of your project I’m sure many here would have good ideas to help you on your way.

I don’t normally recommend hiring consultants – I like to think that most projects can be accomplished with a bit of effort – that’s the best part of RapidWeaver. But this sounds like a really big project – with a lot of time invested – perhaps investing a bit of money in hiring a consultant to help re-architect this massive beast into something a bit less beastly. There are lots of good for-hire consultants here on the forum and, in this case it would probably be money well spent.


(Mark Buehl) #12

Thanks for the info, but I did not want to go with a database approach for a number of reasons. Since the publish part is the only time consuming part, I guess I will have to live with it.


(Doug Bennett) #13

There are a few folks that have quite large sites with rapidweaver. Most have broken the sites into Multiple project files. Might make things a bit more manageable as well as faster.
You may want to consider searching the forum for some hints.


(Isaiah Carew) #14

As I mentioned, if you want to avoid a database, then you need to go with a static site generator. It gets the same job done, without the complications or security risks of a DB.

However, in my expert opinion, the path you’re on will only lead to frustration, pain, and eventual failure. I cannot be any more blunt: you must to do something different. The sooner the better.

However if you choose to ignore my previous advice, listen to this one thing: make a backup. I’m afraid that you’ll probably need it.

Isaiah


(Greg Schneck) #15

First, carefully consider what everyone else here has said. There is a lot of wisdom and expertise involved.

Secondly, the site I maintain is 1500 pages and it is easy to maintain because I have it broken down into several separate project files. But the TOTAL size of all project files is about 550 mb. Why is your 2k pages 34 gb??? Isaiah is right… that is insane.

Third, I can’t remember the last time I updated (published) the entire web site. Something is added every day so I do publish daily but it is generally one or two new pages and 2-3 updated pages. I just use single page publishing. Navigation changes or anything else (partials, etc) that effect the entire site are rare so there is no need to upload the entire 1500 pages to add or change one. I have the site broken down into “reports,” “store,” “directory,” and such. And at that, if I need publish an entire “section” (project) I publish locally and then use ftp. But I reset, 90% of my daily “publishing” (upload) is a single page here… a single page there… You just need to consider if your change is affecting one page, a “section” of the site, or the entire site.

But again… consider the advice here. A 34 gb project taking hours to upload… that is just not right. Period.