🦄 UPDATED 🦄 RapidWaver 7.1 Developer Beta 2 (18041b)

RapidWeaver Developer Network
1

Important Developer Changes - Read Now!

A lot of things have changed in the way we load and save plugins, please test your plugins thoroughly and be on the look out for data-loss and corruption.

RapidWeaver 7.1 is pretty spesh, as we’re now lazy loading plugins, this makes opening RW projects super, super quick. In our tests ALL sites now open in under a few seconds, no matter the size they are. It’s pretty incredible and we think users are really going to get a kick out of it.

We’ve also added a new area in the page source list for plugin settings, details here: http://forums.realmacsoftware.com/t/api-for-hiding-and-showing-the-page-inspector-for-your-plugin/6916/18?u=simon and here:

What’s new in 7.1?

  • Fix the typo in +clearSharedPluinDataForDocument:
  • Removed RWPathUtilities from RMKit (FYI: just incase you were using it)
  • Improved the global plugin data storage
  • Added back in support for re-publishing all files for a single page
  • SiteLok plugin no longers breaks Preview mode
  • Colour arithmetic in themes now works again
  • Fixed an issue where the Filename extension was ignored when exporting (again)
  • Fixed an issue where an updated resource would not get re-published
  • Resizing Gifs now works (animation is not supported in resized gifs, yet)
  • Cleaned up plugin loading code, it’s good to be clean!

WARNING: Do not use this build on production sites, always work with back-up projects:
I agree to only use this on projects that are backed up, gimme the build now… OLD BUILD REMOVED

UPDATE: New build 7.1 (18052b ) is now available (18th July 2016) https://dl.devmate.com/com.realmacsoftware.rapidweaver/18052b/1468849490/RapidWeaver-18052b.zip

Basic stacks based sites should work fine, however you may see some data loss or corruption if your project uses partials. @isaiah is currently working on a new build of Stacks that will fix this and play nicely with RW7.1 :slight_smile:

On the public release we’ll block loading of previous versions of Stacks that are not compatible so as to protect all our users.

Hopefully @isaiah can post a new beta of Stacks here that is compatible with 7.1 for us all to use and test!

If you have any questions or concerns please post here and we’ll help you out as always.

Happy Weaving!

Cheers,

Dan

4 Likes
2

@dan all problems corrected. Stacks 3.1.1 beta 8 on the Slack channel.

I’ll post a public beta later today.

2 Likes
3

@isaiah Awesome, nice work :smiley:

Fingers crossed that means we should be able to do a public beta of RW 7.1 next week! :tada:

1 Like
4

yes. i need to release another variant for Stacks 2 to make the same architectural updates — but it is a much smaller problem there (no partials).

isaiah

2 Likes
5

BTW: schedule-wise i’m trying to put the finishing touches on this and want to have it all squared away before the weekend

2 Likes
6

Dropped beta 9 – and put on the forum for everyone.

I think we’re good to go. I’ll release a Stacks 2 update in a couple days – but no need to hold up RW 7.1 for that.

Thanks for all the big API updates guys. Awesome work!

isaiah

2 Likes
7

Nice work @isaiah, we’ll probably push out a public beta of 7.1 on Monday/Tuesday all being well :grinning:

Happy times!

8

:thumbsup: good plan

isaiah

9

@isaiah I’ve just had a play with 3.1.1b9 and found a problem with partials not showing here when the project is loaded. After clicking on the pages where the partials exist, they do show just fine.

Image 2016-07-15 at 9.53.19 am.png966×496 24.2 KB

Test project: https://cl.ly/2g2R0D3C0x2Q

10

such bug report. many steps. wow. :dog2:
i will post my bug reports on forum too? :stuck_out_tongue:

seriously tho…
it’s expected behavior in 7.1.
bummer.

tested other things.

  • migrate method: slower than 7.0
  • move to global storage – not backward compatible (see prev discussion)

if there is a way to use the new api that’s backward compatible and performant i’m all ears.

11

I’ll try and ignore the snarky comment at the start of the message. I asked Tom to post it here to keep all 3rd party devs in the loop.

hmm, I don’t think that’s expected behaviour :frowning: Shouldn’t it just work with the new build of Stacks?

Do we need to change anything to get this to work, or is it something you can fix up in Stacks?

Let us know, we want to make this smooth for all the weavers out there!

12

lots more details. 99% snark free:

hmm, I don’t think that’s expected behaviour :frowning: Shouldn’t it just work with the new build of Stacks?

to be honest, i thought this was the behavior you were aiming for: loading the content later. i hadn’t even considered that you wanted me to try to keep the 7.0 behavior until this morning.

i have been working on ways to create the 7.0 behavior – but i assumed that would be subversive – going against what you were trying to achieve.

just taking a step back for a moment to talk about the intent of the new API from a 10,000ft view:
if you look at a lot of the large documents (that are stacks heavy) – they use partials a lot. Not always, but Foundation encourages it – which means it’s a popular thing. If I load those partials at the time the document loaded it will undo a large part of the performance gain that you’ve just made. Right? Or what am I missing?

OK, tech details

other than the current beta build these are a couple things I’ve at least tried:

  1. i have a branch that does do the deserialize/unarchive on each page, every time, in the migrate method. verdict: it’s slow. it’s doing all the work of 7.0 to deserialize/unarchive everything each time (to find the partials and load them) – plus some extra because it’s sort of out of order – then it has to do it again later as the pages are viewed.
    but, this is my fallback plan if support load from the hidden partials becomes too high. i think it does work. but i have not tested a bunch of the weird corner cases like loading S2 docs etc.

  2. i can do the deserialize/unarchive stuff just once, then save it to the global store. that’s great and how the api is surely meant to work. but this makes large files slow again because a lot of their bulk is in those partials. AND it is also incompatible with older versions of stacks/rw.

and it’s not a minor incompatibility. if i made this change and moved partials into the global store – and a user opened the doc with Stacks 3.0 the file would appear to have holes everywhere a partial was. with auto-save enabled those holes would be written immediately and become permeant. so even an accidental click could be catastrophic data loss.

Tom mentioned in an email that we could prevent users from doing it – but it would cause a crash: “which causes a crash when you try and open a document from a newer version”. A crash is better than data loss for sure, but not exactly a solid win either.

so options with pros/cons

  1. quirky partial behavior
  • low risk (zero change)
  • already done
  • fast
  • no crashes
  • backwards compatible
  1. read everything all the time
  • medium risk (small change – new load sequence)
  • implemented but untested
  • very slow
  • no crashes
  • backwards compatible
  1. read once and move to global store
  • high risk (large change – new format)
  • 2-3 weeks work?
  • fast(er) – but probably not with large files – where it really counts
  • crashes when opened in old version
  • NOT backward compatible

so far i’ve chosen #1 with an option to quickly move to #2 if support load for the quirky behavior becomes to high.

i’ve not given up tho. i think there might be some more non-obvious solutions to this: breaking partials up / storing them differently / removing images from the equation / etc. – that might be both fast and compatible. but i don’t have any concrete ideas yet. and i think everything in that realm is likely to be a much larger/longer coding effort.

Isaiah

13

also probably worth mentioning…

all of this partials stuff is something that has been entirely on the back burner for me. i’ve been mulling it over – but not really working on it much.

the changes that i’ve been making for the past week have nothing to do with this – they involve an API specific feature – the node IDs – which have traditionally been document-wide-unique (and to answer Tom from last week – these are completely separate from the each stacks UUID – which they do have and work just fine). Node IDs are used by the 3rd party stacks API as a way to identify each element. They’re: unique, persistent, and SEO friendly (which is why we don’t use the UUID – which are pretty unfriendly).

Ensuring document uniqueness in the new API has become very difficult. It’s possible using the migration feature, but has the same slow vs. incompatible challenges I mentioned above. But YEAH!!! For this problem, there IS a 4th option that was really nice…

I simply removed the guarantee of document-wide uniqueness and then cope with those consequences. This is easy – the only challenge is ensuring the downside is very minor (TEST TEST TES).

There are two other API specific features that rely on document-uniqueness, but i simply mangle those names a bit now for partials and a similar kludge for Global-Content-IDs. Both changes are unlikely to affect 3rd party stacks and – no matter if the do – I’ve updated the Stack API version number – so in the rare case that it does have an effect those stacks can use the Min/Max versioning of the stacks API to migrate to the new version without fear of data loss.

Better yet, these changes flow seamlessly back and forth between versions. To be honest, this was unplanned serendipity. It’s just that the document-wide uniqueness property was only important for those two very limited cases. Done and Done!!!

There is still a case: Creating a document, with global content, in Stacks 2, with RapidWeaver 7.1 – that could potentially cause newly created global stacks to accidentally acquire the global content ID of a pre-exiting stack on another page – and thus inherit its content.
I’ll release a new S2 version shortly to fix this too.
But I’ve tested this and although strange and surprising when seemingly random content POPs into view in a new stack, it doesn’t really do any harm, and the solution is simple: update. Case closed.

1 Like
14

Hey @isaiah,

From the user’s perspective we would like stacks to function exactly the same in 7.1 as it does in 7.0

So, this discounts option 1 because the partials don’t show up in the stacks browser until you select a page that contains a partial.

Option 2 isn’t actually going to work. The migrate method has been designed to help migrate data from each page to a global storage area. It will only be called ONCE. After the migration has completed, the page is marked as migrated and migration will not occur again for that page.

So option 3 is the only viable option here.

I’d like to make it clear that RW7.1 documents will not open in RW7.0 and if a user tries to do so it will result in a crash. That’s a bug on our part and it’s been corrected so future versions will not suffer from this. A crash is better than data corruption so it’s acceptable in our eyes.

We’ve added in a version check for Stacks that stops versions <= 3.1.0 from loading. The reason for this is that if a user tries to use Stacks 3.1.0 in RW7.1 it will fail to load the partials, and potentially this could cause a lot of confusion / support. There isn’t anything you can do to stop this on your side, hence us implementing the limit.

I’ve not tested Stacks 2 with this limit in place. I’m pretty sure RW7.1 will reject it, but we can easily add a separate check for Stacks 2 if necessary.

15

About #1 - OK. That’s good I suppose. We can ignore the beta then. I’m not going to pull it as it’s no harm to 7.0 users and fixes a Sierra bug.

About #2 - OK. Super bummer. I didn’t completely grok that. You can ignore the build I sent you yesterday – it did exactly that.

About #3 - Like I said, this is 2-3 weeks work (wild ass guess). To be honest I’m not at all sure how to approach this – it requires slicing out the partial Node tree saving it separately – and then re-injecting it at load-time on the fly. It seems feasible. In theory. :grimacing:

Moving data (especially images) is really hard. I spent, no joke, several months beta testing that stuff in S3. Mostly because I lost all of 2012 to the dreaded “Missing Images Problem” – that nearly killed my company – mostly due to the high price of bourbon and psychiatric care in the US. (joking. mostly. ok not really.) :cocktail::sob:

Attempting this in such a short schedule makes me reach for a paper bag to breath into.

Give me a couple days to mull this over. Maybe there’s some way to cut this Gordian knot. I usually think of something. :bulb:

We’ve added in a version check for Stacks that stops versions <= 3.1.0 from loading

Thanks.
Does that also crash? Or not?
It seemed like from your email that it would. But now I’m not sure.

assuming it doesn’t crash (omg :tada:):

  • What sort of error message do you throw?
  • Does it just give the generic “plugin not installed” deal?
  • Is there any API for that sort of thing – it would be great if I had some control over that – rather than hard coding the version numbers into the app.
    e.g.: the sandwich format has a version numbering deal built in – it would be great if there was a way to say what the min/max version number of a specific plugin or a specific sandwich was capable of.

I’ve not tested Stacks 2 with this limit in place. I’m pretty sure RW7.1 will reject it, but we can easily add a separate check for Stacks 2 if necessary.

I actually don’t think this is necessary. Don’t worry about Global ID issue in Stacks 2.

That said, the rest is purely for educational value in case you’re curious…

TL; DR: it’s rare. it’s easy to fix. and it’s not very dangerous. meh!

When you asked last week I think I said it was a problem – I was wrong – sorry about that. My assumption that the uniqueness was relied upon was simply incorrect.

You can see the Global Content ID by editing a Text stack. In edit mode it will be the last field in the Info sidebar. This issue exists in Stacks 3 too – but with the advent of partials – globals are rarely used. Even in S2 it was well hidden. Only a few brave souls used it.

The problem: the default Global Content ID is generated with the Node ID. If duplicate Node IDs exist then duplicate Global IDs do. When this happens, and the user has global content enabled, there is a chance that the user will mistakenly link stacks together that should not be. I suspect would be quite difficult to see in practice because not every stack is global – so the overlap of IDs that are being used is probably very very rare.

The solution for S3 is straightforward: remove the UI. The feature has been on the chopping block for a while. It will improve everything.

For S2 it’s nearly as trivial, just use the node’s UUID instead. It’s ugly UX-wise – but functional – and the user can change it if they like. If this was a oft-used feature, I’d be more concerned, but it isn’t, so I’m not.

For the existing versions out there: it’s rare and mostly just a little confusing to users. No big deal.

Isaiah

16

Yo!

The check for version <= 3.1.0 does not crash! :tada:

The check is only in RW7.1 and will show something like the following

Image 2016-07-16 at 7.10.54 pm.png1064×592 75.9 KB

You can test this yourself too, just install an older version of Stacks and run RW7.1 :wink:

It’s only RW7.0 that will crash if the user attempts to open any document saved in RW7.1. When we save a document we store the version number of RW in the document itself. So when RW7.0 comes to open a 7.1 document it attempts to tell the user it’s incompatible. Unfortunately, the alert was shown on a background thread, hence the crash.

I’d rather not hard code version numbers into RW, but I think it actually makes sense. When we’re changing things, we can prevent certain existing plugins from loading if they are known to cause problems. There isn’t an API for this that I know of - if there’s demand for it we could look at adding it.

See how you get on with the migration of partials to the global store. I appreciate it’s the trickiest option, but I think it’s the best moving forward. Keep us in the loop :thumbsup:

Ps. Missing image problem sounds like a nightmare!

17

if there’s demand for it we could look at adding it

trying so hard not to go full-snark on this one.

See how you get on with the migration of partials to the global store.

Did some initial planning today. Scoped out the points at which I can make the cuts to the object graph.

Still holding out hope that maybe there’s a more clever approach to this.

missing image problem sounds like a nightmare

Speaking of unique IDs… take it from me, don’t rely heavily on the uniqueness of RWDocument uniqueID.

Actually, “don’t rely on any code you don’t own” is probably even better.

Isaiah

18

@tpbradley @dan

hey guys, bug report:

synopsis:
the files dictionary of the sandwich sometimes returns paths that may no longer exist

severity:
show stopper. data deletion. file corruption. the symptom is identical to the Missing Images bug. danger Wil Robinson. i’m having a panic attack right now.

example document:
https://dl.dropboxusercontent.com/u/433436/image-load-demo2.rw.zip

requirements:
RW 7.1 both betas and Stacks 3.x any version.

to repeat:

  1. open the file in RW 7.1
    you should see the homepage – an empty Styled Text page
  2. in the Finder, rename the file to “renamed.rw”
  3. in RapidWeaver click on the Stacks page

expected result:
the page should have an “I :heart: RW” button image.

actual result:
the page has a broken image

reasons:
when the “initWithSandwich:” method is called on the the paths in the sandwich files dictionary appear as if it had never been renamed.

with the file on my desktop I see a path like this:

    "74818DA1-CB06-433C-84D3-8E27261C7906.jpg" = "/Users/iac/Desktop/image-load-demo.rw/Pages/2-Two/Data/74818DA1-CB06-433C-84D3-8E27261C7906.jpg";

note the path contains the original file name “image-load-demo.rw”, not “renamed.rw”

corollaries:
a similar issue exists when the file is deleted instead of renaming. in that case the path returned is to a non-existent file.

In all of these cases the RW 7.0 with all versions of Stacks behaves as expected.

a plugin bug?:

this seems very unlikely.

in the example the plugin is never provided with information about the new name of the file, and there doesn’t seem an obvious way to find that information.

in the deleted file corollary the image data is actually gone. the plugin was not given any opportunity to prevent the deletion, cache the files, or create an NSFileWrapper to maintain the inode.

solution:

going back to 2008 and the release of Collage 2 that exhibited a similar bug, i’ve recommend (ok, maybe pleaded would be a better word) that the contents of the sandwich be cached or use an NSFileWrapper. NSFileWrapper seems to be the best practice recommended by Apple for using bundle-folder files and seems as if it would fix this problem.

Isaiah

19

Thanks @isaiah - I’ve now got this ticketed to look at first thing on Monday, will keep you posted on the progress!

20

@isaiah Ok I’ve replicated this problem here and I’m pretty sure this is a bug in the sandwich stuff - investigating now :wink:

I completely agree with you - I would love to use NSFileWrapper for this stuff! It makes so much more sense. I did actually attempt this as part of 7.1 but had to pull it. The damn sandwich format holds the full path to every file in the files array. NSFileWrapper doesn’t allow access to the url - and rightly so. It’s based on inodes instead so renaming files don’t ever cause problems. So moving to NSFileWrapper would pretty much break every single plugin out there :confused: