So I cannot believe that I never realized this…
The cache busting links option in RW is great. It ensures that when you republish all files, that browsers will redownload an asset because of the parameters added to the asset URLs.
However, the rwcache key that gets added to the URLs is unique for each and every page. This means that the browser will never cache any assets because the URL to them is unique on every page.
What needs to happen is that the rwcache value needs to be the same on all pages when republishing. It can change for each and every publish, but if I republish all files, that rwcache keep should be the exact same value on every page on my site.
Hope that this makes sense.