I have now had the precise same issue on two sites: getting invalid certificate messages when testing connections that work with FTP but not SFTP or FTPS. In both cases I have not had problems connecting via FTPS on Cyberduck.
In the one I am currently working on I had RW SFTP working fine when connecting to sandbox.domain, but when I switched over to another subdomain.domain I got FTP to work, but am getting a certificate invalid messages when testing SFTP (or FTPS), using the same setup as worked for sandbox.domain, except for the new username and password.
I went into the certificate and checked “always trust” wherever I could. Looking into the certificate details I see there is Basic Constraints extension that is critical but “certificate authority” is set to “no”
