No, I'm fully aware we don't trust the CAs with our personal data. We're trusting the CAs to vouch for the organizations to whom they issue certificates. But now there are hordes of CAs, some of whom may not be particularly trustworthy, but the browser makers don't descriminate (much).
As a result, we have CAs that we're supposed to trust because our browsers accept them, but those CAs are passing out SSL certs like candy to anyone with a few bucks.
While we're not directly giving our personal data to the CAs, we're trusting the organizations they vouch for on the basis of the supposed trustworthiness of the CAs, when in fact most of them are utterly opaque and unknown to us, thus indirectly trusting them to protect our personal data.
Again I say, anyone on the internet should look at the diagram, look at the list of signing authorities their browsers trust, and ask themselves, "who the hell are all these people and why do I trust them?"