SSL Certificate Verification Issue

[furkansimsekli]

aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host ee.hacettepe.edu.tr:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'ee.hacettepe.edu.tr'. (_ssl.c:000)")]

We had this issue last year, and that time it wasn't due to our infrastructure but due to IT people of Hacettepe. Disabling ssl verification could be a solution. I thought it would get fixed but still is an issue after 2 weeks.

[div72]

@furkansimsekli This looks different, the TLS verification complains about the hostname mismatch (no idea how they managed to break that unless a self-signed cert leaked). That should also break the site on browsers, so should be fixed relatively "quickly".

---

The previous problem was that servers were not serving intermediate certificates in chain leading to the scraper not being able to verify the chain. I researched it a bit, apparently browsers use a method called AIA Fetching/Chasing to work around the issue. If the issue presents itself again, it might be worthwhile to look for a Python package which implements that.

[furkansimsekli]

@div72
For the following websites, bot keeps getting ssl verification issues:

• https://sksdb.hacettepe.edu.tr
• https://ie.hacettepe.edu.tr
• https://mat.hacettepe.edu.tr
• https://ee.hacettepe.edu.tr
• https://edebiyat.hacettepe.edu.tr
• https://abofisi.hacettepe.edu.tr
• https://bidb.hacettepe.edu.tr
• https://jeomuh.hacettepe.edu.tr
• https://hidro.hacettepe.edu.tr
• https://ide.hacettepe.edu.tr
• https://sporbilimleri.hacettepe.edu.tr
• https://shmyo.hacettepe.edu.tr

If I run the bot in my own computer, I still get these errors for the same departments. I'm going to inform Hacettepe IT about this.

Firefox can open these websites without any warning about SSL certificates.

[div72]

@furkansimsekli See the logs, the latest error is unable to get local issuer certificate. That was the issue we got last year, where the server did not provide intermediate certs. That can be solved with the method I mentioned in my last comment.

[furkansimsekli]

Problem seems to be resolved for every department except https://ee.hacettepe.edu.tr

We already talked about this, but let me write here as well. The solution you mentioned might require the use of packages outside the Python, I honestly don't feel very good about this, especially if it affects the development/test.

[div72]

There are two packages, one of which requires OpenSSL. Is that what you meant by "packages outside of Python"? Nevertheless, neither of the packages are packaged in NixOS which (while is not a deal breaker) requires some work that I'd rather avoid.

[furkansimsekli]

All problems have been fixed a few weeks ago thanks to our own attempts, but I want to implement a new feature to disable ssl verification temporarily, therefore I will make this an enhancement issue rather than bug.