PHP stream wrappers and Windows Certificate Store with Proxy - php

Question

Setup/Environment:
In our PHP application, we sometimes need to make HTTPS requests from PHP to other servers. The setup in question is as follows:
We are using PHP stream wrappers for doing the HTTP requests (using Guzzle HTTP). We are doing this because stream wrappers support using the Windows Certficiate Store for certificate verification.
The server runs on Windows.
We use a proxy on for the HTTPS requests.
The firewalls are configured to allow
Access to the servers we are doing our requests to.
Access to all certificate revocation lists relevant for the certificates used.
Our problem:
Sometimes, out of the blue, our HTTPS requests fail, with certificate validation errors. This problem persists, until someone opens a remote desktop session to the server and requests the very same URL we are trying to query in the servers Internet Explorer. After that, our PHP application can do its requests as it should.
Question:
What is the problem here? And what can we do to analyse this further?

Answer 1

If that were a Guzzle problem, it would happen every time.
However, do try to issue the same HTTPS call using cURL to both verify this is the case, and see if by any chance the cURL request also temporarily clears the issue, just as Internet Explorer does.
But this rather looks like a caching problem - the PHP server request is not able to properly access (priming certificates) the Certificate Store, it is only able to use its services after someone else has gained access, and only as long as the cache does not expire. To be sure this is the case, simply issue calls periodically and mark the time elapsed between user logging in and using IE, and Guzzle calls starting to fail. If I am right, that time will always be the same.
It could be a permission problem (I think it probably is, but what permissions to give to what, that I'm at a loss to guess at). Maybe calls aren't allowed unless fresh CRLs for that URL are available, and PHP doesn't get them). This situation could also either be fixed temporarily by running a IE connection attempt to the same URL from a PowerShell script launched by PHP in case of error, or (more likely, and hopefully) attempting to run said script will elicit some more informative error message.
update
I have looked into how PHP on Windows handles TLS through Guzzle, and nothing obvious came out. But I found an interesting page about TLS/SSL quirks.
More interestingly, I also found out several references on how PHP ends up using Schannel for TLS connections, and how Windows and specifically Internet Explorer have a, let us say, cavalier attitude about interoperability. So I would suggest you try activating the Schannel log on Windows and see whether anything comes out of it.
Additionally, on the linked page there is a reference to a client cache being used, and the related page ends up here ("ClientCacheTime").

Answer 2

Its not an application problem.
I am 99% sure this is routing problem and in some circumstances packets are dropped in the router. I would look at the network, change the environment or, if possible, do some network sniffing or monitoring.
If You have a decent network infrastructure, You can do SNPM traps for request count and timeout data collecting (from routers and switches) and ingest it in Elastic APM. This would give You quite detailed time-series analysis.

Answer 3

You can see this https://github.com/guzzle/guzzle/issues/394 verifyis the problem. and if you make the verify to be false that will make your system expose to security attack.
// Use the system's CA bundle (this is the default setting)
$client->request('GET', '/', ['verify' => true]);
// Use a custom SSL certificate on disk.
$client->request('GET', '/', ['verify' => '/path/to/cert.pem']);
// Disable validation entirely (don't do this!).
$client->request('GET', '/', ['verify' => false]);
These are the Request Options and you can see how to do the SSL certificate verification. They describe the issue as the following
Not all system's have a known CA bundle on disk. For example, Windows
and OS X do not have a single common location for CA bundles. When
setting "verify" to true, Guzzle will do its best to find the most
appropriate CA bundle on your system. When using cURL or the PHP
stream wrapper on PHP versions >= 5.6, this happens by default. When
using the PHP stream wrapper on versions < 5.6, Guzzle tries to find
your CA bundle in the following order:
Check if openssl.cafile is set in your php.ini file.
Check if curl.cainfo is set in your php.ini file.
Check if /etc/pki/tls/certs/ca-bundle.crt exists (Red Hat, CentOS, Fedora;
provided by the ca-certificates package)
Check if /etc/ssl/certs/ca-certificates.crt exists (Ubuntu, Debian; provided by
the ca-certificates package)
Check if /usr/local/share/certs/ca-root-nss.crt exists (FreeBSD; provided by
the ca_root_nss package)
Check if /usr/local/etc/openssl/cert.pem (OS X; provided by homebrew)
Check if C:\windows\system32\curl-ca-bundle.crt exists (Windows)
Check if C:\windows\curl-ca-bundle.crt exists (Windows)
The result of this lookup is cached in memory so that subsequent calls in the same
process will return very quickly. However, when sending only a single
request per-process in something like Apache, you should consider
setting the openssl.cafile environment variable to the path on disk to
the file so that this entire process is skipped
See also and how to ignore invalid ssl certificate errors in-guzzle 5 and guzzle-request-fails