PDO connection times out in try but works in catch - php

Question

We have an autoscaling group in AWS where upon launch database connections timeout. This is a Zend Framework 1 app using PHP 7.0.22-0ubuntu0.16.04.1 on Ubuntu 16.04.3 LTS
The code is baked into the AMI but during launch a userdata script will pull from git and phing configure the app. The database domain hasn't changed in years, the configure is mostly to take care of which elasticache instance to use. In other words, the baked in code already has the database configured and just gets overwritten with same value during configure step.
Once the ec2 instance is in the ELB, it begins getting hit on /health-check to see if things are OK by the load balancer. Inside this controller is the following code:
public function healthCheckAction() {
try {
/* #var $DBConn Zend_Db_Adapter_Pdo_Mysql */
$DBConn = Zend_Registry::get('multidb')->getDb();
// test guide service (most likely will be from memcache, unlikely to hit db)
$guideService = $this->_apiGuideService();
$guideService->isLoaded();
// this line fails and throws an exception
// I put host in here just so an error would include it in throw during this phase instead of catch phase (where it works)
// test raw db connection
$dbh = new PDO("mysql:host={$DBConn->getConfig()['host']};dbname={$DBConn->getConfig()['dbname']}", $DBConn->getConfig()['username'], $DBConn->getConfig()['password']);
$data = $dbh->query("SELECT '{$DBConn->getConfig()['host']}' as host, now()")->fetchObject();
// test database connectivity
// I put host in here just so an error would include it in throw during this phase instead of catch phase (where it works)
$sql = "SELECT '{$DBConn->getConfig()['host']}' as host, now()";
$DBConn->fetchRow($sql);
// test cache
/* #var $cache Zend_Cache_Core */
$cache = Zend_Registry::get('cachemanager')->getCache('default');
if (!$cache->load('health_check')) {
$cache->save(true, 'health_check');
}
echo 'Instance is healthy';
}
catch (Exception $e) {
header('HTTP/1.1 500 Internal Server Error');
echo 'Instance is unhealthy';
// get instance id
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://169.254.169.254/latest/meta-data/public-ipv4');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// get instance ip
$ip = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, 'http://169.254.169.254/latest/meta-data/instance-id');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$instance = curl_exec($ch);
// email us some info
$message = "Instance $instance failed health check. ssh ubuntu#$ip to investigate<br><br>" . $e->getLine() . " " . $e->getCode() . "<br>" . $e->getMessage() . "<br>" . $e->getTraceAsString(). "<br><br>";
ob_start();
// this works and returns access denied, not timeout
$this->runCommand('mysql -u examplecom_platform -h sg-rds-example.us-east-1.rds.amazonaws.com');
echo "testing DB with php<br>";
try {
echo "write host: " . $DBConn->getConfig()['host'] . "<br>";
echo "read host: " . $DBConn->getConfig()['host'] . "<br>";
$dbh = new PDO("mysql:host={$DBConn->getConfig()['host']};dbname={$DBConn->getConfig()['dbname']}", $DBConn->getConfig()['username'], $DBConn->getConfig()['password']);
$data = $dbh->query('select now()')->fetchObject();
echo "query database without zend:<br>";
print_r($data);
// this line works and prints out
// stdClass Object
// (
// [now()] => 2018-01-09 14:47:12
// )
$dbh = null;
} catch (PDOException $e) {
print "Error: " . $e->getMessage() . "<br/>";
}
// this all work/show correct IP
$this->runCommand('nc -vz sg-rds-example.us-east-1.rds.amazonaws.com 3306');
$this->runCommand('host sg-rds-example.us-east-1.rds.amazonaws.com');
$this->runCommand('dig sg-rds-example.us-east-1.rds.amazonaws.com');
$debug = ob_get_contents();
ob_end_clean();
$message .= "<br><br>" . str_replace("\n", "<br>", $debug);
$mail = new Zend_Mail();
$mail->setSubject('[examplecom] Instance Failed Healthcheck v2')
->setFrom('noreply#example.com')
->addTo('alerts#example.com')
->setBodyHtml($message)
->send();
}
}
As I kept debugging things I would add more and more stuff to test the connection
The try statement throws an error of SQLSTATE[HY000] [2002] Connection timed out
But this exact same connection works in the catch and is able to query now() from the database.
This is where I am stumped, how could the same process timeout on first connection but work during the error catching?
Also I will only get 1 or 2 of these emails saying it cant connect, but then eventually by the time I can login to test some things it is working and connecting fine. Health-check reports happy and instance is kept in ELB.
Any ideas or suggestions to add more debugging?