I'm calling a common function from Lumen queue and cron job. During user interaction this function is called from queue ( async purpose ), if something goes wrong with queue execution for example lock wait timeout etc. I have a cron job which is scheduled for every 15 mins to process older transactions. There is no issue when cron is executed but whenever queue is executed MySQL server has gone away error is occurring. Error is occurring at the line DB::connection()->getpdo()->exec('BEGIN');
I did some research on internet, those articles are saying this kind of error will raise when we try to insert large data and we can avoid this error my increasing max_allowed_packet. But error is occurring at beginning line itself, I'm not trying to insert large data ( mostly it should be in KBs ) and moreover same function is working when cron executes for every 15 mins. Following are my code snippets,
public function processTransaction($data)
{
try {
$this->validate($data);
DB::connection()->getpdo()->exec('BEGIN');
// Save & Update DB
DB::connection()->getpdo()->exec('COMMIT');
} catch (Exception $ex) {
DB::connection()->getpdo()->exec('ROLLBACK');
Log::error($ex->getMessage() . '-' . $ex->getTraceAsString());
throw new AppException($ex->getMessage(), $ex->getCode());
}
}
Above is the initial version I tried, in this case error was at ROLLBACK statement. Later I had updated to following
public function processTransaction($data)
{
try {
$this->validate($data);
DB::connection()->getpdo()->exec('BEGIN');
// Save & Update DB
DB::connection()->getpdo()->exec('COMMIT');
} catch (Exception $ex) {
Log::error($ex->getMessage() . '-' . $ex->getTraceAsString());
try {
DB::connection()->getpdo()->exec('ROLLBACK');
} catch (Exception $ex) {
Log::error($ex->getMessage() . '-' . $ex->getTraceAsString());
}
throw new AppException($ex->getMessage(), $ex->getCode());
}
}
Here the error is at BEGIN statement and PDO exception error code is in string, which is also creating the issue for argument 2 AppException (extends Exception class), which excepts argument 2 as integer. I think PDO exception issue can be handled by separately catching PDO exception but I want to know why MySQL server has gone error is getting.
Following is the error logged at BEGIN statement
Error while sending QUERY packet. PID=28968-#0 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns{closure}(2, 'Error while sen...', '/web-server/...', 164, Array)
Related
I call an API to send SMS and save it's response using Redis::throttle to limit the rate of the call to 5 calls every 60s with :
Redis::throttle('throttle:sms')->allow(5)->every(60)->then(function(){
//->API call
//->Save response
},function($error){//could not obtain lock
return $this->release(60);//Put back in queue in 60s
});
I didn't specify any $tries because if the lock cannot be obtain, it count as a try and if I process a long queue and the lock cannot be obtain many time the job will fail without any "real" errors.
But I dont want the job to be processed forever, if there is a real error (like if the response cannot be saved) it should fail without retry especially if the error appends after the API call as a retry will send another SMS (which I definitely don't want).
What I've tried :
Redis::throttle('throttle')->allow(5)->every(60)->then(function(){
try{
$response = MyAPICall();
$test = 8/0;
saveResponse($response);
} catch(\LimiterTimeoutException $e){
throw $e;
} catch(Exception $e){
Log::error($e);
$this->fail($exception = null);
//$this->delete();
}
},function($error){//could not obtain lock
Log::info($error);
return $this->release(60);//Put back in queue in 60s
});
If there is an exception because the lock cannot be obtain, I throw it back to let the queue handle it but if it's another exception, I log the error and fail or delete the job.
But it's not working with either delete() or fail(), the job is always retried.
How can I remove the job if there is an exception other than the LimiterTimeoutException ?
I was missing a "\" before Exception in my catch. Here is the fix code :
Redis::throttle('throttle:sms')->allow(5)->every(60)->then(function(){
$response = myAPICall();
try{
$test = 8/0;
SaveResponse($response);
}
catch (LimiterTimeoutException $exception){
throw $exception;//throw back exception to let redis handle it below
}
catch (\Exception $exception) {
Log::error($exception);
$this->fail($exception);
}
},function($error){//could not obtain lock
return $this->release(60);//Put back in queue in 60s
});
I added $this->fail($exception) to make the job to show up as "failed" in Horizon.
I had MappingException somewhere in my code. Please help me how to find place in a code where that exception is raising?
[29-Mar-2019 22:30:26 Europe/Moscow] PHP Fatal error: Uncaught Doctrine\Common\Persistence\Mapping\MappingException: Class 'Client' does not exist in /var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/MappingException.php:93
Stack trace:
#0 /var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/RuntimeReflectionService.php(24): Doctrine\Common\Persistence\Mapping\MappingException::nonExistingClass('Client')
#1 /var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/AbstractClassMetadataFactory.php(251): Doctrine\Common\Persistence\Mapping\RuntimeReflectionService->getParentClasses('Client')
#2 /var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/AbstractClassMetadataFactory.php(284): Doctrine\Common\Persistence\Mapping\AbstractClassMetadataFactory->getParentClasses('Client')
#3 /var/www/virtual/account/htdocs/vendor/doctrine/orm/lib/Doc in /var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/MappingException.php on line 93
If i will do something like that will it help or i'll get the same stack trace which i have now?
try {
// application init
$app->init();
} catch (Doctrine\Common\Persistence\Mapping\MappingException $e) {
error_log("MappingException: " . $e->getTraceAsString());
throw $e;
}
UPDATE
Problem wasn't related to Doctrine exactly but to length of stack trace which too big and doesn't fit to 1024 bytes by (default php's setting). So, just need to increase max length of stack trace which can be done by follow command:
ini_set('log_errors_max_len', 0); // 0 - infinity length allow
The exception was thrown at:
file:
/var/www/virtual/account/htdocs/vendor/doctrine/persistence/lib/Doctrine/Common/Persistence/Mapping/MappingException.php
line: 93
If you want to see what happened before (a cause) you can check trace of your code. I don't see whole your code but:
If you have Uncaught exception even you wrapped your code with try catch block there are only two main ways here:
1. You don't catch the exception. Probably because of not escaping the exception you want to catch or the exception is different than you try to catch. Or you re-throw it (probably your case)
try this way of catching with \Exception
try {
// application init
$app->init();
} catch (\Exception $e) {
error_log("MappingException: " . $e->getTraceAsString());
//no throw
}
if this works then you may change your code to have such way of catching exceptions
try {
// application init
$app->init();
//note backslash
} catch (\Doctrine\Common\Persistence\Mapping\MappingException $e) {
error_log("MappingException: " . $e->getTraceAsString());
//no throw
} catch (\Exception $e) {
error_log("GeneralExcpeption: " . $e->getTraceAsString());
//no throw
}
You always catch exceptions starting from most specific (Doctrine in your case) to less specific (\Exception)
Probably it will be much easier to work with getTrace than with getTraceAsString() to see the execution order that led to exception.
but do yourself a favor and limit the amount of entries you display ie this way:
try {
// application init
$app->init();
//note backslash
} catch (\Doctrine\Common\Persistence\Mapping\MappingException $e) {
var_export(array_slice($e->getTrace(), 0, 5));
//no throw
} catch (\Exception $e) {
var_export(array_slice($e->getTrace(), 0, 5));
//no throw
}
and go backwards to the place where you used Client if number 5 is not enough then increase it to 10, 15 etc to be able to go backwards where the execution was before an exception happened. Each entry has file and line number so you'll be able to recreate what happend.
Look at your code where you have your Client and check that you used proper use of that Client in the namespace you are. If there is a namespace keyword at the top of the php file where you use Client you need to be sure the Client is part of that namespace or use \Client to escape the namespace and treat the name Client as if there was no namespace.
2. The exception is thrown outside of your try catch block (probably not)
You can try to catch exceptions that are outside your try/catch block by registering your own function with handler. The tricky part is that this must be done before an exception is thrown so perfectly at the first line of the php code. Note this might be different line than the first line you wrote for example this can be application bootstrap.php.
you may check what is your app first line by running app with Xdebug, into debug mode.
here's example of putting exeption handler:
<?php
function exception_handler($e) {
echo "MyUncaughtException: " , $e->getMessage(), "\n";
}
set_exception_handler('exception_handler');
Problem wasn't related to Doctrine exactly but to length of stack trace which too big and doesn't fit to 1024 bytes by (default php's setting). So, just need to increase max length of stack trace which can be done by follow command:
ini_set('log_errors_max_len', 0); // 0 - infinity length allow
I just upgraded from 4.2 to 5.0. I got all of my commands working, but I noticed in one script I see an error (an expected error) which reports ErrorException. The problem is, it breaks my script from continuing instead of moving on the the next step in a foreach loop. The same error with the same script on 4.2 will report the error and continue.
4.2: Cannot connect to xyz.IP Error 60. Operation timed out
5.0: [ErrorException]
Cannot connect to xyz.IP Error 60. Operation timed out
For more context: I'm using the script to SSH into a couple of servers and run a Ping command. I'm using Phpseclib 1.0. I've tested phpseclib on my old 4.2 build and it works fine. 5.0 is where the problem started occuring.
Does anyone know how I can make the script continue to run after an ErrorException?
foreach ($query as $key => $value) {
$ssh = new Net_SSH2($value->origin_ip);
$key = new Crypt_RSA();
$key->loadKey(decryptIt($value->password));
if (!$ssh->login($value->username, $key)) {
exit('Login Failed');
}
$this->info(' Running Ping');
//$ssh->setTimeout(1);
if ($ssh->read('/.*#.*[$|#]/', NET_SSH2_READ_REGEX)) {
//echo "reading";
//$this->info(' Running Ping');
//$ssh->setTimeout(4);
$statusOutput=$ssh->exec("ping -c 1 -W 1 ".$value->destination_ip." >/dev/null 2>&1; echo $? ");
} else {
//echo "not reading";
$this->error("Unable to Read Ping");
}
}
To work with exceptions inline within a script, use a try...catch block:
try {
$value = someFunctionThatMayCauseAnException();
} catch (Exception $e) {
$errorMessage = $e->getMessage();
}
For more information, see the PHP manual entry for Exceptions
"Uncaught" exceptions will halt your script. Sometimes, that is the desired effect. For example, the SSH library you're using does not catch the exceptions that occur within the methods, they are allowed to bubble out to the calling script. Maybe your calling script catches them, or maybe you let them keep bubbling to your global exception handler. There are a number of ways to work with exceptions, but the general rule of thumb is that you don't catch them unless you're going to do something with it, like show an error message.
Your script would continue in the previous version because the error was, most likely, emitted as a warning or notice and returning false to indicate failure. With the newer PHP version, the library began emitting exceptions instead, at once indicating failure AND providing an exception object with details about the failure.
This means you'll have to restructure the logic within your loop instead of directly calling the function in a conditional if. You didn't specify which line is emitting the exception in your example, but for instance, this is one way that you could restructure to work with exceptions:
$errorMessage = false;
try {
$ssh->login($value->username, $key); // this code is attempted
} catch (Exception $e) {
// if an exception is emitted
// in the try block above, this block
// is reached. Otherwise, it is skipped
$errorMessage = $e->getMessage();
}
// $errorMessage can only be other than false from the exception catch block above
if ($errorMessage !== false) {
exit($errorMessage);
}
Solved it. I had a try catch which I modified.
Make sure the catch has a backwards slash like this:
try {
//code here
} catch (\Exception $e) {
print_r($e->getMessage());
}
When my Beanstalkd job has an error, like "exception 'ErrorException' with message 'Notice: Undefined index: id in /var/www/mysite/app/libraries/lib.php line 248' in /var/www/mysite/app/libraries/lib.php:248, how should Beanstalkd knows that an error has occured and mark it as failed so it can be retried again?
Install a monitor for beanstalkd which can be useful when developing/testing your app. Some alternatives that uses PHP: http://mnapoli.github.io/phpBeanstalkdAdmin/ and https://github.com/ptrofimov/beanstalk_console
As for handling the errors, you could define your own error handler for beanstalkd jobs and in that handler decide if you want to:
bury (put the job aside for later inspection)
kick (put it back into the queue)
delete (remove it, if this is safe for your application)
EDIT - Did you manage to solve your problem?
The best way might be to use a try/catch around your jobs to catch the exceptions, and then bury it if the exception is raised at the worker. If the exception is raised at the producer it is probably never added into the queue so no need to bury() then, but use a monitor to make sure.
If you want to try to define an own error handler for your object I´ve done something similar before by setting up a custom error handler for your class. It might be a ugly hack by trying to get the pheanstalk (job) object through $errcontext - but might be something to try.. Here is some pseudo-code I quickly put together if you want to try it out (created a subclass to avoid put code into pheanstalk class):
class MyPheanstalk extends Pheanstalk {
function __construct() {
//register your custom error_handler for objects of this class
set_error_handler(array($this, 'myPheanstalk_error_handler'));
//call parent constructor
parent::__construct();
}
function myPheanstalk_error_handler($errno, $errstr, $errfile, $errline, $errcontext) {
// get the current job that failed
foreach($errcontext as $val) //print_r($errcontext) to find info on the object(job) you are looking for
{
if(is_object($val)) {
if(get_class($val) == 'Pheanstalk') { //and replace with correct class here
//optionally check errstr to decide if you want to delete() or kick() instead of bury()
$this->bury($val);
}
}
}
}
}
Its your script that has the error, not beanstalkd.
try {
//your code from Line 248 here
} catch (ErrorException $e) { //this section catches the error.
//you can print the error
echo 'An error occurred'. $e->getMessage();
//or do something else like try reporting the ID is bad.
if (!isset($id)) {
echo 'The id was not set!';
}
} //end of try/catch
Here is the problem
There is a script that after X amount of time (unknown amount between 5 and 40 minutes) throws the following error: MySQL server has gone away which Kohana turns into a Database_Exception 2006 As a result some of the information is not saved to the DB.
Here is what I think might work
class Model_Bar extends ORM {
protected $_belongs_to = array(
'foo' => array()
);
public function save(){ //Extends the save method
try {
$result = parent::save(); //Try parent save
} catch (Database_Exception $e) { //Catch exception
if ($e->getCode() == 2006) { //If exception code == 2006 then DB has gone away
mysqli_ping(); //Try to refresh DB link
$result = parent::save(); //Try parent save again
} else { //Exception code != 2006
throw new Exception($e); //Throw new DB exception
}
}
return $result; // Return the result from parent::save()
}
}
The Question: How can I refresh the link to the DB in Kohana's ORM?
More Info:
Using Kohana 3.0.8
Possible solution (I dont know how to try it in Kohana)
Thanks!
That's a problem of either
Reached the MYSQL configured timeout
Exceeded packet size
Loss of packets
If you are executing a long insert, like a bulk insert, if your DB isn't configured for that, no change in your code will have any effect. You may try to reconfigure your MYSQL instance, then rule out the MYSQL blame, then after you try to amend your code (which I doubt is the source of the problem). Retrying saving won't help much, but to make the DB even busier.
Another thing, if you are using a proxy (like HAProxy), check the timeouts on that as well.