Apache (2.4.25 FPM/FastCGI) stop responding and crashes after a while on my wordpress multisite website.
For some reason it used to run fine several days until I changed the wordpress(mu) domain using this script github.com/interconnectit/Search-Replace-DB (probably not the reason).
When restarted it works for about an hour then choked again.
I have added a mysql slow query log but its empty.
Apache error_log has multiple errors like this one:
[Wed Sep 06 08:50:27.941819 2017] [proxy_fcgi:error] [pid 25444:tid 140610719610624] (70007)The timeout specified has expired: [client x.x.x.x:53398] AH01075: Error dispatching request to : (polling)
and occasionally this error appears too:
[Wed Sep 06 09:13:33.296777 2017] [core:notice] [pid 10710:tid 140611211331392] AH00051: child pid 25582 exit signal Segmentation fault (11), possible coredump in /opt/bitnami/apache2
another occasional error (expected due to the multiple AH01075 timeout errors):
[Mon Sep 04 20:18:58.758718 2017] [mpm_event:error] [pid 19928:tid 140675798046528] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
Stopping mysql server kills my site but seems to keep apache working fine, at least for an hour or two that i`ve been testing.
Is 457452 value is normal for mysqld.bin in the following process list? (ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS)
2128 /sbin/init
2400 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
3840 -bash
4244 sshd: ubuntu [priv]
6928 /usr/bin/gonit
11036 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11320 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11364 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11592 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
20592 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
57668 php-fpm: pool wordpress
57716 php-fpm: pool wordpress
65252 php-fpm: pool wordpress
67608 php-fpm: pool wordpress
68776 php-fpm: pool wordpress
457452 /opt/bitnami/mysql/bin/mysqld.bin --defaults-file=/opt/bitnami/mysql/my.cnf --basedir=/opt/bi
PHP Version 5.6.30
mysql.bin Ver 14.14 Distrib 5.6.36
Ubuntu 14.04.5 (bitnami image on AWS)
Memory usage while apache not responding: used:883mb of 992mb
php.ini script memory_limit = 128M
Was a bit long to put it in a comment, so replying as an answer.
Are you logging any php errors? From what you presented so far, my theory would be that:
You used that script to do a search & replace of db but it executed poorly causing malformed or corrupted data.
That caused your php scripts that rely on mysql to go haywire due to malformed or corrupted data.
Because your php on fcgi isn't responding, your apache waits for it and holds onto the resources.
All the requests are hanging and now you run out of resources and start seg-faulting.
And then stupid apache, seeing that you're out of child count asks you to raise it not realizing you're already exhausted on physical resource beyond reason. Typical apache...
Because apache is now out of free children, it can't take anymore requests and it appears as if all of apache is broken even though it's merely clogged up by bad fcgi processes.
If my theory is correct, you should.
Check that none of the mysql data is corrupted
Check that the data that was replaced in mysql was as what you expected
Check for PHP error logs as they should be logging something if they can.
You should isolate the PHP processes that are actually the cause of clogging up and remove it if cleaning up data alone wasn't enough. Ideally, no matter how bad the data, the php program should've died cleanly.
Good luck hunting.
Related
I am using Yii 1 with PHP 7.1.33 running on an Apache/2.4.41 server macOS Catalina 10.15.3
In my application some of the requests are dying unexpectedly and randomly like this
In this image two calls are being made to the same api but one is succeeding and other one is returning no response.
I know the parameters are different but even when I am doing a die dump at the start of controller method, I'm getting the same result, first one succeeding, second one failing.
The same error is occurring with other apis which are in no way related to this api. And when this error is occurring, I am getting
[Tue Mar 17 16:34:39.572039 2020] [core:notice] [pid 697] AH00052: child pid 1233 exit signal Segmentation fault (11)
in my apache logs.
question is pretty simple even if not so obvious.
I managing a website that's showing issues with connection timeout, making the web service unaccessible and not working properly when it happens.
The actual environment is WooCommerce on a self-managed secured VPS.
php7, php-fpm, wordpress 5.+, apache2, mysql, cPanel, centOs
For a couple of time the site became unreachable showing Connection Timeout error, even if all services was up and running.
How to track or log requests originating the connection timeout error?
For example, with an application ran on the server that:
log every requests that occurs with more than 30 seconds execution time
the moment of the request
the request URL.
Any software, wordpress, cPanel extension or service in order to detect the issue?
I get the solution in matter of detecting the situation when it happens.
The key information can be found in PHP-FPM log (deducted that by the fact that restarting PHP-FPM and not Apache2 was solving the problem, due to a probable PHP-FPM thread appended).
i.e., in a cPanel enviroment you would find the log at the path:
/opt/cpanel/ea-php$$/root/usr/var/log/php-fpm/error.log
In the file the following lines are noteful:
[Tue Jul 16 09:55:30.677693 2019] [proxy_fcgi:error] [pid 17283] (70007)The timeout specified has expired: [client 162.158.91.192:16946] AH01075: Error dispatching request to : (polling), referer: https://www.example.com/my-resource.php
[Tue Jul 16 10:00:30.738947 2019] [proxy_fcgi:error] [pid 17283] (70007)The timeout specified has expired: [client 162.158.91.192:16946] AH01075: Error dispatching request to : (polling), referer: https://www.example.com/my-resource.php
So I've been able to inspect the shown resource and replicate and solve the issue.
I am trying to do a massive select * { insert into..} on a php page using xampp (v 5.5.30) on windows (64bit, 4 Gb ram).
After a few minutes I however get either two error messages:
A) "the connection was reset" in the browsers, or sometimes:
B) "mysql_query(): (HY000/2008): Out of memory..." during the insertion query.
On both occasions the PHP script itself is halted even using the directive:
ignore_user_abort(true);
I can't figure out why this is happening. Things I tried so for:
1) added in my apache config file:
<IfModule mpm_winnt_module> ThreadStackSize 8388608 </IfModule> because in the error log I noticed the line at the time of the connection reset:
[Mon Dec 07 13:01:23.540742 2015] [mpm_winnt:notice] [pid 2344:tid
504] AH00428: Parent: child process 4036 exited with status 3221225477
-- Restarting.
[Mon Dec 07 12:09:39.584814 2015] [mpm_winnt:notice] [pid 1684:tid
528] AH00354: Child: Starting 150 worker threads.
It seems like this error is linked to my problem in some way. Changing the config file however did not help and the error still appears.
2) Changed php ini setting:
ini_set("memory_limit","750MB");
3) Removed all mysql cache using:
RESET QUERY CACHE
4) Set PHP timeout:
set_time_limit(60*60);
The error occurs after a few minutes already.
Additional note: the filesize of the table used in the main select-query (mytable.MYD) is 400Mb, of which I only use a few columns of in the select. Once the query is loaded in the PHP memory, before executing the while loop, the memory footprint is 60Mb (using memory_get_usage())
Any idea what I can do to solve this?
thanks
You have to set execution time also.
set execution time to 0,it is functionally equivalent to turning the execution limit off completely:
ini_set('max_execution_time', 0);
Hope it will solve your problem
I'm running a PHP script that converts a few hundred thousand records from a remote MySQL server to the local one. A few times during the transfer, the error_logs shows ...
[Wed Jul 03 17:02:36 2013] [notice] Graceful restart requested, doing restart
[Wed Jul 03 17:02:37 2013] [notice] SSL FIPS mode disabled
[Wed Jul 03 17:02:37 2013] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0-fips mod_bwlimited/1.4 PHP/5.3.16 configured -- resuming normal operations
It's rare, and I can't find a reason for it, but the script seems to continue as though nothing happened. Testing if data was lost would be very cumbersome. So I'm wondering, what happens if a MySQL query is interrupted? And why does PHP seems to ignore the restart entirely, and just continue running as though nothing happened? Any thoughts on how/what to test?
That's an Apache log, not a PHP log. Your script is probably running in CLI (command line) so an Apache restart won't affect it. Otherwise it may have been restarted with -k graceful which will allow child processes to complete.
So I'm wondering, what happens if a MySQL query is interrupted
If MySQL is restarted, the script will fail (unless the programmer incorporated some extra robust error handling to deal with that situation).
In my Plesk 11, PHP 5.4.7. Apache/2.4.3 system, when I make cron jobs I get this error from error.log file.
[Tue Mar 19 21:33:18 2013] [warn] [client 88.44.55.66] mod_fcgid: read data timeout in 45 seconds
[Tue Mar 19 21:33:18 2013] [error] [client 88.44.55.66] Premature end of script headers: index.php
So I checked for execution time with this:
echo ini_get('max_execution_time');
I got this value: 1560000
It doesn't seem a timeout problem but I get timeout error?
Answer for the future reference:
Edit /etc/httpd/conf.d/fcgid.conf file with vim.
Change FcgidIOTimeout 45 to FcgidIOTimeout 600.
Restart Apache.
Then Fast CGI timeout will be solved.
That's a mod_fcgid timeout, not a PHP timeout. Take a look at the mod_fcgid documentation to find out how to change it, especially at FcgidIOTimeout.
Cron scripts if build correctly should not be run through apache. Your error shows that you are running into a timeout set by mod_fcgi.
If you access it through http in your cronjob, then it will work as a webpage request and then it may be affected by apache timeout because apache is involved in the process. You can run the php script directly with an absolute path using php interpreter to bypass apache