I’ve been having a major issue with my Drupal website for the past couple of weeks. I’ve been trying to home in on where the issue is coming from but at the moment it has been hard to work out if it is module related or server related.
Essentially my server is hitting 100% CPU and doesn’t stop until I reset APACHE. I’m not getting the WSOD with a PHP error and there are no new entries in my PHP error log. Also, because PHP doesn’t give me an error I haven’t been able to use xhprof to debug.
I found that this would happen when I had multiple users browsing the website and also when I quickly opened multiple tabs in my browser. I had at first thought it was because I had some views embedded in a page using the views field module, each of which I had contextual filters enabled. These pages were usually where the issue would arise.
I have however been able to replicate the crash by quickly opening pages in multiple tabs where views aren’t present. This then started to make me think it was server related, but because I don’t have much experience in the server side of things I wasn’t sure what to look for or how to debug.
The details of my server and module setup are below. I’d appreciate any help or direction on this as it’s become a show stopper as far as launching our site is concerned.
- Server:Windows server 2012 – Intel Xeon 2.2GHz, 8GB RAM, 64-bit Operating System
- APACHE: Apache/2.2.25
- PHP: php-5.4.20-Win32-VC9-x86 - Memory limit 120M
- Database: Microsoft SQL Server 10.50.1600 (using sqlsrv module) located in a different server
- Drupal: 7.23
- Memcache: 3.0.7
- Modules that I thought could be linked to the issue (but might not be): views, entity reference, views field
The APACHE error log has the following entries when I replicate the crash:
- [Wed Mar 12 18:46:37 2014] [notice] Parent: child process exited with status 255 -- - Restarting.
- [Wed Mar 12 18:46:37 2014] [notice] Apache/2.2.25 (Win32) PHP/5.4.20 configured -- resuming normal operations
- [Wed Mar 12 18:46:37 2014] [notice] Server built: Jun 29 2013 19:23:21
- [Wed Mar 12 18:46:37 2014] [notice] Parent: Created child process 5084
- [Wed Mar 12 18:46:37 2014] [notice] Child 5084: Child process is running
- [Wed Mar 12 18:46:37 2014] [notice] Child 5084: Acquired the start mutex.
- [Wed Mar 12 18:46:37 2014] [notice] Child 5084: Starting 150 worker threads.
- [Wed Mar 12 18:46:37 2014] [notice] Child 5084: Starting thread to listen on port 80.
- [Wed Mar 12 18:46:37 2014] [notice] Child 5084: Starting thread to listen on port 80.
- [Wed Mar 12 18:46:41 2014] [notice] Parent: child process exited with status 255 -- Restarting.
- [Wed Mar 12 18:46:41 2014] [notice] Apache/2.2.25 (Win32) PHP/5.4.20 configured -- resuming normal operations
- [Wed Mar 12 18:46:41 2014] [notice] Server built: Jun 29 2013 19:23:21
- [Wed Mar 12 18:46:41 2014] [notice] Parent: Created child process 1628
- [Wed Mar 12 18:46:41 2014] [notice] Child 1628: Child process is running
- [Wed Mar 12 18:46:41 2014] [notice] Child 1628: Acquired the start mutex.
- [Wed Mar 12 18:46:41 2014] [notice] Child 1628: Starting 150 worker threads.
- [Wed Mar 12 18:46:41 2014] [notice] Child 1628: Starting thread to listen on port 80.
- [Wed Mar 12 18:46:41 2014] [notice] Child 1628: Starting thread to listen on port 80.
- [Wed Mar 12 18:51:33 2014] [notice] Parent: Received shutdown signal -- Shutting down the server.
- [Wed Mar 12 18:51:35 2014] [notice] Child 1628: Exit event signaled. Child process is ending.
- [Wed Mar 12 18:51:36 2014] [notice] Child 1628: Released the start mutex
- [Wed Mar 12 18:51:53 2014] [notice] Parent: Child process exited successfully.
The windows evernt log has the following:
- Faulting application name: httpd.exe, version: 2.2.25.0, time stamp: 0x51cf187f
- Faulting module name: php5ts.dll, version: 5.4.20.0, time stamp: 0x523a144e
- Exception code: 0xc0000005
- Fault offset: 0x0004b4dd
- Faulting process id: 0x65c
- Faulting application start time: 0x01cf3e2368a98606
- Faulting application path: C:\Apache2\bin\httpd.exe
- Faulting module path: C:\php-5.4.20-Win32-VC9-x86\php5ts.dll
- Report Id: a868e8ac-aa16-11e3-93f4-0050568b2e3b
- Faulting package full name:
- Faulting package-relative application ID:
Sorry about the amount of writing but I wanted to make sure I explained as much as I could.
Thanks
Related
Okay, first of all I understand and ... permit ... you all to laugh: I have a local server running php 4,4 / apache / mysql - installed with xampp distribution. Just simply haven't got the time to upgrade all the code that has been serving me for years and is still serving me well :-)
To the issue - I can't make a try/catch ... beeeecause of the old php version. I have a script that parses a dir and to a lot of things to all the files ... and after several 100 thousand files over the years without problems ... I now find that the server shuts down / browser looses connection probably because of a single file ... but for my life I can't figure out which. So thought maybe I could figure it out through the error log. But ... I didn't learn much - and thought I might ask some wise guys here if there is anything I can learn from the following from my error log - regarding maybe what function/file or whatever it is that causes the breaking...:
[Tue Feb 14 20:47:26 2017] [crit] (22)Invalid argument: unable to replace stderr with error_log
[Tue Feb 14 20:47:26 2017] [crit] (2)No such file or directory: unable to replace stderr with /dev/null
[Tue Feb 14 20:47:26 2017] [notice] Apache/2.0.54 (Win32) mod_ssl/2.0.54 OpenSSL/0.9.8 PHP/4.4.0 mod_autoindex_color configured -- resuming normal operations
[Tue Feb 14 20:47:26 2017] [notice] Server built: Jul 28 2005 18:10:37
[Tue Feb 14 20:47:26 2017] [notice] Parent: Created child process 5384
[Tue Feb 14 20:47:27 2017] [notice] Child 5384: Child process is running
[Tue Feb 14 20:47:27 2017] [notice] Child 5384: Acquired the start mutex.
[Tue Feb 14 20:47:27 2017] [notice] Child 5384: Starting 250 worker threads.
[Tue Feb 14 20:47:36 2017] [notice] Parent: child process exited with status 3221225725 -- Restarting.
[Tue Feb 14 20:47:36 2017] [crit] (22)Invalid argument: unable to replace stderr with error_log
[Tue Feb 14 20:47:36 2017] [crit] (2)No such file or directory: unable to replace stderr with /dev/null
[Tue Feb 14 20:47:36 2017] [notice] Apache/2.0.54 (Win32) mod_ssl/2.0.54 OpenSSL/0.9.8 PHP/4.4.0 mod_autoindex_color configured -- resuming normal operations
[Tue Feb 14 20:47:36 2017] [notice] Server built: Jul 28 2005 18:10:37
[Tue Feb 14 20:47:36 2017] [notice] Parent: Created child process 3136
[Tue Feb 14 20:47:36 2017] [notice] Child 3136: Child process is running
[Tue Feb 14 20:47:36 2017] [notice] Child 3136: Acquired the start mutex.
[Tue Feb 14 20:47:36 2017] [notice] Child 3136: Starting 250 worker threads.
Got it ... filesize() was failing (silent aparently) for some file(s). Ran it w/o this function and it worked. Case closed...
Edit: strange though ... filesize() suddenly fails on all files. And they are local. Subdir to the htdocs folder ... and the top level folder to this structure is set to all my users to have full control of this folder and all subfolders... weird...
I am trying to track down the cause of my Apache PHP crashes. I am hoping there is some kind of setting, tool or something to help track down what code, configuration or what the issue is that is causing these php crashes.
In my Windows event log I have (shows apache crashing, but caused by php):
Faulting application httpd.exe, version 2.2.21.0, time stamp 0x4e6b3136, faulting module php5ts.dll, version 5.3.8.0, time stamp 0x4e537a04, exception code 0xc0000005, fault offset 0x0000c7d7, process id 0xbf4, application start time 0x01cd45afc42f0b7d.
The application utilizes Apache, PHP and MySQL on a Windows Server 2008 machine (xampp).
This is the crash in my Apache log:
[Fri Jun 08 15:56:34 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Fri Jun 08 15:56:35 2012] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 08 15:56:35 2012] [notice] Digest: done
[Fri Jun 08 15:56:35 2012] [notice] Apache/2.2.21 (Win32) PHP/5.3.8 configured -- resuming normal operations
[Fri Jun 08 15:56:35 2012] [notice] Server built: Sep 10 2011 11:34:11
[Fri Jun 08 15:56:35 2012] [notice] Parent: Created child process 1220
[Fri Jun 08 15:56:35 2012] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 08 15:56:35 2012] [notice] Digest: done
[Fri Jun 08 15:56:35 2012] [notice] Child 1220: Child process is running
[Fri Jun 08 15:56:35 2012] [notice] Child 1220: Acquired the start mutex.
[Fri Jun 08 15:56:35 2012] [notice] Child 1220: Starting 150 worker threads.
[Fri Jun 08 15:56:35 2012] [notice] Child 1220: Starting thread to listen on port 80.
[Fri Jun 08 15:56:35 2012] [notice] Child 1220: Starting thread to listen on port 80.
[Fri Jun 08 15:57:37 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Fri Jun 08 15:57:37 2012] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 08 15:57:37 2012] [notice] Digest: done
[Fri Jun 08 15:57:37 2012] [notice] Apache/2.2.21 (Win32) PHP/5.3.8 configured -- resuming normal operations
[Fri Jun 08 15:57:37 2012] [notice] Server built: Sep 10 2011 11:34:11
[Fri Jun 08 15:57:37 2012] [notice] Parent: Created child process 3932
[Fri Jun 08 15:57:38 2012] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 08 15:57:38 2012] [notice] Digest: done
[Fri Jun 08 15:57:38 2012] [notice] Child 3932: Child process is running
[Fri Jun 08 15:57:38 2012] [notice] Child 3932: Acquired the start mutex.
[Fri Jun 08 15:57:38 2012] [notice] Child 3932: Starting 150 worker threads.
[Fri Jun 08 15:57:38 2012] [notice] Child 3932: Starting thread to listen on port 80.
[Fri Jun 08 15:57:38 2012] [notice] Child 3932: Starting thread to listen on port 80.
Any thoughts on what I should try, or tools/techniques to trace back to what code, etc. might be causing this?
Since XAMPP's Apache (for Windows) is built with winn_nt MPM (multi-thread) + Thread-Safe PHP, it may be that you are hitting a problem inherent to the supposedly Thread-Safe versions of PHP (see this very good post for more information). This could explain why you never encounter the error when running in a single-user mode.
Please let us know if Uku Loskit's suggestion did any good. If not, I would recommend switching to a Prefork + Non-Thread-Safe PHP setup in such a multi-user environment.
Because MPM must be chosen at compile-time, you would need to either:
compile Apache yourself (technically feasible, but quite a pain in the neck on Windows, as far as I remember)
find a distribution other than XAMPP, which is built with prefork (I do not know any)
Perhaps you'd better try to run PHP with FastCGI (this extra layer basically isolates server threads by starting a separate PHP process for each thread, and much more). This should allow you to verify easily enough my initial assumption.
Just a side note: I do not see the point in installing Apache on a Windows Server. You may want to consider either using IIS, optionally with FastGCI (see instructions here), or (better) switch to a Linux-based stack.
My following setup is Xampp 1.7.7 and here is the info for everything in that package:
- Apache/2.2.21 (Win32) mod_ssl/2.2.21 OpenSSL/1.0.0e PHP/5.3.8 mod_perl/2.0.4 Perl/v5.10.1
I'm running the server on Windows XP SP3 32 bit OS, 4 gigs of ram, Quad Core.
The issue I'm having in my apache error log file is:
[Tue Apr 24 15:55:55 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Tue Apr 24 15:55:57 2012] [notice] Digest: generating secret for digest authentication ...
[Tue Apr 24 15:55:57 2012] [notice] Digest: done
[Tue Apr 24 15:55:59 2012] [notice] Apache/2.2.21 (Win32) mod_ssl/2.2.21 OpenSSL/1.0.0e PHP/5.3.8 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Tue Apr 24 15:55:59 2012] [notice] Server built: Sep 10 2011 11:34:11
[Tue Apr 24 15:55:59 2012] [notice] Parent: Created child process 776
[Tue Apr 24 15:56:00 2012] [notice] Disabled use of AcceptEx() WinSock2 API
[Tue Apr 24 15:56:01 2012] [notice] Digest: generating secret for digest authentication ...
[Tue Apr 24 15:56:01 2012] [notice] Digest: done
[Tue Apr 24 15:56:02 2012] [notice] Child 776: Child process is running
[Tue Apr 24 15:56:02 2012] [notice] Child 776: Acquired the start mutex.
[Tue Apr 24 15:56:02 2012] [notice] Child 776: Starting 350 worker threads.
[Tue Apr 24 15:56:02 2012] [notice] Child 776: Listening on port 443.
[Tue Apr 24 15:56:02 2012] [notice] Child 776: Listening on port 80.
This seems to occur sporadically throughout the day and I even tried using Win32DisableEx, EnableIMAP Off and EnableSendFile Off in the apache conf file. I also tried copying the libmysql.dll file to the system32 and apache/bin folders with no avail.
If anyone knows other reasons this error for the child process exiting and causing apache to crash, info would be greatly appreciated. If any additional log files are needed please let me know.
Tks,
Shane.
The error code 3221225477 is 0xC0000005 in hex, which on Windows is:
#define STATUS_ACCESS_VIOLATION ((NTSTATUS)0xC0000005L)
Access violation is Windows' version of "segmentation fault", which simply said means that the program tried to access a memory which is not allocated. This can happen for a lot of different reasons, but mostly (if not always) is a bug in the program.
Now, my guess for your situation, is that there is either a bug in PHP or in one of PHP's extensions or in Perl or some Perl application. Apache itself is usually very stable, but if you use some unusual extension, it might be the cause, too.
I would suggest updating all your configuration to latest versions. If you want to find the source of the problem for sure, run Apache inside a debugger, like Visual Studio or OllyDbg. When the exception (access violation) happens, it will stop execution (instead of restarting) and you'll see in which module it is.
Also take a look in the access log, if there is a suspicious request with the same timestamp as the error. But it may happen that the crash happens before the request is saved in the logfile.
How do I debug errors that have no error message?
When loading a PHP page I am getting this error in Firefox.
The connection to the server was reset while the page was loading.
It gives no indication as to why other than it appears to be Apache crashing.
Apache error logs shows:
[Wed Nov 03 10:23:04 2010] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Wed Nov 03 10:23:06 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Nov 03 10:23:06 2010] [notice] Digest: done
[Wed Nov 03 10:23:06 2010] [notice] Apache/2.2.14 (Win32) DAV/2 mod_autoindex_color PHP/5.3.1 mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Wed Nov 03 10:23:06 2010] [notice] Server built: Nov 11 2009 14:29:03
[Wed Nov 03 10:23:06 2010] [notice] Parent: Created child process 5100
[Wed Nov 03 10:23:07 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Nov 03 10:23:07 2010] [notice] Digest: done
[Wed Nov 03 10:23:07 2010] [notice] Child 5100: Child process is running
[Wed Nov 03 10:23:07 2010] [notice] Child 5100: Acquired the start mutex.
[Wed Nov 03 10:23:07 2010] [notice] Child 5100: Starting 150 worker threads.
[Wed Nov 03 10:23:07 2010] [notice] Child 5100: Starting thread to listen on port 80.
[Wed Nov 03 10:23:07 2010] [notice] Child 5100: Starting thread to listen on port 80.
I tried searching the 3221225477 code but found nothing that worked.
Also the Windows Event Viewer shows this:
Faulting application name: httpd.exe, version: 2.2.14.0, time stamp: 0x4aeb9704
Faulting module name: php5ts.dll, version: 5.3.1.0, time stamp: 0x4b06c41d
Exception code: 0xc0000005
Fault offset: 0x00082391
Faulting process id: 0x8f4
Faulting application start time: 0x01cb7b85fa74bf83
Faulting application path: C:\xampp\apache\bin\httpd.exe
Faulting module path: C:\xampp\php\php5ts.dll
Report Id: 73996201-e779-11df-938f-e0cb4e5b154a
This is happening when I call:
return mysql_fetch_object($result, $class_name);
Its also happening on another page when I use $_SESSION
If I switch to using CGI instead of the php5ts.dll and php5apache2_2.dll I get:
Premature end of script headers: php-cgi.exe
Its is only happening for some pages. Others work fine.
No PHP errors or exceptions occur. Error reporting is set to E_ALL & ~E_NOTICE & ~E_DEPRECATED. display_errors is on.
Timeout is set to 60 seconds, the error page is returned in about 10 seconds.
The memory usage gets about 16MB, the maximum limit is 256MB.
PHP 5.3.1, Windows 7 64bit.
So does anyone know how to effectively debug this error?
Well, you can start by reading "Generating a Backtrace" and searching for registered PHP bug reports similar to you, and submitting a bug if you've really found none. If you are really interested, you can inspect the PHP Source and propose a patch.
Looks like you are experiencing a bug in PHP, leading to a crash. Try to upgrade to the latest nightly build (http://snaps.php.net/). If that doesn't help, try to get a backtrace and submit a bug on bugs.php.net.
Seems that the script is taking a long time to execute so the browser is timing out. try using the php function set_time_limit to keep the script from dying.
set_time_limit(0); // no time limit
If apache is crashing it may be due to a mangled .htaccess. Is there any log entry before the error message/crash to indicate what Apache thought it was doing?
Upgrade your PHP.
(This is what I did, and now it seems to work, however my question still stands: How do I effectively debug issues like these?)
http://bugs.php.net/bugs-getting-valgrind-log.php
[edit2]
http://en.wikipedia.org/wiki/Strace can also be handy to find the spot that takes you system down.
[edit]
how did you monitor the memory usage?
I have this PHP code:
<?php
$username="root";
$password="******";// censored out
$database="bazadedate";
mysql_connect("127.0.0.1",$username,$password); // i get unknown constant localhost if used instead of the loopback ip
#mysql_select_db($database) or die( "Unable to select database");
$query="SELECT * FROM backup";
$result=mysql_query($query);
$num=mysql_numrows($result);
$i=0;
$raspuns="";
while ($i < $num) {
$data=mysql_result($result,$i,"data");
$suma=mysql_result($result,$i,"suma");
$cv=mysql_result($result,$i,"cv");
$det=mysql_result($result,$i,"detaliu");
$raspuns = $raspuns."#".$data."#".$suma."#".$cv."#".$det."#";
$i++;
}
echo "<b> $raspuns </b>";
mysql_close();
?>
And it should return a single string containing all data from the table. But it says "connection reset when loading page".
the log is :
[Tue Jun 15 16:20:31 2010] [notice] Parent: child process exited with status 255 -- Restarting.
[Tue Jun 15 16:20:31 2010] [notice] Apache/2.2.11 (Win32) PHP/5.3.0 configured -- resuming normal operations
[Tue Jun 15 16:20:31 2010] [notice] Server built: Dec 10 2008 00:10:06
[Tue Jun 15 16:20:31 2010] [notice] Parent: Created child process 2336
[Tue Jun 15 16:20:31 2010] [notice] Child 2336: Child process is running
[Tue Jun 15 16:20:31 2010] [notice] Child 2336: Acquired the start mutex.
[Tue Jun 15 16:20:31 2010] [notice] Child 2336: Starting 64 worker threads.
[Tue Jun 15 16:20:31 2010] [notice] Child 2336: Starting thread to listen on port 80.
[Tue Jun 15 16:20:35 2010] [notice] Parent: child process exited with status 255 -- Restarting.
[Tue Jun 15 16:20:35 2010] [notice] Apache/2.2.11 (Win32) PHP/5.3.0 configured -- resuming normal operations
[Tue Jun 15 16:20:35 2010] [notice] Server built: Dec 10 2008 00:10:06
[Tue Jun 15 16:20:35 2010] [notice] Parent: Created child process 1928
[Tue Jun 15 16:20:35 2010] [notice] Child 1928: Child process is running
[Tue Jun 15 16:20:35 2010] [notice] Child 1928: Acquired the start mutex.
[Tue Jun 15 16:20:35 2010] [notice] Child 1928: Starting 64 worker threads.
[Tue Jun 15 16:20:35 2010] [notice] Child 1928: Starting thread to listen on port 80.
Any idea why it outputs nothing?
$num=mysql_numrows($result);
should be
$num=mysql_num_rows($result);
Thats atleast 1 issue.
You should also look into mysql_fetch_assoc...
# this will loop through each record
while($row = mysql_fetch_assoc($result)) {
$data = $row['data'];
$sum = $row['suma'];
....
}
You need to enable error reporting to see what's going on... Edit php.ini, and set error_reporting=2147483647 and display_errors=1. Then run the script again to check for all errors...
From the log snippet, something's killing the Apache child handling the request, which causes an immediate shutdown of the TCP connection, which your browser sees as a "Connection reset".
You may want to investigate using 'localhost' instead of '127.0.0.1' for your database connection. MySQL treats "localhost" differently than an IP address, and tries to connect via sockets, rather than a TCP connection - this is a speed optimization as local sockets are faster than TCP sockets. Your MySQL install may not be listening for TCP connections (skip_networking = 1).
As well, you should NEVER connect to your database as the root account, especially if this will be a public-facing site. The root account can access anything and do anything in MySQL, which is NOT what you want to expose to the world. Read up on the GRANT syntax and create a MySQL account with limited privileges and use that instead. Most likely your site would only need insert/update/select/delete.
Beyond that, change your query fetching logic to use what Lizard suggested above, and start doing some debugging - figure out which line is causing the error, and how many iterations it took to get there. You may be exhausting some kind of resource which kills Apache/PHP.
I have experienced this error many times and the solution for me was to increase the apache binary (apache.exe or httpd.exe) stack size. You will need the Visual studio to do that, but you can use the trial version, as I did. You don't even need to turn it on. The command is:
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64>editbin /STACK:8000000 "c:\Program Files (x86)\WAMP\apache\bin\apache.exe"
Change the paths above according to your environment of course.
In the Visual studio directory VC/binary/ there are several utilities by the name editbin.exe. Use the one suitable for you platform or try them one by one, until it works (as I did).