Apache crash : Cannot allocate memory - php

Question

I am experiencing a recurring issue why my ubuntu server complete becomes unavailable around the same time for the past two days. I cant even SSH into it during the crash.
The Ubuntu server is on a amazon aws t2.micro instance, running on 1 gb of ram , Ubuntu 16.04
The apache error logs are shown below
[Wed Aug 30 18:02:23.710072 2017] [autoindex:error] [pid 7505] [client 60.191.38.77:57957] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi,$
[Wed Aug 30 18:02:23.710126 2017] [:error] [pid 7505] [client 60.xx.xx.xx:57957] script '/var/www/html/404.php' not found or unable to stat
[Wed Aug 30 19:11:54.375001 2017] [autoindex:error] [pid 11307] [client 45.55.21.189:52050] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi$
[Wed Aug 30 19:11:54.376134 2017] [:error] [pid 11307] [client 45.xx.xx.xx:52050] script '/var/www/html/404.php' not found or unable to stat
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 20:50:00.570286 2017] [core:notice] [pid 31139] AH00051: child pid 17670 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:03:27.234926 2017] [core:notice] [pid 31139] AH00051: child pid 18507 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354905 2017] [core:notice] [pid 31139] AH00051: child pid 18511 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354927 2017] [core:notice] [pid 31139] AH00051: child pid 18512 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:42.865027 2017] [core:notice] [pid 31139] AH00051: child pid 18506 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:46.984235 2017] [core:notice] [pid 31139] AH00051: child pid 18529 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:14:50.194072 2017] [core:notice] [pid 31139] AH00051: child pid 18605 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Aug 30 21:14:50.482541 2017] [core:notice] [pid 31139] AH00051: child pid 18618 exit signal Aborted (6), possible coredump in /etc/apache2
[Thu Aug 31 07:09:50.271441 2017] [mpm_prefork:notice] [pid 1321] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Thu Aug 31 07:09:50.274044 2017] [core:notice] [pid 1321] AH00094: Command line: '/usr/sbin/apache2'
The server becomes inaccessible around this time
Wed Aug 30 19:11:54 based on the logs and traffic monitor on AWS.
I have little understanding of how Apache works in terms of memory and concurrent connections. The website I have in it has low traffic. Is there some config file I might have messed up? .BTW memory_limit is -1 no limit as per the PHP docs
Or what can I do to find the cause of the problem, I am thinking of migrating the server files to a AWS beanstalk instance (A deploying aide plus scaling aide,AFAIK) a fresh start.
Is there away I could know the script(s) that might have caused the crash.I have too many scripts plus running on laravel PHP framework + third party scripts would really make it hard to outline which one is the suspect.
I would greatly appreciate an answer that explains the error log.i.e
A script did this...
Apache seems to have done this...
Ubuntu then did this...
Which led to this...

Answer 1

Borrowing from what #arkascha said in the comments the crash was caused by some script that was using a lot of memory and not well optimised.
The following steps will help others facing similar problems.
Limit the php script memory limit to something reasonable according
to what you running.
Have Swap files, really important I didn't have one which caused the
whole system to crash during memory intensive processing. (In my
case Apache was using all of the available memory from my 1GB,
literally causing the system to crash)
(A swap file is basically a file on a hard disk used to provide space for programs that have been transferred from the processor's memory. An extra RAM,but slow)
Find the root of the problem (I haven't found a technique or tool
outline php memory usage but i think there some available). Another
way is too look for scripts with long queries (multiple joins,etc) these are typically the cause.
Probably obvious but maybe not so obvious, get some additional RAM
for your server. You webapp may require more than you currently
have.
Caching data. There are many techniques in doing this. Basically it
involves loading static data where dynamic loading is not really needed.