PHP can't use 300MB of RAM - php

I'm trying to increase the allowed memory for certain PHP script. No matter what I do, for instance this:
ini_set('memory_limit', '512M');
... the script always runs out of memory at around 300MB:
Fatal error: Out of memory (allocated 25165824) (tried to allocate 343810589 bytes) in \\Foo\phpQuery\phpQuery.php on line 255
I've verified by several means that memory_limit is actually changed. The issue seems to be that PHP can't physically allocate a total of 300 MB of memory (25165824 bytes + 343810589 bytes = 352 MB).
I've tried both PHP/5.3.0 and PHP/5.3.9 in two different Windows-based computers with the following specs:
Windows XP / Windows Server 2003 (both computers are 32-bit boxes with 1GB or RAM)
Official PHP 32-bit VC9 binaries
Running as Apache 2.2 module (third-party 32-bit VC9 binaries)
I understand that using half of the physical RAM will force swapping and slow things down as hell but I just need to make sure the script actually works so it can be deployed to the live server. I've also tried larger values (which procuded the same error) and smaller values (with either made my script hit the limit or made Apache crash).
What can be the source of this apparently hard-coded memory limit?
Update #1: I've done further testing with the Windows Server 2003 box (which is actually a VMWare virtual machine). I've increased the "physical" RAM to 2 GB and I've verified that the paging file is allowed to grow up to 1152 MB. Task manager shows that current transaction load is 886 MB and there're 1,5 GB of free physical memory. However, I'm getting the same error with exactly the same figures.
Update #2: As I said, the memory_limit directive is fine. It shows up in both ini_get() and phpinfo(). The error message you'd get is slightly different from mine; mine indicates a PHP crash. Please compare:
Out of memory (allocated 25165824) (tried to allocate 343810589 bytes)
Allowed memory size of 25165824 bytes exhausted (tried to allocate 343810589 bytes)
I'll try to compose a script to reproduce the issue and report back.

An OOM exception is different to the memory limit warninigs.
This means PHP can't actually allocate the memory because insufficient resources are available within your operating system.
You'll need to check the system has sufficient memory/paging available to support this.

Try with max_input_time, sometimes when PHP says memory_limit it actually means max_input_time (-1 is infinite for this one).

I had a similar problem with out-of-memory errors popping up at numbers as low as 250MB. If you find your apache config file that controls ThreadsPerChild (for me it was /conf/extra/httpd-mpm.conf) and reduce the ThreadsPerChild from 150 to 50 or so under you should see a noticeable improvement... here's a script to test it out:
echo "Memory limit: ".ini_get("memory_limit")."<br><br>";
$a=array();
if (ob_get_level() == 0) ob_start();
for($i=0;$i<200;$i++)
{
$a[]=str_pad('',1024*1024*32);
echo "Pass ".$i.", memory used: ".number_format((memory_get_usage())/(1024*1024),0)." MB<br>";
ob_flush();
flush();
}

Related

What limits PHP memory when memory_limit is set to -1?

Background
I understand how PHP's memory_limit setting can be used to control how much memory is available for PHP to use.
As well as using this property to raise/lower the memory limit for your script, you can also set it to -1 to disable the memory limit altogether.
However, as we all know, a computer does not have infinite memory, therefore all we are really talking about is removing any self-imposed limits implemented by PHP itself.
An illustration
We can demonstrate that this is true, by using the following script:
<?php
print("Original: ");
print(ini_get('memory_limit'));
ini_set('memory_limit', -1);
print(", New: ");
print(ini_get('memory_limit'));
$x = "123456789ABCDEF";
while (true)
$x .= $x;
?>
When running from the command-line, I get the following output:
Original: 128M, New: -1
PHP Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
zend_mm_heap corrupted
And from the web (via Apache) I get something similar:
Original: 128M, New: -1
Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
In my examples, the values are the same (~480MB) so it doesn't appear that the web server is imposing a limit. Also, this is nowhere near the amount of RAM installed in the system (even ignoring virtual memory) so it is not a hardware limitation.
Note that these tests were run on PHP 5.6 on a Windows machine with 4GB of RAM. However, output is similar on other PHP versions.
Finally, we come to a question!
Given the above:
What actually dictates the memory limit when we set it to -1?
Is there a way of finding out what this limit is from within PHP?
When you set memory_limit to -1 the actual limit is memory available to operating system.
By available memory I mean sum of physical memory(RAM) and virtual memory(SWAP) minus memory currently used by others processes, and operating system itself.
This is hardware limitation, you just have to consider other processes as well.
For example given 4GB of RAM, no SWAP, and other processes(DB, WebBrowser, etc.) consuming 3GB of memory, you have ~1GB available to be used by PHP. When you try to use more operating system rejects memory allocation request, and subsequently PHP exits with "out of memory" error.
There is no method which can be used across different operating systems, PHP don't provide functions to check how much memory is available and used in the environment. If you know your application is running on Linux you can try to read memory info from /proc/meminfo.
<?php
echo file_get_contents("/proc/meminfo");
which will output something like:
MemTotal: 1999084 kB
MemFree: 174344 kB
MemAvailable: 1077264 kB
Buffers: 151328 kB
Cached: 726752 kB
...
Based on that you can try to estimate amount of available memory, be advised memory used by other processes will vary with time.

PHP running out of memory with Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)

I'm busy migrating a customer application written in PHP/MySQL from Hetzner to AWS. Everything works fine but a few scripts. These scripts are poorly written and loop through millions of records, creating hundreds of local variables in each run of the loop, writing each row to an excel file, opening another file, writing a status update and closing a file in each run of the loop. The script is spawned as an independent process using shell_exec from the main web application.
When I first tested the script on EC2, it crashed quite quickly as the php memory_limit parameter was set to 128M on my EC2 instance. The error was similar to the one below:
Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)
I increased the memory_limit from 128M to 256M, then to 512M, then to 1024M, 4096M and eventually set to -1 to see where the problem lies.
Setting it to -1 did initially seemed to work but it then froze the entire instance (t2.micro). I then realized it was because of lack of swap space that system memory limits are being reached so just for testing purposes, I added a swap space of 4GB and set memory_limit to -1. This did work as expected and the script never crashed but as the script ran, it became progressively slower. For instance, it wrote the first 10% of records to excel file much faster than the 10% after 50%. All this while I watched the memory usage expand to 6 GB.
Below are screenshots from htop while the script was running:
I'm using the default php.ini that comes php installation except for memory_limit which is set to -1.
However, when I run this very same script on another server (Hetzner shared hosting in this case), the server somehow limits the memory usage and the script runs fine on 128M of memory_limit setting. Although this is not entirely true (as confirmed by htop on the server), it seems like it does not crash on the server with much less memory/swap space than on my EC2 instance.
Below are screenshots from htop running on Hetzner:
The third screenshot was taken towards the end of the script - and as you can see the memory didn't change much between when it started and the end. Here's the php.ini settings from the server:
display_errors=1
memory_limit=128M
max_execution_time=90
max_input_vars=3500
upload_max_filesize=64M
post_max_size=64M
allow_url_fopen=0
The database and code base are exact replicas in both instances.The resultant excel file size is about 225MB in both cases.
So, any ideas what might be causing this behavior and how should I go about fixing it on my EC2 instance?
Thank you for your help!
EasyPHP-Devserver-17-depanne-erreur Fatal error: Allowed memory size
of 134217728 bytes exhausted (tried to allocate 471698889 bytes) in
C:\Program Files
(x86)\EasyPHP-Devserver-17\eds-binaries\httpserver\apache2425vc11x86x180705170153\eds-app-dashboard.php
on line 77
SOLUTION
In my case, it was because Easyphp was trying to load a too large error.log file.
Deleting the server log file in eds-binaries\httpserver\apache2418x160331124251\logs
Test ok

PHP Fatal error out of memory over 1 GB (should be unlimited) [duplicate]

I'm getting the error when I run my PHP script....
Fatal error: Out of memory (allocated 1827405824) (tried to allocate 88800 bytes)
I've added this line to my PHP script..
ini_set("memory_limit","3000M");
This statement does seem to correctly control the memory usage, but I dont seem to be able to get it above about 1.8GB.
Its as if the upper memory limit is being restricted somewhere else.
I've also added to the php.ini...
memory_limit = 3000M
Does anyone know if the memory is restricted elsewhere?
I'm running a local server with Xampp.
I have Windows 7, 64-bit with 4GB RAM.
My script uses PHP's GD image library and I get the error when trying to allocate an image reference with ImageCreateTrueColor().
(I know this is a huge amount of memory - but this is just a one-of script, and its just a lot easier to do it this way.)
Thanks.
Update....
#elusive #Orbling
I expect everybody's bored whith this question, but here is the simplified code which illustrates the problem.
<?php
ini_set("memory_limit","4000000000");
echo "ini_get = " . ini_get('memory_limit') . "<br>\n";
echo "memory_get_usage = " . memory_get_usage(true) . "<br>\n";
$bigImageHandle = imagecreatetruecolor(22200, 24800); //this is line 5
?>
Browser output...
ini_get = 4000000000
memory_get_usage = 524288
Fatal error: Out of memory (allocated 1843396608) (tried to allocate 88800 bytes) in
E:\User\My_Webs\experiments\houseshunting\temp\osMaps\t1.php on line 5
I tested this out with a smaller set of tiles and the memory used by imagecreatetruecolor() and I estimate I need 2.7GB
Using Acquia Dev Desktop, I had many memory limit crashes.
After having increased the memory limit into PHP.ini.
php_value memory_limit 1024M
php_value max_execution_time 3000
This issue was less frequent but still occuring ( Especially with Feature Recreate )
Into my httpd.conf I increased the StackThread to 16M
ThreadStackSize 16*1024*1024
And it solved the memory crash issue.
Hope it can help
You're running on a 64-bit operating system, but Apache and PHP are likely still 32-bit. If you're using mod_php, apache would be the limiting factor here.
32-bit processes are limited about 2GiB of RAM unless you used the /3GB switch and the software is aware of 3GB support.
That still leaves up about 200 MiB that seems unused, but its small enough that it can be used by various libraries that all have to be loaded in memory
As far as I know, the library usage won't show up in the committed memory, but still counts towards the 2GiB limit (much like device memory counts towards the 4GiB limit on 32-bit windows. Where installing 2 GiB graphics card brings you down to under 2GiB of usable RAM).
Most likely solution? Install a 64-bit PHP, and then dispatch it to that (using a system() call, perhaps)
In Ubuntu 18.04
Check version PHP: php -v
In my case I have PHP 7.4
Edit file: nano /etc/php/7.4/apache2/php.ini
Search and change memory_limit = 2048M
Edit file: nano /etc/php/7.4/cli/php.ini
Search and change memory_limit = 2048M
Finally: systemctl restart apache2
Which PHP version are you using?
The memory_limit variable is, or was, contained in a 32-bit integer, so can not go above 2GB.
See: http://bugs.php.net/bug.php?id=39132&edit=1
From the bottom comment on that bug report, it might be the routine that translates the human readable form to a number, try putting it in digits.
Check your Apache config (e.g., httpd.conf). There's likely an RLimitMEM directive limiting the memory allow for children processes handling requests.
So, you can set your PHP limit all you want, if Apache spawns the process with a memory limit, you can't exceed that.
If you're on a hosted service and have a shared server, likely you don't have access to this config and need to work with your provider. As you can see, it's configuration that applies server-wide... you're not likely going to get them to change this. Then again, if you're looking to spawn bigger than 1.5Gig processes, you prolly should be either solving the problem a different way (others have suggested this) or getting a dedicated server of some kind (e.g. EC2).
For example:
/usr/local/apache/conf
#RLimitMEM 85643200 104857600 # Limit to: 80Mb / process, 100Mb total
RLimitMEM 134217728 537395200 # Limit to: 128Mb / Process, 512Mb total
The issue is likely to be caused by running 32-bit apache and php. Try upgrading these to 64-bit binaries and see if that fixes the issue.
I've had exactly the same problem and after searching a lot I discovered it had nothing to do wih memory limit but with a bug in my code: I had a function using an array, located in external file. In this function I had set the array as "global" but missed to include the file with tis array....
Like the spiderplant0's error, this was giving me an error with a very very huge memory allocation.
So check your code and try to see which part create this error.
Try this
set_time_limit(300);
ini_set('memory_limit', '20000M');
Try this:
#php_value memory_limit 300M
#php_value upload_max_filesize 200M
#php_value post_max_size 200M
#php_value max_execution_time 80000
#php_value max_input_time 80000
try ini_set('memory_limit', '-1');

Prevent memory overflow

At some point in my php script that makes use of curl, the following error shows up: "Fatal error: Allowed memory size of 262144 bytes exhausted (tried to
allocate 77824 bytes) in"... It points out this part of the script: "$s = curl_exec($c);"
What is the problem? And how to settle it down?
In this case, your server is misconfigured.
Allowed memory size of 262144 bytes
200 kilobytes of RAM per script are not enough for most PHP scripts. The standard in my experience is 8 MB minimum; 16 MB is normal. A blog system like WordPress (it is admittedly fat, but still one of the most popular blog systems around) chokes on 8 MB and runs half-way decently with 16.
You should change the memory_limit value in your php.ini. If you're on shared hosting, demand that the provider increase it to at least 8M, better 16M or more. If they deny, get out of there: It's sub-standard hosting.
you are tying to allocate more memory than the heap can handle
set your limit higher, for xample
at the top of the script::
ini_set("memory_limit","10M");
or in your php.ini
memory_limit = 10M
this set your memory_limit to 10M
ini_set() is probably better than setting something in php.ini. If you've got a specific application you know needs more than the standard memory - then it's fine to increase the memory limit for that application. You want to be really careful opening up all your code to having a higher memory limit though.
That said, if you set memory_limit to 0, there is no memory limit & the script will use as much memory as it needs (and the system can give it).
Here are a couple of suggestions:
edit your php.ini file and change the line that says memory_limit = .25M so that it says
memory_limit = 16M
make sure you are calling
curl_close($c);
consistently.
If you want to include a larger code snippet, maybe we can see where you have memory leaks.

How else might a PHP CLI script determine its memory limit?

I need to run a PHP CLI script and give it a LOT of memory (it's an automatic documentation generator that needs to cover a large code base). I have a powerful machine and have allocated 5GB to php, both in php.ini and in an inline ini_set('memory_limit','5120M') declaration in the script file.
If I add these lines to top of the script:
phpinfo();
exit();
... it claims that it has a memory limit of 5120M, as I specified.
But the script still errors out, saying
Fatal error: Allowed memory size of 1073741824 bytes exhausted
... which is 1GB, not 5GB as I specified.
Is there any other place that the script might be looking to determine its memory limit? This is running in Fedora Linux.
(Yes, the ultimate solution may be to rewrite the script to be more efficient, but I didn't write it in the first place, so before I resort to that, I want to just throw resources at it and see if that works.)
The heap limit property is a size_t, which is 32 bits on a 32-bit machine. If this is in bytes, this would limit the memory limit to 4 GB. You may try running it on a 64 bit machine, with 64-bit PHP.
Edit: confirmed, heap->limit is a size_t (unsigned int) and is in bytes. A memory_limit of -1 sets the heap->limit to 4GB, and does not disable it as the documentation implies. Setting it to 5GB makes it wrap around to 1GB.

Categories