I need to run a PHP CLI script and give it a LOT of memory (it's an automatic documentation generator that needs to cover a large code base). I have a powerful machine and have allocated 5GB to php, both in php.ini and in an inline ini_set('memory_limit','5120M') declaration in the script file.
If I add these lines to top of the script:
phpinfo();
exit();
... it claims that it has a memory limit of 5120M, as I specified.
But the script still errors out, saying
Fatal error: Allowed memory size of 1073741824 bytes exhausted
... which is 1GB, not 5GB as I specified.
Is there any other place that the script might be looking to determine its memory limit? This is running in Fedora Linux.
(Yes, the ultimate solution may be to rewrite the script to be more efficient, but I didn't write it in the first place, so before I resort to that, I want to just throw resources at it and see if that works.)
The heap limit property is a size_t, which is 32 bits on a 32-bit machine. If this is in bytes, this would limit the memory limit to 4 GB. You may try running it on a 64 bit machine, with 64-bit PHP.
Edit: confirmed, heap->limit is a size_t (unsigned int) and is in bytes. A memory_limit of -1 sets the heap->limit to 4GB, and does not disable it as the documentation implies. Setting it to 5GB makes it wrap around to 1GB.
Related
Background
I understand how PHP's memory_limit setting can be used to control how much memory is available for PHP to use.
As well as using this property to raise/lower the memory limit for your script, you can also set it to -1 to disable the memory limit altogether.
However, as we all know, a computer does not have infinite memory, therefore all we are really talking about is removing any self-imposed limits implemented by PHP itself.
An illustration
We can demonstrate that this is true, by using the following script:
<?php
print("Original: ");
print(ini_get('memory_limit'));
ini_set('memory_limit', -1);
print(", New: ");
print(ini_get('memory_limit'));
$x = "123456789ABCDEF";
while (true)
$x .= $x;
?>
When running from the command-line, I get the following output:
Original: 128M, New: -1
PHP Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
zend_mm_heap corrupted
And from the web (via Apache) I get something similar:
Original: 128M, New: -1
Fatal error: Out of memory (allocated 503840768) (tried to allocate 1006632961 bytes) in test.php on line 14
In my examples, the values are the same (~480MB) so it doesn't appear that the web server is imposing a limit. Also, this is nowhere near the amount of RAM installed in the system (even ignoring virtual memory) so it is not a hardware limitation.
Note that these tests were run on PHP 5.6 on a Windows machine with 4GB of RAM. However, output is similar on other PHP versions.
Finally, we come to a question!
Given the above:
What actually dictates the memory limit when we set it to -1?
Is there a way of finding out what this limit is from within PHP?
When you set memory_limit to -1 the actual limit is memory available to operating system.
By available memory I mean sum of physical memory(RAM) and virtual memory(SWAP) minus memory currently used by others processes, and operating system itself.
This is hardware limitation, you just have to consider other processes as well.
For example given 4GB of RAM, no SWAP, and other processes(DB, WebBrowser, etc.) consuming 3GB of memory, you have ~1GB available to be used by PHP. When you try to use more operating system rejects memory allocation request, and subsequently PHP exits with "out of memory" error.
There is no method which can be used across different operating systems, PHP don't provide functions to check how much memory is available and used in the environment. If you know your application is running on Linux you can try to read memory info from /proc/meminfo.
<?php
echo file_get_contents("/proc/meminfo");
which will output something like:
MemTotal: 1999084 kB
MemFree: 174344 kB
MemAvailable: 1077264 kB
Buffers: 151328 kB
Cached: 726752 kB
...
Based on that you can try to estimate amount of available memory, be advised memory used by other processes will vary with time.
I'm busy migrating a customer application written in PHP/MySQL from Hetzner to AWS. Everything works fine but a few scripts. These scripts are poorly written and loop through millions of records, creating hundreds of local variables in each run of the loop, writing each row to an excel file, opening another file, writing a status update and closing a file in each run of the loop. The script is spawned as an independent process using shell_exec from the main web application.
When I first tested the script on EC2, it crashed quite quickly as the php memory_limit parameter was set to 128M on my EC2 instance. The error was similar to the one below:
Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)
I increased the memory_limit from 128M to 256M, then to 512M, then to 1024M, 4096M and eventually set to -1 to see where the problem lies.
Setting it to -1 did initially seemed to work but it then froze the entire instance (t2.micro). I then realized it was because of lack of swap space that system memory limits are being reached so just for testing purposes, I added a swap space of 4GB and set memory_limit to -1. This did work as expected and the script never crashed but as the script ran, it became progressively slower. For instance, it wrote the first 10% of records to excel file much faster than the 10% after 50%. All this while I watched the memory usage expand to 6 GB.
Below are screenshots from htop while the script was running:
I'm using the default php.ini that comes php installation except for memory_limit which is set to -1.
However, when I run this very same script on another server (Hetzner shared hosting in this case), the server somehow limits the memory usage and the script runs fine on 128M of memory_limit setting. Although this is not entirely true (as confirmed by htop on the server), it seems like it does not crash on the server with much less memory/swap space than on my EC2 instance.
Below are screenshots from htop running on Hetzner:
The third screenshot was taken towards the end of the script - and as you can see the memory didn't change much between when it started and the end. Here's the php.ini settings from the server:
display_errors=1
memory_limit=128M
max_execution_time=90
max_input_vars=3500
upload_max_filesize=64M
post_max_size=64M
allow_url_fopen=0
The database and code base are exact replicas in both instances.The resultant excel file size is about 225MB in both cases.
So, any ideas what might be causing this behavior and how should I go about fixing it on my EC2 instance?
Thank you for your help!
EasyPHP-Devserver-17-depanne-erreur Fatal error: Allowed memory size
of 134217728 bytes exhausted (tried to allocate 471698889 bytes) in
C:\Program Files
(x86)\EasyPHP-Devserver-17\eds-binaries\httpserver\apache2425vc11x86x180705170153\eds-app-dashboard.php
on line 77
SOLUTION
In my case, it was because Easyphp was trying to load a too large error.log file.
Deleting the server log file in eds-binaries\httpserver\apache2418x160331124251\logs
Test ok
Laravel 4.2, Amazon ec2 Linux
PHP Fatal error:
Allowed memory size of 262144 bytes exhausted (tried to allocate 3072 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 78
TL;DR You need to increase your memory_limit setting in php.ini. You're barely letting your scripts use any memory at all.
The Problem
The error message Allowed memory size of 262144... means you have a memory_limit setting of 256 kb. That is far too low. For most useful applications, you need at least a few MB. I would start with 8MB and see how it goes.
The Solution
Change the memory_limit setting. In theory, you can do this two ways: (1) edit php.ini or (2) use ini_set().
In practice, you cannot always use ini_set('memory_limit', value);. For one thing, some extensions, like suhosin, prevent scripts from setting the memory_limit in this way. For another, you have to be careful how you do it. For example, you proposed using ini_set('memory_limit', '1G');. But the shorthand (K, M, G) only works in php.ini, not in ini_set. You would have to enter an actual number of bytes (e.g., 1073741824). Also, 1G is pretty excessive for most purposes; very, very few non-malicious PHP scripts need anything like that. Even pretty heavy frameworks like WordPress typically run well within 64 MB or so, even with lots of plugins loaded up.
How to change the setting in php.ini
Figure out which php.ini file you are using (it's not uncommon to have several floating around, depending on how you installed things). You can do this in two ways:
Put this in a .php file and run it: <?php phpinfo();
On a command line, type php -i | grep php.ini. You should see a line that says something like Loaded Configuration File => /etc/php.ini (the output may vary by system, of course).
Edit the file you just found by changing the line that starts with memory_limit = to something more appropriate, like
memory_limit = 8M
Feel free to bump that number up as needed, but I would recommend starting small (not 1G) to prevent bringing your server to its knees accidentally.
One caveat: if this is a testing/development machine and you are running a debugger or profiler like xdebug, you may want to start much higher, and 1G is not insane. But don't start with such a high number on a production machine; work your way up to it.
I'm trying to increase the allowed memory for certain PHP script. No matter what I do, for instance this:
ini_set('memory_limit', '512M');
... the script always runs out of memory at around 300MB:
Fatal error: Out of memory (allocated 25165824) (tried to allocate 343810589 bytes) in \\Foo\phpQuery\phpQuery.php on line 255
I've verified by several means that memory_limit is actually changed. The issue seems to be that PHP can't physically allocate a total of 300 MB of memory (25165824 bytes + 343810589 bytes = 352 MB).
I've tried both PHP/5.3.0 and PHP/5.3.9 in two different Windows-based computers with the following specs:
Windows XP / Windows Server 2003 (both computers are 32-bit boxes with 1GB or RAM)
Official PHP 32-bit VC9 binaries
Running as Apache 2.2 module (third-party 32-bit VC9 binaries)
I understand that using half of the physical RAM will force swapping and slow things down as hell but I just need to make sure the script actually works so it can be deployed to the live server. I've also tried larger values (which procuded the same error) and smaller values (with either made my script hit the limit or made Apache crash).
What can be the source of this apparently hard-coded memory limit?
Update #1: I've done further testing with the Windows Server 2003 box (which is actually a VMWare virtual machine). I've increased the "physical" RAM to 2 GB and I've verified that the paging file is allowed to grow up to 1152 MB. Task manager shows that current transaction load is 886 MB and there're 1,5 GB of free physical memory. However, I'm getting the same error with exactly the same figures.
Update #2: As I said, the memory_limit directive is fine. It shows up in both ini_get() and phpinfo(). The error message you'd get is slightly different from mine; mine indicates a PHP crash. Please compare:
Out of memory (allocated 25165824) (tried to allocate 343810589 bytes)
Allowed memory size of 25165824 bytes exhausted (tried to allocate 343810589 bytes)
I'll try to compose a script to reproduce the issue and report back.
An OOM exception is different to the memory limit warninigs.
This means PHP can't actually allocate the memory because insufficient resources are available within your operating system.
You'll need to check the system has sufficient memory/paging available to support this.
Try with max_input_time, sometimes when PHP says memory_limit it actually means max_input_time (-1 is infinite for this one).
I had a similar problem with out-of-memory errors popping up at numbers as low as 250MB. If you find your apache config file that controls ThreadsPerChild (for me it was /conf/extra/httpd-mpm.conf) and reduce the ThreadsPerChild from 150 to 50 or so under you should see a noticeable improvement... here's a script to test it out:
echo "Memory limit: ".ini_get("memory_limit")."<br><br>";
$a=array();
if (ob_get_level() == 0) ob_start();
for($i=0;$i<200;$i++)
{
$a[]=str_pad('',1024*1024*32);
echo "Pass ".$i.", memory used: ".number_format((memory_get_usage())/(1024*1024),0)." MB<br>";
ob_flush();
flush();
}
At some point in my php script that makes use of curl, the following error shows up: "Fatal error: Allowed memory size of 262144 bytes exhausted (tried to
allocate 77824 bytes) in"... It points out this part of the script: "$s = curl_exec($c);"
What is the problem? And how to settle it down?
In this case, your server is misconfigured.
Allowed memory size of 262144 bytes
200 kilobytes of RAM per script are not enough for most PHP scripts. The standard in my experience is 8 MB minimum; 16 MB is normal. A blog system like WordPress (it is admittedly fat, but still one of the most popular blog systems around) chokes on 8 MB and runs half-way decently with 16.
You should change the memory_limit value in your php.ini. If you're on shared hosting, demand that the provider increase it to at least 8M, better 16M or more. If they deny, get out of there: It's sub-standard hosting.
you are tying to allocate more memory than the heap can handle
set your limit higher, for xample
at the top of the script::
ini_set("memory_limit","10M");
or in your php.ini
memory_limit = 10M
this set your memory_limit to 10M
ini_set() is probably better than setting something in php.ini. If you've got a specific application you know needs more than the standard memory - then it's fine to increase the memory limit for that application. You want to be really careful opening up all your code to having a higher memory limit though.
That said, if you set memory_limit to 0, there is no memory limit & the script will use as much memory as it needs (and the system can give it).
Here are a couple of suggestions:
edit your php.ini file and change the line that says memory_limit = .25M so that it says
memory_limit = 16M
make sure you are calling
curl_close($c);
consistently.
If you want to include a larger code snippet, maybe we can see where you have memory leaks.