Intermittent "too many open files" when including/requiring PHP scripts - php

On my development box (thank goodness it's not happening in production—that I know of—yet), as I'm working on a PHP site, I get this occasional error:
Warning:
require_once(filename.php): failed
to open stream: Too many open files in
path/functions.php on line 502
Fatal error: require_once(): Failed opening required 'filename.php'
(include_path='/my/include/path') in
path/functions.php on line 502
Line 502 of functions.php is my "autoload" function which automatically requires PHP files containing classes I use:
function autoload($className)
{
require_once $className . ".php"; // <-- Line 502
}
By an "occasional" error, I mean that it'll work fine for about a day of development, then when I first see this, I can refresh the page and it'll be okay again, then refreshing gives me the same error. This happens just a few times before it starts to show it every time. And sometimes the name of the file it's requiring (I have a script split out into several PHP files) is different... it's not always the first or last or middle files that it bombs on.
Restarting php-fpm seems to solve the symptoms, but not the problem in the long run.
I'm running PHP 5.5.3 on my Mac (OS X 10.8) with nginx 1.4.2 via php-fpm.
Running lsof | grep php-fpm | wc -l tells me that php-fpm has 824 files open. When I examined the actual output, I saw that, along with some .so and .dylib files, the vast majority of lines were like this:
php-fpm 4093 myuser 69u unix 0x45bc1a64810eb32b 0t0 ->(none)
The segment "69u" and the 0x45bc1a6481... number are different on each row. What could this mean? Is this the problem? (ulimit is "unlimited")
Incidentally (though perhaps un-related), there's also one or two of these:
php-fpm 4093 myuser 8u IPv4 0x45bc1a646b0f97b3 0t0 TCP 192.168.1.2:59611->rest.nexmo.com:https (CLOSE_WAIT)
(I have some pages which use HttpRequest (PECL libraries) to call out to the Nexmo API. Are these not being closed properly or something? How can I crack down on those?)

Try to set php-fpm (they're set to infinite by default) to more appropriate values on your needs.
For example:
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
Maybe, set this too if your app works with lots of files:
rlimit_files = 1024

Related

How does one enable the OCI8 PHP extension, using the Oracle Instant Client, in Ubuntu 18.04 LTS with PHP-FPM and NGINX?

I'm using the latest PHP packages available from https://launchpad.net/~ondrej/+archive/ubuntu/php .
When I build and install the OCI8 extension, everything appears to be in order, but despite enabling the extension in the PHP-FPM configuration, its presence is not reflected in the output from phpinfo().
The following Gist details the exact process that I'm using to configure, build, and install the OCI8 PHP extension:
https://gist.github.com/cbj4074/fa761f60b6f8db431539d76ebfba828e
The very same process and configuration work perfectly well on Ubuntu 16.04 LTS, so it seems that there is some fundamental difference on Ubuntu 18.04 LTS, whether with the operating system or the PHP packages in question.
As a bit of important (and I suspect relevant to this issue) background information, on Ubuntu 18.04 LTS, the extension fails to load in the CLI environment out-of-the-box, with the error:
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20160303/oci8.so' - libmql1.so: cannot open shared object file: No such file or directory in Unknown on line 0
I resolved the issue like so:
# echo 'LD_LIBRARY_PATH="/opt/oracle/instantclient_12_2"' >> /etc/environment
I thought that perhaps adding the LD_LIBRARY_PATH to the PHP-FPM environment configuration might resolve the equivalent issue there:
# echo "env['LD_LIBRARY_PATH'] = /opt/oracle/instantclient_12_2" >> /etc/php/7.2/fpm/pool.d/www.conf
# systemctl restart php7.2-fpm
This does indeed cause the LD_LIBRARY_PATH value, as specified, to be reflected in both the Environment section of phpinfo() (when rendered via PHP-FPM + NGINX and requested from a browser) and the PHP Variables section, as $_SERVER['LD_LIBRARY_PATH'].
Oddly, even with PHP-FPM's logging set to debug, I don't see any trace of the libmql1.so error that I experience with the CLI. The OCI8 extension simply fails to load, silently. display_startup_errors = On in PHP-FPM's effective php.ini, too.
I elected to see if the OCI8 extension works in Apache, on the same server, and it does, provided I add export LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2 to /etc/apache2/envvars; in its absense, Apache complains on startup:
PHP Warning: PHP Startup: Unable to load dynamic library 'oci8.so' (tried: /usr/lib/php/20170718/oci8.so (libmql1.so: cannot open shared object file: No such file or directory), /usr/lib/php/20170718/oci8.so.so (/usr/lib/php/20170718/oci8.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
None of this business with the LD_LIBRARY_PATH is necessary on Ubuntu 16.04 LTS, and based on my observations herein and the comments regarding https://stackoverflow.com/a/45242468/1772379 , that changed in Ubuntu 17.10 and Ubuntu 18.04 LTS.
Has anybody else tried this, on Ubuntu 18.04 LTS, specifically?
I've tried this on two different Vagrant VMs, laravel/homestead box 6.0.0, and ubuntu/bionic64 box v20180509.0.0, and the behavior is the same in both.
Any other ideas would be most appreciated!
EDIT 1:
I asked about this issue on the package maintainer's GitHub tracker and he suggested that the problem stems from failing to set an appropriate RPATH at compile time.
I explain in my reply that I am setting an appropriate value, but the issue remains closed.
I do notice an interesting detail, however, which is that the compiled extension on Ubuntu 18.04 uses RUNPATH (and not RPATH, which is used in Ubuntu 16.04). If PHP-FPM ignores RUNPATH, and looks only for RPATH, it would explain this behavior.
EDIT 2:
This still-open report looks like an excellent candidate for having introduced the observed behavior:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859732
(discovered through comments on use RPATH but not RUNPATH? )
EDIT 3:
On a commenter's advice, I reexamined updating the ld configuration before building the extension and that resolved the issue! I had tried this before, but must have overlooked something between build attempts:
# echo /opt/oracle/instantclient_12_2 > /etc/ld.so.conf.d/oracle-instantclient.conf
# ldconfig
I still don't know why LD_LIBRARY_PATH doesn't work as it should in this instance, but adding the Instant Client library path to the linker configuration seems a better approach besides.
EDIT 4:
I stated in my previous edit that modifying the ldconfig constitutes a better approach, but came to realize (on a commenter's good advice) that doing so can cause undesirable library conflicts, because the effects are system-wide.
In hindsight, it makes sense to minimize the "collateral damage" from runtime library linkage modifications by limiting them to the execution environment via the LD_LIBRARY_PATH. Accordingly, I am motivated to determine why this does not work on Ubuntu 18.04 LTS.
I feel that I have established definitively that the PHP-FPM daemon ignores LD_LIBRARY_PATH on Ubuntu (and has since at least Ubuntu 16.04 LTS; see Comments for explanation).
The ld.so(8) manpage states (in relation to the order in which runtime library paths are searched):
Using the environment variable LD_LIBRARY_PATH (unless the executable is being run in secure-execution mode; see below). [sic] in which case it is ignored.
As yet, I cannot think of any other reason for which the path would be ignored. Of secure-execution mode, the same document says:
Secure-execution mode
For security reasons, the effects of some environment variables are voided or modified if the dynamic linker determines that the binary
should be run in secure-execution mode. (For details, see the discussion of individual environment variables below.) A binary is exe‐
cuted in secure-execution mode if the AT_SECURE entry in the auxiliary vector (see getauxval(3)) has a nonzero value. This entry may
have a nonzero value for various reasons, including:
* The process's real and effective user IDs differ, or the real and effective group IDs differ. This typically occurs as a result of
executing a set-user-ID or set-group-ID program.
* A process with a non-root user ID executed a binary that conferred capabilities to the process.
* A nonzero value may have been set by a Linux Security Module.
Firstly, Secure-Execution Mode seems not to be in effect, as the PHP executables don't exhibit this flag (AT_SECURE is 0):
LD_SHOW_AUXV=1 /usr/sbin/php-fpm7.1 -daemonize --fpm-config /etc/php/7.1/fpm/php-fpm.conf
AT_SYSINFO_EHDR: 0x7ffc569e1000
AT_HWCAP: 178bfbff
AT_PAGESZ: 4096
AT_CLKTCK: 100
AT_PHDR: 0x55ceab0c4040
AT_PHENT: 56
AT_PHNUM: 9
AT_BASE: 0x7f823c77f000
AT_FLAGS: 0x0
AT_ENTRY: 0x55ceab19e360
AT_UID: 0
AT_EUID: 0
AT_GID: 0
AT_EGID: 0
AT_SECURE: 0
AT_RANDOM: 0x7ffc56962349
AT_HWCAP2: 0x0
AT_EXECFN: /usr/sbin/php-fpm7.1
AT_PLATFORM: x86_64
It occurred to me that the child FPM pool processes might exhibit different AT_SECURE values, but the output is identical for the PHP-FPM daemon itself, as well as any child processes. The parent and the children all have the following values:
# od -t d8 /proc/851/auxv
0000000 33 140722944548864
0000020 16 395049983
0000040 6 4096
0000060 17 100
0000100 3 93903778242624
0000120 4 56
0000140 5 9
0000160 7 140365152313344
0000200 8 0
0000220 9 93903779136352
0000240 11 0
0000260 12 0
0000300 13 0
0000320 14 0
0000340 23 0
0000360 25 140722944193929
0000400 26 0
0000420 31 140722944196579
0000440 15 140722944193945
0000460 0 0
Secondly, none of these reasons seem to apply, given the following:
1) There is no indication that PHP-FPM or its child processes have real and effective user or group IDs that differ (thanks to https://unix.stackexchange.com/a/202359 for this command):
# ps -e -o user= -o ruser= | awk '$1 != $2'
systemd+ systemd-timesync
systemd+ systemd-resolve
beansta+ beanstalkd
message+ messagebus
daemon root
systemd+ systemd-network
# ps -e -o group= -o rgroup= | awk '$1 != $2'
systemd+ systemd-timesync
systemd+ systemd-resolve
beansta+ beanstalkd
message+ messagebus
daemon root
systemd+ systemd-network
2) The binaries in question do not have any capabilities (the following commands produce no output):
# getcap /usr/lib/php/20170718/oci8.so
# getcap -r /opt/oracle/instantclient_12_2/
3) I have ensured that AppArmor is disabled (it doesn't have a policy that should affect PHP-FPM, anyway):
# systemctl disable apparmor
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apparmor
# reboot
# aa-status
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
So, why does PHP-FPM ignore LD_LIBRARY_PATH, if not for any of the aforementioned reasons?
EDIT 5 (Solution):
An astute commenter, #vinc17 , points-out that on systems running systemd, environment variables, such as LD_LIBRARY_PATH, are not necessarily propagated to processes that are started via a systemd Unit.
In other words, PHP-FPM isn't "ignoring" LD_LIBRARY_PATH, but rather, it is not being conveyed to the process. And attempts to set LD_LIBRARY_PATH within the PHP-FPM configuration are futile, because it's too late to do anything useful with the value.
On this advice, it occurred to me to set LD_LIBRARY_PATH in the systemd context, namely, in the Unit file(s) that start the PHP-FPM daemon(s), in which case PHP-FPM loads the OCI8 extension successfully.
Needless to say, we want to avoid editing the package maintainer's file (to avoid conflicts during future upgrades), so we extend it instead:
# mkdir /etc/systemd/system/php7.1-fpm.service.d
# touch /etc/systemd/system/php7.1-fpm.service.d/environment.conf
To this file we add the following:
[Service]
Environment=LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2
And to make the change effective:
# systemctl daemon-reload
# systemctl restart php7.1-fpm
For a more complete example, which addresses multiple co-installed PHP versions, please see my post at https://github.com/oerdnj/deb.sury.org/issues/865#issuecomment-395441936 .
First, Debian bug 859732 is a completely different issue (I would even say an opposite issue): for this bug, several versions of the library are present in the search path (one in some directory specified by LD_LIBRARY_PATH and a different one in some directory specified by the run path), but the wrong one is chosen by the dynamic linker.
In your case, the problem is that the requested library is not found anywhere in the search path. Note also that in your case, it is PHP that seems to try to open the library (via dlopen?), since the message starts with "PHP Warning:". However, it seems that the mechanisms are the same as with usual dynamic linking.
After installing the library, what you need is at least one of:
Nothing special if the library has been installed in a directory that is searched by default. Since you get an error, this is not your case.
Providing the directory in a run path, which must be specified at compile time of the software that will need the library. The problem is that under Linux, this is not done in standard by the build tools, and it may be complex to do it right without breaking other things. However, in the context of dlopen, the software (here, PHP) may have set up what one can call a "plugin search path", where you can put your libraries.
Providing the directory in LD_LIBRARY_PATH. This is what you tried, but your LD_LIBRARY_PATH seems incorrect. Libraries are usually installed in subdirectories named lib (or lib32 or lib64 in specific cases). So, export LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2 seems wrong. Search for the full pathname of the library oci8.so, and just take the directory part of this pathname for LD_LIBRARY_PATH.
Note: strace may be useful to see what directories are considered to search for the libraries. EDIT: ldd and objdump -p are other useful tools to find what's going on with the search paths.
EDIT 2: Another point to note when choosing to use a run path is that indirect library dependencies are found when RPATH is used, but not when RUNPATH is used (so, in this latter case, all dependencies also need to have a run path if they depend on other libraries so that all libraries can be found without resorting to LD_LIBRARY_PATH). This is documented in recent versions of the ld.so(8) man page:
Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present. Such directories are searched only to find those objects required by DT_NEEDED (direct dependencies) entries and do not apply to those objects' children, which must themselves have their own DT_RUNPATH entries. This is unlike DT_RPATH, which is applied to searches for all children in the dependency tree.
This is probably why, without using LD_LIBRARY_PATH, this was working with 16.04 (where RPATH is used) but not with 18.04 (where RUNPATH is used).

Heroku Worker Caching

I'm having trouble getting my worker process to update in Heroku. I have a worker dyno set in my Procfile that is connected to a Redis instance; I cannot see changes that I make to this file after deploying.
I've tried:
Resetting the dyno using heroku ps:restart worker.1 -a [appname]
Restarting all dynos using heroku ps:restart -a [appname]
Changing the contents of the file so the size is different
Changing the Procfile to point to a different PHP file
Nothing works. It looks like it picked up some of my changes overnight (maybe during reboot?) but I can't force it to pick up the changes... any ideas?
Logs to the rescue... I had an issue with my required include file paths, which was causing the build to fail. It was defaulting to the last successful build, which is why it looked like it was caching.
I was able to find this by watching the logs during the build:
heroku[worker.1]: Starting process with command `php bin/worker.php`
heroku[worker.1]: State changed from starting to up
heroku[worker.1]: Process exited with status 255
app[worker.1]: PHP Warning: require_once(../vendor/autoload.php): failed to open stream: No such file or directory in /app/bin/worker.php on line 9
app[worker.1]: PHP Fatal error: require_once(): Failed opening required '../vendor/autoload.php' (include_path='.:/app/.heroku/php/lib/php') in /app/bin/worker.php on line 9
heroku[worker.1]: State changed from up to crashed
Once I solved the build errors it no longer 'cached' and everything worked properly.
The include path that worked:
require_once(__DIR__ . '/../vendor/autoload.php');

Nginx + php-fpm: Bad gateway only when xdebug server is running

Problem
When xdebug server is running from IntelliJ IDEA, I get 502 Bad Gateway from nginx when I try loading my site to trigger breakpoints.
If I stop the xdebug server, the site works as intended.
So, I'm not able to run the debugger, but it did work previously (!). Not able to pinpoint why it suddenly stopped working.
Setup
A short explanation of the setup (let me know if I need to expand on this).
My php app is running in a docker container, and it is linked to nginx running in a different container using volumes_fromin the docker compose config.
After starting the app, I can verify using phpinfo(); the xdebug module is loaded.
My xdebug.ini has the following content:
zend_extension=xdebug.so
xdebug.remote_enable=1
xdebug.remote_host=10.0.2.2
xdebug.remote_connect_back=0
xdebug.remote_port=5555
xdebug.idekey=complex
xdebug.remote_handler=dbgp
xdebug.remote_log=/var/log/xdebug.log
xdebug.remote_autostart=1
I got the ip address for remote_host (where the xdebug server is running) by these steps:
docker-machine ssh default
route -n | awk '/UG[ \t]/{print $2}' <-- Returns 10.0.2.2
To verify I could reach the debugging server from within my php container, I did the following steps
docker exec -it randomhash bash
nc -z -v 10.0.2.2 5555
Giving the following output depending on xdebug server running or not:
Running: Connection to 10.0.2.2 5555 port [tcp/*] succeeded!
Not running: nc: connect to 10.0.2.2 port 5555 (tcp) failed: Connection refused
So IntelliJ IDEA is surely set up to receive connections on 5555. I also did the appropriate path mapping between my source file paths and the remote path (when setting up the PHP Remote Debugging server from within IDEA).
Any ideas? Kind of lost on this one as I don't have much experience with any of these technologies :D
This sometimes happens, the reason is the errors in php-fpm and xdebug (exactly)!
When I refactored my colleagues code, оne page on the project returned 502 Bad Gateway
Here's what I found:
php-fpm.log
WARNING: [pool www] child 158 said into stderr: "*** Error in `php-fpm: pool www': free(): invalid size: 0x00007f1351b7d2a0 ***"
........
........
WARNING: [pool www] child 158 exited on signal 6 (SIGABRT - core dumped) after 38.407847 seconds from start
I found a piece of code that caused the error:
ob_start();
$result = eval("?>".$string."<"."?p"."hp return 1;");
$new_string = ob_get_clean();
But that is not all. The error occurred only in a certain state $string which at first glance, did not differ from the others. In my case, everything is simple. I removed the code that caused the error. This did not affect the functionality of the web page. I continued to debug the code further.
I had the same problem with the Vagrant Homestead Parallels box with a Silicon chip. Switching from php 7.3 to 7.4 fixed the issue for me.

Sending errors to syslog in php-fpm

I'm trying to setup an php-fpm daemon to send error_log to syslog.
My config php-fpm.conf looks thereby.
error_log = syslog
syslog.facility = local4
syslog.ident = php-fpm
Restarting php-fpm daemon I can see in the remote syslog server.
Mar 22 00:32:08 192.168.33.14 php-fpm[20919]: [NOTICE] configuration file /etc/php5/fpm/php-fpm.conf test is successful
Good!
But when I test this with a real php script with errors, message change to..
Mar 22 00:05:59 192.168.33.14 ool www[20889]: PHP Parse error: syntax error, unexpected ''api'' (T_CONSTANT_ENCAPSED_STRING) in /var/api/public/index.php on line 2
ool www is pool + default pool name (www). So two questions:
Why I see ool www instead of php-fpm (syslog.ident value)?
Why is pool title is truncated?
I think its a PHP bug :(
See please
https://bugs.php.net/bug.php?id=67764
This issue will become more prevalent as we start to upgrade older systems to php 5.5+, so posting for future reference.
Here is the fix we used, put it as low level as you can so all your scripts can benefit:
// This was needed for upgrade to php5.5+ with php-fpm
// Source: https://gist.github.com/gjuric/e0c9e45efb3d15e3b949
// Bug History: http://stackoverflow.com/questions/22575152/sending-errors-to-syslog-in-php-fpm
openlog('php', LOG_ODELAY, LOG_USER);

PHP Extension (memcache|memcached) not showing in phpinfo() but showing in php -m and php -i?

I am getting both modules listed as installed / configured when I use:
php -m
or if I use:
php -i
but when I use:
$m = new Memcache;
// or
$m = new Memcache();
// or
$m = new Memcached();
//or
$m = new Memcached;
I get the following error:
Fatal error: Class 'Memcached' not found
I am running on a Mac - OS X (10.5.7) with default install of apache & php. Additionally, I have memcached running as a daemon on 127.0.0.1:11211 and libmemcache as required by the php-memcached library. I have restarted apache tons of times and even done a machine restart.
Does anyone know why the modules/extensions show up in the command line but not in my phpinfo()? I am literally stumped, after 3 hours of googling, I am just about ready to give up.
Also, please note, my phpinfo() outputs my ini files as follows AND they are both the exact same file:
Configuration File (php.ini) Path: /etc
Loaded Configuration File: /private/etc/php.ini
UPDATE:
Apache is failing to load the extension.
[Fri May 14 04:22:26 2010] [warn]
Init: Session Cache is not configured
[hint: SSLSessionCache] PHP Warning:
PHP Startup: Unable to load dynamic
library
'/usr/lib/php/extensions/no-debug-non-zts-20060613/memcached.so'
- (null) in Unknown on line 0 PHP Warning: PHP Startup: Unable to load
dynamic library
'/usr/lib/php/extensions/no-debug-non-zts-20060613/memcache.so'
- (null) in Unknown on line 0
Does anyone know why or how this would happen? Both of the files referenced above DEFINITELY ARE there. Should I move this question to server fault?
Your webserver is probably using mod_php, which is a seperate bit of code from the standalone (CLI) interpreter. If they are both using the same ini file, and the memcache extension is configured in the ini file, then it sounds like for some reason, the mod_php is failing to load the extension - check your webserver error_log for startup errors.
It may be that the mod_php was compiled without memcache support (most extensions need to have a stub file linked into the php code, even though the bulk of the code is not linked until run time). Or it may be a permissions problem on the shared object file. Or your webserver may be running chroot, and unable to find the extension (which would also mean that although the ini files appear to have the same path, this is relative to different roots).
HTH
C.
because both versions use different php.ini
place your php.ini into location noted in the phpinfo() outout
I would suspect that the issue revolves around permissions. When you run php from comand line, it runs as the user invoking it. When run as an apache module, it runs as "nobody".
I would assume that the memcached.so file, or the directory it's in does not have proper permissions.
I stumpled upon this post and was having the exact same problem with an extension in my php -i but not in phpinfo(). Mine was a permissions problem because of selinux on a CentOS machine. I had to change ownership and permissions and now it is working as expected.
Set php path in environment variables as given below.
Right-click on "My Computer"
Properties
Advanced Tab > Environment Variables
Under System variables, scroll down to find "Path", select and click on Edit.
Add path to your php install on the end (make sure to precede with semi-colon ";"). Example: ";C:\php7"
Click Ok.

Categories