I'm debugging a segfault in my php app, I compiled php with --enable-debug option and after php-fpm dumped core during the segfault, I run:
$ gdb /usr/local/sbin/php-fpm core
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/php-fpm...(no debugging symbols found)...done.
[New LWP 2110]
warning: Can't read pathname for load map: No error information.
Core was generated by `php-fpm:'.
Program terminated with signal SIGBUS, Bus error.
#0 0x000055c27ba1f291 in ?? ()
(gdb) bt
#0 0x000055c27ba1f291 in ?? ()
#1 0x000055c27ba21bce in _efree ()
#2 0x000055c27ba6c8b2 in ?? ()
#3 0x000055c27ba70cee in zend_array_destroy ()
#4 0x000055c27ba56fbd in _zval_dtor_func ()
#5 0x000055c27ba6cce4 in ?? ()
#6 0x000055c27ba70cc8 in zend_array_destroy ()
#7 0x000055c27ba56fbd in _zval_dtor_func ()
#8 0x000055c27ba6cce4 in ?? ()
#9 0x000055c27ba70c8a in zend_array_destroy ()
#10 0x000055c27ba56fbd in _zval_dtor_func ()
#11 0x000055c27ba6cce4 in ?? ()
#12 0x000055c27ba70cc8 in zend_array_destroy ()
#13 0x000055c27ba56fbd in _zval_dtor_func ()
#14 0x000055c27ba44b75 in ?? ()
#15 0x000055c27ba450fe in zend_cleanup_user_class_data ()
#16 0x000055c27ba3f473 in ?? ()
#17 0x000055c27ba5971b in zend_deactivate ()
#18 0x000055c27b9bf1df in php_request_shutdown ()
#19 0x000055c27bb55015 in ?? ()
#20 0x00007f7fb743e964 in __libc_start_main () from /lib/ld-musl-x86_64.so.1
#21 0x0000000000000000 in ?? ()
I also put .gdbinit from php sources to the directory with core file and set set auto-load safe-path / to ~/.gdbinit (not sure how it helps though).
As you can see there are ?? in the backtrace which is not helpful at all. Is this how it should be? Is there any way to get the real caller?
Is this how it should be?
No.
Is there any way to get the real caller?
Probably.
Your stack trace indicates that /usr/local/sbin/php-fpm is fully-stripped (has its symbol table removed). You can confirm this with file /usr/local/sbin/php-fpm.
It is not uncommon for make install to fully strip binaries. You should try to use the "as built" binary (instead of "as installed" you are doing now): gdb /path/to/build/tree/php-fpm core.
Related
OS: Ubuntu 20.04 LTS
PHP Version: 7.4.22
I am trying to install a package with composer. But it caused a Segmentation Fault:
root#micro7:/www/wwwroot/micro7.ml# composer require laravel/breeze
Segmentation fault (core dumped)
The problem also appeared when creating project with composer:
root#micro7:~# composer create-project laravel/laravel micro7.ml
Creating a "laravel/laravel" project at "./micro7.ml"
Segmentation fault (core dumped)
I used gdb to get the stack trace of the dumped core, it looked like this:
root#micro7:~# gdb php core
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from php...
[New LWP 583799]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php /usr/bin/composer create-project laravel/laravel micro7.ml'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fec2b356e00 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
(gdb) bt
#0 0x00007fec2b356e00 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#1 0x00007fec2c1ecf6c in SSL_CTX_new () from /usr/local/openssl/lib/libssl.so.1.0.0
#2 0x00007fec2bf0ab3c in ossl_connect_step1 () from /usr/local/lib/libcurl.so.4
#3 0x00007fec2bf0c4ff in ossl_connect_common () from /usr/local/lib/libcurl.so.4
#4 0x00007fec2bf0d47c in Curl_ssl_connect_nonblocking () from /usr/local/lib/libcurl.so.4
#5 0x00007fec2bed2996 in https_connecting () from /usr/local/lib/libcurl.so.4
#6 0x00007fec2bed4383 in Curl_http_connect () from /usr/local/lib/libcurl.so.4
#7 0x00007fec2bee734d in multi_runsingle () from /usr/local/lib/libcurl.so.4
#8 0x00007fec2bee84b1 in curl_multi_perform () from /usr/local/lib/libcurl.so.4
#9 0x000055bdd4a2c339 in zif_curl_multi_exec ()
#10 0x000055bdd5166c3d in execute_ex ()
#11 0x000055bdd517e815 in zend_execute ()
#12 0x000055bdd4fce45b in zend_execute_scripts ()
#13 0x000055bdd4f05a84 in php_execute_script ()
#14 0x000055bdd51815df in do_cli ()
#15 0x000055bdd5182a24 in main ()
If you look at the first line of the output of the bt, you'll find that the Segmentation Fault maybe caused when establishing a SSL connection to the server of packagist.
I tried to reinstall composer, switch to the super user, reboot the server, even reinstalling the OS, but the problem still persist!
I'm trying to run PHP 5.5 (with FPM) with APCu inside a Docker container. I'm using boot2docker on OSX.
When I try to run php-fpm -i, it segfaults.
Running in gdb, I get the following backtrace:
(gdb) run -i
Starting program: /usr/local/sbin/php-fpm -i
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff542474c in __pthread_rwlock_init (rwlock=rwlock#entry=0xffffffffffffffff, attr=attr#entry=0x7ffff1bda988 <apc_lock_attr>) at pthread_rwlock_init.c:40
40 pthread_rwlock_init.c: No such file or directory.
(gdb) bt
#0 0x00007ffff542474c in __pthread_rwlock_init (rwlock=rwlock#entry=0xffffffffffffffff, attr=attr#entry=0x7ffff1bda988 <apc_lock_attr>) at pthread_rwlock_init.c:40
#1 0x00007ffff19ca9d0 in apc_lock_create (lock=lock#entry=0xffffffffffffffff) at /tmp/pear/temp/apcu/apc_lock.c:180
#2 0x00007ffff19d0135 in apc_sma_api_init (sma=0x7ffff1bda540 <apc_sma>, data=<optimized out>, expunge=<optimized out>, num=<optimized out>, size=<optimized out>, mask=<optimized out>) at /tmp/pear/temp/apcu/apc_sma.c:323
#3 0x00007ffff19ccac9 in zm_startup_apcu (type=0, module_number=35) at /tmp/pear/temp/apcu/php_apc.c:284
#4 0x00000000007198b6 in zend_startup_module_ex ()
#5 0x0000000000725565 in zend_hash_apply ()
#6 0x000000000071d32a in zend_startup_modules ()
#7 0x00000000006b9413 in php_module_startup ()
#8 0x00000000007d9da5 in ?? ()
#9 0x000000000042e684 in ?? ()
#10 0x00007ffff5657b45 in __libc_start_main (main=0x42de80, argc=2, argv=0x7fffffffec08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffebf8) at libc-start.c:287
#11 0x000000000042f7fa in _start ()
(gdb)
I thought this might be some general problem with phtreads in a Docker container, but I haven't been able to find any reports of that kind of problem.
Some questions:
Is this a known issue (and, so, trying to fix this through configuration is fruitless)?
What is the significance of 40 pthread_rwlock_init.c: No such file or directory.?
I'm working on a PHP extension that I'm trying to test on travis. Unit tests reveal a segfault that only occur on travis and I can't reproduce it locally no matter what I try.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff1ccc0000
Core was generated by `/home/travis/.phpenv/versions/5.6/bin/php -d output_handler= -d open_basedir= -'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000095c6b8 in gc_zval_check_possible_root (tsrm_ls=<optimized out>, z=<optimized out>) at /tmp/php-build/source/5.6.3/Zend/zend_gc.h:183
183 /tmp/php-build/source/5.6.3/Zend/zend_gc.h: No such file or directory.
Thread 1 (Thread 0x7f759eae4740 (LWP 12532)):
#0 0x000000000095c6b8 in gc_zval_check_possible_root (tsrm_ls=<optimized out>, z=<optimized out>) at /tmp/php-build/source/5.6.3/Zend/zend_gc.h:183
#1 i_zval_ptr_dtor (tsrm_ls=<optimized out>, zval_ptr=<optimized out>) at /tmp/php-build/source/5.6.3/Zend/zend_execute.h:86
#2 zend_do_fcall_common_helper_SPEC (execute_data=<optimized out>, tsrm_ls=0x32fee10) at /tmp/php-build/source/5.6.3/Zend/zend_vm_execute.h:564
#3 0x000000000092194b in execute_ex (execute_data=0x7f759eaa5100, tsrm_ls=0x32fee10) at /tmp/php-build/source/5.6.3/Zend/zend_vm_execute.h:363
#4 0x00000000008b1098 in zend_execute_scripts (type=8, tsrm_ls=0x32fee10, retval=0x0, file_count=3) at /tmp/php-build/source/5.6.3/Zend/zend.c:1344
#5 0x000000000083e404 in php_execute_script (primary_file=0x7fff1cc7ff70, tsrm_ls=0x32fee10) at /tmp/php-build/source/5.6.3/main/main.c:2584
#6 0x0000000000960467 in do_cli (argc=63, argv=0x32fe2f0, tsrm_ls=0x32fee10) at /tmp/php-build/source/5.6.3/sapi/cli/php_cli.c:994
#7 0x000000000044a960 in main (argc=63, argv=0x32fe2f0) at /tmp/php-build/source/5.6.3/sapi/cli/php_cli.c:1378
I really don't know what can cause it, since I can't reproduce.
Ok, I'm running a PHP app on command line on Ubuntu, and it ends with "Segmentation fault (core dumped)".
How do I go from here to debug it? I'm pretty sure there is no memory leak as I checked it already with get_memory_usage().
Edit: Alright, as explained by Brendan and Ulricht, I tried gdb. That's not my environment at all so sorry for the oncoming newbie questions.
I ran my code under gdb and got the segfault. Here are the first 22 lines.
(gdb) bt
#0 0x00000000006f5d36 in ?? ()
#1 0x00000000006f7625 in ?? ()
#2 0x00000000006f7b68 in zend_parse_parameters ()
#3 0x0000000000610584 in zif_array_rand ()
#4 0x00000000006dd9bb in dtrace_execute_internal ()
#5 0x000000000079da15 in ?? ()
#6 0x0000000000717748 in execute_ex ()
#7 0x00000000006dd8b9 in dtrace_execute_ex ()
#8 0x000000000079e060 in ?? ()
#9 0x0000000000717748 in execute_ex ()
#10 0x00000000006dd8b9 in dtrace_execute_ex ()
#11 0x000000000079e060 in ?? ()
#12 0x0000000000717748 in execute_ex ()
#13 0x00000000006dd8b9 in dtrace_execute_ex ()
#14 0x000000000079e060 in ?? ()
#15 0x0000000000717748 in execute_ex ()
#16 0x00000000006dd8b9 in dtrace_execute_ex ()
#17 0x000000000079e060 in ?? ()
#18 0x0000000000717748 in execute_ex ()
#19 0x00000000006dd8b9 in dtrace_execute_ex ()
#20 0x000000000079e060 in ?? ()
#21 0x0000000000717748 in execute_ex ()
#22 0x00000000006dd8b9 in dtrace_execute_ex ()
According to https://bugs.php.net/bugs-generating-backtrace.php, I should get "execute" calls, instead I have "execute_ex". Same thing?
The following command anyway does not return the function name (after having done frame 6):
print (char *)(executor_globals.function_state_ptr->function)->common.function_name
Attempt to extract a component of a value that is not a structure.
Edit2: I'd appreciate knowing why the downvote. I think it's a valid question and I haven't found a similar one for PHP. If there is then you're free to comment.
There are several things to pinpoint the error, but the first is to run the executable in gdb:
> gdb /usr/bin/php
....
(gdb) run path/to/script
Note that you can also load the dumped core into gdb. Other tools for finding out what could cause the issue are strace and valgrind.
I have several crons running at various times every day. Occasionally, one or another will get 'stuck' and its process will never die. This isn't consistent, but after a change in a function in a different file which several of the crons access, there's been a marked increase in the number of times. Not always to the same file, not always to the same time, and running the files manually works fine.
My questions:
If I modify this external file, the running cron should pick up the change, right? (Tried and it isn't working, so either it doesn't, or it didn't get to that point in my code, which brings me to my second question:)
How can I print a stack trace of the running cron in order to see where it gets stuck? Running on Linux, Centos version 7
Edit: pstack gives me main() - how can I get farther in than that?
#0 0x00007f710a129e0d in poll () from /lib64/libc.so.6
#1 0x00007f7102391ab9 in Curl_poll () from /lib64/libcurl.so.4
#2 0x00007f710238aa4d in curl_multi_wait () from /lib64/libcurl.so.4
#3 0x00007f7102383baf in curl_easy_perform () from /lib64/libcurl.so.4
#4 0x00007f71025c96f6 in zif_curl_exec () from /usr/lib64/php/modules/curl.so
#5 0x00007f71029e6a4d in xdebug_execute_internal (current_execute_data=0x7f710d8f14a0, return_value_used=1) at /var/tmp/xdebug/xdebug.c:1547
#6 0x00007f710dc61271 in zend_do_fcall_common_helper_SPEC ()
#7 0x00007f710dbde617 in execute ()
#8 0x00007f71029e5e3a in xdebug_execute (op_array=0x7f70fa316bc8) at /var/tmp/xdebug/xdebug.c:1435
#9 0x00007f710dc6190d in zend_do_fcall_common_helper_SPEC ()
#10 0x00007f710dbde617 in execute ()
#11 0x00007f71029e5e3a in xdebug_execute (op_array=0x7f710f89efb0) at /var/tmp/xdebug/xdebug.c:1435
#12 0x00007f710dc6190d in zend_do_fcall_common_helper_SPEC ()
#13 0x00007f710dbde617 in execute ()
#14 0x00007f71029e5e3a in xdebug_execute (op_array=0x7f710d921838) at /var/tmp/xdebug/xdebug.c:1435
#15 0x00007f710dbb727f in zend_execute_scripts ()
#16 0x00007f710db56656 in php_execute_script ()
#17 0x00007f710dc63548 in do_cli ()
#18 0x00007f710da1015e in main ()
Thanks for any help!
If any of the crons are leaving the file open then this might be the cause when a concurrent access attempt is made to the same file by different cron instances. You could write a small shell script to execute the code file preceded by something like
$ok = `lsof | grep -c <filename>`
if($ok != "0")
then sleep 5
fi