Below is the code I'm trying to finish to view logs on our PBX. It runs fine, sorts through the log files, displays all the data, etc. The problem is that once it's done, it keeps a large amount of data (several hundred MB) in memory. I need it cleared out when it's done but can't figure out how. If it's run a couple of times it sucks all of the RAM from the server and we start getting errors. Thanks in advance!
<!DOCTYPE html>
<html>
<title>Call Search</title>
<body>
<?php
function getBetween($var1, $var2, $pool)
{
$temp1 = strpos($pool, $var1) + strlen($var1);
$result = substr($pool, $temp1, strlen($pool));
$dd=strpos($result, $var2);
if($dd == 0){$dd = strlen($result);}
return substr($result, 0, $dd);
}
if(!isset($_POST['phonenum']) && !isset($_POST['WantedCallID']))
{
echo '<form action="test.php" method="post">';
echo '<h4>Please enter the number you wish to search for...</h4><input type="text" name="phonenum">';
echo '<input type="submit" value="Go">';
echo '</form>';
}
else if(isset($_POST['WantedCallID']))
{
$counter = 0;
$logfile="calllog";
while (file_exists("/var/log/".$logfile))
{
if($counter==0){$logfile="calllog";}
else{$logfile="callog.".$counter;}
$path = "/var/log/".$logfile;
$logtoarray = file($path);
foreach ($logtoarray as $linenumber => $line)
{
if(strpos($line, "[".$_POST['WantedCallID']."]") !== false){echo $line." <br>";}
}
$counter++;
}
}
else
{
$counter = 0;
$logfile="calllog";
$arrayCounter = 0;
while (file_exists("/var/log/".$logfile))
{
if($counter == 0){$logfile = "calllog";}
else{$logfile="calllog.".$counter;}
$path = "/var/log/".$logfile;
if(file_exists($path))
{
$logtoarray = file($path);
$oldCallId = "";
$currentCallId = "";
foreach ($logtoarray as $linenumber => $line)
{
if(strpos($line, $_POST['phonenum']) !== false && strpos($line, "VERBOSE[") !== false)
{
$currentCallId = getBetween("VERBOSE[", "]", $line);
if($currentCallId !== $oldCallId)
{
$callIdArray[$arrayCounter] = "Date: ".substr($line, 0, 22)." Call ID:".$currentCallId."\n";
echo "Date: ".substr($line, 0, 22)." Call ID:".$currentCallId."<br>";
$oldCallId = $currentCallId;
}
}
$arrayCounter++;
}
}
else{break;}
$counter++;
}
echo '<form action="test.php" method="post">';
echo '<h4>Please enter the call ID you wish to search for...</h4><input type="text" name="WantedCallID">';
echo '<input type="submit" value="Go">';
echo '</form>';
}
$variables = array_keys(get_defined_vars());
foreach($variables as $variable) {
unset(${"$variable"});
}
unset($callIdArray);
?>
</body>
</html>
I'm going to presume that you run this script through an Apache server. This is a fairly common problem, though I've more often seen it from resizing images.
Once an Apache child (the process that runs PHP) has had the amount of memory it, or more usually, PHP, increased because of a lot of data being used, it doesn't have a method to return the memory to the OS.
Instead, each child-process has the MaxRequestsPerChild setting. This might be set to 0 - never restart. If the inflating memory size is a problem, then setting this to a few hundred, or thousand, might be useful. After that many requests, the apache child is restarted, and so all the memory captured si returned to the OS, and a new child is spun up if needed. While the requests might take a little longer when the process is being restarted, the freeing of memory would be more useful.
Here is a simple modification to your code that will read the logfiles one line at a time rather than reading the entire file into memory.
while (file_exists("/var/log/".$logfile))
{
if($counter == 0){$logfile = "calllog";}
else{$logfile="calllog.".$counter;}
$path = "/var/log/".$logfile;
if(file_exists($path))
{
$fh = fopen($path, 'r'); //change
$linenumber = 0; //addition
$oldCallId = "";
$currentCallId = "";
while( $line = fgets($fh) ) //change
{
$linenumber++; //addition
if(strpos($line, $_POST['phonenum']) !== false && strpos($line, "VERBOSE[") !== false)
{
$currentCallId = getBetween("VERBOSE[", "]", $line);
if($currentCallId !== $oldCallId)
{
$callIdArray[$arrayCounter] = "Date: ".substr($line, 0, 22)." Call ID:".$currentCallId."\n";
echo "Date: ".substr($line, 0, 22)." Call ID:".$currentCallId."<br>";
$oldCallId = $currentCallId;
}
}
$arrayCounter++;
}
fclose($fh); //addition
}
}
Related
I decided to try out the latest PHP ZIP functions for extracting data and something odd has happened. This PHP script is to read data from packed files of a specific extension (gob or goo) and display them in an XML format for further use later.
$zip = new ZipArchive;
$zipres = $zip->open($zipfilename);
if($zipres === true)
{
for($i = 0; $i < $zip->numFiles; $i++)
{
echo "\t<entry id=\"".$i."\">".$zip->getNameIndex($i)."</entry>\n";
if(strripos($zip->getNameIndex($i),".gob") !== false || strripos($zip->getNameIndex($i),".goo") !== false)
{
$zipentry = $zip->getStreamIndex($i);
return;
if($zipentry)
{
$packagenames[] = $zip->getNameIndex($i);
$wholeGOB[] = stream_get_contents($zipentry);
fclose($zipentry);
}
else
echo "\t<error>Error reading entry ".$zip->getNameIndex($i)."</error>";
}
}
$zip->close();
}
else
echo "\t<error>Invalid ZIP file</error>\n\t<code>".$zipres."</code>\n";
Having the return; before getStreamIndex will print out the <entry>s, but as soon as I move the return; to after getStreamIndex I get a blank page (even in view-source). Am I doing something wrong, or has this function always been a bit off? And what alternative would you recommend?
I have eight csv sheets, which where updated (every two hours) by an external service. I want to analyse these data - showing it in a php page for my boss....
I imagined, that it would be the easiest way to have the .csv in a GTT on my MsSQL Server. Maybe it is possible to create a GTT on ervery use (on commit preserve rows)? I still have no approach to make the csv to a normal format for insert syntax... is it even possible?
Or is there an other option? I try to catch some ideas for the solution...
Thank you!
EDIT
I have php code to read the .csv and put it into an array.
$csv_datei = "applicationData.csv";
$feler_trenner = ";";
$zeilen_trenner = "n";
if (#file_exists($csv_datei) == false) {
echo 'Die CSV Datei: '. $csv_datei.' gibt es nicht!';
}
else {
$datei_inhalt = #file_get_contents($csv_datei);
$zeilen = explode($zeilen_trenner, $datei_inhalt);
if (is_array($zeilen) == true) {
foreach($zeilen as $zeile) {
$felder = explode($feler_trenner, $zeile);
$i = 0;
if (is_array($felder) == true) {
foreach($felder as $felde) {
if ($felde != '') {
echo (($i != 0) ? ', ':'') . str_replace('"', '', $felde);
$i++;
}
}
}
echo '<br>';
}
}
}
source: PHP-space.info
i am currently developing a web crawler in PHP and it still is a simple one but what i want to know is how can i make my crawler to crawl pages in background and not use my bandwidth, do i have to use some cron jobs and i want it to automatically store the data in database.
Here what i have done:
<?php
$conn = mysqli_connect("localhost","root","","crawler") or die(mysqli_error());
ini_set('max_execution_time', 4000);
$to_crawl = "http://hootpile.com";
$c = array();
function get_links($url){
global $c;
$input = file_get_contents($url);
$regexp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
preg_match_all("/$regexp/siU", $input, $matches);
$base_url = parse_url($url, PHP_URL_HOST);
$l = $matches[2];
foreach($l as $link) {
if(strpos($link, "#")) {
$link = substr($link,0, strpos($link, "#"));
}
if(substr($link,0,1) == ".") {
$link = substr($link, 1);
}
if(substr($link,0,7)=="http://") {
$link = $link;
}
else if(substr($link,0,8) =="https://") {
$link = $link;
}
else if(substr($link,0,2) =="//") {
$link = substr($link, 2);
}
else if(substr($link,0,2) =="#") {
$link = $url;
}
else if(substr($link,0,2) =="mailto:") {
$link = "[".$link."]";
}
else {
if(substr($link,0,1) != "/") {
$link = $base_url."/".$link;
}
else {
$link = $base_url.$link;
}
}
if(substr($link, 0, 7)=="http://" && substr($link, 0, 8)!="https://" && substr($link, 0, 1)=="[") {
if(substr($url, 0, 8) == "https://") {
$link = "https://".$link;
}
else {
$link = "http://".$link;
}
}
//echo $link."<br />";
if(!in_array($link,$c)) {
array_push($c,$link);
}
}
}
get_links($to_crawl);
foreach ($c as $page) {
get_links($page);
}
foreach ($c as $page) {
$query = mysqli_query($conn,"INSERT INTO LINKS VALUES('','$page')");
echo $page."<br />";
}
?>
You can use SimpleHTML Dom, But crawling/scraping depend on the web page structure. How many data you want to store, May be you can't found same data and structure on different websites. In case you should make some common program to fetch data from scraped data.
You can use ReactPHP, since it allows you to easily spawn a process that keeps running.
You also can write a hashbang at the very beggining of the file:
#/usr/bin/php
give execution permissions to the file:
chmod a+x your_script_path.php
and execute it with cron or with nohup. If you want to daemonize it, then there is a little more work.
I think you should not use PHP for crawler/scraper because it's not designed for long-running tasks. It will cause memory usage problem, use Python instead (I use Python + BeautifulSoup + urllib for scraper).
Besides you should use crontab and nohup to schedule background jobs.
I have a function that will take an id and with that find out other information in the database relating to it spread among 3 tables. It then compares this to a csv file which at most times is cpu intensive. Running this once with one id takes approx 8 to 10 sec at most but I have been asked to have it run automatically across a varing number of ids in the database. To do this I created an array of the ids that match the criteria in the database at any point and then run a 'while' statement to repeat the function for each element in the array but it gets as far as maybe 4 of them and I get the following error:
Server error!
The server encountered an internal error and was unable to complete
your request. Either the server is overloaded or there was an error in
a CGI script.
If you think this is a server error, please contact the webmaster.
Error 500
I'll admit that my code could be much much cleaner as I'm still learning as I go but the real bottle neck appears to be reading the csv which is a report which size changes each day. I have tried different combinations and the best result is (please don't chastise me for this as I know it is stupid but the other ways haven't works as of yet) to run the code as follows:
$eventArray = eventArray($venueId);
$totalEvents = count($eventArray);
for($i=0; $i<$totalEvents; $i++)
{
$eventId = $eventArray[$i];
echo $eventId;
echo $datename = getEventDetails($eventId, $zone);
// date of event
echo $eventDate = $datename['eventDate'];
// vs team
echo $eventName = $datename['eventName'];
$file_handle = fopen("data/rohm/sales/today.csv", "r");
while (!feof($file_handle) )
{
$line_of_text = fgetcsv($file_handle, 200);
include('finance_logic.php');
}
fclose($file_handle);
}
Yes, it is repeating the reading of the csv every time but I couldn't get it to function at all any other way so if this is the issue I would really appreciate some guidence on dealing with the csv better. Incase it is relevent the code it 'finance_logic.php' is listed below:
if($line_of_text[0] == "Event: $eventName ")
{
$f = 1;
$ticketTotalSet = "no";
$killSet = 'no';
// default totals zero
$totalHolds = 0;
$totalKills = 0;
$ticketSold = 0;
$ticketPrice = 0;
$totalCap = 0;
}
if($f == 1 && $line_of_text[0] == "$eventDate")
{
$f = 2;
}
if($f == 2 && $line_of_text[0] == "Holds")
{
$f = 3;
while($line_of_text[$col] !== "Face Value Amt")
{
$col++;
}
}
if($f == 3 && $line_of_text[0] !== "Face Value Amt")
{
if($f == 3 && $line_of_text[0] == "*: Kill")
{
$totalKills = $line_of_text[$col];
}
$holdsArray[] = $line_of_text[$col];
}
if($f == 3 && $line_of_text[0] == "--")
{
$f = 4;
}
if($f == 4 && $line_of_text[0] == "Capacity")
{
$totalCap = $line_of_text[$col];
$f = 5;
}
if($f == 5 && $line_of_text[0] == "Abbreviated Performance Totals")
{
$f = 6;
}
if($f == 6 && $line_of_text[0] == "$eventName")
{
// change when 1 ticket exists
$ticketTotalSet = "yes";
// set season tickets
include("financial/seasontickets/$orgShortName.php");
// all non season are single tickets
if(!isset($category))
{
$category = 'single';
}
$ticketName = $line_of_text[2];
$ticketSold = $line_of_text[3];
$ticketPrice = $line_of_text[4];
addTicketType($eventId, $ticketName, $category, $ticketSold, $ticketPrice);
unset($category);
}
if($f == 6 && $ticketTotalSet == "yes" && $line_of_text[0] !== "$eventName")
{
$totalHolds = (array_sum($holdsArray) - $totalKills);
// add cap, holds and kills
addKillsHoldsCap($eventId, $totalCap, $eventId, $totalHolds, $totalKills);
// reset everything
$f = 0;
$ticketTotalSet = "no";
echo "$eventName updated!";
}
Thanks in advance!
p.s. The reason the report is called each time is so that the 'eventName' and 'eventDate' are searched for with the 'finance_logic.php'. Obviously if this was set with all event names and dates already it would take one search of the report to find them all but I'm not sure how I could do this dynamically. Any suggestions would be welcome as I'm sure there is something out there that I just haven't learnt yet.
I have some heavy script i use with localhost sometimes and if i don't add anything they will just time out.
A simple solution is to limit the number of execution of your function, then reload the page, then restart where you stopped.
How do you calculate the server load of PHP/apache? I know in vBulletin forums there's the server load displayed like 0.03 0.01 0.04 but it's not really understandable to a average joe. So I thought about a 1-100 percentile scale. I wanted to display a server load visual that reads from a DIV:
$load = $serverLoad*100;
<div class=\"serverLoad\">
<div style=\"width: $load%;\"></div>//show visual for server load on a 1-100 % scale
</div>
<p>Current server load is $load%</p>
</div>
However, I don't know how to detect server load. Is it possible to do turn this server load into a percentile scale? I don't know where to start. May someone please help me out?
Thanks.
I have a very old function that should still do the trick:
function getServerLoad($windows = false){
$os=strtolower(PHP_OS);
if(strpos($os, 'win') === false){
if(file_exists('/proc/loadavg')){
$load = file_get_contents('/proc/loadavg');
$load = explode(' ', $load, 1);
$load = $load[0];
}elseif(function_exists('shell_exec')){
$load = explode(' ', `uptime`);
$load = $load[count($load)-1];
}else{
return false;
}
if(function_exists('shell_exec'))
$cpu_count = shell_exec('cat /proc/cpuinfo | grep processor | wc -l');
return array('load'=>$load, 'procs'=>$cpu_count);
}elseif($windows){
if(class_exists('COM')){
$wmi=new COM('WinMgmts:\\\\.');
$cpus=$wmi->InstancesOf('Win32_Processor');
$load=0;
$cpu_count=0;
if(version_compare('4.50.0', PHP_VERSION) == 1){
while($cpu = $cpus->Next()){
$load += $cpu->LoadPercentage;
$cpu_count++;
}
}else{
foreach($cpus as $cpu){
$load += $cpu->LoadPercentage;
$cpu_count++;
}
}
return array('load'=>$load, 'procs'=>$cpu_count);
}
return false;
}
return false;
}
This returns processor load. You can also use memory_get_usage and memory_get_peak_usage for memory load.
If you can't handle finding percentages based on this... sigh, just post and we'll try together.
This function in PHP might do that trick: sys_getloadavg().
Returns three samples representing the average system load (the number of processes in the system run queue) over the last 1, 5 and 15 minutes, respectively.
function get_server_load()
{
$serverload = array();
// DIRECTORY_SEPARATOR checks if running windows
if(DIRECTORY_SEPARATOR != '\\')
{
if(function_exists("sys_getloadavg"))
{
// sys_getloadavg() will return an array with [0] being load within the last minute.
$serverload = sys_getloadavg();
$serverload[0] = round($serverload[0], 4);
}
else if(#file_exists("/proc/loadavg") && $load = #file_get_contents("/proc/loadavg"))
{
$serverload = explode(" ", $load);
$serverload[0] = round($serverload[0], 4);
}
if(!is_numeric($serverload[0]))
{
if(#ini_get('safe_mode') == 'On')
{
return "Unknown";
}
// Suhosin likes to throw a warning if exec is disabled then die - weird
if($func_blacklist = #ini_get('suhosin.executor.func.blacklist'))
{
if(strpos(",".$func_blacklist.",", 'exec') !== false)
{
return "Unknown";
}
}
// PHP disabled functions?
if($func_blacklist = #ini_get('disable_functions'))
{
if(strpos(",".$func_blacklist.",", 'exec') !== false)
{
return "Unknown";
}
}
$load = #exec("uptime");
$load = explode("load average: ", $load);
$serverload = explode(",", $load[1]);
if(!is_array($serverload))
{
return "Unknown";
}
}
}
else
{
return "Unknown";
}
$returnload = trim($serverload[0]);
return $returnload;
}
If "exec" is activated
/**
* Return the current server load
* It needs "exec" activated
*
* e.g.
* 15:06:37 up 10 days, 5:59, 12 users, load average: 1.40, 1.45, 1.33
* returns
* float(1.40)
*/
public function getServerLoad()
{
preg_match('#(\d\.\d+),[\d\s\.]+,[\d\s\.]+$#', exec("uptime"), $m);
return (float)$m[1];
}
<?php
$load = sys_getloadavg();
if ($load[0] > 0.80) {
header('HTTP/1.1 503 Too busy, try again later');
die('Server too busy. Please try again later.');
}
?>
Its Pretty easy in LAMP environment if you have proper permissions,
print_r(sys_getloadavg());
OUTPUT : Array ( [0] => 0 [1] => 0.01 [2] => 0.05 )
Array values are average load in the past 1, 5 and 15 minutes respectively.
Here is a windows/linux compatible php server load function:
function get_server_load() {
$load = '';
if (stristr(PHP_OS, 'win')) {
$cmd = 'wmic cpu get loadpercentage /all';
#exec($cmd, $output);
if ($output) {
foreach($output as $line) {
if ($line && preg_match('/^[0-9]+$/', $line)) {
$load = $line;
break;
}
}
}
} else {
$sys_load = sys_getloadavg();
$load = $sys_load[0];
}
return $load;
}
function _loadavg()
{
if(class_exists("COM"))
{
$wmi = new COM("WinMgmts:\\\\.");
$cpus = $wmi->InstancesOf("Win32_Processor");
foreach ($cpus as $cpu)
{
return $cpu->LoadPercentage;
}
}
}