I have written a very simple page counter and a logging script that increments a counter stored in a file and logs information about the client's operating system and which browser they use. It's a simple spare time project I've been working on, and as such it is extremely rudimentary, writing the counter and the logged information in a designated folder for each page on the site, in a new file for each day.
The thing is, I recently used blitz.io to test my site, and when I ran a "Rush" of 250 requests per second, the counters and the logs were completely flushed, except for the very last query.
I'm not perfectly sure what happened, but I suspect something along the lines of PHP not properly finishing up the previous query before taking on the next one.
I use file_get_contents()/file_put_contents() for the both of them, instead of file(). Would changing to file() solve the problem?
Here's the counter:
$filename = '.' . $_SERVER['PHP_SELF'];
$counterpath = '/Websites/inc/logs/counters/total/' . getCurrentFileName() . '-counter.txt';
$globalcounter = '/Websites/inc/logs/counters/total/global-counter.txt';
if (file_exists($counterpath)) {
$hit_count = file_get_contents($counterpath);
$hit_count++;
file_put_contents($counterpath,$hit_count);
}
else {
$hit_count = "1";
file_put_contents($counterpath, $hit_count);
}
And here's the logger:
$logdatefolder = '/Websites/inc/logs/ip/' . date('Y-m-d',$_SERVER['REQUEST_TIME_FLOAT']);
$logfile = $logdatefolder . "/" . getCurrentFileName() . '-iplog.html';
$ua = getbrowser();
if (false == (file_exists($logdatefolder))) {
mkdir($logdatefolder);
}
function checkRef() {
if (!isset($_SERVER['HTTP_REFERER'])) {
//If not isset -> set with dummy value
$_SERVER['HTTP_REFERER'] = 'N/A';
}
return $_SERVER['HTTP_REFERER'];
}
/* Main logger */
$logheader = "<!DOCTYPE html><html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en-US\"><head><title>" . getCurrentFileName() . " log</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" /></head><body>";
$logentry = date("Y-m-d, H:i:s, O T") . ":" .
"<br />- Requesting: http://giphtbase.org" . $_SERVER['REQUEST_URI'] .
"<br />- Arriving from: " . checkRef() .
"<br />- Browser: " . $ua['browser'] .
"<br />- Full browser name: " . $ua['name'] .
"<br />- Operating system: " . $ua['platform'] .
"<br />- Full user agent: " . $ua['userAgent'] .
"<br />";
$logfooter = "<!-- Bottom --></body></html>";
if (file_exists($logfile)) {
$logPage = file_get_contents($logfile);
$logContents = str_replace("<!-- Bottom --></body></html>","",$logPage);
file_put_contents($logfile, $logContents . $logentry . $logfooter);
}
elseif (false == (file_exists($logfile))) {
file_put_contents($logfile, $logheader . $logentry . $logfooter);
}
You should use the FILE_APPEND flag in your file_put_contents() otherwise you will only ever see the last entry:
file_put_contents($logfile, $logContents . $logentry . $logfooter, FILE_APPEND);
As for the counter, it looks like the file is trying to be written to too many times by different threads, causing it to be inaccessible. You should either use a database, or create a file_lock, or create temporary files and run a cronjob to do the math.
Related
I have a project where I have to make a new log file if the current log file is larger than 50mb. I know that we have to test for the size of the file.
I am starting out with this $log variable:
$log =
"Branch: ".$branchDir . PHP_EOL.
"Phase: ". $phaseDir . PHP_EOL.
"Total number of results files deleted: ". count($folderCounter). PHP_EOL.
"File names: " . $filePathToDelete . PHP_EOL.
"Starting CustomerID directory name: " . $CustIDvalue . PHP_EOL;
// $log = 'this is a test' . $branch . PHP_EOL;
$dt = time();
$mysql_datetime = strftime("%Y-%m-%d %H:%M:%S", $dt);
$mysql_datetimes = preg_replace('/\s+/', '', $mysql_datetime);
$logFileName = "ResultsFileDeletion" . $mysql_datetimes . ".txt";
if (unlink($filePathToDelete)) {
//this echo string will be moved to a log file
echo "first unlink: {$filePathToDelete} was deleted <br>";
$myFile =
file_put_contents("c:\\sites\\EtonBio/sequencing/logs/{$logFileName}",
$log, FILE_APPEND);
echo 'my file: ' . $myFile . '<br>';
}
I found this code snippet:
$size = filesize($_FILES['foto']['tmp_name']);
I believe my if statement would look something like this:
if($size < 52428800) {
*creates another log file
}
I want to format the $size variable correctly and I am also not sure how to make a new log file.
$size = filesize($_FILES['foto']['tmp_name']); only works if you are uploading the file.
If you want to check a file already in you server ou can simply do
$size = filesize("path/to/file");
Also, there is a ' in your $log declaration that shouldn't be there
My php is running out of memory with a server error "Out of memory:Kill process..about 25% of the way through the process" Although it searches through about 10,000 lines, the number of lines that match the criteria, and therefore need to be stored and written to the file at the end of the process, are less than 200. So I am not sure why it is running out of memory.
Am I receiving this error because I am not clearing variables after each loop, or do I need to increase the memory on the server?
The process in brief is:
- LOOPA - loop through list of 400 zip codes
- using one api call for each zip - get list of all places within each zip (typically about 40-50)
-- SUBLOOP1 - for each place found, use an api call to get all events for that place
---- SUBLOOP1A loop through events to count the number for each place
zips = file($configFile, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$dnis = file($dniFile, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$s3->registerStreamWrapper();
$file = fopen("s3://{$bucket}/{$key}", 'w') or die("Unable to open file!");
fwrite($file, $type . " id" . "\t" . $type . " name" . "\t" . "zip" . "\t" . "event count" . "\n" );
foreach($zips as $n => $zip){
//first line is the lable to describe zips, so skip it
if ($n < 1) continue;
$params = $url;
$params .= "&q=" . $zip;
$more_node_pages = true;
while ($more_node_pages){
$res = fetchEvents($params);
//Now find the number of events for each place
foreach($res->data as $node){
//first check if on Do Not Include list
$countevents = true;
foreach($dnis as $dni) {
if ($dni == $node->id) {
echo "Not going to get events for ". $node->name . " id# " . $dni . "\n\n";
$countevents = false;
break;
}
}
//if it found a match, skip this and go to the next
if (!$countevents) continue;
$params = $url . $node->id . "/events/?fields=start_time.order(reverse_chronological)&limit=" . $limit . "&access_token=". $access_token;
//Count the number of valid upcoming events for that node
$event_count = 0;
$more_pages = true;
$more_events = true;
while ($more_pages) {
$evResponse = fetchEvents($params);
if (!empty($evResponse->error)) {
checkError($evResponse->error->message, $evResponse->error->code, $file);
}
//if it finds any events for that place, go throught each event for that place one by one to count until you reach today
foreach($evResponse->data as $event){
if(strtotime($event->start_time) > strtotime('now')){
$event_count++;
}
//else we have reached today's events for this node, so get out of this loop, and don't retrieve any more events for this node
else {
$more_events = false;
break;
}
}
if (!empty($evResponse->paging->next) and $more_events) $params = $evResponse->paging->next;
else $more_pages = false;
} //end while loop looking for more pages with more events for that node (page)
if ($event_count > "0") {
fwrite($file, $node->id . "\t" . $node->name . "\t" . $zip . "\t" . $event_count . "\n");
echo $event_count . "\n";
}
} // loop back to the next place until done
//test to see if there is an additional page
if (!empty($res->paging->next)) $params = $res->paging->next; else $more_node_pages = false;
} //close while loop for $more_node_pages containing additional nodes for that zip
} // loop back to the next zip until done
fclose($file);
I would highly recommend adding output to the beginning of each nested loop. I think you most likely have an infinite loop, which is causing the script to run out of memory.
If that isn't the case, then you can try increasing the memory limit for your PHP script by adding this line of PHP to the top of your script:
ini_set("memory_limit", "5G");
If it takes more than 5GB of RAM for your script to process the 400 zip codes, I would recommend breaking your script up so that you can run zip codes 0-10 and then 11-20, then 21-30, etc.
Hope this helps, cheers.
You need to find out where the memory is being lost and then you can either take care of it or work around it. memory_get_usage() is your friend - print it at the top (or bottom) of each loop with some identifier so you can see when & where you are using up memory.
I want to synchronize files between two servers.
The idea is to check if a file exists on the "local" server and if true, check if it the same as a remote server. If not, the update it.
If it doesn't exist, add it.
The code seems to work but always one file is changed, even when it hasn't and its a different one each time.
Is there an error or what is the reason?
if (file_exists($img)) {
$image_hashfile_remote = sha1_file($image_url[$urlIndex]);
$image_hashfile_local = sha1_file($img);
if ($image_hashfile_local == $image_hashfile_remote) {
$num_same++;
} else {
file_put_contents($img, file_get_contents($image_url[$urlIndex]));
echo "Image file as changed since last synchronization. " .$img . " updated. <br />";
echo $image_hashfile_local . " " .$image_hashfile_remote . "<br />";
$num_saved++;
}
} else {
file_put_contents($img, file_get_contents($image_url[$urlIndex]));
echo "New image file found. " . $img . " added. <br />";
$num_saved++;
}
My Requirement is as follows:
When user uploads a file i should check for "File already Exists", if file exists i must show confirm box if 'OK' i have to replace and if cancel the reverse.
This is my following code
if (file_exists($path . $documentName)) {
$msg = $documentName . " already exists. ";
?>
<script type="text/javascript">
var res = confirm('File already exists Do you want to replace?');
if (res == false) {
<?php
$msg = 'File Upload cancelled';
?>
} else {
<?php
if (move_uploaded_file($_FILES["document"]["tmp_name"], $path . $documentName)) {
$msg = $documentName . " File Replaced Successfully";
$successURL = $document_path . $documentName;
}
else
$msg = $documentName . "Upload Failed";
?>
}
</script>";
<?
}
My problem is even if i give cancel the file is getting replaced.
just let me know where I'm wrong or Is there any other approach?
Please help me to close this issue
Note:jquery Not allowed.
Your problem is that you mix javascript and PHP. The PHP-Code will be run on the server and generates the HTML-document. At this point, the file gets replaced already.
Then, this document (with the javascript-code inside) will then be send to the user and there the javascript-code is run. And in that moment, the user gets to see the confirmaion-dialog, even though the file already was replaced!
Take a look at the source-code that your php-code is generating and you will see what I mean.
A solution would be to add a checkbox to confirm overwriting files. Then after hitting the upload-/submit-button, your php-script would check if this box was checked and either replace the file or not.
#Gogul, honestly, this is not the right way to go. Better that you handle the file submission with an AJAX request which receives a response back from your server (either uploaded successfully, or file exists) which you handle appropriately. If presenting the user an option to replace the file, again handle that action with AJAX.
You can do AJAX request in raw JavaScript (jQuery not required) - see here: http://www.w3schools.com/ajax/ajax_xmlhttprequest_send.asp
You are mixing server side code with client side javascript. The solving of your problem is more complicated if you don't want the user to reupload the document:
Store the file in a temporary location under random filename. Output a yes/no form to the user, including the random filename and original filename.
If the user answers yes, move from temporary location to $path, else remove the file from temporary location.
Guys i came with with this following solution
upload
uploaddocument.php
$documentName = preg_replace('/[^a-zA-Z0-9.]/s', '_', $_FILES["document"]["name"]);
if (file_exists($path . $documentName)) {
move_uploaded_file($_FILES["document"]["tmp_name"], "F:\\Content\\enews_files\\temp\\" . $documentName);
$msg = $documentName . " already exists. <a href='confirm.php?confirm=1&filename=" . $documentName . "&language=" . $lang . "'>Replace</a>||<a href='confirm.php?confirm=0&filename=" . $documentName . "'>Cancel</a>";
} else {
if (move_uploaded_file($_FILES["document"]["tmp_name"], $path . $documentName)) {
$msg = $documentName . " Upload Success";
$successURL = $document_path . $lang . '/' . $documentName;
}
else
$msg = $documentName . " Upload Failed";
}
confirm.php
include("config_enews.php");
$lang = $_GET['language'];
$path = "F:\\Content\\enews_files\\" . $lang . "\\";
//$path = "D:\\test\\test\\" . $lang . "\\";
$documentName = preg_replace('/[^a-zA-Z0-9.]/s', '_', $_GET["filename"]);
if ($_GET['confirm'] == 1) {
//echo sys_get_temp_dir();die;
if (copy("F:\\Content\\enews_files\\temp\\" . $_GET["filename"], $path . $documentName)) {
unlink("F:\\Content\\enews_files\\temp\\" . $_GET["filename"]);
header("Location: uploaddocument.php?message=success&fname=$documentName&lang=$lang");
} else {
echo $res = move_uploaded_file($_GET["tempname"], $path . $documentName);
echo $msg = $documentName . " Upload Failed";
header("Location: uploaddocument.php?message=failed&fname=$documentName");
}
} else {
unlink("F:\\Content\\enews_files\\temp\\" . $_GET["filename"]);
header("Location: uploaddocument.php?message=cancelled&fname=$documentName");
}
I got this spark from #Marek. If any one has better solution kindly provide.
I don't have enough reputations to vote your answers sorry.
Thank you so much for all your support.
This script is supposed to write log files using file locks etc to make sure that scripts running at the same time don't have any read/write complications. I got it off someone on php.net. When I tried to run it twice at the same time, I noticed that it completely ignored the lock file. However, when I ran them consecutively, the lock file worked just fine.
That doesn't make any sense whatsoever. The script just checks if a file exists, and acts based on that. Whether another script is running or not, shouldn't influence it at all. I double checked to make sure the lock file was created in both cases; it was.
So I started to do some testing.
First instance started at 11:21:00 outputs:
Started at: 2012-04-12 11:21:00
Checking if weblog/20120412test.txt.1.wlock exists
Got lock: weblog/20120412test.txt.1.wlock
log file not exists, make new
log file was either appended to or create anew
Wrote: 2012-04-12 11:21:00 xx.xx.xx.xxx "testmsg"
1
Second instance started at 11:21:03 outputs:
Started at: 2012-04-12 11:21:00
Checking if weblog/20120412test.txt.1.wlock exists
Got lock: weblog/20120412test.txt.1.wlock
log file not exists, make new
log file was either appended to or create anew
Wrote: 2012-04-12 11:21:00 xx.xx.xx.xxx "testmsg"
1
So there are two things wrong here. The timestamp, and the fact that the script sais the lock file doesn't exist even though it most certainly does.
It's almost as if the second instance of the script simply outputs what the first one did.
<?php
function Weblog_debug($input)
{
echo $input."<br/>";
}
function Weblog($directory, $logfile, $message)
{
// Created 15 september 2010: Mirco Babin
$curtime = time();
$startedat = date('Y-m-d',$curtime) . "\t" . date('H:i:s', $curtime) . "\t";
Weblog_debug("Started at: $startedat");
$logfile = date('Ymd',$curtime) . $logfile;
//Set directory correctly
if (!isset($directory) || $directory === false)
$directory = './';
if (substr($directory,-1) !== '/')
$directory = $directory . '/';
$count = 1;
while(1)
{
//*dir*/*file*.*count*
$logfilename = $directory . $logfile . '.' . $count;
//*dir*/*file*.*count*.lock
$lockfile = $logfilename . '.wlock';
$lockhandle = false;
Weblog_debug("Checking if $lockfile exists");
if (!file_exists($lockfile))
{
$lockhandle = #fopen($lockfile, 'xb'); //lock handle true if lock file opened
Weblog_debug("Got lock: $lockfile");
}
if ($lockhandle !== false) break; //break loop if we got lock
$count++;
if ($count > 100) return false;
}
//log file exists, append
if (file_exists($logfilename))
{
Weblog_debug("log file exists, append");
$created = false;
$loghandle = #fopen($logfilename, 'ab');
}
//log file not exists, make new
else
{
Weblog_debug("log file not exists, make new");
$loghandle = #fopen($logfilename, 'xb');
if ($loghandle !== false) //Did we make it?
{
$created = true;
$str = '#version: 1.0' . "\r\n" .
'#Fields: date time c-ip x-msg' . "\r\n";
fwrite($loghandle,$str);
}
}
//was log file either appended to or create anew?
if ($loghandle !== false)
{
Weblog_debug("log file was either appended to or create anew");
$str = date('Y-m-d',$curtime) . "\t" .
date('H:i:s', $curtime) . "\t" .
(isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : '-') . "\t" .
'"' . str_replace('"', '""', $message) . '"' . "\r\n";
fwrite($loghandle,$str);
Weblog_debug("Wrote: $str");
fclose($loghandle);
//Only chmod if new file
if ($created) chmod($logfilename,0644); // Read and write for owner, read for everybody else
$result = true;
}
else
{
Weblog_debug("log file was not appended to or create anew");
$result = false;
}
/**
Sleep & disable unlinking of lock file, both for testing purposes.
*/
//Sleep for 10sec to allow other instance(s) of script to run while this one still in progress.
sleep(10);
//fclose($lockhandle);
//#unlink($lockfile);
return $result;
}
echo Weblog("weblog", "test.txt", "testmsg");
?>
UPDATE:
Here's a simple script that just shows the timestamp. I tried it on a different host so I don't think it's a problem with my server;
<?php
function Weblog_debug($input)
{
echo $input."<br/>";
}
$curtime = time();
$startedat = date('Y-m-d',$curtime) . "\t" . date('H:i:s', $curtime) . "\t";
Weblog_debug("Started at: $startedat");
$timediff = time() - $curtime;
while($timediff < 5)
{
$timediff = time() - $curtime;
}
Weblog_debug("OK");
?>
Again, if I start the second instance of the script while the first is in the while loop, the second script will state it started at the same time as the first.
I can't fricking believe this myself, but it turns out this is just a "feature" in Opera. The script works as intended in Firefox. I kinda wish I tested that before I went all berserk on this but there ya go.