I have been working on this for days, I thought I had it but was wrong.
$done=0;
$filename = "raw_urls.txt";
if(! ($fhandle = fopen($filename, "r")))
{ echo "File failed to open";
Exit; }
while((fscanf($fhandle, "%s\n",$url_full))!== false)
{
print (mysql_error());
if(strlen($url_full) > 3)
{
$url_stat++;
$end_st = strlen($url_full)-29;
$s_url= substr($url_full,29,$end_st);
}
else{
$done++;
}
$res1=sql("SELECT * FROM `spy3` WHERE `Landingpage` LIKE '%$s_url%' LIMIT 0, 30 ",$o);
if($row=mysql_fetch_array($res1))
{
$lp=$row[6];
$found++;
}
else{
$nfound++;
}
sql("insert into sitemap (url, stat_url,nf, s_recno)
values (
'$url_full',
'$lp',
'$nfound',
'$url_stat'
)", $o);
print (mysql_error());
$found=0;
$nfound=0;
}
?>
I have tried fgets, changed txt files, it always stops between 128 and 132 lines of text. There are 2500 lines in the text file. Php.ini memory is very big. If I cut the txt file where it stops and save it, its 9k big.
have seen this link..?
Please Click here
OR
you can also use fseek($handle, 0); AND refer this link
Thanks.
Related
I have a 197gb text file that I want to read and push the contents into MySql database. I know, I can't put that big file in PHP buffer and read it as whole, So I want to read few hundred lines as a time and keep on reading next and next to read the whole file.
I am trying it with this but the page returns nothing
<?php
$i = 0;
$handle = fopen("./data/200gbfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (($line = fgets($handle)) !== false) {
echo $line . "<br />";
if ($i > 100) {
exit;
}
$i++;
}
fclose($handle);
} else {
echo "Error Opeing File!";
}
?>
Is there a limit of the max file size to be handled in php setting?
EDIT: for the 197gb file in question, fopen is failing to return anything and
the output page is just going blank.
You can read the file in chunks to save memory:
For example:
$fd = #fopen("./data/200gbfile.txt", "r");
while (!feof($fd)) {
$data = fread($fd, 1024); // read the file in 1024kb chunks
// handle current data (read line by line for example)
}
fclose($fd);
But no idea if that works with a file with 100Gbytes+.
Edit: # with fopen is required as suggested by Roman.
you can use ini_set('memory_limit','16M'); to set size accordingly but i don't wether it will handle such huge file. never tested that..
I am having some difficulty with reading info from a text file. Is it possible to use php and get one line at a time, and compare that line to a variable, one character at a time? Every time I add the character searching algorithm it messes up. or does the file reading only do full files/lines/character
ex:
$file=fopen("text/dialogue.txt","r") or exit("unable to open dialogue file");
if($file == true) {
echo "File is open";
fgets($file);
$c = "";
while(!feof($file)) {
$line = fgets($file)
while($temp = fgetc($line)) {
$c = $c . $temp;
//if statement and comparrison
}
}
} else {
echo "File not open";
}
fclose($file);
You may use php file function to read a file line by line
<?php
$lines = file("myfile.txt");
foreach($lines as $line){
## do whatever you like here
echo($line);
}
?>
Please check php manual
http://php.net/manual/en/function.file.php
I'm really new to php. I decided to make a counter based off a script I've seen. I've made changes to it. I'm trying to figure out how to reset the counter.
$userCount = file_get_contents("count.txt");
$userCount = trim($userCount);
$userCount = $userCount + 1;
$countReset = $userCount - $userCount;
$file = fopen("count.txt","w+");
fwrite($file,$userCount);
fclose($file);
print "The number of visitors is: $userCount";
if ($userCount < 20){
echo 'not yet';
}
else {
echo 'done!';
}
if ($userCount > 40){
fwrite($file,$countReset);
fclose($file);
}
I tried subtracting the counter from itself to get it back to 0.
$countReset = $userCount - $userCount;
However, it doesn't seem to work. the counter itself works, but I am unable to get it to reset back to 0.
This is just an impression like script I'm doing as a way to learn php. Also, sorry about the format, struggling with this post editor.
Any help with the script?
You've closed the file before trying to write to it again. Before your second fwrite, add:
$file = fopen("count.txt","w+");
Try to simply set value 0 to the var:
$countReset = 0;
Couldn't you just edit the count.txt file?
Doing it in PHP, you could do
fwrite($file,'0');
EDIT: Like CanSpice said, you shouldn't close the file before you're done with it. Remove the first fclose, and it should work.
I wouldn't mix fopen() and file_get_content() functions in this context, either use fopen(), fread() and fwrite() or use file_get_contents() and file_put_contents().
If you need just reset counter and you don't need previous value, than use:
file_put_contents('count.txt', '0');
If you need update value you may use either:
$count = file_get_contents( 'count.txt');
$count++;
// Reset?
if( $reset){
$count = 0;
}
file_put_contents( 'count.txt', "$count");
Or rather:
$fp = fopen( 'count.txt', 'r+') or die( 'Cannot use counter');
$count = trim( fread( $fp, 1024));
$count++;
// Reset?
if( $reset){
$count = 0;
}
ftruncate( $fp, 0);
fseek( 0, SEEK_SET)
fwrite( $fp, "$count");
fclose( $fp);
Here are the manual pages for ftruncate() and fseek() + you should probably study flock() so two scripts wouldn't overwrite the content at the same time.
/****************************************************************************
* read the current count from the counter file, increment
* it by 1, and re-save the new value off to the file
***************************************************************************/
function getAndIncrementCount(){
// attempt to open the file
// a+ means keep the contents so we can read it at least once,
// but allow us to overwrite the value once we increment it
if (($fHandle = fopen('count.txt','a+')) !== FALSE){
// read in the count (also cast to an int so if there is
// an invalid (or absent) value it will default to a 0
$count = (int) fread($fHandle, 100);
// increase the counter
$count++;
// go back to the beginning of the file
fseek($fHandle, 0);
// re-write the new count back to the file
fwrite($fHandle, $count);
// cose the file now that we're done with it
fclose($fHandle);
// return back the count
return $count;
}
// we couldn't get to the file, so return an error flag
return FALSE;
}
/****************************************************************************
* write the specified value to the counter file
***************************************************************************/
function setCount($count = 0){
// attempt to open the file with over-write permissions
// w+ will open the file and clear it
if (($fHandle = fopen('count.txt','w+')){
// write the counter to the file
fwrite($fHandle, $count);
// close the file now that we're done
fclose($fHandle);
// return the newly saved count
return $count;
}
// we couldn't get to the file, so return an error flag
return FALSE;
}
And applied in practice:
$userCount = getAndIncrementCount();
echo "The number of visitors is: {$userCount}";
if ($userCount < 20){
echo "Not Yet";
}else{
echo "Done!";
}
if ($userCount > 40){
setCount(0);
}
It's because you are not rewriting the file contents but adding to them when you use the fwrite second time, so $countReset gets appended to the content already in the file. Try this:
$userCount = file_get_contents("count.txt");
$userCount = $userCount + 1;
$countReset = $userCount - $userCount;
file_put_contents("count.txt", $userCount);
print "The number of visitors is: $userCount\n";
if ($userCount < 20){
echo 'not yet';
}
else {
echo 'done!';
}
if ($userCount > 40){
file_put_contents("count.txt", $countReset);
}
$done=0;
$filename = "raw_urls.txt";
if(! ($fhandle = fopen($filename, "r")))
{ echo "File failed to open";
Exit; }
//
// main loop reads sitemap url list
//
while($url_full_raw = fgets($fhandle,4096))
{
print (mysql_error());
$url_full= preg_replace('/[\x00-\x1F\x80-\xFF]/', '', $url_full_raw);
if(strlen($url_full) > 3)
{
$url_stat++;
// Echo ' tp1 Url from sitemap:',$url_stat,' - ' ,$url_full,'<br>';
$end_st = strlen($url_full)-29;
$s_url= substr($url_full,29,$end_st);
//Echo 'short:',$s_url,'<br>';
$url_full_raw= '';
}
else{
$done++;
Echo '----------- short string ---------------';
}
//
// Check for url
//
$res1=sql("SELECT * FROM `spy3` WHERE `Landingpage` LIKE '%$s_url%' LIMIT 0, 30 ",$o);
if($row=mysql_fetch_array($res1))
{
$lp=$row[6];
$found++;
// Echo '------->Url from sitemap:',$url_full,'<br>';
}
else{
Echo 'Not Found:-> ',$s_url,'<br>';
$nfound++;
}
sql("insert into sitemap (url, stat_url,nf, s_recno)
values (
'$url_full',
'$lp',
'$nfound',
'$url_stat'
)", $o);
print (mysql_error());
$found=0;
$nfound=0;
}
So the code works great. Except for one problem, after about 130 lines, it stops. It exits the program with no error. Yes full error reporting is on. PHP.ini memory is huge.
If I edit the txt file and take out some lines, no difference. I have been working on this for many hours.
Try doing it like they do in their example... with !== false. i.e,
while(($url_full_raw = fgets($fhandle,4096))!==false) {
I'm guessing your content is evaluating to false for whatever reason. That just happens to be at 130 lines (throw the 130 lines into a text file and see if the file size is close to 4 KB).
Also, you might want to fix your formatting for next time. Makes it very hard for us to read and help you.
again. First of all, bear with me, it's late at night, and I'm tired, so this is probably a simple answer...
I'm gathering info on products, one by one, then writing each product as a line in a CSV file. However, I don't want one huge CSV file, so I'm trying to split the files up by size. As each new product is gathered, the script should check whether or not the file it's about to write to is less than the limit. If it is, then it should go ahead and write to that file. If not, then a new file should be created and the product added to the new file.
Here's what I curently have trying to limit the size of the file:
$f = 1;
if(file_exists($shop_path.'items/items'.$f.'.txt'))
{
if(filesize($shop_path.'items/items'.$f.'.txt') >= 512000)
{
$f++;
$fh = fopen($shop_path.'items/items'.$f.'.txt', "a+");
}
else
{
$fh = fopen($shop_path.'items/items'.$f.'.txt', "a+");
}
}
else
{
$fh = fopen($shop_path.'items/items'.$f.'.txt', "a+");
}
if(fwrite($fh, $prod_str) === TRUE)
{
echo 'Done!<br />';
}
else
{
echo 'Could not write to file<br />';
}
fclose($fh);
However, I just keep getting the "Could not write to file" error. Why is this?
fwrite doesnt return a boolean value, only on failure (false).
"fwrite() returns the number of bytes written, or FALSE on error."
Check out: http://hu2.php.net/manual/en/function.fwrite.php
Maybe it works:
if(fwrite($fh, $prod_str))
{
echo 'Done!<br />';
}
else
{
echo 'Could not write to file<br />';
}
Edit:
Or even better:
if(fwrite($fh, $prod_str) !== false)
{
echo 'Done!<br />';
}
else
{
echo 'Could not write to file<br />';
}
The ftell() function will always tell you your current byte offset in a file. Test the value of this before each write, and if it's over your predefined value then you can close that file and open a new one.