Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
I have a PHP function that reads the contents of a file given its path and removes first characters until it finds a '#' (first correct character in my file).This function works fine but how do I reduce the execution time ?
Please suggest/advice.
function foo($filepath)
{
if(($contents = file_get_contents($filepath)) !== false)
{
while ($contents[0] != '#')
$contents = substr($contents, 1);
file_put_contents($filepath, $contents);
}
return false;
}
This can be optimised in two ways: speed and memory management. You reading the entire file into memory is quite expensive and may fail entirely on large files. Instead, this'll be more memory efficient, but requires a second temporary file:
$fh = fopen($filepath, 'r');
do {
$chr = fread($fh, 1);
} while ($chr != '#' && !feof($fh));
fseek($fh, -1, SEEK_CUR);
$temppath = tempnam(sys_get_temp_dir(), 'substr');
$tempfh = fopen($temppath, 'w');
stream_copy_to_stream($fh, $tempfh);
fclose($fh);
fclose($tempfh);
rename($temppath, $filepath);
Speed-wise your existing solution can be simplified to:
if (($contents = file_get_contents($filepath)) !== false) {
$index = strpos($contents, '#');
file_put_contents($filepath, substr($contents, $index));
}
But again, it's reading everything into memory, which may be an important bottleneck to begin with.
function foo($filepath)
{
if(($contents = file_get_contents($filepath)) !== false)
{
$contentExplode = explode("#",$contents );
array_shift($contentExplode);//remove chars to the first #
$contents = implode("#",$contentExplode);
file_put_contents($filepath, $contents);
}
return false;
}
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
i have a file (in my case debug.log) and there is a lot of source code from many files in it. I want to extract these lines of code in seperate files.
Structure of my debug.log:
#NewFile#path/to/file.php
<?php
class ClassA {
function A() { do smth(); }
}
#NewFile#path/to/nextFile.php
<?php
class ClassA {
function A() { do smth(); }
}
#NewFile#path/to/thirdFile.php
...
Now i want to split by #NewFile# and want to save the Content in a new .php File.
This is my code for doing this:
$handle = fopen('debug.log', 'r');
$index = 1;
$filename = '/home/myuser/folder/file';
while (($line = fgets($handle)) !== false) {
if (strpos($line, '#NewFile#') !== false) {
$content = file_get_contents($filename . $index . '.php');
file_put_contents($filename . $index . '.php', $content . $line);
} else {
$index++;
}
}
fclose($handle);
Thanks for your help :)
Apart from the fact that a file called debug.log seems to contain PHP source (which, no matter how you look at it, is really weird), it's a fairly trivial thing to do:
The simplest way to reliably parse php files in php is to use the token_get_all function. In this case, it's a matter of doing something like this:
$tokens = token_get_all(file_get_contents('input_file.php'));
$file = null;
$contents = [];
foreach ($tokens as $token) {
//comment with #NewFile# in there?
if ($token[0] === T_COMMENT && strstr($token[1]{0}, '#NewFile#')) {
if ($file) {
//write code to file
file_put_contents($file, implode(PHP_EOL, $contents));
}
$contents = ['<?php '];
$file = str_replace('#NewFile#', '', $token[1]);//set file path
} else {
//use line numbers as key, append value of current token to the line
$contents[$token[2]] .= $token[1];
}
}
//write the last file
if ($file) {
file_put_contents($file, implode(PHP_EOL, $contents));
}
I'm iterating over all the parser tokens. If I encounter a T_COMMENT token containing the string #NewFile#, I take that as sign that I need to write my current buffer ($contents into the file that I last read from the previous comment. After that, I reassign $file to point to a new file (again, path and name taken from the comment), and start building the $contents buffer again.
After the loop, $file and $contents will contain all the tokens that should go in the last file, so I just do a quick check (make sure $file is set), and write whatever is in the buffer to that file.
Here is my own solution for my Problem, that solved it :)
$handle = fopen(dirname(__FILE__) . '/debug.log', 'r');
$fileName = '/file';
$dir = '/home/myuser/folder';
while (($line = fgets($handle)) !== false) {
if (strpos($line, '#NewFile#') === false) {
if (file_exists($dir . $fileName)) {
file_put_contents($dir . $fileName, $line, FILE_APPEND);
} else {
preg_match("/(\/.*\/)/", $fileName, $path);
if (!is_dir($dir . $path[0])) {
mkdir($dir . $path[0], 0777, true);
}
file_put_contents($dir . $fileName, $line);
}
} else {
$fileName = str_replace(".#NewFile#", '', $line);
$fileName = str_replace("#NewFile#", '', $fileName);
}
}
fclose($handle);
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am using Windows and working with CSV/XLS data. I have a data that need to edit with PHP, but I'm not sure how.
Here is sample the data:
code1,data1
D0001,X1235466
D0002,X1232265
D0003,X1235166
D0004,X5235466
D0005,X6232265
D0006,X7235166
.. and so on--about 20,000 lines.
I am trying to make the file look like this:
code1,data1,code2,data2,code3,data3
D0001,X1235466,D0002,X1232265,D0003,X1235166
D0004,X5235466,D0005,X6232265,D0006,X7235166
and so on, then save it to new CSV file. Thanks.
Alright, I was bored, so I wrote this. You really need to attempt writing the code before posting the question, though, and show your work.
Here goes:
<?php
$desiredColumns = 3;
$inputFd = fopen('data.csv', 'r');
$outputFd = fopen('output.csv', 'w');
$buffer = [];
// Write the CSV header.
fputcsv($outputFd, [
'code1', 'data1', 'code2', 'data2','code3', 'data3'
]);
//Skip header line.
fgetcsv($inputFd);
while (!feof($inputFd)) {
$line = fgetcsv($inputFd);
if (empty($line) || empty($line[0])) {
continue;
}
$line = array_map('trim', $line);
$buffer[] = $line;
// If our buffer has $desiredColumns in it, flush the buffer
// to our output file.
if (count($buffer) == $desiredColumns) {
outputCsv($outputFd, $buffer);
$buffer = [];
}
}
// If we had a total number of entries that is not a multiple
// of 3, we need to output the remaining records.
if (!empty($buffer)) {
outputCsv($outputFd, $buffer);
}
// Function for flushing a record buffer to the output file.
function outputCsv($outputFd, $buffer)
{
$outputData = [];
foreach ($buffer as $record) {
$outputData[] = $record[0];
$outputData[] = $record[1];
}
fputcsv($outputFd, $outputData);
}
Now, as an example of this being run:
$ cat data.csv
code1,data1
D0001,X1235466
D0002,X1232265
D0003,X1235166
D0004,X5235466
D0005,X6232265
D0006,X7235166
$ php so.php
$ cat output.csv
code1,data1,code2,data2,code3,data3
D0001,X1235466,D0002,X1232265,D0003,X1235166
D0004,X5235466,D0005,X6232265,D0006,X7235166
This question already has an answer here:
Why preg_replace throws me a "Unknown modifier" error? [duplicate]
(1 answer)
Closed 7 years ago.
I was trying to make a "search replace between for a huge xml file (1GB).
I found this great code that is work perfectly while using str_replace on my file-
<?php
function replace_file($path, $string, $replace)
{
set_time_limit(0);
if (is_file($path) === true)
{
$file = fopen($path, 'r');
$temp = tempnam('./', 'tmp');
if (is_resource($file) === true)
{
while (feof($file) === false)
{
file_put_contents($temp, str_replace($string, $replace, fgets($file)), FILE_APPEND);
}
fclose($file);
}
unlink($path);
}
return rename($temp, $path);
}
replace_file('myfile.xml', '<search>', '<replace>');
so far so good and it works great.
Now I changed the str_replace to preg_replace and the search value to '/^[^]/' so the code looks like this-
<?php
function replace_file($path, $string, $replace)
{
set_time_limit(0);
if (is_file($path) === true)
{
$file = fopen($path, 'r');
$temp = tempnam('./', 'tmp');
if (is_resource($file) === true)
{
while (feof($file) === false)
{
file_put_contents($temp, preg_replace($string, $replace, fgets($file)), FILE_APPEND);
}
fclose($file);
}
unlink($path);
}
return rename($temp, $path);
}
replace_file('myfile.xml', '/[^<search>](.*)[^</search>]/', '<replace>');
I get an error "preg_replace unknown modifier" 'd' on line 16
line 16 is -
file_put_contents($temp, preg_replace($string, $replace, fgets($file)), FILE_APPEND);
[] in PCRE is a character class. With [^<category>] you're actually matching the same as with [^<>acegorty]. You're matching characters (or bytes), not words.
PCRE is not best solution for this anyway. Use XMLReader and XMLWriter.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 8 years ago.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Improve this question
someone could please tell why this code does not work? it does not create data.txt. and the file does not save anything.
<?php
$txt = "data.txt";
if(isset($_POST['info1']) && isset($_POST['info2']) && isset($_POST['info3']) && isset($_POST['info4']) && isset($_POST['info5']) && isset($_POST['info6'])) {
// check if both fields are set
//open file in read mode to get number of lines
$handle = fopen($txt, 'r');
//check file opened
if($handle) {
//get number of lines
$count = 1;
while(fgets($handle) !== false) {
$count++;
}
fclose($handle);
//open in append mode.
$handle = fopen($txt, 'a');
//prepare data to be writen
$txt = $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n";
//write data to file
fwrite($handle,$txt);
fclose($handle);
}
}
?>
In fopen function mode param with value r doesn't make file being created.
See fopen documentation.
Also the issue could be write permissions for user that runs that PHP script.
These is only my guesses. Anyway you should configure your PHP (in dev environment) to show you every error/warning in order to debug such issues. You can do that in php.ini file or straight in PHP file with this code:
error_reporting(E_ALL);
ini_set('display_errors', 1);
You are first trying to open the file read only, and if that succeeds you close it and open it again with write access.
This will only work if the file already exists.
If the file doesn't exist, your text will not get written to it.
Move the closing bracket if if ($handle) { in front of the second fopen call.
if($handle) {
//get number of lines
$count = 1;
while(fgets($handle) !== false) {
$count++;
}
fclose($handle);
}
//open in append mode.
$handle = fopen($txt, 'a');
//prepare data to be writen
$txt = $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n";
//write data to file
fwrite($handle,$txt);
fclose($handle);
by the way, you can reduce this to three lines of code.
$lines = file($txt);
$count = sizeof($lines);
file_put_contents($txt, $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n", FILE_APPEND);
to check for multiple variables in isset create only one isset to check all veriables
change this
if(isset($_POST['info1']) && isset($_POST['info2']) && isset($_POST['info3']) && isset($_POST['info4']) && isset($_POST['info5']) && isset($_POST['info6']))
to
if(isset($_POST['info1'],$_POST['info2'],$_POST['info3'],$_POST['info4'],$_POST['info5'],$_POST['info6']))
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have a csv file, containing millions of email addresses which I want to upload fast into a mysql database with PHP.
Right now I'm using a single threaded program which takes too much time to upload.
//get the csv file
$file = $_FILES['csv']['tmp_name'];
$handle = fopen($file,"r");
//loop through the csv file and insert into database
do {
if ($data[0]) {
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
if (preg_match($expression, $data[0])) {
$query=mysql_query("SELECT * FROM `postfix`.`recipient_access` where recipient='".$data[0]."'");
mysql_query("SET NAMES utf8");
$fetch=mysql_fetch_array($query);
if($fetch['recipient']!=$data[0]){
$query=mysql_query("INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES('".addslashes($data[0])."','".$_POST['note']."')");
}
}
}
} while ($data = fgetcsv($handle,1000,",","'"));
First of all, I can't stress enough; fix your indentation - it will make life easier for everyone.
Secondly, the answer depends a lot on the actual bottlenecks you are encountering:
Regular expressions are very slow, especially when they're in a loop.
Databases tend to either work well for WRITES or for READS but not BOTH: try decreasing the amount of queries beforehand.
It stands to reason that the less PHP code in your loop, the faster it will work. Consider decreasing conditions (for instance).
For the record, your code is not safe against mysql injection: filter $_POST before hand [*]
[*] speaking of which, it's faster to access a variable than the index of an array, like $_POST.
You can simulate multithreading by having your main program split the huge CSV file into a smaller one and run each file into a different process.
common.php
class FileLineFinder {
protected $handle, $length, $curpos;
public function __construct($file){
$handle = fopen($file, 'r');
$length = strlen(PHP_EOL);
}
public function next_line(){
while(!feof($this->handle)){
$b = fread($this->handle, $this->length);
$this->curpos += $this->length;
if ($b == PHP_EOL) return $this->curpos;
}
return false;
}
public function skip_lines($count){
for($i = 0; $i < $count; $i++)
$this->next_line();
}
public function __destruct(){
fclose($this->handle);
}
}
function exec_async($cmd, $outfile, $pidfile){
exec(sprintf("%s > %s 2>&1 & echo $! >> %s", $cmd, $outfile, $pidfile));
}
main.php
require('common.php');
$maxlines = 200; // maximum lines subtask will be processing at a time
$note = $_POST['note'];
$file = $_FILES['csv']['tmp_name'];
$outdir = dirname(__FILE__) . DIRECTORY_SEPARATOR . 'out' . DIRECTORY_SEPARATOR;
//make sure our output directory exists
if(!is_dir($outdir))
if(!mkdir($outdir, 0755, true))
die('Cannot create output directory: '.$outdir);
// run a task for each chunk of lines in the csv file
$i = 0; $pos = 0;
$l = new FileLineFinder($file);
do {
$i++;
exec_async(
'php -f sub.php -- '.$pos.' '.$maxlines.' '.escapeshellarg($file).' '.escapeshellarg($note),
$outdir.'proc'.$i.'.log',
$outdir.'proc'.$i.'.pid'
);
$l->skip_lines($maxlines);
} while($pos = $l->next_line());
// wait for each task to finish
do {
$tasks = count(glob($outdir.'proc*.pid'));
echo 'Remaining Tasks: '.$tasks.PHP_EOL;
} while ($tasks > 0);
echo 'Finished!'.PHP_EOL;
sub.php
require('common.php');
$start = (int)$argv[1];
$count = (int)$argv[2];
$file = $argv[3];
$note = mysql_real_escape_string($argv[4]);
$lines = 0;
$handle = fopen($file, 'r');
fseek($handle, $start, SEEK_SET);
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
mysql_query('SET NAMES utf8');
//loop through the csv file and insert into database
do {
$lines++;
if ($data[0]) {
if (preg_match($expression, $data[0])) {
$query = mysql_query('SELECT * FROM `postfix`.`recipient_access` where recipient="'.$data[0].'"');
$fetch = mysql_fetch_array($query);
if($fetch['recipient'] != $data[0]){
$query = mysql_query('INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES("'.$data[0].'","'.$note.'")');
}
}
}
} while (($data = fgetcsv($handle, 1000, ',', '\'')) && ($lines < $count));
Credits
https://stackoverflow.com/a/2162528/314056
https://stackoverflow.com/a/45966/314056
The most pressing thing to do is to make sure your database is properly indexed so the lookup query you do for every row is as fast as possible.
Other than that, there simply isn't that much you can do. For a multithreaded solution, you'll have to go outside PHP.
You could also just import the CSV file in mySQL, and then weed out the superfluous data using your PHP script - that is likely to be the fastest way.
Just a general suggestion: The key to speed up any program is to know which part take most of the time.
And then figure out how to reduce it. Sometimes you will be very surprised by the actual result.
btw, I don't think multithreading would solve your your problem.
Put the whole loop inside an SQL transaction. That will speed things up by an order of magnitude.