Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am using Windows and working with CSV/XLS data. I have a data that need to edit with PHP, but I'm not sure how.
Here is sample the data:
code1,data1
D0001,X1235466
D0002,X1232265
D0003,X1235166
D0004,X5235466
D0005,X6232265
D0006,X7235166
.. and so on--about 20,000 lines.
I am trying to make the file look like this:
code1,data1,code2,data2,code3,data3
D0001,X1235466,D0002,X1232265,D0003,X1235166
D0004,X5235466,D0005,X6232265,D0006,X7235166
and so on, then save it to new CSV file. Thanks.
Alright, I was bored, so I wrote this. You really need to attempt writing the code before posting the question, though, and show your work.
Here goes:
<?php
$desiredColumns = 3;
$inputFd = fopen('data.csv', 'r');
$outputFd = fopen('output.csv', 'w');
$buffer = [];
// Write the CSV header.
fputcsv($outputFd, [
'code1', 'data1', 'code2', 'data2','code3', 'data3'
]);
//Skip header line.
fgetcsv($inputFd);
while (!feof($inputFd)) {
$line = fgetcsv($inputFd);
if (empty($line) || empty($line[0])) {
continue;
}
$line = array_map('trim', $line);
$buffer[] = $line;
// If our buffer has $desiredColumns in it, flush the buffer
// to our output file.
if (count($buffer) == $desiredColumns) {
outputCsv($outputFd, $buffer);
$buffer = [];
}
}
// If we had a total number of entries that is not a multiple
// of 3, we need to output the remaining records.
if (!empty($buffer)) {
outputCsv($outputFd, $buffer);
}
// Function for flushing a record buffer to the output file.
function outputCsv($outputFd, $buffer)
{
$outputData = [];
foreach ($buffer as $record) {
$outputData[] = $record[0];
$outputData[] = $record[1];
}
fputcsv($outputFd, $outputData);
}
Now, as an example of this being run:
$ cat data.csv
code1,data1
D0001,X1235466
D0002,X1232265
D0003,X1235166
D0004,X5235466
D0005,X6232265
D0006,X7235166
$ php so.php
$ cat output.csv
code1,data1,code2,data2,code3,data3
D0001,X1235466,D0002,X1232265,D0003,X1235166
D0004,X5235466,D0005,X6232265,D0006,X7235166
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
I'm writing a php function which must open and read line by line the content of a txt file saved in the same directory of the php file and checks if the first value of the line is equal to a number which I give it as a parameter of the function and if this number isn't in the file I will create a new line with the number followed by another information separated by a space. This funcion is in a webhook file of a Facebook Messenger bot and it doesn't work if the function isn't commented out and I don't know why because I checked the code several times and I haven't found the error.
The code of the function is:
function messagesNumber($userid)
{
$fo = ("url of the files", "a+");
$found = false;
if($fo) {
$line = fgets($fo);
while(! feof($fo) && $found == false) {
$words = explode(" ", $line, 2);
if($words[0] == $userid) {
$found = true;
$words[1] += 1;
$messagges = $words [1];
str_replace($line, $words, $fo);
}
$line = fgets($fo);
}
if(! $found) {
fprintf($fo, "%d 1", $userid);
$messages = 1;
}
}
else
$messagges = -1;
fclose($fo);
return $messagges;
}
It returns: -1 if the file can't be open or an integer that coincides with the second number of every line.
It is an example of the txt file:
282318736127 5
827813487273 3
...
The first number is the id which Facebook assign to every user and the second number is a counter who counts how many messages a user writes to the bot and I increase this last number every times that a specific user sends a message to the bot.
easy ... your missing a $
if(! $found) {
fprintf($fo, "%d 1", $userid);
messages = 1; // missing $
}
to
if(! $found) {
fprintf($fo, "%d 1", $userid);
$messages = 1;
}
okay edit ...
easy your missing an fopen
$fo = ("url of the files", "a+");
to
$fo = fopen("url of the files", "a+");
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have a ip.txt file which contains
127.0.0.1
I need to get the text file from a form and to display the ping details in the new window using php. how to read ip from a text file so as to ping it from the read information.
Try this code
$ips = array();
$file = #fopen("ip.txt", "r");
if ($file) {
while (($buffer = fgets($file, 4096)) !== false) {
$ips[] = $buffer;
}
if (!feof($file)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($file);
echo '<pre>'; print_r($ips); echo '</pre>';
}
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
I have a PHP function that reads the contents of a file given its path and removes first characters until it finds a '#' (first correct character in my file).This function works fine but how do I reduce the execution time ?
Please suggest/advice.
function foo($filepath)
{
if(($contents = file_get_contents($filepath)) !== false)
{
while ($contents[0] != '#')
$contents = substr($contents, 1);
file_put_contents($filepath, $contents);
}
return false;
}
This can be optimised in two ways: speed and memory management. You reading the entire file into memory is quite expensive and may fail entirely on large files. Instead, this'll be more memory efficient, but requires a second temporary file:
$fh = fopen($filepath, 'r');
do {
$chr = fread($fh, 1);
} while ($chr != '#' && !feof($fh));
fseek($fh, -1, SEEK_CUR);
$temppath = tempnam(sys_get_temp_dir(), 'substr');
$tempfh = fopen($temppath, 'w');
stream_copy_to_stream($fh, $tempfh);
fclose($fh);
fclose($tempfh);
rename($temppath, $filepath);
Speed-wise your existing solution can be simplified to:
if (($contents = file_get_contents($filepath)) !== false) {
$index = strpos($contents, '#');
file_put_contents($filepath, substr($contents, $index));
}
But again, it's reading everything into memory, which may be an important bottleneck to begin with.
function foo($filepath)
{
if(($contents = file_get_contents($filepath)) !== false)
{
$contentExplode = explode("#",$contents );
array_shift($contentExplode);//remove chars to the first #
$contents = implode("#",$contentExplode);
file_put_contents($filepath, $contents);
}
return false;
}
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 8 years ago.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Improve this question
someone could please tell why this code does not work? it does not create data.txt. and the file does not save anything.
<?php
$txt = "data.txt";
if(isset($_POST['info1']) && isset($_POST['info2']) && isset($_POST['info3']) && isset($_POST['info4']) && isset($_POST['info5']) && isset($_POST['info6'])) {
// check if both fields are set
//open file in read mode to get number of lines
$handle = fopen($txt, 'r');
//check file opened
if($handle) {
//get number of lines
$count = 1;
while(fgets($handle) !== false) {
$count++;
}
fclose($handle);
//open in append mode.
$handle = fopen($txt, 'a');
//prepare data to be writen
$txt = $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n";
//write data to file
fwrite($handle,$txt);
fclose($handle);
}
}
?>
In fopen function mode param with value r doesn't make file being created.
See fopen documentation.
Also the issue could be write permissions for user that runs that PHP script.
These is only my guesses. Anyway you should configure your PHP (in dev environment) to show you every error/warning in order to debug such issues. You can do that in php.ini file or straight in PHP file with this code:
error_reporting(E_ALL);
ini_set('display_errors', 1);
You are first trying to open the file read only, and if that succeeds you close it and open it again with write access.
This will only work if the file already exists.
If the file doesn't exist, your text will not get written to it.
Move the closing bracket if if ($handle) { in front of the second fopen call.
if($handle) {
//get number of lines
$count = 1;
while(fgets($handle) !== false) {
$count++;
}
fclose($handle);
}
//open in append mode.
$handle = fopen($txt, 'a');
//prepare data to be writen
$txt = $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n";
//write data to file
fwrite($handle,$txt);
fclose($handle);
by the way, you can reduce this to three lines of code.
$lines = file($txt);
$count = sizeof($lines);
file_put_contents($txt, $count . ' ' . $_POST['info1'].'/'.$_POST['info2'].'/'.$_POST['info3'].'/'.$_POST['info4'].'/'.$_POST['info5'].'/'.$_POST['info6']. "\r\n", FILE_APPEND);
to check for multiple variables in isset create only one isset to check all veriables
change this
if(isset($_POST['info1']) && isset($_POST['info2']) && isset($_POST['info3']) && isset($_POST['info4']) && isset($_POST['info5']) && isset($_POST['info6']))
to
if(isset($_POST['info1'],$_POST['info2'],$_POST['info3'],$_POST['info4'],$_POST['info5'],$_POST['info6']))
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have a csv file, containing millions of email addresses which I want to upload fast into a mysql database with PHP.
Right now I'm using a single threaded program which takes too much time to upload.
//get the csv file
$file = $_FILES['csv']['tmp_name'];
$handle = fopen($file,"r");
//loop through the csv file and insert into database
do {
if ($data[0]) {
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
if (preg_match($expression, $data[0])) {
$query=mysql_query("SELECT * FROM `postfix`.`recipient_access` where recipient='".$data[0]."'");
mysql_query("SET NAMES utf8");
$fetch=mysql_fetch_array($query);
if($fetch['recipient']!=$data[0]){
$query=mysql_query("INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES('".addslashes($data[0])."','".$_POST['note']."')");
}
}
}
} while ($data = fgetcsv($handle,1000,",","'"));
First of all, I can't stress enough; fix your indentation - it will make life easier for everyone.
Secondly, the answer depends a lot on the actual bottlenecks you are encountering:
Regular expressions are very slow, especially when they're in a loop.
Databases tend to either work well for WRITES or for READS but not BOTH: try decreasing the amount of queries beforehand.
It stands to reason that the less PHP code in your loop, the faster it will work. Consider decreasing conditions (for instance).
For the record, your code is not safe against mysql injection: filter $_POST before hand [*]
[*] speaking of which, it's faster to access a variable than the index of an array, like $_POST.
You can simulate multithreading by having your main program split the huge CSV file into a smaller one and run each file into a different process.
common.php
class FileLineFinder {
protected $handle, $length, $curpos;
public function __construct($file){
$handle = fopen($file, 'r');
$length = strlen(PHP_EOL);
}
public function next_line(){
while(!feof($this->handle)){
$b = fread($this->handle, $this->length);
$this->curpos += $this->length;
if ($b == PHP_EOL) return $this->curpos;
}
return false;
}
public function skip_lines($count){
for($i = 0; $i < $count; $i++)
$this->next_line();
}
public function __destruct(){
fclose($this->handle);
}
}
function exec_async($cmd, $outfile, $pidfile){
exec(sprintf("%s > %s 2>&1 & echo $! >> %s", $cmd, $outfile, $pidfile));
}
main.php
require('common.php');
$maxlines = 200; // maximum lines subtask will be processing at a time
$note = $_POST['note'];
$file = $_FILES['csv']['tmp_name'];
$outdir = dirname(__FILE__) . DIRECTORY_SEPARATOR . 'out' . DIRECTORY_SEPARATOR;
//make sure our output directory exists
if(!is_dir($outdir))
if(!mkdir($outdir, 0755, true))
die('Cannot create output directory: '.$outdir);
// run a task for each chunk of lines in the csv file
$i = 0; $pos = 0;
$l = new FileLineFinder($file);
do {
$i++;
exec_async(
'php -f sub.php -- '.$pos.' '.$maxlines.' '.escapeshellarg($file).' '.escapeshellarg($note),
$outdir.'proc'.$i.'.log',
$outdir.'proc'.$i.'.pid'
);
$l->skip_lines($maxlines);
} while($pos = $l->next_line());
// wait for each task to finish
do {
$tasks = count(glob($outdir.'proc*.pid'));
echo 'Remaining Tasks: '.$tasks.PHP_EOL;
} while ($tasks > 0);
echo 'Finished!'.PHP_EOL;
sub.php
require('common.php');
$start = (int)$argv[1];
$count = (int)$argv[2];
$file = $argv[3];
$note = mysql_real_escape_string($argv[4]);
$lines = 0;
$handle = fopen($file, 'r');
fseek($handle, $start, SEEK_SET);
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
mysql_query('SET NAMES utf8');
//loop through the csv file and insert into database
do {
$lines++;
if ($data[0]) {
if (preg_match($expression, $data[0])) {
$query = mysql_query('SELECT * FROM `postfix`.`recipient_access` where recipient="'.$data[0].'"');
$fetch = mysql_fetch_array($query);
if($fetch['recipient'] != $data[0]){
$query = mysql_query('INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES("'.$data[0].'","'.$note.'")');
}
}
}
} while (($data = fgetcsv($handle, 1000, ',', '\'')) && ($lines < $count));
Credits
https://stackoverflow.com/a/2162528/314056
https://stackoverflow.com/a/45966/314056
The most pressing thing to do is to make sure your database is properly indexed so the lookup query you do for every row is as fast as possible.
Other than that, there simply isn't that much you can do. For a multithreaded solution, you'll have to go outside PHP.
You could also just import the CSV file in mySQL, and then weed out the superfluous data using your PHP script - that is likely to be the fastest way.
Just a general suggestion: The key to speed up any program is to know which part take most of the time.
And then figure out how to reduce it. Sometimes you will be very surprised by the actual result.
btw, I don't think multithreading would solve your your problem.
Put the whole loop inside an SQL transaction. That will speed things up by an order of magnitude.