Fastest way to reduce a csv file to two columns - php

I have a csv file that has like 30,000 rows in it. It also has like 9 columns. In the interest of speeding up the processing of everything I want to reduce the file to the two columns that I need and remove the rest. here is what I have done.
$retardment=1;//17;// 151; //499;// 991;// 1877
if (($handle = fopen($source, "r")) !== FALSE) {
$stock_handle = fopen($source_stock, "w+");
$row=0;
$col=array();
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
unset($line[1]);
unset($line[2]);
unset($line[3]);
unset($line[4]);
unset($line[5]);
unset($line[6]);
unset($line[8]);
unset($line[9]);
if($row%$retardment<1){
fputcsv($stock_handle, $line);
}
unset($line);
$row++;
}
fclose($handle);
fclose($stock_handle);
}
I am coping it to a new file and this works... but it seems to be pretty slow. Any ideas on how to make it faster? Thank you for the help.
Cheers -Jeremy
{EDIT}
So far this seems to take just as long. But works just fine
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row%$retardment<1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
$row++;
}

You could replace those unset() calls with...
$line = array($line[0], $line[7]);
Alternatively, remember that unset() takes multiple arguments...
unset($line[1], $line[2], ...);

You can speed it up fractionally more, but again it's a microtime()-measurable improvement: perception is that it won't be noticeably faster.
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row++ % $retardment < 1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
}
but as your script is IO-bound, it's the actual reads and writes that are the slowest functions, and you can't speed those up.
Using stream_copy_to_stream() with a stream input filter might be another approach, but you won't see much noticeable improvement unless you can reduce disk access times

Related

PHP Cron Read CSV from URL and Convert to Array

I am trying to develop email system that need to be send every month. So i need to build cron job file from php. Anyone know how to read file CSV or Excel file from url such as:
http://yourdomain.com/cron.php?file=http://google.com/monthly.csv
I am stuck when try to read file from url.
This is my recent code:
<?php
$url = 'http://www1.intranet.com/reportingtool.asp?settings=var&export=ok';
$tmpfname = tempnam(sys_get_temp_dir());
file_put_contents(
$tmpfname,
file_get_contents($url)
);
?>
If you're dealing with remote files, you should always keep in mind that
The connection between you and the remote can break, and you won't get the full file content;
The file can be too big to read it on-the-fly.
In both cases, file_get_contents() is not a very good thing to use: you should consider cURL functions for that. However, if the concerns above are negligible, you should be okay with the following (as the example here suggests):
$url = 'http://www1.intranet.com/reportingtool.asp?settings=var&export=ok';
$tmpfname = tempnam(sys_get_temp_dir());
file_put_contents(
$tmpfname,
file_get_contents($url)
);
if (($handle = fopen($tmpfname, "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
// Do something with $data, which comes in
// as an array of values from the current CSV line.
}
}
Someone would need one more example, so leaving it here:
$dataSource can be any file on drive or just online link to parse csv.
private array $data = [];
private string $dataSource = "https://any.csv";
if (($open = fopen($dataSource, 'rb')) !== false)
{
while (($data = fgetcsv($open, 1000, ";")) !== false)
{
$data[] = $data;
}
fclose($open);
}

Run PHP function in While Loop

I'd like to run a function within a While loop.
Unfortunately, it works but not quite as desired.
My goal is to read out from a CSV file, each line and is stored in the MySQL database. This also works flawlessly.
In addition, however, I want to let miteinpflegen a value by a numeric random.
Here again my scripts to generate the numerical codes.
function generateCode($length){
$string = "";
$numbers = "0123456789";
for($i=0;$i < $length;$i++) {
$char = $numbers[mt_rand(0, strlen($numbers)-1)];
$string .= $char;
}
return $string;
}
$gencode = generateCode(8);
Works great.
And here my while loop ($handle is in my case the var to open CSV file)
while (($data = fgetcsv($handle, 10000, ";")) !== FALSE) {
$import = "INSERT into jobs(id,link_id,random number) values('NULL','$linkid','$gencode')";
mysql_query($import) or die(mysql_error());
}
But how do I get it back now that each line of the CSV file gets its own random number or rather, which is calculated in each row $gencode new?
At the moment i get one random number for all rows.
For help, I am very grateful.
I just can no longer see the forest for the trees.
Thank you
You can simply call your own function generateCode() like any other function inside your loop:
while (($data = fgetcsv($handle, 10000, ";")) !== FALSE) {
$gencode = generateCode(8);
// the rest of your code
}
Just make sure that generateCode() is already defined when you call it.

Parsing a large CSV in PHP

I have a large CSV I am writing a PHP CLI script to import into an existing PHP application, while utilizing the existing application's ORM to do relationship management between fields (trust me, I looked at a SQL CSV import, its easier this way). The issue at hand is that the CSV data I have is either malformed, or I have my fgetcsv() call wrong.
This is an example data set i'm aiming to import:
id,fname,lname,email\n
1,John,Public,johnqpublic#mailinator.com\n
1,Jane,Public,janeqpublic#mailinator.com\n
And the CSV Import code pretty much takes from the PHP Docs on fgetcsv():
function import_users($filepath) {
$row = 0;
$linesExecuted = 0;
if(($file = fopen($filepath, 'r')) !== false) {
$header = fgetcsv($file); //Loads line 1
while(($data = fgetcsv($file, 0, ",")) !== false) {
$userObj = user_record_generate_stdclass($data);
//A future method actually pipes the data through via the ORM
$row++;
}
fclose($file);
} else {
echo "It's going horribly wrong";
}
echo $row." records imported.";
}
The resulting logic of this method is pretty much a % sign, which is baffling. Am I overlooking something?

PHP - Out of Memory while very simple loop, w/o object

i have some trouble for a very simple loop because i goes out of memory and i really dont know why.
Hope someone will be able to help me.
Here is my code:
$full_list = array();
$fp = fopen($file_name, 'r');
while (($line = fgetcsv($fp, 0, $delimiter)) !== FALSE)
{
$val = array_slice($line, 0, 1);
$line = NULL;
unset($line);
if (in_array($val, $full_list) === FALSE)
$full_list[] = $val;
$val = NULL;
unset($val);
}
fclose($fp);
I tried the $line = NULL && then unset it as u can see but even like that it doesnt work, if the file is too big i'll get out of memory...
To be honest, i dont even understand why the memory even increase during the loop....
yes, you will always run out of memory, because, variable $full_list will always grow, till you will reach out of memory.
you need to do your stuff directly in while cycle.
and instead use array_slice($line, 0, 1) just use: $line[0]

Read .csv file and save its values in a list of arrays

I am new at php programming but I have been stuck with this code for some time.
I would like to read a .csv file line by line and then save its values in a list of arrays.
$file = fopen('Sub-Companies.csv', 'r');
while (($line =
fgetcsv($file)) !== FALSE) {
print_r($line);
list($customer_id[],$company_name[],$department[],$employee[],$country[],$zipcode[],$address[],$city[],
$smth1[], $smth2[], $phone_no1[],$phone_no2[],$email[],$website[],
$customer_no[],$problem1[],$problem2[]) = explode(";",$line); }
fclose($file); var_dump($customer_id);
The problem is that, although it is read correctly the file, then the explode is not working and the arrays appear to be null.
One thing that I am considering is that some arrays have more ";" than others, so that might be a problem, that is why I have the arrays $problem1 and $problem2, in order to store the values of this arrays.
Any help would be great!
You're using fgetcsv() in the wrong way.
We've come to this solution while chatting here on StackOverflow.
<?php
// Create file data.csv with your data
$handle = fopen('Sub-Companies.csv', 'r');
$customer_id = array();
$xyz_array = array();
// ...
// Better use a specified length (second parameter) instead of 0
// It slows down the whole process of reading the data!
while (($line = fgetcsv($handle, 0, ';')) !== FALSE) {
$customer_id[] = $line[0];
$xyz_array[] = $line[1];
}

Categories