I have a csv file that has like 30,000 rows in it. It also has like 9 columns. In the interest of speeding up the processing of everything I want to reduce the file to the two columns that I need and remove the rest. here is what I have done.
$retardment=1;//17;// 151; //499;// 991;// 1877
if (($handle = fopen($source, "r")) !== FALSE) {
$stock_handle = fopen($source_stock, "w+");
$row=0;
$col=array();
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
unset($line[1]);
unset($line[2]);
unset($line[3]);
unset($line[4]);
unset($line[5]);
unset($line[6]);
unset($line[8]);
unset($line[9]);
if($row%$retardment<1){
fputcsv($stock_handle, $line);
}
unset($line);
$row++;
}
fclose($handle);
fclose($stock_handle);
}
I am coping it to a new file and this works... but it seems to be pretty slow. Any ideas on how to make it faster? Thank you for the help.
Cheers -Jeremy
{EDIT}
So far this seems to take just as long. But works just fine
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row%$retardment<1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
$row++;
}
You could replace those unset() calls with...
$line = array($line[0], $line[7]);
Alternatively, remember that unset() takes multiple arguments...
unset($line[1], $line[2], ...);
You can speed it up fractionally more, but again it's a microtime()-measurable improvement: perception is that it won't be noticeably faster.
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row++ % $retardment < 1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
}
but as your script is IO-bound, it's the actual reads and writes that are the slowest functions, and you can't speed those up.
Using stream_copy_to_stream() with a stream input filter might be another approach, but you won't see much noticeable improvement unless you can reduce disk access times
Related
I am trying to develop email system that need to be send every month. So i need to build cron job file from php. Anyone know how to read file CSV or Excel file from url such as:
http://yourdomain.com/cron.php?file=http://google.com/monthly.csv
I am stuck when try to read file from url.
This is my recent code:
<?php
$url = 'http://www1.intranet.com/reportingtool.asp?settings=var&export=ok';
$tmpfname = tempnam(sys_get_temp_dir());
file_put_contents(
$tmpfname,
file_get_contents($url)
);
?>
If you're dealing with remote files, you should always keep in mind that
The connection between you and the remote can break, and you won't get the full file content;
The file can be too big to read it on-the-fly.
In both cases, file_get_contents() is not a very good thing to use: you should consider cURL functions for that. However, if the concerns above are negligible, you should be okay with the following (as the example here suggests):
$url = 'http://www1.intranet.com/reportingtool.asp?settings=var&export=ok';
$tmpfname = tempnam(sys_get_temp_dir());
file_put_contents(
$tmpfname,
file_get_contents($url)
);
if (($handle = fopen($tmpfname, "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
// Do something with $data, which comes in
// as an array of values from the current CSV line.
}
}
Someone would need one more example, so leaving it here:
$dataSource can be any file on drive or just online link to parse csv.
private array $data = [];
private string $dataSource = "https://any.csv";
if (($open = fopen($dataSource, 'rb')) !== false)
{
while (($data = fgetcsv($open, 1000, ";")) !== false)
{
$data[] = $data;
}
fclose($open);
}
I'd like to run a function within a While loop.
Unfortunately, it works but not quite as desired.
My goal is to read out from a CSV file, each line and is stored in the MySQL database. This also works flawlessly.
In addition, however, I want to let miteinpflegen a value by a numeric random.
Here again my scripts to generate the numerical codes.
function generateCode($length){
$string = "";
$numbers = "0123456789";
for($i=0;$i < $length;$i++) {
$char = $numbers[mt_rand(0, strlen($numbers)-1)];
$string .= $char;
}
return $string;
}
$gencode = generateCode(8);
Works great.
And here my while loop ($handle is in my case the var to open CSV file)
while (($data = fgetcsv($handle, 10000, ";")) !== FALSE) {
$import = "INSERT into jobs(id,link_id,random number) values('NULL','$linkid','$gencode')";
mysql_query($import) or die(mysql_error());
}
But how do I get it back now that each line of the CSV file gets its own random number or rather, which is calculated in each row $gencode new?
At the moment i get one random number for all rows.
For help, I am very grateful.
I just can no longer see the forest for the trees.
Thank you
You can simply call your own function generateCode() like any other function inside your loop:
while (($data = fgetcsv($handle, 10000, ";")) !== FALSE) {
$gencode = generateCode(8);
// the rest of your code
}
Just make sure that generateCode() is already defined when you call it.
I have a large CSV I am writing a PHP CLI script to import into an existing PHP application, while utilizing the existing application's ORM to do relationship management between fields (trust me, I looked at a SQL CSV import, its easier this way). The issue at hand is that the CSV data I have is either malformed, or I have my fgetcsv() call wrong.
This is an example data set i'm aiming to import:
id,fname,lname,email\n
1,John,Public,johnqpublic#mailinator.com\n
1,Jane,Public,janeqpublic#mailinator.com\n
And the CSV Import code pretty much takes from the PHP Docs on fgetcsv():
function import_users($filepath) {
$row = 0;
$linesExecuted = 0;
if(($file = fopen($filepath, 'r')) !== false) {
$header = fgetcsv($file); //Loads line 1
while(($data = fgetcsv($file, 0, ",")) !== false) {
$userObj = user_record_generate_stdclass($data);
//A future method actually pipes the data through via the ORM
$row++;
}
fclose($file);
} else {
echo "It's going horribly wrong";
}
echo $row." records imported.";
}
The resulting logic of this method is pretty much a % sign, which is baffling. Am I overlooking something?
i have some trouble for a very simple loop because i goes out of memory and i really dont know why.
Hope someone will be able to help me.
Here is my code:
$full_list = array();
$fp = fopen($file_name, 'r');
while (($line = fgetcsv($fp, 0, $delimiter)) !== FALSE)
{
$val = array_slice($line, 0, 1);
$line = NULL;
unset($line);
if (in_array($val, $full_list) === FALSE)
$full_list[] = $val;
$val = NULL;
unset($val);
}
fclose($fp);
I tried the $line = NULL && then unset it as u can see but even like that it doesnt work, if the file is too big i'll get out of memory...
To be honest, i dont even understand why the memory even increase during the loop....
yes, you will always run out of memory, because, variable $full_list will always grow, till you will reach out of memory.
you need to do your stuff directly in while cycle.
and instead use array_slice($line, 0, 1) just use: $line[0]
I am new at php programming but I have been stuck with this code for some time.
I would like to read a .csv file line by line and then save its values in a list of arrays.
$file = fopen('Sub-Companies.csv', 'r');
while (($line =
fgetcsv($file)) !== FALSE) {
print_r($line);
list($customer_id[],$company_name[],$department[],$employee[],$country[],$zipcode[],$address[],$city[],
$smth1[], $smth2[], $phone_no1[],$phone_no2[],$email[],$website[],
$customer_no[],$problem1[],$problem2[]) = explode(";",$line); }
fclose($file); var_dump($customer_id);
The problem is that, although it is read correctly the file, then the explode is not working and the arrays appear to be null.
One thing that I am considering is that some arrays have more ";" than others, so that might be a problem, that is why I have the arrays $problem1 and $problem2, in order to store the values of this arrays.
Any help would be great!
You're using fgetcsv() in the wrong way.
We've come to this solution while chatting here on StackOverflow.
<?php
// Create file data.csv with your data
$handle = fopen('Sub-Companies.csv', 'r');
$customer_id = array();
$xyz_array = array();
// ...
// Better use a specified length (second parameter) instead of 0
// It slows down the whole process of reading the data!
while (($line = fgetcsv($handle, 0, ';')) !== FALSE) {
$customer_id[] = $line[0];
$xyz_array[] = $line[1];
}