When I run the code it is ridiculously slow even if I comment out the second half where I fopen the same file. I have changed the length of stream_get_line. File is 64MB, 73500 lines, 159 columns. Any help to optimize would be much appreciated.
<?php
include "configure.php";
$row = 1;
if(($handle = fopen("ACC_half.txt", "r")) !== false)
{
global $arraySamples;
global $arrayTrans;
global $num;
global $lines;
while (!feof($handle))
{
$data = stream_get_line($handle, 1000000, "\n");
$num = count($data);
$lines = count(file("ACC_half.txt"));
//Get all the names of the samples into array
if($row == 1)
{
for($i=0;$i<$num;$i++)
{
//echo $data[$i];
$arraySamples[]=$data[$i];
}
//Get all the names of the first row(headers) into array for use
}
else
{
$arrayTrans[]=$data[0];
}
$row++;
}
}
fclose($handle);
$row = 1;
$transInc=1;
if(($handle1 = fopen("ACC_half.txt", "r")) !== false)
{
while (($data2 = stream_get_line($handle,1000000,"\n\r")) !== false)
{
if(($row == 2) || ($row==1))
{
$row++;continue;
}
$jnum=count($data2);
for($j=1;$j<$jnum;$j+=2)
{
$g=$j+1;
$h=$j+2;
$import = mysql_query("INSERT into acc (form,sample,raw,scale)
VALUES ('$arrayTrans[$transInc]', '$arraySamples[$g]', '$data2[$j]', '$data2[$g]')")
or die (mysql_error());
}
$transInc++;
}
}
fclose($handle1);
?>
I believe that one of the main problems is that you open the same big file twice. You should open this file once not twice
Related
I'm trying to post 3 arrays using foreach and for some reason the break at the end isn't working and it outputs the whole list (40+) on to the page.
$file = fopen('names.csv', 'r');
while (($line = fgetcsv($file)) !== FALSE) {
//$line is an array of the csv elements
shuffle($line);
$i = 0;
foreach ($line as $number) {
{
if($i==3){ break; } else {
$rtime = mt_rand(1, 7);
echo $number; }
$i++;
}
}
}
fclose($file);
This is kind of how it looks: take.ms/cLgIh, instead it should only show 3 of these usernames.
<?php
$i = 0;
//I have opened my contact.csv :P
$file = fopen('contact.csv', 'r');
while (($line = fgetcsv($file)) !== FALSE) {
//$line is an array of the csv elements
shuffle($line);
foreach ($line as $number) {
{
if($i==3){ exit(); } else {
$rtime = mt_rand(1, 7);
echo "<br/> i = ".$i.$number.", "; }
}
$i++;
}
}
fclose($file);
?>
I have downloade first sample CSV from here:-http://www.sample-videos.com/download-sample-csv.php
And this code works for me:-
<?php
$file = fopen('SampleCSVFile_2kb.csv', 'r');
while (($line = fgetcsv($file)) !== FALSE) {
//$line is an array of the csv elements
shuffle($line);
$i = 0;
foreach ($line as $number) {
if($i==3){
exit;
} else {
$rtime = mt_rand(1, 7);
echo $number.'<br/>';
echo $i.'<br/>'; // you can remove this line
}
$i++;
}
}
fclose($file);
?>
Output on each page refresh:-
http://prntscr.com/cln2ju
http://prntscr.com/cln2nf
Note:- if still not work then check your CSV file. May be it is corrupted.
Conclusion:- And after all discussion it comes to an end that your CSV file is corrupted. But yes code improvement is needed too
You need to increment $i otherwise it's value will always be 0
$i = 0;
foreach ($line as $number) {
$rtime = mt_rand(1, 7);
echo "$number";
if($i==3) break;
$i++;
}
Also you need to check if your while statement is closing
$file = fopen('names.csv', 'r');
while (($line = fgetcsv($file)) !== FALSE) {
//$line is an array of the csv elements
shuffle($line);
$i = 0;
foreach ($line as $number) {
$rtime = mt_rand(1, 7);
echo $number;
if($i==3) break;
$i++;
}
} //check for this
As I can see in your code example it is missing the closing brace
Also, remove the double quotes from your $number, it's not necessary.
echo $number
You have a mistake is that the break are in the if($i==3) and you declared $i =0, but never increment this. So $i never arrive at 3.
I have a simple CSV-file which looks like this:
Value:
AAA
Value:
BBB
Value:
AAA
I want to count the number of times a certain value shows up (e.g. AAA).
To start, I want to get the lines which read "Value:" and just echo the following line "line[$i+1] which would be the corresponding value.
Here's the code:
<?php
$file_handle = fopen("rowa.csv", "r");
$i = 0;
while (!feof($file_handle) ) {
$line_of_text = fgetcsv($file_handle, 1024);
$line[$i] = $line_of_text[0];
if($line[$i] == "Value:"){
echo $line[$i+1]."<br />";
}
$i++;
}
fclose($file_handle);
?>
The outcome should look like this:
AAA
BBB
AAA
Unfortunately, this doesn't work..It just gives me "<*br /">s
If you are printing on command line or a file, you need to use \n instead of <br/>. That only works if your output is HTML. Also every time you want to move two lines. the logic should look like this:
if($line[$i] == "Value:"){
echo $line[$i+1]."\n"; // add a new line
}
$i+=2; // you want to move two lines
This doesn't look like a normal everyday CSV file, but here's an example that should work.
$fh = fopen('rowa.csv', 'r');
$OUT = array();
$C = 0;
while( ! feof($fh) ) {
// read 1 line, trim new line characters.
$line = trim(fgets($fh, 1024));
// skip empty lines
if ( empty($line) ) continue;
// if it's a value line we increase the counter & skip to next line
if( $line === 'Value:' ) {
$C++;
continue;
}
// append contents to array using the counter as an index
$OUT[$C] = $line;
}
fclose($fh);
var_dump($OUT);
This is not a CSV file. The file() command will load the lines of a file into an array. The for loop prints every second line.
$lines = file("thefile.txt");
for ($i = 1; $i < count($lines); $i = $i + 2) {
echo $lines[$i] . "<br/>" . PHP_EOL;
}
As PHP.net example provides, you can use this modified code:
<?php
$count = 0;
if (($handle = fopen("test.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
for ($c=0; $c < $num; $c++)
{
if (!strcmp($data[$c], 'Value:')) continue;
if (!strcmp($data[$c], 'AAA')) $count++;
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
?>
UPDATE
Try this new code, we use the value as array key and increment the count for that "key".
<?php
$counts = array();
if (($handle = fopen("test.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
for ($c=0; $c < $num; $c++)
{
if (strcmp($data[$c], 'Value:'))
{
if (!isset($counts[$data[$c]]))
{
$counts[$data[$c]] = 0;
}
$counts[$data[$c]]++;
}
else
{
// Do something
}
}
}
fclose($handle);
}
var_dump($counts);
?>
You can print the array like this:
foreach ($counts as $key => $count)
{
printf('%s: %d<br/>' . "\n", $key, $count);
}
I am trying to import a CSV file. Due to the program we use, the first row is basically all headers that I would like to skip since I've already put my own headers in via HTML. How can I get the code to skip the first row of the CSV? (the strpos command is to cut off the first field in all the rows.)
<?php
$row = 1;
if (($handle = fopen("ptt.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++) {
if(strpos($data[$c], 'Finished') !== false) {
$c++;
echo "<TR> <TD nowrap>" . $data[$c] . "</ TD>"; }
Else{
echo "<TD nowrap>" . $data[$c] . "</ TD>";
}
}
}
fclose($handle);
}
?>
Rather than using if condition for checking whether it is the first row, a better solution is to just add an extra line of code before the line from where the while loop starts as shown below :
....
.....
fgetcsv($handle);//Adding this line will skip the reading of th first line from the csv file and the reading process will begin from the second line onwards
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
.......
.......
It is just as simple......
As you are keeping track of the row number anyway, you can use continue to skip the rest of the loop for the first row.
For example, add this at the start of your while loop (just above $num = count($data)):
if($row == 1){ $row++; continue; }
There are other ways to do this, but just make sure that when you continue, $row is still being incremented or you'll get an infinite loop!
Please use the following lines of code
$flag = true;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if($flag) { $flag = false; continue; }
//your code for insert
}
Having the flag variable as true and setting it to false will skip the first line of the CSV file. This is simple and easy to implement.
put this inside your while loop:
if ($row == 1) continue;
Add this in the body of the while loop above the $row++;:
if ($row == 1) {
continue;
}
$count = 0;
while (($fields = fgetcsv($handle, 0, ",")) !== FALSE) {
$count++;
if ($count == 1) { continue; }
this worked for me:
$count = 0;
while(! feof($file))
{
$entry = fgetcsv($file, 0, ';');
if ($count > 0) {
//skip first line, header
}
$count++;
}
use this code
// mysql hostname
$hostname = 'localhost';
// mysql username
$username = 'root';
// mysql password
$password = '';
if (isset($_FILES['file']))
{
// get the csv file and open it up
$file = $_FILES['file']['tmp_name'];
//$handle is a valid file pointer to a file successfully opened by fopen(), popen(), or fsockopen().
$handle = fopen($file, "r");
try {
// Database Connection using PDO
$dbh = new PDO("mysql:host=$hostname;dbname=clasdb", $username, $password);
// prepare for insertion
$STM = $dbh->prepare('INSERT INTO statstrackertemp (ServerName, HiMemUti, AvgMemUti, HiCpuUti, AvgCpuUti, HiIOPerSec, AvgIOPerSec, HiDiskUsage, AvgDsikUsage) VALUES (?, ?, ?, ?, ?,?, ?, ?, ? )');
if ($handle !== FALSE)
{
// fgets() Gets a line from file pointer and read the first line from $handle and ignore it.
fgets($handle);
// created loop here
while (($data = fgetcsv($handle, 1000, ',')) !== FALSE)
{
$STM->execute($data);
}
fclose($handle);
}
}
catch(PDOException $e)
{
die($e->getMessage());
}
echo 'Data imported';
}
else
{
echo 'Could not import Data';
}
?>
I would like to convert a CSV to Json, use the header row as a key, and each line as object. How do I go about doing this?
----------------------------------CSV---------------------------------
InvKey,DocNum,CardCode
11704,1611704,BENV1072
11703,1611703,BENV1073
---------------------------------PHP-----------------------------------
if (($handle = fopen('upload/BEN-new.csv'. '', "r")) !== FALSE) {
while (($row_array = fgetcsv($handle, 1024, ","))) {
while ($val != '') {
foreach ($row_array as $key => $val) {
$row_array[] = $val;
}
}
$complete[] = $row_array;
}
fclose($handle);
}
echo json_encode($complete);
Just read the first line separately and merge it into every row:
if (($handle = fopen('upload/BEN-new.csv', 'r')) === false) {
die('Error opening file');
}
$headers = fgetcsv($handle, 1024, ',');
$complete = array();
while ($row = fgetcsv($handle, 1024, ',')) {
$complete[] = array_combine($headers, $row);
}
fclose($handle);
echo json_encode($complete);
I find myself converting csv strings to arrays or objects every few months.
I created a class because I'm lazy and dont like copy/pasting code.
This class will convert a csv string to custom class objects:
Convert csv string to arrays or objects in PHP
$feed="https://gist.githubusercontent.com/devfaysal/9143ca22afcbf252d521f5bf2bdc6194/raw/ec46f6c2017325345e7df2483d8829231049bce8/data.csv";
//Read the csv and return as array
$data = array_map('str_getcsv', file($feed));
//Get the first raw as the key
$keys = array_shift($data);
//Add label to each value
$newArray = array_map(function($values) use ($keys){
return array_combine($keys, $values);
}, $data);
// Print it out as JSON
header('Content-Type: application/json');
echo json_encode($newArray);
Main gist:
https://gist.github.com/devfaysal/9143ca22afcbf252d521f5bf2bdc6194
For those who'd like things spelled out a little more + some room to further parse any row / column without additional loops:
function csv_to_json_byheader($filename){
$json = array();
if (($handle = fopen($filename, "r")) !== FALSE) {
$rownum = 0;
$header = array();
while (($row = fgetcsv($handle, 1024, ",")) !== FALSE) {
if ($rownum === 0) {
for($i=0; $i < count($row); $i++){
// maybe you want to strip special characters or merge duplicate columns here?
$header[$i] = trim($row[$i]);
}
} else {
if (count($row) === count($header)) {
$rowJson = array();
foreach($header as $i=>$head) {
// maybe handle special row/cell parsing here, per column header
$rowJson[$head] = $row[$i];
}
array_push($json, $rowJson);
}
}
$rownum++;
}
fclose($handle);
}
return $json;
}
I can't get the new code I've written to skip the first row (header) as the code I was using before did (see bottom).
I'm not receiving any errors, but just can't get it to omit first row.
$file = fopen($uploadcsv,"r");
$column_headers = array();
$row_count = 0;
while(!feof($file)) {
if ($row_count==0){
$column_headers = $file;
} else {
print_r(fgetcsv($file));
}
++$row_count;
}
fclose($file);
Below is the old source that skipped the header, for reference and comparison.
$handle = fopen($uploadcsv, 'r');
$column_headers = array();
$row_count = 0;
while (($data = fgetcsv($handle, 100000, ",")) !== FALSE) {
if ($row_count==0){
$column_headers = $data;
} else {
print_r($data);
}
++$row_count;
}
fclose($handle);
Why even count? Just get the headers before looping.
$column_headers = fgetcsv($file);
while(!feof($file)) {
...
Also, you're only assigning the file pointer to the variable.
When $row_count is 0 you are not reading any row.
Change
if ($row_count==0){
$column_headers = $file; // just assigning file handle.
}
to
if ($row_count==0){
$column_headers = fgetcsv($file); // read the row.
}