I have a CSV file that contains around 8500 lines but I'm getting a really weird "bug".
I'm validating the data inside the CSV to make sure the data is cool to import into the database. I currently just log the data errors to a log file, but when I open it I see error reports for rows upto 8800 (give or take).
I did some basic debugging to see what's what and did this to begin with:
foreach ($csv as $key => $row)
{
if ($key > 8500) {
echo '<pre>';
print_r($row);
echo '</pre>';
}
}
and that only returned about 50/60 more which is fine as the total rows is around that number.
I then tried doing this to get the end array result:
$last = end($csv);
print_r($last);
and that showed an array with data as expected. However when I do this:
var_dump(array_keys($csv));
then it shows 8800 (give or take) values. Doing count($csv) returns the same number.
I've tried going into the actual CSV and highlighting everything below the last row and hitting clear but it still has the same affect..
Here's how I build my $csv array:
$skus = $csv = [];
if (($handle = fopen($fileTmp, 'r')) !== false) {
set_time_limit(0);
$i = 0;
while (($csvData = fgetcsv($handle, 1000, ',')) !== false)
{
$colCount = count($csvData);
$csv[$i]['sku'] = $csvData[0];
$csv[$i]['desc'] = $csvData[1];
$csv[$i]['ean'] = $csvData[2];
$csv[$i]['rrp_less_vat'] = $csvData[3];
$csv[$i]['rrp_inc_vat'] = $csvData[4];
$csv[$i]['stock'] = $csvData[5];
$csv[$i]['est_delivery'] = $csvData[6];
$csv[$i]['img_name'] = $csvData[7];
$csv[$i]['vatable'] = $csvData[8];
$csv[$i]['obsolete'] = $csvData[9];
$csv[$i]['dead'] = $csvData[10];
$csv[$i]['replacement_product'] = $csvData[11];
$csv[$i]['brand'] = $csvData[12];
$csv[$i]['ext_desc'] = $csvData[13];
$i++;
}
fclose($handle);
}
Am I doing something wrong that I can't see in building the array or is this unexpected behaviour?
PHP version: 7.1
OS: Linux Mint
You have lines that are longer than the $length argument you are passing to fgetcsv(). From the documentation, emphasis mine:
Must be greater than the longest line (in characters) to be found in the CSV file (allowing for trailing line-end characters). Otherwise the line is split in chunks of length characters, unless the split would occur inside an enclosure.
The easiest fix is to stop limiting the length of the line to 1000:
while (($csvData = fgetcsv($handle)) !== false)
Related
I have a csv file with a very large number of item (5000 lines) in this format
storeId,bookId,nb
124,48361,0
124,48363,6
125,48362,8
125,48363,2
126,28933,4
142,55433,6
142,55434,10
171,55871,7
171,55872,6
I need to count the number of stores in the file, so for exemple with the line above the result should be 5. But I need to doo it with 5000 lines so I can't just loop.
How can I achieve that?
I also need too return the max quantity, so 10
I began by converting the file into an array:
if (file_exists($file)) {
$csv = array_map('str_getcsv', file($file));
#Stores
$storeIds = array_column($csv, 0);
$eachStoreNb = array_count_values($storeIds);
$storeCount = count($eachStoreNb);
}
print_r($storeCount);
Is there a better way to do it? Faster ? Maybe without using the array
Faster here would come in the context of micro-optimization, however you can see an improvement in memory usage.
You could just read the file line by line instead of collecting all store IDs in an array and then doing an array_count_values() saving you an extra loop and unnecessary linear storage of all duplicate values.
Store IDs would just be made as a key for an associative array.
For max NB, you can just keep a max variable keeping the track of max value using max() function. Rest is self-explanatory.
Snippet:
<?php
$file = 'test.csv';
if (file_exists($file)) {
$fp = fopen($file ,'r');
$max_nb = 0;
$store_set = [];
fgetcsv($fp); // ignoring headers
while(!feof($fp)){
$row = fgetcsv($fp);
$store_set[$row[0]] = true;
$max_nb = max($max_nb,end($row));
}
fclose($fp);
echo "Num Stores : ",count($store_set),"<br/>";
echo "Max NB : ",$max_nb;
}else{
echo "No such CSV file found.";
}
Note: For profiling, I suggest you to try both scripts using xdebug
What if you looped through the file line by line?
I mean ...
$datas = [];
$handle = fopen("filename.csv", "r");
$flagFirstLine = true;
while(!feof($handle)){
//dont read first line
if($flagFirstLine) continue;
$flagFirstLine = false;
$csvLine = fgetcsv($handle);
$storeID = $csvLine[0];
$datas[] = $storeID;
}
echo "all row: " . count($datas);
echo "\nnum store: " . count(array_unique($datas));
What 'nice_dev' says, but a little more compact.
$fp = fopen('<your_file>', 'r');
fseek($fp, strpos($content, "\n") + 1); // skip first line
$stores = [];
while($row = fgetcsv($fp)) {
$stores[$row[0]] = max([($stores[$row[0]] ?? 0), $row[2]]);
}
Working example.
An answer with awk would be:
awk -F, 'BEGIN {getline}
{ a[$1]++; m=$3>m?$3:m }
END{ for (i in a){ print i, a[i] };
print "Number of stores",length(a), "max:",m}' testfile
getline to skip the first line
increment the element with the value of the first column $1 in array a with one, and keep the max value in m
loop over the array a and print all counts (optional)
print the total 'Number of stores', and the max value.
output:
124 52
125 52
126 26
142 52
171 52
Number of stores 5 max: 10
Solution in AWK, to compare the difference. This includes the count of each store as well. AWK should be able to process millions in less than 1 second. I use the same to filter duplicates from a file.
BEGIN{ # Set some variables initially
FS="," # field separator for INPUT
mymax=0 # init variable mymax
}
NR>1 { # skip the header line, this matches line 2 onwards
mycount[$1]++ # increase associative array at that position
if ($3>mymax){ # compare with max
mymax=$3
}
}
END{ # finally print results
for (i in mycount){
if (length(i)>0){
print "value " i " has " mycount[i]
}
}
print "Maximum value is " mymax
}
After opening CSV files with fopen(), I'm currently skipping the three first lines of these files like this:
fgetcsv($file, 0, ';');
fgetcsv($file, 0, ';');
fgetcsv($file, 0, ';');
Is there a nicer method to skip the n first lines ?
If you need to skip more than a handful of lines, simply add one instance of the "throw-away read" code you already have inside a for loop. Number of loops equals lines skipped (e.g. 8 lines):
for ($i = 0; $i < 8; $i++) {
fgetcsv($file, 0, ';');
}
Do this before you begin your main while loop getting the CSV lines you care about. You could turn the loop into a utility function -- if you find yourself doing this often at different skip lengths.
function fskipcsv_lines($handle, int $lines) {
for ($i = 0; $i < $lines; $i++) {
fgetcsv($handle, 0, ';');
}
}
Simple enough. Same construct applies for the "dumb" repetition of any other function you don't need to get a a return value from, that simply needs to be called N times.
P.S. Don't place the "skip line" check routine inside your main CSV iteration loop. Why? Because the check (e.g. $row < 3; or in_array($row, $skips)) would happen at each loop iteration. Suppose you need to read 100,000+ lines, the overhead will start adding up, unnecessarily burdening each loop iteration for the first (now past) few lines' sake.
Your are doing right, but this code may help if you are going to skip many. :
$skippTheseLines = array(1,2,3);
$i = 0;
$totlLines= 10000;
while (($emapData = fgetcsv($file, $totalLines, ";")) !== FALSE) {
if(in_array($i, $skippTheseLines)) {
continue;
}else{
// rest of your code
}
$i = $i + 1;
}
Check out this code. Similar questions: skip first line of fgetcsv method in php
$file = fopen('example.csv', 'r'); // Here example is a CSV name
$row = 1;
$number_to_skip = 3
while (($line = fgetcsv($file,0, ",")) !== FALSE) {
// $line is an array of the csv elements
if($row < $number_to_skip)
{
$row++; continue; // continue is used for skip row 0,1,2
}
print_r($line);
}
So my csv file format would be like this
Site：
Site ID：
Owner:
Interval:
Location:
Last Record:
---Data From User File---：
Date, Content
2019-01-10-12:15, 1
2019-01-10-12:15, 1.5
2019-01-10-12:15, 0.8
2019-01-10-12:15, 1.4
2019-01-10-12:15, 1.9
2019-01-10-12:15, 1.4
2019-01-10-12:15, 1.6
2019-01-10-12:15, 1.7
2019-01-10-12:15, 1.1
And I need do start to read csv file from line 10 (2019-01-10-12:15, 1) to end of the file by using php. I have check most of methods online and it did not help me to solve the problem, my code is provided below.
and it turned out to be reading the previous 8 lines(which is opposite of what I want).
<?php
$index = 8;
$i = 1;
$file = fopen("data.csv", "r");
$data = fgetcsv($file);
while (! feof($file)) {
if ($index >= $i) {
print_r($data = fgetcsv($file));
}
$i++;
}
fclose($file);
?>
Need to read from line 10
<?php
$skip_lines = 8;
$i = 0;
$file = fopen('/path/to.csv', 'r');
while (($row = fgetcsv($file)) !== FALSE) {
if($i++ >= $skip_lines) {
print_r($row);
}
}
fclose($file);
There can be several solutions to process your data.
As your other lines does not start with a number, what I would do is check first character of new line is numeric or not (get first character by using PHP's substr() function and then check that variable it with is_numeric() function). If you get a numeric value, that is your first useful line.
Hello everyone and I immediately apologize, as
I have seen various threads on the site, but unfortunately my knowledge is still insufficient to complete my project.
I have a text file and I have to do the sum of each column (just need the total):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26
I need to get this(I have this file on linux then we can use Bash, PHP, Mysql... ):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26 <br>
xxxx|Total |4873.00|30.00|332.00|0.00|0.00|0.00|0.00|362.00 |326.45|35.55|47865.86
Where xxxx is the Id number (No sum here).
I've been trying to do this in PHP and MySQL -- No luck so far.
try something like:
$file = '/path/to/your_file.txt';
if ( ($file = fopen($file, "r")) !== FALSE) {
$total = 0;
$row_1 = 0;
while (($line = fgetcsv($file, 1000, "|")) !== FALSE) {
// brutal dirt sanitization
foreach ( $line as $k => $v ) {
$line[$k] = (float) preg_replace('#[^0-9\.]#','', $v);
}
$total = $total + array_sum(array_slice($line, 2));
$row_1 = $row_1 + array_sum(array_slice($line, 2, 1));
//...
}
echo $total.' | '.$row_1; //...
}
else echo 'error ...';
also, you can sanitize each row by replacing array_sum() by array_map() wih a callback function
Psuedocode:
open source file for reading
open destination file for writing
initialise totaling array to zero values
while not EOF
read in line from file
explode line into working array
for x=2 ; x<14; x++
add totalling array with floatval( working array )
write line out to destination file
close read file
write out totals array to destination file
close destingation file
Try to get the text file data into an excel spreadsheet and then add up the columns.
You can use VB to get the text into excel and then continue adding up the values of each column.
1) replace all | chars with , using str_replace
2) Use str_getcsv to create array out of the above resulting csv string
3) use foreach and loop through each row and calculate total
some PHP code
$str = file_get_contents('myfile.txt');
$str = str_replace('|', ',', $str);
$csv = str_getcsv($str);
$totals = array(0,0,0,0);
foreach ($csv as $row) {
$totals[0] += trim($row[0]);
$totals[1] += trim($row[2]);
$totals[2] += trim($row[3]);
$totals[3] += trim($row[4]);
}
the $totals array contains all totals!
Thank you for taking the time to read this and I will appreciate every single response no mater the quality of content. :)
Using PHP, I'm trying to get the last 15 lines of a text document (.txt) and store that data into a php variable. I understand that this is possible, however when I do get the last 15 lines, is it possible to retain the order? For example:
text document:
A
B
C
When I grab the text document from the last 15 characters, I don't want the echo to end up like:
C
B
A
All assistance is appreciated and I look forward to your replies; thank you. :) If I didn't explain anything clearly and/or you'd like me to explain in more detail, please reply. :)
Thank you.
Try using array_slice, which will return a part of an array. In this case you want it to return the last 15 lines of the array, so:
$filearray = file("filename");
$lastfifteenlines = array_slice($filearray,-15);
If you don't mind loading the entire file into memory:
$lines = array_slice(file('test.txt'), -15);
print_r($lines );
If the file is too large to fit into memory you can use a circular method:
// Read the last $num lines from stream $fp
function read_last_lines($fp, $num)
{
$idx = 0;
$lines = array();
while(($line = fgets($fp)))
{
$lines[$idx] = $line;
$idx = ($idx + 1) % $num;
}
$p1 = array_slice($lines, $idx);
$p2 = array_slice($lines, 0, $idx);
$ordered_lines = array_merge($p1, $p2);
return $ordered_lines;
}
// Open the file and read the last 15 lines
$fp = fopen('test.txt', 'r');
$lines = read_last_lines($fp, 15);
fclose($fp);
// Output array
print_r($lines);
This method will also work if the file has less than 15 lines- returning an array with however many lines are in the file.
You can use fseek with a negative position to seek backwards through the file, counting newlines as you go.
I'm too tired to write up copy/past-able code, but there are some examples in the comments to the manual page for fseek that are very close to what you want.
If the file isn't bigger than available memory you can do this:
$fArray = file("filename");
$len = sizeof($fArray);
for($i=$len -15;$i<$len ;$i++)
{
echo $fArray[$i];
}
If you have a file that is hundreds of megabytes :
$rc = fopen("file","r");
for ($i=0; $line = fgets($file) ;$i++)
{
if ($i%15 == 0)
{
$last15 = array();
}
$last15[] = $line;
}
echo join("\n",$last15);
the longer array solution:
array_slice(explode("\n",file_get_contents($file)),-15);
the shorter array solution:
array_slice(file($file),-15);
This code will open the file, show the total lines, show the header of file and show the last lines of file defined in $limit.
<?php
// open the file in read mode
$file = new SplFileObject('file.csv', 'r');
// get the total lines
$file->seek(PHP_INT_MAX);
$last_line = $file->key();
echo $last_line;
echo "<br>";
// Rewind to first line to get header
$file->rewind();
// Output first line if you need use the header to make something
echo $file->current();
echo "<br>";
// selecting the limit
$limit = 6;
// selecting the last lines using the $limit
$lines = new LimitIterator($file, $last_line - $limit, $last_line);
//print all the last 6 lines array
//print_r(iterator_to_array($lines));
//echo "<br>";
// Loop over whole file to use a single line
foreach ($lines as $line) {
print_r($line);
echo "<br>";
}