PHP fgetcsv array issues with comma delimiting - php

I seem to be having a problem with my fgetcsv command not comma delimiting, basically what is happening is my data is just one really long string in an array. I need the data to be in an array format for the implode function to work properly for uploading into a mysql database.
Currently the mysql upload looks like...
INSERT INTO metars VALUES('PAJZ 011132Z AUTO 1 3/4SM BR FEW009 BKN019 OVC026 08/07 A2959 RMK AO2 PWINO TSNO P0001,PAJZ,2013-07-01T11:32:00Z,59.73,-157.27,8.0,7.0,,,,1.75,29.589567,,,TRUE,TRUE,,,TRUE,,TRUE,BR,FEW,900,BKN,1900,OVC,2600,,,IFR,,,,,,0.01,,,,,,SPECI,82.0"
This should have single quotes around each element, however since its a string the implode isn't working =\
<?php
require_once("../config/dbmetar.php");
$file = "metars.csv";
$db = new mysqli(DB_HOST_METAR, DB_USER_METAR, DB_PASS_METAR, DB_NAME_METAR);
$r = 0;
if (($handle = fopen($file, "r")) !== FALSE) {
while (($data = fgetcsv($handle, 3000, ",", '"')) !== FALSE) {
if ($r >= 6) { //skips the header
foreach($data as $i => $content) {
$data[$i] = $db->real_escape_string($content);
}
//echo "INSERT INTO metars VALUES('" . implode("','", $data) . '"' ;
//echo var_dump($data);
$db->query("INSERT INTO metars VALUES('" . implode("','", $data) . "');");
}
$r++;
}
fclose($handle);
}
?>
Here's an example of the csv file
No errors
Max results hit (1000): not all possible results returned;
59 ms
data source=metars
948 results
raw_text,station_id,observation_time,latitude,longitude,temp_c,dewpoint_c,wind_dir_degrees,wind_speed_kt,wind_gust_kt,visibility_statute_mi,altim_in_hg,sea_level_pressure_mb,corrected,auto,auto_station,maintenance_indicator_on,no_signal,lightning_sensor_off,freezing_rain_sensor_off,present_weather_sensor_off,wx_string,sky_cover,cloud_base_ft_agl,sky_cover,cloud_base_ft_agl,sky_cover,cloud_base_ft_agl,sky_cover,cloud_base_ft_agl,flight_category,three_hr_pressure_tendency_mb,maxT_c,minT_c,maxT24hr_c,minT24hr_c,precip_in,pcp3hr_in,pcp6hr_in,pcp24hr_in,snow_in,vert_vis_ft,metar_type,elevation_m"
"PAJZ 011132Z AUTO 1 3/4SM BR FEW009 BKN019 OVC026 08/07 A2959 RMK AO2 PWINO TSNO P0001,PAJZ,2013-07-01T11:32:00Z,59.73,-157.27,8.0,7.0,,,,1.75,29.589567,,,TRUE,TRUE,,,TRUE,,TRUE,BR,FEW,900,BKN,1900,OVC,2600,,,IFR,,,,,,0.01,,,,,,SPECI,82.0"
"CYOD 011131Z 18002KT 1/2SM FG FEW250 RMK CI0,CYOD,2013-07-01T11:31:00Z,54.4,-110.28,,,180,2,,0.5,,,,,,,,,,,FG,FEW,25000,,,,,,,LIFR,,,,,,,,,,,,SPECI,544.0"
"CYYD 011131Z AUTO VRB06KT 5SM -RA BR FEW007 BKN070 OVC084 15/14 A3005 RMK PRESRR PCPN 1.0MM PAST HR SLP173 DENSITY ALT 1900FT,CYYD,2013-07-01T11:31:00Z,54.82,-127.18,15.0,14.0,0,6,,5.0,30.050198,1017.3,,TRUE,,,,,,,-RA BR,FEW,700,BKN,7000,OVC,8400,,,MVFR,,,,,,,,,,,,SPECI,523.0"
"KNSE 011131Z 00000KT 10SM -RA FEW025 BKN070 BKN120 BKN250 23/19 A2980 RMK AO2 WSHFT 1045 RAB29 P0000 $ ,KNSE,2013-07-01T11:31:00Z,30.72,-87.02,23.0,19.0,0,0,,10.0,29.799213,,,,TRUE,TRUE,,,,,-RA,FEW,2500,BKN,7000,BKN,12000,BKN,25000,VFR,,,,,,0.0050,,,,,,SPECI,61.0"
I've been trying to solve this dilemma for about 6 hours with no luck, i've tried different csv pull methods such as str_getcsv, fgetcsv, i've tried both foreach and for... the simplified version is what I have below and is the easiest way for updating header information.
Purpose: this php script will be ran about every 60sec-150sec in a cron scheduler, if you have any other suggestions regarding this, I would appreciate that information as well.
I do appreciate any help in this matter,
Thanks,
-Mikael

You might want to consider using file() for this type of operation so you can trim the first few lines out and trim the quotes off the ends of the CSV string. The fgetcsv parameters you passed are for each column to be wrapped in quotes, not the entire string.
From there, you can then use str_getcsv()
Example (untested):
<?php
$file = file("metars.csv");
for($i = 0; $i < 6; $i++) array_shift($file);
foreach($file as $line) {
$line = ltrim($line, '"');
$line = rtrim($line, '"');
$csv = str_getcsv($line);
/** do query, preferably a prepared statement **/
}
?>
Of course, if your file is massive - this may not be the best solution.

Related

array_keys returning higher number than end($array)

I have a CSV file that contains around 8500 lines but I'm getting a really weird "bug".
I'm validating the data inside the CSV to make sure the data is cool to import into the database. I currently just log the data errors to a log file, but when I open it I see error reports for rows upto 8800 (give or take).
I did some basic debugging to see what's what and did this to begin with:
foreach ($csv as $key => $row)
{
if ($key > 8500) {
echo '<pre>';
print_r($row);
echo '</pre>';
}
}
and that only returned about 50/60 more which is fine as the total rows is around that number.
I then tried doing this to get the end array result:
$last = end($csv);
print_r($last);
and that showed an array with data as expected. However when I do this:
var_dump(array_keys($csv));
then it shows 8800 (give or take) values. Doing count($csv) returns the same number.
I've tried going into the actual CSV and highlighting everything below the last row and hitting clear but it still has the same affect..
Here's how I build my $csv array:
$skus = $csv = [];
if (($handle = fopen($fileTmp, 'r')) !== false) {
set_time_limit(0);
$i = 0;
while (($csvData = fgetcsv($handle, 1000, ',')) !== false)
{
$colCount = count($csvData);
$csv[$i]['sku'] = $csvData[0];
$csv[$i]['desc'] = $csvData[1];
$csv[$i]['ean'] = $csvData[2];
$csv[$i]['rrp_less_vat'] = $csvData[3];
$csv[$i]['rrp_inc_vat'] = $csvData[4];
$csv[$i]['stock'] = $csvData[5];
$csv[$i]['est_delivery'] = $csvData[6];
$csv[$i]['img_name'] = $csvData[7];
$csv[$i]['vatable'] = $csvData[8];
$csv[$i]['obsolete'] = $csvData[9];
$csv[$i]['dead'] = $csvData[10];
$csv[$i]['replacement_product'] = $csvData[11];
$csv[$i]['brand'] = $csvData[12];
$csv[$i]['ext_desc'] = $csvData[13];
$i++;
}
fclose($handle);
}
Am I doing something wrong that I can't see in building the array or is this unexpected behaviour?
PHP version: 7.1
OS: Linux Mint
You have lines that are longer than the $length argument you are passing to fgetcsv(). From the documentation, emphasis mine:
Must be greater than the longest line (in characters) to be found in the CSV file (allowing for trailing line-end characters). Otherwise the line is split in chunks of length characters, unless the split would occur inside an enclosure.
The easiest fix is to stop limiting the length of the line to 1000:
while (($csvData = fgetcsv($handle)) !== false)

PHP probleme while remove columns in csv

I use the php script below to remove some columns from a csv, order them new and save it as new file.
And it works for the file i made it for.
Now i need to do the same with another csv but i don't know whats wrong. I always get a comma befor the data in the first column.
This is what i have, but it doesn't really work.
<?php
$input = 'http://***/original.csv';
$output = 'new.csv';
if (false !== ($ih = fopen($input, 'r'))) {
$oh = fopen($output, 'w');
while (false !== ($data = fgetcsv($ih))) {
// this is where you build your new row
$outputData = array($data[4], $data[0]);
fputcsv($oh, $outputData);
}
fclose($ih);
fclose($oh);
}
The original.csv looks that:
subproduct_number barcode stock week_number qty_estimate productid variantid
05096470000 4024144513543 J 3 6 35016
ae214 848518017215 N 23 0 7 35015
05097280000 4024144513727 J 1 32 34990
The seperator is ';'. The same seperator is used in the file that is working
But here it will be go wrong because my saved new.csv looks like this:
subproduct_number barcode stock week_number qty_estimate productid variantid
,05096470000 4024144513543 J 3 6 35016
,ae214 848518017215 N 23 0 7 35015
,05097280000 4024144513727 J 1 32 34990
But what i need is a new csv that looks like this:
qty_estimate subproduct_number
3 05096470000
0 ae214
1 05097280000
As you can see, i need only the 5. column ($data[4]) as first and the first column ($data[0]) as the second one.
I hope someone can point me in the reight direction.
Thanks
You can do so:
while (false !== ($data = fgetcsv($ih))) {
$data = explode(';', $data[0]);
$outputData = array($data[4], $data[0]);
fputcsv($oh, $outputData, ';');
}

PHP - write to CSV issue

Read from a csv file and pick certain columns and write again to another csv file. Everything fine except that the last column is always get wrap around with a pair of double quotes and the last double quotes is in a new line, which is strange. Below is my code and the output
if( file_exists($file_path) && filesize($file_path) > 0){
$counter = 0;
if(false !== ($read_file = fopen($file_path,'r')) ){
$output_file = fopen($file_output,'w');
while(false !== ($data = fgetcsv($read_file))){
$outputData = array($data[1], $data[6], $data[19]);
fputcsv($output_file, $outputData,',');
}
}
fclose($read_file);
fclose($output_file);
unset($outputData);
}
Output: data1,data6,"data9
"
Yes, so the last double quotes is in the newline and the last column is wrapped with a double quotes regards of what the data type is.
Thank you very much

How to sum column text file

Hello everyone and I immediately apologize, as
I have seen various threads on the site, but unfortunately my knowledge is still insufficient to complete my project.
I have a text file and I have to do the sum of each column (just need the total):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26
I need to get this(I have this file on linux then we can use Bash, PHP, Mysql... ):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26 <br>
xxxx|Total |4873.00|30.00|332.00|0.00|0.00|0.00|0.00|362.00 |326.45|35.55|47865.86
Where xxxx is the Id number (No sum here).
I've been trying to do this in PHP and MySQL -- No luck so far.
try something like:
$file = '/path/to/your_file.txt';
if ( ($file = fopen($file, "r")) !== FALSE) {
$total = 0;
$row_1 = 0;
while (($line = fgetcsv($file, 1000, "|")) !== FALSE) {
// brutal dirt sanitization
foreach ( $line as $k => $v ) {
$line[$k] = (float) preg_replace('#[^0-9\.]#','', $v);
}
$total = $total + array_sum(array_slice($line, 2));
$row_1 = $row_1 + array_sum(array_slice($line, 2, 1));
//...
}
echo $total.' | '.$row_1; //...
}
else echo 'error ...';
also, you can sanitize each row by replacing array_sum() by array_map() wih a callback function
Psuedocode:
open source file for reading
open destination file for writing
initialise totaling array to zero values
while not EOF
read in line from file
explode line into working array
for x=2 ; x<14; x++
add totalling array with floatval( working array )
write line out to destination file
close read file
write out totals array to destination file
close destingation file
Try to get the text file data into an excel spreadsheet and then add up the columns.
You can use VB to get the text into excel and then continue adding up the values of each column.
1) replace all | chars with , using str_replace
2) Use str_getcsv to create array out of the above resulting csv string
3) use foreach and loop through each row and calculate total
some PHP code
$str = file_get_contents('myfile.txt');
$str = str_replace('|', ',', $str);
$csv = str_getcsv($str);
$totals = array(0,0,0,0);
foreach ($csv as $row) {
$totals[0] += trim($row[0]);
$totals[1] += trim($row[2]);
$totals[2] += trim($row[3]);
$totals[3] += trim($row[4]);
}
the $totals array contains all totals!

php nested for statements?

I'm trying to process a for loop within a for loop, and just a little wary of the syntax... Will this work? Essentially, I want to run code for every 1,000 records while the count is equal to or less than the $count... Will the syntax below work, or is there a better way?
for($x = 0; $x <= 700000; $x++) {
for($i = 0; $i <= 1000; $i++) {
//run the code
}
}
The syntax you have will work, but I don't think it's going to do exactly what you want. Right now, it's going to do the outer loop 700,001 times, and for every single one of those 700,001 times, it's going to do the inner loop.
That means, in total, the inner loop is going to run 700,001 x 1001 = about 700.7 million times.
If this isn't what you want, can you give a bit more information? I can't really work out what "I want to run code for every 1,000 records while the count is equal to or less than the $count" means. I don't see any variable named $count at all.
Well, essentially, I'm reading in a text file and inserting each of the lines into a db. I did originally try while(!feof($f)) [where $f = filename], but it keeps complaining of a broken pipe. I thought this would be another way to go
$f should be file-handle returned by fopen(), not a filename.
$file_handle = fopen($filename, 'r');
while(!feof($file_handle)) {
$line = fgets($file_handle);
$line = trim($line); // remove space chars at beginning and end
if(!$line) continue; // we don't need empty lines
mysql_query('INSERT INTO table (column) '
.'VALUES ("'.mysql_real_escape_string($line).'")');
}
Read through the documentation at php.net for fopen(), fgets(). You might also need explode() if you need to split your string.
If your file isn't big, you might want to read it into an array at once like this:
$filelines = file($filename, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
foreach($filelines as $line) {
do_stuff_with($line);
}
Hmm. Ok... Well, essentially, I'm reading in a text file and inserting each of the lines into a db. I did originally try while(!feof($f)) [where $f = filename], but it keeps complaining of a broken pipe. I thought this would be another way to go..
To read a text file line by line I usually:
$file = file("path to file")
foreach($file as $line){
//insert $line into db
}
Strictly answering the question, you'd want something more like this:
// $x would be 0, then 1000, then 2000, then 3000
for($x = 0; $x < 700000; $x += 1000) {
// $i would be $x through $x + 999
for($i = $x; $i < $x + 1000; $i++) {
//run the code
}
}
However, you should really consider one of the other methods for importing files to a database.

Categories