PHP Split CSV file and update to Database doesn't work - php

I would like to split a very large CSV file (20 000 lines) to be able to put everything in my MySQL database (after I use cronjob for load the php file every hours).
So I have found code to split my file :
$inputFile = 'Shipping.csv';
$outputFile = 'output';
$splitSize = 1000;
$in = fopen($inputFile, 'r');
$rowCount = 0;
$fileCount = 1;
while (!feof($in)) {
if (($rowCount % $splitSize) == 0) {
if ($rowCount > 0) {
fclose($out);
}
$out = fopen($outputFile . $fileCount++ . '.csv', 'w');
}
$data = fgetcsv($in);
if ($data)
fputcsv($out, $data);
$rowCount++;
}
fclose($out);
I have tried to do this but they don't work :
require_once dirname(__DIR__).'/pdo.php';
$inputFile = 'Shipping.csv';
$outputFile = 'output';
$splitSize = 1000;
//open uploaded csv file with read only mode
$in = fopen($inputFile, 'r');
//skip first line
fgetcsv($in);
$rowCount = 0;
$fileCount = 1;
while (!feof($in)) {
if (($rowCount % $splitSize) == 0) {
if ($rowCount > 0) {
fclose($out);
}
$out = fopen($outputFile . $fileCount++ . '.csv', 'w');
}
$data = fgetcsv($in);
if ($data){
fputcsv($out, $data);
}
while(($line = fgetcsv($out)) !== FALSE){
//check whether member already exists in database with same email
$prevQuery = "SELECT id FROM shipbp WHERE license = '".$line[1]."'";
$prevResult = $bdd->query($prevQuery);
if($prevResult->rowCount() > 0)
{
$bdd->query("UPDATE shipbp SET license = '".$line[0]."'");
}
else{
$bdd->query("INSERT INTO shipbp (license) VALUES ('".$line[0]."')");
}
}
$rowCount++;
}
fclose($out);
CSV file look like :
P135460,002,00003,250,AS44563,0.35,,Blabla,17/3/2017 00:00:00,SB,Blabla
How I can do ?
Thank you for your help

Related

How to write and save CSV file to server in PHP?

I am trying to create a new CSV file using PHP and upload or move it to a new part of the server but the spreadsheet it returns is a spreadsheet that has only the first cell in the first row with a value of either 404 or 1. What am I doing wrong?
My code is attached below.
// genrate new general spreadsheet
$filepath = substr($file_path, 1);
$data = load_csv_file($filepath);
header('Content-type: text/csv');
header('Content-Disposition: attachment; filename="file-saved.csv"');
$fp = fopen('php://output', 'wb');
foreach ($data as $row) {
$output = fputcsv($fp, $row);
}
$filename = "file-saved.csv";
file_put_contents( $filename, $output);
fclose($fp);
The $data variable is an array of values from another CSV file.
$output = [];
foreach($data as $row) {
$output[] = ..
}
...
error_reporting(0);
$file_n = public_path('/csv_file/product_details.csv');
$infoPath = pathinfo($file_n);
if($infoPath['extension'] == 'csv'){
$file = fopen($file_n, "r");
$i = 0;
$all_data = array();
while ( ($filedata = fgetcsv($file, null, "|")) !==FALSE) {
$num = count($filedata );
for ($c=0; $c < $num; $c++) {
$all_data[$i][] = $filedata [$c];
}
$i++;
}
fclose($file);
foreach($all_data as $importData){
$insertData = array(
"article_number"=>$importData[0],
"article_name"=>$importData[1],
"article_description"=>$importData[2],
"article_price"=>$importData[3],
"article_manufacturer"=>$importData[6],
"article_productgroupkey"=>$importData[7],
"article_productgroup"=>$importData[8],
"article_ean"=>$importData[9],
"article_hbnr"=>$importData[10],
"article_shippingcosttext"=>$importData[11],
"article_amount"=>$importData[12],
"article_paymentinadvance"=>$importData[13],
"article_maxdeliveryamount"=>$importData[14],
"article_energyefficiencyclass"=>$importData[15]
);
insertData($insertData);
}
}else{
echo "Invalid file extension.";
}
function insertData($data){
if($article_number->count() == 0){
//write your insert query here for $data
}elseif($article_number->count() > 0){
//article_number already present then update the table.UPDATE QUERY
}
}

“How to export other next file‘ after 2000 rows data

I am using PHP for export CSV file this is working, but i export 2000 or greater rows how to create automatic next CSV file.
How to Move other file on after 2000 rows?
<?php
header('Content-type: application/csv');
header('Content-Disposition: attachment; filename = records.csv');
echo $header = "Name";
echo "\r\n";
$sql = mysql_query(“Select * from table”);
while ($getData = mysql_fetch_assoc($sql)) {
echo '"'.$name.'"';
echo "\r\n";
}
exit;
?>
You can use array_chunk function to keep the records of 2000 and export them in csv.
For example
$rowData = [
[1,2,3],
[11,21,31],
[21,22,32],
[31,42,73],
[111,222,333]
];
foreach(array_chunk($rowData, 2) as $key => $records){
$file_name = 'export_data_'.$key.'.csv';
writeToCsv($file_name,$records);
}
//funtion to export data to csv
function writeToCsv($fileName, $rowData){
$fp = fopen($fileName, 'w');
foreach ($rowData as $fields) {
fputcsv($fp, $fields);
}
fclose($fp);
}
In your case use array_chunk($rowData, 2000)
<?php
$rowLimit = 2000;
$fileIndex = 0;
$i = 1;
$handle = null;
$fileList = array();
$timestampFolder = strtotime("now").'/';
mkdir($timestampFolder);
$sql = mysql_query("Select * from table");
while ($getData = mysql_fetch_assoc($sql)) {
if( ($i%$rowLimit) == 1) {
$fileIndex+=1;
if(!is_null($handle)) {
fclose($handle);
}
$fileName = "records".$fileIndex.".csv";
$handle = fopen($timestampFolder.$fileName, "a");
$fileList[] = $fileName;
}
fputcsv($handle, $getData);
$i++;
}
foreach($fileList as $file) {
echo ''.$file.'<br>';
}

Php code to read multiple file and return the last line

Am trying to read multiple file and display the last line of each file but my code is not working very fine i don't know what am doing wrong.
<?php
if(isset($_GET['fid'], $_GET['timestamp']) && !empty($_GET['fid'])){
$lastmodif = isset( $_GET['timestamp']) ? $_GET['timestamp']: 0 ;
$fl_session = preg_replace("/[^0-9=]+/", "", $_GET['fid']);
if(strpos($fl_session, '==') !== false) {
$fl_session = explode('==', $fl_session);
foreach($fl_session as $fn){
$filename = dirname(__FILE__).'/logs/foo_'.$fn.'.txt';
$currentmodif = filemtime($filename);
while($currentmodif <= $lastmodif) {
usleep(10000);
clearstatcache();
$currentmodif = filemtime($filename);
}
if(file_exists($filename)){
$myfile = fopen($filename, "r") or die("Unable to load readme file!");
if($myfile){
$log = fread($myfile, filesize($filename));
fclose($myfile);
}
$log = explode("\n", trim($log));
for ($i = 0; $i < count($log); $i++) {
$log[$i] = trim($log[$i]);
$log[$i] = explode('|', $log[$i]);
}
foreach ($log as $logline) {
$response['id'] = trim($logline['0']);
$response['type'] = trim($logline['3']);
}
}
$response[$fn]['timestamp'] = $currentmodif;
//$response['timestamp'] = $currentmodif;
//var_export($response);
echo json_encode($response);
}
}
}
In my GET['fid'] parameter it contain something like this 12345==09765==87765, so i will explode it and loop on every id to locate the file name foo_12345.txt, foo_09765.txt, foo_87765.txt. This current code returned something like this from one txt file and ignored other files.
{ "id":"58287469", "type":"client", "8942180223":{
"timestamp":1505409432 } } { "id":"53094615", "type":"client",
"8942180223":{ "timestamp":1505409432 }, "4582422934":{
"timestamp":1505409420 } }

Export row from CSV file to JSON

I want to export a row in CSV file to JSON file with require : 1 line in CSV export 1 file JSON. I success to export file,but can't filter the row that i want to export. Can anyone help me?
<?php
header('Content-type: application/json');
$feed = 'jsondata_palpad.csv';
$keys = array();
$newArray = array();
function csvToArray($file, $delimiter) {
if (($handle = fopen($file, 'r')) !== FALSE) {
$i = 0;
while (($lineArray = fgetcsv($handle, 4000, $delimiter, '"')) !== FALSE) {
for ($j = 0; $j < count($lineArray); $j++) {
$arr[$i][$j] = $lineArray[$j];
}
$i++;
}
fclose($handle);
}
return $arr;
}
$data = csvToArray($feed, ',');
$count = count($data) - 1;
$labels = array_shift($data);
foreach ($labels as $label) {
$keys[] = $label;
}
$keys[] = 'id';
for ($i = 0; $i < $count; $i++) {
$data[$i][] = $i;
}
for ($j = 0; $j < $count; $j++) {
$d = array_combine($keys, $data[$j]);
$newArray[$j] = $d;
}
//Xuat file JSON
for ($i = 0; $i < 5; $i++) {
$json = json_encode($newArray[$i]);
echo $json . "<br />\n";
$filename = $data[$i][1] . ".json";
echo $filename . "<br />\n";
//echo "title[" . $data[$i][4] . "]";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
}
?>
If recommend = 2,it will export line whre id=2.
I want to export id,product_id,title,outline to JSON file.This is sample JSON file:
http://img839.imageshack.us/img839/5277/21527799.png
This is CSV file :
http://img690.imageshack.us/img690/1526/43004273.png
You are looping through all 6 lines of the csv file and saving them to the JSON file:
for ($i = 0; $i < 5; $i++) {
...
}
If you know what row number it is you want, don't loop through every row - just call the code inside the loop for that row. For example, if you wanted the 2nd row:
$row_we_want = 1;
$json = json_encode($newArray[$row_we_want]);
$filename = $data[$row_we_want][1] . ".json";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
If you are unsure about which row it is and need to check each one, you could do so in the loop:
for ($i = 0; $i < 5; $i++) {
if (some_condition) {
$json = json_encode($newArray[$i]);
$filename = $i][1] . ".json";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
}
}
Also it might be worth replacing 5 in your loop with the length of the array using the count function if it is not always a fixed length.

Writing to multiple (splitting) CSV files with PHP

I'm using a simple function to write write arrays to a CSV-file, which look like this:
function writeToCSV($array) {
$fp = fopen('programmes.csv', 'a');
fputcsv($fp, $array);
fclose($fp);
}
Simple as a pie. However, is there anyway to know what line-number the pointer is at? Because I want to be able to after 1000 lines to begin writing to a new file. Why? Because I need to be able to import them to a database later with some memory constraints, and to parse a CSV-file with 15000 lines is a no-no.
function writeToCSV($array) {
$i = 1;
$j = 1;
$fp = fopen('programmes' . $j . '.csv', 'a');
foreach($array as $fields) {
if ($i % 1000 == 0) {
fclose($fp);
$fp = fopen('programmes' . $j . '.csv', 'a');
$j = $j + 1;
}
fputcsv($fp, $fields);
$i = $i + 1;
}
fclose($fp);
}
Try this:
count(file('programmes.csv'));
This will give you the number of lines in a file.
I haven't tried if this works, but i would do something like this:
<?php
function writeToCSV($array) {
// count lines in the current file
$linecount = 0;
$fh = fopen('programmes.csv','rb') or die("ERROR OPENING DATA");
while (fgets($fh) !== false) $linecount++;
fclose($fh);
$aSize = sizeof($array);
if (($linecount + $aSize) > 1000) {
// split array
$limit = 1000 - $linecount;
$a = array_slice($array, 0, $limit);
$b = array_slice($array, $limit);
// write into first file
$fp = fopen('programmes.csv', 'a');
foreach($a as $field) fputcsv($fp, $field);
fclose($fp);
// write into second file
$fp = fopen('programmes2.csv', 'a');
foreach($b as $field) fputcsv($fp, $field);
fclose($fp);
} else {
$fp = fopen('programmes.csv', 'a');
$idx = 0;
while ($linecount < 1000) {
// fill the file to the 1000 lines
fputcsv($fp, $array[$idx]);
++$linecount;
++$idx;
}
fclose($fp);
if ($idx != $aSize) {
// create new file
$fp = fopen('programmes.csv', 'a');
while ($idx< $aSize) {
// fill the file to the 1000 lines
fputcsv($fp, $array[$idx]);
++$idx;
}
fclose($fp);
}
}
}
?>

Categories