I would like to split a very large CSV file (20 000 lines) to be able to put everything in my MySQL database (after I use cronjob for load the php file every hours).
So I have found code to split my file :
$inputFile = 'Shipping.csv';
$outputFile = 'output';
$splitSize = 1000;
$in = fopen($inputFile, 'r');
$rowCount = 0;
$fileCount = 1;
while (!feof($in)) {
if (($rowCount % $splitSize) == 0) {
if ($rowCount > 0) {
fclose($out);
}
$out = fopen($outputFile . $fileCount++ . '.csv', 'w');
}
$data = fgetcsv($in);
if ($data)
fputcsv($out, $data);
$rowCount++;
}
fclose($out);
I have tried to do this but they don't work :
require_once dirname(__DIR__).'/pdo.php';
$inputFile = 'Shipping.csv';
$outputFile = 'output';
$splitSize = 1000;
//open uploaded csv file with read only mode
$in = fopen($inputFile, 'r');
//skip first line
fgetcsv($in);
$rowCount = 0;
$fileCount = 1;
while (!feof($in)) {
if (($rowCount % $splitSize) == 0) {
if ($rowCount > 0) {
fclose($out);
}
$out = fopen($outputFile . $fileCount++ . '.csv', 'w');
}
$data = fgetcsv($in);
if ($data){
fputcsv($out, $data);
}
while(($line = fgetcsv($out)) !== FALSE){
//check whether member already exists in database with same email
$prevQuery = "SELECT id FROM shipbp WHERE license = '".$line[1]."'";
$prevResult = $bdd->query($prevQuery);
if($prevResult->rowCount() > 0)
{
$bdd->query("UPDATE shipbp SET license = '".$line[0]."'");
}
else{
$bdd->query("INSERT INTO shipbp (license) VALUES ('".$line[0]."')");
}
}
$rowCount++;
}
fclose($out);
CSV file look like :
P135460,002,00003,250,AS44563,0.35,,Blabla,17/3/2017 00:00:00,SB,Blabla
How I can do ?
Thank you for your help
Related
I am trying to create a new CSV file using PHP and upload or move it to a new part of the server but the spreadsheet it returns is a spreadsheet that has only the first cell in the first row with a value of either 404 or 1. What am I doing wrong?
My code is attached below.
// genrate new general spreadsheet
$filepath = substr($file_path, 1);
$data = load_csv_file($filepath);
header('Content-type: text/csv');
header('Content-Disposition: attachment; filename="file-saved.csv"');
$fp = fopen('php://output', 'wb');
foreach ($data as $row) {
$output = fputcsv($fp, $row);
}
$filename = "file-saved.csv";
file_put_contents( $filename, $output);
fclose($fp);
The $data variable is an array of values from another CSV file.
$output = [];
foreach($data as $row) {
$output[] = ..
}
...
error_reporting(0);
$file_n = public_path('/csv_file/product_details.csv');
$infoPath = pathinfo($file_n);
if($infoPath['extension'] == 'csv'){
$file = fopen($file_n, "r");
$i = 0;
$all_data = array();
while ( ($filedata = fgetcsv($file, null, "|")) !==FALSE) {
$num = count($filedata );
for ($c=0; $c < $num; $c++) {
$all_data[$i][] = $filedata [$c];
}
$i++;
}
fclose($file);
foreach($all_data as $importData){
$insertData = array(
"article_number"=>$importData[0],
"article_name"=>$importData[1],
"article_description"=>$importData[2],
"article_price"=>$importData[3],
"article_manufacturer"=>$importData[6],
"article_productgroupkey"=>$importData[7],
"article_productgroup"=>$importData[8],
"article_ean"=>$importData[9],
"article_hbnr"=>$importData[10],
"article_shippingcosttext"=>$importData[11],
"article_amount"=>$importData[12],
"article_paymentinadvance"=>$importData[13],
"article_maxdeliveryamount"=>$importData[14],
"article_energyefficiencyclass"=>$importData[15]
);
insertData($insertData);
}
}else{
echo "Invalid file extension.";
}
function insertData($data){
if($article_number->count() == 0){
//write your insert query here for $data
}elseif($article_number->count() > 0){
//article_number already present then update the table.UPDATE QUERY
}
}
I am using PHP for export CSV file this is working, but i export 2000 or greater rows how to create automatic next CSV file.
How to Move other file on after 2000 rows?
<?php
header('Content-type: application/csv');
header('Content-Disposition: attachment; filename = records.csv');
echo $header = "Name";
echo "\r\n";
$sql = mysql_query(“Select * from table”);
while ($getData = mysql_fetch_assoc($sql)) {
echo '"'.$name.'"';
echo "\r\n";
}
exit;
?>
You can use array_chunk function to keep the records of 2000 and export them in csv.
For example
$rowData = [
[1,2,3],
[11,21,31],
[21,22,32],
[31,42,73],
[111,222,333]
];
foreach(array_chunk($rowData, 2) as $key => $records){
$file_name = 'export_data_'.$key.'.csv';
writeToCsv($file_name,$records);
}
//funtion to export data to csv
function writeToCsv($fileName, $rowData){
$fp = fopen($fileName, 'w');
foreach ($rowData as $fields) {
fputcsv($fp, $fields);
}
fclose($fp);
}
In your case use array_chunk($rowData, 2000)
<?php
$rowLimit = 2000;
$fileIndex = 0;
$i = 1;
$handle = null;
$fileList = array();
$timestampFolder = strtotime("now").'/';
mkdir($timestampFolder);
$sql = mysql_query("Select * from table");
while ($getData = mysql_fetch_assoc($sql)) {
if( ($i%$rowLimit) == 1) {
$fileIndex+=1;
if(!is_null($handle)) {
fclose($handle);
}
$fileName = "records".$fileIndex.".csv";
$handle = fopen($timestampFolder.$fileName, "a");
$fileList[] = $fileName;
}
fputcsv($handle, $getData);
$i++;
}
foreach($fileList as $file) {
echo ''.$file.'<br>';
}
Am trying to read multiple file and display the last line of each file but my code is not working very fine i don't know what am doing wrong.
<?php
if(isset($_GET['fid'], $_GET['timestamp']) && !empty($_GET['fid'])){
$lastmodif = isset( $_GET['timestamp']) ? $_GET['timestamp']: 0 ;
$fl_session = preg_replace("/[^0-9=]+/", "", $_GET['fid']);
if(strpos($fl_session, '==') !== false) {
$fl_session = explode('==', $fl_session);
foreach($fl_session as $fn){
$filename = dirname(__FILE__).'/logs/foo_'.$fn.'.txt';
$currentmodif = filemtime($filename);
while($currentmodif <= $lastmodif) {
usleep(10000);
clearstatcache();
$currentmodif = filemtime($filename);
}
if(file_exists($filename)){
$myfile = fopen($filename, "r") or die("Unable to load readme file!");
if($myfile){
$log = fread($myfile, filesize($filename));
fclose($myfile);
}
$log = explode("\n", trim($log));
for ($i = 0; $i < count($log); $i++) {
$log[$i] = trim($log[$i]);
$log[$i] = explode('|', $log[$i]);
}
foreach ($log as $logline) {
$response['id'] = trim($logline['0']);
$response['type'] = trim($logline['3']);
}
}
$response[$fn]['timestamp'] = $currentmodif;
//$response['timestamp'] = $currentmodif;
//var_export($response);
echo json_encode($response);
}
}
}
In my GET['fid'] parameter it contain something like this 12345==09765==87765, so i will explode it and loop on every id to locate the file name foo_12345.txt, foo_09765.txt, foo_87765.txt. This current code returned something like this from one txt file and ignored other files.
{ "id":"58287469", "type":"client", "8942180223":{
"timestamp":1505409432 } } { "id":"53094615", "type":"client",
"8942180223":{ "timestamp":1505409432 }, "4582422934":{
"timestamp":1505409420 } }
I want to export a row in CSV file to JSON file with require : 1 line in CSV export 1 file JSON. I success to export file,but can't filter the row that i want to export. Can anyone help me?
<?php
header('Content-type: application/json');
$feed = 'jsondata_palpad.csv';
$keys = array();
$newArray = array();
function csvToArray($file, $delimiter) {
if (($handle = fopen($file, 'r')) !== FALSE) {
$i = 0;
while (($lineArray = fgetcsv($handle, 4000, $delimiter, '"')) !== FALSE) {
for ($j = 0; $j < count($lineArray); $j++) {
$arr[$i][$j] = $lineArray[$j];
}
$i++;
}
fclose($handle);
}
return $arr;
}
$data = csvToArray($feed, ',');
$count = count($data) - 1;
$labels = array_shift($data);
foreach ($labels as $label) {
$keys[] = $label;
}
$keys[] = 'id';
for ($i = 0; $i < $count; $i++) {
$data[$i][] = $i;
}
for ($j = 0; $j < $count; $j++) {
$d = array_combine($keys, $data[$j]);
$newArray[$j] = $d;
}
//Xuat file JSON
for ($i = 0; $i < 5; $i++) {
$json = json_encode($newArray[$i]);
echo $json . "<br />\n";
$filename = $data[$i][1] . ".json";
echo $filename . "<br />\n";
//echo "title[" . $data[$i][4] . "]";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
}
?>
If recommend = 2,it will export line whre id=2.
I want to export id,product_id,title,outline to JSON file.This is sample JSON file:
http://img839.imageshack.us/img839/5277/21527799.png
This is CSV file :
http://img690.imageshack.us/img690/1526/43004273.png
You are looping through all 6 lines of the csv file and saving them to the JSON file:
for ($i = 0; $i < 5; $i++) {
...
}
If you know what row number it is you want, don't loop through every row - just call the code inside the loop for that row. For example, if you wanted the 2nd row:
$row_we_want = 1;
$json = json_encode($newArray[$row_we_want]);
$filename = $data[$row_we_want][1] . ".json";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
If you are unsure about which row it is and need to check each one, you could do so in the loop:
for ($i = 0; $i < 5; $i++) {
if (some_condition) {
$json = json_encode($newArray[$i]);
$filename = $i][1] . ".json";
$handle = fopen("./" . $filename, "w");
fwrite($handle, $json);
fclose($handle);
}
}
Also it might be worth replacing 5 in your loop with the length of the array using the count function if it is not always a fixed length.
I'm using a simple function to write write arrays to a CSV-file, which look like this:
function writeToCSV($array) {
$fp = fopen('programmes.csv', 'a');
fputcsv($fp, $array);
fclose($fp);
}
Simple as a pie. However, is there anyway to know what line-number the pointer is at? Because I want to be able to after 1000 lines to begin writing to a new file. Why? Because I need to be able to import them to a database later with some memory constraints, and to parse a CSV-file with 15000 lines is a no-no.
function writeToCSV($array) {
$i = 1;
$j = 1;
$fp = fopen('programmes' . $j . '.csv', 'a');
foreach($array as $fields) {
if ($i % 1000 == 0) {
fclose($fp);
$fp = fopen('programmes' . $j . '.csv', 'a');
$j = $j + 1;
}
fputcsv($fp, $fields);
$i = $i + 1;
}
fclose($fp);
}
Try this:
count(file('programmes.csv'));
This will give you the number of lines in a file.
I haven't tried if this works, but i would do something like this:
<?php
function writeToCSV($array) {
// count lines in the current file
$linecount = 0;
$fh = fopen('programmes.csv','rb') or die("ERROR OPENING DATA");
while (fgets($fh) !== false) $linecount++;
fclose($fh);
$aSize = sizeof($array);
if (($linecount + $aSize) > 1000) {
// split array
$limit = 1000 - $linecount;
$a = array_slice($array, 0, $limit);
$b = array_slice($array, $limit);
// write into first file
$fp = fopen('programmes.csv', 'a');
foreach($a as $field) fputcsv($fp, $field);
fclose($fp);
// write into second file
$fp = fopen('programmes2.csv', 'a');
foreach($b as $field) fputcsv($fp, $field);
fclose($fp);
} else {
$fp = fopen('programmes.csv', 'a');
$idx = 0;
while ($linecount < 1000) {
// fill the file to the 1000 lines
fputcsv($fp, $array[$idx]);
++$linecount;
++$idx;
}
fclose($fp);
if ($idx != $aSize) {
// create new file
$fp = fopen('programmes.csv', 'a');
while ($idx< $aSize) {
// fill the file to the 1000 lines
fputcsv($fp, $array[$idx]);
++$idx;
}
fclose($fp);
}
}
}
?>