Im working with someone else's code.
lets say I have an id have something like this in a csv text file(no header this is just an example and is imported into a database)
|id|partNumber|vendorNumber|dateModified|
123|18-302|fe32l|8/27/2020
123|18-302|fe32l|8/22/2020
123|18-302|fe32l|8/27/2021
321|18-3032|fe32l|8/27/2020
321|18-3032|fe32l|5/24/2022
My conclusion is because it gabs everything, rather than the newest, as a result it doesn't import the newest information.
Question
How do I modify the text file it creates to only write then newest dates for each id.(which is the first and last field in the text file). It can even delete the fields after it writes it.
I think the file is created within lines 253-297
protected function writeImportFile($tmpFile)
{
$data = file_get_contents($tmpFile);
$data = str_replace('\\', '', $data);
$data = str_replace("\t", '', $data);
file_put_contents($tmpFile, $data);
unset($data);
$stopwatch = new Stopwatch();
$stopwatch->start('write');
$tmpDir = $this->tmpDir;
$fileImport = $tmpDir . $this->tableName . '.txt';
$fileRead = new \SplFileObject($tmpFile);
$reader = new CsvReader($fileRead, '|');
$reader->setHeaderRowNumber(0);
$writer = new CsvWriter('|');
$writer->setStream(fopen($fileImport, 'w'));
$numLines = $reader->count();
$i=0;
$this->modColumns();
foreach ($reader as $row) {
$newRowArray = $this->modRow($row);
if (!empty($newRowArray)) {
$writer->writeItem($newRowArray);
}
$i++;
if($i%1000 == 0){
$this->writeLog(' import file has wrote '.$i.' of '.$numLines);
}
}
unlink($tmpFile);
$exTime = $stopwatch->stop('write');
$timeMil = $exTime->getDuration();
$this->writeLog(' Time to write import file for ' . $this->tableNameSw . ' ' . gmdate("H:i:s", $timeMil / 1000));
return $fileImport;
}
Related
I was working on the eCommerce site and uploading the CSV file in the PHP database of Size that having the comma. The outcome of the result is the database is coming up with backward slash and double-quotes.
Please help me in rectifying that issue as had wasted my two days working on it.
CSV Format in notepad
Product Name,Footware Size
Shirt,"""35,36,34"""
Image of my csv file
CSV File
But it saved in the table
Table Screenshot
Code OF upload CSV File into the database
if($_FILES['csv_file']['name'])
{
$filename = explode(".", $_FILES['csv_file']['name']);
if(end($filename) == "csv")
{
$handle = fopen($_FILES['csv_file']['tmp_name'], "r");
$find_header = 0;
while($data = fgetcsv($handle,6000,",",'"'))
{
$find_header++;
if($find_header > 1){
$name = $database->escape_string($data[0]);
$foot_size = trim(addslashes($data[2]), '"');;
$products = new Product();
$products->product_name = $name;
$products->created_at = $time;
$products->updated_at = $time;
$result = $products->save();
if($result){
$product_id = $products->id;
if(!empty($foot_size)){
$sizes = explode(',', $foot_size);
$size_str = '';
foreach($sizes as $size){
$size_str .= $size.',';
}
$p_size = rtrim($size_str,",");
$product_size = new FootSize();
$product_size->product_id = $product_id;
$product_size->foot_size = $p_size;
$product_size->date = $time;
$product_size->save();
}
}
}
}
if($result === true){
$session->message('Product File Uploaded Successfully.');
fclose($handle);
redirect_to('add_product_csv');
}
}
else
{
$message = '<label class="text-danger">Please Select CSV File only</label>';
}
}
Problem
You have an error in this line:
$foot_size = trim(addslashes($data[2]), '"');
What it does is to first escape double quotes:
"35,36,34" --> \"35,36,34\"
And then trim them:
\"35,36,34\" --> \"35,36,34\
Solution
Depending on if you actually want to have the quotes stored in your DB or not, call either trim or addslahes (but not both) or none of the two:
Strip quotes:
$foot_size = trim($data[2], '"');
Keep quotes:
$foot_size = $data[2];
# your framework *might* require explicitly escaping of quote chars:
$foot_size = addslashes($data[2]);
# even better:
$foot_size = $database->escape_string($data[2]);
I'm a beginner level developer learning php.The task that i need to do is upload a 6gb CSV file which contains data, into the data base.I need to access the data i.e reading the file through controller.php file and then splitting that huge CSV file into 10,000 row output CSV files and writing data into those output CSV files. I have been through this task a week already and dint figure it out yet.Would you guys please help me in solving this issue.
<?php
namespace App\Http\Controllers;
use Illuminate\Queue\SerializesModels;
use App\User;
use DateTime;
use Illuminate\Http\Request;
use Storage;
use Validator;
use GuzzleHttp\Client;
use GuzzleHttp\RequestOptions;
use Queue;
use App\model;
class Name extends Controller
{
public function Post(Request $request)
{
if($request->hasfile('upload')){
ini_set('auto_detect_line_endings', TRUE);
$main_input = $request->file('upload');
$main_output = 'output';
$filesize = 10000;
$input = fopen($main_input,'r');
$rowcount = 0;
$filecount = 1;
$output = '';
// echo "here1";
while(!feof($input)){
if(($rowcount % $filesize) == 0){
if($rowcount>0) {
fclose($output);
}
$output = fopen(storage_path(). "/tmp/".$main_output.$filecount++ . '.csv','w');
}
$data = fgetcsv($input);
print_r($data);
if($data) {
fputcsv($output, $data);
}
$rowcount++;
}
fclose($output);
}
}
}
Maybe it's because you are creating a new $output file handler for each iteration.
I've made some adjustments, so that we only create a file when the rowCount = 0 and close it when the fileSize is reached. Also the rowCount has to be reset to 0 each time we close the file.
public function Post(Request $request)
{
if($request->hasfile('upload')){
ini_set('auto_detect_line_endings', TRUE);
$main_input = $request->file('upload');
$main_output = 'output';
$filesize = 10000;
$input = fopen($main_input,'r');
$rowcount = 0;
$filecount = 1;
$output = '';
// echo "here1";
while(!feof($input)){
if ($rowCount == 0) {
$output = fopen('php://output', storage_path(). "/tmp/".$main_output.$filecount++ . '.csv','w');
}
if(($rowcount % $filesize) == 0){
if($rowcount>0) {
fclose($output);
$rowCount = 0;
continue;
}
}
$data = fgetcsv($input);
print_r($data);
if($data) {
fputcsv($output, $data);
}
$rowcount++;
}
fclose($output);
}
}
Here is working example of splitting CSV file by the amount of lines (defined by$numberOfLines). Just set your path in $filePath and run the script in shell for example:
php -f convert.php
script code:
convert.php
<?php
$filePath = 'data.csv';
$numberOfLines = 10000;
$file = new SplFileObject($filePath);
//get header of the csv
$header = $file->fgets();
$outputBuffer = '';
$outputFileNamePrefix = 'datasplit-';
$readLinesCount = 1;
$readlLinesTotalCount = 1;
$suffix=0;
$outputBuffer .= $header;
while ($currentLine = $file->fgets()) {
$outputBuffer .= $currentLine;
$readLinesCount++;
$readlLinesTotalCount++;
if ($readLinesCount >= $numberOfLines) {
$outputFilename = $outputFileNamePrefix . $suffix . '.csv';
file_put_contents($outputFilename, $outputBuffer);
echo 'Wrote ' . $readLinesCount . ' lines to: ' . $outputFilename . PHP_EOL;
$outputBuffer = $header;
$readLinesCount = 0;
$suffix++;
}
}
//write remainings of output buffer if it is not empty
if ($outputBuffer !== $header) {
$outputFilename = $outputFileNamePrefix . $suffix . '.csv';
file_put_contents($outputFilename, $outputBuffer);
echo 'Wrote (last time)' . $readLinesCount . ' lines to: ' . $outputFilename . PHP_EOL;
$outputBuffer = '';
$readLinesCount = 0;
}
you will not be able to convert such amount of data in one php execution if it is run form web because of the maximum execution time of php scripts that is usually between 30-60sec and there is a reason for that - don't event try to extend it to some huge number. If you want your script to run even for hours you need to call it from command line, but you also can call it similar way from another script (for example the controller you have)
You do that this way:
exec('php -f convert.php');
and that's it.
The controller you have will not be able to tell if the whole data was converted because before that happens it will be terminated. What you can do is to write your own code in convert.php that updates some field in database and other controller in your application can read that and print to the user the progress of the runnig convert.php.
The other approach is to crate job/jobs that you can put in the queue and can be run by job manager process with workers that can take care for the conversion but I think that would be an overkill for your need.
Keep in mind that if you split something and on different location join you may have problem of getting something wrong in that process the method that would assure you that you split, transferred, joined your data successfully is to calculate HASH ie SHA-1 of the whole 6GB file before split, send that HASH to destination where all small parts of data needs to be combined, combine them into one 6GB file, calculate HASH of that file and compare with the one that was send. Keep in mind that each of small parts of your data after splitting has their own header to be CSV file easy to interpret (import), where in the original file you have only one header row.
I'm trying to generate a CSV file on the fly, depending on what the user selects as report output. Retrieving the data and writing it to a file using CakeResponse is done, however I'm struggling to set the file extension to '.csv', the file get downloaded as a normal text file.
CakePHP documentation suggests I do this:
$this->response->type('csv');
..but even this is not working, I'm still getting a text file. Can anyone shed some light? Please note, I'm not looking for new methods to generate a CSV file, I just want to change the extension. Thank you.
This is how I download the file:
$this->response->body($this->constructFileBody($logs));
return $this->response;
This is the method 'constructFileBody', although I think its beyond the scope of this question:
public function constructFileBody($logs = array()){
$content = "";
for($i = 0; $i < count($logs); $i++){
$row = $logs[$i]['EventLog'];
$line = $row['description'] . "," . $row['user'] . "," . $row['affected_user'] . "," . $row['report_title'] . "," . $row['date_created'] . "\n";
$content = $content . $line;
}
return $content;
}
As i saw your code, I don't think you used the header anywhere, try this code:
//create a file
$filename = "export_".date("Y.m.d").".csv";
$csv_file = fopen('php://output', 'w');
header('Content-type: application/csv');
header('Content-Disposition: attachment; filename="'.$filename.'"');
$results = $this->ModelName->query($sql); // This is your sql query to pull that data you need exported
//or
$results = $this->ModelName->find('all', array());
// The column headings of your .csv file
$header_row = array("ID", "Received", "Status", "Content", "Name", "Email", "Source", "Created");//columns you want in csv file
fputcsv($csv_file,$header_row,',','"');
// Each iteration of this while loop will be a row in your .csv file where each field corresponds to the heading of the column
foreach($results as $result)
{
// Array indexes correspond to the field names in your db table(s)
$row = array(
$result['ModelName']['id'],
$result['ModelName']['received'],
$result['ModelName']['status'],
$result['ModelName']['content'],
$result['ModelName']['name'],
$result['ModelName']['email'],
$result['ModelName']['source'],
$result['ModelName']['created']
);
fputcsv($csv_file,$row,',','"');
}
fclose($csv_file);
Now look at your code and get the line of code mine which needs to be replaced.
I need a script that is finding and then replacing a sertain line in a CSV like file.
The file looks like this:
18:110327,98414,127500,114185,121701,89379,89385,89382,92223,89388,89366,89362,89372,89369
21:82297,79292,89359,89382,83486,99100
98:110327,98414,127500,114185,121701
24:82297,79292,89359,89382,83486,99100
Now i need to change the line 21.
This is wat i got so far.
The first 2 to 4 digits folowed by : ar a catergory number. Every number after this(followed by a ,) is a id of a page.
I acces te id's i want (i.e. 82297 and so on) from database.
//test 2
$sQry = "SELECT * FROM artikelen WHERE adviesprijs <>''";
$rQuery = mysql_query ($sQry);
if ( $rQuery === false )
{
echo mysql_error ();
exit ;
}
$aResult = array ();
while ( $r = mysql_fetch_assoc ($rQuery) )
{
$aResult[] = $r['artikelid'];
}
$replace_val_dirty = join(",",$aResult);
$replace_val= "21:".$replace_val_dirty;
// file location
$file='../../data/articles/index.lst';
// read the file index.lst
$file1 = file_get_contents($file);
//strip eerde artikel id van index.lst
$file3='../../data/articles/index_grp21.lst';
$file3_contents = file_get_contents($file3);
$file2 = str_replace($file3_contents, $replace_val, $file1);
if (file_exists($file)) {
echo "The file $filename exists";
} else {
echo "The file $filename does not exist";
}
if (file_exists($file3)) {
echo "The file $filename exists";
} else {
echo "The file $filename does not exist";
}
// replace the data
$file_val = $file2;
// write the file
file_put_contents($file, $file_val);
//write index_grp98.lst
file_put_contents($file3, $replace_val);
mail('info#', 'Aanbieding catergorie geupdate', 'Aanbieding catergorie geupdate');
Can anyone point me in the right direction to do this?
Any help would be appreciated.
You need to open the original file and go through each line. When you find the line to be changed, change that line.
As you can not edit the file while you do that, you write a temporary file while doing this, so you copy over line-by-line and in case the line needs a change, you change that line.
When you're done with the whole file, you copy over the temporary file to the original file.
Example Code:
$path = 'file';
$category = 21;
$articles = [111182297, 79292, 89359, 89382, 83486, 99100];
$prefix = $category . ':';
$prefixLen = strlen($prefix);
$newLine = $prefix . implode(',', $articles);
This part is just setting up the basics: The category, the IDs of the articles and then building the related strings.
Now opening the file to change the line in:
$file = new SplFileObject($path, 'r+');
$file->setFlags(SplFileObject::DROP_NEW_LINE | SplFileObject::SKIP_EMPTY);
$file->flock(LOCK_EX);
The file is locked so that no other process can edit the file while it gets changed. Next to that file, the temporary file is needed, too:
$temp = new SplTempFileObject(4096);
After setting up the two files, let's go over each line in $file and compare if it needs to be replaced:
foreach ($file as $line) {
$isCategoryLine = substr($line, 0, $prefixLen) === $prefix;
if ($isCategoryLine) {
$line = $newLine;
}
$temp->fwrite($line."\n");
}
Now the $temporary file contains already the changed line. Take note that I used UNIX type of EOF (End Of Line) character (\n), depending on your concrete file-type this may vary.
So now, the temporary file needs to be copied over to the original file. Let's rewind the file, truncate it and then write all lines again:
$file->seek(0);
$file->ftruncate(0);
foreach ($temp as $line) {
$file->fwrite($line);
}
And finally you need to lift the lock:
$file->flock(LOCK_UN);
And that's it, in $file, the line has been replaced.
Example at once:
$path = 'file';
$category = 21;
$articles = [111182297, 79292, 89359, 89382, 83486, 99100];
$prefix = $category . ':';
$prefixLen = strlen($prefix);
$newLine = $prefix . implode(',', $articles);
$file = new SplFileObject($path, 'r+');
$file->setFlags(SplFileObject::DROP_NEW_LINE | SplFileObject::SKIP_EMPTY);
$file->flock(LOCK_EX);
$temp = new SplTempFileObject(4096);
foreach ($file as $line) {
$isCategoryLine = substr($line, 0, $prefixLen) === $prefix;
if ($isCategoryLine) {
$line = $newLine;
}
$temp->fwrite($line."\n");
}
$file->seek(0);
$file->ftruncate(0);
foreach ($temp as $line) {
$file->fwrite($line);
}
$file->flock(LOCK_UN);
Should work with PHP 5.2 and above, I use PHP 5.4 array syntax, you can replace [111182297, ...] with array(111182297, ...) in case you're using PHP 5.2 / 5.3.
I made a script a while ago that wrote to a file, I did the same thing here, only added a part to read the file and write it again. What I am trying to achive is quite simple, but the problem is eluding me, I am trying to make my script write to a file basically holding the following information
views:{viewcount}
date-last-visited:{MM/DD/YYYY}
last-ip:{IP-Adress}
Now I have done a bit of research, and tried several methods to reading the data, none have returned anything. My current code is as follows.
<?php
$filemade = 0;
if(!file_exists("stats")){
if(!mkdir("stats")){
exit();
}
$filemade = 1;
}
echo $filemade;
$hwrite = fopen("stats/statistics.txt", 'w');
$icount = 0;
if(filemade == 0){
$data0 = file_get_contents("stats/statistics.txt");
$data2 = explode("\n", $data0);
$data1 = $data_1[0];
$ccount = explode(":", data1);
$icount = $ccount[1] + 1;
echo "<br>icount:".$icount."<br>";
echo "data1:".$data1."<br>";
echo "ccount:".$ccount."<br>";
echo "ccount[0]:".$ccount1[0]."<br>";
echo "ccount[1]:".$ccount1[1]."<br>";
}
$date = getdate();
$ip=#$REMOTE_ADDR;
fwrite($hwrite, "views:" . $icount . "\nlast-viewed:" . $date[5] . "/" . $date[3] . $date[2] . "/" . $date[6] . "\nlast-ip:" . $ip);
fclose($hwrite);
?>
the result is always:
views:1
last-viewed://
last-ip:
the views never go up, the date never works, and the IP address never shows.
I have looked at many sources before finally deciding to ask, I figured I'd get more relevant information this way.
Looking forward to some replies. PHP is my newest language, and so I don't know much.
What I have tried.
I have tried:
$handle_read = fopen("stats/statistics.txt", "r");//make a new file handle in read mode
$data = fgets($handle_read);//get first line
$data_array = explode(":", $data);//split first line by ":"
$current_count = $data_array[1];//get second item, the value
and
$handle_read = fopen("stats/statistics.txt", "r");//make a new file handle in read mode
$pre_data = fread($handle_read, filesize($handle_read));//read all the file data
$pre_data_array = explode("\n", $pre_data);//split the file by lines
$data = pre_data_array[0];//get first line
$data_array = explode(":", $data);//split first line by ":"
$current_count = $data_array[1];//get second item, the value
I have also tried split instead of explode, but I was told split is deprecated and explode is up-to-date.
Any help would be great, thank you for your time.
Try the following:
<?php
if(!file_exists("stats")){
if(!mkdir("stats")) die("Could not create folder");
}
// file() returns an array of file contents or false
$data = file("stats/statistics.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
if(!$data){
if(!touch("stats/statistics.txt")) die("Could not create file");
// Default Values
$data = array("views:0", "date-last-visited:01/01/2000", "last-ip:0.0.0.0");
}
// Update the data
foreach($data as $key => $val){
// Limit explode to 2 chunks because we could have
// IPv6 Addrs (e.x ::1)
$line = explode(':', $val, 2);
switch($key){
case 0:
$line[1]++;
break;
case 1:
$line[1] = date('m/d/Y');
break;
case 2:
$line[1] = $_SERVER['REMOTE_ADDR'];
break;
}
$data[$key] = implode(':', $line);
echo $data[$key]. "<br />";
}
// Write the data back into the file
if(!file_put_contents("stats/statistics.txt", implode(PHP_EOL, $data))) die("Could not write file");
?>