Parse large CSV into mysql db - php

I want to parse very large csv into mysql using php. My idea is to make it with executing 4-5 times the request which will paste data into mysql, after each iteration request should start from rows which has not been pasted yet (with one request iteration script can cover 400 rows). But I don't know to skip rows in csv, which has been already pasted into database. Maybe I should check number of rows in the table then define some variable with this number and make iteration according on it. But I don't know how to operate with csv using foreach, I only have code with "while" from example.
Here is my current code:
public function action_index(){
if(($handle = fopen('data_wpic.csv', 'r')) !== false)
{
$header = fgetcsv($handle);
while(($data = fgetcsv($handle)) !== false)
{
$model = ORM::factory('Drug');
$image_path = $data[18];
if(strlen($image_path) > 5) {
$path= 'drug_images/' . $image_path;
$image = ORM::factory('Image')->remote($path);
if ($image) {
$model->image_id = $image;
unlink($path);
}
}
$model->drugGenericName = $data[17];
$model->drugForm = $data[4];
$model->drugProperties = $data[7];
$model->drugIndication = $data[2];
$model->drugDosage = $data[13];
$model->drugSide = $data[11];
$model->drugContrIndication = $data[12];
$model->drugInteractions = $data[15];
$model->drugSpecial = $data[0];
$model->drugExpiry = $data[3];
$model->drugRealCondition = $data[8];
$model->tradeName = $data[16];
$model->save();
unset($data);
}
fclose($handle);
}
}
I have limited web hosting, thats why I try to solve this routine
If someone knows better aproach - I will be glad to hear him

Related

Encode-Decode issue php (CSV->MySQL)

I'm trying to pull data in CSV format for a server, process and store in a MySQL database:
When I insert into my database, the text is "Atila%27s+Goonies" I want it to be "Atila's Goonies"
I have been reading through all the text encoding topics, but I think I'm missing a very simple function.
My requirement is to have this text and store in the database as a properly formatted (or encoded) string, so I can query later and display online. Database column is collated as "utf8_unicode_ci". I'm not sure if it's relevant because if I want to display on the browser, I cannot get to the correct format either. thanks.
Any help would be appreciated.
My code below:
if (($handle = fopen("alliances.txt", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
$row++;
$alliance_id = $data[0];
$alliance_name = $data[1];
$alliance_points = $data[2];
$alliance_villages = $data[3];
$alliance_members = $data[4];
$alliance_rank = $data[5];
$sql_ia = "insert into alliances (alliance_id,alliance_name, alliance_villages,alliance_members,alliance_rank,timestamp)
values (:alliance_id,:alliance_name, :alliance_villages,:alliance_members,:alliance_rank,:timestamp)";
$st_ia = $DBcon->prepare($sql_ia);
$st_ia->bindParam(':alliance_rank', $alliance_rank,PDO::PARAM_INT);
$st_ia->bindParam(':alliance_id', $alliance_id,PDO::PARAM_INT);
$st_ia->bindParam(':alliance_name',$alliance_name,PDO::PARAM_STR);
$st_ia->bindParam(':alliance_members',$alliance_members,PDO::PARAM_INT);
$st_ia->bindParam(':alliance_villages',$alliance_villages,PDO::PARAM_INT);
$st_ia->bindParam(':timestamp',$timestamp,PDO::PARAM_INT);
$author_name = $ag_row[$agency];
$st_ia->execute();
}
fclose($handle);
}
Try this.
<?php
$encoded = 'Atila%27s+Goonies';
// Your string is URL encoded. Use urldecode() to decode.
$decoded = urldecode($encoded);
var_dump($decoded); // string(15) "Atila's Goonies"

CSV file upload with comma value using Php and mysql?

I was working on the eCommerce site and uploading the CSV file in the PHP database of Size that having the comma. The outcome of the result is the database is coming up with backward slash and double-quotes.
Please help me in rectifying that issue as had wasted my two days working on it.
CSV Format in notepad
Product Name,Footware Size
Shirt,"""35,36,34"""
Image of my csv file
CSV File
But it saved in the table
Table Screenshot
Code OF upload CSV File into the database
if($_FILES['csv_file']['name'])
{
$filename = explode(".", $_FILES['csv_file']['name']);
if(end($filename) == "csv")
{
$handle = fopen($_FILES['csv_file']['tmp_name'], "r");
$find_header = 0;
while($data = fgetcsv($handle,6000,",",'"'))
{
$find_header++;
if($find_header > 1){
$name = $database->escape_string($data[0]);
$foot_size = trim(addslashes($data[2]), '"');;
$products = new Product();
$products->product_name = $name;
$products->created_at = $time;
$products->updated_at = $time;
$result = $products->save();
if($result){
$product_id = $products->id;
if(!empty($foot_size)){
$sizes = explode(',', $foot_size);
$size_str = '';
foreach($sizes as $size){
$size_str .= $size.',';
}
$p_size = rtrim($size_str,",");
$product_size = new FootSize();
$product_size->product_id = $product_id;
$product_size->foot_size = $p_size;
$product_size->date = $time;
$product_size->save();
}
}
}
}
if($result === true){
$session->message('Product File Uploaded Successfully.');
fclose($handle);
redirect_to('add_product_csv');
}
}
else
{
$message = '<label class="text-danger">Please Select CSV File only</label>';
}
}
Problem
You have an error in this line:
$foot_size = trim(addslashes($data[2]), '"');
What it does is to first escape double quotes:
"35,36,34" --> \"35,36,34\"
And then trim them:
\"35,36,34\" --> \"35,36,34\
Solution
Depending on if you actually want to have the quotes stored in your DB or not, call either trim or addslahes (but not both) or none of the two:
Strip quotes:
$foot_size = trim($data[2], '"');
Keep quotes:
$foot_size = $data[2];
# your framework *might* require explicitly escaping of quote chars:
$foot_size = addslashes($data[2]);
# even better:
$foot_size = $database->escape_string($data[2]);

PDF Files from database keep getting corrupted

So I am storing my files in a database. Don't ask why, just know that I am not in control of this. Next, I am able to successfully store them as a hexidecimal representation and then spit them back for display with no problem, but then I attach them to an email using PHPMailer and they get sent properly with the right name and all, but they are corrupted. I will walk you through step by step below so that you know exactly how it is being stored, and this may help me debug my issue. (Please note that all code is paraphrased to save space and only show what is needed)
STEP 1
File is grabbed and then processed
$name = $_FILES['file_data']['name'];
$file = prepareImageDBString($_FILES['file_data']['tmp_name']);
$mime_type = $_FILES['file_data']['type'];
name, file, and mime_type are stored
here is the function prepareImageDBString()
function prepareImageDBString($filepath){
$out = 'null';
$handle = #fopen($filepath, 'r');
if($handle){
$content = #fread($handle, filesize($filepath));
$content = bin2hex($content);
#fclose($handle);
$out = $content;
}
return $out;
}
STEP 2
When the file is being viewed I show it as an embedded object. This file is small so I just posted the whole code. Do note that the file shows up with no problems here.
$q = "SELECT lease_doc_file_data FROM lease_doc_file WHERE lease_doc__id ='".$_GET['id']."'";
$file = "";
foreach($CONN->query($q) as $row){
$file = $row['lease_doc_file_data'];
}
if(!empty($file)){
header("Content-type: application/pdf");
ob_clean();
flush();
echo hextobin($file);
}
Here is the function hextobin()
function hextobin($hexstr){
$n = strlen($hexstr);
$sbin = "";
$i = 0;
while($i < $n){
$a = substr($hexstr,$i,2);
$c = pack("H*", $a);
if ( $i == 0 ){ $sbin = $c; }
else { $sbin .= $c;}
$i += 2;
}
return $sbin;
}
STEP 3
Finally the part where I go to send it as a mailer.
$q = "SELECT lease_doc_file_data, lease_doc_file_name, lease_doc_file_type FROM lease_doc_file WHERE lease_doc__id ='$id'";
$file_data = "";
$file_name = "";
$file_type = "";
foreach($CONN->query($q) as $row){
$file_data = $row['lease_doc_file_data'];
$file_name = $row['lease_doc_file_name'];
$file_type = $row['lease_doc_file_type'];
}
$file_data = hextobin($file_data);
$mail->AddStringAttachment($file_data, $file_name, 'binary', $file_type);
So this is the three step process and I"m not sure where the error is coming from. Hopefully someone can help! Thank you for all help in advance!

Updating CSV Column Values Using UA-Parser Library

I'm using the ua-parser library to identify the device family for a number of user agent strings in a spreadsheet column. The problem I'm running into is that it doesn't seem like my function is really running. The value output for detectAgent($data[2]) is not always accurate.
Here's a code sample. I feel like I must be missing something related to the limitations of creating objects over and over again.
Thanks in advance for any help.
<?php
require_once 'vendor/autoload.php';
use UAParser\Parser;
function detectAgent($ua) {
$parser = Parser::create();
$result = $parser->parse($ua);
return $result->os->family;
}
$input_file = "input.csv";
$output_file = "output.csv";
if (($handle1 = fopen($input_file, "r")) !== FALSE) {
if (($handle2 = fopen($output_file, "w")) !== FALSE) {
while (($data = fgetcsv($handle1, 5000000, ",")) !== FALSE) {
// Alter your data
#print $data . "<br />";
$data[2] = detectAgent($data[2]); //identify browser family
// Write back to CSV format
fputcsv($handle2, $data);
}
fclose($handle2);
}
fclose($handle1);
}
?>
This was a silly mistake. I was writing to the wrong column in $data[2] = detectAgent($data[2]);.
If anyone else runs into the same problem, the code is working now and I've posted an example here.

How can I get the total number of rows in a CSV file with PHP?

Using PHP, how can I get the total number of rows that are in a CSV file? I'm using this method but cannot get it to work properly.
if (($fp = fopen("test.csv", "r")) !== FALSE) {
while (($record = fgetcsv($fp)) !== FALSE) {
$row++;
}
echo $row;
}
Create a new file reference using SplFileObject:
$file = new SplFileObject('test.csv', 'r');
Try to seek to the highest Int PHP can handle:
$file->seek(PHP_INT_MAX);
Then actually it will seek to the highest line it could in the file, there is your last line and the last line + 1 is equals to your total lines:
echo $file->key() + 1;
Tricky, but this will avoid you from loading the file contents into memory, which is a very cool thing to do when dealing with really large files.
Here's another option using file() to read the entire file into an array, automatically parsing new lines etc:
$fp = file('test.csv');
echo count($fp);
Also, since PHP5, you can pass in the FILE_SKIP_EMPTY_LINES... to skip empty lines, if you want to:
$fp = file('test.csv', FILE_SKIP_EMPTY_LINES);
Manual: http://php.net/manual/en/function.file.php
Try
$c =0;
$fp = fopen("test.csv","r");
if($fp){
while(!feof($fp)){
$content = fgets($fp);
if($content) $c++;
}
}
fclose($fp);
echo $c;
I know that this is pretty old, but actually I ran into the same question.
As a solution I would assume to use linux specific logic:
$rows = shell_exec('$(/bin/which cat) file.csv | $(/bin/which tr) "\r" "\n" | $(which wc) -l');
NOTE: this only works for linux only and this only should be used if you are 100% certain that your file has no multiline-cells
CSV rows are separated by line breaks. Therefore, split the rows by line breaks, and you will get an array of rows, which is countable.
if (($fp = fopen("test.csv", "r")) !== FALSE) {
$rows = explode("\n", $fp);
$length = count($rows);
echo $length;
}
Note; none of higher-upvoted solutions that count lines in the file are reliable, as they are only counting the lines, not the csv entries (which can contain newline characters)
I'm using a similar solution to op, and it works perfectly, but with op's code the while part can break on empty lines, which is potentially his problem.
So it looks like this (edited op's code)
$rowCount=0;
if (($fp = fopen("test.csv", "r")) !== FALSE) {
while(!feof($fp)) {
$data = fgetcsv($fp , 0 , ',' , '"', '"' );
if(empty($data)) continue; //empty row
$rowCount++;
}
fclose($fp);
}
echo $rowCount;
I find this the most reliable:
$file = new SplFileObject('file.csv', 'r');
$file->setFlags(
SplFileObject::READ_CSV |
SplFileObject::READ_AHEAD |
SplFileObject::SKIP_EMPTY |
SplFileObject::DROP_NEW_LINE
);
$file->seek(PHP_INT_MAX);
$lineCount = $file->key() + 1;
I know this is an old post, but I've been googling this issue, and found that the only problem with the original code was that you need to define $row outside the while loop, like this:
if (($fp = fopen("test.csv", "r")) !== FALSE) {
$row = 1;
while (($record = fgetcsv($fp)) !== FALSE) {
$row++;
}
Just in case it helps someone :)
echo $row;
}
In case you are getting the file from a form
$file = $_FILES['csv']['tmp_name'];
$fp = new SplFileObject($file, 'r');
$fp->seek(PHP_INT_MAX);
echo $fp->key() + 1;
$fp->rewind();
Works like charm!!!!!!!!!!!!!!!!!!
$filename=$_FILES['sel_file']['tmp_name'];
$file=fopen($filename,"r");
$RowCount=0;
while ((fgetcsv($file)) !== FALSE)
{
$RowCount++;
}
echo $RowCount;
fclose($file);

Categories