code:
while (($row= fgetcsv($file_data, 10000, ",")) !== FALSE)
{
$product_id = date('mdHis');
$data[] = array(
'product_id' => $product_id
);
}
In this code I am importing csv file which work perfectly. Now, When I insert csv file data into my database then I am also insert an id i.e. product_id Now, when I click on submit button then It store same value but I want to store different product_id for a different row. So, How can I do this? Please help me.
Thank You
You may want to just use auto-increment in the database, you can additionally use a date_created column, with the time. A loop is too fast for date() (s is seconds!), but even microtime() would not really make much sense.
If you want to really do this, why ever, in php:
function generateTimeID($start, $format_string) {
while (True) {
yield date($format_string) . $start;
$start ++;
}
}
$time_generator = generateTimeID($last_id_from_database, 'mdHis-');
while (($row= fgetcsv($file_data, 10000, ",")) !== FALSE)
{
$product_id = $time_generator->value();
$time_generator->next();
$data[] = array(
'product_id' => $product_id
);
}
Related
What I am trying to do is Upload a CSV file with Php. The first line is the Column names and below that the data (of course). Each column name can change depends on the end user uploads. So the main column names we need can change spots (A1 or B1 etc...) So lets say the column I need is B1 and I need to get all the data in B. Not sure on how to go by it. So far this is what I have. Any ideas?
ini_set("allow_url_fopen", 1);
$handle = fopen($_FILES['fileToUpload']['tmp_name'], 'r') or die ('cannot open the file');
while(!feof($handle)) {
$data[] = fgetcsv($handle);
}
var_dump($data);
fclose($handle);
UPDATE:
I am importing this file from .CSV to PHP
I need to search for column header that starts with “SKU” and then “COST”
From there once those are found then I want the whole column… B, E. But those column letters can change, depends on how it is being exported by the end user. I do not need the rows, just columns.
Once the file is uploaded into the server, use something like the following code to parse it and actually use it as an array[];
Code:
$filename = "upload/sample.csv";
if (($handle = fopen($filename, 'r')) !== FALSE){
while (($row = fgetcsv($handle, 1000, ",")) !== FALSE){
print_r($row);
}
}
That's one way of doing it, you could also read more about it here.
If you want the value of a specific column for each row then you need to loop through the results and pick it out. It looks like you are getting an array of arrays so...(EDITED to get the column based on the header name):
$header = $data[0];
unset($data[0]); // delete the header row so those values don't show in results
$sku_index = '';
$cost_index = '';
// get the index of the desired columns by name
for($i=0; $i < count($header); $i++) {
if($header[$i] == 'SKU') {
$sku_index = $i;
}
if($header[$i] == 'COST') {
$cost_index = $i;
}
}
// loop through each row and grab the values for the desired columns
foreach($data as $row) {
echo $row[$sku_index];
echo $row[$cost_index];
}
Should get what you want.
I have a CSV file with more than 100.000 lines, each line has 3 values separated by semicolon. Total filesize is approx. 5MB.
CSV file is in this format:
stock_id;product_id;amount
==========================
1;1234;0
1;1235;1
1;1236;0
...
2;1234;3
2;1235;2
2;1236;13
...
3;1234;0
3;1235;2
3;1236;0
...
We have 10 stocks which are indexed 1-10 in CSV. In database we have them saved as 22-31.
CSV is sorted by stock_id, product_id but I think it doesn't matter.
What I have
<?php
session_start();
require_once ('db.php');
echo '<meta charset="iso-8859-2">';
// convert table: `CSV stock id => DB stock id`
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
$sql = $mysqli->query("SELECT product_id FROM table WHERE fielddef_id = 1");
while ($row = $sql->fetch_assoc()) {
$products[$row['product_id']] = 1;
}
$csv = file('export.csv');
// go thru CSV file and prepare SQL UPDATE query
foreach ($csv as $row) {
$data = explode(';', $row);
// $data[0] - stock_id
// $data[1] - product_id
// $data[2] - amount
if (isset($products[$data[1]])) {
// in CSV are products which aren't in database
// there is echo which should show me queries
echo " UPDATE t
SET value = " . (int)$data[2] . "
WHERE fielddef_id = " . (int)$stocks[$data[0]] . " AND
product_id = '" . $data[1] . "' -- product_id isn't just numeric
LIMIT 1<br>";
}
}
Problem is that writing down 100k lines by echo is soooo slow, takes long minutes. I'm not sure what MySQL will do, if it will be faster, or take ± the same time. I have no testing machine here, so I'm worry about testing in on prod server.
My idea was to load CSV file into more variables (better array) like below, but I don't know why.
$csv[0] = lines 0 - 10.000;
$csv[1] = lines 10.001 - 20.000;
$csv[2] = lines 20.001 - 30.000;
$csv[3] = lines 30.001 - 40.000;
etc.
I found eg. Efficiently counting the number of lines of a text file. (200mb+), but I'm not sure how it can help me.
When I replace foreach for print_r, I get dump in < 1 sec. The task is to make the foreach loop with database update faster.
Any ideas how to updates so many records in database?
Thanks.
Something like this (please note this is 100% untested and off top of my head may need some tweaking to actually work :) )
//define array may (probably better ways of doing this
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
$handle = fopen("file.csv", "r")); //open file
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
//loop through csv
$updatesql = "UPDATE t SET `value` = ".$data[2]." WHERE fielddef_id = ".$stocks[$data[0]]." AND product_id = ".$data[1];
echo "$updatesql<br>";//for debug only comment out on live
}
There is no need to do your initial select since you're only ever setting your product data to 1 anyway in your code and it looks from your description that your product id's are always correct its just your fielddef column which has the map.
Also just for live don't forget to put your actual mysqli execute command in on your $updatesql;
To give you a comparison to actual usage code (I can benchmark against!)
This is some code I use for an importer of an uploaded file (its not perfect but it does its job)
if (isset($_POST['action']) && $_POST['action']=="beginimport") {
echo "<h4>Starting Import</h4><br />";
// Ignore user abort and expand time limit
//ignore_user_abort(true);
set_time_limit(60);
if (($handle = fopen($_FILES['clientimport']['tmp_name'], "r")) !== FALSE) {
$row = 0;
//defaults
$sitetype = 3;
$sitestatus = 1;
$startdate = "2013-01-01 00:00:00";
$enddate = "2013-12-31 23:59:59";
$createdby = 1;
//loop and insert
while (($data = fgetcsv($handle, 10000, ",")) !== FALSE) { // loop through each line of CSV. Returns array of that line each time so we can hard reference it if we want.
if ($row>0) {
if (strlen($data[1])>0) {
$clientshortcode = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[0])));
$sitename = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[0]))." ".trim(stripslashes($data[1])));
$address = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[1])).",".trim(stripslashes($data[2])).",".trim(stripslashes($data[3])));
$postcode = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[4])));
//look up client ID
$client = $db->queryUniqueObject("SELECT ID FROM tblclients WHERE ShortCode='$clientshortcode'",ENABLE_DEBUG);
if ($client->ID>0 && is_numeric($client->ID)) {
//got client ID so now check if site already exists we can trust the site name here since we only care about double matching against already imported sites.
$sitecount = $db->countOf("tblsites","SiteName='$sitename'");
if ($sitecount>0) {
//site exists
echo "<strong style=\"color:orange;\">SITE $sitename ALREADY EXISTS SKIPPING</strong><br />";
} else {
//site doesn't exist so do import
$db->execute("INSERT INTO tblsites (SiteName,SiteAddress,SitePostcode,SiteType,SiteStatus,CreatedBy,StartDate,EndDate,CompanyID) VALUES
('$sitename','$address','$postcode',$sitetype,$sitestatus,$createdby,'$startdate','$enddate',".$client->ID.")",ENABLE_DEBUG);
echo "IMPORTED - ".$data[0]." - ".$data[1]."<br />";
}
} else {
echo "<strong style=\"color:red;\">CLIENT $clientshortcode NOT FOUND PLEASE ENTER AND RE-IMPORT</strong><br />";
}
fcflush();
set_time_limit(60); // reset timer on loop
}
} else {
$row++;
}
}
echo "<br />COMPLETED<br />";
}
fclose($handle);
unlink($_FILES['clientimport']['tmp_name']);
echo "All Imports finished do not reload this page";
}
That imported 150k rows in about 10 seconds
Due to answers and comments for the question, I have the solution. The base for that is from #Dave, I've only updated it to pass better to question.
<?php
require_once 'include.php';
// stock convert table (key is ID in CSV, value ID in database)
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
// product IDs in CSV (value) and Database (product_id) are different. We need to take both IDs from database and create an array of e-shop products
$sql = mysql_query("SELECT product_id, value FROM cms_module_products_fieldvals WHERE fielddef_id = 1") or die(mysql_error());
while ($row = mysql_fetch_assoc($sql)) {
$products[$row['value']] = $row['product_id'];
}
$handle = fopen('import.csv', 'r');
$i = 1;
while (($data = fgetcsv($handle, 1000, ';')) !== FALSE) {
$p_id = (int)$products[$data[1]];
if ($p_id > 0) {
// if product exists in database, continue. Without this condition it works but we do many invalid queries to database (... WHERE product_id = 0 updates nothing, but take a time)
if ($i % 300 === 0) {
// optional, we'll see what it do with the real traffic
sleep(1);
}
$updatesql = "UPDATE table SET value = " . (int)$data[2] . " WHERE fielddef_id = " . $stocks[$data[0]] . " AND product_id = " . (int)$p_id . " LIMIT 1";
echo "$updatesql<br>";//for debug only comment out on live
$i++;
}
}
// cca 1.5sec to import 100.000k+ records
fclose($handle);
Like I said in the comment, use SPLFileObject to iterate over the CSV file. Use Prepared statements to reduce performance overhead of calling the UPDATE in each loop. Also, merge your two queries together, there isn't any reason to pull all of the product rows first and check them against the CSV. You can use a JOIN to ensure that only those stocks in the second table that are related to the product in the first and that is the current CSV row will get updated:
/* First the CSV is pulled in */
$export_csv = new SplFileObject('export.csv');
$export_csv->setFlags(SplFileObject::READ_CSV | SplFileObject::DROP_NEW_LINE | SplFileObject::READ_AHEAD);
$export_csv->setCsvControl(';');
/* Next you prepare your statement object */
$stmt = $mysqli->prepare("
UPDATE stocks, products
SET value = ?
WHERE
stocks.fielddef_id = ? AND
product_id = ? AND
products.fielddef_id = 1
LIMIT 1
");
$stmt->bind_param('iis', $amount, $fielddef_id, $product_id);
/* Now you can loop through the CSV and set the fields to match the integers bound to the prepared statement and execute the update on each loop. */
foreach ($export_csv as $csv_row) {
list($stock_id, $product_id, $amount) = $csv_row;
$fielddef_id = $stock_id + 21;
if(!empty($stock_id)) {
$stmt->execute();
}
}
$stmt->close();
Make the query bigger, i.e. use the loop to compile a larger query. You may need to split it up into chunks (e.g. process 100 at a time), but certainly don't do one query at a time (applies for any kind, insert, update, even select if possible). This should greatly increase the performance.
It's generally recommended that you don't query in a loop.
Updating every record every time will be too expensive (mostly due to seeks, but also from writing).
You should TRUNCATE the table first and then insert all the records again (assuming you won't have external foreign keys linking to this table).
To make it even faster, you should lock the table before the insert and unlock it afterwards. This will prevent the indexing from happening at every insert.
I have a csv file in a format resembling the following. There are
no column heads in the actual file - They are shown here for clarity.
id|user|date|description
0123456789|115|2011-10-12:14:29|bar rafael
0123456789|110|2012-01-10:01:34|bar rafael
0123456902|120|2011-01-10:14:55|foo fighter
0123456902|152|2012-01-05:07:17|foo fighter
0123456902|131|2011-11-21:19:48|foo fighter
For each ID, I need to keep the most recent record only, and write
the results back to the file.
The result should be:
0123456789|110|2012-01-10:01:34|bar rafael
0123456902|152|2012-01-05:07:17|foo fighter
I have looked at the array functions and don't see anything that
will do this without some kind of nested loop.
Is there a better way?
const F_ID = 0;
const F_USER = 1;
const F_DATE = 2;
const F_DESCRIPTION = 3;
$array = array();
if (($handle = fopen('test.csv', 'r')) !== FALSE) {
while (($data = fgetcsv($handle, 1000, '|')) !== FALSE) {
if (count($data) != 4)
continue; //skip lines that have a different number of cells
if (!array_key_exists($data[F_ID], $array)
|| strtotime($data[F_DATE]) > strtotime($array[$data[F_ID]][F_DATE]))
$array[$data[F_ID]] = $data;
}
}
You'll have, in $array, what you want. You can write it using fputcsv.
NOTE. I didn't test this code, it's meant to provide a basic idea of how this would work.
The idea is to store the rows you want into $array, using the first value (ID) as the key. This way, on each line you read, you can check if you already have a record with that ID, and only replace it if the date is more recent.
Each time you encounter a new id, put it in your $out array. If the id already exists, overwrite it if the value is newer. Something like:
$in_array = file('myfile.txt');
$out_array = array();
$fields = array('id', 'user', 'date', 'description');
foreach($in_array as $line) {
$row = array_combine($fields, explode('|', $line) );
//New id? Just add it.
if ( !isset($out_array[ $row['id'] ]) ) {
$out_array[ $row['id'] ] = $row;
}
//Existing id? Overwrite if newer.
else if (strcmp( $row['date'], $out_array[ $row['id'] ]['date'] ) > 0 ) {
$out_array[ $row['id'] ] = $row;
}
//Otherwise ignore
}
//$out_array now has the newest row for each id, keyed by id.
I was wondering if any of you guys have tried something about CSV migration in a live form(multipart/form-data) mine is working the only thing I hate about is that it consumes so much of time and it's reaching the maximum execution timeout. The quick fix I made is by setting the maximum execution time in my php.ini(or set_time_limit()) but it's really annoying me to wait for half an hour just to import the whole data though it's not more than 100kb. Am I just overreacting or something?
This is the code:
function upload($id, $old_eid)
{
$filename = $_FILES['event_file']['tmp_name'];
$handle = fopen($filename, "r");
while(($data = fgetcsv($handle, 1000, ",")) !== FALSE){
$id = $id;
$id2 = $data[2];
$ckr = $this->Manager_model->check_if_record_exists($id, $id2);
if(count($ckr) > 0):
$this->session->set_flashdata('err', '<div class="error">Duplicated record</div>');
redirect("manager/csver/$id");
else:
$data['col1'] = $data[0];
$data['col2'] = $id;
$data['col3'] = $data[3].' '.$data[4];
$data['col4'] = $data[2];
$data['col5'] = $data[6];
$data['col6'] = $data[1];
$data['col7'] = $data[7];
$data['col8'] = mt_rand(11111, 99999);
$data['col9'] = $old_eid;
$this->Manager_model->add_csv($data);
$this->Manager_model->add_csv_to_photo($data);
endif;
}
fclose($handle);
$this->session->set_flashdata('success', '<div class="success">CSV successfully uploaded</div>');
redirect("manager/records/$id");
//$this->session->set_flashdata('msg', '<div class="success">Records successfully uploaded</div>');
}
My Manager_model:
function add_csv($data)
{
$src = array(
'col1'=> $data['col1'],
'col2' => $data['col2'],
'col3' => $data['col3'],
'col4' => $data['col4'],
'col5' => $data['col5'],
'col6' => $data['col6'],
'col7' => $data['col7'],
'col8' => $data['col8'],
);
$this->db->insert('e_records2', $src);
if($this->db->affected_rows() == '1'):
return TRUE;
endif;
return FALSE;
}
function add_csv_to_photo($data) {
$src = array(
'col1'=> $data['col1'],
'col2' => $data['col2'],
'col3' => $data['col3'],
'col4' => $data['col4'],
'col5'=> $data['col5'],
'col6'=> $data['col6'],
);
$this->db->insert('e_records', $src);
if($this->db->affected_rows() == '1'):
return TRUE;
endif;
return FALSE;
} function check_if_record_exists($id, $id2)
{
$eid = $id;
$id2 = $id2;
$query = $this->db->query("select * from races_results where eid = $eid AND id2 = $id2");
return $query->result();
}
P.S.
I'm not talking about PhpMyAdmin here cos I know how import csv file works there. And plus it would create a lot of trivial tasks to have a file to migrate using the bone.
Why not run the profiler to optimize your code? Codeigniter includes this useful piece for problems like this http://codeigniter.com/user_guide/general/profiling.html
It will give you a breakdown of your SQL queries and what is taking long, and where.
$this->output->enable_profiler(TRUE);
The problem seems to me that you are querying the DB once (or twice ?) per line in your CSV file.
Of course you're going to get horrible performance.
You can do the whole query in one go and have the DB make the CSV for you in no time.
SELECT DISTINCT f1,f2,f3,... FROM tablex WHERE .. INTO OUTFILE 'c:/dir/ca.csv'
FIELDS ESCAPED BY '"' FIELDS TERMINATED BY ';' LINES TERMINATED BY '\n';
//note the use of forward slashes even on Windows.
See: http://dev.mysql.com/doc/refman/5.0/en/select-into.html
The speed of the select itself is the limiting factor here.
Make sure you have write permissions on the directory and note that MySQL will never overwrite files.
This command is very fast on MySQL.
$id = $id;
really?
$ckr = $this->Manager_model->check_if_record_exists($id, $id2);
One obvious way to make it go faster would be to have a unique index on eid and id2 and ignore duplicate row errors on the INSERT.
But really, f you want it to go much faster, just tell mysql to parse and load the data.
hi
so this is the setup: i need to update some prices from a csv file called pricelist.csv. the database table is called products and there is a column called product_id, which contains the product ids which can also be found in the first column of the csv file and the prices and lastly i need are located in the 7th column of the csv file. i need to write these to the price column of my database.
i have tried my best to come up with the code, but it just seems too much for my skill level. here is what i made:
<?php
include("admin/include/db.php");
$res=mysql_query("select * from products");
$row = 1;
$mycsvfile = array(); //define the main array.
if (($handle = fopen("pricelist.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE)
{
$num = count($data);
$row++;
$mycsvfile[] = $data;
}
fclose($handle);
}
$row['product_id'] = $mycsvfile[$which_row][1] //trying to find the row in the csv
$mycsvfile[$which_row][7] = $price; //should get price, but previous line does not work
while($row=mysql_fetch_array($res))
{
mysql_query("update products set price='".$price."', isavailable='1' where id='".$row['id']."'");
}
?>
any sort of help is welcome! thanks
I think you are looking for this (but I didn't test it):
<?php
include("admin/include/db.php");
if( ( $handle = fopen("pricelist.csv", "r") ) !== FALSE )
{
while( ( $r = fgetcsv( $handle, 1000, ";") ) !== FALSE )
{
mysql_query('UPDATE products SET "price"="'.$r[6].'", "isavailable"="1" where "id"="'.$r[0].'"');
}
}
Disclaimer: Yes I know I didn't sanitize the data, but I don't feel like working with outdated mysql functions.
You can use file() to read the CSV file into an array. Then use str_getcsv() to read each item from the file array and turn it into an array reprensnting a row from the CSV file. Then you pull the data from that array into an update query and run it.
Like this:
$id = 0;
$price = 6;
$csv_file = file('pricelist.csv');
foreach($csv_file as $row)
{
$data = str_getcsv($row);
$query = "UPDATE products SET price = {$data[$price]}, isavailable='1' WHERE `id` = {$data[$id]}";
//run the query
}