Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
These are my databases:
database.csv:
barcode, Name , Qty ,Code
123456 ,Rothmans Blue , 40 ,RB44
234567 ,Rothmans Red , 40 ,RB30
345678 ,Rothmans Green , 40 ,RB20
456789 ,Rothmans Purple, 40 ,RB10
567890 ,Rothmans Orange, 40 ,RB55
stocktakemain.csv:
barcode, Name , Qty ,Code,ScQty
123456 ,Rothmans Blue , 40 ,RB44, 1
234567 ,Rothmans Red , 40 ,RB30, 1
Process:
The website has an input scan that is posted to "barcode".
It will check that the barcode exists within 'database.csv'
IF the barcode exists and is NOT within the 'stocktakemain.csv', it will add it to 'stocktakemain.csv' with a ScQty of 1. Please see Section 2 within code below.
ELSE when an existing barcode within stocktakemain.csv is scanned, append 'stocktakemain.csv' with +1 (an addition of 1) to ScQty for that particular line.
Bolded above is not working
Code:
function searchForBarcode($id, $array)
{
foreach ($array as $key => $val)
{
if (in_array($id, $val))
{
return $key;
}
}
return null;
}
$post = $_POST["barcode"];
$dbcsv = fopen('databases/database.csv', 'r');
$csvArray = array();
while(! feof($dbcsv))
{
$csvArray[] = fgetcsv($dbcsv);
}
fclose($dbcsv);
$searchf = searchForBarcode($post, $csvArray);
$result = $csvArray[$searchf];
$final = $result[+3];
if ($searchf !== NULL){
$stcsv = fopen('databases/stocktakemain.csv', 'r+');
$stArray = array();
while(! feof($stcsv))
{
$stArray[] = fgetcsv($stcsv);
}
fclose($stcsv);
$searchs = searchForBarcode($post, $stArray);
if ($searchs === NULL) {
$filew = 'databases/stocktakemain.csv';
$write = file_get_contents($filew);
$write .= print_r(implode(",",$result), true).",1\n";
file_put_contents($filew, $write);
}
else {
$filew = 'databases/stocktakemain.csv';
$resultexisting = $stArray[$searchs];
print_r($resultexisting);
echo "<br/>";
$getfilecont = file_get_contents($filew);
$getfilecont = trim($getfilecont);
$existing = explode(",", $getfilecont);
$existing[4] = trim($existing[4]);
++$existing[4];
print_r($existing);
echo "<br/>";
$writeto = print_r(implode(",",$existing), true);
print_r($writeto);
file_put_contents($filew, $writeto);
}
}
Here's some conclusions I've made from reading your code:
The else block is executed if an item is scanned that is already in the stocktakemain.csv file
$searchs contains the index of the row of the item that was scanned
$stArray contains a 2D array of the stocktakemain.csv contents - the first index is the line number, starting at 0, and the next index is the column number
Based on this, I think you need to rewrite your else block to be something like:
$scQtyColumn = 4;
// what is the current quantity?
$scQty = intval($stArray[$searchs][$scQtyColumn]);
// update the quantity in the stocktakemain.csv contents array
$stArray[$searchs][$scQtyColumn] = $scQty + 1;
// write each line to file
$output = fopen('databases/stocktakemain.csv', 'w');
foreach($stArray, $line) {
fputcsv($output, $line);
}
Could you try that out and see if it does the trick?
Related
I have a CSV file with more than 100.000 lines, each line has 3 values separated by semicolon. Total filesize is approx. 5MB.
CSV file is in this format:
stock_id;product_id;amount
==========================
1;1234;0
1;1235;1
1;1236;0
...
2;1234;3
2;1235;2
2;1236;13
...
3;1234;0
3;1235;2
3;1236;0
...
We have 10 stocks which are indexed 1-10 in CSV. In database we have them saved as 22-31.
CSV is sorted by stock_id, product_id but I think it doesn't matter.
What I have
<?php
session_start();
require_once ('db.php');
echo '<meta charset="iso-8859-2">';
// convert table: `CSV stock id => DB stock id`
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
$sql = $mysqli->query("SELECT product_id FROM table WHERE fielddef_id = 1");
while ($row = $sql->fetch_assoc()) {
$products[$row['product_id']] = 1;
}
$csv = file('export.csv');
// go thru CSV file and prepare SQL UPDATE query
foreach ($csv as $row) {
$data = explode(';', $row);
// $data[0] - stock_id
// $data[1] - product_id
// $data[2] - amount
if (isset($products[$data[1]])) {
// in CSV are products which aren't in database
// there is echo which should show me queries
echo " UPDATE t
SET value = " . (int)$data[2] . "
WHERE fielddef_id = " . (int)$stocks[$data[0]] . " AND
product_id = '" . $data[1] . "' -- product_id isn't just numeric
LIMIT 1<br>";
}
}
Problem is that writing down 100k lines by echo is soooo slow, takes long minutes. I'm not sure what MySQL will do, if it will be faster, or take ± the same time. I have no testing machine here, so I'm worry about testing in on prod server.
My idea was to load CSV file into more variables (better array) like below, but I don't know why.
$csv[0] = lines 0 - 10.000;
$csv[1] = lines 10.001 - 20.000;
$csv[2] = lines 20.001 - 30.000;
$csv[3] = lines 30.001 - 40.000;
etc.
I found eg. Efficiently counting the number of lines of a text file. (200mb+), but I'm not sure how it can help me.
When I replace foreach for print_r, I get dump in < 1 sec. The task is to make the foreach loop with database update faster.
Any ideas how to updates so many records in database?
Thanks.
Something like this (please note this is 100% untested and off top of my head may need some tweaking to actually work :) )
//define array may (probably better ways of doing this
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
$handle = fopen("file.csv", "r")); //open file
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
//loop through csv
$updatesql = "UPDATE t SET `value` = ".$data[2]." WHERE fielddef_id = ".$stocks[$data[0]]." AND product_id = ".$data[1];
echo "$updatesql<br>";//for debug only comment out on live
}
There is no need to do your initial select since you're only ever setting your product data to 1 anyway in your code and it looks from your description that your product id's are always correct its just your fielddef column which has the map.
Also just for live don't forget to put your actual mysqli execute command in on your $updatesql;
To give you a comparison to actual usage code (I can benchmark against!)
This is some code I use for an importer of an uploaded file (its not perfect but it does its job)
if (isset($_POST['action']) && $_POST['action']=="beginimport") {
echo "<h4>Starting Import</h4><br />";
// Ignore user abort and expand time limit
//ignore_user_abort(true);
set_time_limit(60);
if (($handle = fopen($_FILES['clientimport']['tmp_name'], "r")) !== FALSE) {
$row = 0;
//defaults
$sitetype = 3;
$sitestatus = 1;
$startdate = "2013-01-01 00:00:00";
$enddate = "2013-12-31 23:59:59";
$createdby = 1;
//loop and insert
while (($data = fgetcsv($handle, 10000, ",")) !== FALSE) { // loop through each line of CSV. Returns array of that line each time so we can hard reference it if we want.
if ($row>0) {
if (strlen($data[1])>0) {
$clientshortcode = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[0])));
$sitename = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[0]))." ".trim(stripslashes($data[1])));
$address = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[1])).",".trim(stripslashes($data[2])).",".trim(stripslashes($data[3])));
$postcode = mysqli_real_escape_string($db->mysqli,trim(stripslashes($data[4])));
//look up client ID
$client = $db->queryUniqueObject("SELECT ID FROM tblclients WHERE ShortCode='$clientshortcode'",ENABLE_DEBUG);
if ($client->ID>0 && is_numeric($client->ID)) {
//got client ID so now check if site already exists we can trust the site name here since we only care about double matching against already imported sites.
$sitecount = $db->countOf("tblsites","SiteName='$sitename'");
if ($sitecount>0) {
//site exists
echo "<strong style=\"color:orange;\">SITE $sitename ALREADY EXISTS SKIPPING</strong><br />";
} else {
//site doesn't exist so do import
$db->execute("INSERT INTO tblsites (SiteName,SiteAddress,SitePostcode,SiteType,SiteStatus,CreatedBy,StartDate,EndDate,CompanyID) VALUES
('$sitename','$address','$postcode',$sitetype,$sitestatus,$createdby,'$startdate','$enddate',".$client->ID.")",ENABLE_DEBUG);
echo "IMPORTED - ".$data[0]." - ".$data[1]."<br />";
}
} else {
echo "<strong style=\"color:red;\">CLIENT $clientshortcode NOT FOUND PLEASE ENTER AND RE-IMPORT</strong><br />";
}
fcflush();
set_time_limit(60); // reset timer on loop
}
} else {
$row++;
}
}
echo "<br />COMPLETED<br />";
}
fclose($handle);
unlink($_FILES['clientimport']['tmp_name']);
echo "All Imports finished do not reload this page";
}
That imported 150k rows in about 10 seconds
Due to answers and comments for the question, I have the solution. The base for that is from #Dave, I've only updated it to pass better to question.
<?php
require_once 'include.php';
// stock convert table (key is ID in CSV, value ID in database)
$stocks = array(
1 => 22,
2 => 23,
3 => 24,
4 => 25,
5 => 26,
6 => 27,
7 => 28,
8 => 29,
9 => 30,
10 => 31
);
// product IDs in CSV (value) and Database (product_id) are different. We need to take both IDs from database and create an array of e-shop products
$sql = mysql_query("SELECT product_id, value FROM cms_module_products_fieldvals WHERE fielddef_id = 1") or die(mysql_error());
while ($row = mysql_fetch_assoc($sql)) {
$products[$row['value']] = $row['product_id'];
}
$handle = fopen('import.csv', 'r');
$i = 1;
while (($data = fgetcsv($handle, 1000, ';')) !== FALSE) {
$p_id = (int)$products[$data[1]];
if ($p_id > 0) {
// if product exists in database, continue. Without this condition it works but we do many invalid queries to database (... WHERE product_id = 0 updates nothing, but take a time)
if ($i % 300 === 0) {
// optional, we'll see what it do with the real traffic
sleep(1);
}
$updatesql = "UPDATE table SET value = " . (int)$data[2] . " WHERE fielddef_id = " . $stocks[$data[0]] . " AND product_id = " . (int)$p_id . " LIMIT 1";
echo "$updatesql<br>";//for debug only comment out on live
$i++;
}
}
// cca 1.5sec to import 100.000k+ records
fclose($handle);
Like I said in the comment, use SPLFileObject to iterate over the CSV file. Use Prepared statements to reduce performance overhead of calling the UPDATE in each loop. Also, merge your two queries together, there isn't any reason to pull all of the product rows first and check them against the CSV. You can use a JOIN to ensure that only those stocks in the second table that are related to the product in the first and that is the current CSV row will get updated:
/* First the CSV is pulled in */
$export_csv = new SplFileObject('export.csv');
$export_csv->setFlags(SplFileObject::READ_CSV | SplFileObject::DROP_NEW_LINE | SplFileObject::READ_AHEAD);
$export_csv->setCsvControl(';');
/* Next you prepare your statement object */
$stmt = $mysqli->prepare("
UPDATE stocks, products
SET value = ?
WHERE
stocks.fielddef_id = ? AND
product_id = ? AND
products.fielddef_id = 1
LIMIT 1
");
$stmt->bind_param('iis', $amount, $fielddef_id, $product_id);
/* Now you can loop through the CSV and set the fields to match the integers bound to the prepared statement and execute the update on each loop. */
foreach ($export_csv as $csv_row) {
list($stock_id, $product_id, $amount) = $csv_row;
$fielddef_id = $stock_id + 21;
if(!empty($stock_id)) {
$stmt->execute();
}
}
$stmt->close();
Make the query bigger, i.e. use the loop to compile a larger query. You may need to split it up into chunks (e.g. process 100 at a time), but certainly don't do one query at a time (applies for any kind, insert, update, even select if possible). This should greatly increase the performance.
It's generally recommended that you don't query in a loop.
Updating every record every time will be too expensive (mostly due to seeks, but also from writing).
You should TRUNCATE the table first and then insert all the records again (assuming you won't have external foreign keys linking to this table).
To make it even faster, you should lock the table before the insert and unlock it afterwards. This will prevent the indexing from happening at every insert.
My PHP code snippet is as follows:
$sql = " SELECT * FROM ".TBL_QUESTIONS." WHERE question_subject_id=".$subject_id;
$sql .= " AND question_topic_id=".$topic_id;
$this->mDb->Query($sql);
$questions_data = $this->mDb->FetchArray();
$questions = $questions_data;
$exclude_words = array('which','who','what','how','when','whom','wherever','the');
/*This loop removes all the words of $exclude_words array from all questions*/
foreach($questions as $index=>$arr) {
$questions_array = explode(' ',$arr['question_text']);
$clean_questions = array_diff($questions_array, $exclude_words);
$questions[$index]['question_text'] = implode(' ',$clean_questions );
}
/*Now the actual comparison of each question with every other question stats here*/
foreach ($questions as $index=>$outer_data) {
$questions_data[$index]['similar_questions_ids_and_percentage'] = Array();
$outer_question = $outer_data['question_text'];
$qpcnt = 0;
foreach ($questions as $inner_data) {
/*This is to avoid comparing the question with itself*/
if ($outer_data['question_id'] != $inner_data['question_id']) {
$inner_question = $inner_data['question_text'];
/*This is to calculate percentage of match between each question with every other question*/
/*In this loop I want single time comparison of each question with every other question Now it's getting repeated, please help me here*/
$same_chars = similar_text($outer_question, $inner_question, $percent);
$percentage = number_format((float)$percent, 2, '.', '');
/*If $percentage is >= $percent_match only then push the respective question_id into an array*/
if($percentage >= 50) {
$questions_data[$index]['similar_questions_ids_and_percentage'][$qpcnt]['question_id'] = $inner_data['question_id'];
$questions_data[$index]['similar_questions_ids_and_percentage'][$qpcnt]['percentage'] = $percentage;
$qpcnt++;
}
}
}
}
Actually I want to avoid the re-comparison in a inner foreach loop from above code.
For example suppose there are ten questions and each one is comparing with other all remaining questions. If Q. No.1 is compared with Q. No. 8 then again when the time comes for Q.8, it's getting compared with Q.1 again. I want to avoid this. I want that if Q.1 is compared with Q.8 then again it should not be compared with Q. No.1 when the turn for Q. 8 comes.
Can anyone please help me out in this regard? any kind of help would be hoghly appreciated.
Maybe you can update the second foreach to also get the index:
foreach ($questions as $inner_data) {
changed to:
foreach ($questions as $secondIndex=>$inner_data){
and just under it add:
if ($secondIndex <= $index)
continue;
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I made a code that reads a table from another website and writes it on mine. Now I want to read just specific rows/columns and write it on my site. The table is filled with weather data and it refreshes every 5 minutes. I need only values for full and half hours and not all the values in the row, but just temperature. For example, there's a row for every five minutes containing temperature value, humidity, sun radiation etc. I need to find a value of, let's say 05:00, and read/write only temperature column of that row. In this case it would be: 05:00 12,5°C. And I need 48 values, because there's 24 hours per day and including another 24 half hours it's 48 all together, right..
This is a part of my code:
<?php
$trazi = ':00';
$citaj = file('proba.txt');
foreach($citaj as $linija)
{
if(strpos($linija, $trazi) !== false)
echo $linija;
}
$traziURL = "somepage";
$stranica = file_get_contents($traziURL);
$tablica = '/(<table.*<\/table>)/s';
preg_match_all($tablica, $stranica, $zeit);
echo $zeit[0][0];
$ime = "proba.txt";
$table = fopen($ime, 'w') or die ("Error!");
$podaci = $zeit[0][0];
fwrite($table, $podaci);
fclose($table);
?>
There's a chance that it won't work for you 'cause some parts are missing, but just to give you the idea.
I'm sure there are multiple other ways to do this, but I'd do it like this.
<?php
/**
* #author Bart Degryse
* #copyright 2013
*/
function getData() {
//Get the html page
$url = "http://www.essen-wetter.de/table.php";
$content = file_get_contents($url);
//Turn it into a dom document searchable by xpath
$dom = new DOMDocument();
$dom->loadHTML($content);
$xpath = new DOMXPath($dom);
//Get field names
$query = "//tr/td[position()=1 and normalize-space(text()) = 'Zeit']";
$entries = $xpath->query($query);
$entry = $entries->item(0);
$tr = $entry->parentNode;
foreach ($tr->getElementsByTagName("td") as $td) {
$fieldnames[] = $td->textContent;
}
//Get field data
$query = "//tr/td[position()=1 and (substring-after(normalize-space(text()),':') = '00' or substring-after(normalize-space(text()),':') = '30')]";
$entries = $xpath->query($query);
foreach ($entries as $entry) {
$fieldvalues = array();
$tr = $entry->parentNode;
foreach ($tr->getElementsByTagName("td") as $td) {
$fieldvalues[] = $td->textContent;
}
$data[] = array_combine($fieldnames, $fieldvalues);
}
//Return data set
return $data;
}
//Gather the data
$data = getData();
//Do something with it
echo "<pre>\n";
foreach ($data as $row) {
echo "Temperature at {$row['Zeit']} was {$row['Temperatur']}.\n";
}
echo "</pre><hr><pre>\n";
print_r($data);
echo "</pre>\n";
?>
If you're going to display the data on a UTF-8 compatible terminal or on a web page that's declared as being UTF-8 encoded this should do it.
If you're want to use single-byte ISO-8859-1 encoding however you'll have to change this line:
$fieldnames[] = $td->textContent;
into this:
$fieldvalues[] = utf8_decode($td->textContent);
Remark
Please note that while doing this is technically not that hard legally you're on loose ground. The data on that page is copyrighted and owned by Markus Wolter. Using his data for your own purposes without his consent is considered theft.
I have a requirement where I need to check a pipe | in the database. If found I need to play around differently.
Here how my db table looks like //Please check the | character in row 11
And if I run a group by sql command myresult will be
Which is correct.
But my requirement is to break the | in any cell and give the count accordingly. The expected result as
Can this be done using MySQL commands alone or do I need to use some php script as well?
Any snippet will be helpful.
Hope this script might help u
$frt =array();
$stmt = $mysqli->prepare("select `fruits` from `meva`") or $mysqli->error ;
$stmt->execute();
$stmt->bind_result($fruits);
while ($stmt->fetch()) {
$frt[]=$fruits;
}
// var_dump($frt); //check all the fruits is in array
$res = array();
$tot = count($frt);
for($i=0;$i<=$tot;$i++)
{
if(preg_match("/\|/", $frt[$i]))
{
$res[] =explode( '|', $frt[$i]);
}else
{
$res[] = $frt[$i];
}
}
// var_dump($res);
$it = new RecursiveIteratorIterator(new RecursiveArrayIterator($res));
foreach($it as $v) {
$ary[]=$v;
}
$all_fruits = array();
$tot_ary = count($ary);
for($io=0;$io<=$tot_ary;$io++)
{
if(isset($ary[$io])!='')
{
$all_fruits[] = trim($ary[$io]);
}else
{
continue;
}
}
// var_dump($all_fruits);
$newArray = array_count_values($all_fruits);
foreach ($newArray as $key => $value) {
echo "$key - <strong>$value</strong> <br />";
}
I think you should fix your data. You can run these two statements in a row until all the data is fixed:
INSERT INTO meva (fruits)
SELECT SUBSTR(fruits, LOCATE('|', fruits) - 1) FROM meva
WHERE LOCATE('|', fruits) > 0;
UPDATE meva
SET fruits = SUBSTR(fruits, LOCATE('|', fruits) + 1)
WHERE LOCATE('|', fruits) > 0;
This will fix the table.
However, if it is your interview question (or a school assignment) just to count from the table as it is, then you can only do it if you know the maximum number of pipes in a given row.
So, if the maximum number of pipes in a row is 1, then your select statement would be:
SELECT count(*),
CASE WHEN LOCATE('|', fruits) > 0 THEN SUBSTR(fruits, LOCATE('|', fruits) - 1) ELSE fruits END
FROM meva
GROUP BY CASE WHEN LOCATE('|', fruits) > 0 THEN SUBSTR(fruits, LOCATE('|', fruits) - 1) ELSE fruits END
If you can have more than one pipe in a row, then your CASE statement will be more complex
Actually the best solution is to change your data structure. This current structure is not recommended. each 'cell' has to contain only one value. If you need to store several fuirts for a specific ID, use
id fruit
11 Apple
11 Mango
this might require some adjustments to your code / tables, but it will prevent the need for more future hacks.
You can use php and do it like below
$query = mysql_query("SELECT fruit FROM meva");
$cnt_array = array();
while($row = mysql_fetch_assoc($query)){
$fruits = $row["fruit"];
$fruit = explode("|", $fruits);
foreach($fruit as $fru){
if(array_key_exists($fru,$cnt_array)){
$cnt_array[$fru] = $cnt_array[$fru]+1;
}
else{
$cnt_array[$fru] = 1;
}
}
}
print_r($cnt_array);
NOTE : This code is not tested,please try it and edit accordingly
I have a csv file in a format resembling the following. There are
no column heads in the actual file - They are shown here for clarity.
id|user|date|description
0123456789|115|2011-10-12:14:29|bar rafael
0123456789|110|2012-01-10:01:34|bar rafael
0123456902|120|2011-01-10:14:55|foo fighter
0123456902|152|2012-01-05:07:17|foo fighter
0123456902|131|2011-11-21:19:48|foo fighter
For each ID, I need to keep the most recent record only, and write
the results back to the file.
The result should be:
0123456789|110|2012-01-10:01:34|bar rafael
0123456902|152|2012-01-05:07:17|foo fighter
I have looked at the array functions and don't see anything that
will do this without some kind of nested loop.
Is there a better way?
const F_ID = 0;
const F_USER = 1;
const F_DATE = 2;
const F_DESCRIPTION = 3;
$array = array();
if (($handle = fopen('test.csv', 'r')) !== FALSE) {
while (($data = fgetcsv($handle, 1000, '|')) !== FALSE) {
if (count($data) != 4)
continue; //skip lines that have a different number of cells
if (!array_key_exists($data[F_ID], $array)
|| strtotime($data[F_DATE]) > strtotime($array[$data[F_ID]][F_DATE]))
$array[$data[F_ID]] = $data;
}
}
You'll have, in $array, what you want. You can write it using fputcsv.
NOTE. I didn't test this code, it's meant to provide a basic idea of how this would work.
The idea is to store the rows you want into $array, using the first value (ID) as the key. This way, on each line you read, you can check if you already have a record with that ID, and only replace it if the date is more recent.
Each time you encounter a new id, put it in your $out array. If the id already exists, overwrite it if the value is newer. Something like:
$in_array = file('myfile.txt');
$out_array = array();
$fields = array('id', 'user', 'date', 'description');
foreach($in_array as $line) {
$row = array_combine($fields, explode('|', $line) );
//New id? Just add it.
if ( !isset($out_array[ $row['id'] ]) ) {
$out_array[ $row['id'] ] = $row;
}
//Existing id? Overwrite if newer.
else if (strcmp( $row['date'], $out_array[ $row['id'] ]['date'] ) > 0 ) {
$out_array[ $row['id'] ] = $row;
}
//Otherwise ignore
}
//$out_array now has the newest row for each id, keyed by id.