I have to INSERT thousand of records.
I use msqli::multi_query() in a loop and want to 'multi_query' block of n query (where 'n' is a parameter).
first INSERT goes ok, the second goes wrong because I have to manage result like this :
while($mysqli->more_results())
{
$mysqli->next_result();
if($res = $mysqli->store_result()) // added closing bracket
{
$res->free();
}
}
The problem is that this chek is slow.
Question is : how can I optimize this bulk INSERT makeing faster the manage of result ?
If you are inserting in a one table just use batch mod in INSERT.
Example:
INSERT INTO TABLE (field1,field2,field3) VALUES (value1,value2,value3),(value4,value5,value6)
Use for or foreach in php to make query and then simply use mysqli_query. Judging by errors you mentioned in comment section, you violating ket restrictions meaning you are trying to insert the same value twice.
If you are inserting in a different tables, you can use multi_query interface.
Prepare one query Insert1;Insert2;
And use multi_query like this
$con->multi_query($query);
while (mysqli_more_results($con)) {
mysqli_use_result($con);
mysqli_next_result($con);
}
Related
I have a function that is designed to copy a product with all attributes with help of sql querys. My problem is to return new_product_id to php after completion.
If i run sql script in phpmyadmin all is working.
If i run sql script with php function all is working.
What i need help with is how to assign mysql-set-variable: #new_product_id from last query to php variable that I want to return.
----- sql query ------
CREATE TEMPORARY TABLE tmptable SELECT * FROM product WHERE id='19' AND site_id='1';
UPDATE tmptable SET id = 0,parent_id='19',status_id='1',name_internal=concat('NEW ',name_internal);
INSERT INTO product SELECT * FROM tmptable;
SET #new_product_id = LAST_INSERT_ID();
DROP TABLE tmptable;
CREATE TEMPORARY TABLE tmptable SELECT * FROM product_abcd WHERE product_id='19' AND site_id='1';
UPDATE tmptable SET product_id = #new_product_id,id=0;
INSERT INTO product_abcd SELECT * FROM tmptable;
DROP TABLE tmptable;
CREATE TEMPORARY TABLE tmptable SELECT * FROM product_efgh WHERE product_id='19' AND site_id='1';
UPDATE tmptable SET product_id = #new_product_id,id=0;
INSERT INTO product_efgh SELECT * FROM tmptable;
DROP TABLE tmptable;
(Here is more correct SQL insert statements)
SELECT #new_product_id AS new_product_id;
----- sql query ------
----- php function (not complete)------
This function is working making a new copy of product, code below is not complete but works so please only focus on multiquery part.
//return 0 for fail or new product_id (!=0) for success
public function copyProduct($data){
$res=0;
//if something, build sql-query as
$sql="sql from above";
//if we have a query to run
if(!empty($sql)){
//this is multi query, use correct function
if ($this->connect()->multi_query($sql) === TRUE) {
//loop it
while ($this->connect()->more_results()){
$result=$this->connect()->next_result();
}//while more results
}//if multiquery ok
return $res;
}//end function copy
----- php function (not complete)------
above code works, i get a nice copy of product with
result =0 for fail and
result 1 for success, (this works)
How i would like it to work is
result= 0 for fail and
result= new_product_id for success
so i can redirect user to the newly created product and therefore save user one click.
Results from query, same from phpmyadmin as from php (all good so far, no incorrect querys at this time)
Mysql returned empty results (no rows) (create temporary table)
1 row affected (update tmpt table)
1 row insert (insert into product)
mysql returned emtpy result (set $new_product_id)
mysql returened empty results (drop tmp table)
mysql returned empty result (create temporary table)
mysql x row affected (update tmp table)
mysql x row affected (insert into table)
mysql returned empty results (drop table tmptable)
mysql returned empty results (create temporary table)
.... N.....
last query "showing rows 0-0 ( 1 total) (select #new_product_id)
new_product_id=25
What have I tried?
I placed the select variable as my final query, i thought it was smart only check last query and assign variable there, but i failed due to php mysqli fetch_assoc is not possible on non object.
so next up was not so bright, i know i have 16 results from mysql and i only need the result from one of them, but anyway i places this inside multiquery
----- php function (not complete)------
This function is working making a new copy of product, NOT WORKING assigning new_product_id
//return 0 for fail or new product_id (!=0) for success
public function copyProduct($data){
$res=0;
//if something, build sql-query as
$sql="sql from above";
//if we have a query to run
if(!empty($sql)){
//this is multi query, use correct function
if ($this->connect()->multi_query($sql) === TRUE) {
//loop it
while ($this->connect()->more_results()){
//insert,update,drop will return false even if sql is ok, this would be sufficient for us now
if ($result = $this->connect()->store_result()) {
$row = $result->fetch_assoc();
if(isset($row["new_product_id"])){
//new return value of newly copied product
$res=$row["new_product_id"];
$result->free();
}
}
$result=$this->connect()->next_result();
}//while more results
}//if multiquery ok
return $res;
}//end function copy
----- php function (not complete)------
Checking other questions on stackoverflow recommended sending multiple normal querys, this seems like a bad solution when multi_query exists.
checking php library for multiquery did me no good, i cant understand how it works, as many others pointed out the documentation seems like a copy from another function.
Remember that multi_query() sends a clump of SQL queries to MySQL server but waits for the execution of only the first one. If you want to execute SQL using multi_query() and get only the result of the last query ignoring the previous ones then you need to perform a blocking loop and buffer the results into PHP array. Iterate over all results waiting for MySQL to process each query and once MySQL responds there are no more results you can keep the last fetched result.
For example, consider this function. It sends a bunch of concatenated SQL queries to the MySQL server and then waits for MySQL to process each query one by one. Every result is fetched into PHP array and the last available array is returned from the function.
function executeMultiQueryAndGetOnlyLastResult(mysqli $mysqli):array {
$mysqli->multi_query('
SELECT "a";
SELECT 2;
SELECT "val";
');
$values = [];
do {
$result = $mysqli->use_result();
if ($result) {
// process the results here
$values = $result->fetch_all();
$result->free();
}
} while ($mysqli->next_result()); // next_result will block and wait for next query to finish on MySQL server
$mysqli->store_result(); // Needed to fetch the error as exception
return $values;
}
Obviously it would be much easier to send each query separately to MySQL instead. multi_query() is very complicated and has very limited use. It can be useful if you have a number of SQL queries which you cannot execute separately via PHP, but most of the time you should be using prepared statements and send each query separately.
Another one bites the dust, I gave up and defined an array of sql querys from 0 to 14 and run it as mysqli->query() instead. Thank you all for comments and your time.
You could try using .multi_query() for all the queries in your operation except the last one, the SELECT that returns the id you want. Then run that SELECT as a single query.
This is a robust solution to your problem: #-variables belong to MySql connections and persist for the lifetimes of those connections.
And, it makes for clean and predictable operation of your software. When you need a result set returned to your program, use a single query.
Function render makes website 500% slow! Can anyone fix that please ?
Someone told me :
because it sends a database request on each iteration of the loop (it's not the only problem with this chunk of code but it's the most taxing one)
Yes I understand what that means. His way is:
you need to get all of the data before you start building the menu,
then you just insert the data instead of requesting more data on each
iteration
But i don't know how i must do it!
<?php
$menu_html='';
function render_menu($parent_id,$actmenuid)
{
$obj = new Database();
$con = $obj->dbconnectt();
global $menu_html;
$result=mysqli_query($con, "select * from tbl_menu where parent_id='$parent_id'");
if(mysqli_num_rows($result)==0) return;
if($parent_id==0){
$menu_html.='<ul class="topnav">';
}else{
$menu_html.='<ul>';
}
while($row=mysqli_fetch_array($result)) {
$childnum = $obj->recordcount("SELECT * FROM tbl_menu WHERE parent_id='".$row['id']."'");
if($childnum == 0){
$linkvalue='/category/'.$row['id'].'.html';
} else{
$linkvalue='#';
}
if($row['id']==$actmenuid && $actmenuid !=NULL){
$actv='class="active"';
}else{
$actv='';
}
$menu_html.='<li '.$actv.'>'.$row['title'].'';
render_menu($row['id'],$actmenuid);
$menu_html.='</li>';
}
$menu_html.='</ul>';return $menu_html;
}
if($isDsh==false){
echo render_menu(0,$actmenuid);
}
?>
Depending on how many records you have, try removing this query from inside the loop since it's running for every record on the first query.
$childnum = $obj->recordcount("SELECT * FROM tbl_menu WHERE parent_id='".$row['id']."'");
Change it a single query like this where it returns counts for each parent idea, and place it outside of the loop:
$parentcount = mysqli_query($con, ("SELECT parent_id, count(*) FROM tbl_menu GROUP BY parent_id");
There may be other issues, so please post the database structure and number of records that you're working with too.
Don't make recursive queries.
Having "more than 1000" rows is not too big. You can simply call everything from the table into php, then perform the recursive html build in php this will have a memory overhead, but far less processing overhead because you only ever make one trip to the db.
Alternatively (when your db table is prohibitively large), you should avoid gathering rows unnecessarily by adding a new column. The new column will store all "descendants" for the respective row when the row is INSERTed or update it when it is UPDATEd. Then you only need to reference this column when needing to call specific rows. In other words, do the recursive processing only once (when writing to the db) AND not when needing to display the data. This will, again, produce a finite result set in one query which can then be recursively traversed to build the desired output.
basically you need to do what #spudly has suggested.
But there is a small catch in his solution which depending on the number of the rows in yous tbl_menu table you may use a big chunk of memory to fetch all the records.
you can optimise it more with using his solution but changing the query to:
select
parent_tbl_menu.id,
count(child_tbl_menu.id) as cnt
from
tbl_menu as parent_tbl_menu
left join
tbl_menu as child_tbl_menu
on parent_tbl_menu.id = child_tbl_menu.parent_id
where
parent_tbl_menu.parent_id = ?
group by
parent_tbl_menu.id
This way you will only fetch the child records of a specific parent.
And please consider using prepared statements as your code has sql injection vulnerability.
Connect (from PHP to MySQL) only once for the entire web page.
Don't put a SELECT inside a loop if you can do all the work in a single SELECT, such as with a JOIN. (Exception: A "hierarchical" table needs the nested SELECT. Exception to the exception: MySQL 8.0 and MariaDB 10.2 can do it with a "recursive CTE".)
Don't fetch all the columns (SELECT *) when all you want it is a recordcount. Instead, SELECT COUNT(*) ... and use the number returned.
1000 of anything is probably excessive for a web page. Re-think the UI.
I write c++ to do 30000*30000 query MySQL insert
Example
for(i=0;i<30000*30000;i++){
// do the MySQL insert
call the function to insert code (maybe just query insert the i value)
}
and I am using mysqlsart() and mysqlclose() sub-function
void mysqlstart()
{
//MYSQL *conn_ptr;
conn_ptr = mysql_init(NULL);
if(!conn_ptr)
{
fprintf(stderr,"mysql_init failed\n");
//return EXIT_FAILURE;
}
conn_ptr = mysql_real_connect(conn_ptr,"localhost","root","nlpgroup","testdb",0,NULL,0);
}
void mysqlclose()
{
mysql_close(conn_ptr);
}
First time I call the function like
mysqlstart();
for loop
mysqlclose();
That will be great in the beginning and after few days I found the error : MySQL server has gone away
And find some solution from internet maybe change max_allowed_packet or some else...
And I don't know what is suitable settings for 30000*30000
And I want to is there something I can change in my code or way to speed up query
You can insert multiple rows at once in this way
http://dev.mysql.com/doc/refman/5.5/en/insert.html
INSERT statements that use VALUES syntax can insert multiple rows. To
do this, include multiple lists of column values, each enclosed within
parentheses and separated by commas. Example:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
I have pulled in the data from a mysql database using select * with the intention of using the data several times without doing repeated sql enquiries using WHERE.
Using this data I am extracting rows that contain a search element using
while($row=mysql_fetch_array($query_result)){ <<<if match add to new array>>> }
As there are thousands of rows this is taking a longer time than I want.
I am trying to use:
$row=mysql_fetch_array($query_result);
$a = array_search($word_to_check, $row);
echo $a;
This extracts the correct sql headings but not the row number. What I want to achieve is
if $word is found in mysql_fetch_array($query_result) the add the row where it was found into the new array for processing.
Any thoughts? Thanks in advance.
Don't use mysql_* functions they are depracated. Use mysqli or pdo instead.
It's not wise to search in array of mysql results in php while it can be done in mysql. Let's say you have table and you want to find all numbers in number column that are greater than 5
SELECT FROM table_name WHERE number>5
to find text you can use simple clause
SELECT FROM table_name WHERE name = 'username'
You can also create more complex conditions.
From MYSQL manual:
WHERE clause, if given, indicates the condition or conditions that rows must satisfy to be selected. where_condition is an expression that evaluates to true for each row to be selected. The statement selects all rows if there is no WHERE clause
Check this link
If you want to limit the query to only once, fetch all the results into temporary array and do the search from it like below
<?php
$all_rows=array();
$match_rows=array();
$i=0;
$limit=100000;
while($row=mysql_fetch_array($query_result)){
$all_rows[]=$row;
if($i % $limit == 0){ // this part only functions every 100,000 cycles.
foreach($all_rows as $search_row){
if(array_search($word_to_check, $search_row)
$match_rows[]=$search_row;
}
$all_rows=array();//reset temporary array
}
$i++;
}
//This solution assumes the required word can be found in mulitple columns
In the MySQL Reference Manual, there's distinction between data definition statements and data manipulation statements.
Now I want to know if a query inserts a database record, updates one, deletes one or modifies the table structure and so on, or, more precisely, the exact number of affected rows, but only if it is applicable.
For example, the statement
SELECT *
FROM SomeTable
WHERE id=1 OR id=2
returns a number of affected rows (in this case 2), but with the SELECT statement, there's nothing modified in the database, so that number would be 0.
How to get the type of query?
I was looking for the same answer and stumbled across this article. It was last updated in August. In it, there is a section: "Determining the Type of a Statement" You basically can make the following assumptions: (copied from the article)
If columnCount() is zero, the statement did not produce a result set. Instead, it modified rows and you can invoke rowCount() to determine the number of affected rows.
If columnCount() is greater than zero, the statement produced a result set and you can fetch the rows. To determine how many rows there are, count them as you fetch them.
I'll save you the trouble and just paste the code sample here
$sth = $dbh->prepare ($stmt);
$sth->execute ();
if ($sth->columnCount () == 0)
{
# there is no result set, so the statement modifies rows
printf ("Number of rows affected: %d\n", $sth->rowCount ());
}
else
{
# there is a result set
printf ("Number of columns in result set: %d\n", $sth->columnCount ());
$count = 0;
while ($row = $sth->fetch (PDO::FETCH_NUM))
{
# display column values separated by commas
print (join (", ", $row) . "\n");
$count++;
}
}
I have been thinking of the same issue, and come to conclusion that I don't need no automation in this matter.
The only use for such an auto-detect is some magic function which will return number of affected rows. But such a magic, although adding a little sugar to the syntax, always makes code support a nightmare:
When you're calling a function, and it can return values of different types depends on the context, you cannot tell which one is returned at every particular moment. So, it makes debugging harder.
So, for sake of readability, just call appropriate function to get the result you need at the moment - affectedRows or numRows. It won't make your code bloated, but make it a lot readable.
I'm using this:
substr($statement->queryString, 0, strpos($statement->queryString, ' '));
where $statement is a PDOStatement object, a few things to note here are that you should verify before using this that $statement is a PDOStatement object, also we should probably take the strpos out of the substr statement in case strpos returns false, which would probably cause an error, finally, this only works with one word statement types, like SELECT, INSERT, etc and not multi-word statement types like ALTER TABLE