I have array like this:
$array = array("AAA,http://aaa.com,bbb,http://bbb.com,ccc,http://ccc.com");
How can I take this array and insert into database like this:
No name url
1. AAA. Http://aaa.com
2. BBB. Http://bbb.com
3. CCC. Http://ccc.com
Using PHP and MySQL.
Thank you.
Assembling one INSERT statement with multiple rows is much faster in MySQL than one INSERT statement per row.
That said, it sounds like you might be running into string-handling problems in PHP, which is really an algorithm problem, not a language one. Basically, when working with large strings, you want to minimize unnecessary copying. Primarily, this means you want to avoid concatenation. The fastest and most memory efficient way to build a large string, such as for inserting hundreds of rows at one, is to take advantage of the implode() function and array assignment.
$sql = array();
foreach( $data as $row ) {
$sql[] = '("'.mysql_real_escape_string($row['text']).'", '.$row['category_id'].')';
}
mysql_query('INSERT INTO table (text, category) VALUES '.implode(',', $sql));
The advantage of this approach is that you don't copy and re-copy the SQL statement you've so far assembled with each concatenation; instead, PHP does this once in the implode() statement. This is a big win.
If you have lots of columns to put together, and one or more are very long, you could also build an inner loop to do the same thing and use implode() to assign the values clause to the outer array.
please try and good luck
Related
My use case:
I have multiple scripts inserting into a table in the order of several inserts per second. I am seeing performance degradation, so I think there would be performance benefits in "batching queries" and inserting several hundred rows every minute or so.
Question:
How would I go about doing this using mysqli? My current code uses a wrapper (pastebin), and looks like:
$array = array();\\BIG ARRAY OF VALUES (more than 100k rows worth)
foreach($array AS $key => $value){
$db -> q('INSERT INTO `player_items_attributes` (`column1`, `column2`, `column3`) VALUES (?, ?, ?)', 'iii', $value['test1'], $value['test2'], $value['test3']);
}
Notes:
I looked at using transactions, but it sounds like those would still hit the server, instead of queuing. I would prefer to use a wrapper (feel free to suggest one with similar functionality to what my current one offers), but if not possible I will try to build suggestions into the wrapper I use.
Sources:
Wrapper came from here
Edit:
I am trying optimize table speed, rather than script speed. This table has more than 35million rows, and has a few indexes.
The MySQL INSERT syntax allows for one INSERT query to insert multiple rows, like this:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
where each set of parenthesised values represents another row in your table. So, by working down the array you could create multiple rows in one query.
There is one major limitation: the total size of the query must not exceed the configured limit. For 100k rows you'd probably have to break this down into blocks of, say, 250 rows, reducing your 100k queries to 400. You might be able to go further.
I'm not going to attempt to code this - you'd have to code something and try it in your environment.
Here's a pseudo-code version:
escape entire array // array_walk(), real_escape_string()
block_size = 250; // number of rows to insert per query
current_block = 0;
rows_array = [];
while (next-element <= number of rows) {
create parenthesised set and push to rows_array // implode()
if (current_block == block_size) {
implode rows_array and append to query
execute query
set current_block = 0
reset rows_array
reset query
}
current_block++
next_element++
}
if (there are any records left over) {
implode rows_array and append to query
execute the query for the last block
}
I can already think of a potentially faster implementation with array_map() - try it.
i was playing with PDO on PostgreSQL 9.2.4 and was trying to fetch data from a table having millions on rows. My query returns about 100.000 rows.
I do not use any of PDOStatements's fetch function, i simply use the result from the PDO Objecte itels and loop through it.
But its getting slower and slower by time. At the beginning it was fetching like 200 rows per second. But the close it comes to its end, it gets slower. Now being at row 30.000 it fetches only 1 row per second. Why is it getting slower.
I do this, its pretty simple:
$dbh = new PDO("pgsql...");
$sql = "SELECT x, y FROM point WHERE name is NOT NULL and place IN ('area1', 'area2')";
$res = $dbh->query($sql);
$ins_sql = "INSERT INTO mypoints (x, y) VALUES ";
$ins_vals = [];
$ins_placeholders = [];
foreach($res as $row) {
$ins_placeholders[] = "(?,?)";
$ins_vals = array_merge($ins_vals, [$row['x'], $row['y']]);
printCounter();
}
// now build up one insert query using placeholders and values,
// to insert all of them in one shot into table mypoints
Function printCounter simply increases an int var and prints it. So i can see how many rows it has put already in that array before i create my insert statement out of it. I use one shot inserts to speed things up, better than doing 100.000 inserts.
But that foreach loop is getting slower by time. How can i increase the speed.
Is there a difference between fetch() and the simple loop method using the pdostatement in foreach?
when i start this php script, it takes like 5-10 seconds for the query. So this has nothing to do with how the table is setup and if i need indexes.
I have other tables returning 1 million rows, im not sure what is the best way to fetch them. I can raise PHP's memory_limit if needed, so the most important thing for me is SPEED.
Appreciate any help.
It's not likely that the slowness is related to the database, because after the $dbh->query() call, the query is finished and the resulting rows are all in memory (they are not in PHP variables yet, but they're in memory accessible at the pgsql module level).
The more likely culprit is the array_merge operation. The array becomes larger at every loop iteration, and the operation recreates the entire array each time.
You may want to do instead:
$ins_vals[] = [$row['x'], $row['y']];
Although personally, when concerned with speed, I'd use an even simpler flat structure:
$ins_vals[] = $x;
$ins_vals[] = $y;
Another unrelated point is that it seems to build a query with a huge number of placeholders, which is not how placeholders are normally used. To send large numbers of values to the server, the efficient way is to use COPY, possibly into a temporary table followed by server-side merge operations if it's not a plain insertion.
I dont know why, but using fetch() method instead and doing the $ins_val filling like this:
$ins_vals[] = $x;
$ins_vals[] = $y;
and using beginTransaction and commit makes now my script unbelievable fast.
Now it takes only about 1 minute to add my 100.000 points.
i think both array_merge and that "ugly" looping through the PDOStatement slowed down my script.
And why the heck someone downvoted my question? Are you punishing me because of my missing knowledge? Thanks.
Ok i generated a class where i set the sql and then put the values for each row with a method call. Whenever it reaches a specific limit, it starts a transaction, prepares the statement with as many placeholders as i have put values, then executes it with the array having all the values, then commit.
This seems to be fast enough, at least it doesnt get slower anymore.
For some reason its faster to add values in a flat structure as Daniel suggested. Thats enough for me.
Sometimes its good to have a function doing one step of insertion, because when the function returns, all the memory used in the function will be freed, so your memory usage stays low.
Merely due to the sake of being efficient I would like to hear some advises from the PHP expers here. (Actually this question goes out all developers that mess with database connections on a regular basis)
Assume you have one personId and this id has multiple numbers. You have a table with columns pID,nums
From the php side you retrieve these numbers in an array. Now my question comes to place. Do you do something like;
for($i=0;$i<count($arr);$i++)
{
//Call the insert query over and over again
}
Or there is obviously a better solution?
Essentially you want to do a bulk insert - it's faster to execute one composite statement, than lots of individual inserts, so try:
$data is your array
$sql = array();
foreach( $data as $row ) {
$sql[] = '("'.mysql_real_escape_string($row['field1']).'")';
}
mysql_query('INSERT INTO table (field) VALUES '.implode(',', $sql));
If your data set is really large, you can look at dumping your values into a file, and then executing "LOAD DATA INFILE" (will only work if MySQL runs on the same host as your script).
Alternatively, you can use one insert query:
INSERT INTO table(pid,nums) VALUES (1,2), (1,3), (1,4), ... -- etc
with multiple values.
This is what i am doing now : - in PHP
foreach($array as $value)
{
$query = select abc from tblname where colname =" .$value.
// fire query
}
then i create array of these values and display accordingly.
The PROBLEM: -
I have applied foreach, which fires the query every time it encounters a value in the array.
result, if i have 10 values in my array it fires 10 queries. and uses network 10 times, result slow output.
What i want -
I want to give the array to a stored procedure which shall give me a resultset which will have the outputs corresponding to all the elements in the array.
I know this can be done but do not know how.
the mysql doesnot take arrays as datatype.
the result shall be that network shall be used only once, despit of any number of values in the array.
LIKE -
StoredProcedure(inputMysqlARRAY) // not possible, need a workaroung
{
// fire simple select(same) query for each value.
}
then call this stored procedure from PHP and input array. // need workaround.
You just have to be smarter about your calls. For instance, keeping cached DB objects around and that sort of thing.
Without knowing more about your code (your question is fairly garbled), it seems that if your query is something like this:
$query = "select abc from tblname where colname =" .$value; // run 10 times.
You really just need to write smarter code:
$values = array(); // Now, populate this array.
// When you're done, run the query:
$query = 'select abc from tblname where colname IN (\''.implode('\',\'', $values).'\')';
Generally, we refer to this as Dynamic SQL and is the underpinning for how things are typically done today. A stored procedure (or, based on how I read your question, stored function) is useful at times, but is somewhat antiquated as a first-order methodology for interfacing with SQL. The DB guys still sometimes swear by it, but I think that even they are fairly well in consensus that smarter queries are always better.
Say I have an array of strings in a php array called $foo with a few hundred entries, and I have a MySQL table 'people' that has a field named 'name' with a few thousand entries. What is an efficient way to find out which strings in $foo aren't a 'name' in an entry in 'people' without submitting a query for every string in $foo?
So I want to find out what strings in $foo have not already been entered in 'people.'
Note that it is clear that all of the data will have to be on one box at one point. The goal would be doing this at the same time minimizing the number of queries and the amount of php processing.
I'd put your $foo data in another table and do a LEFT OUTER JOIN with your names table. Otherwise, there aren't a lot of great ways to do this that don't involve iteration at some point.
The best I can come up with without using a temporary table is:
$list = join(",", $foo);
// fetch all rows of the result of
// "SELECT name FROM people WHERE name IN($list)"
// into an array $result
$missing_names = array_diff($foo, $result);
Note that if $foo contains user input it would have to be escaped first.
What about the following:
Get the list of names that are already in the db, using something like:
SELECT name FROM people WHERE name IN (imploded list of names)
Insert each item from the return of array_diff()
If you want to do it completely in SQL:
Create a temp table with every name in the PHP array.
Perform a query to populate a second temp table that will only include the new names.
Do an INSERT ... SELECT from the second temp table into the people table.
Neither will be terribly fast, although the second option might be slightly faster.
CREATE TEMPORARY TABLE PhpArray (name varchar(50));
-- you can probably do this more efficiently
INSERT INTO PhpArray VALUES ($foo[0]), ($foo[1]), ...;
SELECT People.*
FROM People
LEFT OUTER JOIN PhpArray USING (name)
WHERE PhpArray.name IS NULL;
For a few hundred entries, just use array_diff() or array_diff_assoc()
$query = 'SELECT name FROM table WHERE name != '.implode(' OR name != '. $foo);
Yeash, that doesn't look like it would scale well at all.
I'm not sure there is a more efficient way to do this other than to submit all the strings to the database.
Basically there are two options: get a list of all the strings in MySQL and pull them into PHP and do the comparisons, or send the list of all the strings to the MySQL server and let it do the comparisons. MySQL is going to do the comparisons much faster than PHP, unless the list in the database is a great deal smaller than the list in PHP.
You can either create a temporary table, but either way your pushing all the data to the database.