I am pulling data from a IDX/RETS server and trying to insert the data into my own MySQL data base using php.
The data comes in as multiple associative arrays and i loop through them entering each array into its own row.
The table itself has 215 columns, in my first test I only pull 10 records/array's and after running my script I only get 1 row entered.
I have been able to get all 10 rows to insert but only by reducing the number of columns to about 6. For what ever reason when trying to enter all 215 columns of the 10 records it keeps timing out.
I have tried:
ini_set('max_execution_time', 500);
&
set_time_limit(0);
I have also tried entering these values into my php.ini file but what ever i do the script only seems to run for about 15-30 seconds.
Is there something else I am missing or should be doing when entering so many columns??
My code is just a simple while loop that is looping through the arrays. And my insert is like this:
$sql = "INSERT INTO rets_property_residentialproperty";
$sql .= " (`".implode("`, `", array_keys($my_array))."`)";
$sql .= " VALUES ('".implode("', '", $my_array)."') ";
$result = mysqli_query($dbcon, $sql) or die(mysql_error());
For one thing, don't mix mysql and mysqli interface functions. The call to mysql_error should be replaced with a call to mysqli_error.
Not at all clear what's "timing out".
Most likely, the INSERT statement is throwing an error, the code is going into the die, and the problem is with the mysql_error function. Get that function replaced with mysqli_error, and I venture that you'll get a MySQL error.
... or die(mysqli_error($dbcon));
^ ^^^^^^
Likely, the underlying issue is a SQL syntax error. For example, one of the values in the array may contain a character that needs to be escaped, such as a single quote.
(But, I'm just guessing here.)
For debugging, you might consider echoing out the SQL text
echo $sql;
before the call to mysqli_query. (Then you can compare the error to the actual SQL text that you intended to send to the database.
To handle single quotes and other "dangerous" characters in the values, (to close a gaping SQL Injection vulnerability) you'd either need to call mysqli_real_escape_string on each of the array values, before the value is included in the SQL text. Or, use a prepared statement with bind placeholders, and then supply the array values as the values for the bind placeholders.
Related
There is a microsoft SQL server that I need to get results from. I have been given a specific query. When I use this query in the HeidiSQL application it works. I can always select and display all 100 rows.
I've tried the exact same query in a PHP script. It manages to select and show some of the rows, but it fails to show everything, even though the script fully runs.
Heavily simplified, the PHP script looks like this:
$serverName = "ServerName";
$connectionInfo = array( "Database"=>"Databasename", "UID"=>"username", "PWD"=>"***");
$MSdb = sqlsrv_connect( $serverName, $connectionInfo);
if ($MSdb) {
$query=" SELECT field1,field2,field3 FROM Table1 ";
$result=sqlsrv_query($MSdb,$query);
while ($row=sqlsrv_fetch_array($result)) {
$field1=$row['field1'];
$field2=$row['field2'];
$field3=$row['field3'];
?><br><?=$field1?> , <?=$field2?> , <?=$field3?> <?php
// process the fields and save them into a mysql database
}
}
?>END!<?php
It seems to fail when the data in the SQL server has a single quote in it.
field3 of row 5 is for example this:
aaa ‘bb’ cccc
If I modify the query with TOP, OFFSET or FETCH NEXT ROWS , then I'm for example able to select and display the first 4 rows, or row 6 till 10. Everything works, as long as I'm excluding row 5 from the query.
If I only select field1,field2 , then I'm able to get all 100 rows with the PHP script.
So somehow this field with the quotes (and several others like it) stop the execution of the query. If I avoid those fields, everything works fine. The PHP script itself will be executed fully, as it always outputs the 'END!' .
I'm assuming that I have to modify the query so that the quotes can be handled better. However, anything I search for is about escaping quotes for INSERT queries, not SELECT queries. So I'm a bit lost here. What do I need to modify?
It's because the quote that you are showing it's a unicode format. You have to convert them to a normal single quote or change the charset from the output.
Where ‘ isn't the same as ', and can break your code if you don't have the proper charset.
Edit: You can try to use str_replace() on field 3 to see if it works before you change the charset.
I am trying to display the data from 'table' if a key inputted by the user is found in the database. Currently I have it set up so that the database checks if the key exists, like so:
//Select all from table if a key entry that matches the user specified key exists
$sql = 'SELECT * FROM `table` WHERE EXISTS(SELECT * FROM `keys` WHERE `key` = :key)';
//Prepare the SQL query
$query = $db->prepare($sql);
//Substitute the :key placeholder for the $key variable specified by the user
$query->execute(array(':key' => $key));
//While fetched data from the query exists. While $r is true
while($r = $query->fetch(PDO::FETCH_ASSOC)) {
//Debug: Display the data
echo $r['data'] . '<br>';
}
These aren't the only SQL statements in the program that are required. Later, an INSERT query along with possibly another SELECT query need to be made.
Now, to my understanding, using WHERE EXISTS isn't always efficient. However, would it be more efficient to split the query into two separate statements and just have PHP check if any rows are returned when looking for a matching key?
I took a look at a similar question, however it compares multiple statements on a much larger scale, as opposed to a single statement vs a single condition.
#MarkBaker Join doesn't have to be faster than exists statement. Query optymalizer is able to rewrite the query live if it sees better way to accomplish query. Exists statement is more readable than join.
Fetching all the data and making filtering directly in PHP is always bad idea. What if your table grow up to milions of records? MySQL is going to find the best execute plan for you. It will automaticaly cache the query if it is going to improve performance.
In other words, your made everything correctly as far as we can see your code now. For futher analyse show us all of your queries.
I have an html page where I collect an array of values from checkboxes to insert in a database. The html page posts to a PHP page that collects the data and then stores in the database.
For each value, there are a few other fields I would like to include that are the same for all the values such as time entered. I can easily convert the captured array into a comma delimited list using implode. I use such a comma delimited list of ids to update and delete records. However, when I want to insert them, MYSQL does not seem to allow you to use a comma delimited list. My question is, what is the easiest way to insert records, one for each value in the c comma delimited list, without using a loop.
html page
<input type="checkbox" name="var[]" value=1>
<input type="checkbox" name="var{}" value=2>
PHP page
$vars = $_POST['var'];
This gives me an array that I can convert to a comma delimited list using implode.
To delete, I can go
$sql = "DELETE * from table WHERE id in '$vars'";
To update I can go
$sql = "UPDATE table, WHERE id in '$vars'";
But there does not seem to be an equivalent for Insert.
Following would work:
$sql = "INSERT into table (var, timeentered) values (1,now()) (2,now())";
However, that's not how I have my data. what I would like to do is something like
$sql = "INSERT into table (var,timeentered) values($vars), now()" but of course that doesn't work.
Do I have to convert my nice comma delimited list that works so well for update and delete into something that looks like (1,now) (2, now()) for inserting or is there an alternative?
Thanks for any suggestions.
Unfortunately you have to build whole query by yourself:
$sql ="insert into table (var, timeentered) values ";
$sql .= "(".implode(", now()), (", $vars).")";
You need to loop through your data set and create the multi-line insert query manually. There is no other alternative if you want to insert multiple rows with a single query. That is outside of using certain DB frameworks which might present a better interface for doing this. Of course at the end of the day, such a DB framework would in essence be building the multi-item insert query manually at some point.
The question might come done to one of how many items are you going to insert. If you are only going to be inserting a few records at a time, then you might want to consider just using prepared statements with individual inserts. However if you are going to be inserting hundreds of records at a time, that would probably not be a good idea.
In your mysql database you can set the default for the column "time_created" to be a TIMESTAMP default CURRENT_TIMESTAMP. This way you don't have to worry about it. Just use the regular insert and it will automatically set the "time_created" column.
For your other issue of multi-line inserts you can create an $update array and use a foreach loop to issue a sql insert command on every row of data.
Two options I can think of.
Build a dynamic insert query like you suggest. However do not call now() each time but just insert a single date ie
$time = gmdate();
$sql = "INSERT into table (var, timeentered) values (1,$time) (2,$time)";
Or use a prepared statement of the single insert below, turn off autocommit, start a transaction and execute the prepared statement in a for loop for the number of inserts needed, then commit the transaction.
$sql = "INSERT into table (var, timeentered) values (?,?)"
Mostly you will have to build your query using some type of looping structure. Convention and best practice aside if you just want to know how to make your array acceptable for a multiple insert statement then why not just do this:
$values = '('.implode('),(', $array).')';
or if already CSV then:
$values = '('.implode('),(', explode(',' $csv)).')';
then you can just use $values in your query using double quotes.
Can the php function mysql_insert_id() return no result after processing the INSERT query in mysql db?
Just to clarify. There was a script performing by cron on the production site. It contained a cycle for generating invoices for users. Each iteration consists of a INSERT db query and the mysql_insert_id() operation going right after the query - to fetch the generated invoice number. A set of iterations were performed without fetching the last inserted number.
Can it be caused by high db server load or by some other reasons that are not linked to the problem at the php code site?
Any help would be appreciated!
Offhand, I can think of a few cases where MySQL wouldn't return the ID:
The table you're inserting into doesn't have an AUTO_INCREMENTed primary key.
You're inserting multiple rows at once.
You're calling mysql_insert_id() from a different connection than the INSERT query was executed.
The INSERT query didn't succeed (for instance, it encountered a deadlock). Make sure you are checking the return value from mysql_query(), then use mysql_errno() and mysql_error().
MySQL docs have a full list of conditions and details on how this function works.
Of course, it's also possible there is a bug in MySQL, which would depend on which version of MySQL you are using.
If you're running the commands through a shell script, and run them both separately as in;
mysql -e "insert into table ( field1 ) values ( 'val1' );" "database"
lastId=`mysql -e "select last_insert_id();" "database"`
Then that won't work as the second call makes a new connection to the server. You need to do something like the following, as it is all done within a single database call / connection;
lastId=`mysql -e "
insert into table ( field1 ) values ( 'val1' );
select last_insert_id();
" "database"`
You'll need to look up the extra parameters required for the MySQL command to remove formatting and header row - I'm afraid I can't remember them off the top of my head!
I have a array with a variable amount of values.
Is there a more efficient or better way to INSERT them into my DB besides a loop with a query inside it?
At this site, there is a nice example of MySQL with a multi-insert query. It is valid SQL to
INSERT INTO [table]
VALUES
(row1),
(row2),
...
On request: a php snippet:
$query="INSERT INTO mytable\nVALUES\n (".$values[0].")";
array_shift( $values );
foreach( $values as $value ) {
$query .= ",(".$value.")";
}
In my experience multi-row inserts are processed MUCH faster than an equivalent number of single row inserts, if you're inserting a large amount of data at a time, that's a good way to go. I've watched a process of entering thousands of rows of data be condensed from 5-10 minutes down to literally seconds using this method.
As far as the code part, I've been a fan of using implode() to join arrays of fields & values together. No reason you can't do the same for rows of data, you just need to be able to identify which fields need to be quoted, escaped, etc.
For the sake of argument assume $rows is an array of properly formatted SQL values...
$sql = "INSERT INTO `table` VALUES (" . implode("), (", $rows) . ")";
You could apply something similar to assemble the individual fields should you desire.
If the DB you are using allows multiple value insert, then you could create an multi-insert statement and send that to the DB - one connect with one command to do multiple inserts.
If you cannot do multiple inserts - (as MSSQL does not allow) - then I think you are stuck.