I'm new one in php, and I have a question.
$result = mssql_query('stored proc. with some parameters');
while ($row = mssql_fetch_row($result)){
mysqli_query('INSERT INTO dbname VALUES ($row[0], $row[1], ... , $row[15] );') or die ('Insertion mistake.');
}
The problem is, I've got die() exception after ~430 insertions. The size of $result query can be up to 100.000 results. I cannot access the MS base and touch stored procedures, and this data transfer in needable. Can somebody help me, with a solution, or show the place, I can read more about that kind of problem?
Creating a single query might be faster.
$result = mssql_query('stored proc. with some parameters');
$query = '';
while ($row = mssql_fetch_row($result)){
$query .= "INSERT INTO dbname VALUES ($row[0], $row[1], ... , $row[15]),";
}
mysqli_query($query) or die('Insertion mistake.');
Ore better this I think:
while ($row = mssql_fetch_row($result)){
$values = "({$row[0]}, {$row[1]}, ... , {$row[15]})";
}
$query = 'INSERT INTO dbname VALUES ' . implode(',', $values);
mysqli_query($query) or die('Insertion mistake.');
And I didn't show it, but you need to run the data $row[0], etc... through mysqli_real_escape_string() before inserting.
Also, in the query, you need to quote the data that is not destined for a numeric column type: '{$row[0]}' etc. if it is text.
I think you are running out of memory. You can add to the top of the script:
ini_set('memory_limit','512M');
This will set the maximum memory limit for your script to 512MB.
You can also try to change memory_limitat php.ini
Related
I have a database and a string with about 100,000 key / value-pair records and I want to create a function to find the value.
Is it better to use a string or a database, considering performance (page load time) and crash safety? Here are my two code examples:
1.
echo find("Mohammad");
function find($value){
$sql = mysql_query("SELECT * FROM `table` WHERE `name`='$value' LIMIT 1");
$count = mysql_num_rows($sql);
if($count > 0){
$row = mysql_fetch_array($sql);
return $row["family"];
} else {
return 'NULL';
}
}
2.
$string = "Ali:Golam,Mohammad:Offer,Reza:Now,Saber:Yes";
echo find($string,"Mohammad");
function find($string,$value){
$array = explode(",",$string);
foreach($array as $into) {
$new = explode(":",$into);
if($new[0] == $value) {
return $new[1];
break;
}
}
}
The database is pretty sure a good idea.
Databases are fast, maybe they are not as fast as basic String operations in PHP, but if there is a lot of data, databases will probably be faster. A basic select Query takes (on my current default Desktop Hardware) about 15ms, cached less than 1ms, and this is pretty much independend of the number of names in your table, if the indexes are correct. So your site will always be fast.
Databases won't cause a StackOverflow or an out of memory error and crash your site (this is very depending on your PHP-Settings and Hardware)
Databases are more flexible, imagine you want to add / remove / edit names after creating the first Data-Set. It's very simple with "INSERT INTO", "DELETE FROM" and "UPDATE" to modify the data, better than editing a string somewhere in your code with about 100.000 entries.
Your Code
You definitly need to use MySQLi or PDO instead, code maybe like this:
$mysqli = new mysqli("host", "username", "password", "dbname");
$stmt = $mysqli->prepare(
'SELECT string FROM table WHERE name = ? LIMIT 1'
);
$stmt->bind_param("s", $value);
$stmt->execute();
$stmt->store_result();
$stmt->bind_result($string);
$stmt->fetch();
$stmt->close();
This uses MySQLi and Prepared Statements (for security), instead of the depracated MySQL extension (in PHP 5.5)
I ran into the following question while writing a PHP script. I need to store the first two integers from an array of variable lenght into a database table, remove them and repeat this until the array is empty. I could do it with a while loop, but I read that you should avoid writing SQL statements inside a loop because of the performance hit.
A simpliefied example:
while(count($array) > 0){
if ($sql = $db_connect->prepare("INSERT INTO table (number1, number2) VALUES (?,?)")){
$sql->bind_param('ii',$array[0],$array[1]);
$sql->execute();
$sql->close();
}
array_shift($array);
array_shift($array);
}
Is this the best way, and if not, what's a better approach?
You can do something like this, which is way faster aswell:
Psuedo code:
$stack = array();
while(count($array) > 0){
array_push($stack, "(" . $array[0] . ", " . $array[1] . ")");
array_shift($array);
array_shift($array);
}
if ($sql = $db_connect->prepare("INSERT INTO table (number1, number2)
VALUES " . implode(',', $stack))){
$sql->execute();
$sql->close();
}
The only issue here is that it's not a "MySQL Safe" insert, you will need to fix that!
This will generate and Array that holds the values. Within 1 query it will insert all values at once, where you need less MySQL time.
Whether you run them one by one or in an array, an INSERT statement is not going to make a noticeable performance hit, from my experience.
The database connection is only opened once, so it is not a huge issue. I guess if you are doing some insane amount of queries, it could be.
I think as long as your loop condition is safe ( will break in time ) and you got something from it .. it's ok
You would be better off writing a bulk insert statement, less hits on mysql
$sql = "INSERT INTO table(number1, number2) VALUES";
$params = array();
foreach( $array as $item ) {
$sql .= "(?,?),\n";
$params[] = $item;
}
$sql = rtrim( $sql, ",\n" ) . ';';
$sql = $db_connect->prepare( $sql );
foreach( $params as $param ) {
$sql->bind_param( 'ii', $param[ 0 ], $param[ 1 ] );
}
$sql->execute();
$sql->close();
In ColdFusion you can put your loop inside the query instead of the other way around. I'm not a php programmer but my general belief is that most things that can be done in language a can also be done in language b. This code shows the concept. You should be able to figure out a php version.
<cfquery>
insert into mytable
(field1, field2)
select null, null
from SomeSmallTable
where 1=2
<cfloop from="1' to="#arrayLen(myArray)#" index="i">
select <cfqueryparam value="myArray[i][1]
, <cfqueryparam value="myArray[i][]
from SomeSmallTable
</cfloop>
</cfquery>
When I've looked at this approach myself, I've found it to be faster than query inside loop with oracle and sql server. I found it to be slower with redbrick.
There is a limitation with this approach. Sql server has a maximum number of parameters it will accept and a maximum query length. Other db engines might as well, I've just not discovered them yet.
i'm trying to make a long mysql query and process and update the row founded:
$query = 'SELECT tvshows.id_show, tvshows.actors FROM tvshows where tvshows.actors is not NULL';
$result = mysql_query($query);
$total = mysql_num_rows($result);
echo $total;
while ($db_row = mysql_fetch_assoc($result))
{
//process row
}
but after 60 second give me a timeout request, i have try to insert these in my php code:
set_time_limit(400);
but it's the same, how i can do?
EDIT:
only the query:
$query = 'SELECT tvshows.id_show, tvshows.actors FROM tvshows where tvshows.actors is not NULL';
takes 2-3 second to perform, so i think the problem is when in php i iterate all the result to insert to row or update it, so i think the problem is in the php, how i can change the timeout?
EDIT:
here is the complete code, i don't think is a problem here in the code...
$query = 'SELECT tvshows.id_show, tvshows.actors FROM tvshows where tvshows.actors is not NULL';
$result = mysql_query($query);
$total = mysql_num_rows($result);
echo $total;
while ($db_row = mysql_fetch_assoc($result)) {
//print $db_row['id_show']."-".$db_row['actors']."<BR>";
$explode = explode("|", $db_row['actors']);
foreach ($explode as $value) {
if ($value != "") {
$checkactor = mysql_query(sprintf("SELECT id_actor,name FROM actors WHERE name = '%s'",mysql_real_escape_string($value))) or die(mysql_error());
if (mysql_num_rows($checkactor) != 0) {
$actorrow = mysql_fetch_row($checkactor);
$checkrole = mysql_query(sprintf("SELECT id_show,id_actor FROM actor_role WHERE id_show = %d AND id_actor = %d",$db_row['id_show'],$actorrow[0])) or die(mysql_error());
if (mysql_num_rows($checkrole) == 0) {
$insertactorrole = mysql_query(sprintf("INSERT INTO actor_role (id_show, id_actor) VALUES (%d, %d)",$db_row['id_show'],$actorrow[0])) or die(mysql_error());
}
} else {
$insertactor = mysql_query(sprintf("INSERT INTO actors (name) VALUES ('%s')",mysql_real_escape_string($value))) or die(mysql_error());
$insertactorrole = mysql_query(sprintf("INSERT INTO actor_role (id_show, id_actor, role) VALUES (%d, %d,'')",$db_row['id_show'],mysql_insert_id())) or die(mysql_error());
}
}
}
}
Should definitely try what #rid suggested, and to execute the query on the server and see the results/duration to debug - if the query is not a simple one, construct it as you would in your PHP script, and only echo the SQL command, don't have to execute it, and just copy that in to the server MySQL command line or whichever tool you use.
If you have shell access, use the top command after running the above script again, and see if the MySQL demon server is spiking in resources to see if it really is the cause.
Can you also try a simpler query in place of the longer one? Like just a simple SELECT count(*) FROM tvshows and see if that also takes a long time to return a value?
Hope these suggestions help.
There are so many problems with your code.
Don't store multiple values in a single column. Your actors column is pipe-delimited text. This is a big no-no.
Use JOINs instead of additional queries. You can (or could, if the above weren't true) get all of this data in a single query.
All of your code can be done in a single query on the server. As I see it, it takes no input from the user and produces no output. It just updates a table. Why do this in PHP? Learn about INSERT...SELECT....
Here are some resources to get you started (from Googling, but hopefully they'll be good enough):
http://www.sitepoint.com/understanding-sql-joins-mysql-database/
http://dev.mysql.com/doc/refman/5.1/en/join.html
http://dev.mysql.com/doc/refman/5.1/en/insert-select.html
What is Normalisation (or Normalization)?
Let me know if you have any further questions.
I'm trying to insert some data into my mysql database. The connection is working fine but im having a problem with sending the query correctly to the database. Below you can find the code in my php file. I also post what for type of fields they are in the Database.
Fields in the mysql database:
Reservaties_id = int
Materialen_id = int
aantal = int
effectief_gebruikt = tinyint
opmerking = Varchar2
datum_van = date
datum_tot = date
$resID = $_REQUEST['resID'];
$materialen_id = $_REQUEST['materialen_id'];
$aantal = $_REQUEST['aantal'];
$effectief_gebruikt = $_REQUEST['effectief_gebruikt'];
$opmerking = $_REQUEST['opmerking'];
$datum_van = date('YYYY-MM-DD',$_REQUEST['datum_van']);
$datum_tot = date('YYYY-MM-DD',$_REQUEST['datum_tot']);
$string = "INSERT INTO `materialen_per_reservatie`(`reservaties_id`, `materialen_id`, `aantal`, `effectief_gebruikt`, `opmerking`, `datum_van`, `datum_tot`) VALUES ($resID, $materialen_id, $aantal, $effectief_gebruikt, '$opmerking', $datum_van, $datum_tot)";
mysql_query($string);
you have to include single quotes for the date fields '$dataum_van'
$string = "INSERT INTO `materialen_per_reservatie`(reservaties_id, materialen_id, aantal, effectief_gebruikt, opmerking, datum_van, datum_tot) VALUES ($resID, $materialen_id, $aantal, $effectief_gebruikt, '$opmerking', '$datum_van', '$datum_tot')";
and this is only a example query, while implementing don't forget to sanitize your inputs
Your code has some serious problems that you should fix. For one, it is not doing any error checking, so it's no surprise the query breaks silently when it fails. Check for errors and it will tell you what goes wrong - how to do it is outlined in the manual on mysql_query() or in this reference question.. Example:
$result = mysql_query($string);
// Bail out on error
if (!$result)
{
trigger_error("Database error: ".mysql_error(), E_USER_ERROR);
die();
}
In this specific case, I'm fairly sure it's because you are not putting your values into quotes after the VALUES keyword.
Also, the code you show is vulnerable to SQL injection. You need to escape every value you use like so:
$resID = mysql_real_escape_string($_REQUEST['resID']);
for this to work, you need to put every value in your query into quotes.
try this
$string = "INSERT INTO `materialen_per_reservatie`(`reservaties_id`) VALUES ('".$resID."')";
$db = mysql_connect("localhost","root","123");
mysql_select_db("website_categorization") or die("\n error selecting database" );
$keyword_array = preg_split('/[\s,]+/', $tag);
foreach($keyword_array as $tag1)
{
mysql_query("INSERT INTO category_keyword(ID_Category, Keyword) VALUES(2,$tag1)");
}
echo "\nAffected rows are ".mysql_affected_rows()."\n";
mysql_close($db);
Can u tell me what is the problem with this code??...I intend to insert rows into the category_keyword table from an array $keyword_array. I get errors "Affected rows are -1" and insertion does not work
You should quote and escape string values.
You should also handle errors, to be notified of them.
You should also write distinct statements, to be able to read your code later (as well as let others to read it).
$tag1 = mysql_real_escape_string($tag1);
$sql = "INSERT INTO category_keyword(ID_Category, Keyword) VALUES(2,'$tag1')";
mysql_query($sql) or trigger_error(mysql_error()." in ".$sql);
insert multiple rows via a php array into mysql
You need to encapsulte the string $tag in a query, otherwise mysql will think its a column name
mysql_query("INSERT INTO category_keyword(ID_Category, Keyword) VALUES(2,'".mysql_real_escape_string($tag1)."')");
You should quote and escape your string columns
$tag1 =
mysql_real_escape_string($tag1);
mysql_query("INSERT INTO
category_keyword(ID_Category, Keyword)
VALUES(2,'$tag1')");
You should also handle the mysql query errors to know why the query get failed. With the current code you never know why it is failing.It is better to handle mysql errors.
mysql_query('Your query') or trigger_error(mysql_error());
You can use this:
mysql_query("INSERT INTO category_keyword SET ID_Category=2, Keyword=".$tag1.");
Better syntax to understand :)