500 Internal Server Error PHP timing out - php

I am receiving an internal server error for my PHP script timing out. I am taking 8000 rows from mySQL database and breaking each row into 3 different tables.
I did some research on stack and ran into this as a solution:
set_time_limit(0);
ignore_user_abort(1);
Now, I am just not receiving the error anymore in chrome, but still not adding to the database.
Here is a little bit of pseudo of what I am doing:
$db = new myConnectDB();
$query = "SELECT * FROM tableName";
$result = $db->query($query);
$numRows = $result->num_rows;
//beerName', '$style' , '$brewery', '$abv','$ibu','$breweryID', '$icon', '$iconBrew' , '$lat', '$long' , '$beerRating')";
for ($i=0; $i < 8000 ; $i++) {
//fetch associate
$row = $result->fetch_assoc();
//get all 13 variables from rom
$db2 = new myConnectDB();
//insert some variables into 1st table
mysqli_close($db2);
//todo: insert BeerTable / check and
$db3 = new myConnectDB();
//insert variables into second table
mysqli_close($db3);
//todo: insert BreweryTable / check and update
$db4 = new myConnectDB();
//insert variables into third table.
mysqli_close($db4);
}
echo $i;

This is bad practice to establish and break connection inside a loop. Try to refactor your code this way:
$db = new myConnectDB();
$db2 = new myConnectDB();
$db3 = new myConnectDB();
$db4 = new myConnectDB();
$query = "SELECT * FROM tableName";
$result = $db->query($query);
$numRows = $result->num_rows;
$insertQuery1 = "INSERT INTO tbl1 (col1, col2, col3) VALUES";
$insertQuery2 = "INSERT INTO tbl2 (col1, col2, col3) VALUES";
$insertQuery3 = "INSERT INTO tbl3 (col1, col2, col3) VALUES";
for ($i=0; $i < 8000 ; $i++) {
//fetch associate
$row = $result->fetch_assoc();
//get all 13 variables from rom
//insert some variables into 1st table
$insertQuery1 .= '(' . $val1 . ',' . $val2 . ',' . $val3 ')';
//todo: insert BeerTable / check and
//insert variables into second table
$insertQuery2 .= '(' . $val4 . ',' . $val5 . ',' . $val6 ')';
//todo: insert BreweryTable / check and update
//insert variables into third table.
$insertQuery1 .= '(' . $val7 . ',' . $val8 . ',' . $val9 ')';
}
$db1->query($insertQuery1);
$db2->query($insertQuery2);
$db3->query($insertQuery3);
mysqli_close($db2);
mysqli_close($db3);
mysqli_close($db4);
In this concept you won't query 3 inserts for each row. You collect 3 big insert queries and execute them after the loop. So this code would work faster.

Related

PHP pDO prepared statements to insert data in Access 2007

I am trying to insert data from an Excel file into an Access 2007 database using PDO prepared statements. What the code is trying to do is to check if the value in the Excel sheet exists in the database and if not to add it but it doesn't do anything
Here is the code:
$conn = connect_to_impact();
//******************************************
//select statement
$select = 'SELECT < ? FROM < ?';
$select_query = $conn->prepare($select);
for ($h = 2; $h<count($clmn[0]); $h++){
$value = $clmn[0][$h];
// query execution
$select_query->execute(array($clmn[0][1], $clmn[0][1]));
$impact_no_result = $select_query->fetchAll();
//query result in multidimensional array
$impact_no_arr = impact_no_select($impact_no_result);
// create an indexed array of results
$impact_no_r = impact_no_indexed($impact_no_arr, $clmn[0][1]);
if(is_null($impact_no_r)){
$insert_impact_no = 'INSERT INTO < ? (< ?) VALUES (< ?)';
$simple_arr = [$clmn[0][1], $clmn[0][1], $value];
$insert_query = $conn->prepare($insert_impact_no);
$insert_query->execute($simple_arr);
}
// then if the value in the column is not in the indexed array insert it
elseif(!in_array($value, $impact_no_r)){
$insert_impact_no = 'INSERT INTO < ? (< ?) VALUES (< ?)';
$simple_arr = [$clmn[0][1], $clmn[0][1], $value];
$insert_query = $conn->prepare($insert_impact_no);
$insert_query->execute($simple_arr);
}
}
Indeed replacements cannot be used for table name and column names in PDO prepared statements. Here is the code that fixed the issueȘ
$conn = connect_to_impact();
//******************************************
//select statement
$select = "SELECT ". $clmn[0][1] . " FROM " . $clmn[0][1];
//repeat for each row the the excel sheet starting from row 2
for ($h = 2; $h<count($clmn[0]); $h++){
//take each value from each row from column A
$value = $clmn[0][$h];
// query execution
$impact_no_result = $conn->query($select);
if($impact_no_result){
$impact_no_result_ass = $impact_no_result->fetchAll();
}
//if there is no result in the select query
if($impact_no_result == false){
$insert_impact_no = "INSERT INTO ". $clmn[0][1] . "(" . $clmn[0][1] . ") " . "VALUES ('" . $value . "');";
$execute = $conn->query($insert_impact_no);
}
//if there is a result in the select query
else {
//put query result in array
$impact_no_arr = impact_no_select($impact_no_result_ass);
//if value is not in array insert it
if(!in_array($value, $impact_no_arr)){
$insert_impact_no = "INSERT INTO ". $clmn[0][1] . "(" . $clmn[0][1] . ") " . "VALUES ('" . $value . "');";
$execute = $conn->query($insert_impact_no);
}
}
}

Move values from one database to another in mysql and Codeigniter

I have a very large log table from which I want to move rows between specific dates, putting them into the same table structure on an other database. I don't want to copy everything, only specific date rows, so that the table on the main database stays small-ish. So I have to SELECT the data I want and only move (and delete) that.
Keep in mind that there is a lot of data, and I don't want to copy it all with a mysqldump. I want to create a php file or function and I will add a crone job which will run after few days and move all specific data to other database from main database.
Use something like this:
$sql = $db->query("SELECT * FROM old_table LIMIT 100");
while($row = $sql->fetch_assoc()){
$values = "";
foreach($row as $v)
$values .= "'" . $db->real_escape_string($v) . "', ";
$values = rtrim($values, ", ");
$db->query("INSERT INTO new_table (" . implode(",", array_keys($row) . ") VALUES ($values)");
$db->query("DELETE FROM old_table WHERE `ID` = {$row->ID} LIMIT 1");
}
For two databases, use this instead:
$db = new MySQLi("hostname", "user", "password", "database");
$db2 = new MySQLi("hostname", "user", "password", "database");
$sql = $db->query("SELECT * FROM old_table LIMIT 100");
while($row = $sql->fetch_assoc()){
$values = "";
foreach($row as $v)
$values .= "'" . $db->real_escape_string($v) . "', ";
$values = rtrim($values, ", ");
$db2->query("INSERT INTO new_table (" . implode(",", array_keys($row) . ") VALUES ($values)");
$db->query("DELETE FROM old_table WHERE `ID` = {$row->ID} LIMIT 1");
}

PHP PDO MySQL parsing SQL query results efficiently

I am using PHP PDOs to parse the results I am receiving from MySQL queries against a database. I am now running into an issue with running out of allocated memory. Any suggestions on how to improve the efficiency of my code or another more efficient way to handle query results other than parsing PDOs? Thanks.
Here is my code (which will return whether or not a DB has referential integrity):
function generateDSN($host, $dbname)
{
return 'mysql:host='. $host . ';dbname=' . $dbname;
}
$dbName = $argv[2];
//Used for query construction
$dsn = generateDSN($argv[1], $argv[2]);
//create DSN
$link = new PDO($dsn, "username", "password");
//Connect to the database
//PDO($dsn, username, password)
if ($link->connect_error)
{
die("Connection failed: " . $link->connect_error);
}
else
{
echo "Connected successfully\n";
}
//validate database connection
$query = "SELECT TABLE_CONSTRAINTS.TABLE_NAME, KEY_COLUMN_USAGE.COLUMN_NAME, KEY_COLUMN_USAGE.REFERENCED_TABLE_NAME, KEY_COLUMN_USAGE.REFERENCED_COLUMN_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS RIGHT OUTER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE ON INFORMATION_SCHEMA.TABLE_CONSTRAINTS.CONSTRAINT_NAME = INFORMATION_SCHEMA.KEY_COLUMN_USAGE.CONSTRAINT_NAME WHERE INFORMATION_SCHEMA.KEY_COLUMN_USAGE.CONSTRAINT_NAME <> 'PRIMARY' AND INFORMATION_SCHEMA.TABLE_CONSTRAINTS.CONSTRAINT_TYPE = 'FOREIGN KEY' AND INFORMATION_SCHEMA.TABLE_CONSTRAINTS.TABLE_SCHEMA = '".$dbName."';";
//Create query to return list of tables with foreign keys
/* Query return format:
+--------------+------------------------+-----------------------+------------------------+
| TABLE_NAME | COLUMN_NAME | REFERENCED_TABLE_NAME | REFERENCED_COLUMN_NAME |
+--------------+------------------------+-----------------------+------------------------+
*/
$result = $link->query($query);
//Get query results
$x = 0;
$tableA = [];
$tableB = [];
//Create arrays to hold data from FK queries
while($tables = $result->fetch(PDO::FETCH_ASSOC))
{
$test[] = $tables;
$assoc = $test[$x];
//assign assoc[] the value of $query (will iterate through query table row by row)
$tableName = $assoc['TABLE_NAME'];
$columnName = $assoc['COLUMN_NAME'];
$refTableName = $assoc['REFERENCED_TABLE_NAME'];
$refColumnName = $assoc['REFERENCED_COLUMN_NAME'];
//get data values for each column
$fkQueryA = "SELECT DISTINCT " . $columnName . " FROM " . $dbName . "." . $tableName . " ORDER BY ".$columnName." ;";
$fkQueryB = "SELECT DISTINCT " . $refColumnName . " FROM " . $dbName . "." . $refTableName . " ORDER BY ".$refColumnName." ;";
//A -- Table with column that is the foreign key
//B -- Table with column that the foreign key references
$resultA = $link->query($fkQueryA);
$resultB = $link->query($fkQueryB);
//Get query results
while($var = $resultA->fetchColumn())
{
$tableA[] = $var;
}
//Push query results to table
while($vari = $resultB->fetchColumn())
{
$tableB[] = $vari;
}
//Push query results to table
$x++;
//increment counter to move through $tables
}
$resultCompAB = array_diff($tableA, $tableB);
//return array with all values of A that are not in B
if(empty($resultCompAB))
{
echo "Database ".$dbName." has referential integrity.";
}
else
{
echo "Orphan Values in database ".$dbName.": \n";
array_diff($tableB, $resultCompAB);
}
//print the results
Instead of the array_diff and storing GIGANT php arrays, you should get it with a Query.
This query, will give you values of A that are not in B:
$tableName = $assoc['TABLE_NAME'];
$columnName = $assoc['COLUMN_NAME'];
$refTableName = $assoc['REFERENCED_TABLE_NAME'];
$refColumnName = $assoc['REFERENCED_COLUMN_NAME'];
$sql = "SELECT * FROM {$tableName} LEFT JOIN {$refTableName}
ON {$tableName}.{$columnName} = {$refTableName}.{$refColumnName}
WHERE {$refTableName}.{$refColumnName} IS NULL
";

MySQL batch insert fails, no error

I'm trying to create a php function to split up the data into batches as it fails when I try to insert them fairly quickly.
I'm trying to insert thousands of records of user-data into a different format in the same database, later to be exported to a seperate database. However the query fails.
Based on comments and answers below I've updated the code to the following. Still fails, though.
The code inserting values:
function insertUsers( $users ){
error_reporting(E_ALL);
ini_set('display_errors',1);
global $pdo;
//insert into database
$i = 0;
$base = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered,
user_firstname, user_lastname ) VALUES ';
$sql = '';
var_dump($users);
while( $i < count( $users ) ){
$sql = $sql . ' ("' .
$users[$i]['user_login'] . '", "' .
$users[$i]['user_pass'] . '", "' .
$users[$i]['user_email'] . '", "' .
$users[$i]['user_registered'] . '", "' .
$users[$i]['meta_value']['first_name'] . '", "' .
$users[$i]['meta_value']['last_name'] . '")';
if (!( $i % 25 === 0 )){
$sql = $sql . ', ';
}
if ($i % 25 === 0) {
//execute $base + $query here
$sql = $base . $sql;
$query = $pdo->prepare( $sql );
echo 'check query: <br />';
print_r( $query );
if( $query->execute() ){
echo '50 users succesfully added to the database';
} else {
echo 'Query failed: ';
print_r( $pdo->errorInfo() );
echo '<br />';
}
$sql = ''; //Re-init query string
}
$i++;
}
if ( strlen( $sql ) > 0 ) { // Execute remainder, if any
//execute $base + $query here
$sql = $base . $sql;
$query = $pdo->prepare( $sql );
echo 'check query: <br />';
print_r($query);
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
echo 'Query failed: ';
print_r( $pdo->errorInfo() );
echo '<br />';
}
}
}
check query:
PDOStatement Object ( [queryString] => INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname ) VALUES ("John Smith", "4\/\/350M3 P4sS\/\/0r|)", "john.smith#greatmail.com", "2013-04-11 11:18:58", "John", "Smith") )
Query failed: Array ( [0] => 00000 [1] => [2] => )
Tried it with a %25 and %50, both don't work. Keep getting the 00000 error which is supposed to lead to victory (success, though for me it still fails, nothing in the DB)
I'd do it manually if I had the time but this won't be a one-time event so I need a solution to this issue. Is there a good way to split up the query into batches (and how?) that would allow this to be repeated and queries to be executed one after the other? I've been looking at a whole bunch of questions on SO (and elsewhere) already and can't find one that suits my needs.
UPDATE - has been answered, need a small modification as shown below:
if (!( $i % 25 === 0 )){
if(!( $i == ( count( $users ) - 1 ))){
$sql = $sql . ', ';
}
}
You're mixing PDO with mysql_ functions. Pick one and follow the respective library's error handling.
It would also be beneficial print out your $sql to yourself to see if it's formatted correctly. Additionally, if you're handling POSTed data, you will want to use prepared statements,
You might need to increase the max_allowed_packet value which defaults to 1Mb. If you want to split up the query in batches you can do so using the modulus operator.
$base = 'INSERT INTO ...';
$sql = '';
while( $i < count( $users ) ){
$sql = $sql . ' ("' ... //etc.
if ($i % 50 === 0) {
//execute $base + $qry here
$sql = ''; //Re-init query string
}
}
if (strlen($qry)>0) { // Execute remainder, if any
//execute $base + $query here
}
Or as an array (described here):
$base = 'INSERT INTO ...';
$sql = array();
while( $i < count( $users ) ){
$sql[] = ' ("' ... //etc.
if ($i % 50 === 0) {
//execute $base + implode(',', $sql) here
$sql = array(); //Re-init query string
}
}
if (sizeof($qry)>0) { // Execute remainder, if any
//execute $base + implode(',', $sql) here
}
Also please make sure you're using prepared statements correctly so you're not vulnerable to SQL injections.
Finally: you might need to enable error reporting so failures won't be silent; if they're still silent after enabling error reporting (e.g. error_reporting(-1);) you might need to set MySQL to strict mode (not sure if that will help). If they still fail silently file a bugreport.
Edit
Oh, I missed the fact that you're mixing mysql_ and PDO; that will probably be the reason why you're not seeing any errors... D'uh. Go read the manual on PDO error handling.
#Downvoters: If you're downvoting at least have the decency to leave a comment on why.
Is there a reason why you want to do it as one single statement? Why not just iterate through the users and insert each one with something like:
while($i < count( $users )) {
$sql = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname ) VALUES ';
$sql = $sql + "(";
$sql = $sql + "'" . $users[$i]['user_login'] . '",';
$sql = $sql + "'" . $users[$i]['user_pass'] . '",';
$sql = $sql + "'" . $users[$i]['user_email'] . '",';
$sql = $sql + "'" . $users[$i]['user_registered'] . '",';
$sql = $sql + "'" . $users[$i]['meta_value']['first_name'] . '",';
$sql = $sql + "'" . $users[$i]['meta_value']['last_name'] . '"';
$sql = $sql + ")";
$query = $pdo->prepare( $sql );
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
die ("Query failed: " . $pdo->errorInfo());
}
}
Even better is to prepare the statement once and then bind the parameters. As pointed out by RobIII, you can introduce SQL injection vulnerabilities by building up your SQL statements as strings so instead, you could do something like:
$sql = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname) ';
$sql += ' VALUES (:user_login, :user_pass, :user_email, :user_registered, :user_firstname, :user_lastname)';
$query = $pdo->prepare( $sql );
while ($i < count($users)) {
$query->bindParam(":user_login", $users[$i]['user_login']);
$query->bindParam(":user_pass", $users[$i]['user_pass']);
$query->bindParam(":user_email", $users[$i]['user_email']);
$query->bindParam(":user_registered", $users[$i]['user_registered']);
$query->bindParam(":user_firstname", $users[$i]['user_firstname']);
$query->bindParam(":user_lastname", $users[$i]['user_lastname']);
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
die ("Query failed: " . $pdo->errorInfo());
}
}
Another issue you may be encountering is a simple fact that the user's name / information may contain invalid characters and killing your command. Take for instance a person's last name is "O'Maley". When building your string of values and wrapping the name with hard quotes, you would get
'O'Maley'
I ran into something similar to this about 8 years ago and had to validate certain values within the data. Once you find it, you'll know how to correct each respective value, then go back to batch mode.
Also, consider someone supplying a bogus value of a bogus / bad value of "--" which indicates rest of line is a comment... or even a ";" to indicate end of statement. That is most likely what you are running into.
As RobIII responded, and I was too quick in my response, don't test one at a time, but query the data an look at it for such anomolies / bad escape posts in the data. Fix what you need to BEFORE trying the insert.

Query to import from Oracle to MySQL returning null object

I'm inserting data from oracle database to mysql database with php than insert and update, but I have a strange behaviour. The table oracle have 20851 records.
The problem is, in the record x, the query to mysql return me a empty result object, but the same query executed on MySQL returned objects with data.
With the following code I can insert and update data from oracle to mysql.
$stid = oci_parse($conn, 'SELECT * FROM B_PROGRAMA_EVALUACION_BIPS'); //oracle db
oci_execute($stid); //oracle db
while ($row = oci_fetch_array($stid, OCI_ASSOC+OCI_RETURN_NULLS)) {
$sql = "select ID_ITEM, ID_PROGRAMA from B_PROGRAMA_EVALUACION_BIPS WHERE ID_ITEM=".$row['ID_ITEM']." AND ID_PROGRAMA=".$row['ID_PROGRAMA'];
$result = $db->query($sql);
$rows = mysqli_fetch_array($result);
if ($rows['ID_ITEM'] == $row['ID_ITEM'] && $rows['ID_PROGRAMA'] == $row['ID_PROGRAMA']) {
$sql = "UPDATE B_PROGRAMA_EVALUACION_BIPS SET ID_PROGRAMA='".$row['ID_PROGRAMA'] . "', ANO='" . $row['ANO'] . "', COPIA='" . $row['COPIA'] . "', TIPO_PROGRAMA_ER='"
. $row['TIPO_PROGRAMA_ER'] . "', EVALUACION='".mysqli_real_escape_string($db, $row['EVALUACION'])."', ID_CARACTERICACION='".$row['ID_CARACTERICACION']."', NOTA='".$row['NOTA']."' WHERE ID_ITEM=".$row['ID_ITEM'];
} else {
$sql = "INSERT INTO B_PROGRAMA_EVALUACION_BIPS VALUES('".$row['ID_PROGRAMA'] . "','" . $row['ANO'] . "','" . $row['COPIA'] . "','"
. $row['ID_ITEM'] . "','".$row['ID_CARACTERICACION']."','".mysqli_real_escape_string($db, $row['EVALUACION'])."','".$row['TIPO_PROGRAMA_ER']."','".$row['NOTA']."')";
}
$result = $db->query($sql);
if ($result != 1) {
$resultado['B_PROGRAMA_EVALUACION_BIPS'] = $resultado['B_PROGRAMA_EVALUACION_BIPS'] + 1;
}
}
If I delete this line the second execute query, the first query working perfectly.
$result = $db->query($sql);
You should add single quotes around the values:
$sql = "select ID_ITEM, ID_PROGRAMA from B_PROGRAMA_EVALUACION_BIPS WHERE ID_ITEM='".$row['ID_ITEM']."' AND ID_PROGRAMA='".$row['ID_PROGRAMA']"'";

Categories