I have a PHP script that is passing large quantities of queries to a DB very quickly. Does a MySQL DB queue the queries as they come in if it can't process them at the same speed they are being passed, or do they get lost?
My program has written and passed syntactically correct queries to the DB, but the DB is very far behind in terms of information contained in tables and number of tables.
Some example code (I am slightly new to PHP, so my code/coding style may be horrifying):
//If table has one primary key
$val = $tblColPkArray[0];
$pkInsert = ", PRIMARY KEY (". $val['COLUMN_NAME'] .")";
$pkColName = $val['COLUMN_NAME'];
$string = ltrim($string, ",");
$oneCreateTableQuery = $beginning . $string . $pkInsert . $end;
echo $oneCreateTableQuery . "\n";
$newLink->query($oneCreateTableQuery);
$pkValuesInOld = "SELECT " . $pkColName . " FROM " . $tables . ";";
$pkValsResult = $link->query($pkValuesInOld);
while($pkValues = $pkValsResult->fetch(PDO::FETCH_ASSOC))
{
$pkRowValuesQuery = "SELECT * FROM " . $tables . " WHERE " . $pkColName . " = '" . $pkValues[$pkColName] . "';";
echo $pkRowValuesQuery . "\n";
$valsOfPkInOldTable = $link->query($pkRowValuesQuery);
while($pkVals = $valsOfPkInOldTable->fetch(PDO::FETCH_ASSOC))
{
//var_dump($ckVals);
$insertBeginning = "INSERT INTO " . $tables . "(";
$insertValuesSection = ") VALUES (";
$insertEnd = ");";
$keys = "";
$rowValues = "";
foreach($pkVals as $key => $value)
{
$keys = $keys . ", " . $key;
$rowValues = $rowValues . ", '" . $value . "'";
}
$insertStatement = $insertBeginning . ltrim($keys, ",") . $insertValuesSection . ltrim($rowValues, ",") . $insertEnd;
echo $insertStatement . "\n";
$newLink->query($insertStatement);
}//While loop: Inserting values of old table into new table via INSERT INTO statement
//unset();
} //While loop: Selecting all values from old table based on PK values per table, pass result of that query to next while loop
You can do this in a single query, instead of calling multiple insert statements.
For instance, instead of running these 3 queries,
INSERT INTO table VALUES(1, 2);
INSERT INTO table VALUES(3, 4);
INSERT INTO table VALUES(5, 6);
...
You could run this query:
INSERT INTO table VALUES(1, 2), (3, 4), (5, 6), ...;
Looks like you could do even combine the INSERT and SELECT:
INSERT INTO table (...)
SELECT ... FROM ...;
Furthermore, your nested loops look like this might work:
INSERT INTO table (...)
SELECT ... FROM ...
JOIN ...;
That would get it down to one call to ->query() and eliminate most of your code.
Related
This is part of the query where it supposed to search through morning, afternoon and evening columns in posts table.
$add_here .= " AND posts.".$col." IN ('" . implode("',' ",$days) . "')";
$col is the variable that holds array or single value for column name supplied by user. Namely 'morning','afternoon','evening'. Do you know how to alter the query above to search in the array($col)?
The below works because I specified column name which is 'afternoon'. I wonder how to replace the text 'afternoon' with array of values!
$add_here .= " AND posts.afternoon IN ('" . implode("',' ",$days) . "')";
Use array_map to build a series of conditions grouped with OR :
$days = ['sun','mon','tue'];
$col = ['afternoon','morning','night'];
$add_here = '';
$add_here .= sprintf(" AND (%s)",
implode(' OR ', array_map(
function($colx) use ($days) {
return sprintf("posts.%s IN ('%s')", $colx, implode("','",$days));
}, is_array($col) ? $col : array($col))
)
);
AND (posts.afternoon IN ('sun','mon','tue') OR posts.morning IN
('sun','mon','tue') OR posts.night IN ('sun','mon','tue'))
I'm new to PHP, and I do not it's syntax and principles very well.
I have such code:
function exportFromTransbase($table_name) {
//$odbc_query = "SELECT * FROM " . $table_name. " WHERE ((CDS_CTM subrange(248 cast integer) = 1) AND (CDS_LNG_ID = 16))";
$odbc_query = "SELECT * FROM " . $table_name. "";
$data = odbc_exec($this->odbc_id, $odbc_query);
odbc_longreadlen($data, 10485760);
$oufile=fopen($table_name.".sql", 'w') or die("error writing file");
$q1='INSERT INTO `' . substr($table_name, 4) . '` VALUES';
fwrite($oufile, $q1);
while($row = odbc_fetch_array($data))
{
foreach($row as $key => $value) {
$keys[] = "`" . $key . "`";
if ($value == ""){
$value = 'NULL';
$values[] = "" . mysql_real_escape_string($value) . "";
}
else{
$values[] = "'" . mysql_real_escape_string($value) . "'";
}
//echo "\n \n ololo ".$value;
}
$mysql_query = "\n (".implode(",", $values).")," ;
fwrite($oufile, $mysql_query);
//mysql_query($mysql_query);
set_time_limit(0);
unset($keys);
unset($values);
unset($row);
}
$stat = fstat($oufile);
ftruncate($oufile, $stat['size']-1);
fseek($oufile, 0, SEEK_END);
fwrite($oufile, ";".$r);
//} while ($r < 5 );
fclose($oufile);
if ($mysql_query){
print "Ýêñïîðò äàííûõ èç òàáëèöû " . $table_name . " çàâåðøåí!";
//strtolower(substr($table_name, 4))
}
}
what and where i need to custom, so that i export all table fields except one, for example called Size, i must insert in db is this field nulls....
Also if it is easy, how to split my sql query in batches of 5000 rows? so insert (5000 rows) then insert another 5000....
But first i need to export all fields, except one...
To the best of my knowledge, it can't be possible.
You can use simply:
SELECT field1, field2, field3, field4 FROM table
See Select all columns except one in MySQL?
And if you have more fields in your table then you can use * only.
At the time of insert you can make a condition which ignores the field which you don't want in new table.
Can you use
"select * from {table} where {sth} != {condition}"
Then you can fetch all data except one. The one is excrpted follow your condition.
Meanwhile,if you want to insert by 5000, you can read 5000 lines one time and insert into.
I am writing a code, where I have to produce a query with many OR statements, and I think there is a more comfortable way to this than:
foreach ($plz as &$value) {
if (empty($i)) {
$query = "WHERE plz='$value'";
} else {
$query = "$query OR plz='$value'";
}
$i++;
}
$sql = mysql_query("SELECT * FROM table $query");
while ($data = mysql_fetch_array($sql)) {
//do something
}
If you have multiple values a column may take, just connect them using the IN operator:
Instead of writing
... WHERE col=1 OR col=2 OR col=3
just write
... WHERE col IN (1,2,3)
To collect all entries in PHP, use an array and implode() later on:
// collecting values
$vals = array();
$vals[] = 1;
$vals[] = 2;
// ...
// add them to your query
$query .= ' WHERE col IN ( ' . implode( ',', $vals ) . ')';
// execute the query ...
In case your values are not integer, but need to be enclosed in apostrophes within the query, insert them that way into the array in the first place:
$vals[] = "'my string value'";
You're looking for ;
SELECT * FROM table WHERE plz in ('value1', 'value2', 'value3')
Be aware of SQL injections...
If the column plz is INT type, and all $plz are also integers, then:
$query = 'WHERE plz IN( ' . implode(',', $plz) . ')';
would work. Otherwise, trying this might work(not tested):
$query = 'WHERE plz IN( \'' . implode("','", $plz) . '\')';
My part_no column has the following format: 000-00000-00 for all records.
I need to extract the five middle characters from part_no and place it in the core column when I create the record.
I can't get my script to work.
I'm not getting any errors. Just not working.
$order = "INSERT INTO cartons_added (add_time, type, part_no, add_type, add_qty, add_ref, add_by, add_notes)
VALUES
('$date',
'$_POST[type]',
'$_POST[part_no]',
'$_POST[add_type]',
'$_POST[add_qty]',
'$_POST[add_ref]',
'$_POST[add_by]',
'$_POST[add_notes]')";
$result = mysql_query($order);
$query2 = "select part_no from cartons_current";
$sel = mysql_query($query2);
$res = mysql_result($sel);
while($row = mysql_fetch_row($res)) {
$core_digits = split('-',$row[0]);
$core =$core_digits[1];
$query3 = "insert into cartons_current(core) values($core)";
$sel2 = mysql_query($query3);
}
You can update your cartons_current table based on your cartons_added table with something like:
INSERT INTO cartons_current(core)
SELECT SUBSTR(part_no, 5, 5) FROM cartons_added
You will probably want to limit that with a WHERE clause or maybe deal with what happens when this value is already in cartons_current (use either INSERT IGNORE or ON DUPLICATE KEY UPDATE)
You are right, the script has no error.
I think the problem is on your SQL that made you can't insert a new row, specifically on the table structure. Maybe you defined a PRIMARY KEY without AUTO_INCREMENT, defined a INDEX or UNIQUE key that is not the core key or there have some other key that did not have default value. Remember that you can't insert a row without defining all required field.
You script is selecting all part_no and for every part_no you are inserting a new row in the same table, so maybe there is the problem.
I think what you want is update every result to add they core value, you can do that with UPDATE as this code:
function getValue($value) {
return "'" . trim(mysql_real_escape_string($value)) . "'";
}
mysql_query('INSERT INTO `cartons_added` (`add_time`, `type`, `part_no`, `add_type`, `add_qty`, `add_ref`, `add_by`, `add_notes`)
VALUES (' .
getValue($date) . ', ' .
getValue($_POST[type]) . ', ' .
getValue($_POST[part_no]) . ', ' .
getValue($_POST[add_type]) . ', ' .
getValue($_POST[add_qty]) . ', ' .
getValue($_POST[add_ref]) . ', ' .
getValue($_POST[add_by]) . ', ' .
getValue($_POST[add_notes]) .
')');
$partNoQuery = mysql_query('SELECT `part_no` FROM `cartons_current`');
while($partNoResult = mysql_fetch_assoc($partNoQuery)) {
list($prefix, $core, $suffix) = explode('-', $partNoResult['part_no']);
mysql_query('UPDATE cartons_current SET `core` = \'' . $core . '\' WHERE `part_no` = \'' . $partNoResult['part_no'] . '\'');
}
I added getValue function to escape posted data to prevent SQL injection.
Try removing this
$res = mysql_result($sel);
And change your while to reference the main query resource
while($row = mysql_fetch_row($sel)) {
I don't understand your logic with your tables though. You're inserting data into the cartons_added table but then you're selecting from cartons_current?
Also, split is deprecated as of PHP 5.3.0
You said five middle "characters", so I'd add quotes around your variable like so:
$query3 = "insert into cartons_current(core) values('$core')";
(Also, there's only about a gazillion answers on SO about SQL injection, and using pdo)
INSERT INTO cartons_current(core)
SELECT
substr(part_no,position('-' IN part_no)+1,position('-' IN substr(part_no,position('-' IN part_no)+1))-1)
FROM cartons_added;
I've got a SQL query within a foreach loop. Sometimes there can be many, and I mean a lot of queries to do, depending on several criteria, up to 78 queries potentially.
Now, I know that premature optimisation is root cause of all evil, but I don't want to see 78 queries - it's just not healthy.
Here's the code:
$crumbs = explode(",", $user['data']['depts']);
foreach ($crumbs as &$value) {
$data = $db->query("SELECT id FROM tbl_depts WHERE id = '" . $value . "'");
$crumb = $data->fetch_assoc();
$dsn = $db->query("SELECT msg, datetime FROM tbl_motd WHERE deptid = '" . $value . "'");
$motd = $dsn->fetch_assoc();
if ($motd['msg'] != "") {
<?php echo $motd['msg']; ?>
}
}
Can I make it any better?
Use IN MySQL operator to search over a set of values for id:
$ids = '"' . implode('", "',$crumbs) . '"';
$query1 = "SELECT id FROM tbl_depts WHERE id IN (" . $ids . ")";
$query2 = "SELECT msg, datetime FROM tbl_motd WHERE deptid IN (" . $ids . ")";
And so you will not need to retrieve all data you need using foreach loop, so you will have only 2 queries instead of 78.
Example: I have a table named table with 10 records which ids are: 1,2,3,4,5,6,7,8,9,10 (auto-incremented). I know I need records with ids 1,5,8. My query will be:
$sql = "SELECT * FROM `table` WHERE id in (1,5,8);";
And I don't understand why do you need to use & operator in foreach loop if you don't modify the $crubms arrays values.
I think this is want you want.
SELECT msg, datetime
FROM tbl_depts td
INNER JOIN tbl_motd tm ON td.id = tm.deptid