I have one table based on which one I have to update 6 rows in the other table for matching ids. It is total of over 1000 records so most of the time I get timeout error with current script.
The way I do it now is, I select the range of ids between two dates from the first table, store it into an array and then run foreach loop making update in the second table where the ids are the same, so basically I run a query for every single id.
Is there anyway I could speed it up the process?
I found only a way to generate the each within the foreach loop
UPDATE product SET price = CASE
WHEN ID = $ID1 THEN $price1
WHEN ID = $ID1 THEN $price2
END
But I don't know how could I modify this to update multiple rows at the same time not just one.
My script code look like that
$sql = "SELECT * FROM `games` where (ev_tstamp >= '".$timestamp1."' and ev_tstamp <= '".$timestamp2."')";
while($row = mysqli_fetch_array($sql1)){
$one_of =[
"fix_id" =>$row['fix_id'],
"t1_res" =>$row['t1_res'],
"t2_res" =>$row['t2_res'],
"ht_res_t1" =>$row['ht_res_t1'],
"ht_res_t2" =>$row['ht_res_t2'],
"y_card_t1" =>$row['y_card_t1'],
"y_card_t2" =>$row['y_card_t2'],
"t1_corners" =>$row['t1_corners'],
"t2_corners" =>$row['t2_corners'],
"red_card_t1" =>$row['red_card_t1'],
"red_card_t2" =>$row['red_card_t2']
];
array_push($today_games,$one_of);
}
foreach($today_games as $key=>$val){
$cards_t1=$val['red_card_t1']+$val['y_card_t1'];
$cards_t2=$val['red_card_t2']+$val['y_card_t2'];
$sql = "Update sights SET t1_res='".$val['t1_res']."',
t2_res='".$val['t2_res']."', ev_tstamp='".$val['ev_tstamp']."',
ht_res_t1='".$val['ht_res_t1']."', ht_res_t2='".$val['ht_res_t2']."',
t1_corners='".$val['t1_corners']."',t2_corners='".$val['t2_corners']."',
t1_cards='".$cards_t1."',t2_cards='".$cards_t2."'
where fix_id='".$val['fix_id']."' "
}
Consider an UPDATE...JOIN query using fix_id as join column. Below runs mysqli parameterized query using timestamps. No loop needed.
$sql = "UPDATE sights s
INNER JOIN `games` g
ON s.fix_id = g.fix_id
AND g.ev_tstamp >= ? and g.ev_tstamp <= ?
SET s.t1_res. = g.t1_res,
s.t2_res. = g.t2_res,
s.ev_tstamp = g.ev_tstamp,
s.ht_res_t1 = g.ht_res_t1,
s.ht_res_t2 = g.ht_res_t2,
s.t1_corners = g.t1_corners,
s.t2_corners = g.t2_corners,
s.t1_cards = (g.red_card_t1 + g.y_card_t1),
s.t2_cards = (g.red_card_t2 + g.y_card_t2)";
$stmt = mysqli_prepare($conn, $sql);
mysqli_stmt_bind_param($stmt, 'ss', $timestamp1, $timestamp2);
mysqli_stmt_execute($stmt);
I have a table like this:
I receive an array like this:
$data = array(
array('m_id'=>1,'d_id'=>101,'available'=>0),
array('m_id'=>1,'d_id'=>102,'available'=>1),
array('m_id'=>1,'d_id'=>103,'available'=>1),
array('m_id'=>1,'d_id'=>104,'available'=>0),
array('m_id'=>1,'d_id'=>105,'available'=>0)
);
My question is how I can update the table just with one query like this:
$query = "UPDATE tbl_name SET available='".$data[0]['available']."'" WHERE conditon1";
I mean update table once instead of 5 query.
In pure MySQL, you could use a CASE expression in the UPDATE to handle this:
UPDATE tbl_name
SET available = CASE WHEN d_id = 101 THEN 0
WHEN d_id = 102 THEN 1
WHEN d_id = 103 THEN 1
WHEN d_id = 104 THEN 0
WHEN d_id = 105 THEN 0 END
WHERE d_id IN (101, 102, 103, 104, 105)
If, on the other hand, you wanted to iterate over your array of arrays and then issue UPDATE statements, you could try the following:
foreach ($data as $entry) {
$query = "UPDATE tbl_name SET available=".$entry['available']." WHERE d_id=".$entry['d_id'];
// execute $query ...
}
Possible duplicate of this one. In a nutshell - you need to use either CASE conditional or INSERT ... ON DUPLICATE KEY UPDATE construct.
Based on the data you have provided, where all rows reference the same columns, and always update the same column, you can create a simple CASE expression:
/* Mostly untested */
function update_array ($data)
{
if (empty ($data))
return NULL;
if (count ($data) === 1) {
/* Simple query for single update */
$mi = $data[0]['m_id'];
$di = $data[0]['d_id'];
$av = $data[0]['available'];
$sql = <<<_
UPDATE `tbl_name`
SET `available` = $av
WHERE `m_id`= $mi AND `d_id`= $di
_;
} else {
/* CASE WHEN query for multiple updates */
$sql = "UPDATE `tbl_name`\n SET `available` = \n CASE ";
foreach ($data as $d) {
$mi = $d['m_id'];
$di = $d['d_id'];
$av = $d['available'];
$sql .= "WHEN `m_id`=$mi AND `d_id`=$di THEN $av\n ";
}
/* Never forget the ELSE clause! */
$sql .= "ELSE `available`\n END\n";
}
return $sql;
}
This test program:
$data = array(
array('m_id'=>1, 'd_id'=>101, 'available'=>0),
array('m_id'=>1, 'd_id'=>102, 'available'=>1),
array('m_id'=>1, 'd_id'=>103, 'available'=>1),
array('m_id'=>1, 'd_id'=>104, 'available'=>0),
array('m_id'=>1, 'd_id'=>105, 'available'=>0)
);
echo update_array ($data);
outputs:
UPDATE `tbl_name`
SET `available` =
CASE WHEN `m_id`=1 AND `d_id`=101 THEN 0
WHEN `m_id`=1 AND `d_id`=102 THEN 1
WHEN `m_id`=1 AND `d_id`=103 THEN 1
WHEN `m_id`=1 AND `d_id`=104 THEN 0
WHEN `m_id`=1 AND `d_id`=105 THEN 0
ELSE `available`
END
Note:
Never forget the ELSE clause in a CASE WHEN update. If you do, then all the non-matching rows will be overwritten with a blank value.
This code blindly assumes that the data contains the expected indexes, and that the data is of the appropriate type.
Some say that you should include a WHERE clause to limit the number of rows to be updated for performance reasons. I haven't seen any data to back this up, but if it is a concern, then a simple improvement might be to narrow it down using the d_id column. Just add the follow at the end of the CASE WHEN query:
$ids = array_column ($data, 'd_id');
$sql .= ' WHERE `d_id` IN (' . implode (', ', $ids) . ")\n";
You still need the ELSE in the CASE expression because this WHERE clause is not guaranteed to exclude all the unwanted rows. Writing a complete WHERE clause will probably be much more complicated, depending on the exact nature of your data.
So I have 2 tables named accounts and tblDataNumbers. The accounts table has a sponsorID field while the tblDataNumbers table has xsponsorID field. So first, this is the code:
for ($x = 501; $x <= 1061; $x++) {
$data = $conn->query("SELECT * FROM tblDataNumbers WHERE tableID = $x");
foreach ($data as $d) {
$xsponsorID = $d['xsponsorID'];
}
$checkSponsor = $conn->query("SELECT * FROM accounts WHERE sponsorID = '".$xsponsorID."' ");
if(count($checkSponsor) > 0) {
$xsid = $xsponsorID;
} else {
$xsid = "";
}
$stmt = $conn->prepare('UPDATE tblDataNumbers SET xsponsorID=:xsponsorID WHERE tableID=:x');
$stmt->execute(array(
':xsponsorID' => $xsid,
':x' => $x
));
}
The code above will check if the $xsponsorID is existing on the sponsorID field of the accounts table. If yes, it will set $xsid to the same value which is $xsponsorID and if not, it will be set to blank or "". When I run the code, nothing happens. There are lots of numbers in the xsponsorID field of the tblDataNumbersthat table that is not present in the sponsorID field of the accounts table so basically something should be set to blank but not even one.
What I wanted to happen with this is to repeat the code inside from row 501 to 1061 of table tblDataNumbers but nothing happens. Is there something wrong with my code? Thank you.
I don't know if this is what you are attempting but that could potentially kill the server for a while as there will be 560 x records (from 1st query ) = lots!
I think there must be a neater solution that does the work in one query - I shall look at the queries in detail to see if there is a solution.
for( $x = 501; $x <= 1061; $x++ ) {
$data = $conn->query("SELECT * FROM tblDataNumbers WHERE tableID = $x");
/* This loop will only ever get one result..
foreach( $data as $d ){
$xsponsorID = $d['xsponsorID'];
}
*/
foreach( $data as $d ){
$xsponsorID = $d['xsponsorID'];
$checkSponsor = $conn->query("SELECT * FROM accounts WHERE sponsorID = '".$xsponsorID."' ");
$xsid=( count( $checkSponsor ) > 0 ) ? $xsponsorID : '';
$stmt = $conn->prepare('UPDATE `tblDataNumbers` SET `xsponsorID`=:xsponsorID WHERE `tableID`=:x;');
$stmt->execute(array(
':xsponsorID' => $xsid,
':x' => $x
));
}
}
After looking at this again and reading your comments I wondered if something along the lines of the following might do the trick. I created a new database with the same two named tables and populated some test data ( though I do not know if it resembles your own data ) and used the following as a test within heidi ( mysql gui app )
set #tblid=1;
update `tblDataNumbers` t set t.`xsponsorID`=(
case
when ( t.`xsponsorID` not in ( select `sponsorID` from `accounts` ) ) then
null
else
t.`xsponsorID`
end
)
where t.`tableID`=#tblid;
Clearly this updates a record where the tableID is equal to the #tblid var only. However, to update all the records that match the criteria, remove the where clause and it will apply the NULL value to any record in the tblDataNumbers table that does not have a corresponding sponsorID in the accounts table. Before running this I would strongly advise making a backup of your data though!!!!
Without the mysql variable the query becomes the following and means you would not need the loops.
update `tblDataNumbers` t set t.`xsponsorID`=(
case
when ( t.`xsponsorID` not in ( select `sponsorID` from `accounts` ) ) then
null
else
t.`xsponsorID`
end
)
If it is important to update the records only between 501 and 1061 then you could add a where clause such as:-
where t.`tableID` between 501 and 1061
Thanks to #RamRaider, managed to fix it by adding a single line and removing the count() function.
for( $x = 501; $x <= 1061; $x++ ) {
$data = $conn->query("SELECT * FROM tblDataNumbers WHERE tableID = $x");
foreach( $data as $d ){
$xsponsorID = $d['xsponsorID'];
$checkSponsor = $conn->query("SELECT * FROM accounts WHERE sponsorID = '".$xsponsorID."' ");
$row = $checkSponsor->fetch(PDO::FETCH_ASSOC); //I forgot this line
$xsid=( $row > 0 ) ? $xsponsorID : '';
$stmt = $conn->prepare('UPDATE `tblDataNumbers` SET `xsponsorID`=:xsponsorID WHERE `tableID`=:x;');
$stmt->execute(array(
':xsponsorID' => $xsid,
':x' => $x
));
}
}
I have one table with some rows and lot of columns (about 50)
- I don't want to list all the column names, co I use a while loop to go through the table and a foreach command to find the column names and values.
Now I would like to add these two rows into one row (where id is 1) and sum the values together...
id | col 1 | col 2 | ...
1 30 21
1 11 16
2 75 0
It should look like this
id | col1 | col2 | ...
1 41 37
2 75 0
This is what I have...
$query = mysql_query("SELECT * FROM `table` WHERE `id`='1'");
while ($row = mysql_fetch_assoc($query)) {
foreach($row as $key => $val) {
if($key != 'id') {
//the sum code...?
}
}
}
Could you please help me? Thank you a lot...
Based on your code which indicates you're using mysql I'm going to give you a mysql solution (which can easily be ported to most RDBMS). What you're looking for can easily be accomplished with aggregate functions. Follow the link to read about all the aggregate functions mysql has.
SELECT id, SUM(col1), SUM(col2) FROM table GROUP BY id;
just sum the results in mysql.. databases are made to handle things like this so it'll be a faster solution than doing it just in php.
SELECT
id,
SUM(col1),
SUM(col2)
FROM table
GROUP BY id;
Try this: See the Explanation as comments:
<?php
$query = mysql_query("SELECT * FROM `table` WHERE `id`='1'");
//Initially Set all the columns as zero.
$col1 = 0; $col2 = 0;
//Etc.
while ($row = mysql_fetch_assoc($query)) {
foreach($row as $key => $val) {
if($key != 'id') {
$$key += $val;
//Like: $col1 += 30;
}
}
}
//Now All the variables are ready added:
//Like: $col1 = 41;
//Like: $col2 = 37;
//Use them However you like:
//To update:
//1. First Delete both rows:
$query = mysql_query("DELETE FROM `table` WHERE `id`='1'");
//2. Insert
$query = mysql_query("INSERT INTO `table` (`id`,`col1`,`col2`) VALUES ('1','{$col1}','{$col2}') ");
//And so on
?>
Right, this code goes through a rather large multidimensional array (has about 28,000 rows and 16 parts).
Order of events:
Check if the data exists in the database
if it exists - Update it with the new data
if it doesn't exist - Insert it
Simple.
But right now to go through this it has taken over 30min i think and Still going.
$counter = 0;
$events = sizeof($feed_array['1'])-1;
while($counter <= $events ) {
$num_rows = mysql_num_rows(mysql_query("SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"));
if($num_rows) {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
mysql_query("UPDATE `it_raw` SET
`eventtime` = '".$eventUnixTime."',
`eventname` = '".addslashes($feed_array['3'][$counter])."',
`venuename` = '".addslashes($feed_array['4'][$counter])."',
`venueregion` = '".addslashes($feed_array['5'][$counter])."',
`venuepostcode` = '".addslashes($feed_array['6'][$counter])."',
`country` = '".addslashes($feed_array['7'][$counter])."',
`minprice` = '".addslashes($feed_array['8'][$counter])."',
`available` = '".addslashes($feed_array['9'][$counter])."',
`link` = '".addslashes($feed_array['10'][$counter])."',
`eventtype` = '".addslashes($feed_array['11'][$counter])."',
`seaOnSaleDate` = '".addslashes($feed_array['12'][$counter])."',
`perOnSaleDate` = '".addslashes($feed_array['13'][$counter])."',
`soldOut` = '".addslashes($feed_array['14'][$counter])."',
`eventImageURL` = '".addslashes($feed_array['15'][$counter])."',
`perfID`= '".addslashes($feed_array['16'][$counter])."'
WHERE `perfID` = ".$feed_array['16'][$counter]." LIMIT 1 ;");
echo "UPDATE ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
} else {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql = "INSERT INTO `dante_tickets`.`it_raw` (
`id` ,
`eventtime` ,
`eventname` ,
`venuename` ,
`venueregion` ,
`venuepostcode` ,
`country` ,
`minprice` ,
`available` ,
`link` ,
`eventtype` ,
`seaOnSaleDate` ,
`perOnSaleDate` ,
`soldOut` ,
`eventImageURL` ,
`perfID`
)
VALUES (
NULL ,
'".$eventUnixTime."',
'".addslashes($feed_array['3'][$counter])."',
'".addslashes($feed_array['4'][$counter])."',
'".addslashes($feed_array['5'][$counter])."',
'".addslashes($feed_array['6'][$counter])."',
'".addslashes($feed_array['7'][$counter])."',
'".addslashes($feed_array['8'][$counter])."',
'".addslashes($feed_array['9'][$counter])."',
'".addslashes($feed_array['10'][$counter])."',
'".addslashes($feed_array['11'][$counter])."',
'".addslashes($feed_array['12'][$counter])."',
'".addslashes($feed_array['13'][$counter])."',
'".addslashes($feed_array['14'][$counter])."',
'".addslashes($feed_array['15'][$counter])."',
'".addslashes($feed_array['16'][$counter])."'
);";
mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
}
unset($sql);
$counter++;
}
UPDATE
I just carried out profiling one one of the rows:
mysql> EXPLAIN SELECT * FROM it_raw WHERE perfID = 210968;
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
| 1 | SIMPLE | it_raw | ref | perfID | perfID | 4 | const | 1 | |
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
1 row in set (0.07 sec)
UPDATE 2
To try and "speed" things up, instead of carrying out the UPDATE and INSERT statements straight away, i've now placed them in a variable (so only the initial select runs - to check a duplicate - then stores the action [insert or update]). At the end of the loop it executes all the statments.
Except now, it's coming up with MySQL error that the syntax is incorrect. (when initially there was nothing wrong).
I've simply replaced the mysql_query with:
$sql_exec .= "SELECT.... ;";
is there something i'm missing here for the formatting?
UPDATE 3
OK finally fixed it
Lessons Learned:
Do logic search first on database
Carry out insert/updates in bulk.
Here is the final code which now takes about 60 seconds to run (from over 30min+)
while($counter <= $events ) {
$num_rows = mysql_num_rows(mysql_query("SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"));
if($num_rows) {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql_exec[] = "UPDATE `it_raw` SET `eventtime` = '".$eventUnixTime."',`eventname` = '".addslashes($feed_array['3'][$counter])."',`venuename` = '".addslashes($feed_array['4'][$counter])."',`venueregion` = '".addslashes($feed_array['5'][$counter])."',`venuepostcode` = '".addslashes($feed_array['6'][$counter])."',`country` = '".addslashes($feed_array['7'][$counter])."',`minprice` = '".addslashes($feed_array['8'][$counter])."',`available` = '".addslashes($feed_array['9'][$counter])."',`link` = '".addslashes($feed_array['10'][$counter])."',`eventtype` = '".addslashes($feed_array['11'][$counter])."',`seaOnSaleDate` = '".addslashes($feed_array['12'][$counter])."',`perOnSaleDate` = '".addslashes($feed_array['13'][$counter])."',`soldOut` = '".addslashes($feed_array['14'][$counter])."',`eventImageURL` = '".addslashes($feed_array['15'][$counter])."',`perfID`='".addslashes($feed_array['16'][$counter])."' WHERE `perfID` = ".$feed_array['16'][$counter]." LIMIT 1;";
echo "UPDATE ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
} else {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql_exec[] = "INSERT INTO `it_raw` (`id` ,`eventtime` ,`eventname` ,`venuename` ,`venueregion` ,`venuepostcode` ,`country` ,`minprice` ,`available` ,`link` ,`eventtype` ,`seaOnSaleDate` ,
`perOnSaleDate` ,`soldOut` ,`eventImageURL` ,`perfID`) VALUES ( NULL ,'".$eventUnixTime."','".addslashes($feed_array['3'][$counter])."','".addslashes($feed_array['4'][$counter])."','".addslashes($feed_array['5'][$counter])."','".addslashes($feed_array['6'][$counter])."','".addslashes($feed_array['7'][$counter])."','".addslashes($feed_array['8'][$counter])."','".addslashes($feed_array['9'][$counter])."','".addslashes($feed_array['10'][$counter])."','".addslashes($feed_array['11'][$counter])."','".addslashes($feed_array['12'][$counter])."','".addslashes($feed_array['13'][$counter])."','".addslashes($feed_array['14'][$counter])."','".addslashes($feed_array['15'][$counter])."','".addslashes($feed_array['16'][$counter])."');";
//mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
}
unset($sql);
$counter++;
}
foreach($sql_exec as $value) {
mysql_query($value) or die (mysql_error().": ".$value);
}
You could try grouping inserts and updates into groups so the code runs less queries.
For example, you could group all of the inserts into one very large insert, or maybe group every 100 inserts.
Also using prepared statements as gradbot suggested may help.
Other than that, it's not very easy to say which part of it is the major contributor to slowness. You should use a profiler to determine that, for example by using a smaller dataset so the profiled script runs in a reasonable time.
You can do a number of things.
on duplicate key instead of select
prepared statement if you have access to PDO MySQL.
stored procedure in MySQL
Try This. I'm unable to test it but the syntax should be right.
$counter = 0;
$events = sizeof($feed_array['1']) - 1;
while($counter <= $events )
{
$eventDate = explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime = explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime = mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$data = array(
'eventtime' => $eventUnixTime,
'eventname' => addslashes($feed_array['3'][$counter]),
'venuename' => addslashes($feed_array['4'][$counter]),
'venueregion' => addslashes($feed_array['5'][$counter]),
'venuepostcode' => addslashes($feed_array['6'][$counter]),
'country' => addslashes($feed_array['7'][$counter]),
'minprice' => addslashes($feed_array['8'][$counter]),
'available' => addslashes($feed_array['9'][$counter]),
'link' => addslashes($feed_array['10'][$counter]),
'eventtype' => addslashes($feed_array['11'][$counter]),
'seaOnSaleDate' => addslashes($feed_array['12'][$counter]),
'perOnSaleDate' => addslashes($feed_array['13'][$counter]),
'soldOut' => addslashes($feed_array['14'][$counter]),
'eventImageURL' => addslashes($feed_array['15'][$counter]),
'perfID' => addslashes($feed_array['16'][$counter]),
);
$update = array();
foreach ($data as $key => $value)
$update[] = "`$key` = '$value'";
$sql = "INSERT INTO `dante_tickets`.`it_raw`" .
'(`id`, `'. implode ('`,`', array_keys($data)) . '`) VALUES ' .
'(NULL, ' . implode (',', $data) . ') ON DUPLICATE KEY UPDATE ' .
implode (',', $update);
mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted or Updated".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
unset($sql);
$counter++;
}
I forgot to mention this requires that perfID is a unique key.
Have you profiled this query?
"SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"
Because you run it 28000 times.. so unless it is REALLY quick, it will cause you a head ache. Use the EXPLAIN syntax and add an appropriate index if needed.
EDIT: With profile, I mean that you should try to use EXPLAIN on the mysql-prompt to see what execution plan the MySQL Query Optimizer suggests for this query. I.e, on the prompt, run:
EXPLAIN SELECT * FROM it_raw WHERE perfID = 426;
# Change this id to something existing and valid
What you want to see is that it is using an index, and ONLY an index. If you don't understand the output, copy and paste it here so can I go through it with you.
UPDATE: As you can see, it takes 0.07 seconds for EVERY row in your data array, plus the time to actually query the database, transfer the result etc. This is roughly 28000 * 0.07 = 1960 seconds, or 32 minutes, just to check if the data exists or not. You need to come up with another way of checking if the data already exists... One, very simple optimization might be:
EXPLAIN SELECT perfId FROM it_raw WHERE perfID = 210968;
This way, you can use the index on perfId and don't need to visit the table
If it is possible, try to avoid quering the database for each run in the loop. Perhaps it is possible to fetch the ids from the database into a big array of ids that will fit in PHP memory? This will be MUCH quicker than quering the database for every row in your big data array.
This is exactly the scenario for which prepared statements were made:
$prepared_statement =
$DB->prepare('INSERT INTO table(column, column) VALUES(?, ?)');
loop {
$prepared_statement->execute(array('value1', 'value2');
}
It's implemented in the MySQLi and PDO wrappers. It only compiles the query once and automagically sanitizes the given data, saving time (both developmental and executional) and headache.