Hi is there a way to get the value from each foreach result.
For Example I have a table named tblsamp.
id | data |
1 | d1 |
2 | d2 |
3 | d3 |
then put some query:
$query = $this->db->query("SELECT * FROM tblsamp");
foreach ($query->result() as $row){
echo $row->data .'<br>';
}
Result is:
d1
d2
d3
Now what I want is I want to get the value of the first result and compare the value from another table, and so on...
For Example:
if('d1' == '(value from another table)'{
\\do something here
} else if ('d2' == '(value from another table)'{
\\and so on.....
}
how can I do this guys? TIA!.
It looks like you are using codeigniter based off the $this->db->query syntax. With codeigniter you can change a table into an array using $query->result_array().
If you are sure both tables have the same amount of rows, you can use something like the following.
$query = $this->db->query("SELECT * FROM tbl1");
$table1 = $query->result_array();
$query = $this->db->query("SELECT * FROM tbl2");
$table2 = $query->result_array();
for ($x = 0; $x <= count($table1); $x++) {
if ($table1[$x] === $table2[$x]) {
//DO SOMETHING HERE
}
}
If $table1 might have more rows than table2, I would change it to
$query = $this->db->query("SELECT * FROM tbl1");
$table1 = $query->result_array();
$query = $this->db->query("SELECT * FROM tbl2");
$table2 = $query->result_array();
for ($x = 0; $x <= count($table1); $x++) {
if (isset($table1[$x]) && isset($table2[$x])) {
if ($table1[$x] === $table2[$x]) {
//DO SOMETHING HERE
}
} else {
break;
}
}
You can find you have matches in 2 tables however you may want to do more research about How SQL query can return data from multiple tables
How can an SQL query return data from multiple tables
This may work for your basic example
$query = $this->db->query("SELECT * FROM tblsamp,tblother");
$rst = mysql_query($query);
if(mysql_affected_rows() > 0) {
while($row = mysql_fetch_array($rst)) :
if($row['data'] == '$row['otherdata']') {
echo 'you got a match' . $row['data'] . ' In table sample to ' . $row['otherdata'] . ' In other table<br>';
}
else {
echo ' no match found in results;
}
}
endwhile;
You could add a query within your query, but this is not recommended -- things can get REALLY slow. It might be something like this:
// your original query
$query = $this->db->query("SELECT * FROM tblsamp");
foreach ($query->result() as $row){
// for each record in your original query, we run a new query derived from the result
// OBVIOUSLY you need to change this line of code to be specific to
// your database table's structure because some_col is probably not
// the name of a column in your database
$subquery = $this->db->query("SELECT * FROM other_table WHERE foo=" . $row->some_col);
$subrows = $subquery->result();
// output $subrows here?
// note that $subrows might be an array with any number any number of records
}
the exact code will depend on the db classes you are using. It's codeigniter, if I'm not mistaken? EDIT: it is codeigniter.
The correct solution here is going to depend on your database table definitions. I would strongly suggest creating a JOIN using CodeIgniter, but can't offer much more advice without some idea of what your db structures are going to look like and how many records in the second table exist for each record in the first table.
I have a table with a lot of fields, and I need to return the COUNT of NOT NULL for each field.
I make a loop in PHP and I execute 1 request by field...
$result = array();
$champs = array("field1", "field2", "field3", "field4", "field5", "field6", "field7", "field8", "field9", "field10", "field11", "field12", "field13", "field14", "field15", "field16", "field17", "field18", "field19", "field20");
foreach ($champs as $value) {
$and = empty($filtroAdd) ? "$value IS NOT NULL" : "AND $value IS NOT NULL";
$requete = "SELECT COUNT($value)
FROM my_table
WHERE $filtroAdd $and";
$r = $db->query($requete)->fetch(PDO::FETCH_NUM);
$result[$value] = $r[0];
}
print_r( $result );
I think it is not the better way, and we surely can return the same final result with a single postgreSQL request ?!
Thanks for help
You can use a single query:
SELECT COUNT(field1), COUNT(field2), COUNT(field3), . . .
FROM my_table;
There is no need to do a separate query for each field. COUNT(<field>) returns the number of rows with non-NULL values in that column.
If you want the number of different values, then use COUNT(DISTINCT):
SELECT COUNT(DISTINCT field1), COUNT(DISTINCT field2), COUNT(DISTINCT field3), . . .
FROM my_table;
You can also embed CASE logic, if you like. I usually do this using SUM():
SELECT SUM(CASE WHEN field1 > 0 THEN 1 ELSE 0 END),
I have a method, where about 20 rows must changed. Im passing per ajax a string like "14-33-61-10 ..." for saving the order of some rows.
Here is my actual (working) code:
$explode = explode('--',$_POST['data']); // -> array('0'=>'8', '1'=>'4', '2'=> ... )
$i = 1;
foreach( $explode as $task ) {
if( !is_int($task) ) continue;
$exec = $project->exec("UPDATE tasks SET pos=$i WHERE rowid=". $task );
$i++;
}
My problem is, that it takes about 1 second.
My question: Is there a way to edit multiple rows with just one query?
Sure, send a SQL string like:
update tasks
set pos =
case rowid
when 13 then 1
when 33 then 2
when 61 then 3
when 10 then 4
end
where rowid in (13,33,61,10)
The translation from 13-33-61-10 to (13,33,61,10) is best done in PHP.
If the range is short then we can use the ID IN (ids) argument.
UPDATE `profile` SET `app_status`=3 WHERE `id` IN (3,37,95,136,1087,1795,1817)
If it's long and we know that range, let's use the BETWEEN value AND value argument.
UPDATE `profile` SET `app_status`=3 WHERE `id` BETWEEN 3904 AND 3918
$explode = explode('--',$_POST['data']); // -> array('0'=>'8', '1'=>'4', ... )
$i = 1;
$sql = "";
foreach($explode as $task)
{
if(!is_int($task)) continue;
$sql .= "UPDATE tasks SET pos=$i WHERE rowid=$task;";
$i++;
}
$project->exec($sql);
I was just hoping someone could help me speed up 4 queries with a multi query.
GOAL: a single multi query to function as the single queries below.
Simple queries, i am checking one table to see if user is banned, then if not, i am getting row for the id and updating it's view count by 1. If user is banned, i do not want the last to queries to complete.
Thank you in advance for your help.
current performance is around 1200ms. (+1000ms avg for facebook graph api query).
NOTE: af_freefeed.pageid & af_ban.pageid are both indexed in database.
ALSO: I have been studying and referencing from http://www.php.net/manual/en/mysqli.multi-query.php i just can not see how to get this config into multi with the if()
$fconn = new mysqli($fdbhost, $fdbuser, $fdbpass, $fdbname) or die ('Error connecting to mysqli');
// 12,000 rows for af_ban - bigint(255) : indexed
$q = sprintf('SELECT COUNT(pageid) AS numrowst FROM af_ban WHERE pageid = %s', $banpage);
$readtop = $fconn->query($q);
$rowtop = $readtop->fetch_assoc();
// 1.17 million rows for af_freefeed - bigint(255) : indexed
if($rowtop[numrowst] == 0){
$q = sprintf('SELECT COUNT(pageid) AS numrowsf FROM af_freefeed WHERE pageid = %s', $banpage);
$readf = $fconn->query($q);
$rowf = $readf->fetch_assoc();
// increment views
$read = $fconn->query("Update af_freefeed SET views = views + 1 WHERE pageid = ".$banpage."");
}
$q=$fconn->query("SELECT pagename,views,pageid FROM af_freefeed ORDER BY views DESC LIMIT 0, 20");
unset($q);
unset($rowf);
unset($rowtop);
mysqli_close($fconn);
actual request times.
grah api: 1127.04610825ms.
conncect: 1.20711326599ms.
check banned: 0.405788421631ms.
get row: 418.189229965ms.
increment views: 472.24655151ms.
get top20: 94.31447983ms.
Multi_query #1 How to stop the multi query if user is banned?
Possible Contender: 943.8181ms. if added : 933.1279ms. if banned
10ms difference if exit loop for banned. This leads me to believe the loop is completing all the queries before they are actually supposed to be executed, "next_result". Or i have an error in how i looped the functions.
replaced exit; with $thread_id = $fconn->thread_id; $fconn->kill($thread_id);
if banned 953.4719ms. no gain.
$banpage='234232874008';
$query = "SELECT pagename,views,pageid FROM af_freefeed ORDER BY views DESC LIMIT 0, 2;";
$query .= "SELECT pageid AS isbanned FROM af_ban WHERE pageid = \"".$banpage."\";";
$query .= "SELECT pageid AS isadded FROM af_freefeed WHERE pageid = \"".$banpage."\";";
$query .= "Update af_freefeed SET views = views + 1 WHERE pageid = \"".$banpage."\"";
/* execute multi query */
if ($fconn->multi_query($query)) {
if ($result = $fconn->store_result()) {
while ($row = $result->fetch_row()) {
print_r($row).'<br />';
}
$result->free();
}
if ($fconn->more_results()) {
while ($fconn->next_result()){
if($thisresult = $fconn->store_result()){
while (is_array($row = $thisresult->fetch_array())) {
if(isset($row['isbanned'])){
if($row['isbanned']===''.$banpage.''){
$thread_id = $fconn->thread_id;
$fconn->kill($thread_id);
// exit;
}
}
}
}
}
}
}
unset($query);
unset($result);
unset($thisresult);
Multi_query #2 "current for benchmark" How to remove duplicate fields in result set after next_result()?
2.667ms. / 1032.2499ms. but print_r is showing duplicate fields in $thisresults?
**Array
(
[0] => 37
[id] => 37
[1] => 159616034235
[pageid] => 159616034235
[2] =>
[userid] =>
[3] => 30343
[views] => 30343
[4] => Walmart
[pagename] => Walmart
)**
$query = "SELECT pageid AS isbanned FROM af_ban WHERE pageid = \"".$banpage."\";";
$query .= "SELECT pageid AS isadded FROM af_freefeed WHERE pageid = \"".$banpage."\";";
$query .= "SELECT * FROM af_freefeed ORDER BY views DESC LIMIT 0, 20";
//$query .= "Update af_freefeed SET views = views + 1 WHERE pageid = \"".$banpage."\"";
/* execute multi query */
echo '<pre>';
$i=0;
if ($fconn->multi_query($query)) {
if ($result = $fconn->store_result()) {
//$row = $result->fetch_assoc();
while ($row = $result->fetch_assoc()) {
print_r($row).'<br />';
}
$result->free();
}
if ($fconn->more_results()) {
while ($fconn->next_result()){
if($thisresult = $fconn->store_result()){
while ($row2 = $thisresult->fetch_array()) {
if(isset($row2['isadded'])){
if($row2['isadded']===''.$banpage.''){
$addone = $fconn->query("Update af_freefeed SET views = views + 1 WHERE pageid = ".$banpage."");
}
}
print_r($row2);
}
}
}
}
}
/* determine our thread id */
$thread_id = $fconn->thread_id;
/* Kill connection */
$fconn->kill($thread_id);
//
echo '</pre><hr/>';
Try to use index in getting count. Like for example COUNT(pageid). It will speed up your query.
Update
You can also try this link for further explanation
EDIT : So now, the conclusion: (test case below)
You cannot control the execution of subsequent statements of a multi-statement query.
You can therefore not use multi_query() in the way you wanted to.
Execute them all, or execute none.
Regarding
Multi_query #2 "current for benchmark" How to remove duplicate fields in result set after next_result()?
Use fetch_assoc() or fetch_array(MYSQLI_ASSOC) (both practically the same) instead of fetch_array().
About multi_query():
I recently worked on a program using the MySQL C API, which mysqli uses, too.
About multiple-statement query support the documentation states:
Executing a multiple-statement string can produce multiple result sets or row-count indicators. Processing these results involves a different approach than for the single-statement case: After handling the result from the first statement, it is necessary to check whether more results exist and process them in turn if so. To support multiple-result processing, the C API includes the mysql_more_results() and mysql_next_result() functions. These functions are used at the end of a loop that iterates as long as more results are available. Failure to process the result this way may result in a dropped connection to the server.
(emphasize added)
This leads to the conclusion, that aborting a multiple-statement query is not an intended feature.
Moreover, I didn't find any resource explaining when subsequent queries are actually executed.
Calling next_result() doesn't neccessarily mean that the query hasn't been executed already.
EDIT : TEST CASE
To prove what I previously assumed, I created a test case:
<?php
$db = new mysqli('localhost', 'root', '', 'common');
$query = 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
if($db->multi_query($query)) {
// Current time
echo "'multi_query()' executed at:\n\t\t"
.date('Y-m-d H:i:s')."\n";
// First result
if($result = $db->store_result()) {
$i = 1;
$row = $result->fetch_assoc();
echo "'NOW()' of result $i:\n\t\t".$row['time']."\n";
$result->free();
// Wait 5 seconds
sleep(5);
// Subsequent results
while($db->more_results() && $db->next_result()) {
$result = $db->store_result();
$row = $result->fetch_assoc();
$i++;
echo "'NOW()' of result $i:\n\t\t".$row['time']."\n";
// Wait 5 seconds
sleep(5);
$result->free();
}
}
}
$db->close();
?>
This results in:
'multi_query()' executed at:
2013-05-10 10:18:47
'NOW()' of result 1:
2013-05-10 10:18:47
'NOW()' of result 2:
2013-05-10 10:18:47
'NOW()' of result 3:
2013-05-10 10:18:47
'NOW()' of result 4:
2013-05-10 10:18:47
Given that, it is obvious that all four statements of the query were executed directly after the call to multi_query().
If they were only executed after calling next_result() there would be a 5 second delay caused by sleep(5) calls I added between the loop iterations.
Please run following query in mysql and check your query run time :
CREATE INDEX pagidIndex ON af_ban (pageid(11));
CREATE INDEX pagidFeedIndex ON af_freefeed (pageid(11));
CREATE INDEX viewsIndex ON af_freefeed (views(11));
i am checking one table to see if user is banned, then if not, i am getting row for the id and updating it's view count by 1.
The following query may help you to update the view count. I assume that you already know the page_id.
UPDATE af_freefeed SET views=views+1 WHERE page_id=%s and page_id not in (select page_id from af_ban WHERE page_id=%s);
You could try something along these lines:
$sql = sprintf("SELECT af_freefeed.pageid FROM af_freefeed left join af_ban ".
"on (af_freefeed.pageid = af_ban.pageid) ".
"where af_freefeed.pageid = %s and ".
"af_ban.pageid is null limit 1", $pageid);
to replace your first two queries.
The existence of a record in the results should indicate an unbanned user requesting the resource. Then you can do your update your views.
Hope this helps.
Right, this code goes through a rather large multidimensional array (has about 28,000 rows and 16 parts).
Order of events:
Check if the data exists in the database
if it exists - Update it with the new data
if it doesn't exist - Insert it
Simple.
But right now to go through this it has taken over 30min i think and Still going.
$counter = 0;
$events = sizeof($feed_array['1'])-1;
while($counter <= $events ) {
$num_rows = mysql_num_rows(mysql_query("SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"));
if($num_rows) {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
mysql_query("UPDATE `it_raw` SET
`eventtime` = '".$eventUnixTime."',
`eventname` = '".addslashes($feed_array['3'][$counter])."',
`venuename` = '".addslashes($feed_array['4'][$counter])."',
`venueregion` = '".addslashes($feed_array['5'][$counter])."',
`venuepostcode` = '".addslashes($feed_array['6'][$counter])."',
`country` = '".addslashes($feed_array['7'][$counter])."',
`minprice` = '".addslashes($feed_array['8'][$counter])."',
`available` = '".addslashes($feed_array['9'][$counter])."',
`link` = '".addslashes($feed_array['10'][$counter])."',
`eventtype` = '".addslashes($feed_array['11'][$counter])."',
`seaOnSaleDate` = '".addslashes($feed_array['12'][$counter])."',
`perOnSaleDate` = '".addslashes($feed_array['13'][$counter])."',
`soldOut` = '".addslashes($feed_array['14'][$counter])."',
`eventImageURL` = '".addslashes($feed_array['15'][$counter])."',
`perfID`= '".addslashes($feed_array['16'][$counter])."'
WHERE `perfID` = ".$feed_array['16'][$counter]." LIMIT 1 ;");
echo "UPDATE ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
} else {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql = "INSERT INTO `dante_tickets`.`it_raw` (
`id` ,
`eventtime` ,
`eventname` ,
`venuename` ,
`venueregion` ,
`venuepostcode` ,
`country` ,
`minprice` ,
`available` ,
`link` ,
`eventtype` ,
`seaOnSaleDate` ,
`perOnSaleDate` ,
`soldOut` ,
`eventImageURL` ,
`perfID`
)
VALUES (
NULL ,
'".$eventUnixTime."',
'".addslashes($feed_array['3'][$counter])."',
'".addslashes($feed_array['4'][$counter])."',
'".addslashes($feed_array['5'][$counter])."',
'".addslashes($feed_array['6'][$counter])."',
'".addslashes($feed_array['7'][$counter])."',
'".addslashes($feed_array['8'][$counter])."',
'".addslashes($feed_array['9'][$counter])."',
'".addslashes($feed_array['10'][$counter])."',
'".addslashes($feed_array['11'][$counter])."',
'".addslashes($feed_array['12'][$counter])."',
'".addslashes($feed_array['13'][$counter])."',
'".addslashes($feed_array['14'][$counter])."',
'".addslashes($feed_array['15'][$counter])."',
'".addslashes($feed_array['16'][$counter])."'
);";
mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
}
unset($sql);
$counter++;
}
UPDATE
I just carried out profiling one one of the rows:
mysql> EXPLAIN SELECT * FROM it_raw WHERE perfID = 210968;
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
| 1 | SIMPLE | it_raw | ref | perfID | perfID | 4 | const | 1 | |
+----+-------------+--------+------+---------------+--------+---------+-------+------+-------+
1 row in set (0.07 sec)
UPDATE 2
To try and "speed" things up, instead of carrying out the UPDATE and INSERT statements straight away, i've now placed them in a variable (so only the initial select runs - to check a duplicate - then stores the action [insert or update]). At the end of the loop it executes all the statments.
Except now, it's coming up with MySQL error that the syntax is incorrect. (when initially there was nothing wrong).
I've simply replaced the mysql_query with:
$sql_exec .= "SELECT.... ;";
is there something i'm missing here for the formatting?
UPDATE 3
OK finally fixed it
Lessons Learned:
Do logic search first on database
Carry out insert/updates in bulk.
Here is the final code which now takes about 60 seconds to run (from over 30min+)
while($counter <= $events ) {
$num_rows = mysql_num_rows(mysql_query("SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"));
if($num_rows) {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql_exec[] = "UPDATE `it_raw` SET `eventtime` = '".$eventUnixTime."',`eventname` = '".addslashes($feed_array['3'][$counter])."',`venuename` = '".addslashes($feed_array['4'][$counter])."',`venueregion` = '".addslashes($feed_array['5'][$counter])."',`venuepostcode` = '".addslashes($feed_array['6'][$counter])."',`country` = '".addslashes($feed_array['7'][$counter])."',`minprice` = '".addslashes($feed_array['8'][$counter])."',`available` = '".addslashes($feed_array['9'][$counter])."',`link` = '".addslashes($feed_array['10'][$counter])."',`eventtype` = '".addslashes($feed_array['11'][$counter])."',`seaOnSaleDate` = '".addslashes($feed_array['12'][$counter])."',`perOnSaleDate` = '".addslashes($feed_array['13'][$counter])."',`soldOut` = '".addslashes($feed_array['14'][$counter])."',`eventImageURL` = '".addslashes($feed_array['15'][$counter])."',`perfID`='".addslashes($feed_array['16'][$counter])."' WHERE `perfID` = ".$feed_array['16'][$counter]." LIMIT 1;";
echo "UPDATE ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
} else {
$eventDate=explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime=explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime=mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$sql_exec[] = "INSERT INTO `it_raw` (`id` ,`eventtime` ,`eventname` ,`venuename` ,`venueregion` ,`venuepostcode` ,`country` ,`minprice` ,`available` ,`link` ,`eventtype` ,`seaOnSaleDate` ,
`perOnSaleDate` ,`soldOut` ,`eventImageURL` ,`perfID`) VALUES ( NULL ,'".$eventUnixTime."','".addslashes($feed_array['3'][$counter])."','".addslashes($feed_array['4'][$counter])."','".addslashes($feed_array['5'][$counter])."','".addslashes($feed_array['6'][$counter])."','".addslashes($feed_array['7'][$counter])."','".addslashes($feed_array['8'][$counter])."','".addslashes($feed_array['9'][$counter])."','".addslashes($feed_array['10'][$counter])."','".addslashes($feed_array['11'][$counter])."','".addslashes($feed_array['12'][$counter])."','".addslashes($feed_array['13'][$counter])."','".addslashes($feed_array['14'][$counter])."','".addslashes($feed_array['15'][$counter])."','".addslashes($feed_array['16'][$counter])."');";
//mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted ".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
}
unset($sql);
$counter++;
}
foreach($sql_exec as $value) {
mysql_query($value) or die (mysql_error().": ".$value);
}
You could try grouping inserts and updates into groups so the code runs less queries.
For example, you could group all of the inserts into one very large insert, or maybe group every 100 inserts.
Also using prepared statements as gradbot suggested may help.
Other than that, it's not very easy to say which part of it is the major contributor to slowness. You should use a profiler to determine that, for example by using a smaller dataset so the profiled script runs in a reasonable time.
You can do a number of things.
on duplicate key instead of select
prepared statement if you have access to PDO MySQL.
stored procedure in MySQL
Try This. I'm unable to test it but the syntax should be right.
$counter = 0;
$events = sizeof($feed_array['1']) - 1;
while($counter <= $events )
{
$eventDate = explode("/", $feed_array['1'][$counter]); //print_r($eventDate);
$eventTime = explode(":", $feed_array['2'][$counter]); //print_r($eventTime);
$eventUnixTime = mktime($eventTime[0], $eventTime[1], "00", $eventDate[1], $eventDate[0], $eventDate[2]);
$data = array(
'eventtime' => $eventUnixTime,
'eventname' => addslashes($feed_array['3'][$counter]),
'venuename' => addslashes($feed_array['4'][$counter]),
'venueregion' => addslashes($feed_array['5'][$counter]),
'venuepostcode' => addslashes($feed_array['6'][$counter]),
'country' => addslashes($feed_array['7'][$counter]),
'minprice' => addslashes($feed_array['8'][$counter]),
'available' => addslashes($feed_array['9'][$counter]),
'link' => addslashes($feed_array['10'][$counter]),
'eventtype' => addslashes($feed_array['11'][$counter]),
'seaOnSaleDate' => addslashes($feed_array['12'][$counter]),
'perOnSaleDate' => addslashes($feed_array['13'][$counter]),
'soldOut' => addslashes($feed_array['14'][$counter]),
'eventImageURL' => addslashes($feed_array['15'][$counter]),
'perfID' => addslashes($feed_array['16'][$counter]),
);
$update = array();
foreach ($data as $key => $value)
$update[] = "`$key` = '$value'";
$sql = "INSERT INTO `dante_tickets`.`it_raw`" .
'(`id`, `'. implode ('`,`', array_keys($data)) . '`) VALUES ' .
'(NULL, ' . implode (',', $data) . ') ON DUPLICATE KEY UPDATE ' .
implode (',', $update);
mysql_query($sql) or die(mysql_error().":".$sql);
echo "Inserted or Updated".$feed_array['16'][$counter].": ".addslashes($feed_array['3'][$counter])."\n";
unset($sql);
$counter++;
}
I forgot to mention this requires that perfID is a unique key.
Have you profiled this query?
"SELECT * FROM it_raw WHERE perfID = '".addslashes($feed_array['16'][$counter])."'"
Because you run it 28000 times.. so unless it is REALLY quick, it will cause you a head ache. Use the EXPLAIN syntax and add an appropriate index if needed.
EDIT: With profile, I mean that you should try to use EXPLAIN on the mysql-prompt to see what execution plan the MySQL Query Optimizer suggests for this query. I.e, on the prompt, run:
EXPLAIN SELECT * FROM it_raw WHERE perfID = 426;
# Change this id to something existing and valid
What you want to see is that it is using an index, and ONLY an index. If you don't understand the output, copy and paste it here so can I go through it with you.
UPDATE: As you can see, it takes 0.07 seconds for EVERY row in your data array, plus the time to actually query the database, transfer the result etc. This is roughly 28000 * 0.07 = 1960 seconds, or 32 minutes, just to check if the data exists or not. You need to come up with another way of checking if the data already exists... One, very simple optimization might be:
EXPLAIN SELECT perfId FROM it_raw WHERE perfID = 210968;
This way, you can use the index on perfId and don't need to visit the table
If it is possible, try to avoid quering the database for each run in the loop. Perhaps it is possible to fetch the ids from the database into a big array of ids that will fit in PHP memory? This will be MUCH quicker than quering the database for every row in your big data array.
This is exactly the scenario for which prepared statements were made:
$prepared_statement =
$DB->prepare('INSERT INTO table(column, column) VALUES(?, ?)');
loop {
$prepared_statement->execute(array('value1', 'value2');
}
It's implemented in the MySQLi and PDO wrappers. It only compiles the query once and automagically sanitizes the given data, saving time (both developmental and executional) and headache.