I'm running a comparison of some data using PHP to access MySQL tables. The script takes well over an hour to run. We're dealing with about 100k rows in the source data and 300k in each of the target tables, i.e. two tables.
There's no more than 10 columns total in each table.
Here is the script:
$query = "SELECT bbva_check_deductions.empno, bbva_check_deductions.idr, bbva_check_deductions.current1, bbva_job.paygroup
FROM bbva_check_deductions
LEFT JOIN bbva_job
ON bbva_check_deductions.empno = bbva_job.file_nbr
ORDER BY current1 DESC, EmpNo DESC, IDR DESC";
$result_set = mysql_query($query,$connection);
while ($result = mysql_fetch_assoc($result_set)) {
$emplid = compare_filter($result['empno']);
$oldCode = compare_filter($result['idr']);
$origCode = $oldCode;
$payGroup = compare_filter($result['paygroup']);
$codes = array('DC6', 'HTH', 'DM4', '4UP', 'DC1', 'DC3', 'DC4', 'DEN', 'DCR', 'MDR', 'PRK', 'VIS', '41K');
//CALCULATE WEEK NO
if ($payGroup == "J2C") {$week = 24;}
elseif ($payGroup == "J2E") {$week = 26;}
else {$week = 22;}
//QUERY TRANSLATION TABLE FOR TRANSLATED VALUE
$translate_query = "SELECT * FROM translation WHERE sourceValue = '{$oldCode}'
AND sourceField LIKE '%ded_code%' LIMIT 1";
$translate_result_set = mysql_query($translate_query, $connection);
if (mysql_num_rows($translate_result_set) > 0) {
$translate_result = mysql_fetch_assoc($translate_result_set);
$oldCode = compare_filter($translate_result['translatedValue']);
}
//DETERMINE IF MEMO
if ($oldCode == "GTL" || $oldCode == "BLI" || $oldCode == "BLT" || $oldCode == "HIC" || $oldCode == "DM4") {
$queryField = "memo_code"; $queryTable = "pf_check_memo";
} else {
$queryField = "deduction_code"; $queryTable = "pf_check_deductions";
}
$compare_query = "SELECT {$queryField}, amount_field_for_payroll
FROM {$queryTable}
WHERE empl_id = {$emplid}
AND {$queryField} LIKE '%{$oldCode}%'
AND week_number = {$week}
LIMIT 1";
$compare_result_set = mysql_query($compare_query, $connection);
if (mysql_num_rows($compare_result_set) != 0) {
$compare_result = mysql_fetch_assoc($compare_result_set);
//PREP COMPARE FIELDS
$amount = compare_filter($result['current1']);
$compare_amount = compare_filter($compare_result['amount_field_for_payroll']);
$negCodes = array('DC6', 'HTH', 'DM4', '4UP', 'DC1', 'DC3', 'DC4', 'DEN', 'DCR', 'MDR', 'PRK', 'VIS', '41K');
if (in_array($oldCode,$negCodes)) {$compare_amount = $compare_amount * -1;}
//AMOUNT COMPARE
direct_compare_basic($amount,$compare_amount,"amount","bbva_check_deductions","Y",$origCode,$oldCode);
} else {$content .= $emplid . $separator . "" . $separator . "Emplid + Deduction Code + Pay Group" . $separator . 'bbva_check_deductions' .
$separator . field_prep($emplid . " - " . $oldCode . " - " . $payGroup) . $separator . "No Record Found in PF for week {$week}" . $separator . "Y" . $separator . "Y" .
$separator . $origCode . $separator . $oldCode . $break;}
Then it just writes to a .CSV file.
My functions do not call any additional queries, just formatting stuff. I'm running InnoDB tables and have indexes for all columns being used. My PC is running WAMP, it's got 16G of RAM and a quad core processor. The HDD is 7200 RPM.
Seems like it is taking far too long to me. Any ideas?
Related
I have this code:
<?php
$episodios = $SQL->query("SELECT * FROM episodios order by id desc LIMIT 500");
$temporadas = $SQL->query("SELECT * from temporadas");
foreach($episodios as $episodio) {
//review performance
foreach($temporadas as $temporada) {
if ($episodio['url_anime'] == $temporada['url_anime'] && $episodio['temporada'] == $temporada['temporada']) {
$tempName = (($temporada['showname'] == 1) ? '- '.$temporada['nome'].'' : '');
}
}
}
?>
But i'm worried about the code performance, because this loop inside another..
Someone told me to use cache and only update when get new data, but have other way to do the same thing with better performance? this code is displayed on my home website page.
Thanks in advance.
Please use the below query for improve code performance:-
$tempName = $SQL->query("Select temporadas.nome from temporadas INNER JOIN episodios ON
temporadas.url_anime = episodios.url_anime AND temporadas.temporada =
episodios.temporada WHERE temporadas.nome = 1 LIMIT 500");
pls check
<?php
//with code
$episodios = $SQL->query("SELECT * FROM episodios order by id desc LIMIT 500");
$temporadas = $SQL->query("SELECT * from temporadas");
$items = [];
foreach ($episodios as $episodio) {
if (!isset($items[$episodio['url_anime'] . "||" . $episodio['temporada']])) {
$items[$episodio['url_anime'] . "||" . $episodio['temporada']] = "-";
}
}
foreach ($temporadas as $temporada) {
if (isset($items[$temporada['url_anime'] . "||" . $temporada['temporada']])) {
$items[$temporada['url_anime'] . "||" . $temporada['temporada']] = (($temporada['showname'] == 1) ? '- ' . $temporada['nome'] . '' : '');
}
}
var_dump($items);
//There is a possibility of a minor error because I have real data
//or with query
#"SELECT * from temporadas join episodios on temporadas.url_anime==episodios.url_anime and temporadas.temporada==episodios.temporada"
I have this coding :
foreach ($users_id_array[REGION_NORTH_REFID][813] as $key) {
$query_course = "SELECT ut_lp_marks.obj_id, object_data.title, read_event.spent_seconds, " .
"read_event.read_count, ut_lp_marks.status, ut_lp_marks.percentage, ut_lp_marks.u_comment FROM ut_lp_marks ".
"LEFT JOIN object_data ON (object_data.obj_id = ut_lp_marks.obj_id) ".
"LEFT JOIN read_event ON (read_event.obj_id = object_data.obj_id AND read_event.usr_id = ut_lp_marks.usr_id) ".
"WHERE ut_lp_marks.usr_id=$key AND object_data.type = 'crs'";
$set_course = mysql_query($query_course);
while($rec_course = mysql_fetch_assoc($set_course))
{
if ($rec_course['status'] == 0) {
$total_regna++;
}
if ($rec_course['status'] == 1) {
$total_reginprogress++;
}
if ($rec_course['status'] == 2) {
$total_regpassed++;
}
if ($rec_course['status'] == 3) {
$total_regfailed++;
}
// $total_attempt = $total_attempt + $rec_course['read_count'];
// $total_spent = $total_spent + $rec_course['spent_seconds'];
}
$no_test++;
}
the variable that i used is for each :
813, 945, 835, 777
My problems is my coding right now is only for 1 variable.
How can i used this same code for different variable,do i need to make array or a function? How can i access .. different $total_regna++; $total_reginprogress++; $total_regpassed++; $total_regfailed++; if i am using 1 code for 4 variable?..
You can make an array of these values such as
$values = array('813','945','835','777');
and instead of using for each, you can implode this array values in query.
$query_course = "SELECT ut_lp_marks.obj_id, object_data.title, read_event.spent_seconds, " .
"read_event.read_count, ut_lp_marks.status, ut_lp_marks.percentage, ut_lp_marks.u_comment FROM ut_lp_marks ".
"LEFT JOIN object_data ON (object_data.obj_id = ut_lp_marks.obj_id) ".
"LEFT JOIN read_event ON (read_event.obj_id = object_data.obj_id AND read_event.usr_id = ut_lp_marks.usr_id) ".
"WHERE ut_lp_marks.usr_id in (".implode(',',$values).") AND object_data.type = 'crs'";
Now while loop will be same.
I think it will be good in your case.
$ids=array(813, 945, 835, 777);
for ($i=0;$i<4;$i++){
foreach ($users_id_array[REGION_NORTH_REFID][$ids[$i]] as $key) {
...
$slider = tep_db_query("select slider_config_value, slider_config_id from " . TABLE_SLIDER_CONFIG . " ");
while ($theslider = tep_db_fetch_array($slider)) {
$autoStart = $theslider['slider_config_value'];
}
What I'm trying to do is select the slider_config_value for certain slider_config_id with code as short as possible. Because I know I could use 'where slider_config_id = ...' but I'd have to have a query for each of em.
What I need is someting like
$autoStart = $theslider['slider_config_value', '1'];
$loop = $theslider['slider_config_value', '22'];
$margin = $theslider['slider_config_value', '4'];
etc.
I have been trying to find a solution, but haven't dealt with this before.
$slider = tep_db_query("select slider_config_value, slider_config_id from " . TABLE_SLIDER_CONFIG . " ");
$slider_configs = array();
while ($theslider = tep_db_fetch_array($slider)) {
$slider_configs[$theslider['slider_config_id']] = $theslider['slider_config_value'];
}
$autoStart = $slider_configs[1];
$loop = $slider_configs[22];
$margin = $slider_configs[4];
I have two queries sent to a database bring back posts (op_ideas 16 cols) followed by another which holds the votes per post (op_idea_vote cols 4) with matching idea_id's
Example of Data:
Query: op_ideas:
[{"idea_id":"2211","author_id":"100000", "date":"2012-09-06
10:02:28","idea_title":"Another test","4" etc etc
Query: op_idea_votes:
idea_id = 2211, agree=3, disagree=1, abstain=0
The code below ought to look at op_ideas, and then cycle over op_ideas_vote until it finds a match under 'idea_id'. Then it goes to the next record under op_ideas, and again using that idea_id search for it within the op_idea_vote list, find a match, and add it to the array.
This works for only the first record, not for the other three. I am testing, so I have 3 rows in each that match idea_id with different results in the op_idea_vote.
$votes = mysql_query($commentVotes);
$result = mysql_query($gl_query);
while ($gce_result = mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
while($allvotes= mysql_fetch_array($votes)) {
if($voteid = $allvotes['idea_id'])
{
//echo $voteid . " main idea and the votes: " . $allvotes;
$gce_result["agree"] = $allvotes['agree'];
$gce_result["disagree"] = $allvotes['disagree'];
$gce_result["abstain"] = $allvotes['obstain'];
}
else
{
$gce_result["agree"] = 0;
$gce_result["disagree"] = 0;
$gce_result["abstain"] = 0;
}
//print_r($gce_result);
}
$data_result[] = $gce_result;
}
echo json_encode($data_result);
If I use print_f(&gce_result) it works fine in phpfiddle. But when i use the code above, it works for the first record, but it's complete missing the second two. It seems to be missing the second while, as it does not even give me the 0 0 0 results.
Query for op_ideas:
$gl_query = "SELECT DISTINCT * FROM heroku_056eb661631f253.op_ideas INNER JOIN op_organs ORDER BY date ASC;";
if (!mysql_query($gl_query)) {
die('Error: ' . $gl_query . " " . mysql_error());
}
$result = mysql_query($gl_query);
Query For op_idea_vote :
$commentVotes = "SELECT v.idea_id, COUNT(v.agree = 1 or null) as agree, COUNT(v.disagree = 1 or null) as disagree, COUNT(v.obstain = 1 or null) as obstain FROM op_idea_vote v GROUP BY v.idea_id";
if (!mysql_query($commentVotes)) {
die('Error: ' . $commentVotes . " " . mysql_error());
}
$votes = mysql_query($commentVotes);
You can scan a resource only once.
So the inner while will be run only one time.
use == instead of = for checking condition of if & while
in the while loop ,you have to assign the value of $allvotes ,but you never assigned,
while ($gce_result == mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
while($allvotes== mysql_fetch_array($votes)) {
if($voteid == $allvotes['idea_id'])
{
//echo $voteid . " main idea and the votes: " . $allvotes;
$gce_result["agree"] = $allvotes['agree'];
$gce_result["disagree"] = $allvotes['disagree'];
$gce_result["abstain"] = $allvotes['obstain'];
}
else
{
$gce_result["agree"] = 0;
$gce_result["disagree"] = 0;
$gce_result["abstain"] = 0;
}
$data_result[] = $gce_result;
}
}
Your problem is trying to scan over the $votes result more than once.
You should store the result of that query first.
Eg.
while ($vote = mysql_fetch_array($votes)) {
$allvotes['idea_id'] = $vote;
}
while ($gce_result = mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
if (array_key_exists[$voteid, $allvotes]) {
//assign results
} else {
//default
}
}
Another option would be to do the query with a join, so you can do everything in one query. Then just loop over that result.
I currently have a php page that grabs information from a database and produces HTML with data attributes that are filled in by from the MySQL query. The database is going to be used to search, with many different options for searches.
What I need help with is knowing a way so to organize how the many variables are handled. It's a really big mess of code, and even with all the comments I put it gives me a headache trying to figure out how to add another variable to the search.
All the variables, except for the LIMIT to which row and how many results, are optional. So if someone leaves everything except that blank, I still want it to function as well as if they meticulously filled in all the fields.
Here's what I have, with 6 variables.
<?php
$product_size = "(".$_GET['size']." BETWEEN productsizeDOWN AND productsizeUP)"; // This code sets the variable to input into the MySQL string based on the URL
$product_size_check = $_GET['size']; // the _checks check are used to see if the value is or isn't empty using if statements below
$manufacturer = $_GET['manufacturer'];
$product_manufacterer_check = $_GET['manufacturer']; // _check
$product_invisible = "(hideproduct = '".$_GET['invisible']."')"; // Checks if product is hidden
$product_invisible_check = $_GET['invisible']; // _check
$product_instock_check = $_GET['instock']; // _check
$product_limit0 = $_GET['startat']; // This is the first number after LIMIT; the row to start in.
$product_limit1 = $_GET['results']; // This is how many results to load.
$manufacturer_array = explode(",", $manufacturer); // The manufacturer comes in as "Nike,Addidas,Rebok" and is turned into an array
$manufacturer_imploded = implode("' OR productmanufacturer = '", $manufacturer_array); // Puts it back together with "OR productmanufacturer =" between each name.
$product_manufacterer = ("(productmanufacturer = '".$manufacturer_imploded."')"); // formats it so it can be directly inserted into MySQL string with a WHERE in front.
if($product_invisible_check == ""){
$product_invisible = "";
}else{$where = "WHERE ";}; //Useless code that I havn't deleted that I tried to use when I searched the entire database
if($product_size_check == ""){
$product_size = "";
}else{$where = "WHERE ";};
if($product_manufacterer_check == ""){
$product_manufacterer = "";
}else{$where = "WHERE ";};
if($product_instock_check == "N"){
$product_instock = "(stockstatus <= '0' AND donotallowbackorders = 'Y') AND "; // Checks if product is in stock (Allowing backordering OR stock >1)
$where = "WHERE ";
}
elseif($product_instock_check == "Y") {
$product_instock = "(stockstatus > '0' OR donotallowbackorders = 'N') AND ";
$where = "WHERE ";
}
else {
$product_instock = "";
};
$sql="Select * FROM ioa7pd_Products WHERE ".$product_instock.$product_size."AND".$product_manufacterer_and.$product_manufacterer."".$product_invisible." LIMIT ".$product_limit0.", ".$product_limit1; // The end result of it all.
echo $sql;
?>
When the URL is
test.php?size=5&manufacturer=Nike,Addidas,Rebok&invisible=N&instock=Y&startat=0&results=30
the resulting SQL query is
Select * FROM ioa7pd_Products WHERE (stockstatus > '0' OR donotallowbackorders = 'N') AND (5 BETWEEN productsizeDOWN AND productsizeUP)AND(productmanufacturer = 'Nike' OR productmanufacturer = 'Addidas' OR productmanufacturer = 'Rebok')(hideproduct = 'N') LIMIT 0, 30
But I plan to add more options to the search.
My main question is simply: What way can I organize this to make it simple to add more variables? Tiered if statements?
Travesty has been helping me with my code and has really been great in organizing it.
Here is the current code. It needs to be secure to prevent injection.
// Database connection
$con = mysql_connect("[CENSORED]","[CENSORED]","[CENSORED]")
or die("Could not connect: " . mysql_error());
mysql_select_db("[CENSORED]") or die('Could not select database');
// Begin organization of URL variables into MYSQL Query
$get_size = $_GET['size'];
$get_manufacturer = $_GET['manufacturer'];
$get_invisible = $_GET['invisible'];
$get_instock = $_GET['instock'];
$get_sex = $_GET['sex'];
$get_startat = $_GET['startat'];
$get_results = $_GET['results'];
if ($get_size != ""){
$all_selectors[] = "(".$get_size." BETWEEN productsizeDOWN AND productsizeUP)"; // Add to array if size is not blank.
};
if ($get_manufacturer != ""){
$manufacturer_exploded = explode(",", $get_manufacturer);
$manufacturer_imploded = implode("' OR productmanufacturer = '", $manufacturer_exploded);
$all_selectors[] = ("(productmanufacturer = '".$manufacturer_imploded."')");
};
if ($get_invisible != ""){
$all_selectors[] = "(hideproduct = '".$get_invisible."')";
};
if($get_instock == "N" or $get_instock == "n"){
$all_selectors[] = "(stockstatus <= '0' AND donotallowbackorders = 'Y')";
}elseif($get_instock == "Y" or $get_instock == "y") {
$all_selectors[] = "(stockstatus > '0' OR donotallowbackorders = 'N')";
};
if ($get_startat != "" or $get_results != ""){
$number_results = "LIMIT ".$get_startat.", ".$get_results;
} else {
$number_results = "LIMIT 0, 15";
};
// All variables are now in an array, except "startat" and "results"
$all_selectors0 = "WHERE ".implode(" AND ", $all_selectors);
// Create SQL query
$sql="Select * FROM sadsads_Products ".$all_selectors0." ".$number_results;
I would do something more like this. It's not tested and probably not 100% complete...you may need to do some further customization, particularly with adding more special cases to the switch statement, but this will make adding more variables much easier:
REMOVED OLD EXAMPLE, SEE UPDATED EXAMPLE BELOW
One key thing to note is that you aren't sanitizing your database inputs. Your code is vulnerable to SQL injection. My example above helps to solve that, but this code isn't fully tested, so you should ensure that all user input is sanitized before using it in any query.
If your field names don't match up with your MySQL columns (which it looks like they don't), then you can fix them with an associative array:
$columns = array(
// [form field] => [mysql column]
'size' => 'product_size',
'manufacturer' => 'product_manufacturer',
'invisible' => 'hideproduct'
// ...
);
And then in your switch statement, do something more like this:
$whereClause[] = "{$columns[$key]} = '{$value}'";
FINAL UPDATE:
DOCUMENTED SAMPLE - has plenty of comments and extra stuff to make it work on Codepad
EXACT WORKING CODE - you should be able to copy and paste this (and add your DB credentials) and it should work:
$con = mysqli_connect("[CENSORED]", "[CENSORED]", "[CENSORED]") or die("Could not connect: ". mysqli_error());
mysqli_select_db("[CENSORED]") or die("Could not select database");
$columns = array(
'size' => 'product_size',
'manufacturer' => 'product_manufacturer',
'invisible' => 'hideproduct'
);
$whereClause = array();
$limit = array("startat" => 0, "results" => 15);
foreach ($_GET as $key=>$value) {
$key = mysqli_real_escape_string($key);
if (is_array($value)) {
for ($i = 0; $i < count($value); $i++) {
$value[$i] = mysqli_real_escape_string($value[$i]);
}
} else {
$value = mysqli_real_escape_string($value);
}
switch ($key) {
case 'size':
$whereClause[] = "({$value} BETWEEN productsizeDOWN AND productsizeUP)";
break;
case 'startat':
case 'results':
$limit[$key] = $value;
break;
case 'instock':
$whereClause[] = "(stockstatus ". ($value == 'N' ? "<=" : ">") ." '0' ". ($value == 'N' ? "AND" : "OR") ." donotallowbackorders = '". ($value == 'N' ? "Y" : "N") ."')";
break;
default: {
if (is_array($value)) {
$whereClause[] = "{$columns[$key]} IN ('". implode("', '", $value) ."')";
} else {
$whereClause[] = "{$columns[$key]} = '{$value}'";
}
}
}
}
$sql = "SELECT * FROM ioa7pd_Products". (empty($whereClause) ? "" : " WHERE ". implode(" AND ", $whereClause)) ." LIMIT {$limit['startat']}, {$limit['results']}";
echo $sql;
after
else {
$product_instock = "";
};
do:
$limit = '';
if( !empty($product_limit0) && !empty($product_limit1) )
$limit = " LIMIT $product_limit0, $product_limit1";
$sql="Select * FROM ioa7pd_Products WHERE ".$product_instock.$product_size."AND".$product_manufacterer_and.$product_manufacterer."".$product_invisible." $limit"; // The end result of it all.
echo $sql;
If you have separate params in $_GET, you would have to traverse with multiple if statements. you can pass the params as an array into $_GET, with numeric keys, that would help a bunch.