I'm running a stock update on our company website, and the query is as follows;
$count = count($stock);
for($i = 0; $i < $count; $i++)
{
if(strtolower($stock[$i]['stockloc']) == 'retail')
{
if($stock[$i]['avail'] < 0)
{
$stock[$i]['avail'] = 0;
}
mysql_query("UPDATE `xcart_products`
SET `avail` = ".$stock[$i]['avail']."
WHERE `productid` IN
(SELECT `productid`
FROM `xcart_extra_field_values`
WHERE `productid` = ".$stock[$i]['productid']."
AND `fieldid` = 5
AND LOWER(value) = 'retail')");
}
}
Eessentially it runs in a for loop and iterates through thousands of products, updating stock levels. It runs fine for about 4 minutes, then things start to lag on our website, mysql seems to become backlogged with other website db queries that it can't run (they all appear locked), which then leads to the inevitable;
Internal Server Error The server encountered an internal error or
misconfiguration and was unable to complete your request.
Please contact the server administrator to inform of the time the
error occurred and of anything you might have done that may have
caused the error.
More information about this error may be available in the server error
log.
It doesn't seem to be a timeout I've upped the limit in PHP.ini to no avail. The query isn't particularly complex, however it does run several thousand times. Maybe as many as 5000+. I'm not sure what the problem is, as we ran a similar update system on our old website, and although it wasn't one I designed, it didn't have trouble uploading this many stock quantity changes.
I'll be happy to expand my question if further information is needed, but honestly, I'm at a bit of a loss even how to trace the problem.
The $stock array is an associative array which has the following format;
Array
(
[0] => Array
(
[avail] => 55
[productid] => 17761
[sku] => AER-TOUCH-2100
[stockloc] => retail
)
[1] => Array
(
[avail] => 166
[productid] => 22653
[sku] => CAS-AER-112
[stockloc] => retail
)
[2] => Array
(
[avail] => 272
[productid] => 22952
[sku] => CAS-ANT-001
[stockloc] => retail
)
[3] => Array
(
[avail] => 5
[productid] => 22956
[sku] => CAS-ANT-005
[stockloc] => retail
)
[4] => Array
(
[avail] => 12
[productid] => 22958
[sku] => CAS-ANT-007
[stockloc] => retail
I have tried running a delay every x iterations of the loop, but I'm still getting the 500 internal server error message. Clearly I need a better solution to updating my table.
Related
I have a tool that allows data to be exported based on different time ranges. The problem I run into is that the longer the time range, the more data, and eventually the data set is too large -- resulting in either a timeout error or a memory allocation error.
Short of changing php.ini to have a larger max_execution_time, etc, is there a better way of doing this?
Here's my script to build the CSV (the user selects export, then this page is loaded):
$fileName = "export-".date("YmdHis");
$exportSignature = array(" ","Export","Generated by: ".$_SESSION['name'],date("Y-m-d H:i:s"));
header('Content-Type: text/csv; charset=utf-8');
header('Content-Disposition: attachment; filename='.$fileName.'.csv');
// create a file pointer connected to the output stream
$output = fopen('php://output', 'w');
// run all the queries for the data
if($exportType == "list") {
include('include/query-compare.php');
} elseif($exportType == "analyze") {
include('include/query-analyze.php');
} elseif($exportType == "funnel") {
include('include/query-funnel.php');
} elseif($exportType == "funnelTrend") {
include('include/query-funnel-trends.php');
}
// add column headers
fputcsv($output,$columnHeaders);
// add row data
if($exportType == "list") {
foreach($comparison as $account) {
fputcsv($output,$account);
}
} elseif($exportType == "analyze") {
foreach($analyze as $account) {
fputcsv($output,$account);
}
} elseif($exportType == "funnel" || $exportType == "funnelTrend") {
foreach($funnel as $line) {
fputcsv($output,$line);
}
}
// add export signature
foreach($exportSignature as $line) {
fputcsv($output,array($line));
}
Here's a sample of the array that's used to create the rows. This is the variable -- if the time range is small there might be a few hundred entries in the array, if it's large, there's tens of thousands.
Array
(
[0] => Array
(
[0] => 1
[firstName] => Marco
[lastName] => R.
[title] => D
[email] => test#test.com
[company] => xx
[lSource] => xx
[lDetail] => xx
[lExact] => xx
[createdAt] => 2017-06-26 00:00:00
[nDate] => 2017-08-15
[cDate] => 2017-08-15
[cMoment] => xx
[mDate] => 2017-08-15
[mMoment] => xx
[Id] => 003Axx
[Type] => Contact
[accountId] => 001A0xx
[parentAccountId] =>
[accountOwner] => Kevin S.
[xx] => Nicholas W.
[accountType] => Prospect
[totalARR] => 0.00
[accountTier] =>
[ty] => XX
[industry] => Corporate Services
[secondaryIndustry] => IT Services and Consulting
[billingCountry] => Germany
[1] => 006Axx
[2] => PS (New Business)
[3] => Kevin S.
[4] => Nicholas W.
[5] => cc
[6] => New Business
[7] => Identify
[8] => 40000.00
[9] => 2017-08-16
[10] => 2018-07-27
[11] => 2017-08-16
[12] => 2017-08-21
)
)
I'd be fine with scheduling the export and emailing it to the user as well, I'm just not familiar with how I can have this large CSV constructed in the background without timing out.
EDIT: would it be a better option to queue the export in a database and have a cron job that looks for the exports, then runs the export in the background until complete? Since it's running via the cron it shouldn't have a timeout, right?
Use set_time_limit ( 0 ); at the beginning of your file. For example:
set_time_limit ( 0 );
$fileName = "export-".date("YmdHis");
Using set_time_limit ( 0 ); didn't help since I'm running on nginx and would run into gateway errors. Instead, I wound up queuing the downloads into a database table and executing the actual queries via a cron job, as those aren't subject to the same timeout limitations.
I have a grid of img squares that can be dragged into any order using the sortable library. Each img is a visual representation of a result from a mySQL db query that selects any image that shares an 'imageparent' identifier. The order they're presented in the grid is taken from the 'imageorder' column in the database and starts at 0 and works in sequence up to the nth number of images returned.
The purpose of dragging the img grid is to be able to change the 'imageorder' index. On completion of the drag, the sortable library POSTS an 'imageorder' var by ajax to service.php and is received correctly. So rather than the original 0,1,2,3,4,5,6,7 order of the original, it sends a string like 2,1,0,3,4,5,7,6. Not too hard to grasp. After I switch the order the orderList var sent to service.php is always correct, but the array I end up sending to the db and setting as my session var becomes a little garbled in order after the second or third drag and I'm not quite sure why.
Code Examples and Comments
$_SESSION['selectedCsImages'] Array structure:
[0] => Array
(
[imagename] => "Title"
[imageorder] => 0
[imageid] => 43
)
[1] => Array
(
[imagename] => "Title"
[imageorder] => 1
[imageid] => 21
)
[2] => Array
(
[imagename] => "Title"
[imageorder] => 2
[imageid] => 3
)
etc...
Services.php extract:
if (session_status() == PHP_SESSION_NONE) {
session_start();
}
// Turn the orderList posted into an array
$removeChars = array('"','[',']');
$orderList = str_replace($removeChars, "", $_POST['order']); // POST received fine.
$listArray = explode(",",$orderList);
// Retrieve the session array
$sorting = $_SESSION['selectedCsImages'];
/* My logic is that I compare the $sorting array to $listArray and reorder $sorting by 'imageorder' to match $listarray */
usort($sorting, function($a, $b) use ($listArray) {
return array_search($a['imageorder'], $listArray) - array_search($b['imageorder'], $listArray);
});
/* I now have a $sorting array that (sometimes, hence the problem) matches the order that the images had just been dragged into by the user. Typically, as I mentioned above, it's correct after the first drag, but not always after the second or third where it creates a new order that I can't see a pattern or logic in. */
/* Had there not been errors with the usort function, I (would) have a $sorting array in the order I want but with imageorder values referring to pre-sorting. I iterate through the array and set each key to 0, 1, 2, etc. so that I have an array in the correct order and with each imageorder correctly stating its place.*/
$i = 0;
foreach ($sorting as $key => $value) {
$sorting[$key]['imageorder'] = $i;
$i++;
}
/* The information is attempted to be sent to the db and, on success I update the session var */
// Database code (runs succesfully and updates the db as per the image orders found in the $sorting array)
$_SESSION['selectedCsImages'] = $sorting;
Debugging:
From debugging, it appears that something happens with the usort function when I call this page from ajax for the second or third time. Everything after this follows through fine and processes the correct or incorrect order as per expectations. The orderList var posted by sortable is correct each time. I'd provide a sample of the $sorted var after usort each time but it's as simple to describe it as the above array example in an order I didn't specify after dragging and I can't see a pattern in the seemingly random order it outputs.
From researching, I had thought that it was an issue with session vars being retained until the page is refreshed but it appears that the ajax call to services.php should refresh the $_SESSION['selectedCsImages'] var. I had also read that, perhaps, I was unknowingly using referenced array values and - as I source from a session var to a new array and, ultimately, save back to this session var from this array - I may have created some messy referencing feedback. However, I tried using $sorted = (array)clone(object)$_SESSION['selectedCsImages']; before attempting usort and the results didn't change.
PHP error logs are showing nothing.
Updates:
Per the suggestion of #Ayaou, I've checked the output of $listArray and am getting some unexpected results. I'd wrongly assumed that as the posted $orderList was correct, that the exploded array would not be a culprit.
Here's the output of print_r($listArray) after completing the following order swaps of 16 img elements: 1st with 2nd, 2nd last with last,6th with 7th:
1st and 2nd:
(
[0] => 1
[1] => 0
[2] => 2
[3] => 3
[4] => 4
[5] => 5
[6] => 6
[7] => 7
[8] => 8
[9] => 9
[10] => 10
[11] => 11
[12] => 12
[13] => 13
[14] => 14
[15] => 15
)
last and 2nd last:
(
[0] => 1
[1] => 0
[2] => 2
[3] => 3
[4] => 4
[5] => 5
[6] => 6
[7] => 7
[8] => 8
[9] => 9
[10] => 10
[11] => 11
[12] => 12
[13] => 13
[14] => 15
[15] => 14
)
6th with 7th:
(
[0] => 1
[1] => 0
[2] => 2
[3] => 3
[4] => 4
[5] => 6
[6] => 5
[7] => 7
[8] => 8
[9] => 9
[10] => 10
[11] => 11
[12] => 12
[13] => 13
[14] => 15
[15] => 14
)
I was progressing with the idea that $listArray would show a sequential 0,1,2,3,etc. each time with only the two swapped items showing order changes. As it's not, I'll look back again at $orderList and check if my sortable library is updating the orders it's obtaining correctly from the updated session var. Older order swaps are being retained somewhere along the chain where they shouldn't.
The solution is on your sortable form (on the front end), so instead of sending the imageorder on your 'order' post data, send the imageid index.
Then change your sort callback like this
//Use imageid index instead of imageorder
usort($sorting, function($a, $b) use ($listArray) {
return array_search($a['imageid'], $listArray) - array_search($b['imageid'], $listArray);
});
on my ratings table for my software i have 4 fields.
id autoincrement
rvid vendor id
ratedate date of rating
rating the actual numeric rating
I have done alot with it over the last few months but this time im stumped and i cant get a clear picture in my head of the best way to do this. What i am trying to do is find out if the vendor has had 3 low 'consecutive' ratings. If their last three ratings have been < 3 then i want to flag them.
I have been playing with this for a few hours now so i thought i would ask (not for the answer) but for some path direction just to push me forward, im stuck in thought going in circles here.
I have tried GROUP BY and several ORDER BY but those attempts did not go well and so i am wondering if this is not a mysql answer but a php answer. In other words maybe i just need to take what i have so far and just move to the php side of things via usort and the like and do it that way.
Here is what i have so far i did select id as well at first thinking that was the best way to get the last consective but then i had a small breakthrough that if they have had 3 in a row the id does not matter, so i took it out of the query.
$sql = "SELECT `rvid`, `rating` FROM `vendor_ratings_archive` WHERE `rating` <= '3' ORDER BY `rvid` DESC";
which give me this
Array
(
[0] => Array
(
[rvid] => 7
[rating] => 2
)
[1] => Array
(
[rvid] => 5
[rating] => 1
)
[2] => Array
(
[rvid] => 5
[rating] => 0
)
[3] => Array
(
[rvid] => 5
[rating] => 3
)
)
this is just just samples i tossed in the fields, and there are only 4 rows here where as in live it will be tons of rows. But basically this tells me that these are the vendors that have low ratings in the table. And that is where i get stumpted. I can only do one sort in the query so that is why i am thinking that i need to take this and move to the php side to finish it off.
I think i need to sort the elements by rvid with php first i think, and then see if three elements in a row are the same vender (rvid).
Hope that makes sense. My brain hurts lol...
update - here is all of the table data using *
Array
(
[0] => Array
(
[id] => 7
[rvid] => 7
[ratedate] => 2016-05-01
[rating] => 2
)
[1] => Array
(
[id] => 8
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 1
)
[2] => Array
(
[id] => 6
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 0
)
[3] => Array
(
[id] => 5
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 3
)
)
Here's one way you can begin approaching this - completely in SQL:
Get the last rating for the vendor. ORDER BY date DESC, limit 1.
Get the second to last rating for the vendor. ORDER BY date DESC, limit 1, OFFSET 1.
Then write a query that does a LEFT join of the first two tables. You will have a dataset that has three columns:
vendor id
last rating
second to last rating
Then you can write an expression that says "if column1 is <3 and column2 < 3, then this new column is true"
You should be able to extend this to three columns relatively easily.
Here is what a came up with to solve this riddle. I think explaining it on here helped as well as Alex also helped as he keyed my brain on using the date. I first started looking at using if statment inside of the query and actually that got my brain out of the box and then it hit me what to do.
It is not perfect and certainly could use some trimming to reduce the code, but i understand it and it seems to work, so that is par for me on this course.
the query...
$sql = "SELECT `rvid`, `ratedate`,`rating` FROM `vendor_ratings_archive` WHERE `rating` <= '3' ORDER BY `ratedate`, `rvid` DESC";
which gives me this
Array
(
[0] => Array
(
[rvid] => 7
[ratedate] => 2016-05-01
[rating] => 2
)
[1] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 1
)
[2] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 0
)
[3] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 3
)
)
notice how vendor (rvid) 5 is grouped together which is an added plus.
next a simple foreach to load a new array
foreach($results as $yield)
{
$rvidarray[] = $yield['rvid'];
}//close foreach
which gives me this
Array
(
[0] => 7
[1] => 5
[2] => 5
[3] => 5
)
then we count the array values to group dups
$rvidcounter = array_count_values($rvidarray);
which results in this
Array(
[7] => 1
[5] => 3
)
so now vender 7 as 1 low score and vendor 5 has 3 low scores and since they were already sorted by date i know that its consecutive. Well it sounds good anyway lol ")
then we create our final array with another foreach
foreach($rvidcounter as $key => $value)
{
//anything 3 or over is your watchlist
if($value > 2)
{
$watchlist[] = $key; //rvid number stored
}
}//close foreach
which gives me this
Array
(
[0] => 5
)
this was all done in a service function. So the final deal is everyone in this array has over 3 consecutive low ratings and then i just use a returned array back in my normal php process file and grab the name of each vender by id and pass that to the html and print out the list.
done...
please feel free to improve on this if you like. I may or may not use it because the above code makes sense to me. Something more complicated may not make sense to me 6 mos from now lol But it would be interesting to see what someone comes up with to shorten the process a bit.
Thanks so much and Happy Coding !!!!!
Dave :)
You could do it in SQL like that:
SET #rvid = -1;
SELECT DISTINCT rvid FROM
(
SELECT
rvid,
#neg := rating<3, /* 0 or 1 for addition in next line */
#count := IF(#rvid <> rvid , #neg, #count+#neg) AS `count`, /* set or add */
#rvid := rvid /* remember last row */
FROM
testdb.venrate
ORDER BY
rvid, datetime desc
) subq
WHERE count>=3
;
You set a variable to a non existing id. In each chronologically sorted row you check if rating is too low, that results in 1 (too low) or 0 (ok). If rvid is not equal to the last rvid, it means a new vender section is beginning. On begin of section set the value 0 or 1, else add this value. Finally store the current row's rvid for comparison in next row process.
The code above is looking for 3 consecutive low ratings (low means a value less than 3) over all the time.
A small modification checks if all the latest 3 ratings has been equal to or less than 3:
SET #rvid = -1;
SELECT DISTINCT
rvid
FROM
(
SELECT
rvid,
#high_found := rating>3 OR (#rvid = rvid AND #high_found) unflag,
#count := IF(#rvid <> rvid , 1, #count+1) AS `count`,
#rvid := rvid /* remember last row */
FROM
testdb.venrate
ORDER BY
rvid, datetime desc
) subq
WHERE count=3 AND NOT unflag
;
I have the following code:
$this->api = new App_Other_SphinxSearch();
$this->api->SetServer($host, $port);
$this->api->SetConnectTimeout(1);
$this->api->SetArrayResult(true);
$results = $this->api->Query("#(title,content) test", 'members');
echo "<pre>";print_r($results);die;
According to their documentation, a syntax like #(field_1,field_2) query should return docs which match the string query in either field_1 or field_2.
The PHP SDK returns something entirely different:
Array
(
[error] =>
[warning] =>
[status] => 0
[fields] => Array
(
[0] => title
[1] => content
)
[attrs] => Array
(
[created] => 2
[content] => 7
)
[total] => 0
[total_found] => 0
[time] => 0.000
[words] => Array
(
[title] => Array
(
[docs] => 10
[hits] => 34
)
[content] => Array
(
[docs] => 34
[hits] => 139
)
[test] => Array
(
[docs] => 26
[hits] => 34
)
)
)
There is no matches key in the array, however it got some hits. I don't really understand why this is happening, especially because if I try the same query from the command line, everything works correctly.
Any help?
Edit: Querying like this works though: #* test. Not really what I want though, cause that searches in all fields.
[total_found] => 0 says there where no matches.
The words array, just tells you now many documents and how many times that word appears in ANY (and all) fields. (WITHOUT regard to your specific query)
Sphinx caches those word stats, which help it make quick sanity checks on queries. When the query runs it can end up not matching any documents (because only then does it apply the field level filters), even though the individual words are found.
That explains your misinpretation of the results, but not why getting the results you get.
You are entering a Extended Mode query (as evidenced by the link to the sphinx documentation) , but the sphinx API defaults to ALL query mode. Notice also that title and content are in the words array, so are being taken as plain keywords not as syntax.
So you need to include a
$this->api->SetMatchMode(SPH_MATCH_EXTENDED);
btw, #* test, works because the #* are simply ignored in ALL query mode.
I'm using the Nestoria API to retrieve property results.
Everything is working quite well and one can return up to 50 properties using this method.
I would like to show 10 items at a time and allow for the user to paginate through them, but for some reason I'm having difficulty doing this.
The code snippet around the section that controls this is as follows:
$page = isset($_REQUEST["page"]) ? (int)$_REQUEST["page"] : 1;
$page = $page-1;
$pagination = new pagination;
$propertyResults = $pagination->generate($nestoria->decodedData->response->listings, 10);
foreach($propertyResults as $listing) {
//do stuff
}
A snippet of the data array would be:
Array
(
[0] => stdClass Object
(
[auction_date] =>
[property_type] => house
[summary] => Located in North Kingston a two double bedroom Victorian house presented in...
[title] => York Road, Kingston, KT2 - Reception
[updated_in_days] => 6.5
[updated_in_days_formatted] => this week
)
[1] => stdClass Object
(
[auction_date] =>
[property_type] => house
[summary] => Fine home was built about 50 years ago and enjoys one of the best locations...
[title] => Coombe Hill, KT2 - Conservatory
[updated_in_days] => 2.5
[updated_in_days_formatted] => this week
)
....
(sample cut down due to size of array elements)
Now I have been staring at this for way too long now and I've drawn a blank.
This code works correctly except if I try go to any other page other than 1, then the page doesn't finish loading, it just carries on until Firefox says: "The page isn't redirecting properly".
So basically, pagination is able to cut my data array up correctly, but is failing to "paginate" correctly.
Any help?
Turns out the problem with the redirect was actually an .htaccess issue which was using the $_GET["page"] variable and was therefore getting confused, so I renamed all references to the $_GET["page"] to $_GET["_page"] in this app.