I have a tool that allows data to be exported based on different time ranges. The problem I run into is that the longer the time range, the more data, and eventually the data set is too large -- resulting in either a timeout error or a memory allocation error.
Short of changing php.ini to have a larger max_execution_time, etc, is there a better way of doing this?
Here's my script to build the CSV (the user selects export, then this page is loaded):
$fileName = "export-".date("YmdHis");
$exportSignature = array(" ","Export","Generated by: ".$_SESSION['name'],date("Y-m-d H:i:s"));
header('Content-Type: text/csv; charset=utf-8');
header('Content-Disposition: attachment; filename='.$fileName.'.csv');
// create a file pointer connected to the output stream
$output = fopen('php://output', 'w');
// run all the queries for the data
if($exportType == "list") {
include('include/query-compare.php');
} elseif($exportType == "analyze") {
include('include/query-analyze.php');
} elseif($exportType == "funnel") {
include('include/query-funnel.php');
} elseif($exportType == "funnelTrend") {
include('include/query-funnel-trends.php');
}
// add column headers
fputcsv($output,$columnHeaders);
// add row data
if($exportType == "list") {
foreach($comparison as $account) {
fputcsv($output,$account);
}
} elseif($exportType == "analyze") {
foreach($analyze as $account) {
fputcsv($output,$account);
}
} elseif($exportType == "funnel" || $exportType == "funnelTrend") {
foreach($funnel as $line) {
fputcsv($output,$line);
}
}
// add export signature
foreach($exportSignature as $line) {
fputcsv($output,array($line));
}
Here's a sample of the array that's used to create the rows. This is the variable -- if the time range is small there might be a few hundred entries in the array, if it's large, there's tens of thousands.
Array
(
[0] => Array
(
[0] => 1
[firstName] => Marco
[lastName] => R.
[title] => D
[email] => test#test.com
[company] => xx
[lSource] => xx
[lDetail] => xx
[lExact] => xx
[createdAt] => 2017-06-26 00:00:00
[nDate] => 2017-08-15
[cDate] => 2017-08-15
[cMoment] => xx
[mDate] => 2017-08-15
[mMoment] => xx
[Id] => 003Axx
[Type] => Contact
[accountId] => 001A0xx
[parentAccountId] =>
[accountOwner] => Kevin S.
[xx] => Nicholas W.
[accountType] => Prospect
[totalARR] => 0.00
[accountTier] =>
[ty] => XX
[industry] => Corporate Services
[secondaryIndustry] => IT Services and Consulting
[billingCountry] => Germany
[1] => 006Axx
[2] => PS (New Business)
[3] => Kevin S.
[4] => Nicholas W.
[5] => cc
[6] => New Business
[7] => Identify
[8] => 40000.00
[9] => 2017-08-16
[10] => 2018-07-27
[11] => 2017-08-16
[12] => 2017-08-21
)
)
I'd be fine with scheduling the export and emailing it to the user as well, I'm just not familiar with how I can have this large CSV constructed in the background without timing out.
EDIT: would it be a better option to queue the export in a database and have a cron job that looks for the exports, then runs the export in the background until complete? Since it's running via the cron it shouldn't have a timeout, right?
Use set_time_limit ( 0 ); at the beginning of your file. For example:
set_time_limit ( 0 );
$fileName = "export-".date("YmdHis");
Using set_time_limit ( 0 ); didn't help since I'm running on nginx and would run into gateway errors. Instead, I wound up queuing the downloads into a database table and executing the actual queries via a cron job, as those aren't subject to the same timeout limitations.
I'm ranking a list based on a set of tie-breaking criteria using usort in PHP. My problem is that the tiebreakers aren't applied correctly when there are 3 or more ties to break. The function only sorts between 2 entries as it iterates through the array.
When 2 entries have identical Overall wins, the function checks Division wins; if those are identical, it checks Head to head; then Points if the tie still is not broken. This works great to break the tie between 2 entries, but with 3 or more ties, the iterative nature of this process is an issue.
Example:
3 entries have identical Overall wins and Division wins, so we check Head to head results.
Team 1 beat Team 2.
Team 2 beat Team 3.
Team 3 beat Team 1.
Therefore, we should move on to Points because the tie cannot be broken with Head to head as each team has one win over the other two teams. But my function doesn't ever make that third check, I guess because Team A and Team C aren't next to each other when usort() does it's thing...I'm not sure.
Here is the comparison function I pass into usort:
function set_division_order($a, $b) {
if ($a['wins'] == $b['wins'] && $a['losses'] == $b['losses']) { //same overall record
if ($a['div_wins'] == $b['div_wins'] && $a['div_losses'] == $b['div_losses']) { //same division record
$h2h = get_h2h_result($year, $a['team_id'], $b['team_id']);
if ($h2h == 0) { //same head-to-head record
return ($b['pts'] - $a['pts']);
}
else { //head-to-head winner
return $h2h;
}
}
else { //different division record
return ($b['div_wins'] - $a['div_wins']);
}
}
else { //different overall record
return ($b['wins'] - $a['wins']);
}
}
The array getting passed into usort($array, 'set_division_order') which is causing problems:
Array
(
[0] => Array
(
[team_id] => 1
[wins] => 3
[losses] => 2
[ties] => 0
[div_wins] => 3
[div_losses] => 2
[div_ties] => 0
[pts] => 513.9
)
[1] => Array
(
[team_id] => 2
[wins] => 3
[losses] => 2
[ties] => 0
[div_wins] => 3
[div_losses] => 2
[div_ties] => 0
[pts] => 504.1
)
[2] => Array
(
[team_id] => 3
[wins] => 3
[losses] => 2
[ties] => 0
[div_wins] => 3
[div_losses] => 2
[div_ties] => 0
[pts] => 517.7
)
[3] => Array
(
[team_id] => 4
[wins] => 4
[losses] => 1
[ties] => 0
[div_wins] => 4
[div_losses] => 1
[div_ties] => 0
[pts] => 491.9
)
[4] => Array
(
[team_id] => 5
[wins] => 2
[losses] => 3
[ties] => 0
[div_wins] => 2
[div_losses] => 3
[div_ties] => 0
[pts] => 393.3
)
[5] => Array
(
[team_id] => 6
[wins] => 0
[losses] => 5
[ties] => 0
[div_wins] => 0
[div_losses] => 5
[div_ties] => 0
[pts] => 377.9
)
)
Sorry this was kind of long, but hopefully the problem is pretty evident. Now it's just time for a solution! I've scoured the PHP resources for the various sort functions, with no luck. I can't be the only one trying to break a 3-way tie with PHP. :)
If I can get just the tied teams matching that criteria into a separate array, I could use a function to re-sort the original sorted array against the values of this new sub-array to order the tied teams within the main array.
...And my head just exploded.
Thanks!
Simple question here. Why is this giving an error?
for ($k = 1; $k < 17; $k++){
echo $result[0]->answer_."$k";
}
Edit: contents of $result:
Array ( [0] => stdClass Object ( [entry_id] => 37 [answer_1] => 20-24 yrs old [answer_2] => No [answer_3] => Yes [answer_4] => No [answer_5] => 236.7 [answer_6] => Ideal blood pressure – upper figure between 91 & 120, lower figure between 61& 80 [answer_7] => No, I haven’t had it checked [answer_8] => No, I quit more than 10 years ago [answer_9] => I sometimes get 150 minutes or more per week [answer_10] => 2-5 hours per day [answer_11] => Every day [answer_12] => Less often or never [answer_13] => Every day [answer_14] => 3-4 days a week [answer_15] => Never [answer_16] => imperial ) )
Try
for ($k = 1; $k < 17; $k++){
echo $result[0]->{'answer_'.$k};
}
Updated based on comment from mpen below
I am trying to calculate the winning order of golfers when they are tied in a competition.
These golf competitions are using the "stableford" points scoring system, where you score points per hole with the highest points winning. Compared to normal golf "stroke play" where the lowest score wins (though this also has the countback system, only calculating the lowest score in the event of a tie...)
The rules are to use a "countback". i.e., if scores are tied after 9 holes, the best placed of the ties is the best score from the last 8 holes. then 7 holes, etc.
The best I can come up with is 2 arrays.
An array with all the players who tied in a given round. ($ties)
One which has the full score data in (referencing the database playerid) for all 9 holes. ($tie_perhole)
I loop through array 1, pulling data from array 2 and using the following formula to create a temporary array with the highest score:
$max = array_keys($array,max($array));
If $max only has 1 item, this player is the highest scorer. the loop through the first array is "by reference", so on the next iteration of the loop, his playerid is now longer in the array, thus ignored. this continues until there is only 1 playerid left in the first array.
However, it only works if a single player wins in each iteration. The scenario that doesn't work is if a sub-set of players tie on any iterations / countbacks.
I think my problem is the current structure I have will need the original $ties array to become split, and then to continue to iterate through the split arrays in the same way...
As an example...
The $ties array is as follows:
Array
(
[18] => Array
(
[0] => 77
[1] => 79
[2] => 76
[3] => 78
)
)
The $tie_perhole (score data) array is as follows:
Array
(
[18] => Array
(
[77] => Array
(
[9] => 18
[8] => 16
[7] => 14
[6] => 12
[5] => 10
[4] => 8
[3] => 6
[2] => 4
[1] => 2
)
[79] => Array
(
[9] => 18
[8] => 17
[7] => 15
[6] => 14
[5] => 11
[4] => 9
[3] => 7
[2] => 5
[1] => 3
)
[76] => Array
(
[9] => 18
[8] => 16
[7] => 14
[6] => 12
[5] => 10
[4] => 8
[3] => 6
[2] => 4
[1] => 2
)
[78] => Array
(
[9] => 18
[8] => 17
[7] => 15
[6] => 13
[5] => 11
[4] => 9
[3] => 7
[2] => 5
[1] => 3
)
)
)
So in this competition, player's 78 and 79 score highest on the 8th hole countback (17pts), so 1st and 2nd should be between them. Player 79 should then be 1st on the 6th hole countback (14pts, compared to 13pts). The same should occur for 3rd and 4th place with the 2 remaining other players.
There are other scenarios that can occur here, in that within a competition, there will likely be many groups of players (of different amounts) on different tied points through the leaderboard.
Also note, there will be some players on the leaderboard who are NOT tied and stay in their current outright position.
The basics of the working code I have is:
foreach ($ties as $comparekey => &$compareval) {
$tie_loop = 0;
for ($m = 9; $m >= 1; $m--) {
$compare = array();
foreach ($compareval as $tie) {
$compare[$tie] = $tie_perhole[$comparekey][$tie][$m];
}
$row = array_keys($compare,max($compare));
if (count($row) == 1) {
$indexties = array_search($row[0], $ties[$comparekey]);
unset($ties[$comparekey][$indexties]);
// Now update this "winners" finishing position in a sorted array
// This is a multidimensional array too, with custom function...
$indexresults = searchForId($row[0], $comp_results_arr);
$comp_results_arr[$indexresults][position] = $tie_loop;
$tie_loop++;
}
// I think I need conditions here to filter if a subset of players tie
// Other than count($row) == 1
// And possibly splitting out into multiple $ties arrays for each thread...
if (empty($ties[$comparekey])) {
break;
}
}
}
usort($comp_results_arr, 'compare_posn_asc');
foreach($comp_results_arr as $row) {
//echo an HTML table...
}
Thanks in advance for any helpful insights, tips, thoughts, etc...
Robert Cathay asked for more scenarios. So here is another...
The leaderboard actually has more entrants (player 26 had a bad round...), but the code i need help with is only bothered about the ties within the leaderboard.
Summary leaderboard:
Points Player
21 48
21 75
20 73
20 1
13 26
This example produces a $tie_perhole array of:
Array
(
[21] => Array
(
[75] => Array
(
[9] => 21
[8] => 19
[7] => 16
[6] => 14
[5] => 12
[4] => 9
[3] => 7
[2] => 5
[1] => 3
)
[48] => Array
(
[9] => 21
[8] => 19
[7] => 16
[6] => 13
[5] => 11
[4] => 9
[3] => 8
[2] => 5
[1] => 3
)
)
[20] => Array
(
[73] => Array
(
[9] => 20
[8] => 18
[7] => 16
[6] => 13
[5] => 11
[4] => 8
[3] => 6
[2] => 5
[1] => 3
)
[1] => Array
(
[9] => 20
[8] => 17
[7] => 16
[6] => 14
[5] => 12
[4] => 9
[3] => 7
[2] => 4
[1] => 2
)
)
)
In this example, the array shows that players 75 and 48 scored 21 points that player 75 will eventually win on the 6th hole countback (14pts compared to 13pts) and player 48 is 2nd. In the next tied group, players 73 and 1 scored 20 points, and player 73 will win this group on the 8th hole countback and finishes 3rd (18 pts compared to 17 pts), with player 1 in 4th. player 26 is then 5th.
Note, the $tie_loop is added to another array to calculate the 1st to 5th place finishing positions, so that is working.
Hopefully that is enough to help.
Ok, so I don't understand golf at all... hahaha BUT! I think I got the gist of this problem, so heres my solution.
<?php
/**
* Author : Carlos Alaniz
* Email : Carlos.glvn1993#gmail.com
* Porpuse : Stackoverflow example
* Date : Aug/04/2015
**/
$golfers = [
"A" => [1,5,9,1,1,2,3,4,9],
"B" => [2,6,4,2,4,4,1,9,3],
"C" => [3,4,9,8,1,1,5,1,3],
"D" => [1,5,1,1,1,5,4,5,8]
];
//Iterate over scores.
function get_winners(&$golfers, $hole = 9){
$positions = array(); // The score numer is the key!
foreach ($golfers as $golfer=>$score ) { // Get key and value
$score_sub = array_slice($score,0,$hole); // Get the scores subset, first iteration is always all holes
$total_score = (string)array_sum($score_sub); // Get the key
if(!isset($positions[$total_score])){
$positions[$total_score] = array(); // Make array
}
$positions[$total_score][] = $golfer; // Add Golpher to score.
}
ksort($positions, SORT_NUMERIC); // Sort based on key, low -> high
return array(end($positions), key($positions)); // The last shall be first
}
//Recursion is Awsome
function getWinner(&$golfers, $hole = 9){
if ($hole == 0) return;
$winner = get_winners($golfers,$hole); // Get all ties, if any.
if(count($winner[0]) > 1){ // If theirs ties, filter again!
$sub_golfers =
array_intersect_key($golfers,
array_flip($winner[0])); // Only the Worthy Shall Pass.
$winner = getWinner($sub_golfers,$hole - 1); // And again...
}
return $winner; // We got a winner, unless they really tie...
}
echo "<pre>";
print_R(getWinner($golfers));
echo "</pre>";
Ok... Now ill explain my method...
Since we need to know the highest score and it might be ties, it makes no sense to me to maintain all that in separate arrays, instead I just reversed the
golfer => scores
to
Tota_score => golfers
That way when we can sort the array by key and obtain all the golfers with the highest score.
Now total_score is the total sum of a subset of the holes scores array. So... the first time this function runs, it will add all 9 holes, in this case theres 3 golfers that end up with the same score.
Array
(
[0] => Array
(
[0] => A
[1] => B
[2] => C
)
[1] => 35
)
Since the total count of golfers is not 1 and we are still in the 9th hole, we run this again, but this time only against those 3 golfers and the current hole - 1, so we are only adding up to the 8th hole this time.
Array
(
[0] => Array
(
[0] => B
[1] => C
)
[1] => 32
)
We had another tie.... this process will continue until we reach the final hole, or a winner.
Array
(
[0] => Array
(
[0] => C
)
[1] => 31
)
EDIT
<?php
/**
* Author : Carlos Alaniz
* Email : Carlos.glvn1993#gmail.com
* Porpuse : Stackoverflow example
**/
$golfers = [
"77" => [2,4,6,8,10,12,14,16,18],
"79" => [3,5,7,9,11,14,15,17,18],
"76" => [2,4,6,8,10,12,14,16,18],
"78" => [3,5,7,9,11,13,15,17,18]
];
//Iterate over scores.
function get_winners(&$golfers, $hole = 9){
$positions = array(); // The score numer is the key!
foreach ($golfers as $golfer => $score) { // Get key and value
//$score_sub = array_slice($score,0,$hole); // Get the scores subset, first iteration is always all holes
$total_score = (string)$score[$hole-1]; // Get the key
if(!isset($positions[$total_score])){
$positions[$total_score] = array(); // Make array
}
$positions[$total_score][] = $golfer; // Add Golpher to score.
}
ksort($positions, SORT_NUMERIC); // Sort based on key, low -> high
return [
"winner"=> end($positions),
"score" => key($positions),
"tiebreaker_hole" => [
"hole"=>$hole,
"score"=> key($positions)],
]; // The last shall be first
}
//Recursion is Awsome
function getWinner(&$golfers, $hole = 9){
if ($hole == 0) return;
$highest = get_winners($golfers,$hole); // Get all ties, if any.
$winner = $highest;
if(count($winner["winner"]) > 1){ // If theirs ties, filter again!
$sub_golfers =
array_intersect_key($golfers,
array_flip($winner["winner"])); // Only the Worthy Shall Pass.
$winner = getWinner($sub_golfers,$hole - 1); // And again...
}
$winner["score"] = $highest["score"];
return $winner; // We got a winner, unless they really tie...
}
echo "<pre>";
print_R(getWinner($golfers));
echo "</pre>";
Result:
Array
(
[winner] => Array
(
[0] => 79
)
[score] => 18
[tiebreaker_hole] => Array
(
[hole] => 6
[score] => 14
)
)
Im trying to fetch data from mysql but only the first character for each field is loaded. However this only happens when I upload my site to my domain. On localhost it works fine. So i can only assume its a server issue?
my Query is:
$stmt = $mysqli->prepare("SELECT Prd_ID, product_title, model_no, part_no, category_id, subcat_id, weights, ean, condition_id, brand_id, warranty_id, collect_serial, retail, listprice, short_desc, full_desc, tax_class_id, web_status_id FROM product WHERE Prd_ID = ?");
$stmt->bind_param("s", $_GET['pid'] );
$stmt->store_result();
$stmt->bind_result($pid, $product_title, $model_no, $part_no, $category_id, $subcat_id, $weights, $ean, $condition_id, $brand_id, $warranty_id, $collect_serial, $retail, $listprice, $short_desc, $full_desc, $tax_class_id, $web_status_id);
if( $stmt->execute() ) {
// Loads of code.. to much to paste here
}
i have used the following to see the output on both localhost and domain online:
print_r(get_defined_vars(),true);
The following are the results i got.
Localhost
[pid] => 5
[product_title] => HP Envy 15-J140SA Intel Core i5-4200M 8GB 1TB Nvidia GT840 2GB Windows 8.1
[model_no] => 15-J140SA
[part_no] => J0B97EA
[category_id] => 1
[subcat_id] => 2
[weights] => 2.50
[ean] => 0888793319512
[condition_id] => 4
[brand_id] => 19
[warranty_id] => 3
[collect_serial] => 2
[retail] => 599.99
[listprice] => 499.99
[short_desc] =>
[full_desc] =>
[tax_class_id] => 1
[web_status_id] => 2
[row] => 1
)
Online
[pid] => 5
[product_title] => H
[model_no] => 1
[part_no] => J
[category_id] => 1
[subcat_id] => 2
[weights] => 2
[ean] => 0
[condition_id] => 4
[brand_id] => 19
[warranty_id] => 3
[collect_serial] => 2
[retail] => 5.00
[listprice] => 4.00
[short_desc] =>
[full_desc] =>
[tax_class_id] => 1
[web_status_id] => 2
[row] => 1
)
The PHP code is a carbon copy on both sides. (localhost and online) and the databases are the same as well. I've been banging my head tryna figure this out. I'd like to know if this is a server problem since it works fine on localhost.
Thanks in advance.
EDIT:
$row = $stmt->fetch();
if( $pid == $_GET['pid'] ){
// Pre-load details
$isCopy = true;
$product_title = $product_title;
$model_no = $model_no;
$part_no = $part_no;
$category_id = $category_id;
$subcat_id = $subcat_id;
$weights = $weights;
$ean = $ean;
$condition_id = $condition_id;
$brand_id = $brand_id;
$warranty_id = $warranty_id;
$collect_serial = $collect_serial;
$retail = number_format($retail, 2);
$listprice = number_format($listprice, 2);
$short_desc = $short_desc;
$full_desc = $full_desc;
$tax_class_id = $tax_class_id;
$web_status_id = $web_status_id;
$stmt->close();
}
Solved it. Removed the $full_desc from the paramaters and it worked. I dont know why it was this particular field and nothing else.