PHP Memory exceeded while writing to array in a loop - php

I need to update a value in an array inside a loop. Normally this works fine, but something about this loop is causing the memory to max out.
while ($watering_window['total_run_time'] > $water_window_total)
{
foreach ($master_schedule as &$_ms)
{
$current_soak_time = $_ms['between_starts'] - $_ms['total_run_time'];
if ($current_soak_time > 0)
{
$new_soak_time = ceil($current_soak_time * 0.9);
$_ms['between_starts'] = $_ms['total_run_time'] + $new_soak_time;
$watering_window['total_run_time'] -= $current_soak_time - $new_soak_time;
}
}
}
Something about writing to $_ms['between_starts'] (an existing key) is causing the memory problem. If I change it to write a NEW key (like $_ms['between_starts_new'], no memory problem. I also tried writing a constant value to it (instead of something mathy) and it still timed out.

You'll need to increase the memory_limit.
Add the following at the top of your php script:
<?php
ini_set('memory_limit','256M');
...
Adjust the value according to your needs.

Try unsetting the key that is causing the problem before you set it.
foreach ($master_schedule as &$_ms)
{
$current_soak_time = $_ms['between_starts'] - $_ms['total_run_time'];
if ($current_soak_time > 0)
{
$new_soak_time = ceil($current_soak_time * 0.9);
unset($master_schedule['between_starts']);
$_ms['between_starts'] = $_ms['total_run_time'] + $new_soak_time;
$watering_window['total_run_time'] -= $current_soak_time - $new_soak_time;
}
}

Related

usleep() in while loop causing issues - whole process freezes

I am using php, Laravel, Redis, and SQL on an Ubuntu localhost server. I have made a bunch of methods that return results from API searches after some processing. I am calling 5 of these methods which will be very slow if done synchronously, so I've been experimenting with async approaches (which I know php isn't optimised for). After a few approaches I have found some success with pcntl_fork(), but I'm running into some nasty problems.
Edit: After some messing around I have found that if I remove the while loop then the code afterward executes properly, I have removed the while loop and placed it in the second 'search' method. However it still causes a freeze of the system. This makes no sense as there shouldn't be an infinite loop as if I manually query the Redis db, all 5 results are there.
This is my code: (I have a few custom classes for making and processing the API calls, fyi these methods work flawlessly)
//this caches the individual api results to a Redis list
public static function cacheAsyncApiSearch(string $searchQuery, int $maxResults = 20)
{
$key = "search:".$searchQuery; //for Redis
if(!Redis::client()->exists($key)) {
for ($i = 0; $i < 5; $i++) {
// Create a child process
$pid = pcntl_fork();
if ($pid == -1) {
// Fork failed
exit(1);
} elseif ($pid) {
// This is the parent process
// I have tried many versions of pcntl_wait, none work! They all still don't allow code to be ran afterwards (even within this elseif block), and the best it does is cache the 1st api case (YouTube)
// while (!pcntl_wait($status, WNOHANG)) {
// $exitStatus = pcntl_wexitstatus($status);
// // Do something with the exit status of the child process
// }
// dd($pid);
// pcntl_waitpid($pid, $status, WUNTRACED);
} else {
//child processes
switch ($i) {
case 0:
$results = YouTube::search($searchQuery, $maxResults)['results'];
Redis::client()->rPush($key,SearchResultDTO::jsonEncodeArray($results));
SearchResultDTO::convertResultDTOToModels($results);
break;
case 1:
$results = Dailymotion::search($searchQuery, $maxResults)['results'];
Redis::client()->rPush($key,SearchResultDTO::jsonEncodeArray($results));
SearchResultDTO::convertResultDTOToModels($results);
break;
case 2:
$results = Vimeo::search($searchQuery, $maxResults)['results'];
Redis::client()->rPush($key,SearchResultDTO::jsonEncodeArray($results));
SearchResultDTO::convertResultDTOToModels($results);
break;
case 3:
$results = Twitch::search($searchQuery, 2)['results'];
Redis::client()->rPush($key,SearchResultDTO::jsonEncodeArray($results));
SearchResultDTO::convertResultDTOToModels($results);
break;
case 4:
$results = Podcasts::getPodcastsFromItunesResults(Podcasts::search($searchQuery, 2)["response"]->results);
Redis::client()->rPush($key,SearchResultDTO::jsonEncodeArray($results));
SearchResultDTO::convertResultDTOToModels($results);
break;
}
$i = 10000;
exit(0);
}
}
// for noting the process id of the given process that gets to this point
Redis::client()->lPush("search_pid:".$searchQuery, $pid);
// sets a time out for the redis cache
Redis::client()->expire($key, 60*60*4);
while (is_numeric( Redis::client()->lLen($key)) && Redis::client()->lLen($key) < 5) {
usleep(500000); // 0.5 seconds
// pcntl_waitpid(-1, $status); //does this even do anything? not for me
}
return false; // not already cached
}
return true; // already cached
}
This code somewhat works, It performs the api calls and caches the Redis perfectly. However when the method is ran, no code will be ran after it (unless redis has found a cached version and the process is not forked).
This made me think that all processes are being exited (possibly true? if so i dont know why), so I tried writing a version without the exit(0) line. This works, I can then perform code after the method call, however I noticed (when getting SQL race conditions) that all 6 (5 child, 1 parent) processes continued to run their own version of the code after this method (e.g. some database writes)
public static function search(string $searchQuery, int $maxResults = 20): array
{
$key = "search:".$searchQuery;
$results = [];
// the quoted method above
self::cacheAsyncApiSearch($searchQuery, $maxResults);
foreach (Redis::client()->lRange($key,0,-1) as $result){
$results = array_merge($results, SearchResultDTO::jsonDecodeArray($result));
}
$creatorDTOs = [];
$videoDTOs = [];
$streamDTOs = [];
$playlistDTOs = [];
$podcastDTOs = [];
/** #var SearchResultDTO $result */
foreach ($results as $result) {
match ($result->kind) {
Kind::Creator => $creatorDTOs[] = $result,
Kind::Video => $videoDTOs[] = $result,
Kind::Stream => $streamDTOs[] = $result,
Kind::Playlist => $playlistDTOs[] = $result,
Kind::Podcast => $podcastDTOs[] = $result,
};
}
// did this to test how many times the code was being ran (the list has 6 1's in it)
Redis::client()->lPush("here", '1');
// I know this code isn't completely efficient since I already called these conversion methods before, however I am just trying to get the forking stuff to work right now.
return [
"creators" => SearchResultDTO::convertResultDTOToModels($creatorDTOs),
"videos" => SearchResultDTO::convertResultDTOToModels($videoDTOs),
"streams" => SearchResultDTO::convertResultDTOToModels($streamDTOs),
"playlists" => SearchResultDTO::convertResultDTOToModels($playlistDTOs),
"podcasts" => SearchResultDTO::convertResultDTOToModels($podcastDTOs)
];
}
These DTO's (Data Transfer Objects) are being used to populate a UI. So for example, when I make a search (that isn't cached), the page is blank forever. But if I refresh the page (after the search is cached) then the results show just fine.
This is the most bizarre problem I have ever ran into and I really appreciate any help.
Edit please read:
After some messing around I have found that if I remove the while loop then the code afterward executes properly, I have removed the while loop and placed it in the second 'search' method. However it still causes a freeze of the system. This makes no sense as there shouldn't be an infinite loop as if I manually query the Redis db, all 5 results are there. And the dd("two") can never be excecated unless the usleep() is removed. Hopefully this narrows the problem down.
Edit 2 please read:
I have figured out that I can get the dd("two") to work when usleep() is reduced to 0.05s from 0.5 seconds, but it still doesnt seem to run long enough for it to work.
if(!self::cacheAsyncApiSearch($searchQuery, $maxResults))
{
// make sure Redis is properly returning a number not object
$len = Redis::client()->lLen($key);
while(!is_numeric($len)){
usleep(500000); // 0.5 seconds
$len = Redis::client()->lLen($key);
}
//dd($len); //this dd() works
while ($len < 5) {
dd("one"); // this dd() works
usleep(500000); // 0.5 seconds
dd("two"); **//$this does not work, why?**
$len = Redis::client()->lLen($key);
}
}

Solution for calling a function doing lots of stuff in it by Cron?

function cronProcess() {
# > 100,000 users
$users = $this->UserModel->getUsers();
foreach ($users as $user) {
# Do lots of database Insert/Update/Delete, HTTP request stuff
}
}
The problem happens when the number of users reaches ~ 100,000.
I called the function by CURL via CronTab.
So what is the best solution for this?
I do a lot of bulk tasks in CakePHP, some processing millions of records. It's certainly possible to do, the key as others suggested is small batches in a loop.
If this is something you're calling from Cron, it's probably easier to use a Shell (< v3.5) or the newer Command class (v3.6+) than cURL.
Here's generally how I paginate large batches, including some helpful optional things like a progress bar, turning off hydration to speed things up slightly, and showing how many users/second the script was able to process:
<?php
namespace App\Command;
use Cake\Console\Arguments;
use Cake\Console\Command;
use Cake\Console\ConsoleIo;
class UsersCommand extends Command
{
public function execute(Arguments $args, ConsoleIo $io)
{
// I'd guess a Finder would be a more Cake-y way of getting users than a custom "getUsers" function:
// See https://book.cakephp.org/3.0/en/orm/retrieving-data-and-resultsets.html#custom-finder-methods
$usersQuery = $this->UserModel->find('users');
// Get a total so we know how many we're gonna have to process (optional)
$total = $usersQuery->count();
if ($total === 0) {
$this->abort("No users found, stopping..");
}
// Hydration takes extra processing time & memory, which can add up in bulk. Optionally if able, skip it & work with $user as an array not an object:
$usersQuery->enableHydration(false);
$this->info("Grabbing $total users for processing");
// Optionally show the progress so we can visually see how far we are in the process
$progress = $io->helper('Progress')->init([
'total' => 10
]);
// Tune this page value to a size that solves your problem:
$limit = 1000;
$offset = 0;
// Simply drawing the progress bar every loop can slow things down, optionally draw it only every n-loops,
// this sets it to 1/5th the page size:
$progressInterval = $limit / 5;
// Optionally track the rate so we can evaluate the speed of the process, helpful tuning limit and evaluating enableHydration effects
$startTime = microtime(true);
do {
$users = $usersQuery->offset($offset)->toArray();
$count = count($users);
$index = 0;
foreach ($users as $user) {
$progress->increment(1);
// Only draw occasionally, for speed
if ($index % $progressInterval === 0) {
$progress->draw();
}
### WORK TIME
# Do your lots of database Insert/Update/Delete, HTTP request stuff etc. here
###
}
$progress->draw();
$offset += $limit; // Increment your offset to the next page
} while ($count > 0);
$totalTime = microtime(true) - $startTime;
$this->out("\nProcessed an average " . ($total / $totalTime) . " Users/sec\n");
}
}
Checkout these sections in the CakePHP Docs:
Console Commands
Command Helpers
Using Finders & Disabling Hydration
Hope this helps!

Php Recursive Function Infinite loop Error

I am having a php recursive function to calculate nearest sale price. but i don't know why its run infinite time and throw error of maximum execution.
Its look like below:
function getamazonsaleper($portal)
{
$cp = floatval($this->input->post('cp')); //user provided inputs
$sp = floatval($this->input->post('sp')); //user provided input
$gst = floatval($this->input->post('gst')); //user provided input
$rfsp = floatval($this->input->post('rfsp')); //user provided input
$mcp = (int)($this->input->post('mcp')); //user provided input
$weight = floatval($this->input->post('weight')); //user provided input
$output = $this->getsalepercent($cp,$sp,$gst,$rfsp,$mcp,$weight,$portal);
return $output;
}
function getsalepercent($cp,$sp,$gst,$rfsp,$mcp,$weight,$portal) //recursive funtion
{
$spcost = ((($sp/100)*$cp));
$gstamount= (($spcost/(100+$gst))*$gst);
$rfspamount= ($spcost*($rfsp/100));
$mcpamount= ($cp*($mcp/100));
$fixedfee=$this->getfixedfee($portal,$spcost);
$weightfee=$this->getweightprice($portal,$weight);
$totalcost=$fixedfee+$weightfee+$rfspamount;
$gstinput=($totalcost*(18/100));
$remittances = $spcost-$totalcost-$gstinput;
$actualprofit= $remittances-$cp-$gstamount+$gstinput;
$actualprofitpercent = ($actualprofit/$cp)*100;
if( $actualprofitpercent >= $mcp)
{
return $sp;
}elseif($actualprofitpercent < $mcp)
{
$newsp = (int)($sp+10) ;
$this->getsalepercent($cp,$newsp,$gst,$rfsp,$mcp,$weight,$portal);
}
}
Can anybody tell me how can resolve this issue? Thanks in advance.
Edited :
Perameters
$cp=100;
$sp=200;
$mcp=20;
$weight=0.5;
$gst=28;
$rfsp=6.5;
First a couple of side notes:
- the way you use $gstinput it cancels itself out when you calculate $actualprofit (it's -$gstinput in $remittances which gets added to +$gstinput).
- $mcpamount seems to go completely unused in the code... I thought for a second you might vahe simply confused vars when doing the comparison, but of course for $cp = 100 it makes no difference.
Even so when I made a few calculations using the example values you gave for $sp = 200 (and growing by 10), I got:
Value of $actualprofit, which for $cp = 100 is also the value of $actualprofitpercent...
for $sp = 200:
43.25 - $fixedfee - $weightfee
for $sp = 210:
50.4125 - $fixedfee - $weightfee
for $sp = 220:
57.575 - $fixedfee - $weightfee
so for each $sp = $sp + 10 recursion the value of $actualprofitpercent (without taking into account $fixedfee and $weightfee) seems to grow by 7.1625.
The value of $weightfee should stay the same, but the value of $fixedfee depends on the value of $sp... Could it be that at each recursion getfixedfee() returns a value which grows faster than 7.1625?

PHP Laravel get millions record using MYSQL

I have been using laravel + mysql for my project but and it was working perfectly fine until now. The records keep on increasing and now they have reached almost a million record. Problem is when i try to fetch sms from my database using this query
$smsHistory = SmsLog::where('created_at', '>=', $startDate)->where('created_at', '<=', $endDate)->whereNotNull('gateway_statuscode')->get();
It gives a 500 error without showing anything in error log. What i assume is that as i decreased the time period it gives me record so the problem is the bulk it can not handle. What could be the possible solution as i have to give it today.
I am not worried about the error log.. i want to get a count of those million record but i want to apply some algorithm on it before doing it
This is the check which i have to perform afterwards to see how many sms are in sms log
foreach ($smsHistory as $sms) {
$sms_content = SmsService::_getSmsContent($sms);
if ($sms_content->business_id && array_key_exists($sms_content->business_id, $smsCredits )) {
if (floor(strlen($sms_content->content) / 160) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 1;
}
if (floor(strlen($sms_content->content) / 160) == 1 && floor(strlen($sms_content->content) / 306) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 2;
}
if (floor(strlen($sms_content->content) / 306) == 1 && floor(strlen($sms_content->content) / 459) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 3;
}
if (floor(strlen($sms_content->content) / 459) == 1 && floor(strlen($sms_content->content) / 621) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 4;
}
if (floor(strlen($sms_content->content) / 621) == 1 && floor(strlen($sms_content->content) / 774) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 5;
}
if (floor(strlen($sms_content->content) / 774) == 1 && floor(strlen($sms_content->content) / 927) == 0) {
$smsCredits[$sms_content->business_id]['count'] += 6;
}
}
this is the database field
content is the sms that i have to get and count
Regarding the 500 error:
If you are getting a 500 error, there should hopefully be some clue in the actual server error log (your laravel application error log may not have caught it depending on the error handlers, etc and what the cause was). php_info() should show you the location of the physical error log with the error_log setting:
<?php phpinfo(); ?>
If I had to guess, possibly something memory related that is causing it to choke. But that is just a guess.
As a side question, why are you trying to retrieve so many at once? Can you split them up somehow?
Edit based on updated question:
You may need to use some raw-expressions here to get what you really want: https://www.laravel.com/docs/4.2/queries#raw-expressions
MySQL for example provides the ability to get a column length:
FLOOR(CHAR_LENGTH(content) / 160) as len1, FLOOR(CHAR_LENGTH(content) / 306) as len2, FLOOR(CHAR_LENGTH(content) / 459) as len3
So probably with some magic we could take your query and let the database system do it all for you. And there are probably more efficient ways to do it, if I knew more about the significance of those numbers, but I am trying to help you at least get on one possible path.
You should setup index for mysql, or implement a search engine like elastic search

Calculating similarity -> python code to php code - Whats wrong?

I am trying to convert the following python code to a PHP code. Can you please explain me what is wrong in my PHP code, because I do not get the same results. If you need example data please let me know.
# Returns a distance-based similarity score for person1 and person2
def sim_distance(prefs,person1,person2):
# Get the list of shared_items
si={}
for item in prefs[person1]:
if item in prefs[person2]: si[item]=1
# if they have no ratings in common, return 0
if len(si)==0: return 0
# Add up the squares of all the differences
sum_of_squares=sum([pow(prefs[person1][item]-prefs[person2][item],2)
for item in prefs[person1] if item in prefs[person2]])
return 1/(1+sum_of_squares)
My PHP code:
$sum = 0.0;
foreach($arr[$person1] as $item => $val)
{
if(array_key_exists($item, $arr[$person2]))
{
$p = sqrt(pow($arr[$person1][$item] - $arr[$person2][$item], 2));
$sum = $sum + $p;
}
}
$sum = 1 / (1 + $sum);
echo $sum;
Thanks for helping!
The main difference is that you've added sqrt to the PHP code. The PHP also doesn't handle the special case of no prefs in common, which gives 0 in the python version and 1 in the PHP version.
I tested both versions and those are the only differences I found.
this is close as i could make a direct translation... (untested)
function sim_distance($prefs, $person1, $person2) {
$si = array();
foreach($prefs[$person1] as $item) {
if($item in $prefs[$person2]) $si[$item]=1;
}
if(count($si)==0) return 0;
$squares = array();
foreach($prefs[$person1] as $item) {
if(array_key_exists($item,$prefs[$person2])) {
$squares[] = pow($prefs[$person1][$item]-$prefs[$person2][$item],2);
}
}
$sum_of_squares = array_sum($squares);
return 1/(1+$sum_of_squares);
}
I don't really know what you're trying to do, or if I've interpreted the indentation correctly...but maybe this'll help. I'm assuming your data structures have the same layout as in the python script.
oh...and i'm interpreting the python as this:
def sim_distance(prefs,person1,person2):
# Get the list of shared_items
si={}
for item in prefs[person1]:
if item in prefs[person2]: si[item]=1
# if they have no ratings in common, return 0
if len(si)==0: return 0
# Add up the squares of all the differences
sum_of_squares=sum([pow(prefs[person1][item]-prefs[person2][item],2) for item in prefs[person1] if item in prefs[person2]])
return 1/(1+sum_of_squares)

Categories