I am using this library to work with fractions in PHP. This works fine but sometimes, I have to loop over a lot of values and this results in the following error:
Allowed memory size of 134217728 bytes exhausted
I can allocate more memory using PHP ini but that is a slippery slope. At some point, I am going to run out of memory when the loops are big enough.
Here is my current code:
for($q = 10; $q <= 20; $q++) {
for($r= 10; $r <= 20; $r++) {
for($p = 10; $p <= 20; $p++) {
for($s = 10; $s <= 20; $s++) {
for($x = 50; $x <= 100; $x++) {
for($y = 50; $y <= 100; $y++) {
$den = ($q + $r + 1000) - ($p + $s);
$num = $x + $y;
$c_diff = new Fraction($num, $den);
}
}
}
}
}
}
I used memory_get_peak_usage(true)/(1024*1024) to keep track of the memory the script is using. The total memory used was just 2MB until I added the line that creates a new fraction.
Could anyone please guide me on how to get rid of this error. I went through the code of the library posted on GitHub here but can't figure out how to get rid of the exhausted memory error. Is this because of the static keyword? I am beginner so I am not entirely sure what's going on.
The library code is about a 100 lines after removing the empty lines and comments. Any help would be highly appreciated.
UPDATE:
The script exhausts its memory even if I use just this block of code and nothing else. I definitely know that creating a new Fraction object is the cause of exhausting memory.
I thought that there was not need to unset() anything because the same one variable to store the new fractional value over and over again.
This leads me to think that whenever I creating a new Fraction object something else happens which in the library code that takes up memory which is not released on rewriting the value in the $c_diff variable.
I am not very good at this so I thought it has something to do with the static keyword used at a couple of places. Could anyone confirm it for me?
If this issue can indeed be resolved by using unset(), should I place it at the end of the loop?
Various Possible fixes and efficiencies:
You have 6 for loops, each loop cycles a single integer value within various ranges.
But your calculation only uses 3 values and so it doesn't matter if $p = 10; $s = 14; or $p = 13; $s = 11; These are entirely equivilant in the calculation.
All you need is the sum; so once you've found that the value 24 works; you can find all the parts (over the minimum value of 10) that fit that value: ie (24 (sum) - 10 (min) = 14), then collect the values within the range; so there are 10,14, 11,13 , 12,12, 13,11, 14,10 valid values. savng yourself 80%+ processing work on the inner for loops.
$pairs = "p,s<BR>"; //the set of paired values found
$other = $sum - $min;
if($other > $max){
$other = $sum - $max;
}
$hardMin = $min;
while ($other >= $hardMin && $min >= $hardMin && $min <= $max){
$pairs .= $min.", ".$other."<BR>";
$other--; // -1
$min++; // +1
}
print $pairs;
Giving:
p,s
10,14
11,13
12,12
13,11
14,10
So for this for loop already, you may only need to do ~10% of the total work cycling the inner loops.
Stop instantiating new classes. Creating a class is expensive. Instad you create one class and simply plug the values in:
Example:
$c_diff = new Fraction();
for(...){
for(...){
$c_diff->checkValuesOrWhateverMethod($num, $den)
}
}
This will save you significant overhead (depending on the structure of the class)
The code you linked on GitHub is simply to turn the value into a fraction and seems to be highly inefficient.
All you need is this:
function float2frac($n, $tolerance = 1.e-6) {
$h1=1; $h2=0;
$k1=0; $k2=1;
$b = 1/$n;
do {
$b = 1/$b;
$a = floor($b);
$aux = $h1; $h1 = $a*$h1+$h2; $h2 = $aux;
$aux = $k1; $k1 = $a*$k1+$k2; $k2 = $aux;
$b = $b-$a;
} while (abs($n-$h1/$k1) > $n*$tolerance);
return $h1."/".$k1;
}
Taken from this excellent answer.
Example:
for(...){
for(...){
$den = ($q + $r + 1000) - ($p + $s);
$num = $x + $y;
$value = $num/den;
$c_diff = float2frac($value);
unset($value,den,$num);
}
}
If you need more precision you can read this question and update PHP.ini as appropriate, but personally I would recommend you use more specialist maths languages such as Matlab or Haskell.
Putting it all together:
You want to check three values, and then find the equivilant part of each one.
You want to simply find the lowest common denominator fraction (I think).
So:
/***
* to generate a fraction with Lowest Common Denominator
***/
function float2frac($n, $tolerance = 1.e-6) {
$h1=1; $h2=0;
$k1=0; $k2=1;
$b = 1/$n;
do {
$b = 1/$b;
$a = floor($b);
$aux = $h1; $h1 = $a*$h1+$h2; $h2 = $aux;
$aux = $k1; $k1 = $a*$k1+$k2; $k2 = $aux;
$b = $b-$a;
} while (abs($n-$h1/$k1) > $n*$tolerance);
return $h1."/".$k1;
}
/***
* To find equivilants
***/
function find_equivs($sum = 1, $min = 1, $max = 2){
$value_A = $sum - $min;
$value_B = $min;
if($value_A > $max){
$value_B = $sum - $max;
$value_A = $max;
}
$output = "";
while ($value_A >= $min && $value_B <= $max){
if($value_A + $value_B == $sum){
$output .= $value_A . ", " . $value_B . "<BR>";
}
$value_A--; // -1
$value_B++; // +1
}
return $output;
}
/***
* Script...
***/
$c_diff = []; // an array of results.
for($qr = 20; $qr <= 40; $qr++) {
for($ps = 20; $ps <= 40; $ps++) {
for($xy = 100; $x <= 200; $xy++) {
$den = ($qr + 1000) - $ps;
$num = $xy;
$value = $num/$den; // decimalised
$c_diff[] = float2frac($num, $den);
/***
What is your criteria for success?
***/
if(success){
$qr_text = "Q,R<BR>";
$qr_text .= find_equivs($qr,10,20);
$sp_text = "S,P<BR>";
$sp_text .= find_equivs($sp,10,20);
$xy_text = "X,Y<BR>";
$xy_text .= find_equivs($sp,50,100);
}
}
}
}
This should do only a small percentage of the original looping.
I guess this isn't the entire block of code you are using.
This loop creates 50*50*10*10*10*10 = 25.000.000 Fraction objects. Consider using PHP's unset() to clean up memory, since you are allocating memory to create objects, but you never free it up.
editing for clarification
When you create anything in PHP, be it variable, array, object, etc. PHP allocates memory to store it and usually, the allocated memory is freed when script execution ends.
unset() is the way to tell PHP, "hey, I don't need this anymore. Can you, pretty please, free up the memory it takes?". PHP takes this into consideration and frees up the memory, when its garbage collector runs.
It is better to prevent memory exhaustion rather than feeding your script with more memory.
Allowed memory size of 134217728 bytes exhausted
134217728 bytes = 134.218 megabytes
Can you try this?
ini_set('memory_limit', '140M')
/* loop code below */
Related
I have a loop that generates all the possible combinations of bits by giving the number of bits desired, bu the issue is that I got out of memory when number of bits goes beyond 20, is there any optimizations that I can do, to solve this issue.
here my code :
function bitsGenerator($N)
{
$listN = $N;
$bits = ['0', '1'];
//check if input is valid or not
if (!is_int($listN)) {
echo 'Input must be numeric!';
}
if ($listN >= 1 && $listN <= 65) {
if ($listN == 1) {
echo '1';
exit;
}
for ($i = 1; $i <= ($listN - 1); $i++) {
$reverseBits = array_reverse($bits);
$prefixBit = preg_filter('/^/', '0', $bits);
$prefixReverseBits = preg_filter('/^/', '1', $reverseBits);
$bits = array_merge($prefixBit, $prefixReverseBits);
unset($prefixBit, $prefixReverseBits, $reverseBits);
}
$finalBits = array_slice($bits, -$listN);
foreach ($finalBits as $k => $v) {
echo $v . "\n";
}
} else {
echo 'Invalid input!';
}
}
The purpose of this function is to get the last $N combinations and display them all other combinations are thrown away, I'm looking for some kind of optimization to my code so that the $bits array will not store more than 65 item because the maximum number to bits thus the maximum number of combination to display is 65.
Thanks to every one for helping me.
The main problem is the size of the $bits array holding all your results. The array will contain 2^$N elements, each of $N length times 8 bits for each character (because you are using strings to have leading zeroes) so you'll end up with a memory consumption of approx (2^$N)*$N*8 which is 167772160 bytes. It won't get any smaller when using RAM.
Your working copies of $bits, preg_filter and array_merge will also consume a lot of RAM. Running your function with $N = 20 consumes 180375552 (172MiB).
BTW: unseting the variables will not reduce the consumed RAM because they would get overwritten in the next iteration anyway (or destroyed at the end of the function)
The following function was my first sketch based on your function and uses a bit less RAM: 171970560 bytes or 164MiB (vs 172MiB).
function myBitsGenerator($length)
{
if (!is_int($length)) {
die('Input must be numeric!');
}
if ($length < 1 || $length > 65) {
die('Input must be between 1 and 65');
}
$bitsArray = ['0', '1'];
$count = 2;
if ($length > 1) {
for ($i = 0; $i < $length; $i++) {
for ($j = 0; $j < $count; $j++) {
$bitsArray[] = $bitsArray[$j] . '1';
$bitsArray[$j] = $bitsArray[$j] . '0';
}
$count += $j;
}
}
$printArray = array_slice($bitsArray, $count - $length);
array_walk(
$printArray,
function ($value) {
echo $value . PHP_EOL;
}
);
}
However, the generation of those numbers is too complicated and should be simplified:
Theoretical: A binary number can be written in decimal and vice versa. The number of possible combinations of binary numbers with the fixed length of $N are x = 2 ^ $N. Each decimal number from 0 to x represents a binary number in the results.
Practical example: (binary) (0b101 === 5) (int)
All you have to do is to pad the calculated binary number with zeroes.
The simplified generator looks like:
$n = pow(2, $length);
for ($i = 0; $i < $iterations; $i++) {
$binary = str_pad(decbin($i), $length, '0', STR_PAD_LEFT);
}
You can use this generator to generate
If you really need to use even less RAM you should think about storing it in a file, which makes the whole think slower but it will use way less RAM.
function fileBitsGenerator($length)
{
$handle = fopen('bits.txt', 'w+');
$iterations = pow(2, $length);
for ($i = 0; $i < $iterations; $i++) {
fwrite($handle, str_pad(decbin($i), $length, '0', STR_PAD_LEFT) . PHP_EOL);
}
}
This consumes just 2097152 bytes and scales!
But be aware that the performance will depend on your HDD/SSD speed and it executes a lot of write operations (which might shorten your SSD life span). For example: the resulting file bits.txt is 92MB big if length = 22
I have an array of numbers shown below. However, I can only calculate the standard deviation of the whole array with only output 1 result as shown below:
Codes:
function standard_deviation_sample ($a)
{
//variable and initializations
$the_standard_deviation = 0.0;
$the_variance = 0.0;
$the_mean = 0.0;
$the_array_sum = array_sum($a); //sum the elements
$number_elements = count($a); //count the number of elements
//calculate the mean
$the_mean = $the_array_sum / $number_elements;
//calculate the variance
for ($i = 0; $i < $number_elements; $i++)
{
//sum the array
$the_variance = $the_variance + ($a[$i] - $the_mean) * ($a[$i] - $the_mean);
}
$the_variance = $the_variance / ($number_elements - 1.0);
//calculate the standard deviation
$the_standard_deviation = pow( $the_variance, 0.5);
//return the variance
return $the_standard_deviation;
}
$a = array(1,2,3,4,9,6,7,8,9,20,11,12,13,14,2,16,17,18,19,27);
$standard_deviation = standard_deviation_sample ($a);
echo "standard_deviation = $standard_deviation<br>";
Results:
standard_deviation = 7.10004
Does anyone know how to compute standard deviation for every 5 numbers instead? So that the output will be:
standard_deviation_1 = 3.11448
standard_deviation_2 = 5.70088
standard_deviation_3 = 4.82701
standard_deviation_4 = 4.39318
You're looking for array_chunk(); It'll split your array up into smaller arrays of a given size, so your code would now be:
$a = array_chunk(array(1,2,3,4,9,6,7,8,9,20,11,12,13,14,2,16,17,18,19,27), 5);
foreach($a as $b){
echo "standard_deviation: " . standard_deviation_sample($b);
}
Reference: http://uk1.php.net/array_chunk
If you require the number at the end of your standard_deviation_1 output, you could change the loop to a for() loop like so:
for($i = 0; $i < count($a); $i++){
echo "standard_deviation_" . ($i + 1) . " " . standard_deviation_sample($a[$i]);
}
The answer is to use array_chunk(), but there's a few more things you can improve on the code itself:
function standard_deviation_sample(array $a)
{
$the_mean = array_sum($a) / count($a);
return sqrt(array_reduce($a, function($result, $item) use ($the_mean) {
return $result + pow($item - $the_mean, 2);
}, 0) / (count($a) - 1));
}
The function itself can be greatly reduced by only using variables where they're needed; secondly, calculating the squared standard deviation is a great example of something you can solve using array_reduce(); you start with the initial value of 0 and the inner function keeps adding the squared differences together.
$a = array(1,2,3,4,9,6,7,8,9,20,11,12,13,14,2,16,17,18,19,27);
foreach (array_chunk($a, 5) as $k => $sample) {
printf("standard_deviation_%d = %.5f<br>\n",
$k + 1,
standard_deviation_sample($sample));
}
Is there a way to perform sorting on integers or strings in an instance of the SplFixedArray class? Is converting to a PHP's array, sorting, and then converting back being the only option?
Firstly, congratulations on finding and using SplFixedArrays! I think they're a highly under-utilised feature in vanilla PHP ...
As you've probably appreciated, their performance is unrivalled (compared to the usual PHP arrays) - but this does come at some trade-offs, including a lack of PHP functions to sort them (which is a shame)!
Implementing your own bubble-sort is a relatively easy and efficient solution. Just iterate through, looking at each consecutive pairs of elements, putting the highest on the right. Rinse and repeat until the array is sorted:
<?php
$arr = new SplFixedArray(10);
$arr[0] = 2345;
$arr[1] = 314;
$arr[2] = 3666;
$arr[3] = 93;
$arr[4] = 7542;
$arr[5] = 4253;
$arr[6] = 2343;
$arr[7] = 32;
$arr[8] = 6324;
$arr[9] = 1;
$moved = 0;
while ($moved < sizeof($arr) - 1) {
$i = 0;
while ($i < sizeof($arr) - 1 - $moved) {
if ($arr[$i] > $arr[$i + 1]) {
$tmp = $arr[$i + 1];
$arr[$i + 1] = $arr[$i];
$arr[$i] = $tmp;
}
$i++;
var_dump ($arr);
}
$moved++;
}
It's not fast, it's not efficient. For that you might consider Quicksort - there's documented examples online including this one at wikibooks.org (will need modification of to work with SplFixedArrays).
Seriously, beyond getting your question answered, I truly feel that forcing yourself to ask why things like SplFixedArray exist and forcing yourself to understand what goes on behind a "quick call to array_sort()" (and why it quickly takes a very long time to run) make the difference between programmers and programmers. I applaud your question!
Here's my adaptation of bubble sort using splFixedArrays. In PHP 7 this simple program is twice as fast as the regular bubblesort
function bubbleSort(SplFixedArray $a)
{
$len = $a->getSize() - 1;
$sorted = false;
while (!$sorted) {
$sorted = true;
for ($i = 0; $i < $len; $i++)
{
$current = $a->offsetGet($i);
$next = $a->offsetGet($i + 1);
if ( $next < $current ) {
$a->offsetSet($i, $next);
$a->offsetSet($i + 1, $current);
$sorted = false;
}
}
}
return $a
}
$starttime = microtime(true);
$array = SplFixedArray::fromArray([3,4,1,3,5,1,92,2,4124,424,52,12]);
$array = bubbleSort($array);
print_r($array->toArray());
echo (microtime(true) - $starttime) * 1000, PHP_EOL;
I'm trying to make something like this to my variable data value...
$maxvalue = 0;
$basevalue = 0;
if($basevalue == 0) {$maxvalue = 0;}
else if ($basevalue == 1) {$maxvalue = 884;}
else if ($basevalue == 2) {$maxvalue = 1819;}
else if ($basevalue == 3) {$maxvalue = 2839;}
and so on.. i believe there is no exact computation on how the $maxvalue shifts as the basevalue increase. Can someone suggest me a simplier way to do this? thanks in advance!
$maxvalues = array(0, 884, 1819, 2839, ...);
$maxvalue = $maxvalues[$basevalue];
It looks like there's a pattern, almost like a faculty, but also with some other calculations. All numbers are multiples of 17. The following function returns the numbers you provided, so I think it might work for the higher numbers too:
function getMaxValue($base)
{
// Factor of $base = 51 + $base^2 + Factor($base - 1). You
// could solve that in a recursion, but a loop is generally better.
$factor = 0;
for ($i = 1; $i <= $base; $i++)
$factor += 51 + ($i * $i);
return $factor * 17;
}
// Test
for ($i = 0; $i < 100; $i++)
{
echo "$i -- " . getMaxValue($i) . "<br>\n";
}
Here's the solution that prevented me putting all of them in an array..
$maxvalue = 17/6*(2*($basevalue*$basevalue*$basevalue)+3
($basevalue*$basevalue)+307*$basevalue);
Thanks for all the help
So, for my Cryptography Library, I have a base converter that I use quite often. It's not the most efficient thing in the world, but it works quite well for all ranges of input.
The bulk of the work is done by the callback loop:
$callback = function($source, $src, $dst) {
$div = array();
$remainder = 0;
foreach ($source as $n) {
$e = floor(($n + $remainder * $src) / $dst);
$remainder = ($n + $remainder * $src) % $dst;
if ($div || $e) {
$div[] = $e;
}
}
return array(
$div,
$remainder
);
};
while ($source) {
list ($source, $remainder) = $callback($source, $srcBase, $dstBase);
$result[] = $remainder;
}
Basically, it takes an array of numbers in $srcBase and converts them to an array of numbers in $dstBase. So, an example input would be array(1, 1), 2, 10 which would give array(3) as a result. Another example would be array(1, 0, 0), 256, 10 which would give array(1, 6, 7, 7, 7, 2, 1, 6) (each element of the array is a single "digit" in the $dstBase.
The problem I'm now facing, is if I feed it 2kb of data, it takes almost 10 seconds to run. So I've set out to optimize it. So far, I have it down to about 4 seconds by replacing that whole structure with this recursive loop:
while ($source) {
$div = array();
$remainder = 0;
foreach ($source as $n) {
$dividend = $n + $remainder * $srcBase;
$res = (int) ($dividend / $dstBase);
$remainder = $dividend % $dstBase;
if ($div || $res) {
$div[] = $res;
}
}
$result[] = $remainder;
$source = $div;
}
The problem I'm facing, is how to optimize it further (if that's even possible). I think the problem is the shear number of iterations it takes for the large input (for a 2000 element array, from base 256 to base 10, it takes 4,815,076 iterations in total).
Any thoughts?
99.9% of the time taken to execute this script originates from the inherent need to iterate through an input. Since the code inside the foreach is very basic, the only way of decreasing execution time is to reduce the number of iterations. If that is not possible, then you have the most efficient version of this function.
Yes, it can be optimized a little:
$source_count = count($source);
while ($source) {
$remainder = $i = 0;
foreach ($source AS &$n) {
$dividend = $n + $remainder * $srcBase;
$remainder = $dividend % $dstBase;
$res = ($dividend - $remainder) / $dstBase;
if ($i || $res)
$source[$i++] = $res;
}
for ($j=$i; $j < $source_count; $j++)
unset($source[$i]);
$source_count=$i;
$result[] = $remainder;
}
or even faster, but more obscure:
$source_count = count($source);
while ($source) {
$remainder = $i = 0;
foreach ($source AS &$n) {
if (($res = ($dividend - ($remainder = ($dividend = $n + $remainder * $srcBase) % $dstBase)) / $dstBase) || $i)
$source[$i++] = $res;
}
for ($j=$i; $j < $source_count; $j++)
unset($source[$i]);
$source_count=$i;
$result[] = $remainder;
}
You will get some memory and CPU usage reduction and it is much more fun but of cource unreadable (:.
But personally I think you are doing it the wrong way. I think you should use some fast C code for this kind of task(by using system call or writing/installing existing PHP module). And I think that code optimizers/compilers like Hip-Hop PHP,Zend Optimized etc. can dramatically increase performance in this case.
I'm not sure, but
$dividend = $remainder * $srcBase + $n;
could be a little bit faster...