I'm building a system where I need to assign users access to a specific (individual) number of assets. These assets (potentially numbering in the tens of thousands).
I thought to do it with bitwise comparisons, so something like storing the value of 3 if a user has access to assets 1 and 2, a value of 7 for access to 1, 2 and 3, etc. etc.
The access is not necessarily sequential, so a user could easily have access to assets 10, 12 and 24324.
I quickly ran into a problem using bits where the server wouldn't pick up assets beyond the 63rd bit, so obviously I've either misunderstood something, or bits is a dumb way to store this kind of info.
My code, running on a 64-bit Linux system, is this (just for testing purposes obviously, to discover just such limitations as this):
<?php
$bitwise = $_GET['bitwise'];
if (isset($bitwise)) {
echo "<br/>bitwise input: ";
echo $bitwise;
$bitcount = 0;
for ($i=1;$i<=$bitwise;$i*=2) {
if (($i & $bitwise) > 0) {
$bitcount++;
echo "<br/>{$bitcount}: " . $i . " is in " . $bitwise;
}
}
}
?>
And I input test values via the querystring. However, no matter what value I input, the maximum count I can get to is 63.
So, my question is: Is this simply because I'm using bitwise comparisons for something they're not ideal for (my theory), or is my implementation of it just wrong?
My next go-to solution would be to store the "bits" in arrays, so if someone has access to assets 1, 2 and 3 I'll store their list as [1,2,3]. It's unlikely that someone has access to more than, say, a hundred specific assets. Is this a reasonable way to do it? I realize this puts the question somewhat into discussion-worthy territory, but hopefully it's still specific enough.
Paramount concerns are, of course, performance if the server has to serve a large number of clients at the same time.
(please excuse wrong terminology where applicable, hopefully my meaning is clear).
This is standard behavior — on 64 bit compiled PHP, integers have a maximum length of 64 bits. While it warms my secret grey beard heart, if you have more than 64 different roles, a bitwise solution is the wrong one for access control.
Two other things worth mentioning.
First, doing this for performance reasons is probably a premature optimization for a web application. ACL lookups aren't going to be the bottleneck in your system for a long time, if at all. Also, it's not clear if bitwise operators offer that much PHP performance benefit, given the language's dynamically typed nature.
Second, the reason you're limited to 63 bits is because PHP (appears to?) use Two's compliment for their implementation of signed integers. The final bit is for representing positive or negative numbers. I asked this question about the bitwise NOT a while back, which is why this question caught my eye.
Related
Further to my question here, I'll be using the random_compat polyfill (which uses /dev/urandom) to generate random numbers in the 1 to 10,000,000 range.
I do realise, that all things being correct with how I code my project, the above tools should produce good (as in random/secure etc) data. However, I'd like to add extra sources of randomness into the mix - just in case 6 months down the line I read there is patch available for my specific OS version to fix a major bug in /dev/urandom (or any other issue).
So, I was thinking I can get numbers from random.org and fourmilab.ch/hotbits
An alternative source would be some logs from a web site I operate - timed to the microsecond, if I ignore the date/time part and just take the microseconds - this has in effect been generated by when humans decide to click on a link. I know this may be classed as haphazard rather than random, but would it be good for my use?
Edit re timestamp logs - will use PHP microtime() which will creaet a log like:
0.**832742**00 1438282477
0.**57241**000 1438282483
0.**437752**00 1438282538
0.**622097**00 1438282572
I will just use the bolded portion.
So let's say I take two sources of extra random numbers, A and B, and the output of /dev/urandom, call that U and set ranges as follows:
A and B are 1 - 500,000
U is 1 - 9,000,000
Final random number is A+B+U
I will be needing several million final numbers between 1 and 10,000,000
But the pool of A and B numbers will only contain a few thousand, but I think by using prime number amounts I can stretch that into millions of A&B combinations like so
// this pool will be integers from two sources and contain a larger prime number
// of members instead of the 7 & 11 here - this sequence repeats at 77
$numbers = array("One","Two","Three","Four","Five","Six","Seven");
$colors = array("Silver","Gray","Black","Red","Maroon","Yellow","Olive","Lime","Green","Aqua","Orange");
$ni=0;
$ci=0;
for ($i=0;$i<$num_numbers_required;$i++)
{
$offset = $numbers[$ni] + $colors[$ci];
if ($ni==6) // reset at prime num 7
$ni=0;
else
$ni++;
if ($ci==10) // reset at prime num 11
$ci=0;
else
$ci++;
}
Does this plan make sense - is there any possibility I can actually make my end result less secure by doing all this? And what of my idea to use timestamp data?
Thanks in advance.
I would suggest reading RFC4086, section 5. Basically it talks about how to "mix" different entropy sources without compromising security or bias.
In short, you need a "mixing function". You can do this with xor, where you simply set the result to the xor of the inputs: result = A xor B.
The problem with xor is that if the numbers are correlated in any way, it can introduce strong bias into the result. For example, if bits 1-4 of A and B are the current timestamp, then the result's first 4 bits will always be 0.
Instead, you can use a stronger mixing function based on a cryptographic hash function. So instead of A xor B you can do HMAC-SHA256(A, B). This is slower, but also prevents any correlation from biasing the result.
This is the strategy that I used in RandomLib. I did this because not every system has every method of generation. So I pull as many methods as I can, and mix them strongly. That way the result is never weaker than the strongest method.
HOWEVER, I would ask why. If /dev/urandom is available, you're not going to get better than it. The reason is simple, even if you call random.org for more entropy, your call is encrypted using random keys generated from /dev/urandom. Meaning if an attacker can compromise /dev/urandom, your server is toast and you will be spinning your wheels trying to make it better.
Instead, simply use /dev/urandom and keep your OS updated...
I'm making a little web application which needs to randomize stuff.
Just a little example of what's it gonna have: returns a random number between and 10 to the user.
I'm planning to do it using Javascript and jQuery (the calculator itself).
My question is: How can I make its functions truly random and not pseudo-random? Is the PHP function perhaps more random than the Javascript ones?
For the sake of the question, let's say what I want is a true random number between X and Y.
No function that you call will be "truly random". They are all PRNGs; most PRNGs are, in fact, quite good and I'd be surprised if they were inadequate for your application. Also, while a few PRNGs are notorious for their small period (Java's Random comes to mind), every modern language that I know of—including JavaScript, PHP, and (with its crypto packages) Java—have very good PRNGs.
The best way to collect "more random" data is to obtain it from an external random source. One possibility is to ask the user to move the mouse around in a window for a certain period of time, collecting the mouse coordinates at regular intervals. Some military, banking, and other high-security systems use hardware like thermal noise sensors to obtain data that is as close to random as one can hope; however, such hardware support is not going to be available to a web app.
Note that hacks like using the system clock are no better than PRNGs; most PRNGs initialize their seed with a combination of such data.
You have not understood the Matrix movies. ;) One function is not "more random" than one in another language. All functions are equally pseudo random. "Pseudo" means that the computer cannot, by definition, pull a random number out of thin air. It just can't. A computer computes, strictly, based on rules, accurately. There's no randomness anywhere in the system. For all its supposed power, randomness is the one thing a computer simply cannot do (that, and making coffee).
For true randomness, you need an outside, natural source. Like measuring atomic decay, or some such thing which is not predictable and truly random. Anything else is just pseudo-randomness, which may be of varying quality.
Good PRNGs try to collect "outside interference" in an entropy pool; e.g. Linux' /dev/random takes into account system driver "noise" which may be based on "random" packets hitting the ethernet port, or the user's mouse movements. How truly random that is is debatable, but it's very very hard to predict at least and suitably random for most purposes.
I don't think there's any way to remove the deterministic aspect of randomness from a program completely. You can do all you want to minimize, mitigate, and obfuscate whatever process you're using to "pull numbers from a hat", but you can never truly make it perfectly random.
You can fashion out a process with sufficient detail to make it practically random, but true random may be impossible.
While you can't achive true random with your code in php you can use random.org API. You could connect through curl in php or through ajax in javascript.
They are using atmospheric noise as a random seed as far as i know.
It is not possible to generate truly random variables on a computer. However, you may improve standard generators. Suppose, you have two basic generators. You create a table and fill it with values from the first generator. Then, if you want to get a number, the second one generates an index and returns correspondent value from the table. This value is then replaced with the new one... I forgot how this generator is called... Hope, it helps.
P.S. Sorry for my English.
My suggestion is that you generate a binary random string by encrypting the local time and date by an encryption algorithm. In this case try to gather all possible sources of "random" data, and load these both as input message and input key.
As you have seen from previous answers, above, your use and your requirements of your random data are important. A number is "random" if it is difficult or impossible for your application to guess its value in advance. Note that the same number source may be considered random for some applications while they are not random in others. Evidently you will have serious problems in case you need high quality random numbers for a demanding application.
TRNG98 True Random Numbers
I have just taken some online tutorials on how Bits work. Though I have a couple of questions. I have searched the internet but didn't find what i was looking for. I may have been searching for some incorrect key words.
Lets say i wish to build an option or permission system using bitwise (i think that is the correct terminology). I have the following hestitations:
1) Is it possible to end up with collisions when using & etc?
2) If there is collision opportunities what sort of steps should i take when im designing my permissions? Does the permission numbers change if i have a considerably large set of permissions e.g. over 500?
Hopefully i got my question across correctly, if not please let me know and i will try to rephrase.
EDIT::
Similar Question here which i believe has been answered
User role permissions for different modules using bitwise operators
What kind of collisions?
For 500 different permissions, you'd have to store 500 bits. There's no computers in existence which can directly handle a 500+bit value. With bit systems, you're basically restricted to using what the underlying cpu can provide, e.g. 8, 16, 32, 64-bit sized values. Anything beyond that has to be split into multiple different memory chunks.
e.g.
permission_flags && (1 << 475)
is going to fail on every platform in existence, because once you shift past bit #63, you're past what the cpu can directly support.
In nearly any programming language, if I do $number = rand(1,100) then I have created a flat probability, in which each number has a 1% chance of coming up.
What if I'm trying to abstract something weird, like launching rockets into space, so I want a curved (or angled) probability chart. But I don't want a "stepped" chart. (important: I'm not a math nerd, so there are probably terms or concepts that I'm completely skipping or ignorant of!) An angled chart is fine though.
So, if I wanted a probability that gave results of 1 through 100... 1 would be the most common result. 2 the next most common. In a straight line until a certain point - lets say 50, then the chart angles, and the probability of rolling 51 is less than that of rolling 49. Then it angles again at 75, so the probability of getting a result above 75 is not simply 25%, but instead is some incredibly smaller number, depending on the chart, perhaps only 10% or 5% or so.
Does this question make any sense? I'd specifically like to see how this can be done in PHP, but I wager the required logic will be rather portable.
The short answers to your questions are, yes this makes sense, and yes it is possible.
The technical term for what you're talking about is a probability density function. Intuitively, it's just what it sounds like: It is a function that tells you, if you draw random samples, how densely those samples will cluster (and what those clusters look like.) What you identify as a "flat" function is also called a uniform density; another very common one often built into standard libraries is a "normal" or Gaussian distribution. You've seen it, it's also called a bell curve distribution.
But subject to some limitations, you can have any distribution you like, and it's relatively straightforward to build one from the other.
That's the good news. The bad news is that it's math nerd territory. The ideas behind probability density functions are pretty intuitive and easy to understand, but the full power of working with them is only unlocked with a little bit of calculus. For instance, one of the limitations on your function is that the total probability has to be unity, which is the same as saying that the area under your curve needs to be exactly one. In the exact case you describe, the function is all straight lines, so you don't strictly need calculus to help you with that constraint... but in the general case, you really do.
Two good terms to look for are "Transformation methods" (there are several) and "rejection sampling." The basic idea behind rejection sampling is that you have a function you can use (in this case, your uniform distribution) and a function you want. You use the uniform distribution to make a bunch of points (x,y), and then use your desired function as a test vs the y coordinate to accept or reject the x coordinates.
That makes almost no sense without pictures, though, and unfortunately, all the best ways to talk about this are calculus based. The link below has a pretty good description and pretty good illustrations.
http://www.stats.bris.ac.uk/~manpw/teaching/folien2.pdf
Essentially you need only to pick a random number and then feed into a function, probably exponential, to pick the number.
Figuring out how weighted you want the results to be will make the formula you use different.
Assuming PHP has a random double function, I'm going to call it random.
$num = 100 * pow(random(), 2);
This will cause the random number to multiply by itself twice, and since it returns a number between 0 and 1, it will get smaller, thus increasing the chance to be a lower number. To get the exact ratio you'd just have to play with this format.
To me it seems like you need a logarithmic function (which is curved). You'd still pull a random number, but the value that you'd get would be closer to 1 than 100 most of the time. So I guess this could work:
function random_value($min=0, $max=100) {
return log(rand($min, $max), 10) * 10;
}
However you may want to look into it yourself to make sure.
The easiest way to achieve a curved probability is to think how you want to distribute for example a prize in a game across many winners and loosers. To simplify your example I take 16 players and 4 prizes. Then I make an array with a symbol of the prize (1,2,2,3,3,3,3,3,4,4,4,4,4,4,4) and pick randomly a number out of this array. Mathematically you would have a probability for prize 1 = 1:16, for prize 2 3:16, for prize 3 5:16 and for prize 4 7:16.
I am a PHP developer and I have always thought that micro-optimizations are not worth the time. If you really need that extra performance, you would either write your software so that it's architecturally faster, or you write a C++ extension to handle slow tasks (or better yet, compile the code using HipHop). However, today a work mate told me that there is a big difference in
is_array($array)
and
$array === (array) $array
and I was like "eh, that's a pointless comparison really", but he wouldn't agree with me.. and he is the best developer in our company and is taking charge of a website that does about 50 million SQL queries per day -- for instance. So, I am wondering here: could he be wrong or is micro-optimization really worth the time and when?
Well, for a trivially small array, $array === (array) $array is significantly faster than is_array($array). On the order of over 7 times faster. But each call is only on the order of 1.0 x 10 ^ -6 seconds (0.000001 seconds). So unless you're calling it literally thousands of times, it's not going to be worth it. And if you are calling it thousands of times, I'd suggest you're doing something wrong...
The difference comes when you're dealing with a large array. Since $array === (array) $array requires a new variable to be copied requires the array to be iterated over internally for the comparison, it'll likely be SIGNIFICANTLY slower for a large array. For example, on an array with 100 integer elements, is_array($array) is within a margin of error (< 2%) of is_array() with a small array (coming in at 0.0909 seconds for 10,000 iterations). But $array = (array) $array is extremely slow. For only 100 elements, it's already over twice as slow as is_array() (coming in at 0.203 seconds). For 1000 elements, is_array stayed the same, yet the cast comparison increased to 2.0699 seconds...
The reason it's faster for small arrays is that is_array() has the overhead of being a function call, where the cast operation is a simple language construct... And iterating over a small variable (in C code) will typically be cheaper than the function call overhead. But, for larger variables, the difference grows...
It's a tradeoff. If the array is small enough, iteration will be more efficient. But as the size of the array grows, it will becomes increasingly slower (and hence the function call will become faster).
Another Way To Look At It
Another way to look at it would be to examine the algorithmic complexity of each cast.
Let's take a look at is_array() first. It's source code basically shows it's an O(1) operation. Meaning it's a constant time operation. But we also need to look at the function call. In PHP, function calls with a single array parameter are either O(1) or O(n) depending on if copy-on-write needs to be triggered. If you call is_array($array) when $array is a variable reference, copy-on-write will be triggered and a full copy of the variable will occur.
So therefore is_array() is a best case O(1) and worst-case O(n). But as long as you're not using references, it's always O(1)...
The cast version, on the other hand, does two operations. It does a cast, then it does an equality check. So let's look at each separately. The cast operator handler first forces a copy of the input variable. No matter if it's a reference or not. So simply using the (array) casting operator forces an O(n) iteration over the array to cast it (via the copy_ctor call).
Then, it converts the new copy to an array. This is O(1) for arrays and primitives, but O(n) for objects.
Then, the identical operator executes. The handler is just a proxy to the is_identical_function(). Now, is_identical will short-circuit if $array is not an array. Therefore, it has a best case of O(1). But if $array is an array, it can short-circuit again if the hash tables are identical (meaning both variables are copy-on-write copies of each other). So that case is O(1) as well. But remember that we forced a copy above, so we can't do that if it's an array. So it's O(n) thanks to zend_hash_compare...
So the end result is this table of worst-case runtime:
+----------+-------+-----------+-----------+---------------+
| | array | array+ref | non-array | non-array+ref |
+----------+-------+-----------+-----------+---------------+
| is_array | O(1) | O(n) | O(1) | O(n) |
+----------+-------+-----------+-----------+---------------+
| (array) | O(n) | O(n) | O(n) | O(n) |
+----------+-------+-----------+-----------+---------------+
Note that it looks like they scale the same for references. They don't. They both scale linearly for referenced variables. But the constant factor changes. For example, in a referenced array of size 5, is_array will perform 5 memory allocations, and 5 memory copies, followed by 1 type check. The cast version, on the other hand, will perform 5 memory allocations, 5 memory copies, followed by 2 type checks, followed by 5 type checks and 5 equality checks (memcmp() or the like). So n=5 yields 11 ops for is_array, yet 22 ops for ===(array)...
Now, is_array() does have the O(1) overhead of a stack push (due to the function call), but that'll only dominate runtime for extremely small values of n (we saw in the benchmark above just 10 array elements was enough to completely eliminate all difference).
The Bottom Line
I'd suggest going for readability though. I find is_array($array) to be far more readable than $array === (array) $array. So you get the best of both worlds.
The script I used for the benchmark:
$elements = 1000;
$iterations = 10000;
$array = array();
for ($i = 0; $i < $elements; $i++) $array[] = $i;
$s = microtime(true);
for ($i = 0; $i < $iterations; $i++) is_array($array);
$e = microtime(true);
echo "is_array completed in " . ($e - $s) ." Seconds\n";
$s = microtime(true);
for ($i = 0; $i < $iterations; $i++) $array === (array) $array;
$e = microtime(true);
echo "Cast completed in " . ($e - $s) ." Seconds\n";
Edit: For the record, these results were with 5.3.2 on Linux...
Edit2: Fixed the reason the array is slower (it's due to the iterated comparison instead of memory reasons). See compare_function for the iteration code...
Micro-optimisation is worth it when you have evidence that you're optimising a bottleneck.
Usually it's not worth it - write the most readable code you can, and use realistic benchmarks to check the performance. If and when you find you've got a bottleneck, micro-optimise just that bit of code (measuring as you go). Sometimes a small amount of micro-optimisation can make a huge difference.
But don't micro-optimise all your code... it will end up being far harder to maintain, and you'll quite possibly find you've either missed the real bottleneck, or that your micro-optimisations are harming performance instead of helping.
Is micro-optimization worth the time?
No, unless it is.
In other words, a-priori, the answer is "no", but after you know a specific line of code consumes a healthy percent of clock time, then and only then is it worth optimizing.
In other words, profile first, because otherwise you don't have that knowledge. This is the method I rely on, regardless of language or OS.
Added: When many programmers discuss performance, from experts on down, they tend to talk about "where" the program spends its time. There is a sneaky ambiguity in that "where" that leads them away from the things that could save the most time, namely, function call sites. After all, the "call Main" at the top of an app is a "place" that the program is almost never "at", but is responsible for 100% of the time. Now you're not going to get rid of "call Main", but there are nearly always other calls that you can get rid of.
While the program is opening or closing a file, or formatting some data into a line of text, or waiting for a socket connection, or "new"-ing a chunk of memory, or passing a notification throughout a large data structure, it is spending great amounts of time in calls to functions, but is that "where" it is? Anyway, those calls are quickly found with stack samples.
As the cliche goes, micro-optimization is generally worth the time only in the smallest, most performance-critical hotspots of your code, only after you've proven that's where the bottleneck is. However, I'd like to flesh this out a little, to point out some exceptions and areas of misunderstanding.
This doesn't mean that performance should not be considered at all upfront. I define micro-optimization as optimizations based on low-level details of the compiler/interpreter, the hardware, etc. By definition, a micro-optimization does not affect big-O complexity. Macro-optimizations should be considered upfront, especially when they have a major impact on high-level design. For example, it's pretty safe to say that if you have a large, frequently accessed data structure, an O(N) linear search isn't going to cut it. Even things that are only constant terms but have a large and obvious overhead might be worth considering upfront. Two big examples are excessive memory allocation/data copying and computing the same thing twice when you could be computing it once and storing/reusing the result.
If you're doing something that's been done before in a slightly different context, there may be some bottlenecks that are so well-known that it's reasonable to consider them upfront. For example, I was recently working on an implementation of the FFT (fast Fourier Transform) algorithm for the D standard library. Since so many FFTs have been written in other languages before, it's very well-known that the biggest bottleneck is cache performance, so I went into the project immediately thinking about how to optimize this.
In general you should not write any optimisation which makes your code more ugly or harder to understand; in my book this definitely falls into this category.
It is much harder to go back and change old code than write new code, because you have to do regression testing. So in general, no code already in production should be changed for frivolous reasons.
PHP is such an incredibly inefficient language that if you have performance problems, you should probably look to refactor hot spots so they execute less PHP code anyway.
So I'd say in general no, and in this case no, and in cases where you absolutely need it AND have measured that it makes a provable difference AND is the quickest win (low-hanging fruit), yes.
Certainly scattering micro-optimisations like this throughout your existing, working, tested code is a terrible thing to do, it will definitely introduce regressions and almost certainly make no noticable difference.
Well, I'm going to assume that is_array($array) is the preferred way, and $array === (array) $array is the allegedly faster way (which bring up the question why isn't is_array implemented using that comparison, but I digress).
I will hardly ever go back into my code and insert a micro-optimization*, but I will often put them into the code as I write it, provided:
it doesn't slow my typing down.
the intent of the code is still clear.
That particular optimization fails on both counts.
* OK, actually I do, but that has more to do with me having a touch of OCD rather than good development practices.
We had one place where the optimisation was really helpful.
Here some comparison:
is_array($v) : 10 sec
$v === (array)$v : 3,3 sec
($v.'') === 'Array' : 2,6 sec
The last one does cast to string, an Array is always casted to a string with value 'Array'. This check will be wrong, if the $v is a string with value 'Array' (never happens in our case).
Well, there's more things than speed to take into consideration. When you read that 'faster' alternative, do you instantly think "Oh, this is checking to see if the variable is an array", or do you think "...wtf"?
Because really - when considering this method, how often is it called? What is the exact speed benefit? Does this stack up when the array is larger or smaller? One cannot do optimizations without benchmarks.
Also, one shouldn't do optimizations if they reduce the readability of the code. In fact, reducing that amount of queries by a few hundred thousand (and this is often easier than one would think) or optimizing them if applicable would be much, much more beneficial to performance than this micro-optimization.
Also, don't be intimidated by the guy's experience, as others have said, and think for yourself.
Micro-optimization is not worth it. Code readability is much more important than micro-optimization.
Great article about useless micro-optimization by Fabien Potencier (creator of the Symfony framework):
print vs echo, which one is faster?
Print uses one more opcode because it actually returns something. We
can conclude that echo is faster than print. But one opcode costs
nothing, really nothing. Even if a script have hundreds of calls to
print. I have tried on a fresh WordPress installation. The script
halts before it ends with a "Bus Error" on my laptop, but the number
of opcodes was already at more than 2.3 millions. Enough said.
IMHO, micro-optimizations are actually even more relevant than algorithmic optimizations today if you are working in a performance-critical field. This might be a big if because many people don't actually work in performance-critical areas even for performance-critical software since they might just be making high-level calls into a third party library which does the actual performance-critical work. For example, many people these days trying to write an image or video software might write non-performance-critical code expressing they want at the image level, not having to manually loop through several million pixels themselves at 100+ frames per second. The library does that for them.
When I say that micro-optimizations are more relevant than algorithmic ones today, I don't mean that, say, parallelized SIMD code that minimizes cache misses applying a bubble sort will beat an introsort or radix sort. What I mean is that professionals don't bubble sort large input sizes.
If you take any reasonably high-level language today, of which I include C++, you already have your share of reasonably efficient general-purpose data structures and algorithms at your fingertips. There's no excuse unless you're a beginning computer science student just getting your feet wet and reinventing the most primitive of wheels to be applying quadratic complexity sorts to massive input sizes or linear-time searches which can be accomplished in constant-time with the appropriate data structures.
So once you get past this beginner level, performance-critical applications still have wildly varying performance characteristics. Why? Why would one video processing software have three times the frame rate and more interactive video previews than the other when the developers aren't doing anything extremely dumb algorithmically? Why would one server doing a very similar thing be able to handle ten times the queries with the same hardware? Why would this software load a scene in 5 seconds while the other takes 5 minutes loading the same data? Why would this beautiful game have silky smooth and consistent frame rates while the other one is uglier, more primitive-looking with its graphics and lighting, and stutters here and there while taking twice the memory?
And that boils down to micro-optimizations, not algorithmic differences. Furthermore our memory hierarchy today is so skewed in performance, making previous algorithms that were thought to be good a couple of decades ago no longer as good if they exhibit poor locality of reference.
So if you want to write competitively-efficient software today, far more often than not, that will boil down to things like multithreading, SIMD, GPU, GPGPU, improving locality of reference with better memory access patterns (loop tiling, SoA, hot/cold field splitting, etc.), maybe even optimizing for branch prediction in extreme cases, and so forth, not so much algorithmic breakthroughs unless you're tackling an extremely unexplored territory where no programmers have ventured before.
There are still occasionally algorithmic breakthroughs that are potential game changers, like voxel-cone tracing recently. But those are exceptions and the people who come up with these often invest their lives to R&D (they're generally not people writing and maintaining large scale codebases), and it still boils down to micro-optimizations whether voxel-cone tracing can be applied to real-time environments like games or not. If you aren't good at micro-optimizations, you simply won't get the adequate framerates even using these algorithmic breakthroughs.