Let's say I have the string "Hey". I would like to determine all combinations of characters that exist in this string as fast as possible. The resulting algorithm should generate this:
H, e, y, He, ey, Hey
The algorithm should not produce the string "Hy" because it does not exist in the string as a substring.
There are O(n^2) of those substrings, of lengths [1,n], so any algorithm to generate all of them will be O(n^2) * O(n) = O(n^3):
(*) See Edit2 at the end - depending on the implementation of the string - the complexity can vary from O(n^2) to O(n^3)
Pseudo code:
result <- {} #result is a set if dupes should be terminated, otherwise - it is a multiset.
for i from 0 to s.length:
for j from i+1 to s.length:
result.add(s.substring(i,j))
return result
Note however, that you can do some "cheating", by creating an iterator and generate the substrings on the fly, it should look something like this [pseudo code]:
class MyIterator:
String s
int i,j
MyIterator(String s):
this.s = s
i = 0
j = 0
next():
j = j + 1
if (j >= s.length):
i = i + 1
j = i + 1
if (i >= s.length):
throw exception
return s.substring(i,j)
Note that creating the iterator is O(1), and each iteration is O(n) - but to actually produce all the elements, you need O(n^2) steps, so complexity remains O(n^3) overall, but you decrease the latency of your application.
EDIT:
I editted complexity, claiming it is O(n^2) is wrong, the complexity is O(n^3) since you need to generate strings of variable lengths, some of them are long. At least half of the generated substrings will be of length n/2 - thus the total complexity is Theta(n^3)
EDIT2:
In some cases it can actually be O(n^2) - depending on the string implementation. In java for example - it uses a single char[], and only "plays" with the offset and length - so in java - creation is actually O(n^2), since creating a substring is O(1)
In C however - it is O(n^3), since every substring needs to be copied to a different char[].
Check for the implementation of n-grams in php.
In your example string: Hey
H, E, Y are unigrams
HE, EY are bigrams
HEY is a trigram
Related
I am trying to solve a CTF in which the juggling type should be used. The code is:
if ($_GET["hash"] == hash("ripemd160", $_GET["hash"]))
{
echo $flag;
}
else
{
echo "<h1>Bad Hash</h1>";
}
I made a script in python which checks random hashes in ripemd160 that begins with "0e" and ends with only numbers. The code is:
def id_generator(size, chars=string.digits):
return ''.join(random.choice(chars) for _ in range(size))
param = "0e"
results = []
while True:
h = hashlib.new('ripemd160')
h.update("{0}".format(str(param)).encode('utf-8'))
hashed = h.hexdigest()
if param not in results:
print(param)
if hashed.startswith("0e") and hashed[2:].isdigit():
print(param)
print(hashed)
break
results.append(param)
else:
print("CHECKED")
param = "0e" + str(id_generator(size=10))
Any suggestions on how to solve it? Thank you!
There seems to be a bit of misunderstanding in the comments, so I'll start by explaining the problem a little more:
Type juggling refers to the behaviour of PHP whereby variables are implicitly cast to different data types under certain conditions. For example, all the following logical expressions will evaluate to true in PHP:
0 == 0 // int vs. int
"0" == 0 // str -> int
"abc" == 0 // any non-numerical string -> 0
"1.234E+03" == "0.1234E+04" // string that looks like a float -> float
"0e215962017" == 0 // another string that looks like a float
The last of these examples is interesting because its MD5 hash value is another string consisting of 0e followed by a bunch of decimal digits (0e291242476940776845150308577824). So here's another logical expression in PHP that will evaluate to true:
"0e215962017" == md5("0e215962017")
To solve this CTF challenge, you have to find a string that is "equal" to its own hash value, but using the RIPEMD160 algorithm instead of MD5. When this is provided as a query string variable (e.g., ?hash=0e215962017), then the PHP script will disclose the value of a flag.
Fake hash collisions like this aren't difficult to find. Roughly 1 in every 256 MD5 hashes will start with '0e', and the probability that the remaining 30 characters are all digits is (10/16)^30. If you do the maths, you'll find that the probability of an MD5 hash equating to zero in PHP is approximately one in 340 million. It took me about a minute (almost 216 million attempts) to find the above example.
Exactly the same method can be used to find similar values that work with RIPEMD160. You just need to test more hashes, since the extra hash digits mean that the probability of a "collision" will be approximately one in 14.6 billion. Quite a lot, but still tractable (in fact, I found a solution to this challenge in about 15 minutes, but I'm not posting it here).
Your code, on the other hand, will take much, much longer to find a solution. First of all, there is absolutely no point in generating random inputs. Sequential values will work just as well, and will be much faster to generate.
If you use sequential input values, then you also won't need to worry about repeating the same hash calculations. Your code uses a list structure to store previously hashed values. This is a terrible idea. Searching for an item in a list is an O(n) operation, so once your code has (unsuccessfully) tested a billion inputs, it will have to compare every new input against each of these billion inputs at each iteration, causing your code to grind to a complete standstill. Your code would actually run a lot faster if you didn't bother checking for duplicates. When you have time, I suggest you learn when to use lists, dicts and sets in Python.
Another problem is that your code only tests 10-digit numbers, which means it can only test a maximum of 10 billion possible inputs. Based on the numbers given above, are you sure this is a sensible limit?
Finally, your code is printing every single input string before you calculate its hash. Before your program outputs a solution, you can expect it to print out somewhere in the order of a billion screenfuls of incorrect guesses. Is there any point in doing this? No.
Here's the code I used to find the MD5 collision I mentioned earlier. You can easily adapt it to work with RIPEMD160, and you can convert it to Python if you like (although the PHP code is much simpler):
$n = 0;
while (1) {
$s = "0e$n";
$h = md5($s);
if ($s == $h) break;
$n++;
}
echo "$s : $h\n";
Note: Use PHP's hash_equals() function and strict comparison operators to avoid this sort of vulnerability in your own code.
I am currently trying to make a function that can tell if number a is closer to c than the number b is to c.
I have tried to do comparisons like so: (taking a and b from c and comparing those)
$c = 10;
$b = 2;
$a = 3;
$b_check = $c - $b; // = 8.
$a_check = $c - $a; // = 7.
In my head, I thought that whatever number (a or b) is smaller means that it's gonna be the closer number to c, though that worked using positive integers, but when it came to negative integers it gave the complete wrong outcome.
I was wondering if maybe there was an in-built function in PHP for this or if there's a better mathematical method for achieving this?
If (a - c) * (a - c) < (b - c) * (b - c) would do it. This gets round the negative number issue and is the way us old cats do it in C.
Else you could use Math.abs(a - c) < Math.abs(b - c) but the other way can be quicker for some types, but you need to take care not to overflow your type and there's a potential for a ruinous wraparound effect for some types too.
Profile it.
The one with the smallest absolute value is always the closest. You're thinking in terms of a number line, but it's easier to imagine if you know vectors in 2D or 3D space. Then the distance between two points is the square root of the sum of squares of the difference between their components. The 1D case falls out as a special case where two components are zero.
double distance = Math.sqrt((x2-x1)*(x2-x1)) = Math.abs(x2-x1)
I need to create a function which takes a single integer as argument in the range 0-N and returns a seemingly random number in the same range.
Each input number should always have exactly one output and it should always be the same.
Such a function would produce something like this:
f(1) = 4
f(2) = 1
f(3) = 5
f(4) = 2
f(5) = 3
I believe this could be accomplished by some kind of a hashing algorithm? I don't need anything complex, just not something too simple like f(1) = 2, f(2) = 3 etc.
The biggest issue is that I need this to be reversible. E.g. the above table should be true left-to-right as well as right-to-left, using a different function for the right-to-left conversion is fine.
I know the easiest way is to create an array, shuffle it and just store the relations in a db or something, but as I need N to be quite large I'd like to avoid this if possible.
Edit: For my particular case N is a specific number, it's exactly 16777216 (64^4).
If the range is always a power of two -- like [0,16777216) -- then you can use exclusive-or just as #MarkBaker suggested. It just doesn't work so easily if your range is not a power of two.
You can use addition and subtraction modulo N, although these alone are too obvious, so you have to combine it with something else.
You can also do multiplication modulo-N, but reversing that is complicated. To make it simpler, we can isolate the bottom eight bits and multiply those and add them in a way that doesn't interfere with those bits so we can use them again to reverse the operation.
I don't know PHP so I'm going to give an example in C, instead. Maybe it's the same.
int enc(int x) {
x = x + 4799 * 256 * (x % 256);
x = x + 8896843;
x = x ^ 4777277;
return (x + 1073741824) % 16777216;
}
And to decode, play the operations back in reverse order:
int dec(int x) {
x = x + 1073741824;
x = x ^ 4777277;
x = x - 8896843;
x = x - 4799 * 256 * (x % 256);
return x % 16777216;
}
That 1073741824 must be a multiple of N, and 256 must be a factor of N, and if N is not a power of two then you can't (necessarily) use exclusive-or (^ is exclusive-or in C and I assume in PHP too). The other numbers you can fiddle with, and add and remove stages, at your leisure.
The addition of 1073741824 in both functions is to ensure that x stays positive; this is so that the modulo operation doesn't ever give a negative result, even after we've subtracted values from x which might have made it go negative in the interim.
I offered to describe how I "randomly" scramble up 9-digit SSNs when producing research data sets. This does not replace or hash an SSN. It re-orders the digits. It is difficult to put the digits back in the correct order if you don't know the order in which they were scrambled. I have a gut feeling that this is not what the questioner really wants. So, I am happy to delete this answer if it is deemed off-topic.
I know that I have 9 digits. So, I start with an array that has 9 index values in order:
$a = array(0,1,2,3,4,5,6,7,8);
Now, I need to turn a key that I can remember into a way to shuffle the array. The shuffling has to be the same order for the same key every time. I use a couple tricks. I use crc32 to turn a word into a number. I use srand/rand to get a predictable order of random values. Note: mt_rand no longer produces the same sequence of random digits with the same seed, so I have to use rand.
srand(crc32("My secret key"));
usort($a, function($a, $b) { return rand(-1,1); });
The array $a still has the digits 0 through 8, but they are shuffled. If I use the same keyword I will get the same shuffled order every time. That lets me repeat this every month and get the same result. Then, with a shuffled array, I can pick the digits off the SSN. First, I ensure it has 9 characters (some SSNs are sent as integers and a leading 0 is omitted). Then, I build a masked SSN by picking the digits using $a.
$ssn = str_pad($ssn, 9, '0', STR_PAD_LEFT);
$masked_ssn = '';
foreach($a as $i) $masked_ssn.= $ssn{$i};
$masked_ssn will now have all the digits in $ssn, but in a different order. Technically, there are keywords that make $a become the original ordered array after shuffling, but that is very very rare.
Hopefully this makes sense. If so, you can do it all much faster. If you turn the original string into an array of characters, you can shuffle the array of characters. You just need to reseed rand every time.
$ssn = "111223333"; // Assume I'm using a proper 9-digit SSN
$a = str_split($ssn);
srand(crc32("My secret key"));
usort($a, function($a, $b) { return rand(-1,1); });
$masked_ssn = implode('', $a);
This is not really faster in a runtime way because rand is a rather expensive function and you run rand a hell of lot more here. If you are masking thousands of values as I do, you will want to use an index array that is shuffled just once, not a shuffling for every value.
Now, how do I undo it? Assume I'm using the first method with the index array. It will be something like $a = {5, 3, 6, 1, 0, 2, 7, 8, 4}. Those are the indexes for the original SSN in the masked order. So, I can easily build the original SSN.
$ssn = '000000000'; // I like to define all 9 characters before I start
foreach($a as $i=>$j) $ssn[$j] = $masked_ssn{$i};
As you can see, $i counts from 0 to 8 across the masked SSN. $j counts 5, 3, 6... and puts each value from the masked SSN in the correct place in the original SSN.
Looks like you've got good answer, but still there is an alternative. Linear Congruential Generator (LCG) could provide 1-to-1 mapping and it is known to be a reversible using Euclid's algorithm. For 24bit
Xi = [(A * Xi-1) + C] Mod M
where M = 2^24 = 16,777,216
A = 16,598,013
C = 12,820,163
For LCG reversability take a look at Reversible pseudo-random sequence generator
How can I convert a uniform distribution (as most random number generators produce, e.g. between 0.0 and 1.0) into a normal distribution? What if I want a mean and standard deviation of my choosing?
There are plenty of methods:
Do not use Box Muller. Especially if you draw many gaussian numbers. Box Muller yields a result which is clamped between -6 and 6 (assuming double precision. Things worsen with floats.). And it is really less efficient than other available methods.
Ziggurat is fine, but needs a table lookup (and some platform-specific tweaking due to cache size issues)
Ratio-of-uniforms is my favorite, only a few addition/multiplications and a log 1/50th of the time (eg. look there).
Inverting the CDF is efficient (and overlooked, why ?), you have fast implementations of it available if you search google. It is mandatory for Quasi-Random numbers.
The Ziggurat algorithm is pretty efficient for this, although the Box-Muller transform is easier to implement from scratch (and not crazy slow).
Changing the distribution of any function to another involves using the inverse of the function you want.
In other words, if you aim for a specific probability function p(x) you get the distribution by integrating over it -> d(x) = integral(p(x)) and use its inverse: Inv(d(x)). Now use the random probability function (which have uniform distribution) and cast the result value through the function Inv(d(x)). You should get random values cast with distribution according to the function you chose.
This is the generic math approach - by using it you can now choose any probability or distribution function you have as long as it have inverse or good inverse approximation.
Hope this helped and thanks for the small remark about using the distribution and not the probability itself.
Here is a javascript implementation using the polar form of the Box-Muller transformation.
/*
* Returns member of set with a given mean and standard deviation
* mean: mean
* standard deviation: std_dev
*/
function createMemberInNormalDistribution(mean,std_dev){
return mean + (gaussRandom()*std_dev);
}
/*
* Returns random number in normal distribution centering on 0.
* ~95% of numbers returned should fall between -2 and 2
* ie within two standard deviations
*/
function gaussRandom() {
var u = 2*Math.random()-1;
var v = 2*Math.random()-1;
var r = u*u + v*v;
/*if outside interval [0,1] start over*/
if(r == 0 || r >= 1) return gaussRandom();
var c = Math.sqrt(-2*Math.log(r)/r);
return u*c;
/* todo: optimize this algorithm by caching (v*c)
* and returning next time gaussRandom() is called.
* left out for simplicity */
}
Where R1, R2 are random uniform numbers:
NORMAL DISTRIBUTION, with SD of 1:
sqrt(-2*log(R1))*cos(2*pi*R2)
This is exact... no need to do all those slow loops!
Reference: dspguide.com/ch2/6.htm
Use the central limit theorem wikipedia entry mathworld entry to your advantage.
Generate n of the uniformly distributed numbers, sum them, subtract n*0.5 and you have the output of an approximately normal distribution with mean equal to 0 and variance equal to (1/12) * (1/sqrt(N)) (see wikipedia on uniform distributions for that last one)
n=10 gives you something half decent fast. If you want something more than half decent go for tylers solution (as noted in the wikipedia entry on normal distributions)
I would use Box-Muller. Two things about this:
You end up with two values per iteration
Typically, you cache one value and return the other. On the next call for a sample, you return the cached value.
Box-Muller gives a Z-score
You have to then scale the Z-score by the standard deviation and add the mean to get the full value in the normal distribution.
It seems incredible that I could add something to this after eight years, but for the case of Java I would like to point readers to the Random.nextGaussian() method, which generates a Gaussian distribution with mean 0.0 and standard deviation 1.0 for you.
A simple addition and/or multiplication will change the mean and standard deviation to your needs.
The standard Python library module random has what you want:
normalvariate(mu, sigma)
Normal distribution. mu is the mean, and sigma is the standard deviation.
For the algorithm itself, take a look at the function in random.py in the Python library.
The manual entry is here
This is a Matlab implementation using the polar form of the Box-Muller transformation:
Function randn_box_muller.m:
function [values] = randn_box_muller(n, mean, std_dev)
if nargin == 1
mean = 0;
std_dev = 1;
end
r = gaussRandomN(n);
values = r.*std_dev - mean;
end
function [values] = gaussRandomN(n)
[u, v, r] = gaussRandomNValid(n);
c = sqrt(-2*log(r)./r);
values = u.*c;
end
function [u, v, r] = gaussRandomNValid(n)
r = zeros(n, 1);
u = zeros(n, 1);
v = zeros(n, 1);
filter = r==0 | r>=1;
% if outside interval [0,1] start over
while n ~= 0
u(filter) = 2*rand(n, 1)-1;
v(filter) = 2*rand(n, 1)-1;
r(filter) = u(filter).*u(filter) + v(filter).*v(filter);
filter = r==0 | r>=1;
n = size(r(filter),1);
end
end
And invoking histfit(randn_box_muller(10000000),100); this is the result:
Obviously it is really inefficient compared with the Matlab built-in randn.
This is my JavaScript implementation of Algorithm P (Polar method for normal deviates) from Section 3.4.1 of Donald Knuth's book The Art of Computer Programming:
function normal_random(mean,stddev)
{
var V1
var V2
var S
do{
var U1 = Math.random() // return uniform distributed in [0,1[
var U2 = Math.random()
V1 = 2*U1-1
V2 = 2*U2-1
S = V1*V1+V2*V2
}while(S >= 1)
if(S===0) return 0
return mean+stddev*(V1*Math.sqrt(-2*Math.log(S)/S))
}
I thing you should try this in EXCEL: =norminv(rand();0;1). This will product the random numbers which should be normally distributed with the zero mean and unite variance. "0" can be supplied with any value, so that the numbers will be of desired mean, and by changing "1", you will get the variance equal to the square of your input.
For example: =norminv(rand();50;3) will yield to the normally distributed numbers with MEAN = 50 VARIANCE = 9.
Q How can I convert a uniform distribution (as most random number generators produce, e.g. between 0.0 and 1.0) into a normal distribution?
For software implementation I know couple random generator names which give you a pseudo uniform random sequence in [0,1] (Mersenne Twister, Linear Congruate Generator). Let's call it U(x)
It is exist mathematical area which called probibility theory.
First thing: If you want to model r.v. with integral distribution F then you can try just to evaluate F^-1(U(x)). In pr.theory it was proved that such r.v. will have integral distribution F.
Step 2 can be appliable to generate r.v.~F without usage of any counting methods when F^-1 can be derived analytically without problems. (e.g. exp.distribution)
To model normal distribution you can cacculate y1*cos(y2), where y1~is uniform in[0,2pi]. and y2 is the relei distribution.
Q: What if I want a mean and standard deviation of my choosing?
You can calculate sigma*N(0,1)+m.
It can be shown that such shifting and scaling lead to N(m,sigma)
I have the following code which maybe could help:
set.seed(123)
n <- 1000
u <- runif(n) #creates U
x <- -log(u)
y <- runif(n, max=u*sqrt((2*exp(1))/pi)) #create Y
z <- ifelse (y < dnorm(x)/2, -x, NA)
z <- ifelse ((y > dnorm(x)/2) & (y < dnorm(x)), x, z)
z <- z[!is.na(z)]
It is also easier to use the implemented function rnorm() since it is faster than writing a random number generator for the normal distribution. See the following code as prove
n <- length(z)
t0 <- Sys.time()
z <- rnorm(n)
t1 <- Sys.time()
t1-t0
function distRandom(){
do{
x=random(DISTRIBUTION_DOMAIN);
}while(random(DISTRIBUTION_RANGE)>=distributionFunction(x));
return x;
}
I have an alphabet array with 26 characters A..Z .
I am searching for a performant algorithm that lists all permutations that fill an array of length X without any repeating characters.
Examples:
X=3 . Target array: _ _ _
Permutations are A B C until Z Y X .
X=4 . Target array: _ _ _ _
Permutations are A B C D until Z Y X W
X=5 . Target array: _ _ _ _ _
Permutations are A B C D E until Z Y X W V
(Sorry, I don't know how this kind of algorithm is named)
Thanks in advance.
Code in C, Delphi or Java is also OK, since it can be easy translated.
A simple solution is a recursive one
char current_combination[27];
int char_used[26];
void enumerate(int i, int n)
{
for (int j=0; j<26; j++)
{
if (!char_used[j])
{
char_used[j] = 1;
current_combination[i] = 'A' + j;
if (i+1 == n)
{
puts(current_combination);
}
else
{
enumerate(i+1, n);
}
char_used[j] = 0;
}
}
}
The function above accepts the index i of the character to be computed and the total number n of characters in a combination (the code assumes i<n). It keeps the current combination and the array of flags for already used variables in globals to avoiding copying them around.
To generate for example all combinations of length 5 call enumerate(0, 5).
Note that the total number of combinations grows very fast. For example for n=6 there are 165,765,600 combinations, with more than 1Gb of output.
I'd take a simple brute force approach though do understand the number of permutations can get up there as the number is 26!/(26-x)! which can be rather large as for 3 there are 15,600 permutations and for 5 there are 7,893,600 permutations which isn't exactly small. Basically you could just loop through all the values with loops in loops that unfortunately would be O(n^x) where x is the number of characters since the nesting of loops causing the complexity jump.
Something to consider is how finely are you examining complexity here. For example, while you could consider ways to go about being clever in the first pair of loops to avoid duplication, the third loop in becomes a bit trickier though if you started with a List of 26 letters and removed the previous ones, this would make the last loop be a simply iterative as you know there isn't any duplicates though this can be expensive in terms of memory consumed in having to make copies of the list on each pass from the outer loop. Thus the first time, you'd go through AB_ and then AC_ and so forth, but the copying of the list may be where this gets expensive in terms of operations as there would be thousands of times that the list is copied that one could wonder if that is more efficient than doing comparisons.
Are you sure you want to see all permutations? If you have X=3, you will have 26*25*24 combinations = 15600. And if X=5 number of combinations is equal to 7893600.
You need to randomly select one letter(or array index) and store it somewhere and on each iteration you should check if this letter(or index) has been already selected on one of the previous iteration. After this you will get random sequence which length is X characters. You need to store it too. Then you need to reapeat all operation made on the previous step and also you have to check if there is random sequense with subsequence you have been generating now.
Or you could use direct enumeration.
Sorry for not satisfactory english. I tried to be clear.
I hope it will be usefull.