I'm creating a link shortening service and I'm using base64 encoding/decoding of an incremented ID field to create my urls. A url with the ID "6" would be: http://mysite.com/Ng==
I need to also allow users to create a custom url name, like http://mysite.com/music
Here's my (possibly faulty) approach so far. Help in fixing it would be appreciated.
When someone creates a new link:
I get the largest link ID from the database (it's not auto incremented)
Increment the ID by 1
Generate a short URL code (http://website.com/[short url name]) by base64_encoding that ID
Insert into links table: id, short_url_code, destination_url
When someone creates a new link and passes a custom short URL:
My plan was base64_decode their custom string and use that as the link ID, but I didn't realize that you can't just base64_decode any alphanumeric string and turn it into a number.
Is there a better encoding method that will let me turn any number into a short string, and any string into a number, so I can always lookup short urls (whether custom or autogenerated) by turning the name into a number and querying for a link with an ID equal to that number?
First and foremost, make sure you have unicity constraints in place on the ID and short_url_code columns.
When someone creates a new link:
Get the next largest link ID from the database (for performance reasons you should really REALLY use autoincrement or SEQUENCE, depending on what your RDBMS offers; otherwise go ahead and select MAX(ID)+1 )
Generate a short URL code (http://website.com/[short url name]) from ID using base64_encode or any other custom or standard encoding scheme
Insert into the links table: ID, short_url_code, destination_url
If the insert fails because of a constraint violation go back to step 1 to try a new ID; you may have had a violation because:
the same ID has already been used (i.e. inserted) in parallel by another thread/process etc. (this will not happen if you used autoincrement or SEQUENCE, and may happen quite often otherwise), and/or
the same short_url_code has already been used as a custom URL (this will happen very seldomly unless someone is trying to cause trouble on your site)
If the insert succeeded, commit and return the short URL to the user
When someone creates a new link and passes a custom short URL:
Perform the same step 1 as above
Instead of generating the short URL part from ID as in step 2 above, use the custom short_url_code provided by the user
Perform the same step 3 as above
If the insert failed because of:
a constraint violation on ID: go back to step 1 to try a new ID
a constraint violation on short_url_code: return an error to the user asking him to pick a different custom URL, as the short URL he/she provided has already been used
Perform the same step 5 as above
base64 can be used to make short urls, but it can also make the url longer. For instance the base64_encode of the number 1 is 'MQ==' which is 4 times the size. Base64 will always have 2 characters to obtain the 64bits, which is not ideal for short urls.
If size is the most important factor then you maybe able to produce the shortest urls by relying on internationalization.
This can make a URI rather long (up to 9 ASCII characters for a single Unicode character), but the intention is that browsers only need to display the decoded form, and many protocols can send UTF-8 without the %HH escaping.
Keep in mind that Browsers work quite well with UTF-8, and twitter will have no trouble with these urls.
Related
I'm creating a pretty simple application which allows references (of potential employees) to upload their own reference letters. Here's how it works:
The applicant submits the references email
The reference receives an automatically generated email containing a unique URL (for security reasons)
Reference follows the link, answers security question (in case he wishes to access the site more than once)
Uploads letter
I'm stuck on how to generate a completely (okay, okay, quasi will do) random URL. What's more: how do I ensure that following the link will direct the references to the correct page? Do I have to create a new page containing the drop box every time I send out a random URL?
Thanks for any suggestions on how to go about this :)
You might try using a hash algorithm which generates the unique-esque checksum from the contents of the file. Usually (for example with md5()) one byte change in the original content results in a completely different hash. (Notice: md5 has some collision vulnerabilities.)
If you store the uploaded file with the filename of the hash, you will be able to retrieve it at a later date, but for more complex system, there should be a database set up which makes the relation between the random URL and the stored content.
If you don't want to hash, there code snippet below could help to generate random URLs (but make sure that if an URL is already used, you prevent accidental overwrites):
md5( sha1( time() + rand(0, time()) ) );
I presume that when you say that you want to generate a random URL, you are essentially asking to generate a random string. This is potentially very simple; here's some pseudocode:
for i = 1 to stringLength
randomString[i] = floor(random() * 26) + 'a'
end
In other words, generate a random number between 0 and 25, and add it to the ASCII value for the character 'a'. This would generate a random string of lowercase letters, which I think should be sufficient for your task. In PHP, you would use the rand function. It would be advisable to use the srand function to seed the random number generator with the current time, like the example at the end of the given link.
As for the second part; I recommend that you simplify things; rather than generating an actual page with a random URL, why not simply pass a random string into the query string such as:
www.mydomain.com/uploadReference.php?id=xxxxxxx
Where xxxxxxx is your random string. You can then verify the string and look it up in a database using PHP. This seems, for your purposes, by far the easiest way.
You can make a unique string based on some form of hash of the current time stamp or the reference's unique credentials (e.g. username or something). You could then create one page for the dropbox that would accept that unique string in the URL to be used for a script on the page which would retrieve the relevant data mapped by that string in a database.
You could also create a random permutation of numerals and characters so that
hash($previous) // is unique
The basic idea is that the 'hash'-function depends only on the previous value, creating a new unique value. For example so that '0' -> '1', '1' -> '2', '9' -> 'a', 'z' -> '10', 'z0' -> '11'. Such an algorithm is relatively easy to devise
I never worked with barcode and now i must design a whole app with barcode support. I was wondering what type of barcode i can use, how can i make shure that barcode string is uniqe and how would i store that in MySQL.
I was thinkin about generating some barcode strings and print them to stickers so my clients can use them. I was thinking to do generating part in php/mysql then prepare for printing (render in pdf). Let's say i generated 100 strings and store them to database and next time i want to generate another 200 that must be unique.
I don't even know where to begin with string. What information can i store in barcode string?
Can i do this: XXX-ZZZZZ-YYYY-autincrementID?
Where XXX is country ID, ZZZZZ is client ID, YYYY is barcode string ID. Should i use surrogative key for my primary key or should i split those to multiple tables?
Did i mentioned that all autoincrementID's should start from 1 for each client :) I am sooooo confused about all this.
Thanks
First decide on the barcode format you want to use.
Then check if there is a PHP implementation out there (there will be for most - if not all - barcode formats).
A basic example (using PEAR Image_Barcode) can be found at Using barcodes in your web application.
You just store the text in the DB and can generate the corresponding image using the Image_Barcode class (it supports Code 39, Code 128, EAN 13, INT 25, PostNet and UPCA).
I once wrote an app creating EAN 13 barcodes, don't remember which lib I used though (I'll check at home if I can find the source).
We need to separate some concerns.
First is the action of printing any given string as a barcode. The other answers talk about how to do that.
The other action has nothing to do with barcodes and is about database design. Your example suggests the barcode will be a combination of values. However, I get the idea (correct me if I am wrong) that the larger application is not yet clearly spelled out. Therefore it does not matter what kind of "play" table you create for unique codes right now -- create whatever you want. When you know what values must be printed as barcodes, then we are into a database design question.
A barcode is just a way to print and/or read a string. It involves
special fonts,
some calculation (for check digits)
Your first step should be to identify wich barcode you need to support. Many companies manufacturing barcode printers and readers also provide some help about that.
I found some great help here, including free fonts. It's a french site but a few things are available in English.
We use UUIDs for our primary keys in our db (generated by php, stored in mysql). The problem is that when someone wants to edit something or view their profile, they have this huge, scary, ugly uuid string at the end of the url. (edit?id=.....)
Would it be safe (read: still unique) if we only used the first 8 characters, everything before the first hyphen?
If it is NOT safe, is there some way to translate it into something else shorter for use in the url that could be translated back into the hex to use as a lookup? I know that I can base64 encode it to bring it down to 22 characters, but is there something even shorter?
EDIT
I have read this question and it said to use base64. again, anything shorter?
Shortening the UUID increases the probability of a collision. You can do it, but it's a bad idea. Using only 8 characters means just 4 bytes of data, so you'd expect a collision once you have about 2^16 IDs - far from ideal.
Your best option is to take the raw bytes of the UUID (not the hex representation) and encode it using base64. Or, just don't worry much, because I seriously doubt your users care what's in the URL.
Don't cut a single bit out of that UUID: You have no control over the algorithm that produced it, there are multiple possible implementation, algorithm implementation is subject to change (example: changed with the version of PHP you're using)
If you ask me an UUID in the address bar doesn't look scary or difficult at all, even a simple google search for "UUID" produces worst looking URL's, and everybody's used to looking at google URL's!
If you want nicer looking URL's, take a look at the address bar of this stackoverflow.com article. They're using the article ID followed by the title of the question. Only the ID part is relevant, everything else is there to make it easy on the eyes of readers (go ahead and try it, you can delete anything after the ID, you can replace it with junk - doesn't matter).
It is not safe to truncate uuid's. Also, they are designed to be globally unique, so you aren't going to have luck shortening them. Your best bet is to either assign each user a unique number, or let users pick a custom (unique) string (like a username, or nick name) that can be decoded. So you could have edit?id=.... or edit?name=blah and you then decode name into the uuid in your script.
It depends on how you're generating the UUID - if you're using PHP's uniqid then it's the right-most digits that are more "unique". However, if you're going to truncate the data, then there's no real guarantee that it'll be unique anyway.
Irrespective, I'd say that this is a somewhat sub-optimal approach - is there no way you can use a unique (and ideally meaningful) textual reference string instead of an ID in the query string? (Hard to know without more knowledge of the problem domain, but it's always a better approach in my opinion, even if SEO, etc. isn't a factor.)
If you were using this approach, you could also let MySQL generate the unique IDs, which is probably a considerably more sane approach than attempting to handle this in PHP.
If you're worried about scaring users with the UUID in the URL, why not write it out to a hidden form field instead?
I want to create shortened links for specific pieces of content on my site. To view these pages now, I pull the relevant content via the content ID passed via GET (ie, mysite.com/content/?id=332). To obfuscate the ID, I want to use base64 to encode and decode it into a short alphanumeric string (like 34sa6), which I already know how to do.
My question is this: does it make more sense to store this string as a database field on creation of each piece of content, or simply decode the string on the fly when a user visits mysite.com/content/34sa6 (which means visiting mysite.com/content/?id=332 will also load the correct page). If I store this instead, it will become the defacto primary key for my purposes, as all related content will be queried based on it, so just trying to figure out the wisest way to do it.
If you decode and encode it during the request, you can't switch to a different method of encoding in the future.
Storing it in the database allows you to change the encoding whenever you want, because when the full URL needs to be retrieved, the script only looks for a matching database entry.
I've always wondered how and why they do this...an example: http://youtube.com/watch?v=DnAMjq0haic
How are these IDs generated such that there are no duplicates, and what advantage does this have over having a simple auto incrementing numeric ID?
How do one keep it short but still keep it's uniqueness? The string uniqid creates are pretty long.
Kevin van Zonneveld has written an excellent article including a PHP function to do exactly this. His approach is the best I've found while researching this topic.
His function is quite clever. It uses a fixed $index variable so problematic characters can be removed (vowels for instance, or to avoid O and 0 confusion). It also has an option to obfuscate ids so that they are not easily guessable.
Try this: http://php.net/manual/en/function.uniqid.php
uniqid — Generate a unique ID...
Gets a prefixed unique identifier based on the current time in microseconds.
Caution
This function does not generate cryptographically secure values, and should not be used for cryptographic purposes. If you need a cryptographically secure value, consider using random_int(), random_bytes(), or openssl_random_pseudo_bytes() instead.
Warning
This function does not guarantee uniqueness of return value. Since most systems adjust system clock by NTP or like, system time is changed constantly. Therefore, it is possible that this function does not return unique ID for the process/thread. Use more_entropy to increase likelihood of uniqueness...
base62 or base64 encode your primary key's value then store it in another field.
example base62 for primary key 12443 = 3eH
saves some space, which is why im sure youtube is using it.
doing a base62(A-Za-z0-9) encode on your PK or unique identifier will prevent the overhead of having to check to see if the key already exists :)
I had a similar issue - I had primary id's in the database, but I did not want to expose them to the user - it would've been much better to show some sort of a hash instead. So, I wrote hashids.
Documentation: http://www.hashids.org/php/
Souce: https://github.com/ivanakimov/hashids.php
Hashes created with this class are unique and decryptable. You can provide a custom salt value, so others cannot decrypt your hashes (not that it's a big problem, but still a "good-to-have").
To encrypt a number your would do this:
require('lib/Hashids/Hashids.php');
$hashids = new Hashids\Hashids('this is my salt');
$hash = $hashids->encrypt(123);
Your $hash would now be: YDx
You can also set minimum hash length as the second parameter to the constructor so your hashes can be longer. Or if you have a complex clustered system you could even encrypt several numbers into one hash:
$hash = $hashids->encrypt(2, 456); /* aXupK */
(for example, if you have a user in cluster 2 and an object with primary id 456) Decryption works the same way:
$numbers = $hashids->decrypt('aXupK');
$numbers would then be: [2, 456].
The good thing about this is you don't even have to store these hashes in the database. You could get the hash from url once request comes in and decrypt it on the fly - and then pull by primary id's from the database (which is obviously an advantage in speed).
Same with output - you could encrypt the id's on the way out, and display the hash to the user.
EDIT:
Changed urls to include both doc website and code source
Changed example code to adjust to the main lib updates (current PHP lib version is 0.3.0 - thanks to all the open-source community for improving the lib)
Auto-incrementing can easily be crawled. These cannot be predicted, and therefore cannot be sequentially crawled.
I suggest going with a double-url format (Similar to the SO URLs):
yoursite.com/video_idkey/url_friendly_video_title
If you required both the id, and the title in the url, you could then use simple numbers like 0001, 0002, 0003, etc.
Generating these keys can be really simple. You could use the uniqid() function in PHP to generate 13 chars, or 23 with more entropy.
If you want short URLs and predictability is not a concern, you can convert the auto-incrementing ID to a higher base.
Here is a small function that generates unique key randomly each time. It has very fewer chances to repeat same unique ID.
function uniqueKey($limit = 10) {
$characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
$randstring = '';
for ($i = 0; $i < $limit; $i++) {
$randstring .= $characters[rand(0, strlen($characters))];
}
return $randstring;
}
source: generate random unique IDs like YouTube or TinyURL in PHP
Consider using something like:
$id = base64_encode(md5(uniqid(),true));
uniqid will get you a unique identifier. MD5 will diffuse it giving you a 128 bit result. Base 64 encoding that will give you 6 bits per character in an identifier suitable for use on the web, weighing in around 23 characters and computationally intractable to guess. If you want to be even more paranoid ugrade from md5 to sha1 or higher.
A way to do it is by a hash function with unique input every time.
example (you've tagged the question with php therfore):
$uniqueID = null
do {
$uniqueID = sha1( $fileName + date() );
} while ( !isUnique($uniqueID) )
There should be a library for PHP to generate these IDs. If not, it's not difficult to implement it.
The advantage is that later you won't have name conflicts, when you try to reorganize or merge different server resources. With numeric ids you would have to change some of them to resolve conflicts and that will result in Url change leading to SEO hit.
So much of this depends on what you need to do. How 'unique' is unique? Are you serving up the unique ID's, and do they mean something in your DB? if so, a sequential # might be ok.
ON the other hand, if you use sequential #'s someone could systematically steal your content by iterating thru the numbers.
There are filesystem commands that will generate unique file names - you could use those.
Or GUID's.
Results of hash functions like SHA-1 or MD5 and GUIDs tend to become very long, which is probably something you don't want. (You've specifically mentioned YouTube as an example: Their identifiers stay relatively short even with the bazillion videos they are hosting.)
This is why you might want to look into converting your numeric IDs, which you are using behind the scenes, into another base when putting them into URLs. Flickr e.g. uses Base58 for their canonical short URLs. Details about this are available here: http://www.flickr.com/groups/api/discuss/72157616713786392/. If you are looking for a generic solution, have a look at the PEAR package Mathe_Basex.
Please note that even in another base, the IDs can still be predicted from outside of your application.
I don't have a formula but we do this on a project that I'm on. (I can't share it). But we basically generate one character at a time and append the string.
Once we have a completed string, we check it against the database. If there is no other, we go with it. If it is a duplicate, we start the process over. Not very complicated.
The advantage is, I guess that of a GUID.
This is NOT PHP but can be converted to php or as it's Javascript & so clinetside without the need to slow down the server.. it can be used as you post whatever needs a unique id to your php.
Here is a way to create unique ids limited to
9 007 199 254 740 992 unique id's
it always returns 9 charachters.
where iE2XnNGpF is 9 007 199 254 740 992
You can encode a long Number and then decode the 9char generated String
and it returns the number.
basically this function uses the 62base index Math.log() and Math.Power to get the right index based on the number.. i would explain more about the function but ifound it some time ago and can't find the site anymore and it toke me very long time to get how this works... anyway i rewrote the function from 0.. and this one is 2-3 times faster than the one that i found.
i looped through 10million checking if the number is the same as the enc dec process and it toke 33sec with this one and the other one 90sec.
var UID={
ix:'abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ',
enc:function(N){
N<=9007199254740992||(alert('OMG no more uid\'s'));
var M=Math,F=M.floor,L=M.log,P=M.pow,r='',I=UID.ix,l=I.length,i;
for(i=F(L(N)/L(l));i>=0;i--){
r+=I.substr((F(N/P(l,i))%l),1)
};
return UID.rev(new Array(10-r.length).join('a')+r)
},
dec:function(S){
var S=UID.rev(S),r=0,i,l=S.length,I=UID.ix,j=I.length,P=Math.pow;
for(i=0;i<=(l-1);i++){r+=I.indexOf(S.substr(i,1))*P(j,(l-1-i))};
return r
},
rev:function(a){return a.split('').reverse().join('')}
};
As i wanted a 9 character string i also appended a's on the generated string which are 0's.
To encode a number you need to pass a Number and not a string.
var uniqueId=UID.enc(9007199254740992);
To decode the Number again you need to pass the 9char generated String
var id=UID.dec(uniqueId);
here are some numbers
console.log(UID.enc(9007199254740992))//9 biliardi o 9 milioni di miliardi
console.log(UID.enc(1)) //baaaaaaaa
console.log(UID.enc(10)) //kaaaaaaaa
console.log(UID.enc(100)) //Cbaaaaaaa
console.log(UID.enc(1000)) //iqaaaaaaa
console.log(UID.enc(10000)) //sBcaaaaaa
console.log(UID.enc(100000)) //Ua0aaaaaa
console.log(UID.enc(1000000)) //cjmeaaaaa
console.log(UID.enc(10000000)) //u2XFaaaaa
console.log(UID.enc(100000000)) //o9ALgaaaa
console.log(UID.enc(1000000000)) //qGTFfbaaa
console.log(UID.enc(10000000000)) //AOYKUkaaa
console.log(UID.enc(100000000000)) //OjO9jLbaa
console.log(UID.enc(1000000000000)) //eAfM7Braa
console.log(UID.enc(10000000000000)) //EOTK1dQca
console.log(UID.enc(100000000000000)) //2ka938y2a
As you can see there are alot of a's and you don't want that... so just start with a high number.
let's say you DB id is 1 .. just add 100000000000000 so that you have 100000000000001
and you unique id looks like youtube's id 3ka938y2a
i don't think it's easy to fulfill the other 8907199254740992 unique id's