I have a piece of shareware that I wrote that I'd like to distribute on the internet. I have a serial number type thing set up but there is still a ton of key sharing :(. I'd like to add a system where once the user enters the serial, it is checked with my server to make sure that it is valid. Simplicity is key.
-Client sends MD5 of serial number to web page using php type thing "www.mywebsite.com?key=3434343"
-Php script takes MD5 and checks it against simple database. If MD5 key exists in database, it increments a counter associated with that key, and generates a webpage that displays a 1. If no entry in the database, the script generates a webpage that displays 0.
-Client reads webpage and checks for 0 or 1 and acts accordingly.
I will manually enter in valid keys (md5'd) in the database and monitor active keys for too many activations. I don't know anything about php so I don't know if this will work. Does this seem good enough? Are there blatant security holes? I will be using a shared host. Is there a better way? Thanks
There are actually several blatant security holes in your scheme.
The first is that users can redirect their local internet traffic to a site that pretends to be you, but always displays a "1".
The second is that each key would only be good for X "activations", but if the message is lost in transit, too bad - the counter is still incremented. You need some way to validate that everything is OK; viewing the page is not enough.
The third is that hacking the program to think it got a "1" from your site would be the easiest thing in the world.
Don't implement your own copy protection. It will just annoy legitimate users, while not even slowing down real pirates. Either use a sophisticated commercial system, or (better yet!) try to strike at whatever motive is leading users to steal your program. This might be that your price is too high, or perhaps they don't feel you respect them. People do things for a reason.
You're lacking a vital understanding of the key... the key is in form of MD5 which will contain both alpha and numeric characters, consider this as an example:
3c102d6b53a7a8723e694f1417c2abfe
How are you going to generate the key? On what basis you generate the key from?
I also, see in this is the passing of the key to the website using the parameter like this:
www.mywebsite.com?key=3434343
It will take a while for the hacker to find a key...what with the advent of processor speed and key generation.... it will be cracked in no time, further more, you will be exposing your script.... do not underestimate what hackers/crackers can do to break the scripts... which is something you did not mention in your posting, no security defences of any kind!
Also, there's no mention of using security certificates to authenticate and not to scare off the end user or even worse, an antivirus scanner may flag the site as suspicious thus the end user is locked out of the application.....
BOTTOM LINE: Do not try security schemes like this as they are false economy, either way, it would be equivalent of a moth attracted to a flame, when an application "is protected", hackers/crackers will attempt to bypass it regardless of what you think..and you would be kidding yourself into thinking and deluding yourself that you're a brilliant programmer and that your dreamt-up-scheme is foolproof - DON'T...Stay away from these kind of schemes....
I would echo the points that have been raised in the comments about why you should consider not doing this, or at least not doing it yourself. But if you really want to (and/or for educational purposes), for a system like this I'd go for asymmetric encryption.
Basically, the idea is that you generate a public/private key pair, embed the public key in your application, and keep the private key on your server. In order to get registered, a copy of your application would send the serial number to your server. The PHP script on the server would check its database for that serial number, decide whether the registration is valid, and if it is, the server sends back the serial number encrypted with its private key. Your program can then use the public key to decrypt the message received, compare the decrypted version against the serial number, and if it matches, it can activate itself. Of course, this won't foil a (semi-)determined hacker, but it does eliminate the easiest way to get around the scheme you gave in your question, namely redirecting the request to a site which always returns a 1.
If you're worried about serial numbers traveling over the internet unencrypted, you could use the public key to encrypt the serial number before the program sends it to the server. And when the server responds, instead of sending back the encrypted serial number, it would send back (1) a random number generated on the server and (2) an encrypted hash of the random number and the serial number. Or use HTTPS, which does something similar. But if you're trying to protect against someone who has the means and motivation to intercept communications between your program and the server, you'll need more than just this simple scheme because that person almost certainly has more sophisticated tools at his/her disposal. In that case you would need a reputable commercial DRM system to have the best chance of protecting your program. And at that point, I'd ask yourself whether it's really worth it.
Related
Imagine a pretty standard website, with user authenticating with email/password pair. For passwords, it already ha shashing with random salt, but the rest of data is kept unencrypted.
We do another step forward and encrypt the sensitive data with a password key, the key, obviously, shall be known to the application to be able to decript the data for its operation.
we don't want to have it in the source code, so it's kept in a file and read by the app when it needs it.
we've secured the file so that only user which executes the app can read it
(this point has appeared after some discussions below) We have already considered buying hardware HSM and found that not possible (for instance we are running the server on a virtual machine)
this way we are relatively protected from complete DB stealing, right? However, the key might become known if someone gets access to the OS user with read rights.
the question is: what are the best practices for keeping such key secure?
Buy a hardware security module and keep the key in it. The key will not be able to be read.
Yubi makes a reasonably priced hsm. $500 if I recall correctly.
While we're here, your db server should be on a different box in a different network zone as your web server.
I have a php/js site where the information is encoded and put into the database. The encryption key for the information is randomly generated, then given back to the users after they send a post through a form. The encryption key is not stored in my database at all. A seperate, randomly generated, ID is formed and stored in the database, used to lookup the item itself before deciphering it.
My question is, is it possible at all to look through the logs and find information that would reveal the key? I am trying to make it impossible to read any of the SQL data without either being the person who has the code (who can do whatever he wants with it), or by a brute force attack (unavoidable if someone gets my SQL database)?
Just to re-iterate my steps:
User sends information through POST
php file generates random ID and access key. The data is encrypted with the access key then put in the php database with the ID as the PRIMARY KEY.
php file echos just the random ID and the access key.
website uses jQuery to create a link from the key and mysite.com?i=cYFogD3Se8RkLSE1CA [9 digit A-Ba-b09 = ID][9 digit A-Ba-b09 = key]
Is there any possibility if someone had access to my server that can read the information? I want it to be information for me to read the messages myself. The information has to be decodable, it can't be a one way encoding.
I like your system of the URL containing the decryption key, so that not even you, without having data available only on the user's computer, will be able to access.
I still see a few gotchas in this.
URLs are often saved in web server logs. If you're logging to disk, and they get the disk, then they get the keys.
If the attacker has access to your database, he may have enough access to your system to secretly install software that logs the URLs. He could even do something as prosaic as turn logging back on.
The person visiting your site will have the URL bookmarked at least (otherwise it is useless to him) and it will likely appear in his browser history. Normally, bookmarks and history are not considered secure data. Thus, an attacker to a user's computer (either by sitting down directly or if the computer is compromised by malware) can access the data as well. If the payload is desirable enough, someone could create a virus or malware that specifically mines for your static authentication token, and could achieve a reasonable hit rate. The URLs could be available to browser plugins, even, or other applications acting under a seemingly reasonable guise of "import your bookmarks now".
So it seems to me that the best security is then for the client to not just have the bookmark (which, while it is information, it is not kept in anyone's head so can be considered "something he has"), but also for him to have to present "something he knows", too. So encrypt with his password, too, and don't save the password. When he presents the URL, ask for a password, and then decrypt with both (serially or in combination) and the data is secure.
Finally, I know that Google's two-factor authentication can be used by third parties (for example, I use it with Dropbox). This creates another "something you have" by requiring the person accessing the resource to have his cell phone, or nothing. Yes, there is recourse if you lose your cell phone, but it usually involves another phone number, or a special Google-supplied one-time long password that has been printed out and stashed in one's wallet.
Let's start with some basic definitions:
Code Protecting data by translating it to another language, usually a private language. English translated to Spanish is encoded but its not very secure since many people understand Spanish.
Cipher Protecting data by scrambling it up using a key. A letter substitution cipher first documented by Julius Caesar is an example of this. Modern techniques involve mathematical manipulation of binary data using prime numbers. The best techniques use asymmetric keys; the key that is used to encipher the data cannot decipher it, a different key is needed. This allows the public key to be published and is the basis of SSL browser communication.
Encryption Protecting data by encoding and/or enciphering it.
All of these terms are often used interchangeably but they are different and the differences are sometimes important. What you are trying to do is to protect the data by a cipher.
If the data is "in clear" then if it is intercepted it is lost. If it is enciphered, then both the data and the key need to be intercepted. If it is enciphered and encoded, then the data, the key and the code need to be intercepted.
Where is your data vulnerable?
The most vulnerable place for any data is when it is in clear the personal possession of somebody, on a storage device (USB, CD, piece of paper) or inside their head since that person is vulnerable to inducement or coercion. This is the foundation of Wikileaks - people who are trusted with in confidence information are induced to betray that confidence - the ethics of this I leave to your individual consciences.
When it is in transit between the client and the server and vice versa. Except for data of national security importance the SSL method of encryption should be adequate.
When it is in memory in your program. The source code of your program is the best place to store your keys, however, they themselves need to be stored encrypted with a password that you enter each time your program runs (best), that is entered when you compile and publish or that is embedded in your code (worst). Unless you have a very good reason one key should be adequate; not one per user. You should also keep in-memory data encrypted except when you actually need it and you should use any in-memory in-clear data structures immediately and destroy them as soon as you are finished with them. The key has to be stored somewhere or else the data is irrecoverable. But consider, who has access to the source code (including backups and superseded versions) and how can you check for backdoors or trojans?
When it is in transit between your program's machine and the data store. If you only send encrypted data between the program and the data store and DO NOT store the key in the data store this should be OK.
When it is stored in the data store. Ditto.
Do not overlook physical security, quite often the easiest way to steal data is to walk up to the server and copy the hard drive. Many companies (and sadly defence/security forces) spend millions on on-line data security and then put their data in a room with no lock. They also have access protocols that a 10 year old child could circumvent.
You now have lovely encrypted data - how are you going to stop your program from serving it up in the clear to anyone who asks for it?
This brings us to identification, validation and authorisation. More definitions:
Identification A claim made by a person that they are so-and-so. This is usually handled in a computer program by a user name. In physical security applications it is by a person presenting themselves and saying "I am so-and-so"; this can explicitly be by a verbal statement or by presenting an identity document like a passport or implicitly by a guard you know recognising you.
Validation This is the proof that a person is who they say they are. In a computer this is the role of the password; more accurately, this proves that they know the person they say they are's password which is the big, massive, huge and insurmountable problem in the whole thing. In physical security it is by comparing physical metrics (appearance, height etc) as documented in a trusted document (like a passport) against the claim; you need to have protocols in place to ensure that you can trust the document. Incidentally, this is the main cause of problems with face recognition technology to identify bad guys – it uses a validation technique to try and identify someone. “This guy looks like Bad Guy #1”; guess what? So do a lot of people in a population of 7 billion.
Authorisation Once a person has been identified and validated they are then given authorisation to do certain things and go to certain places. They may be given a temporary identification document for this; think of a visitor id badge or a cookie. Depending on where they go they may be required to reidentify and revalidate themselves; think of a bank’s website; you identify and validate yourself to see your bank accounts and you do it again to make transfers or payments.
By and large, this is the weakest part of any computer security system; it is hard for me to steal you data, it is far easier for me to steal your identity and have the data given to me.
In your case, this is probably not your concern, providing that you do the normal thing of allowing the user to set, change and retrieve their password in the normal commercial manner, you have probably done all you can.
Remember, data security is a trade off between security on the one hand and trust and usability on the other. Make things too hard (like high complexity passwords for low value data) and you compromise the whole system (because people are people and they write them down).
Like everything in computers – users are a problem!
Why are you protecting this data, and what are you willing to spend to do so?
This is a classic risk management question. In effect, you need to consider the adverse consequences of losing this data, the risk of this happening with your present level of safeguards and if the reduction in risk that additional safeguards will cost is worth it.
Losing the data can mean any or all of:
Having it made public
Having if fall into the wrong person’s hands
Having it destroyed maliciously or accidently. (Backup, people!)
Having it changed. If you know it has been changed this is equivalent to losing it; if you don’t this can be much, much worse since you may be acting on false data.
This type of thinking is what leads to the classification of data in defence and government into Top Secret, Secret, Restricted and Unrestricted (Australian classifications). The human element intervenes again here; due to the nature of bureaucracy there is no incentive to give a document a low classification and plenty of disincentive; so documents are routinely over-classified. This means that because many documents with a Restricted classification need to be distributed to people who don’t have the appropriate clearance simply to make the damn thing work, this is what happens.
You can think of this as a hierarchy as well; my personal way of thinking about it is:
Defence of the Realm Compromise will have serious adverse consequences for the strategic survival of my country/corporation/family whatever level you are thinking about.
Life and Death Compromise will put someone’s life or health in danger.
Financial Compromise will allow someone to have money/car/boat/space shuttle stolen.
Commercial Compromise will cause loss of future financial gain.
Humiliating Compromise will cause embarrassment. Of course, if you are a politician this is probably No 1.
Personal These are details that you would rather not have released but aren’t particularly earth shaking. I would put my personal medical history in here but, the impact of contravening privacy laws may push it up to Humiliating (if people find out) or Financial (if you get sued or prosecuted).
Private This is stuff that is nobody else’s business but doesn’t actually hurt you if they find out.
Public Print it in the paper for all anyone cares.
Irrespective of the level, you don’t want any of this data lost or changed but if it is, you need to know that this has happened. For the Nazi’s, having their Enigma cipher broken was bad; not knowing it had happened was catastrophic.
In the comments below, I have been asked to describe best practice. This is impossible without knowing the risk of the data (and risk tolerance of the organisation). Spending too much on data security is as bad as spending too little.
First and most importantly, you need a really good, watertight legal disclaimer.
Second, don’t store the user’s data at all.
Instead when the user submits the data (using SSL), generate a hash of the SessionID and your system’s datetime. Store this hash in your table along with the datetime and get the record ID. Encrypt the user’s data with this hash and generate a URL with the record ID and the data within it and send this back to the user (again using SSL). Security of this URL is now the user’s problem and you no longer have any record of what they sent (make sure it is not logged).
Routinely, delete stale (4h,24h?) records from the database.
When a retrieval request comes in (using SSL) lookup the hash, if it’s not there tell the user the URL is stale. If it is, decrypt the data they sent and send it back (using SSL) and delete the record from your database.
Lets have a little think
Use SSL - Data is encrypted
Use username/password for authorisation
IF someboby breaks that - you do have a problem with security
Spend the effort on fixing that. Disaster recover is a waste of effort in this case. Just get the base cases correct.
I'm interested in your advice/opinion on this security problem.
I was thinking on doing something like this:
Get hash MAC (sha256) from string built from userId + expirationTime and as secret key string built from some secret string and $_SERVER['HTTP_USER_AGENT'].
Get hash MAC (sha256) from userId + expirationTime and as secret key previously made hash (from step 1).
Build string from userId|expiration| and previously made hash (from step 2).
Encrypt given string (from step 3) with 'rijndael-256' algo. (mcrypt family of functions).
Encode to base64.
Set cookie with given value.
What do you think. Is this ok?
What else could I implement with $_SERVER['HTTP_USER_AGENT'] check, to make sure that the cookie isn't stolen (except IP address)?
P.S. From sensitive data cookie would contain only userId.
EDIT:
Ok to clear some things.
I'm trying to make "safe" auth system that doesn't rely on sessions. The app in question is build more or less as pure restful api.
Step 2:
Problem:
"Fu’s protocol does not provide an answer to this
question. There is only one key involved in Fu’s proto-
col, namely the server key. One straightforward solu-
tion is to use this server key to encrypt the data field
of every cookie; however, this solution is not secure."
Solution:
"Our solution to this problem is simple and efficient.
We propose to use HMAC(user name|expiration time,
sk) as the encryption key. This solution has the fol-
lowing three good properties. First, the encryption key
is unique for each different cookie because of the user
name and expiration time. Note that whenever a new
cookie is created, a new expiration time is included in
the cookie. Second, the encryption key is unforgeable
because the server key is kept secret. Third, the encryp-
tion key of each cookie does not require any storage on
the server side or within the cookie, rather, it is com-
puted by a server dynamically.
"
From paper "A Secure Cookie Protocol" by Alex X. Liu1 , Jason M. Kovacs
Step 4:
Encrypts data (which would look something like this: 'marko#example.com|34234324234|324erfkh42fx34gc4fgcc423g4'), so that even client couldn't know exactly what's inside.
Step 5:
Base64 encode is there just to make final value pretty.
I'll bite.
In order to maintain any semblance of state you need to identify the user using a key of some type. That key is sent to the browser as a cookie OR through query string parameters.
Now, the validation of that key can occur inside the web server itself (session) or through checking some other storage mechanism, usually a database record.
The key itself should be obfuscated using some mechanism. The reason for the obfuscation is simply to make it harder to guess what values other keys might have if the originating user or someone else decides to inspect the value. For example, if the key is your user id (not recommended) and you are using incrementing ints then it's trivial to guess the other user keys. I want to stress that obfuscating ( or even downright encrypting ) the key provides absolutely no protection against a hijacked session. ALL it does is make it harder to guess other peoples session keys.
That said, I believe the key should have nothing at all to do with your user id and instead be some other near random value like a generated GUID. Quite frankly a base 64 encoded GUID is at the exact same level of security as encrypting user id + time. It's just that one is more computationally intensive on your server than the other.
Of course, this key could change upon each request. Browser posts something, you generate a new key and send it back. In the event the browser posts an out of date key then log it and kick them back to the login screen. This should prevent replay attacks .. to a degree. However, it introduces other challenges such as using the Back button on various browsers. So, you may not want to go down this path.
That said you can't depend on the client IP address because the same user might send follow up requests using a different IP. You can't depend on browser fingerprinting because any decent hacking tool will capture that and submit the same values regardless of whatever they are using.
Now, if you really want to do this right you should have SSL turned on. Otherwise you're wasting your time. The entire conversation (from the login screen on) needs to be encrypted. If it's not then someone could simply listen for that cookie, replay it immediately and hijack the session. Point is that they don't need to know the values contained therein to use them. So all of that hashing, etc you have is just fluff that will increase your server load.
Did I say use SSL? ;) This will encrypt the traffic from the beginning of the conversation and an attacker cannot replay the same packets as they would have to negotiate their own handshake with the server. Which means all you have to do is ensure that whatever session id you use is non-guessable so that one logged in user can't take over another's session.
So, to sum up: the method you posted is a waste of time.
You are much better off just getting a $10 SSL certificate and using a base 64 encoded GUID as the session ID. How you store that session info on your server doesn't really matter... except in load balanced situations. At which point it needs to be out-of-process and backed by a database server.. but that's another question.
#Marko A few comments about how secure this kind of "session in a cookie" approach is:
First of all, as said by others as well, you need a secure connection. There is no realiable way around this requirement. It is a must.
Other than that, there are quite a few pitfalls regarding to implement a secure encryption/authentication system. For example you need to make the MAC verification "constant-time", you need to pay attention how do you implement the encryption/authentication (mode of operation, IV creation etc.). And so on.
If you are unsure about such issues, I recommend you to take a look at TCrypto (which I maintain):
TCrypto
It is a small PHP 5.3+ key-value storage library (cookies will be used as a storage backend by default). Designed exactly for (scalable) "session in a cookie" usage. Feel free to use it :) Also, if you are interested about the low-level implementation, take a look at the code. The codebase is not that huge, I guess it would do quite well, demonstrating encryption related code usage in PHP applications.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I'm making a web application and would like to have a secure area where you can only sign in with your finger print. My original idea was to just use a usb barcode reader and you scan that, and it outputs the ID into a text box, but that's not very secure. So I want to use a USB fingerprint reader to maybe generate a hash for each person and store that in a text box instead. Which then will check that against a database with everyone's hash value. Does anyone know if there is a fingerprint reader out there that can do this, or of a way I can easily integrate a reader into a website?
EDIT:
The Idea for this website is that it's like a login system (Sort of like how you can clock in and out if you get paid hourly) The idea is that no one can sign someone else in and out. IF you just use a password, than someone can just tell their friend the password who can type it in. That's why I thought of a fingerprint, or something similar... I'm open to other suggestions
Also, I'm using PHP
EDIT 2: The Basic idea, is I have to come up with a way to Prove someone was there by checking in. I don't want to use passwords, because then someone can just tell someone else their password to type in. Any other suggestions? It doesn't have to be fingerprints.
Biometrics are a very bad way to do authentication for many reasons:
They're essentially just a password that you can never change. (At least not without some serious pain!) With traditional password schemes, if your password gets stolen or guessed, you can at least change it. But if someone steals your fingerprint, now what are you going to do?
Biometrics are not secret. Every time you touch something, you're leaving your password laying around. Every time your picture gets taken your facial image / retinal image gets copied. Passwords have to remain secret to be useful.
Like Borealid said, Biometrics are never scanned exactly the same twice. So when you do matching, there has to be some kind of fudge factor in allowing input. This:
Just makes it easier for attackers to copy your data and replay it, since they don't have to get an exact match. An attacker only has to get close to get accepted.
It forces the authentication server to store the your biometric info in plaintext. You can't hash biometric data like you can passwords, since then you'd have to match exactly with the hashed value.
So don't do it!
You can't do what you want, exactly.
Fingerprints never exactly match. Even if you scan your own right index finger twice in a row, the scans won't be the same. So "hashing the fingerprint" won't work - two hashes of the same finger would be indistinguishable from two hashes of two different fingers (with a good, cryptographically strong hash).
Fingerprint readers work by storing some key onboard, and letting that key out if and only if the fingerprint given is close enough to what they expect. The fingerprint itself is not used to get direct access to anything outside the reader.
Sending the fingerprint as seen by the reader over the network is not acceptable - people are nervous about giving their prints to police. You think they'll be OK giving them to you?
Also not acceptable is having the reader say "finger 2 is OK". This could be easily spoofed.
Instead, have your user use X.509 (SSL) client certificates to access your site. They may, if they wish, control access to their private key via the fingerprint reader.
EDIT: updating this answer. There is, now in the year 2014, a standard from the FIDO Alliance called "UAF" which allows sites to use fingerprint authentication in a way that works across different sites. Rumor has it Paypal is going to begin using it soon.
Biometry for remote authentication is never secure. You can't know if a real finger with that fingerprint is in a reader, or if the user only sends you an image. So a fingerprint just becomes a password the user can never change, which is the same for every service you login to, and which is left on every object the user touches.
Biometry can only work for local authentication where you trust the reader to be not hacked(i.e. you have physical control over the reader) and the reader can distinguish a real finger/eye/... from a fake one. Which most can't.
It is possible to create a one way hash of a finger print. You first have to extract a number of observables just like you might do for a fuzzy match. But since you need an exact match for a hash you need to throw error codes at the values you extracted which then can correct the minor differences on each measurement. It's not easy to code and it doesn't fix the fundamental problems listed above, but it should be possible.
So you want to use a local authentication mechanism to authenticate a remote resource? There are many issues here to suggest this would not be a smart choice. E.g., how does the web app know the hash belongs to the original user and not someone who has a duplicate?
What I would suggest instead is to go the route of banks from several years ago when they had smartcard readers sent to credit card customers. Use the fingerprint scanner to store a local copy of the user's login name, requiring a second form of authentication such as a password.
This is so bogus. Why can't you encrypt a fingerprint? Your master is stored encrypted, you send an encrypted one to the server. The server software un-encrypts both and does the comparison. Simple. Your 'expert' solutions are bogus.
why not use face match software for authentication.
http://www.oculislabs.com/products/privateeye
I'm trying to make a "normal" username/password login form secure, without needing HTTPS. My idea is this:
Server generates a keypair for some kind of assymetric encryption algorithm. It stores this keypair in a temporary table of sorts (or perhaps the local session data).
Server sends the form to the client and includes the public key.
User fills in the form.
Before it's sent to the server, Javascript encrypts the password using the given public key.
Form is sent.
Server decrypts the password with it's private key (which it gets from the temporary table, using the public key to find it).
What I need to know for this is:
Which encryption method is the best to use? RSA?
How can I decrypt the password in PHP?
And probably the most difficult one, how can I make Javascript encrypt the password?
In advance: I'm sorry for being negative, however;
Implementing your own security protocol is never a good idea, unless you're a highly trained security expert, or you actually don't really care about the security and only want to create an impression of security (marketing) and stop the script kiddies.
SSL is definitely not a fingerprint lock, as so say in your comments, JCryption and your proposal are equal to having a door where you can enter a two-digit code to open the door and you have infinite many retries. It's hard to break if you're not really interested and just passing by, but if you want to get in that house (and you probably do, else security wouldn't be needed), you will get in.
Another point is that people often forget to mention what they want to achieve. Security has the famous three components called CIA, namely confidentiality, integrity and availability. Is it for you important that the data you transport is confidential, or is integrity important (i.e. you're sure that the sent data comes from the one you expect and not the man in the middle)?
To make it concrete in this case, the only thing that you achieve here is that a passive attacker cannot see whats passing by on the line. As soon as your attacker gets active and changes the messages on their route, your whole security falls apart. So my advice would be to just stick with the solution the experts have come up with (TLS in this case, not ssl since that is the old version of it) and just make sure your server supports it.
edit:
Btw, SSL/TLS cannot work without certificates. The whole point in public key crypto is that there should be at least somewhere some trusted party.
On the other hand, if you don't care that your users will get an "invalid certificate" message, you can just create your own certificate which is really easy. In that case your certificate isn't trusted by the browsers, however, you can be sure that at least your communication is safe (okay, there are exceptions in this case, but still ...)
The argument that certificates should be for free is really from a perspective point of view. I think people who claim it is bogus/idiotic don't know what it takes to be a certification authority. These companies invest millions in order to keep the communication secure, and sure they make nice money out of selling certificates, but hey its their job and they also deserve to make money, just like any others.
edit2: after comments
I indeed say that you have a secure communication. However, you miss the point that with self signed certificates you dont know to whom you talk securely. Imagine a dark room which is completely isolated from eavesdropping a conversation. Now imagine the difference between such a room with and without light. If the room has light, you can actually see to whom you're talking securely and only choose to talk to people you like to trust. Now imagine doing the same in a completely dark room. You can only hope that the guy you talk to inside this dark room is just an ally and not your adversary. However, you cannot know that, just hope that it's ok. And although your conversation itself is secure, nobody can listen in, you still dont have "full" security.
If I, being a crook, do a man-in-the-middle attack, I can create a self signed certificate without the user noticing. So the advantage of using TLS with self signed certificates is that you have at least the implementation of the protocol corrent (and even implementing this is far from easy). Moreover you can avoid the ugly warnings by advising your users to manually trust the certificate once. However, this is only possible if you have a relatively small group of returning visitors, for a public website this is not really a solution.
This doesn't seem that secure from the perspective of the client. Two (related) problems:
How does the client trust the server? How can it verify that the key the sever's presenting is the one that belongs to it?
It's possible to do man-in-the-middle attacks. A malicious proxy could strip out and store the public key before the client sees it, substituting its own, then decrypt the password when the client authenticates, store it, and re-encrypt and send the response on so the client doesn't realise something's up.
What's wrong with ordinary SSL? There has to be a consensus that it's secure, otherwise vendors and organisations would drop support for it overnight. By contrast, most attempts to invent a funky new way to do security "on the cheap" usually miss something fundamental.
It looks like a lot of what you want to do is supplied by the jquery plugin JCryption. It even assumes PHP as the backend, so a good fit for you.
Livejournal does something similar to what you want where:
Server generates a challenge string, inserts this into form. [1]
Client generates response by MD5 hashing the password, then MD5 hashing the previous hash with the challenge prepended [2].
Server gets response, checks challenge validity, then does same as step 2, comparing the result to the response.
This is a very good idea, and it's already been done. See jCryption.
jCryption looks interesting, I've not seen it before.
But I have to ask what is wrong with SSL?
Encryption code is notoriously hard to do right, and you can bet that the SSL implementations found in browsers and http servers are much more rigorously tested and reviewed than the jCryption stuff.
That said, jCryption looks neat if you absolutely need to avoid SSL, and you're not dealing with super-sensitive information.
By storing the passwords in encrypted method on the server the server can retrieve the passwords and verify checksum sent by client. Send a session password and ask client to make a hash of session password and the user inputted password, do the same on the server and compare the two hashes.
This will not secure users from MITM attacks - local admins, NSA, telecom, router hijacks, but it will keep the password safe in open wlan.