Secure Authentication without SSL

Secure Authentication without SSL - php

I thought of an authentication system without SSL that seems reasonably secure. Am I overlooking something important?
User hits the login page
Server generates a salt for transmission (t-salt) and stores it in the session
Server sends the t-salt to the user as part of the login page that loads
User types in their username and password and clicks submit
Browser MD5 encrypts their password along with the t-salt
Browser sends username and MD5 (password + t-salt) to the server
Server retrieves password from database using username (*) Note below
Server MD5 encrypts password retrieved from step 7 along with the t-salt that was stored in the session in step 2
Server compares both of the MD5s from step 6 and step 8
If they are identical, the login is successfully authenticated
The server removes the t-salt from the session (added in step 2) to prevent potential replay attacks
* Note that the password retrieved in step 7 cannot be 1-way encrypted (as is common practice) in order for step 8 to work. But 2-way encryption systems can still be used to secure passwords at the database level. (Hey, that comes with the side benefit of allowing a more user friendly password recovery process.)
Aside from my note immediately above, what are the strengths and weaknesses of this scheme?

This'd be wide-open to man-in-the-middle attacks. Nothing would stop an attacker from sniffing the link and getting the salt as it goes from server->client or the hashed password as it goes from client->server, and replaying either.
Invalidating and generating a new salt after each attempt would prevent simple replays, but then it comes down to a race condition - can the attack submit their login attempt before the user? Even then, since the attacker's sitting in the middle, they could presumably interrupt the user's link, capture the salted password, and submit it themselves and capture the session.

You send the t-salt and the hashing algorythm. It wouldn't take long to calculate the password inside the hash.
You should reconsider SSL in my opinion.

While I think that your intentions are good, the fact of the matter is that there really is no security offered at all in your approach. As others have pointed out, any reasonably competent hacker will be able to intercept data going over the wire and execute a replay attack. That is not to mention the fact that any data going over the wire will be unencrypted, which exposes your users' potentially sensitive information to anyone.

The problem with this is that you're making the assumption that SSL is purely about encryption. Take a look at SSL is not about encryption. On this basis, your scheme falls apart at step 1 because you have no assurance that when the login page is loaded it has actually come from the site you think it has and that it hasn't been tampered with.
There are many precedents of this happening, the Tunisian government harvesting usernames and passwords is a good one and you'd be wide open to this style of attack as your login page could be altered before it even hits the browser.
Then of course you have the Firesheep problem in that your auth persistence (which I assume you do via cookies), is going backwards and forwards across an unencrypted network. It doesn't matter what you encrypt inside these cookies, if someone is able to grab it and reissue a request (very easily done at a public wifi hotspot), then you've got a session hijacking problem.
Then there's also the known weaknesses in MD5 but even using a more secure hashing algorithm won't save you from the other problems described above. Spend a tiny bit of money, do some minimal configuration and make SSL part of your login process. Refer to the OWASP Top 10 on Insufficient Transport Layer Security for more info.
Finally, SSL is not intended to be a panacea; it won't protect you from key loggers, it won't protect you from having your database breached and it won't protect you from someone whacking your end users over the head with a wrench until they disclose their password. But it's not meant to do any of these things, it's simply intended to be one more layer of defence - albeit an essential one - which is part of a broader security strategy.

One thing to think about is that SSL doesn't just provide confidentiality and integrity protection to the overall data stream (via encryption) it also provides a level of validation of the identity of the server.
In your example there's no way (that I can see) for the client to validate that they are speaking to the real server before providing their password, as such a DNS spoofing attack or some other MITM attack would be effective.
Also as mentioned, it would be possible to brute-force a users password quite easily as the attacker can intercept the salt going from server-->client and then the hashed password coming back. MD5 being a fast hash algorithm it would likely be quite an effective attack against standard user passwords.

The data transmitted while authenticated won't be secure, and implementing your own schemes is usually a pretty bad idea.

Related

Sending passwords to server in a secure way

I am currently working on a better login-routine for one of my websites and I wanted to adress securely transmitting login-Data to the server. There are several articles and posts concerning this topic, but they frequently have quite different opinions on how it has to be done.
Starting off with SSL, pretty everyone is on the same page here: You should use SSL, and I do. Then there are those who say: "That is enough, with SSL you can send username & PW as plaintext". I disagree. Other say that it should still be hashed. And I read several posts where I had the feeling that people were concerned about different aspects of a login routine and proposed mechanisms to only handle their security aspect.
So what I want to know, if the routine I have so far elaborated would be adequate, too much or too little. I will try to explain why I chose to implement a certain feature and which security aspect I try to cover:
SSL:
The communication between server and client should always be https:// - nevertheless I read several articles warning that SSL is "no silver bullet", but it is a good start.
Hash PW clientside (SHA3, ARGON2i, BCRYPT):
Many comments did reject hasing PW. Using a hash, comparing it to a HASHed PW in the database would simply change the PW from the userinput to the HASH - an attacker would still have access by simply getting hands on the HASH. I agree. BUT (this is what I meant that people read about different aspects of security) the ones claiming that it is better than sending plaintext because in that case ONLY your system, but not other systems with the same PW would be compromised (unless, of course they use also hashed PWs). So I would implement HASHing of the password before sending it via SSL.
encrypting the HASH:
Let's assume SSL could not hide the data we send to the server and an attacker would read the HASHed PW. The only way that I could think of to adjust the security to this scenario, would be to encrypt (e.g. AES CBC) the client-side HASHed PW with a key that has been send beforehand by the server, and that has a short expiration period. The key would have to be generated randomly. Like that, the server can decrypt the data, and then compare the HASH with the one it its database.
To sum it up:
-> Clients wants to Login via SSL -> Server sends back a key -> Clientside hashing of the PW -> clientside encryption of teh HASH with the key and a random IV -> Server decrypts the data with the key (stored in $_SESSION, with an expiration timestamp) and compares the HASH with the HASH in his DB (if the expiration timestamp is still valid).
Would this be a good approach? Or is this too much? (Can there be too much security?) Or do you have any alternative solutions?

Or is this too much? (Can there be too much security?)
You're talking about it like security is a liquid that must fill-up a container without overflowing it. That's not how it works and you're asking the wrong questions, which means you're trying to solve the wrong problem. It has nothing to do with the amount of measures that you pile up, but whether and how they address a specific problem.
If the problem is protecting data in transit, then the solution is TLS (SSL) - that's what it is specifically designed for, and anything that you can come up with would, in the very best case scenario, be a poor alternative to it. You can't outsmart the decades of research and practice that has gone into TLS.
That was already answered by Jay Blanchard though ... I want to point out the mistakes you've made, because otherwise it looks like one man's word vs. another (and you may listen, but other readers might not):
SSL:
The communication between server and client should always be https:// - nevertheless I read several articles warning that SSL is "no silver bullet", but it is a good start.
It is both a silver bullet and not a silver bullet, depending on how you look at it.
When we talk about protecting data in transit, it is THE solution - a silver bullet in a way.
But that doesn't mean flaws won't be found in time, or that you just switch it On and say "I have TLS, I am secure!" - no, it still requires proper configuration, maintenance and adjustments over time. In that sense, it's not a silver bullet.
It also doesn't solve many other security issues, so when someone asks "How do I make my application secure?", of course you would say that it is not a silver bullet - many threats need to be separately addressed and there's no one-stop shop for them all.
Hash PW clientside (SHA3, ARGON2i, BCRYPT):
Many comments did reject hasing PW. Using a hash, comparing it to a HASHed PW in the database would simply change the PW from the userinput to the HASH - an attacker would still have access by simply getting hands on the HASH. I agree. BUT (this is what I meant that people read about different aspects of security) the ones claiming that it is better than sending plaintext because in that case ONLY your system, but not other systems with the same PW would be compromised (unless, of course they use also hashed PWs). So I would implement HASHing of the password before sending it via SSL.
It's the exact opposite - when you hash the password on the client side, that only makes accounts on your site easier to compromise after a data breach.
Lookup the hash in the database - got the password right there, that's the part you figured out. But that hash is still the result of a certain user-provided string ... Nothing stops an attacker from applying the same techniques to break the hash in order to compromise accounts on other servers.
So, this doesn't in any way solve the problem, but you'll probably think that in the worst case it doesn't do anything bad ... Well, indirectly it does - you have to make a considerable effort, to implement something with a lot of potential for mistakes.
In the best case scenario, you're just wasting your time, but a minor mistake can be a major vulnerability.
Also, SHA-3 is a cryptographic primitive - it has many appliances, but mostly as a building block. You can't just put one round of it on a password and be happy with the resulting hash.
For a comparison, bcrypt uses Blowfish internally (as a primitive of the same kind as SHA-3), but you cannot equate Blowfish to bcrypt.
encrypting the HASH:
Let's assume SSL could not hide the data we send to the server and an attacker would read the HASHed PW. The only way that I could think of to adjust the security to this scenario, would be to encrypt (e.g. AES CBC) the client-side HASHed PW with a key that has been send beforehand by the server, and that has a short expiration period. The key would have to be generated randomly. Like that, the server can decrypt the data, and then compare the HASH with the one it its database.
There are valid reasons to encrypt password hashes, but not for this purpose and certainly not on the client side.
You need a secure key-exchange protocol for this to even work. And guess how you do that? TLS.
There's literally nothing different between communicating an encryption key or a password over the wire. So, even if this was somehow a solution to protecting a password, how would you apply it again on the key itself? It makes no sense.

SSL is just fine, I have no idea why you disagree. Client-side hashing still leaves the PW visible on the client-side as well as the hash, so nothing gained there.
The question comes down to, "what are you protecting?" My guess is that you're not guarding anything needing more security than banks, and probably way less than that.
You're spending a lot of time trying to re-invent the wheel here, rather than relying on tried and true methods. Stick with what is proven.

-> Clients wants to Login via SSL -> Server sends back a key -> Clientside hashing of the PW -> clientside encryption of teh HASH with the key and a random IV -> Server decrypts the data with the key (stored in $_SESSION, with an expiration timestamp) and compares the HASH with the HASH in his DB (if the expiration timestamp is still valid).
Why encrypt a hash? That implies that the hash isn't secure enough. OK, fine, let's go with that. So let's assume an attacker is able to read the hash, and that's the reason you want to protect it with an additional layer. If an attacker is in a position to read the hash, they're also in a position to read the key sent by the server to the client, and the Javascript which contains the encryption algorithm (assuming you're talking about an HTML scenario here). Now the attacker has everything to replicate and reverse the encryption, and in fact they are probably also in a position to alter the Javascript sent from the server to the client in the first place.
To protect against that happening, you'd need some wrapper that protects all the communication between the client and the server, like, oh, uhm, say… SSL.
Since SSL already protects the communication from 3rd party interference… what do you think you're adding with that additional song and dance exactly? I'll tell you: nothing.

Is this mechanism to authenticate a user secure?

I am working on a website to learn more about web programming and also to launch same as a start-up. The first problem that i came across was how to implement a secure login system. Currently i have take steps like escaping the password and then hashing it, using a salt. But i was wondering is the following mechanism secure,
I will make the user enter username and will keep checking if the user has entered his username (when textbox loses focus or a button for submitting username, also to prevent listing of usernames ill block the user by setting cookie on system if multiple incorrect attempts are made, or maybe ill use captcha for each ), once entered username, ill send back random stored salt to user.
Using that salt and password entered, user will hash the password and send it in form
I'll verify the password by comparing hashes
I think it will be beneficial since server side i don't have to do any processing, and hence i don't have to worry about DoS attack, as i read somewhere that using some slow hashing like BCrypt will expose site to DoS attacks.
Also user's password is never communicated over the network, making it secure from person sniffing the network.
Please do point me to some reference or anything that may help me to implement securely. Consider me noob because i have still started learning, and would like to know what you think of it, what possible flaws are there ? and what must be secure strategy.
UPDATE-
Many answers i am getting, all usually assume that i am thinking this as an alternative to SSL; no it's not the case. By protection against sniffing i meant protection just in case some attacker might make the user use a SSL proxy.
just for reference - https://security.stackexchange.com/questions/19616/why-is-it-possible-to-sniff-an-https-ssl-request

Client side hashing can have its advantages, but you cannot do without server side hashing. In your scenario, the calculated hash acts as the new password. An attacker with read access to the database (SQL-injection) will see this hash and can use it directly as password to login.
Using a slow hash with a cost factor is mandatory, usually it is done server-side, because client-side languages are slower and can do less rounds. Of course somebody can use it to make a DoS attack, but this can be done with every other page as well. The size of the password doesn't matter (as one can read ocassionally), because after the first round only the hash will be hashed.
If you plan to do a client-side hashing, don't forget to calculate a (fast) hash on the server as well. And you have to ensure, that the hashing was done correctly client-side. Much more important is, that you use SSL to send the credentials.
The question Secure authentication: partial client-side key stretching… could be of interest to you.
EDIT:
I will try to sum up the important points for client-side hashing.
A slow hashing algorithm with a salt and a cost factor (BCrypt/PBKDF2/SCrypt) is mandataory, this is the only thing that makes it hard to retrieve the original password from a hash, if the password is weak. It is possible to do this client-side.
Server-side hashing is mandatory too, to prevent an attacker from using stored hashes directly as passwords, if he knows them. The hash can be fast without a salt (SHA-256), because the input (BCrypt hash) has enough entropy. Such a strong "password" with 60 characters cannot be successfully brute-forced.
If the attacker cannot crack the fast SHA-256 hash because the input is too strong, he can try to brute-force with the original passwords (from a dictionary). But to do this he would first have to calculate the slow BCrypt hash and afterwards the fast SHA-256 hash.
Client-side languages like JavaScript are usually interpreted and much slower than compiled code, so you can do less rounds in the same time as you could do on the server (this weakens security). If you have the possibility to run native code on the client, there is no disadvantage to do the slow hash client side.

no, you should not send any 'salt' to the user.
It can be sniffed.
What you're basically doing is a send something like a (csrf-)token that can be used once. Nothing wrong with that, but you seem to be re-inventing the wheel.

Seriously, I think your solution is good only for hackers. If I sniff communication I will get gradually username, salt and password hash. You have to send all those values over the network (username to get salt, password hash to auth attempt). Now I can use sniffed password hash directly in malicious login request or start cracking password locally (users usually have same password for more services). All checks and limits on auth attempts are out of game because I don't need send request to guess password. Depends on hash algorithm it will more or less time consumption. I think network sniffing is the main purpose why you calculate with not sending plain password over the network.
You can secure your network communication with TLS but then all things with sending salt and hashing password on client are unneccessary. You can just send password in plain text to server. But yes, hashing password on client, why not if you want. You can use ie. sha1 also on server if you think that bcrypt is performance issue and possibly DOS cause.
Good example of protocol that is used for transmitting infomation through unsecure network is OAuth 1.0a and even in it you need some cryptography or TLS for transfering consumer secret.
Let me know if I am something understood incorrectly.

I guess only drawback i can see is using this on low-end mobile devices.

android, ios and php encryption/decryption [duplicate]

For a webapplication, when HTTPS is not available as a security measure, is it possible to still make the login somewhat secure? E.g.:
Tokenize logins, to make repeat attacks difficult?
Somehow encrypt the sent password from a HTML password field?
In particular I'm using CakePHP and an AJAX POST call to trigger authentication (includes provided username and password).
Update on the problem:
HTTPS is not available. Period. If you don't like the the situation, consider it a theoretical question.
There are no explicit requirements, you have whatever HTTP, PHP and a browser (cookies, JavaScript etc.) offers in real life (no magic RSA binaries, PGP plugins).
Question is, what is the best, you can make out of this situation, that is better than sending the passwords plaintext. Knowing the drawbacks of each such solutions is a plus.
Any improvement better than plain passwords is welcome. We do not aim for a 100% l33tG0Dhx0r-proff solution. Difficult to crack is better than complicated to hack which is better than a trivial sniffing revealing the password.

It is a bad engineering practice to reinvent the wheel. Engineers who do this are falling victim to the "Not Invented Here" bias, which can cause a lot of damage when it is a security critical system.
SSL/TLS which is behind HTTPS is absolutely vital in maintaining a secure connection between a website and a browser. Public wifi networks put users at risk, and when used correctly, HTTPS is the only tool that can protect user accounts from this vulnerability.
In the case of two clients that need secure end-to-end (e2e) encryption then there is the open source and vetted Signal Protocol which has received number opens source ports on github and a wide adoption from popular apps like WhatsApp. There is no need to brew your own, these protocols work well for a reason.
If your host doesn't support HTTPS then a service like Cloudflare Universal SSL can be used to ensure all browsers connect to your site using HTTPS, even if your server doesn't support SSL/TLS. The connection between Cloudflare and your website will still be unprotected, but this Cloudflare service is intended to protect users against threats found on public wifi networks. From the perspective of a penetration tester, not providing HTTPS is highly suspect, if you aren't providing a basic security requirement as delivering traffic, then what other security requirements are you missing? HTTPS certificates can be obtained for free using Let's Encrypt or Start SSL, there is no legitimate reason not to support HTTPS.
HTTPS is vital because it does lot more than just "encrypt passwords". Another important role is that it should prevent the user from giving logging into a malicious server that is impersonating a real server. Using a system to protect the password alone is still a violation of OWASP A9 - Insufficient Transport Layer Protection because you would still be transmitting session credentials in plain text which is all the attacker needs (Firesheep).
JavaScript-based cryptography cannot be used to construct a secure transport layer.
"Tokenize logins": If an attacker is sniffing
the traffic, they'll have the plain text username/password and then
they can just login with these new credentials. (Replay attack)
"Somehow encrypt the transmitted password": After the person has logged in
an attacker can sniff the traffic to get the valid session id
(cookie) and then just use this instead of logging in. If the
entire session was protected with SSL/TLS then this is not a problem.
There are other more complex attacks that affect both this system and our current SSL infrastructure. The SSLStrip attack goes into greater detail. I highly recommend watching Moxie Marlinspike's Blackhat 2009 talk, which lead to the HTTP-Strict-Transport-Security standard.

Since you cannot do SSL at the web server, and you are not a security expert, look for an existing secure authentication service that you can utilize, and let them handle both the SSL and the complexities of handling credentials for you.
In particular, I would suggest that you use a free third-party authentication service, such as OpenID. They have libraries for PHP including one for CakePHP.
Edit: (about risks)
While using a 3rd-party secure authentication service (that uses HTTPS itself) can mitigate the problem doing authentication itself without using HTTPS (on your server), it does not entirely eliminate the possibility of attacks.
The most common two attacks would be replay attacks, and session-hijacking where the attacker is able to either re-uses a genuine login session token later, or use a valid session token for their own malicious purpose.
The replay attack can be mitigated by having the session token expiry, and preferably by using a nonce to prevent session replay and to reduces the risk of session hijacking. With a nonce, a legitimate session generates an error if successfully hijacked, because the nonce has expired (been used), so their own session is no longer valid.
If you cannot use HTTPS to encrypt the session token while being transmitted to and from your server, you cannot entirely prevent active attacks such as session-hijacking or man-in-the-middle attack. This may be acceptable in some cases, such as websites with a small user base for non-commercial usage.

The short answer is that without SSL endpoint to endpoint encryption, it's impossible to do it securely...
One of the primary reasons for this is that you can't do secure crypto in a browser. See this reference - Javascript Cryptography Considered Harmful.
Additionally, there's no way that you can be sure that the source of the credentials are indeed who you're talking to. Meaning that there's absolutely no way without SSL to be sure that there's not a Man-In-The-Middle Attack going on.
So no, you can't do it.
Additionally, don't even try. Get SSL. You can get free certificates. Hosts will usually give you a dedicated IP for a few $$$ per month. And if you really care about security, you'd be using at least a VM with a dedicated IP address anyway.
To even attempt this would be Security Through Obscurity at best, and nothing at worst. SSL is a solved problem. Why not use that solution. Security is not something to guess at. Use the proper techniques. Don't try to invent your own. It won't work...

As you suggested, you may be able to generate a unique token each time the page is created. That same token would need to be sent back with the form data and could not be reused. You could also keep the password safe by using JavaScript to hash it, if you can rely on it being enabled by your users.
This scheme is still not secure, however. An attacker could still see everything going across the wire. They could intercept the token and send a response back to you before the user does. Or they could just wait for someone to login, steal that person's credentials (as they are sent over the wire), and just make their own login request later on.
Bottom Line - you need to use HTTPS to guarantee the site is secure.

You can encrypt the password using Javascript and decrypt it on the server.
I would recommend generating an RSA keypair on the server, send the public key along with a timed salt to the browser, then encrypting the password, combined with the salt, using the public key in Javascript.
You can find an RSA implementation in Javascript here
You should include both the IP address and the entire X-FORWARDED-FOR hedaer in the authentication cookies to prevent cookie theft behind proxies.
If you're dealing with sensitive data, you could generate a random AES key in Javascript, then send it to the server along with the password encrypted with RSA.
You could then make the entire application use encrypted AJAX requests from a single page and not use an auth cookie at all.
Note that it is not possible to protect against an active man-in-the-middle attack without SSL. An active attacker can completely replace your site with his own proxy, and there isn't any way to defend against that. (Since there cannot be any known good code)

You can use HTTP Digest authentication, which is supported by most browsers and does not send the password in clear over the wire.
The downside is the ugly log in box displayed by broswer. If you preffer to stick with forms, then you can implement exactly the same protocol as HTTP Digest in your forms authnetication: send hidden fields containing the realm and the challenge, and have the client add in JavaScript the nonce and compute the digest. This way you'll use a well known and proven exhange protocol, rather than roll your own.
HTTP Digest requires only hash operations.

Create a public/private key pair using an asymmetric cipher.
Create a symmetric key on the server.
Send the public key down to the client side.
Create a random key for the symmetric cipher client side.
Encrypt that random key using the public key client side.
Send the encrypted key to the server.
Server does the following:
a. Decrypts the random symmetric key using the private key.
b. Creates a token containing the generated client key.
c. Signs the token.
d. Encrypts the token using the server symmetric key.
e. Encrypts the already encrypted token with the client generated key.
f. Sends the encrypted token down.
The client receives this token and does the following:
a. Decrypts the token with the key it generated.
b. Stores the decrypted token.
c. At this point the stored token is only encrypted with the server symmetric key.
On every from the client to the server:
a. Encrypt the outbound data using the client generated key.
b. Send the token + encrypted data
On every request the server receives:
a. Decrypt the token using the server symmetric key.
b. Verify the signature.
c. Decrypt the data using the client generated key stored in the token.

What about HTTP Digest Authentication? It provides security by MD5-hashing username, password and a nonce (among other things) before sending it to the server. MD5 isn't really secure, but it's a good way for simple security with HTTP.
Of course this doesn't prevent hackers from changing the message... but it secures your password.

HTTPS has numerous use cases, most of which are designed to defend against Man-in-the-middle attacks. Anyone with a hacker's mindset will shudder to tell you that there is no way other than the established way to accomplish something. The fact is that just because you use TLS (the standard which modern HTTPS uses), does not mean you are using it well. Additionally, just using TLS does not prevent someone from exploiting known weaknesses. Just as you may be finding creative ways to secure your data, there are people who are finding creative ways to exploit your security measures.
So, what to do?
First of all, if you're going to forego TLS, it is helpful to understand how it works. And it is all about a handshake.
Once the client and server have agreed to use TLS, they negotiate a
stateful connection by using a handshaking procedure.[7] During this
handshake, the client and server agree on various parameters used to
establish the connection's security:
The handshake begins when a client connects to a TLS-enabled server
requesting a secure connection and presents a list of supported cipher
suites (ciphers and hash functions).
From this list, the server picks
a cipher and hash function that it also supports and notifies the
client of the decision.
The server sends back its identification in
the form of a digital certificate.[contradiction] The certificate
usually contains the server name, the trusted certificate authority
(CA) and the server's public encryption key.
The client may contact
the server that issued the certificate (the trusted CA as above) and
confirm the validity of the certificate before proceeding.
In order to
generate the session keys used for the secure connection, the client
encrypts a random number with the server's public key and sends the
result to the server. Only the server should be able to decrypt it,
with its private key.
From the random number, both parties generate
key material for encryption and decryption.[contradiction] This
concludes the handshake and begins the secured connection, which is
encrypted and decrypted with the key material until the connection
closes.
If any one of the above steps fails, the TLS handshake fails, and the
connection is not created.
Source: Wikipedia
So, is it possible? Yes. I was taught that anything is possible. It may be expensive, but it is always possible.
I want to fully disclose that I am NOT a security professional, just an enthusiast. I do not recommend attempting this for a production-grade project or anything other than your own edification. You should DEFINITELY check out this SO post which provides an excellent explanation as to roadblocks in setting up your own security protocol.
However, if you want to move on, here are some thoughts that come to mind. These are realities that will exist regardless of which direct you went with this project.
HTTPS is supported by all major modern browsers. Even with this reality, HTTPS load times are slower than plain HTTP. Without extensive production, it is highly likely your alternative implementation will be a fraction as secure while being significantly slower. This will be a drawback of any homegrown implementation unless you are utilizing browser features, which brings us full circle back to using TLS, which is what modern HTTPS utilizes.
If you manage to encrypt your password without TLS on the browser side using Javascript in an unpredictable enough fashion that an MiTM attack would be difficult, don't rest there. You also should be securing the data you send back and forth. Otherwise the password being encrypted really is irrelevant. Sure, an attacker might not know bobsmith109's password, but he doesn't need it, because he can sniff every single activity on the network. He knows what times bobsmith109 logs in, can probably trace his IP, and any other sensitive piece of data you send back and forth.
No matter what security measures you take, there is security in depth. So one thing that you can do right off the bat is make sure you encrypt your data in the database while also requiring strong passwords.
I reiterate that I am not a security professional and strongly discourage this as anything other than to satiate your curiosity. It is astronomically improbable that you can create a viable alternative to TLS without an extraordinarily large group of security professionals contributing to a project for years if not decades, which is what SSL/TLS can boast. That being said, a good starting point if you choose to go forward is to look at the handshake model above and see how you can implement a version of this without TLS.
I would be remiss to not share in my post that most real-life barriers to using HTTPS are being actively fought against. One of the largest - cost - is very close to becoming a non-issue. A free certificate authority will be coming out 2Q 2015 is supported by some big guns, including Mozilla and Akamai, to name a few. Here is an article.

Login without HTTPS, how to secure?
Since there is no secure channel between your server and your client:
because there is no secure channel, anybody can snoop your traffic.
because anybody can snoop the traffic, you are open to a MITM attack.
because you are open to MITM attack, there is no guarantee you client will see a legitimate page.
because the pages are not legitimate and your page is in effect not being served (the guy in the middle is serving the pages), all tricks used server-side are rendered useless.
What can you do? Theorically?
both client and server need to use encryption to make snooping / MITM less susceptible.
assume you cannot have a handshake,
assume your client already has your key and knows how to speak the same gibberish as your server.
how about some SSL over HTTP but wrapped in base64-encoded message for some gibberish?
But wait... Since you said no magic binary, or plugin, not even RSA, I don't know if any of this is possible save for (some potentially very weak) in-house encryption.
--

You can try to replicate it to some point, by using public key encryption (GPG maybe) and making use of browser caching.
This is not something secure, even just putting up SSL won't be enough for a sophisticated attacker, you need to make use of HSTS, public key pinning etc to just to consider a web site secure today.
The rest of the answer is just food for thought.
Create a public-private key pair. Keep private one secure.
Create a js file containing the public key and a encrypt function, find a secure encryption algorithm. This function should encrypt a given string (serialized form) with an additional timestamp, to avoid a replication attack.
Serve this file with Cache-Control:public, max-age=31536000 HTTP header. We try to mitigate when the attacker tries to replace the script. The file will always be served from the browser cache.
Send all the forms via Javascript, using the encrypt function. Serve these with the same header as above.
At the server side, decrypt the data, check the timestamp, if it's still valid. Do you thing, if not, discard it.
Create a cookie token which can only be used once for a very short amount of time. If the attacker captures a cookie, he won't have much time to do stuff. However, if the attacker is fast enough, then he might log the original user out.
Change the cookies with every response. But then what do you do when the user sends multiple requests at once and then they arrive in the reverse order? Which cookie is valid? This creates tons of problems at the cost of a false sense of security.
Any listeners won't be able to make use of the data going back and forth, and they won't be able to change/inject the existing JS files until the cache expires / user clears the cache. However, any sophisticated attacker can replace the whole HTML file which would discard all the security measurements I have just mentioned. If you can at least serve this file / form over HTTPS, you might get away with it, put them on github pages or whatever. However, if you put the file some other domain, then you need to set up CORS for the receiving domain for this to work.
Another try
One time passwords sent to email.
User fills out their email, clicks a link which then sends a link to their email with a token that will enable them logging in.
User clicks the link
Server checks the token, logs the user in.
Rolls the cookies like the previous example.
All in all, whatever you do, it is not secure. Given a fast, sophisticated attacker, nothing stands in the way.
Get SSL, if the infrastructure does not support it, change it. If your manager does not believe in SSL, convince him/her. Don't create a false sense of security. Protect your user's data, depending on your location, you are legally required to protect the data of your users.
Then let's talk about how to make a site secure with SSL.

Have a look at "The Secure Remote Password Protocol".
Instead of formulating it myself, let me quote from their webite:
The Secure Remote Password protocol performs secure remote authentication of short human-memorizable passwords and resists both passive and active network attacks.
and:
[The] protocol combines techniques of zero-knowledge proofs with asymmetric key exchange protocols and offers significantly improved performance over comparably strong extended methods that resist stolen-verifier attacks such as Augmented EKE or B-SPEKE.
Although the Stanford University doesn't provide implementations for PHP and JavaScript themselves, they link to some 3rd-party implementations.
One of those links leads to "Clipperz", which is an online password manager. It is also available as a community edition on GitHub. There they host their "javascript-crypto-library", which implements the protocol and the "password-manager" itself, which contains backends written in PHP and Python.
I can't say how difficult it would be to extract the relevant portions of code, but maybe you can reuse their implementation (it's licensed under AGPL).
Edit 2014/10/24:
Wikipedia's article on SRP lists some more implementations. Relevant for PHP/JS:
srp-client (JS)
srp-6a-demo (PHP/JS)

The best solution I have seen for somewhat secure HTTP connections is to use a Javascript implementation of md5sum (or some other hash) to avoid transmitting the password in plaintext. You can create a form onsubmit handler in Javascript that replaces the password field with a hash of the original value. This adds a modest amount of security to an unsecure connection, but relies on Javascript running in the browser to work properly.

I guess you care about secure transmission of password to the server? My answer is: dont transmit passwords to the server :)
Infact you may not transmit anything from browser (user) to server to authenticate the user, as an attacker who is spying http traffic would also be able to retransmit the data and authenticate.
Proposal:
Obvious solution would be to use a one-way, one-time transaction authentication originating from server; like a transaction number which can only be used once. Eventually, you still need a secure channel once to sync the list of transaction numbers with user.
You could use something google authenticator, yet you need a secure channel once to setup parameters on either side. If you consider email to be secure, that would be a way to go.

I have the same issue on a system of mine. I have taken steps to try and increase security without compromising the user experience with convoluted mechanisms. What I noticed was that the vast majority of users logged in from the same machine using the same browser, (but not necessarily the same IP address), or from a couple of browsers (eg: desktop or mobile). I decided I could use this to identify a pattern.
1) During registration, users are required to have strong passwords (to prevent dictionary attacks), a security question/answer and standard email verification (as proof of real person)
2) During login, after 5 failed login attempts (not before), a captcha is displayed to prevent brute force attacks.
3) Finally, I created a hash of parts of the user-agent string following a successful login, containing the users OS, browser (general not versions) and language - forming a sort of secondary password. If the useragent hash is significantly different on next login, the user is asked to answer the security question. Then, if this is answered satisfactory, the new UA string is hashed and added to their "safe machines" list, so that they wont be asked again from this machine. This is similar to a mechanism employed by the Steam gaming system.
This has been in use for over a year very successfully with about 700 users and it had the additional benefit of preventing "login sharing" - a problem where multiple users were using the same credentials for convenience!

The answer is shorter, and if you really matter about security you always have options that different levels of bureauocracy.
Absolut security does not exists. The number one flaw is always on the client side, with trojans ans keyloggers. SSL doesn't help with that.
1) Token generators: banks use them, blizzard uses then. It can be a device or an app. Well.. it's expensive.
2) SMS pins. interesting and affordable solution. There is a lot of good prices from trnasactional sms on the market and everyone has a phone capable of receiving it.
3) If you have to use HTTP, you can force a third party oauth service, like google or facebook. That's the best you can do without a token generator.

Use hashing mechanisms to store password and always compare the hashed password then nobody knows the real password even you.
It is very simple but it is effective.However, nothing is completely secure and there are some ways to broke the scurity layers.

Try this : On each request of the login page, send across a nonce and a timestamp.
While posting to server, send the following four details :
The username, the nonce and the timestamp in plaintext.
Then concatenate the above with a separator (Eg: newline) and encrypt using the user's password as encryption in chained-block-cipher mode.
On the server end use the username to lookup the password and verify the encrypted string.
Since the password is never sent across in clear, it is secure and the timestamp can be used to avoid a re-submit of the same data.
To avoid hijacking of session by obtaining the session key through a man-in-the-middle attack, the password or a hash of the password can be stored in-memory by the application on the client end and be used for generating unique session keys for validation by server.
Taking a look at OAuth 1.0 is also not a bad idea.

If you can't use HTTPS or you don't want to use HTTPS, consider using jCryption. jCryption offers encryption for the data being sent through HTTP requests (POST, GET etc.).
You can test the technique here: http://www.jcryption.org/#examples
If you're using Firebug, you'll see that all the data is encrypted.
It has jQuery library to encrypt the data on the front-end and a PHP library to decrypt the data in the back-end.

It is hard to secure the communication without a trusted third party, however, there are some security tricks for you:
DO NOT expose users' sensitive information to public network.
Every sensitive information should be well hashed or public-key encrypted. Pay attention: If you choose to encrypt users' sensitive information by a public-key, please make sure that the user can verify the public-key. For example, you could send some kind of public-key fingerprint to user via SMS or even an auto-call.
Generate a SHARED SECRET after log on successfully
After a secure log on transaction, a shared secret should be generate. The generation procedure could refer to SSL Handshake. Pay attention: Once the shared secret is generated, it must on be transported anymore. The only function of it is to encrypt/decrypt the data between Server and Broswer
There SHOULD be a two-step-verification to avoid repeat attack
May these tricks will help you

How to secure the sign up process if there is no ssl

I am building a sign up page for user to sign up as a member, and am wondering how to keep the user's password secure if I have no ssl-server.
The only way I can imagine is to md5 encrypt the user's password before sending back to server for storing, and next time while in login page, the password input will be md5 with a dynamic secret seed before sending back to the server to autheticate if the user is a member.
Is it a good idea? Any good suggestion? Do I have other option?
Thanks a lot for any good idea.

The problem is that you need some kind of shared secret between client and server that a possible eavesdropper does not know to be able to encrypt it. As the eavesdropper can also listen to all traffic between client and server beforehand, you have some kind of a chicken and egg situation.
Only way out: use public/private key encryption. The client encrypts the password with the public key of the server and then sends it. The only one who might open it is the owner of the private key, presumably your server.
Have a look at http://www.jcryption.org, it might do what you want.

First, it's worth trying to protect passwords even if the assets you're protecting do not require a high security approach - since too many people use the same password for different sites - however for a secure, public facing system there is no substitute for SSL.
It is possible to do this - if you hash the submitted password with a challenge from the server. And you've already got a suitable challenge available in the form of the PHP session id (although you need to ensure that you're not susceptible to session fixation, and there are also some security constraints around allowing the session cookie to be read from Javascript).
This of course depends on having an un-hashed password on the server to create a comparison value from. And this is a definite no-no.
So....you store the password hashed with a known salt (S1) on the server. When someone wants to login you send them a session id (S2) and S1 and they send back:
md5(S2 . md5(S1 .password));
There are javascript implementations of md5.
Md5 is approaching the end of usefulness as a secure cipher - however for the puproses stated here it should be adequate.

I guess you don't really need to decrypt it, as people normally only store the password hash in the database. (unless you want to know/harvest their password).
A common way is to has the password and send only the hash.
I'll suggest you to pass "salt" from the server side to the form, and hash the password and the salt together to make it more random.
To be really secure, you'll have to implement/find a public key encryption algorithm implemented in javascript. Using any symmetric key encryption would still be vulnerable to man in the middle attack as your key has to be transferred to the client side.

The real fix for this problem is to use real SSL/HTTPS connection.
Rationale:
If the content that is available after logging in is worth protecting, the whole session must be protected, not just the user password.
If the content is not worth protecting why require logging in at all?
Note that you do not need to use paid SSL certificate to get the benefits of SSL. You may sign your own (see http://www.debian-administration.org/articles/284 for an example). However, recent versions of Firefox have made use of self signed certificates a pain in the ass unless the user is ready to install you as CA. (By some weird logic Firefox displays much less alarming dialog for adding a new CA than adding an exception for a singler server that uses self signed certificate. This is really bad because accepting a new CA accepts all certificates that will be signed by that CA in the future!)
However, if you insist on not using real SSL, you may implement encryption with JavaScript: http://www.hanewin.net/encrypt/ or http://www.jcryption.org/ - be warned though that this requires a lot of work and the end result may end up nearly as protected as SSL if implemented correctly in every little detail. The end result will never be as safe as SSL because you have to transfer the JS script to the visitor's user agent (browser) without encryption and as a result, the visitor may end up running JavaScript selected by the attacker (the attacker can execute Man-in-the-Middle attack because otherwise you don't need any protection to the user password, either).

Certificate Authorities(CAs)'s Public keys stored in web browser is the only thing that prevents SSL/TSL to be not vulnerable to man in the middle attack. So, there is no way to protection these solutions.
All of these solutions are vulnerable to man in the middle attack.
Note that Mallory (Malicious active attacker) can replace his public key in the page.

You should be using SSL(the rest is NOT that secure) to do authentication and luckily you can by using open-source OpenID just like stackoverflow.com does. You can for example read why Stackoverflow.com has switched to OpenID by reading this article(good read).
There is a very user-friendly OpenID library available from LightOpenID. I have created a little library "Openid for PHP with an user-friendly way to select an OpenID thanks openid-selector and LightOpenID" available at github. I also have put a demo on my webhosting available at http://westerveld.name/php-openid/

Login without HTTPS, how to secure?

For a webapplication, when HTTPS is not available as a security measure, is it possible to still make the login somewhat secure? E.g.:
Tokenize logins, to make repeat attacks difficult?
Somehow encrypt the sent password from a HTML password field?
In particular I'm using CakePHP and an AJAX POST call to trigger authentication (includes provided username and password).
Update on the problem:
HTTPS is not available. Period. If you don't like the the situation, consider it a theoretical question.
There are no explicit requirements, you have whatever HTTP, PHP and a browser (cookies, JavaScript etc.) offers in real life (no magic RSA binaries, PGP plugins).
Question is, what is the best, you can make out of this situation, that is better than sending the passwords plaintext. Knowing the drawbacks of each such solutions is a plus.
Any improvement better than plain passwords is welcome. We do not aim for a 100% l33tG0Dhx0r-proff solution. Difficult to crack is better than complicated to hack which is better than a trivial sniffing revealing the password.

It is a bad engineering practice to reinvent the wheel. Engineers who do this are falling victim to the "Not Invented Here" bias, which can cause a lot of damage when it is a security critical system.
SSL/TLS which is behind HTTPS is absolutely vital in maintaining a secure connection between a website and a browser. Public wifi networks put users at risk, and when used correctly, HTTPS is the only tool that can protect user accounts from this vulnerability.
In the case of two clients that need secure end-to-end (e2e) encryption then there is the open source and vetted Signal Protocol which has received number opens source ports on github and a wide adoption from popular apps like WhatsApp. There is no need to brew your own, these protocols work well for a reason.
If your host doesn't support HTTPS then a service like Cloudflare Universal SSL can be used to ensure all browsers connect to your site using HTTPS, even if your server doesn't support SSL/TLS. The connection between Cloudflare and your website will still be unprotected, but this Cloudflare service is intended to protect users against threats found on public wifi networks. From the perspective of a penetration tester, not providing HTTPS is highly suspect, if you aren't providing a basic security requirement as delivering traffic, then what other security requirements are you missing? HTTPS certificates can be obtained for free using Let's Encrypt or Start SSL, there is no legitimate reason not to support HTTPS.
HTTPS is vital because it does lot more than just "encrypt passwords". Another important role is that it should prevent the user from giving logging into a malicious server that is impersonating a real server. Using a system to protect the password alone is still a violation of OWASP A9 - Insufficient Transport Layer Protection because you would still be transmitting session credentials in plain text which is all the attacker needs (Firesheep).
JavaScript-based cryptography cannot be used to construct a secure transport layer.
"Tokenize logins": If an attacker is sniffing
the traffic, they'll have the plain text username/password and then
they can just login with these new credentials. (Replay attack)
"Somehow encrypt the transmitted password": After the person has logged in
an attacker can sniff the traffic to get the valid session id
(cookie) and then just use this instead of logging in. If the
entire session was protected with SSL/TLS then this is not a problem.
There are other more complex attacks that affect both this system and our current SSL infrastructure. The SSLStrip attack goes into greater detail. I highly recommend watching Moxie Marlinspike's Blackhat 2009 talk, which lead to the HTTP-Strict-Transport-Security standard.

The short answer is that without SSL endpoint to endpoint encryption, it's impossible to do it securely...
One of the primary reasons for this is that you can't do secure crypto in a browser. See this reference - Javascript Cryptography Considered Harmful.
Additionally, there's no way that you can be sure that the source of the credentials are indeed who you're talking to. Meaning that there's absolutely no way without SSL to be sure that there's not a Man-In-The-Middle Attack going on.
So no, you can't do it.
Additionally, don't even try. Get SSL. You can get free certificates. Hosts will usually give you a dedicated IP for a few $$$ per month. And if you really care about security, you'd be using at least a VM with a dedicated IP address anyway.
To even attempt this would be Security Through Obscurity at best, and nothing at worst. SSL is a solved problem. Why not use that solution. Security is not something to guess at. Use the proper techniques. Don't try to invent your own. It won't work...

As you suggested, you may be able to generate a unique token each time the page is created. That same token would need to be sent back with the form data and could not be reused. You could also keep the password safe by using JavaScript to hash it, if you can rely on it being enabled by your users.
This scheme is still not secure, however. An attacker could still see everything going across the wire. They could intercept the token and send a response back to you before the user does. Or they could just wait for someone to login, steal that person's credentials (as they are sent over the wire), and just make their own login request later on.
Bottom Line - you need to use HTTPS to guarantee the site is secure.

You can encrypt the password using Javascript and decrypt it on the server.
I would recommend generating an RSA keypair on the server, send the public key along with a timed salt to the browser, then encrypting the password, combined with the salt, using the public key in Javascript.
You can find an RSA implementation in Javascript here
You should include both the IP address and the entire X-FORWARDED-FOR hedaer in the authentication cookies to prevent cookie theft behind proxies.
If you're dealing with sensitive data, you could generate a random AES key in Javascript, then send it to the server along with the password encrypted with RSA.
You could then make the entire application use encrypted AJAX requests from a single page and not use an auth cookie at all.
Note that it is not possible to protect against an active man-in-the-middle attack without SSL. An active attacker can completely replace your site with his own proxy, and there isn't any way to defend against that. (Since there cannot be any known good code)

You can use HTTP Digest authentication, which is supported by most browsers and does not send the password in clear over the wire.
The downside is the ugly log in box displayed by broswer. If you preffer to stick with forms, then you can implement exactly the same protocol as HTTP Digest in your forms authnetication: send hidden fields containing the realm and the challenge, and have the client add in JavaScript the nonce and compute the digest. This way you'll use a well known and proven exhange protocol, rather than roll your own.
HTTP Digest requires only hash operations.

Create a public/private key pair using an asymmetric cipher.
Create a symmetric key on the server.
Send the public key down to the client side.
Create a random key for the symmetric cipher client side.
Encrypt that random key using the public key client side.
Send the encrypted key to the server.
Server does the following:
a. Decrypts the random symmetric key using the private key.
b. Creates a token containing the generated client key.
c. Signs the token.
d. Encrypts the token using the server symmetric key.
e. Encrypts the already encrypted token with the client generated key.
f. Sends the encrypted token down.
The client receives this token and does the following:
a. Decrypts the token with the key it generated.
b. Stores the decrypted token.
c. At this point the stored token is only encrypted with the server symmetric key.
On every from the client to the server:
a. Encrypt the outbound data using the client generated key.
b. Send the token + encrypted data
On every request the server receives:
a. Decrypt the token using the server symmetric key.
b. Verify the signature.
c. Decrypt the data using the client generated key stored in the token.

What about HTTP Digest Authentication? It provides security by MD5-hashing username, password and a nonce (among other things) before sending it to the server. MD5 isn't really secure, but it's a good way for simple security with HTTP.
Of course this doesn't prevent hackers from changing the message... but it secures your password.

Login without HTTPS, how to secure?
Since there is no secure channel between your server and your client:
because there is no secure channel, anybody can snoop your traffic.
because anybody can snoop the traffic, you are open to a MITM attack.
because you are open to MITM attack, there is no guarantee you client will see a legitimate page.
because the pages are not legitimate and your page is in effect not being served (the guy in the middle is serving the pages), all tricks used server-side are rendered useless.
What can you do? Theorically?
both client and server need to use encryption to make snooping / MITM less susceptible.
assume you cannot have a handshake,
assume your client already has your key and knows how to speak the same gibberish as your server.
how about some SSL over HTTP but wrapped in base64-encoded message for some gibberish?
But wait... Since you said no magic binary, or plugin, not even RSA, I don't know if any of this is possible save for (some potentially very weak) in-house encryption.
--

You can try to replicate it to some point, by using public key encryption (GPG maybe) and making use of browser caching.
This is not something secure, even just putting up SSL won't be enough for a sophisticated attacker, you need to make use of HSTS, public key pinning etc to just to consider a web site secure today.
The rest of the answer is just food for thought.
Create a public-private key pair. Keep private one secure.
Create a js file containing the public key and a encrypt function, find a secure encryption algorithm. This function should encrypt a given string (serialized form) with an additional timestamp, to avoid a replication attack.
Serve this file with Cache-Control:public, max-age=31536000 HTTP header. We try to mitigate when the attacker tries to replace the script. The file will always be served from the browser cache.
Send all the forms via Javascript, using the encrypt function. Serve these with the same header as above.
At the server side, decrypt the data, check the timestamp, if it's still valid. Do you thing, if not, discard it.
Create a cookie token which can only be used once for a very short amount of time. If the attacker captures a cookie, he won't have much time to do stuff. However, if the attacker is fast enough, then he might log the original user out.
Change the cookies with every response. But then what do you do when the user sends multiple requests at once and then they arrive in the reverse order? Which cookie is valid? This creates tons of problems at the cost of a false sense of security.
Any listeners won't be able to make use of the data going back and forth, and they won't be able to change/inject the existing JS files until the cache expires / user clears the cache. However, any sophisticated attacker can replace the whole HTML file which would discard all the security measurements I have just mentioned. If you can at least serve this file / form over HTTPS, you might get away with it, put them on github pages or whatever. However, if you put the file some other domain, then you need to set up CORS for the receiving domain for this to work.
Another try
One time passwords sent to email.
User fills out their email, clicks a link which then sends a link to their email with a token that will enable them logging in.
User clicks the link
Server checks the token, logs the user in.
Rolls the cookies like the previous example.
All in all, whatever you do, it is not secure. Given a fast, sophisticated attacker, nothing stands in the way.
Get SSL, if the infrastructure does not support it, change it. If your manager does not believe in SSL, convince him/her. Don't create a false sense of security. Protect your user's data, depending on your location, you are legally required to protect the data of your users.
Then let's talk about how to make a site secure with SSL.

Have a look at "The Secure Remote Password Protocol".
Instead of formulating it myself, let me quote from their webite:
The Secure Remote Password protocol performs secure remote authentication of short human-memorizable passwords and resists both passive and active network attacks.
and:
[The] protocol combines techniques of zero-knowledge proofs with asymmetric key exchange protocols and offers significantly improved performance over comparably strong extended methods that resist stolen-verifier attacks such as Augmented EKE or B-SPEKE.
Although the Stanford University doesn't provide implementations for PHP and JavaScript themselves, they link to some 3rd-party implementations.
One of those links leads to "Clipperz", which is an online password manager. It is also available as a community edition on GitHub. There they host their "javascript-crypto-library", which implements the protocol and the "password-manager" itself, which contains backends written in PHP and Python.
I can't say how difficult it would be to extract the relevant portions of code, but maybe you can reuse their implementation (it's licensed under AGPL).
Edit 2014/10/24:
Wikipedia's article on SRP lists some more implementations. Relevant for PHP/JS:
srp-client (JS)
srp-6a-demo (PHP/JS)

The best solution I have seen for somewhat secure HTTP connections is to use a Javascript implementation of md5sum (or some other hash) to avoid transmitting the password in plaintext. You can create a form onsubmit handler in Javascript that replaces the password field with a hash of the original value. This adds a modest amount of security to an unsecure connection, but relies on Javascript running in the browser to work properly.

I guess you care about secure transmission of password to the server? My answer is: dont transmit passwords to the server :)
Infact you may not transmit anything from browser (user) to server to authenticate the user, as an attacker who is spying http traffic would also be able to retransmit the data and authenticate.
Proposal:
Obvious solution would be to use a one-way, one-time transaction authentication originating from server; like a transaction number which can only be used once. Eventually, you still need a secure channel once to sync the list of transaction numbers with user.
You could use something google authenticator, yet you need a secure channel once to setup parameters on either side. If you consider email to be secure, that would be a way to go.

I have the same issue on a system of mine. I have taken steps to try and increase security without compromising the user experience with convoluted mechanisms. What I noticed was that the vast majority of users logged in from the same machine using the same browser, (but not necessarily the same IP address), or from a couple of browsers (eg: desktop or mobile). I decided I could use this to identify a pattern.
1) During registration, users are required to have strong passwords (to prevent dictionary attacks), a security question/answer and standard email verification (as proof of real person)
2) During login, after 5 failed login attempts (not before), a captcha is displayed to prevent brute force attacks.
3) Finally, I created a hash of parts of the user-agent string following a successful login, containing the users OS, browser (general not versions) and language - forming a sort of secondary password. If the useragent hash is significantly different on next login, the user is asked to answer the security question. Then, if this is answered satisfactory, the new UA string is hashed and added to their "safe machines" list, so that they wont be asked again from this machine. This is similar to a mechanism employed by the Steam gaming system.
This has been in use for over a year very successfully with about 700 users and it had the additional benefit of preventing "login sharing" - a problem where multiple users were using the same credentials for convenience!

The answer is shorter, and if you really matter about security you always have options that different levels of bureauocracy.
Absolut security does not exists. The number one flaw is always on the client side, with trojans ans keyloggers. SSL doesn't help with that.
1) Token generators: banks use them, blizzard uses then. It can be a device or an app. Well.. it's expensive.
2) SMS pins. interesting and affordable solution. There is a lot of good prices from trnasactional sms on the market and everyone has a phone capable of receiving it.
3) If you have to use HTTP, you can force a third party oauth service, like google or facebook. That's the best you can do without a token generator.

Use hashing mechanisms to store password and always compare the hashed password then nobody knows the real password even you.
It is very simple but it is effective.However, nothing is completely secure and there are some ways to broke the scurity layers.

Try this : On each request of the login page, send across a nonce and a timestamp.
While posting to server, send the following four details :
The username, the nonce and the timestamp in plaintext.
Then concatenate the above with a separator (Eg: newline) and encrypt using the user's password as encryption in chained-block-cipher mode.
On the server end use the username to lookup the password and verify the encrypted string.
Since the password is never sent across in clear, it is secure and the timestamp can be used to avoid a re-submit of the same data.
To avoid hijacking of session by obtaining the session key through a man-in-the-middle attack, the password or a hash of the password can be stored in-memory by the application on the client end and be used for generating unique session keys for validation by server.
Taking a look at OAuth 1.0 is also not a bad idea.

If you can't use HTTPS or you don't want to use HTTPS, consider using jCryption. jCryption offers encryption for the data being sent through HTTP requests (POST, GET etc.).
You can test the technique here: http://www.jcryption.org/#examples
If you're using Firebug, you'll see that all the data is encrypted.
It has jQuery library to encrypt the data on the front-end and a PHP library to decrypt the data in the back-end.

It is hard to secure the communication without a trusted third party, however, there are some security tricks for you:
DO NOT expose users' sensitive information to public network.
Every sensitive information should be well hashed or public-key encrypted. Pay attention: If you choose to encrypt users' sensitive information by a public-key, please make sure that the user can verify the public-key. For example, you could send some kind of public-key fingerprint to user via SMS or even an auto-call.
Generate a SHARED SECRET after log on successfully
After a secure log on transaction, a shared secret should be generate. The generation procedure could refer to SSL Handshake. Pay attention: Once the shared secret is generated, it must on be transported anymore. The only function of it is to encrypt/decrypt the data between Server and Broswer
There SHOULD be a two-step-verification to avoid repeat attack
May these tricks will help you

We Keep Coding

PHP, A popular general-purpose scripting language that is especially suited to web development.