There's a webpage that I need to log in to. I used CURL with post to login, but it's not enough. When you log in from the website the post also includes a string that is always changing. Is threre a way to get over that?
I use this:
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; he-IL; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_POST, TRUE);
$post = "username=$username&password=$password";
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
It's like I need the code to actually go to the webpage and fill the form regularly.
I looked everywhere but all I could find was using post data.
Thanks!
To pass that you need to visit the page with the form, grab the field and then use it in POST request when you submit the form.
I suggest you visit the form page not only for that, but also for the following reasons (some of which can be used to figure people using automatic requests):
You recieve cookies
You don't fake referrer, you actually visited the page
You might want to check form fields to see if there's any new ones added since you wrote the script. That could be the case if form setup changes and you might want to adapt to that, if you don't then your script might stop working one day
Related
I'm opening a page as logged user, and it kind of seems to work, except the website has some sort of a protection system. If I do this normally, I'll get the page I want, but if I do it with cURL, I'll get 'Welcome back user (userid)' and a link to the page I requested. Once I click the link, I'll get where I want to be. Now I tried faking the referer and checking the data that gets sent to the page, there's nothing special there. When I click the link, I simply get redirected to the page I wanted in the first place. My question is why doesn't this code get me there as well:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL , "http://www.site.com/sell/index");
curl_setopt($ch, CURLOPT_REFERER, 'http://www.site.com');
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_FAILONERROR, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
$response = curl_exec($ch);
curl_close($ch);
echo $response;
Just before I do this, I perform login procedure and grab the cookie. And I do get to open the page as logged in user, I just can't seem to access it without clicking the ahref.
PS. The same thing would happen if I logged in, open the page I wanted, closed browser and opened it again. So I'm thinking it has to do with referer?
cookie-jar means it will save your cookie from curl's response. That's why it is not working for you. Instead use cookie-file so that your curl send stored cookie with request:
curl_setopt($ch, CURLOPT_COOKEFILE, "cookie.txt");
Also, use absolute path(/var/tmp/cookie.txt) instead of relative path.
Now, Be Happy!
I'm trying to make a script that would run through some sites that I visit every day and get the most interesting info/statistics from them. I wanted to use curl for this purpose, because some of these sites require authentification. Everything was ok until I bumped into the site: rossnet.pl which seems to be somehow secured 'cause I can't authenticate myself at all.
The form that I want to use can be found here:
https://www.rossnet.pl/rossnetlogin.aspx
On the left, under the text: "Mam konto w Rossnet.pl - Logowanie". It doesn't seem to have any hidden input fields, only two text fields for credentials, called:
- "dnn$ctr1203$ViewLogin$txtUserLogin"
- "dnn$ctr1203$ViewLogin$txtUserPass"
I'm using the code shown below but the page returned by the server seems as if exactly nothing happened (no error messages, it seems to look the same as when I don't send any POST data).
Does anyone have a clue about what may be wrong? In the code below I put in actual account credentials for you to be able to test the script if you wish to help me.
Here you can see how does the script below work on my server:
http://kremuwa.netii.net/rossman/skrypt.php
<?php
$url = "https://www.rossnet.pl/rossnetlogin.aspx";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'dnn$ctr1203$ViewLogin$txtUserLogin=warzywko3000&dnn$ctr1203$ViewLogin$txtUserPass=password123');
$output = curl_exec($ch);
curl_close($ch);
echo $output;
?>
Login forms are sometimes protected with challenges that prevent you from directly submitting the form without loading the page first. I've listed a few options that could stand in your way.
One option is cookie challenges, it's also the easiest to deal with by just loading the page (fetch the cookie) and send it along with the form submission.
Another option is a hidden field challenge; a hidden form field is populated with a challenge code and the submission expects that value to be sent as well.
The last option I can think of is an even more difficult approach involving JavaScript; the page would use JavaScript to load the challenge string, maybe obfuscate it a bit and then send it along (via hidden form field or ajax request).
I am trying to automate the login progress on a captcha protected page. I am using Death By Captcha to translate the image into text and it seems to be working well. I am using curl to load the login page, retrieve the captcha image url, send it to DBC, get the text back and submit a POST request to the login page with the captcha text.
The problem that I'm having is that the captcha image changes when I submit the post request. Since I do not get the same behavior when reloading/or wrongly submitting the form through a browser (I get the same image over and over again), I am assuming that the problem has to do with the cookies or something else that I'm missing that relates to the session.
This is the code that I use to retrieve the data and submit the form:
$ch = curl_init();
// Not sure that I need it, just make sure that the session doesn't change...
curl_setopt($ch, CURLOPT_COOKIESESSION, false);
curl_setopt($ch, CURLOPT_URL, $loginUrl);
// It seems that PHPSESSID cookie parameter might be the parameter that keep the image the same, but it didn't work. I even read it dynamically from the cookie file but it still didn't work
//curl_setopt($ch, CURLOPT_COOKIE, "PHPSESSID=2bp3nhkp3bgftfrr1rjekg03o2");
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieName);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieName);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $loginUrl);
$result = curl_exec($ch);
// Resolve the captcha and append it to the post parameters
$captchaText = $this->resolveCaptcha($result);
$postData .= '&LoginForm%5BverifyCode%5D='.$captchaText;
// Resubmit the form with the updated form data
curl_setopt($ch, CURLOPT_REFERER, $loginUrl);
curl_setopt($ch, CURLOPT_URL, $loginUrl);
curl_setopt ($ch, CURLOPT_POST, 1); //FIXED
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postData);
$result = curl_exec($ch);
When I print the end result, I can see that the captcha text was submitted successfully but that the image itself has changed...
I am also attaching a screenshot of the request params as captured with Tamper in a standard Firefox session (so someone might spot if I'm missing something).
The PHP/curl submit code is fully working for non-captcha based sites so the POST parameters submission seems to be working.
It could be that I'm missing something very basic here, any help will be much appreciated.
I also took a look at these posts but couldn't find the answer that I'm looking for.
How CURL Login with Captcha and Session
How to retrieve captcha and save session with PHP cURL?
https://stackoverflow.com/questions/8633282/curl-to-download-a-captcha-and-submit-it
you're using
curl_setopt ($ch, CURLOPT_POST, 0);
in second curl_exec. shoudn't it be
curl_setopt ($ch, CURLOPT_POST, 1);
?
Using PHP and cURL, I'd like to check if I can login to a website using the provided user credentials. For that I'm currently retrieving the entire website and then use regex to filter for keywords that might indicate the login didn't work.
The url itself contains the string "errormessage" if a wrong username/password has been entered. Is it possible to only use curl to get the url address, without the contents to speed it up?
Here's my curl PHP code:
function curl_get_request($referer, $submit_url, $ch)
{
global $cookie_path;
// sends a request via curl to the string specifics listed
$agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)";
curl_setopt($ch, CURLOPT_URL, $submit_url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_path);
return $result = curl_exec ($ch);
}
Also, if somebody has a better idea on how to handle a problem like this, please let me know!
What you should do is check the URL each time there is a redirect. Most redirects are going to be done with the proper HTTP headers. If that is the case, see this answer:
PHP: cURL and keep track of all redirections
Basically, turn off automatic redirection following, and check the HTTP status code for 301 or 302. If you get one of those, you can continue to follow the redirection if needed, or exit from there.
If instead, the redirection is happening client side, you will have to parse the page with a DOM parser.
i tried many tutorials but all failed, i know for an experienced user it might be obvious, thx anyway.
there is the simple form:
https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp?MODE=SHAB
here is my script which returns only the empty form instead of my POST search:
(i used tamper-data to get the Post-variables, i also use https)
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp?MODE=SHAB');
curl_setopt($ch, CURLOPT_POSTFIELDS,'KEYWORDS=&NOTICE_NR=&TIMESPAN=TODAY&STAT_TM_1=&STAT_TM_2=&SELTYPE=HR&TYPE_CD_AW=&TYPE_CD_AN=&TYPE_CD_BL=&TYPE_CD_VM=&TYPE_CD_HR=HR01&LEGAL_FORM_NR_HR=&FIRM_ID_HR=&HR_CANTON_AG=ON&HR_CANTON_BE=ON&TYPE_CD_IS=&TYPE_CD_KK=&YN_KK=&TYPE_CD_IP=&TYPE_CD_NA=&YN_NA=&TYPE_CD_SB=&YN_SB=&TYPE_CD_SR=&FIRM_NAME_TX_UP=&FIRM_CITY_TX_UP=&command=Recherchieren');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_REFERER,"https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp%3Fcategory%3DHR");
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
$result = curl_exec($ch);
echo $result;
This has strangely been written into my_cookies.txt
www.shab.ch FALSE /shabforms FALSE 0 JSESSIONID E884A3B4187C68253CEEBCD58E7E934E
www.shab.ch FALSE / FALSE 1287673522 BC_HA_C30B29681466613B 131BDF
What is wrong? :)
UPDATE:
Ok, i got the error. it was related to the post-url. the script on the website seems to do the process by ajax,... without changing the url to send (i could not even find the correct url in tamper data!!).
Fortunately i could figure that out, its "shabforms/servlet/web/DocumentSearch".
Now it works, thx
I just ran this script and got a German website saved in $result.
Maybe your curl setup needs tweeking? Have you got it working with another site?
it was related to the post-url. the script on the website seems to do the process by ajax,... without changing the url to send (i could not even find the correct url in tamper data!!). Fortunately i could figure that out, its "shabforms/servlet/web/DocumentSearch".
Now it works, thx