I want get data from this website: https://vrl.lta.gov.sg/lta/vrl/action/pubfunc?ID=EnquireTransferFee
This is my code using CURL:
function postPage($url, $pvars, $referer, $timeout){
if(!isset($timeout))
$timeout=30;
$curl = curl_init();
$post = http_build_query($pvars);
if(isset($referer)){
curl_setopt ($curl, CURLOPT_REFERER, $referer);
}
curl_setopt ($curl, CURLOPT_URL, $url);
curl_setopt ($curl, CURLOPT_TIMEOUT, $timeout);
curl_setopt ($curl, CURLOPT_USERAGENT, sprintf("Mozilla/%d.0",rand(4,5)));
curl_setopt ($curl, CURLOPT_HEADER, 0);
curl_setopt ($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt ($curl, CURLOPT_POST, 1);
curl_setopt ($curl, CURLOPT_POSTFIELDS, $post);
curl_setopt ($curl, CURLOPT_HTTPHEADER,
array("Content-type: application/x-www-form-urlencoded"));
$html = curl_exec ($curl);
curl_close ($curl);
return $html;
}
$vars = array(
'vehicleNo' => 'SFT4228H',
'transferDate' => '10042014'
);
$result = postPage('test.php', $vars, 'https://vrl.lta.gov.sg/lta/vrl/action/pubfunc?ID=EnquireTransferFee', '30');
print "Result:".$result; //Show data in my website
=> How to get this data
As you have it here, it appears that you are using curl to request "test.php".
Try switching them around in your function call:
$result = postPage('https://vrl.lta.gov.sg/lta/vrl/action/pubfunc?ID=EnquireTransferFee', $vars, 'test.php', '30');
Related
I am trying to scrape the contents of the a site with login secured
but unable to do it
The site's login has three options username,password,passcode
here is the code I am using
<?php
// HTTP authentication
$url = "http://aftabcurrency.com/login_script.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$cookie = 'cookies.txt';
$timeout = 30;
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($curl, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch,CURLOPT_POSTFIELDS,"user_name=user&user_password=pass&passcode=code");
$result = curl_exec($ch);
curl_close($ch);
echo $result;
?>
you need to do a POST to http://aftabcurrency.com/login_script.php
your curl needs also to accept cookies.
After the authentification the script will redirect you, so you need also to add CURLOPT_FOLLOWACTION.
here is a edited version of your script, I can't test it on http://aftabcurrency.com/ hope it works:
$url = "http://aftabcurrency.com/login_script.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$cookie = 'cookies.txt';
$timeout = 30;
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch,CURLOPT_POSTFIELDS,"user_name=user&user_password=pass&passcode=code");
$result = curl_exec($ch);
/* //OPTIONAL - Redirect to another page after login
$url = "http://aftabcurrency.com/some_other_page";
curl_setopt ($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
*/ //end OPTIONAL
curl_close($ch);
echo $result;
You need to POST your username/password/passcode to that page. What you are trying to do right now is http authentication.
So instead of this
curl_setopt($ch, CURLOPT_USERPWD, "demo:demopass:demopasscode");
you need this
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, "user_name=xxxxx&user_password=xxxxxx&passcode=xxxxx");
I am trying to login to a page, and then go to a different page, submit a form, and then I get another form on the same link, submit that, and then get another form on the same link, and submit that.
However, from what I see, I login fine ... but after that it keeps showing me the same page (the one after I login).
$username = 'xxxxxxx';
$password = 'yyyyyyy';
$useragent = $_SERVER["HTTP_USER_AGENT"];
$fields = array(
'username='. urlencode($username),
'password=' . urlencode($password),
);
$postdata = implode('&', $fields);
$ch = curl_init();
//Main Login
curl_setopt ($ch, CURLOPT_URL,"https://xxxxxxxxxxxx.com/index.php");
curl_setopt ($ch, CURLOPT_HEADER, true);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt ($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt ($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt ($ch, CURLOPT_REFERER, "https://xxxxxxxxxxx.com/index.php");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
echo $result;
echo "<pre>";
print_r ($info);
echo "</pre>";
print curl_error($ch);
$contractid = "12345";
$fields = array(
'contract_id='. urlencode($contractid),
'submit='. urlencode("Next"),
);
$postdata2 = implode('&', $fields);
curl_setopt ($ch, CURLOPT_URL,"https://xxxxxxxxxxxx.com/index.php?page=Recovery");
curl_setopt ($ch, CURLOPT_HEADER, true);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt ($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt ($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt ($ch, CURLOPT_REFERER, "https://xxxxxxxxxxxxx.com/index.php?page=Recovery");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata2);
curl_setopt ($ch, CURLOPT_POST, 1);
$result_2 = curl_exec($ch);
$info = curl_getinfo($ch);
echo $result;
echo "<pre>";
print_r ($info);
echo "</pre>";
print curl_error($ch);
and twice more for each step, the same way and ending it with curl_close($ch);.
However, like I stated, it still shows only the main page.
After first curl request close curl and initialize again. Otherwise curl will not write anything to cookie file and at next call no cookies are sent!
CURLOPT_COOKIEJAR
The name of a file to save all internal cookies to when the handle is
closed, e.g. after a call to curl_close.
// after end of first curl call
curl_close($ch);
// before second curl call
$ch = curl_init();
Hey am I missing a curl_opt or something? This code was working yesterday and now it doesnt work. It is suppose to return a json string but instead returns empty string.
<?php
session_start();
$UCID=$_POST['UCID'];
$Pass=$_POST['Pass'];
$credentials=array("Username"=>$UCID, "Password"=>$Pass);
$post_field_string = http_build_query($credentials, '', '&');
$url1="https://web.njit.edu/~vm276/connect.php";
//$url1="https://www.njit.edu/cp/login.php";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url1);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_PROTOCOLS, CURLPROTO_HTTPS);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_REFERER, $url1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $post_field_string);
curl_setopt ($ch, CURLOPT_POST, true);
$result = curl_exec($ch);
curl_close($ch);
$json=json_decode($result);
echo $json->result;
?>
You can get information about errors by adding to your code:
$result= curl_exec($ch);
$error = curl_error($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if(200 == $httpCode) {
//...
}
else {
echo "httpCode: ".$httpCode."\n";
var_dump($error);
exit();
}
http://php.net/manual/en/function.curl-error.php
http://php.net/manual/en/function.curl-getinfo.php
I need to retrieve the XML content from this URL :
http://www.ardmasr.com/MasrenaRss.aspx?Category=4
always return an empty string with out any errors, code:
$curl = curl_init();
curl_setopt ($curl, CURLOPT_CONNECTTIMEOUT,10);
curl_setopt ($curl, CURLOPT_USERAGENT,'smartic-rss-feeds');
curl_setopt ($curl, CURLOPT_URL, $url);
curl_setopt ($curl, CURLOPT_TIMEOUT,120);
curl_setopt ($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt ($curl, CURLOPT_SSL_VERIFYPEER, 0);
if($curlFollowRedirects){
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, false);
}
$content = curl_exec($curl);
// Check if any error occurred
if($content === false)
{
echo 'Curl error: ' . curl_error($curl);
}
curl_close ($curl);
// if there is a redirect
if(preg_match('#Location: (.*)#',$content, $r)){
$l = trim($r[1]);
$content = $this->fetch_data($l);
}
return $content;
My problem in encoding the URL i used this function to fix my problem :
utf8_decode($url)
UPDATE: I am now able to to access the main homepage, but i am not logged in.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.xxx.com/?cmd=home');
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, 'emailAddress=jeffanderson#tradermail.info&persist=on&pswd=qweqwe&from=/?cmd=home');
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
$content = curl_exec ($ch);
curl_close ($ch);
echo $content;
curl_exec() returns false if there was a problem:
$content = curl_exec($ch);
if ($content === FALSE) {
die(curl_error($ch));
}
return($content);
You need to do two things. First, add the following two options to CURL:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
That will tell CURL to follow any redirects. Then, change return to echo. Echo will send the data to the browser. Then you are all set!
Add curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);.
after all honorable answer..u put all together in below way
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.xxx.com/?cmd=sb-login&from=/?cmd=home');
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, 'emailAddress=xxx#tradermail.info&persist=on&pswd=xxx&from=/?cmd=home');
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_MAXREDIRS, 10);
$content = curl_exec($ch);
if (FALSE===$content {
echo "Follwowing error:".curl_error();
die(curl_errorno($ch));
}
curl_close ($ch);
return($content);