cURL not returning anything? - php

<?php
error_reporting(-1);
$config = array
(
"siteURL" => "http://domain.com",
"loginCheck" => "checkuser.php",
"userAgent" => "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16"
);
$postFields = "username=user&password=pass&submit= Login ";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $config['siteURL'] . $config['loginCheck']);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_USERAGENT, config['userAgent']);
curl_setopt($ch, CURLOPT_REFERER, config['siteURL']);
$content = curl_exec($ch);
curl_close($ch);
echo $content;
?>
I expect this to be echoing the result of the curl, but its not. am I doing anything wrong?

It does, if you set a complete URL with a '/' at the end (fix two other typos):
error_reporting(-1);
$config = array
(
"siteURL" => "http://www.apple.com/",
"loginCheck" => "checkuser.php",
"userAgent" => "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16"
);
$postFields = "username=user&password=pass&submit= Login ";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $config['siteURL'] . $config['loginCheck']);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $config['userAgent']);
curl_setopt($ch, CURLOPT_REFERER, $config['siteURL']);
$content = curl_exec($ch);
curl_close($ch);
echo $content;

If curl doesn't return anything then you have to check for the reason why that happen. Check for the last curl error message:
if(curl_exec($ch) === false)
{
echo 'Curl error: ' . curl_error($ch);
}
else
{
echo 'Operation completed without any errors, you have the response';
}
Basically it would be good to always check on that. You shouldn't assume that curl request succeeded.

Two of your variables are named badly:
curl_setopt($ch, CURLOPT_USERAGENT, config['userAgent']);
curl_setopt($ch, CURLOPT_REFERER, config['siteURL']);
Should be:
curl_setopt($ch, CURLOPT_USERAGENT, $config['userAgent']);
curl_setopt($ch, CURLOPT_REFERER, $config['siteURL']);
My guess is that PHP faulted to a blank page (since the variable is seen as a constant but constants must be scalar).
Another issue may be the user agent. I've seen servers that completely refuse to reply if no user agent was set.

You can use a class that I wrote that wraps around CURL. To install, see: https://github.com/homer6/altumo
It's much easier to use and can give you some easy to access debugging. For example:
try{
//load class autoloader
require_once( __DIR__ . '/loader.php' ); //you should ensure that is is the correct path for this file
//make the response; return the response with the headers
$client = new \Altumo\Http\OutgoingHttpRequest( 'http://www.domain.com/checkuser.php', array(
'username' => 'user',
'password' => 'pass',
'submit' => ' Login '
));
$client->setRequestMethod( \Altumo\Http\OutgoingHttpRequest::HTTP_METHOD_POST );
//send the request (with optional arguments for debugging)
//the first true will return the response headers
//the second true will turn on curl info so that it can be retrieved later
$response = $client->send( true, true );
//output the response and curl info
\Altumo\Utils\Debug::dump( $response, $client->getCurlInfo() );
//alternatively, you can get the response wrapped in an object that allows you to retrieve parts of the response
$http_response = $client->sendAndGetResponseMessage( true );
$status_code = $http_response->getStatusCode();
$message_body = $http_response->getMessageBody();
$full_http_response = $http_response->getRawHttpResponse();
\Altumo\Utils\Debug::dump( $status_code, $message_body, $full_http_response );
}catch( \Exception $e ){
//This will display an error if any exceptions have been thrown
echo 'Error: ' . $e->getMessage();
}
Hope that helps...

Related

php get error to curl url from array get in file

I get Error When Curl Data From URL ( Url From Array get in file ) .
When I use Normaly URL , It Work Correctly !
But When I get URL from Array and run a loop for it , and call function curl to run every line url , it not work ?
I don't know why , but i need someone help me .
I had turn on error report php , but nothing error !
Example File "link.txt"
https://www.fshare.vn/file/4UNG2NRPW5OQ
https://www.fshare.vn/file/CFTJYXKPMN7Q
https://www.fshare.vn/file/RMHD2XBFY93F
https://www.fshare.vn/file/I4TIUE6E9QOV
https://www.fshare.vn/file/5PV15EB38M7T
https://www.fshare.vn/file/ZDLJNDNWCNYM
https://www.fshare.vn/file/O1VYZUXXJNCI
https://www.fshare.vn/file/SD5C1VZ38IPN
And My PHP code
<?php
error_reporting(E_ALL);
$lines = file('link.txt', FILE_IGNORE_NEW_LINES); // Turn Every Line To Array From File , Or use other below Method
//$lines = explode("\n",fopen('link.txt',"r")); // Other Method Get Content To Array
//$lines = explode("\n",file_get_contents('link.txt')); // Other Method Get Content To Array
foreach($lines as $key => $value){
echo $value . "</br>";
echo get_data($value); // It not Work , Why is that ?
echo get_data("https://www.fshare.vn/file/ACKM6ONXFZJE"); // It Work Normally
die();
}
function get_data($url = null) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type' => 'application/x-www-form-urlencoded', 'charset' => 'utf-8'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0");
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com.vn/');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Right now , i have tested again and it work correctly !
Don't know what happened !

How to retrieve the data from a URL with an exclamation mark inside it?

I would like the retrieve the content of a website but the website is build with a exclamation mark in the URL and this doesn't seem to work.
Things I tried:
<?php
echo file_get_contents('https://domain.com/path/!weird.formatted?url=1');
echo file_get_contents('https://domain.com/path/%21weird.formatted?url=1');
echo file_get_contents(urlencode('https://domain.com/path/!weird.formatted?url=1'));
echo file_get_contents(rawurlencode('https://domain.com/path/!weird.formatted?url=1'));
I also tried to retrieve the content with PHP Curl but here seems the exclamation mark also a problem.
So how do I retrieve this webpage? Any suggestions would be highly appreciate.
Update
The URL I try to retrieve the content from:
https://loket.bunnik.nl/mozard/!suite86.scherm0325?mPag=1070
So the problem is that the web page was checking for a valid user agent/cookie. The code I used to fix the issue:
<?php
echo getPage("https://loket.bunnik.nl/mozard/!suite86.scherm0325?mPag=1070");
function getPage ($url) {
$useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36';
$timeout= 120;
$dir = dirname(__FILE__);
$cookie_file = $dir . '/cookies/' . md5($_SERVER['REMOTE_ADDR']) . '.txt';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt($ch, CURLOPT_ENCODING, "" );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt($ch, CURLOPT_AUTOREFERER, true );
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_MAXREDIRS, 10 );
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
$content = curl_exec($ch);
if(curl_errno($ch))
{
echo 'error:' . curl_error($ch);
}
else
{
return $content;
}
curl_close($ch);
}
?>

Symfony2 Authentication via HTTP headers

I have Symfony2 app running, that has login mechanism implemented.
However I need somehow to send request from other app, that checks whether requested URL exists on that system:
For example there's request http://myapp/order/1111 that I need to check whether it returns 200 OK header or 404 Not Found status;
When I send request it's unauthorized and get's redirected to login page /login which is returns 200 OK status header.
What I need is to somehow safely bypass login mechanism. Is there a way that I could add login credentials to request, or create additional login method?
EDIT:
As suggested in comments, I tried to use curl. But I can't figure out how make it work. Here's what I'v got, but It doesn't work.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://localhost/en/login');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6');
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_POST, true);
$html = new DOMDocument();
$html->loadHTML(curl_exec($ch));
if (curl_error($ch)) {
echo(curl_error($ch));
}
$csrf_token = $html->getElementById('csrf_token')->getAttribute('value');
curl_setopt($ch, CURLOPT_URL, 'http://localhost/en/login');
//curl_setopt($ch, CURLOPT_REFERER, 'http://localhost/en/login');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query(array(
'_username' => 'login_username',
'_password' => 'login_password',
'_remember_me' => 'on',
'_csrf_token' => $csrf_token
)));
curl_setopt($ch, CURLOPT_POST, true);
$result = curl_exec($ch);
if (curl_error($ch)) {
echo(curl_error($ch));
}
echo $result;
curl_close($ch);
Any suggestions, how to work with curl on this case?
I wrote a solution to this problem if anyone else struggling with this:
$ch = curl_init();
//get login form
curl_setopt($ch, CURLOPT_URL, $login_url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6');
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_path);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_path);
//parse html to get csrf_token, I added id attribute to it for easier retrieving
$html = new DOMDocument();
$html->loadHTML(curl_exec($ch));
$csrf_token = $html->getElementById('csrf_token')->getAttribute('value');
if (curl_error($ch)) {
throw new Exception(curl_error($ch));
}
//send post request to login_check with parameters (csrf_token included)
curl_setopt($ch, CURLOPT_URL, $check_url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query(array(
'_username' => $username,
'_password' => $password,
'_remember_me' => 'on', //if you use remember me functionality
'_csrf_token' => $csrf_token
)));
curl_exec($ch);
if (curl_error($ch)) {
throw new Exception(curl_error($ch));
}
//request login protected urls
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, false);
curl_exec($ch);
if (curl_error($ch)) {
throw new Exception(curl_error($ch));
}
$info = curl_getinfo($ch);
curl_close($ch);
//I used this code in a function returning url and http_code for further processing
return array(
'url' => $info['url'],
'http_code' => $info['http_code']
);
I was having a problem with localization. I used to pass login_check url with locale like this http://localhost/en/login_check, by removing locate I fixed the problem: http://localhost/login_check

Using Curl to Login to Vimeo

I'm trying to use CURL with PHP to login to Vimeo.com, Vimeo login.
To provide the data for CURL to use (cookie and field data), I'm using a browser extension to read the field data off of the webpage and get the cookies. I'm then passing that data through to my server and am trying to login using curl.
I'm quite positive that the browser extension part works correctly (gets the correct data) because I can verify what it's passing with what it should be passing, and it matches correctly.
Additionally, I've used this on other sites as well, and it has no problem logging in, but on vimeo the exec returns false.
Any thoughts?
function curlpage(){
$ch = curl_init();
$url = $this->input->post('url');
$data = $this->input->post('data');
$cookie = $this->input->post('cookie');
$method = $this->input->post('method');
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS,$data);
if(strtolower($method)=="put"){
curl_setopt($ch, CURLOPT_PUT, 1);
}
else{
curl_setopt($ch, CURLOPT_PUT, 0);
}
if(strtolower($method)=="get"){
curl_setopt($ch, CURLOPT_HTTPGET, 1);
}
else{
curl_setopt($ch, CURLOPT_HTTPGET, 0);
}
if(strtolower($method)=="post"){
curl_setopt($ch, CURLOPT_POST, 1);
}
else{
curl_setopt($ch, CURLOPT_POST, 0);
}
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_CAINFO, getcwd().'/certificates/BuiltinObjectToken-EquifaxSecureCA.crt');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20100101 Firefox/17.0");
curl_setopt($ch, CURLOPT_REFERER, $url);
$error = curl_error($ch);
$url=curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
if(!preg_match('/^http(s)?:\/\//', $url)){
$url = 'http://' . $url;
}
$host = parse_url($url, PHP_URL_HOST);
$page = curl_exec($ch);
curl_close($ch);
return array('page'=>$page, 'url'=>$host, 'error'=>$error);
}
Here's a sample of the data I'm sending to the above function on my server (with a bogus email an password and altered cookies):
data=action%253Dlogin%2526service%253Dvimeo%2526email%253Dhou%2540fah.com%2526password%253Dudwt%2526token%253D6b2fc081bcdf02b1f58a390d6a3f8b83
cookie=__utma%3D18392654.1284111214.1456668252.1456678435.1456181183.3%3B__utmb%3D18302654.2.10.1454681883%3B__utmc%3D18232154%3B__utmz%3D17202654.1456675435.2.2.utmcsr%3Dgoogle%7Cutmccn%3D(organic)%7Cutmcmd%3Dorganic%7Cutmctr%3D(not%2520provided)%3B
method=POST
url=http%3A%2F%2Fvimeo.com%2Flog_in
$ret = customSendDataByCurl("https://vimeo.com/log_in");
preg_match("/xsrft: \'(.*)\',/i",$ret,$token);
$token = $token[1];
echo "$token <hr>";
$cookie = '(copy from your browser using tamper data)... xsrft='.$token;
$headers = array(
"Referer: https://vimeo.com/log_in",
"User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Accept: application/json",
"Accept-Language: en-US,en;q=0.5",
"X-Requested-With: XMLHttpRequest",
"X-Request: JSON",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
);
$ret = customSendDataByCurl("https://vimeo.com/log_in?action=warm", "POSTDATA=email=(email url encoded)&token=".$token, $headers, $cookie);
$fields = array(
"action" => "login",
"service" => "vimeo",
"email" => "(email)",
"password" => "(pass)",
"token" => $token,
);
$headers = array(
"Referer: https://vimeo.com/log_in",
"User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
);
$ret = customSendDataByCurl("https://vimeo.com/log_in", http_build_query($fields), $headers, $cookie);
$ret = customSendDataByCurl("https://vimeo.com/stats/video/84142281/totals/export:csv", http_build_query($fields), $headers, $cookie);
var_export($ret);
function customSendDataByCurl($agateway, $apostfields=null, $headers=array(), $cookie="") {
if(is_array($apostfields)) $apostfields = http_build_query($apostfields);
$cookiesFile = 'cookies.txt';
$ch = curl_init($agateway);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt( $ch, CURLOPT_COOKIESESSION, true );
if ($headers) curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt( $ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0' );
curl_setopt($ch,CURLOPT_COOKIEJAR,$cookiesFile);
curl_setopt($ch,CURLOPT_COOKIEFILE,$cookiesFile);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
if(!empty($apostfields)) curl_setopt($ch, CURLOPT_POSTFIELDS, $apostfields);
$response = curl_exec($ch);
if(!$response) $response="CURL #".curl_errno($ch).": ".curl_error($ch);
return $response;
}
Vimeo, I've actually found to be a weird case for websites. They don't set all of their cookies up front, but rather set certain cookies needed to login when the form is submitted. So one of my problems was that I was not submitting all of the correct cookie information.
My second problem was that I was not correctly encoding my data as it was being sent.
All said and done, I now have got it to work!

Php curl problem

<?php
set_time_limit(0);
$php_userid = "my yahoo id";
$php_password = "my yahoo password";
$agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)";
$reffer = "http://mail.yahoo.com/";
$LOGINURL = "https://login.yahoo.com/config/login?";
$POSTFIELDS = ".tries=1&.src=ym&.intl=us&.u=3jtlosl6ju4sc.v=0&.challenge=NZYhS1spj7zunoVhpd6KRNqaF5Kz&hasMsgr=0&.chkP=Y&.done=http://mail.yahoo.com&.pd=ym_ver=0&c=&ivt=&sg=&pad=3&aad=3&login".$php_userid."&passwd".$php_password."";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$LOGINURL);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$POSTFIELDS);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
$result = curl_exec ($ch);
curl_close ($ch);
echo $result;
?>
Is there any error? I'm doing something wrong, but i can not find it. please let me know if you can see where is my error? i need to set up this curl class.
The POST data is malformed, it should be :
[...] "login=" . urlencode($php_userid). "&passwd=" . urlencode($php_password) . ""
Instead of
[...] "login".$php_userid."&passwd".$php_password.""
You should use urlencode to ensure that the data passed in POST data is properly sent and you where missing an = for the login and passwd value.
You should use curl_error to determine what is failing during the cURL process. You can see a list of cURL error codes here: http://curl.haxx.se/libcurl/c/libcurl-errors.html
$result = curl_exec($ch);
$error = curl_error($ch);
print $error;
curl_close ($ch);
I assume the .challenge field gets dynamically generated. Your curl request uses an invalid challenge and so the server blocks it.
$cookie = cookie.txt";
Insert after: curl_setopt($ch, CURLOPT_POSTFIELDS,$POSTFIELDS); this: curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);

Categories