I have this script:
$ch = curl_init($url_path.'admin/');
$cookiefile = $srv_path."admin/cookie.txt" ;
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec ($ch);
//
curl_setopt($ch, CURLOPT_PROXY, "http://127.0.0.1.:8888");
curl_setopt($ch, CURLOPT_PROXYPORT, 8888);
curl_setopt($ch, CURLOPT_HEADER, 1);
preg_match('/^Set-Cookie: (.*?);/m', curl_exec($ch), $m);
$xid = substr($m[1], 10);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url_path.'admin/login.php');
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);
//SET POST PARAMETERS : FORM VALUES FOR EACH FIELD
curl_setopt($ch, CURLOPT_POSTFIELDS, 'xid_7d781='.$xid.'$username=*****&password=******&mode=login&usertype=P&redirect=admin');
# Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL # not to print out the results of its query.
# Instead, it will return the results as a string return value # from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // Allow redirection
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
//
//echo('EXECUTE 1st REQUEST (FORM LOGIN)');
$page = curl_exec ($ch);
curl_close ($ch);
echo $page;
but when i call it, the form fields are blank. Furthermore the method that returns is GET and not POST. Why returns GET?
The same code if i try it for another domain run correctly and that is also very strange. Is it possible the post method blocked by something?
I tried to echo any possible errors with var_dump(curl_error($ch)); but the string is empty.
Related
I'm having trouble understanding how CURL handles headers.
I have a site.com/page1 that I want to access with CURL it does a 308 redirect to site.com/page2/file.zip
What I need is to go through site.com/page1 with CURL but download site.com/page2/file.zip directly from site.com
I'm using this code but it does not work as expected. It accesses site.com/page1 redirects to site.com/page2/file.zip but opens the file in the browser
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $_cookie_file);
curl_setopt($ch, CURLOPT_COOKIEFILE, $_cookie_file);
curl_setopt($ch, CURLOPT_REFERER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_exec($ch);
$error = curl_getinfo($ch);
curl_close($ch);
I guess if I can keep the response headers I'll fix the problem. But how do I do it ?? How do I use the same headers for the CURL visitor that the site I am accessing is sending me.
You want the cURL option RETURNTRANSFER set to true so what is returned comes back to you. Since you are trying to save a ZIP file you'll also need to open a file and use the CURLOPT_FILE option to tell cURL where to save your ZIP file.
curl_setopt ($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt ($ch, CURLOPT_TIMEOUT,'180'); # 3 minute timeout
$FileOut = fopen('MyZIP_File.zip','w') or die('Could not open the output data file');
curl_setopt ($ch, CURLOPT_FILE,$FileOut);
curl_exec ($ch);
fclose($FileOut) or die('We ran into a problem saving data file');
This solved the problem.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $_cookie_file);
curl_setopt($ch, CURLOPT_COOKIEFILE, $_cookie_file);
curl_setopt($ch, CURLOPT_REFERER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HEADER, true);
$result = curl_exec($ch);
if (preg_match('~Location: (.*)~i', $result, $match)) {
$location = trim($match[1]);
header('Location:' . $location);
}
I'm trying to login into twitter with php + curl, but I think there's something wrong with my request because I get this as response:
Something is technically wrong.
Thanks for noticing—we\'re going to fix it up and have things back to
normal soon.
The php code I'm using is:
<?php
$ch = curl_init($sTarget);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $sPost);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $_CKS);
curl_setopt($ch, CURLOPT_COOKIEFILE, $_CKS);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-type: multipart/form-data"));
curl_setopt($ch, CURLOPT_REFERER, "https://twitter.com");
?>
$sPost looks like this:
session[username_or_email]=user&session[password]=password&remember_me=1&scribe_log=&redirect_after_login=&authenticity_token=6e706165609354bd7a92a99cf94e09140ea86c6f
The code first fetches the login form fields so that the post variables do have the proper values(auth token). I just tried about everything and yes I know there's an api for this but I'd rather find out how to do it manual for learning's sake.
Thanks in advance.
The CURLOPT_COOKIESESSION is used to indicate a new session. That's not what you want, since you need to send the session cookie in the second post.
I got the twitter login to work with this code:
<?php
# First call gets hidden form field authenticity_token
# and session cookie
$ch = curl_init();
$sTarget = "https://twitter.com/";
curl_setopt($ch, CURLOPT_URL, $sTarget);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/cookie.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://twitter.com/");
$html = curl_exec($ch);
# parse authenticity_token out of html response
preg_match('/<input type="hidden" value="([a-zA-Z0-9]*)" name="authenticity_token"\/>/', $html, $match);
$authenticity_token = $match[1];
$username = "your#email.com";
$password = "password";
# set post data
$sPost = "session[username_or_email]=$username&session[password]=$password&return_to_ssl=true&scribe_log=&redirect_after_login=%2F&authenticity_token=$authenticity_token";
# second call is a post and performs login
$sTarget = "https://twitter.com/sessions";
curl_setopt($ch, CURLOPT_URL, $sTarget);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $sPost);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-type: application/x-www-form-urlencoded"));
# display server response
curl_exec($ch);
curl_close($ch);
?>
PS: Sorry for not reading your post properly the first time.
I noticed two things:
1) Try to URL encode your POST data
such as:
session%5Busername_or_email%5D=user&session%5Bpassword%5D=password...
instead of:
session[username_or_email]=user&session[password]=password...
2) twitter has a hidden field named authenticity_token in the login form. It is bound to the session. Thus you cannot use a static authenticity_token, you have to read the login form first and use the authenticity_token field from there.
Hope that helps.
In the below code i am trying to post the username and password but not able to create the cookie using post, if i do the same with get method it creates the cookie.
$BASEURL="http://localhost/openx/www/api/json/index.php/main/authenticate/";
$POSTFIELDS='username="'.$username.'"&password="'.$password.'"';
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FAILONERROR, 0);
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $POSTFIELDS);
curl_setopt($ch, CURLOPT_URL, $BASEURL);
$returned = curl_exec($ch);
The curl_getinfo($ch) returns 200 OK http code, can anybody please advice what is wrong with the above code.
Thanks in advance.
The cookiejar is only saved when you close the curl handle using curl_close($ch).
From the manual:
CURLOPT_COOKIEFILE
The name of the file containing the cookie data. The cookie file can be in Netscape format, or just plain HTTP-style headers dumped into a file. If the name is an empty string, no cookies are loaded, but cookie handling is still enabled.
CURLOPT_COOKIEJAR
The name of a file to save all internal cookies to when the handle is closed, e.g. after a call to curl_close.
http://www.php.net/manual/en/function.curl-setopt.php
I have changed my code to below and now working for me.
$ckfile = tempnam ("/tmp/", "CURLCOOKIE");
$BASEURL='http://localhost/openx/www/api/json/index.php/main/authenticate/';
$POSTFIELDS='username='.$username.'&password='.$password.'';
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookieFileName");
curl_setopt($ch, CURLOPT_URL,'http://localhost/openx/www/api/json/index.php/main/authenticate/');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $POSTFIELDS);
ob_start(); // prevent any output
$result=curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
$result = curl_exec($ch);
curl_close($ch);
I am looking to set curl to use a proxy server. The url is provided by an html form, which has not been a problem. Without the proxy it works fine. I have found code on this and other sites, but they do not work. Any help in finding the correct solution would be much appreciated. I feel that the bellow are close, but that I am missing something. Thank You.
The bellow code I adapted from here http://www.webmasterworld.com/forum88/10572.htm but it returns an error message about a missing T_VARIABLE on line 12.
<?
$url = '$_POST[1]';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_PROXY, '66.96.200.39:80');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST,'GET');
curl_setopt ($ch, CURLOPT_HEADER, 1)
curl_exec ($ch);
$curl_info = curl_getinfo($ch);
curl_close($ch);
echo '<br />';
print_r($curl_info);
?>
The bellow is from curl through proxy returns no content
<?
$proxy = "66.96.200.39:80";
$proxy = explode(':', $proxy);
$url = "$_POST[1]";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
?>
is currently live on pelican-cement.com but also does not work.
UPDATE:
Thank you for all your help, I made the above changes. Now it only returns a blank screen.
<?
$url = $_POST['1'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_PROXY, '66.96.200.39:80');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST,'GET');
curl_setopt ($ch, CURLOPT_HEADER, 1);
curl_exec ($ch);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
?>
Here is a working version with your bugs removed.
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '127.0.0.1:8888';
//$proxyauth = 'user:password';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
I have added CURLOPT_PROXYUSERPWD in case any of your proxies require a user name and password.
I set CURLOPT_RETURNTRANSFER to 1, so that the data will be returned to $curl_scraped_page variable.
I removed a second extra curl_exec($ch); which would stop the variable being returned.
I consolidated your proxy IP and port into one setting.
I also removed CURLOPT_HTTPPROXYTUNNEL and CURLOPT_CUSTOMREQUEST as it was the default.
If you don't want the headers returned, comment out CURLOPT_HEADER.
To disable the proxy simply set it to null.
curl_setopt($ch, CURLOPT_PROXY, null);
Any questions feel free to ask, I work with cURL every day.
I have explained use of various CURL options required for CURL PROXY.
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '127.0.0.1:8888';
$proxyauth = 'user:password';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); // URL for CURL call
curl_setopt($ch, CURLOPT_PROXY, $proxy); // PROXY details with port
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth); // Use if proxy have username and password
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5); // If expected to call with specific PROXY type
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // If url has redirects then go to the final redirected URL.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0); // Do not outputting it out directly on screen.
curl_setopt($ch, CURLOPT_HEADER, 1); // If you want Header information of response else make 0
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
root#APPLICATIOSERVER:/var/www/html# php connectiontest.php
61e23468-949e-4103-8e08-9db09249e8s1 OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 10.172.123.1:80 root#APPLICATIOSERVER:/var/www/html#
Post declaring the proxy settings in the php script file issue has been fixed.
$proxy = '10.172.123.1:80';
curl_setopt($cSession, CURLOPT_PROXY, $proxy); // PROXY details with port
Here is a well tested function which i used for my projects with detailed self explanatory comments
There are many times when the ports other than 80 are blocked by server firewall so the code appears to be working fine on localhost but not on the server
function get_page($url){
global $proxy;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 200); // http request timeout 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection responce
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_ENCODING, "gzip"); // the page encoding
$data = curl_exec($ch); // execute the http request
curl_close($ch); // close the connection
return $data;
}
I'm using cUrl(PHP) to post a login request and store response in cookie file.
In my second request I'm passing cookie in header and post data to verify it.
Issue is that cookie file is not created in first succesful request results in failure for second request. Please suggest me where I'm doing wrong.
$cookiefile="/var/www/html/dimdim/cook.txt";
$url_log="http://my.dimdim.com/api/auth/login";
$p_log='request={"account":"bin6k","password":"password","group":"all"}';
$url_ver="http://my.dimdim.com/api/auth/verify";
$p_ver='request={"account":"bin6k","password":"password","group":"all"}';
$ch = curl_init();
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_URL,$url_log);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $p_log);
ob_start(); // prevent any output
$retval=curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);
//print_r($retval);
unset($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_URL,$url_ver);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $p_log);
$buf2 = curl_exec ($ch);
curl_close ($ch);
echo "<PRE>".htmlentities($buf2);
i had the same problem use this for running in windows localhost server
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__) . "/cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . "/cookies.txt");
Try adding a curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile); in there as well.
Also, there's no need to start a fresh curl instance for each request. You can reuse the same instance for multiple requests. Just set the appropriate CURLOPT (url, postfields, get, etc...) each time and curl will sort things out internally.
Try this function the cookijar option is added:
function execute($toLoad) {
if ( !preg_match( '/^http/', $toLoad ) ) {
$toLoad = 'http://'.$toLoad;
}
$cookiefile = APP_PATH.'tmp/cookie.txt';
$data = array();
$ch = curl_init();
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT,"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.6) Gecko/20091216 Fedora/3.5.6-1.fc11 Firefox/3.5.6 GTB6");
curl_setopt($ch, CURLOPT_URL, $toLoad); // The page to download
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, 'http://somesite.com/' );
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
$data['data'] = curl_exec($ch);
$data['status'] = curl_getinfo($ch);
//$this->out(curl_error($ch));
//$this->out(curl_getinfo($ch));
//$this->out('');
//$this->out($data);
//$this->out('');
curl_close($ch);
return $data;
}
you could also try reading the header of the returned data for Set-Cookie: SOMEKEY
and then in the curlopt use
if ($header) {
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Cookie:'.$header));
}
Thanks