I am trying to cURL a remote site that results in a 403 Forbidden error. From the same server I can run the following two commands via Terminal. The first fails while the second works. How can I get my PHP code to match up to the second terminal command?
This Terminal command results in nothing returned.
curl http://www.barneys.com
This Terminal command results in a proper result (the HTML of the page)
curl -L http://www.barneys.com
My PHP Code:
$ch = curl_init('http://www.barneys.com');
$http_headers = array(
'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)' . 'Gecko/20100101 Firefox/40.0',
'Accept: */*',
'X-Requested-With: XMLHttpRequest',
'Referer: http://www.barneys.com', # IMPORTANT
'Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3',
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.barneys.com');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$response = curl_exec($ch);
$redirectURL = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL);
curl_close($ch);
echo $response;
EDIT: Here is the log from the cURL request via PHP:
* About to connect() to www.barneys.com port 80 (#0)
* Trying 23.204.27.110... * connected
* Connected to www.barneys.com (23.204.27.110) port 80 (#0)
> GET / HTTP/1.1
Host: www.barneys.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)Gecko/20100101 Firefox/40.0
Accept: */*
X-Requested-With: XMLHttpRequest
Referer: http://www.barneys.com
Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3
Connection: keep-alive
< HTTP/1.1 403 Forbidden
< Server: AkamaiGHost
< Mime-Version: 1.0
< Content-Type: text/html
< Content-Length: 265
< Expires: Mon, 11 Apr 2016 23:22:16 GMT
< Date: Mon, 11 Apr 2016 23:22:16 GMT
< Connection: close
<
* Closing connection #0
There is no request header included in your php cURL request. To fix that, add the following line right before your setting the CURLOPT_HTTPHEADER option:
curl_setopt($ch, CURLOPT_HEADER, true);
From the PHP curl docs:
CURLOPT_HEADER TRUE to include the header in the output.
Also, if you add a trailing slash to the url, the URL won't need to be rebuilt by cURL. A lot of your code appears to be unnecessary. Here is a trimmed down working example:
<?php
$ch = curl_init('http://www.barneys.com/');
$http_headers = array(
'User-Agent: Junk', // Any User-Agent will do here
);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
echo $response;
Related
I created a script for send the xml request using POST,It works perfectly on linux command line but not in php curl and this is my xml file and the command which I used.
<?xml version="1.0"?>
<methodCall>
<methodName>test.echo</methodName>
<params>
<param><value><string>nbb</string></value></param>
<param><value><string>health check: 1436777963</string></value></param>
</params></methodCall>
curl -H "Content-Type: text/xml" -d #test.xml -X POST http:my.host:5862 -v
This is my php script,
$url = 'http://my.host:5862';
#$requested = xml_gen($function_name,$epos,$time);
$requested = '<?xml version="1.0"?><methodCall><methodName>test.echo</methodName><params><param><value><string>nbb</string></value></param><param><value><string>health check: 1436777963</string></value></param></params></methodCall>';
#echo($requested);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_ENCODING ,"ISO-8859-1");
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
curl_setopt($ch, CURLOPT_POSTFIELDS, $requested);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
echo $response = curl_exec($ch);
curl_close($ch);
//Decoding the response to be displayed
echo xmlrpc_decode($response);
This is the error I get with PHP's cURL.
HTTP/1.1 400 Bad Request
Server: nginx
Date: Mon, 13 Jul 2015 15:26:05 GMT
Content-Type: text/XML
Content-Length: 0
Connection: keep-alive
Keep-Alive: timeout=20
I found the solution, Have to add the user agent on curl request.
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
I used below function to get redirected (final) url of these links
http://iprice.my/coupons/zalora/
function curlFileGetContents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0');
$result = curl_exec($ch);
$info = curl_getinfo($ch);
$effectiveUrl = $info['url'];
curl_close($ch);
return $effectiveUrl;
}
but I couldn't get anything, I wonder why? Eg I did curlFileGetContents('http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6');
The URL isn't being redirected - the links you are referring to (that only spawn on that page) are pop unders. A look at the curl header shows the URL is not being re-directed:
curl --head http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.5.9-1ubuntu4.9
Cache-Control: no-cache
Date: Fri, 22 May 2015 15:06:31 GMT
X-iPrice-Cached: 1
X-iPrice-Cached-Type: Response
The approach you currently have won't work for what you want to do.
This is an old script that is no longer working that logins to Paypal, I had a working version but deleted it by accident now I'm trying to get an old back up working.
Initialize curl:
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_REFERER, '');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLOPT_COOKIEJAR, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
cp_post_page function
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query_string);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
ob_start();
curl_exec($ch);
$response = ob_get_contents();
ob_end_clean();
return $response;
Rest of the script:
$response = cp_get_page($ch, 'https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now');
$query_string = "CONTEXT=" . $form_context. "&login_cmd=&login_params=&login_email=" . PAYPAL_EMAIL . "&login_password=" . PAYPAL_PASSWORD
. "&submit.x=Log%20In&form_charset=UTF-8&auth=$form_auth"
. "&browser_name=Firefox&browser_version=3&browser_version_full=3&operating_system=Linux";
$response = cp_post_page($ch, $form_action, $query_string); <--- Fails here
Response from the curl:
HTTP/1.1 200 OK Server: Apache Strict-Transport-Security: max-age=14400 Strict-Transport-Security: max-age=14400 Content-Type: text/html DC: slc-a-origin-www-1.paypal.com Date: Mon, 04 Nov 2013 23:45:19 GMT Content-Length: 54 Connection: keep-alive Set-Cookie: X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2402383954; domain=.paypal.com; path=/; Secure; HttpOnly Set-Cookie: X-PP-SILOVER=; Expires=Thu, 01 Jan 1970 00:00:01 GMT
Fatal Failure
Headers being sent:
POST /ca/cgi-bin/webscr?cmd=_flow&SESSION=[removed]&dispatch=[removed] HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10
Host: www.paypal.com
Accept: */*
Referer: https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now
Cookie: aksession=1383609018~id=cookieHTdyV99GKjx4ataOFl8HX+fgn4AgJSYxaLCcm+N/2KWZPsBAQbqDZ0ek9tQy9J9/gwTMcvHTSYGX65BNgg10oVSLJurTnMsIlySSW7uFaZjrVKxpCVApCbxyp5lfygq/QA1GvRvOk0k=; DC=slc-a-origin-www-1.paypal.com; X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2385606738; Apache=10.73.8.47.1383608717943546; navlns=0.0; pNTcMTtQfrJuaJiwEnWXQ6yNxfq=cRcI945EJH9tChcYgpHw1EwU36Z-qza0kxriR6IYWTNRRqPyItIzb1qCgt5K-W3DjQPwjI8yCfYaNInqtDcheZgtxQX9L7xLZM8pY7bKHS_XWsWt759waXfATBCGKYeusuJuPFdeRH2_qHRlS6s31k4inXdD-TZnRI8OEdaArFLEBx3t4-5d4NV5aeqdVSL8TuDf-kqWJFvs4Xzs2wdBEmpoocMLXGm_igzYEYHmP9KqDIUaXAMPiZUeMmPfAJiBxC8-EN5zJqI7dqs3-BgIPpCi5Is5IQe_84xDMHVBIAAgDgSUByR3-FkmBtPlfDB6rLoItmY0kT9L7yUZFW48kP3yNWHhWQ1o-InAmm; navcmd=_ship-now; cookie_check=yes; KHcl0EuY7AKSMgfvHl7J5E7hPtK=kCH9bvOH2hK9miohP-LJoRVMGNwgry4awBca8g8fKl4vOhFrS5fM82dAIUPhBGuKkYwgeLhgsPV2tVS-; cwrClyrK4LoCV1fydGbAxiNL6iG=yGLUV2f-3wPoFZV-vMEGW6jlZr7im-N5EYbO1KWk2loskGrFWziDsIkn0xLI8kQo5MScMg9TWryYuevj3SLa07p2m_IyjtTAa2W_iF4rbbPYPPsEdbypbjQjxWgd3RZw9IzCAaPJ3ZLS1R1-kJWYRevJBbZaqTApiIlRA8ALAZlOJ1g9ft_FL2GsOERgVaKpjz_aZcVeaKInfcPRHoGc9EMQTz9bFsIardyUhnQxw4Zu19vefkGYYk-1NCtLctqJ1jQ5HVn-3d7clgyddNul7JockOlurWRgPjbfkDjQ7-eXuleFhb9LkfUgpnQXPyvYTUmWh5QYnOEr9q_cRNFCRHs4vIvmhvaziv8Eyg7gFmg8v6--xOQy-a-gBqj7JEgd0kOKOWIRbneb1mm1Icd_o3lkuISss1xKvcIzXrx2scz3fq6Ys8z1VNWDNBK
Content-Length: 376
Content-Type: application/x-www-form-urlencoded
I verified that that the query string contains the right data, it is not urlencoded because curl will encode it from my understanding. I compared the POST data to a form submit using Chrome and the same number of fields are being sent to Paypal.
From what I am seeing the correct post data is being sent, the cookie exists but the login fails because it is missing something (I think it is a cookie issue, it is not a javascript issue as I login with javascript disabled). I'm not sure what other trouble shooting steps I can do to help pin point the issue.
I've solved the issue, given that one input is shown below:
<input type="hidden" id="CONTEXT_CGI_VAR" name="CONTEXT" value="REMOVED">
Even after analyzing a sample response, the reference to CONTEXT should be changed to CONTEXT_CGI_VAR in order for the curl to work correctly. Why? No idea.
I am trying to login into Tesco Mobile, using PHP curl, I have the following code:
$postData='user=XXX&password=XXXXX';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do');
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_REFERER, 'https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do');
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$result = curl_exec ($ch);
curl_close($ch);
But have been unsuccessful, please can anyone help how I can achieve this. Please find the output from Live HTTP below:
Thanks in advance.
https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do
POST /orderentry/LoginSubmit.do HTTP/1.1 Host:
paymonthly.tescomobile.com User-Agent: Mozilla/5.0 (Macintosh; Intel
Mac OS X 10.7; rv:18.0) Gecko/20100101 Firefox/18.0 Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate
Referer:
https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do;jsessionid=60af415cd448d5728a9fa98cdf99
Cookie: JSESSIONID=60af415cd448d5728a9fa98cdf99;
reevoomark_marker=349049566;
reevoo_utma=1.324660794.1360800582.1360800582.1360800582.1;
reevoo_utmb=1.5.6.1360800582; reevoo_utmc=1;
reevoo_utmz=1.1360800582.1.1.utmgclid=CK6Vp5nBtLUCFeXKtAodVGIAVA|utmccn=(not%20set)|utmcmd=(not%20set)|utmctr=tescomobile;
reevoomark_bh_session=0af0a16fc00f5fbc8b0d28c571721a72;
CUSTOMER_UUID=04fdc3e6-59e1-4038-81b7-ec7c5d63c9d4;
__utma=152930744.490151867.1360800710.1360800710.1360800710.1; __utmb=152930744; __utmc=152930744; __utmz=152930744.1360800710.1.1.utmccn=(referral)|utmcsr=phone-shop.tesco.com|utmcct=/tesco-mobile/my-tesco-mobile/|utmcmd=referral
Connection: keep-alive Content-Type: application/x-www-form-urlencoded
Content-Length: 32 user=dinoc83&password=c079378643 HTTP/1.1 200 OK
X-Powered-By: JSP/2.1 Server: Sun GlassFish Enterprise Server v2.1.1
Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date:
Thu, 14 Feb 2013 00:15:27 GMT
Try to change :
$postData='user=XXX&password=XXXXX';
by $postData=array('user=XXX&password=XXXXX');
There are some code samples in : http://php.net/manual/en/function.curl-setopt.php
Regards,
Just checked form
Form attribute action have parameter jsessionid In your code you don't use it. I think you should in first load page with form, then parse response code and get correct URL for post. And this need for initialize cookies.
Try to this in this way it's working for us posting data in another server
$postData='user=XXX&password=XXXXX'
$url ="https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do";
$ch = curl_init() or die(curl_error());
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData);
curl_setopt($ch, CURLOPT_URL,$url);// here the request is sent to payment gateway
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result=curl_exec($ch) or die(curl_error());
curl_close($ch);
You can also set $postData value this way:
$postData = array( 'user'=>'XXX', 'password'=>'XXXX' );
And note what:
Passing an array to CURLOPT_POSTFIELDS will encode the data as multipart/form-data, while passing a URL-encoded string will encode the data as application/x-www-form-urlencoded.
I am using the following code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $real_url);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $fscookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $fscookiefile);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 GTB7.1');
$r = curl_exec($ch) or die('WHY DOES THIS DIE!!!');
curl_close($ch);
Which always results in WHY DOES THIS DIE!!! or the page keeps loading for sometime and then shows nothing if I don't use
or die('WHY DOES THIS DIE!!!')
If I set CURLOPT_NOBODY to false
I get
HTTP/1.1 200 OK Date: Thu, 09 Jun 2011 01:35:07 GMT Pragma: public Expires: 0 Cache-Control: must-revalidate, post-check=0, pre-check=0, private=false Content-Type: application/force-download Content-Transfer-Encoding: binary Keep-Alive: 1000 Content-Disposition: attachment; filename*= UTF-8''Ever17.part3.rar Accept-Ranges: bytes Content-Length: 520000000 Server: Jetty(6.1.16) Rar!zs sdt�NP�(�&R�%G��65,��Ever17\Ever_17_-_The_Out_of_Infinity_CD3.iso��"j����M�K�U:� >�b�2Q�sT?G��+iM���qN�'��&�-d�z�����} xi�<�3\�d��Z:|�/���#�gl��$��~#��u�וۊ \��jմg�)�L��#&��;��5M�s}���(�Đ$�yGwӏ$�E����$�Z���:�/�����r͐�8�&������#Q�00��>�)�n�̌}o�H����2Z���y�S㞝�H��=�9�[������E�nЭ��\�9Ǘ!{&y�j/���Ȇg������FW��~G�5������F-&����ƭ��9<���^7����(�HY(a����%�ɚQYL_���ܽ�X|��&
Why am I not getting any output with CURLOPT_NOBODY set to true ????
I am using PHP 5.1.6 on Apache/2.2.3 and have curl-7.15.5-9.el5_6.2.i386 installed
(That's a seriously outdated curl version)
Try using the command line version (preferably from the same host running that PHP script) like 'curl -I [URL]' and make sure the web site truly responds nice to a HEAD request.
Add -v or --trace-ascii for full debug/trace output.