get redirect url using curl_getinfo failed - php

I used below function to get redirected (final) url of these links
http://iprice.my/coupons/zalora/
function curlFileGetContents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0');
$result = curl_exec($ch);
$info = curl_getinfo($ch);
$effectiveUrl = $info['url'];
curl_close($ch);
return $effectiveUrl;
}
but I couldn't get anything, I wonder why? Eg I did curlFileGetContents('http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6');

The URL isn't being redirected - the links you are referring to (that only spawn on that page) are pop unders. A look at the curl header shows the URL is not being re-directed:
curl --head http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.5.9-1ubuntu4.9
Cache-Control: no-cache
Date: Fri, 22 May 2015 15:06:31 GMT
X-iPrice-Cached: 1
X-iPrice-Cached-Type: Response
The approach you currently have won't work for what you want to do.

Related

I cant reach the url desired with curl

I try curl a website for show it in mine but always I stuck in a white page. I think because it have a redirection to a login form but I dont sure if is the reason because I failed. You can access the url that I use without login.
Here the code
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec($ch);
curl_close($ch);
htmlentities($output);
The html result:
<html>
<head>
<title></title>
</head>
<body>
<!--
<h1>No se ha aceptado el cookie</h1>
-->
</body>
</html>
Your request url only validate requests which have JSESSIONID cookie. So before you must grab valid cookie:
<?php
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
$useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
This gives you three headers for requests:
HTTP/1.1 302 Movido temporalmente
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=648967C5FC907B1225EC61E9A65443E5; Path=/pnfg
Location: http://www.faf.es/pnfg/NLogin
Content-Length: 0 Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
HTTP/1.1 302 Movido temporalmente
Server: Apache-Coyote/1.1
Cache-Control: no-cache
Pragma: no-cache
Set-Cookie: JSESSIONID=9973308FA3AF81BE66C2F8C124671870; Path=/pnfg
Location: http://www.faf.es/pnfg/NLogin?NSess=1
Content-Length: 0 Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
HTTP/1.1 200 OK Server: Apache-Coyote/1.1
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html;charset=ISO-8859-15
Content-Length: 117
Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
We have three header result because we said, request has CURLOPT_FOLLOWLOCATION option. So, we made three request in curl.
I identified, second JSESSIONID cookie value is valid. So, for getting web site information we must use cookie as below:
<?php
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
$useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36';
curl_setopt($ch, CURLOPT_URL, $url);
// Secondary request cookie.
curl_setopt($ch, CURLOPT_COOKIE, "JSESSIONID=9973308FA3AF81BE66C2F8C124671870");
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
When web site content comes, do whatever you want.
You should revisit the htmlentities() documentation again; it doesn't do what it appears you believe it does.
In short, htmlentities() returns a string. You aren't capturing or otherwise outputting this string result anywhere. Use echo (or similar) to make sure the result of htmlentities($output) is output to where you expect it.
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec($ch);
curl_close($ch);
echo htmlentities($output);

PHP cURL results in 403 Forbidden

I am trying to cURL a remote site that results in a 403 Forbidden error. From the same server I can run the following two commands via Terminal. The first fails while the second works. How can I get my PHP code to match up to the second terminal command?
This Terminal command results in nothing returned.
curl http://www.barneys.com
This Terminal command results in a proper result (the HTML of the page)
curl -L http://www.barneys.com
My PHP Code:
$ch = curl_init('http://www.barneys.com');
$http_headers = array(
'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)' . 'Gecko/20100101 Firefox/40.0',
'Accept: */*',
'X-Requested-With: XMLHttpRequest',
'Referer: http://www.barneys.com', # IMPORTANT
'Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3',
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.barneys.com');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$response = curl_exec($ch);
$redirectURL = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL);
curl_close($ch);
echo $response;
EDIT: Here is the log from the cURL request via PHP:
* About to connect() to www.barneys.com port 80 (#0)
* Trying 23.204.27.110... * connected
* Connected to www.barneys.com (23.204.27.110) port 80 (#0)
> GET / HTTP/1.1
Host: www.barneys.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)Gecko/20100101 Firefox/40.0
Accept: */*
X-Requested-With: XMLHttpRequest
Referer: http://www.barneys.com
Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3
Connection: keep-alive
< HTTP/1.1 403 Forbidden
< Server: AkamaiGHost
< Mime-Version: 1.0
< Content-Type: text/html
< Content-Length: 265
< Expires: Mon, 11 Apr 2016 23:22:16 GMT
< Date: Mon, 11 Apr 2016 23:22:16 GMT
< Connection: close
<
* Closing connection #0
There is no request header included in your php cURL request. To fix that, add the following line right before your setting the CURLOPT_HTTPHEADER option:
curl_setopt($ch, CURLOPT_HEADER, true);
From the PHP curl docs:
CURLOPT_HEADER TRUE to include the header in the output.
Also, if you add a trailing slash to the url, the URL won't need to be rebuilt by cURL. A lot of your code appears to be unnecessary. Here is a trimmed down working example:
<?php
$ch = curl_init('http://www.barneys.com/');
$http_headers = array(
'User-Agent: Junk', // Any User-Agent will do here
);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
echo $response;

Trouble shooting PHP cURL

This is an old script that is no longer working that logins to Paypal, I had a working version but deleted it by accident now I'm trying to get an old back up working.
Initialize curl:
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_REFERER, '');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLOPT_COOKIEJAR, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
cp_post_page function
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query_string);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
ob_start();
curl_exec($ch);
$response = ob_get_contents();
ob_end_clean();
return $response;
Rest of the script:
$response = cp_get_page($ch, 'https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now');
$query_string = "CONTEXT=" . $form_context. "&login_cmd=&login_params=&login_email=" . PAYPAL_EMAIL . "&login_password=" . PAYPAL_PASSWORD
. "&submit.x=Log%20In&form_charset=UTF-8&auth=$form_auth"
. "&browser_name=Firefox&browser_version=3&browser_version_full=3&operating_system=Linux";
$response = cp_post_page($ch, $form_action, $query_string); <--- Fails here
Response from the curl:
HTTP/1.1 200 OK Server: Apache Strict-Transport-Security: max-age=14400 Strict-Transport-Security: max-age=14400 Content-Type: text/html DC: slc-a-origin-www-1.paypal.com Date: Mon, 04 Nov 2013 23:45:19 GMT Content-Length: 54 Connection: keep-alive Set-Cookie: X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2402383954; domain=.paypal.com; path=/; Secure; HttpOnly Set-Cookie: X-PP-SILOVER=; Expires=Thu, 01 Jan 1970 00:00:01 GMT
Fatal Failure
Headers being sent:
POST /ca/cgi-bin/webscr?cmd=_flow&SESSION=[removed]&dispatch=[removed] HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10
Host: www.paypal.com
Accept: */*
Referer: https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now
Cookie: aksession=1383609018~id=cookieHTdyV99GKjx4ataOFl8HX+fgn4AgJSYxaLCcm+N/2KWZPsBAQbqDZ0ek9tQy9J9/gwTMcvHTSYGX65BNgg10oVSLJurTnMsIlySSW7uFaZjrVKxpCVApCbxyp5lfygq/QA1GvRvOk0k=; DC=slc-a-origin-www-1.paypal.com; X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2385606738; Apache=10.73.8.47.1383608717943546; navlns=0.0; pNTcMTtQfrJuaJiwEnWXQ6yNxfq=cRcI945EJH9tChcYgpHw1EwU36Z-qza0kxriR6IYWTNRRqPyItIzb1qCgt5K-W3DjQPwjI8yCfYaNInqtDcheZgtxQX9L7xLZM8pY7bKHS_XWsWt759waXfATBCGKYeusuJuPFdeRH2_qHRlS6s31k4inXdD-TZnRI8OEdaArFLEBx3t4-5d4NV5aeqdVSL8TuDf-kqWJFvs4Xzs2wdBEmpoocMLXGm_igzYEYHmP9KqDIUaXAMPiZUeMmPfAJiBxC8-EN5zJqI7dqs3-BgIPpCi5Is5IQe_84xDMHVBIAAgDgSUByR3-FkmBtPlfDB6rLoItmY0kT9L7yUZFW48kP3yNWHhWQ1o-InAmm; navcmd=_ship-now; cookie_check=yes; KHcl0EuY7AKSMgfvHl7J5E7hPtK=kCH9bvOH2hK9miohP-LJoRVMGNwgry4awBca8g8fKl4vOhFrS5fM82dAIUPhBGuKkYwgeLhgsPV2tVS-; cwrClyrK4LoCV1fydGbAxiNL6iG=yGLUV2f-3wPoFZV-vMEGW6jlZr7im-N5EYbO1KWk2loskGrFWziDsIkn0xLI8kQo5MScMg9TWryYuevj3SLa07p2m_IyjtTAa2W_iF4rbbPYPPsEdbypbjQjxWgd3RZw9IzCAaPJ3ZLS1R1-kJWYRevJBbZaqTApiIlRA8ALAZlOJ1g9ft_FL2GsOERgVaKpjz_aZcVeaKInfcPRHoGc9EMQTz9bFsIardyUhnQxw4Zu19vefkGYYk-1NCtLctqJ1jQ5HVn-3d7clgyddNul7JockOlurWRgPjbfkDjQ7-eXuleFhb9LkfUgpnQXPyvYTUmWh5QYnOEr9q_cRNFCRHs4vIvmhvaziv8Eyg7gFmg8v6--xOQy-a-gBqj7JEgd0kOKOWIRbneb1mm1Icd_o3lkuISss1xKvcIzXrx2scz3fq6Ys8z1VNWDNBK
Content-Length: 376
Content-Type: application/x-www-form-urlencoded
I verified that that the query string contains the right data, it is not urlencoded because curl will encode it from my understanding. I compared the POST data to a form submit using Chrome and the same number of fields are being sent to Paypal.
From what I am seeing the correct post data is being sent, the cookie exists but the login fails because it is missing something (I think it is a cookie issue, it is not a javascript issue as I login with javascript disabled). I'm not sure what other trouble shooting steps I can do to help pin point the issue.
I've solved the issue, given that one input is shown below:
<input type="hidden" id="CONTEXT_CGI_VAR" name="CONTEXT" value="REMOVED">
Even after analyzing a sample response, the reference to CONTEXT should be changed to CONTEXT_CGI_VAR in order for the curl to work correctly. Why? No idea.

How request xml via PHP-cURL POST request

I want a PHP-cURL script to do the following request.
http://site5.way2sms.com/QuickContacts
POST /QuickContacts HTTP/1.1
Host: site5.way2sms.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://site5.way2sms.com/Main.action?id=0CD36BD332A5C7AE77FDBA1CBDBFFBB6.w809
Content-Length: 16
gads=ID=e2cdae9b764355fa:T=1333862873:S=ALNI_MYvxochQ56ILMvBDr4oyyqCIDVn3w
Pragma: no-cache
Cache-Control: no-cache
folder=DashBoard
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html
Content-Length: 1901
Date: Sun, 08 Apr 2012 06:07:52 GMT
Connection: close
I think the returned content is in XML format. How to handle them?
If you want generate a post request to the URL and catch the returned data using cURL you can use the following function.
function cURL($url, $header=NULL, $cookie=NULL, $p=NULL)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_NOBODY, $header);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
if ($p) {
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $p);
}
$result = curl_exec($ch);
if ($result) {
return $result;
} else {
return curl_error($ch);
}
curl_close($ch);
}
Like,
$data = cURL("http://site5.way2sms.com/QuickContacts", NULL, NULL, array("post_var" => value, "another_post_var" => val2));

CURLOPT_NOBODY not working

I am using the following code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $real_url);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $fscookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $fscookiefile);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 GTB7.1');
$r = curl_exec($ch) or die('WHY DOES THIS DIE!!!');
curl_close($ch);
Which always results in WHY DOES THIS DIE!!! or the page keeps loading for sometime and then shows nothing if I don't use
or die('WHY DOES THIS DIE!!!')
If I set CURLOPT_NOBODY to false
I get
HTTP/1.1 200 OK Date: Thu, 09 Jun 2011 01:35:07 GMT Pragma: public Expires: 0 Cache-Control: must-revalidate, post-check=0, pre-check=0, private=false Content-Type: application/force-download Content-Transfer-Encoding: binary Keep-Alive: 1000 Content-Disposition: attachment; filename*= UTF-8''Ever17.part3.rar Accept-Ranges: bytes Content-Length: 520000000 Server: Jetty(6.1.16) Rar!zs sdt�NP�(�&R�%G��65,��Ever17\Ever_17_-_The_Out_of_Infinity_CD3.iso��"j����M�K�U:� >�b�2Q�sT?G��+iM���qN�'��&�-d�z�����} xi�<�3\�d��Z:|�/���#�gl��$��~#��u�וۊ \��jմg�)�L��#&��;��5M�s}���(�Đ$�yGwӏ$�E����$�Z���:�/�����r͐�8�&������#Q�00��>�)�n�̌}o�H����2Z���y�S㞝�H��=�9�[������E�nЭ��\�׊9Ǘ!{&y�j/���Ȇg������FW��~G�5������F-&����ƭ��9<���^7����(�HY(a����%�ɚQYL_���ܽ�X|��&
Why am I not getting any output with CURLOPT_NOBODY set to true ????
I am using PHP 5.1.6 on Apache/2.2.3 and have curl-7.15.5-9.el5_6.2.i386 installed
(That's a seriously outdated curl version)
Try using the command line version (preferably from the same host running that PHP script) like 'curl -I [URL]' and make sure the web site truly responds nice to a HEAD request.
Add -v or --trace-ascii for full debug/trace output.

Categories