I'm trying to get a json result from a website to analyze in my system.
In the first request of the day I receive Status 200 OK I get the result that I'm expecting.
But if I try to get the result again making a refresh in the page for example, I receive the following error
"close Cache-Control: no-cache Pragma: no-cache "
With the following HTTP HEADER
"HTTP/1.1 302 Found : Moved Temporarily Location: http://localhost Cache-Control: private, no-cache, max-age=0, no-store, must-revalidate Expires: Thu, 01 Jan 1970 00:00:01 +0000 Pragma: no-cache
For some reason the server returns error 302 and move to localhost and I have to wait until the next day to get the result 200 again.
Running var_dump(curl_error($curl));. I get bool(false)
My code
<?php $cookie="files/cookies.txt";
$url = "https://extra.bet365.com/ResultsApi/GetResults?sportName=sport&sportId=146&fixtureId=104716787&competitionId=20700663&fromDate=2021-07-01T04:05:00&toDate=2021-07-01T04:05:00&challengeId=62233643&marketOverride=";
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($curl, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($curl, CURLOPT_COOKIEFILE, $cookie);
$headers = array(
"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:89.0) Gecko/20100101 Firefox/89.0",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3",
"Connection: keep-alive",
);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$resp = curl_exec($curl);
curl_close($curl);
var_dump($resp); ?>
Related
I used below function to get redirected (final) url of these links
http://iprice.my/coupons/zalora/
function curlFileGetContents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0');
$result = curl_exec($ch);
$info = curl_getinfo($ch);
$effectiveUrl = $info['url'];
curl_close($ch);
return $effectiveUrl;
}
but I couldn't get anything, I wonder why? Eg I did curlFileGetContents('http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6');
The URL isn't being redirected - the links you are referring to (that only spawn on that page) are pop unders. A look at the curl header shows the URL is not being re-directed:
curl --head http://iprice.my/coupons/zalora/#007b9a6024d19edec91d04c2e92e143e744c83b6
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.5.9-1ubuntu4.9
Cache-Control: no-cache
Date: Fri, 22 May 2015 15:06:31 GMT
X-iPrice-Cached: 1
X-iPrice-Cached-Type: Response
The approach you currently have won't work for what you want to do.
I'm writing a web scraper for a project, which needs to login and save some pages, but after even login, saving the cookie.txt it redirects back to login page. Looks like it's not logging in.
Here is my code:
<?php
$ch = curl_init();
$cookie_file_path = 'cookie.txt';
$cookie_file_path = realpath($cookie_file_path);
$data = array();
$data['txtUser'] = "username";
$data['txtPass'] = "password";
$postStr = "";
foreach($data as $key=>$d){
$postStr .= $key.'='.urlencode($d).'&';
}
$postStr = substr($postStr,0,-1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$agent = $_SERVER["HTTP_USER_AGENT"];
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
//new ones
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_URL,"http://madstore.su/login.php");
curl_setopt($ch,CURLOPT_POST,TRUE);
curl_setopt($ch,CURLOPT_POSTFIELDS,$postStr);
curl_exec ($ch); // execute the curl command
echo 'Curl error: ' . curl_error($ch); //no errrors
curl_close ($ch);
unset($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_URL,"http://madstore.su/index.php");
//new ones
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_exec ($ch);
echo 'Curl error: ' . curl_error($ch);
curl_close ($ch);
?>
I have read about all questions on StackOverflow and searching on Google since hours.
Here is what I'm getting in cookie.txt:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_.madstore.su TRUE / FALSE 1577145000 __cfduid d3f365e8218ab84f921e43db0d1500e7c1391327438626
madstore.su FALSE / FALSE 0 PHPSESSID t41g1j9cdl800e9qdj2pq96ef1
Here is what I'm getting as curl error:
HTTP/1.1 302 Found
Server: cloudflare-nginx
Date: Sun, 02 Feb 2014 08:01:40 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.1.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
location: login.php
CF-RAY: f655ad243d007e5-LAX
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Sun, 02 Feb 2014 08:01:41 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.1.6
CF-RAY: f655ad5141f07e5-LAX
I will highly appreciate if anyone can help me with this issue.
Your approach is wrong. First browse the login page(http://madstore.su/login.php) using curl so that it can store the cookies into the file.
Then do the curl posting and use the cookie that saved earlier. Also you are missing this POST parameter, so add this with your data.
$data['btnLogin'] = "Log in";
When the login is done, then browse the required page using your final curl's GET.
This is an old script that is no longer working that logins to Paypal, I had a working version but deleted it by accident now I'm trying to get an old back up working.
Initialize curl:
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_REFERER, '');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLOPT_COOKIEJAR, PAYPAL_COOKIE_FILE);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
cp_post_page function
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query_string);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
ob_start();
curl_exec($ch);
$response = ob_get_contents();
ob_end_clean();
return $response;
Rest of the script:
$response = cp_get_page($ch, 'https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now');
$query_string = "CONTEXT=" . $form_context. "&login_cmd=&login_params=&login_email=" . PAYPAL_EMAIL . "&login_password=" . PAYPAL_PASSWORD
. "&submit.x=Log%20In&form_charset=UTF-8&auth=$form_auth"
. "&browser_name=Firefox&browser_version=3&browser_version_full=3&operating_system=Linux";
$response = cp_post_page($ch, $form_action, $query_string); <--- Fails here
Response from the curl:
HTTP/1.1 200 OK Server: Apache Strict-Transport-Security: max-age=14400 Strict-Transport-Security: max-age=14400 Content-Type: text/html DC: slc-a-origin-www-1.paypal.com Date: Mon, 04 Nov 2013 23:45:19 GMT Content-Length: 54 Connection: keep-alive Set-Cookie: X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2402383954; domain=.paypal.com; path=/; Secure; HttpOnly Set-Cookie: X-PP-SILOVER=; Expires=Thu, 01 Jan 1970 00:00:01 GMT
Fatal Failure
Headers being sent:
POST /ca/cgi-bin/webscr?cmd=_flow&SESSION=[removed]&dispatch=[removed] HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10
Host: www.paypal.com
Accept: */*
Referer: https://www.paypal.com/ca/cgi-bin/webscr?cmd=_ship-now
Cookie: aksession=1383609018~id=cookieHTdyV99GKjx4ataOFl8HX+fgn4AgJSYxaLCcm+N/2KWZPsBAQbqDZ0ek9tQy9J9/gwTMcvHTSYGX65BNgg10oVSLJurTnMsIlySSW7uFaZjrVKxpCVApCbxyp5lfygq/QA1GvRvOk0k=; DC=slc-a-origin-www-1.paypal.com; X-PP-SILOVER=name%3DLIVE5.WEB.1%26silo_version%3D880%26app%3Dslingshot%26TIME%3D2385606738; Apache=10.73.8.47.1383608717943546; navlns=0.0; pNTcMTtQfrJuaJiwEnWXQ6yNxfq=cRcI945EJH9tChcYgpHw1EwU36Z-qza0kxriR6IYWTNRRqPyItIzb1qCgt5K-W3DjQPwjI8yCfYaNInqtDcheZgtxQX9L7xLZM8pY7bKHS_XWsWt759waXfATBCGKYeusuJuPFdeRH2_qHRlS6s31k4inXdD-TZnRI8OEdaArFLEBx3t4-5d4NV5aeqdVSL8TuDf-kqWJFvs4Xzs2wdBEmpoocMLXGm_igzYEYHmP9KqDIUaXAMPiZUeMmPfAJiBxC8-EN5zJqI7dqs3-BgIPpCi5Is5IQe_84xDMHVBIAAgDgSUByR3-FkmBtPlfDB6rLoItmY0kT9L7yUZFW48kP3yNWHhWQ1o-InAmm; navcmd=_ship-now; cookie_check=yes; KHcl0EuY7AKSMgfvHl7J5E7hPtK=kCH9bvOH2hK9miohP-LJoRVMGNwgry4awBca8g8fKl4vOhFrS5fM82dAIUPhBGuKkYwgeLhgsPV2tVS-; cwrClyrK4LoCV1fydGbAxiNL6iG=yGLUV2f-3wPoFZV-vMEGW6jlZr7im-N5EYbO1KWk2loskGrFWziDsIkn0xLI8kQo5MScMg9TWryYuevj3SLa07p2m_IyjtTAa2W_iF4rbbPYPPsEdbypbjQjxWgd3RZw9IzCAaPJ3ZLS1R1-kJWYRevJBbZaqTApiIlRA8ALAZlOJ1g9ft_FL2GsOERgVaKpjz_aZcVeaKInfcPRHoGc9EMQTz9bFsIardyUhnQxw4Zu19vefkGYYk-1NCtLctqJ1jQ5HVn-3d7clgyddNul7JockOlurWRgPjbfkDjQ7-eXuleFhb9LkfUgpnQXPyvYTUmWh5QYnOEr9q_cRNFCRHs4vIvmhvaziv8Eyg7gFmg8v6--xOQy-a-gBqj7JEgd0kOKOWIRbneb1mm1Icd_o3lkuISss1xKvcIzXrx2scz3fq6Ys8z1VNWDNBK
Content-Length: 376
Content-Type: application/x-www-form-urlencoded
I verified that that the query string contains the right data, it is not urlencoded because curl will encode it from my understanding. I compared the POST data to a form submit using Chrome and the same number of fields are being sent to Paypal.
From what I am seeing the correct post data is being sent, the cookie exists but the login fails because it is missing something (I think it is a cookie issue, it is not a javascript issue as I login with javascript disabled). I'm not sure what other trouble shooting steps I can do to help pin point the issue.
I've solved the issue, given that one input is shown below:
<input type="hidden" id="CONTEXT_CGI_VAR" name="CONTEXT" value="REMOVED">
Even after analyzing a sample response, the reference to CONTEXT should be changed to CONTEXT_CGI_VAR in order for the curl to work correctly. Why? No idea.
I am getting some info from a https web server using PHP plus cURL. All info got as HTTP GET is ok but when I need to do some HTTP POST I get a no sense output. Web server is ok as If i get the info from a web browser all works ok.
I am using following code:
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1");
if($method == "POST"){
print_r($post_fields);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch,CURLOPT_HTTPHEADER, array (
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3",
"Accept-Encoding: gzip, deflate",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7"
));
}
if ($usecookie) {
curl_setopt($ch, CURLOPT_COOKIEJAR, $usecookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $usecookie);
}
if ($refer != "") {
curl_setopt($ch, CURLOPT_REFERER, $refer );
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
Answer header is:
HTTP/1.1 200 OK
Date: Sun, 20 Nov 2011 11:04:39 GMT
Server: Apache
Cache-Control: must-revalidate
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
max-age: Thu, 01 Jan 1970 00:00:00 GMT
X-Powered-By: Servlet/2.4 JSP/2.0
idWl: PRO-LOW16_6604
Vary: Accept-Encoding
Content-Encoding: gzip
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Any idea about where the problem could be?
It clearly shows its gziped...
Content-Encoding: gzip
Transfer-Encoding: chunked
Passing the returned data through the below function will inflate it back to readable content.
function gzdecoder($d){
$f=ord(substr($d,3,1));
$h=10;$e=0;
if($f&4){
$e=unpack('v',substr($d,10,2));
$e=$e[1];$h+=2+$e;
}
if($f&8){
$h=strpos($d,chr(0),$h)+1;
}
if($f&16){
$h=strpos($d,chr(0),$h)+1;
}
if($f&2){
$h+=2;
}
$u = gzinflate(substr($d,$h));
if($u===FALSE){
$u=$d;
}
return $u;
}
I am using the following code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $real_url);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $fscookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $fscookiefile);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 GTB7.1');
$r = curl_exec($ch) or die('WHY DOES THIS DIE!!!');
curl_close($ch);
Which always results in WHY DOES THIS DIE!!! or the page keeps loading for sometime and then shows nothing if I don't use
or die('WHY DOES THIS DIE!!!')
If I set CURLOPT_NOBODY to false
I get
HTTP/1.1 200 OK Date: Thu, 09 Jun 2011 01:35:07 GMT Pragma: public Expires: 0 Cache-Control: must-revalidate, post-check=0, pre-check=0, private=false Content-Type: application/force-download Content-Transfer-Encoding: binary Keep-Alive: 1000 Content-Disposition: attachment; filename*= UTF-8''Ever17.part3.rar Accept-Ranges: bytes Content-Length: 520000000 Server: Jetty(6.1.16) Rar!zs sdt�NP�(�&R�%G��65,��Ever17\Ever_17_-_The_Out_of_Infinity_CD3.iso��"j����M�K�U:� >�b�2Q�sT?G��+iM���qN�'��&�-d�z�����} xi�<�3\�d��Z:|�/���#�gl��$��~#��u�וۊ \��jմg�)�L��#&��;��5M�s}���(�Đ$�yGwӏ$�E����$�Z���:�/�����r͐�8�&������#Q�00��>�)�n�̌}o�H����2Z���y�S㞝�H��=�9�[������E�nЭ��\�9Ǘ!{&y�j/���Ȇg������FW��~G�5������F-&����ƭ��9<���^7����(�HY(a����%�ɚQYL_���ܽ�X|��&
Why am I not getting any output with CURLOPT_NOBODY set to true ????
I am using PHP 5.1.6 on Apache/2.2.3 and have curl-7.15.5-9.el5_6.2.i386 installed
(That's a seriously outdated curl version)
Try using the command line version (preferably from the same host running that PHP script) like 'curl -I [URL]' and make sure the web site truly responds nice to a HEAD request.
Add -v or --trace-ascii for full debug/trace output.