I have been trying to get data from this web page. I found that I need to pass the camera id to this url using GET method to get image.
But every time I do a curl request I got http_code 500: internal server error
This is my code (I'm still trying out with static camera id):
$curl = curl_init('http://www.mytransport.sg/content/mytransport/home/myconcierge/trafficcameras/jcr:content/par/cameras_slideshow.singleresult.html?cameraid=4701');
curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.28 Safari/537.36");
curl_setopt($curl, CURLOPT_REFERER, "http://www.mytransport.sg/content/mytransport/home/myconcierge/trafficcameras.html");
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl,CURLOPT_FOLLOWLOCATION,true);
$resp = curl_exec($curl);
echo $resp;
$header = curl_getinfo($curl);
echo "<pre>";
print_r($header);
echo "</pre>";
curl_close($curl);
Result I get:
HTTP/1.1 500 Internal Server Error
Date: Mon, 09 Sep 2013 15:26:56 GMT
Server: Microsoft-IIS/5.0
Last-Modified: Thu, 04 Oct 2012 15:55:23 GMT
ETag: "7d2c5-402e-4cb3dc86ef0c0"
Accept-Ranges: bytes
Content-Length: 16430
Vary: Accept-Encoding
Cache-Control: must-revalidate, proxy-revalidate, private, must-revaildate, proxy-revalidate, private Connection: close
Content-Type: text/html
Related
I've run manually the url from postman, and I've successfully connected and it printed it results.
But when I tried calling it from a method, no result is printing.
here's the curl code I used:
public function curlHandle($apiPath, $postArray)
{
$CI = & get_instance();
$conf = $CI->config->config;
$curlHandle = curl_init($apiPath);
curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlHandle, CURLOPT_USERAGENT,"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36");
curl_setopt($curlHandle, CURLOPT_TIMEOUT , 300);
curl_setopt($curlHandle, CURLOPT_POST, 1);
curl_setopt($curlHandle, CURLOPT_POSTFIELDS, $postArray);
curl_setopt($curlHandle, CURLOPT_VERBOSE, 1);
$fp = fopen('/var/www/html/logs/curl_log.txt', 'w');
curl_setopt($curlHandle, CURLOPT_STDERR, $fp);
$result = curl_exec($curlHandle);
print_r(curl_error($curlHandle));
if ($result === FALSE) {
die(curl_error($curlHandle));
} else {
$data = json_decode($result, TRUE);
}
curl_close($curlHandle);
return $data;
}
Here's the content of curl_log.txt
Hostname localhost/codeignitertest was found in DNS cache
Trying 127.0.0.1...
TCP_NODELAY set
Connected to alocalhost/codeignitertest (127.0.0.1) port 80 (#0)
POST /test/process HTTP/1.1 Host: localhost/codeignitertest User-Agent: Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57
Safari/537.36 Accept: / Content-Length: 142 Content-Type:
multipart/form-data; boundary=------------------------78a116d61ad4939d
< HTTP/1.1 200 OK < Date: Tue, 30 Apr 2019 21:19:43 GMT < Server:
Apache/2.4.29 (Ubuntu) < Cache-Control: no-store, no-cache,
must-revalidate < Pragma: no-cache < Set-Cookie:
ci_session=ag85kv438n0ukb1rvrkb5cdfkrd8s2q6; expires=Tue, 30-Apr-2019
21:49:43 GMT; Max-Age=1800; path=/; HttpOnly < Expires: Thu, 19 Nov
1981 08:52:00 GMT < Content-Length: 14 < Content-Type: text/html;
charset=UTF-8 <
* Connection #0 to host localhost/codeignitertest left intact
also note that curlHandle() is from another project and localhost/codeignitertest but both are running in my localhost
You've issued this option:
curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, true);
Which means that curl returns the output from the remote server instead of printing it.
You then capture that output into a variable:
$result = curl_exec($curlHandle);
Which you then encode and return to your caller:
return $data;
This code shouldn't print the results. It returns the results.
Code:
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, "https://detail.1688.com/offer/543783250479.html?sk=consign");
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$a = curl_exec($ch);
$data = curl_exec($ch);
curl_close($ch);
var_dump($data);
Result:
string(339) "HTTP/1.1 403 Forbidden
Server: nginx/1.11.1
Date: Tue, 16 May 2017 03:46:32 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 169
Connection: keep-alive
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.11.1</center>
</body>
</html>
"
While I run curl -I https://detail.1688.com/offer/543783250479.html?sk=consign in my shell, it returns 200:
HTTP/1.1 200 OK
Date: Tue, 16 May 2017 03:46:51 GMT
Content-Type: text/html;charset=GBK
Connection: keep-alive
Vary: Accept-Encoding
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Cache-Control: max-age=0,s-maxage=0
b2c_auction: 543783250479
atp_isdpp: 99vb2b-2295161471
page_cache_info: {"is_new":true,"create_time":"2017-05-16T11:46:51","expire_time":3600000}
X-Cache: MISS TCP_REFRESH_MISS dirn:-2:-2
Via: aserver010103196008.et2[69,200-0,M]
url-hash: id=543783250479&detail.1688.com
Server: Tengine/Aserver
Strict-Transport-Security: max-age=31536000
Timing-Allow-Origin: *
EagleEye-TraceId: 0b83e0c214949064118297808e926
Could anyone please give me some hints about why I get 403 by cURL in PHP?
Environment:
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.0.8-2+deb.sury.org~xenial+1, Copyright (c) 1999-2016, by Zend Technologies
with blackfire v1.10.6, https://blackfire.io, by Blackfireio Inc.
The returning headers without using an useragent:
HTTP/1.1 302 Moved Temporarily
Date: Tue, 16 May 2017 04:15:46 GMT
Content-Type: text/html
Content-Length: 266
Connection: keep-alive
Location: http://127.0.0.1/?sk=consign
X-Cache: MISS TCP_MISS dirn:-2:-2
Via: aserver010103196008.et2[0,302-0,M]
url-hash: id=543783250479&detail.1688.com
Server: Tengine/Aserver
Strict-Transport-Security: max-age=31536000
Timing-Allow-Origin: *
EagleEye-TraceId: 0b83dc9c14949081466171756eb58d
The important part is:
Location: http://127.0.0.1/?sk=consign
if I use an useragent, I get:
HTTP/1.1 200 OK
Date: Tue, 16 May 2017 04:17:30 GMT
Content-Type: text/html;charset=GBK
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Cache-Control: max-age=0,s-maxage=0
b2c_auction: 543783250479
atp_isdpp: 99vb2b-2295161471
page_cache_info: {"is_new":true,"create_time":"2017-05-16T12:17:30","expire_time":3600000}
X-Cache: MISS TCP_MISS dirn:-2:-2
Via: aserver011128044194.eu13[106,200-0,M]
url-hash: id=543783250479&detail.1688.com
Server: Tengine/Aserver
Strict-Transport-Security: max-age=31536000
Timing-Allow-Origin: *
EagleEye-TraceId: 0b802cd414949082503341644e23a0
Which is correct and it returns the desired html
Code used:
$url = "http://detail.1688.com/offer/543783250479.html?sk=consign";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0");
$html = curl_exec($ch);
curl_close($ch);
print $html;
In curl -I stands for head request. Try setting the head request in your php as
:
curl_setopt($ch, CURLOPT_NOBODY, true);
And give it a try
I'm trying to embed Facebook's posts (e.g. video) using oEmbed format. According to Facebook documentation, oEmbed is now supported. I'm trying this PHP code:
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url={MY VIDEO URL HERE}');
$oembed = json_decode($json_post);
var_dump($oembed);
I already used the same code for Instagram with success, now I'm getting a NULL result. oEmbed works good if i directly write the URL on the browser. Am i missing something?
Thanks.
Update
I tried with Curl:
$url='https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
//curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$page = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
print_r($page);
curl_close($ch);
Now i get:
HTTP/1.1 302 Found
Location: https://www.facebook.com/unsupportedbrowser
access-control-allow-method: OPTIONS
Access-Control-Expose-Headers: X-FB-Debug, X-Loader-Length
Access-Control-Allow-Origin: https://www.facebook.com
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: text/html
X-FB-Debug: gGcZzyllZadlcn/6jz2HqqouIcDnhTzxzR+etWXhZEnOcditfsaIUw0WjgO3nELHzveRCYa1UM86D3LA/nLnNw==
Date: Wed, 11 Jan 2017 10:18:47 GMT
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
X-XSS-Protection: 0
public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
Pragma: no-cache
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=15552000; preload
X-Frame-Options: DENY
Vary: Accept-Encoding
Content-Type: text/html
X-FB-Debug: zwArox8KyM3BtwLymhiARCTltrrcE/pDqSWdqbHgstXVBEbIXG57Od2MfDnqgqSX5Tj43qoe8uYhphzwoZcXeg==
Date: Wed, 11 Jan 2017 10:18:48 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Still waiting for a reply.
Thank You.
Set the user agent with the curl and try,
$browser = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \Chrome/24.0.1304.0 Safari/537.16';
curl_setopt($ch, CURLOPT_USERAGENT, $browser);
Here is the answer with file_get_content,
$options = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie: foo=bar\r\n" . // check function.stream-context-create on php.net
"User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n" // i.e. An iPad
)
);
$context = stream_context_create($options);
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F', false, $context);
$oembed = json_decode($json_post);
var_dump($oembed)
I tried translate this PHP function
function checkLogin($username, $password)
{
$URL = "https://www.yookos.com/cs_login";
$COOKIE = "cookie_".$username.".txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $COOKIE);
curl_setopt($ch, CURLOPT_REFERER, $URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=".$username."&password=".$password);
$result = curl_exec ($ch);
$loggedIn = 0;
if (strpos($result, "I agree to the Terms & Conditions"))
{
// ---- Accept terms and conditions
curl_setopt($ch, CURLOPT_URL, 'https://www.yookos.com/terms-and-conditions.jspa');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "url=&agree=true&method%3Aexecute=Accept");
$result = curl_exec ($ch);
$loggedIn = 1;
} elseif (strpos($result, "/people/".$username)) {
$loggedIn = 2;
} else {
$loggedIn = 3;
}
curl_close($ch);
unlink($COOKIE);
return $loggedIn;
}
To Delphi XE6:
procedure TfrmMain.prcYOOKOS_AcceptTerms(User, Pass: string);
var
sResponse: string;
Param: TStringStream;
HTTP: TidHTTP;
SSL: TIdSSLIOHandlerSocketOpenSSL;
Cookie: TIdCookieManager;
//U : TidURI;
begin
//U := TidURI.Create('www.yookos.com');
HTTP := TIdHTTP.Create(nil);
SSL := TIdSSLIOHandlerSocketOpenSSL.Create(HTTP); // nil
Cookie := TIdCookieManager.Create(HTTP);
HTTP.IOHandler := SSL;
HTTP.AllowCookies := True;
HTTP.CookieManager := Cookie;
//
// ---- Login page
//
m.Lines.Add('------------------------------------');
try
sResponse := HTTP.Get('https://www.yookos.com/login.jspa');
except
on E: Exception do
m.Lines.Add('Error on request 0: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Post DATA
//
m.Lines.Add('------------------------------------');
try
Param := TStringStream.Create('username='+User+'&password='+Pass, TEncoding.UTF8);
HTTP.Request.ContentType := 'application/x-www-form-urlencoded';
HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6'; //HTTP.Request.CustomHeaders.AddValue('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6');
HTTP.Request.Referer := 'https://www.yookos.com/cs_login';
HTTP.Post('https://www.yookos.com/cs_login', Param);
FreeAndNil(Param);
except
on E: Exception do
m.Lines.Add('Error on request 1: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Handle "Location:"
//
while (Pos('condition',HTTP.Response.Location)=0) and (Pos('login.jspa',HTTP.Response.Location)=0) do
begin
m.Lines.Add('------------------------------------');
try
//HTTP.CookieManager.AddServerCookie('', U);
sResponse := HTTP.Get(HTTP.Response.Location);
except
on E: Exception do
m.Lines.Add('Error on request 0: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
end;
//
// ---- ACCEPT TERMS
//
m.Lines.Add('------------------------------------');
try
Param := TStringStream.Create('url=&agree=true&method:execute=Accept', TEncoding.UTF8);
sResponse := HTTP.Post('https://www.yookos.com/terms-and-conditions.jspa', Param);
FreeAndNil(Param);
except
on E: Exception do
m.Lines.Add('Error on request 2: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Free OBJECTS from memory
//
FreeAndNil(SSL);
FreeAndNil(Cookie);
FreeAndNil(HTTP);
end;
procedure TfrmMain.Button1Click(Sender: TObject);
begin
prcYOOKOS_AcceptTerms(A.Text, B.Text);
end;
But when i test it, the JSESSIONID is not held by application, i know is something about Cookie Management, but i dont have experience using cookies and IdHTTP on Delphi.
I need to know how to implement a EQUIVALENT CURL COOKIEJAR and access TERMS-AND-CONDITIONS page...
Some test accounts:
bamidelebatigo21911
yookos1
ekeminibiney37611
yookos1
sojiedna3511
yookos1
I hope someone can help me.
Edit:
Nework Activity:
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/cs_login
Request Method:POST
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:49
Content-Type:application/x-www-form-urlencoded
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; JSESSIONID=4453E0FF613AD064007787E047D0DB01.; jive.security.context="RbpSeLe7t/WPkLYju8/vVkwHJGRKmHXKBce5/ZQD60vRMQ9fvd/9cUbo9eCMywxd.LTE="; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH
Host:www.yookos.com
Origin:https://www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Form Dataview sourceview URL encoded
username:bamidelebatigo21911
password:yookos1
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:keep-alive
Content-Encoding:gzip
Content-Length:20
Content-Type:text/plain
Date:Mon, 03 Nov 2014 03:06:08 GMT
Expires:Mon, 03 Nov 2014 03:06:08 GMT
Location:https://www.yookos.com/
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Set-Cookie:jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="; Version=1; Max-Age=2592000; Expires=Wed, 03-Dec-2014 03:06:08 GMT; Path=/;HttpOnly
Set-Cookie:JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; Path=/;HttpOnly
Set-Cookie:jive.user.loggedIn=true; Path=/;HttpOnly
Set-Cookie:jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; Version=1; Path=/;HttpOnly
Vary:Accept-Encoding
X-Cnection:close
X-JSL:D=20228 t=1414983968813575
--------------------------------------------------------------------------------
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/
Request Method:GET
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH; JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; jive.user.loggedIn=true; jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="
Host:www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:keep-alive
Content-Encoding:gzip
Content-Length:20
Content-Type:text/plain
Date:Mon, 03 Nov 2014 03:06:09 GMT
Expires:Mon, 03 Nov 2014 03:06:09 GMT
Location:https://www.yookos.com/terms-and-conditions!input.jspa
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Vary:Accept-Encoding
X-Cnection:close
X-JSL:D=2883 t=1414983969114534
--------------------------------------------------------------------------------
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/terms-and-conditions!input.jspa
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH; JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; jive.user.loggedIn=true; jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="
Host:www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:Transfer-Encoding
Connection:keep-alive
Content-Encoding:gzip
Content-Language:en-US
Content-Type:text/html;charset=UTF-8
Date:Mon, 03 Nov 2014 03:06:10 GMT
Expires:Mon, 03 Nov 2014 03:06:09 GMT
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Transfer-Encoding:chunked
Vary:Accept-Encoding
X-Cnection:close
X-JIVE-USER-ID:44470875
X-JSL:D=161278 t=1414983969375279
I'm trying to implement a functionnality like facebook, when you paste a link it's grabbing some information (h1, desc, images, ...) from the page and display them.
I already face several issues that I managed to fix (gzip, cookies, user agent, ...) but on this one I'm not sure what is blocking my request.
The link in question is http://www.mixcloud.com
Here is my PHP script:
protected function getContent()
{
$ch = curl_init();
$headers = array(
'Accept: */*',
// 'Accept-Encoding: gzip,deflate,sdch',
// 'Accept-Language: en-US,en;q=0.8,es;q=0.6,fr;q=0.4,pt;q=0.2',
// 'Cache-Control: no-cache',
// 'Connection: keep-alive'
);
$debug = TRUE;
// Set the request type
curl_setopt($ch, CURLOPT_VERBOSE, $debug);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_NOBODY, FALSE);
curl_setopt($ch, CURLOPT_URL, $this->url);
curl_setopt($ch, CURLOPT_USERAGENT, $this->userAgent);
curl_setopt($ch, CURLOPT_REFERER, $this->referrer);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, $debug);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_ENCODING , 'gzip');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5');
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookies.txt');
$data = curl_exec($ch);
var_dump($data);die;
return curl_exec($ch);
}
Here is the verbose response:
* Adding handle: conn: 0x7f937504e400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f937504e400) send_pipe: 1, recv_pipe: 0
* About to connect() to www.mixcloud.com port 80 (#0)
* Trying 46.23.65.210...
* Connected to www.mixcloud.com (46.23.65.210) port 80 (#0)
> GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5
Host: www.mixcloud.com
Accept-Encoding: gzip
Referer: https://www.google.com.au
Accept: */*
< HTTP/1.1 403 Forbidden
* Server nginx/1.5.8 is not blacklisted
< Server: nginx/1.5.8
< Date: Tue, 18 Feb 2014 06:39:45 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Content-Encoding: gzip
<
* Connection #0 to host www.mixcloud.com left intact
string(376) "HTTP/1.1 403 Forbidden\r\nServer: nginx/1.5.8\r\nDate: Tue, 18 Feb 2014 06:39:45 GMT\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\n\r\n<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor="white">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.5.8</center>\r\n</body>\r\n</html>\r\n"
Now if I try to execute the curl command in the shell it's working fine:
$ curl -i 'http://www.mixcloud.com' -v
* Adding handle: conn: 0x7fe28b004000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fe28b004000) send_pipe: 1, recv_pipe: 0
* About to connect() to www.mixcloud.com port 80 (#0)
* Trying 46.23.65.210...
* Connected to www.mixcloud.com (46.23.65.210) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.mixcloud.com
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 18 Feb 2014 06:41:30 GMT
Date: Tue, 18 Feb 2014 06:41:30 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 194847
Content-Length: 194847
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding
* Server gunicorn/0.17.4 is not blacklisted
< Server: gunicorn/0.17.4
Server: gunicorn/0.17.4
< Vary: Cookie, User-Agent, X-Requested-With, X-Ignore-Block
Vary: Cookie, User-Agent, X-Requested-With, X-Ignore-Block
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< Set-Cookie: csrftoken=ciOosbUNp5EL8t5tiQQzkoeaJIDJ3VfO; Domain=.mixcloud.com; expires=Tue, 17-Feb-2015 06:41:30 GMT; Max-Age=31449600; Path=/
Set-Cookie: csrftoken=ciOosbUNp5EL8t5tiQQzkoeaJIDJ3VfO; Domain=.mixcloud.com; expires=Tue, 17-Feb-2015 06:41:30 GMT; Max-Age=31449600; Path=/
< Set-Cookie: eventstream=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
Set-Cookie: eventstream=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
<
<!DOCTYPE html> ...
I know that the cURL for PHP and cURL are different, but I can't see what I am missing.
Anyone?
Cheers,
Maxime
Ok I've found what was the issue. It was the user-agent.
It's really weird. I was using this user-agent:
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5
With this user agent I was getting a 403. I've updated it using the following one:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
And it's now working well. I can't believe that people are still rejecting request for specific user agent...