I'm trying to create a function which would be able to log me into a ROBLOX account
This is the code i've created and tried
<?php
$ch = curl_init('http://www.roblox.com/newlogin');
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
username => 'Username',
password => '#######',
submitLogin => 'Log In'
));
$result = curl_exec($ch);
echo($result);
?>
The only problem is that it fails to log me in, and it also outputs Object moved to here
Object moved Here
Is there any reason why it failed to log me in?! and why the object moved here echoes instead of ROBLOX's newlogin page?!
Here is the neccessary information if you'd like
Request URL:https://www.roblox.com/newlogin
Request Method:POST
Status Code:302 Found
Remote Address:8.42.96.39:443
Response Headers
view source
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:https://www.roblox.com
Access-Control-Expose-Headers:X-CSRF-TOKEN
Cache-Control:private
Content-Length:130
Content-Type:text/html; charset=utf-8
Date:Fri, 26 Feb 2016 10:15:03 GMT
Location:/home?nl=true
P3P:CP="CAO DSP COR CURa ADMa DEVa OUR IND PHY ONL UNI COM NAV INT DEM PRE"
Set-Cookie:.ROBLOSECURITY=_|WARNING:-DO-NOT-SHARE-THIS.--Sharing-this-will-allow-someone-to-log-in-as-you-and-to-steal-your-ROBUX-and-items.|_C1183ACB7F4DF9061B7F4E0CF17A7E5A6556D75C4B3B05467092952A6A2554429AC0FE5A24B70282702CE582A8AC040790564311FF4F47841B87D9DE5BB613DF7B564B39F6D65E4A53991E42463BFE959E5E7C638FD204F81F9A5F1A08D208DB5D662DAC02D7F09E1F689E055053866CB4794C56F18C0FE372469732888A0F8FCDC8704319BBBDAFFD0D98243681A912872AF00218773E150788096CFBAB158A1176D65A5BE24896064903CFB77318EB8875351072CE35CB059641BB986038C9AC5863D11206E330A2FC27DC62E55D65EA032A1790331B86BC9B5352EAB131321FFE94599005693977C5E2A054CF6945FF8D325443103AFE414369EF5564E5C65A76A2E8A8EE9CC4E8546E9098FF6768A4A6A88215F23CBCBF6227A34BE0C2DBFE9F4802058419E614CE30FA52486789F0E86FE5B45E1C66D3A47B3F695A4DDCDC6B3EB3BEC4731C09BEC3188E5AD456599AFB18884828064D131E7004F054298D5EE3FC144F8; domain=.roblox.com; expires=Sun, 27-Mar-2016 10:15:03 GMT; path=/; HttpOnly
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:59
Content-Type:application/x-www-form-urlencoded
Cookie:RBXSource=rbx_acquisition_time=2/17/2016 11:34:48 AM&rbx_acquisition_referrer=&rbx_medium=Direct&rbx_source=&rbx_campaign=&rbx_adgroup=&rbx_keyword=&rbx_matchtype=&rbx_send_info=1; RBXViralAcquisition=time=2/17/2016 11:34:48 AM&referrer=&originatingsite=; GuestData=UserID=-1661021767; RBXMarketing=FirstHomePageVisit=1; __gads=ID=85bb563f4079b12f:T=1455730802:S=ALNI_MbDjPeXjVm69YJMcFdVnOyw2XxWvQ; __ssid=aaf89930-bda3-4057-9fed-d5324ec16615; __RequestVerificationToken=NnycdFxv3QtmdyJPW9irh64pVurQGL4BHEjZI9kq3uET0G_NT_f0JUQlIhVsE6gUC5clWd4QoPQ3gwwDo45kif1JQHw1; glt_3_OsvmtBbTg6S_EUbwTPtbbmoihFY5ON6v6hbVrTbuqpBs7SyF_LQaJwtwKJ60sY1p=LT3_p97tsEiCJbASoKTsdz1ye4a8igDizZThXseKb-HwEYg%7CUUID%3D4169b82697734de882c125e3b3e4bc17; _gig_llp=facebook; _gig_llu=Baheeg; RBXEventTrackerV2=CreateDate=2/26/2016 4:01:29 AM&rbxid=96092679&browserid=4151477874; rbx-ip=; __qca=P0-1972999082-1456481070051; RBXImageCache=timg=62646533346162632D643361312D343366392D396536622D386432646561316636313662253130392E3137372E32382E32323425322F32362F3230313620393A35313A353820414D534CED27903239AA1175AB86097D565F4BC07C53; __utmt_b=1; __utma=200924205.672426321.1455730486.1456473772.1456480004.45; __utmb=200924205.66.7.1456481717857; __utmc=200924205; __utmz=200924205.1456426636.42.6.utmcsr=forum.roblox.com|utmccn=(referral)|utmcmd=referral|utmcct=/Forum/ShowPost.aspx
Host:www.roblox.com
Origin:https://www.roblox.com
Referer:https://www.roblox.com/newlogin
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
Form Data
view source
view URL encoded
Username:########
Password:########
ReturnUrl:
Note: This is not for malicious uses
You didn't ask curl to follow redirects. CURLOPT_FOLLOWLOCATION
You will probably also need to enable cookies so that the correct set is passed on in the redirect-to request.
Related
I'm trying to embed Facebook's posts (e.g. video) using oEmbed format. According to Facebook documentation, oEmbed is now supported. I'm trying this PHP code:
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url={MY VIDEO URL HERE}');
$oembed = json_decode($json_post);
var_dump($oembed);
I already used the same code for Instagram with success, now I'm getting a NULL result. oEmbed works good if i directly write the URL on the browser. Am i missing something?
Thanks.
Update
I tried with Curl:
$url='https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
//curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$page = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
print_r($page);
curl_close($ch);
Now i get:
HTTP/1.1 302 Found
Location: https://www.facebook.com/unsupportedbrowser
access-control-allow-method: OPTIONS
Access-Control-Expose-Headers: X-FB-Debug, X-Loader-Length
Access-Control-Allow-Origin: https://www.facebook.com
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: text/html
X-FB-Debug: gGcZzyllZadlcn/6jz2HqqouIcDnhTzxzR+etWXhZEnOcditfsaIUw0WjgO3nELHzveRCYa1UM86D3LA/nLnNw==
Date: Wed, 11 Jan 2017 10:18:47 GMT
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
X-XSS-Protection: 0
public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
Pragma: no-cache
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=15552000; preload
X-Frame-Options: DENY
Vary: Accept-Encoding
Content-Type: text/html
X-FB-Debug: zwArox8KyM3BtwLymhiARCTltrrcE/pDqSWdqbHgstXVBEbIXG57Od2MfDnqgqSX5Tj43qoe8uYhphzwoZcXeg==
Date: Wed, 11 Jan 2017 10:18:48 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Still waiting for a reply.
Thank You.
Set the user agent with the curl and try,
$browser = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \Chrome/24.0.1304.0 Safari/537.16';
curl_setopt($ch, CURLOPT_USERAGENT, $browser);
Here is the answer with file_get_content,
$options = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie: foo=bar\r\n" . // check function.stream-context-create on php.net
"User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n" // i.e. An iPad
)
);
$context = stream_context_create($options);
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F', false, $context);
$oembed = json_decode($json_post);
var_dump($oembed)
I want to scrap some data, but I need to log in. So, my idea is to copy the cookies when I log in to my program. But I don't know why, but if I'm using my program, I kept redirect to login pages. I already compared it, but the cookies are same.
Here's the header if I login using my google chrome, (Copied it from request header):
GET /example/data/data.jsp?date=01-Jan-2001&_=1439020103330 HTTP/1.1
Host: www.example.com
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,ms;q=0.6
Cookie: SESSIONID=BA4BA42C628D5C6EB959D49DB745D94A.NGXA; __utma=77920972.1013585791.1438786361.1438966138.1439020034.5; __utmc=77920972; __utmz=77920972.1438786423.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
My curl code:
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$f = fopen('request.txt', 'w');
curl_setopt($ch, CURLOPT_STDERR , $f);
curl_setopt($ch, CURLOPT_HTTPHEADER,array('
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding: gzip, deflate, sdch',
'Accept-Language: en-US,en;q=0.8,ms;q=0.6',
'Cookie: SESSIONID=BA4BA42C628D5C6EB959D49DB745D94A.NGXA; __utma=77920972.1013585791.1438786361.1438966138.1439020034.5; __utmc=77920972; __utmz=77920972.1438786423.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)',
'Upgrade-Insecure-Requests: 1',
'User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36',
'X-DevTools-Emulate-Network-Conditions-Client-Id: 3A45EE97-D41F-45A3-AFCD-1540014377A7
'));
Here's my request.txt used to debug my program header:
* About to connect() to www.example.com port 80 (#0)
* Trying 202.43.163.203... * connected
* Connected to www.example.com (202.43.163.203) port 80 (#0)
> GET /example/data/data.jsp?date=01-Jan-2001&_=1439020103330 HTTP/1.1
Host: www.example.com
Accept: */*
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,ms;q=0.6
Cookie: SESSIONID=BA4BA42C628D5C6EB959D49DB745D94A.NGXA; __utma=77920972.1013585791.1438786361.1438966138.1439020034.5; __utmc=77920972; __utmz=77920972.1438786423.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36
X-DevTools-Emulate-Network-Conditions-Client-Id: 3A45EE97-D41F-45A3-AFCD-1540014377A7
< HTTP/1.1 302 Found
< Date: Sat, 08 Aug 2015 09:44:06 GMT
< Server: Apache
< Set-Cookie: SESSIONID=7C8779894A3CE29D4BCED4B4D311E07E.NGXA; Path=/example/; HttpOnly
< Location: http://www.example.com/login.jsp
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host www.example.com left intact
* Closing connection #0
This my first time scrapping data with login user, but did I miss something?
Cookies can be set via the CURLOPT_COOKIE. In your case
curl_setopt($ch, CURLOPT_COOKIE, 'SESSIONID=BA4BA42C628D5C6EB959D49DB745D94A.NGXA');
With semicolon space you can add more cookies to the request. See http://php.net/manual/en/function.curl-setopt.php for more information.
If you want to store and re-use the cookies you can also use CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR.
I tried translate this PHP function
function checkLogin($username, $password)
{
$URL = "https://www.yookos.com/cs_login";
$COOKIE = "cookie_".$username.".txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $COOKIE);
curl_setopt($ch, CURLOPT_REFERER, $URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=".$username."&password=".$password);
$result = curl_exec ($ch);
$loggedIn = 0;
if (strpos($result, "I agree to the Terms & Conditions"))
{
// ---- Accept terms and conditions
curl_setopt($ch, CURLOPT_URL, 'https://www.yookos.com/terms-and-conditions.jspa');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "url=&agree=true&method%3Aexecute=Accept");
$result = curl_exec ($ch);
$loggedIn = 1;
} elseif (strpos($result, "/people/".$username)) {
$loggedIn = 2;
} else {
$loggedIn = 3;
}
curl_close($ch);
unlink($COOKIE);
return $loggedIn;
}
To Delphi XE6:
procedure TfrmMain.prcYOOKOS_AcceptTerms(User, Pass: string);
var
sResponse: string;
Param: TStringStream;
HTTP: TidHTTP;
SSL: TIdSSLIOHandlerSocketOpenSSL;
Cookie: TIdCookieManager;
//U : TidURI;
begin
//U := TidURI.Create('www.yookos.com');
HTTP := TIdHTTP.Create(nil);
SSL := TIdSSLIOHandlerSocketOpenSSL.Create(HTTP); // nil
Cookie := TIdCookieManager.Create(HTTP);
HTTP.IOHandler := SSL;
HTTP.AllowCookies := True;
HTTP.CookieManager := Cookie;
//
// ---- Login page
//
m.Lines.Add('------------------------------------');
try
sResponse := HTTP.Get('https://www.yookos.com/login.jspa');
except
on E: Exception do
m.Lines.Add('Error on request 0: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Post DATA
//
m.Lines.Add('------------------------------------');
try
Param := TStringStream.Create('username='+User+'&password='+Pass, TEncoding.UTF8);
HTTP.Request.ContentType := 'application/x-www-form-urlencoded';
HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6'; //HTTP.Request.CustomHeaders.AddValue('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6');
HTTP.Request.Referer := 'https://www.yookos.com/cs_login';
HTTP.Post('https://www.yookos.com/cs_login', Param);
FreeAndNil(Param);
except
on E: Exception do
m.Lines.Add('Error on request 1: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Handle "Location:"
//
while (Pos('condition',HTTP.Response.Location)=0) and (Pos('login.jspa',HTTP.Response.Location)=0) do
begin
m.Lines.Add('------------------------------------');
try
//HTTP.CookieManager.AddServerCookie('', U);
sResponse := HTTP.Get(HTTP.Response.Location);
except
on E: Exception do
m.Lines.Add('Error on request 0: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
end;
//
// ---- ACCEPT TERMS
//
m.Lines.Add('------------------------------------');
try
Param := TStringStream.Create('url=&agree=true&method:execute=Accept', TEncoding.UTF8);
sResponse := HTTP.Post('https://www.yookos.com/terms-and-conditions.jspa', Param);
FreeAndNil(Param);
except
on E: Exception do
m.Lines.Add('Error on request 2: ' + e.Message);
end;
m.Lines.Add(HTTP.Response.RawHeaders.GetText);
m.Lines.Add(HTTP.Response.Location);
m.Lines.Add('------------------------------------');
//
// ---- Free OBJECTS from memory
//
FreeAndNil(SSL);
FreeAndNil(Cookie);
FreeAndNil(HTTP);
end;
procedure TfrmMain.Button1Click(Sender: TObject);
begin
prcYOOKOS_AcceptTerms(A.Text, B.Text);
end;
But when i test it, the JSESSIONID is not held by application, i know is something about Cookie Management, but i dont have experience using cookies and IdHTTP on Delphi.
I need to know how to implement a EQUIVALENT CURL COOKIEJAR and access TERMS-AND-CONDITIONS page...
Some test accounts:
bamidelebatigo21911
yookos1
ekeminibiney37611
yookos1
sojiedna3511
yookos1
I hope someone can help me.
Edit:
Nework Activity:
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/cs_login
Request Method:POST
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:49
Content-Type:application/x-www-form-urlencoded
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; JSESSIONID=4453E0FF613AD064007787E047D0DB01.; jive.security.context="RbpSeLe7t/WPkLYju8/vVkwHJGRKmHXKBce5/ZQD60vRMQ9fvd/9cUbo9eCMywxd.LTE="; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH
Host:www.yookos.com
Origin:https://www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Form Dataview sourceview URL encoded
username:bamidelebatigo21911
password:yookos1
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:keep-alive
Content-Encoding:gzip
Content-Length:20
Content-Type:text/plain
Date:Mon, 03 Nov 2014 03:06:08 GMT
Expires:Mon, 03 Nov 2014 03:06:08 GMT
Location:https://www.yookos.com/
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Set-Cookie:jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="; Version=1; Max-Age=2592000; Expires=Wed, 03-Dec-2014 03:06:08 GMT; Path=/;HttpOnly
Set-Cookie:JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; Path=/;HttpOnly
Set-Cookie:jive.user.loggedIn=true; Path=/;HttpOnly
Set-Cookie:jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; Version=1; Path=/;HttpOnly
Vary:Accept-Encoding
X-Cnection:close
X-JSL:D=20228 t=1414983968813575
--------------------------------------------------------------------------------
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/
Request Method:GET
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH; JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; jive.user.loggedIn=true; jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="
Host:www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:keep-alive
Content-Encoding:gzip
Content-Length:20
Content-Type:text/plain
Date:Mon, 03 Nov 2014 03:06:09 GMT
Expires:Mon, 03 Nov 2014 03:06:09 GMT
Location:https://www.yookos.com/terms-and-conditions!input.jspa
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Vary:Accept-Encoding
X-Cnection:close
X-JSL:D=2883 t=1414983969114534
--------------------------------------------------------------------------------
Remote Address:23.216.189.124:443
Request URL:https://www.yookos.com/terms-and-conditions!input.jspa
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:__gads=ID=ff29271d05b213c8:T=1414610836:S=ALNI_MYDwjPO57uPfIaGmdVfqnjoWHxsbg; BIGipServeryookos-bm-pool=171466762.20480.0000; _ga=GA1.2.1807302990.1414610813; __zlcmid=RceB82JzC9uSzH; JSESSIONID=C363F991321B5C68D0A6B31C8600F98F.; jive.user.loggedIn=true; jive.server.info="serverName=www.yookos.com:serverPort=443:contextPath=:localName=localhost:localPort=9200:localAddr=127.0.0.1"; jive.security.context="0acwivaPVzRzAfYY/qpfog==.NDQ0NzA4NzU="
Host:www.yookos.com
Referer:https://www.yookos.com/login.jspa
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, private, max-age=0
Connection:Transfer-Encoding
Connection:keep-alive
Content-Encoding:gzip
Content-Language:en-US
Content-Type:text/html;charset=UTF-8
Date:Mon, 03 Nov 2014 03:06:10 GMT
Expires:Mon, 03 Nov 2014 03:06:09 GMT
P3P:CP="CAO PSA OUR"
Server:Apache-Coyote/1.1
Transfer-Encoding:chunked
Vary:Accept-Encoding
X-Cnection:close
X-JIVE-USER-ID:44470875
X-JSL:D=161278 t=1414983969375279
I have been trying to get data from this web page. I found that I need to pass the camera id to this url using GET method to get image.
But every time I do a curl request I got http_code 500: internal server error
This is my code (I'm still trying out with static camera id):
$curl = curl_init('http://www.mytransport.sg/content/mytransport/home/myconcierge/trafficcameras/jcr:content/par/cameras_slideshow.singleresult.html?cameraid=4701');
curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.28 Safari/537.36");
curl_setopt($curl, CURLOPT_REFERER, "http://www.mytransport.sg/content/mytransport/home/myconcierge/trafficcameras.html");
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl,CURLOPT_FOLLOWLOCATION,true);
$resp = curl_exec($curl);
echo $resp;
$header = curl_getinfo($curl);
echo "<pre>";
print_r($header);
echo "</pre>";
curl_close($curl);
Result I get:
HTTP/1.1 500 Internal Server Error
Date: Mon, 09 Sep 2013 15:26:56 GMT
Server: Microsoft-IIS/5.0
Last-Modified: Thu, 04 Oct 2012 15:55:23 GMT
ETag: "7d2c5-402e-4cb3dc86ef0c0"
Accept-Ranges: bytes
Content-Length: 16430
Vary: Accept-Encoding
Cache-Control: must-revalidate, proxy-revalidate, private, must-revaildate, proxy-revalidate, private Connection: close
Content-Type: text/html
I have this cURL code in php.
curl_setopt($ch, CURLOPT_URL, trim("http://stackoverflow.com/questions/tagged/java"));
curl_setopt($ch, CURLOPT_PORT, 80); //ignore explicit setting of port 80
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_HTTPHEADER, $v);
curl_setopt($ch, CURLOPT_VERBOSE, true);
The contents of HTTPHEADER are ;
Proxy-Connection: Close
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1017.2 Safari/535.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: __qca=blabla
Connection: Close
Each of them individual items in the array $v.
When I upload the file on my host and run the code, what I get is :
400 Bad request
Your browser sent an invalid request.
But when I run it on my system using command line PHP, what I get is
< HTTP/1.1 200 OK
< Vary: Accept-Encoding
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Content-Encoding: gzip
< Date: Sat, 03 Mar 2012 21:50:17 GMT
< Connection: close
< Set-Cookie: buncha cokkies; path=/; HttpOnly
< Content-Length: 22151
<
* Closing connection #0
.
It's not only on stackoverflow, this happens, it happens also on 4shared, but works on google and others.
Thanks for any help.
This is more a comment than an answer: From your question it's not clear what specifically triggers the 400 error nor what especially means it or more concrete: the source of it.
Is that the output by your server? Is that some feedback (the curl response) that you output with your script?
To better debug things, I've come up with a slightly different form of configuration you might be interested in when using the curl extension. There is a nice function called curl_setopt_array which allows you to set multiple options at once. It will return false if one of the options fails. It allows you to configure your request in complete in front. So you can more easily inject and replace it with a second (debug) configuration:
$curlDefault = array(
CURLOPT_PORT => 80, // ignore explicit setting of port 80
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_ENCODING => '',
CURLOPT_HTTPHEADER => array(
'Proxy-Connection: Close',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1017.2 Safari/535.19',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding: gzip,deflate,sdch',
'Accept-Language: en-US,en;q=0.8',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Cookie: __qca=blabla',
'Connection: Close',
),
CURLOPT_VERBOSE => TRUE, // TRUE to output verbose information. Writes output to STDERR, or the file specified using CURLOPT_STDERR.
);
$url = "http://stackoverflow.com/questions/tagged/java";
$handle = curl_init($url);
curl_setopt_array($handle, $curlDefault);
$html = curl_exec($handle);
curl_close($handle);
This might help you to improve the code and to debug things.
Furthermore you're making use of the CURLOPT_VERBOSE option. This will put the verbose information into STDERR - so you can't track it any longer. Instead you can add it to the output as well to better see what's going on:
...
CURLOPT_VERBOSE => TRUE, // TRUE to output verbose information. Writes output to STDERR, or the file specified using CURLOPT_STDERR.
CURLOPT_STDERR => $verbose = fopen('php://temp', 'rw+'),
);
$url = "http://stackoverflow.com/questions/tagged/java";
$handle = curl_init($url);
curl_setopt_array($handle, $curlDefault);
$html = curl_exec($handle);
$urlEndpoint = curl_getinfo($handle, CURLINFO_EFFECTIVE_URL);
echo "Verbose information:\n<pre>", !rewind($verbose), htmlspecialchars(stream_get_contents($verbose)), "</pre>\n";
curl_close($handle);
Which gives sort of the following output:
Verbose information:
* About to connect() to stackoverflow.com port 80 (#0)
* Trying 64.34.119.12...
* connected
* Connected to stackoverflow.com (64.34.119.12) port 80 (#0)
> GET /questions/tagged/java HTTP/1.1
Host: stackoverflow.com
Proxy-Connection: Close
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1017.2 Safari/535.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: __qca=blabla
Connection: Close
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Content-Encoding: gzip
< Vary: Accept-Encoding
< Date: Mon, 05 Mar 2012 17:33:11 GMT
< Connection: close
< Content-Length: 10537
<
* Closing connection #0
Which should provide you the information needed to track things down if they are request/curl related. You can then easily change parameters and see if it makes a difference. Also compare the curl version you have installed locally with the one on the server. To obtain it, use curl_version:
$curlVersion = curl_version();
echo $curlVersion['version']; // e.g. 7.24.0
Hope this helps you to track things down.
according to http://php.net/manual/en/function.curl-setopt.php
try setting CURLOPT_ENCODING to "gzip"
also, i'd try to avoid as many header lines as possible, for example use CURLOPT_COOKIE instead of Cookie: __qca__=blabla or CURLOPT_USERAGENT
EDIT: it seems that you're not using an array (key => value) for CURLOPT_HTTPHEADER, are you? in this case, use the array and with the other stuff, i wrote, you'll be fine. (how this is done, read the manual :P)
hope that helps.
this worked for me
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
$response = curl_exec($ch);
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "Verbose information:\n<pre>", htmlspecialchars($verboseLog), "</pre>\n";