There is a website http://www.tremorgames.com and I want to do a cURL login. I have a code which I use on my server http://shalva97.x10.mx/tremorlogin/ but it just does not work, but after a lot of testing I decided to install Apache on my PC and tested the exactly same code and it worked.
Here is the code :
<?php
$url="http://www.tremorgames.com/index.php";
$postdata = "loginuser=shalva&loginpassword=shalva&Submit=&saveme=1";
$ch = curl_init();
$f = fopen('request.txt', 'w');
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_VERBOSE => 1,
CURLOPT_STDERR => $f,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0',
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $postdata,
CURLOPT_HEADER => 1,
CURLOPT_HTTPHEADER => array("Accept-Language: en-US;q=0.6,en;q=0.4", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Connection: keep-alive"),
CURLOPT_COOKIEFILE => "cookies.txt", //Save cookies
CURLOPT_COOKIEJAR => "cookies.txt", //Cookies located
CURLOPT_REFERER => 'http://www.tremorgames.com/index.php',
CURLOPT_ENCODING => 'gzip,deflate'
));
echo curl_exec($ch);
fclose($f);
curl_close($ch);
?>
Executing this same code on both my PC and server, I get different headers, on PC it receives multiple "Set-Cookie" header, but on server it is only one.
Why it does not work on the server? Why it works on my PC?
well after a lot of thinking i found out that it is not only headers is sent but the ip and my location. The site does not allow logins from other countries thats why it didnot loged in, the code is correct and it was all about server location. i hope this will help others who also try to do same thing.
Related
I need to log into the TD Ameritrade website with my account. Note, I am not talking about the official API, I'm talking about their actual UI. I need to scrape from the site itself, unfortunately. I'm using curl in PHP, but any language you can get to work is probably fine. I've started with postman, which I'm usually able to get to work for this kind of thing, but it isn't returning anything when I copy the transactions from Chrome Dev Tools and importing into Postman. For example, it either loops back to the login site, or if I play with the settings it will flat out not return anything. The current issue with my php is that I get Maximum (10) redirects followed, it's an infinite loop where the login forwards back to itself. Unfortunately, for obvious reasons, I cannot post my username and password, so you would need an account to test. They are free, but not the easiest to set up. In any case, if you have any ideas that would be greatly appreciated.
<?php
header('Content-type: text/javascript');
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'https://auth.tdameritrade.com/oauth?client_id=MOBI%40AMER.OAUTHAP&response_type=code&redirect_uri=https%3A%2F%2Finvest.ameritrade.com%2Fgrid%2Fp%2Fsite&code_challenge=xNghkUhiiE_DSuVXLcSF1ufBVSPTQRa1ITybIyw2USc&code_challenge_method=S256',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_HEADER => 1, // needed for getting cookies
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1",
CURLOPT_POSTFIELDS => 'su_username=' . $username . '&su_password=' . $password,
CURLOPT_VERBOSE => true,
CURLOPT_POST => true,
CURLOPT_AUTOREFERER => true,
// CURLOPT_HTTPHEADER => array(
// 'Content-Type: application/x-www-form-urlencoded'
// ),
));
//curl_setopt($curl, CURLOPT_USERPWD, $username . ":" . $password);
// $cookie = dirname(__FILE__) . "/cookie.txt";
// curl_setopt($curl, CURLOPT_COOKIEFILE, $cookie);
// curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
$res = (object)array(
'response' => curl_exec($curl),
'info' => (object)curl_getinfo($curl),
'errors' => curl_error($curl)
);
print_r($res);
I've got a problem with my PHP script using CURL. I want to download file from another server (Probably with hotlink protection). The URL is like this: example.com/script.extension?id=12345&type=xxx&another=qwerty
When i hit that link from my website, It sends 'Referer:' in headers, so the download is not starting. (without the header it works)
I have to modify the file's name, so i've used CURL to download that to my server. When I'm doing that with that script, the file is created, but it is empty:
function my_function($the_url) {
$fp = fopen ('tmp_file.extension', 'w+');
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_VERBOSE => 1,
CURLOPT_URL => $the_url,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POST => false,
CURLOPT_USERAGENT => $user_agent,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_AUTOREFERER => false,
CURLOPT_REFERER => "http://example.com/",
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$my_exec = curl_exec($ch);
fwrite($fp, $my_exec);
curl_close($ch);
fclose($fp);
}
But it works well with file which is hosted on my site (eg. mysite.com/file.zip)
Also, when I run command through the shell:
$ curl -L -o file.tmp --referer http://example.com/ full url here
It downloaded the file properly.
What's the reason I cannot do it with my PHP function?
I am making an HTTP API for use in Roblox. I am using PHP and cURL to do so.
However, there is one method on which I am stuck. That is sending messages through roblox in PHP. I have captured the message sending request using Fiddler. I have put all the headers into my cURL request.
I also am fetching the .ROBLOSECURITY cookie, the X-CSRF Token, and the RBXSessionTracker and __RequestVerification cookies.
Here is the messaging code for those of you want to help:
$RecipientId=$_POST['RecipientId'];
$Subject=$_POST['Subject'];
$Message=$_POST['Message'];
$AuthCookie=decodeJSON($_POST['Login']);
$AuthCookie=$AuthCookie[0];
$TokenCurl=curl_init('roblox.com/build/upload?groupId=1');
curl_setopt_array($TokenCurl,array(
CURLOPT_HTTPGET => 1,
CURLOPT_FRESH_CONNECT => 1,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HEADER => 1,
CURLOPT_HTTPHEADER => array('Cookie: .ROBLOSECURITY='.$AuthCookie),
));
$TokenResponse=curl_exec($TokenCurl);
$TokenHeaderSize=curl_getinfo($TokenCurl,CURLINFO_HEADER_SIZE);
$TokenCurlError=curl_error($TokenCurl);
$Cookies=GetCookies(substr($TokenResponse,0,$TokenHeaderSize));
$SessionTracker=$Cookies['RBXSessionTracker'];
$RequestVerification=$Cookies['__RequestVerificationToken'];
$AuthCookie='.ROBLOSECURITY='.$AuthCookie.'; RBXSessionTracker='.$SessionTracker.'; __RequestVerificationToken='.$RequestVerification.';';
curl_close($TokenCurl);
if($TokenCurlError!==''){
die($TokenCurlError);
};
$Cache=time();
preg_match("\`<script\stype=\"text/javascript\">\s*?Roblox\.XsrfToken\.setToken\('(.*?)'\);\s*?</script>\`",$TokenResponse,$Matches);
$Token=$Matches[1];
if(empty($Token)){
die("Could not get X-CSRF Token");
};
$PostData=json_encode(array(subject=>$Subject,body=>$Message,recipientid=>$RecipientId,cacheBuster=>$Cache));
$Curl=curl_init('roblox.com/messages/send');
curl_setopt_array($Curl,array(
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0',
CURLOPT_POST => 1,
CURLOPT_FRESH_CONNECT => 1,
CURLOPT_HEADER => 1,
CURLOPT_POSTFIELDS => $PostData,
CURLOPT_HTTPHEADER => array('Content-Length: '.strlen($PostData),'X-CSRF-TOKEN: '.$Token,'Referer: www.roblox.com/messages/compose?recipientId='.$RecipientId,'X-Requested-With: XMLHttpRequest','Cookie: '.$AuthCookie,'Accept: application/json, text/javascript, */*; q=0.01','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Connection: keep-alive','Pragma: no-cache','Cache-Control: no-cache'),
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_VERBOSE => 1
));
$ResponseBody=curl_exec($Curl);
echo $ResponseBody;
$HTTPResponseCode=curl_getinfo($Curl,CURLINFO_HTTP_CODE);
$HTTPHeaderSize=curl_getinfo($Curl,CURLINFO_HEADER_SIZE);
$ResponseHeader=substr($ResponseBody,0,$HTTPHeaderSize);
$CurlError=curl_error($Curl);
curl_close($Curl);
if($CurlError!==''){
die($CurlError);
};
$MessageResult=decodeJSON(substr($ResponseBody,$HTTPHeaderSize));
echo json_encode(array(Message=>$MessageResult['message']));
I removed the 'http://' from the links because of reputation.
Everything works, it just will not send the message. Does anyone know what exactly you need to send a message?
I am using curl to hit a url and get contents from it and below is the code
$url = 'http://uber.com';
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \ Chrome/24.0.1304.0 Safari/537.16',
CURLOPT_CONNECTTIMEOUT => 180,
CURLOPT_TIMEOUT => 180 + 60,
CURLOPT_COOKIE => 'lang=en;'
));
$response = curl_exec($ch);
var_dump($response);
It gives me the content when I execute the script on local server but while executing this on staging and production server, situated in US, it give 403-forbidden error. I also tested the same from another server in US and it works fine.
Any help is appreciated.
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>