i'm trying to call a url in a page on a remote site. Decided on using curl. On the remote site the url vars are showing as:
$_REQUEST Array
(
[var1] => val1
[amp;var2] => val2
[amp;var3] => val3
)
the url being called is:
http://site.com/controller/action.php?var1=val1&var2=val2&var3=val3
Notice that I'm not using & in the url, but the request global has it, or nearly - it has amp; instead of & and not & like I used!!! Not that it should have any of them.
The curl object has logged in to a form, set cookies, posted a form on another page, and now this is the last link I have to follow.
here is the php i'm using:
curl_setopt($this->ch, CURLOPT_URL, "http://site.com/controller/action.php?var1=val1&var2=val2&var3=val3");
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($this->ch, CURLOPT_REFERER, "http://site.com/");
curl_setopt($this->ch, CURLOPT_VERBOSE, 1);
curl_setopt($this->ch, CURLOPT_COOKIEJAR, $this->cookie);
curl_setopt($this->ch, CURLOPT_COOKIEFILE, $this->cookie);
curl_setopt($this->ch, CURLOPT_POST, 0);
curl_setopt($this->ch, CURLOPT_HTTPGET, 1);
$output = curl_exec($this->ch);
$info = curl_getinfo($this->ch);
I've ran another curl request after this and I'm still logged in so issue is not cookies. Because the last request (the login form) used $_POST have have set the CURLOPT_POST to 0 and the CURLOPT_HTTPGET to 1.
Here is the output from $info:
info Array
(
[url] => http://site.com/controller/action.php?var1=val1&var2=val2&var3=val3
[content_type] => text/html; charset=utf-8
[http_code] => 403
[header_size] => 403
[request_size] => 541
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.781186
[namelookup_time] => 3.7E-5
[connect_time] => 3.7E-5
[pretransfer_time] => 4.2E-5
[size_upload] => 1093
[size_download] => 3264
[speed_download] => 4178
[speed_upload] => 1399
[download_content_length] => 3264
[upload_content_length] => 0
[starttransfer_time] => 0.781078
[redirect_time] => 0
[certinfo] => Array
(
)
)
if i copy and past $info['url'] into the browser it works. Completely lost, hours lost on the clock, any help would be appreciated ;)
Try modifying your code as follows:
$data = array('val1'=>"val1", 'val2'=>"val2", 'val3'=>"val3");
curl_setopt($this->ch, CURLOPT_URL, "http://site.com/controller/action.php");
curl_setopt($this->ch, CURLOPT_POSTFIELDS, http_build_query($data));
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($this->ch, CURLOPT_REFERER, "http://site.com/");
curl_setopt($this->ch, CURLOPT_VERBOSE, 1);
curl_setopt($this->ch, CURLOPT_COOKIEJAR, $this->cookie);
curl_setopt($this->ch, CURLOPT_COOKIEFILE, $this->cookie);
curl_setopt($this->ch, CURLOPT_POST, 0);
curl_setopt($this->ch, CURLOPT_HTTPGET, 1);
$output = curl_exec($this->ch);
$info = curl_getinfo($this->ch);
EDIT
Removed custom encode function as per Brad's comment - you don't need it, as PHP has a build-in function that does the same thing.
Related
I'm trying to get Robinhood option data in PHP, which requires authentication. I feel like I am a breath away from my solution, but after trying for a day I am ready to ask for help.
So far, I have been able to log in to Robinhood and get the token, then use that token to authenticate a request for a second (oauth) token successfully. But for some reason, I am unable to get options data for the option of my choice (MSFT Put 75 Exp 1/17/2020, found here with proper authentication https://api.robinhood.com/marketdata/options/0fd40096-9cbc-4b14-9df4-c1c9ea5f5729/ )
Here is how I login
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://api.robinhood.com/api-token-auth/");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=example#gmail.com&password=mypassword");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec ($ch);
$result = json_decode($server_output);
$token = $result->token;
curl_close ($ch);
Then I take that token and convert it
$url = 'https://api.robinhood.com/oauth2/migrate_token/';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Authorization: Token '.$token));
$server_output = curl_exec ($ch);
$result = json_decode($server_output);
$oauth_token = $result->access_token;
curl_close ($ch);
Up to here so far so good, but I am only getting a blank response when I try the following:
$url = 'https://api.robinhood.com/marketdata/options/0fd40096-9cbc-4b14-9df4-c1c9ea5f5729/';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Authorization: Bearer '.$oauth_token));
$server_output = curl_exec ($ch);
var_dump($server_output);
curl_close ($ch);
Any help or ideas why I'm having issues on the last part would be immensely appreciated :)
EDIT: In answer to WebCode.ie, the result of print_r(curl_getinfo($ch)) is:
Array
(
[url] => https://api.robinhood.com/marketdata/options/0fd40096-
cbc-4b14-9df4-c1c9ea5f5729/
[content_type] => application/json
[http_code] => 405
[header_size] => 187
[request_size] => 453
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.386518
[namelookup_time] => 2.8E-5
[connect_time] => 0.094845
[pretransfer_time] => 0.289176
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => 0
[upload_content_length] => -1
[starttransfer_time] => 0.386491
[redirect_time] => 0
[redirect_url] =>
[primary_ip] => 52.200.3.207
[certinfo] => Array
(
)
[primary_port] => 443
[local_ip] => 84.x.x.x //my IP
[local_port] => 60974
)
Looks like I was making POST call when I should have been making a GET call. If you are having trouble remove this line from my last code block.
curl_setopt($ch, CURLOPT_POST, 1);
I'm trying to scrap a website but it always said that Empty Reply from server
can any one look at the code and tell me what am I doing wrong?
Here is the code
function spider($url){
$header = array(
"Host" => "www.example.net",
//"Accept-Encoding:gzip,deflate,sdch",
"Accept-Language:en-US,en;q=0.8",
"Cache-Control:max-age=0",
"Connection:keep-alive","Content-Length:725","Content-Type:application/x-www-form-urlencoded",
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
,"X-Requested-With:XMLHttpRequest"
);
$cookie = "cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 200); // http request time-out 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the URL changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection response
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, realpath( $cookie)); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, realpath( $cookie));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,"view=ViewDistrict¶m=7&uniqueid=1397991494188&PHPSESSID=f134vrnv7glosgojvf4n1mp7o2&page=http%3A%2F%2Fwww.example.com%2Fxhr.php");
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_REFERER, "http://www.example.com/");
$data = curl_exec($ch); // execute the http request
$info = curl_getinfo($ch);
curl_close($ch); // close the connection
return $data;
}
Here is function call
echo spider("http://www.example.net/");
Edit
Array ( [url] => http://www.example.net/ [content_type] => text/html [http_code] => 301 [header_size] => 196 [request_size] => 840 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 1 [total_time] => 61.359 [namelookup_time] => 0 [connect_time] => 0.281 [pretransfer_time] => 0.281 [size_upload] => 0 [size_download] => 0 [speed_download] => 0 [speed_upload] => 0 [download_content_length] => 178 [upload_content_length] => 0 [starttransfer_time] => 60.593 [redirect_time] => 0.766 [certinfo] => Array ( ) [redirect_url] => ) Empty reply from server
this is the header now
also I'd updated my post data
it's now
curl_setopt($ch, CURLOPT_POSTFIELDS,"view=ViewDistrict¶m=7&uniqueid=".time(). rand(101,500)."&PHPSESSID=f134vrnv7glosgojvf4n1mp7o2&page=http%3A%2F%2Fexample.com%2Fxhr.php");
and also had removed "X-Requested-With:XMLHttpRequest" from headers
Have you tried removing this from the headers ?
X-Requested-With:XMLHttpRequest
My guess is that your problem is in this line:
curl_setopt(
$ch,
CURLOPT_POSTFIELDS,
"view=ViewDistrict¶m=7&uniqueid=1397991494188&PHPSESSID=f134vrnv7glosgojvf4n1mp7o2&page=http%3A%2F%2Fwww.example.com%2Fxhr.php"
);
Notice that you're passing a value for PHPSESSID. I'm guessing you copied & pasted a URL from a visit to the site, right? That session ID was probably valid when you visited the site, but the odds of it being valid now are pretty slim. And if the server doesn't like the session ID, chances are it's not going to give you any data.
I'm trying to POST some data to a web service with cURL, using the following code:
$response = "<p>Here is your RMA information. Please ship the product to the address below.<br>
<br>
IMPORTANT:<br>
1) Refunds can only be issues for purchases made within 30 days. Products missing any accessories or original packaging will require a 20% restocking fee.<br>Please see our RMA Guidelines here:<br>
We will update you with the progress of the RMA once we receive your RMA. We appreciate your patience and understanding.<br>
</p>";
$url = "https://mysite.desk.com/api/v2/cases/18/notes";
$username = "my username";
$password = "my password";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, '{"body":"' . $response . '"}');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$output = curl_exec($ch);
it gives me following error
[url] => https://mysite.desk.com/api/v2/cases/18/notes
[content_type] =>
[http_code] => 500
[header_size] => 165
[request_size] => 228
[filetime] => -1
[ssl_verify_result] => 20
[redirect_count] => 0
[total_time] => 1.373
[namelookup_time] => 0
[connect_time] => 0.281
[pretransfer_time] => 0.811
[size_upload] => 2235
[size_download] => 0
[speed_download] => 0
[speed_upload] => 1627
[download_content_length] => -1
[upload_content_length] => 2235
[starttransfer_time] => 1.092
[redirect_time] => 0
I don't understand exact problem. please help me.
try array if json format data seem to be complicated
$post_array=array('body'=>$response);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_array;
It is solved by changing following lines
$data = array();
$data['body'] = preg_replace("/&#?[a-z0-9]{2,8};/i", "", strip_tags($_POST['response']));
and
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data));
I created an API for customers using cURL. I just moved to a new server this domain and now the api doesnt not work. Everything seems to be working fine modules wise but I can't get it to work:
This is the response I get.
Array (
[url] => https://www.1800pay.com/api/process.php
[content_type] => text/html; charset=iso-8859-1
[http_code] => 404
[header_size] => 179
[request_size] => 506
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.038607
[namelookup_time] => 0.002688
[connect_time] => 0.002737
[pretransfer_time] => 0.038372
[size_upload] => 0
[size_download] => 294
[speed_download] => 7615
[speed_upload] => 0
[download_content_length] => 294
[upload_content_length] => 0
[starttransfer_time] => 0.038597
[redirect_time] => 0 )
Curl error:
Not Found
The requested URL /api/process.php was not found on this server.
Apache/2.2.3 (CentOS) Server at www.1800pay.com Port 443
CODE USED:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.1800pay.com/api/process.php");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
//curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $output_transaction);
curl_setopt($ch, CURLOPT_POST, 1);
if (!($data = curl_exec($ch))) {print_r(curl_error($ch));echo "error";
return ERROR;
}
print_r(curl_getinfo($ch));
echo 'Curl error: ' . curl_error($ch);
curl_close($ch);
print_r($data);
Thanks for the help :)
And YES THE FILE EXISTS ON THE SERVER..... :|
The answer is within the response.. a not found error is exactly that. Just make sure the file api/process.php exists on the new domain.
I'm not sure what your problem is but when I try to curl the URL using this:
function curl_get($url)
{
$curl_handle = curl_init();
curl_setopt($curl_handle, CURLOPT_URL, $url);
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl_handle, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($curl_handle, CURLOPT_COOKIEFILE, "cookie.txt");
//curl_setopt($curl_handle, CURLOPT_USERPWD, $co.":".$apikey);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_DNS_USE_GLOBAL_CACHE, FALSE);
$buffer = curl_exec($curl_handle);
$error = curl_error($curl_handle);
curl_close($curl_handle);
//if (empty($buffer))
//echo "No response from server: ".$error;
//else
return $buffer;
}
I get this:
Error [Code:1043] Payment ()Key - incorrect data
Trying to do some local testing on a virtual server, the problem is cURL is returning a http_code => 0
I think it's to do with my virtual host naming.
Virtual Host Name: dev.project
the cURL request is adding http://
if I ping: dev.project from the command line, I get a hit.
If I try it with http://dev.project I get unknown host.
Is there a curl_setopt option just to use the hostname? I'm no sure if I can use the IP as there are several projects on the server, or would Apache handle this?
Here is what I have tried:
$request_url = 'dev.project';
$request_args = 'parm=1234';
$user_agent = 'Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.151 Safari/534.16';
$ch = curl_init();
// set curl options (GET)
curl_setopt($ch, CURLOPT_URL, $request_url.'?'.$request_args);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
//curl_setopt($ch, CURLOPT_FAILONERROR, TRUE);
//curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
//curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
//curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); // CHANGE THIS TO TRUE
// Set curl options (POST)
//curl_setopt($ch, CURLOPT_URL, $request_url);
//curl_setopt($ch, CURLOPT_POST, TRUE);
//curl_setopt($ch, CURLOPT_POSTFIELDS, $request_args);
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
//curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
//curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
//curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$response = curl_exec($ch);
echo "<pre>".print_r($response,true)."</pre><br />\n"; // nothing is returned
print_r(curl_getinfo($ch));
curl_close($ch);
Response from curl_getinfo() : (NOTE: the http:// is pre-pended in the url)
Array
(
[url] => http://dev.project?parm=1234
[http_code] => 0
[header_size] => 0
[request_size] => 0
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0
[namelookup_time] => 0
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => 0
[upload_content_length] => 0
[starttransfer_time] => 0
[redirect_time] => 0
)
If anyone has this issue, here is the fix:
// This is your Virtual Hosts name
$request_host = 'dev.project';
// This is the IP
$request_url = '192.168.0.1';
$headers = array("Host: ".$request_host);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_URL, $request_url.'?'.$request_args);
curl_setopt($ch, CURLOPT_HEADER, FALSE);