I have some PHP code that was at one point WORKING FINE. It makes a call out to an external API, the API has NOT CHANGED AT ALL. The PHP code has also NOT CHANGED AT ALL. But suddenly, I am getting no results back for this function:
if (!function_exists(setFieldsAndCallURL))
{
function setFieldsAndCallURL($url,$fields)
{
//url-ify the data for the POST
$fields_string='';
foreach($fields as $key=>$value)
{ $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string,'&');
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute the jump
$result = '';
$result = curl_exec($ch);
//close connection
curl_close($ch);
return $result;
}
}
Before, it would return a text GUID when called:
$userURL ='https://api.nottherealendpointurl.net/public/user/authenticate';
$userFields = array(
'username'=>$username,
'lastName'=>$lastname,
'firstName'=>$firstname,
'email'=>$email,
'token'=>urlencode($adminKey),
);
//Login this particular user
$userKey = setFieldsAndCallURL($userURL,$userFields);
But suddenly it has started returning "" (empty string) and I have no idea why.
Is there any way to get more info and spy on the inner workings of this thing? See the call it is making using HTTP header logging software? Or anything?
NOTE: I have already tested the POST manually to the API and it is working as expected, I am still getting back the proper GUID. For some reason doing it through this curl thing just suddenly quit doing it properly. Nobody has any idea what could be different now.
Propably the ip of the server is blocked by now, where your local ip is not?
You might want to add
$headerFile = fopen(filepath_to_header_file);
$errorFile = fopen(filepath_to_error_file);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_WRITEHEADER, $headerFile );
curl_setopt($ch, CURLOPT_STDERR, $errorFile );
to get the header of the response and the errors into files and look their content up.
edit:
To verify if the ip of the server is blocked you could try something like this
$host = "ssl://api.nottherealendpointurl.net/";
$port = 443;
$url = "/public/user/authenticate";
$timeout = 30;
$errno = "";
$errstr= "";
$fp = fsockopen($host, $port, $errno, $errstr, $timeout);
if($fp)
{
$request = "GET ".$url." HTTP/1.1\r\n";
$request.= "Host: ".$host."\r\n";
$request.= "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.7.12) Gecko/20050919 Firefox/1.0.7\r\n";
$request.= "Connection: Close\r\n\r\n";
fwrite($fp, $request);
while (!feof($fp))
{
$data .= fgets($fp, 128);
}
fclose($fp);
echo $data;
}
else
{
echo "ERROR: ".$errstr;
}
where $data contains the response from the remote server-
Depending on what platform you're on you can look at the raw packets, for linux command line only, that'd be tcpdump for widows/others you can use wireshark.
tcpdump -i eth1 tcp port 80
or
http://www.wireshark.org/download.html
Related
I want to parse a lot of URLs to only get their status codes.
So what I did is:
$handle = curl_init($url -> loc);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_HEADER , true); // we want headers
curl_setopt($handle, CURLOPT_NOBODY , true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$response = curl_exec($handle);
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
curl_close($handle);
But as soon as the "nobody"-option is set to true, the returned status codes are incorrect (google.com returns 302, other sites return 303).
Setting this option to false is not possible because of the performance loss.
Any ideas?
The default HTTP request method for curl is GET. If you want only the response headers, you can use the HTTP method HEAD.
curl_setopt($handle, CURLOPT_CUSTOMREQUEST, 'HEAD');
According to #Dai's answer, the NOBODY is already using the HEAD method. So the above method will not work.
Another option would be to use fsockopen to open a connection, write the headers using fwrite. Read the response using fgets until the first occurrence of \r\n\r\n to get the complete header. Since you need only the status code, you just need to read the first 13 characters.
<?php
$fp = fsockopen("www.google.com", 80, $errno, $errstr, 30);
if ($fp) {
$out = "GET / HTTP/1.1\r\n";
$out .= "Host: www.google.com\r\n";
$out .= "Accept-Encoding: gzip, deflate, sdch\r\n";
$out .= "Accept-Language: en-GB,en-US;q=0.8,en;q=0.6\r\n";
$out .= "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36\r\n";
$out .= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
$tmp = explode(' ', fgets($fp, 13));
echo $tmp[1];
fclose($fp);
}
cURL's nobody option has it use the HEAD HTTP verb, I'd wager the majority of non-static web applications I the wild don't handle this verb correctly, hence the problems you're seeing with different results. I suggest making a normal GET request and discarding the response.
i suggest get_headers() instead:
<?php
$url = 'http://www.example.com';
print_r(get_headers($url));
print_r(get_headers($url, 1));
?>
I am building a web application for my router, it will be my Bachelor's Thesis.
The bad thing is that I can't display my router's informations using my cURL function because I get bad router username and password error. I didn't found any problem at all:
The cURL function:
function myCurl($url, $post="")
{
global $status;
$header = 'Authorization: Basic YWRtaW46YWRtaW4=';
$cookiepath_tmp = "c:/xampp/htdocs/wifi/cookie.txt";
$resp = array();
$ch = curl_init();
curl_setopt($ch,CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
curl_setopt($ch,CURLOPT_URL, trim($url));
curl_setopt($ch,CURLOPT_REFERER, trim($url));
curl_setopt($ch,CURLOPT_COOKIEJAR,$cookiepath_tmp);
curl_setopt($ch,CURLOPT_COOKIEFILE,$cookiepath_tmp);
curl_setopt($ch,CURLOPT_COOKIESESSION, true);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_MAXREDIRS, 10);
curl_setopt($ch,CURLOPT_ENCODING, "");
curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
#curl_setopt($ch,CURLOPT_AUTOREFERER, true);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch,CURLOPT_TIMEOUT, 15);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false );
curl_setopt($ch,CURLOPT_HEADER, 0);
curl_setopt($ch,CURLOPT_HTTPHEADER, array( 'Expect:' ) );
curl_setopt($ch,CURLOPT_VERBOSE, 1);
#curl_setopt($ch,CURLOPT_FAILONERROR, true);
if($post) { curl_setopt($ch, CURLOPT_POST,1); curl_setopt($ch,CURLOPT_POSTFIELDS,$post); }
$returned = curl_exec($ch);
$resp['returned'] = $returned;
$status=curl_getinfo($ch);
$resp['status'] = $status;
curl_close($ch);
return $resp;
}
I am trying to display the informations using PHP:
The PHP code:
<?php echo $success_msg;
$url = "http://192.168.0.1/session.cgi";
$post = "REPORT_METHOD=xml&ACTION=login_plaintext&USER=admin&PASSWD=admin&CAPTCHA=";
$data = myCurl($url, $post);
#$url = "http://192.168.0.1/st_log.php";
#$data = myCurl($url);
echo $data['returned'];
?>
The error is:
Username or Password is incorrect.
However, The username and password admin are correct.
I have added the following code into myCurl function but still doesn't work:
$header = 'Authorization: Basic YWRtaW46YWRtaW4=';
YWRtaW46YWRtaW4= is the encoded username:password in Base64.
LAST EDIT:
I set the CURLOPT_HEADER to true, and I got this text displayed:
HTTP/1.1 501 Not Implemented Server: Router Webserver Connection: close WWW-Authenticate: Basic realm="TP-LINK Wireless Lite N Router WR740N" Content-Type: text/html
Any solution for this?
I really appreciate your help! Thank you!
I don't known what is your router (vendor / model) but most of them use HTTP basic authentication. And, when the authentication is empty or wrong you get a HTTP 401 error: Unauthorized, which could correspond to your error string.
So you should try to insert a HTTP authorization header in the cURL request:
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
Is there a way for PHP CURL functions to get the contents of the website, but stopped on the characters that we just ask. I think this sort of buffer.
so the script did not call the overall page
So schemes like this:
: curl execution
<html>
->
->
->
-> Title Detected
: curl close
->
->
->
->
</ html>
Please this is not a DOM problem. But how to curl stops when it finds that we ask.
this is my code :
function curl_download($Url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_REFERER, $Url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
If cURL can't handle this problem, how about fopen? and do you have an example?
Thanks before.Also please give me your example code for me,, thanks
Here is a very simple example using fsockopen(). Extend it to fit your needs.
$host = 'www.site.com';
$port = 80;
$sock = fsockopen($host, $port, $errno, $errstr, 30);
if (!$sock) {
die("Failed to connect. $errno: $errstr");
}
// write http request to socket:
$request = "GET /file.html HTTP/1.0\r\n"
."Host: $host\r\n"
."User-Agent: some-user-agent\r\n"
."Connection: close\r\n"
."\r\n";
fwrite($sock, $request);
$buffer = ''; // buffer for storing response
while (!feof($sock)) {
$buffer .= fgets($sock, 1024); // read 1024 bytes from socket, append to buffer
if (strpos($buffer, '</title>') !== false)) { // title was found
fclose($sock);
break;
}
}
So we connect to the HTTP server on the remote host, issue a simple HTTP/1.0 request, and read the response 1024 bytes at a time until the closing title tag is detected. Once it is found, the connection is closed.
Note, even though we didn't read the entire response from the socket, the underlying system (PHP and the OS socket layer) may have read more (or possibly all depending on size) of the response. In either case, you did prevent PHP from reading most of the response. If the pages are very big, closing the socket early will likely prevent a bulk of the data from actually ever being received.
Hope that helps.
I do not think you can parse the DOM with CURL
I advise you to use the string function like strstr, strtok...
http://www.php.net/manual/en/ref.strings.php
I need to send a curl request with the user's ip address not the server one. I tried this with no luck:
curl_setopt( $ch, CURLOPT_INTERFACE, $ip );
Any ideas?
Ok, so there's no way to safely spoof the ip address of a curl request, but I found a non-safe way, it depends on the server script receiving the request, but it worked for me to trick the API I was making the request to:
curl_setopt( $ch, CURLOPT_HTTPHEADER, array("REMOTE_ADDR: $ip", "HTTP_X_FORWARDED_FOR: $ip"));
This won't always work, but in this case it worked for me.
Thanks everyone for the help!
It doesn't work with curl for me so i found a way around it, I just had to do this and as long as the IP is assigned to your server, then:
echo http_socket::download('http://something.com', '55.55.44.33');
final class http_socket
{
static public function download($url, $bind_ip = false)
{
$components = parse_url($url);
if(!isset($components['query'])) $components['query'] = false;
if(!$bind_ip)
{
$bind_ip = $_SERVER['SERVER_ADDR'];
}
$header = array();
$header[] = 'GET ' . $components['path'] . ($components['query'] ? '?' . $components['query'] : '');
$header[] = 'Host: ' . $components['host'];
$header[] = 'User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100106 Ubuntu/9.10 (karmic) Firefox/3.5.7';
$header[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
$header[] = 'Accept-Language: en-us,en;q=0.5';
$header[] = 'Accept-Encoding: gzip,deflate';
$header[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7';
$header[] = 'Keep-Alive: 300';
$header[] = 'Connection: keep-alive';
$header = implode("\n", $header) . "\n\n";
$packet = $header;
//----------------------------------------------------------------------
// Connect to server
//----------------------------------------------------------------------
$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
socket_bind($socket, $bind_ip);
socket_connect($socket, $components['host'], 80);
//----------------------------------------------------------------------
// Send First Packet to Server
//----------------------------------------------------------------------
socket_write($socket, $packet);
//----------------------------------------------------------------------
// Receive First Packet to Server
//----------------------------------------------------------------------
$html = '';
while(1) {
socket_recv($socket, $packet, 4096, MSG_WAITALL);
if(empty($packet)) break;
$html .= $packet;
}
socket_close($socket);
return $html;
}
}
Spoofing an IP address is not something cURL can do. That's a lower-level operation requiring manipulation of raw socket connections.
None of upper solutions has worked for me. However making a request through a proxy works very well:
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '127.0.0.1:8888'; //put your proxy here
//$proxyauth = 'user:password';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
here are the lines of php code which may work.
You may use any other method to set "X-Forwarded-For" header.
$httpClient = new Zend_Http_Client($reqUrl);
$httpClient->setHeaders("X-Forwarded-For","127.0.0.1"); //---this sets the desired ip address
That's because you are supposed to put your server's ip address there.
You cannot forge an IP packet with fake source address using curl.
Use HTTP_X_REAL_IP header in addition with HTTP_X_FORWARDED_FOR and REMOTE_ADDR like
"HTTP_X_REAL_IP: xxx.xxx.xxx.xx"
Use it into the header add tags like REMOTE_ADDR: majbase HTTP_X_FORWARDED_FOR: codix like "HTTP_X_REAL_IP: xxx.xxx.xxx.xx"
In PHP, how can I replicate the expand/contract feature for Tinyurls as on search.twitter.com?
If you want to find out where a tinyurl is going, use fsockopen to get a connection to tinyurl.com on port 80, and send it an HTTP request like this
GET /dmsfm HTTP/1.0
Host: tinyurl.com
The response you get back will look like
HTTP/1.0 301 Moved Permanently
Connection: close
X-Powered-By: PHP/5.2.6
Location: http://en.wikipedia.org/wiki/TinyURL
Content-type: text/html
Content-Length: 0
Date: Mon, 15 Sep 2008 12:29:04 GMT
Server: TinyURL/1.6
example code...
<?php
$tinyurl="dmsfm";
$fp = fsockopen("tinyurl.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /$tinyurl HTTP/1.0\r\n";
$out .= "Host: tinyurl.com\r\n";
$out .= "Connection: Close\r\n\r\n";
$response="";
fwrite($fp, $out);
while (!feof($fp)) {
$response.=fgets($fp, 128);
}
fclose($fp);
//now parse the Location: header out of the response
}
?>
And here is how to contract an arbitrary URL using the TinyURL API. The general call pattern goes like this, it's a simple HTTP request with parameters:
http://tinyurl.com/api-create.php?url=http://insertyourstuffhere.com
This will return the corresponding TinyURL for http://insertyourstuffhere.com. In PHP, you can wrap this in an fsockopen() call or, for convenience, just use the file() function to retrieve it:
function make_tinyurl($longurl)
{
return(implode('', file(
'http://tinyurl.com/api-create.php?url='.urlencode($longurl))));
}
// make an example call
print(make_tinyurl('http://www.joelonsoftware.com/items/2008/09/15.html'));
As people have answered programatically how to create and resolve tinyurl.com redirects, I'd like to (strongly) suggest something: caching.
In the twitter example, if every time you clicked the "expand" button, it did an XmlHTTPRequest to, say, /api/resolve_tinyurl/http://tinyurl.com/abcd, then the server created a HTTP connection to tinyurl.com, and inspected the header - it would destroy both twitter and tinyurl's servers..
An infinitely more sensible method would be to do something like this Python'y pseudo-code..
def resolve_tinyurl(url):
key = md5( url.lower_case() )
if cache.has_key(key)
return cache[md5]
else:
resolved = query_tinyurl(url)
cache[key] = resolved
return resolved
Where cache's items magically get backed up into memory, and/or a file, and query_tinyurl() works as Paul Dixon's answer does.
Here is another way to decode short urls via CURL library:
function doShortURLDecode($url) {
$ch = #curl_init($url);
#curl_setopt($ch, CURLOPT_HEADER, TRUE);
#curl_setopt($ch, CURLOPT_NOBODY, TRUE);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
#curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$response = #curl_exec($ch);
preg_match('/Location: (.*)\n/', $response, $a);
if (!isset($a[1])) return $url;
return $a[1];
}
It's described here.
Another simple and easy way:
<?php
function getTinyUrl($url) {
return file_get_contents('http://tinyurl.com/api-create.php?url='.$url);
}
?>
If you just want the location, then do a HEAD request instead of GET.
$tinyurl = 'http://tinyurl.com/3fvbx8';
$context = stream_context_create(array('http' => array('method' => 'HEAD')));
$response = file_get_contents($tinyurl, null, $context);
$location = '';
foreach ($http_response_header as $header) {
if (strpos($header, 'Location:') === 0) {
$location = trim(strrchr($header, ' '));
break;
}
}
echo $location;
// http://www.pingdom.com/reports/vb1395a6sww3/check_overview/?name=twitter.com%2Fhome
In PHP there is also a get_headers function that can be used to decode tiny urls.
The Solution here from #Pons solution, didn't work alone on my php7.3 server reslolving stackexchange URLs like https://stackoverflow.com/q/62317
This solved it:
public function doShortURLDecode($url) {
$ch = #curl_init($url);
#curl_setopt($ch, CURLOPT_HEADER, TRUE);
#curl_setopt($ch, CURLOPT_NOBODY, TRUE);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
#curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$response = #curl_exec($ch);
$cleanresponse= preg_replace('/[^A-Za-z0-9\- _,.:\n\/]/', '', $response);
preg_match('/Location: (.*)[\n\r]/', $cleanresponse, $a);
if (!isset($a[1])) return $url;
return parse_url($url, PHP_URL_SCHEME).'://'.parse_url($url, PHP_URL_HOST).$a[1];
}