I'm building a Curl web automation app and am having some issue with not getting the desired outcome of my POST action, I am having some trouble figuring out how I can show the full POST request I am sending over (with headers), I have been searching on this but everything that comes up is the response headers, actually I want these too but also the request, which none of the posts I find on google seem to mention..
I know I can display the result of a curl request using something like this (forgive me if my syntax is off, I already shut down my virtual machine with my ide and code to refer to
$result = curl($curl_exect) ;
Anyways, I would greatly appreciate any advice on how to view the full headers, thanks
Here is all you need:
curl_setopt($curlHandle, CURLINFO_HEADER_OUT, true); // enable tracking
... // do curl request
$headerSent = curl_getinfo($curlHandle, CURLINFO_HEADER_OUT ); // request headers
You can see the information regarding the transfer by doing:
curl_setopt($curl_exect, CURLINFO_HEADER_OUT, true);
before the request, and
$information = curl_getinfo($curl_exect);
after the request
View: http://www.php.net/manual/en/function.curl-getinfo.php
You can also use the CURLOPT_HEADER in your curl_setopt
curl_setopt($curl_exect, CURLOPT_HEADER, true);
$httpcode = curl_getinfo($c, CURLINFO_HTTP_CODE);
return $httpcode == 200;
These are just some methods of using the headers.
You can save all headers sent by curl to a file using :
$f = fopen('request.txt', 'w');
curl_setopt($ch,CURLOPT_VERBOSE,true);
curl_setopt($ch,CURLOPT_STDERR ,$f);
You can make you request headers by yourself using:
// open a socket connection on port 80
$fp = fsockopen($host, 80);
// send the request headers:
fputs($fp, "POST $path HTTP/1.1\r\n");
fputs($fp, "Host: $host\r\n");
fputs($fp, "Referer: $referer\r\n");
fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
fputs($fp, "Content-length: ". strlen($data) ."\r\n");
fputs($fp, "Connection: close\r\n\r\n");
fputs($fp, $data);
$result = '';
while(!feof($fp)) {
// receive the results of the request
$result .= fgets($fp, 128);
}
// close the socket connection:
fclose($fp);
Like writen on how make request
I had exactly the same problem lately, and I installed Wireshark (it is a network monitoring tool). You can see everything with this, except encrypted traffic (HTTPS).
Related
I am downloading a recording from an external url and saving it using CURL as follows:
$ch = curl_init($Recording);
$fp = fopen($recording_file_loc, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
I need to change the file permissions once file is completely downloaded as follows.
chmod($recording_file_loc , 0640);
How can i check and ensure that file is completely downloaded before executing chmod??
updated:
I updated my code as follows:
$statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
and
if($statusCode == 200){
chmod($recording_file_loc , 0640);
}
else{
echo $statusCode;
}
You need to put to check if the download process is complete.
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'progress'); // call progress function
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
Then you need to define a function which checks the download progress
// progress function definition
function progress($resource,$download_size, $downloaded, $upload_size, $uploaded)
{
// Progress
if($download_size > 0)
echo $downloaded / $download_size * 100;
if($downloaded / $download_size == 1){
// chmod code here
}
}
Check this link cURL download progress in PHP
on transfers where curl detected any errors, curl_errno($ch) should no longer return 0, so if(curl_errno($ch)!==0), something bad probably happened to your download.
another thing, as pointed out by #Pamela in a comment, if the response code is not 2XX (like HTTP 200 OK or HTTP 204 No Content), that's another sign something probably went wrong, which can be detected by doing if(((string)curl_getinfo($ch,CURLINFO_RESPONSE_CODE))[0]!=='2')
so..
if(curl_errno($ch)!==0 || ((string)curl_getinfo($ch,CURLINFO_RESPONSE_CODE))[0]!=='2'){
// the download probably failed.
}
generally speaking, this may be impossible to detect on servers that doesn't implement "Content-Length" headers, if you're downloading from a server that doesn't support Content-Length, then there may be no standardized way to detect the broken download at all.. (you may have to inspect what you've downloaded to make sure it's what you expect or something, idk)
for example, on transfers where the body length doesn't match the "Content-Length" header, curl_errno($ch) returns int(56) (instead of the usual int(0)), and curl_exec($ch) returns bool(false) (PS! if you used CURLOPT_RETURNTRANSFER, then it may contain a string instead of bool)
here's a little HTTP server sending "Content-Length: 3", then cutting the connection after just sending 2 (of allegedly 3) bytes of the body:
<?php
$port=1234;
$srv=socket_create_listen($port);
while(($conn=socket_accept($srv))){
$headers=implode("\r\n",array(
"HTTP/1.1 200 OK",
"Content-Type: text/plain",
"Content-Length: 3",
"Connection: close",
"","",
));
// i lied! i said 3 bytes body, but only send 2 bytes body
$body="ab";
$response=$headers.$body;
var_dump(strlen($response),socket_write($conn,$response));
socket_close($conn);
}
and an accompanying test script:
<?php
$ch=curl_init("http://127.0.0.1:1234");
var_dump(curl_exec($ch));
var_dump(curl_errno($ch),curl_error($ch));
printing:
abbool(false)
int(56)
string(38) "Recv failure: Connection reset by peer"
Get the info like this:
$ch = curl_init($Recording);
$fp = fopen($recording_file_loc, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
fclose($fp);
Then check the "download_content_length" against "size_download" like this:
if($info["download_content_length"]==$info["size_download"])
{
//Download complete!
}
else
{
//Error
}
Note that it works only if server sends the Content-Length header in advance.
A server I am working on appears to be denying outbound HTTP requests. The reason I think this is because I've tried both Guzzle and curl requests to the API.
The API lives on the same domain as the web server (this is temporary at clients request). I can make requests to the API server via Postman (Chrome plugin), but when I run that same request on the server, it doesn't return anything.
Here are the headers from the 'Postman' request:
POST /api2/user/session HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 49
Cache-Control: no-cache
Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: PHPSESSID=d9ad79c4c0822fc5c86f4d8799307f1b; _ga=GA1.2.1674422587.1425409444
Post data:
token=a559d5bba5a9e9517d5c3ed7aeb62db6&user=30972
This works. It returns the data. But when I call the same endpoint from within my web app, I get nothing.
$data = urlencode("token=a559d5bba5a9e9517d5c3ed7aeb62db6&user=30972");
$ch = curl_init('http://example.com/api2/user/session');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: ' . strlen($data))
);
$result = curl_exec($ch);
What I don't understand is I can run the following, and it returns the content:
print file_get_contents("http://www.google.com");
When I var_dump the $_POST fields on the endpoint user/session it returns the array of postdata using Postman but $_POST fields are blank when sending via the web app. Even before it makes any request to the database, the post fields should be set right?
Via SSH this also works:
curl -F token=a559d5bba5a9e9517d5c3ed7aeb62db6 -F user=30972 http://example.com/api2/user/session
As suggested in comments I've tried:
var_dump(function_exists('curl_version'));
// bool(true)
I can't figure out what's going on.
Edit: This works ... but I don't want to use sockets. Must be a curl issue.
$fp = fsockopen('example.com', 80);
$vars = array(
'token' => 'a559d5bba5a9e9517d5c3ed7aeb62db6',
'user' => '30972'
);
$content = http_build_query($vars);
fwrite($fp, "POST /api2/user/session HTTP/1.1\r\n");
fwrite($fp, "Host: example.com\r\n");
fwrite($fp, "Content-Type: application/x-www-form-urlencoded\r\n");
fwrite($fp, "Content-Length: ".strlen($content)."\r\n");
fwrite($fp, "Connection: close\r\n");
fwrite($fp, "\r\n");
fwrite($fp, $content);
header('Content-type: text/plain');
while (!feof($fp)) {
echo fgets($fp, 1024);
}
Edit:
curl_error() also returns no error.
To better understand the differences between the PHP code and cURL, I created a RequestBin instance and tried both on it. They yielded drastically different results:
It seemed like the POST data from the PHP script yielded an incorrect result for what was sent. This can be fixed by using a built-in PHP function http_build_query.
It will yield a more apt result:
This can be caused by a session lock... If you use curl to access the same server, the same session is used. While the script is running, the session is locked by default, this means that the current request has to finish before another is handled for the same session. This would explain a timeout of the request in curl, as your first request is not completed and another is made...
Using session_write_close() before the curl_exec will unlock the session and correct the problem.
It turns out I needed to use http_build_query.
$vars = array(
'token' => 'a559d5bba5a9e9517d5c3ed7aeb62db6',
'user' => '30972'
);
$content = http_build_query($vars);
$ch = curl_init('http://example.com/api2/user/session');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: ' . strlen($content))
);
$result = curl_exec($ch);
I want to parse a lot of URLs to only get their status codes.
So what I did is:
$handle = curl_init($url -> loc);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_HEADER , true); // we want headers
curl_setopt($handle, CURLOPT_NOBODY , true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$response = curl_exec($handle);
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
curl_close($handle);
But as soon as the "nobody"-option is set to true, the returned status codes are incorrect (google.com returns 302, other sites return 303).
Setting this option to false is not possible because of the performance loss.
Any ideas?
The default HTTP request method for curl is GET. If you want only the response headers, you can use the HTTP method HEAD.
curl_setopt($handle, CURLOPT_CUSTOMREQUEST, 'HEAD');
According to #Dai's answer, the NOBODY is already using the HEAD method. So the above method will not work.
Another option would be to use fsockopen to open a connection, write the headers using fwrite. Read the response using fgets until the first occurrence of \r\n\r\n to get the complete header. Since you need only the status code, you just need to read the first 13 characters.
<?php
$fp = fsockopen("www.google.com", 80, $errno, $errstr, 30);
if ($fp) {
$out = "GET / HTTP/1.1\r\n";
$out .= "Host: www.google.com\r\n";
$out .= "Accept-Encoding: gzip, deflate, sdch\r\n";
$out .= "Accept-Language: en-GB,en-US;q=0.8,en;q=0.6\r\n";
$out .= "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36\r\n";
$out .= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
$tmp = explode(' ', fgets($fp, 13));
echo $tmp[1];
fclose($fp);
}
cURL's nobody option has it use the HEAD HTTP verb, I'd wager the majority of non-static web applications I the wild don't handle this verb correctly, hence the problems you're seeing with different results. I suggest making a normal GET request and discarding the response.
i suggest get_headers() instead:
<?php
$url = 'http://www.example.com';
print_r(get_headers($url));
print_r(get_headers($url, 1));
?>
Scenario : I have a c# .net web page. I want the user to be able to download a file placed on a remote server from a link on my page. However while downloading there should be minimum load on my server. Hence i tried creating a HttpWebRequest instance, passed the download.php path
e.g.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://servername/download.php");
myHttpWebRequest.Headers.Add
("Content-disposition", "attachment;filename=XXX.pdf");
myHttpWebRequest.ContentType = "application/pdf";
Passed the httprequest object in the session; however while reading the httpwebresponse on another page the contenttype is reset to "text/html".
Also the php file readers the headers and uses a readfile command to download the file. It gives the following error. Warning: readfile() [function.readfile]: URL file-access is disabled in the server configuration in
I don't entirely understand the scenario, but on PHP side, if fopen() URL access is disabled, your next port of call should be the curl family of functions. (Or, of course, activate URL access using the allow_url_fopen php.ini option but it sounds like you can't do that.)
The text/html header is probably due to the download failing.
A very rudimentary example:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch); // $result will contain the contents of the request
curl_close($ch);
?>
You can get around allow_url_fopen restrictions by using fsockopen. Here's a (rudimentary) implementation:
function fsock_get_contents($url) {
$fp = fsockopen($url, 80, $errno, $errstr, 20);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
return false;
} else {
$out = "GET / HTTP/1.1\r\n";
$out .= "Host: " . parse_url($url, PHP_URL_HOST) . "\r\n";
$out .= "Connection: Close\r\n\r\n";
$contents = '';
fwrite($fp, $out);
while (!feof($fp)) {
$contents .= fgets($fp, 128);
} fclose($fp);
return $contents;
}
}
echo fsock_get_contents('www.google.com');
I want to send a raw http packet to a webserver and recieve its response but i cant find out a way to do it. im inexperianced with sockets and every link i find uses sockets to send udp packets. any help would be great.
Take a look at this simple example from the fsockopen manual page:
<?php
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET / HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
echo fgets($fp, 128);
}
fclose($fp);
}
?>
The connection to the server is established with fsockpen. $out holds the HTTP request that’s then send with frwite. The HTTP response is then read with fgets.
If all you want to do is perform a GET request and receive the body of the response, most of the file functions support using urls:
<?php
$html = file_get_contents('http://google.com');
?>
<?php
$fh = fopen('http://google.com', 'r');
while (!feof($fh)) {
$html .= fread($fh);
}
fclose($fh);
?>
For more than simple GETs, use curl (you have to compile it into php). With curl you can do POST and HEAD requests, as well as set various headers.
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://google.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$html = curl_exec($ch);
?>
cURL is easier than implementing client side HTTP. All you have to do is set a few options and cURL handles the rest.
$curl = curl_init($URL);
curl_setopt_array($curl,
array(
CURLOPT_USERAGENT => 'Mozilla/5.0 (PLAYSTATION 3; 2.00)',
CURLOPT_HTTPAUTH => CURLAUTH_ANY,
CURLOPT_USERPWD => 'User:Password',
CURLOPT_RETURNTRANSFER => True,
CURLOPT_FOLLOWLOCATION => True
// set CURLOPT_HEADER to True if you want headers in the result.
)
);
$result = curl_exec($curl);
If you need to set a header that cURL doesn't support, use the CURLOPT_HTTPHEADER option, passing an array of additional headers. Set CURLOPT_HEADERFUNCTION to a callback if you need to parse headers. Read the docs for curl_setopt for more options.