I'm using the following code to post information to a URL.
$query = http_build_query($myvars);
$options = array(
'http' => array(
'header' => "Content-Type: application/x-www-form-urlencoded\r\n".
"Content-Length: ".strlen($query)."\r\n".
"User-Agent:MyAgent/1.0\r\n",
'method' => "POST",
'content' => $query,
),
);
$context = stream_context_create($options);
$response = file_get_contents($url, false, $context);
Is it possible to show the complete header information of the response.
First I used curl, but this took to much cpu.
With curl I used the following option:
curl_setopt($ch, CURLOPT_HEADER, 1);
And I received the following header information:
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Mon, 21 Sep 2015 07:06:35 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.11
Content-Description: File Transfer
Content-Disposition: attachment; filename=File.txt
Content-Transfer-Encoding: binary
Content-Length: 333
Cache-Control: must-revalidate, post-check=0, pre-check=0
Expires: 0
Pragma: public
Vary: Accept-Encoding
Content-Type: text/plain
Is the with file_get_contents also possible?
file_get_contents("http://example.com");
var_dump($http_response_header);
http://php.net/manual/en/reserved.variables.httpresponseheader.php
Related
I'm working on attaching files via PHP to confluence (version 5.9.10)
Here is my code
$ch=curl_init();
$headers = array(
'X-Atlassian-Token: no-check'
);
$data = array('file' => '#test.txt');
curl_setopt_array(
$ch,
array(
CURLOPT_URL=>'https://<path_to_confluence>/rest/api/content/<page_id>/child/attachment',
CURLOPT_POST=>true,
CURLOPT_VERBOSE=>1,
CURLOPT_POSTFIELDS=>$data,
CURLOPT_SSL_VERIFYHOST=> 0,
CURLOPT_SSL_VERIFYPEER=> 0,
CURLOPT_RETURNTRANSFER=>true,
CURLOPT_HEADER=>false,
CURLOPT_HTTPHEADER=> $headers,
CURLOPT_USERPWD=>C_USERNAME.":".C_PASSWORD
)
);
$result=curl_exec($ch);
$ch_error = curl_error($ch);
if ($ch_error) {
echo "cURL Error: $ch_error";
} else {
var_dump($result);
}
curl_close($ch);
But after running script I've next error:
>
HTTP/1.1 500 Internal Server Error
Server: nginx/1.5.12
Date: Thu, 03 Nov 2016 10:12:44 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-ASEN: SEN-2160053
Set-Cookie: JSESSIONID=EE3116DFC552C7D4571608BFCF410559; Path=/; HttpOnly
X-Seraph-LoginReason: OK
X-AUSERNAME: user
Cache-Control: no-cache, must-revalidate
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Content-Type-Options: nosniff
HTTP error before end of send, stop sending
Closing connection 0
string(93) "statusCode":500,"message":"java.lang.IllegalArgumentException:File name must not be null"
What I'm doing wrong? What have I missed?
Try this:
$data = array('file' => '#' . realpath('test.txt'));
...
CURLOPT_POSTFIELDS=>$data
If the filename is required, you can try this instead:
$data = array('file' => '#' . realpath('test.txt') . ';filename=test.txt'));
...
CURLOPT_POSTFIELDS=>$data
I encountered this error when I was using the lib cURL and was doing it the old way, like:
CURLOPT_POSTFIELDS => [
'file' => '#/pathToFile'
]
But since it's deprecated from PHP 5.5, you need to use it the new way:
Imperative way:
CURLOPT_POSTFIELDS => [
'file' => curl_file_create($filePath)
]
Object way:
CURLOPT_POSTFIELDS => [
'file' => new CurlFile($filePath)
]
I am trying to use twitter API with PHP CURL (previously I used file_get_contents which I had to abandon due to file_get_content not able to respond effectively to twitter rate limits).
When I run the code using file_get_content, I get a string in response, on which I can easily use json_decode() and my work is done. But...
When I run the code using curl, in the response i get a one huge string that has the same data that I obtained using the above method plus some additional information. And because of that I cant use json_decode() on that response. Following are the two responses.
Response From file_get_content
string(1964) "{"id":2988119635,"id_str":"2988119635","name":"Michael Jackson","screen_name":"Yahoo6464","location":"","description":"","url":null,"entities":{"description":{"urls":[]}},"protected":false,"followers_count":4,"friends_count":2,"listed_count":2,"created_at":"Sun Jan 18 07:18:41 +0000 2015","favourites_count":6,"utc_offset":null,"time_zone":null,"geo_enabled":false,"verified":false,"statuses_count":144,"lang":"en","status":{"created_at":"Tue Mar 15 07:09:21 +0000 2016","id":709637592510308352,"id_str":"709637592510308352","text":"noooooooooooooooooooo"}"
Response From CURL
string(2942) "HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-disposition: attachment; filename=json.json
content-length: 1964
content-type: application/json;charset=utf-8
date: Wed, 16 Mar 2016 06:58:18 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Wed, 16 Mar 2016 06:58:18 GMT
pragma: no-cache
server: tsa_f
set-cookie: lang=en; Path=/
set-cookie: guest_id=v1%3A145811149839783410; Domain=.twitter.com; Path=/; Expires=Fri, 16-Mar-2018 06:58:18 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read-write
x-connection-hash: e23e2992def7a3837cdbb3a3201bf7de
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-rate-limit-limit: 15
x-rate-limit-remaining: 12
x-rate-limit-reset: 1458112290
x-response-time: 171
x-transaction: 3fca373925e5f65b
x-twitter-response-tags: BouncerExempt
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block
{"id":2988119635,"id_str":"2988119635","name":"Michael Jackson","screen_name":"Yahoo6464","location":"","description":"","url":null,"entities":{"description":{"urls":[]}},"protected":false,"followers_count":4,"friends_count":2,"listed_count":2,"created_at":"Sun Jan 18 07:18:41 +0000 2015","favourites_count":6,"utc_offset":null,"time_zone":null,"geo_enabled":false,"verified":false,"statuses_count":144,"lang":"en","status":{"created_at":"Tue Mar 15 07:09:21 +0000 2016","id":709637592510308352,"id_str":"709637592510308352","text":"noooooooooooooooooooo"}"
Here is my curl options
$options = array(
CURLOPT_URL => $url,
CURLOPT_HEADER => true,
CURLINFO_HEADER_OUT => true,
CURLOPT_HTTPHEADER => $header,
CURLOPT_POST => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_CUSTOMREQUEST => 'GET',
);
$ci = curl_init();
$d = curl_setopt_array($ci, $options);
$response = curl_exec($ci);
Just set CURLOPT_HEADER to false.
I need to get the location header the following request:
This is what the request looks like from the chrome network dev tool when I execute it in chrome.
here is the redirect request in chrome network dev tool, just in case its needed for some reason.
I've tried the following things:
this seems to only give me an empty answer:
<?php
$url = $_GET["url"];
if (0 === preg_match('/form_build_id" value="form-([^"]*)"/', file_get_contents('http://anything2mp3.com/'), $matches)) exit("false");
$id = $matches[1];
$params = array('http' => array(
'method' => 'POST',
'content' => array(
'url' => $url,
'op' => "Convert",
'form_build_id' => "form-" . $id,
'form_id' => "videoconverter_convert_form"
),
'max_redirects' => "0",
'header' => "Content-Type:application/x-www-form-urlencoded"
));
stream_context_set_default($params);
$headers = get_headers("http://anything2mp3.com/?q=&mobile_detect_caching_notmobile&mobile_detect_caching_nottablet", false, $ctx);
echo implode("\n", $headers);
?>
i've also tried this:
<?php
$url = $_GET["url"];
if (0 === preg_match('/form_build_id" value="form-([^"]*)"/', file_get_contents('http://anything2mp3.com/'), $matches)) exit("false");
$id = $matches[1];
$params = array('http' => array(
'method' => 'POST',
'content' => array(
'url' => $url,
'op' => "Convert",
'form_build_id' => "form-" . $id,
'form_id' => "videoconverter_convert_form"
),
'max_redirects' => "0",
'header' => "Content-Type:application/x-www-form-urlencoded"
));
$ctx = stream_context_create($params);
$file = fopen("http://anything2mp3.com/?q=&mobile_detect_caching_notmobile&mobile_detect_caching_nottablet", 'rb', false, $ctx);
$headers = $http_response_header;
echo implode("\n", $headers);
?>
the last one gives me the following result:
HTTP/1.1 200 OK
Date: Sun, 26 Jul 2015 18:20:20 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Set-Cookie: __cfduid=db7807d30a61600d31a18cc1a725150811437934820; expires=Mon, 25-Jul-16 18:20:20 GMT; path=/; domain=.anything2mp3.com; HttpOnly
Vary: Accept-Encoding
X-Cookie-Domain: .anything2mp3.com
Expires: Tue, 24 Jan 1984 08:00:00 GMT
Last-Modified: Sun, 26 Jul 2015 18:20:20 GMT
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: W/"1437934820"
Content-Language: en
X-Device: normal
X-GeoIP-Country-Code: US
X-GeoIP-Country-Name: United States
X-Speed-Cache-Key: /?q=&mobile_detect_caching_notmobile&mobile_detect_caching_nottablet
X-NoCache: Method
X-This-Proto: http
X-Server-Name: anything2mp3.com
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Server: cloudflare-nginx
CF-RAY: 20c21e3333eb23a8-IAD
as you can see, there is no location header.
note that I've seen some similar posts already (like question 6532753, I couldn't link it because I'm already at my 2 link limit), but they don't seem to work for me.
How do I retrieve the location header from request like marked in the original screenshot? What am I missing?
edit: I messed up pretty hard when copying my code, sorry for that
I eventually got it to work using curl like #frz3993 suggested (and after sending in a blank 'cookies' header, the server didn't seem to like it without that)
here is the code that I use now:
<?php
$url = $_GET["url"];
if (0 === preg_match('/form_build_id" value="form-([^"]*)"/', file_get_contents('http://anything2mp3.com/'), $matches)) exit("false");
$longid = $matches[1];
$ch = curl_init("http://anything2mp3.com/?q=&mobile_detect_caching_notmobile&mobile_detect_caching_nottablet");
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query(array(
'url' => $url,
'op' => "Convert",
'form_build_id' => "form-" . $longid,
'form_id' => "videoconverter_convert_form"
)));
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Content-Type:application/x-www-form-urlencoded",
"Cookie:"
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
$data = curl_exec($ch);
if (0 === preg_match('/Location: (.*id=(\d+))/', $data, $matches)) exit("false");
$redirect = $matches[1];
$id = $matches[2];
echo $redirect;
echo " - ";
echo $longid;
die();
?>
note that even tho the second suggestion from my OP gave me headers, it did not give me the location header which curl does now. (don't ask me why :p)
I have a PHP script, that should connect to a proxy, chosen from a proxy list and download a file. Some of the proxies (out of 200-400 working ones) work perfectly, but others don't, and I cannot find out why.
Here the code that connects through the proxy:
$proxy = determine_proxy ($proxyList);
$proxyString = 'tcp://' . $proxy['ip'] . ':' . $proxy['port'];
$userAgent = $userAgents [rand (0, $agentsCount - 1)];
// set up our headers
$hdrs = array( 'http' => array(
'method' => "GET",
'header'=> "Host: www.example.net\r\n" .
// "User-Agent: $userAgent\r\n" .
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" .
"Accept-Language: en-us,en;q=0.5\r\n" .
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n" .
"Keep-Alive: 115\r\n" .
"Proxy-Connection: keep-alive\r\n" .
"Referer: http://$url", // Setting the http-referer
'proxy' => "$proxyString",
'request_fulluri' => true
)
);
echo "Using proxy: "; print_r ($proxy); echo '<br>';
$context = stream_context_create ($hdrs); // set up the context
$timeout = 3;
$oldTimeout = ini_set('default_socket_timeout', $timeout);
$oldAgent = ini_set ('user_agent', $userAgent);
$fp = fopen ("http://www.example.net$file", 'r', false, $context); // open the file
if (!$fp) {
echo 'fopen failed! Skipping this proxy for now...<br>';
print_r ($http_response_header); echo '<br />';
unset ($http_response_header);
flush(); #ob_flush();
ini_set ('user_agent', $oldAgent);
ini_set('default_socket_timeout', $oldTimeout);
continue;
}
print_r ($http_response_header); echo '<br />';
unset ($http_response_header);
The bizarre thing is that the response header for the failed tries is sometimes empty, and sometimes it's the following:
Array (
[0] => HTTP/1.0 200 OK
[1] => Server: falcon
[2] => Date: Sun, 16 Jan 2011 14:06:37 GMT
[3] => Content-Type: application/x-bittorrent
[4] => Cache-Control: must-revalidate, post-check=0, pre-check=0
[5] => Content-Disposition: attachment; filename="example.torrent"
[6] => Vary: Accept-Encoding,User-Agent
[7] => Connection: close
)
And sometimes, it's this:
Array (
[0] => HTTP/1.0 200 OK
[1] => Server: falcon
[2] => Date: Sun, 16 Jan 2011 14:06:47 GMT
[3] => Content-Type: application/x-bittorrent
[4] => Cache-Control: must-revalidate, post-check=0, pre-check=0
[5] => Content-Disposition: attachment; filename="example2.torrent"
[6] => Vary: Accept-Encoding,User-Agent
[7] => X-Cache: MISS from proxy
[8] => Proxy-Connection: close
)
This is a response header from a successful attempt:
HTTP/1.0 200 OK
Server: falcon
Date: Fri, 21 Jan 2011 18:53:00 GMT
Content-Type: application/x-bittorrent
Cache-Control: must-revalidate, post-check=0, pre-check=0
Content-Disposition: attachment; filename="example3.torrent"
Vary: Accept-Encoding,User-Agent
X-Cache: MISS from www.example.com
X-Cache-Lookup: MISS from www.example.com:3128
Via: 1.0 www.example.com (squid/3.0.STABLE23-BZR)
Proxy-Connection: close
I am setting the user agent to be a valid user agent string, I have checked allow_url_fopen and it is set to On.
From RFC-2616, section 10:
200 OK
The request has succeeded. The
information returned with the response
is dependent on the method used in the
request, for example:
GET an entity corresponding to the
requested resource is sent in the
response;
How is it possible, that the server via the proxy returns a status of 200, and still fopen fails? Does anybody have an idea about the problem and how to fix it?
The problem was the fact, that I was setting a socket timeout that in some cases was too low for fopen to manage and download all the data. After the timeout period has gone and fopen still hadn't downloaded the data, it returned FALSE and threw an "HTTP reqeust failed" error.
the server reported 200 OK , but the proxy still didn't know where to forward this data to , so you got a Request Failed ...
try using the VIA header
I am building a basic link checker at work using cURL. My application has a function called getHeaders() that returns an array of HTTP headers:
function getHeaders($url) {
if(function_exists('curl_init')) {
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
$options = array(
CURLOPT_URL => $url,
CURLOPT_HEADER => true,
CURLOPT_NOBODY => true,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_RETURNTRANSFER => true );
curl_setopt_array($ch, $options);
// grab URL and pass it to the browser
curl_exec($ch);
$headers = curl_getinfo($ch);
// close cURL resource, and free up system resources
curl_close($ch);
} else {
echo "Error: cURL is not installed on the web server. Unable to continue.";
return false;
}
return $headers;
}
print_r(getHeaders('mail.google.com'));
Which yields the following results:
Array
(
[url] => http://mail.google.com
[content_type] => text/html; charset=UTF-8
[http_code] => 404
[header_size] => 338
[request_size] => 55
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.128
[namelookup_time] => 0.042
[connect_time] => 0.095
[pretransfer_time] => 0.097
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => 0
[upload_content_length] => 0
[starttransfer_time] => 0.128
[redirect_time] => 0
)
I've tested it with several long links, and the function acknowledges redirects, all apart from mail.google.com it seems.
For fun, I passed the same URL (mail.google.com) to the W3C link checker, which produced:
Results
Links
Valid links!
List of redirects
The links below are not broken, but the document does not use the exact URL, and the links were redirected. It may be a good idea to link to the final location, for the sake of speed.
warning Line: 1 http://mail.google.com/mail/ redirected to
https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=zpwhtygjntrz&scc=1<mpl=default<mplcache=2
Status: 302 -> 200 OK
This is a temporary redirect. Update the link if you believe it makes sense, or leave it as is.
Anchors
Found 0 anchors.
Checked 1 document in 4.50 seconds.
Which is correct, as the address above is where I am redirected to when I enter mail.google.com into my browser.
What cURL options would I need to use to make my function return 200 for mail.google.com?
Why is it that the function above returns 404 status code as opposed to 302 status code?
TIA
The problem is that the redirect is specified through methods that cURL won't follow.
Here is the response from http://mail.google.com:
HTTP/1.1 200 OK
Cache-Control: public, max-age=604800
Expires: Mon, 22 Jun 2009 14:58:18 GMT
Date: Mon, 15 Jun 2009 14:58:18 GMT
Refresh: 0;URL=http://mail.google.com/mail/
Content-Type: text/html; charset=ISO-8859-1
X-Content-Type-Options: nosniff
Transfer-Encoding: chunked
Server: GFE/1.3
<html>
<head>
<meta http-equiv="Refresh" content="0;URL=http://mail.google.com/mail/" />
</head>
<body>
<script type="text/javascript" language="javascript">
<!--
location.replace("http://mail.google.com/mail/")
-->
</script>
</body>
</html>
As you can see, the page uses both a Refresh header (and HTML meta equivalent) and javascript in the body to change location to http://mail.google.com/mail/.
If you then request http://mail.google.com/mail/, you will be redirected (with the Location header, which cURL follows) to the page you had previously mentioned W3C correctly identifies.
HTTP/1.1 302 Moved Temporarily
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Mon, 15 Jun 2009 15:07:56 GMT
Location: https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=zpwhtygjntrz&scc=1<mpl=default<mplcache=2
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Transfer-Encoding: chunked
Server: GFE/1.3
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Cache-control: no-cache, no-store
Pragma: no-cache
Expires: Mon, 01-Jan-1990 00:00:00 GMT
Set-Cookie: GALX=B8zH60M78Ys;Path=/accounts;Secure
Date: Mon, 15 Jun 2009 15:07:56 GMT
X-Content-Type-Options: nosniff
Content-Length: 19939
Server: GFE/2.0
(HTML page content here, removed)
Perhaps you should add an additional step in your script to check for a Refresh header.
Another possible error is that you have open_basedir set in your PHP configuration, which would disable CURLOPT_FOLLOWLOCATION - you can check this quickly by turning on error reporting, as a message is generated as either a warning or notice.
The results above were all obtained with the following cURL setup:
$useragent="Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$res = curl_exec($ch);
curl_close($ch);
Could it be that
mail.google.com -> mail.google.com/mail is a 404 and then a hard redirect
and
mail.google.com/mail -> https://www.google.com/accounts... etc is a 302 redirect