I was thinking of doing a head request with cURL, was wondering if this is the way to go?
The best solution which follows the KISS principle
$head = array_change_key_case(get_headers("http://example.com/file.ext", 1));
$filesize = $head['content-length'];
I'm guessing using curl to send a HEAD request is a nice possibility ; something like this would probably do :
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://sstatic.net/so/img/logo.png');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_exec($ch);
$size = curl_getinfo($ch, CURLINFO_CONTENT_LENGTH_DOWNLOAD);
var_dump($size);
And will get you :
float 3438
This way, you are using a HEAD request, and not downloading the whole file -- still, you depend on the remote server send a correct Content-length header.
Another option you might think about would be to use filesize... But this will fail : the documentation states (quoting) :
As of PHP 5.0.0, this function can
also be used with some URL wrappers.
Refer to List of Supported
Protocols/Wrappers for a listing
of which wrappers support stat()
family of functionality.
And, unfortunately, with HTTP and HTTPS wrappers, stat() is not supported...
If you try, you'll get an error, like this :
Warning: filesize() [function.filesize]: stat failed
for http://sstatic.net/so/img/logo.png
Too bad :-(
Yes. Since the file is remote, you're completely dependent on the value of the Content-Length header (unless you want to download the whole file). You'll want to curl_setopt($ch, CURLOPT_NOBODY, true) and curl_setopt($ch, CURLOPT_HEADER, true).
Using a HEAD request and checking for Content-Length is the standard way to do it, but you can't rely on it in general, since the server might not support it. The Content-Length header is optional, and further the server might not even implement the HEAD method. If you know which server you're probing, then you can test if it works, but as a general solution it isn't bullet proof.
If you don't need a bulletproof solution you can just do:
strlen(file_get_contents($url));
Related
I am just wondering if there is a way to check if an encrypted image exist over https.
I am using this code below to check whether an image exist. It will return 200 if it does, and 404 if it doesn't. But this only work on image URL that does not contain https.
function exists_url($uri){
$ch = curl_init($uri);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_exec($ch);
return $code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
}
If I try to verify this url ( https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcTup5KSMveqkgrDKZR6p-0ANhPkJ7srbJOlKR78DUqqh85I_3MUrw ) it gives me a result of 0 instead of 200 or 404.
I have thought to use getimagesize() function to accomplish this task. But even this function only works for image that is not send over HTTPS.
Since you're using SSL/TLS you must point cURL to a valid CA Certificate bundle. The bundle must be in .pem format. You can obtain such a bundle from http://curl.haxx.se/ca/.
Next you need to tell cURL to use your CA bundle. You can do this with the CURLOPT_CAINFO option.
curl_setopt($ch, CURLOPT_CAINFO, '/path_to_file/cacert.pem');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
You could simply deactivate the verify of peer with
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
So you have not to worry about nothing.
Watch out: #user555 answer is better for security purposes
A response code of 0 from Curl typically means there was an error connecting to the server or something along those lines (See PHP cURL HTTP CODE return 0 and countless other questions).
I did however run this code myself using PHP 5.5.x and it returned a 200 as expected. getimagesize() not working also leads me to think it's most likely a problem with curl or php on your system, what versions of them do you have?
For some reason my curl call is very slow. Here is the code I used.
$postData = "test"
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$result = curl_exec($ch);
Executing this code takes on average 250ms to finish.
However when I just open the url in a browser, firebug says it only takes about 80ms.
Is there something I am doing wrong? Or is this the overhead associated with PHP Curl.
It's the call to
curl_exec
That is taking up all the time.
UPDATE:
So I figured out right after I posted this that if I set the curl option
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
It significantly slows down
curl_exec
The post data could be anything and it will slow it down.
Even if I set
curl_setopt($ch, CURLOPT_POST, false);
It's slow.
I'll try to work around it by just adding the parameters to the URI as a query string.
SECOND UPDATE:
Confirmed that if I just call the URI using GET and passing parameters
as a query string it is much faster than using POST and putting the parameters in the body.
CURL has some problems with DNS look-ups. Try using IP address instead of domain name.
Curl has the ability to tell exactly how long each piece took and where the slowness is (name lookup, connect, transfer time). Use curl_getinfo (http://www.php.net/manual/en/function.curl-getinfo.php) after you run curl_exec.
If curl is slow, it is generally not the PHP code, it's almost always network related.
try this
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
Adding "curl_setopt($ch, CURLOPT_POSTREDIR, CURL_REDIR_POST_ALL);" solved here. Any problem with this solution?
I just resolved this exact problem by removing the following two options:
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
Somehow on the site I was fetching, the POST request to over ten full seconds. If it's GET, it's less than a second.
So... in my wrapper function that does the Curl requests, it now only sets those two options when there is something in $postData
I just experienced a massive speed-up through compression. By adding the Accept-Encoding header to "gzip, deflate", or just to all formats which Curl supports, my ~200MB download took 6s instead of 20s:
curl_setopt($ch, CURLOPT_ENCODING, '');
Notes:
If an empty string, "", is set, a header containing all supported encoding types is sent.
you do not even have to care about decompression after the download, as this is done by Curl internally.
CURLOPT_ENCODING requires Curl 7.10+
The curl functions in php directly use the curl command line tool under *nix systems.
Therefore it really only depends on the network speed since in general curl itself is much faster than a webbrowser since it (by default) does not load any additional data like included pictures, stylesheets etc. of a website.
It might be possible that you are not aware, that the network performance of the server on which you were testing your php script is way worse than on your local computer where you were testing with the browser. Therefore both measurements are not really comparable.
generally thats acceptable when you are loading contents or posting to slower end of world. curl call are directly proportional to your network speed and throughput of your webserver
I am trying to use an rss feed from a domain that does not have a crossdomain file and because of that I am going to use a web service in the middle where I will be just getting the rss feed from a url (let's say the url is: www.example.com/feed) and then just print it to a page.
The service would work like: www.mywebservice.com/feed.php?word=something) and that will just go print the rss feed for: www.example.com/feed&q=word).
I used:
<?php
$word = $_GET["word"];
$ch=curl_init("http://example.com/feed.php?word=".$word."");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
print $data;
?>
But this did not work, it gives me (SYSTEM ERROR: we're sorry but a serious error has occurred in the system). I am on shared hosting
Any help?
The readfile function reads a file and writes it to the output buffer.
readfile('http://example.com/feed.rss');
A URL can be used as a filename with this function if the fopen wrappers have been enabled. See fopen() for more details on how to specify the filename. See the List of Supported Protocols/Wrappers for links to information about what abilities the various wrappers have, notes on their usage, and information on any predefined variables they may provide.
If you need to do anything with the XML, use one of PHP's many XML libraries, preferably DOM, but there is also SimpleXml or XMLReader. As an alternative, you could use Zend_Feed from the Zend Framework as a standalone component to work with the RSS feed.
If you cannot enable allow_url_fopen on your server, try cURL like Matchu suggested or go with Artefacto's suggestion.
Consider doing this with mod_rewrite (using the P flag) or setting up a reverse proxy with ProxyPass.
Since you say you can't do the fancy URL-file-opening shortcuts due to server restrictions, you will need to use PHP's cURL module to send an HTTP request.
If you want to also parse XML and process it further, be sure to look into SimpleXML. Let's you parse and manipulate the feed.
So at the end I ended up doing this:
$word = $_GET["word"];
$url = "http://www.example.com/feed.php?q=".$word;
$curl = #curl_init ($url);
#curl_setopt ($curl, CURLOPT_HEADER, FALSE);
#curl_setopt ($curl, CURLOPT_RETURNTRANSFER, TRUE);
#curl_setopt ($curl, CURLOPT_FOLLOWLOCATION, TRUE);
#curl_setopt ($curl, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
$source = #curl_exec ($curl);
#curl_close ($curl);
print $source;
I hope this is considered as an answer not an edit (if an edit please tell me so I can just delete this answer and edit the post)
There is an epic lack of PHP cURL love on the Internet for beginners like me. I was wondering how to use cURL to download & display an ICS file (They're plain text to me...) in my PHP code. Unless fopen() is 1,000 times easier, I'd like to stick with cURL for this one.
If your webserver allows it, file_get_contents() is even easier.
echo file_get_contents('http://www.example.com/path/to/your/file.ics');
If you can not open URLs with file_get_contents() check out all the stuff on Stack Overflow, which I believe should be fine for a beginner.
If remote file_get_contents is not enabled, cURL can indeed do this.
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://example.com/file.ics');
// this is the key option - sets curl_exec to return the HTTP response
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$file_contents = curl_exec($curl);
What is the best way to check if a given url points to a valid file (i.e. not return a 404/301/etc.)? I've got a script that will load certain .js files on a page, but I need a way to verify each URL it receives points to a valid file.
I'm still poking around the PHP manual to see which file functions (if any) will actually work with remote URLs. I'll edit my post as I find more details, but if anyone has already been down this path feel free to chime in.
The file_get_contents is a bit overshooting the purpose as it is enough to have the HTTP header to make the decision, so you'll need to use curl to do so:
<?php
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
one such way would be to request the url and get a response with a status code of 200 back, aside from that, there's really no good way because the server has the option of handling the request however it likes (including giving you other status codes for files that exist, but you don't have access to for a number of reasons).
If your server doesn't have fopen wrappers enabled (any server with decent security won't), then you'll have to use the CURL functions.