When I execute the following code it takes between 10-12 seconds to respond.
Is the problem with Twitter or with our server?
I really need to know as this is part of the code to display tweets on our website and a 12 second load time is just not acceptable!
function get_latest_tweets($username)
{
print "<font color=red>**". time()."**</font><br>";
$path = 'http://api.twitter.com/1/statuses/user_timeline/' . $username.'.json?include_rts=true&count=2';
$jason = file_get_contents($path);
print "<font color=red>**". time()."**</font><br>";
}
Thanks
When you put the URL into your browser (http://api.twitter.com/1/statuses/user_timeline/username.json?include_rts=true&count=2) how long does it take for the page to appear? If it's quick then you need to start the search at your server.
use curl instead of file_get_contents() to request, so that response will be compressed. Here is the curl function which iam using.
function curl_file_get_contents($url)
{
$curl = curl_init();
curl_setopt($curl,CURLOPT_URL,$url); //The URL to fetch. This can also be set when initializing a session with curl_init().
curl_setopt($curl,CURLOPT_RETURNTRANSFER,TRUE); //TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
curl_setopt($curl,CURLOPT_ENCODING , "gzip");
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE); //To fail silently if the HTTP code returned is greater than or equal to 400.
curl_setopt($curl,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
$contents = curl_exec($curl);
curl_close($curl);
return $contents;
}
Related
I'm new to cURL & I'm trying to retrieve data from a dynamic URL. I've successfully retrieved data from 1 page only, but what I want is retrieving data from all pages. The problem here is that pages are variable; I don't know how many time the code should loop to loop over all pages. i.e. the number of pages varies from one case to another, and a good program is the one that works in as many cases as possible. Consequently, putting the links in an array and looping over them isn't the right solution for this.
Here's a quick explanation of the URL I'm trying to retrieve data from:
https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner
&advertiser-ids=1513033&records-per-page=100&page-number=' . $num
Did you notice the last variable $num? That should represent the number of the page from where data will be retrieved from. In some cases, it could be only 1, and in some other cases, it could be 10, 12 or 15 (it varies) depending on the parameters I choose, whether I want to see everything or filter some information.
Now here is the problem. How to dynamically increment that number as long as the request returns data? and if not, cURL should stop running?
Here's the code:
<?php
$num = 1;
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "XGET");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($curl, CURLOPT_ENCODING, "UTF-8");
curl_setopt($curl, CURLOPT_URL, 'https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner&advertiser-ids=1513033&records-per-page=100&page-number=' . $num);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_COOKIE,true);
$result = curl_exec($curl);
$xml = simplexml_load_string($result) or die("Error: Cannot create object");
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
}
curl_close($curl);
?>
You can define your curl request inside the function and call that function like this:
<?php
$GLOBALS['num'] = 1;
function curlRequest()
{
// HERE DEFINE YOUR CURL REQUEST
// https://yourUrl.com?$GLOBALS['num']
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
// Exit from function in case there is no output
return;
} else {
$GLOBALS['num']++;
// Call the function to fetch NEXT page
curlRequest();
}
}
// Call the function for first time
curlRequest();
Assuming that curl will fire error if page number not exist or you can implement if condition from the result you are getting. Hope you will get some idea from this.
The main part here is if & else. You should implement if condition on the output you are getting and if page not exist then output will be different so you can simply return from the function in this case.
I'm trying to write a simple curl function that queries the freegeoip.net site with the IP address of a site visitor. This is usually done by typing "https://freegeoip.net/csv/{IP Address}" in the browser address line. The site then processes the request and returns a csv file that can be opened or saved. I'm trying to access the csv data directly so that I can parse and use it. This is the code that I am using:
<?php
$ip=$_SERVER["REMOTE_ADDR"];
$geturl = "http://freegeoip.net/csv/".$ip;
$data = curl_get_contents($geturl);
echo ("<br>Data = '".$data."'<br>");
function curl_get_contents($url)
{
$ch = curl_init($url);
if($ch)
{
$tmp = curl_exec($ch);
curl_close($ch);
return $tmp;
}
else
{
echo "Curl not loaded!<br>";
}
}
?>
This is what I am getting back:
...,US,United States,ST,State,City,?????,America/New_York,.*****,-.****,***
Data = '1'
As you can see, my function is accessing and showing the csv data but not returning it to the $data variable. Apparently, the data is being shown when the "curl_exec($ch);" command is being executed. I want to parse and use the returned data but can't until the data is returned. What am I doing wrong?
The documentation of curl_exec() says:
Return Values
Returns TRUE on success or FALSE on failure. However, if the CURLOPT_RETURNTRANSFER option is set, it will return the result on success, FALSE on failure.
What it doesn't say is explained in the documentation page of curl_setopt(), on the CURLOPT_RETURNTRANSFER option:
Option: CURLOPT_RETURNTRANSFER
Set value to: TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
That is, by default, curl_exec() outputs the body of the response it gets. In order to make it return the value and not output it, you have to use curl_setopt():
$ch = curl_init($url);
curl_exec($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
you need to add following line before curl_exec other wise the result will output instead of returning it to $tmp variable.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
You aren't telling Curl that you want the data to be returned rather than output:
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, True);
curl_setopt($ch, CURLOPT_URL, "http://freegeoip.net/csv/".$ip);
$csv=curl_exec($ch);
But this is rather verbose when, depending on your config, you can:
$csv=file_get_contents("http://freegeoip.net/csv/".$ip);
I have a PHP Code, which reloads itself with another GET value. Like: example.com?number=453 and it keeps doing this for days. I was doing this in the browser. But i found cron job is way better.
So, I need to use CURL to reload the page with a new GET value like ?number=550. So this is the code, which i use (found it on stackoverflow)
function curl_download($Url){
// is cURL installed yet?
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
// OK cool - then let's create a new cURL resource handle
$ch = curl_init();
// Now set some options (most are optional)
// Set URL to download
curl_setopt($ch, CURLOPT_URL, $Url);
// Set a referer
curl_setopt($ch, CURLOPT_REFERER, "http://www.example.org/yay.htm");
// User agent
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
// Include header in result? (0 = yes, 1 = no)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Should cURL return or print out the data? (true = return, false = print)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Timeout in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
// Download the given URL, and return output
$output = curl_exec($ch);
// Close the cURL resource, and free system resources
curl_close($ch);
return $output;
}
$output = curl_download("http://www.example.com/yourscript.php?number='$requestsDone'");
And at the end of the page, i use this, to call the function
curl_download($Url);
But i am getting this error:
Notice: Undefined variable: Url
in the last line, i.e curl_download($Url);
This is how we call the function right? What's wrong? Also is there any mistake or improvement, I can make in the code?
You not assign values to variables $Url.
Before you can assign values to variables $Url. That like $Url = "http://domaintest.com/?number=550";
curl_download($Url);
Below code snippet is to get the final URL(which has media/zip/rar file) from redirect URL by using cURL. It gets the final URL, no doubt about it, but what it does is according to the size of file it varies in time to get URL.
Suppose file at final URL is 1MB, it will take around 5sec to retrieve. But if the file is about 35MB, it takes time about 150 sec. I think cURL is downloading result and finally fetching the URL from result.
<?php
echo get_rurl("x_url");//1.2MB -> 5-10sec
//echo get_rurl("y_url");//31.6MB -> 150sec
function get_rurl($url){
// initialize cURL
$curl = curl_init($url);
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
));
// execute the request
$result = curl_exec($curl);
// fail if the request was not successful
if ($result === false) {
curl_close($curl);
return null;
}
// extract the target url
$redirectUrl = curl_getinfo($curl, CURLINFO_EFFECTIVE_URL);
curl_close($curl);
return $redirectUrl;
}
?>
i cant use file_get_content() because i just want to get the final URL from given redirect URL.
So in short - how to get the final URL from redirect URL without downloading results.
Hope i make it clear. Any help will be appreciated.
This works fine with CURLINFO_EFFECTIVE_URL, but for it the option CURLOPT_FOLLOWLOCATION must set to TRUE. This is on the grounds that CURLINFO_EFFECTIVE_URL returns precisely what it says, the effective url that ends up getting loaded. If the CURLOPT_FOLLOWLOCATION=False then the effective url will be requested url, else it will be final url that is redirected to.
I did this using curl_getinfo. which gives me information regarding the last transfer
<?php
echo get_rurl("xurl");
//echo get_rurl("yurl");
function get_rurl($url){
// initialize cURL
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); //specify your URL
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); //disable follow redirects
$http_data = curl_exec($ch); //hit the $url
$curl_info = curl_getinfo($ch);
return $curl_info['redirect_url'];// extract final url
}
?>
or
Even you can use CURLINFO_REDIRECT_URL or CURLINFO_EFFECTIVE_URL depending upon your use cases. refer here
<?php
echo get_rurl("xurl");
//echo get_rurl("yurl");
function get_rurl($url){
// initialize cURL
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); //specify your URL
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); //disable follow redirects
$http_data = curl_exec($ch); //hit the $url
return curl_getinfo($ch, CURLINFO_REDIRECT_URL);
}
?>
Hope this helps to others users too.
According to the documentation of libcurl (https://curl.haxx.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html), this is exactly as is expected when using CURLOPT_FOLLOWLOCATION => true,. You probably want to change this to false.
I am usig curl_setopt to eval another file from a different server, the link is something like this "http://somewhere.com/index.php?vars=hello"
now in somewhere.com/index.php i need to get the value of vars that was passed using curl, but so far i cant get any values at all.
here is my sample code for your reference this is from the file calling somewhere.com:
$d = "http://somewhere.com/index.php?vars=hello";
$ch1 = curl_init();
curl_setopt($ch1, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch1, CURLOPT_URL, $d);
curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, TRUE);
$data1 = curl_exec($ch1);
eval($data1);
curl_close($ch1);
in somewhere.com/index.php i already did print_r($_GET); to view any passed values to the file but it returned nothing.
What is happening in the index.php file with the variable "hello"? When you do a curl post to another page that other page typically will echo out a response and that response is what you evaluate. Can you post your code from index.php so we can see what you are trying to do?
Edit: Also you can use chrome inspector, fiddler or some other network monitor to see if the http request actually is being fired off and to check that you are actually getting a 200 response back.
Edit: I don't know what you are using eval either, just echo the response. If you have an output buffer issue then start buffering before you echo the response like:
ob_start():
echo $data1;
ob_end_clean();
Also if you want to see if you are getting any errors just do this:
if(curl_exec($ch1) === false)
{
echo 'Curl error: ' . curl_error($ch1);
}
else
{
echo $data1;
}
// Close handle
curl_close($ch1);
can't you just use $_GET[]?
i.e.
$variables = $_GET['vars'];