I can't work out why this URL is not being found by CURL. The CURL engine is simply taken to a 400 error page.
My code is very simple and works fantastically with non-dynamic URLs.
I am hoping it's something easy to spot, for example, a missing CURL option.
I have tried using $url = urlencode($url) but that didn't work either.
Here's the code:
$url = 'http://www.destinations-uk.com/accommodations.php?link=accommodations&country=england&category=Reviews&id=1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
$r = curl_exec($ch);
$r = explode("\n", $r);
$keys = array();
if(!empty($r)) $keys[] = array_shift($r);
foreach($r as $line){
preg_match('/.+:\s/',$line,$match);
if($match) $keys[substr($match[0],0,-2)] = preg_replace('/.+:\s/','', $line);
}
print_r($keys);
Perhaps, this is something on the server-side done to prevent automated requests.
Related
My problem is somehow peculiar. I have this bulksms api from my provider:
http://www.estoresms.com/smsapi.php?username=user&password=1234&sender=##sender##&recipient=##recipient##&m
essage=##message##&
then i wrapped it in PHP and passed it in cURL:
$api = "http://www.estoresms.com/smsapi.php?username=".$sms_user."&password=".$sms_pwd."&sender=".$sender_id."&recipient=".$numbers."&message=".$text."&";
function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$send_it = curl_get_contents($api);
ordinarily, it worked fine, but when $recepient (phone numbers) are more than say 300, i get an error:
Request-URI Too Long
The requested URL's length exceeds the capacity limit for this server.
Additionally, a 414 Request-URI Too Long error was encountered while trying to use an ErrorDocument to handle the request.
But BulkSMS should be able to send to thousands of numbers at a time.
From my research, i found out that there's a limit to URL. I'm not the server owner. i working on a shared hosting plan. pls how can i get around this problem. I know there's a solution to it that would not mean buying my own server.
Thanks
Can you try to make the API use POST instead of GET. It would solve the issue.
Edit:
I'm not sure your API check POST, but try that:
$api = "http://www.estoresms.com/smsapi.php";
$data = array('username' => $sms_user, 'password' => $sms_pwd, 'sender' => $sender_id , 'recipient' => $numbers , 'message' => $text);
function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt($handle, CURLOPT_POST, true);
curl_setopt($handle, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$send_it = curl_get_contents($api);
Have a look at this code example (from bulksms.com).
http://developer.bulksms.com/eapi/code-samples/php/send_sms/
So then, i had to find a way around my own problem. if the API will not allow thousands of numbers at a time, then let's break it into chunks at the point of execution.
function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$how_many = count(explode(',', $numbers));
if ($how_many > 250){
$swi = range(0, ceil($how_many/250)-1);
foreach ($swi as $sw){$numbers_a = implode(',', (array_slice(explode(',', $numbers), $sw*250, 250)));
$api = "http://www.estoresms.com/smsapi.php?username=".$sms_user."&password=".$sms_pwd."&sender=".$sender_id."&recipient=".$numbers_a."&message=".$text."&";
$send_it = curl_get_contents($api);
}
}
if ($how_many <= 250){
$api = "http://www.estoresms.com/smsapi.php?username=".$sms_user."&password=".$sms_pwd."&sender=".$sender_id."&recipient=".$numbers."&message=".$text."&";
$send_it = curl_get_contents($api);
}
I have a task: get by inputed keyword Wikipedia article, save it to database and then make a search inside them.
The problem is: how to access api and retrieve data from wikipedia, I've tried this url (at the begining i've tried json format):
$url = 'https://en.wikipedia.org/w/api.php?action=query&titles=Dog&prop=revisions&rvprop=content&format=xml';
and this php code:
$ch=curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
$res = curl_exec($ch);
if (!$res) {
echo 'cURL Error: '.curl_error($ch);
}
var_dump($res);
but nothing happend. Is it possible to access data with curl?
At the end one code worked with url above:
ini_set('user_agent','TestText');
$xmlDoc = new \DOMDocument();
$xmlDoc->load($url);
echo($xmlDoc->saveXML());
and then I get the text like this
{{about|the domestic dog|related species known as "dogs"|Canidae|other
uses|Dog (disambiguation)|}} {{Redirect|Doggie|the Danish
artist|Doggie (artist)}} {{pp-semi-indef}} {{pp-move-indef}} {{Taxobox
| name = Domestic dog | fossil_range = {{Fossil
range|0.033|0}}[[Pleistocene]] – [[Recent]] |
How can I handle it to be prettier (text with paragraphes or at liest plain text)?
So, There are two questions:
1. Is it possible to access wiki data with php curl and how I should improve my code?
2. How do I make wiki xml code prettier?
My question about code, especially about curl. Why it doesn't work?
And also, answer to another question says only about wikipedia api urls. By only changing url I can't solve problem.
I've found the solution, CURLOPT_SSL_VERIFYPEER was needed:
$url = 'http://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&explaintext=&titles=Dog';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
$res = curl_exec($ch);
//$json_data = mb_substr($res, curl_getinfo($ch, CURLINFO_HEADER_SIZE));
curl_close($ch);
$json = json_decode($res);
$content = $json->query->pages;
$wiki_id = '';
foreach ($content as $key => $value) {
$wiki_id = $key;
}
echo $content = $content->$wiki_id->extract;
I have the following bit of PHP, it works locally (via apache and localhost) but not on my hosting - $response is always empty:
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$api_key = 'my_api_key';
$randomString = generateRandomString(10);
$endLabel = sha1(md5($randomString));
$user_id = $endLabel;
$amount_doge = '5';
$url = "https://dogeapi.com/wow/?api_key=".$api_key."&a=get_new_address&address_label=".$user_id;
$response = get_data($url);
I wondered if this could be because I'm hosted on HTTP (no SSL option) and I'm calling a HTTPS domain? If so, is there a way around this? I've tried curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); but it doesn't seem to do anything :(
try to use echo curl_error($ch) after $data = curl_exec($ch); to see what curl says
it wil lreport you what happened
I have a Affiliate URL Like http://track.abc.com/?affid=1234
open this link will go to http://www.abc.com
now i want to execute the http://track.abc.com/?affid=1234 Using CURL
and now how i can Get http://www.abc.com
with Curl ?
If you want cURL to follow redirect headers from the responses it receives, you need to set that option with:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
You may also want to limit the number of redirects it follows using:
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
So you'd using something similar to this:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://track.abc.com/?affid=1234");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$data = curl_exec($ch);
Edit: Question wasn't exactly clear but from the comment below, if you want to get the redirect location, you need to get the headers from cURL and parse them for the Location header:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://track.abc.com/?affid=1234");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
This will give you the headers returned by the server in $data, simply parse through them to get the location header and you'll get your result. This question shows you how to do that.
I wrote a function that will extract any header from a cURL header response.
function getHeader($headerString, $key) {
preg_match('#\s\b' . $key . '\b:\s.*\s#', $headerString, $header);
return substr($header[0], strlen($key) + 3, -2);
}
In this case, you're looking for the value of the header Location. I tested the function by retrieving headers from a TinyURL, that redirects to http://google.se, using cURL.
$url = "http://tinyurl.com/dtrkv";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
$location = getHeader($data, 'Location');
var_dump($location);
Output from the var_dump.
string(16) "http://google.se"
I am using curl, I am wondering how would I send post/submit data on my page to those websites? The web site has "host, time, port". My MYSQL database has a list of urls. I was thinking of curl_multi but I am not sure.
Please someone post examples. It has to be a fast method.
Basically feteches the url and post.
while($resultSet = mysql_fetch_array($SQL)){
$ch = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
}
The PHP cURL reference says that the CURLOPT_POST option, set to true, makes it a POST request. CURLOPT_POSTFIELDS sets the fields that you will send in foo=bar&spam=eggs format (which one can build from an array with http_build_query).
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'foo=bar&spam=eggs');
Here is an example on how to do it with curl_multi. Although you should break it up so you only have a certain amount of URLs going out at once (i.e. 30). I added the follow location directive, which you usually want.
$mh = curl_multi_init();
$ch = array();
while($resultSet = mysql_fetch_array($SQL)){
$ch[$i] = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch[$i], CURLOPT_TIMEOUT, 2);
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch[$i], CURLOPT_FOLLOWLOCATION, true);
curl_multi_add_handle($mh, $ch[$i]);
}
$running = null;
do {
curl_multi_exec($mh,$running);
} while ($running > 0);
$num = count($ch);
for ($i=0; $i<$num; $i++ ) {
curl_multi_remove_handle($mh, $ch[$i]);
}
curl_multi_close($mh);
Give this a shot:
while ($resultSet = mysql_fetch_assoc($SQL)) {
$ch = curl_init($resultSet['url']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fullcurl);
$response = curl_exec($ch);
curl_close();
}