How to use curl and an array of URLs together - php

I found this code that does what I'm looking for on a single domain check. But I have a list of URLs that I would like to check.
Can you and how would you create an array with curl?
$url = 'http://www.example.com';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true); // we want headers
curl_setopt($ch, CURLOPT_NOBODY, true); // we don't need body
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
$output = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
echo 'HTTP code: ' . $httpcode;

If all you want are headers, then instead of curl try:
$urls = array('http://www.example.com','http://www.example.com2','http://www.example.com3','http://www.example4.com');
$httpCodes = [];
foreach($urls as $url)
{
$httpCodes[$url] = get_headers($url)[0];
}
var_dump($httpCodes);
This uses get_headers() to get the header data from a URL's response.
Edit: Or if you really want to use curl. you can use the following code with curl_multi instead:
$urls = array('http://www.example.com','http://www.example.com2','http://www.example.com3','http://www.example4.com');
$curlMultiReq = curl_multi_init();
$ch = [];
foreach($urls as $key => $url)
{
$ch[$key] = curl_init($url);
curl_setopt($ch[$key], CURLOPT_HEADER, true); // we want headers
curl_setopt($ch[$key], CURLOPT_NOBODY, true); // we don't need body
curl_setopt($ch[$key], CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch[$key], CURLOPT_TIMEOUT,10);
curl_multi_add_handle($curlMultiReq, $ch[$key]);
}
$running = '';
do {
curl_multi_exec($curlMultiReq, $running);
curl_multi_select($curlMultiReq);
} while ($running > 0);
foreach(array_keys($ch) as $key){
echo curl_getinfo($ch[$key], CURLINFO_HTTP_CODE);
echo "\n";
curl_multi_remove_handle($curlMultiReq, $ch[$key]);
}
curl_multi_close($curlMultiReq);

Related

php multiple curl urls with while loop

I am developing a small script and I am bit messed up with using a couple of curl and while loop.
I want to stop processing curl at a point when one of the URL is giving me a information. Note: I have multiple curl requests.
So my concept is,
I have a couple of URLS, which I have to process and get information from. If information is found on a particular URL, it will be giving me a string. If no information is found, it will give me no value. So I have nearly 10 URLs to process approximately. In all cases, any one of the URL will be giving me information, so the remaining urls will be producing no value. Since processing there much URLS, latency is a issue. So suppose in the sample code below, if the url ends with value2.php gives me a result, then I immediately wanted to stop processing the other URLs. Because I already got the result and no point in running other curl. Then finally I have to print the result.
Also I have a condition where none of the URL produce any result and it will be great if someone shows me how to handle that also.
My sample code.
<?php
///functions here
do {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value1.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value2.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value3.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value4.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value5.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value6.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value7.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value8.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value9.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value10.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
} while (strlen($combined) != 0);
echo $combied;
///functions here
?>
Try the following:
<?php
function callCURL($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);
return $combined;
}
function getResult($urls) {
$return = array();
foreach ($urls as $url) {
$response = callCURL($url);
if (strlen($response) !== 0) {
$return[] = $response;
break;
}
}
return $return;
}
$urls = array("example.com/value1.php?process=$param", "example.com/value2.php?process=$param", "example.com/value3.php?process=$param")
$result = getResult($urls);
Like you have it in your question and how the other answers have it, the problem is that for each request you have to send it, wait for a response, process the data and only then do you make your subsequent requests. This works fine, but it is highly time-inefficient. Say, for example, each request takes 100 ms to make (which probably not unrealistic). For 10 requests you're looking at 1 second of load time. Instead I would recommend forgetting about trying to stop making requests after you find your result and send all the requests... simultaneously. This can be accomplished with PHP's curl_multi_* functions.
// Put all of your URLs in here. I'm just using google for
// all as an example:
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
// Get cURL handles
foreach ($urls as $key => $url) {
$chs[$key] = curl_init();
// Set all your options for each connection here
curl_setopt($chs[$key], CURLOPT_URL, $url);
curl_setopt($chs[$key], CURLOPT_HEADER, 0);
}
//create the multiple cURL handle
$mh = curl_multi_init();
//add the handles
foreach ($chs as &$ch) {
curl_multi_add_handle($mh,$ch);
}
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
foreach ($chs as $url=>&$ch) {
$html = curl_multi_getcontent($ch);
// [do what you want with the HTML]
curl_multi_remove_handle($mh, $ch); // remove the handle (assuming you are done with it);
}
curl_multi_close($mh);
You can use something like this. I didn't check the code, after debug it should work.
function callcurl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value5.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$ret = curl_exec ($ch);
curl_close ($ch);
}
$urls = array('url1', 'url2'); /// etc
$combined = ''
$cnt = 0;
do {
$combined = callcurl($urls[$cnt++]);
} while (strlen($combined) != 0 && $cnt < count($urls)); //
print $combined;
You can achieve this using a for loop.
$fileNames = array(
'0' => 'valueabc',
'1' => 'valuedef',
[...] => [...]
);
$combined = array();
for ($i = 1; $i < 10; $i++)
{
$ch = curl_init();
$url = "example[dot]com/" . $fileNames[$i] . ".php?process=" . $param;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined[$i] = curl_exec ($ch);
curl_close ($ch);
}
To see all the responses you can foreach $combined.
foreach ($combined as $value=>$response)
{
// TODO: Work with the response
echo "[" . $value . "]" . $response;
}

PHP curl questions

<?php
ini_set('display_errors',1);
$url = 'www.google.com.my';
$header = true;
$returntransfer = true;
$connecttimeout = 3;
$timeout = 60;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, $returntransfer);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $connecttimeout);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$execute = curl_exec($ch);
$info = curl_getinfo($ch);
header('Content-Type: text/plain');
echo $execute;
echo "\n\ncurl_getinfo() said:\n", str_repeat('-', 31 + strlen($url)), "\n";
foreach ($info as $label => $value)
{
printf("%-30s %s\n", $label, $value);
}
echo str_repeat('-', 31 + strlen($url));
?>
Here is my questions:
(1) I want to test the web services using PHP curl. Am I missing something based from the above code?
(2) If I have 2 URLs, should I use curl_setopt or curl_multi_init?
I really hope that someone will answer my questions.
It's a simultaneous requests:
$ch_1 = curl_init('http://url.one.com/');
$ch_2 = curl_init('http://url.two.com/');
curl_setopt($ch_1, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch_2, CURLOPT_RETURNTRANSFER, true);
// build the multi-curl handle, adding both $ch
$mh = curl_multi_init();
curl_multi_add_handle($mh, $ch_1);
curl_multi_add_handle($mh, $ch_2);
// execute all queries simultaneously, and continue when all are complete
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running);
// all of our requests are done, we can now access the results
$response_1 = curl_multi_getcontent($ch_1);
$response_2 = curl_multi_getcontent($ch_2);
echo "$response_1 $response_2"; // same output as first example

How to send multiple URLs simultaneously with PHP?

I wanted to send different url at same time.this is the code which i used.Can you please tell me the error in here?
<?php
$data = R::find('savedata','list_name = ? order by id desc',array($i));
foreach ( $data as $list):
$mh = curl_multi_init(); //set up a cURL multiple execution handle
$ch = curl_init("https://example.com/save_data.php?NUM=$list->id&MSG=$message");
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_multi_add_handle($mh, $ch);
endforeach;
curl_multi_exec($mh,);
curl_multi_close($mh)
?>
Thank you all.I have corrected this by my self.I am sharing the correct code with you,
$mobile = R::find('add_numbers', 'list_name = ? order by id desc', array($i));
foreach ($mobile as $list):
try{
$url = "https://example.com/save_data.php?NUM=$list->id&MSG=****";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // TRUE
curl_setopt($ch, CURLOPT_HEADER, 0); // DO NOT RETURN HTTP HEADERS
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // RETURN THE CONTENTS OF THE CALL
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$results = curl_exec($ch);
$i[]=$results;
}

How to read CURL POST on remote server?

This is my cURL POST function:
public function curlPost($url, $data)
{
$fields = '';
foreach($data as $key => $value) {
$fields .= $key . '=' . $value . '&';
}
rtrim($fields, '&');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, count($data));
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
}
$this->curlPost('remoteServer', array(data));
How do I read the POST on the remote server?
The remote server is using PHP... but what var in $_POST[] should I read
for e.g:- $_POST['fields'] or $_POST['result']
You code works but i'll advice you to add 2 other things
A. CURLOPT_FOLLOWLOCATION because of HTTP 302
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
B. return in case you need to output the result
return $result ;
Example
function curlPost($url, $data) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
return $result;
}
print(curlPost("http://yahoo.com", array()));
Another Example
print(curlPost("http://your_SITE", array("greeting"=>"Hello World")));
To read your post you can use
print($_REQUEST['greeting']);
or
print($_POST['greeting']);
as a normal POST request ... all data posted can be found in $_POST ... except files of course :) add an &action=request1 for example to URL
if ($_GET['action'] == 'request1') {
print_r ($_POST);
}
EDIT: To see the POST vars use the folowing in your POST handler file
if ($_GET['action'] == 'request1') {
ob_start();
print_r($_POST);
$contents = ob_get_contents();
ob_end_clean();
error_log($contents, 3, 'log.txt' );
}

PHP - How to send request to web sites?

I am using curl, I am wondering how would I send post/submit data on my page to those websites? The web site has "host, time, port". My MYSQL database has a list of urls. I was thinking of curl_multi but I am not sure.
Please someone post examples. It has to be a fast method.
Basically feteches the url and post.
while($resultSet = mysql_fetch_array($SQL)){
$ch = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
}
The PHP cURL reference says that the CURLOPT_POST option, set to true, makes it a POST request. CURLOPT_POSTFIELDS sets the fields that you will send in foo=bar&spam=eggs format (which one can build from an array with http_build_query).
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'foo=bar&spam=eggs');
Here is an example on how to do it with curl_multi. Although you should break it up so you only have a certain amount of URLs going out at once (i.e. 30). I added the follow location directive, which you usually want.
$mh = curl_multi_init();
$ch = array();
while($resultSet = mysql_fetch_array($SQL)){
$ch[$i] = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch[$i], CURLOPT_TIMEOUT, 2);
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch[$i], CURLOPT_FOLLOWLOCATION, true);
curl_multi_add_handle($mh, $ch[$i]);
}
$running = null;
do {
curl_multi_exec($mh,$running);
} while ($running > 0);
$num = count($ch);
for ($i=0; $i<$num; $i++ ) {
curl_multi_remove_handle($mh, $ch[$i]);
}
curl_multi_close($mh);
Give this a shot:
while ($resultSet = mysql_fetch_assoc($SQL)) {
$ch = curl_init($resultSet['url']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fullcurl);
$response = curl_exec($ch);
curl_close();
}

Categories