PHP - How to send request to web sites? - php

I am using curl, I am wondering how would I send post/submit data on my page to those websites? The web site has "host, time, port". My MYSQL database has a list of urls. I was thinking of curl_multi but I am not sure.
Please someone post examples. It has to be a fast method.
Basically feteches the url and post.
while($resultSet = mysql_fetch_array($SQL)){
$ch = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
}

The PHP cURL reference says that the CURLOPT_POST option, set to true, makes it a POST request. CURLOPT_POSTFIELDS sets the fields that you will send in foo=bar&spam=eggs format (which one can build from an array with http_build_query).
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'foo=bar&spam=eggs');

Here is an example on how to do it with curl_multi. Although you should break it up so you only have a certain amount of URLs going out at once (i.e. 30). I added the follow location directive, which you usually want.
$mh = curl_multi_init();
$ch = array();
while($resultSet = mysql_fetch_array($SQL)){
$ch[$i] = curl_init($resultSet['url'] . $fullcurl);
curl_setopt($ch[$i], CURLOPT_TIMEOUT, 2);
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch[$i], CURLOPT_FOLLOWLOCATION, true);
curl_multi_add_handle($mh, $ch[$i]);
}
$running = null;
do {
curl_multi_exec($mh,$running);
} while ($running > 0);
$num = count($ch);
for ($i=0; $i<$num; $i++ ) {
curl_multi_remove_handle($mh, $ch[$i]);
}
curl_multi_close($mh);

Give this a shot:
while ($resultSet = mysql_fetch_assoc($SQL)) {
$ch = curl_init($resultSet['url']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fullcurl);
$response = curl_exec($ch);
curl_close();
}

Related

What on my PHP Curl call do I need to add to read a response's custom headers?

I tried a few several ways to read the responses custom header but have not been able to. I know the response I get is served by nginx and the custom header names start with X-......
$endpoint = 'url here';
$ch = curl_init( $endpoint );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'cbFunc');
$result = curl_exec($ch);
print_r( curl_getinfo($ch ) );
The PHP manual is an excellent reference guide an a good starting point when you run into problems like this.
CURLOPT_HEADERFUNCTION [Set value to] A callback accepting five parameters.
hence
log_headers('init');
$endpoint = 'url here';
$ch = curl_init( $endpoint );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'log_headers');
$result = curl_exec($ch);
$headers=log_headers();
print_r($headers);
function log_headers($ch=false, $headers=false)
{
static $hdrs;
if (is_array($hrs) && $ch===$headers===false) {
return $hdrs[];
} elseif ($ch==='init') {
$hdrs=array();
return 0;
}
$hdrs[]=$headers;
return strlen($headers);
}

How to use curl and an array of URLs together

I found this code that does what I'm looking for on a single domain check. But I have a list of URLs that I would like to check.
Can you and how would you create an array with curl?
$url = 'http://www.example.com';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true); // we want headers
curl_setopt($ch, CURLOPT_NOBODY, true); // we don't need body
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
$output = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
echo 'HTTP code: ' . $httpcode;
If all you want are headers, then instead of curl try:
$urls = array('http://www.example.com','http://www.example.com2','http://www.example.com3','http://www.example4.com');
$httpCodes = [];
foreach($urls as $url)
{
$httpCodes[$url] = get_headers($url)[0];
}
var_dump($httpCodes);
This uses get_headers() to get the header data from a URL's response.
Edit: Or if you really want to use curl. you can use the following code with curl_multi instead:
$urls = array('http://www.example.com','http://www.example.com2','http://www.example.com3','http://www.example4.com');
$curlMultiReq = curl_multi_init();
$ch = [];
foreach($urls as $key => $url)
{
$ch[$key] = curl_init($url);
curl_setopt($ch[$key], CURLOPT_HEADER, true); // we want headers
curl_setopt($ch[$key], CURLOPT_NOBODY, true); // we don't need body
curl_setopt($ch[$key], CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch[$key], CURLOPT_TIMEOUT,10);
curl_multi_add_handle($curlMultiReq, $ch[$key]);
}
$running = '';
do {
curl_multi_exec($curlMultiReq, $running);
curl_multi_select($curlMultiReq);
} while ($running > 0);
foreach(array_keys($ch) as $key){
echo curl_getinfo($ch[$key], CURLINFO_HTTP_CODE);
echo "\n";
curl_multi_remove_handle($curlMultiReq, $ch[$key]);
}
curl_multi_close($curlMultiReq);

Reduce the load php cURL puts on the server

I'm currently using php URL to browse over 500 web pages a day with cookies.
I have to check each page to ensure that the account is still logged in and the pages are being viewed as a member, not a guest.
The script takes an hour or two to complete as it sleeps in between views.
I just want to know if there's anything I can do to reduce the load this script puts on the local server, I've made sure to clear variables at the end of each loop but is there anything I'm missing that would help?
Any new cURL settings that would help?
$i = 0;
$useragents = array();
foreach($urls as $url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, str_replace('\\','/',dirname(__FILE__)).'/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, str_replace('\\','/',dirname(__FILE__)).'/cookies.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $useragents[array_rand($useragents)]);
$html = curl_exec($ch);
curl_close($ch);
if(!$html)
die("No HTML - Not logged in");
if($i %10 != 0)
sleep(rand(5,20));
else
sleep(rand(rand(60,180), rand(300,660)));
$i++;
$html = '';
}
You could reuse your curl handle instead of creating a new one for each connection.
Clearing $html at the end of each iteration won't reduce memory usage and just adds an extra operation because it already gets reset in the next iteration.
$i = 0;
$useragents = array();
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, str_replace('\\','/',dirname(__FILE__)).'/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, str_replace('\\','/',dirname(__FILE__)).'/cookies.txt');
foreach($urls as $url){
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $useragents[array_rand($useragents)]);
$html = curl_exec($ch);
if(!$html)
die("No HTML - Not logged in");
if($i++ % 10 != 0)
sleep(rand(5,20));
else
sleep(rand(rand(60,180), rand(300,660)));
}
curl_close($ch);

How to send multiple URLs simultaneously with PHP?

I wanted to send different url at same time.this is the code which i used.Can you please tell me the error in here?
<?php
$data = R::find('savedata','list_name = ? order by id desc',array($i));
foreach ( $data as $list):
$mh = curl_multi_init(); //set up a cURL multiple execution handle
$ch = curl_init("https://example.com/save_data.php?NUM=$list->id&MSG=$message");
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_multi_add_handle($mh, $ch);
endforeach;
curl_multi_exec($mh,);
curl_multi_close($mh)
?>
Thank you all.I have corrected this by my self.I am sharing the correct code with you,
$mobile = R::find('add_numbers', 'list_name = ? order by id desc', array($i));
foreach ($mobile as $list):
try{
$url = "https://example.com/save_data.php?NUM=$list->id&MSG=****";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // TRUE
curl_setopt($ch, CURLOPT_HEADER, 0); // DO NOT RETURN HTTP HEADERS
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // RETURN THE CONTENTS OF THE CALL
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$results = curl_exec($ch);
$i[]=$results;
}

php CURL not working with dynamic URLs

I can't work out why this URL is not being found by CURL. The CURL engine is simply taken to a 400 error page.
My code is very simple and works fantastically with non-dynamic URLs.
I am hoping it's something easy to spot, for example, a missing CURL option.
I have tried using $url = urlencode($url) but that didn't work either.
Here's the code:
$url = 'http://www.destinations-uk.com/accommodations.php?link=accommodations&country=england&category=Reviews&id=1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
$r = curl_exec($ch);
$r = explode("\n", $r);
$keys = array();
if(!empty($r)) $keys[] = array_shift($r);
foreach($r as $line){
preg_match('/.+:\s/',$line,$match);
if($match) $keys[substr($match[0],0,-2)] = preg_replace('/.+:\s/','', $line);
}
print_r($keys);
Perhaps, this is something on the server-side done to prevent automated requests.

Categories