Need to ping 1000 urls every 2 minutes - php

Question

I have 1000 feed urls sitting in a MySQL database table. I need to do a http request to all these urls every 2 minutes. I wrote a php script to do that, but the script takes 5min 30sec to run.
I want to be able to finish all the 1000 requests in under a minute. Is there a way to run multiple async processes to get the job done faster? Any help is appreciated. Thanks in advance.

Answer 1

Since your question is about sending http requests, not really ping, you can use Grequests (Requests+gevent) to do it easily and fast (in my experience seconds for a couple hundred url requests):
import grequests
urls = [
'http://www.python.org',
'http://python-requests.org',
'http://www.google.com',
]
rs = (grequests.get(u) for u in urls)
grequests.map(rs) # [<Response [200]>, <Response [200]>, <Response [200]>]
Your Php script takes 5 mins to run because it is synchronous code, which means that for every request you sent, you have to wait for response to arrive before moving onto sending the next request.
The trick here is not to wait (or block as many would call) for responses but go straight to make the next request, and you can achieve it easily with gevent(coroutine-based) or nodejs. You can read more on it here.

Answer 2

Have a look at the AnyEvent::Ping or AnyEvent::FastPing modules on CPAN.
Below is straightforward example of using AnyEvent::Ping to ping 10000 urls:
use strict;
use warnings;
use AnyEvent;
use AnyEvent::Ping;
my $cv = AnyEvent->condvar;
my $ping = AnyEvent::Ping->new;
my #results;
for my $url (get_ten_thousand_urls()) {
$cv->begin;
# ping each URL just once
$ping->ping( $url, 1, sub {
# [ url address, ping status ]
push #results, [ $url, $_[0]->[0][0] ];
$cv->end;
});
}
$cv->recv;
# now do something with #results
Some quick tests of above using 10,000 random URLs all took just over 7 seconds to run on my Macbook Air. With tweaking and/or using faster event loop then this time will drop further (above used default pure Perl event loop).
NB. AnyEvent is an abstraction library which will allow you to use the async event system provided by (or installed on) your system. If you want to use a specific event loop then remember to install the relevant Perl module from CPAN, for e.g. EV if using libev. AnyEvent will default to a pure Perl event loop if nothing else is found (installed).
BTW, If you just need to check an HTTP request (ie. not ping) then simply replace AnyEvent::Ping part with AnyEvent::HTTP.

Answer 3

You tagged this with "python", so I'll assume that using Python is an option here. Look at the multiprocessing module. For example:
#!/usr/bin/env python
import multiprocessing
import os
import requests
import subprocess
addresses = ['1.2.3.4', '1.2.3.5', '4.2.2.1', '8.8.8.8']
null = open(os.devnull, 'w')
def fbstatus(count):
"""Returns the address, and True if the ping returned in under 5 seconds or
else False"""
return (count,
requests.get('http://www.facebook.com/status.php').status_code)
def ping(address):
"""Returns the address, and True if the ping returned in under 5 seconds or
else False"""
return address, not subprocess.call(['ping', '-c1', '-W5', address],
stdout=null)
pool = multiprocessing.Pool(15)
if False:
print pool.map(ping, addresses)
else:
pool.map(fbstatus, range(1000))
New - Fetching pages
The fbstatus() function fetches a page from Facebook. This scaled almost linearly with the size of the pool up through 30 concurrent processes. It averaged a total runtime of about 80 seconds on my laptop. At 30 workers, it took a total of about 3.75 wall clock seconds to finish.
Old - Pinging
This uses the subprocess module to call the ping command with a 5 second timeout and a count of 1. It uses the return value of ping (0 for success, 1 for failure) and negates it to get False for failure and True for success. The ping() function returns the address it was called with plus that boolean result.
The last bit creates a multiprocessing pool with 5 child processes, then calls ping() on each of the values in addresses. Since ping() returns its address, it's really easy to see the result of pinging each of those addresses.
Running it, I get this output:
[('1.2.3.4', False), ('1.2.3.5', False), ('4.2.2.1', True), ('8.8.8.8', True)]
That run took 5.039 seconds of wallclock time and 0% CPU. In other words, it spent almost 100% of its time waiting for ping to return. In your script, you'd want to use something like Requests to fetch your feed URLs (and not the literal ping command that I was using as an example), but the basic structure could be nearly identical.

Answer 4

You could try multithreading ping on python.
Here is good example.
#!/usr/bin/env python2.5
from threading import Thread
import subprocess
from Queue import Queue
num_threads = 4
queue = Queue()
ips = ["10.0.1.1", "10.0.1.3", "10.0.1.11", "10.0.1.51"]
#wraps system ping command
def pinger(i, q):
"""Pings subnet"""
while True:
ip = q.get()
print "Thread %s: Pinging %s" % (i, ip)
ret = subprocess.call("ping -c 1 %s" % ip,
shell=True,
stdout=open('/dev/null', 'w'),
stderr=subprocess.STDOUT)
if ret == 0:
print "%s: is alive" % ip
else:
print "%s: did not respond" % ip
q.task_done()
#Spawn thread pool
for i in range(num_threads):
worker = Thread(target=pinger, args=(i, queue))
worker.setDaemon(True)
worker.start()
#Place work in queue
for ip in ips:
queue.put(ip)
#Wait until worker threads are done to exit
queue.join()

Answer 5

I used Perl's POE Ping Component module for this task quite extensively.

Answer 6

[Update: Re-tested this with maxSockets = 100 and while connected to a very good network connection. The script finished in < 1 second, meaning the biggest factor is probably network thruput / latency, as previously noted. Your results will almost certainly vary. ;) ]
You can use node.js for this, as it's API for doing HTTP is powerful, clean, and simple. E.g. The following script fetches ~1000 requests in 10 seconds less than one second on my MacBook Pro:
test.js
var http = require('http');
// # of simultaneouse requests allowed
http.globalAgent.maxSockets = 100;
var n = 0;
var start = Date.now();
function getOne(url) {
var id = n++;
var req = http.get(url, function(res) {
res.on('data', function(chunk){
// do whatever with response data here
});
res.on('end', function(){
console.log('Response #' + id + ' complete');
n--;
if (n == 0) {
console.log('DONE in ' + (Date.now() - start)/1000 + ' secs');
}
});
});
}
// Set # of simultaneous connections allowed
for (var i = 0; i < 1000; i++) {
getOne('http://www.facebook.com/status.php');
}
Outputs ...
$ node test.js
Response #3 complete
Response #0 complete
Response #2 complete
...
Response #999 complete
DONE in 0.658 secs

Answer 7

Thanks Alex Lunix for the suggestion. I looked up curl_multi_* and found a solution to do it in curl, so I don't have to change my code much. But thank you all the others for the answers. Here is what I did:
<?php
require("class.php");
$obj=new module();
$det=$obj->get_url();
$batch_size = 40;
function curlTest2($urls) {
clearstatcache();
$batch_size = count($urls);
$return = '';
echo "<br/><br/>Batch:";
foreach ($urls as &$url)
{
echo "<br/>".$url;
if(substr($url,0,4)!="http") $url = "http://".$url;
$url = "https://ajax.googleapis.com/ajax/services/feed/load?v=1.0&num=-1&q=".$url;
}
$userAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
$chs = array();
for ($i = 0; $i < $batch_size; $i++)
{
$ch = curl_init();
array_push($chs, $ch);
}
for ($i = 0; $i < $batch_size; $i++)
{
curl_setopt($chs[$i], CURLOPT_HEADER, 1);
curl_setopt($chs[$i], CURLOPT_NOBODY, 1);
curl_setopt($chs[$i], CURLOPT_USERAGENT, $userAgent);
curl_setopt($chs[$i], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($chs[$i], CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($chs[$i], CURLOPT_FAILONERROR, 1);
curl_setopt($chs[$i], CURLOPT_FRESH_CONNECT, 1);
curl_setopt($chs[$i], CURLOPT_URL, $urls[$i]);
}
$mh = curl_multi_init();
for ($i = 0; $i < $batch_size; $i++)
{
curl_multi_add_handle($mh, $chs[$i]);
}
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
for ($i = 0; $i < $batch_size; $i++)
{
curl_multi_remove_handle($mh, $chs[$i]);
}
curl_multi_close($mh);
}
$startTime = time();
$urls = array();
foreach($det as $key=>$value){
array_push($urls, $value['url']);
if (count($urls) == $batch_size)
{
curlTest2($urls);
$urls = array();
}
}
echo "<br/><br/>Time: ".(time() - $startTime)."sec";
?>
This brought down my processing time from 332sec to 18sec. The code probably can be optimized a little but you get the gist of it.