Is excessive cURL intensive on the server? - php

Question

I am setting up a site where my users can create lists of names that gets stored in the database. They can then "check" these lists, and each name in the list is run through a cURL function, checking an external site to see if that name is available or taken (for domain names, Twitter names, Facebook names, gaming names, etc). There will be a drop down for them to select which type of name they want to find, and it checks that site.
Here's a code sample for a Runescape name checker:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=" . $name);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$output = curl_exec($ch);
if (stristr($output,"non-member account")) {
echo 'Not available';
}
elseif (stristr($output,"private profile")) {
echo 'Not available';
}
elseif (stristr($output,"top skills")) {
echo 'Not available';
}
else {
echo 'Available';
}
curl_close($ch);
Will this cause too much stress on the server? I'm thinking also of capping lists, so maybe only 1,000 names per list for free members or something, and they can upgrade to run bigger lists (maybe even smaller than 1,000 for free users). Another thing I could do is store the results locally (which I'll do anyway), and load it from there if the name was searched recently. But then it's not completely accurate.

Answer 1

The answer can only be "it depends." It depends on how many users you have, how often those users hit the page in question, how beefy your hardware is, how much bandwidth your host allows, how much data is being transferred, and a million other things.
In general, you should locally (as in, on your server) cache as much data as you can from API responses. That prevents unnecessary duplicate API requests for data that you already had at some point previously. As for what data makes sense to cache, that is completely application/API specific, and something you will have to decide. In general, good candidates for caching are things that don't change very often and are either easy to determine when they are changed, or not important enough that somewhat stale data will be a big deal.

Answer 2

CURL requests are fundamentally slow, and PHP is, for the most part, a synchronous language, so unless you want to wait for each request to return (which, when I tested your command took ~1.2 seconds per request) your best bet it to either have PHP fork the curl requests using your OS's curl command via exec or to use non-blocking sockets. This article has a good explanation of how to do it:
https://segment.io/blog/how-to-make-async-requests-in-php/
However, you're still going to run into issues where the receiving host may not be able to handle the volume of requests you're sending (or it will blacklist you). You might have an easier time breaking the requests into batches (say ten names at a time) and then run those requests simultaneously against each host (Runescape, FB, etc)... this will let you run a few hundred simultaneous requests without hitting any one host too hard... It's still going to be a slowish process, and you might get your IP banned, but it's a reasonable approach.
Also, you might think about having the whole process broken down over a long-ish period of time... so a user uploads the list, and your server says "thanks, you'll receive an email when we're done"... then use a cron job to schedule the subsequent cURL requests over the course of an hour or so... which should help with all the above issues.