Submit POST data in loop without cURL and get response data - CodeIgniter - php

I have a code that sends SMS to customers that is set up as cron job. My problem is, when I use cURL, it's sending 2-part messages and our wallet is being charged twice. We have an average of 500 messages to send per day. My goal is to make the message 1-part only.
So I'm trying to find the best way to send messages in loop without using cURL.
I have thought of saving the post data from the loop in an array and send it to my view. Then when the view is called, I will automatically send the form withouth hitting the submit button and using javascript to auto submit. After the view is called, I will use file_get_contents() to get the response from the URL. But I cant make it work:
The view is not working inside the loop. It just works outside the loop.
If I pass the loop inside the view and not in the controller, how can I get the response data each loop?
My current code in CURL:
public function send_sms_globe(){
error_reporting(-1);
ini_set('display_errors', 1);
set_time_limit(0);
//get all data from the database pull with status = queue
$globe_data = $this->New_Sms_Api_model->get_queued_data('globe_api');
$passphrase = '[our_pass_phrase]';
$app_id = '[our_app_id]';
$app_secret = '[our_app_secret]';
$url = 'https://devapi.globelabs.com.ph/smsmessaging/v1/outbound/<our_shortcode>/requests/';
$ch = curl_init($url);
$limit = 0;
foreach($globe_data AS $records_data){
if($limit == 49){
break;
}
switch($limit) {
case 49:
$limit = 0;
break;
default:
if($records_data['Remarks'] == 'LOADED'){
if($records_data['sent_to'] == 'sender'){
$address = $records_data['sender_phone_number'];
}else if($records_data['sent_to'] == 'consignee'){
$address = $records_data['consignee_phone_number'];
}
} else {
$address = $records_data['sender_phone_number'];
}
//$address = '+63917*******';//address : *subscriber number $records_data['phone_number'];
$message = (isset($records_data['Message']) && $records_data['Message'] != '') ? $records_data['Message']:''; //message : *sms content
$str = iconv('UTF-8', 'ASCII//IGNORE//TRANSLIT', $message);
$post_data = [
'app_id' => $app_id,
'app_secret' => $app_secret,
'passphrase' => $passphrase,
'message' => rawurlencode($str),
'address' => $address
];
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
// execute!
$response = curl_exec($ch);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$queue_id = $records_data['ID'];
$invoice_number = $records_data['InvoicecNumber'];
$status_remark = $records_data['Remarks'];
$record_id = $records_data['record_id'];
if($http_code == 201 || $http_code == 200){
$no_of_tries = $records_data['no_of_tries'];
if($no_of_tries == 0){
$no_of_tries = 1;
} else {
$no_of_tries = $records_data['no_of_tries'];
}
$status = 'sent';
$retry = 0;
} else {
$no_of_tries = $records_data['no_of_tries'] + 1;
if($no_of_tries == 3){
$status = 'failed';
$retry = 0;
} else {
$status = 'retry';
$retry = 1;
}
}
$update_queued_data = $this->New_Sms_Api_model->update_queued_data($queue_id, $invoice_number, $status, $retry, $no_of_tries);
if($update_queued_data){
if($status == 'sent'){
if($status_remark == 'LOADED'){
$sent_to = $records_data['sent_to'];
} else {
$sent_to = NULL;
}
$this->New_Sms_Api_model->save_to_cq_sms($invoice_number, $status_remark, $record_id,$sent_to);
echo $records_data['record_id'].' ---- '.$status;
}
$limit++;
}
}
}
// close the connection, release resources used
curl_close($ch);
}
We have a message with 157 characters (160 max). I already talked to the API support we're using. First they suggested to format my message as URL encoding, and so I did. So from 3-part message it became 2-part. And then they said it will send as 2-part even if it's formatted that way because we use cURL. They suggested that we use PostMan but it's not for free so it's not an option.
Any ideas that can replace my current code? Thanks!

Sorry for the confusion. I was able to fix the issue without changing all of my code. I just removed rawurlencode in my string message and it's sending a 1-part message now.
Maybe using $str = iconv('UTF-8', 'ASCII//IGNORE//TRANSLIT', $message); already did the trick and using rawurlencode after just added additional characters ex: %20.
Thanks everyone!

Related

SMS Reminder remain from Server Side in php

what is doing on SMS send Reminder to remain from Server Side for package end within 7 days or less than 10 days? any code or example video how to implement query.
I want the alert will come out about 2 weeks before the expiry date.
Many Paid SMS api available in online.
Here i'm using SmsHorizon
// Replace with your username
$user = "yourname";
// Replace with your API KEY (We have sent API KEY on activation email, also available on panel)
$apikey = "ABCBEFGH786756";
// Replace if you have your own Sender ID, else donot change
$senderid = "WEBSMS";
// For Plain Text, use "txt" ; for Unicode symbols or regional Languages like hindi/tamil/kannada use "uni"
$type = "txt";
// Database Configuration
$con = mysql_connect("143.114.0.49","username","password");
$con_db = mysql_select_db("db_name",$con);
/***********************Over Speed Alert***/ //Run every 1 minute
$i = 0;
$message = "2 Week Before Expiry Alert.\n" ;
$device_query = mysql_query("SELECT * FROM table_name); // Write whatever the query you've required.
while($device_value = mysql_fetch_array($device_query))
{
$message.= ""; // Content you load here.
$i++;
}
// Replace with the destination mobile Number to which you want to send sms
$mobile = array('9999999999');
// Replace with your Message content
$message = urlencode($message);
if($i > 0)
{
for($j = 0; $j < count($mobile); $j++)
{
$ch = curl_init("http://smshorizon.co.in/api/sendsms.php?user=".$user."&apikey=".$apikey."&mobile=".$mobile[$j]."&senderid=".$senderid."&message=".$message."&type=".$type."");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//$output = curl_exec($ch);
if(curl_errno($ch))
{
echo 'Curl error: ' . curl_error($ch);
}
curl_close($ch);
// Display MSGID of the successful sms push
//echo $output;
}
}
Hope it help you.

Telegram webhook php bot doesn't answer

I'm trying to set up a telegram bot with a webhook. I can get it to work with getUpdates, but I want it to work with a webhook.
My site (that hosts the bot php script) has the SSL certificate working (I get the green lock in the address bar):
I set up the webhook with
https://api.telegram.org/bot<token>/setwebhook?url=https://www.example.com/bot/bot.php
And I got: {"ok":true,"result":true,"description":"Webhook was set"}
(I don't know if this matters, but I have given rwx rights to both the folder and the script)
The php bot: (https://www.example.com/bot/bot.php)
<?php
$botToken = <token>;
$website = "https://api.telegram.org/bot".$botToken;
#$update = url_get_contents('php://input');
$update = file_get_contents('php://input');
$update = json_decode($update, TRUE);
$chatId = $update["message"]["chat"]["id"];
$message = $update["message"]["text"];
switch($message) {
case "/test":
sendMessage($chatId, "test");
break;
case "/hi":
sendMessage($chatId, "hi there!");
break;
default:
sendMessage($chatId, "default");
}
function sendMessage ($chatId, $message) {
$url = $GLOBALS[website]."/sendMessage?chat_id=".$chatId."&text=".urlencode($message);
url_get_contents($url);
}
function url_get_contents($Url) {
if(!function_exists('curl_init')) {
die('CURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
?>
But when I write anything to the bot I receive no answers...
Any ideas why?
Thanks
In your question it's not clear the script location. Seeing your code, it seems that you try to load a request through url_get_contents to retrieve telegram server response. This is the correct method if your bot works without webhook. Otherwise, after setting webhook, you have to process incoming requests.
I.e., if you set webhook to https://example.com/mywebhook.php, in your https://example.com/mywebhook.php script you have to write something like this:
<?php
$request = file_get_contents( 'php://input' );
# ↑↑↑↑
$request = json_decode( $request, TRUE );
if( !$request )
{
// Some Error output (request is not valid JSON)
}
elseif( !isset($request['update_id']) || !isset($request['message']) )
{
// Some Error output (request has not message)
}
else
{
$chatId = $request['message']['chat']['id'];
$message = $request['message']['text'];
switch( $message )
{
// Process your message here
}
}

Sending a payment gateway POST request through a RESTful API

I am trying to integrate a payment gateway into a website that is being driven by AngularJS on the front-end and PHP Yii Framework on the backend. The backend is a REST API.
I have a page on my website with all my buying options. When a user clicks on the "Buy" button alongside any of these options I need the user to be sent to the payment page of the payment gateway service provider along with the required details sent as a POST request.
To do this, first I am sending a JSON to one of my APIs. This JSON contains some product related details and that's about it. It is as follows.
$scope.payment = {
key: '',
txnid: '',
amount: '1250',
productinfo: '3',
firstname: '',
email: '',
phone: '',
surl: '',
furl: '',
hash: '',
service_provider: ''
};
Except the amount and product info, all other values are empty.
Once this JSON is received by the API, the API decodes this JSON and fills it up with all the other values. The API code is as follows.
public function actionMakePayment () {
$returnInfo = array("requireLogin"=>false);
if (!$this->isLogedIn()) {
$returnInfo['requireLogin'] = true; // Checking if user is logged in
} else {
$userId = Yii::app()->user->id; // Getting user id
$currentUserModel = User::model()->findByPk($userId); // Extracting user model
$email = $currentUserModel->email; // Extracting email ID
$phone = $currentUserModel->contact_no; // Extracting contact number
$first_name = $currentUserModel->first_name; // Extracting first name
$action = '';
$json = file_get_contents('php://input'); // Taking in the posted JSON
$posted = json_decode($json, true); // Decoding JSON
$MERCHANT_KEY = "XXXXXX"; // Setting merchant key
$SALT = "XXXXXXXX"; // Setting merchant SALT
$PAYU_BASE_URL = "https://paymentgateway.com"; // Gateway domain name
$SERVICE_PROVIDER = "service_provider"; // Gateway provider ID
$RETURN_URL = "http://domain.com/rest/api/resolvePayment"; // Setting URL for redirection after payment is made or cancelled
$formError = 0; // POST data error check
// Assigning txnid
if (empty($posted['txnid'])) {
$txnid = substr(hash('sha256', mt_rand() . microtime()), 0, 20);
$posted['txnid'] = $txnid;
} else {
$txnid = $posted['txnid'];
}
$posted['key'] = $MERCHANT_KEY; // assigning the merchant key
$posted['surl'] = $RETURN_URL; // assigning success URL
$posted['furl'] = $RETURN_URL; // assigning failure URL
$posted['service_provider'] = $SERVICE_PROVIDER; // assigning
$posted['firstname'] = $first_name; // assigning name
$posted['phone'] = $phone; // assigning contact number
$posted['email'] = $email;
$hash = '';
$hashSequence = "key|txnid|amount|productinfo|firstname|email|udf1|udf2|udf3|udf4|udf5|udf6|udf7|udf8|udf9|udf10";
if (empty($posted['hash']) && sizeof($posted) > 0) {
if (
empty($posted['key']) ||
empty($posted['txnid']) ||
empty($posted['amount']) ||
empty($posted['firstname']) ||
empty($posted['email']) ||
empty($posted['phone']) ||
empty($posted['productinfo']) ||
empty($posted['surl']) ||
empty($posted['furl']) ||
empty($posted['service_provider'])
) {
$formError = 1;
} else {
$hashVarsSeq = explode('|', $hashSequence);
$hash_string = '';
foreach($hashVarsSeq as $hash_var) {
$hash_string .= isset($posted[$hash_var]) ? $posted[$hash_var] : '';
$hash_string .= '|';
}
$hash_string .= $SALT;
$hash = strtolower(hash('sha512', $hash_string));
$posted['hash'] = $hash;
$action = $PAYU_BASE_URL . '/_payment';
}
} else if (!empty($posted['hash'])) {
$hash = $posted['hash'];
$action = $PAYU_BASE_URL . '/_payment';
}
}
echo json_encode($posted);
**$this->send_post($action, $posted);**
}
When I echo $posted as a response to the API, it returns the exact JSON I am required to POST to the payment gateway URL. Now comes the part where I am struggling. I am using the last line of code to send the data as POST request to the URL $action. The code for the function "send_post" is as follows.
private function send_post($url, $data) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); // set url to post to
// curl_setopt($ch, CURLOPT_FAILonerror, TRUE); //Fail on error
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // return into a variable
curl_setopt($ch, CURLOPT_POST, TRUE); // set POST method
curl_setopt($ch, CURLOPT_POSTFIELDS, $data); // add POST fields
$result = curl_exec($ch); // run the whole process
curl_close($ch);
return $result;
}
I had to comment out the CURLOPT_FAILonerror option as it throws an error. Do not know why. Also, after putting up all this code, when I click on the "buy" button on the front-end, the API executes and return $posted, but I am not taken to the payment page, and I don't know if the data gets posted or not. There is no error in the response to the API call. It executes perfectly as far as I can see.
Can someone help me with this? I am posting the data correctly? Or am I supposed to post the $hash_string variable? The last few parts are confusing me.
This is not strictly an answer to the issue I faced with CURL. But I figured a work around. I sent the parameters as GET variables to an intermediate PHP page that I built. I then captured these GET variables in the PHP page and sent them as a POST request to the payment gateway.
I don't know why CURL did not work, and after I spoke to the payment gateway tech guys they were very much against using CURL. So I tried to send a POST request to the gateway using AngularJS itself. But this was another huge (and seemingly popular) problem with response headers in Angular. I went through a lot of forums to check for solutions but nothing worked for me. So I finally decided to build and intermediate PHP page and work it out as described above.

Output PHP Parser Script Variables to PHP form via Sessions

New to PHP, so bear with me...
I'm trying to send/make available the output variables from this simpleXml parser script to this other PHP file, which is supposed to send data to Brightcove's Media API.
Sending Script:
<?php
session_name("FeedParse");
session_start();
$_SESSION['bcName'] = $title;
$_SESSION['shortDescription'] = $description;
$_SESSION['remoteUrl'] = $videoFile;
$html = "";
$url = "http://feeds.nascar.com/feeds/video?command=search_videos&media_delivery=http&custom_fields=adtitle%2cfranchise&page_size=100&sort_by=PUBLISH_DATE:DESC&token=217e0d96-bd4a-4451-88ec-404debfaf425&any=franchise:%20Preview%20Show&any=franchise:%20Weekend%20Top%205&any=franchise:Up%20to%20Speed&any=franchise:Press%20Pass&any=franchise:Sprint%20Cup%20Practice%20Clips&any=franchise:Sprint%20Cup%20Highlights&any=franchise:Sprint%20Cup%20Final%20Laps&any=franchise:Sprint%20Cup%20Victory%20Lane&any=franchise:Sprint%20Cup%20Post%20Race%20Reactions&any=franchise:All%20Access&any=franchise:Nationwide%20Series%20Qualifying%20Clips&any=franchise:Nationwide%20Series%20Highlights&any=franchise:Nationwide%20Series%20Final%20Laps&any=franchise:Nationwide%20Series%20Victory%20Lane&any=franchise:Nationwide%20Series%20Post%20Race%20Reactions&any=franchise:Truck%20Series%20Qualifying%20Clips&any=franchise:Truck%20Series%20Highlights&any=franchise:Truck%20Series%20Final%20Laps&any=franchise:Truck%20Series%20Victory%20Lane&any=franchise:Truck%20Series%20Post%20Race%20Reactions&output=mrss";
$xml = simplexml_load_file($url);
$namespaces = $xml->getNamespaces(true); // get namespaces
for($i = 0; $i < 50; $i++){ // will return the 50 most recent videos
$title = $xml->channel->item[$i]->video;
$link = $xml->channel->item[$i]->link;
$title = $xml->channel->item[$i]->title;
$pubDate = $xml->channel->item[$i]->pubDate;
$description = $xml->channel->item[$i]->description;
$titleid = $xml->channel->item[$i]->children($namespaces['bc'])->titleid;
$m_attrs = $xml->channel->item[$i]->children($namespaces['media'])->content[0]->attributes();
$videoFile = $m_attrs["url"];
$html .= //"<h3>$title</h3>$description<p>$pubDate<p>$url<p>Video ID: $titleid<p>
print $title;
print $description;
print $videoFile;
// echo $html;/* tutorial for this script is here https://www.youtube.com/watch?v=4ZLZkdiKGE0 */
}
//http://support.brightcove.com/en/video-cloud/docs/media-write-api-php-example-upload-video
?>
Receiving Script:
<?php
session_start();
$title = $_SESSION['bcName'];
$description = $_SESSION['shortDescription'];
$videoFile = $_SESSION['remoteUrl'];
// Instantiate the Brightcove class
$bc = new Brightcove(
'//readtoken//', //Read Token BC
'//writetoken//' //Write Token BC
);
// Set the data for the new video DTO using the form values
$metaData = array(
'$title' => $_POST['bcName'],
'$description' => $_POST['bcShortDescription'],
);
//changed all the code below to what i think works for remoteUrl and URLs as opposed to actual video files
// Rename the file to its original file name (instead of temp names like "a445ertd3")
$url = $_URL['remoteUrl'];
//rename($url['tmp_name'], '/tmp/' . $url['name']);
//$url = '/tmp/' . $url['name'];
// Send the file to Brightcove
//Actually, this has been changed to send URL to BC, not file
echo $bc->createVideo($url,$metaData);
class Brightcove {
public $token_read = 'UmILcDyAFKzjtWO90HNzc67X-wLZK_OUEZliwd9b3lZPWosBPgm1AQ..'; //Read Token from USA Today Sports BC
public $token_write = 'svP0oJ8lx3zVkIrMROb6gEkMW6wlX_CK1MoJxTbIajxdn_ElL8MZVg..'; //Write Token from USA Today Sports BC
public $read_url = 'http://api.brightcove.com/services/library?';
public $write_url = 'http://api.brightcove.com/services/post';
public function __construct($token_read, $token_write = NULL ) {
$this->token_read = $token_read;
$this->token_write = $token_write;
}
public function createVideo($url = NULL, $meta) {
$request = array();
$post = array();
$params = array();
$video = array();
foreach($meta as $key => $value) {
$video[$key] = $value;
}
$params['token'] = $this->token_write;
$params['video'] = $video;
$post['method'] = 'create_video';
$post['params'] = $params;
$request['json'] = json_encode($post);
if($file) {
$request['file'] = '#' . $file;
}
// Utilize CURL library to handle HTTP request
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $this->write_url);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $request);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_VERBOSE, TRUE );
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 300);
curl_setopt($curl, CURLOPT_TIMEOUT, 300);
$response = curl_exec($curl);
curl_close($curl);
// Responses are transfered in JSON, decode into PHP object
$json = json_decode($response);
// Check request error code and re-call createVideo if request
// returned a 213 error. A 213 error occurs when you have
// exceeded your allowed number of concurrent write requests
if(isset($json->error)) {
if($json->error->code == 213) {
return $this->createVideo($url, $meta);
} else {
return FALSE;
}
} else {
return $response;
}
}
}
?>​
Did I set up sessions to work correctly here? Any ideas on why the receiving PHP script isn't picking up the data/variables outputted by the PHP feed parser script?
In your sending script, it looks like you're setting the session variables at the beginning of the script and you're expecting them to get updated whenever the local variable changes. This won't happen with the way you've written it.
You could make this happen by assigning the variables by reference, by putting an ampersand (&) before the local variable name, but this can get kinda tricky in some scenarios, and it might be best to skip that headache and instead just update the session variable directly.
However, another issue is that you're attempting to store multiple values (50, as indicated by the code comments) into a scalar session variable. So every time your loop iterates, it would overwrite the previous value. Perhaps what would be better would be to use an array structure:
<?php
session_name("FeedParse");
session_start();
$_SESSION['videos'] = array(); // initialize a session variable 'videos' to be an array
$url = "http://feeds.nascar.com/feeds/video?command=search_videos&media_delivery=http&custom_fields=adtitle%2cfranchise&page_size=100&sort_by=PUBLISH_DATE:DESC&token=217e0d96-bd4a-4451-88ec-404debfaf425&any=franchise:%20Preview%20Show&any=franchise:%20Weekend%20Top%205&any=franchise:Up%20to%20Speed&any=franchise:Press%20Pass&any=franchise:Sprint%20Cup%20Practice%20Clips&any=franchise:Sprint%20Cup%20Highlights&any=franchise:Sprint%20Cup%20Final%20Laps&any=franchise:Sprint%20Cup%20Victory%20Lane&any=franchise:Sprint%20Cup%20Post%20Race%20Reactions&any=franchise:All%20Access&any=franchise:Nationwide%20Series%20Qualifying%20Clips&any=franchise:Nationwide%20Series%20Highlights&any=franchise:Nationwide%20Series%20Final%20Laps&any=franchise:Nationwide%20Series%20Victory%20Lane&any=franchise:Nationwide%20Series%20Post%20Race%20Reactions&any=franchise:Truck%20Series%20Qualifying%20Clips&any=franchise:Truck%20Series%20Highlights&any=franchise:Truck%20Series%20Final%20Laps&any=franchise:Truck%20Series%20Victory%20Lane&any=franchise:Truck%20Series%20Post%20Race%20Reactions&output=mrss";
$xml = simplexml_load_file($url);
$namespaces = $xml->getNamespaces(true); // get namespaces
for($i = 0; $i < 50; $i++){ // will return the 50 most recent videos
$m_attrs = $xml->channel->item[$i]->children($namespaces['media'])->content[0]->attributes();
// on each loop iteration, create a new array structure with the video info in it and
// push it onto the 'video' array session variable
$video = array(
'bcName' => $xml->channel->item[$i]->video,
'shortDescription' => $xml->channel->item[$i]->description,
'remoteUrl' => $m_attrs["url"],
);
$_SESSION['videos'][] = $video;
}
Then, on your receiving script, you'll loop through $_SESSION['videos']:
<?php
session_start();
// Instantiate the Brightcove class
$bc = new Brightcove(
'UmILcDyAFKzjtWO90HNzc67X-wLZK_OUEZliwd9b3lZPWosBPgm1AQ..', //Read Token from USA Today Sports BC
'svP0oJ8lx3zVkIrMROb6gEkMW6wlX_CK1MoJxTbIajxdn_ElL8MZVg..' //Write Token from USA Today Sports BC
foreach ((array)$_SESSION['videos'] as $video) {
$title = $video['bcName'];
$description = $video['shortDescription'];
$videoFile = $video['remoteUrl'];
// The code below this line may need to be adjusted. It does not seem quite right.
// Are you actually sending anything to this script via POST? Or should those just
// be the values we set above?
// What is $_URL? Should that just be the $videoFile value?
// Set the data for the new video DTO using the form values
$metaData = array(
'$title' => $_POST['bcName'],
'$description' => $_POST['bcShortDescription'],
);
//changed all the code below to what i think works for remoteUrl and URLs as opposed to actual video files
// Rename the file to its original file name (instead of temp names like "a445ertd3")
$url = $_URL['remoteUrl'];
//rename($url['tmp_name'], '/tmp/' . $url['name']);
//$url = '/tmp/' . $url['name'];
// Send the file to Brightcove
//Actually, this has been changed to send URL to BC, not file
echo $bc->createVideo($url,$metaData);
}
IMPORTANT NOTE:
Keep in mind that this will call the API once for each video in the session (sounds like up to 50 each time). So you'll be making 50 cURL requests on each run of this script. That seems a bit heavy, but perhaps that's expected. It would be worth investigating if their API allows you to compile the data into one call and send it all up at once, as opposed to connecting, sending the data, parsing the response, and disconnecting, 50 times.

multi curl unable to handle more than 200 request at a time

Could you please tell me , is there any limitation to send a request using multi_curl.
When I tried to send a request more than 200 , it was getting timeout.
see the below code ..............
.........................................
foreach($newUrlArry as $url){
$gatherUrl[] = $url['url'];
}
/*...................Array slice----------------------*/
$totalUrlRequest = count($gatherUrl);
if($totalUrlRequest > 10){
$offset = 10;
$index = 0;
$matchedAnchors = array();
$dom = new DOMDocument;
$NoOfTilesRequest = ceil($totalUrlRequest/$offset);
for($sl = 0; $sl<$NoOfTilesRequest;$sl++){
$output = array_slice($gatherUrl, $index, $offset);
$index = $offset+$index;
$responseAction = $this->multiRequestAction($output);
$k=0;
foreach($responseAction as $responseHtml){
#$dom->loadHTML($responseHtml);
$documentLinks = $dom->getElementsByTagName("a");
$chieldFlag = false;
for($i=0;$i<$documentLinks->length;$i++) {
$documentLink = $documentLinks->item($i);
if ($documentLink->hasAttribute('href') AND substr($documentLink->getAttribute('href'), 0, strlen($match)) == $match) {
$description = $documentLink->childNodes;
foreach($description as $words) {
$name = trim($words->nodeName);
if($name == 'em' || $name == 'b' || $name=="span" || $name=="p") {
if(!empty($words->nodeValue)) {
$matchedAnchors[$sl][$k]['anchor'] = trim($words->nodeValue);
$matchedAnchors[$sl][$k]['img'] = 0;
if($documentLink->hasAttribute('rel'))
$matchedAnchors[$sl][$k]['rel'] = 'Y';
else
$matchedAnchors[$sl][$k]['rel'] = 'N';
$chieldFlag = true;
break;
}
}
elseif($name == 'img' ) {
$alt= $words->getAttribute('alt');
if(!empty($alt)) {
$matchedAnchors[$sl][$k]['anchor'] = trim($words->getAttribute('alt'));
$matchedAnchors[$sl][$k]['img'] = 1;
if($documentLink->hasAttribute('rel'))
$matchedAnchors[$sl][$k]['rel'] = 'Y';
else
$matchedAnchors[$sl][$k]['rel'] = 'N';
$chieldFlag = true;
break;
}
}
}
if(!$chieldFlag){
$matchedAnchors[$sl][$k]['anchor'] = $documentLink->nodeValue;
$matchedAnchors[$sl][$k]['img'] = 0;
if($documentLink->hasAttribute('rel'))
$matchedAnchors[$sl][$k]['rel'] = 'Y';
else
$matchedAnchors[$sl][$k]['rel'] = 'N';
}
}
}$k++;
}
}
}
Both #Phliplip & #lunixbochs have mentioned common cURL pitfalls (max execution time & denied by the target server.)
When sending that many cURL requests to the same server I try to "be nice" and place voluntarily sleep periods so I don't bombard the host. For a low-end site, 1000+ requests could be like a mini DDOS!
Here's code that's worked for me. I it used to scrape a client's product data from their old site, since the data was locked in a proprietary database system with NO export function.
<?php
header('Content-type: text/html; charset=utf-8', true);
set_time_limit(0);
$urls = array(
'http://www.example.com/cgi-bin/product?id=500',
'http://www.example.com/cgi-bin/product?id=501',
'http://www.example.com/cgi-bin/product?id=502',
'http://www.example.com/cgi-bin/product?id=503',
'http://www.example.com/cgi-bin/product?id=504',
);
$i = 0;
foreach($urls as $url){
echo $url."\n";
$curl = curl_init($url);
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
curl_setopt($curl, CURLOPT_USERAGENT, $userAgent);
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt($curl, CURLOPT_TIMEOUT, 25 );
$html = curl_exec($curl);
$html = #mb_convert_encoding($html, 'HTML-ENTITIES', 'utf-8');
curl_close($curl);
// now do something with info returned by curl
$i++;
if($i%10==0){
sleep(20);
} else {
sleep(2);
}
}
?>
The main features are:
no max execution time
voluntary sleep-ing
new curl init & exec for each request.
In my experience, going to sleep() will stop servers from denying you.
However if by "different different server" you mean that you are sending a small number of requests a large number of servers, for example:
$urls = array(
'http://www.example-one.com/',
'http://www.example-two.com/',
'http://www.example-three.com/',
'http://www.example-four.com/',
'http://www.example-five.com/',
'http://www.example-six.com/'
);
And you are using set_time_limit(0); then something then an error may be causing your code to die; try
ini_set('display_errors',1);
error_reporting(E_ALL);
And tell us the error message you are getting.
PHP doesn't place a restriction on the number of connections using curl_multi_init, but memory usage and time limits will be an issue.
Check your memory_limit setting in your php.ini and try to increase it to see if that helps you.

Categories