How to access long polling results in wordpress plugin? - php

I am using Peepso plugin for my Wordpress website. Peepso is using Long polling for the part of the chats in order to create real-time chat. I don't know much on Long polling so I am confused in the way it is written inside the add-on file. What I want is to connect this website with my android application so I want to get those data into the android application in real-time. Here is the code of Peepso messages:
/**
* Creates a new message thread and adds the first message.
* #param PeepSoAjaxResponse $resp
*/
public function new_message(PeepSoAjaxResponse $resp)
{
$subject = '';
$message = htmlspecialchars($this->_input->raw('message'));
// SQL safe, forced to int
$participants = array_map('intval',$this->_input->value('recipients', array(), FALSE));
$creator_id = get_current_user_id();
$model = new PeepSoMessagesModel();
$msg_id = $model->create_new_conversation($creator_id, $message, $subject, $participants);
if (is_wp_error($msg_id)) {
$resp->success(FALSE);
$resp->error($msg_id->get_error_message());
$resp->set('subject', $subject);
$resp->set('creator_id', $creator_id);
} else {
$ps_messages = PeepSoMessages::get_instance();
do_action('peepso_messages_new_message', $msg_id);
$resp->success(TRUE);
$resp->notice(__('Message sent.', 'msgso'));
$resp->set('msg_id', $msg_id);
$resp->set('url', $ps_messages->get_message_url($msg_id));
}
}
/**
* Add a message to an existing conversation.
* #param PeepSoAjaxResponse $resp The response object to be sent.
*/
public function add_message(PeepSoAjaxResponse $resp)
{
$message = htmlspecialchars($this->_input->raw('content'));
$parent_id = $this->_input->int('parent_id');
$user_id = get_current_user_id();
// Make sure the "I am typing" flag is deleted
$mayfly = 'msgso_typing_' . $user_id. '_' . $parent_id;
PeepSo3_Mayfly::del($mayfly);
// And that another one can't be set for another full second
$mayfly_just_posted = "msgso_posted__{$user_id}_{$parent_id}";
PeepSo3_Mayfly::set($mayfly_just_posted,1,1);
$participants = new PeepSoMessageParticipants();
if (FALSE === $participants->in_conversation($user_id, $parent_id)) {
$resp->success(FALSE);
$resp->error(__('You are not part of this conversation.', 'msgso'));
} else {
$model = new PeepSoMessagesModel();
$msg_id = $model->add_to_conversation($user_id, $parent_id, $message);
if (FALSE === $msg_id) {
$resp->success(FALSE);
$resp->error(__('Failed to send message.', 'msgso'));
} else {
$resp->success(TRUE);
$resp->notice(__('Message sent.', 'msgso'));
$resp->set('msg_id', $msg_id);
}
}
}
And here is the js file that controls the long polling:
/**
* Starts long-polling to get chat stack state.
*/
startLongPolling: function () {
if (!SSE_ENABLED && !this.sessionExpired) {
this.stopLongPolling();
this.fetchChatState().done(
$.proxy(function () {
this.checkChatState();
}, this)
);
}
},
/**
* Stops long-polling to get chat stack state.
*/
stopLongPolling: function () {
if (!SSE_ENABLED) {
clearTimeout(this.checkChatTimer);
this.checkChatXHR && this.checkChatXHR.abort();
this.fetchChatXHR && this.fetchChatXHR.ret && this.fetchChatXHR.ret.abort();
this.checkChatXHR = false;
this.fetchChatXHR = false;
}
}
I tried to get those data by adding this line into the code echo json_encode($resp, JSON_PRETTY_PRINT); but nothing happened. So, how can I get the data from this code above? If you need any further information please let me know as I am new on long polling and how to connect it with android application. Any answer will be appreciated.

Related

Drupal 9 Webforms - Attach/Trigger Remote Post Webform Handler on Programmatic Webform Submission Update (CRON)

I have been digging around for about a week trying to figure this out and still have not come across anything that works or is even close.
The idea is that I have a custom webform. This webform has an Remote POST submit handler. Part of that submission and submit handler puts a value from the response into the webform via TOKENS. If an Remote POST submission fails, this value will be empty. EDIT FOR CLARITY: The default value for this "api_response" field (which is hidden on the form) is the token
[webform:handler:remote_post:completed:0:inserted:[webform_submission:values:external_id]][webform:handler:remote_post_1:completed:0:inserted:[webform_submission:values:external_id]]
In a CRON job, a query is run to see which webform submissions do not have this value and then it adds them to a Queue.
The queue worker then loops through and simply attempts to resubmit the Webform Submission if it failed earlier for whatever reason. By my initial understanding, when the submission update is submited, it should trigger the handlers. However, in the programmatic Cron, submission update does not appear to trigger any handlers. I have found some stuff for simple email handlers, but they do not have remotely the same setup or methods for \Drupal\webform\Plugin\WebformHandler\RemotePostWebformHandler. No idea how to ensure this gets re-ran with a webform submission update.
This is my queue worker file.
<?php
namespace Drupal\my_module\Plugin\QueueWorker;
use Drupal\Core\Entity\EntityStorageException;
use Drupal\Core\Queue\QueueWorkerBase;
use Drupal\webform\Entity\Webform;
use Drupal\webform\Entity\WebformSubmission;
use Drupal\webform\Plugin\WebformHandler\RemotePostWebformHandler;
use Drupal\webform\WebformSubmissionForm;
/**
* Product Offer Scheduler Queue Worker.
*
* #QueueWorker(
* id = "webform_submission_replay_queue_worker",
* title = #Translation("Webform Submission Replay Queue Worker: WebformSubmission"),
* cron = {"time" = 180}
* )
*/
class WebformSubmissionReplayQueueWorker extends QueueWorkerBase {
/**
* A function to run the appropriate handlers on each webform.
*
* #param WebformSubmission $webformSubmission
* #param string $webformId
*
* #throws EntityStorageException
*/
private function processWebformSubmission($webformSubmission, $webformId) {
switch ($webformId) {
case 'form_id':
$accountId = $webformSubmission->getElementData('account_id');
$accountType = $webformSubmission->getElementData('account_type');
$webform = $webformSubmission->getWebform();
// $webform->getState();
if ($accountId == "" || $accountType != "customer") {
// $config = \Drupal::configFactory()->getEditable('webform.webform.' . $webformId);
// $config_key = 'handlers.remote_post';
// $config->
// $webform->addWebformHandler($webformSubmission->getWebform()->getHandler('remote_post'));
// $webform->save();
// /** #var RemotePostWebformHandler $handler */
// $handler = $webformSubmission->getWebform()->getHandler('remote_post');
// $handler->setWebformSubmission($webformSubmission);
// $handler->submitForm();
// $message = $handler->getMessage($webformSubmission);
// $handler->sendMessage($webformSubmission, $message);
} elseif ($accountId != "" && $accountType == "customer") {
// $webform->addWebformHandler($webformSubmission->getWebform()->getHandler('remote_post_1'));
// $webform->save();
// $handler = $webformSubmission->getWebform()->getHandler('remote_post_1');
// $handler->setWebformSubmission($webformSubmission);
// $handler->submitForm();
// $message = $handler->getMessage($webformSubmission);
// $handler->sendMessage($webformSubmission, $message);
}
break;
default:
// do nothing
}
}
public function processItem($data) {
$webformId = $data->webform_id;
$submissionSerial = $data->serial;
$webform = Webform::load($webformId);
$isOpen = WebformSubmissionForm::isOpen($webform);
if($isOpen === TRUE){
$webformSubmission = WebformSubmission::load($submissionSerial);
// $token = \Drupal::token();
// $elem = $webform->getElement('api_response');
// $default = $token->replace($elem['#value']);
// \Drupal::logger('my_module')->debug('tokens: <pre>#res</pre>', [
// '#res' => $token->replace($elem['#value'])
// ]);
// $webformSubmission->setElementData('api_response', $token->replace($elem['#value']));
// $fields = $webformSubmission->getFields();
// \Drupal::logger('my_module_'.$webformId)->debug('Fields: <pre>#data</pre>', [
// '#data' => print_r(array_keys($fields), true)
// ]);
$errors = WebformSubmissionForm::validateWebformSubmission($webformSubmission);
if(!empty($errors)) {
\Drupal::logger('my_module_reroll_errors_'.$webformId)
->debug('Submission ID: #id - Errors: <pre>#err</pre>', [
'#id' => $submissionSerial,
'#err' => print_r($errors, true)
]);
} else {
// $ws = WebformSubmission::;
// $webformSubmission = WebformSubmissionForm::submitWebformSubmission($webformSubmission);
// $webformSubmission->save();
// $handlers = $webform->getHandlers();
// \Drupal::logger('my_module')->debug('tokens: <pre>#res</pre>', [
// '#res' => $token->replace($elem['#value'])
// ]);
$this->processWebformSubmission($webformSubmission, $webformId);
\Drupal::logger('my_module_reroll_'.$webformId)->debug('sid: #sid --- serial: #serial', [
'#sid' => $webformSubmission->id(),
'#serial' => $submissionSerial
]);
}
}
}
}

Form Submission Not Working with Gsuite email - but is with any other

I am a reasonably new developer and finally hit my first unsolvable problem!
I have used a reusable form and successfully tested it on two of my own Gmail accounts (example#gmail.com), however as soon as I use the client's professional G suite email (info#example.co.uk) it doesn't work, the email never arrives. I have checked & rechecked obvious things such as the spelling, cached files, spam & junk folders & re-uploaded the code several times. I have been through all of the Gsuite email settings and even added the IP to both the email whitelist & inbound gateway. Nothing works.
I am wondering if there is something in the MX records I have to do? I didn't try this as it worked with my other emails and there was nothing in the info about this.
I also considered contacting Gsuite to see if some kind of block exists in professional email?
Here is the code i am using:
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
/*
Tested working with PHP5.4 and above (including PHP 7 )
*/
require_once './vendor/autoload.php';
use FormGuide\Handlx\FormHandler;
$pp = new FormHandler();
$validator = $pp->getValidator();
$validator->fields(['name','email'])->areRequired()->maxLength(50);
$validator->field('email')->isEmail();
$validator->field('comments')->maxLength(6000);
$pp->sendEmailTo('info#example.co.uk'); // ← Your email here
echo $pp->process($_POST);
Update, I checked the Logs and I am getting this error Message:
PHP Fatal error: Uncaught Error: Class 'FormGuide\PHPFormValidator\FormValidator' not found in /home4/macatac5/public_html/anderscapeslandscaping.co.uk/src/FormHandler.php:41
This is the code it relates to:
<?php
namespace FormGuide\Handlx;
use FormGuide\PHPFormValidator\FormValidator;
use PHPMailer;
use FormGuide\Handlx\Microtemplate;
use Gregwar\Captcha\CaptchaBuilder;
/**
* FormHandler
* A wrapper class that handles common form handling tasks
* - handles Form validations using PHPFormValidator class
* - sends email using PHPMailer
* - can handle captcha validation
* - can handle file uploads and attaching the upload to email
*
* ==== Sample usage ====
* $fh = FormHandler::create()->validate(function($validator)
* {
* $validator->fields(['name','email'])
* ->areRequired()->maxLength(50);
* $validator->field('email')->isEmail();
*
* })->useMailTemplate(__DIR__.'/templ/email.php')
* ->sendEmailTo('you#website.com');
*
* $fh->process($_POST);
*/
class FormHandler
{
private $emails;
public $validator;
private $mailer;
private $mail_template;
private $captcha;
private $attachments;
private $recaptcha;
public function __construct()
{
$this->emails = array();
$this->validator = FormValidator::create();
$this->mailer = new PHPMailer;
$this->mail_template='';
$this->mailer->Subject = "Contact Form Submission ";
$host = isset($_SERVER['SERVER_NAME'])?$_SERVER['SERVER_NAME']:'localhost';
$from_email ='forms#'.$host;
$this->mailer->setFrom($from_email,'Contact Form',false);
$this->captcha = false;
$this->attachments = [];
$this->recaptcha =null;
}
/**
* sendEmailTo: add a recipient email address
* #param string/array $email_s one or more emails. If more than one emails, pass the emails as array
* #return The form handler object itself so that the methods can be chained
*/
public function sendEmailTo($email_s)
{
if(is_array($email_s))
{
$this->emails =array_merge($this->emails, $email_s);
}
else
{
$this->emails[] = $email_s;
}
return $this;
}
public function useMailTemplate($templ_path)
{
$this->mail_template = $templ_path;
return $this;
}
/**
* [attachFiles find the file uplods and attach to the email]
* #param array $fields The array of field names
*/
public function attachFiles($fields)
{
$this->attachments = array_merge($this->attachments, $fields);
return $this;
}
public function getRecipients()
{
return $this->emails;
}
/**
* [validate add Validations. This function takes a call back function which receives the PHPFormValidator object]
* #param function $validator_fn The funtion gets a validator parameter using which, you can add validations
*/
public function validate($validator_fn)
{
$validator_fn($this->validator);
return $this;
}
public function requireReCaptcha($config_fn=null)
{
$this->recaptcha = new ReCaptchaValidator();
$this->recaptcha->enable(true);
if($config_fn)
{
$config_fn($this->recaptcha);
}
return $this;
}
public function getReCaptcha()
{
return $this->recaptcha;
}
public function requireCaptcha($enable=true)
{
$this->captcha = $enable;
return $this;
}
public function getValidator()
{
return $this->validator;
}
public function configMailer($mailconfig_fn)
{
$mailconfig_fn($this->mailer);
return $this;
}
public function getMailer()
{
return $this->mailer;
}
public static function create()
{
return new FormHandler();
}
public function process($post_data)
{
if($this->captcha === true)
{
$res = $this->validate_captcha($post_data);
if($res !== true)
{
return $res;
}
}
if($this->recaptcha !== null &&
$this->recaptcha->isEnabled())
{
if($this->recaptcha->validate() !== true)
{
return json_encode([
'result'=>'recaptcha_validation_failed',
'errors'=>['captcha'=>'ReCaptcha Validation Failed.']
]);
}
}
$this->validator->test($post_data);
//if(false == $this->validator->test($post_data))
if($this->validator->hasErrors())
{
return json_encode([
'result'=>'validation_failed',
'errors'=>$this->validator->getErrors(/*associative*/ true)
]);
}
if(!empty($this->emails))
{
foreach($this->emails as $email)
{
$this->mailer->addAddress($email);
}
$this->compose_mail($post_data);
if(!empty($this->attachments))
{
$this->attach_files();
}
if(!$this->mailer->send())
{
return json_encode([
'result'=>'error_sending_email',
'errors'=> ['mail'=> $this->mailer->ErrorInfo]
]);
}
}
return json_encode(['result'=>'success']);
}
private function validate_captcha($post)
{
#session_start();
if(empty($post['captcha']))
{
return json_encode([
'result'=>'captcha_error',
'errors'=>['captcha'=>'Captcha code not entered']
]);
}
else
{
$usercaptcha = trim($post['captcha']);
if($_SESSION['user_phrase'] !== $usercaptcha)
{
return json_encode([
'result'=>'captcha_error',
'errors'=>['captcha'=>'Captcha code does not match']
]);
}
}
return true;
}
private function attach_files()
{
foreach($this->attachments as $file_field)
{
if (!array_key_exists($file_field, $_FILES))
{
continue;
}
$filename = $_FILES[$file_field]['name'];
$uploadfile = tempnam(sys_get_temp_dir(), sha1($filename));
if (!move_uploaded_file($_FILES[$file_field]['tmp_name'],
$uploadfile))
{
continue;
}
$this->mailer->addAttachment($uploadfile, $filename);
}
}
private function compose_mail($post)
{
$content = "Form submission: \n\n";
foreach($post as $name=>$value)
{
$content .= ucwords($name).":\n";
$content .= "$value\n\n";
}
$this->mailer->Body = $content;
}
}
Just a school boy MX error in the end. Hadn't double checked the details were matching and had an old record in there causing confusion. Thanks for the help anyways!
If you say that sending emails to a normal Gmail free account account works but you use a G Suite account it won't most likely is some G Suite settings or the DNS Domain settings the G Suite Domain.
If is as you said the MX record then refer to this:
Have you tried to send email from gmail to G suite (or from any other client, it doesn't matter), does it work?
To check if you MX record point at Goolge you can use this toolbox, you only have to put your domain and select MX: https://toolbox.googleapps.com/apps/dig/#MX/
If there are none (the records are shown in Bold), then you need to add the MX record for the Google Server, which are the following:
ASPMX.L.GOOGLE.COM : priorità 1
ALT1.ASPMX.L.GOOGLE.COM : priorità 5
ALT2.ASPMX.L.GOOGLE.COM : priorità 5
ALT3.ASPMX.L.GOOGLE.COM : priorità 10
ALT4.ASPMX.L.GOOGLE.COM : priorità 10
Set up MX records for G Suite Gmail: https://support.google.com/a/answer/140034?hl=en
If this is not the issue then it may be something else, if I understood right your code is sending to an email, and not actually using the account to send via SMPT
right?
Are there any other errors?
Are you the Super Admin of this G Suite Account?
Have tried to enable Less secure Apps? Control access to less secure apps: https://support.google.com/a/answer/6260879?hl=en

PHP: Error handling without throwing an exception

I'm sending multiple emails from a PHP application, and I want to inform the user of the emails that failed to be sent.
What is the most elegant way to do the error handling when
I don't want to throw an exception that terminates the sending of the remaining emails
The call goes through several method calls
What I want is to get the $notificationSucceeded back from Suggestion::notifyDeletionToAll() to SuggestionController somehow nicely from all notifications.
The depth of the call stack made me doubt if returning it through all the methods is the most elegant way, especially when I already have a return value from Suggestion::cancel().
Is there a better way?
Controller:
class SuggestionController {
function cancelSuggestion($suggestionId)
{
$suggestion = new Suggestion();
$suggestion->fetch($suggestionId);
$suggestionDeleted = $suggestion->cancel();
print json_encode(array(
'status' => 'ok',
'suggestionDeleted' => $suggestionDeleted,
));
}
}
Suggestion class:
class Suggestion {
/**
* Cancels membership of current user in the suggestion
*/
public function cancel()
{
$this->cancelMembership();
if (!$this->hasAcceptedMembers()) {
$this->deleteAndNotify();
return true;
}
return false;
}
/**
* Deletes the suggestion and notifies all the users in it
*/
private function deleteAndNotify()
{
$this->notifyDeletionToAll();
DB::inst()->query("DELETE FROM suggestions WHERE id = {$this->id}");
}
/**
* Notifies about the deletion of the suggestion to all members (users in the suggestion)
*/
private function notifyDeletionToAll()
{
$result = DB::inst()->query("SELECT user_id FROM suggestions_users
WHERE suggestion_id = {$this->id}");
while ($member_id = DB::inst()->fetchFirstField($result)) {
$member = new User();
$member->fetch($member_id);
$notificationSucceeded = $member->notifySuggestionDeleted($this);
}
}
}
I can't understand your question clearly. but i hope this will help you
$successfully_sent_arr = array();
$failure_notsent_arr = array();
if($mail->Send())
{
$successfully_sent_arr[] = $to_email;
//Once the last loop occurs, update this array in database
}
else
{
$failure_notsent_arr[] = $to_email;
//Once the last loop occurs, update this array in database
}

AJAX calls return results in a (seemingly) synchronous manner

All,
Background of how problem was detected
My question concerns the performance of a web app, mainly the index page. I noticed the problem when I was giving a demonstration at a local branch of my company that has slow internet (I don't know the exact speeds, or ping rate) judged by the fact that Google took about 10 seconds to load. My index page took ~10-20 times longer to load. I was under the assumption that my app did most of the work on the server side (as php is making all of the database queries...). But this led me to look at the network tool of Chrome and see the latency times of these 4 divs being loaded by ajax (I'll elaborate in a bit). Interestingly, the scripts being called appear to run sequentially, but not necessarily in the order I invoked the ajax calls (sometimes they do, other times they don't).
What are these divs / ajax requests?
Here is a code snippets of a request:
Yii::app()->clientScript->registerScript('leftDiv', '
$( "#left_dash" ).load(
"'.$this->createUrl("/site/page?view=leftDashLoad") .'",
function(){
$("#left_dash p a").click(function() {
$(this).parent().parent().find("div.scroll100").slideUp();
$(this).parent().next().stop(false, false).slideDown();
});
$("p:first-child").next().slideDown();
}
);
' );
Here is the page requested:
$this->widget('widgets.ScrollList',array(
'condition'=>
function($muddJob,$scrollList)
{
$job = $muddJob->Job;; //returns a job or empty array
if(!empty($job) )
{
if( $muddJob->uploadArtwork == null && $muddJob->uploadData == null ) {
array_push($scrollList->_models,$job);
$scrollList->columnValues = array($muddJob->jobDescription,$muddJob->dropDate1);
return true;
}
}
return false;
},
'columns' => array('col1'=>"MuddJob#",'col2'=>"Desc",'col3'=>"Dealer Name"),
'name'=> "Print New Ticket",
'muddJobs' => $currentExchanges->getCurrentMuddExchanges(),
)
);
Imagine that page (the page that ajax has called) having 6 similar declarations that create widgets. The goal is to return html to put back in place of a loading gif on the index page.
Here is the scroll widget:
<?php
Yii::import('widgets.ScrollListBase');
include_once Yii::app()->extensionPath . "/BusinessDay.php";
class ScrollList extends ScrollListBase
{
private $_content;
public $columns = array();
public $columnValues;
private $_listInfo;
public $name;
public $_models = array();
public $condition;
public $muddJobs; //object to pass
public $jobsMailingTodayArray = array();
public function init()
{
//$this->init();
$this->_listInfo = $this->generateListInfo($this->columns);
//$muddJobs = $this->getCurrentMuddExchanges();
$listInfo = $this->newScrollList($this->muddJobs);
$contents = $this->createContent($listInfo,$this->name);
$this->_content = $contents[0];
// $this->_fullTableContent = $contents[1];
//$this->_listInfo = $contents[2];
}
public function run()
{
//if($this->data['isVisible'])
echo $this->_content;
Yii::app()->session["exploded_content_{$this->name}"] = $this->_models;
}
private function newScrollList($muddJobs)
{
$listInfo = $this->_listInfo;
$tempCount = 0;
foreach($muddJobs as $muddJob)
{
$condition = $this->condition;
if($condition($muddJob,$this) && empty($this->jobsMailingTodayArray) ) //if no job exists for the muddExchange...
{
$tempArray = $this->createWidgetLinks($tempCount,$listInfo,$muddJob,$this->columnValues);
$listInfo = $tempArray[0];
$tempCount = $tempArray[1];
}
elseif ( !empty($this->jobsMailingTodayArray ) )
{
foreach ($this->jobsMailingTodayArray as $jobMailingToday) //change to for loop over the length of the jobsMailingToday
{
$tempArray = $this->createWidgetLinks($tempCount,$listInfo,$muddJob,$this->columnValues);
$listInfo = $tempArray[0];
$tempCount = $tempArray[1];
}
$this->jobsMailingTodayArray = array();
}
}
return array($listInfo,$tempCount);
}
}
?>
Here is it's parent:
<?php
class ScrollListBase extends CWidget
{
private $content = "<p>";
private $divDeclaration = "<div class='scroll100'>\n<table class='quickInfoTable'>\n<thead>\n";
private $headTag = "<th>";
private $headTagClose = "</th>\n";
private $theadTagClose = "</thead>\n";
private $bodyTag = "<tbody>\n";
private $listInfo = "<div class='scroll100'>\n<table class='quickInfoTable'>\n<thead>\n<th>Job#</th>\n<th>Package#</th>\n<th>Entry Date</th>\n</thead>\n<tbody>\n";
/**
* Initializes the widget.
*/
public function createContent($listInfo,$name)
{
$largeHref = Yii::app()->request->baseUrl . '/index.php/site/fullTableView';
$this->content .= "<span class='badge' >{$listInfo[1]} </span> <a href='#'>{$name} </a> <a href='$largeHref/Name/{$name}'> <small>(view larger)</small> </a> </p>";
if( $listInfo[1] > 0 )
{
// $this->fullTable .= substr($listInfo[0],22);
// $this->fullTableContent= $this->fullContent .= $this->fullTable . "</tbody>\n</table>\n</div>";
$this->content .= $listInfo[0] . "</tbody>\n</table>\n</div>";
}
return array($this->content);
}
//Helper Methods
/**
*
* #param type $attributeArray. send an accociative array
* #return type = either a job or an empty array
*/
protected function getJobByAttributes($attributeArray)
{
return Jobs::model()->with('MuddExchange')->findByAttributes($attributeArray);
}
protected function createWidgetLinks($tempCount,$listInfo,$muddJob,$columnValues,$url="/MuddExchange/")
{
$tempCount++;
$viewIndex = $muddJob->exchange_id;
$model = $muddJob;
$job = $muddJob->Job;
if ( isset($job ))
{
$model = $job;
$url = "/Jobs/";
$viewIndex = $model->job_id;
}
$link = CHtml::link("$model->jobNumber",array("{$url}{$viewIndex}"));
$listInfo .= "<tr>\n<td>$link</td>\n";
foreach ($columnValues as $columnValue)
{
$listInfo .= "<td>{$columnValue}</td>\n";
}
$listInfo .= "</tr>";
return array($listInfo,$tempCount);
}
protected function getListInfo()
{
return $this->listInfo;
}
/**
* Takes an array of strings to generate the column names for a particular list.
* #param array $heads
* #return string
*
*/
protected function generateListInfo($heads)
{
//<th>Job#</th>\n<th>Package#</th>\n<th>Entry Date</th>\n</thead>\n<tbody>\n";
$htmlScrollStart = $this->divDeclaration;
foreach ($heads as $tableColumn => $name)
{
$htmlScrollStart .= $this->headTag . $name . $this->headTagClose;
}
$htmlScrollStart .= $this->theadTagClose . $this->bodyTag;
return $htmlScrollStart;
}
public function calculateDueDate($jobsMailDate,$job)
{
$package = PackageSchedule::model()->findByAttributes(array('package_id'=>$job->packageID));
$projectedDays = $package->projected_days_before_mail_date;
$dropDate1 = $jobsMailDate->projected_mail_date;
$dropDate = wrapBusinessDay($dropDate1); //use this for actual command...
$toSec = 24*60*60;
$dayInt =0;
$secDropDate = strtotime($dropDate1);
do{
$dayInt +=1;
$daysInSec = ($dayInt) * $toSec ;
$secGuessDueDate = $secDropDate - $daysInSec;
$dueDate = date('Y-m-d',$secGuessDueDate);
$difference = $dropDate->difference($dueDate);
}while( $difference != $projectedDays);
return $dueDate;
}
}
?>
Why I think this behavior is odd
The whole slow internet thing is a beast in and of itself, but I don't think that is in the scope of StackOverflow. I'm more concerned about the loading of these divs. The div that loads last, i.e., takes on average 1.5 to 2 seconds, is an ajax request to a page that creates a single widget. The logic behind it is here:
<?php
include_once Yii::app()->extensionPath . "/CurrentExchanges.php";
$currentExchanges = Yii::app()->session['currentExchanges'];
$this->layout = 'barebones';
$this->widget('widgets.ScrollList',array(
'condition'=>
function($muddJob,$scrollList)
{
if ($muddJob->dropDate1 != null && $muddJob->dropDate1 != '0000-00-00')
{
$job = $muddJob->Job;;
if(!empty($job) && $job->packageID != null) //if job exists for the muddExchange and has a package
{
if($job->uploadArtwork == null )
{
$jobsMailDate = JobsMailDate::model()->findByAttributes(array("job_id"=>$job->job_id,'sequence_num'=>1));
if(!empty($jobsMailDate))
{
$calculatedDueDate = $scrollList->calculateDueDate($jobsMailDate,$job);
if (strtotime($calculatedDueDate) <= strtotime(date("Y-m-d")) )
{
array_push($scrollList->_models , $job);
$scrollList->columnValues = array($muddJob->jobDescription,$muddJob->dropDate1,$jobsMailDate->projected_mail_date);
return true;
}
}
}
}
}
return false;
},
'columns' => array('col1'=>"MuddJob#",'col2'=>"Desc",'col3'=>"Drop Date", 'col4' =>'Projected Drop Date'),
'name'=> "Artwork Due Today",
'muddJobs' => $currentExchanges->getCurrentMuddExchanges(),
)
);
?>
The calculateduedate method makes 2 additional calls to the server.
What I'm failing to understand is why the left div (with the most proccessing to do) is usually the first to return and the artLoad is usually the last to load (by a substantial difference). Here are some times returned by chromes network tool:
leftDashLoad: 475ms
rightDashLoad: 593ms
dataLoad: 825ms
artLoad: 1.41s
dataLoad: 453ms
rightDashLoad: 660ms
leftDashLoad: 919ms
artLoad: 1.51s
rightDashLoad: 559ms
leftDashLoad: 1.17s
dataLoad: 1.65s
artLoad: 2.01s
I just can't fathom why the left/right dashloads return so much faster than the artLoad. The code for artLoad and dataLoad are nearly identical save the actual comparison (the one if statement). If this were truly asynchronous, I'd expect the order to be art/dataLoad, rightDashLoad and leftDashLoad based purely on the amounts of computation done in each page. Perhaps the server isn't multithreading, or there is some weird configuration, but if that were the case, I don't see why the effects of the loading would be hit so hard by slow internet.
If I have overlooked something obvious or failed to use google appropriately, I do apologize. Thanks for any help you can offer!
Language/other tech info
The app was developed using the Yii framework. PHP 5.3. Apache Server. INNODB Tables. Server is hosted in dreamhost.
Update
I've changed the view page so that the ajax calls are now calling a controller action. It seems to have made the loading times more similar (asynchronous?) on my local dev environment, but really slowed it down on the QA environment (hosted on dreamhost...). Here is screen shot of the local network tools info:
dev environment
and the qa site (note, that the databases have about the same amounts of data...)
qa environment
Thoughts? It seems to me that my original problem may be solved, as the return times (on my local dev) look more like I expect them to.
Also, my own ignorance as to how NetBeans debugs was playing a part in this synchronous loading thing as xdebug is using a session. I believe this was forcing the ajax calls to wait their turn.
Thanks to #Rowan for helping me diagnose this strange behavior. PHP was trying to request a session before the session was closed in hopes to prevent a race hazard. There were session requests in my code, and there was a session started by my IDE (NetBeans). Removing all references to sessions in the ajax called pages and having ajax call actions that use renderPartial() proved to return the expected behavior (and much better code IMO). Let me know how to properly thank users in terms of StackOverflow (can I up comments, or what is there available? ). Thank you all!

Grab SearchEngine results as much as possible for a survey

For a survey I am trying to grab google search result from my php page. I grab six results then i click next button to get next page records.but after 9 pages e.g. 64 results it gives following error:
stdClass Object
(
[responseData] =>
[responseDetails] => out of range start
[responseStatus] => 400
)
I just want as much data as possible. i dont mind if it is google search engine or any other search engine. but for accurate result of survey I want large number of resultset. Can anybody knows how can i do that ?
Is it possible to grab results through cron ? Is there any other way ?
ATTENTION
Google tries to prevent scraping and so servers will be blocked and requests will be dropped when they suspect scraping. So you can use this if you occassionally need to get some google results.
Check google-scraper.squabbel.com for a proxy based scraper and more information on googles blocking mechanism. Its also against their policy and thus illigal.
The google api will not allow for more then 64 results, so if you need more you need to scrape the site yourself. As it was a fun project, I have created a class to do this for you.
It requires the free PHP Simple HTML DOM Parser so you need download this code aswell.
it will output an array like
array(100) {
[0]=>
array(3) {
["title"]=>
string(67) "Online Tests - Online aptitude tests for interview, competitive ..."
["href"]=>
string(36) "http://www.indiabix.com/online-test/"
["description"]=>
string(168) "Online aptitude tests for competitive examination, entrance examination and
campus interview. Take various tests and find out how much you score before
you ... "
}
[1]=>
array(3) {
["title"]=>
string(37) "Test your English - Cambridge English"
["href"]=>
string(50) "http://www.cambridgeenglish.org/test-your-english/"
["description"]=>
string(179) "Test Your English. This is a quick, free online test. It will tell you which Cambridge
English exam may be best for you. Click 'Begin Test' and answer each of the ... "
}
//removed for better visibility
}
How to use:
//start the scraper for google.com (english results)
$gs = new GoogleScraper();
//start the scraper for google.nl (dutch results)
//$gs = new GoogleScraper('https://www.google.nl');
//set your search query
$gs->SearchQuery('online exams');
//start loading the pages. You can enter any integer above 0
$gs->LoadPages(10);
//dump the results, but its just an array so you can also do other things with it.
echo '<pre>';
var_dump($gs->GetResults());
echo '</pre>';
?>
And then the GoogleScraper.php
<?php
require_once('simple_html_dom.php');
class GoogleScraper
{
private $_results;
private $_baseUrl;
private $_searchQuery;
private $_resultsPerPage;
/**
* constructor
* I use the constructor to set all the defaults to keep it all in one place
*/
final public function __construct($baseUrl='')
{
$this->_results = array();
$this->_resultsPerPage = 100;
if (empty($baseUrl)) {
$this->_baseUrl = 'https://www.google.com';
} else {
$this->_baseUrl = $baseUrl;
}
}
/**
* cleanup
*/
final public function __destruct()
{
unset($this->_results);
unset($this->_baseUrl);
unset($this->_searchQuery);
}
/**
* Set the query
*/
final public function SearchQuery($searchQuery)
{
if (!(is_string($searchQuery) || is_numeric($searchQuery)))
{
throw new Exception('Invalid query type');
}
$this->_searchQuery = $searchQuery;
}
/**
* Set the number of results per page
*/
final public function ResultsPerPage($resultsPerPage)
{
if (!is_int($resultsPerPage) || $resultsPerPage<10 || $resultsPerPage>100)
{
throw new Exception('Results per page must be value between 10 and 100');
}
$this->_resultsPerPage = $resultsPerPage;
}
/**
* Get the result
*/
final public function GetResults()
{
return $this->_results;
}
/**
* Scrape the search results
*/
final public function LoadPages($pages=1)
{
if (!is_int($pages) || $pages<1)
{
throw new Exception('Invalid number of pages');
}
if (empty($this->_searchQuery))
{
throw new Exception('Missing search query');
}
$url = $this->_baseUrl . '/search?num='.$this->_resultsPerPage.'&q=' . urlencode($this->_searchQuery);
$currentPage = 1;
while($pages--) {
if ($content = $this->LoadUrl($url)) {
/*
Load content in to simple html dom
*/
$html = new simple_html_dom();
$html->load($content);
/*
Find and handle search results
*/
$items = $html->find('div#ires li');
foreach($items as $item) {
/*
Only normal search results have this container. Special results like found images or news dont have it.
*/
$check = $item->find('div.s');
if (count($check)!=1) {
continue;
}
$head = $item->find('h3.r a', 0);
$result['title'] = $head->plaintext;
/*
If we dont have a title, there is no point in continuing
*/
if (empty($result['title'])) {
continue;
}
$result['href'] = $head->href;
/*
Check if we can parse the URL for the actual url
*/
if (!empty($result['href'])) {
$qs = explode('?', $result['href']);
if (!empty($qs[1])) {
parse_str($qs[1], $querystring);
if (!empty($querystring['q'])) {
$result['href'] = $querystring['q'];
}
}
}
/*
Try to find the description
*/
$info = $item->find('span.st', 0);
$result['description'] = $info->plaintext;
/*
Add the results to the total
*/
$this->_results[] = $result;
}
/*
Find next page
*/
$url = $this->_baseUrl . '/search?num='.$this->_resultsPerPage.'&q=' . urlencode($this->_searchQuery) . '$start=' . ($currentPage*$this->_resultsPerPage);
} else {
throw new Exception('Failed to load page');
}
$currentPage++;
}
}
/**
* Load the url
*/
final private function LoadUrl($url)
{
if (!is_string($url))
{
throw new Exception('Invalid url');
}
$options['http'] = array(
'user_agent' => "GoogleScraper",
'timeout' => 0.5
);
$context = stream_context_create($options);
$content = file_get_contents($url, null, $context);
if (!empty($http_response_header))
{
return (substr_count($http_response_header[0], ' 200 OK')>0) ? $content : false;
}
return false;
}
}
?>
Check this PHP Fiddle to see it in action. Because this could be used quite often from this server, there is a chance for 503 errors from google.
You should add a sleep(1) as cooldown between calls, otherwise might get bebanned.
Did you consider going the official way and getting a google API key?

Categories