slow php imap email fetching ( >10 seconds ) - php

I am building my own webmail client. Like Roundcube or Squirrelmail, for example. The problem is that my version is very slow, while Roundcube is fast and I cannot understand why is that (Roundcube's source ir very big and I am unable to dive in it..)
The goal - fetch last 50 messages from mailbox. My strategy:
Get number of messages in mailbox by imap_num_msg()
Make array of sequence numbers from max to (max-50)
For each sequence number I ran functions imap_header() and imap_fetchstructure()
It takes 10-15 seconds. It allows me to get each messages title, date, whether is has attachments or not, from, to and other information.
However, Roundcube displays the same info, but load time is only 3 seconds or so. My strategy seems to be very wrong. How can I do it faster? I'm pretty sure that it must be slow to ran imap_header and imap_fetchstructure for each sequence number, but I think there is no other way to get that information.. I'm doing something like this:
function getMessageBySequenceNumber($imapStream, $sequence_number){
$header = imap_header($imapStream, $sequence_number);
$structure = imap_fetchstructure($imapStream, $sequence_number);
/*
... some code parsing $structure to find out whether this emails has any attachments or not
*/
return [
'uid' => imap_uid($imapStream, $i),
'subject' => $header->subject,
'timestamp' => $header->udate,
'unseen' => $header->Unseen,
'star' => $header->Flagged,
'draft' => $header->Draft,
'size' => $header->size,
'has_attachments_bool' => $has_attachments_bool,
];
}
$imapStream = imap_open();
$first_sequence_number = imap_num_msg(); // lets imagine it returns 100
$last_sequence_number = $first_sequence_number-50;
$sequence_numbers = [100,99,88 ..., 51, 50];
$messages = [];
foreach($sequence_numbers as $sequence_number){
$messages[] = getMessageBySequenceNumber($imapStream, $sequence_number);
}
return $messages;

You are fetching the messages one-by-one. This means that your PHP code has to wait for the remote IMAP server to answer you, then your PHP code is going to process the (partial) response, send the data back to the server, etc.
Use an IMAP library which allows batched operations, and read RFC 3501 to understand how to use it.

Related

GMail API - Getting certain message headers or fields

I have successfully connected GMail API via G Suite account and service account. I can get a message list and I can retrieve messages by IDs. I'm working with PHP.
What I'm having problems with is to get for example the FROM or TO headers, SUBJECT or the snippet field.
$optParam = array('format' => 'metadata', 'metadataHeaders'=>['subject','from'], 'fields'=>['snippet','labelIds']);
$fullMessage = $service->users_messages->get($user, $id, $optParam);
This will return the snippet, but not the subject or from or the labelIds.
If I use the GMail "Try this API" and use the id of the message and use "snippet" in the "fields" entry, I just get the snippet back as:
{
"snippet": "Short snippet of the message"
}
If I use:
$optParam = array('format' => 'metadata', 'metadataHeaders'=>['subject','from','to']);
I do get the 3 headers, but I also get a lot more information, including the labels and snippet - about 3K for each message.
I just can't seem to be able to specify a small subset of the data. All I need is to show messages as a list with the subject, date/time, from/to.
I don't care so much about the amount of data, but it takes on average about 3.5 seconds to retrieve the data for just 14 message!
Is there a way to restrict this so I don't get all the "extra" data or speed the retrieval up somehow?
Sending the request would involve specifying the metadata keys as well as the parameter names of the fields you want to obtain. You can use an HTTP GET request with the URI to get the ‘to’, ‘from’, ‘subject’ and ‘snippet’ with https://www.googleapis.com/gmail/v1/users/me/messages/<MESSAGE_ID>?format=metadata&metadataHeaders=to&metadataHeaders=from&metadataHeaders=subject&fields=snippet%2C+payload%2Fheaders, which will also limit the headers you obtain.
In PHP you can use this:
$optParam = array('format' => 'metadata', 'metadataHeaders'=>['subject', 'from', 'to'], 'fields'=>'payload/headers,snippet');
Note that the fields parameter needs to be sent as a string and not an array.
Also be aware there is a known issue with the Gmail API where using the https://www.googleapis.com/auth/gmail.metadata scope will not return the snippet.
You’ll need to use https://www.googleapis.com/auth/gmail.readonly instead.
You can also make a batch of requests in one network call which will help speed up overall execution time as documented here.

PHP Azure SDK Service Bus returns Malformed Response

I'm working on trace logger of sorts that pushes log message requests onto a Queue on a Service Bus, to later be picked off by a worker role which would insert them into the table store. While running on my machine, this works just fine (since I'm the only one using it), but once I put it up on a server to test, it produced the following error:
HTTP_Request2_MessageException: Malformed response: in D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 1013
0 HTTP_Request2_Response->__construct('', true, Object(Net_URL2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:1013
1 HTTP_Request2_Adapter_Socket->readResponse() D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:139
2 HTTP_Request2_Adapter_Socket->sendRequest(Object(HTTP_Request2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2.php:939
3 HTTP_Request2->send() D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\Http\HttpClient.php:262
4 WindowsAzure\Common\Internal\Http\HttpClient->send(Array, Object(WindowsAzure\Common\Internal\Http\Url)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\RestProxy.php:141
5 WindowsAzure\Common\Internal\RestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\ServiceRestProxy.php:86
6 WindowsAzure\Common\Internal\ServiceRestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:139
7 WindowsAzure\ServiceBus\ServiceBusRestProxy->sendMessage('<queuename>/mes…', Object(WindowsAzure\ServiceBus\Models\BrokeredMessage)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:155
⋮
I've seen previous posts that describe similar issues; Namely:
Windows Azure PHP Queue REST Proxy Limit (Stack Overflow)
Operations on HTTPS do not work correctly (GitHub)
That imply that this is a known issue regarding the PHP Azure Storage libraries, where there are a limited amount of HTTPS connections allowed. Before requirements were changed, I was accessing the table store directly, and ran into this same issue, and fixed it in the way the first link describes.
The problem is that the Service Bus endpoint in the connection string, unlike Table Store (etc.) connection string endpoints, MUST be 'HTTPS'. Trying to use it with 'HTTP' will return a 400 - Bad Request error.
I was wondering if anyone had any ideas on a potential workaround. Any advice would be greatly appreciated.
Thanks!
EDIT (After Gary Liu's Comment):
Here's the code I use to add items to the queue:
private function logToAzureSB($source, $msg, $severity, $machine)
{
// Gather all relevant information
$msgInfo = array(
"Severity" => $severity,
"Message" => $msg,
"Machine" => $machine,
"Source" => $source
);
// Encode it to a JSON string, and add it to a Brokered message.
$encoded = json_encode($msgInfo);
$message = new BrokeredMessage($encoded);
$message->setContentType("application/json");
// Attempt to push the message onto the Queue
try
{
$this->sbRestProxy->sendQueueMessage($this->azureQueueName, $message);
}
catch(ServiceException $e)
{
throw new \DatabaseException($e->getMessage, $e->getCode, $e->getPrevious);
}
}
Here, $this->sbRestProxy is a Service Bus REST Proxy, set up when the logging class initializes.
On the recieving end of things, here's the code on the Worker role side of this:
public override void Run()
{
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
Client.OnMessage((receivedMessage) =>
{
try
{
// Pull the Message from the recieved object.
Stream stream = receivedMessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string message = reader.ReadToEnd();
LoggingMessage mMsg = JsonConvert.DeserializeObject<LoggingMessage>(message);
// Create an entry with the information given.
LogEntry entry = new LogEntry(mMsg);
// Set the Logger to the appropriate table store, and insert the entry into the table.
Logger.InsertIntoLog(entry, mMsg.Service);
}
catch
{
// Handle any message processing specific exceptions here
}
});
CompletedEvent.WaitOne();
}
Where Logging Message is a simple object that basically contains the same fields as the Message Logged in PHP (Used for JSON Deserialization), LogEntry is a TableEntity which contains these fields as well, and Logger is an instance of a Table Store Logger, set up during the worker role's OnStart method.
This was a known issue with the Windows Azure PHP, which hasn't been looked at in a long time, nor has it been fixed. In the time between when I posted this and now, We ended up writing a separate API web service for logging, and had our PHP Code send JSON strings to it over cURL, which works well enough as a temporary work around. We're moving off of PHP now, so this wont be an issue for much longer anyways.

Clickatell batch templates slow (PHP, http)

I'm trying to setup clickatell at the moment for sending out batch sms'. I've got it working but it's quite slow. About 20 seconds to send 5 test sms and 30 seconds for 10 test sms.
$nums = array(
"44-227811116" => "1",
"44-227819885" => "2",
"44-227819314" => "3",
"44-227815413" => "4",
"44-227819326" => "5"
);
//login
$url="https://api.clickatell.com/http/auth?api_id=xxxxx&user=xxxxx&password=xxxxx";
$page=Utilities::getWebPage($url);
//session
$clicksessionparts=explode(":", $page);
$clicksession=trim($clicksessionparts[1]);
//batch
$from=xxxxx;
$batchTemplate = urlencode("Test message #field1#");
$url="https://api.clickatell.com/http_batch/startbatch?session_id=$clicksession&template=$batchTemplate&from=$from&deliv_ack=1";
$page=Utilities::getWebPage($url);
$batchId=explode(":",$page);
$batchId=trim($batchId[1]);
foreach ($nums as $k => $v)
{
$start = new DateTime();
print_r($start->format("H i:s"));
$url="https://api.clickatell.com/http_batch/senditem?session_id=$clicksession&batch_id=$batchId&to=xxxxx&field1=$v";
$page=Utilities::getWebPage($url);
echo "<pre>";
print_r($page);
echo "</pre>";
$end = new DateTime();
print_r($end->format("H i:s"));
echo "<br><br>";
}
You should be able to submit over 100 messages per second to the HTTP API comfortably.
Creating HTTPS connections is a very slow process (compared to HTTP). If you want better performance with HTTPS, you will have to reuse the connections.
I am guessing that Utilities::getWebPage() is creating a new HTTPS connection each time? For PHP I would suggest you look at using cURL.
If you want to go another step further (I doubt you need to go this far), you can consider using curl_multi... Its a bit more work though and most people dont need so much speed (some find it easier just to use another API like the SMTP API so they have many messages in 1 email).
Also, you technically do not need to use the batch commands on the HTTP API to send your messages (unless you want to). You can send millions with just api.clickatell.com/http/sendmsg?.... in which case there is no need to do a start batch call.
With something like an SMTP API, you can put 100 000 messages in one email (if you need unique text per message, you would use the batch facility on that API).

what is the best way to fetch a list of hashes with Redis in php

I am building a notification system using Redis and predis (not mobile push notification, just a visual warning on a website).
For example, when user1 sends a message to user2, I create an entry in Redis.
On that type of event, I create a hash with all the information about the notification (date, content, sender...).
Then I add the hash key to a list that is specific to the user.
//creating the notification
notificationId = uniqid();
$this->redis->hmset($notificationId, array(
"sender" => "sender's Name",
"type" => "message",
"user_id" => "recipient's id",
"content" => "message content",
"date" => new \Datetime()
)
);
//adding the id of the notification in the user's message notifications list
$this->redis->lpush("messageList".$userId, $notificationId);
Then when I want to retrieve all the message notifications for the user I do :
$listName = "messageList:".$userId;
$arrNotifications = $this->redis->pipeline(function ($pipe) use ($listName) {
foreach ($pipe->getClient()->lrange($listName, 0, -1) as $key => $id) {
$arrNotif[] = $pipe->hgetall($id);
}
});
I get all the desired results with this method, but if the message list contains a couple thousands entries, the operation takes 0.5 seconds. It looks kinda slow, Redis is well known for being super fast.
So I am wondering if I am doing things correctly.
Any advice ?
Thanks
You may be much more satisfied with performance if you use phpredis.
It's a php C extension you have to compile in your server.
I started with predis too, and was glad to benchmark phpredis a better solution.

Keep messages from ACKing on get() using pecl-amqp

I'm attempting to use pecl-amqp for a project of mine. I'm having difficulties though with the ACK process. I need to manually ACK each message I receive off a queue, but the messages appear to be auto-ACKing when the message is retrieved.
I've set my queue to AMQP_NOACK and am using AMQPQueue->get(AMQP_NOACK) but none of it seems to have any affect, the messages are still removed from the queue without me sending AMQPQueue->ack().
If anyone has any experience with the pecl-amqp I would appreciate the help.
I have been having the same problem but have managed to get the acknowledge mechanism to work using the consume method. Using rabbitmqctl to list the entries in the queue it appears to work OK, though I seems to be getting the messages off the queue in a different order to that in which they were sent - which kind of defeats the object of a queue. My code is as follows:
// Create the queue to be:
// AMQP_DURABLE - messages will withstand a broker restart (i.e. they are written to disk).
// AMQP_NOACK - when consumed messages will not be marked as delivered until an explicit ACK is received.
$q->declare($queueName, AMQP_DURABLE | AMQP_NOACK );
// Bind it on the exchange to routing key.
$q->bind($exchangeName, $routingKey);
// Set the options for our consumption of the messages:
// Get a minimum of 0 msg.
// Get a maximum of 1 msg.
// Don't ACK the message on consumption i.e. explicitly acknoledge later.
$options = array(
'min' => 0,
'max' => 1,
'ack' => false
);
// Get the messages
$results_array = $q->consume($options);
// show the message
print_r($results_array);
$delivery_tag = $results_array[0]['delivery_tag'];
echo 'delivery_tag: [' . $delivery_tag . "].\r\n";
// Acknowledge receipt of the message.
$q->ack($delivery_tag);
For reference, the AMQP extension did not support the AMQP_NOACK correctly. You can get it to do what you want, but it isnt pretty. I am working on fixing that now. FYI, AMQP_NOACK means that you will not have to come back to acknowledge the message later, i.e. as soon as the server sends the message to the client, the server marks the message as ack'ed. There has been some confusion around this, so I wanted to clarify.

Categories