How to fetch all inbox messages in php in a dedicated server? - php

Question

In our application we need to connect to the user email inbox and fetch all of messages.
The application will run in a dedicated server by Ubuntu OS.
I have ran some test in my pc by using imap extension (imap_open, imap_fetchheader, imap_fetchbody, ...).
There are multiple problems in using this approach:
1- The connection and fetching speed is very slow.
2- The php script will time out in case of many messages exists in inbox.
3- The application is multiuser and the number of requests is high.
My search results to solve above problems:
1- We have to rent a static IP from google or other email servers for imap connections !
2- We have to use PHP CLI to fetch orders from database and fetch inbox messages.
3- Multithreading in php.
Summary:
I want to write a bot in php to Connect to mail servers everyday. What connection and fetching way do you suggest? (library, method, language and etc). Thanks
**UPDATE : The code that I had ran to fetch 30 first messages:
$mbox = imap_open('{imap.gmail.com:993/imap/ssl/novalidate-cert}', 'myemailaddress', 'mypassword');
$count = imap_num_msg($mbox);
for($i = $count; $i >= $count - 30; $i--){
$header = imap_fetchheader($mbox, $i);
/* process header & body */
}
imap_close($mbox);
This code works well but when I change imap_fetchheader to imap_fetchbody it takes more time.
Update :
Based on the arkascha answer, I worked on my design and architecture and I found out the low speed reason is in the connections to the mail server. Because this is a I/O bounded task, I cached the imap connection to each mail server and the hole speed improved but not so much.

Answer 1

Sorry, but your search results are simply wrong.
you do not need a static IP address to fetch messages from an imap server. And certainly not one "rented from google". Why should that be required?
I see no reasons why the cli variant of php should be better suited here. In contrary, it is less efficient, since it has a much bigger startup load bacause a process must be spawned for each and every request.
why should you need multithreading? What benefits should it offer for this situation? Multithreading is only of interest for interactive stuff where responsiveness might be an issue. And even then this can be the wrong idea.
I myself implemented an imap client a few times and ran into no such problems. You certainly can implement a robust and efficient solution based on the phps imap extension. Speed of connection and transfer depend on many details, I doubt that php imap implementation is the cause of your problems. There might be additional issues which lead you to your statements, but you do not specify such in your question.
In general you should never fetch a huge list of data in one go and then process it. That is simply really bad design which cannot scale in any way. Instead you should always process lists in a sequential strategy: fetch a single data unit and process it. Then head on to the next. That way your memory footprint stays small, you prevent hitting php limits. This also enables you to process the whole list in smaller chunks (or even one message at a time) which prevents you from hitting phps execution time limit or a timeout on client side. These are general successful implementation patterns which are the base of a robust processing implementation which also apply to email messages.
To answer your bottom line question:
nothing speaks against using phps imap extension
use a cron job for this and do it on a regular base, not just once a day
invest into planning a clean architecture before starting the implementation

Answer 2

I might be biased but I'd use Zend Framework with Zend_Mail_Storage classes: http://framework.zend.com/manual/1.7/en/zend.mail.read.html You get a couple of options like switching to directly accessing mbox or maildir files if possible or getting closer to the underlying protocol and hack something in the zend_mail_protocol_imap to improve speed.