I've been debugging some Soap requests we are making between two servers on the same VLAN. The app on one server is written in PHP, the app on the other is written in Java. I can control and make changes to the PHP code, but I can't affect the Java server. The PHP app forms the XML using the DOMDocument objects, then sends the request using the cURL extension.
When the soap request took longer than 5 minutes to complete, it would always wait until the max timeout limit and exit with a message like this:
Operation timed out after 900000 milliseconds with 0 bytes received
After sniffing the packets that were being sent, it turns out that the problem was caused by a 5 minute timeout in the network that was closing what it thought was a stale connection. There were two ways to fix it: bump up the timeout in iptables, or start sending KeepAlive packets over the request.
To be thorough, I would like to implement both solutions. Bumping up the timeout was easy for ops to do, but sending KeepAlive packets is turning out to be difficult. The cURL library itself supports this (see the --keepalive-time flag for the CLI app), but it doesn't appear that this has been implemented in the PHP cURL library. I even checked the source to make sure it wasn't an undocumented feature.
So my question is this: How the heck can I get these packets sent? I see a few clear options, but I don't like any of them:
Write a wrapper that will kick off the request by shell_execing the CLI app. This is a hack that I just don't like
Update the cURL extension to support this. This is a non-option according to Ops.
Open the socket myself. I know just enough to be dangerous. I also haven't seen a way to do this with fsockopen, but I could be missing something.
Switch to another library. What exists that supports this?
Thanks for any help you can offer.
Related
The problem is I have to use curl and sometimes the curl requests take a long time because of the timeouts. I have set the timeouts to 1 second so no request should take more than 1 second but still the server is unable to process other php requests.
My question is how many concurrent scripts(running at the same time) can nginx/php-fpm handle. What I see is that a few requests lasting 1 second make the whole server unresponsive. What are the settings that I can change so more requests can be processed at the same time?
Multicurl is indeed not the solution to your probleme, but asynchrousity probably is. I am not sure that the solution is tweaking Nginx. It would scale better if you were to consider one of the following options :
You can abstract Curl with Guzzle http://docs.guzzlephp.org/en/latest/ and use their approach to async call and promises.
You can use Gearmand http:/gearman.org/getting-started/ which will enable you to send an async message to a remote server which will process the instruction based on a script you register to your message. (I use this mechanism for non blocking logging)
Either way, your call will be made in milliseconds and won't block your nginx but your code will have to change a little bit.
Php-curl did not respond in a timely manner because of DNS.
The problem was that I had to access files from a CDN but the IP behind the domain changed frequently and unfortunately curl keeps a DNS cache.
So from time to time it would try to access files from IPs that were not valid anymore, but they were still in the DNS cache of php-curl.
I had to drop php-curl completely and use a plain file_get_contents(...) request. This completely solved the problem.
I have a PHP site in which I make an ajax call , in that ajax call I make call to an API that returns XML and I parse it, The problem it sometimes the xML is so huge that it takes many time, The load balancer in EC2 have timeout value of 20 minutes, so If my call is greater than this I get 504 Error, How can I solve this issue? I know its a server issue but how I can solve this?I dont think php.ini is helpful here
HTTP is a stateless protocol. It works best when responses to requests are made within a few seconds of the request. When you don't respond quickly, timeouts start coming into play. This might be a timeout you can control (fcgi process timeout) or one you can't control (third party proxy, client browser).
So what do you do when you have work that will take longer than a few seconds? Use a message queue of course.
The cheap way to do this is store the job in a db table and have cron read from the table and process the work. This can work on a small scale, but it has some issues when you try to get larger.
The proper way to do this is use a real message queue system. Amazon has SQS, but could just as well use Gearman, zeroMQ, rabbitMQ, and others to handle this.
Is there a way to efficiently set a timeout on an https request in php?
There is a solution with curl simulating a SOAP request, however it is significantly slower.
Benchmarking it against a slow service shows an average of 6.5 seconds instead of 5.5 seconds the SOAP Client takes for the same request.
There is an alternative of stream socket
(implemented by Zend Framework on ZendHttpClientAdapterSocket) however the
stream_set_timeout function does not seem to work on https connections.
Please note that the issue is the read timeout (time to get a response) and not the connect timeout (time to do the handshake) which works on both http and soap requests
Finding a solution to make curl faster would resolve the issue as well.
USER STORY
I am making requests on an external https soap webservice using zend soap client.
The service usually responds on average on 5.5 seconds when the network is ok.
When there are network issues however, some requests take up to 7 to 8 minutes
consuming server resources.
I can use curl and force a timeout and then i am solving my problems when there are network issues with the webservice.
However my average response time goes up to 6.5 seconds when the network is ok
The business requirement suggests that requests that take longer than 30 seconds should rather be dropped in order to ensure stability of the system.
That depends on what you're using other than cURL.
If you're using streams you can just use stream_set_timeout (which sets the read-write timeout).
The connect timeout you can specify in fsockopen or however you create your stream.
See if you can specify a read-write timeout in your SOAP client?
So a friend and I are building a web based, AJAX chat software with a jQuery and PHP core. Up to now, we've been using the standard procedure of calling the sever every two seconds or so looking for updates. However I've come to dislike this method as it's not fast, nor is it "cost effective" in that there are tons of requests going back and forth from the server, even if no data is returned.
One of our project supporters recommended we look into a technique known as COMET, or more specifically, Long Polling. However after reading about it in different articles and blog posts, I've found that it isn't all that practical when used with Apache servers. It seems that most people just say "It isn't a good idea", but don't give much in the way of specifics in the way of how many requests can Apache handle at one time.
The whole purpose of PureChat is to provide people with a chat that looks great, goes fast, and works on most servers. As such, I'm assuming that about 96% of our users will being using Apache, and not Lighttpd or Nginx, which are supposedly more suited for long polling.
Getting to the Point:
In your opinion, is it better to continue using setInterval and repeatedly request new data? Or is it better to go with Long Polling, despite the fact that most users will be using Apache? Also, it possible to get a more specific rundown on approximately how many people can be using the chat before an Apache server rolls over and dies?
As Andrew stated, a socket connection is the ultimate solution for asynchronous communication with a server, although only the most cutting edge browsers support WebSockets at this point. socket.io is an open source API you can use which will initiate a WebSocket connection if the browser supports it, but will fall back to a Flash alternative if the browser does not support it. This would be transparent to the coder using the API however.
Socket connections basically keep open communication between the browser and the server so that each can send messages to each other at any time. The socket server daemon would keep a list of connected subscribers, and when it receives a message from one of the subscribers, it can immediately send this message back out to all of the subscribers.
For socket connections however, you need a socket server daemon running full time on your server. While this can be done with command line PHP (no Apache needed), it is better suited for something like node.js, a non-blocking server-side JavaScript api.
node.js would also be better for what you are talking about, long polling. Basically node.js is event driven and single threaded. This means you can keep many connections open without having to open as many threads, which would eat up tons of memory (Apaches problem). This allows for high availability. What you have to keep in mind however is that even if you were using a non-blocking file server like Nginx, PHP has many blocking network calls. Since It is running on a single thread, each (for instance) MySQL call would basically halt the server until a response for that MySQL call is returned. Nothing else would get done while this is happening, making your non-blocking server useless. If however you used a non-blocking language like JavaScript (node.js) for your network calls, this would not be an issue. Instead of waiting for a response from MySQL, it would set a handler function to handle the response whenever it becomes available, allowing the server to handle other requests while it is waiting.
For long polling, you would basically send a request, the server would wait 50 seconds before responding. It will respond sooner than 50 seconds if it has anything to report, otherwise it waits. If there is nothing to report after 50 seconds, it sends a response anyways so that the browser does not time out. The response would trigger the browser to send another request, and the process starts over again. This allows for fewer requests and snappier responses, but again, not as good as a socket connection.
I need to develop HTTP push using php, it sends back a message to the browser every 20 seconds.. that i need to display on the server
Is there any way of implementing http push in php with out using any sockets or library?
Sure: don't let your PHP script terminate for as long as you want the connection to remain open. Since you are generating output every 20 sec, the connection is much less likely to timeout.
There are many ways to achieve this including blocking (e.g. sleep() below) or busy waiting, but the simplest solution would be:
while( true ){
generate_output();
sleep(20);
}
You may need to handle unexpected connection termination on the client-side, but this is the jist of it. Check your system configuration to see how many open connections you can handle. (In Apache, see MaxClients)
As for the client-side, see this answer https://stackoverflow.com/a/7953033/329062
The only way to do this is to use WebSockets, "Long Polling", or an AJAX request every 20 seconds (even if there is nothing to update). Your solution will depend on which browsers you want to support. For example, newer browsers support Web Sockets, but older browsers don't. I would recommend researching Web Sockets. They are a great solution for this type of situation.