I have setup a local server on a regular desktop (not a server desktop) and have 3-4 client machines accessing the local web application I developed from the server via a WIFI router (server is connected to router via cable. All clients via WIFI).
When two of the clients are connected to the application all is well, but when a third (or more) machine joins in there are periods where each machine does not get any service from the server (the application webpage remains loading until I manually reset Apache on the server via services). At times the server responds when one of the clients refresh their page but most of the time I have to perform a reset of the Apache server.
This occurs roughly once an hour on average (6 hours of continuous usage) as the clients are using the application.
Server is running Windows 7 and Apache v2.4 with PHP v5.4
Server and all client machines are running AVG internet security
Firewall is handled by AVG Internet Security
Is this issue due to the code in my application, desktop not being able to manage requests like a server machine, antivirus or a mix of the three?
If so, how can I set-up the server to reset automatically?
Thanks
UPDATE
It is a application where users write reports on files after reviewing information
-Frequent sql requests for file data
-No images
-Some pages contain multiple sql queries that represent the page content
-Network has no internet connection
-Code does not make requests for external information from the internet
-All client machines run the application on Google Chrome web browser
But it rarely happens but sometimes the amount of connection is restricted by the third-party interface being used by the application. We are unable to figure out the reason unless having more details like what content of your app, and the error code apache or HTTP returning.
This kind of situations is difficult to track, especially on Windows where diagnostic tools aren't as readily available as on other platforms.
I suppose you can try and check the antivirus by either running server and clients with no antivirus at all for some hours, or disabling and re-enabling the antivirus when the hangup occurs.
Apart from that, you would need to pinpoint where the error occurs:
in the connection stage (Windows OS is the problem)
in the response stage (Apache is the problem - try fiddling with the child spawning parameters)
in the management stage (PHP is the problem - you can probably check this by changing the setup from PHP-as-a-module, and PHP-as-CGI-application)
in the response stage (that is, connection to the SQL server). You can check this by setting up some pages that use different combinations of session, database, and output buffering and see whether those pages remain reachable even when the application is hung up.
For an example of the last, if a page such as
<?php print date("H:i:s"); phpinfo(); ?>
remains reachable and correctly refreshes (that's why the date() command) even when the app does not respond, this demonstrates that Windows, Apache and PHP are "innocent", and either the SQL server has issues, or you do not interface with it correctly. It might be for example be the case (though unlikely in this instance) that the resident PHP module is accumulating connections to the SQL server and not releasing them, so that after a while you need to stop Apache (thereby freeing the module) and restart.
If this were the case, even if it's not a "real" fix, you can set up Apache so that all children die and are replaced after a small number of requests (once it was 150, but when leaks all but disappeared, I believe that the default became 0 -- Apache children no longer die. Check it out, I might well misremember).
Related
We have a product where a local linux machine "SCANNER" does some polling on a local network and then stuffs information into a database in our cloud server over an OpenVPN connection, using pg_query (server is running PostgreSQL 9.5.5).
There is a PHP (5.5.9) daemon on SCANNER that checks the database in a 'while' loop for work to be done. This has always worked great, and continues to work great on all of our client networks, except for one, which has recently developed a bizarre problem.
After they upgraded their firewall (a Checkpoint 5200, and as far as we can tell, all the rules are correct to allow traffic from SCANNER to our cloud server over the VPN), one query in one script hangs indefinitely. Here are the symptoms we have noted:
Most of the time, the query works fine and the script continues. Every once in a while tough, the pg_query() call blocks and never returns. It's not that there's an error; the call literally blocks forever (or many hours until we manually restart).
This query has been the same for a long time, and we have never had this problem at any of our other clients, nor at this client until they changed their firewall.
We can tell from watching the pg_stat_activity table on the cloud server that the query does make it to the cloud, and then sits in that table forever. In every case, pg_stat_activity.state='idle' and pg_stat_activity.waiting=false
During this time, we can still ping the cloud server from SCANNER over the VPN, and we can continue to successfully query its remote database from other scripts on SCANNER, and from SCANNER's command line.
This client happens to have two different SCANNER machines, on different subnets but behind the same firewall. This problem can occur at any time on either one, but doesn't necessarily occur at the same time on both (at least not with any statistical significance).
If we restart the daemon, the problem is temporarily resolved. But it usually recurs sometime between 2 seconds and a few hours later.
We are looking for any input that might solve the problem, whether it is related to our application or the firewall itself (which we have been given permission to modify as needed). Feel free to ask any clarifying questions.
Thanks in advance!
Check Point firewall has a lot of advanced threat protections, depending of acquired blades, the device can be blocking communication between your database and application. Try to use Check Point log tools (Tracker, SmartEvent or NGSE) to filter Firewall logs. Filter all events where your source or destination are your Scanner or database server IP adresses. If you get some drop ou block into your TCP Packets, logs will show that.
If your firewall topology uses Cluster configurations try to check your configuration, it could be misconfigured and database TCP session are active but packets are using another network path.
If you are using Check Point VPN Client, try to update it to last version and update your firewall device with last update packages (Take).
If the problem persists, get evidences to comprove that your communication without Check Point hardware doesn't stop and some evidences of your problem and ask to your Check Point solution provider to open a case with Check Point.
We solved the problem. Technically, we never figured out exactly why it was happening, but UDP packets were getting received by the VPN server in an incorrect order, or sometimes not at all. The firewall didn't give us any indication that it was doing this, so as best we can figure it was an unlogged side-effect of CheckPoint's multiple layers of threat protection.
We switched over to using OpenVPN over TCP instead of UDP, and that seems to have solved the problem. Hopefully we don't suffer any adverse effects from that in the long-term!
I have a network of one load balanced server (using nginx) lb1 which routes traffic between four web servers web1, web2, web3, web4. These four webservers are routed to using round-robin in nginx.
All servers are set to max_fails=1 and fail_timeout=5s, so when a server is down, it should be ignored fairly quickly if it is not online.
I should note that the average response time of the web pages from each web servers is around 50-150ms, if all four web servers are online. The issue arises when just ONE web server is offline. When one goes offline and a user tries to load another page, the response time varies anywhere from 50ms-25s. Yes, 25 seconds.
I am confused, because I would think that the round-robin and fail_timeout settings would make it so the offline server would be ignored.
Additional, possibly relevant notes: All four web servers are running apache with php5, and memcached is enabled between the four.
UPDATE
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
This is pretty strange to me, I have a server downstairs that hosts my websites and MySQL servers and it has been running for years without many issues. I have 2 routers bridged together behind my modem and my server is behind one of them. All other devices connect via WiFi. All of the proper ports are open on the router and I have users configured in MySQL that haven't changed and have been working fine this whole time.
So last night I was working on a project and I decided to sync everything with a backup on my SkyDrive. I have a scheduled backup for MySQL that runs at midnight (daily) and it just turned over to midnight so I decided to open my network and watch the file get populated before I sent a copy to my SkyDrive. After the backup was complete (which it did successfully), I was going to continue to work on my program but all of a sudden I can no longer connect via my local network to the MySQL server. I'm using PHP and my connection string never changed and all other MySQL admin tools don't connect. The live site works fine, so MySQL was definitely running and working but no remote connections were being accepted. Why is this happening all of a sudden?
Things I've tried :
I did notice that my logs were packed full of BINLOG errors so I turned off the binlog since I recently turned it on (a couple weeks ago).
Restarting MySQL
Turning off Windows Server 2008 firewall (temporarily)
Connecting from a different device (mobile phone, tablet), no luck
Temporarily allowing port 3306 on my router
Checked server logs for intrusion attempts, none present...
Setup :
PHP 5.4 on local machine and server
Windows Server 2008 Enterprise (on server...)
MySQL Version 5.5.25a
Does anyone have any clue as to what's going on here? I'm going to reboot my server when the load is low and see if this helps any, I will update this once it comes back online.
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
I have a small home server (Ubuntu+XAMPP) and 2 PHP scripts: server.php and client.php, which both communicate to each other via sockets.
When I run server.php / client.php on the same machine (localhost), it works fine. Also, when I run server.php on the server, and client.php on the same server but from other local PC (i.e. local_server_ip/client.php), all works fine as well.
However, when I run server.php on the server and client.php on the other PC on the same network (replacing localhost with local_server_ip_addr in the client.php script), it fails with the actively refused connection error.
All necessary ports are forwarded in the router. I guess it is kind of security block on XAMPP/Linux and can be eliminated by some configuration file. I replaced Deny from all in the New XAMPP security concept with Allow from all in httpd-xamp.conf file, but it still fails.
Any help would be much appreciated.
(PS: server/client scripts examples taken from http://i-novice.net/sokety-v-php/ )
UPD: Have modified port 8080 (the one is dedicated for sockets in my system) to XXXXX. All works fine!
In the event that this happens dependably, it actually implies that the machine exists however that it has no administrations listening on the predetermined port, or there is a firewall ceasing you.
In the event that it happens periodically - you utilized "now and then" - and retrying succeeds, it is likely in light of the fact that the server has a full 'overabundance'.
When you are holding up to be acknowledged on a listening attachment, you are put in a build-up. This overabundance is limited and short - estimations of 1, 2 or 3 are not bizarre - thus the OS may be not able to line your solicitation for the "acknowledge" to devour.
The build-up is a parameter on the listen capacity - all dialects and stages have essentially the same API in such manner, even the C# one. This parameter is frequently configurable in the event that you control the server, and is likely read from a few settings document or the registry. Research how to arrange your server.
In the event that you composed the server, you may have substantial preparing in the acknowledge of your attachment, and this can be better moved to a different specialist string so your acknowledge is constantly prepared to get associations. There are different structural engineering decisions you can investigate that alleviate lining up customers and handling them successively.
Notwithstanding whether you can build the server build-up, you do need retry rationale in your customer code to adapt to this issue - as even with a long build-up the server may be getting bunches of different demands on that port around then.
There is an uncommon probability where a NAT switch would give this blunder if its ports for mappings be depleted. I think we can toss this probability as a lot of a long shot however, since the switch has 64K concurrent associations with the same destination location/port before weariness.
If MySQL is dropping connections in a PHP application, and MySQL connection limit is set above the number of concurrent users in the application, which other factors can contribute to this behavior? Also, analyzing moodle logs (the only app running at these servers), yesterday I had 4 times more activity and it didn't drop anything, but today there were times where it was frustrating the number of dropped connections.
My main question here is why the database is rejecting connections when it didn't before with 4 times more activity (and everything it's the same, I didn't change anything in between).
Some background: I've got 2 servers contracted at my hosting:
shared server running Debian Linux/PHP 5.3 (FastCGI)
VPS running Debian Linux/MySQL 5.1.39
On this environment I'm running only moodle 1.9.12 (for the database connection using adoDB and persistent connections), php part on the shared server, database on the VPS. I suspect that, by PHP running on a shared server, other hosting accounts are affecting me (the database rejecting connections I mean, I really don't care about RAM/CPU).
Reading about the issue I've seen in places that persistent connections don't work well with PHP as CGI/FastCGI and that if both servers are in the local lan it really doesn't matter using persistent or not persistent connections because of the connection is going to be quick anyway. So now I'm using not persistent connections. I guess that may be part of the problem, but I fail to understand why it worked with more load. Which PHP/Apache settings are involved here?
Since your database and web server are on two different machines, there are two other possible causes: the network in between, and the network layer of the operating system.
If you did not change anything in your configuration and it worked with higher loads before, it is more likely to be an issue with the network connectivity between the two machines. Since the database machine is a VPS you also don't know how much real network load it is handling. If your ISP has competent support personnel (which unfortunately isn't always the case) it can't hurt to ask them if they have an explanation.
The same goes for your "shared" web-server. While it is unlikely, it is not impossible that it is a an issue of too many connections on that machine.
It would also help to know how exactly you are measuring dropped connections. If you are looking at the aborted-connections counter of mySQL, it is not necessarily a measure of an actual problem: http://dev.mysql.com/doc/refman/5.1/en/communication-errors.html. A user aborting a page load already may increase this counter.
If PHP throws an error, because it could not connect to the server or lost connection to the server during a query, then it is an issue.