I am running a PHP script which basically tries to find matching names from MYSQL database and then assigns the same number to records with identical names.
My problem is that the number of records are around 1.5 million. The script always runs for about 14 hours every time and gives this error : mysql_query unable to save result set in xx on line xxx. and phpmyadmin gives this error #2008 out of memeory
Here is my php code
mysql_query("SET SQL_BIG_TABLES=1");
$res = mysql_query("SELECT company_name, country, id FROM proj")
or die (mysql_error());
while ($row = mysql_fetch_array($res, MYSQL_NUM)) {
$res1 = mysql_query("SELECT company_name, id FROM proj WHERE country='$row[1]'"+
"AND id<>'$row[2]'") or die ("here".mysql_error().mysql_errno());
while ($row1 = mysql_fetch_array($res1, MYSQL_NUM)) {
//My calculations here
}
}
Ok. Your query is incredibly inefficient. You say in a comment that there's 1.5 million rows.
Outside query creates a result set with all 1.5 million rows
Inside query creates a new result set with all 1.5 million rows, EXCEPT the row that has the same of the row you're looping on. So basically you're doing 1.5 million * (1.5 million - 1) rows =
2,249,998,500,000 = 2.25 trillion rows
In other words, not only incredibly inefficient - it's absolutely ludicrous. Given you seem to want to fetch only rows by country, why not do something like:
$sql1 = "SELECT DISTINCT country FROM proj";
$res1 = mysql_query($sql1) or die(mysql_error());
while($row1 = mysql_fetch_associ($res1)) {
$country = $row1['country'];
$escaped_country = mysql_real_escape_string($country);
$sql2 = "SELECT company_name, id FROM proj WHERE country='$country'";
$res2 = mysql_query($sql2) or die(mysql_error());
while ($row2 = mysql_fetch_assoc($res2)) {
... calculations ...
}
}
This'd fetch only 1.5 million + # of country records from the database, which is far far far less than the 2.3 trillion your version has.
Related
I am using the following php to display the number of records returned in a db search.
$sql = "SELECT COUNT(id) FROM authorsbooks WHERE author LIKE '%$searchquery%'";
$query = mysqli_query($dbc, $sql);
$row = mysqli_fetch_row($query);
$rows = $row[0];
$textline1 = "Your Search Returned (<b>$rows</b>) Records";
<?php echo $textline1; ?>
This seems to work fine.
However, I cannot get the total number of records in the actual db to display.
Can anyone explain a way of getting the total number of records in the database. Btw, I have tried using $total = mysqli_num_rows($query) but it keeps returning 1 as an answer. Thanks for any help.
For that you've to fire another SQL query. Like this,
$sql = "SELECT COUNT(id) FROM authorsbooks";
$query = mysqli_query($dbc, $sql);
$row = mysqli_fetch_row($query);
$rows = $row[0];
echo $rows; // will return total rows in database.
SELECT COUNT(*) FROM authorsbooks
It's true that $total = mysqli_num_rows($query) should return one row. When you do a SELECT COUNT(*) then the query returns 1 row telling you how many matches there were in the table.
I want to make a mysql query after a few seconds the website user has performed an action on my website. I think I can do it with php sleep, but i don't know if it is the best way to do it. What are the options for doing that?
Thanks for your answers.
Edited:
The idea is: I have a lobby, where website registered users can join (When they join a mysql query is performed updating the info on their sql row, setting that they are connected to lobby number x, and on slot x, but they aren't joined to the server). When they join to the lobby i want to give them x time (suppose 5 minutes) to join a counter strike source server (This updates the user row setting that they are joined to the server). But, i want to kick them of the lobby (this is just updating again their row in the mysql database), if in those 5 minutes they haven't joined the server. I got all of this coded except the part of wait 5 minutes and check if they are connected.
The code is:
$PlayerSlot=mysql_real_escape_string($_POST['playern']);
$sID = mysql_real_escape_string($_POST['jsID']);
//Check if actual player is already in another lobby
$query = sprintf("SELECT `playern` FROM `WebsiteUsers` WHERE `userID`='%s'", mysql_real_escape_string($_SESSION['userID']));
$result = mysql_query($query, $con) or die("Error checking user state: ".mysql_error());
$row = mysql_fetch_row($result);
$playerPlaying = $row[0];
//Check if the slot is free
$query = "SELECT `".$PlayerSlot."` FROM `ServerPlayers`";
$result = mysql_query($query, $con);
$PlayerSlotFree = mysql_fetch_row($result);
if($playerPlaying == NULL && $PlayerSlotFree[0] == NULL) {
$query = "UPDATE `WebsiteUsers` SET `playern`='".$PlayerSlot."', `servern`='".$sID."' WHERE `userID`='".$_SESSION['userID']."'";
$result = mysql_query($query, $con) or die(mysql_error());
$query = "UPDATE `ServerPlayers`SET `".$PlayerSlot."`='".$_SESSION['userID']."', WHERE `serverID`='".$sID."'";
$result = mysql_query($query, $con) or die(mysql_error());
//Update Current Players
$query = "UPDATE `Servers` SET `serverPlayers`=serverPlayers + 1 WHERE `serverID`='".$sID."'";
$result = mysql_query($query, $con) or die(mysql_error());
//TO-DO Here Wait 5 minutes and perform another query
}
Use setTimeout() in javascript
// Wait for 3 Seconds then execute
setTimeout(function(){
// your action
}, 3000);
// In Every 3 Second Execute
setInterval(function(){
// your action
}, 3000);
Was hoping someone could give me some help. I have been trying to learn pagination but have run into an issue. when I run this to get my total rows count:
$sql = "SELECT COUNT (*) FROM item WHERE fid='17'";
$query = mysqli_query($con, $sql);
$row = mysqli_fetch_row($query);
$rows = $row[0];
$rows comes back with no value, I am under the impression that $rows should contain the total number of records from item with the fid of 17, when I run SELECT COUNT (*) FROM item WHERE fid='17' in phpmyadmin it returns 98 which is the correct count. Directly before the above code I use this code to connect to the db, which I use again later to display the records which works fine.
$con=mysqli_connect("$host","$username","$password","$dbname");
// Check connection
if (mysqli_connect_errno())
{
echo "Failed to connect to MySQL: " . mysqli_connect_error();
}
This statement displays records later in the script
$sql = mysqli_query($con,"SELECT * FROM item WHERE fid='17' ORDER BY id DESC $limit ");
So there is data and the info is correct.
I have been following this tutorial on the subject http://www.developphp.com/view.php?tid=1349 and it works like a charm on his example in the video, all I have changed is the database particulars to mine. Can't figure this one out, been stuck for days so now I am bothering you fine folks.
Update: Changed
$query = mysqli_query($con, $sql);
$row = mysqli_fetch_row($query);
$rows = $row[0];
to
if ($result=mysqli_query($con,$sql))
{
// Return the number of rows in result set
$rows=mysqli_num_rows($result);
// Free result set
mysqli_free_result($result);
}
And everything is working now. Still wish I knew what was wrong with my original code but I will be moving on. Thanks Phil Perry for all your help this morning!
Well, you could SELECT * FROM... and then call mysqli_num_rows($query). Probably slower than SELECT count(*). BTW, I presume that item is not a reserved word in SQL. You can always wrap table and field names in backticks (not quotes ' or "). You could also try SELECT count(*) AS myCount....
try this,
$sql = "SELECT COUNT(*) FROM item WHERE fid='17'";
$query = mysqli_query($sql);
$row = mysqli_fetch_array($query);
$rows = $row[0];
This script uses php and mysql to compute a one minute rolling average to reduce the impact of outliers on the my data (one minute = 6 10-second rows). It computes everything correctly, but is not efficient enough to do more than 150 rows at a time. I'd like to do as many rows as I can at a time, possibly between 5-10,000 as my table is over 150,000 and I input approximately 8,000 rows per day.
Does anyone have any suggestions as to how I can make this script run more efficiently?
Thanks!
<?php
//connect to database
mysql_connect("localhost","user","password");//database connection
mysql_select_db("database");
$result = mysql_query("SELECT Timestamp FROM table");
if (!$result) {
die('Could not query:' . mysql_error());
}
//get number of rows in table
$resultA = mysql_query("SELECT * FROM table");
$num_rows = mysql_num_rows($result);
echo "There are $num_rows rows.</br>";
//select column to be averaged
$resultB = mysql_query("SELECT PortRPMSignal FROM table");
if (!$resultB) {
die('Could not query:' . mysql_error());
}
//set start equal to the first row you want to calculate the averages from, likely the first null row
$start = 5;
//calculate 1 minute average, the average is correct
for($i = $start; $i<$num_rows; $i++){
$output = mysql_result($result,$i);
$test = mysql_result($resultB,$i)+mysql_result($resultB,$i-1)+mysql_result($resultB,$i-2)+mysql_result($resultB,$i-3)+mysql_result($resultB,$i-4)+mysql_result($resultB,$i-5);
$test2 = $test/6;
$round = round($test2,4);
$temp = mysql_query("SELECT Timestamp FROM table");
if(!$temp){
die('Could not query:' . mysql_error());
}
//gets timestamp at row $i, and inserts new average value into that row in RPMAve column
$time = mysql_result($result,$i);
mysql_query("UPDATE table SET PortMinuteAveRPM = $round WHERE Timestamp = '$time'");
}
For starters, the initial "count" block here can be cleaned up by adding the COUNT() aggregate:
$resultA = mysql_query("SELECT * FROM table");
$num_rows = mysql_num_rows($result);
echo "There are $num_rows rows.</br>";
Change to:
$resultA = mysql_query("SELECT COUNT(*) FROM table");
$row = mysql_fetch_array($result);
$num_rows = $row[0];
echo "There are $num_rows rows.</br>";
That should speed things up considerably on its own. Without it, you're selecting all of the data from the table - a query that will only grow slower the more you put into the table.
For the averages you're computing, is there any logic required that can't be accomplished directly in a MySQL query? Something such as:
UPDATE table SET PortMinuteAveRPM=(SELECT AVG(PortRPMSignal) FROM table WHERE Timestamp BETWEEN '$startTime' AND '$endTime') WHERE TimeStamp='$endTime'
This may save you from looping through results, if it's plausible.
It sounds like you're trying to calculate an autoregressive moving average (ARMA) but there's numerous issues with your interpretation of your data and how you are capturing it.
If you've got a complete set of data (though your question implies that you don't), then work out what time interval contains the required amount of records and get it direct from the database, e.g.
SELECT a.timestamp as base, AVG(b.PortRPMSignal)
FROM table a, table b
WHERE b.timestamp BETWEEN a.timestamp AND a.timestamp+INTERVAL 6 HOUR
GROUP BY a.timestamp
If you want to thin out the datapoints, then try something like....
SELECT a.timestamp as base, AVG(b.PortRPMSignal)
FROM table a, table b
WHERE b.timestamp BETWEEN a.timestamp AND a.timestamp+INTERVAL 6 HOUR
AND DATE_FORMAT(a.timestamp, '%i%s')='0000'
GROUP BY a.timestamp
Although a better solution if you've not got a complete dataset but there's only a small amount of jitter would be to use the modulus of an auto-increment id to pick out fewer rows from 'a'
It's only a start, but you can bin this bit
//get number of rows in table
$resultA = mysql_query("SELECT * FROM table");
$num_rows = mysql_num_rows($result);
echo "There are $num_rows rows.</br>";
Because the following line
$resultB = mysql_query("SELECT PortRPMSignal FROM table");
...will give you a result set that you can use mysql_num_rows on.
Using the * in a query increases the load on the database.
In your for loop you then have this
$temp = mysql_query("SELECT Timestamp FROM table");
if(!$temp){
die('Could not query:' . mysql_error());
}
which means this query runs every time you loop and you're not even using the results.
I don't know if mysqli will give you better performance, but you should use it.
I set up a cron job to run the following script every 30 min, Basically this script selects article info from stories table where field showing=0 And it than counts votes related to that article, the votes are located in votes table. itemname_field in votes table is equal to id field in stories table.
If you need more information on database structure and voting system, I describe it in depth here:
Set a cron job to update article information depending on its vote values
<?php
mysql_connect("localhost","username","password") or die (mysql_error("There was an error in connection"));
mysql_select_db("database") or die (mysql_error("There was an error in connection"));
$query = "select sum(vt.vote_value) as vote_value, vt.item_name from Votes vt join stories st on st.id = vt.item_name and st.showing = 0 group by vt.item_name";
$result = mysql_query($query);
while($row = mysql_fetch_array($result))
{
if($row['vote_value'] > 9) {
$showing = 1;
}
else if($row['vote_value'] < -9) {
$showing = 2;
}
else {
$showing = 0;
}
$query2 = "UPDATE `stories` SET `showing` = $showing WHERE `id` = '".$row['item_name']."'";
mysql_query($query2);
}
?>
Can anyone suggest the reason, why this script is not working? If everything is alright with script, could it be something in cron job that needs to be done?
EDIT: when I run script in browser it gives the following error:
Warning: mysql_fetch_array(): supplied argument is not a valid MySQL
Which is this line while($row = mysql_fetch_array($result))
The error message you quote means you have an error in your query. Run it though a mysql client to get more info, or echo mysql_error() in php.
In such cases mysql_query() returns boolean false, you can use this for error handling.
The issue was in this part:
$query = "... from Votes vt join ...";
it should not be capital "V".