So I'm having a very strange issue and am not quite sure what to think about it...
I have a standard database query doing a select on a table. It returns a resource, num_rows = 101, and it will fetch rows up to row 42 at which point it simply stops all script execution, no errors, not timing out (takes less than a sec to get the 42 rows) ... I can't provide a link but here's the code in question...
$select2 = "SELECT * FROM `$dbname`.`$file" . "_settings` WHERE `weight` <> 0 ORDER BY `count` ASC";
$result = mysql_query($select2);
$lpcnt = 0;
$numrows = mysql_num_rows($result);
while ($display = mysql_fetch_array($result, MYSQL_ASSOC)){
/* This will print out 42 times */
echo '<pre>In The LOOP, row('.$lpcnt.'):<br>NumRows: '.$numrows.'<br>';
print_r($display);
echo '</pre><br/>';
$domRows[] = $display;
$aTotal[] = $display['count'];
$aTotWeight[] = $display['weight'];
//debug vars
$d['url'] = $display['url'];
$d['total'] = $total;
$d['count'] = $display['count'];
$d['weight'] = $display['weight'];
$dbug['domRows'][] = $d;
$lpcnt++;
}
/* Never Reaches this */
echo '<pre>';die(print_r('Loop Finished'));
At a loss as to what's causing the failure midway through the results loop...
also..I know, I know...myql_ is depreciated, but's that's what I have to work with!
Thanks in advance for any light anyone can shed...this is really hurting the site!
EDIT: also this doesn't break all the time, so far it seems to be related somehow to the number of results...for example, if I run through a result set that has 39 rows, it will proccess all of them...and it's consitently failing at 42 rows... on my tables that have 100+ records
EDIT FINAL: Ok figured it out! Turns out we had our memory limit to low, so it was trying to allocate an illegal amount of memory! So we upped it and now it works! Also I had my error reporting in a spot where it was being conditionally turned back off by other code...that's why I wasn't seeing errors! Duh... anyway, thanks for the stabs, to those that responded Merry x-mas and all that jazz...
Related
I'm in need of some expertise here, I have a massive SQL database that I use in conjunction with my Mobile app I program. I'm getting some very long times to fetch a result from the database, at times upwards of 20 to 25 seconds. I've managed to increase the speed and it is were it is now from 40 seconds to retrieve a result. I am hoping someone may have some insight on how I can speed up the query speed and return a result faster then 20 seconds.
Main table is 4 columns + 1 for the "id" column, the database contains 15,254,543 rows of data. Currently it is setup as an InnoDB, with 4 indexes, and it about 1.3GB for size.
My server is a GoDaddy VPS, 1 CPU, with 4 GB of Ram. The is dedicated and I do not share resources with anyone else, its only purpose beside a very basic website is the SQL database.
Just to note, the database record count is not going to get any larger, I really just need to figure out a better way to return a query faster then 20 seconds.
In more detail, the Android app connects to my website via a php document to query and return the results, I have a thought that there maybe a better way to go about this and this maybe were the pitfall is. An interesting note is that when I'm in PHP My Admin, I can do a search and get a result back in under 3 seconds, which also points me to the issue might be in my php document. Here is the php document below that I wrote to do the work.
<?php
require "conn.php";
$FSC = $_POST["FSC"];
$PART_NUMBER = $_POST["NIIN"];
$mysql_qry_1 = "select * from MyTable where PART_NUMBER like '$PART_NUMBER';";
$result_1 = mysqli_query($conn ,$mysql_qry_1);
if(mysqli_num_rows($result_1) > 0) {
$row = mysqli_fetch_assoc($result_1);
$PART_NUMBER = $row["PART_NUMBER"];
$FSC = $row["FSC"];
$NIIN = $row["NIIN"];
$ITEM_NAME = $row["ITEM_NAME"];
echo $ITEM_NAME, ",>" .$PART_NUMBER, ",>" .$FSC, "" .$NIIN;
//usage stats
$sql = "INSERT INTO USAGE_STATS (ITEM_NAME, FSC, NIIN, PART_NUMBER)
VALUES ('$ITEM_NAME', '$FSC', '$NIIN', '$PART_NUMBER')";
if ($conn->query($sql) === TRUE) {
$row = mysqli_insert_id($conn);
} else {
//do nothing
}
//
} else {
echo "NO RESULT CHECK TO ENSURE CORRECT PART NUMBER WAS ENTERED ,> | ,>0000000000000";
}
$mysql_qry_2 = "select * from MYTAB where FSC like '$FSC' and NIIN like '$NIIN';";
$result_2 = mysqli_query($conn ,$mysql_qry_2);
if(mysqli_num_rows($result_2) > 0) {
$row = mysqli_fetch_assoc($result_2);
$AD_PART_NUMBER = $row["PART_NUMBER"];
if(mysqli_num_rows($result_2) > 1){
echo ",>";
while($row = mysqli_fetch_assoc($result_2)) {
$AD_PART_NUMBER = $row["PART_NUMBER"];
echo $AD_PART_NUMBER, ", ";
}
} else {
echo ",> | NO ADDITIONAL INFO FOUND | ";
}
} else {
echo ",> | NO ADDITIONAL INFO FOUND | ";
}
mysqli_close($con);
?>
So my question here is how can I improve the read speed with the available resources I have or is there an issue with my current PHP document that is causing the bottle neck here?
Instead of using LIKE you would get much faster reads by selecting a specific column that was indexed.
SELECT * FROM table_name FORCE INDEX (index_list) WHERE condition;
The other thing that speeds up Mysql greatly is the use of an SSD drive on the VPS server. A SSD drive will greatly decrease the amount of time it takes to scan a database that large.
I have a trouble.
I have a SQL table with processes which are finalized, closed, cancelled and working on, which also are associateds by an area (talent, outsourcing, digital, etc).
I want to do an AVG by the area and total by the states of the processes.
so i have:
$tiempoQueryFinalizados = mysqli_query($con,
"SELECT IFNULL(SUM(IFNULL(timestampdiff(DAY,fecha_creacion,IFNULL(fecha_cerrado,NOW())),0)),0) AS finalizados,
(Select count(id_proceso) from proceso where estado ='FINALIZADO' and fecha_cerrado is not null) as c
FROM proceso
WHERE estado = 'FINALIZADO' and fecha_cerrado IS NOT NULL and area ='".$rowArea['id_area']."' ");
Then I use a simple division in PHP with
mysqli_fetch_assoc($tiempoQueryFinalizados)['finalizados'] / mysqli_fetch_assoc($tiempoQueryFinalizados)['c']
but I get the error Warning: Division by zero, so I check if was zero but it doesn't. In this example mysqli_fetch_assoc($tiempoQueryFinalizados)['c'] = 2 but still counting as 0
any help, please?
EDIT: the area is returned by a fetch array of $areaQuery = mysqli_query($con,"SELECT * from area where id_area!=0");
This is not a simple division
mysqli_fetch_assoc($tiempoQueryFinalizados)['finalizados'] /
mysqli_fetch_assoc($tiempoQueryFinalizados)['c']
Each call to mysqli_fetch_assoc() gets a new row from your resultset. So as the query only returns one row, the second call will return FALSE which equates to zero.
So this would be a better way of doing the retrieval and calculation
$row = mysqli_fetch_assoc($tiempoQueryFinalizados);
$calc = $row['finalizados']] / $row['c'];
Ok I forgot when every call you do to mysqli_fetch* you are jumping to the next register, so only I needed to edit
$tiempoQueryFinalizados = mysqli_query(...
by
$tiempoQueryFinalizados = mysqli_fetch_assoc(mysqli_query(...
now
$tiempoQueryFinalizados['finalizados'] / $tiempoQueryFinalizados['c']
works very nice :)
thank you all!
I seriously need help with GDS and PHP-GDS Library.
I am doing a fetch from outside google appengine using the fantastic php-gds library. So far the library works fine.
My problem is that my data fetch from GDS returns inconsistent results and I have no idea what the issue might be.
Please see code below:
<?php
//...
//...
$offset = 0;
do{
$query = "SELECT * FROM `KIND` order by _URI ASC limit 300 offset ".$offset;
$tableList=[];
$tableList = $this->obj_store->fetchAll($query);
$offset += count($tableList);
$allTables[] = $tableList;
$totalRecords = $offset;
}while(!empty($tableList));
echo $totalRecords; // i expect total records to be equal to the number of entities in the KIND.
// but the results are inconsistent. In some cases, it is correct
// but in most cases it is far less than the total records.
// I could have a KIND with 750 entities and only 721 will be returned in total.
// I could have a KIND with 900 entities and all entities will be returned.
// I could have a KIND with 4000 entities and only 1200 will be returned.
?>
Please help. Also when I run the exact same query in the cloud console I get the right entity count. (Hope this helps someone)
UPDATE
I ended up using cursors. New code below:
<?php
$query = "SELECT * FROM `KIND`";
$tableList=[];
$queryInit = $this->obj_store->query($query);
do{
$page = $this->obj_store->fetchPage(300);// fetch page.
$tableList = am($tableList,$page); //merge with existing records.
$this->obj_store->setCursor($this->obj_store->getCursor());//set next cursor to previous last cursor
}while(!empty($page)); //as long as page result is not empty.
?>
Try using a cursor instead of an offset. See the discussion of cursors (including samples in PHP) here:
https://cloud.google.com/datastore/docs/concepts/queries#datastore-cursor-paging-php
This is the query I have:
$sqlw = "SELECT * FROM coverages where user_id='3828' ORDER BY sp_id ASC";
$resultw = mysql_query($sqlw);
$roww = mysql_fetch_array($resultw);
while ($roww = mysql_fetch_array($resultw)) {
echo $roww['sp_id']."<br>";
}
echo "TOTAL:".mysql_num_rows($resultw)."<br>";
As you can see its very basic
the results show : TOTAL:29
But when I count the list of the items returned back its only 28.
I ran the query on phpmyadmin it shows a total of 29 rows, I did count them and they are 29.
I ran different other simple queries and it always does the same thing: one row is missing. This could be trivial maybe I am missing something or maybe its server related? any help/ideas would be greatly appreciated. Thank you
Your call to mysql_fetch_array() before the loop disposes of a row.
You have a classic off-by-one error.
There is an extra $roww = mysql_fetch_array($resultw); before your loop starts. This means you're throwing away the first row.
Looking for some ideas here... I have a MySQL table that has 100K rows of test data in it.
I am using a PHP script to fetch rows out of that table, and in this test case, the script is fetching all 100,000 rows (doing some profiling and optimization for large datasets).
I connect to the DB, and execute an unbuffered query:
$result = mysql_unbuffered_query("SELECT * FROM TestTable", $connection) or die('Errant query: ' . $query);
Then I iterate over the results with:
if ($result) {
while($tweet = mysql_fetch_assoc($result)) {
$ctr++;
if ($ctr > $kMAX_RECORDS) {
$masterCount += $ctr;
processResults($results);
$results = array();
$ctr = 1;
}
$results[] = array('tweet' => $tweet);
}
echo "<p/>FINISHED GATHERING RESULTS";
}
function processResults($resultSet) {
echo "<br/>PROCESSED " . count($resultSet) . " RECORDS";
}
$kMAX_RECORDS = 40000 right now, so I would expect to see output like:
PROCESSED 40000 RECORDS PROCESSED 40000 RECORDS PROCESSED
20000 RECORDS FINISHED GATHERING RESULTS
However, I am consistently seeing:
PROCESSED 39999 RECORDS
PROCESSED 40000 RECORDS
FINISHED GATHERING RESULTS
If I add the output of $ctr right after $ctr++, I get the full 100K records, so it seems to me to be some sort of timing issue or problem with fetching the data from the back-end with MYSQL_FETCH_ASSOC.
On a related note, the code in the while loop is there because prior to breaking up the $results array like this the while loop would just fall over at around 45000 records (same place every time). Is this due to a setting somewhere that I have missed?
Thanks for any input... just need some thoughts on where to look for the answer to this.
Cheers!
You're building an array of results, and counting that new array's members. So yes, it is expected behavior that after fetching the first row, you'll get "1 result", then "2 results", etc...
If you want to get the total number of rows expected, you'll need to use mysql_num_rows()
When you start going through your result $ctr has no value and doing first incrementation will evaluate it to 0. But when reaching $kMAX_RECORDS you reset it to 1 instead of 0. I don't know however why you see 1 row less in the first time of calling processResults(). I think it should be one more.
As to missing last 20000 rows notice that you are running processResults() only after $ctr exceeds $kMAX_RECORDS