Google data store inconsistent result fetch from select - php

I seriously need help with GDS and PHP-GDS Library.
I am doing a fetch from outside google appengine using the fantastic php-gds library. So far the library works fine.
My problem is that my data fetch from GDS returns inconsistent results and I have no idea what the issue might be.
Please see code below:
<?php
//...
//...
$offset = 0;
do{
$query = "SELECT * FROM `KIND` order by _URI ASC limit 300 offset ".$offset;
$tableList=[];
$tableList = $this->obj_store->fetchAll($query);
$offset += count($tableList);
$allTables[] = $tableList;
$totalRecords = $offset;
}while(!empty($tableList));
echo $totalRecords; // i expect total records to be equal to the number of entities in the KIND.
// but the results are inconsistent. In some cases, it is correct
// but in most cases it is far less than the total records.
// I could have a KIND with 750 entities and only 721 will be returned in total.
// I could have a KIND with 900 entities and all entities will be returned.
// I could have a KIND with 4000 entities and only 1200 will be returned.
?>
Please help. Also when I run the exact same query in the cloud console I get the right entity count. (Hope this helps someone)
UPDATE
I ended up using cursors. New code below:
<?php
$query = "SELECT * FROM `KIND`";
$tableList=[];
$queryInit = $this->obj_store->query($query);
do{
$page = $this->obj_store->fetchPage(300);// fetch page.
$tableList = am($tableList,$page); //merge with existing records.
$this->obj_store->setCursor($this->obj_store->getCursor());//set next cursor to previous last cursor
}while(!empty($page)); //as long as page result is not empty.
?>

Try using a cursor instead of an offset. See the discussion of cursors (including samples in PHP) here:
https://cloud.google.com/datastore/docs/concepts/queries#datastore-cursor-paging-php

Related

how to subtract a specific amount from the COUNT(*) result

I'm new to PHP and i want to know how i can subtract a specific amount from the results from counting the total amount of rows in a table. In this case i'd like to minus the value 3 from whatever the value of the total rows is. But i keep getting an error. Below is my code.
$cartwork = $con->query("SELECT count(*) FROM table");
$vs = '3';
$camount = $cartwork - $vs;
echo "$camount";
When the code runs i get the error "Object of class mysqli_result could not be converted to int" what can i do to fix this and get it to work properly.
The query returns a result set. You need to parse through the result set(s) in order to access the values returned. That's basically what the error states.
Please see here for documentation on the PHP function for fetching rows:
http://php.net/manual/en/function.mysql-fetch-row.php
So basically you would need
$row=$cartwork->mysql_fetch_row();
$cartWork_value = $row[0];
$vs = '3';
$camount = $cartwork_Value - $vs;
echo "$camount";
Note - this assumes that you get back exactly one result row (which should be the case with your query).
You can simply change your query to:
$cartwork = $con->query("SELECT count(*)-3 FROM table");
It doesn't smell particularly good though.

Best way to show large amount of data

What is the best way to handle a large amount of data entries on a web page?
Let's assume I am having a database with 5000 records on a table that contain song_name,author_name,song_id,posted_by; I want to build a playlist with all the songs on a single page. Also on that page there is a player that plays songs according to the playlist entries that is shown on the page.
I have tried to pull all 5000 entries from that table and build a javascript object with them, and handling that object I have built the playlist, search in playlist, and so forth. But that takes a very large amount of resources ( un the user end ) and a lot of page loading time ( because there are a lot of entries! ) and the page is very slow.
Is it better to load all the data into an object and paginate by JavaScript each 100 records of the playlist or is it better to get the results paginated from the database and just update the playlist? ( This taking in consideration the fact that I if the player has the shuffle button activated, it may shuffle to ANY song in the user's database, not only on the current songs from the visible playlist )
I think pagination is your best option. Just create a limit of 100 (for example) and use AJAX to extract the next 100. If the client turns on shuffle, just send another request to the server and let it call a function that does the following:
Count total rows in database
Use a randomize function to get 100 random numbers
Now create a slightly tricky query to get records from the db based
on their rownumber:
function getRandomTracks($limit) {
$total = $this->db->count_all('table_tracks');
//Get random values. Speed optimization by predetermine random rownumbers using php
$arr = array();
while (count($arr) < $limit) {
$x = mt_rand(0, $total); //get random value between limit and 0
if (!isset($arr[$x])) { //Random value must be unique
//using random value as key and check using isset is faster then in_array
$arr[$x] = true;
}
}
//Create IN string
$in = implode(',', array_keys($arr));
//Selection based on random rownumbers
$query = $this->db->query('SELECT * FROM
(SELECT #row := #row + 1 as row, t.*
FROM `table_tracks` t, (SELECT #row := 0) r) AS tracks
WHERE `row` IN(' . $in . ')');
return $query->result();
}
I'm using a similar function, also to deal will large amounts of tracks (over 300.000) so I'm sure this will work!
It is very hard to load the "entire" data to client program even if you are using jQuery or other library else, as the key factor is not what code/sdk you are using but the browser itself!
By the way, chrome is the most fast and IE(before ver.10) is the lowest.
You can refer the links below:
http://www.infoq.com/news/2010/09/IE-Subsystems-Spends-Time
http://www.nczonline.net/blog/2009/01/05/what-determines-that-a-script-is-long-running/
http://www.zdnet.com/browser-benchmarks-ie-firefox-opera-and-safari-3039420732/
http://msdn.microsoft.com/en-us/library/Bb250448
http://support.microsoft.com/kb/175500/en-us
So what you should do is to move your client logic to the server-side just as other people suggesting.
As you mentioned to get paginated but with just javascript for all your data, it is the same as none paginate in essence.
use ajax to load the data in steps of 100 (or more, just try)
do a loop over your recordsets and increase the limit each time:
<?php
$Step = 100;
$u_limit = 0;
$sql = "SELECT song_id FROM $MySQL_DB.songs";
$data = mysql_query($sql, $dblk);
$numer_of_entries = mysql_num_rows($data);
while($o_limit < $numnumer_of_entries)
{
$o_limit = u_limit + $Step;
$sql = "SELECT * FROM $MySQL_DB.songs order by id DESC LIMIT $u_limit, $o_limit";
$data = mysql_query($sql, $dblk);
while($row = mysql_fetch_array($data))
{
// built an array and return this to ajax
}
$u_limit += $Step;
}
Try this: http://www.datatables.net/
I wonder but maybe it's works.

Sum variables in an array using while loop only 1 iteration per refresh PHP SQL

Hello stackoverflow, long time lurker first time asker. Anyways I am creating a project for my school and it is almost completely finished but I think I need help with the while loop. I am trying to sum the variables that are stored in the array and then output that into a useable variable that I can sum with another variable. The problem is the loop only does 1 iteration every time I refresh to page. So it will eventually get the full array but only one item at a time. Please let me know what I am going wrong, I'm sure its something dumb!
$bill= mysql_query("SELECT Transaction.date, Transaction.Price, Transaction.customer_customer_id, Service.cost, Service.user_id, Service.uname
FROM *.Transaction,*.Service
WHERE Transaction.customer_customer_id = Service.user_id AND Service.uname = '$uuname'");
$query_row=mysql_fetch_array($bill);
$userservice = ($query_row[cost]);
$userprice = ($query_row[Price]);
$row2=array();
while($row = mysql_fetch_array($bill))
{
$row2[] += $row['cost'];
}
$totalservice = array_sum($row2);
Thanks for any help you guys may have. This one is frying my brain.
Both your SQL and your PHP make no sense.
FROM *.Transaction,*.Service
...will throw an error in MySQL, but your code doesn't check for errors. I suspect it should be:
FROM Transaction, Service
While the PHP will parse and run.....
while($row = mysql_fetch_array($bill)) {
$row2[] += $row['cost'];
}
$totalservice = array_sum($row2);
Is a very strange way to populate an array. Why not just....
while($row = mysql_fetch_array($bill)) {
$totalservice+=$row['cost'];
}
Indeed, if you are just throwing away the rest of the data, then why are you fetching it from the database?
SELECT SUM(Service.cost)
FROM Transaction,Service
WHERE Transaction.customer_customer_id = Service.user_id
AND Service.uname = '$uuname'
In which case the join is also redundant:
SELECT SUM(Service.cost)
FROM Service
WHERE Service.uname = '$uuname'
Rewrite query as
SELECT Service.user_id, Service.uname, SUM(Service.cost) as cost
FROM djqrico_hotel.Transaction LEFT JOIN djqrico_hotel.Service ON Transaction.customer_customer_id = Service.user_id
WHERE Service.uname = '$uuname' GROUP BY Service.user_id
if you want to get sum of all user's transactions.
The problem is the loop only does 1 iteration every time I refresh to page.
Have you ran the query against the database and checked if it's returning more than one row?
Also, if all you want to do is sum the cost column you could do that in sql:
SELECT SUM(cost)[....]

Query on large mysql database

i've got a script which is supposed to run through a mysql database and preform a certain 'test'on the cases. Simplified the database contains records which represent trips that have been made by persons. Each record is a singel trip. But I want to use only roundway trips. So I need to search the database and match two trips to each other; the trip to and the trip from a certain location.
The script is working fine. The problem is that the database contains more then 600.000 cases. I know this should be avoided if possible. But for the purpose of this script and the use of the database records later on, everything has to stick together.
Executing the script takes hours right now, when executing on my iMac using MAMP. Off course I made sure that it can use a lot of memory etcetare.
My question is how could I speed things up, what's the best approach to do this?
Here's the script I have right now:
$table = $_GET['table'];
$output = '';
//Select all cases that has not been marked as invalid in previous test
$query = "SELECT persid, ritid, vertpc, aankpc, jaar, maand, dag FROM MON.$table WHERE reasonInvalid != '1' OR reasonInvalid IS NULL";
$result = mysql_query($query)or die($output .= mysql_error());
$totalCountValid = '';
$totalCountInvalid = '';
$totalCount = '';
//For each record:
while($row = mysql_fetch_array($result)){
$totalCount += 1;
//Do another query, get all the rows for this persons ID and that share postal codes. Postal codes revert between the two trips
$persid = $row['persid'];
$ritid = $row['ritid'];
$pcD = $row['vertpc'];
$pcA = $row['aankpc'];
$jaar = $row['jaar'];
$maand = $row['maand'];
$dag = $row['dag'];
$thecountquery = "SELECT * FROM MON.$table WHERE persid=$persid AND vertpc=$pcA AND aankpc=$pcD AND jaar = $jaar AND maand = $maand AND dag = $dag";
$thecount = mysql_num_rows(mysql_query($thecountquery));
if($thecount >= 1){
//No worries, this person ID has multiple trips attached
$totalCountValid += 1;
}else{
//Ow my, the case is invalid!
$totalCountInvalid += 1;
//Call the markInvalid from functions.php
$totalCountValid += 1;
markInvalid($table, '2', 'ritid', $ritid);
}
}
//Echo the result
$output .= 'Total cases: '.$totalCount.'<br>Valid: '.$totalCountValid.'<br>Invalid: '.$totalCountInvalid; echo $output;
Your basic problem is that you are doing the following.
1) Getting all cases that haven't been marked as invalid.
2) Looping through the cases obtained in step 1).
What you can easily do is to combine the queries written for 1) and 2) in a single query and loop over the data. This will speed up the things a bit.
Also bear in mind the following tips.
1) Selecting all columns is not at all a good thing to do. It takes ample amount of time for the data to traverse over the network. I would recommend replacing the wild-card with all columns that you really need.
SELECT * <ALL_COlumns>
2) Use indexes - sparingly, efficiently and appropriately. Understand when to use them and when not to.
3) Use views if you can.
4) Enable MySQL slow query log to understand which queries you need to work on and optimize.
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes
5) Use correct MySQL field types and the storage engine (Very very important)
6) Use EXPLAIN to analyze your query - EXPLAIN is a useful command in MySQL which can provide you some great details about how a query is ran, what index is used, how many rows it needs to check through and if it needs to do file sorts, temporary tables and other nasty things you want to avoid.
Good luck.

First record not picked up in pagination script (despite the fact it is set to show all records > NOW() as query)

This is my pagination script, which I've been working on:
http://pastebin.com/4mpjdWKD
This is the query page which displays the records - but it omits the first record:
http://pastebin.com/kJZy9fv0
The actual query is this:
<?php
//Require the file that contains the required classes
include("pmcPagination.php");
//PhpMyCoder Paginator
$paginator = new pmcPagination(20, "page");
//Connect to the database
mysql_connect("localhost","root","PASSWORD HIDDEN FOR SECURITY REASONS");
//Select DB
mysql_select_db("housemde");
//Select only results for today and future
$result = mysql_query("SELECT * FROM housemd WHERE expiration > NOW()
order by airdate;");
$recordCount = mysql_num_rows($result);
//You can also add reuslts to paginate here
mysql_data_seek($result,0) ;
while ($row = mysql_fetch_array($result))
{
$paginator->add(new paginationData($row['programme'],
$row['channel'],
$row['airdate'],
$row['expiration'],
$row['episode'],
$row['series'],
$row['epno'],
$row['setreminder']));
}
?>
and the dates are all in the future, yet to even get the first record to show, I have to duplicate it in PhpMyadmin - so what is wrong with my script?
Is it something to do with mysql_fetch_array, and if so, where do I need to fix the script?
In any case, here's the data from the database, all showing records for future events:
http://pastebin.com/gwcv7qza
In short, the basic problem is, I have to manually duplicate a record to get it to show in the script sometimes, and I'm trying to find a fix - but I'm having trouble looking for a solution. I've tried myself, and it didn't work.
I hope I've explained this well enough.
Any help on fixing this is appreciated (just post the solutions in my pastebin links where it allows you to submit a correction/amendment to the pastebin, if you want.)
Thanks!
I think the problem is in your Paginatop class
this row $result = ($page - 1) * $resultsPerPage + 1; will return 1 for the first page and you will show the second element in $this->items[$result] array - $this->items[1], not the first $this->items[0].
Give it a try with $result = ($page - 1) * $resultsPerPage;
for page 1 $result will be 0
for page 2 $result will be 19 (if showing 20 records per page)

Categories