I am creating a project which involves getting some questions from mysql database. For instance, if I have 200 questions in my database, I want to randomly choose 20 questions in such a way that no one question will be repeated twice. That is, I want to be able to have an array of 20 different questions from the 200 I have every time the user tries to get the list of questions to answer. I will really appreciate your help.
SELECT * FROM questions ORDER BY RAND() LIMIT 20;
PS^ This method not possible for very big tables
Use Google to find a function to create an array with 20 unique numbers, with a minimum and a maximum. Use this array to prepare an SQL query such as:
expression IN (value1, value2, .... value_n);
More on the SQL here.
Possible array filling function here too.
Assuming you have contiguously number questions in your database, you just need a list of 20 random numbers. Also assuming you want the user to be able to take more than one test and get another 20 questions without duplicates then you could start with a randomised array of 200 numbers and select blocks of 20 sequentially from that set i.e.
$startQuestion=1;
$maxQuestion=200;
$numberlist= range(1,$maxQuestion);
shuffle($numberlist);
function getQuestionSet( $noOfQuestions )
{
global $numberlist, $maxQuestion, $startQuestion;
$start= $startQuestion;
if( ($startQuestion+$noOfQuestions) > $maxQuestion)
{
echo "shuffle...\n";
shuffle($numberlist);
$startQuestion=1;
}else
$startQuestion+= $noOfQuestions;
return array_slice($numberlist,$start,$noOfQuestions);
}
// debug...
for($i=0; $i<42; $i++)
{
$questionset= getQuestionSet( 20 );
foreach( $questionset as $num )
echo $num." ";
echo "\n";
}
then use $questionset to retrieve your questions
If you know how many rows there are in the table, you could do use LIMIT to your advantage. With limit you specify a random offset; syntax: LIMIT offset,count. Example:
<?php
$totalRows = 200; // get his value dynamically or whatever...
$limit = 2; // num of rows to select
$rand = mt_rand(0,$totalRows-$limit-1);
$query = 'SELECT * FROM `table` LIMIT '.$rand.','.$limit;
// execute query
?>
This should be safe for big tables, however it will select adjacent rows. You could then mix up the result set via array_rand or shuffle:
<?php
// ... continued
$resultSet = $pdoStm->fetchAll();
$randResultKeys = array_rand($resultSet,$limit); // using array_rand
shuffle($resultSet); // or using shuffle
?>
Related
I want get results of my query (with limit 10) + count possible results.
I know there is similar questions and answers.
for example here
but if i trying get count possible rows (via getSingleScalarResult()) i will get excepton: The query returned multiple rows. Change the query or use a different result function like getScalarResult().
$query = $repository
->createQueryBuilder('t')
->select('COUNT(t.katId)', 't.hotel', 't.title', 't.desc', 'picture', 'MIN(t.price) AS price');
$query->where('t.visible = (:visible)')->setParameter('visible', 1);
// + some wheres, where in, more than....
$query->groupBy('t.hotel');
$query->setMaxResults(10);
echo $query->getQuery()->getSingleScalarResult();
exit();
I just need one integer whitch represent all results from my query.
How can i get this count number? Ideal in one shot to db.
EDIT:
if i remove $query->groupBy('t.hotel'); and in select keep only ->select('count(t.katId)'); then it work. But i need groupBy because it makes real count of results.
SOLUTION
I divided it on two queries so - to get results i rolled back changes to state before trying any count information, and make clone this query (before set setMaxResults and groupBy), change select (keep all wheres) and get count information.
I will be grateful if someone offers better solution
Get results:
removed COUNT() from select
asking for results changed to 'normal' ->getArrayResults
Get count:
$q = clone $query;
$q->select('count(distinct t.hotel) as count');
$r = $q->getQuery()->getArrayResult();
echo $r[0]['count'];
exit();
If you need keep the groupBy:
$query = $repository->createQueryBuilder('t')
$query->select('COUNT(t.katId)', 't.hotel', 't.title', 't.desc', 'picture', 'MIN(t.price) AS price');
$query->from(ENTITY STRING, 't', 't.hotel'); //here defined your array result key
$query->where('t.visible = (:visible)')->setParameter('visible', 1);
$query->groupBy('t.hotel');
$query->setMaxResults(10);
echo $query->getQuery()->getScalarResult();
exit();
Edit : New edit works ?
You are only interested in COUNT(t.katId), so you should drop other returned fields 't.hotel', 't.title', etc.
The result will then contain a single return value (single scalar result), so $query->setMaxResults(10) is not needed.
What is the best way to handle a large amount of data entries on a web page?
Let's assume I am having a database with 5000 records on a table that contain song_name,author_name,song_id,posted_by; I want to build a playlist with all the songs on a single page. Also on that page there is a player that plays songs according to the playlist entries that is shown on the page.
I have tried to pull all 5000 entries from that table and build a javascript object with them, and handling that object I have built the playlist, search in playlist, and so forth. But that takes a very large amount of resources ( un the user end ) and a lot of page loading time ( because there are a lot of entries! ) and the page is very slow.
Is it better to load all the data into an object and paginate by JavaScript each 100 records of the playlist or is it better to get the results paginated from the database and just update the playlist? ( This taking in consideration the fact that I if the player has the shuffle button activated, it may shuffle to ANY song in the user's database, not only on the current songs from the visible playlist )
I think pagination is your best option. Just create a limit of 100 (for example) and use AJAX to extract the next 100. If the client turns on shuffle, just send another request to the server and let it call a function that does the following:
Count total rows in database
Use a randomize function to get 100 random numbers
Now create a slightly tricky query to get records from the db based
on their rownumber:
function getRandomTracks($limit) {
$total = $this->db->count_all('table_tracks');
//Get random values. Speed optimization by predetermine random rownumbers using php
$arr = array();
while (count($arr) < $limit) {
$x = mt_rand(0, $total); //get random value between limit and 0
if (!isset($arr[$x])) { //Random value must be unique
//using random value as key and check using isset is faster then in_array
$arr[$x] = true;
}
}
//Create IN string
$in = implode(',', array_keys($arr));
//Selection based on random rownumbers
$query = $this->db->query('SELECT * FROM
(SELECT #row := #row + 1 as row, t.*
FROM `table_tracks` t, (SELECT #row := 0) r) AS tracks
WHERE `row` IN(' . $in . ')');
return $query->result();
}
I'm using a similar function, also to deal will large amounts of tracks (over 300.000) so I'm sure this will work!
It is very hard to load the "entire" data to client program even if you are using jQuery or other library else, as the key factor is not what code/sdk you are using but the browser itself!
By the way, chrome is the most fast and IE(before ver.10) is the lowest.
You can refer the links below:
http://www.infoq.com/news/2010/09/IE-Subsystems-Spends-Time
http://www.nczonline.net/blog/2009/01/05/what-determines-that-a-script-is-long-running/
http://www.zdnet.com/browser-benchmarks-ie-firefox-opera-and-safari-3039420732/
http://msdn.microsoft.com/en-us/library/Bb250448
http://support.microsoft.com/kb/175500/en-us
So what you should do is to move your client logic to the server-side just as other people suggesting.
As you mentioned to get paginated but with just javascript for all your data, it is the same as none paginate in essence.
use ajax to load the data in steps of 100 (or more, just try)
do a loop over your recordsets and increase the limit each time:
<?php
$Step = 100;
$u_limit = 0;
$sql = "SELECT song_id FROM $MySQL_DB.songs";
$data = mysql_query($sql, $dblk);
$numer_of_entries = mysql_num_rows($data);
while($o_limit < $numnumer_of_entries)
{
$o_limit = u_limit + $Step;
$sql = "SELECT * FROM $MySQL_DB.songs order by id DESC LIMIT $u_limit, $o_limit";
$data = mysql_query($sql, $dblk);
while($row = mysql_fetch_array($data))
{
// built an array and return this to ajax
}
$u_limit += $Step;
}
Try this: http://www.datatables.net/
I wonder but maybe it's works.
I wrote a function which makes a random id makeid(); Just to ensure the id is unique I have a SQL statement which checks if the id already exists.
$does_id_exist = mysql_query("SELECT COUNT(*) AS count FROM signups WHERE affid='$affid'");
if(mysql_num_rows($does_id_exist) == 1)
{
#loop function and perform query again
}
else
{
#insert record
}
So I'm having trouble with looping the function. How do I loop my function makeid() and perform the $does_id_exist check to ensure that each ID is unique.
--UPDATE-- Just to clarify- My code makes an id like YES#281E But before I INSERT this id into the users record. I just need to verify IF any other user already has this id. IF another user has this id that event must trigger my function to create a new id e.g. WOW!29E3 and again check the sql/query to ensure no other user has that id. Continue to loop if fails or end and INSERT if the id is available.
You can either just use a primary key on your database table, or something like this:
<?php
// the id to insert
$newId = null;
// populate with results from a SELECT `aff_id` FROM `table`
$currentIds = array();
// prepopulate
for( $i=0; $i<100000; $i++ )
{
$currentIds[] = "STRING_" + rand();
}
// generate at least one id
do
{
$newId = "STRING_" + rand();
}
// while the id is taken (cached in $currentIds)
while( in_array($newId, $currentIds) );
// when we get here, we have an id that's not taken.
echo $newId;
?>
Output:
STRING_905649971 (run time 95ms);
I'd definitely not recommend running the query repeatedly. Perhaps a final check before you insert, if your traffic volume is high enough.
Do not do COUNT(*), because you do not need to know how many rows is there (it should be 0 or 1 as you need Id unique), so even DB finds your row it will still be checking for the whole table to count. You really care if you got 1 row, so just select for row with that ID and this sufficient. You should also avoid using rand() - this does not help as you see and you cannot predict how many loops you can do before you find "free slot". use something predictable, like date prefix, or prefix incremented each day. anything that would help you narrow the data set. But for now (pseudocode!):
$id = null;
while( $id == null ) {
$newId = 'prefix' . rand();
mysql_query("SELECT `affid` FROM `signups` WHERE `affid`='${newId}'");
if( mysql_num_rows() == 0) {
$id = newId;
break;
}
}
Ensure you got DB indexed, to speed things up.
EDIT: I do agree that any cache would be useful to speed things up (you can add it easily yourself based on #Josh example), still, I think this is fixing at wrong place. If possible rethink the way you generate your ID. It does not really need to be auto increment, but something more predictable than rand() would help you. If your ID does not need to be easily memorable and it is not any security concern to have them sequential, maybe use numbers with other base than 10 (i.e. using 26 would use all digits + letters so you'd end with PREFIX-AX3TK, so string as you want, and at the same time you would easily be able to quickly generate next Id
I'm trying to generate select names from a mysql database by using the rand() function in php. The program works fine but is it possible to make the results more random because some of the records that I'm getting is the same(whenever I clear the session and start over again), and others might not get selected. I have currently 91 records in the table.
if(!isset($_SESSION['max'])){
$max = $db->get_var("SELECT COUNT(*) FROM tbl_participants");
$_SESSION['max'] = $max;
}else{
$max = $_SESSION['max'];
}
if(!empty($_GET['clear'])){
$_SESSION['used_up'] = Array();
$_SESSION['names'] = Array();
}
//print_r($_SESSION['used_up']);
$current_number = rand(0, $max);
$exists = check_existing($current_number, $_SESSION['used_up']);
if($exists == 2){
$_SESSION['used_up'][] = (int)$current_number;
$name = $db->get_var("SELECT participant FROM tbl_participants WHERE participant_id='$current_number'");
$_SESSION['names'][] = $name;
foreach($_SESSION['names'] as $k=>$v){
echo '<li>'.$v.'</li>';
}
}
function check_existing($item, $array){
if(in_array($item, $array)){
return 1;
}else{
return 2;
}
}
Try mt_rand instead of rand. (http://php.net/manual/en/function.mt-rand.php)
Edit:
A gentleman in this post:
What is the best way to generate a random key within PHP?
suggested using
sha1(microtime(true).mt_rand(10000,90000))
According to PHP manual there is no need to seed the random number generator (at least since PHP 4.2.0). So it should return different number every time the php script is run. You can try to use also mt_rand(), but I don't think it will give you better randomness then rand(). The manual just says that it is 4x faster and that on some old libc versions the rand() was buggy - but I think that was written long time ago.
How many records do you need? You can just use SELECT * FROM table ORDER BY rand() LIMIT 5, however that can be slow if you have many large records in table (1000+).
If you have only 91 records it is quite possible that sometimes the generator will generate the same number again. For code on how to generate X unique random numbers, you can check answers to this question: Generating UNIQUE Random Numbers within a range - PHP
lets say;
I have a $friends array with 2,000 different friendID numbers
+
I have a $bulletins array with 10,000 bulletinID numbers, the $bulletins array will also have another value with a userID of who posted the bulletin entry
Now is it possible to get all the bulletinID numbers that have a userID matching a userID in the friends array? And if it is even possible, would this be fast or slow or not generally a good method? I am trying to get bulletin type posts on my site and only show ones posted by a friend of a user but some users have a few thousand friends and bulletins could be in the thousands but only some of them a user is allowed to view
Also if this is possible, could I limit it to oly get like the first 50 bulletins ID's that match a friendID
Where are you getting these arrays of thousands of friends/bulletins? If the answer is a relational database (MySQL, PostgreSQL), then this should be done using a SQL query as it is quite trivial, and much more efficient than anything you could do in PHP.
Here is an example of how this could be done in SQL:
SELECT
posts.id
FROM posts
JOIN users ON posts.user_id = users.id
JOIN user_friends ON user_friends.user_id = users.id
WHERE posts.type = 'bulletin'
AND user_friends.user_id = 7
LIMIT 50;
Obviously it is done with no knowledge of your actual database structure, if any, and thus will not work as-is, but should get you on the right path.
Ok, so it sounds like you have an array of friend ids that is not associative, ie array(0 => 'userid0', 1 => 'userid1', etc), and an array of bulletin ids that is associative, ie. array('bulletin1' => 'userid1', 'bulletin2' => 'userid2', etc).
Going with that assumption, you can get all the matching bulletins using array_intersect(). You can then take the first fifty bulletin keys with array_slice():
$matchingBulletins = array_intersect($bulletins, $friends);
$first50 = array_slice(array_keys($matchingBulletins),0,50);
It sounds like you might be getting this data out of a database however, in which case it would be much more prudent to filter your database results somehow and avoid returning 10,000 ids each time. You could do the sorting and filtering using JOINs and WHEREs on the right tables.
I'm assuming here that your $friends array is just an array of ints and each item in your $bulletins is an array with userId and some extra fields.
$len = count($bulletins);
$matchedBulletins = array();
for ($i = 0; $i < $len; $i++) {
if (in_array($bulletins[$i]['userId'], $friends) {
$matchedBulletins[] = $bulletins[$i];
}
}
If you wan't to limit this array to like 50 first records only just add a condition inside a loop.
$len = count($bulletins);
$matchedBulletins = array();
$bulletinsCount = 0;
for ($i = 0; $i < $len; $i++) {
if (in_array($bulletins[$i]['userId'], $friends) {
$matchedBulletins[] = $bulletins[$i];
$bulletinsCount++
if ($bulletinsCount == 50) {
break;
}
}
}
If you'd post a bit of each array (not all 10,000 items, the first 10 would do) you may get more bites.
Check out array_search() in the meantime.