i will keep it simple to understand :
1) i have a database with 2 columns - word and text
2) Each word has approx. 600,000 lines of text associated with it.
3) Iam a .net person shifting to php and mysql - so a little knowledge.
MY requirement :
1) I will pass the word through a form
2) The form will connect to the database and should display 2000 random lines from those 600,000 which should be non-repetitive
My current progress :
<?php
$con = mysql_connect("localhost","text_minx","pwd");
if (!$con)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db("my_db", $con);
$result = mysql_query("SELECT * FROM Data
WHERE word='health'");
while($row = mysql_fetch_array($result))
{
echo $row['lines'];
echo "<br />";
}
?>
This shows all the lines. What i want is to read this $row['lines'] in a possible array and randomly select 2000 lines from it - and they should be non-repetitive.
Any help please?
Something like this:
$result = mysql_query("SELECT DISTINCT * FROM Data WHERE word='health' ORDER BY RAND() LIMIT 2000");
It's more efficient to select the 2000 random lines by MySQL instead of by PHP, as above.
Also note that SELECT DISTINCT will select only unique rows, which you should probably remove anyway. If you specify the column names instead of using * then you can choose which columns you want to be unique - although this also depends somewhat on how your table is built.
i think this is the way you want
SELECT * FROM Data
WHERE word='health' ORDER BY RAND() LIMIT 0,2000
this will give you 2000 records sorted by any random order
You need to split your row result into array (as it's currently a string), and then you can select 2000 lines randomly.
Assuming that the text has like breaks as line separators, it would look like that:
echo $row['lines']; //starts from this line of your code
$data = explode("\n", $row['lines']);
shuffle($data);
$random_lines = array_slice($data, 0, 2000);
This does not take care of non-repetition, though. If I understand your need correctly, you can use array_unique() function before passing it to shuffle().
Related
I'm trying to figure out a way to limit the amount of information pulled from a database, that is displayed within a table as like an overview type function.
Let's say the information pulled from the database has 1000 words, but I only want the table to display 150 characters and then display "..." to indicated there is more information.
Is this possible? Like a maxlength or overflow property?
Thanks.
You could use substr() to limit the length of returned items.
$result = mysql_query("SELECT * FROM table");
while($row = mysql_fetch_array($result)) {
// returns the first 150 characters followed by "..."
$strIn = $row['database_column'];
$strOut = substr($strIn, 150) . '...';
echo('<p>'.$strOut.'</p>');
}
If you want to check to make sure it splits properly between words, you'll want to do some further string manipulation.
I am creating a project which involves getting some questions from mysql database. For instance, if I have 200 questions in my database, I want to randomly choose 20 questions in such a way that no one question will be repeated twice. That is, I want to be able to have an array of 20 different questions from the 200 I have every time the user tries to get the list of questions to answer. I will really appreciate your help.
SELECT * FROM questions ORDER BY RAND() LIMIT 20;
PS^ This method not possible for very big tables
Use Google to find a function to create an array with 20 unique numbers, with a minimum and a maximum. Use this array to prepare an SQL query such as:
expression IN (value1, value2, .... value_n);
More on the SQL here.
Possible array filling function here too.
Assuming you have contiguously number questions in your database, you just need a list of 20 random numbers. Also assuming you want the user to be able to take more than one test and get another 20 questions without duplicates then you could start with a randomised array of 200 numbers and select blocks of 20 sequentially from that set i.e.
$startQuestion=1;
$maxQuestion=200;
$numberlist= range(1,$maxQuestion);
shuffle($numberlist);
function getQuestionSet( $noOfQuestions )
{
global $numberlist, $maxQuestion, $startQuestion;
$start= $startQuestion;
if( ($startQuestion+$noOfQuestions) > $maxQuestion)
{
echo "shuffle...\n";
shuffle($numberlist);
$startQuestion=1;
}else
$startQuestion+= $noOfQuestions;
return array_slice($numberlist,$start,$noOfQuestions);
}
// debug...
for($i=0; $i<42; $i++)
{
$questionset= getQuestionSet( 20 );
foreach( $questionset as $num )
echo $num." ";
echo "\n";
}
then use $questionset to retrieve your questions
If you know how many rows there are in the table, you could do use LIMIT to your advantage. With limit you specify a random offset; syntax: LIMIT offset,count. Example:
<?php
$totalRows = 200; // get his value dynamically or whatever...
$limit = 2; // num of rows to select
$rand = mt_rand(0,$totalRows-$limit-1);
$query = 'SELECT * FROM `table` LIMIT '.$rand.','.$limit;
// execute query
?>
This should be safe for big tables, however it will select adjacent rows. You could then mix up the result set via array_rand or shuffle:
<?php
// ... continued
$resultSet = $pdoStm->fetchAll();
$randResultKeys = array_rand($resultSet,$limit); // using array_rand
shuffle($resultSet); // or using shuffle
?>
So i'm working with huge MYSQL database of english words. the list has upwards of 180k words. Im trying to run a query, and then compute something using every word in the database. The problem is the database is so big, and it seems to be freezing my browser (Im working locally with MAMP). Is there a way I can do five queries at a time, instead of trying to run through the whole thing, so i can see the results coming in gradually in my browser? Heres the code:
<?php
require 'class.thing.php';
require 'db_config.php';
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query");
while($row = mysql_fetch_array($result))
{
$word = $row['word'];
$compute = $word."thing";
$finished = new method( $compute );
if ($finished->successfull()) {
echo "<br/>";
echo "worked!";
echo "<br/>";
} else {
echo "uh oh..";
}
}
?>
** im looking for something like:
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query LIMIT 0,5");
get results
wait 5 seconds
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query LIMIT 0,5");
get results
etc...
You should take a look at this related question:
What is the best way to paginate results in SQL Server
Note that you can compare words in SQL alphabetically (check whether your settings use case sensitive sorting or not), so you can do something like:
SELECT * FROM words WHERE word <= "ferrous" ORDER BY word
If you're using an auto-incrementing id, you can use that to subdivide as well (it'd probably be easier).
EDIT:
As pointed out below, you can use LIMIT in MySQL.
Example:
SELECT * FROM words LIMIT 0, 1000
(This would display the first 1000 results).
I have a question on how to go about the next phase of a project I am working on.
Phase I:
create a php script that scraped directory for all .txt file..
Open/parse each line, explode into array...
Loop through array picking out pieces of data that were needed and INSERTING everything into the database (120+ .txt files & 100k records inserted)..
this leads me to my next step,
Phase II:
I need to take a 'list' of several 10's of thousand of numbers..
loop through each one, using that piece of data (number) as the search term to QUERY the database.. if a match is found I need to grab a piece of data in a different column of the same record/row..
General thoughts/steps I plan to take
scrape directory to find 'source' text file.
open/parse 'source file'.... line by line...
explode each line by its delimiting character.. and grab the 'target search number'
dump each number into a 'master list' array...
loop through my 'master list' array.. using each number in my search (SELECT) statement..
if a match is found, grab a piece of data in another column in the matching/returned row (record)...
output this data.. either to screen or .txt file (havent decided on that step yet,..most likely text file through each returned number on a new line)
Specifics:
I am not sure how to go about doing a 'multiple' search/select statement like this?
How can I do multiple SELECT statements each with a unique search term? and also collect the returned column data?
is the DB fast enough to return the matching value/data in a loop like this? Do I need to wait/pause/delay somehow for the return data before iterating through the loop again?
thanks!
current function I am using/trying:
this is where I am currently:
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
$query = "SELECT guar_nu FROM placements WHERE har_id IN ($harNumArray2)";
echo $query;
$match = mysql_query($query);
//$match = mysql_query('"' . $query . '"');
$results = $match;
echo("<BR><BR>");
print_r($results);
I get these outputs respectively:
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN ("108383442","106620416","109570835","109700427","100022236")
&
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN (108383442,106620416,109570835,109700427,100022236)
Where do I stick this to actually execute it now?
thanks!
update:
this code seems to be working 'ok'.. but I dont understand on how to handle the retirned data correctly.. I seem to only be outputting (printing) the last variable/rows data..instead of the entire list..
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
//$query = "'SELECT guar_num FROM placements WHERE har_id IN ($harNumArray2)'";
$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (" . $harNumArray2 . ")")
//$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (0108383442,0106620416)")
or die(mysql_error());
// store the record of the "example" table into $row
$row = mysql_fetch_array($result);
$numRows = mysql_num_rows($result);
/*
while($row = #mysql_fetch_assoc($result) ){
// do something
echo("something <BR>");
}
*/
// Print out the contents of the entry
echo("TOTAL ROWS RETURNED : " . $numRows . "<BR>");
echo "HAR ID: ".$row['har_id'];
echo " GUAR ID: ".$row['guar_num'];
How do I handle this returned data properly?
thanks!
I don't know if this answers your question but I think you're asking about sub-queries. They're pretty straightforward and just look something like this
SELECT * FROM tbl1 WHERE id = (SELECT num FROM tbl2 WHERE id = 1);
That will only work if there is one unique value to that second subquery. If it returns multiple rows it will return a parse error. If you have to select multiple rows research JOIN statements. This can get you started
http://www.w3schools.com/sql/sql_join.asp
I am not sure how to go about doing a 'multiple' search/select statement like this?
With regards to a multiple select, (and I'll assume that you're using MySQL) you can perform that simply with the "IN" keyword:
for example:
SELECT *
FROM YOUR_TABLE
WHERE COLUMN_NAME IN (LIST, OF, SEARCH, VALUES, SEPARATED, BY COMMAS)
EDIT: following your updated code in the question.
just a point before we go on... you should try to avoid the mysql_ functions in PHP for new code, as they are about to be deprecated. Think about using the generic PHP DB handler PDO or the newer mysqli_ functions. More help on choosing the "right" API for you is here.
How do I handle this returned data properly?
For handling more than one row of data (which you are), you should use a loop. Something like the following should do it (and my example will use the mysqli_ functions - which are probably a little more similar to the API you've been using):
$mysqli = mysqli_connect("localhost", "user", "pass");
mysqli_select_db($mysqli, "YOUR_DB");
// make a comma separated list of the $ids.
$ids = join(", ", $id_list);
// note: you need to pass the db connection to many of these methods with the mysqli_ API
$results = mysqli_query($mysqli, "SELECT har_id, guar_num FROM placements WHERE har_id IN ($ids)");
$num_rows = mysqli_num_rows($results);
while ($row = mysqli_fetch_assoc($results)) {
echo "HAR_ID: ". $row["har_id"]. "\tGUAR_NUM: " . $row["guar_num"] . "\n";
}
Please be aware that this is very basic (and untested!) code, just to show the bare minimum of the steps. :)
I am trying to query an SQL table, and then have all the values from one column (it's a "tinyint(1)") added up and then printed, using array_sum. (Currently there are only two rows).
When both rows have a "1" in that column, it gives 2, the correct answer. And when they are both "0" it gives 0, correct again. But when one is "1" and one "0" it gives 2, not 1.
This is my code
$con = mysql_connect("XXX","XXX","XXX");
if (!$con)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db("XXX_test", $con);
$day = $_GET["day"];
$query = mysql_query("SELECT $day FROM sites");
if (!$query)
{
die('Could not query: ' . mysql_error());
}
$rows = mysql_fetch_array($query);
echo array_sum($rows);
mysql_close($con);
?>
Thanks for any help.
--UPDATE--
I just solved this myself.
With this, if you want to know:
$query = mysql_query("SELECT name, SUM($day) AS Alerts FROM sites");
while($row = mysql_fetch_array($query))
{
echo $row['Alerts'];
echo "<br />";
}
This in fact never works correctly.
$rows = mysql_fetch_array($query);
That line only ever fetches the first row. The second row is never retrieved.
The reason it gives 2 using array_sum is that mysql_fetch_array gets the result in two different formats, with keys both by name and number. So you'll get something like this:
array(
0 => '1',
'Tuesday' => '1'
)
Obviously this will always be 2 or 0.
You should almost certainly do a count on the DB server:
SELECT COUNT(*) FROM sites WHERE $day=1
Note, however, that your biggest current problem is the enormous gaping SQL injection hole. Sanitise your input.
mysql_fetch_array() only fetches one row at a time. You need to fetch the rows in a while loop. Then you can either append the results to an array and sum those, or sum them in the loop. You can also SUM in mysql anyway, so you might look into using that.
I'm also obligated to suggest that you use PDO or some other DB wrapper instead of mysql_. You also need to escape variables you use in queries under most circumstances, especially if they come from _GET.
I would suggest doing it in the mysql query:
SELECT SUM($day) FROM sites
also, please escape the $day variable.
More of a comment:
array_sum can not work with the array you're passing to it. You need to pass an array for what the function actually works, like array(1, 2, 3) which will give you 6 then.
Next to that, let MySQL itself calculate the SUM (or COUNT?), so it's easier to handle in your script as others have posted.
I just solved this myself.
With this, if you want to know:
$query = mysql_query("SELECT name, SUM($day) AS Alerts FROM sites");
while($row = mysql_fetch_array($query))
{
echo $row['Alerts'];
echo "<br />";
}