So i'm working with huge MYSQL database of english words. the list has upwards of 180k words. Im trying to run a query, and then compute something using every word in the database. The problem is the database is so big, and it seems to be freezing my browser (Im working locally with MAMP). Is there a way I can do five queries at a time, instead of trying to run through the whole thing, so i can see the results coming in gradually in my browser? Heres the code:
<?php
require 'class.thing.php';
require 'db_config.php';
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query");
while($row = mysql_fetch_array($result))
{
$word = $row['word'];
$compute = $word."thing";
$finished = new method( $compute );
if ($finished->successfull()) {
echo "<br/>";
echo "worked!";
echo "<br/>";
} else {
echo "uh oh..";
}
}
?>
** im looking for something like:
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query LIMIT 0,5");
get results
wait 5 seconds
$result = mysql_query("SELECT * FROM words") or die ("Could not complete query LIMIT 0,5");
get results
etc...
You should take a look at this related question:
What is the best way to paginate results in SQL Server
Note that you can compare words in SQL alphabetically (check whether your settings use case sensitive sorting or not), so you can do something like:
SELECT * FROM words WHERE word <= "ferrous" ORDER BY word
If you're using an auto-incrementing id, you can use that to subdivide as well (it'd probably be easier).
EDIT:
As pointed out below, you can use LIMIT in MySQL.
Example:
SELECT * FROM words LIMIT 0, 1000
(This would display the first 1000 results).
Related
I am having some trouble displaying a single field from the db based on an ID number.
I want to trying to display the "C_Type_of_Referral" if the ID matches. I am sure the code is a mess of crap but I am still at the VERY early stages of learning how to do things in mysql and php.
$query = "SELECT C_Type_of_Referral FROM Ctable WHERE C_ID = '70'";
$result = mysql_query($query) or die('Error : ' . mysql_error());
list($C_Type_of_Referral) = mysql_fetch_array($result);
Then I am trying to display the result using the code below:
echo ((isset ($_POST['C_Type_of_Referral']) && $_POST['C_Type_of_Referral'] == 'AS') || ($C_Type_of_Referral == 'AS'));
The code above returns the value of 1.
echo ($_POST['C_Type_of_Referral']);
The code above returns nothing.
Where am I going wrong?
Thanks for any help!
Let me start with: Use mysqli instead of mysql, especially when you are starting. Mysql is getting outdate, so start with the right one :)
$query = "SELECT C_Type_of_Referral FROM Ctable WHERE C_ID = 70 LIMIT 1";
$result = mysql_query($query) or die('Error : ' . mysql_error());
$fetch = mysql_fetch_assoc($result);
echo $fetch['C_Type_of_Referral']; // <- this is what you are looking for
if( $fetch['C_Type_of_Referral']=='AS'){ echo 'AdServices'; }
elseif( $fetch['C_Type_of_Referral']=='VS'){ echo 'VisServices'; }
else{ echo 'Nothing of the kind'; }
Things I've done:
- removed quotes arround 70. Integers should not use quotes.
- Added 'limit 1'. This will speed things up in the long run. If you want 1 line, use limit 1
- Changed fetch_array to fetch_assoc ( you'll have to find out why yourself ;) )
Small note: keep table and columnnames lowercase, some systems might give you troubles when you use uppercase characters
I have a question on how to go about the next phase of a project I am working on.
Phase I:
create a php script that scraped directory for all .txt file..
Open/parse each line, explode into array...
Loop through array picking out pieces of data that were needed and INSERTING everything into the database (120+ .txt files & 100k records inserted)..
this leads me to my next step,
Phase II:
I need to take a 'list' of several 10's of thousand of numbers..
loop through each one, using that piece of data (number) as the search term to QUERY the database.. if a match is found I need to grab a piece of data in a different column of the same record/row..
General thoughts/steps I plan to take
scrape directory to find 'source' text file.
open/parse 'source file'.... line by line...
explode each line by its delimiting character.. and grab the 'target search number'
dump each number into a 'master list' array...
loop through my 'master list' array.. using each number in my search (SELECT) statement..
if a match is found, grab a piece of data in another column in the matching/returned row (record)...
output this data.. either to screen or .txt file (havent decided on that step yet,..most likely text file through each returned number on a new line)
Specifics:
I am not sure how to go about doing a 'multiple' search/select statement like this?
How can I do multiple SELECT statements each with a unique search term? and also collect the returned column data?
is the DB fast enough to return the matching value/data in a loop like this? Do I need to wait/pause/delay somehow for the return data before iterating through the loop again?
thanks!
current function I am using/trying:
this is where I am currently:
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
$query = "SELECT guar_nu FROM placements WHERE har_id IN ($harNumArray2)";
echo $query;
$match = mysql_query($query);
//$match = mysql_query('"' . $query . '"');
$results = $match;
echo("<BR><BR>");
print_r($results);
I get these outputs respectively:
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN ("108383442","106620416","109570835","109700427","100022236")
&
Array ( [0] => sample_source.txt )
Total FILES TO GRAB HAR ID's FROM: 1
TOAL HARS FOUND IN ALL FILES: 5
SELECT guar_nu FROM placements WHERE har_id IN (108383442,106620416,109570835,109700427,100022236)
Where do I stick this to actually execute it now?
thanks!
update:
this code seems to be working 'ok'.. but I dont understand on how to handle the retirned data correctly.. I seem to only be outputting (printing) the last variable/rows data..instead of the entire list..
$harNumArray2 = implode(',', $harNumArray);
//$harNumArray2 = '"' . implode('","', $harNumArray) . '"';
//$query = "'SELECT guar_num FROM placements WHERE har_id IN ($harNumArray2)'";
$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (" . $harNumArray2 . ")")
//$result = mysql_query("SELECT har_id, guar_num FROM placements WHERE har_id IN (0108383442,0106620416)")
or die(mysql_error());
// store the record of the "example" table into $row
$row = mysql_fetch_array($result);
$numRows = mysql_num_rows($result);
/*
while($row = #mysql_fetch_assoc($result) ){
// do something
echo("something <BR>");
}
*/
// Print out the contents of the entry
echo("TOTAL ROWS RETURNED : " . $numRows . "<BR>");
echo "HAR ID: ".$row['har_id'];
echo " GUAR ID: ".$row['guar_num'];
How do I handle this returned data properly?
thanks!
I don't know if this answers your question but I think you're asking about sub-queries. They're pretty straightforward and just look something like this
SELECT * FROM tbl1 WHERE id = (SELECT num FROM tbl2 WHERE id = 1);
That will only work if there is one unique value to that second subquery. If it returns multiple rows it will return a parse error. If you have to select multiple rows research JOIN statements. This can get you started
http://www.w3schools.com/sql/sql_join.asp
I am not sure how to go about doing a 'multiple' search/select statement like this?
With regards to a multiple select, (and I'll assume that you're using MySQL) you can perform that simply with the "IN" keyword:
for example:
SELECT *
FROM YOUR_TABLE
WHERE COLUMN_NAME IN (LIST, OF, SEARCH, VALUES, SEPARATED, BY COMMAS)
EDIT: following your updated code in the question.
just a point before we go on... you should try to avoid the mysql_ functions in PHP for new code, as they are about to be deprecated. Think about using the generic PHP DB handler PDO or the newer mysqli_ functions. More help on choosing the "right" API for you is here.
How do I handle this returned data properly?
For handling more than one row of data (which you are), you should use a loop. Something like the following should do it (and my example will use the mysqli_ functions - which are probably a little more similar to the API you've been using):
$mysqli = mysqli_connect("localhost", "user", "pass");
mysqli_select_db($mysqli, "YOUR_DB");
// make a comma separated list of the $ids.
$ids = join(", ", $id_list);
// note: you need to pass the db connection to many of these methods with the mysqli_ API
$results = mysqli_query($mysqli, "SELECT har_id, guar_num FROM placements WHERE har_id IN ($ids)");
$num_rows = mysqli_num_rows($results);
while ($row = mysqli_fetch_assoc($results)) {
echo "HAR_ID: ". $row["har_id"]. "\tGUAR_NUM: " . $row["guar_num"] . "\n";
}
Please be aware that this is very basic (and untested!) code, just to show the bare minimum of the steps. :)
I am currently still learning PHP so some things I still struggle with.
I have been taking it slowly and reading tutorials which has helped but I can't figure this one out.
I have a database table (in mysql) with let's say, 100 urls. There is a column called 'url' and a second column 'text'. I already have the pagination code which works, so will also be using that.
What I want to do is echo out the URLs (which are all in folder called blog in the root of my site), but use the text as the link.
So for example the first three rows in my table might be:
url
001.php
002.php
003.php
text
random text
some random text
more text
when echoed out the links show the text from the column text like:
random text
some random text
more text
and will open to the relevant url when clicked
I'm guessing it will need some kind of loop to collect all the URLs and save me adding the link text in manually, and then my pagination code will split them up.
This is my first time asking a question on here, so if it wasn't clear enough or you need more info, let me know.
I have done multiple searches on the internet but can't seem to find a tutorial.
Assuming you connect to a local mysql server with username "root" and password "root", and have your url's stored in a table named url_table in a database named url_database you could do something like:
$connection = mysql_connect("127.0.0.1","root","root"); // Connect to the mysql server
mysql_select_db("url_database"); // Open the desired database
$query = "SELECT url,text FROM url_table"; // Query to select the fields in each row
$result = mysql_query($query); // Run the query and store the result in $result
while($row = mysql_fetch_assoc($result)) // While there are still rows, create an array of each
{
echo "<a href='".$row['url']."'>".$row['text']."</a>"; // Write an anchor with the url as href, and text as value/content
}
mysql_close($connection); // close the previously opened connection to the database
What you need is to:
get your result array from the database. Use something like
$query = "SELECT * FROM urls";
$result = mysql_query($query);
For every row in your results table, show the corresponding url. Note that calling mysql_fetch_array on a result resource, returns the first row of the results table when called for the first time, the second on second time etc. The function returns false when there are no more rows to return.
(See more on that in the mysql_fetch_array() documentation)
While( $row = mysql_fetch_array($result) ){
echo '<a href='.$row['url'].'>'.$row['text'].'</a>';
}
Here is a sample you can start with:
$con = mysql_connect("host","user","password");
if (!$con) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db("my_db", $con);
$result = mysql_query("SELECT the_url, the_text FROM my_table");
while($row = mysql_fetch_array($result))
{
echo '"' . $row['the_text'] . ' <br />';
}
mysql_close($con);
First, you want to select only the relevant lines in your database for the page that the user is currently viewing. For example, if the user is viewing page 2, with entries 15-30 present, we only want to pull those entries from the database. This is an efficiency concern.
The code you're after is something like this:
$link = mysql_connect(/*connection parameters go here*/);
if ($link === false)
die;
mysql_select_db('my_database');
$result = mysql_query("SELECT text, url FROM my_table LIMIT 15,30");
print "<ul>\n";
while ($row = mysql_fetch_assoc($result)) {
print "<li>{$row['text']}</li>\n";
}
print "</ul>\n";
mysql_close();
The first three lines establish a connection to the database, and exit the script if an error occurs.
The next line selects the appropriate database on the server.
The next block runs the appropriate query for the second page. After the query is run, a 'result' is stored in $result. This result is comprised of a number of rows, and mysql_fetch_assoc($result) obtains those lines one at a time. It then formats these lines into the appropriate link format and outputs them. The entire set of links is wrapped in a dot-point list.
Finally, mysql_close() closes the connection to the database.
I'm assuming since you just started you're probably doing all this procedurally.
First you want to query the database and get the info you need.
<?php
$result = mysqli_query($link, "SELECT url, text FROM table_name");
while ($row = mysqli_fetch_array($result)) $hrefs[] = $row;
foreach ($hrefs as $href) {
echo "".$href['text']."";
}
?>
Please note I've done no error handling here.
i will keep it simple to understand :
1) i have a database with 2 columns - word and text
2) Each word has approx. 600,000 lines of text associated with it.
3) Iam a .net person shifting to php and mysql - so a little knowledge.
MY requirement :
1) I will pass the word through a form
2) The form will connect to the database and should display 2000 random lines from those 600,000 which should be non-repetitive
My current progress :
<?php
$con = mysql_connect("localhost","text_minx","pwd");
if (!$con)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db("my_db", $con);
$result = mysql_query("SELECT * FROM Data
WHERE word='health'");
while($row = mysql_fetch_array($result))
{
echo $row['lines'];
echo "<br />";
}
?>
This shows all the lines. What i want is to read this $row['lines'] in a possible array and randomly select 2000 lines from it - and they should be non-repetitive.
Any help please?
Something like this:
$result = mysql_query("SELECT DISTINCT * FROM Data WHERE word='health' ORDER BY RAND() LIMIT 2000");
It's more efficient to select the 2000 random lines by MySQL instead of by PHP, as above.
Also note that SELECT DISTINCT will select only unique rows, which you should probably remove anyway. If you specify the column names instead of using * then you can choose which columns you want to be unique - although this also depends somewhat on how your table is built.
i think this is the way you want
SELECT * FROM Data
WHERE word='health' ORDER BY RAND() LIMIT 0,2000
this will give you 2000 records sorted by any random order
You need to split your row result into array (as it's currently a string), and then you can select 2000 lines randomly.
Assuming that the text has like breaks as line separators, it would look like that:
echo $row['lines']; //starts from this line of your code
$data = explode("\n", $row['lines']);
shuffle($data);
$random_lines = array_slice($data, 0, 2000);
This does not take care of non-repetition, though. If I understand your need correctly, you can use array_unique() function before passing it to shuffle().
I tried to use this function
$conn = db_connect();
while ($newsfeed = $conn->query("select info, username, time from newsfeed ORDER BY time DESC LIMIT 10"))
{
(...)
echo "<p>User $newsfeed_username just registerted ".$minutes." min ago </p><br>";
but it only shows the latest row over and over again. I want to loop through all the queries from
select info, username, time from newsfeed ORDER BY time DESC LIMIT 10
in descending order.
Here's the basic template for this kind of thing, using built-in php functions (assuming old-style mysql, but similar using other database back-ends, or higher-level libraries). In this example, errors are handled by throwing exceptions, but that's just one way to do it.
Connect to the database
Make sure connection was successful
Run the query
Make sure the query didn't fail for some reason (usually a SQL syntax error). If it did fail, find out why and handle that error
Check that the query returned at least one row (zero rows typically is a special case)
Loop over the returned rows, doing whatever it is you need done.
The exception classes would need to be defined (they're the only non-built-in syntax here, but you shouldn't throw plain-vanilla Exceptions).
Example Code:
<?PHP
//try to connect to your database.
$conn = mysql_connect(...);
//handle errors if connection failed.
if (! $conn){
throw new Db_Connect_Error(..);
}
// (try to) run your query.
$resultset = mysql_query('SELECT ...');
//handle errors if query failed. mysql_error() will give you some handy hints.
if (! $resultset){
// probably a syntax error in your SQL,
// but could be some other error
throw new Db_Query_Exception("DB Error: " . mysql_error());
}
//so now we know we have a valid resultset
//zero-length results are usually a a special case
if (mysql_num_rows($resultset) == 0){
//do something sensible, like tell the user no records match, etc....
}else{
// our query returned at least one result. loop over results and do stuff.
while($row = mysql_fetch_assoc($resultset)){
//do something with the contents of $row
}
}
First you don't want to loop though queries. You want to loop through records which query will return.
Second you could do that, this way:
$conn = db_connect();
$query = mysql_query("SELECT info, username, time FROM newsfeed ORDER BY time DESC LIMIT 10");
while(($row = mysql_fetch_assoc($query)) != NULL) {
echo "<p>User {$row['username']} just registered {$minutes} min ago</p><br />";
}
NB! Assuming, that this db_connect() makes a mysql connection.
You need to store the result of $conn-query() in a variable before entering the loop. Right now you're running the query over and over again with each iteration of the loop which will always give you the first result.
Example
$conn = db_connect();
$result = $conn->query("select info, username, time from newsfeed ORDER BY time DESC LIMIT 10");
foreach ($result as $newsfeed)
{
(...)
echo "<p>User $newsfeed_username just registerted ".$minutes." min ago </p><br>";