I made a simple search box on a page, where a user can type in keywords to look for photos of certain items, using PHP. I'm using an MySQL database. I trim the result and show only 10 to make the loading quicker, but certain set of keywords causes the browser to hang on both IE and Firefox. When this happens on IE, I can see outlines of photos (just the silhouette) beyond the 10 results with an "X" mark at the top right corner, similar to when you load a photo and the photo doesn't exist on a webpage, even though I wrote the code to show only 10 results. The database has over 10,000 entries, and I'm thinking maybe it's trying to display the entire set of photos in the database. Here are some code that I'm using.
I'm using the function below to create the query. $keyword is an array of the keywords that the user has typed in.
function create_multiword_query($keywords) {
// Creates multi-word text search query
$q = 'SELECT * FROM catalog WHERE ';
$num = 0;
foreach($keywords as $val) { // Multi-word search
$num++;
if ($num == 1) {
$q = $q . "name LIKE '%$val%'"; }
else {
$q = $q . " AND name LIKE '%$val%'";}
}
$q = $q . ' ORDER BY name';
return $q;
//$q = "SELECT * FROM catalog WHERE name LIKE \"%$trimmed%\" ORDER BY name";
}
And display the result. MAX_DISPLAY_NUM is 10.
$num = 0;
while (($row = mysqli_fetch_assoc($r)) && ($num < MAX_DISPLAY_NUM)) { // add max search result!
$num++;
print_images($row['img_url'], '/_', '.jpg'); // just prints photos
}
I'm very much a novice with PHP, but I can't seem to find anything wrong with my code. Maybe the way I wrote these algorithms are not quite right for PHP or MySQL? Can you guys help me out with this? I can post more code as necessary. TIA!!
Don't limit your search results in PHP, limit them in the SQL query with the LIMIT keyword.
As in:
select * form yourtable where ... order by ... limit 10;
BTW, those LIKE '%something%' can be expensive. Maybe you should look at Full text indexing and searching.
If you want to show a More... link or something like that, one way of doing it would be to limit your query to 11 and only show the first ten.
Apart from the LIMIT in your query, I would check out mysql full text search (if your tables have the MyISAM format).
Why don't use use MySQL to limit the number of search results returned?
http://dev.mysql.com/doc/refman/5.0/en/select.html
add LIMIT to your query.
you are retrieving all rows from DB (lot of bytes traveling from DB to server) and then you are filtering the first 10 rows.
try
$q = $q . ' ORDER BY name LIMIT 10';
LIKE is slow also according to Flickr(slides 24-26). You should first try to use FULL TEXT indexes instead. If your site still seems slow there are also some other really fast(er)/popular alternatives available:
sphinx
elasticsearch
solr
The only thing that is a little bit annoying that you need to learn/install these technologies, but are well worth the investment when needed.
Related
I have the following query for a type-ahead search (as you type into the form it displays matches in a drop down). This query worked well until I switched to a database with about a million records. Now it takes 15 seconds for the match to be displayed.
Because search hits are displayed as you type, the query is inside a loop. Is there anything about this query that can be changed to speed it up?
$diagnosis = isset($_GET['diagnosis']) ? $_GET['diagnosis'] : '';
$data = array();
if ($diagnosis) {
$query = explode(' ', $diagnosis);
for ($i = 0, $c = count($query); $i < $c; $i ++) {
$query[$i] = '+' . mysql_real_escape_string($query[$i]) . '*';
}
$query = implode(' ', $query);
$sql = "SELECT diagnosis, icd9, MATCH(diagnosis) AGAINST('$query' IN BOOLEAN MODE) AS relevance
FROM icd10 WHERE MATCH(diagnosis) AGAINST('$query' IN BOOLEAN MODE) HAVING relevance > 0 ORDER BY relevance ";
$r = mysql_query($sql);
while ($row = mysql_fetch_array($r)) {
$data[] = $row;
}
}
echo json_encode($data);
exit;
You can try some stuff:
First, make sure you have a fulltext index for diagnosis. Second, make sure you have a fulltext index for diagnosis! A million rows isn't that much (depending on the number of words in diagnosis of course), so that just already might be the problem.
Then try the following code:
SELECT diagnosis, icd9, MATCH(diagnosis) AGAINST('$query' IN BOOLEAN MODE) AS relevance
FROM icd10 ORDER BY relevance desc limit 30
(It might not be obvious that this is faster, and it might not be, so just try it).
If you need to support short words, e.g. if 3 digit icd9-codes are entered often, you should check your ft_min_word_len / innodb_ft_min_token_size-values (depending on your database) to make sure they are included in the index - but be aware it will increase your index size. Maybe check the stopwords.
You didn't specify your setup; you can often improve general database performance by e.g. changing settings, hdds or ram. Especially ram.
Some general ideas: You might want to call the function asynchronously (the user should be able to type while the query runs). As soon as you hit less than 30 results (or whatever limit you set), you can just filter the remaining results on the fly in php (as long as the query gets longer/no words are removed) - it's the closest you get to a cache. Or set the limit to 1000 and filter manually afterwards, php regex is fast too, you just need a score-function.
Depending on your data, you might want to not run the query when you just add a single letter to the query (every text will contain a word beginning with an "a", so you might not get a better result - that might not be the case for "q" though). That won't reduce runtime of the query, but you can just save one execution.
I'm trying to create a simple search engine where users can query a database and be returned results that both match and are close to their query. At first I was just using wildcards (%) to find results that were relevant to a users search. The PHP for that looked something like this:
// Users search terms is saved in $_POST['q']
$q = $_POST['q'];
// Prepare statement
$search = $db->prepare("SELECT `id`, `name` FROM `users` WHERE `name` LIKE ?");
// Execute with wildcards
$search->execute(array("%$q%"));
// Echo results
foreach($search as $s) {
echo $s['name'];
}
The above code works fine, however, it's rather limited. While it can fetch results that are close to but don't exactly match the users query (because of the wildcards), it still doesn't return all relevant results; the user's query still has to have an exact match to something in the database. For example, if I had a database with the name "Tim" as a row, searching for "Timothy" wouldn't work. So my new approach looks something like this:
// Users search terms is saved in $_POST['q']
$q = $_POST['q'];
// Create array for the names that are close to or match the search term
$results = array();
foreach($db->query('SELECT `id`, `name` FROM `users`') as $name) {
// Keep only relevant results
if (levenshtein($q, $name['name']) < 4) {
array_push($results,$name['name']);
}
}
// Echo out results
foreach ($results as $result) {
echo $result."\n";
}
This code technically works, however, it's pretty inefficient and I'm wondering how it can be improved. The biggest problem is that as all results have to be retrieved from the database and then sorted, an unnecessarily large SQL query is created, which is especially problematic as I have a big database. Furthermore I wanted to know if simply using the levenshtein function is sufficient for getting relevant results, or if there is a better way to sort out the irrelevant results. Some other ways of sorting the relevant results I came up with:
if (levenshtein(metaphone($q), metaphone($name['name'])) < 4) {
array_push($results,$name['name']);
}
or
if (similar_text(metaphone($q), metaphone($name)['name']) < 2) {
array_push($results,$name['name']);
}
or
if (similar_text($q, $name['name']) > 2) {
array_push($results,$name['name']);
}
I think using levenshtein with metaphone may actually work the best as it would better take into account simple spelling errors. But I'm not sure which would be the best to use, especially considering that the way I'm doing it now is already very expensive (the large SQL query + the expensive functions that take place in a loop don't bode well for performance).
Thanks in advance
I wrote a PHP/MySQLi frontend, in which the user can enter SQL queries, and the server then returns the results in a table (or prints OK on INSERTs and UPDATEs)
As printing the results can take a very long time (e.g. SELECT * FROM movies) in a IMDb extract with about 1.6M movies, 1.9M actors and 3.2M keywords, I limited the output to 50 rows by cancelling the printing for-loop after 50 iterations.
However, the queries themselves also take quite some time, so I hoped that it might be possible to set a global maximum row return value, nevertheless whether the LIMIT keyword is used or not. I only intended to use the server for my own practice, but as some people in my class are struggling with the frontend provided by the teacher (Windows EXE, but half of the class uses Mac/Linux), I decided to make it accessible to them, too. But I want to keep my Debian VM from crashing because of - well, basically it would be a DDoS.
For clarification (examples with a global limit of 50):
SELECT * FROM movies;
> First 50 rows
SELECT * FROM movies LIMIT 10;
> First 10 rows
SELECT * FROM movies LIMIT 50,100;
> 50 rows (from 50 to 99)
Is there any possibility to limit the number of returned values using either PHP/MySQLi or the MySQL server itself? Or would I have to append/replace LIMIT to/in the queries?
You can use there queries and add "LIMIT 50" to it.
And if they added LIMIT by them self just filter it out with regex and still add your LIMIT.
I believe you have to build yourself a paginator anyway, avoiding to use LIMIT statement is not really possible i believe.
Here is what I would suggest for a Paginator:
if($_REQUEST['page'] == ""){
$page = 1;
}else{
$page = $_REQUEST['page']; // perhaps double check if numeric
}
$perpage = 50;
$start = ($page - 1) * $perpage;
$limit_string = " LIMIT ". $start . "," . $perpage ;
$query = "SELECT * FROM movies";
$query .= $limit_string;
Hope that helps
You can create a function.
https://dev.mysql.com/doc/refman/5.0/en/create-function.html
Let us know if this helps.
I am pulling some data from mysql database using PHP, now I would like to place that data in a html table which I have completed, how can I make the table pageable if it goes over say 10 records? Is there a tutorial I can look into or any information where I can get this? Maybe a tool I can implement easily? I just haven't found anything online about this topic but perhaps anyone here can lead me in the correct direction
I am currently using just a simple <table></table> html
You can achieve paging with MySQL's LIMIT keyword.
You can then use a query string to tell the website which page to get.
First we need to set a default page number and define how many results we want to display in the page:
$items_per_page = 10;
$page = 1;
if(isset($_GET['page'])) {
$page = (int)$_GET['page'];
}
The LIMIT keyword works by providing an offset and the number of rows you want to limit to. So now we need to figure out the offset:
$offset = ($page - 1) * $items_per_page;
Now we have all of the information we need to limit the results correctly based on the page number in our query string:
$query = "SELECT column_1, column_2 FROM your_table LIMIT {$offset}, {$items_per_page};";
$result = mysql_query($query) or die('Error, query failed');
while($row = mysql_fetch_assoc($result)) {
echo $row['column_1'] . '<br />';
}
Now to show your different pages you just add the query string to the end of your page URI.
For example my_page.php?page=1 or my_page.php?page=2
Perhaps you could try to figure out how to create the paging links by yourself and post more if you can't get it to work.
You just need to find out the total rows in your query with COUNT in MySQL and you can do all of the maths from there ;)
You'll need some javascript in order to do paging..
Take a look at http://flexigrid.info/
In summary, what you should do is have your php script return the tabular data as JSON or XML, and then feed it to flexgrid.
You can also have flexgrid request records when they're needed using additional requests.
I'm writing a filter/sorting feature for an application right now that will have text fields above each column. As the user types in each field, requests will be sent to the back-end for sorting. Since there are going to be around 6 text fields, I was wondering if there's a better way to sort instead of using if statements to check for each variable, and writing specific queries if say all fields were entered, just one, or just two fields, etc.
Seems like there would be a lot of if statements. Is there a more intuitive way of accomplishing this?
Thanks!
Any initial data manipulation, such as sorting, is usually done by the database engine.
Put an ORDER BY clause in there, unless you have a specific reason the sorting needs done in the application itself.
Edit: You now say that you want to filter the data instead. I would still do this at the database level. There is no sense in sending a huge dataset to PHP, just for PHP to have to wade through it and filter out data there. In most cases, doing this within MySQL will be far more efficient than what you can build in PHP.
Since there are going to be around 6 text fields, I was wondering if there's a better way to sort instead of using if statements to check for each variable
Definitely NO.
First, nothing wrong in using several if's in order.
Trust me - I myself being a huge fan of reducing repetitions of code, but consider these manually written blocks being the best solution.
Next, although there can be a way to wrap these condition ns some loop, most of time different conditions require different treatment.
however, in your next statements you are wrong:
and writing specific queries
you need only one query
Seems like there would be a lot of if statements.
why? no more than number of fields you have.
here goes a complete example of custom search query building code:
$w = array();
$where = '';
if (!empty($_GET['rooms'])) $w[]="rooms='".mesc($_GET['rooms'])."'";
if (!empty($_GET['space'])) $w[]="space='".mesc($_GET['space'])."'";
if (!empty($_GET['max_price'])) $w[]="price < '".mesc($_GET['max_price'])."'";
if (count($w)) $where="WHERE ".implode(' AND ',$w);
$query="select * from table $where";
the only fields filled by the user going to the query.
the ordering is going to be pretty the same way.
mesc is an abbreviation for the mysql_real_escape_string or any other applicable database-specific string escaping function
select * from Users
order by Creadted desc, Name asc, LastName desc, Status asc
And your records will be sorted by order from query.
First by Created desc, then by Name asc and so on.
But from your question I can see that you are searching for filtering results.
So to filter by multiple fileds just append your where, or if you are using any ORM you can do it through object methods.
But if its simple you can do it this way
$query = "";
foreach($_POST['grid_fields'] as $key => $value)
{
if(strlen($query) > 0)
$query .= ' and '
$query .= sprintf(" %s LIKE '%s' ", mysql_real_escape_string($key), '%' .mysql_real_escape_string($value) .'%');
}
if(strlen($query) > 0)
$original_query .= ' where ' . $query;
this could help you to achieve your result.
No. You cannot avoid the testing operations when sorting the set, as you have to compare the elements in the set in same way. The vehicle for this is an if statement.
Could you take a look at this?
WHERE (ifnull(#filter1, 1) = 1 or columnFilter1 = #filter1)
and (ifnull(#filter2, 1) = 1 or columnFilter2 = #filter2)
and (ifnull(#filter3, 1) = 1 or columnFilter3 = #filter3)
and (ifnull(#filter4, 1) = 1 or columnFilter4 = #filter4)
and (ifnull(#filter5, 1) = 1 or columnFilter5 = #filter5)
and (ifnull(#filter6, 1) = 1 or columnFilter6 = #filter6)
Please let me know if I'm misunderstanding your question.. It's not like an IF statement batch, and is pretty lengthy, but what do you think?