How can I sort a query based on the average Rating in a field in my table, the field itself is JSON text, structured like:
[
{"Type":1,"Rating":5},
{"Type":2,"Rating":5},
{"Type":3,"Rating":5}
]
I need my query to be sorted by the average of the 3 Ratings. There will always ever be only 3 values for this.
My current query is:
SELECT `Name`, `Town`, `Vehicle`, `Review`, `Rating`, `Pics`, `PostedOn`
FROM `tbl_ShopReviews`
WHERE `Approved` = 1
ORDER BY `PostedOn` DESC
Current results:
Name Town Vehicle Review Rating Pics PostedOn
Kevin Chicopee 94 Corolla Great stuff, very glad I brought it here [{"Type":1,"Rating":5},{"Type":2,"Rating":5},{"Type":3,"Rating":5}] \N
Just for those like me, who googles and tries to find solution for laravel 5.4. You can use -> operator to extract JSON, that is equal to json_extract function or to column->"$.key" syntax.
$users->orderBy('column->key', 'desc');
Looks like it would be very useful in late 2013 (smile).
For example field_name has the value like
{"json_field_key1":"2016/11/24","json_field_key2":"value"}
Use this code for get the json field json_field_key1 based value in ORDER BY case
select table.*, SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1) as json_field_name from table ORDER BY SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1) DESC
If your value is in date format just modify this code like
order by DATE(SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1)) DESC
Would this feature from MySQL 5.7 help? http://blog.ulf-wendel.de/2013/mysql-5-7-sql-functions-for-json-udf/
In my opinion the best solution is to be able to hook up update and insert events in your application for that reviews table and calculate the average in to another field.
Then those queries that need this info will be much easier to handle and will have a better performance.
The better solution is to parse the data before the insert, and have it ready for you in 3 columns or in 1 normalized column.
Saying that, if you're dealing with a non-changeable situation, and have exactly 3 ratings always, you can try this
ORDER BY (substring(json, 21, 1)+
substring(json, 43, 1)+
substring(json,65, 1))/3 desc;
Please consider that this solution is the least maintainable and flexible of them all, and very bug prone. The real solution is restructuring your data.
There isn't an easy way to do this, in fact, I'm not even sure it's possible.
That being said, your database structure really shouldn't contain JSON if it's something you need access to in this respect, instead, why not add a Type field to your database?
the Rating field could be a table with user, type and vlaue as columns where the user is the key. Then you can just use mysql AVG() on the value column where the key match and then sort to that.
hope this help
Related
As i am a junior PHP Developer growing day by day stuck in a performance problem described here:
I am making a search engine in PHP ,my database has one table with 41 column and million's of rows obviously it is a very large dataset. In index.php i have a form for searching data.When user enters search keyword and hit submit the action is on search.php with results.The query is like this.
SELECT * FROM TABLE WHERE product_description LIKE '%mobile%' ORDER BY id ASC LIMIT 10
This is the first query.After result shows i have to run 4 other query like this:
SELECT DISTINCT(weight_u) as weight from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(country_unit) as country_unit from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(country) as country from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(hs_code) as hscode from TABLE WHERE product_description LIKE '%mobile%'
These queries are for FILTERS ,the problem is this when i submit search button ,all queries are running simultaneously at the cost of Performance issue,its very slow.
Is there any other method to fetch weight,country,country_unit,hs_code speeder or how can achieve it.
The same functionality is implemented here,Where the filter bar comes after table is filled with data,How i can achieve it .Please help
Full Functionality implemented here.
I have tried to explain my full problem ,if there is any mistake please let me know i will improve the question,i am also new to stackoverflow.
Firstly - are you sure this code is working as you expect it? The first query retrieves 10 records matching your search term. Those records might have duplicate weight_u, country_unit, country or hs_code values, so when you then execute the next 4 queries for your filter, it's entirely possible that you will get values back which are not in the first query, so the filter might not make sense.
if that's true, I would create the filter values in your client code (PHP)- finding the unique values in 10 records is going to be quick and easy, and reduces the number of database round trips.
Finally, the biggest improvement you can make is to use MySQL's fulltext searching features. The reason your app is slow is because your search terms cannot use an index - you're wild-carding the start as well as the end. It's like searching the phonebook for people whose name contains "ishra" - you have to look at every record to check for a match. Fulltext search indexes are designed for this - they also help with fuzzy matching.
I'll give you some tips that will show useful in many situations when querying a large dataset, or mostly any dataset.
If you can list the fields you want instead of querying for '*' is a better practice. The weight of this increases as you have more columns and more rows.
Always try to use the PK's to look for the data. The more specific the filter, the less it will cost.
An index in this kind of situation would come pretty handy, as it will make the search more agile.
LIKE queries are generally pretty slow and resource heavy, and more in your situation. So again, the more specific you are, the better it will get.
Also add, that if you just want to retrieve data from this tables again and again, maybe a VIEW would fit nicely.
Those are just some tips that came to my mind to ease your problem.
Hope it helps.
I'm storing a list of items in a serialized array within a field in my database (I'm using PHP/MySQL).
I want to have a query that will select all the records that contain a specific one of these items that is in the array.
Something like this:
select * from table WHERE (an item in my array) = '$n'
Hopefully that makes sense.
Any ideas would be greatly appreciated.
Thanks
As GWW says in the comments, if you need to query things this way, you really ought to be considering storing this data as something other than a big-ole-string (which is what your serialized array is).
If that's not possible (or you're just lazy), you can use the fact that the serialized array is just a big-ole-string, and figure out a LIKE clause to find matching records. The way PHP serializes data is pretty easy to figure out (hint: those numbers indicate lengths of things).
Now, if your serialized array is fairly complex, this will break down fast. But if it's a flat array, you should be able to do it.
Of course, you'll be using LIKE '%...%', so you'll get no help from any indicies, and performance will be very poor.
Which is why folks are suggesting you store that data in some normalized fashion, if you need to query "inside" it.
If you have control of the data model, stuffing serialized data in the database will bite you in the long run just about always. However, oftentimes one does not have control over the data model, for example when working with certain open source content management systems. Drupal sticks a lot of serialized data in dumpster columns in lieu of a proper model. For example, ubercart has a 'data' column for all of its orders. Contributed modules need to attach data to the main order entity, so out of convenience they tack it onto the serialized blob. As a third party to this, I still need a way to get at some of the data stuffed in there to answer some questions.
a:4:{s:7:"cc_data";s:112:"6"CrIPY2IsMS1?blpMkwRj[XwCosb]gl<Dw_L(,Tq[xE)~(!$C"9Wn]bKYlAnS{[Kv[&Cq$xN-Jkr1qq<z](td]ve+{Xi!G0x:.O-"=yy*2KP0#z";s:7:"cc_txns";a:1:{s:10:"references";a:1:{i:0;a:2:{s:4:"card";s:4:"3092";s:7:"created";i:1296325512;}}}s:13:"recurring_fee";b:1;s:12:"old_order_id";s:2:"25";}
see that 'old_order_id'? thats the key I need to find out where this recurring order came from, but since not everybody uses the recurring orders module, there isnt a proper place to store it in the database, so the module developer opted to stuff it in that dumpster table.
My solution is to use a few targeted SUBSTRING_INDEX's to chisel off insignificant data until I've sculpted the resultant string into the data gemstone of my desires.
Then I tack on a HAVING clause to find all that match, like so:
SELECT uo.*,
SUBSTRING_INDEX(
SUBSTRING_INDEX(
SUBSTRING_INDEX( uo.data, 'old_order_id' , -1 ),
'";}', 1),
'"',-1)
AS `old order id`
FROM `uc_orders AS `uo`
HAVING `old order id` = 25
The innermost SUBSTRING_INDEX gives me everything past the old_order_id, and the outer two clean up the remainder.
This complicated hackery is not something you want in code that runs more than once, more of a tool to get the data out of a table without having to resort to writing a php script.
Note that this could be simplified to merely
SELECT uo.*,
SUBSTRING_INDEX(
SUBSTRING_INDEX( uo.data, '";}' , 1 ),
'"',-1)
AS `old order id`
FROM `uc_orders` AS `uo`
HAVING `old order id` = 25
but that would only work in this specific case (the value I want is at the end of the data blob)
So you mean to use MySQL to search in a PHP array that has been serialized with the serialize command and stored in a database field? My first reaction would be: OMG. My second reaction would be: why? The sensible thing to do is either:
Retrieve the array into PHP, unserialize it and search in it
Forget about storing the data in MySQL as serialized and store it as a regular table and index it for fast search
I would choose the second option, but I don't know your context.
Of course, if you'd really want to, you could try something with SUBSTRING or another MySQL function and try to manipulate the field, but I don't see why you'd want to. It's cumbersome, and it would be an unnecessary ugly hack. On the other hand, it's a puzzle, and people here tend to like puzzles, so if you really want to then post the contents of your field and we can give it a shot.
You can do it like this:
SELECT * FROM table_name WHERE some_field REGEXP '.*"item_key";s:[0-9]+:"item_value".*'
But anyway you should consider storing that data in a separate table.
How about you serialize the value you're searching for?
$sql = sprintf("select * from tbl WHERE serialized_col like '%%%s%%'", serialize($n));
or
$sql = sprintf("select * from tbl WHERE serialized_col like '%s%s%s'", '%', serialize($n), '%');
Working with php serialized data is obviously quite ugly, but I've got this one liner mix of MySQL functions that help to sort that out:
select REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX(searchColumn, 'fieldNameToExtract', -1), ';', 2), ':', -1), '"', '') AS extractedFieldName
from tableName as t
having extractedFieldName = 'expressionFilter';
Hope this can help!
Well, i had the same issue, and apparently it's a piece of cake, but maybe it needs more tests.
Simply use the IN statement, but put the field itself as array!
Example:
SELECT id, title, page FROM pages WHERE 2 IN (child_of)
~ where '2' is the value i'm looking for inside the field 'child_of' that is a serialized array.
This serialized array was necessary because I cannot duplicate the records just for storing what id they were children of.
Cheers
If I have attribute_dump field in log table and the value in one of its row has
a:69:{s:9:"status_id";s:1:"2";s:2:"id";s:5:"10215"}
If I want to fetch all rows having status_id is equal to 2, then the query would be
SELECT * FROM log WHERE attribute_dump REGEXP '.*"status_id";s:[0-9]+:"2".*'
There is a good REGEX answer above, but it assumes a key and value implementation. If you just have values in your serialized array, this worked for me:
value only
SELECT * FROM table WHERE your_field_here REGEXP '.*;s:[0-9]+:"your_value_here".*'
key and value
SELECT * FROM table WHERE your_field_here REGEXP '.*"array_key_here";s:[0-9]+:"your_value_here".*'
For easy method use :
column_field_name LIKE %VALUE_TO_BE_SEARCHED_FOR%
in MySQL query
You may be looking for an SQL IN statement.
http://www.w3schools.com/sql/sql_in.asp
You'll have to break your array out a bit first, though. You can't just hand an array off to MySQL and expect it will know what to do with it. For that, you may try serializing it out with PHP's explode.
http://php.net/manual/en/function.explode.php
Select * from table where table_field like '%"enter_your_value"%'
select * from postmeta where meta_key = 'your_key' and meta_value REGEXP ('6')
foreach( $result as $value ) {
$hour = unserialize( $value->meta_value );
if( $hour['date'] < $data['from'] ) {
$sum = $sum + $hour['hours'];
}
}
Please don't beat me if this is elementary. I searched and found disjointed stuff relating to pseudo columns. Nothing spot on about what I need.
Anyway... I have a table with some rows. Each record has a unique ID, an ID that relates to another entity and finally a comment that relates to that last entity.
So, I want to COUNT these rows to basically find what entity has the most comments.
Instead of me explaining the query, I'll print it
SELECT entity_id, COUNT(*) AS amount FROM comments GROUP BY entity_id ORDER BY amount DESC
The query does just what I want, but I want to echo the values from that pseudo column, 'amount'
Can it be done, or should I use another method like mysql_num_rows?
Thank you!!!
It's just the same as with the other column – you use the mysql_fetch_* family.
Note that moving to the Mysqli extension is encouraged. See here why.
Once you have the row in, say, $row, you can simply use the value of $row['amount'].
I know i am writing query's wrong and when we get a lot of traffic, our database gets hit HARD and the page slows to a grind...
I think I need to write queries based on CREATE VIEW from the last 30 days from the CURDATE ?? But not sure where to begin or if this will be MORE efficient query for the database?
Anyways, here is a sample query I have written..
$query_Recordset6 = "SELECT `date`, title, category, url, comments
FROM cute_news
WHERE category LIKE '%45%'
ORDER BY `date` DESC";
Any help or suggestions would be great! I have about 11 queries like this, but I am confident if I could get help on one of these, then I can implement them to the rest!!
Putting a wildcard on the left side of a value comparison:
LIKE '%xyz'
...means that an index can not be used, even if one exists. Might want to consider using Full Text Searching (FTS), which means adding full text indexing.
Normalizing the data would be another step to consider - categories should likely be in a separate table.
SELECT `date`, title, category, url, comments
FROM cute_news
WHERE category LIKE '%45%'
ORDER BY `date` DESC
The LIKE '%45%' means a full table scan will need to be performed. Are you perhaps storing a list of categories in the column? If so creating a new table storing category and news_article_id will allow an index to be used to retrieve the matching records much more efficiently.
OK, time for psychic debugging.
In my mind's eye, I see that query performance would be improved considerably through database normalization, specifically by splitting the category multi-valued column into a a separate table that has two columns: the primary key for cute_news and the category ID.
This would also allow you to directly link said table to the categories table without having to parse it first.
Or, as Chris Date said: "Every row-and-column intersection contains exactly one value from the applicable domain (and nothing else)."
Anything with LIKE '%XXX%' is going to be slow. Its a slow operation.
For something like categories, you might want to separate categories out into another table and use a foreign key in the cute_news table. That way you can have category_id, and use that in the query which will be MUCH faster.
Also, I'm not quite sure why you're talking about using CREATE VIEW. Views will not really help you for speed. Not unless its a materialized view, which MySQL doesn't suppose natively.
If your database is getting hit hard, the solution isn't to make a view (the view is still basically the same amount of work for the database to do), the solution is to cache the results.
This is especially applicable since, from what it sounds like, your data only needs to be refreshed once every 30 days.
I'd guess that your category column is a list of category values like "12,34,45,78" ?
This is not good relational database design. One reason it's not good is as you've discovered: it's incredibly slow to search for a substring that might appear in the middle of that list.
Some people have suggested using fulltext search instead of the LIKE predicate with wildcards, but in this case it's simpler to create another table so you can list one category value per row, with a reference back to your cute_news table:
CREATE TABLE cute_news_category (
news_id INT NOT NULL,
category INT NOT NULL,
PRIMARY KEY (news_id, category),
FOREIGN KEY (news_id) REFERENCES cute_news(news_id)
) ENGINE=InnoDB;
Then you can query and it'll go a lot faster:
SELECT n.`date`, n.title, c.category, n.url, n.comments
FROM cute_news n
JOIN cute_news_category c ON (n.news_id = c.news_id)
WHERE c.category = 45
ORDER BY n.`date` DESC
Any answer is a guess, show:
- the relevant SHOW CREATE TABLE outputs
- the EXPLAIN output from your common queries.
And Bill Karwin's comment certainly applies.
After all this & optimizing, sampling the data into a table with only the last 30 days could still be desired, in which case you're better of running a daily cronjob to do just that.
I am stuck with the following query, which won't order by it's date. Any help, or insight into what I am doing wrong will be much appreciated. The query is supposed to get an entry by thread_id, and then show the newest post in the thread, much like with a forum post, which it does fine. But when I try to order the results from newest to oldest using ORDER BY clause, it seems to ignore it.
$query = "SELECT *
FROM messages
WHERE (thread_id, received)
IN (SELECT thread_id, MAX(received)
FROM messages
WHERE receiver='$user' OR sender='$user'
AND is_hidden_receiver!='1'
GROUP BY thread_id)
ORDER BY received DESC";
Cheers, Lea
You were using the PHP time() function to generate a value to be inserted into an INT(11) column. I'm a little mystified as to why this was sorting incorrectly. I will update this answer if I figure out how to explain it concisely.
This feature is built into MySQL, it is the TIMESTAMP column type. You should probably read up on it a bit more before being happy with this solution. It has some interesting properties, depending on how to define your table, a column of type TIMESTAMP can act either as a creation timestamp or a modification timestamp.
Is the problem that it really isn't sorting by "received", or are you just getting different results than you expect? It could be order of operations on the where clause--I'm not sure if AND or OR takes precedence. Maybe try changing this:
receiver='$user' OR sender='$user' AND is_hidden_receiver!='1'
to whichever one of these you are wanting:
(receiver='$user' OR sender='$user') AND is_hidden_receiver!='1'
receiver='$user' OR (sender='$user' AND is_hidden_receiver!='1')