I'm storing a list of items in a serialized array within a field in my database (I'm using PHP/MySQL).
I want to have a query that will select all the records that contain a specific one of these items that is in the array.
Something like this:
select * from table WHERE (an item in my array) = '$n'
Hopefully that makes sense.
Any ideas would be greatly appreciated.
Thanks
As GWW says in the comments, if you need to query things this way, you really ought to be considering storing this data as something other than a big-ole-string (which is what your serialized array is).
If that's not possible (or you're just lazy), you can use the fact that the serialized array is just a big-ole-string, and figure out a LIKE clause to find matching records. The way PHP serializes data is pretty easy to figure out (hint: those numbers indicate lengths of things).
Now, if your serialized array is fairly complex, this will break down fast. But if it's a flat array, you should be able to do it.
Of course, you'll be using LIKE '%...%', so you'll get no help from any indicies, and performance will be very poor.
Which is why folks are suggesting you store that data in some normalized fashion, if you need to query "inside" it.
If you have control of the data model, stuffing serialized data in the database will bite you in the long run just about always. However, oftentimes one does not have control over the data model, for example when working with certain open source content management systems. Drupal sticks a lot of serialized data in dumpster columns in lieu of a proper model. For example, ubercart has a 'data' column for all of its orders. Contributed modules need to attach data to the main order entity, so out of convenience they tack it onto the serialized blob. As a third party to this, I still need a way to get at some of the data stuffed in there to answer some questions.
a:4:{s:7:"cc_data";s:112:"6"CrIPY2IsMS1?blpMkwRj[XwCosb]gl<Dw_L(,Tq[xE)~(!$C"9Wn]bKYlAnS{[Kv[&Cq$xN-Jkr1qq<z](td]ve+{Xi!G0x:.O-"=yy*2KP0#z";s:7:"cc_txns";a:1:{s:10:"references";a:1:{i:0;a:2:{s:4:"card";s:4:"3092";s:7:"created";i:1296325512;}}}s:13:"recurring_fee";b:1;s:12:"old_order_id";s:2:"25";}
see that 'old_order_id'? thats the key I need to find out where this recurring order came from, but since not everybody uses the recurring orders module, there isnt a proper place to store it in the database, so the module developer opted to stuff it in that dumpster table.
My solution is to use a few targeted SUBSTRING_INDEX's to chisel off insignificant data until I've sculpted the resultant string into the data gemstone of my desires.
Then I tack on a HAVING clause to find all that match, like so:
SELECT uo.*,
SUBSTRING_INDEX(
SUBSTRING_INDEX(
SUBSTRING_INDEX( uo.data, 'old_order_id' , -1 ),
'";}', 1),
'"',-1)
AS `old order id`
FROM `uc_orders AS `uo`
HAVING `old order id` = 25
The innermost SUBSTRING_INDEX gives me everything past the old_order_id, and the outer two clean up the remainder.
This complicated hackery is not something you want in code that runs more than once, more of a tool to get the data out of a table without having to resort to writing a php script.
Note that this could be simplified to merely
SELECT uo.*,
SUBSTRING_INDEX(
SUBSTRING_INDEX( uo.data, '";}' , 1 ),
'"',-1)
AS `old order id`
FROM `uc_orders` AS `uo`
HAVING `old order id` = 25
but that would only work in this specific case (the value I want is at the end of the data blob)
So you mean to use MySQL to search in a PHP array that has been serialized with the serialize command and stored in a database field? My first reaction would be: OMG. My second reaction would be: why? The sensible thing to do is either:
Retrieve the array into PHP, unserialize it and search in it
Forget about storing the data in MySQL as serialized and store it as a regular table and index it for fast search
I would choose the second option, but I don't know your context.
Of course, if you'd really want to, you could try something with SUBSTRING or another MySQL function and try to manipulate the field, but I don't see why you'd want to. It's cumbersome, and it would be an unnecessary ugly hack. On the other hand, it's a puzzle, and people here tend to like puzzles, so if you really want to then post the contents of your field and we can give it a shot.
You can do it like this:
SELECT * FROM table_name WHERE some_field REGEXP '.*"item_key";s:[0-9]+:"item_value".*'
But anyway you should consider storing that data in a separate table.
How about you serialize the value you're searching for?
$sql = sprintf("select * from tbl WHERE serialized_col like '%%%s%%'", serialize($n));
or
$sql = sprintf("select * from tbl WHERE serialized_col like '%s%s%s'", '%', serialize($n), '%');
Working with php serialized data is obviously quite ugly, but I've got this one liner mix of MySQL functions that help to sort that out:
select REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX(searchColumn, 'fieldNameToExtract', -1), ';', 2), ':', -1), '"', '') AS extractedFieldName
from tableName as t
having extractedFieldName = 'expressionFilter';
Hope this can help!
Well, i had the same issue, and apparently it's a piece of cake, but maybe it needs more tests.
Simply use the IN statement, but put the field itself as array!
Example:
SELECT id, title, page FROM pages WHERE 2 IN (child_of)
~ where '2' is the value i'm looking for inside the field 'child_of' that is a serialized array.
This serialized array was necessary because I cannot duplicate the records just for storing what id they were children of.
Cheers
If I have attribute_dump field in log table and the value in one of its row has
a:69:{s:9:"status_id";s:1:"2";s:2:"id";s:5:"10215"}
If I want to fetch all rows having status_id is equal to 2, then the query would be
SELECT * FROM log WHERE attribute_dump REGEXP '.*"status_id";s:[0-9]+:"2".*'
There is a good REGEX answer above, but it assumes a key and value implementation. If you just have values in your serialized array, this worked for me:
value only
SELECT * FROM table WHERE your_field_here REGEXP '.*;s:[0-9]+:"your_value_here".*'
key and value
SELECT * FROM table WHERE your_field_here REGEXP '.*"array_key_here";s:[0-9]+:"your_value_here".*'
For easy method use :
column_field_name LIKE %VALUE_TO_BE_SEARCHED_FOR%
in MySQL query
You may be looking for an SQL IN statement.
http://www.w3schools.com/sql/sql_in.asp
You'll have to break your array out a bit first, though. You can't just hand an array off to MySQL and expect it will know what to do with it. For that, you may try serializing it out with PHP's explode.
http://php.net/manual/en/function.explode.php
Select * from table where table_field like '%"enter_your_value"%'
select * from postmeta where meta_key = 'your_key' and meta_value REGEXP ('6')
foreach( $result as $value ) {
$hour = unserialize( $value->meta_value );
if( $hour['date'] < $data['from'] ) {
$sum = $sum + $hour['hours'];
}
}
Related
I just discovered this amazingly useful MySQL function GROUP_CONCAT. It appears so useful and over-simplifying for me that I'm actually afraid of using it. Mainly because it's been quite some time since I started in web-programming and I've never seen it anywhere. A sample of awesome usage would be the following
Table clients holds clients ( you don't say... ) one row per client with unique IDs.
Table currencies has 3 columns client_id, currency and amount.
Now if I wanted to get user 15's name from the clients table and his balances, with the "old" method of array overwriting I would have to do use the following SQL
SELECT id, name, currency, amount
FROM clients LEFT JOIN currencies ON clients.id = client_id
WHERE clients.id = 15
Then in php I would have to loop through the result set and do an array overwrite ( which I'm really not a big fan of, especially in massive result sets ) like
$result = array();
foreach($stmt->fetchAll() as $row){
$result[$row['id']]['name'] = $row['name'];
$result[$row['id']]['currencies'][$row['currency']] = $row['amount'];
}
However with the newly discovered function I can use this
SELECT id, name, GROUP_CONCAT(currency) as currencies GROUP_CONCAT(amount) as amounts
FROM clients LEFT JOIN currencies ON clients.id = client_id
WHERE clients.id = 15
GROUP BY clients.id
Then on application level things are so awesome and pretty
$results = $stmt->fetchAll();
foreach($results as $k => $v){
$results[$k]['currencies'] = array_combine(explode(',', $v['currencies']), explode(',', $v['amounts']));
}
The question I would like to ask is are there any drawbacks to using this function in performance or anything at all, because to me it just looks like pure awesomeness, which makes me think that there must be a reason for people not to be using it quite often.
EDIT:
I want to ask, eventually, what are the other options besides array overwriting to end up with a multidimensional array from a MySQL result set, because if I'm selecting 15 columns it's a really big pain in the neck to write that beast..
Using GROUP_CONCAT() usually invokes the group-by logic and creates temporary tables, which are usually a big negative for performance. Sometimes you can add the right index to avoid the temp table in a group-by query, but not in every case.
As #MarcB points out, the default length limit of a group-concatenated string is pretty short, and many people have been confused by truncated lists. You can increase the limit with group_concat_max_len.
Exploding a string into an array in PHP does not come for free. Just because you can do it in one function call in PHP doesn't mean it's the best for performance. I haven't benchmarked the difference, but I doubt you have either.
GROUP_CONCAT() is a MySQLism. It is not supported widely by other SQL products. In some cases (e.g. SQLite), they have a GROUP_CONCAT() function, but it doesn't work exactly the same as in MySQL, so this can lead to confusing bugs if you have to support multiple RDBMS back-ends. Of course, if you don't need to worry about porting, this is not an issue.
If you want to fetch multiple columns from your currencies table, then you need multiple GROUP_CONCAT() expressions. Are the lists guaranteed to be in the same order? That is, does the third field in one list correspond to the third field in the next list? The answer is no -- not unless you specify the order with an ORDER BY clause inside the GROUP_CONCAT().
I usually favor your first code format, use a conventional result set, and loop over the results, saving to a new array indexed by client id, appending the currencies to an array. This is a straightforward solution, keeps the SQL simple and easier to optimize, and works better if you have multiple columns to fetch.
I'm not trying to say GROUP_CONCAT() is bad! It's really useful in many cases. But trying to make any one-size-fits-all rule to use (or to avoid) any function or language feature is simplistic.
The biggest problem that I see with GROUP_CONCAT is that it is highly specific to MySql: if you want to port your code to run against any other platform, you would have to rewrite all queries that use GROUP_CONCAT. For example, your first query is a lot more portable - you can probably run it against any major RDBMS engine without changing a single character in it.
If you are fine with working only with MySql (say, because you are writing a tool that is meant to be specific to MySql) the queries with GROUP_CONCAT would probably go faster, because the RDBMS would do more work for you, saving on the size of the data transfer.
I have a string like this:
x26y6z8/0|x999y0z1/1|x1y5z40/9999|etc...
Let's say I know this:
x1y5z40
How can I find the whole part of the string I'm looking for? Which is
x1y5z40/9999
It is actually simple, but the way I'm doing it is absolutely not the correct one, as I'm spamming the database with queries and doing it all with php, which obviously results in it being slow.
To make things more difficult, once I found
x1y5z40/9999
I need to replace it with, for example:
x1y5z40/0
I would like to do it entirely with MySQL if possible, maybe with 1 query, somebody got any idea on how could I do?
Use REPLACE().
update TBL
set content = REPLACE(content , 'x1y5z40/9999', 'x1y5z40/0')
Source: http://dev.mysql.com/doc/refman/5.0/en/string-functions.html#function_replace
I think you are leaving out that the components of the string are separated by |. I think you can do this using MySQL string functions. The following should return the second part:
select substring_index(substring_index(col, '|x1y5z40/', 2), '|', 1)
Ah, now to replace the second part with 0. That will be a bit uglier:
select replace(col,
concat('|x1y5z40/',
substring_index(substring_index(col, '|x1y5z40/', 2), '|', 1),
'|'
),
'|x1y5z40/0'
)
You can also express this as an update, if you are trying to change the data in the database.
By the way, storing lists of things in strings is a very bad idea in SQL. Perhaps you don't have choice. But SQL has an excellent data structure for storing lists, with all sorts of built-in functionality. It is called a table. You should have a junction table with one row per entity and value in the string.
How can I sort a query based on the average Rating in a field in my table, the field itself is JSON text, structured like:
[
{"Type":1,"Rating":5},
{"Type":2,"Rating":5},
{"Type":3,"Rating":5}
]
I need my query to be sorted by the average of the 3 Ratings. There will always ever be only 3 values for this.
My current query is:
SELECT `Name`, `Town`, `Vehicle`, `Review`, `Rating`, `Pics`, `PostedOn`
FROM `tbl_ShopReviews`
WHERE `Approved` = 1
ORDER BY `PostedOn` DESC
Current results:
Name Town Vehicle Review Rating Pics PostedOn
Kevin Chicopee 94 Corolla Great stuff, very glad I brought it here [{"Type":1,"Rating":5},{"Type":2,"Rating":5},{"Type":3,"Rating":5}] \N
Just for those like me, who googles and tries to find solution for laravel 5.4. You can use -> operator to extract JSON, that is equal to json_extract function or to column->"$.key" syntax.
$users->orderBy('column->key', 'desc');
Looks like it would be very useful in late 2013 (smile).
For example field_name has the value like
{"json_field_key1":"2016/11/24","json_field_key2":"value"}
Use this code for get the json field json_field_key1 based value in ORDER BY case
select table.*, SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1) as json_field_name from table ORDER BY SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1) DESC
If your value is in date format just modify this code like
order by DATE(SUBSTRING_INDEX(SUBSTRING_INDEX(field_name,'json_field_key1":"',-1),'"',1)) DESC
Would this feature from MySQL 5.7 help? http://blog.ulf-wendel.de/2013/mysql-5-7-sql-functions-for-json-udf/
In my opinion the best solution is to be able to hook up update and insert events in your application for that reviews table and calculate the average in to another field.
Then those queries that need this info will be much easier to handle and will have a better performance.
The better solution is to parse the data before the insert, and have it ready for you in 3 columns or in 1 normalized column.
Saying that, if you're dealing with a non-changeable situation, and have exactly 3 ratings always, you can try this
ORDER BY (substring(json, 21, 1)+
substring(json, 43, 1)+
substring(json,65, 1))/3 desc;
Please consider that this solution is the least maintainable and flexible of them all, and very bug prone. The real solution is restructuring your data.
There isn't an easy way to do this, in fact, I'm not even sure it's possible.
That being said, your database structure really shouldn't contain JSON if it's something you need access to in this respect, instead, why not add a Type field to your database?
the Rating field could be a table with user, type and vlaue as columns where the user is the key. Then you can just use mysql AVG() on the value column where the key match and then sort to that.
hope this help
I have a database and some of the columns contain things like CA, GB etc, although some contain multiple country codes like
US+GB+CA+AU
I'm just wondering what kind of query I would do that would return that row when I'm searching for just CA or just GB, and no necessarily the whole package US+GB+CA+AU
Encase that's a little confusing, basically I just need to return that row based on a search for just CA or just GB etc.
Thanks
Use FIND_IN_SET(), but you'll first need to replace + with , since it expects a comma-separated string. Even without the
REPLACE(), this is will not make use of any index on the countrycodes column.
SELECT * FROM tbl
WHERE FIND_IN_SET('AU', REPLACE(countrycodes, '+', ',')) > 0
The proper long term solution, however, is to change your database structure to normalize these country codes into a table that contains only two columns - a country code, and the id of the associated row from the table you're attempting to query now. You can then index the column appropriately to improve performance (possibly drastically improve it).
I would recommend to normalise it like liquorvicar said.
but using SELECT ... WHERE countrycode LIKE '%GB%' would work.
http://w3schools.com/sql/sql_like.asp
It's not a good solution, but you can use LIKE for your query:
SELECT * FROM `table` WHERE `field` LIKE '%+CA+%' OR `field` LIKE 'CA+%' OR `field` LIKE '%+CA' OR `field` = 'CA'
Last two checks for firs and last values.
I wonder if it is possible to query a specific part of a comma separated string, something like the following:
$results = mysql_query("SELECT * FROM table1 WHERE $pid=table1.recordA[2] ",$con);
$pid is a number
and recordA contains data like
34,9008,606,,416,2
where i want to check the third part (606)
Thank you in advance
Having comma seperated lists or any data seperation within a mySQL field is frowned upon and is to all extents bad practice.
Rather than looking at querying an element of a delimetered list within a mySQL field consider breaking the field into its own table and then creating an adjacency list to create a 1:many relationship between table1 and it's associated variables.
If you are commited to this route, the simplest method would be to use PHP to manage it as mySQL has very few tools (above and beyond regex / text searches) to drill down to the data you want to extract. $results = explode(',',$query); would create an array of your variables from the returned field allowing you to run as many conditional checks against it as needed.
However, consider adding this to your 'need to re-write / re-think' list. A relational tables structure would allow you to query the database for $pid's value directly as it would be contained within it's own field and linked
If the delimetered variable list is of an inderterminate length or the relationships between the variables are heirarchical you'd be better off searching stackoverflow for information on Directed Acyclic Graphs in mySQL to find a better solution to the problem.
Without knowing the nature or the intended purpose for this script I can't answer in any more detail. I hope this has helped a little.
How about this:
SELECT * FROM table1 WHERE FIND_IN_SET({$pid}, recordA) = 3
Make sure to index recordA. I love normalization as much as the next guy, but sometimes breaking it up is just more trouble than it's worth ;)