I have a feed that comes from the State of Florida in a CSV that I need to load daily into MySQL. It is a listing of all homes for sale in my area. One field has a list of codes, separated by commas. Here's one such sample:
C02,C11,U01,U02,D02,D32,D45,D67
These codes all mean something (pool, fenced in area, etc) and I have the meanings in a separate table. My question is, how should I handle loading these? Should I put them in their own field as they are in the CSV? Should I create a separate table that holds them?
If I do leave them as they are in a field (called feature_codes), how could I get the descriptions out of a table that has the descriptions? That table is simply feature_code, feature_code_description. I don't know how to break them apart in my first query to do the join to bring the description in.
Thank you
As a general rule, csv data should never stored in a field, especially if you actually need to consider individual bits of the csv data, instead of just the csv string as a whole.
You SHOULD normalize the design and split each of those sub "fields" into their own table.
That being said, MySQL does have find_in_set() which allows you sort-of search those csv strings and treat each as its own distinct datum. It's not particularly efficient to use this, but it does put a bandaid on the design.
You should keep the information about feature codes in a separate table, where each row is a pair of house identifier, and feature identifier
HouseID FeatureID
1 C07
1 D67
2 D02
You can use explode() to separate your CSV string : http://php.net/manual/en/function.explode.php
$string = 'C02,C11,U01,U02,D02,D32,D45,D67';
$array = explode(',', $string);
Then with your list of feature_codes you can easily retrieve your feature_code_description but you need to do another query to get an array with all your feature_codes and feature_code_description.
Or split your field and put it in another table with the home_id.
You can save it in your DB as is and when you read it out you can run the php function explode. Go check that function out. It will build an array for you out of a string separating the values by whatever you want . In your case you can use:
$array_of_codes = explode(",", $db_return_string);
This will make an array out of each code separating them by the commas between them. Good luck.
Related
I'm new to PHP/MySQL. I want to minimize the number of tables I have so I've been thinking of saving an array of IDs (from checkboxes users tick off) as a string instead of in a separate table. What do you use to format the list of IDs as a string so I can easily parse the IDs for future use in my program?
you can use implode()
and explode for joining the values and separating the values respectively. You can also try serialize() for storing the values. There are a lot of examples in my given links, so they will be helpful for your desirable data format.
Thats not a good idea. I think a 1:N-Relation is the better choice. This givs a better performance and the db supports integrity checks.
So if you want to, you can build a comma-separated list and than use following where-statement:
where CONCAT(',', FIELDWITHIDS, ',') like '%,13,%'
to find datasets that reference the ID 13. Or you have to use explode() and implode() in PHP.
Either comma separated values or php serialize (http://php.net/manual/en/function.serialize.php)
That being said, you lose the benefit of being able to do joins or integrity checks in your DB, so you should generally avoid it.
I wonder if it is possible to query a specific part of a comma separated string, something like the following:
$results = mysql_query("SELECT * FROM table1 WHERE $pid=table1.recordA[2] ",$con);
$pid is a number
and recordA contains data like
34,9008,606,,416,2
where i want to check the third part (606)
Thank you in advance
Having comma seperated lists or any data seperation within a mySQL field is frowned upon and is to all extents bad practice.
Rather than looking at querying an element of a delimetered list within a mySQL field consider breaking the field into its own table and then creating an adjacency list to create a 1:many relationship between table1 and it's associated variables.
If you are commited to this route, the simplest method would be to use PHP to manage it as mySQL has very few tools (above and beyond regex / text searches) to drill down to the data you want to extract. $results = explode(',',$query); would create an array of your variables from the returned field allowing you to run as many conditional checks against it as needed.
However, consider adding this to your 'need to re-write / re-think' list. A relational tables structure would allow you to query the database for $pid's value directly as it would be contained within it's own field and linked
If the delimetered variable list is of an inderterminate length or the relationships between the variables are heirarchical you'd be better off searching stackoverflow for information on Directed Acyclic Graphs in mySQL to find a better solution to the problem.
Without knowing the nature or the intended purpose for this script I can't answer in any more detail. I hope this has helped a little.
How about this:
SELECT * FROM table1 WHERE FIND_IN_SET({$pid}, recordA) = 3
Make sure to index recordA. I love normalization as much as the next guy, but sometimes breaking it up is just more trouble than it's worth ;)
I have a tab delimited text file with the first row being label headings that are also tab delimited, for example:
Name ID Money
Tom 239482 $2093984
Barry 293984 $92938
The only problem is that there are 30 some columns instead of 3 so I'd rather not have to type out all the (name VARCHAR(50),...) if it's avoidable.
How would I go about writing a function that creates the table from scratch in php from the text file, and say the function takes in $file_path and $table_name? Do I have to write all the column names again telling mysql what type they are and chop off the top or is there a more elegant solution when the names are already there?
You would somehow need to map the column type to the columns in your file. You could do this by adding that data to your textfile. For instance
Name|varchar(32) ID|int(8) Money|int(10)
Tom 239482 $2093984
Barry 293984 $92938
or something similar. Then write a function thet get's the column name and columntype using the first line and the data to fill the table with using all the other rows. You might also want to add a way to name the given table etc. However, this would probably be as much work (if not more) than creating SQL queries using you text file. Add a create table statement at the top and insert statements for each line. With search and replace this could be done very fast.
Even if you could find a way to do this, how would you determine the column type? I guess there would be some way to determine the type of the columns through checking for certain attributes (int, string, etc). And then you'd need to handle weird columns like Money, which might be seen as a string because of the dollar sign, but should almost certainly be stored as an integer.
Unless you plan on using this function quite a bit, I wouldn't bother spending time cobbling it together. Just fat finger the table creation. (Ctrl-C, Ctrl-V is your friend)
I have a two separate tables that contain parts of user name (don't ask why)...
t1
---------------
firstName
lastName
t2
---------------
middleName
stage_firstName
stage_middleName
stage_lastName
So before I output the name I run it through a function that capitalizes First letter of Name and uses Stage name if provided.
It works OK, but I now have a case where I need to display multiple names. The question I have, is: shall I use mySQL to store properly formatted name when the user formats it initially, or keep the values in multiple tables and keep on using the function to format them. For some reason I think I can improve some performance by utilizing a single value from a table, even if I add additional table column rather than keeping the fields separately in two separate tables and then parsing each name through this huge function.
Am I wrong with these assumptions?
And if, at some point, you need to extract the name and display/use it in a different format, you will need to then perform some kind of translation on the already formatted string.
You could write the formatting into the query though.
I have a field in a table recipes that has been inserted using mysql_real_escape_string, I want to count the number of line breaks in that field and order the records using this number.
p.s. the field is called Ingredients.
Thanks everyone
This would do it:
SELECT *, LENGTH(Ingredients) - LENGTH(REPLACE(Ingredients, '\n', '')) as Count
FROM Recipes
ORDER BY Count DESC
The way I am getting the amount of linebreaks is a bit of a hack, however, and I don't think there's a better way. I would recommend keeping a column that has the amount of linebreaks if performance is a huge issue. For medium-sized data sets, though, I think the above should be fine.
If you wanted to have a cache column as described above, you would do:
UPDATE
Recipes
SET
IngredientAmount = LENGTH(Ingredients) - LENGTH(REPLACE(Ingredients, '\n', ''))
After that, whenever you are updating/inserting a new row, you could calculate the amounts (probably with PHP) and fill in this column before-hand. Or, if you're into that sort of thing, try out triggers.
I'm assuming a lot here, but from what I'm reading in your post, you could change your database structure a little bit, and both solve this problem and open your dataset up to more interesting uses.
If you separate ingredients into its own table, and use a linking table to index which ingredients occur in which recipes, it'll be much easier to be creative with data manipulation. It becomes easier to count ingredients per recipe, to find similarities in recipes, to search for recipes containing sets of ingredients, etc. also your data would be more normalized and smaller. (storing one global list of all ingredients vs. storing a set for each recipe)
If you're using a single text entry field to enter ingredients for a recipe now, you could do something like break up that input by lines and use each line as an ingredient when saving to the database. You can use something like PHP's built-in levenshtein() or similar_text() functions to deal with misspelled ingredient names and keep the data as normalized as possbile without having to hand-groom your [users'] data entry too much.
This is just a suggestion, take it as you like.
You're going a bit beyond the capabilities and intent of SQL here. You could write a stored procedure to scan the string and return the number and then use this in your query.
However, I think you should revisit the design of whatever is inserting the Ingredients so that you avoid searching strings in of every row whenever you do this query. Add a 'num_linebreaks' column, calculate the number of line breaks and set this column when you're adding the Indgredients.
If you've no control over the app that's doing the insertion, then you could use a stored procedure to update num_linebreaks based on a trigger.
Got it thanks, the php code looks like:
$check = explode("\r\n", $_POST['ingredients']);
$lines = count($check);
So how could I update all the information in the table so Ingred_count based on field Ingredients in one fellow swoop for previous records?