I have a tab delimited text file with the first row being label headings that are also tab delimited, for example:
Name ID Money
Tom 239482 $2093984
Barry 293984 $92938
The only problem is that there are 30 some columns instead of 3 so I'd rather not have to type out all the (name VARCHAR(50),...) if it's avoidable.
How would I go about writing a function that creates the table from scratch in php from the text file, and say the function takes in $file_path and $table_name? Do I have to write all the column names again telling mysql what type they are and chop off the top or is there a more elegant solution when the names are already there?
You would somehow need to map the column type to the columns in your file. You could do this by adding that data to your textfile. For instance
Name|varchar(32) ID|int(8) Money|int(10)
Tom 239482 $2093984
Barry 293984 $92938
or something similar. Then write a function thet get's the column name and columntype using the first line and the data to fill the table with using all the other rows. You might also want to add a way to name the given table etc. However, this would probably be as much work (if not more) than creating SQL queries using you text file. Add a create table statement at the top and insert statements for each line. With search and replace this could be done very fast.
Even if you could find a way to do this, how would you determine the column type? I guess there would be some way to determine the type of the columns through checking for certain attributes (int, string, etc). And then you'd need to handle weird columns like Money, which might be seen as a string because of the dollar sign, but should almost certainly be stored as an integer.
Unless you plan on using this function quite a bit, I wouldn't bother spending time cobbling it together. Just fat finger the table creation. (Ctrl-C, Ctrl-V is your friend)
Related
I have a feed that comes from the State of Florida in a CSV that I need to load daily into MySQL. It is a listing of all homes for sale in my area. One field has a list of codes, separated by commas. Here's one such sample:
C02,C11,U01,U02,D02,D32,D45,D67
These codes all mean something (pool, fenced in area, etc) and I have the meanings in a separate table. My question is, how should I handle loading these? Should I put them in their own field as they are in the CSV? Should I create a separate table that holds them?
If I do leave them as they are in a field (called feature_codes), how could I get the descriptions out of a table that has the descriptions? That table is simply feature_code, feature_code_description. I don't know how to break them apart in my first query to do the join to bring the description in.
Thank you
As a general rule, csv data should never stored in a field, especially if you actually need to consider individual bits of the csv data, instead of just the csv string as a whole.
You SHOULD normalize the design and split each of those sub "fields" into their own table.
That being said, MySQL does have find_in_set() which allows you sort-of search those csv strings and treat each as its own distinct datum. It's not particularly efficient to use this, but it does put a bandaid on the design.
You should keep the information about feature codes in a separate table, where each row is a pair of house identifier, and feature identifier
HouseID FeatureID
1 C07
1 D67
2 D02
You can use explode() to separate your CSV string : http://php.net/manual/en/function.explode.php
$string = 'C02,C11,U01,U02,D02,D32,D45,D67';
$array = explode(',', $string);
Then with your list of feature_codes you can easily retrieve your feature_code_description but you need to do another query to get an array with all your feature_codes and feature_code_description.
Or split your field and put it in another table with the home_id.
You can save it in your DB as is and when you read it out you can run the php function explode. Go check that function out. It will build an array for you out of a string separating the values by whatever you want . In your case you can use:
$array_of_codes = explode(",", $db_return_string);
This will make an array out of each code separating them by the commas between them. Good luck.
I have a mysql table that looks like this:
id author public image1 image2 image3 bio media1 media2 media3 media4 media5 media6
The Field "author" normaly has Firstname (Secondname) Lastname seperated by whitespaces.
How can I sort the array after the Lastname and if just one name is present after this one.
This is the modx query I use to sort after the author but obviously it doesn't use the lastname.
$c = $modx->newQuery('AuthorDe');
$c->sortby('author','ASC');
$authors = $modx->getCollection('AuthorDe',$c);
You're shooting yourself in the foot right now, for a couple of reasons:
When there is only one word in the string, the sorting is hard to predict.
You have indexes for your data for a reason. They make it a lot faster. Using string functions force a table scan. Good enough for 100 data units, slow for 10000 rows and 'database went for a vacation" at 1000000.
Next time you have to use the author field and you realize you have to split it up to words you also have to understand and fix this code snippet on top of the old ones.
That said - I haven't tested it - but try this:
$c->sortby('substring_index(author," ",-1)','ASC');
So to elaborate on the very valuable point jous made, putting multiple points of data in one database column is counter productive.
The sorting you want to do would be simple, fast, & efficient, in a sql query (using the same construct jous showed but without the string operation).
To modify this table you would simply add the following columns to your table in place of author:
firstname
lastname
middlename
To show you how simple this is (and make it even easier) here's the code to do it:
ALTER TABLE [tablename]
ADD COLUMN firstname varchar(32)
ADD COLUMN lastname varchar(32)
ADD COLUMN middlename varchar(32)
DROP COLUMN author;
Then the modx PHP code would be:
$c->sortby('lastname','ASC');
So this is fairly easily done... and if you still need to support other references to author then create a view that returns author in the same way the un-altered table did as shown below (NOTE: you would still have to change the table name reference so it points to the view instead of the table... if this will be a big problem then rename the table and name the view the same as the old table was...):
CREATE VIEW oldtablename AS
SELECT firstname+' '+middlename+' '+lastname' ' AS author
FROM newtablename;
NOTE: if you do create a view like the above then it is probably worth your while to add all of the other columns from the new table (the multiple image & media columns).
NOTE2: I will add, however, that those would ideally be in separate tables with a join table to this one... but if I were in your spot I might agree that expedience might beat utility & future usability.... however if you did put them in different tables you could add those tables to this view (as joins to the new table) and still be able to support existing code that depends on the old table & it's structure.
While the above is all fairly easily done and will work with minor adjustments from you the last part of this is getting your custom table changes to be reflected by xPDO. If you are already comfortable with this and know what to do then great.
If you aren't this is by far the best article on the topic: http://bobsguides.com/custom-db-tables.html
(Yes it is worth getting Bob's code as a snippet so all of this can simply be generated for you once the database changes have been made... (remember you will likely need to delete the existing schema file & xpdo related class files & map files before you run Bob's generation code, or your changes that have the same table name, like the view, won't take effect).
Hope this helps you (or the next person to ask a similar question).
I have a two separate tables that contain parts of user name (don't ask why)...
t1
---------------
firstName
lastName
t2
---------------
middleName
stage_firstName
stage_middleName
stage_lastName
So before I output the name I run it through a function that capitalizes First letter of Name and uses Stage name if provided.
It works OK, but I now have a case where I need to display multiple names. The question I have, is: shall I use mySQL to store properly formatted name when the user formats it initially, or keep the values in multiple tables and keep on using the function to format them. For some reason I think I can improve some performance by utilizing a single value from a table, even if I add additional table column rather than keeping the fields separately in two separate tables and then parsing each name through this huge function.
Am I wrong with these assumptions?
And if, at some point, you need to extract the name and display/use it in a different format, you will need to then perform some kind of translation on the already formatted string.
You could write the formatting into the query though.
I have a table full of data, where one column is full of different entries for each row, where the data is formatted like this:
A:some text|B:some other text|C:some more text|
I want to separate those strings of text into two columns on a new table. So the new table should have one column for A, B, C etc. and the other column will have the rest of the text in their respective rows.
And there is another value (a DATETIME value) in a separate column of the first table that I would like to copy into a third column for each of the separated entries.
Let me know if this needs clarificaiton, I know it's kind of confusing and I'm pretty fuzzy with MySQL. Thanks!
MySQL supports SUBSTRING, together with LOCATE you could probably whip up something nice, based on the pipe symbol you seem to use as a separator.
http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_locate
http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_substring
In most cases I prefer to write "convertors" in a another language than perform it directly on the database, however in this situation it looks like it's not that much data so 'might' work fine..
I think you should better write a simple script in VBScript, PHP or any other scripting language of your choice. All scripting languages provide you with string manipulation and date formatting functions. Database queries won't allow you to handle the "unexpected".
I have a field in a table recipes that has been inserted using mysql_real_escape_string, I want to count the number of line breaks in that field and order the records using this number.
p.s. the field is called Ingredients.
Thanks everyone
This would do it:
SELECT *, LENGTH(Ingredients) - LENGTH(REPLACE(Ingredients, '\n', '')) as Count
FROM Recipes
ORDER BY Count DESC
The way I am getting the amount of linebreaks is a bit of a hack, however, and I don't think there's a better way. I would recommend keeping a column that has the amount of linebreaks if performance is a huge issue. For medium-sized data sets, though, I think the above should be fine.
If you wanted to have a cache column as described above, you would do:
UPDATE
Recipes
SET
IngredientAmount = LENGTH(Ingredients) - LENGTH(REPLACE(Ingredients, '\n', ''))
After that, whenever you are updating/inserting a new row, you could calculate the amounts (probably with PHP) and fill in this column before-hand. Or, if you're into that sort of thing, try out triggers.
I'm assuming a lot here, but from what I'm reading in your post, you could change your database structure a little bit, and both solve this problem and open your dataset up to more interesting uses.
If you separate ingredients into its own table, and use a linking table to index which ingredients occur in which recipes, it'll be much easier to be creative with data manipulation. It becomes easier to count ingredients per recipe, to find similarities in recipes, to search for recipes containing sets of ingredients, etc. also your data would be more normalized and smaller. (storing one global list of all ingredients vs. storing a set for each recipe)
If you're using a single text entry field to enter ingredients for a recipe now, you could do something like break up that input by lines and use each line as an ingredient when saving to the database. You can use something like PHP's built-in levenshtein() or similar_text() functions to deal with misspelled ingredient names and keep the data as normalized as possbile without having to hand-groom your [users'] data entry too much.
This is just a suggestion, take it as you like.
You're going a bit beyond the capabilities and intent of SQL here. You could write a stored procedure to scan the string and return the number and then use this in your query.
However, I think you should revisit the design of whatever is inserting the Ingredients so that you avoid searching strings in of every row whenever you do this query. Add a 'num_linebreaks' column, calculate the number of line breaks and set this column when you're adding the Indgredients.
If you've no control over the app that's doing the insertion, then you could use a stored procedure to update num_linebreaks based on a trigger.
Got it thanks, the php code looks like:
$check = explode("\r\n", $_POST['ingredients']);
$lines = count($check);
So how could I update all the information in the table so Ingred_count based on field Ingredients in one fellow swoop for previous records?