I am downloading new csv's each night using a cron job with PHP. Each csv is normally about the same, possibly one night within a month a field is new. I need to get the new field and append it to the database. I don't know how to get the type of the new field. I saw someone else's question with gettype() but i'm not sure if that would work or not since the data is inside a csv so wouldn't they all be strings when some need to be floats, or ints? How would I go across checking the type?
The second question, is there a way to check if there is not a name inside of a table? For instance, if they add a new field called foo52, and I have foo1 through foo51 in my database, is there a quick way to search for fields that aren't there, or would I have to use a select statement for each one and append it when it's false?
I use MySQL for my database.
Thanks for your help.
The first question on the part about getting the type is the simply try the conversion of the data itself and then seeing if the data is equivalent with a == comparison.
So,
some,data,is,123
After reading in the data you can then try the conversion to various types such as strings, ints, ect...then from that you are able to determine the type of the data.
For the second question you can get the column names by doing:
show columns from db.table_name
Then you can do a simple in_array to test if the new column name is already in the database.
EDIT:
Using array_diff can simplify the finding of the missing/new column names from the CSV.
csv_names = get_csv_column_names();
sql_names = get_sql_column_names();
new_names = array_diff( csv_names, sql_names );
I have found the parsecsv-for-php library very handy for [re|de]constructing CSV data.
For the first question: you can test if it's numeric with is_numeric(). If not, store as string. If yes, create the field as numeric in your database. If you want, you can use regex to check if it's a date or some other datatype you think is required to be stored correctly (i.e. not a a default-text)
for the second question: getting the fieldnames of a table in Postgres is done with the follwoing query
$sql = "SELECT attname FROM pg_catalog.pg_attribute
WHERE attrelid =
(SELECT c.oid FROM pg_catalog.pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname = '$this->tableName' AND n.nspname = 'public')
AND attnum > 0
AND NOT attisdropped";
For MySQL, it should be doable with "show columns from db.table_name".
Once you have the fields, use in_array() to check if it exists already...
Note that: you'd probably need to check for all columns in your CSV if they exist already. If not: add a new colum for it. If yes, leave it as is...
Related
This probably has a simple solution, although it is made a little more difficult because of the way the database is constructed, it isn't mine. A column in the database has a text value that is of the form text1DDtext2DDtext3, where DD is a delimiter that they through in rather than having a separate table for 0 to n values that go in that column.
There is a search that is executed where what I have to start with is:
"text1","text2", "text3", . . .
All I want to do is build on a query that checks to see if any of the "textn" strings are in the column field, although it would be nice to have a query that also checked to see if all of the search string text values are in the column value. The order in which they are stored in the column can vary, as can the search string. If there was a linked table that just had single values in a column it would not be very hard.
I've just various combinations of IN and LIKE, and that doesn't seem to work.
Thanks.
try:
SELECT columnYouWant FROM dbo.table WHERE UPPER(column) LIKE ('%TEXT%');
I am trying to something in MySQL that I do in JavaScript/PHP all the time. I need to concatenate the value of a field in MySQL with a value that is passed into a query from PHP. For example, let's say I have a field called favourites with a value of 27 and I have this query:
UPDATE useraccs SET favourites = favourites + ',30' WHERE id='10'
My desired new value for favourites would be 27,30, but I'm getting 57, where clearly SQL is adding them numerically. I have set the data type for this column as TEXT and was hoping that would force SQL to treat it as a string all the time, but that doesn't seem to be the case.
In my research I read about the CONCAT() function, and I tried this:
UPDATE useraccs SET favourites = CONCAT(favourites,',30') WHERE id='10'
That results in a failed query. The logic feels right but that is obviously not how that function is meant to be used.
I acknowledge that in theory, I could just grab the original value of favourites and concatenate it with the new value in the PHP itself and then send it to MySQL, but I feel like there MUST be a way to do this in one query...if I'm wrong about that so be it, but I'm sure there must be a way.
Use the following to create '27,30':
CONVERT(favourites,char) + ',30'
So I've got a field in my database that will contain serial id numbers separated by commas eg. 2817,2385,4937,3298 I want to be able to add more numbers to the same field over time.
The best way I can think to do this is to get the contents, add the new numbers to it, and insert them back into the database.
What I'm wondering is if there's a more direct way. I had trouble thinking of a good way to word this that yields helpful search results so I'm asking here.
Yes there is.
UPDATE `table` SET `column` = CONCAT(`column`,',new_serial')
However this is not right, you should never store comma separated values. It's called database normalization.
Try this:
UPDATE `tableName` SET `yourColum` = CONCAT(`yourColumn`, ',nextId')
This will update your column as you requested.
Given a result set, how can I determin the actual names of the fields specified in the query (NOT their aliases).
$query = "SELECT first AS First_Name, last AS Last_Name FROM people";
$dbResult = mysql_query($query);
$fieldCount = mysql_num_fields($dbResult);
for ($i=0; $i<$fieldCount; $i++) {
// Set some values
$fieldName = mysql_field_name($dbResult, $i);
}
This example returns field names, but in this example it returns the alias "First_Name" instead of the actual field name "first".
Is it possible to get the actual field name from such a query. Particularly if I am writing a function and have no idea what query will be thrown at it.
If you are using MySQLi:
http://www.php.net/manual/en/mysqli-result.fetch-field.php
The field object has a "orgname" property.
The "classic" MySQL equivalent function doesn't report back the original column names.
Short answer: you don't.
Long answer: Once the dataset is pulled by MySQL and sent back to PHP, the only information PHP now has is the columns, or aliases if you used them. There is no way to look at a result set and determine what the original column names were. You have to switch to another DB driver like mysqli to obtain this info.
Your question doesn't make sense.
What are you going to do if you get a derived column i.e.
select column_a + column_b as order_total from orders;
are you saying you want to know that the original query was column_a + column b ??
if so, you probably need to write a query parser, or get one off the internet.
I think the implementation of that is beyond the scope of your question though :)
I'm not 100% sure about this, but I would say: there is no way.
The MySQL gives you back the result set, nothing more. It does not return the select statement nor any details about it.
So you cannot get the original field names because the server will provide you the information you asked: alias names.
If you don't mind making a second query (and your using MySQL 5 or greater) you can ask information_schema for the names.
Check out MySQL Reference for the details:
SHOW COLUMNS FROM tbl_name;
if you have access to the string of the query you could try a regular expression to parse it.
I'm no regex master but you could chop up the string by looking at the text between 'select' and 'from' then grabbing all the field names as either
field FieldAlias
or
field as FieldAlias
If you're trying to write some functionality to let you know what fields are being fetched for handling updates - the only way to do this correctly is for it to present an SQL-less interface to the code above and manage all SQL generation itself. This is called a data abstraction layer.
Say I have an array of strings in a php array called $foo with a few hundred entries, and I have a MySQL table 'people' that has a field named 'name' with a few thousand entries. What is an efficient way to find out which strings in $foo aren't a 'name' in an entry in 'people' without submitting a query for every string in $foo?
So I want to find out what strings in $foo have not already been entered in 'people.'
Note that it is clear that all of the data will have to be on one box at one point. The goal would be doing this at the same time minimizing the number of queries and the amount of php processing.
I'd put your $foo data in another table and do a LEFT OUTER JOIN with your names table. Otherwise, there aren't a lot of great ways to do this that don't involve iteration at some point.
The best I can come up with without using a temporary table is:
$list = join(",", $foo);
// fetch all rows of the result of
// "SELECT name FROM people WHERE name IN($list)"
// into an array $result
$missing_names = array_diff($foo, $result);
Note that if $foo contains user input it would have to be escaped first.
What about the following:
Get the list of names that are already in the db, using something like:
SELECT name FROM people WHERE name IN (imploded list of names)
Insert each item from the return of array_diff()
If you want to do it completely in SQL:
Create a temp table with every name in the PHP array.
Perform a query to populate a second temp table that will only include the new names.
Do an INSERT ... SELECT from the second temp table into the people table.
Neither will be terribly fast, although the second option might be slightly faster.
CREATE TEMPORARY TABLE PhpArray (name varchar(50));
-- you can probably do this more efficiently
INSERT INTO PhpArray VALUES ($foo[0]), ($foo[1]), ...;
SELECT People.*
FROM People
LEFT OUTER JOIN PhpArray USING (name)
WHERE PhpArray.name IS NULL;
For a few hundred entries, just use array_diff() or array_diff_assoc()
$query = 'SELECT name FROM table WHERE name != '.implode(' OR name != '. $foo);
Yeash, that doesn't look like it would scale well at all.
I'm not sure there is a more efficient way to do this other than to submit all the strings to the database.
Basically there are two options: get a list of all the strings in MySQL and pull them into PHP and do the comparisons, or send the list of all the strings to the MySQL server and let it do the comparisons. MySQL is going to do the comparisons much faster than PHP, unless the list in the database is a great deal smaller than the list in PHP.
You can either create a temporary table, but either way your pushing all the data to the database.