I am attempting to load many CSV files of historical data to a PGSQL server with PHP. I noticed PHP provides a pg_copy_from() function for loading local data into a remote server, but the exact format of the input is vague.
The documentation says it expects an array of rows, each row being a delimited string of the values in that row. PHP's file() function returns an array in a format as described, but feeding it into pg_copy_from() returns an error of:
$rows = file('path/to/data.csv', FILE_IGNORE_NEW_LINES); // Same result without ignore_new_lines
pg_copy_from($db, 'dest_table', $rows, ',');
'PHP Warning: pg_copy_from(): Copy command failed: ERROR: invalid input syntax for integer: ""col_2""'
I checked the array, $rows, and it's of a format like this:
[index]=>""foo",3,7,"bar",500,"z""
Where all string values and the string itself are in double quotes. I assume it's trying to load the entire string into the 1st column of my table, hence the error that col_2 is unable to load (cause there's nothing to load).
Are my assumptions about the array wrong or does the delimited string need a different format (if so, how)?
Thanks for any help/tips!
As noted in comments by #Wolph - just remove first element of your $rows array, because it contains headers. To remove first element of array array_shift can be used without reversing the array:
$rows = file('path/to/data.csv', FILE_IGNORE_NEW_LINES);
array_shift($rows);
pg_copy_from($db, 'dest_table', $rows, ',');
Related
I have database outputs like the following:
$row = '(one,"pika chu",,"")'
If I send this string as parameter to str_getcsv it will output ['one', 'pika chu', '', '']. The third element despite being absent has been turned into an empty string. This is very annoying since I must recognize empty values (no values) from empty strings. The output I would expect is ['one', 'pika chu', null, ''].
The inputs I get are from a PostgreSQL database and are represented as composite values.
By example, if a table is pokemon_id => int4, name => text then a query will output strings like '(1, "pika chu")'. A unique constraint on the name field by example will allow the following two records to exist: (100, '') and (101, null).
When fetched, they are formatted as raw values like:
'98,whatever'
'99,"pika chu"'
'100,""'
'101,'
'102,","'
I need to read those strings and this example must output the following arrays:
['98', 'whatever']
['99', 'pika chu']
['100', '']
['101', null]
['102', ',']
Is there a way to do that in PHP?
Update 1: #deceze kindly sent me this link stating there are no NULLs in CSV (TL;DR because there were no nulls in XML basically, this problem has been tackled since then.) How to parse CSV with NULLs then?
Update 2: I had propositions to create a dedicated parser in PHP using preg_match_* functions. I am a bit reluctant to go that way because 1) of the performance impact compared to str_getcsv and 2) the fact preg_match used to segfault if the string passed was over 8kb (which can happen in a CSV context).
Update 3: I looked at str_getcsvsource code to see if it was possible to propose a patch to add parsing options like it is in some other languages. I now understand PHP’s underlying philosophy better. #daniel-vérité raised the idea to implement a state machine to parse CSV strings. Even though input can have thousands of lines that weight dozens of kilobytes with embedded CSV structures, it might be the best way.
Thank you for your help.
I have a long string of values separated by tabs and I wish to cut the data as if using unix cut -f. If I use cut -f5 it cuts all my data into a single column of the value which is in the 5th position. Is there a PHP function that can do the same?
Below is an example of the raw file with each word in the row separated by a tab
The result would be as follows if I ran cut -f2:
I guess the answer really is "no", as far as I know. But you can combine a few PHP functions to achieve the same result.
You can use file to read the lines from the file into an array
$rows = file($path_to_your_file);
Then convert that array of strings to a multidimensional array
$rows = array_map(function($row){
return str_getcsv($row, "\t");
}, $rows);
Then get the column you want from that array.
$column_5 = array_column($rows, 4);
Not as concise as cut -f5, but PHP rarely is for things like this.
Incidentally, if you don't care that your PHP program will only work on systems that have unix cut, you can actually just use the cut -f in shell_exec.
I simply need to store text (rows have their own separate date).
Either...
1. How do I read (either of) the following value(s) in to a PHP array that I can iterate over via a foreach loop?
...or...
2. how do I INSERT INTO a table a value assigned to a key (which is not clarified at all in the PostgreSQL documentation)?
{".022 x 2.031"}
{".012 x 3.621"}
So I currently have single measurement values (to be treated as text purely for records) formatted as the first format above. I try to convert the row data in PHP and get Invalid argument supplied for foreach() for the following:
$array = json_decode($row1['material_size']);
foreach ($array as $k1) {echo $k1;}
So after doing even more searching and reading I decided to see if I could just convert the data via PostgreSQL directly:
SELECT array_to_json(material_size) AS material_size FROM table;
Now I get the array data correctly...
$array = json_decode($row1['material_size']);
print_r($array);
Array
(
[0] => .022 x 2.031
)
Afaik, dimension is a structured data. If your data is 2 dimensional, you may use the point datatype or create your own composite type. I'd suggest to use Pomm to get it converted to a PHP array.
If you want to stick with Json then you can select on keys like stated in the documentation but you need to overwrite the complete JSON value every time you want to update a value.
I am putting the contents of an text file into an array via the file() command. When I try and search the array for a specific value it does not seem to return any value but when I look at the contents of the array the value I am searching for is there.
Code used for putting text into array:
$usernameFileHandle = fopen("passStuff/usernames.txt", "r+");
$usernameFileContent = file("passStuff/usernames.txt");
fclose($usernameFileHandle);
Code for searching the array
$inFileUsernameKey = array_search($username, $usernameFileContent);
Usernames.txt contains
Noah
Bob
Admin
And so does the $usernameFileContent Array. Why is array_search not working and is there a better way to do this. Please excuse my PHP noob-ness, thanks in advance.
Because file():
Returns the file in an array. Each element of the array corresponds to a line in the file, with the newline still attached
To prove this try the following:
var_dump(array_search('Bob
', $usernameFileContent));
You could use array_map() and trim() to correct the behavior of file(). Or, alternatively, use file_get_contents() and explode().
To quote the docs:
Each element of the array corresponds to a line in the file, with the newline still attached.
That means that when you're doing the search, you're searching for "Noah" in an array that contains "Noah\n" - which doesn't match.
To fix this, you should run trim() on each element of your array before you do the search.
You can do that using array_map() like this:
$usernameFileContent = array_map($usernameFileContent, 'trim');
Note, too, that the file() function operates directly on the provided filename, and does not need a file handle. That means you to do not need to use fopen() or fclose() - You can remove those two lines entirely.
So your final code could look like this:
$usernameFileContent = array_map(file('passStuff/usernames.txt'), 'trim');
$inFileUsernameKey = array_search($username, $usernameFileContent);
I have an array of CLOB datatypes. In my sample data, there are two items in the array. The array is embedded in a FOR loop that will loop until all of the data is read from the array (in this case, twice). I have a function that reads the CLOB data into a string, then returns the string, literally just:
function getCLOB($clob)
{
$str = $clob->read($clob->size());
return $str;
}
My problem is that it reads in the first thing I send it fine and I can get that to display back in the FOR loop. However, when I send the second one, the string is empty. I put in some echos to see what was happening with the data and the $clob variable has data once it's in the function, but the $str variable is empty after the first line of the function. Does anyone have any ideas on why it works once but not the second time?