the NOOB Developer is back with yet another question. I'm working on importing a large dataset into a PostgreSQL 9.5 database. I originally started out using PHP, but once I tried to load the entire 14mb file, it failed. I went on to increase the memory limit within the script, but that didn't help. I thought about using a parsing library, but decided that since I'm using PostgreSQL 9.5, I should just leverage the database instead. My JSON file has repeatable fields, so I could not use JSONB and went with using the plain JSON import. Unfortunately, that worked until I tried to load the entire file. I then got the following error:
ERROR: invalid input syntax for type json
DETAIL: Expected JSON value, but found "]".
CONTEXT: JSON data, line 1: ...:"Commercial & Industrial","u_cost_code":""},"",]
Here is an example of the JSON file content:
Array
(
[result] => Array
(
[0] => Array
(
[field1] => 0
[fiedl2] =>
[field3] => 1900-04-19 19:14:10
[field4] => false
[field5] => XXX.XXXXX.XXX.XXX.XXX
[field6] => ldap:CN=XXXX XXXXXXX,OU=XXXXX,OU=XXXXX,OU=XXX,DC=XXXXXX,DC=XXXX,DC=XXXX
[field7] => 1900-07-18 17:45:08
[field8] => true
[field9] =>
[field10] => false
[field11] => 2
[field12] => 30406
[field13] => T
[field14] => 00000000000000000
[field15] => 1900-01-19 21:33:07
[field16] => Array
(
[link] => https://mozilla.com
[value] => mozilla
)
[field17] => 1601-01-01 06:00:00
[field18] =>
[field19] => false
[field20] => 01001
[field21] =>
)
)
)
Here is the statement I'm using to create my table, this allowed me to import the entire file 14mb without an issue:
CREATE TABLE temp_json
(
ID SERIAL NOT NULL PRIMARY KEY
,TIMESTAMP TIMESTAMP DEFAULT CURRENT_TIMESTAMP
,VALUES TEXT
);
I started following the example of this developer hoping to resolve this issue: how-to-get-json-data-type-into-postgresql
Here is the fairly standard copy command I'm using to import the data into the table:
copy temp_json(values) from 'C:\path\to\my\json_file.json';
I then went on to use the following sql statement in an attempt to move the data into a relational table that I found here, loading-json-data-from-a-file-into-postgres, on stack. I did this in the effort of finding an easier way to move my data set into the table. Here is the sql statement I am trying to get working:
insert into table_to_hold_json
select
values::json->'result'->'calendar_integration' as calendar_integration,
values::json->'result'->'country' as country,
values::json->'result'->'last_login_time' as last_login_time,
values::json->'result'->'u_gartner_acct' as u_gartner_acct,
values::json->'result'->'u_dept_name' as u_dept_name,
values::json->'result'->'source' as source,
values::json->'result'->'sys_updated_on' as sys_updated_on,
values::json->'result'->'u_field_user' as u_field_user
from ( select json_array_elements(replace(values,'\','\\')::json) as values
from temp_json ) a;
However, I'm now getting the same error as I did on the import to the temp_json table. I also tried to escape the '\' with the copy command using:
csv quote e'\x01' delimiter e'\x02'
Unfortunately, I still end up with the same error when I try to query the JSON data. So, now I'm banging my head against the wall in trying to sort out how to escape that darn ']'. Any assistance that is given will be greatly appreciated!
Okay, so I went back and worked out how to break up my file download from the data provider. Now that I'm keeping the data set under the specified timeout period I can use PHP or whatever else I want to parse the data. This is a good reminder to always check your logs or data sets, closely. :-)
Related
I am encountering a weird error.
In a laravel project I am reading a CSV file. In my Controller I am parsing the CSV and creating database records for a mysql database.
I convert the CSV lines into individual arrays which are located within a big array -> two dimensional array. So far so good.
When iterating through my array the first iteration works just fine and I get the database records as expected. However, during the second iteration a Query Exception is thrown.
The stack trace leads me to a line where I am assigning a double value to an attribute of the model.
Transaction::firstOrCreate([
'date' => $date,
'transaction_type' => $the_big_array[$i][2],
'db_counter_party_id' => $db_counter_party_id,
'amount' => $amount,
'currency' => $the_big_array[$i][6],
'usage' => $the_big_array[$i][7],
'category' => "none",
'balance' => $balance_after_transaction,
]);
The last assignment is causing the error. The variable balance_after_transaction is of type double and calculated according to previous transactions. So it is not directly retrieved from the CSV.
Does someone have a clue what is going on?
The error was caused by a wrong character encoding.
I've got the error "SQLSTATE[HY093]: Invalid parameter number: parameter was not defined" while trying to save an array into a related InnoDB, which makes no sense to me because I've already used the same method multiple times in the same Application and the DB connection seems to work as well because I can read and delete entries without any problems.
I've checked a lot of similar questions in hope they would resolve mine but haven't found one that resolved my problem.
This is the method I use to save to create the array and save it into the DB:
public function saveBest(Besten $besten)
{
$GePunkte = (int) $besten->Punkte/$besten->Dauer;
$data = array(
'GePunkte' => $GePunkte,
'Name' => $besten->Name,
'Zeitpunkt' => $besten->Zeitpunkt,
'Punkte' => $besten->Punkte,
'Dauer' => $besten->Dauer,
'GewäKat' => $besten->GewäKat,
);
print_r($data);
$this->tableGateway->insert($data);
}
I've already used "print_r($data)" to check if the data in the array is complete because I suspected this to be the cause of the error.
print_r displayed the following:
Array ( [GePunkte] => 0.125 [Name] => Test [Zeitpunkt] => 1451480345 [Punkte] => 1 [Dauer] => 8 [GewäKat] => Natur )
At the same time I checked if the values of each param matches the datatypes set in the DB. GePunkte is set as float, Name as text, Zeitpunkt as int, Punkte as int, Dauer as int and GewäKat as text, so all values are inside the set boundaries.
I've also checked if I've got all the params named right by checking the insert sql command in Phpmyadmin and it's:
INSERT INTO `bestenliste`(`Sid`, `GePunkte`, `Name`, `Zeitpunkt`, `Punkte`, `Dauer`, `GewäKat`) VALUES ([value-1],[value-2],[value-3],[value-4],[value-5],[value-6],[value-7])
Since Sid is created automatically this seems to be alright too.
I can't figure out why it isn't working as it should, since everything else in the application seams to work perfectly fine, and because the deadline for this application is tomorrow at midnight I'm kinda in a hurry to resolve this problem.
So does anyone know a solution to this problem ?
I'm accessing a MySQL database from a PHP script using the PDO library, which I haven't used before (I am more familiar with MySQLi). I'm seeing extra, unexpected fields in the result. They aren't interfering with my code, but I'd like to know where they've come from.
Here is my database call:
SELECT
author.author_id AS author_id,
author.name_last AS name_last,
author.name_first AS name_first,
author.detail AS detail
FROM join_play_author
LEFT JOIN author
ON join_play_author.author_id = author.author_id
WHERE join_play_author.play_id = :play_id
ORDER BY author.name_last
I then bind and execute successfully, and my results contain all of the fields I've requested, with the appropriate labels. However, each field occurs twice in the result set: once with the label I requested, and once with an extra auto-incremented value. For example, one result set (printed using print_r()) looks like this:
Array
(
[author_id] => 41
[0] => 41
[name_last] => Dekker
[1] => Dekker
[name_first] => Thomas
[2] => Thomas
[detail] => 0
[3] => 0
)
These double fields aren't actively interfering with my code, but they are annoying during debug and I worry that they could affect performance for large result sets, which will be a concern on the production site.
Removing the AS tags doesn't affect the result array.
Any ideas? Is this just a feature of PDO--and if so, is there any way to turn it off? I don't ever remember seeing these extra fields in MySQLi results, although it's possible I missed them.
The PHP library is probably setting PDO::FETCH_BOTH as the result. If you don't want the extra keys set in your array you would need to change it to PDO::FETCH_ASSOC. Most libraries allow you to change this without modifying code though. You will have to read the documentation on that.
Documentation on the PDO statement fetch options.
http://www.php.net/manual/en/pdostatement.fetch.php
As stated by chapka in his comment - use setAttribute to clear the results from double results:
$db->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
I can't express Question well in English
You know my English is poor.
That's not the point
When I use getQueryLog() to get the last executed query.
But I get the response
[query] => select * from table where`teacher_oid` = ? and `class_oid` = ?
[bindings] => Array
(
[0] => 'tid'
[1] => 'cid'
)
[time] => 12.58
Is there a way automatically written bindings[0] to first "?" bindings[1] to second "?" ?
Thank you!
Nope. Unfortunately there is not. This is a limitation of PHP's PDO.
However you can still find the final query.
I accomplish this by enabling MySQL's query logging in /etc/mysql/my.cnf (The location of your mysql configuration file may be different).
You want to enable the general_log inside of my.cnf and then restart MySQL. Then in /var/log/mysql.log, you'll see each query made to the database.
Only do this in a development server, not in production.
I have a PDO issue with PHP 5.3. I am running MaxDB 7.8 for my database - if you haven't heard of that, you're not alone. It's an open-source enterprise database that is connected with MySQL on paper, but is nothing like it.
I'm not sure that this issue is caused by MaxDB, but I wanted to mention it.
I found that when using prepared statements via the ODBC driver, my query fails if ANY of the $data array values are empty strings. For example, a user is presented with a dialog box where 1 value is required ('title'), but 2 are optional ('author', 'version'). If the user chooses not to enter one of the optional values, that array element would be == "". If this is the case, the $sth->execute($data) fails, complaining of null errors.
I have tried the setAttribute command re: Nulls on all 3 settings, I have tried checking if $value == null { $value = ""; }... and numerous other things I've found in articles, all to no avail. If I manually replace any empty values with a string like "(none)", the query works.
Anyways, here's the particulars:
$data:
Array
(
[cust] => 1
[ftype] => 1
[title] => test
[author] =>
[version] =>
[folder] => 0
[modified] => 1337394898
[content] => "this is test content"
[status] => 1
[pages] => 1
)
$sql = "INSERT INTO FORMS (CUST, FTYPE, TITLE, AUTHOR, VERSION, FOLDER, MODIFIED, STATUS, CONTENT, PAGES) VALUES (:cust, :ftype, :title, :author, :version, :folder, :modified, :status, :content, :pages)";
$sth = $dbh->prepare($sql) or $this->error($sql, $dbh->errorInfo());
$sth->execute($data);
Thanks in advance for your assistance!
After days of lost time, I'm giving up on this. If anyone finds a solution, I'd love to hear it but I need to move on.
My duct tape/chicken wire solution to this is to just remove the possibly-empty values from the list of bound parameters, and include them as part of the SQL statement. That way if something happens to equal '', it doesn't freak out & explode.
Thanks to all who tried.