I have a .sql file with 24k INSERT statements as:
INSERT INTO dump(id, title, content, datetime, section) VALUES ('', 'Title up to 200 characters', 'Detailed content with HTML tags', '04-10-2010, 11:48 AM', 1);
Once import is completed, i get a message that says
Import has been successfully finished, 24664 queries executed.
When i browse the table, i find many titles and contents are totally empty or carry part of texts only i.e. few letters.
The only parts that gets inserted properly is: id, datetime and section.
title and content carry heavy contents. Many "content" might carry 250 char+. Title carries around 100 chars max.
On the other hand, i find copy/paste few pieces of INSERT and do it manually via SQL area. Everything gets inserted correctly. This method could take me very long time...
Any idea why i'm facing such a problem?
Use MYSQL to create your dumps try this command
mysqldump --extended-insert=FALSE --complete-insert=TRUE -p db_name
Check the max_allowed_packet size and be sure that is not hurting your inserts.
Related
Our client has sent us a CSV file of data that I need to import into a specific table in our Postgresql 8.3.9 database. The database uses UTF-8 character encoding, i.e. our CMS allows multiple languages such as French which are inputted into the database via the CMS in French. One particular facility is for the client to upload images to the server and then enter "alt" tags for them in French. However, due to a bulk update required, we have been sent a CSV to feed into a particular table - for the image alt tags, in French.
The CSV has some special characters such as "é" - e.g.
"Bottes Adaptées Amora Cuir Faux-Croco Fauve Photo d'Ensemble"
The images themselves are hosted on two places - one is a CDN, and one is a local database backup and local server (web server) file backup. I am using a PHP script to read the CSV file and do the needful so that the "alt" tags are updated on two places - our web database, and the CDN.
However, when I read the CSV (using PHP), the character does not "come out" as expected.
The data is comming as "Bottes Adapt�es Amora Cuir Faux-Croco Fauve Photo d'Ensemble".
I don't think this has anything to do with the database, but it has something to do with my PHP file reading the CSV data. Even if I print the data that it is reading, the special character above does not print as above, it' prints as if the special character is not recognised. Other characters print fine.
Here is the code I'm using (not some special custom functions are used here to interact with the database but they can be ignored). The CSV file is made up of {column 1} for image name, and {column 2} for the ALT tag.
$handle = fopen($conn->getIncludePath() . "cronjobs/GIB_img_alt_tags_fr.csv", "r");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
//normally I run a query here to check if the data exists - "SELECT imageid, image_fileref FROM table1 WHERE image_fileref = '". $data[0]. "'");
if ($conn->Numrows($result)) { //if rows were found -
$row=$conn->fetchArray($result);
//printing the data from $row here
}
}
fclose($handle);
You've still omitted key information - when asking for help with an UPDATE don't delete the UPDATE statement from the code - and your description of the problem is very confused, but there's some hint of what's going on.
Mismatched encodings
It's highly likely that your PHP connection has a client_encoding set to something other than UTF-8. If you're sending UTF-8 data down the connection without conversion, the connection's client_encoding must be UTF-8.
To confirm, run SHOW client_encoding as a SQL statement from PHP and print the result. Add SET client_encoding = 'UTF-8' to your code before importing the CSV and see if that helps. Assuming, of course, that the CSV file is really UTF-8 encoded. If it isn't, you need to either transcode it to UTF-8 or find out what encoding it is in and SET client_encoding to that.
Read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) and the PostgreSQL manual on character set support.
Better approach
The approach you're taking is unnecessarily slow and inefficient, anyway. You should be:
Opening a transaction
Creating a temporary table in the database with the same structure as the CSV file.
Use pg_copy_from to load the CSV into the temp table, with appropriate options to specify the CSV format.
Merge the contents of the temporary table into the destination table with an INSERT then an UPDATE, eg:
INSERT INTO table1 (image_fileref, ... other fields ...)
SELECT n.image_fileref, ... other fields ...
FROM the_temp_table n
WHERE NOT EXISTS (SELECT 1 from table1 o WHERE o.image_fileref = n.image_fileref);
UPDATE table1 o
SET .... data to update ....
FROM the_temp_table n
WHERE o.image_fileref = n.image_fileref;
Commit the transaction
The INSERT may be more efficiently written as a left outer join with an IS NULL filter to exclude matching rows. It depends on the data. Try it.
I probably could've written a faster CTE-based version, but you didn't say what version of Pg you were using, so I didn't know if your server supported CTEs.
Since you left out the UPDATE I can't be more specific about the UPDATE or INSERT statements. If you'd provided the schema for table1 or even just your INSERT or UPDATE I could've said more. Without sample data I haven't been able to run the statements to check them, and I didn't feel like making up some dummy data, so the above is untested. As it is, completing the code is left as a learning exercise. I will not be updating this answer with fully-written-out statements, you get to work that out.
I have a codeigniter controller that receives json content from an API and inserts it to a mysql longtext field unchanged.
Everything's been running smoothly for a while, but recently i've been noticing some of the content is being truncated. Here are two examples.
The table being inserted has three fields
id int
data longtext
date_added datetime
I am receiving the data and directly adding it to the database using the active record insert function. It looks like that
$this->db->insert('received', array('data' => $data, 'date_added' => date('Y-m-d H:i:s')));
Using the profiler, i monitored the queries and found two faulty ones:
INSERT INTO `received` (`data`, `date_added`) VALUES ('{\"status\":{\"lastMaintenanceAt\":0000,\"code\":304,\"period\":900},\"items\":[{\"permalinkUrl\":\"http://example.com\",\"updated\":0000,\"postedTime\":0000,\"summary\":\"Man\'s guarantees are like his words.\",\"title\":\"By: John\",\"content\":\"<p>Man’s guarantees are like his words.</p>\",\"id\":\"http://example.com\",\"actor\":{\"permalinkUrl\":\"\",\"displayName\":\"John\"}}],\"title\":\"Comments on: Banker refuses to give ‘Written Guarantee’\"}', '2012-04-08 00:28:29')
and
INSERT INTO `received` (`data`, `date_added`) VALUES ('{\"status\":{\"code\":304,\"period\":900,\"feed\":\"http://example.com\"},\"items\":[],\"title\":\"Comments on: Making her cry…\"}', '2012-04-08 00:49:35')
The queries seems alright. But only part of the JSON is making it to the table in the first case, it is truncated after "[...] refuses to give" and in the second after making her cry.
The queries are not returning an error and the date is being inserted properly. Moreover, if i copy the queries and execute them in a mysql command prompt, they will insert the full text.
These queries a one a few of hundreds of other successful ones.
Any idea what might be the problem?
thank you
All of the faulty queries share one thing in common: they've got "special characters" which MS Word inserts, e.g. ‘, ’ and …. If you can convert / remove these, then your problems should go away.
Update
For a CodeIgniter solution to this, you could load the Text Helper library and then use the ascii_to_entities function on any strings, e.g.
$this->load->helper('text');
$desc = ascii_to_entities($desc);
PART 01: INTRO
Hey Stackoverflow
Greetings from the Snowy alps of Norway
I have now tried for two full consecutive days to reverse engineer Magic Fields. Why? Because I am a simple man, and want to import data to my wordpress from an exotic database using CSV (and not programming a php routine, which I don't know nor understand).
This has worked for all vanilla Wordpress fields, i.e. post-data, categories and so on. (I used LibreOffice Calc to fine-tune the CSV data, Quest Toad for MySQL import and to create MySQL queries).
My designer has designed this entire thing extensively using the Magic Fields plugin. Therefore I have to deal with Magic Fields for the rest of the data that I need to have imported. It's a movie database, so it's the typical meta-data for movies like "productionyear" "producer" "imdblink" "youtubetrailerlink" and so on.
My question is that I am hoping to get some insights/ideas/support on how to import these data into the magic fields. I have tried and given up the "magic fields importer", it is not documented and does not give any error messages when I try to import various csv formats.
PART 02: Understanding Magic Fields
My question is how can I populate(insert) a magic field with data using a mysql query?
I have tried to understand what it is Magic Fields is doing when I put some data into a magic field and press save, in the admin-edit-post-frontend. It does a lot of things that I can't seem to re-create. For one it is not enough to insert data into the two places where the data is referenced. wp_postmeta and wp_mf_post_meta.
This example post is not sufficient to get the data to "work" i.e. display themselves in wordpress neither in the admin-post-editor nor the front-end for the user:
INSERT INTO `wp_postmeta` (`post_id`,`meta_key`,`meta_value`) VALUES ('474','originaltitle','Die Hard 3');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 1100083, 'originaltitle' , 1,1 ,474 );
Also with that meta_id number, Magic Fields creates a number series of 10 for each post, like 7000-7010 and then 8000-8010. I don't understand where it takes those numbers from, and whether they have to be sequential/consecutive. I have found that once entered by the admin-post-editor frontend, I can change the meta-id's in wp_postmeta and wp_mf_post_meta and it still works. But again when I try to create them myself with a SQL query, "it just doesn't work (TM)".
After enabling MySQL querylogging and examining the logfile from the MySQL db engine gives and then trying to insert the queries manually (that I can make out from the logs), I still can't make Wordpress "find" the data and display them properly on the page.
I can modify the data with mysql queries after I have populated them using the frontend with no problems.
Also using the query logs, I found that after I push "SAVE" in the Wordpress frontend-post-editor it does no less than ~780 mysql query lines (regardless if I populate 1 or 10 of my magic fields)!
The other thing I have tried is to take a snapshot of the database before and after I have done the post of the magic fields, and then used a diff-tool (various, WinMerge and Notepad++ mainly). I can't understand what it's doing but I THINK it is doing some hidden trickery in wp_term_taxonomy. This is just speculation.
Part 03: Conclusion I don't know programming so any practical solution, would be deeply apprechiated.
the full query I want to do per post (I have about 800 of these) is here: >http://pastebin.com/5cZT3AjA
The FULL list of queries that Magic Fields is doing after I push save in the admin-front-end is listed here (which I call the 'robot-at-work'). http://pastebin.com/c2c6qUQt To be sure I have checked three times that it wasn't something extraordinary that it is doing ~780 lines after I push save.
I guess if I could find a way to bulk-edit all of my posts, and then have the "robot" i.e. admin-post-edit frontend do the work to create all these lines in the database. I could then change the fields afterwards... But the bulk-editor doesn't show the magic-field editor either.
I have twisted my head around this problem 10 times now, and can't twist anymore!
SOLVED!!!!
Meta_id 's must be CHRONOLOGICAL and CANNOT be arbitrary (i.e. random).
data must be inserted into wp_mf_postmeta AND wp_postmeta
these records relate to post_id, meta_id and the key-value stores of
the data itself such where key would be "movietitle" and value would be "Die Hard".
I have uploaded the csv I created to do this and create the mysql queries.
I used LibreOffice Calc to finetune the data and Toad for MySQL (by Quest, free)
to create the MySQL queries...
These two records need to be modified for the data to be properly inserted:
NOTE THAT the META_ID MUST be chronological i.e. 11001, 11002 11003 per field per post
so if meta_key is Movietitle is first then that has meta_id of 11001, if productionyear is second then that is 11002 and if country is third then that is 11003
Also note that for the wp_mf_postmeta the fieldname is the SAME VALUE as meta_key
from wp_postmeta , I.e. the meta key.
INSERT INTO wp_postmeta (post_id,meta_key,meta_value) VALUES (346,'produksjonsar','18001');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 18001, 'produksjonsar' , 1,1 ,346 );
The file I have used for this import is available here, if anyone
needs to see an example of how to do this import... Again I used
Toad for MySQL to create the SQL Queries by using the IMPORT function.
http://ge.tt/9Q5h4zC
This also solves this question which I posted on the Wordpress Forums, Stackoverflow and Stackexchange.
http://wordpress.org/support/topic/importing-data-to-custom-fields?replies=2#post-2592976
https://wordpress.stackexchange.com/questions/37691/inserting-data-into-magicfields-using-mysql-queries
Importing Data into Wordpress-Magic Fields using MySQL queries
Massive props to Hameedullah Khan Stackexchange, profile here https://wordpress.stackexchange.com/users/5337/hameedullah-khan , for basically providing the clue needed to find the solution.
I have a large table of about 14 million rows. Each row has contains a block of text. I also have another table with about 6000 rows and each row has a word and six numerical values for each word. I need to take each block of text from the first table and find the amount of times each word in the second table appears then calculate the mean of the six values for each block of text and store it.
I have a debian machine with an i7 and 8gb of memory which should be able to handle it. At the moment I am using the php substr_count() function. However PHP just doesn't feel like its the right solution for this problem. Other than working around time-out and memory limit problems does anyone have a better way of doing this? Is it possible to use just SQL? If not what would be the best way to execute my PHP without overloading the server?
Do each record from the 'big' table one-at-a-time. Load that single 'block' of text into your program (php or what ever), and do the searching and calculation, then save the appropriate values where ever you need them.
Do each record as its own transaction, in isolation from the rest. If you are interrupted, use the saved values to determine where to start again.
Once you are done the existing records, you only need to do this in the future when you enter or update a record, so it's much easier. You just need to take your big bite right now to get the data updated.
What are you trying to do exactly? If you are trying to create something like a search engine with a weighting function, you maybe should drop that and instead use the MySQL fulltext search functions and indices that are there. If you still need to have this specific solution, you can of course do this completely in SQL. You can do this in one query or with a trigger that is run each time after a row is inserted or updated. You wont be able to get this done properly with PHP without jumping through a lot of hoops.
To give you a specific answer, we indeed would need more information about the queries, data structures and what you are trying to do.
Redesign IT()
If for size on disc is not !important just joints table into one
Table with 6000 put into memory [ memory table ] and make backup every one hour
INSERT IGNORE into back.table SELECT * FROM my.table;
Create "own" index in big table eq
Add column "name index" into big table with id of row
--
Need more info about query to find solution
I have an INSERT query that works fine, but when I add an extra field I get the oh-so-helpful:
Invalid object name 'optimizations'
Optimizations being the table name. The query is below... I am totally lost as to why this wouldn't work. If I cut & paste the query into MSSQL Management Studio, the query works fine.
INSERT INTO [optimizations]
([opt_date],[opt_concept_id],[opt_pallet],[opt_turnable],
[opt_sideupok],[opt_endupok],[opt_flatok],[opt_notstickingout],
[opt_notinoverhang],[opt_itemsloaded],[opt_loadedweight],
[opt_pweightutil],[opt_pvolumeutil],[opt_request_xml],
[opt_response_xml],[opt_images],[opt_primary_stage_id])
VALUES
(GETDATE(),'775','20ft','true','false','false','true',
'true','true','10','5952.6','18.6','48.9',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'img1.jpg,img2.jpg,img3.jpg','98');
For some reason, if I include the last field in the query [opt_primary_stage_id] it does not work... if I omit this field it works fine. Interestingly enough if I add [opt_primary_stage_id] to the list of fields being passed, and pass it a value of NULL the query also works fine. The [opt_primary_stage_id] field's data type is INT, I have tried including the last entry of the VALUES() section both as a straight number (3) and with quotes ('3')... neither works.
The server executing the code is running PHP 4.3.9, under IIS and with and MSSQL.
This is a version of the query, with the SAME data that will work.
INSERT INTO [optimizations]
([opt_date],[opt_concept_id],[opt_pallet],[opt_turnable],
[opt_sideupok],[opt_endupok],[opt_flatok],[opt_notstickingout],
[opt_notinoverhang],[opt_itemsloaded],[opt_loadedweight],
[opt_pweightutil],[opt_pvolumeutil],[opt_request_xml],
[opt_response_xml],[opt_images])
VALUES
(GETDATE(),'775','20ft','true','false','false','true','true',
'true','10','5952.6','18.6','48.9',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'img1.jpg,img2.jpg,img3.jpg');
EDIT: Fixed XML to have proper bracketing & added working query.
Adding a field won't cause failure on it's own: there is a valid reason.
You say it runs in SSMS OK then it shows that issue is in PHP or in the call.
First thoughts:
How long is the command. Is it being truncated somewhere in PHP? Is this dynamic SQL IRL?
Is the user context the same from the client? You may have the same table in 2 different schemas for example, with differing permissions.
Wrong database in the connection. What does SELECT DB_NAME() say? Sounds obvious but no object = not there. Don't dismiss this idea and assume it is the correct context: check.