I have a codeigniter controller that receives json content from an API and inserts it to a mysql longtext field unchanged.
Everything's been running smoothly for a while, but recently i've been noticing some of the content is being truncated. Here are two examples.
The table being inserted has three fields
id int
data longtext
date_added datetime
I am receiving the data and directly adding it to the database using the active record insert function. It looks like that
$this->db->insert('received', array('data' => $data, 'date_added' => date('Y-m-d H:i:s')));
Using the profiler, i monitored the queries and found two faulty ones:
INSERT INTO `received` (`data`, `date_added`) VALUES ('{\"status\":{\"lastMaintenanceAt\":0000,\"code\":304,\"period\":900},\"items\":[{\"permalinkUrl\":\"http://example.com\",\"updated\":0000,\"postedTime\":0000,\"summary\":\"Man\'s guarantees are like his words.\",\"title\":\"By: John\",\"content\":\"<p>Man’s guarantees are like his words.</p>\",\"id\":\"http://example.com\",\"actor\":{\"permalinkUrl\":\"\",\"displayName\":\"John\"}}],\"title\":\"Comments on: Banker refuses to give ‘Written Guarantee’\"}', '2012-04-08 00:28:29')
and
INSERT INTO `received` (`data`, `date_added`) VALUES ('{\"status\":{\"code\":304,\"period\":900,\"feed\":\"http://example.com\"},\"items\":[],\"title\":\"Comments on: Making her cry…\"}', '2012-04-08 00:49:35')
The queries seems alright. But only part of the JSON is making it to the table in the first case, it is truncated after "[...] refuses to give" and in the second after making her cry.
The queries are not returning an error and the date is being inserted properly. Moreover, if i copy the queries and execute them in a mysql command prompt, they will insert the full text.
These queries a one a few of hundreds of other successful ones.
Any idea what might be the problem?
thank you
All of the faulty queries share one thing in common: they've got "special characters" which MS Word inserts, e.g. ‘, ’ and …. If you can convert / remove these, then your problems should go away.
Update
For a CodeIgniter solution to this, you could load the Text Helper library and then use the ascii_to_entities function on any strings, e.g.
$this->load->helper('text');
$desc = ascii_to_entities($desc);
Related
I am doing some PostGIS work on a CakePHP application.
Because I've been working with some database functions, I've done raw $this->query() calls to do inserts of data. I'm at a point where I need to get the ID of the result of an insert query, but $this->query() returns an empty array.
Here is the query I'm using for inserts:
INSERT INTO locations (title,company_id,state_id,poly,point)
VALUES ('$title',$company_id,$state_id,ST_GeomFromText('$geom',4269),$point);
The problem is Cake is trying to insert 'ST_GeomFromText('$geom',4269)' as a string, and I cannot figure out how to get it to remove the quotes in the SQL insert statement it prepares via $this->Location->save();.
Any suggestions?
Edit:
Before I was doing $this->query(); and my SQL works fine. That's not the issue.
Now when I do:
$this->Location->create();
$this->Location->set('poly',$poly);
... Set all the rest of the model fields ...
$this->Location->save();
The SQL that gets produced looks like this:
INSERT INTO locations (title,company_id,state_id,poly,point)
VALUES ('Some Title',5,23,'ST_GeomFromText('$geom',4269)',POINT(-72.342,102.23455);
Because ST_GeomFromText gets 'quoted', PostgreSQL throws an error and doesn't insert the geometry.
The solution to my problem turned out to NOT be a CakePHP based one, but a PostgreSQL one!
By appending "RETURNING id" to the query like this:
INSERT INTO locations (title,company_id,state_id,poly,point)
VALUES ('$title',$company_id,$state_id,ST_GeomFromText('$geom',4269),$point) RETURNING id;
The query no longer returns an empty array, and now returns the ID of the row it just created!
Via $this->ModelName->getLastInsertID(); you are able to get the last inserted ID. So, after a save you can call this code to get its id.
I have a .sql file with 24k INSERT statements as:
INSERT INTO dump(id, title, content, datetime, section) VALUES ('', 'Title up to 200 characters', 'Detailed content with HTML tags', '04-10-2010, 11:48 AM', 1);
Once import is completed, i get a message that says
Import has been successfully finished, 24664 queries executed.
When i browse the table, i find many titles and contents are totally empty or carry part of texts only i.e. few letters.
The only parts that gets inserted properly is: id, datetime and section.
title and content carry heavy contents. Many "content" might carry 250 char+. Title carries around 100 chars max.
On the other hand, i find copy/paste few pieces of INSERT and do it manually via SQL area. Everything gets inserted correctly. This method could take me very long time...
Any idea why i'm facing such a problem?
Use MYSQL to create your dumps try this command
mysqldump --extended-insert=FALSE --complete-insert=TRUE -p db_name
Check the max_allowed_packet size and be sure that is not hurting your inserts.
PART 01: INTRO
Hey Stackoverflow
Greetings from the Snowy alps of Norway
I have now tried for two full consecutive days to reverse engineer Magic Fields. Why? Because I am a simple man, and want to import data to my wordpress from an exotic database using CSV (and not programming a php routine, which I don't know nor understand).
This has worked for all vanilla Wordpress fields, i.e. post-data, categories and so on. (I used LibreOffice Calc to fine-tune the CSV data, Quest Toad for MySQL import and to create MySQL queries).
My designer has designed this entire thing extensively using the Magic Fields plugin. Therefore I have to deal with Magic Fields for the rest of the data that I need to have imported. It's a movie database, so it's the typical meta-data for movies like "productionyear" "producer" "imdblink" "youtubetrailerlink" and so on.
My question is that I am hoping to get some insights/ideas/support on how to import these data into the magic fields. I have tried and given up the "magic fields importer", it is not documented and does not give any error messages when I try to import various csv formats.
PART 02: Understanding Magic Fields
My question is how can I populate(insert) a magic field with data using a mysql query?
I have tried to understand what it is Magic Fields is doing when I put some data into a magic field and press save, in the admin-edit-post-frontend. It does a lot of things that I can't seem to re-create. For one it is not enough to insert data into the two places where the data is referenced. wp_postmeta and wp_mf_post_meta.
This example post is not sufficient to get the data to "work" i.e. display themselves in wordpress neither in the admin-post-editor nor the front-end for the user:
INSERT INTO `wp_postmeta` (`post_id`,`meta_key`,`meta_value`) VALUES ('474','originaltitle','Die Hard 3');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 1100083, 'originaltitle' , 1,1 ,474 );
Also with that meta_id number, Magic Fields creates a number series of 10 for each post, like 7000-7010 and then 8000-8010. I don't understand where it takes those numbers from, and whether they have to be sequential/consecutive. I have found that once entered by the admin-post-editor frontend, I can change the meta-id's in wp_postmeta and wp_mf_post_meta and it still works. But again when I try to create them myself with a SQL query, "it just doesn't work (TM)".
After enabling MySQL querylogging and examining the logfile from the MySQL db engine gives and then trying to insert the queries manually (that I can make out from the logs), I still can't make Wordpress "find" the data and display them properly on the page.
I can modify the data with mysql queries after I have populated them using the frontend with no problems.
Also using the query logs, I found that after I push "SAVE" in the Wordpress frontend-post-editor it does no less than ~780 mysql query lines (regardless if I populate 1 or 10 of my magic fields)!
The other thing I have tried is to take a snapshot of the database before and after I have done the post of the magic fields, and then used a diff-tool (various, WinMerge and Notepad++ mainly). I can't understand what it's doing but I THINK it is doing some hidden trickery in wp_term_taxonomy. This is just speculation.
Part 03: Conclusion I don't know programming so any practical solution, would be deeply apprechiated.
the full query I want to do per post (I have about 800 of these) is here: >http://pastebin.com/5cZT3AjA
The FULL list of queries that Magic Fields is doing after I push save in the admin-front-end is listed here (which I call the 'robot-at-work'). http://pastebin.com/c2c6qUQt To be sure I have checked three times that it wasn't something extraordinary that it is doing ~780 lines after I push save.
I guess if I could find a way to bulk-edit all of my posts, and then have the "robot" i.e. admin-post-edit frontend do the work to create all these lines in the database. I could then change the fields afterwards... But the bulk-editor doesn't show the magic-field editor either.
I have twisted my head around this problem 10 times now, and can't twist anymore!
SOLVED!!!!
Meta_id 's must be CHRONOLOGICAL and CANNOT be arbitrary (i.e. random).
data must be inserted into wp_mf_postmeta AND wp_postmeta
these records relate to post_id, meta_id and the key-value stores of
the data itself such where key would be "movietitle" and value would be "Die Hard".
I have uploaded the csv I created to do this and create the mysql queries.
I used LibreOffice Calc to finetune the data and Toad for MySQL (by Quest, free)
to create the MySQL queries...
These two records need to be modified for the data to be properly inserted:
NOTE THAT the META_ID MUST be chronological i.e. 11001, 11002 11003 per field per post
so if meta_key is Movietitle is first then that has meta_id of 11001, if productionyear is second then that is 11002 and if country is third then that is 11003
Also note that for the wp_mf_postmeta the fieldname is the SAME VALUE as meta_key
from wp_postmeta , I.e. the meta key.
INSERT INTO wp_postmeta (post_id,meta_key,meta_value) VALUES (346,'produksjonsar','18001');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 18001, 'produksjonsar' , 1,1 ,346 );
The file I have used for this import is available here, if anyone
needs to see an example of how to do this import... Again I used
Toad for MySQL to create the SQL Queries by using the IMPORT function.
http://ge.tt/9Q5h4zC
This also solves this question which I posted on the Wordpress Forums, Stackoverflow and Stackexchange.
http://wordpress.org/support/topic/importing-data-to-custom-fields?replies=2#post-2592976
https://wordpress.stackexchange.com/questions/37691/inserting-data-into-magicfields-using-mysql-queries
Importing Data into Wordpress-Magic Fields using MySQL queries
Massive props to Hameedullah Khan Stackexchange, profile here https://wordpress.stackexchange.com/users/5337/hameedullah-khan , for basically providing the clue needed to find the solution.
I have an INSERT query that works fine, but when I add an extra field I get the oh-so-helpful:
Invalid object name 'optimizations'
Optimizations being the table name. The query is below... I am totally lost as to why this wouldn't work. If I cut & paste the query into MSSQL Management Studio, the query works fine.
INSERT INTO [optimizations]
([opt_date],[opt_concept_id],[opt_pallet],[opt_turnable],
[opt_sideupok],[opt_endupok],[opt_flatok],[opt_notstickingout],
[opt_notinoverhang],[opt_itemsloaded],[opt_loadedweight],
[opt_pweightutil],[opt_pvolumeutil],[opt_request_xml],
[opt_response_xml],[opt_images],[opt_primary_stage_id])
VALUES
(GETDATE(),'775','20ft','true','false','false','true',
'true','true','10','5952.6','18.6','48.9',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'img1.jpg,img2.jpg,img3.jpg','98');
For some reason, if I include the last field in the query [opt_primary_stage_id] it does not work... if I omit this field it works fine. Interestingly enough if I add [opt_primary_stage_id] to the list of fields being passed, and pass it a value of NULL the query also works fine. The [opt_primary_stage_id] field's data type is INT, I have tried including the last entry of the VALUES() section both as a straight number (3) and with quotes ('3')... neither works.
The server executing the code is running PHP 4.3.9, under IIS and with and MSSQL.
This is a version of the query, with the SAME data that will work.
INSERT INTO [optimizations]
([opt_date],[opt_concept_id],[opt_pallet],[opt_turnable],
[opt_sideupok],[opt_endupok],[opt_flatok],[opt_notstickingout],
[opt_notinoverhang],[opt_itemsloaded],[opt_loadedweight],
[opt_pweightutil],[opt_pvolumeutil],[opt_request_xml],
[opt_response_xml],[opt_images])
VALUES
(GETDATE(),'775','20ft','true','false','false','true','true',
'true','10','5952.6','18.6','48.9',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'<xml><herpa><derpa>xml</derpa></herpa></xml>',
'img1.jpg,img2.jpg,img3.jpg');
EDIT: Fixed XML to have proper bracketing & added working query.
Adding a field won't cause failure on it's own: there is a valid reason.
You say it runs in SSMS OK then it shows that issue is in PHP or in the call.
First thoughts:
How long is the command. Is it being truncated somewhere in PHP? Is this dynamic SQL IRL?
Is the user context the same from the client? You may have the same table in 2 different schemas for example, with differing permissions.
Wrong database in the connection. What does SELECT DB_NAME() say? Sounds obvious but no object = not there. Don't dismiss this idea and assume it is the correct context: check.
I have a mysql database and a table to store stuff from twitter API. When i parse data with PHP from twitter i use mysql_query to insert data into the table.
I have a weird problem with the ID of the tweets:
For example the status update with ID 15861323074113537 (a tweet from google) is stored in the database as: 15861323074114000 (the last 4 digits are altered).
The php query is:
$sql = mysql_query("INSERT INTO $table (id,tw_text) VALUES ($id,'$tw_text')");
If i edit the record via phpmyadmin the correct value is stored (15861323074113537). The column is BIGINT.
So i guess something weird is going on with the php function mysql_query and the INSERT command.
Any ideas/solutions?
Thanks in advance
ok found the problem, it wasn't code related. Twitter returns 2 IDs and one of those has 000 in the end. [id_str] => 15861323074113537 and [id] => 15861323074114000
Damn twitter API!
Sorry for the trouble :/
PHP integers cannot be that large. When a number that doesn't fit into an integer is used, it is converted to a float. What you are probably seeing is that even the float is not precise enough to contain that many digits. Follow webbiedave's advice and keep it as a string to avoid rounding.