PART 01: INTRO
Hey Stackoverflow
Greetings from the Snowy alps of Norway
I have now tried for two full consecutive days to reverse engineer Magic Fields. Why? Because I am a simple man, and want to import data to my wordpress from an exotic database using CSV (and not programming a php routine, which I don't know nor understand).
This has worked for all vanilla Wordpress fields, i.e. post-data, categories and so on. (I used LibreOffice Calc to fine-tune the CSV data, Quest Toad for MySQL import and to create MySQL queries).
My designer has designed this entire thing extensively using the Magic Fields plugin. Therefore I have to deal with Magic Fields for the rest of the data that I need to have imported. It's a movie database, so it's the typical meta-data for movies like "productionyear" "producer" "imdblink" "youtubetrailerlink" and so on.
My question is that I am hoping to get some insights/ideas/support on how to import these data into the magic fields. I have tried and given up the "magic fields importer", it is not documented and does not give any error messages when I try to import various csv formats.
PART 02: Understanding Magic Fields
My question is how can I populate(insert) a magic field with data using a mysql query?
I have tried to understand what it is Magic Fields is doing when I put some data into a magic field and press save, in the admin-edit-post-frontend. It does a lot of things that I can't seem to re-create. For one it is not enough to insert data into the two places where the data is referenced. wp_postmeta and wp_mf_post_meta.
This example post is not sufficient to get the data to "work" i.e. display themselves in wordpress neither in the admin-post-editor nor the front-end for the user:
INSERT INTO `wp_postmeta` (`post_id`,`meta_key`,`meta_value`) VALUES ('474','originaltitle','Die Hard 3');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 1100083, 'originaltitle' , 1,1 ,474 );
Also with that meta_id number, Magic Fields creates a number series of 10 for each post, like 7000-7010 and then 8000-8010. I don't understand where it takes those numbers from, and whether they have to be sequential/consecutive. I have found that once entered by the admin-post-editor frontend, I can change the meta-id's in wp_postmeta and wp_mf_post_meta and it still works. But again when I try to create them myself with a SQL query, "it just doesn't work (TM)".
After enabling MySQL querylogging and examining the logfile from the MySQL db engine gives and then trying to insert the queries manually (that I can make out from the logs), I still can't make Wordpress "find" the data and display them properly on the page.
I can modify the data with mysql queries after I have populated them using the frontend with no problems.
Also using the query logs, I found that after I push "SAVE" in the Wordpress frontend-post-editor it does no less than ~780 mysql query lines (regardless if I populate 1 or 10 of my magic fields)!
The other thing I have tried is to take a snapshot of the database before and after I have done the post of the magic fields, and then used a diff-tool (various, WinMerge and Notepad++ mainly). I can't understand what it's doing but I THINK it is doing some hidden trickery in wp_term_taxonomy. This is just speculation.
Part 03: Conclusion I don't know programming so any practical solution, would be deeply apprechiated.
the full query I want to do per post (I have about 800 of these) is here: >http://pastebin.com/5cZT3AjA
The FULL list of queries that Magic Fields is doing after I push save in the admin-front-end is listed here (which I call the 'robot-at-work'). http://pastebin.com/c2c6qUQt To be sure I have checked three times that it wasn't something extraordinary that it is doing ~780 lines after I push save.
I guess if I could find a way to bulk-edit all of my posts, and then have the "robot" i.e. admin-post-edit frontend do the work to create all these lines in the database. I could then change the fields afterwards... But the bulk-editor doesn't show the magic-field editor either.
I have twisted my head around this problem 10 times now, and can't twist anymore!
SOLVED!!!!
Meta_id 's must be CHRONOLOGICAL and CANNOT be arbitrary (i.e. random).
data must be inserted into wp_mf_postmeta AND wp_postmeta
these records relate to post_id, meta_id and the key-value stores of
the data itself such where key would be "movietitle" and value would be "Die Hard".
I have uploaded the csv I created to do this and create the mysql queries.
I used LibreOffice Calc to finetune the data and Toad for MySQL (by Quest, free)
to create the MySQL queries...
These two records need to be modified for the data to be properly inserted:
NOTE THAT the META_ID MUST be chronological i.e. 11001, 11002 11003 per field per post
so if meta_key is Movietitle is first then that has meta_id of 11001, if productionyear is second then that is 11002 and if country is third then that is 11003
Also note that for the wp_mf_postmeta the fieldname is the SAME VALUE as meta_key
from wp_postmeta , I.e. the meta key.
INSERT INTO wp_postmeta (post_id,meta_key,meta_value) VALUES (346,'produksjonsar','18001');
INSERT INTO wp_mf_post_meta ( meta_id, field_name, field_count, group_count, post_id ) VALUES ( 18001, 'produksjonsar' , 1,1 ,346 );
The file I have used for this import is available here, if anyone
needs to see an example of how to do this import... Again I used
Toad for MySQL to create the SQL Queries by using the IMPORT function.
http://ge.tt/9Q5h4zC
This also solves this question which I posted on the Wordpress Forums, Stackoverflow and Stackexchange.
http://wordpress.org/support/topic/importing-data-to-custom-fields?replies=2#post-2592976
https://wordpress.stackexchange.com/questions/37691/inserting-data-into-magicfields-using-mysql-queries
Importing Data into Wordpress-Magic Fields using MySQL queries
Massive props to Hameedullah Khan Stackexchange, profile here https://wordpress.stackexchange.com/users/5337/hameedullah-khan , for basically providing the clue needed to find the solution.
Related
I have products stored in a MySQL database, it's a Wordpress website, however my data in stored in custom tables. I need to search for products and I'm currently facing some performance issues that I hope someone could help me or point me a way.
Since I receive a file (*.csv) once a day to update all my products (add, update or remove products), I have a process to read the file and populate/update tables. In this process, I add a step to filter data and replace any special character to "unpecial" characters (example: replace 'รก' by 'a').
By now, I have a table (products_search) related to product's table (products) and built from it, I use this table to do searches. When the user search something, I modify the input to replace special characters, so the search would be direct on table.
The problem: searching in "text" columns is slow, even adding index on that column. I'm currently search like this:
select * from products_search
where description like %search_word_1%
or description like %search_word_2% ...
If I get a result, I will get the ID and relate to product table and get all info I might need to show to user.
Solution looked for: I'm looking for a way to search on products_search table with a better performance. The wordpress search engine, as I understand, work only on "posts" table. Is there any way to do a quicker search? Perhaps using a plugin or just change the way the search is doing.
Thanks to all
I think we need to revise the nightly loading in order to make the index creation more efficient.
I'm assuming:
The data in the CSV file replaces the existing data.
You are willing to use FULLTEXT for searching.
Then do:
CREATE TABLE new_data (...) ENGINE=InnoDB;
LOAD DATA INTO new_data ...;
Cleanse the data in new_data.
ALTER TABLE new_data ADD FULLTEXT(...); The column(s) to index here either exist, or are added during step 1 or 3.
RENAME TABLE real_data TO old_data, new_data TO real_data;
DROP TABLE old_data;
Note that this has essentially zero downtime for real_data so you can continue to do SELECTs.
You have not explained how you spray the single CSV file into wp_posts and wp_postmeta. That sounds like a nightmare buried inside my step 3.
FULLTEXT is immensely faster than futzing with wp_postmeta. (I don't know if there is an existing way or plugin to achieve such.)
With `FULLTEXT(description), your snippet of code would use
WHERE MATCH(description) AGAINST ('word1 word2' IN BOOLEAN MODE)
instead of the very slow LIKE with a leading wildcard.
If you must use wp_postmeta, I recommend https://wordpress.org/plugins/index-wp-mysql-for-speed/
I have a database table that looks like this, with some more information
Basically, I'm collecting results from e-Sports games in this table. Now, when I read the data, I'm putting them in tables to see how each team performs. Unfortunately, sometimes the team I have selected is team1, sometimes team2. Displaying this in a table really hurts the readability:
I'd like to have the team I have selected always on the left side. If I'd just switch team1 and team2 in the code, other information (like team1_roster or team1_percent) would be assigned to the wrong team. Is there any way to do this without using complex inconvenient methods?
Thanks in advance!
Do the "change/switch" in whatever code read the SQL and/or renders the table based on the selected condition.
This is no fundamentally different than choosing the column coloring, except it works on the row and not individual cell. This can be written 'generically' outside of the rendering as well; if reading each row as a "pair of data" (each team info is a datum) and then simply swap the pairs based on condition before containing processing.
This change is merely about the view (or model passed to the view generation) and it should not affect the underlying data or column names therein.
If just handled in SQL it would be a "complex inconvenient" query because it is about transposing N columns based on a condition, which is not something SQL likes to express.
I am using WordPress with some custom post types (just to give a description of my DB structure - its WP's).
Each post has custom meta, which is stored in a separate table (postmeta table). In my case, I am storing city and state.
I've added some actions to WP's save_post/trash_post hooks so that the city and state are also stored in a separate table (cities) like so:
ID postID city state
auto int varchar varchar
I did this because I assumed that this table would be faster than querying the rather large postmeta table for a list of available cities and states.
My logic also forced me to add/update cities and states for every post, even though this will cause duplicates (in the city/state fields). This must be so because I must keep track of which states/cities exist (actually have a post associated with them). When a post is added or deleted, it takes its record to or from the cities table with it.
This brings me to my question(s).
Does this logic make sense or do I suck at DB design?
If it does make sense, my real question is this: **would it be faster to use MySQL's "SELECT DISTINCT" or just "SELECT *" and then use PHP's array_unique on the results?**
Edits for comments/answers thus far:
The structure of the table is exactly how I typed it out above. There is an index on ID, but the point of this table isn't to retrieve an indexed list, but to retrieve ALL results (that are unique) for a list of ALL available city/state combos.
I think I may go with (I don't know why I didn't think of this before) just adding a serialized list of city/state combos in ONE record in the wp_options table. Then I can just get that record, and filter out the unique records I need.
Can I get some feedback on this? I would imagine that retrieving and filtering a serialized array would be faster than storing the data in a separate table for retrieval.
To answer your question about using SELECT distinct vs. array_unique, I would say that I would almost always prefer to limit the result set in the database assuming of course that you have an appropriate index on the field for which you are trying to get distinct values. This saves you time in transmitting extra data from DB to application and for the application reading that data into memory where you can work with it.
As far as your separate table design, it is hard to speculate whether this is a good approach or not, this would largely depend on how you are actually preforming your query (i.e. are you doing two separate queries - one for post info and one for city/state info or querying across a join?).
The is really only one definitive way to determine what is fastest approach. That is to test both ways in your environment.
1) Fully normalized table(when it have only integer values and other tables have only one int+varchar) have advantage when you not dooing full table joins often and dooing alot of search on normalized fields. As downside it require large join/sort buffers and result more complex queries=much less chance query will be auto-optimized by mysql. So you have optimize your queries yourself.
2)Select distinct will be faster in almost any cases. Only case when it will be slower - you have low size sort buffer in /etc/my.conf and much more size memory buffer for php.
Distinct select can use indexes, while your code can't.
Also sending large amount of data to your app require alot of mysql cpu time and real time.
I'm running a sort of forum modification code on my server. In my database, I have a HUGE table called core, which basically includes all the admin-editable settings, of which there are a lot. My question is, should I normalize this?
I've created a fiddle to show you how large this table is: http://sqlfiddle.com/#!2/f4536/1
You'll see some columns called gear_notifications, gear_bank, gear_* etc. These indicate whether a certain system is turned on. For example, if gear_bank=1, then the Bank System is turned on. At the moment, in the __construct of my DB file, I run the following query:
$settings = mysql_query("SELECT GROUP_CONCAT(d.domain) AS domains, GROUP_CONCAT(d.zb) AS zb_info, c.* FROM core c JOIN domains d ON d.cid = c.id WHERE c.id ='$cid' LIMIT 1");
Ignoring the JOIN you can straight away see a problem here; the query returns EVERY field from the core table, regardless of whether the corresponding system is turned on. For example, if gear_bank=0, the query will still return bank_name, bank_history_perpage, bank_* etc. While this doesn't present any physical problem with how the code runs (as it can just ignore any data it does not need), I'd much prefer if it didn't have to load that data.
Would I be better off creating one table called core which has all the gear_* values, then corresponding tables (core_bank, core_* etc) for their corresponding values?
I've been querying this (bad pun, sorry!) for a while now and I just don't know enough to work out whether this will provide a performance boost for my code. Note that the above query will be run on EVERY page.
If I were to revert to the new system, i.e the multiple tables, how would I get all the information I need? I don't want to run one query to core to work out which systems are turned on, then subsequently run an extra query on all corresponding tables. I'd much prefer one query which JOINs all necessary tables based on the values of gear_* in core. Would that be possible?
I've not yet released this forum modification so I can make as many changes as I like without any real-world impact :) .
If you split your core into multiple tables you can then use NULL to indicate if a particular entry has info for a particular system. The joins will then work as expected.
However, your table really is not that big and it's doubtful that you will actually notice any speed improvement at the application level by breaking it up.
And if you don't need to query on individual columns then just use one TEXT column, put all your attributes in an array and then serialize the array into the text column.
I'm running a sql query to get basic details from a number of tables. Sorted by the last update date field. Its terribly tricky and I'm thinking if there is an alternate to using the UNION clause instead...I'm working in PHP MYSQL.
Actually I have a few tables containing news, articles, photos, events etc and need to collect all of them in one query to show a simple - whats newly added on the website kind of thing.
Maybe do it in PHP rather than MySQL - if you want the latest n items, then fetch the latest n of each of your news items, articles, photos and events, and sort in PHP (you'll need the last n of each obviously, and you'll then trim the dataset in PHP). This is probably easier than combining those with UNION given they're likely to have lots of data items which are different.
I'm not aware of an alternative to UNION that does what you want, and hopefully those fetches won't be too expensive. It would definitely be wise to profile this though.
If you use Join in your query you can select datas from differents tables who are related with foreign keys.
You can look of this from another angle: do you need absolutely updated information? (the moment someone enters new information it should appear)
If not, you can have a table holding the results of the query in the format you need (serving as cache), and update this table every 5 minutes or so. Then your query problem becomes trivial, as you can have the updates run as several updates in the background.