So yeah, the issue is there are some articles, (very old one 2015~), which anonymous users can not access after i've done rebuilding permissions.
New content does not seem to be affected though. One solution I can do (after researching) is to re-save these articles, BUT I think i can not keep doing this because there are a lot of articles., im telling more than 100K
Is there a better way to resolve this?
PS: I confirm that the permission is set correctly for anonymous users to see published content.
We can only speculate what happen, without diving into your custom code or checking at the list of contrib modules that you are using it is almost impossible to detect where the issue is. If you are sure that re-save makes the problem go away, add views bulk operation module - which will allow you to select all nodes in one go than select publish bulk operation (this will trigger node save). You can narrow down and add filters to your admin/content view to show only older nodes (around 10k nodes takes 15 mins to resave - regular node article) - this will not bring down your site or slow it down since it's bulk operation and you can always do this during the night when there are few users on the site... Do a db backup before, resave nodes on live so the users can access nodes, import the database on your local machine and peacefully (since live is working) hunt down the source of the issue.
Related
We've been developing for Wordpress for several years and whilst our workflow has been upgraded at several points there's one thing that we've never solved... merging a local Wordpress database with a live database.
So I'm talking about having a local version of the site where files and data are changed, whilst the data on the live site is also changing at the same time.
All I can find is the perfect world scenario of pulling the site down, nobody (even customers) touching the live site, then pushing the local site back up. I.e copying one thing over the other.
How can this be done without running a tonne of mysql commands? (it feels like they could fall over if they're not properly checked!) Can this be done via Gulp's (I've seen it mentioned) or a plugin?
Just to be clear, I'm not talking about pushing/pulling data back and forth via something like WP Migrate DB Pro, BackupBuddy or anything similar - this is a merge, not replacing one database with another.
I would love to know how other developers get around this!
File changes are fairly simple to get around, it's when there's data changes that it causes the nightmare.
WP Stagecoach does do a merge but you can't work locally, it creates a staging site from the live site that you're supposed to work on. The merge works great but it's a killer blow not to be able to work locally.
I've also been told by the developers that datahawk.io will do what I want but there's no release date on that.
It sounds like VersionPress might do what you need:
VersionPress staging
A couple of caveats: I haven't used it, so can't vouch for its effectiveness; and it's currently in early access.
Important : Take a backup of Live database before merging Local data to it.
Follow these steps might help in migrating the large percentage of data and merging it to live
Go to wp back-end of Local site Tools->Export.
Select All content radio button (if not selected by default).
This will bring an Xml file containing all the local data comprised of all default post types and custom post types.
Open this XML file in notepad++ or any editor and find and replace the Local URL with the Live URL.
Now visit the Live site and Import the XML under Tools->Import.
Upload the files (images) manually.
This will bring a large percentage of data from Local to Live .
Rest of the data you will have to write custom scripts.
Risk factors are :
When uploading the images from Local to Live , images of same name
will be overriden.
Wordpress saves the images in post_meta generating a serialized data for the images , than should be taken care of when uploading the database.
Serialized data in post_meta for post_type="attachment" saves serialized data for 3 or 4 dimensions of the images.
Usernames or email ids of users when importing the data , can be same (Or wp performs the function of checking unique usernames and emails) then those users will not be imported (might be possible).
If I were you I'd do the following (slow but affords you the greatest chance of success)
First off, set up a third database somewhere. Cloud services would probably be ideal, since you could get a powerful server with an SSD for a couple of hours. You'll need that horsepower.
Second, we're going to mysqldump the first DB and pipe the output into our cloud DB.
mysqldump -u user -ppassword dbname | mysql -u root -ppass -h somecloud.db.internet
Now we have a full copy of DB #1. If your cloud supports snapshotting data, be sure to take one now.
The last step is to write a PHP script that, slowly but surely, selects the data from the second DB and writes it to the third. We want to do this one record at a time. Why? Well, we need to maintain the relationships between records. So let's take comments and posts. When we pull post #1 from DB #2 it won't be able to keep record #1 because DB #1 already had one. So now post #1 becomes post #132. That means that all the comments for post #1 now need to be written as belonging to post #132. You'll also have to pull the records for the users who made those posts, because their user IDs will also change.
There's no easy fix for this but the WP structure isn't terribly complex. Building a simple loop to pull the data and translate it shouldn't be more then a couple of hours of work.
If I understand you, to merge local and live database, until now I'm using other software such as NavicatPremium, it has Data Sycn feature.
This can be achieved live using spring-xd, create a JDBC Stream to pull data from one db and insert into the other. (This acts as streaming so you don't have to disturb any environment)
The first thing you need to do is asses if it would be easier to do some copy-paste data entry instead of a migration script. Sometimes the best answer is to suck it up and do it manually using the CMS interface. This avoids any potential conflicts with merging primary keys, but you may need to watch for references like the creator of a post or similar data.
If it's just outright too much to manually migrate, you're stuck with writing a script or finding one that is already written for you. Assuming there's nothing out there, here's what you do...
ALWAYS MAKE A BACKUP BEFORE RUNNING MIGRATIONS!
1) Make a list of what you need to transfer. Do you need users, posts, etc.? Find the database tables and add them to the list.
2) Make a note all possible foreign keys in the database tables being merged into the new database. For example, wp_posts has post_author referencing wp_users. These will need specific attention during the migration. Use this documentation to help find them.
3) Once you know what tables you need and what they reference, you need to write the script. Start by figuring out what content is new for the other database. The safest way is to do this manually with some kind of side-by-side list. However, you can come up with your own rules on how to automatically match table rows. Maybe to check for $post1->post_content === $post2->post_content in cases the text needs to be the same. The only catch here is the primary/foreign keys are off limits for these rules.
4) How do you merge new content? The general idea is that all primary keys will need to be changed for any new content. You want to use everything except for the id of post and insert that into the new database. There will be an auto-increment to create the new id, so you wont need the previous id (unless you want it for script output/debug).
5) The tricky part is handling the foreign keys. This process is going to vary wildly depending on what you plan on migrating. What you need to know is which foreign key goes to which (possibly new) primary key. If you're only migrating posts, you may need to hard-code a user id to user id mapping for the post_author column, then use this to replace the values.
But what if I don't know the user ids for the mapping because some users also need to be migrated?
This is where is gets tricky. You will need to first define the merge rules to see if a user already exists. For new users, you need record the id of the newly inserted users. Then after all users are migrated, the post_author value will need to be replaced when it references a newly merged user.
6) Write and test the script! Test it on dummy databases first. And again, make backups before using it on your databases!
I've done something simillar with ETL (Extract, Transform, Load) process when I was moving data from one CMS to another.
Rather than writing a script I used a Pentaho Data Integration (Kettle) tool.
The Idea of ETL is pretty much straight forward:
Extract the data (for instance from one database)
Transform it to suit your needs
Load it to the final destination (your second database).
The tool is easy to use and it allows you to experiment with various steps and outputs to investigate the data. When you design a right ETL proces, you are ready to merge those databases of yours.
How can this be done without running a tonne of mysql commands?
No way. If both local and web sites are running at the same time how can you prevent not having the same ids' with different content?
so if you want to do this you can use mysql repication.i think it will help you to merge with different database mysql.
As always, I apologize if this question has been discussed somewhere.
I'm curious for some insight on how everyone is handling related tables and assets when a user is deleted. The project I'm currently working on is mostly a project management / issue tracking system.
When a user is deleted what should happen to their issues, projects, files, etc..?
A few scenarios off the top of my head were to either:
Delete all their issues, files, projects, etc...
Reparent everything to the only (or one of many) admin user
Lock the user account but keep all their assets (but I may need to REALLY delete them at some point if server storage becomes an issue)
Assign a non-existent user_id (0) to related tables (This seems like it would be too problematic when trying to JOIN tables
Any other possible considerations I'm missing? The third solution sounds the best to me right now. Those of you who've worked on large projects, how have you handled deleting users? On a side note, I'm developing the project in php (yii) and mysql.
You should delete all data that is not used by other users.
Because of situations in which other users are affected, like issues or tasks, you should just deactivate the user profile. The best way doing this is by adding a column 'active' to the database and set it to 0 if the user gets deleted. In your system, you can parse this column and display something like 'Deleted user'.
If no one else is assigned to a task, it would be the best if you let them unassigned and make something like a pool in which other users can choose tasks or get assigned to tasks.
The same way I'd do it with other data like projects.
Concerning files it does make sense to delete them if they are not required by anything any more.
Always remember, it's better to have some more data on the server than loosing important information!
Actually deleting a user and related objects should be the last thing you do. Even a deleted user is historical and useful data. You can always archive it if it truly becomes burdensome.
As recommended by Lukas, deactivate the user instead. All of your queries that request user-related objects should then check the 'active' flag to include/exclude the user. When you have that complex join query to retrieve the data, it saves a lot of effort and processing with that flag.
When it comes to 'actual' assets, like a file, those can take up a lot of space, but if they are related to a inactive user, you should not delete those either. Archive again, if necessary. It may require adding an 'archived' flag to an entry pointing to that asset, so that you're not trying to open the file when it does not exist.
i'm going crazy trying to solve a VERY weird error in a PHP script (Joomla). I have a page that displays multiple select dropdown lists, all of them showing the same list of items (the selected item changes from one list to another, but listed items are the same). This list has around 35-40 items. This works fine until a certain amount of selects, but when i put more than 20 or 25 selects on the same page, it doesn't work and shows only a white page. No errors, there is no text displayed, no errors in php logs, nothing; just a white page. If using THE SAME CODE, i display 11 dropdown select lists... it works.
I'm guessing that this problem is related to memory or something like that, but i can't be sure cause as i've said, there is no errors displayed. Does anyone knows about a simmilar issues? can anyone give me a tip about how to address this problem? i don't know what to do and i've tried many things but it still doesn't work. Any help will be very much appreciated and wellcomed...
NOTE: The select list are filled with values from a DB table and each select list has a different selected item based on contents from another table. It's not a very complex code and as i've said, it works fine when i use less select lists on the same page. The problem is when i reach a certain number of select lists on the same page (i think that it's around 20 or 25 input selects). I think that the amount of data is not very exagerated, so i can't understand why it doesn't work¿!?
A quick google for your issue turns this up:
for jos_session, which is the only table I suggested you empty, any logged on users will be logged off...any work in progress (forum posts/articles would be lost)...
I also empty recaptcha...
Please remember to always back up your db first.
I empty these two for a higher volume joomla 1.5 system once a week...we also set the session lifetime (no activity) varies 60-90 minutes...6-7k day volume site...this also helps our akeeba back up, as the two aforementioned tables can get very large without proper maintenance.
Just some general ramblings...
You should also review your mysql site report via phymyadmin "Show MySQL runtime information". Look for things that are in 'red'.
As for your overall question about performance. Please remember that there are many ways to improve websites performance. It's yes another job required by site administrators and at least an interesting process.
The joomla performance forum is a great place to have your site reviewed and get good help tuning your site including the minimum base server you need (shared/vps/dedicicated).
IMHO...First objective is to turn off joomla cache and joomla gzip by enabling standard server modules like mod_deflate and mod_expires (mod_expires is one of the best fixes for returning visitors). Make sure you mysql configuration enabling query_cache are or can be set. You will need a minimum of a vps. and there's more!...jajajaja
A little note about running shared server, and not having certain server modules available,
check this out: http://www.webogroup.com/ It's really one heck of a product. I've used it on the aforementioned site until I could implement the changes on the server. As I implemented each new server module I turned off the webo...site is now boring fast
have fun
Our company has been quite successful in managing its website including all of the business logic and stuff. However, there are also a lot of static content pages which today get served using a templating system which stores the content in serialized PHP objects on the file system.
We are now considering using a "real" CMS, however we have some requirements which sort out more or less all the usual suspects. The most important requirement is our hosting environment:
We have two completely separate hosting locations with a "share nothing" approach for failover. Both locations have separate MySQL instances which are slaves () of our master database which is located on-site at our HQ. Both locations have a certain number of web servers each storing the complete website (again, for failover).
From this architecture, two possible approaches come out naturally:
- A database-driven CMS which gets managed at our HQ and gets replicated over to our hosting locations (and images and stuff which gets replicated using our file sync process)
- A file driven CMS in which not only the attachments, but also the content files get synced using our file sync
The database driven approach seems more flexible to me, however we couldn't find a CMS which works in a "administer locally on a read/write database and serve content using only a read-only slave". The usual suspect for example (Typo3) needs a database to write to for its logging and session management, is therefore not an option. Other CMS seem to share this problem.
So, long story short, is there a (PHP/MySQL-)CMS out there which can handle this? Any suggestions?
Extra points if the CMS can easily integrated with our Zend Framework applications (or vice versa).
You should look into Percona, which is a MySQL Performance Company who put MySQL on steroids. The easily support a Master/Master/Master environment, and can achieve it easily without needing to change auto-increment values from master to master.
They have a product called, XtraDB Cluster. It is a free prodcut, works just like MySQL, installs the same way, but handles clustering at the DB level and does a damn good job.
Once you have your DB's under control, you can install a CMS on one of the servers, make your changes, copy it to all other servers and your entire environment is fail-over ready.
According to the added information in your comment it seems we are talking about static content.
Existing systems
As far as I can see you have a system already in place and stable which is able to deploy static content. In that sense it does not make sense to make more complex situations. Keep it simple will most of the time work best when it fits your needs.
Depending on your needs you can make the choice between your database sync or your file sync.
File sync
If you can send everything totally static your file sync will do just well. Though I see an issue there. For example: If you need a list of latest news items you would also have to generate and sync it. When you use database sync you can just do the simple query needed SELECT * FROM news ORDER BY created_at DESC LIMIT 0,10.
Amount of synchronization actions
Another point is the amount of synchronization you need to do. If you have a screen for example with 10 news items in your backend. The writer created 3 posts and starts clicking the publish button for each of them. In basic your file sync would start syncing after the first one. So it should first sync then new news item. Then it should update the latest news items listing and then sync that one.
That could be kind of bad if they start editing quite well. Because this also needs to be done on a title change for example.
Database solution
This is where a database is better at, you just update the record and it will get there. The latest news section will update itself when the new record becomes available.
Issue with related static content
Now you have one issue here: You have to make sure your process goes in 2 steps:
1. Sync images and other static content
2. Sync and/or publish the news item
Otherwise you will see broken images. This could be done for example by: Make sure that the sync of a new image starts directly. So when it is uploaded to the CMS. Then it will be there before the news item is published. You could verify that based on your needs.
Alternatives
Since this is static content you might be able to process it a bit better. That would require new technology to your stack so first verify whether it is really needed. A news item is a workflow. So, it has steps next to each other. In general for a workflow you could use queues. So you create a queue with new news items, a queue for publishing, one for syncing static items etc.
Queues work quite well with caching systems. And there is a quite interesting part to review I think. Currently there are strong caching systems available which can make the content available and sync them. They are based on the fact that you have lots of servers serving the same content. They might be a very stable and simple solution to get your work done. But as said, if you don't have them yet consider whether you want to add new complexity to your stack.
I've been working on getting a sitemap to generate automatically off a cron job.
This is the one I've been using - http://www.xml-sitemaps.com/
Also I've played with someothers too.
http://www.xml-sitemaps.com
I've found this code to be very very average.
I leaves endless log files and session files the grow every time it runs and need the folder/file perms opened up to cleanup. (eg: to get PHP to clean these files up).
Also it doesn't always generate the sitemap sometimes - just doesn't update for unknown reasons.
Thus I have a question.
How important is a sitemap?
Does it really matter if I don't have one?
Will I really do better in google with one?
Also is there a better product I can use?
Would prefer php code that can audit a domainname - not DB - and I'ved already tried a few that were very average to say the least.
thx