How to export lots of wordpress posts to text document? - php

I need to export all posts from certain category (that contains thousands posts) to a text document. And then someone will make corrections and changes in this document, and after this I have to enter all the updated posts to WP. So I decided that the best way is to make a XML document (by this way it could be easy to enter the posts back).
So my code is:
require_once(dirname(__FILE__) . '/wp-blog-header.php');
$counter = 0;
$recorded = array();
$double=0;
$handle = fopen("all_posts.xml", "w");
fwrite($handle, "<all_posts>" . "\r\n"); // the root XML tag
// get all the categories from the global category
$global_cat = get_categories(array("child_of"=>5, 'pad_counts'=>true, 'hierarchical' =>
false));
foreach($global_cat as $child_cat){
global $post;
$args = array('numberposts' => 50000,'cat' => $child_cat->cat_ID);
print_r($child_cat); echo "<br>" . $counter ."<br>";
$q_posts = get_posts($args);
foreach($q_posts as $post){
setup_postdata($post);
if( in_array($post->ID, $recorded ) ) {continue;}
$recorded[] = $post->ID;
$counter++;
$title = get_the_title();
$cur_categories = get_the_category();
$cur_tags = get_the_tags();
$d = get_the_date();
$cont = get_the_content();
fwrite($handle, "<post>" . "\r\n");
fwrite($handle, "<title>" . $title . "</title>" . "\r\n");
fwrite($handle, "<id>" . $post->ID . "</id>" . "\r\n");
fwrite($handle, "<cur_cat>" . $child_cat->name . "</cur_cat>" . "\r\n");
fwrite($handle, "<categories>\r\n");
foreach ($cur_categories as $cat) {
fwrite($handle, "<cat>" . $cat->cat_name . "</cat>");
}
fwrite($handle, "\r\n</categories>" . "\r\n");
fwrite($handle, "<tags>\r\n");
foreach ($cur_tags as $tag) {
fwrite($handle, "<tag>" . $tag->name . "</tag>");
}
fwrite($handle, "\r\n</tags>" . "\r\n");
fwrite($handle, "<date>" . $d . "</date>\r\n");
fwrite($handle, "<content>\r\n" . $cont . "</content>\r\n\r\n");
fwrite($handle, "</post>" . "\r\n");
}
}
fwrite($handle, "</all_posts>");
fclose($handle);
The problem is, that because there is something like 10,000, the server does not give responce [I think it because that the xml file become to big or because of the excessively long time of procceng php script]. Only when I try to export posts from category that has only something like 2000 posts it works well.
What is the way to fix it?

On the top of your code set the max_execution_time to unlimited (or for some minutes) ...
ini_set('max_execution_time', 0);
Try even to boost the memory limit used by PHP
ini_set('memory_limit', '100M');

You are probably hitting the time limit for scripts. This can sometimes be changed with set_time_limit.
Otherwise you can limit it to export a few hundred pages at a time. Just change the offset option for get_posts between runs.

Try this.
Where you have:
$args = array('numberposts' => 50000,'cat' => $child_cat->cat_ID);
put
$args = array('numberposts' => 50000,'cat' => $child_cat->cat_ID, 'post_status' => 'publish' );
This will make sure you only have the posts that are published. Is this what you want? or do you want all of them.
If you want all of them, try putting all of the code in a while loop based on the categories and insert the category name into the name of the file therefore having multiple files but then you can still call all of them individually. This is probably your best bet since the file would be too large.
Ethan Brouwer

Related

How to stream creation of a JSON File?

Am trying to create a JSON file from a large dump of a database query, and works when I set a LIMIT to 100000 rows being returned, but when I want all rows to be returned, it just goes to a 502 Error (The page request was canceled because it took too long to complete). Wondering if there is a way that I can streamline the creation of a JSON File in bits using php, or if there is a Library out there that will allow me to build the json file in parts?
Basically am running a .php file here to attempt to get all orders in json format from woocommerce, since the plugin I purchased "CSV Import Suite" does not work when importing orders, it just stays in the queue.
So, I decided to try and export all orders myself, but keep hitting a 502 Error Page and it never creates the .json file either, so am thinking I need a way to stream this somehow. Any help on this would be appreciated...
ini_set('memory_limit', '-1');
ini_set('max_execution_time', '-1');
set_time_limit(0);
error_reporting(E_ALL);
ob_implicit_flush(TRUE);
ob_end_flush();
global $wpdb, $root_dir;
if (!defined('ABSPATH'))
$root_dir = dirname(__FILE__) . '/';
else
$root_dir = ABSPATH;
$download = isset($_GET['download']);
// Allows us to use WP functions in a .php file without 404 headers!
require_once($root_dir . 'wp-config.php');
$wp->init();
$wp->parse_request();
$wp->query_posts();
$wp->register_globals();
if (empty($download))
$wp->send_headers();
// exclude
$exclude_post_statuses = array('trash', 'wc-refunded', 'wc_cancelled');
$start_date = !empty($_GET['start_date']) ? DateTime::createFromFormat('Y-m-d', $_GET['start_date']) : '';
$end_date = !empty($_GET['end_date']) ? DateTime::createFromFormat('Y-m-d', $_GET['end_date']) : '';
$order_db = array(
'columns' => array(
'p' => array('ID', 'post_author', 'post_date', 'post_date_gmt', 'post_content', 'post_title', 'post_excerpt', 'post_status', 'comment_status', 'ping_status', 'post_password', 'post_name', 'to_ping', 'pinged', 'post_modified', 'post_modified_gmt', 'post_content_filtered', 'post_parent', 'guid', 'menu_order', 'post_type', 'post_mime_type', 'comment_count'),
'pm' => array('meta_id', 'post_id', 'meta_key', 'meta_value'),
'oi' => array('order_item_id', 'order_item_name', 'order_item_type', 'order_id'),
'oim' => array('meta_id', 'order_item_id', 'meta_key', 'meta_value')
)
);
$select_data = '';
$total_columns = count($order_db['columns']);
$i = 1;
foreach($order_db['columns'] as $column_key => $columns)
{
$select_data .= implode(', ', array_map(
function ($v, $k) { return $k . '.' . $v . ' AS ' . $k . '_' . $v; },
$columns,
array_fill(0, count($columns), $column_key)
));
if ($i < $total_columns)
$select_data .= ', ';
$i++;
}
// HUGE DATABASE DUMP HERE, needs to be converted to JSON, after getting all columns of all tables...
$orders_query = $wpdb->get_results('
SELECT ' . $select_data . '
FROM ' . $wpdb->posts . ' AS p
INNER JOIN ' . $wpdb->postmeta . ' AS pm ON (pm.post_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_items AS oi ON (oi.order_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_itemmeta AS oim ON (oim.order_item_id = oi.order_item_id)
WHERE p.post_type = "shop_order"' . (!empty($exclude_post_statuses) ? ' AND p.post_status NOT IN ("' . implode('","', $exclude_post_statuses) . '")' : '') . (!empty($start_date) ? ' AND post_date >= "' . $start_date->format('Y-m-d H:i:s') . '"' : '') . (!empty($end_date) ? ' AND post_date <= "' . $end_date->format('Y-m-d H:i:s') . '"' : '') . '
ORDER BY p.ID ASC', ARRAY_A);
$json = array();
if (!empty($orders_query))
{
foreach($orders_query as $order_query)
{
if (!isset($json[$order_query['p_post_type']], $json[$order_query['p_post_type']][$order_query['p_post_name']]))
$json[$order_query['p_post_type']][$order_query['p_post_name']] = array(
'posts' => array(),
'postmeta' => array(),
'woocommerce_order_items' => array(),
'woocommerce_order_itemmeta' => array()
);
if (!empty($order_query['p_ID']))
$json[$order_query['p_post_type']][$order_query['p_post_name']]['posts'][$order_query['p_ID']] = array_filter($order_query, function($k) {
$is_p = strpos($k, 'p_');
return $is_p !== FALSE && empty($is_p);
}, ARRAY_FILTER_USE_KEY);
if (!empty($order_query['pm_meta_id']))
$json[$order_query['p_post_type']][$order_query['p_post_name']]['postmeta'][$order_query['pm_meta_id']] = array_filter($order_query, function($k) {
$is_pm = strpos($k, 'pm_');
return $is_pm !== FALSE && empty($is_pm);
}, ARRAY_FILTER_USE_KEY);
if (!empty($order_query['oi_order_item_id']))
$json[$order_query['p_post_type']][$order_query['p_post_name']]['woocommerce_order_items'][$order_query['oi_order_item_id']] = array_filter($order_query, function($k) {
$is_io = strpos($k, 'oi_');
return $is_io !== FALSE && empty($is_io);
}, ARRAY_FILTER_USE_KEY);
if (!empty($order_query['oim_meta_id']))
$json[$order_query['p_post_type']][$order_query['p_post_name']]['woocommerce_order_itemmeta'][$order_query['oim_meta_id']] = array_filter($order_query, function($k) {
$is_oim = strpos($k, 'oim_');
return $is_oim !== FALSE && empty($is_oim);
}, ARRAY_FILTER_USE_KEY);
}
}
// Downloading or viewing?
if (!empty($download))
{
// Outputs json in a textarea for you to copy and paste into a .json file for import...
if (!empty($json))
{
$filename = uniqid('orders_') . '.json';
$fp = fopen($filename, 'w');
fwrite($fp, json_encode($json));
fclose($fp);
$size = filesize($root_dir . '/' . $filename);
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header("Content-Disposition: attachment; filename=\"" . $filename . "\"");
header('Content-Transfer-Encoding: binary');
header('Connection: Keep-Alive');
header('Expires: 0');
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
header('Pragma: public');
header('Content-Length: ' . $size);
readfile($root_dir . '/' . $filename);
}
}
else
{
// Outputs json in a textarea for you to copy and paste into a .json file for import...
if (!empty($json))
echo '<textarea cols="200" rows="50">', json_encode($json), '</textarea>';
}
The JSON File created could be a well over 500 MB, and possibly even up to 1 Gig of data. So, I believe PHP is running out of memory here, and needs to be processed bit by bit somehow, either in the background, or completely, without hitting the php memory limit. I believe the memory limit is set to 1024 MB, which is pretty high, but not high enough and tbh, for what I'm doing, I don't think we can ever have enough memory to perform the operation as is. Something needs to change in how I process the json and/or download it. And I do not want to create multiple json files, please only 1 JSON file.
I think there might be couple of issues. Firstly I would suggest you do some profiling.
// HUGE DATABASE DUMP HERE, needs to be converted to JSON, after getting all columns of all tables...
echo 'Start Time: '. date("Y-m-d H:i:s");
echo ' Memory Usage: ' . (memory_get_usage()/1048576) . ' MB \n';
$orders_query = $wpdb->get_results('
SELECT ' . $select_data . '
FROM ' . $wpdb->posts . ' AS p
INNER JOIN ' . $wpdb->postmeta . ' AS pm ON (pm.post_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_items AS oi ON (oi.order_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_itemmeta AS oim ON (oim.order_item_id = oi.order_item_id)
WHERE p.post_type = "shop_order"' . (!empty($exclude_post_statuses) ? ' AND p.post_status NOT IN ("' . implode('","', $exclude_post_statuses) . '")' : '') . (!empty($start_date) ? ' AND post_date >= "' . $start_date->format('Y-m-d H:i:s') . '"' : '') . (!empty($end_date) ? ' AND post_date <= "' . $end_date->format('Y-m-d H:i:s') . '"' : '') . '
ORDER BY p.ID ASC', ARRAY_A);
echo 'End Time: '. date("Y-m-d H:i:s");
echo ' Memory Usage: ' . (memory_get_usage()/1048576) . ' MB \n';
die('Finished');
$json = array();
The above will help you to know how much memory is in use, upto this point. If it fails before it prints 'Finished', we know it is not a json issue. If the script works fine then we can first create a csv file rather json. Since you are running a select query, (at this point) it does not have to be nested json file which you require. A flat structure can be achieved by just creating a CSV file.
$csvFile = uniqid('orders') . '.csv';
$fp = fopen($csvFile, 'w');
if (!empty($orders_query))
{
$firstRow = true;
foreach($orders_query as $order_query)
{
if(true === $firstRow) {
$keys = array_keys($order_query);
fputcsv($fp, $order_query);
$firstRow = false;
}
fputcsv($fp, $order_query);
}
}
fclose($fp);
If the above works fine you at-least have a csv file to work with.
At this point I am not sure how complex is your data structure nested. For e.g how many distinct values exist for 'p_post_type' and 'p_post_name' you are having. You might require to parse the csv file and create multiple json file for each ['p_post_type']['p_post_name']['posts'], ['p_post_type']['p_post_name']['posts'], ['p_post_type']['p_post_name']['woocommerce_order_items'] and ['p_post_type']['p_post_name']['woocommerce_order_itemmeta'].
If the number of files are few you can write a script to merge them automatically or do them manually. If you have too many nested items, the number of json files that might be created might be a lot and might be hard to merge them and might not be a feasible option.
If the number of json files are lot, I would like to know what is the purpose of having such a huge single json file. If export is an issue import might be an issue too, especially ingesting such a huge json file in memory. I believe if the purpose of creating the json file is to import it in some form, at some stage in future, I think you might have to look at the option of just having a csv file instead, which you use to filter out whatever is required at that point of time.
I hope this helps.
FURTHER UPDATE
It looks to me that $wpdb->get_results is using mysqli_query/mysql_query (depending on your configuration) to fetch the results. See wordpress query docs. It is not memory efficient way to fetch the data this way. I believe you might be failing at this point ($wpdb->get_results) itself. I would suggest you to run the query without using $wpdb. There is a concept of unbuffered query whenever large data retrieval is required, which has very low impact on the memory. Further information can be found here mysql unbuffering.
Even if you get past this point, you will still run into memory issues, due to the way how you are storing everything in $json variable which is eating up lot of your memory. $json is an array and it would interesting to know how PHP array works. PHP arrays are dynamic and they do not allocate extra memory every time a new element is added, since that would be extremely slow. It instead, increases the array size to the power of two, which means whenever the limit is exhausted it increases the array limit to twice its current limit and in the process tries to increase the memory to twice the limit. This has been less of an issue with PHP 7, since they have made some major changes to the php core. So if you have 2GB data that might be required to be stored in $json, the script might easily allocate anywhere between 3-4 GB memory, depending upon when it hits the limit. Further details can be found here php array and How does PHP memory actually work
If you consider the overhead of the $orders_query which is an array combined with overhead of $json it is quite substantial due to the way PHP array works.
You can also try to create another database B. So while you are reading from database A, you simultaneously start writing data to database B. In the end you have database B with all the data in it with the power of MySQL. You could also push the same data into a MongoDB which would be lightning fast and might help you with the json nesting you are after. MongoDB is meant to work really efficiently with large datasets.
JSON STREAMING SOLUTION
Firstly, I would like to say that streaming is sequential/linear process. As such, it is does not have memory of what was added before this point of time or what will added after this point of time. It works in small chunks and that is the reason it is so memory efficient. So when you actually write or read, the responsibility lies with the script, that it maintains a specific order, which is kind of saying you are writing/reading your own json, as streaming only understands text and has no clue about what json is and won't bother itself in writing/reading a correct one.
I have found a library on github https://github.com/skolodyazhnyy/json-stream which would help in you achieving what you want. I have experimented with the code and I can see it will work for you with some tweaks in your code.
I am going to write some pseudo-code for you.
//order is important in this query as streaming would require to maintain a proper order.
$query1 = select distinct p_post_type from ...YOUR QUERY... order by p_post_type;
$result1 = based on $query1;
$filename = 'data.json';
$fh = fopen($filename, "w");
$writer = new Writer($fh);
$writer->enter(Writer::TYPE_OBJECT);
foreach($result1 as $fields1) {
$posttype = $fields1['p_post_type'];
$writer->enter($posttype, Writer::TYPE_ARRAY);
$query2 = select distinct p_post_name from ...YOUR QUERY... YOUR WHERE ... and p_post_type= $posttype order by p_post_type,p_post_name;
$result2 = based on $query2;
foreach($result2 as $fields2) {
$postname = $fields1['p_post_name'];
$writer->enter($postname, Writer::TYPE_ARRAY);
$query3 = select ..YOUR COLUMNS.. from ...YOUR QUERY... YOUR WHERE ... and p_post_type= $posttype and p_post_name=$postname where p_ID is not null order by p_ID;
$result3 = based on $query3;
foreach($result2 as $field3) {
$writer->enter('posts', Writer::TYPE_ARRAY);
// write an array item
$writer->write(null, $field3);
}
$writer->leave();
$query4 = select ..YOUR COLUMNS.. from ...YOUR QUERY... YOUR WHERE ... and p_post_type= $posttype and p_post_name=$postname where pm_meta_id is not null order by pm_meta_id;
$result4 = based on $query4;
foreach($result4 as $field4) {
$writer->enter('postmeta', Writer::TYPE_ARRAY);
// write an array item
$writer->write(null, $field4);
}
$writer->leave();
$query5 = select ..YOUR COLUMNS.. from ...YOUR QUERY... YOUR WHERE ... and p_post_type= $posttype and p_post_name=$postname where oi_order_item_id is not null order by oi_order_item_id;
$result5 = based on $query5;
foreach($result5 as $field5) {
$writer->enter('woocommerce_order_items', Writer::TYPE_ARRAY);
// write an array item
$writer->write(null, $field5);
}
$writer->leave();
$query6 = select ..YOUR COLUMNS.. from ...YOUR QUERY... YOUR WHERE ... and p_post_type= $posttype and p_post_name=$postname where oim_meta_id is not null order by oim_meta_id;
$result6 = based on $query6;
foreach($result6 as $field6) {
$writer->enter('woocommerce_order_itemmeta', Writer::TYPE_ARRAY);
// write an array item
$writer->write(null, $field5);
}
$writer->leave();
}
$writer->leave();
fclose($fh);
You might have to start limiting your queries to 10 something until you get it right. Since the code above might not just work as it is. You should be able to read the code in similar fashion as the same library has got a Reader class to help. I have tested both reader and writer and they seem to work fine.
Creating the file
The problem with your code is you are trying to fit whole dataset into the memory, which eventually will fail as soon as your database gets large enough. To overcome this you have to fetch the data in batches.
We are going to generate the query multiple times so I extracted your query into a function. I skipped passing required parameters though (or making them global if you will) for brevity so you have to get this to work by yourself.
function generate_query($select, $limit = null, $offset = null) {
$query = 'SELECT ' . $select . '
FROM ' . $wpdb->posts . ' AS p
INNER JOIN ' . $wpdb->postmeta . ' AS pm ON (pm.post_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_items AS oi ON (oi.order_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_itemmeta AS oim ON (oim.order_item_id = oi.order_item_id)
WHERE p.post_type = "shop_order"' . (!empty($exclude_post_statuses) ? ' AND p.post_status NOT IN ("' . implode('","', $exclude_post_statuses) . '")' : '') . (!empty($start_date) ? ' AND post_date >= "' . $start_date->format('Y-m-d H:i:s') . '"' : '') . (!empty($end_date) ? ' AND post_date <= "' . $end_date->format('Y-m-d H:i:s') . '"' : '') . '
ORDER BY p.ID ASC';
if ($limit && $offset) {
$query .= ' LIMIT ' . $limit . ' OFFSET ' . $offset;
}
return $query;
}
Now we will get results from the db in batches, we define the batch count that is the number of records per iteration that we will load into the memory. You can later on play with this value to find one that will be fast enough and won't make PHP crash. Keep in mind we want to reduce the number of database queries as much as possible:
define('BATCH_COUNT', 500);
Before we create the loop we need to know how many iterations (database calls) we will make, so we need the total order count. Having this and the batch count, we can calculate this value easily:
$orders_count = $wpdb->get_col(generate_query('COUNT(*)'));
$iteration_count = ceil($orders_count / BATCH_COUNT);
As a result we would like to have a huge JSON string inside the result file. Since with each iteration we will have a separate JSON containing an array of objects, we will simply strip the [ and ] from each side of the JSON string and put that string into the file.
Final code:
define('FILE', 'dump.json');
file_put_contents(FILE, '[');
for ($i = 0; $i < $iteration_count; $i++) {
$offset = $i * BATCH_COUNT;
$result = $wpdb->get_results(
generate_query($select_data, BATCH_COUNT, $offset),
ARRAY_A
);
// do additional work here, add missing arrays etc.
// ...
// I assume here the $result is a valid array ready for
// creating JSON from it
// we append the result file with partial JSON
file_put_contents(FILE, trim(json_encode($result), '[]'), FILE_APPEND);
}
file_put_contents(FILE, ']', FILE_APPEND);
Congratulations, you have just created your first huge JSON dump ;) You should run this script in the command line so it can get as long as it needs to, there's no need to modify the memory limit from now on, because we are never going to hit the limit hopefully.
Sending the file
Streaming large files with PHP is easy and has already been answered on SO many times. However I personally don't recommend you doing anything time consuming in PHP, because it sucks as a long running process, either in the command line or as a file server.
I assume you are using Apache. You should consider using SendFile and let Apache do the hard work for you. This method is far more efficient when dealing with huge files. This method is very easy, all you need to do is pass the path to the file in the header:
header('X-Sendfile: ' . $path_to_the_file);
Should you use Nginx there's XSendFile support as well.
This method does not use a lot of memory, does not block the PHP process. The file does not need to be accessible in the webroot too. I use XSendFile all the time to serve 4K videos to authenticated users.
First, you should ask yourself a question: Do I need to write database dump myself?
If not then you can simply use some service that will do the work for you. Mysqldump-php should be able to do the job.
Then you can simply:
include_once(dirname(__FILE__) . '/mysqldump-php-2.0.0/src/Ifsnop/Mysqldump/Mysqldump.php');
$dump = new Ifsnop\Mysqldump\Mysqldump('mysql:host=localhost;dbname=testdb', 'username', 'password');
$dump->start('storage/work/dump.sql');
This should create .sql file. However, you wanted json file. That shouldn't be a problem though. This tool will do the rest of the job: http://www.csvjson.com/sql2json
You can also find the source code of sql2json on github: https://github.com/martindrapeau/csvjson-app
I believe you may be looking for Generators
http://php.net/manual/en/language.generators.overview.php
https://scotch.io/tutorials/understanding-php-generators
Instead of creating that huge $json array, you iterate over each $order_query and perform operations on each iteration, negating the need to store it in memory.
Your problem is that you get a large result set from your query, which is heavy since you have 3 joins.
You could define a limit and use offset to get the data in chunks and then output your json in parts. The main problem is to somehow get the json data in memory and then access it to output in parts.
For the latter cache or a nosql database could be used. My solution will use cache and in particular memcache:
class Cache {
private $cache;
public function __construct($cache)
{
$this->cache = $cache;
}
public function addPostName($postName)
{
$this->addKeyToJsonObject('postNames', $postName);
}
public function addKeyToJsonObject($rootName, $key)
{
$childNames = $this->cache->get($rootName);
if($childNames === false) {
$this->cache->set($rootName, [$key]);
}
else {
$childNamesList = $childNames;
// not found
if(array_search($key, $childNamesList) === false) {
$childNamesList[] = $key;
$this->cache->set($rootName, $childNamesList);
}
}
}
public function getPostNames()
{
return $this->cache->get('postNames');
}
public function set($key, $value) {
$this->cache->add($key, $value);
}
public function addPostIdsByNameAndType($postName, $type, $pid)
{
$this->addKeyToJsonObject($postName . '-' . $type, $pid);
}
public function getPostIdsByNameAndType($postName, $type)
{
return $this->cache->get($postName . '-' . $type);
}
public function addPostValueByNameTypeAndId($postName, $type, $pid, $value)
{
$this->cache->set($postName . '-' . $type . '-' . $pid, $value);
}
public function getPostValueByNameTypeAndId($postName, $type, $pid)
{
return $this->cache->get($postName . '-' . $type . '-' . $pid);
}
}
and then:
$memcache = new Memcache();
$memcache->connect('127.0.0.1', 11211) or die ("Could not connect");
$memcache->flush();
$cache = new Cache($memcache);
header('Content-disposition: attachment; filename=file.json');
header('Content-type: application/json');
echo '{"shop_order":{';
function getResultSet($wpdb, $offset = 1) {
return $wpdb->get_results('
SELECT ' . $select_data . '
FROM ' . $wpdb->posts . ' AS p
INNER JOIN ' . $wpdb->postmeta . ' AS pm ON (pm.post_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_items AS oi ON (oi.order_id = p.ID)
LEFT JOIN ' . $wpdb->prefix . 'woocommerce_order_itemmeta AS oim ON (oim.order_item_id = oi.order_item_id)
WHERE p.post_type = "shop_order"' . (!empty($exclude_post_statuses) ? ' AND p.post_status NOT IN ("' . implode('","', $exclude_post_statuses) . '")' : '') . (!empty($start_date) ? ' AND post_date >= "' . $start_date->format('Y-m-d H:i:s') . '"' : '') . (!empty($end_date) ? ' AND post_date <= "' . $end_date->format('Y-m-d H:i:s') . '"' : '') . '
ORDER BY p.ID ASC LIMIT 1000 OFFSET ' . $offset, ARRAY_A);
}
$offset = 1;
$orders_query = getResultSet($wpdb, 1);
while(!empty($orders_query)) {
cacheRowData($cache, $orders_query);
$offset = $offset + 1000;
$orders_query = getResultSet($wpdb, $offset);
}
outputRowData($cache);
function cacheRowData($cache, $orders_query)
{
foreach($orders_query as $order_query) {
if(empty($order_query)) { continue; }
$cache->addPostName($order_query['p_post_name']);
// posts
if (!empty($order_query['p_ID'])) {
$cache->addPostIdsByNameAndType($order_query['p_post_name'],'posts', $order_query['p_ID']);
$value = array_filter($order_query, function($k) {
$is_p = strpos($k, 'p_');
return $is_p !== FALSE && empty($is_p);
}, ARRAY_FILTER_USE_KEY);
$cache->addPostValueByNameTypeAndId($order_query['p_post_name'],'posts', $order_query['p_ID'], $value);
}
if (!empty($order_query['pm_meta_id'])) {
$cache->addPostIdsByNameAndType($order_query['p_post_name'],'postmeta', $order_query['pm_meta_id']);
$value = array_filter($order_query, function($k) {
$is_pm = strpos($k, 'pm_');
return $is_pm !== FALSE && empty($is_pm);
}, ARRAY_FILTER_USE_KEY);
$cache->addPostValueByNameTypeAndId($order_query['p_post_name'],'postmeta', $order_query['pm_meta_id'], $value);
}
// here do the same for "woocommerce_order_items" and "woocommerce_order_itemmeta"
}
}
function outputRowData($cache)
{
$cachedPostNames = $cache->getPostNames();
$firstRow = true;
foreach($cachedPostNames as $postName) {
if(empty($postName)) { continue; }
if($firstRow === false) {
echo ',';
}
$firstRow = false;
echo '"' . $postName . '":{';
$postIds = $cache->getPostIdsByNameAndType($postName, 'posts');
if(!$postIds) {
$postIds = [];
}
// generate posts
$postValues = [];
foreach ($postIds as $postId) {
$postValues[$postId] = $cache->getPostValueByNameTypeAndId($postName, 'posts', $postId);
}
$postMetaIds = $cache->getPostIdsByNameAndType($postName, 'postmeta');
if(!$postMetaIds) {
$postMetaIds = [];
}
$postMetaValues = [];
foreach ($postMetaIds as $postMetaId) {
$postMetaValues[$postMetaId] = $cache->getPostValueByNameTypeAndId($postName, 'postmeta', $postMetaId);
}
// here do the same for "woocommerce_order_items" and "woocommerce_order_itemmeta"
echo '"posts":' . json_encode($postValues) . ',';
echo '"postmeta":' . json_encode($postMetaValues);
echo '}';
ob_flush();
flush(); // flush the output to start the download
}
}
echo '}}';
So there are lot of things that you need to get working to this right. I will all the point that I have in mind.
Termination by WebServer
If you use Apache or Nginx/PHP-FPM, both by default have a timeout for the url that is hit. So even though you have used
ini_set('memory_limit', '-1');
ini_set('max_execution_time', '-1');
set_time_limit(0);
To let the script run for long, but Apache, Nginx, PHP-FPM all have a timeout which won't allow your script to work. So you need fix these to get it working. You never mentioned which server you used. But a NGINX+PHP-FPM will result in 502 for sure with the default config.
Memory Usage
Even though you have used
ini_set('memory_limit', '-1');
If your memory needs rises high, PHP may start using paging and your code could become slow.
PHP CLI or PHP Web?
Not sure what is the frequency of execution here, but if it is low you can consider your data dumping script to be run through a PHP-CLI instead of HTTP. This would mean that you would run a PHP script directly through terminal to dump the JSON into a file and later use a URL to download the file directly
Using X-Sendfile or X-Accel-Redirect
If you are using apache you can send a header
header('X-Sendfile: /data/generated.json');
In case of Nginx you can send a
header('X-Accel-Redirect: /data/generated.json');
This you would do only in case you have decided to run the script as web and not as a CLI. When the generation of the JSON has finished, you don't want your script to read the file and server. You just want the webserver to care of it.
Unbuffered query instead of WPDB Query
https://core.trac.wordpress.org/browser/tags/4.9/src/wp-includes/wp-db.php#L2480
By default WPDB query fetches all the data into memory. But you can query the DB yourself using unbuffered query, this will not flood the memory
Example #1 Unbuffered query example: mysqli
<?php
$mysqli = new mysqli("localhost", "my_user", "my_password", "world");
$uresult = $mysqli->query("SELECT Name FROM City", MYSQLI_USE_RESULT);
if ($uresult) {
while ($row = $uresult->fetch_assoc()) {
echo $row['Name'] . PHP_EOL;
}
}
$uresult->close();
?>
Example #2 Unbuffered query example: pdo_mysql
<?php
$pdo = new PDO("mysql:host=localhost;dbname=world", 'my_user', 'my_pass');
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
$uresult = $pdo->query("SELECT Name FROM City");
if ($uresult) {
while ($row = $uresult->fetch(PDO::FETCH_ASSOC)) {
echo $row['Name'] . PHP_EOL;
}
}
?>
Example #3 Unbuffered query example: mysql
<?php
$conn = mysql_connect("localhost", "my_user", "my_pass");
$db = mysql_select_db("world");
$uresult = mysql_unbuffered_query("SELECT Name FROM City");
if ($uresult) {
while ($row = mysql_fetch_assoc($uresult)) {
echo $row['Name'] . PHP_EOL;
}
}
?>
https://secure.php.net/manual/en/mysqlinfo.concepts.buffering.php
PS: There might be few more points that I am missing right now in my head, would update this soon

Echo file content based on separator

I have a txt file with lines in this format
this is a name|this is a type|this is a description
this is a name|this is a type|this is a description
this is a name|this is a type|this is a description
I need to access those lines and echo them like this:
<li type="this is a type" description="this is a description">this is a name</li>
I have no idea how to approach this.
Thanks in advance
Normally I wouldn't write code for you without you having provided an example of what you've tried, but in this case it's pretty basic.
Use PHP's file function to read a file into an array (line by line), then use explode to break that line up:
<?php
$contents = file('yourfile.txt');
foreach($contents as $eachline) {
list($name, $type, $description) = explode("|", $eachline);
echo '<li type="' . $type . '" description="' . $description . '">' . $name . '</li>';
}
?>
PHP manual: http://us2.php.net/manual/en/function.file.php
The first step is to read every line of the file.
How to read a file line by line in php
After that explode the string by the pipe symbol $out = explode("|", $string);
after that you have an array and you can access the values with $out[0]; for example.
This is easypeasy:
$parts = explode("|", $line);
$out = "<li type='$parts[1]' description='$parts[2]'>$parts[0]</li>"

How do I extract from WordPress database to MS Excel?

I am trying to add data into excel file which is extracted from wordpress database, Actually I am trying to export data (tags) from database into excel file. And I write down a code, but when I click on generate button. This generates empty file.
Please guys check what I am doing wrong.
Codes are below:
if (check_admin_referer('tag-export'))
{
$blogname = str_replace(" ", "", get_option('blogname'));
$date = date("m-d-Y");
$xls_file_name = $blogname."-exported-tags-".$date;
$tags = get_terms( 'post_tag' , 'hide_empty=0' );
$count = count($tags);
if ( $count > 0 )
{
echo 'name' . "\t" . 'slug' . "\n";
foreach ( $tags as $tag )
{
echo $tag->name . "\t" . $tag->slug . "\n";
}
}
ob_clean();
echo $xls_file;
header( "Content-Type: application/vnd.ms-excel" );
header( "Content-disposition: attachment; filename=$xls_file_name.xls" );
exit();
}
The above codes are not writing data into excel file. please check and let me know.
Just based on your existing code:
if (check_admin_referer('tag-export'))
{
$blogname = str_replace(" ", "", get_option('blogname'));
$date = date("m-d-Y");
$xls_file_name = $blogname."-exported-tags-".$date;
$tags = get_terms( 'post_tag' , 'hide_empty=0' );
$count = count($tags);
$xls_file = '';
if ( $count > 0 )
{
$xls_file .= 'name' . "\t" . 'slug' . "\n";
foreach ( $tags as $tag )
{
$xls_file .= $tag->name . "\t" . $tag->slug . "\n";
}
}
ob_clean();
header( "Content-Type: application/vnd.ms-excel" );
header( "Content-disposition: attachment; filename=$xls_file_name.xls" );
echo $xls_file;
exit();
}
A more general suggestion, not a solution for your coding problem: create an HTML table file from the code and then open it in Excel for conversion. Doing it so you'll have a better understand on what's going on with your code: you can add var_dumps or simply debug it like a normal web page.
Having an html table is also useful because excel works quite well in converting it to XLS files.
After your HTML file works well, then you can apply necessary formatting/header to the code in order to create the xls file from scratch.

php file put contents reverse

Ok sorry if its a stupid question im a begginer.
Im making a small shoutbox just for practise.
It inserts the shout infos in a txt file.
My problem is that, it lists the text from top to bottom, and i would like to do this reversed.
if(isset($_POST['submit'])) {
$text = $_POST['text'];
if(!empty($text)) {
$text = $_POST['text'];
$name = $_POST['name'];
$time = date("H:i");
$content =
"<div class='text'><em>" . $time . "</em>
<span class='c11'><b>" . "<a href='userinfo_php_willbe_here.php' target='_blank'>" . htmlspecialchars($name) . "</a>" . ":</span></b>
" . htmlspecialchars($text) . "
</div>\n";
file_put_contents($file, $content, FILE_APPEND | LOCK_EX);
}
}
here is my code.
i was googleing around with not much luck maybe i wasnt looking hard enough.
could please someone give me a hint?
thank you
No way to do so with one function call. You need to read the content from your target file, prepend the data in php and rewrite the whole file (see file_get_contents).
$fileContent = file_get_contents($file);
$fileContent = $content . $fileContent;
file_put_contents($file, $fileContent, LOCK_EX);
You can also use the array_reverse like so:
// Data in file separated by new-line
$data = explode("\n",file_get_contents("filename.txt"));
foreach(array_reverse($data) as $value) {
echo $value."\n";
}
You can only prepend to a file by means of reading it and writing afterwards.
file_put_contents($file, $content . file_get_contents($file), LOCK_EX);

PHP dump $_REQUEST to file

I want to dump request variables to a file for debugging. How's this possible?
<?php
$req_dump = print_r($_REQUEST, TRUE);
$fp = fopen('request.log', 'a');
fwrite($fp, $req_dump);
fclose($fp);
Untested but should do the job, just change request.log to the file you want to write to.
I think nowadays this method is easier and faster:
$req_dump = print_r($_REQUEST, true);
$fp = file_put_contents('request.log', $req_dump, FILE_APPEND);
Use serialize() function for dumping. Dump $_SERVER, $_COOKIE, $_POST and $_GET separately (may go to the same file). If you're planning on debugging with the data it helps to know if the data was part of a POST request or a GET request.
Dumping everything is good for debugging in development, but not so in production. If your application does not have many users, it can work in production too. If you anticipate many users, consider dumping just the $_POST data, or limit server variables to those starting with HTTP_.
/* may be late but he can help others.
it's not my code, I get it from :
https://gist.github.com/magnetikonline/650e30e485c0f91f2f40
*/
class DumpHTTPRequestToFile {
public function execute($targetFile) {
$data = sprintf(
"%s %s %s\n\nHTTP headers:\n",
$_SERVER['REQUEST_METHOD'],
$_SERVER['REQUEST_URI'],
$_SERVER['SERVER_PROTOCOL']
);
foreach ($this->getHeaderList() as $name => $value) {
$data .= $name . ': ' . $value . "\n";
}
$data .= "\nRequest body:\n";
file_put_contents(
$targetFile,
$data . file_get_contents('php://input') . "\n"
);
echo("Done!\n\n");
}
private function getHeaderList() {
$headerList = [];
foreach ($_SERVER as $name => $value) {
if (preg_match('/^HTTP_/',$name)) {
// convert HTTP_HEADER_NAME to Header-Name
$name = strtr(substr($name,5),'_',' ');
$name = ucwords(strtolower($name));
$name = strtr($name,' ','-');
// add to list
$headerList[$name] = $value;
}
}
return $headerList;
}
}
(new DumpHTTPRequestToFile)->execute('./dumprequest.txt');
// add this line at the end to create a file for each request with timestamp
$date = new DateTime();
rename("dumprequest.txt", "dumprequest" . $date->format('Y-m-d H:i:sP') . ".txt");
<?php //log
$razdelitel = '--------------------------------------------'.PHP_EOL . date("Y-m-d H:i:s") .PHP_EOL.PHP_EOL;
$data_REQUEST = '$_REQUEST: ' . print_r($_REQUEST, true).PHP_EOL;
$data_POST = '$_POST: ' . print_r($_POST, true).PHP_EOL;
$data_GET = '$_GET: ' . print_r($_GET, true).PHP_EOL;
$data_all = $razdelitel . $data_REQUEST . $data_POST . $data_GET;
$name_txt = __DIR__ . '/log_' . date('m.Y') . '.txt'; //log_12.2021.txt
$chmod = '0244';
chmod($name_txt, $chmod);
file_put_contents($name_txt, $data_all, FILE_APPEND);
//var_dump($name_txt, $chmod); ?>

Categories