I'm trying to export some data into a csv file, I've got the data going into the file, this is my code:
*Please excuse me if the code is bad, I've never done anything like this and this was the only way I could get it working.
/* CSV Export - Create Row */
$csv_row_content .= $userdata->id.',';
$csv_row_content .= $userdata->firstname.' '.$userdata->lastname;
$csv_row_content .= ','.$split_date[2].'-'.$split_date[1].'-'.$split_date[0].',';
$csv_row_content .= $sale->name;
if($sale->optionlabel):
$csv_row_content .= ' ('.$sale->optionlabel.')';
endif;
$csv_row_content .= ',';
$csv_row_content .= $sale->status.',';
$csv_row_content .= number_format($sale->unitprice, 2, '.', '');
$csv_row_content .= "\r\n";
$data_array[] = $csv_row_content;
$csv_file = fopen('../wp-content/plugins/data-export/export_doc.csv', 'w');
foreach ($data_array as $single_line)
{
fputcsv($csv_file,split(',',$single_line));
}
fclose($csv_file);
/* Clear Array */
unset($data_array);
$data_array = array();
It's working except I'm having trouble with the quotations marks on certain items
303,"User Name",12-02-2013,"College Sweater (Black)",,"20.00
207","User Name",30-01-2013,"College Sweater (Black)",,"20.00
"
So I'm not sure what the go is with the first and last items, the one quotation mark show up sometimes and not in others.
Notice the odd quotation mark on row id 207 & on the last value for both row.
Also there's a new line begin made on the third row with just a single quote.
Also on some other items the function is splitting the name of the item into two items. eg:
207","User Name",30-01-2013,"College ","Sweater (Black)",,"22.73
So obviously I'm off base here somewhere, if anyone could help me with this, I'm really keen on learning the correct way this kind of thing should be done, checked the php.net docs quite a bit, but a lot of the time I find that resource incredibly overwhelming and this is one such occasion.
If anyone can point me in the right direction on this I'd really appreciate it.
I'd prefer to understand this than just have a copy and paste solution.
Thanks,
Frank
You are manually creating a CSV string and splitting it before using fputcsv to put it back together. Try this instead:
$row = array(
$userdata->id,
$userdata->firstname . ' ' . $userdata->lastname,
$split_date[2] . '-' . $split_date[1] . '-' . $split_date[0],
$sale->name . ($sale->optionlabel ? $sale->optionlabel : ''),
$sale->status,
number_format($sale->unitprice, 2, '.', '')
);
$data[] = $row;
$csv_file = fopen('../wp-content/plugins/data-export/export_doc.csv', 'w');
foreach ($data as $line) {
fputcsv($csv_file, $line);
}
fclose($csv_file);
This creates an array containing all the fields which can be passed to fputcsv which takes care of enclosing and delimiting fields as well as line endings.
Related
We have 2 millions encrypted records in a table to export. We are using Drupal 8 but we cannot export data it through custom views or using webform export due to encryption of sensitive data. So we have to write a custom function to export data in CSV or Excel. But it throw "Allowed Memory Exhausted" error due large amount of data whenever we tried to export it.
It seems the best option is loading data in smaller chunks and appending to the same sheet. How can we achieve this approach? Or any idea to do it in PHP or Drupal 8.
Exporting to CSV is by far the simpler operation. There are a couple of ways to do this.
1. You could always use mysqldump with text delimiters to avoid PHP memory constraints:
mysqldump -u YOUR_USERNAME -p -t DATABASE_NAME TABLE_NAME
--fields-terminated-by=","
--fields-optionally-enclosed-by="\""
--fields-escaped-by="\""
--lines-terminated-by="\r\n"
--tab /PATH/TO/TARGET_DIR
Line breaks added for readability. By default, mysqldump also generates a .sql file with DROP/CREATE TABLE statements. The -t option skips that.
2. You can make a MySQL query and define INTO OUTFILE with the appropriate delimiters to format your data as CSV and save it into a file:
SELECT * FROM `db_name`.`table_name`
INTO OUTFILE 'path_to_folder/table_dump.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n';"
If you run this on the command line, you can probably get away with a single call without the need to batch it (subject to your server specs and MySQL memory config).
If you do need to batch, then add something like LIMIT 0, 100000 where 100000 is whatever is a good result set size, and adapt your filename to match: table_dump_100000.csv etc. Merging the resulting CSV dumps into one file should be a simple operation.
3. If you do want to run this over PHP, then you most likely have to batch it. Basic steps:
A loop with for($i = 0; $i <= $max_rows; $i += $incr) where $incr is the batch size. In the loop:
Make MySQL query with variables used in the LIMIT clause; as in LIMIT $i, $incr.
Write the rows with fputcsv into your target file. Define your handle before the loop.
The above is more of a homework assignment than an attempt to provide ready code. Get started and ask again (with code shown). Whatever you do, make sure the data variables used for each batch iteration are reused or cleared to prevent massive memory usage buildup.
You can up your script's memory limit with ini_set('memory_limit', '2048M'); (or whatever your server can handle). If you run into max execution time, set_time_limit(600) (10 min; or whatever seems enough) at the start of your script.
Never tried with 2 million records.
But it works with a few hundred thousand, using drush and a script similar to:
<?php
// php -d memory_limit=-1 vendor/bin/drush php:script export_nodes.php
// options
$type = 'the_type';
$csv_file_name = '/path/to/csv_file.csv';
$delimiter = '"';
$separator = ',';
// fields
$fields = [
'nid',
'title',
'field_one',
'field_two',
'field_three',
];
// header
$header = '';
foreach ($fields as $field) {
$header = $header . $delimiter . $field . $delimiter . $separator;
}
$header = $header . PHP_EOL;
file_put_contents ($csv_file_name, $header, FILE_APPEND);
unset ($header);
// get nodes
$nodes = \Drupal::entityTypeManager()
->getStorage('node')
->loadByProperties([
'type' => $type,
]);
// loop nodes
foreach ($nodes as $node) {
$line = '';
// loop fields
foreach ($fields as $field) {
$field_value_array = $node->get($field)->getValue();
if (empty ($field_value_array[0]['value'])) {
$field_value = '';
}
else {
$field_value = $field_value_array[0]['value'];
}
$line = $line . $delimiter . $field_value . $delimiter . $separator;
}
unset ($field_value_array);
unset ($field_value);
// break line
$line = $line . PHP_EOL;
// write line
file_put_contents ($csv_file_name, $line, FILE_APPEND);
unset ($line);
}
unset ($nodes);
So I have this code:
$datas .= '"' . implode('", "', $body["id"].",".$body["prefecture_id"].",".$body["industry_id"].",".$body["offset"].",".$body["name"].",".$email . ",".$body["tel"].",".$body["fax"].",".$body["address"].",".$body["address_coordinate"].",".$body["url"].",".$body["image_url"].",".$body["flexible_1"].",".$body["flexible_2"].",".$body["flexible_3"].",".$body["flexible_4"].",".$body["flexible_5"].",".$body["flexible_6"].",".$body["flexible_7"].",".$body["flexible_8"].",".$body["flexible_9"].",".$body["flexible_10"].",".$body["sequence"].",".$body["del_flg"].",".$body["create_date"].",".$body["create_user"].",".$body["update_date"].",".$body["update_user"]).'"'."\r\n";
I want it to look like this:
"KN01001","01","4","500","Starbuck's","admin#starbucks.com","09-232-821","09-232-822","Upper G/F SM, Juan Luna Extension"
Problem is address has comma, so this would separate it.
I really could not construct it right. Please help
this code is from a CSV convert function. In the CSV file each column is separated by a comma.
But my problem is that I have several columns that contains also comma. I would like that those commas in the columns remain as it is.
Instead of rolling your own CSV encoding function it might be better to use the built in fputcsv().
$out = fopen("php://memory", "w");
foreach($something as $body) {
fputcsv($out, $body);
}
rewind($out);
$data = stream_get_contents($out);
You'll need to adapt the code to order the columns if you want but that's the basic method (Don't forget to close $out either).
To save holding the entire CSV in memory (if this was a file download/export mechanism) you could open php://output instead and write/flush directly to the web-server/browser.
To answer your question directly implode() takes an array as a second argument but you've given it a string so change it to this (and add all the fields):
$datas .= '"' . implode('", "', [$body["id"], $body["prefecture_id"], $body["industry_id"], $body["offset"], ...]) . "\"\r\n";
can you use fgetcsv() instead of implode? This would handle the
$fileHandle = fopen("filename.csv", "r");
while (($row = fgetcsv($fileHandle, 0, ",")) !== FALSE) {
var_dump($row);
//$row is an array of all the columns in the csv regardless of commas in the dat
}
Otherwise if you must use implode, is the data that has a comma enclosed in speech marks: "some date, with a comma". If so you could use a regex to ignore the commas within the speechmarks. I can't see your data so its hard to say.
I have a serious problem with text wrapping via PHPExcel. I have a column, which contains texts in new lines. It does the linebreaks in LibreOffice. In MS Office it is displayed in a single line. In both viewers, it only does the wrap, when I double click into a cell and then click out the cell. I have the following code:
foreach($view->results as $row){
//...
foreach($unserialized as $task){
$value = $field_info['settings']['allowed_values'][$doc['document']];
$current_tasks .= $value . "\n";
}
$active_sheet->setCellValue($letter.$i, $current_tasks);
//...
//end of main foreach loop
$active_sheet->getStyle('L' . $i)->getAlignment()->setWrapText(true);
$i++;
}
//tried this too outside the foreach:
$active_sheet->getStyle('L2:L' . $i)->getAlignment()->setWrapText(true);
They don't seem to be working. Am I doing something wrong? I googled it up and neither of the solutions worked for me.
I only needed to set the height of the rows.
$numtasks = 20;
foreach($unserialized as $task){
$value = $field_info['settings']['allowed_values'][$doc['document']];
$current_tasks .= $value . "\n";
$active_sheet->getRowDimension($i)->setRowHeight($numtasks);
$numtasks += 20; //20 is for 1 cells height in pixels
}
I'm working on a project for a client - a wordpress plugin that creates and maintains a database of organization members. I'll note that this plugin creates a new table within the wordpress database (instead of dealing with the data as custom_post_type meta data). I've made a lot of modifications to much of the plugin, but I'm having an issue with a feature (that I've left unchanged).
One half of this feature does a csv import and insert, and that works great. The other half of this sequence is a feature to download the contents of this table as a csv. This part works fine on my local system, but fails when running from the server. I've poured over each portion of this script and everything seems to make sense. I'm, frankly, at a loss as to why it's failing.
The php file that contains the logic is simply linked to. The file:
<?php
// initiate wordpress
include('../../../wp-blog-header.php');
// phpinfo();
function fputcsv4($fh, $arr) {
$csv = "";
while (list($key, $val) = each($arr)) {
$val = str_replace('"', '""', $val);
$csv .= '"'.$val.'",';
}
$csv = substr($csv, 0, -1);
$csv .= "\n";
if (!#fwrite($fh, $csv))
return FALSE;
}
//get member info and column data
$table_name = $wpdb->prefix . "member_db";
$year = date ('Y');
$members = $wpdb->get_results("SELECT * FROM ".$table_name, ARRAY_A);
$columns = $wpdb->get_results("SHOW COLUMNS FROM ".$table_name, ARRAY_A);
// echo 'SQL: '.$sql.', RESULT: '.$result.'<br>';
//output headers
header("Content-type: application/octet-stream");
header("Content-Disposition: attachment; filename=\"members.csv\"");
//open output stream
$output = fopen("php://output",'w');
//output column headings
$data[0] = "ID";
$i = 1;
foreach ($columns as $column){
//DIAG: echo '<pre>'; print_r($column); echo '</pre>';
$field_name = '';
$words = explode("_", $column['Field']);
foreach ($words as $word) $field_name .= $word.' ';
if ( $column['Field'] != 'id' && $column['Field'] != 'date_updated' ) {
$data[$i] = ucwords($field_name);
$i++;
}
}
$data[$i] = "Date Updated";
fputcsv4($output, $data);
//output data
foreach ($members as $member){
// echo '<pre>'; print_r($member); echo '</pre>';
$data[0] = $member['id'];
$i = 1;
foreach ($columns as $column){
//DIAG: echo '<pre>'; print_r($column); echo '</pre>';
if ( $column['Field'] != 'id' && $column['Field'] != 'date_updated' ) {
$data[$i] = $member[$column['Field']];
$i++;
}
}
$data[$i] = $member['date_updated'];
//echo '<pre>'; print_r($data); echo '</pre>';
fputcsv4($output, $data);
}
fclose($output);
?>
So, obviously, a routine wherein a query is run, $output is established with fopen, each row is then formatted as comma delimited and fwrited, and finally the file is fclosed where it gets pushed to a local system.
The error that I'm getting (from the server) is
Error 6 (net::ERR_FILE_NOT_FOUND): The file or directory could not be found.
But it clearly is getting found, its just failing. If I enable phpinfo() (PHP Version 5.2.17) at the top of the file, I definitely get a response - notably Cannot modify header information (I'm pretty sure because phpinfo() has already generated a header). All the expected data does get printed to the bottom of the page (after all the phpinfo diagnostics), however, so that much at least is working correctly.
I am guessing there is something preventing the fopen, fwrite, or fclose functions from working properly (a server setting?), but I don't have enough experience with this to identify exactly what the problem is.
I'll note again that this works exactly as expected in my test environment (localhost/XAMPP, netbeans).
Any thoughts would be most appreciated.
update
Ok - spent some more time with this today. I've tried each of the suggested fixes, including #Rudu's writeCSVLine fix and #Fernando Costa's file_put_contents() recommendation. The fact is, they all work locally. Either just echoing or the fopen,fwrite,fclose routine, doesn't matter, works great.
What does seem to be a problem is the inclusion of the wp-blog-header.php at the start of the file and then the additional header() calls. (The path is definitely correct on the server, btw.)
If I comment out the include, I get a csv file downloaded with some errors planted in it (because $wpdb doesn't exist. And if comment out the headers, I get all my data printed to the page.
So... any ideas what could be going on here?
Some obvious conflict of the wordpress environment and the proper creation of a file.
Learning a lot, but no closer to an answer... Thinking I may need to just avoid the wordpress stuff and do a manual sql query.
Ok so I'm wondering why you've taken this approach. Nothing wrong with php://output but all it does is allow you to write to the output buffer the same way as print and echo... if you're having trouble with it, just use print or echo :) Any optimizations you could have got from using fwrite on the stream then gets lost by you string-building the $csv variable and then writing that in one go to the output stream (Not that optimizations are particularly necessary). All that in mind my solution (in keeping with your original design) would be this:
function escapeCSVcell($val) {
return str_replace('"','""',$val);
//What about new lines in values? Perhaps not relevant to your
// data but they'll mess up your output ;)
}
function writeCSVLine($arr) {
$first=true;
foreach ($arr as $v) {
if (!$first) {echo ",";}
$first=false;
echo "\"".escapeCSVcell($v)."\"";
}
echo "\n"; // May want to use \r\n depending on consuming script
}
Now use writeCSVLine in place of fputcsv4.
Ran into this same issue. Stumbled upon this thread which does the same thing but hooks into the 'plugins_loaded' action and exports the CSV then. https://wordpress.stackexchange.com/questions/3480/how-can-i-force-a-file-download-in-the-wordpress-backend
Exporting the CSV early eliminates the risk of the headers already being modified before you get to them.
Going around in circles so hopefully someone can put me right.
Generating a CSV through PHP and using my data from MySQL. Data is inputted by the user via TinyMCE so has various "p" tags etc that I obviously need to strip out which I can do successfully, however the string in the csv still gets displayed in various different cells even after being stripped.
I've tried using strip_tags() and preg matching "/n" and "/r" and all sorts to remove line breaks but no luck as it still appears on new cells.
I'm using a semi colon as the seperator in the csv.
An example of a string i'm trying to strip all html from would be<p>I would like to join because it looks fun</p><p> </p><p>Help me.</p>. It only happens when their is more than one tag or something - if the string is simply one line, it works no issues.
Heres the csv code i'm using:
echo 'Username;Email;Date joined;Reason for joining'."\r\n";
foreach($users as $user) {
$csv_data .= $user->username . ';' . $user->email. ';' . date("d/m/y", $user->time_created) . ';' . $user->reason_for_joining .';'."\r\n";
}
echo $csv_data;
Where am i going wrong?
Thanks guys :)
The CSV format has more pitfalls than one might imagine (as you are experiencing). Therefore, don't reinvent the wheel and use fputcsv, which correctly escapes everything that needs escaping.
fputcsv expects to write to a file, but you can give it a fake file to write to instead, as outlined here:
$fp = fopen('php://output', 'w');
$delimiter = ';';
$headers = array('Username', 'Email', 'Date joined', 'Reason for joining');
fputcsv($fp, $headers, $delimiter);
foreach ($users as $user) {
$fields = array($user->...);
fputcsv($fp, $fields, $delimiter);
}
fclose($fp);
you forgot the "\r" for the first echo,
supposed to be:
echo 'Username;Email;Date joined;Reason for joining'."\r\n";
and better set $csv_data to blank before the 'for' loop.
and if either 'username', 'email', 'reason_for_joining' has the semicolon,
the code will generate a mal-formed csv.