I'm trying to use Mailchimp's Export API to generate a CSV file of all members of a given list. Here' the documentation and the example PHP code they give:
$apikey = 'YOUR_API_KEY';
$list_id = 'YOUR_LIST_ID';
$chunk_size = 4096; //in bytes
$url = 'http://us1.api.mailchimp.com/export/1.0/list?apikey='.$apikey.'&id='.$list_id;
/** a more robust client can be built using fsockopen **/
$handle = #fopen($url,'r');
if (!$handle) {
echo "failed to access url\n";
} else {
$i = 0;
$header = array();
while (!feof($handle)) {
$buffer = fgets($handle, $chunk_size);
if (trim($buffer)!=''){
$obj = json_decode($buffer);
if ($i==0){
//store the header row
$header = $obj;
} else {
//echo, write to a file, queue a job, etc.
echo $header[0].': '.$obj[0]."\n";
}
$i++;
}
}
fclose($handle);
}
This works well for me and when I run this file, I end up with a bunch of data in this format:
Email Address: xdf#example.com
Email Address: bob#example.com
Email Address: gerry#example.io
What I want to do is turn this into a CSV (to pass to a place on my server) instead of echoing the data. Is there a library or simple syntax/snippit I can use to make this happen?
If the format simply like:
Email Address, xdf#example.com
Email Address, bob#example.com
Email Address, gerry#example.io
is what you after, then you can do:
$handle = #fopen($url,'r');
$csvOutput = "";
if (!$handle) {
echo "failed to access url\n";
} else {
$i = 0;
$header = array();
while (!feof($handle)) {
$buffer = fgets($handle, $chunk_size);
if (trim($buffer)!=''){
$obj = json_decode($buffer);
if ($i==0){
//store the header row
$header = $obj;
} else {
//echo, write to a file, queue a job, etc.
echo $header[0].', '.$obj[0]."\n";
$csvOutput .= $header[0].', '.$obj[0]."\n";
}
$i++;
}
}
fclose($handle);
}
$filename = "data".date("m.d.y").".csv";
file_put_contents($filename, $csvOutput);
The variable $csvOutput contains the CSV format string.
This ones on me. From now on you might want to actually read some documentation instead of copying and pasting your way through life. other's will not be as nice as i am. here's a list of file system functions from the php website. http://php.net/manual/en/ref.filesystem.php Getting the file output to the desired csv format is an exercise left to the reader.
$apikey = 'YOUR_API_KEY';
$list_id = 'YOUR_LIST_ID';
$chunk_size = 4096; //in bytes
$url = 'http://us1.api.mailchimp.com/export/1.0/list?apikey='.$apikey.'&id='.$list_id;
/** a more robust client can be built using fsockopen **/
$handle = #fopen($url,'r');
if (!$handle) {
echo "failed to access url\n";
} else {
$i = 0;
$header = array();
$output = ''; //output buffer for the file we are going to write.
while (!feof($handle)) {
$buffer = fgets($handle, $chunk_size);
if (trim($buffer)!=''){
$obj = json_decode($buffer);
if ($i==0){
//store the header row
$header = $obj;
} else {
//write data into our output buffer for the file
$output .= $header[0].': '.$obj[0]."\n";
}
$i++;
}
}
fclose($handle);
//now write it to file
$path = '/path/to/where/you/want/to/store/file/';
$file_name = 'somefile.csv';
//create a file resource to write to.
$fh = fopen($path.$file_name,'w+');
//write to the file
fwrite($fh,$output);
//close the file
fclose($fh);
}
Related
I need to transfer files of any type or size over HTTP/GET in ~1k chunks. The resulting file hash needs to match the source file. This needs to be done in native PHP without any special tools. I have a basic strategy but I'm getting odd results. This proof of concept just copies the file locally.
CODE
<?php
$input="/home/lm1/Music/Ellise - Feeling Something Bad.mp3";
$a=pathinfo($input);
$output=$a["basename"];
echo "\n> ".md5_file($input);
$fp=fopen($input,'rb');
if ($fp) {
while(!feof($fp)) {
$buffer=base64_encode(fread($fp,1024));
// echo "\n\n".md5($buffer);
write($output,$buffer);
}
fclose($fp);
echo "\n> ".md5_file($output);
echo "\n";
}
function write($file,$buffer) {
// echo "\n".md5($buffer);
$fp = fopen($file, 'ab');
fwrite($fp, base64_decode($buffer));
fclose($fp);
}
?>
OUTPUT
> d31e102b1cae9c73bbf5a12615a8ea36
> 9f03f6c88ed61c07cb534922d6d31864
Thanks in advance.
fread already advances the file pointer position, so there's no need to keep track of it. Same with frwite, so consecutive calls automatically append to the given file. Thus, you could simplify your approach to (code adapted from this answer on how to efficiently write a large input stream to a file):
$src = "a.test";
$dest = "b.test";
$fp_src = fopen($src, 'rb');
if ($fp_src) {
$fp_dest = fopen($dest, 'wb');
$buffer_size = 1024;
while(!feof($fp_src)) {
fwrite($fp_dest, fread($fp_src, $buffer_size));
}
fclose($fp_src);
fclose($fp_dest);
echo md5_file($src)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7
echo md5_file($dest)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7
}
If you want to keep both processes separated, you'd do:
$src = "a.test";
$dest = "b.test";
if (file_exists($dest)) {
unlink($dest); // So we don't append to an existing file
}
$fp = fopen($src,'rb');
if ($fp) {
while(!feof($fp)){
$buffer = base64_encode(fread($fp, 1024));
write($dest, $buffer);
}
fclose($fp);
}
function write($file, $buffer) {
$fp = fopen($file, 'ab');
fwrite($fp, base64_decode($buffer));
fclose($fp);
}
echo md5_file($src)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7
echo md5_file($dest)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7
As for how to stream files over HTTP, you might want to have a look at:
Streaming a large file using PHP
Here has a question: I need execute a task to put many data to another mysql database per minute; if the first task hasn't finish, the second has start; so,there has a multiple concurrent problem; how to resolve the problem??
I have some ideas, first, Let the task has a execute-time which less than the start time of next task;second, let the task support multi-process; but,i don't the how to write the code?
public function execute(Input $input, Output $output)
{
$tele_data = Telesales::field('*')->where([['create_time','<',time()-48*3600],['customer_label','in',[2,6,7]],['virtual_sale','=','0']])->whereRaw('phone is not null')->select()->toArray();
foreach($tele_data as $key=>$value) {
static::pushTeleToIdc($value);
}
}
private static function pushTeleToIdc($data = []) {
$res = Telesales::where('id',$value['id'])->update(['virtual_sale'=>'1']);
if(!$res) {
return;
}
$url = config('idc.tele_url');
$key = config('idc.tele_key');
$channel = config('idc.tele_channel');
$time = time();
$sign = md5($key.$channel.$time);
$urls = $url."?channel=".$channel."&sign=".$sign."&time=".$time;
$require_params = config('idc.require_params');
foreach($require_params as $key=>$value) {
if(array_key_exists($key,$data) && !empty($data[$key])) {
$d[$key] = $data[$key];
}else{
$d[$key] = empty($value)?'':$value[array_rand($value,1)];
}
}
$d['register_time'] = $d['create_time'];
$res = post_url($urls,$d);
$result = json_decode($res,true);
if (isset($result['code']) && $result['code'] != 0){
Log::init(['single'=>'tpushidc'])->error($res);
}
}
Could you help me resolve the problem?
The easiest thing to do is to setup a flag to tell that the process is already in progress and check if that's the case at the start of the function. I don't know how you want to setup the visibility of your code, so I leave it to you to extract $myFile to the file/class scope (same goes for the file path, you probably want to use some /var or /log folder for such stuff).
So the gist is: we create a file, if it doesn't exist or there is a 0 in it - it means we can start working. On other hand, if the contents of the file is 1, the process will die and it will be so every time you run it, until the first one finishes and rewrites the contents of the file to 0 (which means the process is not in progress anymore).
public function execute(Input $input, Output $output)
{
if ($this->isProcessInProgress()) {
die('Process is in progress');
}
$this->startProcess();
$tele_data = [...];
foreach($tele_data as $key=>$value) {
static::pushTeleToIdc($value);
}
$this->finishProcess();
}
private function isProcessInProgress() {
$myFile = 'tele_to_idc_process.txt';
$handle = fopen($myFile, 'r');
if (!$handle)
return false;
$status = fread($handle, 1);
fclose($handle);
return (bool) $status;
}
private function startProcess() {
$myFile = 'tele_to_idc_process.txt';
$handle = fopen($myFile, 'w');
if (!$handle)
return;
$status = fwrite($handle, '1');
fclose($handle);
}
private function finishProcess() {
$myFile = 'tele_to_idc_process.txt';
$handle = fopen($myFile, 'w');
if (!$handle)
return;
$status = fwrite($handle, '0');
fclose($handle);
}
You might get a warning if the file doesn't exist, you can suppress it with #fopen instead of fopen
So I am storing my files in a database. Don't ask why, just know that I am not in control of this. Next, I am able to successfully store them as a hexidecimal representation and then spit them back for display with no problem, but then I attach them to an email using PHPMailer and they get sent properly with the right name and all, but they are corrupted. I will walk you through step by step below so that you know exactly how it is being stored, and this may help me debug my issue. (Please note that all code is paraphrased to save space and only show what is needed)
STEP 1
File is grabbed and then processed
$name = $_FILES['file_data']['name'];
$file = prepareImageDBString($_FILES['file_data']['tmp_name']);
$mime_type = $_FILES['file_data']['type'];
name, file, and mime_type are stored
here is the function prepareImageDBString()
function prepareImageDBString($filepath){
$out = 'null';
$handle = #fopen($filepath, 'r');
if($handle){
$content = #fread($handle, filesize($filepath));
$content = bin2hex($content);
#fclose($handle);
$out = $content;
}
return $out;
}
STEP 2
When the file is being viewed I show it as an embedded object. This file is small so I just posted the whole code. Do note that the file shows up with no problems here.
$q = "SELECT lease_doc_file_data FROM lease_doc_file WHERE lease_doc__id ='".$_GET['id']."'";
$file = "";
foreach($CONN->query($q) as $row){
$file = $row['lease_doc_file_data'];
}
if(!empty($file)){
header("Content-type: application/pdf");
ob_clean();
flush();
echo hextobin($file);
}
Here is the function hextobin()
function hextobin($hexstr){
$n = strlen($hexstr);
$sbin = "";
$i = 0;
while($i < $n){
$a = substr($hexstr,$i,2);
$c = pack("H*", $a);
if ( $i == 0 ){ $sbin = $c; }
else { $sbin .= $c;}
$i += 2;
}
return $sbin;
}
STEP 3
Finally the part where I go to send it as a mailer.
$q = "SELECT lease_doc_file_data, lease_doc_file_name, lease_doc_file_type FROM lease_doc_file WHERE lease_doc__id ='$id'";
$file_data = "";
$file_name = "";
$file_type = "";
foreach($CONN->query($q) as $row){
$file_data = $row['lease_doc_file_data'];
$file_name = $row['lease_doc_file_name'];
$file_type = $row['lease_doc_file_type'];
}
$file_data = hextobin($file_data);
$mail->AddStringAttachment($file_data, $file_name, 'binary', $file_type);
So this is the three step process and I"m not sure where the error is coming from. Hopefully someone can help! Thank you for all help in advance!
I am trying to add certain variables to couple of files which already have some content.
I am using file_get_contents to copy the contents of a particular file and then using file_put_contents to paste variable values along with the existing contents to that file.
The problem is that, on the first instance it works properly but to the second file it pastes everything that has been stored in the memory. It puts all the contents from the first file along with the contents of the second file.
Is there any way that I can clear the memory before the next file_get_contents executes. Or my concept is false here.
Here is my code...
<?php
if ($_POST["submit"]) {
$ip = $_POST['ip'];
$subnet = $_POST['subnet'];
$gateway = $_POST['gateway'];
$hostname = $_POST['hostname'];
$domain = $_POST['domain'];
$netbios = $_POST['netbios'];
$password = $_POST['password'];
$ipfile = 'one.txt';
$file = fopen($ipfile, "r");
$ipfileContents = fread($file, filesize($ipfile));
$ipcontent = "ip='$ip'\n";
$ipcontent .= "netmask='$subnet'\n";
$ipcontent .= "gw='$gateway'\n";
$conten = $ipcontent . $ipfileContents;
$file = fopen($ipfile, "w");
fwrite($file, $ipfileContents);
fclose($file);
$ipsh = shell_exec('sh path/to/CHANGE_IP.sh');
$hostfile = 'two.txt';
$fileh = fopen($hostfile, "r");
$hostfileContents = fread($fileh, filesize($hostfile));
$hostcontent = "ip='$ip'\n";
$hostcontent .= "m_name='$hostname'\n";
$hostcontent .= "fqdn='$domain'\n";
$conten = $hostcontent . $hostfileContents;
$fileh = fopen($hostfile, "w");
fwrite($fileh, $hostfileContents);
fclose($fileh);
$hostsh = shell_exec('sh path/to/MODIFY_HOSTS.sh');
}
?>
I have tried unset, but didn't work
$ipfilecontents->__destruct();
unset($ipfilecontents);
UPDATE:
file_get_contents & file_put_contents has some concurrency problems. So I had to change my method to fopen/fwrite/fclose and it worked flawlessly. Thanks for your help Jacinto.
if ($_POST["submit"]) {
$ip = $_POST['ip'];
$subnet = $_POST['subnet'];
$gateway = $_POST['gateway'];
$hostname = $_POST['hostname'];
$domain = $_POST['domain'];
$netbios = $_POST['netbios'];
$password = $_POST['password'];
$ipfile = 'one.txt';
$file = fopen($ipfile, "r");
$ipfileContents = fread($file, filesize($ipfile));
$ipcontent = "ip='$ip'\n";
$ipcontent .= "netmask='$subnet'\n";
$ipcontent .= "gw='$gateway'\n";
$content = $ipcontent . $ipfileContents;
$file = fopen($ipfile, "w");
fwrite($file, $content);
fclose($file);
$ipsh = shell_exec('sh path/to/CHANGE_IP.sh');
//do the same to the next file
}
This isn't an answer - I'll delete it in a minute. It's just a convenient place to show how to do trace statements:
$ipfile = 'one.txt';
$ipfileContents = file_get_contents($ipfile);
$ipcontent = "ip='$ip'\n";
$ipcontent .= "netmask='$subnet'\n";
$ipcontent .= "gw='$gateway'\n";
echo "DEBUG: hostcontent=<pre>$ipcontent</pre><br />====<br />hostfileContents=<pre>$ipfileContents</pre><br />\n";
file_put_contents($ipfile, $ipcontent . $ipfileContents, LOCK_EX);
$ipsh = shell_exec('sh path/to/CHANGE_IP.sh');
$hostfile = 'two.txt';
$hostfileContents = file_get_contents($hostfile);
$hostcontent = "ip='$ip'\n";
$hostcontent .= "m_name='$hostname'\n";
$hostcontent .= "fqdn='$domain'\n";
echo "DEBUG: hostcontent=<pre>$hostcontent</pre><br />====<br />hostfileContents=<pre>$hostfileContents</pre><br />\n";
file_put_contents($hostfile, $hostcontent . $hostfileContents, LOCK_EX);
$hostsh = shell_exec('sh path/to/MODIFY_HOSTS.sh');
The most efficient would be to read and write the file in chunks simultaneously:
open the file in read-write mode
read the data chunk that will be overwritten by the new data
reset pointer to begin of read chunk
write new data
make the read data the new data to be written
repeat until there is no data to write
Example:
$bufw = "ip=''\n";
$bufw .= "netmask=''\n";
$bufw .= "gw=''\n";
$bufw_len = strlen($bufw);
$file = fopen($ipfile, 'c+');
while ($bufw_len > 0) {
// read next chunk
$bufr = fread($file, $bufw_len);
$bufr_len = strlen($bufr);
// reset pointer to begin of chunk
fseek($file, -$bufr_len, SEEK_CUR);
// write previous chunk
fwrite($file, $bufw);
// update variables
$bufw = $bufr;
$bufw_len = strlen($bufw);
}
fclose($file);
With this the total memory usage is only up to the length of the new data to be written.
You can make it a function like:
function file_prepend_contents($filename, $data, $flags = 0, $context = null) {
if (!is_null($context)) {
$handle = fopen($filename, 'c+', ($flags & FILE_USE_INCLUDE_PATH) === FILE_USE_INCLUDE_PATH, $context);
} else {
$handle = fopen($filename, 'c+', ($flags & FILE_USE_INCLUDE_PATH) === FILE_USE_INCLUDE_PATH);
}
if (!$handle) return false;
if (($flags & LOCK_EX) === LOCK_EX) {
flock($handle, LOCK_EX);
}
$bytes_written = 0;
$bufw = $data;
$bufw_len = strlen($bufw);
while ($bufw_len > 0) {
// read next chunk
$bufr = fread($handle, $bufw_len);
$bufr_len = strlen($bufr);
// reset pointer
fseek($handle, -$bufr_len, SEEK_CUR);
// write current chunk
if (ftell($handle) === 0) {
$bytes_written = fwrite($handle, $bufw);
} else {
fwrite($handle, $bufw);
}
// update variables
$bufw = $bufr;
$bufw_len = strlen($bufw);
}
fclose($handle);
return $bytes_written;
}
I am trying to export CSV using PHP. But instead of printing the result it's printing Resource id #26 in the generated CSV FILE. If I remove exit from end it print's my whole HTML page content. My code is...
if (isset($_REQUEST['download']) && ($_REQUEST['download'] == "yes")) {
header('Content-Type: text/csv; charset=utf-8');
header('Content-Disposition: attachment; filename=link_point_report.csv');
$output = fopen('php://memory', 'w+');
fputcsv($output, array('Member Id', 'Customer Name', 'Customer Email'));
$customerListAll = $oAdmFun->linkpoint_check_customer('', '');
$oAdmFun->pr($customerListAll, true);
// if (count($customerListAll) > 0) {
// for ($c = 0; $c < count($customerListAll); $c++) {
// fputcsv($output, array('Sankalp', 'Sankalp', 'Sankalp'));
// }
// }
ob_clean();
flush();
exit($output);
}
That's because $output is a resource, which is what fopen() returns. You need to use fread() to get its contents.
EDIT: Now I understand what you were asking. To output the contents of your CSV, you need to get the text output from the resource:
rewind( $output );
$csv_output = stream_get_contents( $output );
fclose( $output );
die( $csv_output );
And it's also a good idea to set the content-length header so that the client knows what to expect:
header('Content-Length: ' . strlen($csv_output) );
Open a chunk of writable memory:
$file = fopen('php://temp/maxmemory:'. (12*1024*1024), 'r+'); // 128mb
Write to it... then fetch the contents with:
rewind($file);
$output = stream_get_contents($file);
fclose($file);
function CSV_HtmlTable($csvFileName)
{
// better to open new webpage
echo "<html>\n<body>\n\t<table style=''>\n\n";
$f = fopen($csvFileName, "r");
$trcount = 0; //start the row count as 0
while (($line = fgetcsv($f)) !== false) {
$trclass = ''; if ($trcount%2==0) { $trclass=' class="dark"'; } //default to nothing, but if it's even apply a class
echo "\t\t<tr".$trclass.">\n"; //same as before, but now this also has the variable $class to setup a class if needed
$tdcount = 0; //reset to 0 for each inner loop
foreach ($line as $cell) {
$tdclass = ''; if ($tdcount%2==0) { $tdclass=' class="dark"'; } //default to nothing, but if it's even apply a class
echo "\t\t\t<td ".$tdclass."style='padding:.4em;'>" . htmlspecialchars($cell) . "</td>"; //same as before, but now this also has the variable $class to setup a class if needed
$tdcount++; //go up one each loop
}
echo "\r</tr>\n";
$trcount++; //go up one each loop
}
fclose($f);
echo "\n\t</table>\n</body>\n</html>";
}