Replace a character in a large file using PHP - php

I tried to replace single quotes in a large XML file(110MB) with this code but an error occured. I need a code that can handle atleast 3GB XML file.
Error Message:
Fatal error: Allowed memory size of 134217728 bytes exhausted
(tried to allocate 20449728 bytes) in C:\xampp\htdocs\replace.php on
line 10
<?php
replace_file('electronics.xml', "'", "'");
function replace_file($path, $string, $replace)
{
$file = fopen($path, 'a+');
while (feof($file) === false)
{
$str=file_get_contents($path);
$str=str_replace($string, $replace, fgets($file));
}
fclose($file);
}
echo "replace done";
?>

Reading a large file into php is not recommended. Call a command line that is appropriate, like sed
Reference: http://www.grymoire.com/Unix/Sed.html

Simplify:
$str = str_replace( "'","'",file_get_contents('electronics.xml'));
This is just very wrong:
Opening XML
$file = fopen($path, 'a+');
While Loop for no reason, fgets reads to end of file, so loop completes on first iteration.
while (feof($file) === false)
{
reading in entire contents of same file file again, for no purpose
$str=file_get_contents($path);
Reading in entire file, no length specified, so reading to EOF
$str=str_replace($string, $replace, fgets($file));
}
fclose($file);
Nothing accomplished.

////
//PHP 5.3 + Class find and replace string in files
//
//by Bruce Afruz
//
//2013
//
//example usage for single file:
//
//$new = new fileReplacement('./');
//$new->setExt("check.php");
//$new->changeContents("hello", "goodbye");
//
//example usage for multiple files:
//
//$new = new fileReplacement('./test');
//$new->setExt("*.html");
//$new->changeContents("hello", "goodbye");
//
//to change directory:
//
//$new = new fileReplacement('./test');
//$new->setDir("./test2");
//$new->setExt("*.html");
//$new->changeContents("hello", "goodbye");
////
class fileReplacement
{
private $ext , $dir ;
public function getDir() {
return $this->dir;
}
public function setDir($dir) {
$this->dir = $dir;
}
public function getExt() {
return $this->ext;
}
public function setExt($ext) {
$this->ext = $ext;
}
function __construct($dir) {
$this->dir = $dir;
}
public function rglob($pattern = '*', $flags = 0, $path = '') {
chdir($this->getDir());
$paths = glob($path . '*', GLOB_MARK | GLOB_ONLYDIR | GLOB_NOSORT);
$files = glob($path . $pattern, $flags);
foreach ($paths as $path) {
$files = array_merge($files, $this->rglob($pattern, $flags, $path));
}
return $files;
}
public function changeContents($replace , $sentence , $flags = 0, $path = '') {
$all = $this->rglob($this->getExt() , $flags, $path);
foreach ($all as $file) {
$filename = $file;
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
$contents = str_replace($replace , $sentence, $contents);
if (is_writable($filename)) {
if (!$handle = fopen($filename, 'w+')) {
echo "Cannot open file ($filename)
";
exit;
}
// Write $contents to our opened file.
if (fwrite($handle, $contents) === FALSE) {
echo "Cannot write to file ($filename)
";
exit;
}
echo "Success, wrote content to file ($filename)
";
fclose($handle);
} else {
echo "The file $filename is not writable
";
}
}
}}

Related

How can I create an array by parsing a large file? [duplicate]

I want to read a file line by line, but without completely loading it in memory.
My file is too large to open in memory, and if try to do so I always get out of memory errors.
The file size is 1 GB.
You can use the fgets() function to read the file line by line:
$handle = fopen("inputfile.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
}
fclose($handle);
}
if ($file = fopen("file.txt", "r")) {
while(!feof($file)) {
$line = fgets($file);
# do same stuff with the $line
}
fclose($file);
}
You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)
<?php
$file = new SplFileObject("file.txt");
// Loop until we reach the end of the file.
while (!$file->eof()) {
// Echo one line from the file.
echo $file->fgets();
}
// Unset the file to call __destruct(), closing the file handle.
$file = null;
If you want to use foreach instead of while when opening a big file, you probably want to encapsulate the while loop inside a Generator to avoid loading the whole file into memory:
/**
* #return Generator
*/
$fileData = function() {
$file = fopen(__DIR__ . '/file.txt', 'r');
if (!$file) {
return; // die() is a bad practice, better to use return
}
while (($line = fgets($file)) !== false) {
yield $line;
}
fclose($file);
};
Use it like this:
foreach ($fileData() as $line) {
// $line contains current line
}
This way you can process individual file lines inside the foreach().
Note: Generators require >= PHP 5.5
There is a file() function that returns an array of the lines contained in the file.
foreach(file('myfile.txt') as $line) {
echo $line. "\n";
}
The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.
$fp = fopen("/path/to/the/file", "r");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
echo $line;
}
fclose($fp);
Use buffering techniques to read the file.
$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer = fread($source_file, 4096); // use a buffer of 4KB
$buffer = str_replace($old,$new,$buffer);
///
}
foreach (new SplFileObject(__FILE__) as $line) {
echo $line;
}
One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.
$handle = fopen("some_file.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
$line = str_replace("\n", "", $line);
}
fclose($handle);
}
This how I manage with very big file (tested with up to 100G). And it's faster than fgets()
$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) {
$left='';
while (!feof($fh)) {// read the file
$temp = fread($fh, $block);
$fgetslines = explode("\n",$temp);
$fgetslines[0]=$left.$fgetslines[0];
if(!feof($fh) )$left = array_pop($lines);
foreach ($fgetslines as $k => $line) {
//do smth with $line
}
}
}
fclose($fh);
Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.
SplFileObject is useful when it comes to dealing with large files.
function parse_file($filename)
{
try {
$file = new SplFileObject($filename);
} catch (LogicException $exception) {
die('SplFileObject : '.$exception->getMessage());
}
while ($file->valid()) {
$line = $file->fgets();
//do something with $line
}
//don't forget to free the file handle.
$file = null;
}
<?php
echo '<meta charset="utf-8">';
$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
$contents = '';
for($i=1;$i<=1500;$i++){
echo $k.' -- '. fgets($fp) .'<br>';$k++;
$contents .= fgets($fp);
}
echo '<hr>';
file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>
Function to Read with array return
function read_file($filename = ''){
$buffer = array();
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer[] = fread($source_file, 4096); // use a buffer of 4KB
}
return $buffer;
}

yii2 REST api for file upload

Im trying to upload a file image/file type from mobile app and store that image in the backend. Im using Yii2 framework API to do this. And im using postman to check the API. Im running the below in my action.
/*Uploading documents*/
public function actionUploading_doc() {
$uploads = \yii\web\UploadedFile::getInstanceByName('upfile');
print_r($uploads);exit;
if (empty($uploads)){
return "Must upload at least 1 file in upfile form-data POST";
}
foreach ($uploads as $file){
$filename = time() . $image->name;
$path = "uploads/" . $filename;
$file->saveAs($path);
}
}
When i run this as POST method from postman.. and print the value of $uploads im getting empty value. It mean its not coming to controller.
Please help me in solving this.
For me this is what i did without the UploadFile class
/*Uploading documents*/
public function actionUploading() {
$uploads = \yii\web\UploadedFile::getInstanceByName('upfile');
\yii::$app->request->enableCsrfValidation = false;
$filename = $uploads->name;
$path = "http://localhost/projects/YiiRestful/api/web/uploads/".$filename;
$putdata = fopen("php://input", "r");
// make sure that you have /web/upload directory (writeable)
// for this to work
$path = "uploads/".$filename;
$fp = fopen($path, "w");
while ($data = fread($putdata, 1024))
fwrite($fp, $data);
/* Close the streams */
fclose($fp);
fclose($putdata);
}
I would try something like this... (not tested)
public function actionUploadingDoc() { // good practice to use camel case for methods
$uploads = \yii\web\UploadedFile::getInstances('upfile');
if (empty($uploads)){
return false;
// handle error reporting somewhere else
}
$path = 'uploads/'; // set your path
foreach ($uploads as $upload){
$filename = $path . time() .'_'. $upload->name ;
$upload->saveAs($filename);
}
return true;
}
You can use base64 string to uplod. define function inside controller like this
public function base64_to_jpeg($base64_string, $output_file) {
$path="your/real/path/";
// open the output file for writing
$ifp = fopen( $path.$output_file, 'wb' );
// split the string on commas
// $data[ 0 ] == "data:image/png;base64"
// $data[ 1 ] == <actual base64 string>
$data = explode( ',', $base64_string );
if(count($data)>1) {
$dataText=$data[ 1 ];
} else {
$dataText=$base64_string;
}
// we could add validation here with ensuring count( $data ) > 1
fwrite( $ifp, base64_decode( $dataText ) );
// clean up the file resource
fclose( $ifp );
return $output_file;
}
And use inside action as
public function actionUpload(){
$imgName=md5(uniqid()).'.jpg';
$this->base64_to_jpeg($base64_string, $imgName);
}

Why won't PHP read this file?

I'm trying to make it retrieve the image files on the server but it won't work if there is a space in the name of the image file .. for example there is a space between dead and air , even if I escape it after adding %20, the function returns an empty string .. but if it is a file with no space in the name like 'http://www.m.trialsite.com/images/thumb/Espresso.jpg'; It will work ! .. where am I going wrong ?
$filename = 'http://www.m.trialsite.com/images/thumb/dead air.jpg';
function readfile_chunked($filename,$retbytes=true) {
$chunksize = 1*(1024*1024); // how many bytes per chunk
$buffer = '';
$cnt =0;
// $handle = fopen($filename, 'rb');
$filename = str_replace(' ','%20',$filename);
$handle = fopen($filename, 'rb');
if ($handle === false) {
return false;
}
$filename = str_replace(' ','%20',$filename);
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer; var_dump($buffer); exit;
ob_flush();
flush();
if ($retbytes) {
$cnt += strlen($buffer);
}
}
$status = fclose($handle);
if ($retbytes && $status) {
return $cnt; // return num. bytes delivered like readfile() does.
}
return $status;
}
use preg_replace("/\s+/","_",$nome); to rename the files and then recovers it it will work
$directory = '/public_html/testfolder/';//example
if ($handle = opendir($directory)) {
while (false !== ($fileName = readdir($handle))) {
$newName = preg_replace("/\s+/","_",$fileName);
rename($directory . $fileName, $directory . $newName);
}
closedir($handle);
}
What if you are doing like this:
$filename = str_replace(' ','%20', 'http://www.m.trialsite.com/images/thumb/dead air.jpg');

How to read a large file line by line?

I want to read a file line by line, but without completely loading it in memory.
My file is too large to open in memory, and if try to do so I always get out of memory errors.
The file size is 1 GB.
You can use the fgets() function to read the file line by line:
$handle = fopen("inputfile.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
}
fclose($handle);
}
if ($file = fopen("file.txt", "r")) {
while(!feof($file)) {
$line = fgets($file);
# do same stuff with the $line
}
fclose($file);
}
You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)
<?php
$file = new SplFileObject("file.txt");
// Loop until we reach the end of the file.
while (!$file->eof()) {
// Echo one line from the file.
echo $file->fgets();
}
// Unset the file to call __destruct(), closing the file handle.
$file = null;
If you want to use foreach instead of while when opening a big file, you probably want to encapsulate the while loop inside a Generator to avoid loading the whole file into memory:
/**
* #return Generator
*/
$fileData = function() {
$file = fopen(__DIR__ . '/file.txt', 'r');
if (!$file) {
return; // die() is a bad practice, better to use return
}
while (($line = fgets($file)) !== false) {
yield $line;
}
fclose($file);
};
Use it like this:
foreach ($fileData() as $line) {
// $line contains current line
}
This way you can process individual file lines inside the foreach().
Note: Generators require >= PHP 5.5
There is a file() function that returns an array of the lines contained in the file.
foreach(file('myfile.txt') as $line) {
echo $line. "\n";
}
The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.
$fp = fopen("/path/to/the/file", "r");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
echo $line;
}
fclose($fp);
Use buffering techniques to read the file.
$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer = fread($source_file, 4096); // use a buffer of 4KB
$buffer = str_replace($old,$new,$buffer);
///
}
foreach (new SplFileObject(__FILE__) as $line) {
echo $line;
}
One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.
$handle = fopen("some_file.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
$line = str_replace("\n", "", $line);
}
fclose($handle);
}
This how I manage with very big file (tested with up to 100G). And it's faster than fgets()
$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) {
$left='';
while (!feof($fh)) {// read the file
$temp = fread($fh, $block);
$fgetslines = explode("\n",$temp);
$fgetslines[0]=$left.$fgetslines[0];
if(!feof($fh) )$left = array_pop($lines);
foreach ($fgetslines as $k => $line) {
//do smth with $line
}
}
}
fclose($fh);
Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.
SplFileObject is useful when it comes to dealing with large files.
function parse_file($filename)
{
try {
$file = new SplFileObject($filename);
} catch (LogicException $exception) {
die('SplFileObject : '.$exception->getMessage());
}
while ($file->valid()) {
$line = $file->fgets();
//do something with $line
}
//don't forget to free the file handle.
$file = null;
}
<?php
echo '<meta charset="utf-8">';
$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
$contents = '';
for($i=1;$i<=1500;$i++){
echo $k.' -- '. fgets($fp) .'<br>';$k++;
$contents .= fgets($fp);
}
echo '<hr>';
file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>
Function to Read with array return
function read_file($filename = ''){
$buffer = array();
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer[] = fread($source_file, 4096); // use a buffer of 4KB
}
return $buffer;
}

Extract sub folders of ZIP file in PHP

I am using a php script to unzip ZIP file. but this script unzip only one level of directories without extracting the sub directories of that file
the script:
$zip = new ZipArchive;
if ($zip->open('test.zip') === TRUE) {
$zip->extractTo('/my/destination/dir/');
$zip->close();
echo 'ok';
} else {
echo 'failed';
}
for example: if the test.zip contains 2 folders: folder1\file.png, folder2\folder3\file3.png
after extracting this ZIP file, i only see the folder1*.* and folder2*.* but without the sub directory folder3.
How can i improve it?
I think this PHP manual will be helpful to you
http://php.net/manual/en/ref.zip.php
<?php
$file = "2537c61ef7f47fc3ae919da08bcc1911.zip";
$dir = getcwd();
function Unzip($dir, $file, $destiny="")
{
$dir .= DIRECTORY_SEPARATOR;
$path_file = $dir . $file;
$zip = zip_open($path_file);
$_tmp = array();
$count=0;
if ($zip)
{
while ($zip_entry = zip_read($zip))
{
$_tmp[$count]["filename"] = zip_entry_name($zip_entry);
$_tmp[$count]["stored_filename"] = zip_entry_name($zip_entry);
$_tmp[$count]["size"] = zip_entry_filesize($zip_entry);
$_tmp[$count]["compressed_size"] = zip_entry_compressedsize($zip_entry);
$_tmp[$count]["mtime"] = "";
$_tmp[$count]["comment"] = "";
$_tmp[$count]["folder"] = dirname(zip_entry_name($zip_entry));
$_tmp[$count]["index"] = $count;
$_tmp[$count]["status"] = "ok";
$_tmp[$count]["method"] = zip_entry_compressionmethod($zip_entry);
if (zip_entry_open($zip, $zip_entry, "r"))
{
$buf = zip_entry_read($zip_entry, zip_entry_filesize($zip_entry));
if($destiny)
{
$path_file = str_replace("/",DIRECTORY_SEPARATOR, $destiny . zip_entry_name($zip_entry));
}
else
{
$path_file = str_replace("/",DIRECTORY_SEPARATOR, $dir . zip_entry_name($zip_entry));
}
$new_dir = dirname($path_file);
// Create Recursive Directory (if not exist)
if (!file_exists($new_dir)) {
mkdir($new_dir, 0700);
}
$fp = fopen($dir . zip_entry_name($zip_entry), "w");
fwrite($fp, $buf);
fclose($fp);
zip_entry_close($zip_entry);
}
echo "\n</pre>";
$count++;
}
zip_close($zip);
}
}
Unzip($dir,$file);
?>
This script extracts all files in the zip file recursively. This will extract them into the current directory.
<?php
set_time_limit(0);
echo "HI<br><br>";
//----------------
//UNZIP a zip file
//----------------
$zipfilename = "site.zip";
//----------------
function unzip($file){
$zip=zip_open(realpath(".")."/".$file);
if(!$zip) {return("Unable to proccess file '{$file}'");}
$e='';
while($zip_entry=zip_read($zip)) {
$zdir=dirname(zip_entry_name($zip_entry));
$zname=zip_entry_name($zip_entry);
if(!zip_entry_open($zip,$zip_entry,"r")) {$e.="Unable to proccess file '{$zname}'";continue;}
if(!is_dir($zdir)) mkdirr($zdir,0777);
#print "{$zdir} | {$zname} \n";
$zip_fs=zip_entry_filesize($zip_entry);
if(empty($zip_fs)) continue;
$zz=zip_entry_read($zip_entry,$zip_fs);
$z=fopen($zname,"w");
fwrite($z,$zz);
fclose($z);
zip_entry_close($zip_entry);
}
zip_close($zip);
return($e);
}
function mkdirr($pn,$mode=null) {
if(is_dir($pn)||empty($pn)) return true;
$pn=str_replace(array('/', ''),DIRECTORY_SEPARATOR,$pn);
if(is_file($pn)) {trigger_error('mkdirr() File exists', E_USER_WARNING);return false;}
$next_pathname=substr($pn,0,strrpos($pn,DIRECTORY_SEPARATOR));
if(mkdirr($next_pathname,$mode)) {if(!file_exists($pn)) {return mkdir($pn,$mode);} }
return false;
}
unzip($zipfilename);
?>
Try this simple way:
system('unzip my_zip_file.zip');

Categories