To populate an sql database I upload .txt files with all data for each field and rows. When my internet connection is bad i have a lot of problems and sometimes process is aborted.
To fix the problem I'd upload .txt files in a compressed format(.zip or if possible also other formats: .rar ...etc). In this way, filesize is 15 times smaller and i can upload it easier and faster.
How can i do?
Here are codes i use at the moment:
upload.php
...
<form action="readfile.php" enctype="multipart/form-data" method="post">
<br>Filename: <input name="userfile" type="file">
<p><input type="reset" value="Reset">
<input type="submit" value="Upload">
</form>
...
readfile.php
...
// Read data posted from form
$browser_name = $_FILES['userfile']['name'];
$temp_name = $_FILES['userfile']['tmp_name'];
$usrname = $_POST['uname'];
// Connect to the database
if (!($connection = # mysql_connect ($hostName,
$username,
$password)))
die ("Could not connect to database");
if (!mysql_select_db ($databaseName, $connection))
showerror();
echo "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\"\"http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd\">";
echo "<html>";
echo "<head>";
echo "<title>Upload file</title>";
echo "</head>";
echo "<body>";
// Was a file uploaded?
if (is_uploaded_file ($temp_name))
{
echo "<h1>File Upload $browser_name</h1>";
echo "<p>Filename - $browser_name\n";
echo "<br>User name - $uname\n";
// Open the file
if (!($file = fopen ($temp_name, "r")))
die ("Could not open the file $file\n");
// Read the first line
$string = fgets ($file, 1024);
...
Where to modify and/or add codes?
If you need more details, please ask me and I paste more: I presumed they're the right lines to explain better everything and to modify for my needs.
Thank you so much in advance for your help.
Mattew
UPDATES:
I made some new modifies using ZipArchive. In a few words now the .zip file is uploaded in a folder and unzipped. So the nomefile.txt file can be read and processed as a normal .txt file inside the specific folder.
Unfortunately the fopen looks for a specific filename. My idea is now to rename any unzipped file (having a random name but .txt extension) to a specific file (for example myfile.txt). In this way i can fix also problems with "different" extensions having small/capitol letters such as .txt and .TXT
HERE IS THE CODE I MENTIONED BEFORE:
...
//unzip del file
$zip = new ZipArchive;
$zip->open('file.zip');
$zip->extractTo('./');
$zip->close();
echo "File unzipped and ready to be processed";
if (!($file = fopen ("namefile.txt", "r")))
die ("Could not open the file $file\n");
// Read the first line
$string = fgets ($file, 1024);
....
Opinions? Helps?
Thanks in advance, I'm very new of php and maybe my problems are elementary, excuse me.
Bye.
Use the ZIP Wrapper:
$filedata = file_get_contents("zip://absolutepathtozip.zip#pathintozip/file.txt");
The approach differs with the compression mechanism you use, since the available libraries all use different interfaces; Exemplary, I'll show bzip2 here since that seems to be the simplest to me. Locally, just execute bzip2 <file> on any *ix command line shell; on Windows you could use e.g. 7-Zip for creating a .bz2 file.
In your php, you will want to read the whole file content instead of single lines (single lines don't make much sense in a compressed file). So you'll have to change your code starting from opening the files:
/*
// Open the file
if (!($file = fopen ($temp_name, "r")))
die ("Could not open the file $file\n");
// Read the first line
$string = fgets ($file, 1024);
*/
$compressed_filecontent = file-get-contents($temp_name);
$filecontent = bzdecompress($compressed_filecontent);
...
// instead of reading from the file line by line,
// you'd then have to split up the $filecontent variable
// in separate lines and work on them
Note: of the "compression" formats where there is a PHP library available for, only LZF is similarly simple to use (you'd just have to replace bzdecompress with lzf-decompress); that is because the other formats (like ZIP and RAR) are not only compression formats but also archiving formats (meaning that such files can hold multiple compressed files). That makes reading them a little harder - see the documentation for more info.
Related
I have cleanup script, which move the XLS files from one place to another. for this file moving process, I have used the rename function. This script is working fine. but when the XLS file is open, when I try to move that xls, I am getting error which simply say Can not rename sample.xls. But I would like to add the functionality like, Check the XLS is open before initiate rename function.
I believe this is function call flock but this is applicable for TXT file alone.
How to check XLS file is opened before call the rename function.
One simple thing you could try is to use flock to acquire a Exclusive Lock on the file and if it fails you will know the file is being used:
<?php
$fp = fopen('c:/your_file.xlsx', 'r+');
if(!flock($fp, LOCK_EX))
{
echo 'File is being used...';
exit(-1);
}
else
{
fclose($fp);
// rename(...);
}
An alternative would be to check the existence of the locking file excel usually creates when a file is being used:
<?php
$file = 'c:/testfile.xlsx';
$lock = 'c:/~$testfile.xlsx';
if (file_exists($lock))
{
echo "Excel $file is locked.";
}
else
{
echo "Excel $file is free.";
}
The hidden file is usually name with the prefix ~$ as for old excel files I believe 2003 and older the lock files are saved on the temp folder with a random name like ~DF7B32A4D388B5854C.TMP so it would be pretty hard to find out.
You should use flock(). This puts a flag on the file so that other scripts are informed that the file is in use. The flag is turned off either intentionally using fclose or implicitly by the end of the script.
Use file lock like:
flock($file,LOCK_EX);
see this
I have a BASE64 string of a zip file that contains one single XML file.
Any ideas on how I could get the contents of the XML file without having to deal with files on the disk?
I would like very much to keep the whole process in the memory as the XML only has 1-5k.
It would be annoying to have to write the zip, extract the XML and then load it up and delete everything.
I had a similar problem, I ended up doing it manually.
https://www.pkware.com/documents/casestudies/APPNOTE.TXT
This extracts a single file (just the first one), no error/crc checks, assumes deflate was used.
// zip in a string
$data = file_get_contents('test.zip');
// magic
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$raw = gzinflate(substr($data,30+$head['namelen']+$head['exlen'],$head['csize']));
// first file uncompressed and ready to use
file_put_contents($filename,$raw);
After some hours of research I think it's surprisingly not possible do handle a zip without a temporary file:
The first try with php://memory will not work, beacuse it's a stream that cannot be read by functions like file_get_contents() or ZipArchive::open(). In the comments is a link to the php-bugtracker for the lack of documentation of this problem.
There is a stream support ZipArchive with ::getStream() but as stated in the manual, it only supports reading operation on an opened file. So you cannot build a archive on-the-fly with that.
The zip:// wrapper is also read-only: Create ZIP file with fopen() wrapper
I also did some attempts with the other php wrappers/protocolls like
file_get_contents("zip://data://text/plain;base64,{$base64_string}#test.txt")
$zip->open("php://filter/read=convert.base64-decode/resource={$base64_string}")
$zip->open("php://filter/read=/resource=php://memory")
but for me they don't work at all, even if there are examples like that in the manual. So you have to swallow the pill and create a temporary file.
Original Answer:
This is just the way of temporary storing. I hope you manage the zip handling and parsing of xml on your own.
Use the php php://memory (doc) wrapper. Be aware, that this is only usefull for small files, because its stored in the memory - obviously. Otherwise use php://temp instead.
<?php
// the decoded content of your zip file
$text = 'base64 _decoded_ zip content';
// this will empty the memory and appen your zip content
$written = file_put_contents('php://memory', $text);
// bytes written to memory
var_dump($written);
// new instance of the ZipArchive
$zip = new ZipArchive;
// success of the archive reading
var_dump(true === $zip->open('php://memory'));
toster-cx had it right,you should award him the points, this is an example where the zip comes from a soap response as a byte array (binary), the content is an XML file:
$objResponse = $objClient->__soapCall("sendBill",array(parameters));
$fileData=unzipByteArray($objResponse->applicationResponse);
header("Content-type: text/xml");
echo $fileData;
function unzipByteArray($data){
/*this firts is a directory*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$if=30+$head['namelen']+$head['exlen']+$head['csize'];
/*this second is the actua file*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,$if,30));
$raw = gzinflate(substr($data,$if+$head['namelen']+$head['exlen']+30,$head['csize']));
/*you can create a loop and continue decompressing more files if the were*/
return $raw;
}
If you know the file name inside the .zip, just do this:
<?php
$xml = file_get_contents('zip://./your-zip.zip#your-file.xml');
If you have a plain string, just do this:
<?php
$xml = file_get_contents('compress.zlib://data://text/plain;base64,'.$base64_encoded_string);
[edit] Documentation is there: http://www.php.net/manual/en/wrappers.php
From the comments: if you don't have a base64 encoded string, you need to urlencode() it before using the data:// wrapper.
<?php
$xml = file_get_contents('compress.zlib://data://text/plain,'.urlencode($text));
[edit 2] Even if you already found a solution with a file, there's a solution (to test) I didn't see in your answer:
<?php
$zip = new ZipArchive;
$zip->open('data::text/plain,'.urlencode($base64_decoded_string));
$zip2 = new ZipArchive;
$zip2->open('data::text/plain;base64,'.urlencode($base64_string));
If you are running on Linux and have administration of the system. You could mount a small ramdisk using tmpfs, the standard file_get / put and ZipArchive functions will then work, except it does not write to disk, it writes to memory.
To have it permanently ready, the fstab is something like:
/media/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=2M 0 0
Set your size and location accordingly so it suits you.
Using php to mount a ramdisk and remove it after using it (if it even has the privileges) is probably less efficient than just writing to disk, unless you have a massive number of files to process in one go.
Although this is not a pure php solution, nor is it portable.
You will still need to remove the "files" after use, or have the OS clean up old files.
They will of coarse not persist over reboots or remounts of the ramdisk.
if you want to read the content of a file from zip like and xml inside you shoud look at this i use it to count words from docx (wich is a zip )
if (!function_exists('docx_word_count')) {
function docx_word_count($filename)
{
$zip = new ZipArchive();
if ($zip->open($filename) === true) {
if (($index = $zip->locateName('docProps/app.xml')) !== false) {
$data = $zip->getFromIndex($index);
$zip->close();
$xml = new SimpleXMLElement($data);
return $xml->Words;
}
$zip->close();
}
return 0;
}
}
The idea comes from toster-cx is pretty useful to approach malformed zip files too!
I had one with missing data in the header, so I had to extract the central directory file header by using his method:
$CDFHoffset = strpos( $zipFile, "\x50\x4b\x01\x02" );
$CDFH = unpack( "Vsig/vverby/vverex/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr( $zipFile, $CDFHoffset, 46 ) );
I have a processing server with my database and a serving database to serve up files with a low bandwidth cost. On the processing server, php is not able to create files so everything must be done with streams and/or stay in memory before being sent over to another server for download. A few days ago I found out about the stream abstraction with 'php://memory' and that I can do something like
$fp=fopen('php://memory','w+');
fwrite($fp,"Hello world");
fseek($fp,0,SEEK_SET);
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.txt",$fp,FTP_BINARY);
to make the file in memory and then allow me to ftp it over to my other server. This is exactly what I want, except I also want to zip the data before sending it -- preferably using only native parts of php like ziparchive and not additional custom classes for special stream manipulation. I know that I am very close with the following...
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$fp = $zip->getStream('test'); if(!$fp) print "no filepointer";
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.zip",$fp,FTP_BINARY);
} else print "couldn't open a zip like that";
The point at which this fails is the call to getStream (which always returns false although I think I am using correctly). It appears that the zip is fine making the file in 'php://memory' but for some reason getStream still fails although perhaps I don't sufficiently understand how ZipArchive makes zips...
How can I go from the string to the zipped filepointer so that I can ftp the zip over to my other server? Remember I can't make any files or else I would just make the zip file then ftp it over.
EDIT: based on skinnynerd's suggestions below I tried the following
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
$fp = fopen('php://memory','r+');
fseek($fp,0,SEEK_SET);
//connect to ftp
$upload = ftp_fput($conn_id,"upload/transfer/helloworld.zip",$fp,FTP_BINARY);
}
This does make a zip and send it over but the zip is 0 bytes large so I don't think that 'php://memory' works the way I thought... it actually fails at the close step -- the $zip->close() returns false which makes me wonder if I can open zips into 'php://memory' at all. Does anyone know what I can try along these line to get the zip?
$zip->getStream('test') is getting a stream to extract the file 'test' from the archive. Since there's no file 'test' in the archive, this fails. This is not the function you want to use.
As you said, what you want to do is send the finished archive to the ftp server. In this case, you would want to close the zip archive, and then reopen php://memory as a normal file (using fopen) to send it.
I don't know, but you may also be able to use $zip as a resource directly, without having to close and reopen the file.
And I think you can try create a stream pipe directly from ftp server
<?php
$zip = new ZipArchive();
if($zip->open('ftp://user:password#ftp.host.com/upload/transfer/helloworld.zip', ZipArchive::CREATE))
{
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
}
else
print "couldn't open zip file on remote ftp host.";
Does it have to be a Zip archive? Since you're trying to save bandwith it could be a gzip too.
<?php
$ftp_credentials = "ftp://USER:PASSWORD#HOST/helloworld.gz";
$gz = gzencode("Hello World!", 9);
$options = array('ftp' => array('overwrite' => true));
$stream_context = stream_context_create($options);
file_put_contents($ftp_credentials, $gz, 0, $stream_context);
?>
I need to read a list of CSV files from an FTP and delete them after I successfully read them.
Until now, i opened the csv file using fopen into a resource and then used fgetcsv to read the csv lines from it.
$res = fopen($url);
while ($csv_row = fgetcsv($res, null, self::DELIMITER)) {
.....
}
The problem is that I need to read a list of csv files and delete them too. the ftp_get function save the file into a local file. I rather avoid that. any way I can keep using the fgetcsv function with the ftp_nlist & ftp_connect functions? ?
You can save the csv file to a temporary file stream using ftp_fget(). This allows you to avoid the "create-read-delete" cycle. Once you close the file stream it's like it magically never existed :)
$ftp_handle = ftp_connect($ftp_server);
$remote_path = "/path/to/file.csv";
$tmp_handle = fopen('php://temp', 'r+');
if (ftp_fget($ftp_handle, $tmp_handle, $remote_path, FTP_ASCII)) {
rewind($tmp_handle);
while ($csv_row = fgetcsv($tmp_handle)) {
// do stuff
}
}
fclose($tmp_handle);
If you wanted to loop over a directory of files just get the list of files and then put the above code in a loop.
I am learning php, trying to use the fopen() function.
The php file I am coding is in this directory /domains/xxxxx.com.au/public_html/phpfile.php
What path do I specify for the file to be opened, the example I am looking at is based on a server on a pc where this is the file path $filename = "c:/newfile.txt"; not an online server.
UPDATE!
This is the whole script, I have the file location correct, now the4 script is returning "couldnt create the file" does this have something to do with ther permission of the folder location of the file?
<?php
$filename = "/domains/xxxxxxxx.com.au/public_html/newfile.txt";
$newfile = #fopen($filename, "w+") or die ("couldnt create the file");
fclose($newfile);
$msg = "<p>File Created</p>";
?>
<HTML>
<HEAD>
</HEAD>
<BODY>
<? echo "$msg" ?>
</BODY>
</HTML>
Assuming that your php file is inside the public_html too, you could use :
$fp = fopen( "newfile.txt", "rt" );
Or, giving the full path :
$fp = fopen( "/domains/xxxxx.com.au/public_html/newfile.txt", "rt" );
This will open it if it already exists.
Refer to this for further details of opening flags.
UPDATE:
You can even use the is_writable/is_readable function to check file access before trying to open it.
Read http://us2.php.net/manual/en/function.fopen.php Example #1 is relevant for a Unix system.