I am trying to download two files from two different FTP servers. One file is 10 MB in size while the other is 3.3GB in size. The 10MB file is downloaded every time without a problem. The 3.3GB file always encounters an error:
Code Error: [2] ftp_get(): Opening BINARY mode data connection for
bigfile.gz (3232089332 bytes).Error/Warning on line 55 in file
script.phpPlease go over the collector code.
The size of the file is exactly 3232089332, so this issue appears after the file has finished downloading completely.
Both files are .gz files (so I know they are binary).
There is enough space on the hard drive (currently free 47GB).
It is worthwhile to note that I am able to download the file without any issues using Filezilla.
Any help would be highly appreciated.
The code is as follows:
function ftpDownload($server, $username, $password, $filename) {
if (strpos($server, '://') !== false) $server = substr($server, strpos($server, '://') + 3);
# set up basic connection
echo "Connecting to $server\n";
$connectionId = ftp_connect($server);
if (!$connectionId) {
return ['success' => false, 'error' => "FTP Connection has Failed"];
}
# login with username and password
echo "Logging in\n";
$loginResult = ftp_login($connectionId, $username, $password);
ftp_pasv($connectionId, true);
# check connection
if (!$loginResult) {
return ['success' => false, 'error' => "Failed to login"];
}
# Verify the file exists
echo "Locating $filename\n";
$result = ftp_size($connectionId, $filename);
if ($result == -1) {
return ['success' => false, 'error' => "Unable to locate $filename in server", 'filename' => false];
}
echo "File size ".number_format(($result / 1024 / 1024))." MBs\n";
# Download the file
echo "Downloading $filename, this may take a while\n";
if (file_exists(__DIR__.'/files/'.$filename)) unlink(__DIR__.'/files/'.$filename);
ftp_pasv($connectionId, true);
$success = ftp_get($connectionId, __DIR__.'/files/'.$filename, $filename, FTP_BINARY);
ftp_close($connectionId);
if ($success == false) {
return ['success' => false, 'error' => "Failed to download file $filename from server, received error from FTP"];
}
if (!file_exists(__DIR__.'/files/'.$filename)) {
return ['success' => false, 'error' => "Unable to locate $filename in server, filename was not properly stored locally"];
}
}
Reached a solution. As the timeout occurred after the file has been fully downloaded, instead of using ftp_get() I used the following:
$fp = fopen($filename, 'w');
#$success = ftp_fget($connectionId, $fp, $filename, FTP_BINARY);
Then to verify check whether the transfer was successful, I just compared the downloaded file's filesize using filesize() with the remote file's filesize which I've obtained beforehand (can be seen in the question's code).
Related
When moving our site to a new host we went from PHP5 (I think) to PHP7. We also added SSL to the site for the first time. Ever since moving the site a function to copy image files to an FTP has been failing randomly.
After doing some research I learned that there is no way to get an error message more detailed that "ftp_put has failed".
$dir = 'path/to/folder';
$a = scandir($dir);
$ftp_server = "ftp.server.com";
$ftp_conn = ftp_connect($ftp_server);
$ftp_username = 'myuser';
$ftp_userpass = 'mypass';
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
ftp_pasv($ftp_conn, true);
foreach ($a as $value) {
if(strlen($value) > 4){
$file = $dir.$value;
$name = $value;
if (ftp_put($ftp_conn, $name, $file, FTP_BINARY)){
echo "<br><br><span style='color: green'>Successfully uploaded $file.</span><br><br>";
}
else{
echo "<br><br><span style='color: green'>Error uploading $file.</span><br><br>";
}
}
}
The output from the code above is:
Successfully uploaded ../../img/bil/AAA123/AAA123-1.jpg.
Successfully uploaded ../../img/bil/AAA123/AAA123-2.jpg.
Successfully uploaded ../../img/bil/AAA123/AAA123-3.jpg.
Error uploading ../../img/bil/AAA123/AAA123-4.jpg.
Error uploading ../../img/bil/AAA123/AAA123-5.jpg.
Successfully uploaded ../../img/bil/AAA123/AAA123-6.jpg.
Error uploading ../../img/bil/AAA123/AAA123-7.jpg.
Successfully uploaded ../../img/bil/AAA123/AAA123-8.jpg.
This output differs, running it again will successfully upload other images and fail other.
I have tried stripping down the code, removing the scandir and foreach parts and using a direct path to one image file as $file with the same result.
I have no idea what could be wrong. I suspect moving to PHP7 and possibly SSL is the problem since this all started then. Not being able to get a detailed error message why ftp_put fails leaves me completely stuck.
Is there anything I can do to find out whats wrong?
edit:
Adding error_reporting(-1) and printing out error_get_last() gives me this:
Array (
[type] => 2
[message] => ftp_put(): Type set to I
[file] => path/to/file.php
[line] => 51
)
Any network communication can fail.
Uploading a large number of files without zipping them up first via FTP is a good way to ensure some of your uploads will fail. If you don't want to zip them up into a single request than a good alternative is to retry failed responses, adding a delay between retries but don't make an infinite loop. Retry 3 times and if it still fails than your problem is much bigger than minor network issues or FTP server bugs and you are better off skipping the file and trying the next. Also don't forget to close any connection you open.
$dir = 'path/to/folder';
$a = scandir($dir);
foreach ($a as $value) {
if(strlen($value) > 4){
for($retry = 0; $retry < 3; $retry++) {
$ftp_server = "ftp.server.com";
$ftp_conn = ftp_connect($ftp_server);
$ftp_username = 'myuser';
$ftp_userpass = 'mypass';
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
ftp_pasv($ftp_conn, true);
$file = $dir.$value;
$name = $value;
if (ftp_put($ftp_conn, $name, $file, FTP_BINARY)){
echo "<br><br><span style='color: green'>Successfully uploaded $file.</span><br><br>";
break;
} else {
if ($retry < 2) {
echo "<br><br><span style='color: green'>Error uploading $file Will retry....</span><br><br>";
sleep(2);
} else {
echo "<br><br><span style='color: green'>Error uploading $file.</span><br><br>";
}
}
ftp_close($ftp_conn);
}
}
}
I'm trying to upload multiple files to a SFTP site from a local directory.
I can get it to work for a single file but I would like to be able to upload multiple files which have variables names too.
$localFile_xml = "C:\xml\Race_" . $value;
chdir($localFile_xml);
//This successfully lists the files
foreach (glob("*.xml") as $filename) {
echo "$filename size " . filesize($filename) . "\n";
}
$remote_XMLfiles = "/FTP/XML/Race_" . $value;
$xmlstream = fopen("ssh2.sftp://$sftp" . $remote_XMLfiles, 'w');
foreach (glob("*.xml") as $filename) {
$xmlfile = file_get_contents($localFile_xml);
fwrite($xmlstream, $xmlfile);
fclose($xmlstream);
}
I believe its there but I cannot get the last bit correct.
Thank you so much in advance
Assuming the remote SSH connection is valid, and that the method you used in your question works for single files, I believe your order of operations needs to be corrected.
As mentioned in my comments, your code appears to be attempting to use file_get_contents on a local directory, which is not permitted. It also appears you attempted the same on the $xmlstream, which must be performed per file, rather than directory. Assuming 'C:\\xml\\Race_' . $value; is a directory like C:\\xml\\Race_1 and not a file.
Some minor issues for validation of resources and Windows specific issues that need to be addressed:
Windows Directory Separators should be written as \\ (even when
using single quotes), since \ is an escape sequence
it causes \x \t \n \r \' \" \\ to be treated as special characters.
When using fopen($path, $mode) it is recommended to specify the b flag as the last character of the mode, to ensure the file is
binary-safe (verbatim) and to avoid ambiguity between operating systems. Alternatively on Windows specify the t mode to transparently translate \n to \r\n (only desirable for plain-text files).
$mode = 'rb' (binary-safe read)
$mode = 'rt' (text-mode translation read)
When working with networking streams, it is recommended to test that the stream has successfully written all of its content. I provided the fwrite_stream function below from the PHP manual.
Example
try {
//--- example purposes only ---
//use your own ssh2_connnect, ssh2_auth_password, ssh2_sftp
if (!$ssh2 = ssh2_connect('192.168.56.101', 22)) {
throw new \RuntimeException('Unable to connect to remote host');
}
if (!ssh2_auth_password($ssh2, 'root', '')) {
throw new \RuntimeException('Unable to Authenticate');
}
if (!$sftp = ssh2_sftp($ssh2)) {
throw new \RuntimeException('Unable to initialize SFTP');
}
$value = '1';
//--- end example purposes only ---
$localFile_xml = 'C:\\xml\\Race_' . $value;
if (!$localFile_xml || !is_dir($localFile_xml)) {
throw new \RuntimeException('Unable to retrieve local directory');
}
//retrieve list of XML files
$iterator = new \GlobIterator($localFile_xml . '/*.xml',
\FilesystemIterator::KEY_AS_PATHNAME |
\FilesystemIterator::CURRENT_AS_FILEINFO |
\FilesystemIterator::SKIP_DOTS
);
if (!$iterator->count()) {
throw new \RuntimeException('Unable to retrieve local files');
}
$success = [];
$remote_XMLfiles = '/FTP/XML/Race_' . $value;
$remote_XMLpath = "ssh2.sftp://$sftp" . $remote_XMLfiles;
//ensure the remote directory exists
if (!#mkdir($remote_XMLpath, 0777, true) && !is_dir($remote_XMLpath)) {
throw new \RuntimeException(sprintf('Unable to create remote directory "%s"', $remote_XMLpath));
}
/**
* #var string $filepath
* #var \SplFileInfo $fileinfo
*/
foreach ($iterator as $filepath => $fileinfo) {
$filesize = $fileinfo->getSize();
printf("%s size %d\n", $filepath, $filesize);
try {
//open local file resource for binary-safe reading
$xmlObj = $fileinfo->openFile('rb');
//retrieve entire file contents
if (!$xmlData = $xmlObj->fread($filesize)) {
//do not permit empty files
printf("No data found for \"%s\"\n", $filepath);
continue;
}
} finally {
//shortcut to close the opened local file resource on success or fail
$xmlObj = null;
unset($xmlObj);
}
try {
$remote_filepath = $remote_XMLpath . '/' . $fileinfo->getBasename();
//open a remote file resource for binary-safe writing
//using current filename, overwriting the file if it already exists
if (!$xmlstream = fopen($remote_filepath, 'wb')) {
throw new \RuntimeException(sprintf('Unable to create remote file "%s"', $remote_filepath));
}
//write the local file data to the remote file stream
if (false !== ($bytes = fwrite_stream($xmlstream, $xmlData))) {
$success[] = [
'filepath' => $filepath,
'remote_filepath' => $remote_filepath,
'bytes' => $bytes,
];
}
} finally {
//shortcut to ensure the xmlstream is closed on success or failure
if (isset($xmlstream) && is_resource($xmlstream)) {
fclose($xmlstream);
}
}
}
//testing purposes only to show the resulting uploads
if (!empty($success)) {
var_export($success);
}
} finally {
//shortcut to disconnect the ssh2 session on success or failure
$sftp = null;
unset($sftp);
if (isset($ssh2) && is_resource($ssh2)) {
ssh2_disconnect($ssh2);
}
}
/*
* Taken from PHP Manual
* Writing to a network stream may end before the whole string is written.
* Return value of fwrite() may be checked
*/
function fwrite_stream($fp, $string)
{
for ($written = 0, $writtenMax = strlen($string); $written < $writtenMax; $written += $fwrite) {
$fwrite = fwrite($fp, substr($string, $written));
if (false === $fwrite) {
return $written;
}
}
return $written;
}
NOTE
All file operations will be created using the ssh2_auth_password
user as the owner/group. You must ensure the specified user has read
and write access to the desired directories.
Use the appropriate file masks to ensure desired file/directory permissions
0777 (default) allows everyone to read, write, execute!
0750 is typically desired for directories
0640 is typically desired for individual files
use chmod($path, 0750) to change permissions on the remote file(s)
use chown($path, 'user') to change the owner on the remote file(s)
use chgrp($path, 'group') to change the group on the remote file(s)
Result
C:\xml\Race_1\file1.xml size 9
C:\xml\Race_1\file2.xml size 11
array (
0 =>
array (
'filepath' => 'C:\\xml\\Race_1\\file1.xml',
'remote_filepath' => 'ssh2.sftp://Resource id #5/FTP/XML/Race_1/file1.xml',
'bytes' => 9,
),
1 =>
array (
'filepath' => 'C:\\xml\\Race_1\\file2.xml',
'remote_filepath' => 'ssh2.sftp://Resource id #5/FTP/XML/Race_1/file2.xml',
'bytes' => 11,
),
)
I am working on to upload the file to FTP using PHP FTP. while putting the file to the server, its throw error.
what I did:
$ftp_conn = ftp_connect(SAP_SERVER_HOST, SAP_SERVER_PORT, 60);
if (!ftp_login($ftp_conn, SAP_SERVER_USER, SAP_SERVER_PASSWORD)) {
echo 'not connected<br/>';
} else {
$localfile = '/abc/txt/15375127769260.txt';
$serverfile = '/folder/15375127769260.txt';
// echo ftp_pwd($ftp_conn);
if (ftp_put($ftp_conn, $serverfile, $localfile, FTP_BINARY)) {
echo "Successfully uploaded $localfile.";
} else {
echo "Error uploading $localfile.";
}
// close connection
ftp_close($ftp_conn);
}
Suggest Me, what I miss in this code.
For anyone stumbling upon this:
my file was sent correctly after adding ftp_pasv($conn_id, true);
Note that it must be added after ftp_login().
ftp-pasv on php.net
are you using right folders and ports ?
$ftp_conn = ftp_connect(SAP_SERVER_HOST, SAP_SERVER_PORT, 60);
it should be port 21
and in local file you must get get the realpath of the file whit realpath() function
and for remote server the path is based on ftp base folder
Take a look of realpath http://php.net/manual/pt_BR/function.realpath.php
I am currently uploading files to a third party server through SFTP by using PHP ssh2
The connection are working fine and I tried to upload files using fwrite
$resourceId = $this->sftp;
$intResourceId = intval($resourceId);
$dir = "ssh2.sftp://$intResourceId/./".$this->un."/".$file_name;
$stream = #fopen($dir, 'w');
if (!$stream) {
return "Could not open file: $file_name";
}
$data_to_send = #file_get_contents(LOCAL_FILE_DIR.$file_name,true);
if ($data_to_send === false){
return "Could not open local file: $file_name.";
}
if (#fwrite($stream, $data_to_send) === false){
return "Could not send data from file: $file_name.";
}
/*
readdir():
Array
(
[0] => 1810171932.txt
[1] => response_dir
)
*/
The files were uploaded when I read the directory but after a few seconds it was removed from the directory.
/*
readdir():
Array
(
[0] => response_dir
)
*/
Any idea why? Does the file being uploaded or there are some permission issue which I need to take care of?
Thanks in advance.
On a server with PHP 5.2.17 running, using any function which makes use of the built in ftp wrapper to upload a file, creates an empty file on the server:
file_put_contents() returns with the accurate number of bytes
copy() also returns with true
Both create the file, but it's empty.
When trying with ftp_put() from the FTP extension, both in binary and ascii mode, it works well.
On my workstation with PHP 5.3.10 it somehow works also with the wrapper.
In code:
$source = '/tmp/testfile';
$target = 'ftp://user:pass#example.com/testfile';
copy($source, $target);
gives no error or warning, but leaves an empty file on the server.
$source = '/tmp/testfile';
$target = 'testfile';
$ftp = ftp_connect('example.com');
ftp_login($ftp, 'user', 'pass');
ftp_put($ftp, $target, $source, FTP_ASCII);
ftp_close($ftp);
works in every respect.
thanks for any suggestion!
Have you tried SSH2 libs? Some sample implementation below:
public function uploadSFTP($host_name, $port, $user, $publicSshKeyPath, $privateSshKeyPath, $remoteFile, $localFile, $fileOperation = 'w')
{
$ssh_conn = ssh2_connect($host_name, $port);
if (ssh2_auth_pubkey_file($ssh_conn, $user, $publicSshKeyPath, $privateSshKeyPath))
{
$sftp_conn = ssh2_sftp($ssh_conn);
$inputfileStream = #fopen('ssh2.sftp://' . $sftp_conn . $remoteFile, $fileOperation);
try
{
if (!$inputfileStream)
throw new Exception('Could open remote file for writing: ' . $remoteFile);
$localFileContents = #file_get_contents($localFile);
if ($localFileContents === FALSE)
throw new Exception('Could not open local file for reading :' . $localFile);
if (#fwrite($inputfileStream, $localFileContents) === FALSE)
throw new Exception('Could not SFTP file');
}
catch (Exception $e)
{
// Do something...
}
fclose($sftpInfileStream);
}
}