Cannot connect to external FTPS server with php - php

I need to extract data from a .gz file which is located on an ftp server. The FTP server works with IP white listing, so I submitted my local IP and my website IP (i used gethostbynameto get the IP) which were both approved.
Locally, I can run this code to reach the file:
$url="ftp://username:password#host/targetfile.gz";
$xml = simplexml_load_file("compress.zlib://$url") or die ("Cannot load file");
This runs perfectly fine and it allows me to extract data from the XML file.
When I run the script on my server however, i'm not getting a connection. I contacted the admin running the ftp server and they told me they only allow FTPS connections.
So, I proceeded with the following code to try and establish a connection:
$conn_id = ftp_ssl_connect($ftp_server);
// login with username and password
$login_result = ftp_login($conn_id, $ftp_user, $ftp_pass);
if (!$login_result) {
die("Cannot login.");
}
echo ftp_pwd($conn_id); // /
// close the ssl connection
ftp_close($conn_id);
This also doesn't connect. I'm new to FTP connections with PHP and FTP in general and i've got no clue on how to proceed from this point.
OpenSSL is enabled.
Can anyone point me in the right direction?
Edit: These are the error messages:
When I use the top block of code, this occurs:
Warning: simplexml_load_file(): I/O warning : failed to load external
entity
"compress.zlib://username:password#ftp.thehost.com/targetfile.gz"; in
/path/to/my/website/folder/htdocs/mydomain.com/feeds/script.php on
line 13
When I use the bottom block of code, I get this error:
Warning: ftp_login() expects parameter 1 to be resource, boolean given
in /path/to/my/website/folder/htdocs/mydomain.com/feeds/script.php on
line 16 Cannot login.
I do not have access to the log file afaik

I do not think there's much we can help you here.
You do not have a network connectivity between the web server and FTP server.
Note that it does not have to be because the web server has not been white-listed on the FTP server. It's also possible that the web server does not allow outgoing connections.

Related

Unable to run FTP commands from AWS AMI

I want to connect to an FTP using PHP to upload the reports generated. As per the remote server, the FTP needs to be in ACTIVE mode.
this is my code:
ini_set('display_errors', '1');
error_reporting(E_ALL);
$conn_id = ftp_connect('myftpserver.com', 21);
if($conn_id)
{
// login with username and password
$login_result = ftp_login($conn_id, 'mysuer', 'password');
$passive = ftp_pasv($conn_id,FALSE);
echo "is active?<br/>";
var_dump($passive);
echo 'Login Result:';
var_dump($login_result);
$files_list = ftp_nlist($conn_id, '/MyFolder/');
echo "<br/>files list ";
var_dump($files_list);
}
else
{
var_dump('Unable to connect to FTP Server');
}
When I am running it from the local machine or a normal shared server, I am able fetch the list, but I am unable to run the script from my AWS AMI instance. For testing purpose, I have even opened all inbound traffic too. Still no luck. Also, the point is that if I try with some other ftp details, I am able to get the response of ftp_nlist. But not for this one. I tried it on 3 AWS instances till yet. Yielded the same result.
All I can say is that this is somewhere the issue at my server security group/firewall. But unable to figure it out. Please help.
The response I get from the server:
is active
bool(true)
Login Result:bool(true)
files list bool(false)
Using FTP Active Mode is problematic with AWS Security Groups.
For active mode to work, you will have to open all inbound ports above 1023. If your client supports restricting the range, do so. You will also need to open both port 20 and port 21 inbound and outbound.
The problem is that the FTP client selects a port that it will listen on. Then the FTP client informs the FTP Server of this port number. The FTP Server then connects to this port. This goes against normal AWS Security Group designs meaning only allow specific ports to be open. You can verify this by opening all ports temporarily, test your FTP client and then closing all the ports.
Active Mode is not secure for the FTP client. Passive Mode is not secure for the FTP server (but the better choice).
NOTE: Rotate your FTP credentials often. Your login and password is sent in the clear and are not encrypted.
FTP is a legacy technology, which is still very popular, that should be stored away in the attic.

Failed to open stream: FTP server reports STOR FTP wrapper

copy(ftp://...#video.calkinsftp.com/videodrive/video_input/Philly/Council Rock South High Schoolers hold mock presidential debate.mp4): failed to open stream: FTP server reports STOR, file: /var/www/html/user-controller.php, line: 138
The file transfers successfully my own network. But it does not work on clients network.
The error message is nonsense, clearly a bug in PHP code.
I have reported this:
Bug #73457 Wrong error message when fopen FTP wrapped fails to open data connection.
Anyway, the root cause is most probably that an FTP data connection cannot be opened.
Most typical cause of the problem is that PHP defaults to the active mode. And in 99% cases, one has to switch to the passive mode, to make the transfer working. Use the ftp_pasv function.
$conn_id = ftp_connect($ftp_server);
// login with username and password
$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass);
// turn passive mode on
ftp_pasv($conn_id, true);
See also:
PHP ftp_put fails with "Warning: ftp_put (): PORT command successful";
my article on the active and passive FTP connection modes.
Please check FTP user authentication and user permission, and check allow particular user uploading file size and your network configuration(NAT/firewall).

Why can't I access BoM's FTP server with PHP?

I'm trying to copy an XML file from the Bureau of Meteorology (Australian) Public Access Data Feeds with PHP to my server. I can open the file in the browser but I can't seem to touch it with PHP using CURL, FTP or simplexml_load_file. I've even tried to copy it with wget and I can't.
Full URL: ftp://ftp2.bom.gov.au/anon/gen/fwo/IDD10150.xml
// connect and login to FTP server
$ftp_username = "anonymous";
$ftp_userpass = "guest";
$ftp_server = "ftp2.bom.gov.au";
$ftp_conn = ftp_connect($ftp_server) or die("Could not connect to $ftp_server");
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
$local_file = "IDD10150.xml";
$server_file = "/anon/gen/fwo/IDD10150.xml";
// download server file
if (ftp_get($ftp_conn, $local_file, $server_file, FTP_ASCII))
{
echo "Successfully written to $local_file.";
}
else
{
echo "Error downloading $server_file.";
}
// close connection
ftp_close($ftp_conn);
Produces the following error
Warning: ftp_get(): Failed to establish connection...
Error downloading /anon/gen/fwo/IDD10150.xml.
Edit: Below is the update code as per suggestions below and the current error messages.
// connect and login to FTP server
$ftp_username = "anonymous";
$ftp_userpass = "guest";
$ftp_server = "ftp2.bom.gov.au";
$ftp_conn = ftp_connect($ftp_server) or die("Could not connect to $ftp_server");
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
ftp_pasv($ftp_conn , TRUE);
$local_file = "IDD10150.xml";
$server_file = "/anon/gen/fwo/";
// download server file
if (ftp_get($ftp_conn, $local_file, $server_file, FTP_BINARY))
{
echo "Successfully written to $local_file.";
}
else
{
echo "Error downloading $server_file.";
}
// close connection
ftp_close($ftp_conn);
Produces the following error
Warning: ftp_get(): php_connect_nonb() failed: Operation now in progress (115) in...
Warning: ftp_get(): Switching to Binary mode. in...
Error downloading /anon/gen/fwo/.
Edit 2: Access via SSH
Last login: Mon Aug 3 11:25:27 on ttys000
MacBook-Pro:~ me$ ssh mysite.com
me#mysite.com's password:
Last login: Mon Aug 3 11:27:37 2015 from IP
me#mysite.com [~]# ftp ftp.bom.gov.au
Connected to ftp.bom.gov.au (134.178.253.145).
220-Welcome to the Bureau of Meteorology FTP service.
220-
220- Disclaimer
220-
220-You accept all risks and responsibility for losses, damages, costs and
220-other consequences resulting directly or indirectly from using this site and
220-any information or material available from it.
220-
220-To the maximum permitted by law, the Bureau of Meteorology excludes all
220-liability to any person arising directly or indirectly from using this
220-site and any information or material available from it.
220-
220-Always Check the Information
220-
220-Information at this site:
220-
220-. is general information provided as part of the Bureau of Meteorology's
220- statutory role in the dissemination of information relating to
220- meteorology.
220-. is subject to the uncertainties of scientific and technical research
220-. may not be accurate, current or complete
220-. is subject to change without notice
220-. is not a substitute for independent professional advice and users
220- should obtain any appropriate professional advice relevant to their
220- particular circumstances
220-. the material on this web site may include the views or recommendations
220- of third parties, which do not necessarily reflect the views of the
220- Bureau of Meteorology or indicate its commitment to a particular course of
220- action.
220
Name (ftp.bom.gov.au:samw): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> get
(remote-file) /anon/gen/fwo/IDQ13015.xml
(local-file) test.xml
local: test.xml remote: /anon/gen/fwo/IDQ13015.xml
227 Entering Passive Mode (134,178,253,145,77,229).
ftp: connect: Connection timed out
Edit 3 & Reason
Ended up being the server Firewall blocking unknown outgoing connections which was found out by contacting BoM and working my server administrator. The BoM IP 134.178.253.145 was added and all was good.
Your server is not in a passive mode, and add this code to process:
ftp_pasv($ftp, true);
For more information look at passive mode on php.net: http://php.net/manual/en/function.ftp-pasv.php Passive mode uses the data initiated by the client rather than the server. So this is why you can't put on server. If this is not set it will fail.
NOTE: Set ftp_pasv() function after ftp_login() function.
Update
Change from ftp_pasv($ftp, true); to ftp_pasv($ftp_conn, true);
First you need to use the passive mode (as suggested by the other answers):
ftp_pasv($ftp_conn, true);
It's unlikely that you succeed to connect in the default active mode as there's typically a firewall between your webserver and the FTP server, which won't allow connections from the FTP server back to your webserver.
See my article on the FTP active/passive connection modes for details.
Regarding the "Operation now in progress (115)" error you get, when using the passive mode.
From the connect man page:
If the connection cannot be established immediately and O_NONBLOCK is set for the file descriptor for the socket, connect() shall fail and set errno to [EINPROGRESS], but the connection request shall not be aborted, and the connection shall be established asynchronously. Subsequent calls to connect() for the same socket, before the connection is established, shall fail and set errno to [EALREADY].
See also TCP Connect error 115 Operation in Progress What is the Cause?
My guess is that the underlying problem is the firewall again, that won't allow the data transfer connection.
If you have a shell access to the webserver, try to connect with the command-line ftp client to verify.
Now that you have tried using the command-line ftp:
As you have verified yourself, you cannot connect even from command-line ftp. There's nothing wrong with your PHP code. As such, your question was resolved from a programmer's point of view. The issue is with the network. What is off-topic on Stack Overflow.
Contact your server administrator. Or consider using the SFTP, if you have the option. It should not suffer from these kind of problems.
ftp_pasv($ftp_conn , TRUE);
ftp_pasv() in w3schools
Edit 01
$local_file = fopen("IDD10150.xml",'w');

Why can I obtain file size and the time stamp with FTP but not be able to retrieve the actual file?

I've been working on this for about 20 hours now and I need some major help. I'm able to get the file size and the timestamp on the file but I am unable to actually obtain the data.
The server I'm trying to get the data from requires FTP over explicit TLS
I'm receiving the same error(s) with both FTP_BINARY and FTP_ASCII in the ftp_fget()
The server the file is coming from is UNIX
If I refresh the page every few hours the errors I get from PHP are different with no change in code
Error 1: 'ftp_fget(): Transfer mode set to BINARY if ftp_get is binary
or it's 'ftp_fget(): Transfer mode set to ASCII' if ftp_get is ascii
Error 2: 'ftp_fget(): Entering Passive Mode(12.345.678.90.12.34)'
On the above errors I read that PASV mode being FALSE is what triggers Error 1, so I think the switching between the errors is for pasv mode working or not working. Not positive though.
<?php
$server = "12.345.678.90";
$local_file = 'inv3.txt';
$file = 'inventory-alp.txt';
$con = ftp_ssl_connect($server,21) or die("Could not connect to $server");
ftp_login($con,"xxxxxx","xxxxxx") or die("Could not login");
ftp_pasv($con,true);
$fsize = ftp_size($con, $file); // works
if ($fsize != -1)
{
echo "</br>$file is $fsize bytes.</br></br>";
}
else
{
echo "</br>Error getting file size.</br></br>";
}
$lastchanged = ftp_mdtm($con, $file); //works
if ($lastchanged != -1)
{
echo date("F d Y H:i:s.",$lastchanged)."</br></br>";
}
else
{
echo "Could not get last modified</br></br>";
}
if (ftp_get($con,$local_file,$file,FTP_ASCII)) //fails
{
echo "successfully written to $local_file";
}
else
{
echo "There was a problem while downloading $file to $local_file";
}
$var = error_get_last();
echo '<pre>';
var_dump($var);
echo '</pre>';
ftp_close($con);
?>
EDIT 1: Solution: I ended up not being able to access what I needed to change the firewall settings and such in php. While this is not the true answer, I did make it work and it is relatively easy.I ended up running across WINSCP, having the ability to connect to the server in a filezilla type layout and then save the session url was nice. All i did was access the saved session in the .exe and was able to set up my connection in half an hour.
What Martin indicated is very true, the SIZE and MDTM commands run synchronized over the main FTP Command Connection only. The transferring of data files, and usually the directory listing also (unless MLST/MSLD is used) requires a separate connection, the Data Connection, which is negotiated by the client and server over the Control Connection using a series of commands, most notably PORT and PASV.
Without going into a ton of detail (There's a link to our white paper later), when the Client & Server negotiate the terms of the Data Connection, one of the end points will tell the other endpoint the specific IP address and Port number for the connection. One endpoint will listen and wait for a connection from the other endpoint. This works great unless there is a firewall in front of the endpoint that is waiting for the inbound connection. If the client/server session is running in Active mode, the Server will actively connect back to the client on the IP/Port which was received by the server from the client in the form of the PORT command. In Passive mode, the Server will passively wait for the client to connect on the IP/Port which was sent by the server to the client in the response to the PASV command sent by the client to the server.
Again, firewalls tend to block FTP data connections, unless the firewall does active FTP NAT'ing or unless Port Forwarding has been set up on the Firewall and a set of passive-ports has been opened and routed to the endpoint specifically.
So check the firewall settings on the client if you want to use Active/Port mode; check the firewall settings on the server if you want to use Passive/PASV mode.
Here's a link to our white paper which outlines the basics of FTP/PASV/PORT, hopefully it'll help you with your issue.
http://www.webdrive.com/wp-content/uploads/FTP_Explained1.pdf
Best of Luck!
Michael
With the FTP protocol, it's perfectly possible that you are able to obtain the file size and the modification timestamp (using SIZE and MDTM commands respectively), but not the file itself.
The SIZE and MDTM commands use the FTP control connection only.
While a file transfer (or a directory listing) requires a separate data connection. And it's likely that there's something that prevents the data connection from being opened.
See (my) article on the FTP connection modes for more details and typical issues with data connections.
Typically a culprit would be a firewall on your webserver. If you have an SSH/terminal access to the webserver, are you able to connect from it to the FTP server?
Another possibility is a misconfigured FTP server. Is the IP address in "Error 2" routable from your web server? (=Is it the real IP address you connect to?)
It is unlikely this is related to an ASCII/BINARY mode. The messages you are getting (Transfer mode set to ...) are status messages, not error messages. They are not related to your problem. It's indeed strange that you got no other message/error.
You can try to use the active mode, instead of the passive.
ftp_pasv($con, false);
But usually the active mode is more problematic.

PHP Script Won't Upload to FTP Server -

Having spent hours on this, I am out of luck. This script worked perfectly until yesterday. The script generates XML and dumps it as a file (5kb) to a remote FTP server. The script has not changed, nor has our host changed anything. The FTP server company has changed something (they said IP change yesterday) (but claims nothing apart from this). This IP resulted in a different ftp_server which has worked fine.
When I attempt to run the script, I get the following error regardless of whether "ftp_pasv($conn_id, true);" is there or not / disabled:
Warning: ftp_fput() [function.ftp-fput]: Opening ASCII mode data connection in ...
Then it gives the line which contains "FTP_ASCII" below.
When I have the ft_pasv section there only (as per original script), an additional error of the following still appears with the following:
Warning: ftp_fput() [function.ftp-fput]: data_accept: SSL/TLS handshake failed in ...
This is for the same line as the above error.
They were originally on a self-signed SSL. Now, due to my issue, they are now on a 'correct' SSL issued by a well known company. No errors display on Filezilla upon connecting.
Importantly, I can upload via Filezilla with no issues, with or without passive mode
Code above the below code in the script is correct for generating the file as it appears on the script page, once loaded. It just won't dump the file on the server. Here is the connecting to the server bit:
//Connect to the FTP server
$ftp_server = 'import.ftpserverdomain.com';
$ftp_user_name = 'CORRECT-USERNAME';
$ftp_user_pass = 'CORRECT-PASSWORD';
// set up basic ssl connection
$conn_id = ftp_ssl_connect($ftp_server) or die("CONNECTION ERROR");
// login with username and password
$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass) or die("LOGIN ERROR");
$directory = ftp_pwd($conn_id); // /
ftp_pasv($conn_id, true);
$fp = fopen($filename, 'w');
fputs($fp,$file_contents);
fclose($fp);
$fp = fopen($filename, 'r');
$path_with_file = $directory.$filename;
if (ftp_fput($conn_id, $path_with_file, $fp, FTP_ASCII)) {
echo "Successfully Uploaded $File\n";
} else {
echo "There was a problem while uploading $File\n";
}
fclose($fp);
unlink($filename);
ftp_close($conn_id);
Any help is much appreciated. Sorry if I lacked any information. I'll be happy to provide any.
It sounds to me like PHP can't find your CA Certs, if you are running this on a linux box, this is something I have encountered before. Some searching should help but basically PHP's OpenSSL integration needs pointing at your cacerts directory so it can use these to validate the SSL connection it is attempting to make.
I fixed this issue. Hopefully this helps someone. The issue turned out to be server (HostGator). They do not allow FTP over TLS on a shared account.
Despite this working for a year, switching hosts resolved the issue.

Categories