I'm trying to copy an XML file from the Bureau of Meteorology (Australian) Public Access Data Feeds with PHP to my server. I can open the file in the browser but I can't seem to touch it with PHP using CURL, FTP or simplexml_load_file. I've even tried to copy it with wget and I can't.
Full URL: ftp://ftp2.bom.gov.au/anon/gen/fwo/IDD10150.xml
// connect and login to FTP server
$ftp_username = "anonymous";
$ftp_userpass = "guest";
$ftp_server = "ftp2.bom.gov.au";
$ftp_conn = ftp_connect($ftp_server) or die("Could not connect to $ftp_server");
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
$local_file = "IDD10150.xml";
$server_file = "/anon/gen/fwo/IDD10150.xml";
// download server file
if (ftp_get($ftp_conn, $local_file, $server_file, FTP_ASCII))
{
echo "Successfully written to $local_file.";
}
else
{
echo "Error downloading $server_file.";
}
// close connection
ftp_close($ftp_conn);
Produces the following error
Warning: ftp_get(): Failed to establish connection...
Error downloading /anon/gen/fwo/IDD10150.xml.
Edit: Below is the update code as per suggestions below and the current error messages.
// connect and login to FTP server
$ftp_username = "anonymous";
$ftp_userpass = "guest";
$ftp_server = "ftp2.bom.gov.au";
$ftp_conn = ftp_connect($ftp_server) or die("Could not connect to $ftp_server");
$login = ftp_login($ftp_conn, $ftp_username, $ftp_userpass);
ftp_pasv($ftp_conn , TRUE);
$local_file = "IDD10150.xml";
$server_file = "/anon/gen/fwo/";
// download server file
if (ftp_get($ftp_conn, $local_file, $server_file, FTP_BINARY))
{
echo "Successfully written to $local_file.";
}
else
{
echo "Error downloading $server_file.";
}
// close connection
ftp_close($ftp_conn);
Produces the following error
Warning: ftp_get(): php_connect_nonb() failed: Operation now in progress (115) in...
Warning: ftp_get(): Switching to Binary mode. in...
Error downloading /anon/gen/fwo/.
Edit 2: Access via SSH
Last login: Mon Aug 3 11:25:27 on ttys000
MacBook-Pro:~ me$ ssh mysite.com
me#mysite.com's password:
Last login: Mon Aug 3 11:27:37 2015 from IP
me#mysite.com [~]# ftp ftp.bom.gov.au
Connected to ftp.bom.gov.au (134.178.253.145).
220-Welcome to the Bureau of Meteorology FTP service.
220-
220- Disclaimer
220-
220-You accept all risks and responsibility for losses, damages, costs and
220-other consequences resulting directly or indirectly from using this site and
220-any information or material available from it.
220-
220-To the maximum permitted by law, the Bureau of Meteorology excludes all
220-liability to any person arising directly or indirectly from using this
220-site and any information or material available from it.
220-
220-Always Check the Information
220-
220-Information at this site:
220-
220-. is general information provided as part of the Bureau of Meteorology's
220- statutory role in the dissemination of information relating to
220- meteorology.
220-. is subject to the uncertainties of scientific and technical research
220-. may not be accurate, current or complete
220-. is subject to change without notice
220-. is not a substitute for independent professional advice and users
220- should obtain any appropriate professional advice relevant to their
220- particular circumstances
220-. the material on this web site may include the views or recommendations
220- of third parties, which do not necessarily reflect the views of the
220- Bureau of Meteorology or indicate its commitment to a particular course of
220- action.
220
Name (ftp.bom.gov.au:samw): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> get
(remote-file) /anon/gen/fwo/IDQ13015.xml
(local-file) test.xml
local: test.xml remote: /anon/gen/fwo/IDQ13015.xml
227 Entering Passive Mode (134,178,253,145,77,229).
ftp: connect: Connection timed out
Edit 3 & Reason
Ended up being the server Firewall blocking unknown outgoing connections which was found out by contacting BoM and working my server administrator. The BoM IP 134.178.253.145 was added and all was good.
Your server is not in a passive mode, and add this code to process:
ftp_pasv($ftp, true);
For more information look at passive mode on php.net: http://php.net/manual/en/function.ftp-pasv.php Passive mode uses the data initiated by the client rather than the server. So this is why you can't put on server. If this is not set it will fail.
NOTE: Set ftp_pasv() function after ftp_login() function.
Update
Change from ftp_pasv($ftp, true); to ftp_pasv($ftp_conn, true);
First you need to use the passive mode (as suggested by the other answers):
ftp_pasv($ftp_conn, true);
It's unlikely that you succeed to connect in the default active mode as there's typically a firewall between your webserver and the FTP server, which won't allow connections from the FTP server back to your webserver.
See my article on the FTP active/passive connection modes for details.
Regarding the "Operation now in progress (115)" error you get, when using the passive mode.
From the connect man page:
If the connection cannot be established immediately and O_NONBLOCK is set for the file descriptor for the socket, connect() shall fail and set errno to [EINPROGRESS], but the connection request shall not be aborted, and the connection shall be established asynchronously. Subsequent calls to connect() for the same socket, before the connection is established, shall fail and set errno to [EALREADY].
See also TCP Connect error 115 Operation in Progress What is the Cause?
My guess is that the underlying problem is the firewall again, that won't allow the data transfer connection.
If you have a shell access to the webserver, try to connect with the command-line ftp client to verify.
Now that you have tried using the command-line ftp:
As you have verified yourself, you cannot connect even from command-line ftp. There's nothing wrong with your PHP code. As such, your question was resolved from a programmer's point of view. The issue is with the network. What is off-topic on Stack Overflow.
Contact your server administrator. Or consider using the SFTP, if you have the option. It should not suffer from these kind of problems.
ftp_pasv($ftp_conn , TRUE);
ftp_pasv() in w3schools
Edit 01
$local_file = fopen("IDD10150.xml",'w');
Related
I want to connect to an FTP using PHP to upload the reports generated. As per the remote server, the FTP needs to be in ACTIVE mode.
this is my code:
ini_set('display_errors', '1');
error_reporting(E_ALL);
$conn_id = ftp_connect('myftpserver.com', 21);
if($conn_id)
{
// login with username and password
$login_result = ftp_login($conn_id, 'mysuer', 'password');
$passive = ftp_pasv($conn_id,FALSE);
echo "is active?<br/>";
var_dump($passive);
echo 'Login Result:';
var_dump($login_result);
$files_list = ftp_nlist($conn_id, '/MyFolder/');
echo "<br/>files list ";
var_dump($files_list);
}
else
{
var_dump('Unable to connect to FTP Server');
}
When I am running it from the local machine or a normal shared server, I am able fetch the list, but I am unable to run the script from my AWS AMI instance. For testing purpose, I have even opened all inbound traffic too. Still no luck. Also, the point is that if I try with some other ftp details, I am able to get the response of ftp_nlist. But not for this one. I tried it on 3 AWS instances till yet. Yielded the same result.
All I can say is that this is somewhere the issue at my server security group/firewall. But unable to figure it out. Please help.
The response I get from the server:
is active
bool(true)
Login Result:bool(true)
files list bool(false)
Using FTP Active Mode is problematic with AWS Security Groups.
For active mode to work, you will have to open all inbound ports above 1023. If your client supports restricting the range, do so. You will also need to open both port 20 and port 21 inbound and outbound.
The problem is that the FTP client selects a port that it will listen on. Then the FTP client informs the FTP Server of this port number. The FTP Server then connects to this port. This goes against normal AWS Security Group designs meaning only allow specific ports to be open. You can verify this by opening all ports temporarily, test your FTP client and then closing all the ports.
Active Mode is not secure for the FTP client. Passive Mode is not secure for the FTP server (but the better choice).
NOTE: Rotate your FTP credentials often. Your login and password is sent in the clear and are not encrypted.
FTP is a legacy technology, which is still very popular, that should be stored away in the attic.
I need to extract data from a .gz file which is located on an ftp server. The FTP server works with IP white listing, so I submitted my local IP and my website IP (i used gethostbynameto get the IP) which were both approved.
Locally, I can run this code to reach the file:
$url="ftp://username:password#host/targetfile.gz";
$xml = simplexml_load_file("compress.zlib://$url") or die ("Cannot load file");
This runs perfectly fine and it allows me to extract data from the XML file.
When I run the script on my server however, i'm not getting a connection. I contacted the admin running the ftp server and they told me they only allow FTPS connections.
So, I proceeded with the following code to try and establish a connection:
$conn_id = ftp_ssl_connect($ftp_server);
// login with username and password
$login_result = ftp_login($conn_id, $ftp_user, $ftp_pass);
if (!$login_result) {
die("Cannot login.");
}
echo ftp_pwd($conn_id); // /
// close the ssl connection
ftp_close($conn_id);
This also doesn't connect. I'm new to FTP connections with PHP and FTP in general and i've got no clue on how to proceed from this point.
OpenSSL is enabled.
Can anyone point me in the right direction?
Edit: These are the error messages:
When I use the top block of code, this occurs:
Warning: simplexml_load_file(): I/O warning : failed to load external
entity
"compress.zlib://username:password#ftp.thehost.com/targetfile.gz"; in
/path/to/my/website/folder/htdocs/mydomain.com/feeds/script.php on
line 13
When I use the bottom block of code, I get this error:
Warning: ftp_login() expects parameter 1 to be resource, boolean given
in /path/to/my/website/folder/htdocs/mydomain.com/feeds/script.php on
line 16 Cannot login.
I do not have access to the log file afaik
I do not think there's much we can help you here.
You do not have a network connectivity between the web server and FTP server.
Note that it does not have to be because the web server has not been white-listed on the FTP server. It's also possible that the web server does not allow outgoing connections.
I have a PC that is running some FTP via PHP that I know used to work 1-2 months ago, but now I return to it I find that the PC is no longer working. I know I have been using the PC but I cannot think of what might have changed.
The PHP is throwing out error messages reading
Unable to build data connection: Connection refused
...when I use the ftp_put() function.
The cut down code I am using is:
<?php
$trackErrors = ini_get('track_errors');
ini_set('track_errors', 1);
$server="***.***.***.***";
$port=21;
echo "<LI>Connecting to $server:$port<BR>";
$conn_id = ftp_connect($server,$port,9999999) or die("<BR>Unable to connect to ".$server.":$port server.");
if ( !$conn_id ) {
$errmsg = $php_errormsg;
echo "<BR><LI>ERR:$errmsg";
}
else {
$passive=false;
echo "<LI>Setting Passive Mode=$passive";
ftp_pasv($conn_id, $passive);
$user="*********";
$pass="*********";
echo "<LI>Connecting as $user/*****";
if (!ftp_login($conn_id, $user, $pass)) {
$msg = "Failed to login to $selected_server as $user; <BR>check logincredentials in the Settings";
echo "<BR><LI>$msg";
$errmsg = $php_errormsg;
echo "<LI>ERR:$errmsg";
return $msg;
}
ftp_set_option($conn_id, FTP_TIMEOUT_SEC, 10000);
if (!#ftp_put($conn_id, "test.txt", "C:......test.txt", FTP_BINARY)) {
echo "<BR><LI>ftp_put failed";
$errmsg = $php_errormsg;
echo "<LI>ERR:$errmsg";
}
echo "<HR>Done";
}
?>
the output when running this as a webpage is
Connecting to ***.***.***.***:21
Setting Passive Mode=
Connecting as *******/*****
ftp_put failed
ERR:ftp_put(): Unable to build data connection: Connection refused
Done
The result is that the ftp_put() gives the error message and leaves a zero (0) byte file with the right filename on the server.
The strange thing is is that
the same code/connection info works on another laptop ok
the same connection info works ok using FileZilla when pushing a file
the problem occurs on several servers (ie. it's not just one specific destination that has the problem)
Also, this doesn't seem to have anything to do with the passive mode (it fails with and without this enabled)
Does anyone have any suggestions?
Thanks
Abe
You are using the active FTP mode. In the active mode the server tries to connect to the client. In most network configurations, that's not possible as the client machine is usually behind a firewall.
That's why the server fails with:
Unable to build data connection: Connection refused
It's specifically ProFTPD error message for this situation.
See my article on the active and passive FTP connection modes for details.
The code can work on other machines, if they have firewall disabled or if they have rules that allow incoming traffic on unprivileged ports.
FileZilla works because it defaults to the passive mode (as most modern FTP clients do).
You have claimed to try the passive mode too, yet to get the same error message.
That's because you are using the ftp_pasv call incorrectly.
You have to move the ftp_pasv call after the ftp_login.
$user = "*********";
$pass = "*********";
echo "<LI>Connecting as $user/*****";
if (!ftp_login($conn_id, $user, $pass)) {
// ...
}
$passive = true;
echo "<LI>Setting Passive Mode=$passive";
ftp_pasv($conn_id, $passive);
The documentation clearly suggests it:
Please note that ftp_pasv() can only be called after a successful login or otherwise it will fail.
For a similar issue (just with Pure-FTPd), see PHP upload via FTP - ftp_put() I won't open a connection to x.x.x.x (only to y.y.y.y).
copy(ftp://...#video.calkinsftp.com/videodrive/video_input/Philly/Council Rock South High Schoolers hold mock presidential debate.mp4): failed to open stream: FTP server reports STOR, file: /var/www/html/user-controller.php, line: 138
The file transfers successfully my own network. But it does not work on clients network.
The error message is nonsense, clearly a bug in PHP code.
I have reported this:
Bug #73457 Wrong error message when fopen FTP wrapped fails to open data connection.
Anyway, the root cause is most probably that an FTP data connection cannot be opened.
Most typical cause of the problem is that PHP defaults to the active mode. And in 99% cases, one has to switch to the passive mode, to make the transfer working. Use the ftp_pasv function.
$conn_id = ftp_connect($ftp_server);
// login with username and password
$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass);
// turn passive mode on
ftp_pasv($conn_id, true);
See also:
PHP ftp_put fails with "Warning: ftp_put (): PORT command successful";
my article on the active and passive FTP connection modes.
Please check FTP user authentication and user permission, and check allow particular user uploading file size and your network configuration(NAT/firewall).
I've been working on this for about 20 hours now and I need some major help. I'm able to get the file size and the timestamp on the file but I am unable to actually obtain the data.
The server I'm trying to get the data from requires FTP over explicit TLS
I'm receiving the same error(s) with both FTP_BINARY and FTP_ASCII in the ftp_fget()
The server the file is coming from is UNIX
If I refresh the page every few hours the errors I get from PHP are different with no change in code
Error 1: 'ftp_fget(): Transfer mode set to BINARY if ftp_get is binary
or it's 'ftp_fget(): Transfer mode set to ASCII' if ftp_get is ascii
Error 2: 'ftp_fget(): Entering Passive Mode(12.345.678.90.12.34)'
On the above errors I read that PASV mode being FALSE is what triggers Error 1, so I think the switching between the errors is for pasv mode working or not working. Not positive though.
<?php
$server = "12.345.678.90";
$local_file = 'inv3.txt';
$file = 'inventory-alp.txt';
$con = ftp_ssl_connect($server,21) or die("Could not connect to $server");
ftp_login($con,"xxxxxx","xxxxxx") or die("Could not login");
ftp_pasv($con,true);
$fsize = ftp_size($con, $file); // works
if ($fsize != -1)
{
echo "</br>$file is $fsize bytes.</br></br>";
}
else
{
echo "</br>Error getting file size.</br></br>";
}
$lastchanged = ftp_mdtm($con, $file); //works
if ($lastchanged != -1)
{
echo date("F d Y H:i:s.",$lastchanged)."</br></br>";
}
else
{
echo "Could not get last modified</br></br>";
}
if (ftp_get($con,$local_file,$file,FTP_ASCII)) //fails
{
echo "successfully written to $local_file";
}
else
{
echo "There was a problem while downloading $file to $local_file";
}
$var = error_get_last();
echo '<pre>';
var_dump($var);
echo '</pre>';
ftp_close($con);
?>
EDIT 1: Solution: I ended up not being able to access what I needed to change the firewall settings and such in php. While this is not the true answer, I did make it work and it is relatively easy.I ended up running across WINSCP, having the ability to connect to the server in a filezilla type layout and then save the session url was nice. All i did was access the saved session in the .exe and was able to set up my connection in half an hour.
What Martin indicated is very true, the SIZE and MDTM commands run synchronized over the main FTP Command Connection only. The transferring of data files, and usually the directory listing also (unless MLST/MSLD is used) requires a separate connection, the Data Connection, which is negotiated by the client and server over the Control Connection using a series of commands, most notably PORT and PASV.
Without going into a ton of detail (There's a link to our white paper later), when the Client & Server negotiate the terms of the Data Connection, one of the end points will tell the other endpoint the specific IP address and Port number for the connection. One endpoint will listen and wait for a connection from the other endpoint. This works great unless there is a firewall in front of the endpoint that is waiting for the inbound connection. If the client/server session is running in Active mode, the Server will actively connect back to the client on the IP/Port which was received by the server from the client in the form of the PORT command. In Passive mode, the Server will passively wait for the client to connect on the IP/Port which was sent by the server to the client in the response to the PASV command sent by the client to the server.
Again, firewalls tend to block FTP data connections, unless the firewall does active FTP NAT'ing or unless Port Forwarding has been set up on the Firewall and a set of passive-ports has been opened and routed to the endpoint specifically.
So check the firewall settings on the client if you want to use Active/Port mode; check the firewall settings on the server if you want to use Passive/PASV mode.
Here's a link to our white paper which outlines the basics of FTP/PASV/PORT, hopefully it'll help you with your issue.
http://www.webdrive.com/wp-content/uploads/FTP_Explained1.pdf
Best of Luck!
Michael
With the FTP protocol, it's perfectly possible that you are able to obtain the file size and the modification timestamp (using SIZE and MDTM commands respectively), but not the file itself.
The SIZE and MDTM commands use the FTP control connection only.
While a file transfer (or a directory listing) requires a separate data connection. And it's likely that there's something that prevents the data connection from being opened.
See (my) article on the FTP connection modes for more details and typical issues with data connections.
Typically a culprit would be a firewall on your webserver. If you have an SSH/terminal access to the webserver, are you able to connect from it to the FTP server?
Another possibility is a misconfigured FTP server. Is the IP address in "Error 2" routable from your web server? (=Is it the real IP address you connect to?)
It is unlikely this is related to an ASCII/BINARY mode. The messages you are getting (Transfer mode set to ...) are status messages, not error messages. They are not related to your problem. It's indeed strange that you got no other message/error.
You can try to use the active mode, instead of the passive.
ftp_pasv($con, false);
But usually the active mode is more problematic.