Working cron job jobs stopped Suddenly, trouble reading external url - php

Question

UPDATE #2:
I have confirmed with my contacts at NOAA that they are having big time interconnectivity problems all across NOAA. For example, they are only getting precipitation data from 2 locations. I am sure this is related. I let NOAA know about this thread and the work you all did to identify this as a connectivity issue.
UPDATE: Now the wget command works from my local server but not from the 1and1.com server. I guess that explains why it works from my browser. Must be a connection issue back east as some of you are also having the same problem. Hopefully this will clear itself as it looks like I can't do anything about it.
EDIT: It is clear that the fetch problem I am having it
unique to NOAA addresses in that there is no problem with my code and other sites
that all fetches work just fine in a normal browser
that no way I have been able to try will fetch the file with code.
My question is how can I make code that will fetch the file as well as the browser?
I have used this command to get an external web page for almost 2 years now
wget -O <my web site>/data.txt http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
I have tried this from two different servers with the same result so I am sure I am not being blocked.
Suddenly this morning it quit working. To make matters worse, it would leave processes running on the server until there were enough that it shut down my account and all my web sites were erroring out until we did a kill one at a time to the 49 sleeping processes.
I got no help from 1and1 tech support. They said it was my cron script, which was just the one line above.
So I decided to re-write the file get using php. I tried file_get_contents. I have tried curl, fgets as well. But none of this worked so I tried lynx.
Nothing loads this particular URL but everything I tried works fine on other urls.
But if I just copy http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt into a browser, no problem - the file displays promptly.
Obviously it is possible to read this file because the browser is doing it. I have tried Chrome, IE, and Firefox and none had a problem loading this page but nothing I have tried in code works.
What I want to do is read this file and write it to the local server to buffer it. Then my code can parse it for various data requests.
What is a reliable way to read this external web page?
It was suggested I add a user agent so I changed my code to the following
function read_url($url){
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
$output = curl_exec($ch);
if(curl_errno($ch)){
echo "<!-- curl_error($ch) -->";
}
curl_close($ch);
return $output;
}
Again, it works on other external web sites but not on this one.
I tried running the wget manually: Here is what I got
(uiserver):u49953355:~ > wget -O <my site>/ships_data.txt http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt
--2013-11-17 15:55:21-- http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt
Resolving www.ndbc.noaa.gov... 140.90.238.27
Connecting to www.ndbc.noaa.gov|140.90.238.27|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 690872 (675K) [text/plain]
Saving to: `<my site>/ships_data.txt'
0% [ ] 1,066 --.-K/s eta 7h 14m
It just stays at 0%
NOTE <my-site> is the web address where my data is stored. I did not want to publish the address of my bugger area but it is like mydomain/buffer/
I just tried the same thing from another server (not 1and1)
dad#myth_desktop:~$ wget -O ships_data.txt http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt
--13:14:32-- http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt
=> `ships_data.txt'
Resolving www.ndbc.noaa.gov... 140.90.238.27
Connecting to www.ndbc.noaa.gov|140.90.238.27|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 690,872 (675K) [text/plain]
3% [====> ] 27,046 --.--K/s ETA 34:18
It is stuck at 3% this time.

Answer 1

Both your wget commands worked for me.
It also seems that NOAA is not blocking your requests either since you get the 200 response code and HTTP headers (content length, type, etc) and part of the data (1066 bytes are somewhere in the row 7-8 of the data).
It may be that your connection (in general or specifically to NOAA) is slow or passing via some buffering proxy. Until the proxy gets all or most of the data, to wget it will look like connection is staling. Does it work to retrieve this file: http://www.ndbc.noaa.gov/robots.txt?
Option --debug of wget might also help to find out the problem.
Anyways, about hanging wget processes, you can use --timeout=60 option to limit the waiting time before failing (http://www.gnu.org/software/wget/manual/wget.html).
wget -O ships_data.txt http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt --timeout=10
If you want to set an user agent (like you did in the PHP script), you can use "--user-agent=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)" option.
wget -O ships_data.txt http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt "--user-agent=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
About curl vs wget, you can just replace the wget commands with curl command (instead of doing it in PHP):
curl -o ships_data.txt http://www.ndbc.noaa.gov/data/realtime2/ship_obs.txt --user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
Andrei

Answer 2

The file is available, but even though its small it is taking a long while to download. In a couple attempts I experienced up to 3 minutes and 47 seconds to fetch this tiny file of 23KB.
It is clearly some issue with their network, not much you can do about it.
Consider using set_time_limit(600) to allow your PHP script to take longer (10 minutes) to download the file, but at the same time not too long so it won't get stuck if it fails.

Answer 3

Since initially, the OP was able to not run the wget command manually, my guess was that the server IP was blocked.
Manually running the following command hung up, so it added weight to my said speculation.
wget -O <my web site>/data.txt http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt on the hosted server
On checking if wget itself was working, OP did wget to a dummy endpoint. wget -O <web-site>/google.log www.google.com which worked.
Since OP mentioned that downloads proceeded sometimes, but not always and it worked from another server from the same hosted solution, I think we can now pin it to be an issue on the other website's network.
My guess is, the crons are being run at a very small frequency (say every minute), like
* * * * * wget -O <my web site>/data.txt http://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
(or at a similar small frequency), and due to whatever kind of server load the external website has, the earlier requests either time out, or do not finish within the time period stipulated for them (1 minute).
Beacuse of this, OP is facing some race condition in which multiple cron processes are trying to write to the same file, but none of them are able to actually write to it completely because of the delay in receving packets for the file (Example, one process hanging from 12:10 AM, another one started at 12:11 AM, and one more started at 12:12 AM, none of them over)
The solution to this would be to make them little more infrequent, or if OP wants to use the same frequencies, then redownload only if a previous version of the download is not currently in progress. For checking if a process is already running, check this