receive csv from server and parse using php - php

Question

this was working up until quite recently and i cannot seem to crack the case.
if you manually visit the url hit against in the script, the results are there..but if i do it in the code, i am having an issue.
you can see in my output test that i am no longer getting any output...
any ideas?
<?
//$ticker=urldecode($_GET["ticker"]);
$ticker='HYG~FBT~';
echo $ticker;
$tickerArray=preg_split("/\~/",$ticker);
// create curl resource
$ch = curl_init();
// set urlm
curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$a='';
$output = curl_exec($ch);
echo "<br><br>OUTPUT TEST: ".($output);
$lineCt=0;
$spaceCt=0;
$splitOutput=preg_split("[\n|\r]",$output);
for($ii=0;$ii<sizeof($tickerArray);$ii++){
$i=0;
$matchSplit[$ii]=-1;
while($i<sizeof($splitOutput) && $matchSplit[$ii]==-1){
$splitOutput2=preg_split("/\,/",$splitOutput[$i]);
if($i>0){
if(strcasecmp($splitOutput2[0],strtoupper($tickerArray[$ii]))==0){
$matchSplit[$ii]=$splitOutput[$i]."#";
}
}
$i++;
}
if($matchSplit[$ii]==-1){
echo "notFound#";
}else{
echo $matchSplit[$ii];
}
}
//echo ($output);
curl_close($ch);
?>

Answer 1

I added a user agent to your script and it seems to work fine here:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/");
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
//Then your output paring code
The output I get:
HYG~FBT~
OUTPUT TEST: Name,Volume,Ask Size,Ask Price,Bid Size,Bid Price,Last Price,Shares Matched,Shares Routed SPY,35641091,0,0.0,0,0.0,134.38,34256509,1384582 BAC,22100508,0,0.0,0,0.0,7.78,20407265,1693243 QQQ,12085707,0,0.0,0,0.0,62.65,11703725,381982 XLF,11642347,0,0.0,0,0.0,14.47,11429581,212766 VXX,9838310,0,0.0,0,0.0,28.2,9525266,313044 EEM,9711498,0,0.0,0,0.0,43.28,9240820,470678 IWM,8272528,0,0.0,0,0.0,81.19,7930349,342179 AAPL,6145951,0,0.0,0,0.0,498.24,4792854,1353097
It is also good practice to close the CURL connection once you are done. I believe that might also play a part in your issues.
If you are still getting issues, check to see if the server the script runs on can access that site.
Update
Upon further investigation, here's what I believe is the root of the problem.
The problem lies with the provider of the CSV file. Perhaps due to some issues on their end, the CSV are generated, but only contain the headers. There were instances where there were indeed data in there.
The data is only avaliable during set hours of the day.
In any case, fetching the empty file = the parser would print #notFound, leading us to assume that there was an issue with CURL.
So my suggestion is to add some further checking to the script to check whether the CSV file actually contains any data at all and is not a file containing just the headings.
Finally, setting a timeout for CURL should fix it as the CSV takes a while to be generated by the provider.