I wonder how to download some .csv file by using SAS.
Browsing on the web, I found it is possible to do that by running the following script:
filename NAME url "http://.../NAME_OF_THE_FILE.csv"
Particularly, I want to understand how such statement works and in which case I cannot use that.
For instance, let's assume one has to download a .csv file that is uploaded on a web page, as, for example, in the web site, where one can find football match data available.
In such case case, by using the following script to download the file:
filename csv url "http://www.football-data.co.uk/mmz4281/1617/E0.csv";
and the following one to import data in SAS:
proc import file = csv
out = junk_00
dbms = csv replace;
delimiter = ",";
run;
everything works fine. This file corresponds to the season 2016/2017 and contains the Premier League data, one can find on the first link.
Instead, in the case of the championship data for the 2016/2017 season, by using the same script as follows:
filename csv url "http://www.football-data.co.uk/mmz4281/1617/E1.csv";
proc import file = csv
out = junk_00
dbms = csv replace;
delimiter = ",";
run;
you get the following error:
Import unsuccessful. See SAS Log for details.
Browsing at the LOG window you can see among the LOG lines the following note/warning:
Invalid data for Date, even if the file is formatted correctly.
I don't understand the reason sometimes the script works and sometimes not, since this happened with other file, although the file are not corrupted and formatted correctly and in the same way.
What's wrong?
Can someone help me to understand why this happens?
Thanks all in advance!
Proc Import has to guess at data types. For some reason it thinks the date field is formatted as MMDDYY, but it's actually DDMMYY. Or maybe it's used inconsistently, I didn't check all, but could see the source of the error immediately.
The solution is to not use PROC IMPORT but to use a data step. If all the files are structured the same, then this works as a solution, but if each file is different then it's not a feasible solution.
Another possible workaround, is to download the data and then set GUESSINGROWS to a large number and then read the files. It will read all the values before guessing at the types so it can be better. This solution does not appear to work when using filename URL, but I don't know why.
I don't think this is a full answer, but it should shed some light on what's happening for you.
Related
ERROR i'm getting :
This page isn’t working didn’t send any data.
ERR_EMPTY_RESPONSE
I am using PHP language for reading the csv file .
My PHP approach is look like for procession the csv data :
$csvAsArray = array_map('str_getcsv', file($tmpName));
I am sure that the above code creating problem afterwords the code is not getting executing .
How i can import more that greater than at least 1 million data at a time ?
Can anyone help me out , which approach should i choose ?
It looks like you're trying to grab the entire contents of the file in one gulp. Don't do that :) PHP array_map() isn't scalable to 1000's ... or millions of lines.
SUGGESTION:
Read your data into a temp file (as you appear to be doing now).
Do a Postgresql COPY
EXAMPLE:
COPY my_table(my_columns, ...)
FROM 'my_file.csv' DELIMITER ',' CSV HEADER;
I would suggest using a league/csv package for CSV parsing and importing. #paulsm4 is correct that it's never needed to put the whole file into memory and then work with it, one should rather read line-by-line. But this package is well-maintained and does this all under the hood and quite effectively. And it is much more flexible than COPY postgres command to my mind. One can filter the contents, map callbacks to fields/rows and all this on PHP side.
Here are the files:
1) In this website:this has excel file, sized 4kb only, that I can't edit, and when I try to get data from it with php, it says that this file is "read only" when it's not. This is price list from accounting program.
2)In this website: is where the program saves the data I think so, but it's coded.
Any suggestions?
i try with all php excel reader i found in net :) i am using win 7, working with xampp in local.
problem is that, i think, that excell file is protected or something, it is exported from program.
i need to get data from that excell file or from another . rea file, they consists data about products : quantity, price, name and so, i can manually add all products to the site, but its better to get data from that program and automaticaly do it, and when there are changes in program it reflects on the site automaticaly, thx for the answers
I am facing a peculiar problem in parsing an excel file (.xls) using PHPExcelReader which actually Spreadsheet_Excel_Reader class. I have used it so many times in different applications and every time I was fine. I am working for an app where there is a ftp folder and an excel file is remotely put there every day. I have a scheduler task that runs every day and read the excel file and update the database.
It was working very fine for couple of months. Now they are adding some new columns in the files and the Spreadsheet_Excel_Reader is unable to read the numeric and date values. But if I just open the file and hit CTRL+S without doing anything, svn says that the file has been modified although I don't see anything changed from 'SVN: Diff with previous version'. However it is doing the magic as I see that the saved file is parsed correctly.
Bellow is the result I see when I try to run the script without touching the file. Please look at index 5 to 9.
Now look at the parse result when I run the script after opening the file and hit CTRL+S. Now entirely sure what is happening. I contacted to them and they said they are not doing anything new.
Any idea about this problem?
Sharing the idea here is much appreciated.
How are you looping through and grabbing the cell values? Have you tried using $cell->val(12,6)? An alternative could be (Thanks to Mark Baker):
$range = 'F'.$row.':'.'J'.$row; # assuming F-J cols are your numeric cols
$objPHPExcel->getActiveSheet()
->getStyle($range)
->getNumberFormat()
->setFormatCode(PHPExcel_Style_NumberFormat::FORMAT_TEXT);
I am looking for the best way to export a CSV file. With MySQL and PHP.
Currently Im generating an CSV with INTO OUTFILE, it works that way but I don't think it's the good way.
Isn't there a better option to make a CSV export download button for every moment a user clicks the download button?
A INTO OUTFILE export is only possible for one instance and is not overwritable.
I have to generate a timestamp and save the file, and then get the latest file from my directory.
This method looks a bit messy for downloading a CSV file from a server...
Has anyone got better solutions?
Thanks!
I think you are well off with exporting via INTO OUTFILE. The reason is that sending the content to the CSV file is done by the MySQL server. Doing it with the PHP Script would be slower (first of all because it is a script, second of all because the data from the SQL server need to be passed to the script) and cost you more resources.
If the CSV file(s) become large you should keep in mind that your Script still may expire. You can encounter this issue by either setting an higher value for the maximum running time of a script in the configuration or have the CSV file being created by an external process/script
Maybe something like this:
`echo $query | mysql > $unique`;
$contents = file($unique);
I am working on a database program using PHP to keep track of the products we manage at my workplace.
For this project, I need to be able to select an .XLS file which contains new product data.
New data consists of the following fields:
Type CHAR(3),
Line CHAR(2),
Number INT,
Measure INT,
Comments VARCHAR(255),
Variation CHAR(1) i.e.('Y' || 'N')
These files are created in Excel, or Google Docs; I have found a wonderful excel_reader which allows me to extract the values from this file.
As this is an action which will happen routinely, as new products are created, so I do not want the file to be stored in my server directory (after a while there would be dozens!).
I would rather that the file simply be read, because the import script I'm writing transfers the file's data into an array.
What I really want to happen is to have the user select the file's location (on their local computer) through an HTML form, and then have the script save that file's contents to a MySQL database without ever sending the file to the Server.
I would greatly appreciate any advice you can offer me, I'm not even sure that my plan is a valid way to handle this situation.
It will have to be stored, at least temporarily. Delete the file after you have what you need from it (presumably after moving it out of the temp directory using move_uploaded_file, to the folder from which you will read it), then remove it using unlink.
As a last point, I would be a little worried about immediate deletion of uploaded files. What if something goes wrong with the script while the file is being parsed and data stored in the database? It would probably be a good idea to have a cron job that periodically deletes the files, to be on the safe side, instead of deleting them immediately.
Since your PHP script is running on the server, the Excel file will have to be saved to the server to be read. Once you've read the file and stored it in the database, just delete it.
I found the answer to my question here. It is a nice tutorial on uploading, moving, reading, and deleting files using PHP.
Thank you to all who contributed.
I was struggling to do the same (for xlsx though).
The solution is to use the $_FILES['file']['tmp_name'],
where file is the input name.
Regards :)