I do not know if this is the right spot to ask my question perhaps I should do it another community but I got a question regarding implementing a website that deals with a large CSV file as input.
I am willing to program a website that have a CSV file as input to generate all kinds of data on my website. Think of each entry has their own geolocations etc. In the end I want to create an informative website where the data of the CSV file is used.
Now I want to know what approach might be the best to do this. I can imagine that storing this CSV file within a Database and retrieving data with PHP might take forever due to large loading time when the website is making a call. Can anyone tell me what approach might work when working which such large CSV file?
To give you an idea. In my CSV file I have the following data:
#, Soccer Club, Street adress, Highest competition,..
1, Soccer club 1, adress 123, 3th division,..
2, Soccer club 2, adress 456, 6th division,..
etc.
Now I want to create a webste that use this data and create all kinds of marker points on a map. I do know how to code this the only problem is that I am searching for a fast way of reading and using the data without having my website to load a long time to generate this information. I did not try it yet but before I spend to much time on it I was wondering which advise could you give me?
BTW Think of CSV files of the size of 1GB/2GB
I would suggest you to import the CSV in to a database table not store the whole file as is and then run queries on it. Databases are optimised and meant for these things.
Related
I'm looking for some advice, to be specific in the generation of data using phpmyadmin and mysql.
I have a website (local) that register ideas from different people, those ideas are registered in a database.
I'm able to retrieve all the data into a table that i have in a reports section, the problem is that when there are at least 1500 records it gets slow (i know the amount of data printed in the table makes it slow).
I generate some charts from that data using charts.js, which is good until there are a lot of records in the webpage. (I tested putting 5000 records, 1500 records, and 2000 (which is the max amount we get with this ideas)).
So, what do I want to know?
What is an efficient way of showing that amount of data to the user without the page getting extremely slow.
What is the best practice to generate charts, from a table that gets its data from database or directly getting the data from the database, if so, which chart framework would you recommend?
I have read about pagination, to retrieve just part of the data by page, but my export to excel use the table data to export it, so it would ruin the exporting option.
I'm really sorry if this is no place for asking this programming question, but I cant think of a way of doing this in an efficient way.
Here is a picture of part of my webpage (in top you can see some charts), (below you can see some records of the table), the middle is just some report generation from the date/week and the export to excel option.
Tech used in the webpage:
PHP
JQUERY
HTML5
CSS3
mysql
Any help will be appreciated.
I am creating a website where I want people to submit location addresses. To avoid spelling mistakes I would like users to select from a list (town name, county).
I came across the following site http://www.doogal.co.uk/london_postcodes.php which allows me to download a 56 mb large csv file containing the location data I need.
However I have never used a csv file larger than 1mb before with more than 30000 rows of data on my websites. Usually I would just upload to PhpMyAdmin.
What is better? Uploading the csv file to PHPmyadmin database or accessing the '.csv' file directly using php?
Regards
It depends on what you want to achieve. I am not so sure that using a CSV is benefitial since using a database will allow you to
Cache data
Create Indexes for fast searching
Create complex queries
Do data manipulation, etc
The only way I would think a CSV is better, is if you would use always, all the data. Otherwise, I would go for a database. The end result would be much more organized, much faster, and you could build on top of it.
Hope this helps.
If you're going to do lookups, I'd recommend you put it into a database table and add indexes on the fields that you will be searching on (https://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html). A flat file is not a good way to store data that you have to access or filter quickly.
I've currently got a database with just short of 2000 client locations in Australia. What I am trying to do is to display this data on a heatmap, to be embedded into an existing website.
I've done a heap of looking around, and can't seem to find exactly what I'm after.
http://www.heatmapapi.com/sample_googlev3.aspx
http://www.heatmaptool.com/documentation.php
These are along the right lines of what I want to achieve, however I cannot see these working with data from a mysql database (require the data to be hard-coded, or uploaded through CSV files).
Has anyone come across this sort of thing before, or managed to achieve it?
Both of the examples you provide would potentially work.
With the first you would need to use the data you have to dynamically generate the javascript, or at least the values that go into the javascript.
The second is probably the better option. You would provide a path to the script that would dynamically generate a CSV file.
So I am a PHP developer really, I have only really used MySQL and some MSSQL datbases, as well as excel back in the day.
I have a client we are building an eCommerce shop for and he wants to save on the time it would take to data-input the products table. The list is provided by another company, think wholesale.
So they have given me a CSV file. a common entry looks like this.
"MAINCAT"^"SUBCAT"^355303^"10931"^"MANUFACTURER"^"Short Description"^"sgl"^"1"^1^.00^.00^.00^.00^.00^.00^6.79^"15561109314 5015561109314"^.20^"This is where the description goes"
I havn't got much experience with CSV files, but from stuff I've looked at, it seems a strange format, and some programmes don't even recognise it as a CSV file.
I have been unable to convert it into MySQL.
My question basically is, is there a way to utilise this CSV file, that will fit in with another PHP and MySQL drive eCommerce platform. ie. can it be easily converted, or utilised as a data platform that will be efficient.
Sorry if this isn't in the right place.
This is hardly a csv file (comma separated values)
Try to convert it as you go, to match this explicit format.
Without knowing the data integrity, an example to make it CSV
$strange_lines = file('strange_csv.csv');
foreach ($strange_lines as $line) {
echo str_replace(array('"^"','^"','"^','^'),',',$line).'<br>';
}
A friend of mine has asked me to figure out a way of getting information from a website and putting it into an excel file.
This is the website in question: http://www.manta.com/world/North+America/Canada/Newfoundland/grocery_stores--B619B/#Location
He wishes to have an excel file with a list of all the names, addresses and phone numbers of all the results of his search.
So far I'm stumped in coming up with an idea. I'm fairly new to internet programming.
I was thinking that maybe I could create a greasemonkey userscript which would search for all the required data on the page and at the click of a button open a pop-up which would have the data in CSV format which could then be copied and pasted into excel. However the phone numbers aren't on the search results page so I don't think this is possible.
My second thought was to create a webpage that would search that site and get all the required data, then provide a "Download data to Excel" option.
Are these ideas possible and how would I best go about doing them? Is there a better way?
Thanks!
This would be easier to answer if we know what languages you're familiar with.
Assuming windows this can be done using jscript or vbscript on WSH using WinHttpRequest , excel may be accessed via ActiveX. If you need a UI i would suggest HTA.