I've already done a lot of the leg work here and I've assumed that I was going to do it this one way, but I'm not sure that it will work now. I have created an inventory system for my business using excel and it has a number of Macros (VBA scripts) - let's call it Inventory Master. I'm trying to keep track of all of my inventory across all of the different selling mediums (Amazon, eBay, personal store).
I've created a PHP script to pull sales data from Amazon and convert the XML date (request data from Amazon) into two separate csv files - for those who care, the way Amazon's API works, I have to make one request to pull of Sales Order IDs for that day and then another request using the Sales Order ID to get the actual order information.
THE PROBLEM is that I'm not sure what the best way to import the data that I need from the two files, into my inventory master. Also, I have to be able to filter the data that I want to import and place it into the appropriate columns in the Inventory Master.
I was going to create an VBA script to import the files, but I'm sure if I can manipulate the data this way, since the import data is a csv and doesn't have macros enabled. I'm sure I could still find a way, but I was then thinking that I might just be able to do all of this via PHP, but the only PHPExcel library I see doesn't work in xlsm formats.
This is where I turn to the internet. Can anyone think of a better way to import this data?
I think you need to do it in stages. Any database manipulation usually goes through a staging table. A temporary table which you manipulate before copying it to the final table.
Andreas suggested you import the entire file into excel. Run your filters through the file in excel before copying it to the Inventory Master.
You can then dispose of the temporary file.
Related
I'm in the middle of a project that I am working on. It's a classified website for lighting distributors created in php. I would like to accept csv data-feed from each distributor and import the data about 3 times a week. I would also like the data-feed to be hosted on the distributors website and I would import the data to the classified website mysql database from the external link that is provide by the distributor.
What would be the best method to import multiple data-feeds from multiple distributors? I am sorry that I have posted this question but I am desperate. I have search the net for answers but came up empty.
Would it be best to create a cron job that calls a script to import each feed? Obviously I would have a test database to first test each data-feed at first to make sure all the data in the csv file is the correct location.
Would I have to use the test database each and every time I import the data? What would be the best way to prevent something from happening to my database if for some reason the distributor changes the feed?
Any information would be greatly appreciated. Thank you in advance for your help.
Welcome to the wonderful world of ETL. While this question is a little too broad for SO, here's how I would go about it (from a high level):
Create a script to import the CSV to your local file system
Import the data from your local file system to a "Stage" table in your database
Check whatever you want to check (did it load without error, does the stage table look correct, etc)
Assuming everything checks out, drop and reload (or upsert or whatever) from your stage table to the live table. Consider adding a new field to your live tables that holds the timestamp from when the data was last loaded for that record
Consider archiving the flat file on your local system for preservation sake
Make a cron job to run your script that does the above steps.
I'm currently building an application which allows a user to import a CSV file to a database. What I'm confused about is the best way to handle the 2nd import mapping process as I'm finding that the file information is lost.
Form first step asks user for a file:
The server handles the post, processes the csv and redirects back to the page with the import mapping and example data:
When the form posts again - how will the server know which file to look for? I was thinking of pushing the saved filepath back and including it as a hidden input, but I realise this is risky as it could be changed by the user. Maybe an encrypted string of some sort?
Any advice would be amazing, or if you can think of a better way of achieving this!
Also - Is there a nice way to only allow users to only select one option from the select dropdowns once?
You could create an “upload” model, that stores the file path in the database. When a user uploads a CSV, you create a model instance and redirect to the form that has the option to map fields. When they set their fields, you process the CSV and delete the record. You can also have a scheduled task that cleans up unprocessed CSV files after a period of time, i.e. 24 hours.
Looking for insight on the best approach for large csv file imports to mysql and managing the dataset. This is for an ecommerce storefront "startup". All product data will be read from csv files which are download via curl (server to server).
Each csv file represents a different supplier/warehouse with up to 100,000 products. In total there are roughly 1.2 million products spread over 90-100 suppliers. At least 75% of the row data (51 columns) is redundant garbage and will not be needed.
Would it be better to use mysqli LOAD DATA LOCAL INFILE to 'temp_products' table. Then, make the needed data adjustments per row, then insert to the live 'products' table or simply use fgetcsv() and go row by row? The import will be handled by a CronJob using the sites php.ini with a memory limit of 128M.
Apache V2.2.29
PHP V5.4.43
MySQL V5.5.42-37.1-log
memory_limit 128M
I'm not looking for "How to's". I'm simply looking for the "best approach" from the communities perspective and experience.
I have direct experience of doing something virtually identical to what you describe -- lots of third party data sources in different formats all needing to go into a single master table.
I needed to take different approaches for different data sources, because some were in XML, some in CSV, some large, some small, etc. For the large CSV ones, I did indeed follow roughly your suggested routed:
I used LOAD DATA INFILE to dump the raw contents into a temporary table.
I took the opportunity to transform or discard some of the data within this query; LOAD DATA INFILE allows some quite complex queries. This allowed me to use the same temp table for several of the import processes even though they had quite different CSV data, which made the next step easier.
I then used a set of secondary SQL queries to pull the temp data into the various main tables. All told, I had about seven steps to the process.
I had a set of PHP classes to do the imports, which all implemented a common interface. This meant that I could have a common front-end program which could run any of the importers.
Since a lot of the importers did similar tasks, I put the commonly used code in traits so that the code could be shared.
Some thoughts based on the things you said in your question:
LOAD DATA INFILE will be orders of magnitude quicker than fgetcsv() with a PHP loop.
LOAD DATA INFILE queries can be very complex and achieve very good data mapping without ever having to run any other code, as long as the imported data is going into a single table.
Your memory limit is likely to need to be raised. However, using LOAD DATA INFILE means that it will be MySQL which will use the memory, not PHP, so the PHP limit won't come into play for that. 128M is still likely to be too low for you though.
-If you struggle to import the whole thing in one go, try using some simple Linux shell commands to split the file into several smaller chunks. CSV data format should make that fairly simple.
I'm currently building a web-app which displays data from .csv files for the user, where they are edited and the results stored in a mySQL database.
For the next phase of the app I'm looking at implementing the functionality to write the results into ** existing .DBF** files using PHP as well as the mySQL database.
Any help would be greatly appreciated. Thanks!
Actually there's a third route which I should have thought of before, and is probably better for what you want. PHP, of course, allows two or more database connections to be open at the same time. And I've just checked, PHP has an extension for dBase. You did not say what database you are actually writing to (several besides the original dBase use .dbf files), so if you have any more questions after this, state what your target database actually is. But this extension would probably work for all of them, I imagine, or check the list of database extensions for PHP at http://php.net/manual/en/refs.database.php. You would have to try it and see.
Then to give an idea on how to open two connections at once, here's a code snippet (it actually has oracle as the second db, but it shows the basic principles):
http://phplens.com/adodb/tutorial.connecting.to.multiple.databases.html
There's a fair bit of guidance and even tutorials on the web about multiple database connections from PHP, so take a look at them as well.
This is a standard kind of situation in data migration projects - how to get data from one database to another. The answer is you have to find out what format the target files (in this case the format of .dbf files) need to be in, then you simply collect the data from your MySQL file, rearrange it into the required format, and write a new file using PHP's file writing functions.
I am not saying it's easy to do; I don't know the format of .dbf files (it was a format used by dBase, but has been used elsewhere as well). You not only have to know the format of the .dbf records, but there will almost certainly be header info if you are creating new files (but you say the files are pre-existing so that shouldn't be a problem for you). But the records may also have a small amount of header data as well, which you would need to write to work out and each one in the form required.
So you need to find out the exact format of .dbf files - no doubt Googling will find you info on that. But I understand even .dbf can have various differences - in which case you would need to look at the structure of your existing files to resolve those if needed).
The alternative solution, if you don't need instant copying to the target database, is that it may have an option to import data in from CSV files, which is much easier - and you have CSV files already. But presumably the order of data fields in those files is different to the order of fields in the target database (unless they came from the target database, but then you wouldn't presumably, be trying to write it back unless they are archived records). The point I'm making, though, is you can write the data into CSV files from the PHP program, in the field order required by your target database, then read them into the target database as a seaparate step. A two stage proces in other words. This is particularly suitable for migrations where you are doing a one off transfer to the new database.
All in all you have a challenging but interesting project!
Well, Maybe 5M is not that much, but it needs to receive a XML based on the following schema
http://www.sat.gob.mx/sitio_internet/cfd/3/cfdv3.xsd
Therefore I need to save almost all the information per row. Now by law we are required to save the information for a very long time and eventually this database will be very very veeeeery big.
Maybe create a table every day? something like _invoices_16_07_2012.
Well, I'm lost..I have no idea how to do this, but I know is possible.
On top of that, I need to create a PDF and 2 more files based on each XML and keep them on HD.
And you should be able to retrieve your files quickly using a web site.
Thats a lot of data to put into one field in a single row (not sure if that was something you were thinking about doing).
Write a script to parse the xml object and save each value from the xml in a separate field or in a way that makes sense for you (so you'll have to create a table with all the appropriate fields). You should be able to input your data as one row per xml sheet.
You'll also want to shard your database and spread it across a cluster of servers on many tables. MySQL does support this but I've only boostrapped my own sharding mechanism before.
Do not create a table per XML sheet as that is overkill.
Now, why do you need mysql for this? Are you querying the data in the XML? If you're storing this data simply for archival purposes, you don't need mysql, but can instead compress the files into, say, a tarball and store them directly on disk. Your website can easily fetch the file in this way.
If you do need a big data store that can handle 5M transactions with as much data as you're saying, you might also want to look into something like Hadoop and store the data in a Distributed File System. If you want to more easily query your data, look into HBase which can run on top of Hadoop.
Hope this helps.