I know the title isn't the most descriptive, but I can't think of a better way to describe it.
Ultimately what I would like to do is this:
I am making a system that handles import of data via CSV from several different companies. Naturally, the format from each company is slightly different, with different amounts of data in each column/heading and a different arrangement.
I could either:
Hard code the file format into the program, which is quick and simple to program, but not abstract or future proof
or provide "dynamicity" by allowing users to configure "file formats" in the system, by dragging and dropping CSV headings and saving presets for each incoming file format.
I would like to do option 2, but how would I go about doing this?
I am guessing a tagging style system would do it... but any advice would be appreciated.
I am using PHP, MySQL, and CodeIgniter 2.0.2 if that helps, and also have access to all the Zend libraries in my app.
Assuming that users will be submitting a subset of possible data types that you define up front, this shouldn't be too tough.
A simple form that allows the user to select the data type and give you the field name in their csv could be used to create the templates you mentioned. Store these in a database table associated with the user and make them choose a template before importing a csv. For customers who have csvs without header rows, a slight variation where they tell you the column numbers for each data type.
I'd also suggest providing pre-defined formats for customers who haven't defined their exports yet. That way they could build their exports to one of your formats.
You could also let users define their default template. That way they wouldn't have to select the template every time they import data if they're always importing data in the same format.
Related
Problem:
I am to build a web-based tool in PHP to help users access information scattered across a collection of XML files that I plan to store data from into tables in MySQL. A lot of examples I see online seem to be based on importing data from a bunch of XML files that all use the same formatting. I do not have that luxury.
How would I be able to parse through XML files that have the following factors?
There are multiple categories of XML files in this collection, each having separate formats and types of information that differentiate each category. Ideally, I would create a separate table for each category. However...
Additional new categories of XML files could be added to this collection without my knowledge ahead of time.
Any existing category could have its format restructured and/or the types of information within could be increased or decreased, also without my knowledge ahead of time.
Even among the same category of XML files, there can be older files that have an outdated formatting version.
Expected Results:
Using an example where the collection of XML is about a group of people, if you search for "brown eyes", you get search results listing pages for everyone listed with brown eyes. One of the pages is "Robert". If you click this result, you'll go to a page where all the information from Robert's XML file is displayed (readable formatting to be handled later).
you could only create a self-learning parser, which adds new columns to the table whenever it finds new properties in the XML. basically there are two options available: either build up a data model, which at some point in time matches all the records - or stuff the mess into a noSQL database, which does not necessarily make the mess any better. "one size fits all" (stuffing unstructured data into a structured database) is not an option.
I need to export data to an Excel template that contains VBA code and data validation in PHP.
I tried using PHPExcel library but it is removing the VBA code and data validations from the template.
I tried using PHPReport library, didn't get proper solution.
The template contains multiple worksheets and they are interdependent.
E.g.: Worksheet 1 contains employee data, then worksheet 2 contains salary with respect to employee name.
I have spent a great deal of time working on this problem, and the problem of memory consumption for large data sets. All of the PHP libs I have found keep all of the cells in memory, which is not viable for anything more than a small sheet.
What I ended up doing was writing a set of Java utilities using Apache POI and packaged them with PHP/Java Bridge so I can call them from PHP. This will allow you to create a new workbook based upon a template, keeping the macros intact. You can also use POI's streaming API, so you can handle massive data sets without crashing your server.
If you have Java chops, I highly recommend going this route, it's really the only way to do brain surgery on Excel files from PHP.
If you have any questions about how to do this or would like some example code, I'll be glad to help.
It's essentially for an "is our service is available in your zip code?" yes/no answer so the lookup is trivial.
But the client wants to be able to add and remove zip codes themselves through WP. So, several thousand very short strings need to be stored. How would you set this up? A special content type? A CSV file that would get overwritten every time?
Personally I would rather handle this outside Wordpress entirely but if forced to keep inside the WP system, what are the best ideas?
Best practice in my opinion would be a WordPress plugin with a migration for a new table that holds your data.
In your backend you would provide a way to edit the zip codes (and/or add the ability to just import a csv file to overwrite them all, if this makes it easier for your client).
In your fronted you can just query the strings as you wish.
A custom post type is too much overhead and an abuse of the format.
I've currently got a database with just short of 2000 client locations in Australia. What I am trying to do is to display this data on a heatmap, to be embedded into an existing website.
I've done a heap of looking around, and can't seem to find exactly what I'm after.
http://www.heatmapapi.com/sample_googlev3.aspx
http://www.heatmaptool.com/documentation.php
These are along the right lines of what I want to achieve, however I cannot see these working with data from a mysql database (require the data to be hard-coded, or uploaded through CSV files).
Has anyone come across this sort of thing before, or managed to achieve it?
Both of the examples you provide would potentially work.
With the first you would need to use the data you have to dynamically generate the javascript, or at least the values that go into the javascript.
The second is probably the better option. You would provide a path to the script that would dynamically generate a CSV file.
So I am a PHP developer really, I have only really used MySQL and some MSSQL datbases, as well as excel back in the day.
I have a client we are building an eCommerce shop for and he wants to save on the time it would take to data-input the products table. The list is provided by another company, think wholesale.
So they have given me a CSV file. a common entry looks like this.
"MAINCAT"^"SUBCAT"^355303^"10931"^"MANUFACTURER"^"Short Description"^"sgl"^"1"^1^.00^.00^.00^.00^.00^.00^6.79^"15561109314 5015561109314"^.20^"This is where the description goes"
I havn't got much experience with CSV files, but from stuff I've looked at, it seems a strange format, and some programmes don't even recognise it as a CSV file.
I have been unable to convert it into MySQL.
My question basically is, is there a way to utilise this CSV file, that will fit in with another PHP and MySQL drive eCommerce platform. ie. can it be easily converted, or utilised as a data platform that will be efficient.
Sorry if this isn't in the right place.
This is hardly a csv file (comma separated values)
Try to convert it as you go, to match this explicit format.
Without knowing the data integrity, an example to make it CSV
$strange_lines = file('strange_csv.csv');
foreach ($strange_lines as $line) {
echo str_replace(array('"^"','^"','"^','^'),',',$line).'<br>';
}