I have a PHP script that imports up to 10 or so different CSV files. Each row of each file contains bank account info, including balance. After I've imported all the CSV data into my database, I'd like to make sure the data got in there correctly by comparing the total account balance in the database to the total account balance of the CSV files.
I see at least a few options:
Manually total up all the account balances in Excel - yuck.
Write a PHP script to read each CSV file and total up the account balances - also yuck.
Some third option that I hope exists. It would be amazing if I could do something like:
excel --file="cd.csv" | sum --column="E"
That's obviously not a real thing but hopefully you get the idea. Using some combination of PHP, MySQL, Linux commands, Excel and/or any other tools, is there a simple way to do this?
Don't have to complete answer for you, but AWK should be able to solve your problem: Have a look at these 2 posts:
https://superuser.com/questions/53652/transforming-csv-file-using-sed
Shell command to sum integers, one per line?
I'm not enough of a AWK expert to give the solution, but perhaps someone else can help us here.
Another option (which you may also consider yuck) is to use a library like PHPExcel
You can iterate over the CSV file using fgetcsv() which converts each line to an array of values. You can accumulate the value of the array element containing the balance as you move thru each iteration until you get the sum total. Use glob to get the list of CSV files in a folder.
You may not have to "manually" total up the account balances, if you can use Excel functions from your application, the Excel formula in VBA would be:
Application.Sum(Range("A:A"))
where A:A is for column A.
Try using CSVFix with it's summary option. It will get you more data than you need, but should be easy to use.
Otherwise, this sounds like a good use for Awk.
Why can't you just automagically total up the account balances in excel with a formula and export them with the rest of the data?
A bit of a different angle: Make use of the MySql CSV engine to expose your CSV files to Mysql and then do a normal SQL SUM.
See also: The CSV Storage Engine
Related
I'm working on a small PHP application which update a stock products regularly, i'm getting the updated file from the server, and i have the old one in my directory, so what is the best way to get only the updated products(lines) between these two files, for information both files contain arround 70000 product lines.
I though to store the data of each file into an array, then use "array_diff" to compare them, it will work theoretically, but will be good idea with 70000 on each array?
Thanks in advance.
I'd use the diff command.
For reference:
https://www.geeksforgeeks.org/diff-command-linux-examples/
I am creating a website where I want people to submit location addresses. To avoid spelling mistakes I would like users to select from a list (town name, county).
I came across the following site http://www.doogal.co.uk/london_postcodes.php which allows me to download a 56 mb large csv file containing the location data I need.
However I have never used a csv file larger than 1mb before with more than 30000 rows of data on my websites. Usually I would just upload to PhpMyAdmin.
What is better? Uploading the csv file to PHPmyadmin database or accessing the '.csv' file directly using php?
Regards
It depends on what you want to achieve. I am not so sure that using a CSV is benefitial since using a database will allow you to
Cache data
Create Indexes for fast searching
Create complex queries
Do data manipulation, etc
The only way I would think a CSV is better, is if you would use always, all the data. Otherwise, I would go for a database. The end result would be much more organized, much faster, and you could build on top of it.
Hope this helps.
If you're going to do lookups, I'd recommend you put it into a database table and add indexes on the fields that you will be searching on (https://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html). A flat file is not a good way to store data that you have to access or filter quickly.
In a PHP script, I got a set of params (zip codes / addresses) that will not change frequently, so I'm looking to move this particular db table to a config file. Would it be faster reading a file containing an array with 1 million lines with zip codes as keys or a db table to scan and get the remaining items of the address (street, city, state).
Thanks,
Try to store data in database rather than file.for million line i guess database if faster than file.
if you want to achieve performance you can use cache like APCCache or use index in databse over zip field.
sphinx is opensource index which allows faster performance over text search.
Based on the number of zip codes I'd say go with the DB instead of the associative array. And you'll be able to search either addresses or zip codes or even ids.
Is it possible to write a program that change the order of columns in sheet1(excel) based on a table in standard sheet2(excel)?
Sheet1 have the same header of each column as sheet2 but the position of columns is not same and also it is possible that some more columns are present in sheet(1). Finally I have to store the standard sheet into the database.
Yes it possible to change the order of columns.
Steps:
Change the excel sheet format file to csv file.
Write a PHP code to arrange the random columns as per the standard sheet.
Write a code to enter this data into the database. Each time you receive the new column append it at the last of the standard column.
Make the respective change in the PHP code for the same.
This is probably a dumb answer, but back in the day I used VB code inside of Excel workbooks to manipulate data including culling information from one worksheet and creating another based on contents.
Try http://msdn.microsoft.com/en-us/library/office/ee814737(v=office.14).aspx for more information.
I'm attempting to clean up data in this (old) spreadsheet and need to remove things like single and double quotes, HTML tags and so on. Trouble is, it's a 3000 row file with 25 columns and every spreadsheet app I've tried (NeoOffice, MS Excel, Apple Numbers) chokes on it. Hard.
Any ideas on how else I can clean this thing up for import to MySQL? Clearly I could go through each record manually, row by row, but would like to avoid that if at all possible. Likewise, I could write a PHP script to handle it on import, but don't want to put the server into a death spiral either.
If it's in Excel format, you might try to access it through python.
http://www.python-excel.org/
I've never done this myself, but it might work in your case.
I found a solution:
Exported the spreadsheet as CSV, and with my desktop running PHP and MySQL I imported the csv file into MySQL using the "LOAD DATA" SQL command, and then used PHP + MySQL to clean up each row.
Much, much faster than asking a desktop app like Excel or Numbers to do a find & replace.