I have this big file containing SWIFT numbers and bank names. I'm using the following php function for reading and comparing data:
function csv_query($blz) {
$cdata = -1;
$fp = fopen(DIR_WS_INCLUDES . 'data/swift.csv', 'r');
while ($data = fgetcsv($fp, 1024, ",")) {
if ($data[0] == $blz){
$cdata = array ('blz' => $data[0],
'bankname' => $data[7]);
// 'prz' => $data[2]
}
}
return $cdata;
}
The csv files looks like that:
"20730054",1,"UniCredit Bank - HypoVereinsbank (ex VereinWest)","21423","Winsen (Luhe)","UniCredit Bk ex VereinWest",,"HYVEDEMM324","68","013765","M",1,"20030000"
"20750000",1,"Sparkasse Harburg-Buxtehude","21045","Hamburg","Spk Harburg-Buxtehude","52002","NOLADE21HAM","00","011993","U",0,"00000000"
"20750000",2,"Sparkasse Harburg-Buxtehude","21605","Buxtehude","Spk Harburg-Buxtehude","52002",,"00","011242","U",0,"00000000"
As you can see from the code, I need the first and the eight string. If the first string has no duplicates everything is ok, but if it has, most likely the eighth field of the duplicate will be empty and I get no result back. So I want to ask how to display that eighth field of the first result if the line has a duplicate.
I guess this will solve your problem :
function csv_query($blz) {
$cdata = -1;
$fp = fopen(DIR_WS_INCLUDES . 'data/swift.csv', 'r');
$counter = 0; // add this line
while ($data = fgetcsv($fp, 1024, ",")) {
if ($data[0] == $blz && !$counter) { //change this line
$cdata = array(
'blz' => $data[0],
'bankname' => $data[7]
);
$counter++; //add this line
}
}
return $cdata;
}
Related
Let's assume I have a CSV file, which looks like:
[Line 1]: ID, Name, Birthday, Phonenumber, Email
[Line 2]:
[Line 3]: 1, Jim, 1978-02-01, 555-23-256, jim.doe#donotreply.com
[Line 4]: 2, Mia, 1985-12-21, 555-23-876, mia#donotreply.com
[Line 5]: 3, Wil, 1962-05-07, 555-23-456, wil.doe#donotreply.com
Now I am trying to wrap my head around this question:
How can I parse this file with PHP to get the following array?
$people = array(
[1] = array(
[Name] => "Jim",
[Birthday] => "1978-02-01",
[Phonenumber] = "555-23-256",
[Email] => "jim.doe#donotreply.com",
),
...
);
Any idea?
Use fopen and fgetcsv
In order to do this you need to first write a function for retrieving the data from the CSV and storing it in a variable, the function below will do this using fopen.
First, you'll need a function to read the CSV:
function utf8_fopen_read($fileName) {
$fc = iconv('windows-1250', 'utf-8//IGNORE', file_get_contents($fileName));
$handle = fopen("php://memory", "rw");
fwrite($handle, $fc);
fseek($handle, 0);
return $handle;
}
Once you've done this you'll want to manipulate the data you've just retrieved so you can create a multi-dimensional array of the results, populating the data into the associative keys like you described in your question.
Now we are going to make a call to the function above and store the results in $data, using a while loop we are going to iterate through the entire CSV and store all of the information into $people row by row.
Populating your array:
$people = [];
if(($handle = utf8_fopen_read('path/to/csv.csv')) !== FALSE) {
while(($data = fgetcsv($handle, 0, ',')) !== FALSE) {
$people[$i] = [];
for($c = 0; $c < count($data); $c++) {
$info = $data[$c];
if($i == 0) $fields[$c] = str_replace(' ', '', $info);
if($i > 0) $people[$ii][$fields[$c]] = $info;
}
if($i > 0) $ii++;
$i++;
}
fclose($handle);
}
I have a text file (which essentially is a csv without the extension) that has 150,000 lines in it. I need to remove duplicates by key then insert them into the database. I'm attempting fgetcvs to read it line by line, but I don't want to do 150,000 queries. So this is what I came up with so far: (keep in mind i'm using laravel)
$count = 0;
$insert = [];
if (($handle = fopen("myHUGEfile.txt", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$count++;
//See if this is the top row, which in this case are column headers
if ($count == 1) continue;
//Get the parts needed for the new part
$quantity = $data[0];
$part_number = $data[1];
$manufacturer = $data[2];
$new_part = [
'manufacturer' => $manufacturer,
'part_number' => $part_number,
'stock' => $quantity,
'price' => '[]',
'approved' => 0,
];
$insert[] = $new_part;
}
fclose($handle);
} else {
throw new Exception('Could not open file for reading.');
}
//Remove duplicates
$newRows = [];
$parsedCount = 0;
foreach ($insert as $row) {
$x = 0;
foreach ($newRows as $n) {
if (strtoupper($row['part_number']) === strtoupper($n['part_number'])) {
$x++;
}
}
if ($x == 0) {
$parsedCount++;
$newRows[] = $row;
}
}
$parsed_rows = array_chunk($newRows, 1000, true);
$x = 0;
foreach ($parsed_rows as $chunk) {
//Insert
if (count($chunk) > 0)
if (DB::table('search_parts')->insert($chunk))
$x++;
}
echo $x . " chunks inserted.<br/>" . $count . " parts started with<br/>" . $parsedCount . " rows after duplicates removed.";
But it's very clunky, I have only tested it with a little over 1000 rows and it works using localhost. But i'm afraid if I push it up to production it won't be able to handle all 150,000 rows. The file is about 4mb.
Can someone show me a better more efficient way to do this?
Right now, you're keeping the first duplicate record. If you're ok keeping the last dupe, you can just change
$insert[] = $new_part;
to
$insert[strtoupper($part_number)] = $new_part
That way, your $insert array will only have one value for each $part_number. Your inserts will be a little slower, but you can drop all of the code which checks for duplicates which looks very, very slow.
4Mb is not remotely a "huge" file. I'd just read the whole thing into an assoc array keyed by part number, which will inherently de-dupe, giving you the last row whenever a duplicate is encountered. Something like this maybe:
$parts = [];
foreach (explode("\n", file_get_contents('file')) as $line) {
$part = str_getcsv($line);
$parts[$part[1]] = [
'manufacturer' => $part[2],
'part_number' => $part[1],
'stock' => $part[0],
'price' => '[]',
'approved' => 0,
];
}
// $parts now contains unique part list
foreach ($parts as $part) {
$db->insert($part);
}
If you don't want duplicates on a certain or multiple keys, you can make it easy on yourself and just add a UNIQUE INDEX on the key you don't want duplicates for on the table.
This way, all you have to worry about is processing the file. When it reaches a duplicate key, it will not be able to insert it and will continue.
It would also make it easier in the future because you wouldn't have to modify your code if you need to do checks on additional columns. Just modify the index.
I know there are a lot of resources out there for putting a CSV into an associative array, but I can't find anything that helps a noob like me do exactly what I want to do.
I currently have an associative array defined inside my PHP file:
$users = array(
'v4f25' => 'Stan Parker',
'ntl35' => 'John Smith',
);
I would like to move that array into a CSV file (users.txt) so:
v4f25, Stan Parker
ntl35, John Smith
The next step is to import users.txt so I can use it precisely like I was using the array $users.
Any help here? The last code I tried returned this: (which is not what I want)
array(2) {
["v4f25"]=>
string(5) "ntl35"
["Stan Parker"]=>
string(10) "John Smith"
}
What about the following?
$data = array();
if ($fp = fopen('csvfile.csv', 'r')) {
while (!feof($fp)) {
$row = fgetcsv($fp);
$data[$row[0]] = $row[1];
}
fclose($fp);
}
$users = array(
'v4f25' => 'Stan Parker',
'ntl35' => 'John Smith',
);
$fp = fopen('users.txt', 'w');
if ($fp) {
foreach ($users as $key => $value) {
fputcsv($fp, array($key, $value));
}
fclose($fp);
} else {
exit('Could not open CSV file')
}
See: fputcsv()
UPDATE - in the comments you're interested in how to read the file and get your users back out. Here's the return trip:
$users = array();
if (($handle = fopen("my-csv-file.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$users[$data[0]] = $data[1];
}
fclose($handle);
} else {
exit('Could not open CSV file');
}
if (count($users) == 0) {
exit('CSV file empty: no users found');
}
Here's a solution using fputcsv() which flattens the key/value pairs to an array before writing to disk.
$filehandle = fopen("csvfile.csv", "w");
if ($filehandle) {
foreach ($users as $key => $value) {
fputcsv($filehandle, array($key, $value);
}
fclose($filehandle);
}
else // couldn't open file
Try this (assuming your strings contain no commas):
$users = array(
'v4f25' => 'Stan Parker',
'ntl35' => 'John Smith',
);
foreach ($users as $k => $v) {
print "$k, $v\n";
}
Obviously you could then create the CSV file like so:
php above_script.php > outfile.csv
Now, to get from CSV back into an array you could use something like:
$file = 'outfile.csv';
$arr = array();
if (file_exists($file)) {
foreach (explode("\n", file_get_contents($file)) as $l) {
list($k, $v) = explode(',', $l);
$arr[trim($k)] = trim($l);
}
}
print_r($arr, true);
NOTES:
If your strings do (or might) contain commas, then you'll probably want to use a PHP builtin function to decode them - in which case the answers by harald and artlung are useful.
RFC 4180 describes how commas (and other values) are encoded in CSV, in case you want to roll your own CSV encoding/decoding functions for whatever reason.
I've seen numerous examples on how to take a CSV file and then create an associative array with the headers as the keys.
For example:
Brand,Model,Part,Test
Honda,Civic,123,244
Honda,Civic,135,434
Toyota,Supra,511,664
Where it would create an Array such as Array[$num][$key] where $key would be Brand, Model, Part, Test.
So If I wanted to access the test value "434" I would have to loop every index in the array and then ignore any Brands that were not honda, and any models that were not Civic
What I need to do is access the value most directly, instead of running through a for loop going through each $num index. I want to be able to access the value test "434" with:
Array['Honda']['Civic']['135']
or control a for statement with looping through every model Honda has... something like
foreach $model in Array['Honda']
At the very least I need to be able to go through every model given a known Brand and access all the relative info for each.
Edit:
Just to confirm I was setting this up an example. My actually data has headers like:
brand model part price shipping description footnote
Of which I need to access all the information tied to the part (price, shipping,desc, footnote)
Too many long solutions. I've always found this to be the simplest:
<?php
/* Map Rows and Loop Through Them */
$rows = array_map('str_getcsv', file('file.csv'));
$header = array_shift($rows);
$csv = array();
foreach($rows as $row) {
$csv[] = array_combine($header, $row);
}
?>
run over the csv file line by line, and insert to array like:
$array = $fields = array(); $i = 0;
$handle = #fopen("file.csv", "r");
if ($handle) {
while (($row = fgetcsv($handle, 4096)) !== false) {
if (empty($fields)) {
$fields = $row;
continue;
}
foreach ($row as $k=>$value) {
$array[$i][$fields[$k]] = $value;
}
$i++;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
To create an associative list array use something like:
$keys = fgetcsv($f);
while (!feof($f)) {
$array[] = array_combine($keys, fgetcsv($f));
}
And to traverse and filter by specific attributes write a function like:
function find($find) {
foreach ($array as $row) {
if (array_intersect_assoc($row, $find) == $find) {
$result[] = $row;
}
}
}
Where you would invoke it with $find = array(Brand=>Honda, Model=>Civic, Part=>135) to filter out the searched models. The other positional array structure seems not very workable, unless you only want to access the "Test" attribute.
Try this simple algorithm:
$assocData = array();
if( ($handle = fopen( $importedCSVFile, "r")) !== FALSE) {
$rowCounter = 0;
while (($rowData = fgetcsv($handle, 0, ",")) !== FALSE) {
if( 0 === $rowCounter) {
$headerRecord = $rowData;
} else {
foreach( $rowData as $key => $value) {
$assocData[ $rowCounter - 1][ $headerRecord[ $key] ] = $value;
}
}
$rowCounter++;
}
fclose($handle);
}
var_dump( $assocData);
Using fgetcsv() seems the most direct and sensible tool for the job.
csv.csv contents:
Brand,Model,Part,Test
Honda,Civic,123,244
Honda,Civic,135,434
Toyota,Supra,511,664
Code:
$assoc_array = [];
if (($handle = fopen("csv.csv", "r")) !== false) { // open for reading
if (($data = fgetcsv($handle, 1000, ",")) !== false) { // extract header data
$keys = $data; // save as keys
}
while (($data = fgetcsv($handle, 1000, ",")) !== false) { // loop remaining rows of data
$assoc_array[] = array_combine($keys, $data); // push associative subarrays
}
fclose($handle); // close when done
}
echo "<pre>";
var_export($assoc_array); // print to screen
echo "</pre>";
Output:
array (
0 =>
array (
'Brand' => 'Honda',
'Model' => 'Civic',
'Part' => '123',
'Test' => '244',
),
1 =>
array (
'Brand' => 'Honda',
'Model' => 'Civic',
'Part' => '135',
'Test' => '434',
),
2 =>
array (
'Brand' => 'Toyota',
'Model' => 'Supra',
'Part' => '511',
'Test' => '664',
),
)
Resource: http://php.net/manual/en/function.fgetcsv.php
Here is a solutions that will work by specifying a local file or URL. You can also switch the association on and off. Hopefully this helps.
class CSVData{
public $file;
public $data;
public $fp;
public $caption=true;
public function CSVData($file=''){
if ($file!='') getData($file);
}
function getData($file){
if (strpos($file, 'tp://')!==false){
copy ($file, '/tmp/csvdata.csv');
if ($this->fp=fopen('/tmp/csvdata.csv', 'r')!==FALSE){
$this->readCSV();
unlink('tmp/csvdata.csv');
}
} else {
$this->fp=fopen($file, 'r');
$this->readCSV();
}
fclose($this->fp);
}
private function readCSV(){
if ($this->caption==true){
if (($captions=fgetcsv($this->fp, 1000, ","))==false) return false;
}
$row=0;
while (($data = fgetcsv($this->fp, 1000, ",")) !== FALSE) {
for ($c=0; $c < count($data); $c++) {
$this->data[$row][$c]=$data[$c];
if ($this->caption==true){
$this->data[$row][$captions[$c]]=$data[$c];
}
}
$row++;
}
}
}
Try this usage:
$o=new CSVData();
$o->getData('/home/site/datafile.csv');
$data=$o->data;
print_r($data);
Here is my solution, similar to others stated but uses a while loop with fgetcsv, and uses a counter and array_combine to set the first row as the keys.
$rownum = 0;
while (($row = fgetcsv($openedFile, 1000, ',')) !== FALSE) {
if ($rownum > 0) {
$row = array_combine($importarray[0], $row);
}
array_push($importarray, $row);
$rownum++;
}
array_shift($importarray);
I have a csv file which looks like this
$lines[0] = "text, with commas", "another text", 123, "text",5;
$lines[1] = "some without commas", "another text", 123, "text";
$lines[2] = "some text with commas or no",, 123, "text";
And I would like to have a table:
$t[0] = array("text, with commas", "another text", "123", "text","5");
$t[1] = array("some without commas", "another text", "123", "text");
$t[2] = array("some text, with comma,s or no", NULL , "123", "text");
If I use split($lines[0],",") I'll get "text" ,"with commas" ...
Is there any elegant way to do it?
You can use fgetcsv to parse a CSV file without having to worry about parsing it yourself.
Example from PHP Manual:
$row = 1;
if (($handle = fopen("test.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
In addition to Matt's suggestion, you can also use SplFileObject to read in the file:
$file = new SplFileObject("data.csv");
$file->setFlags(SplFileObject::READ_CSV);
$file->setCsvControl(',', '"', '\\'); // this is the default anyway though
foreach ($file as $row) {
list ($fruit, $quantity) = $row;
// Do something with values
}
source: http://de.php.net/manual/en/splfileobject.setcsvcontrol.php
you can read the data using the following function.
function readCSV() {
$csv = array_map('str_getcsv', file('data.csv'));
array_shift($csv); //remove headers
}
http://www.pearlbells.co.uk/how-to-sort-a1a2-z9z10aa1aa2-az9az10-using-php/
here is also a simple method to get read csv file.
$sfp = fopen('/path/to/source.csv','r');
$dfp = fopen('/path/to/destination.csv','w');
while ($row = fgetcsv($sfp,10000,",","")) {
$goodstuff = "";
$goodstuff = str_replace("¦",",",$row[2]);
$goodstuff .= "\n";
fwrite($dfp,$goodstuff);
}
fclose($sfp);
fclose($dfp);
Maybe my code solves your problem:
// Parse all content from csv file and generate array from line.
function csv_content_parser($content) {
foreach (explode("\n", $content) as $line) {
// Generator saves state and can be resumed when the next value is required.
yield str_getcsv($line);
}
}
// Get content from csv file.
$content = file_get_contents('your_file.csv');
// Create one array from csv file's lines.
$data = array();
foreach (csv_content_parser($content) as $fields) {
array_push($data, $fields);
}
In result you have an array with all values from csv.
It would be something like:
Array
(
[0] => Array
(
[0] => text, with commas
[1] => another text
[2] => 123
[3] => text
[4] => 5
)
[1] => Array
(
[0] => some without commas
[1] => another text
[2] => 123
[3] => text
)
[2] => Array
(
[0] => some text, with comma,s or no
[1] => NULL
[2] => 123
[3] => text
)
)
Suppose you have a create a function for same things, Then it should look like
function csvtoarray($filename='', $delimiter){
if(!file_exists($filename) || !is_readable($filename)) return FALSE;
$header = NULL;
$data = array();
if (($handle = fopen($filename, 'r')) !== FALSE ) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE)
{
if(!$header){
$header = $row;
}else{
$data[] = array_combine($header, $row);
}
}
fclose($handle);
}
if(file_exists($filename)) #unlink($filename);
return $data;
}
$data = csvtoarray('file.csv', ',');
print_r($data);
You could use something like https://github.com/htmlburger/carbon-csv that allows column mapping:
$csv = new \Carbon_CSV\CsvFile('path-to-file/filename.csv');
$csv->set_column_names([
0 => 'first_name',
1 => 'last_name',
2 => 'company_name',
3 => 'address',
]);
foreach ($csv as $row) {
print_r($row);
}
The result of the below code would be something like:
Array
(
[0] => Array
(
[first_name] => John
[last_name] => Doe
[company_name] => Simple Company Name
[address] => Street Name, 1234, City Name, Country Name
)
[1] => Array
(
[first_name] => Jane
[last_name] => Doe
[company_name] => Nice Company Name
[address] => Street Name, 5678, City Name, Country Name
)
)
Another library that does the same thing(and much more) is http://csv.thephpleague.com/9.0/reader/
When you want to keep the index (first line) for multidimensional result array, you can use:
$delim = ';';
$csvFile = file($csv_file);
$firstline = str_getcsv($csvFile[0], $delim);
$data = array();
foreach ($csvFile as $line) {
$line = str_getcsv($line, $delim);
$data[] = array_combine($firstline, $line);
}
I've built an application to extract data from a CSV file , this php application was used to show a daily quote for users.
The full project on github: 365-quotes-php-csv.
Also this is the class Code for the application i've built
<?php
/*
Main Class
please note :
1- the CSV file must be comma separated (,) and each line must End with (;).
2- Feel free to edit the all.CSV file and add all of your 366 New Quotes.
3- don't change any thing specially the CSV file Location.
---------------------------------------------------------------------------
RISS.WORK all copy rights reserved 2018
please Don't Remove
Github/RissWork
Email : info#riss.work
*/
class Quote{
//properties
private $_quote,$_allQuotes;
private static $_instance = null;
//Constructor
private function __construct(){
//Day Count
$dayCount = date(z);
if($this->readCsvAndGetQuote($dayCount)){
return $this->getQuote();
}else{
echo 'Error Cannot open the .CSV File';
}
}
//Methods
//get Instance
public function getInstance(){
if(!isset(self::$_instance)){
self::$_instance = new Quote();
}
return self::$_instance;
}//end of get Instance
//get daily Quote
public function getQuote(){
return $this->_quote;
}//end of get Quote
//Read CSV
private function readCsvAndGetQuote($dayCount = 1 ){
if(($handel = fopen("csv/all.csv" , "r")) !== false){
$this->_allQuotes = fgetcsv($handel,1000000,';');
$this->_quote = explode(',',$this->_allQuotes[$dayCount]);
return true;
}
return false;
}//end of read CSV
}//end of Class
Return a php mapping array with the column of interests :
public function extractCSVDatas($file_uri) {
$AliasToSystemPathMappingArray = [];
if (($handle = fopen($file_uri, "r")) !== FALSE) {
$csv = array_map('str_getcsv', file($file_uri));
//remove header and choose columns among the list:
foreach((array_slice($csv,1)) as $line) {
list($id, $alias, $systemPath) = explode(';',$line[0]);
$AliasToSystemPathMappingArray[] = [$alias, $systemPath];
}
fclose($handle);
}
return $AliasToSystemPathMappingArray;
}
/**
* #return mixed[]
*/
public function csvToArray(string $delimiter, string $filename = ''): array
{
$data = [];
if (file_exists($filename) && is_readable($filename)) {
$header = null;
if (($handle = fopen($filename, 'r')) !== false) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== false) {
if (!$header) {
$header = $row;
} else {
$data[] = array_combine($header, $row);
}
}
fclose($handle);
}
}
return $data;
}
And why must it be simple when it can be complicated?
Joke aside, here is a quick and easy solution using the PHP function str_getcsv()
Here is an example:
function parse_csv( $filename_or_text, $delimiter=',', $enclosure='"', $linebreak="\n" )
{
$return = array();
if(false !== ($csv = (filter_var($filename_or_text, FILTER_VALIDATE_URL) ? file_get_contents($filename_or_text) : $filename_or_text)))
{
$csv = trim($csv);
$csv = mb_convert_encoding($csv, 'UTF-16LE');
foreach(str_getcsv($csv, $linebreak, $enclosure) as $row){
$col = str_getcsv($row, $delimiter, $enclosure);
$col = array_map('trim', $col);
$return[] = $col;
}
}
else
{
throw new \Exception('Can not open the file.');
$return = false;
}
return $return;
}
Imagine a situation where you need a function that works with both URL and comma delimited text. This is exactly the function that works like that. Just simply insert a CSV URL or comma separated text and it work nicely.
Just created a function that extracts data from .csv file or csv formatted text and then parses it for more convenient usage (presuming the first line is the columns). Works with however many rows/columns you'd like/have, just simply point the function to the file location/string and it will return the results!
function csvParse($input, $callback = false){
$results = [];
$raw_array = (is_file($input)) ? array_map('str_getcsv', file($input)) : array_map('str_getcsv', explode("\n", $input));
$array = array_splice($raw_array, 1, count($raw_array));
foreach($raw_array[0] as $c) $columns[] = $c;
foreach($array as $key0 => $val0)
foreach($val0 as $key1 => $val1)
$results[$key0][$columns[$key1]] = $val1;
if(is_callable($callback)) call_user_func_array($callback, array($results));
else return $results;
}
# Either One Would Work
$input = "dir/file.csv";
$input = "name,age,occupation,city\nCrimin4L,24,Programmer,New York\nMrAwesome,20,Gamer,Los Angeles";
# Usage #1
$array = csvParse($input);
var_export($array);
# Usage #2
csvParse($input, function($array){
var_export($array);
});
Input:
name,age,occupation,city
Crimin4L,24,Programmer,New York
MrAwesome,20,Gamer,Los Angeles
Array Output:
array (
0 =>
array (
'name' => 'Crimin4L',
'age' => '24',
'occupation' => 'programmer',
'city' => 'New York',
),
1 =>
array (
'name' => 'MrAwesome',
'age' => '20',
'occupation' => 'gamer',
'city' => 'Los Angeles',
),
)
Live Demo: https://ideone.com/Sa1aMO