I have exported my sql dump in csv format, so suppose my schema was like name,email ,country, I want to remove email column and all its data from csv. what would be the most optimized way to do that either using a tool or any technique.I tried to load that dump in excel but that didn't looked proper
Thanks
you could copy the table inside the mysql database, delete the email column using some mysql client and export back to csv.
Importing to excel should work with ordered data - you might need to consider alternative delimiters if your data contains commas (such as addresses). If possible use an alternative delimiter, add quote marks around troublesome fields or shift to fixed width output.
Any tool you write or use will need to be able to parse your data and that will always be an issue if the delimiter is scattered through the data.
Alternatively rewrite the view / select / procedure that is generating the data set initially.
This command should do it (assuming a unix* OS):
$ cut -d ',' -f 1,3- dump.csv > newdump.csv
UPDATE: DevZer0 is right, this is unfit for the general case. So you could do (it's tested):
#!/usr/bin/env perl
use Text::ParseWords;
my $file = 'dump.csv';
my $indexOfFieldToBeRemoved = 1; # 0, 1, ...
my #data;
open(my $fh, '<', $file) or die "Can't read file '$file' [$!]\n";
while (my $line = <$fh>) {
chomp $line;
my #fields = Text::ParseWords::parse_line(',', 0, $line);
splice(#fields, $indexOfFieldToBeRemoved, 1);
foreach (#fields) {
print "\"$_\",";
}
print "\n";
}
close $fh;
Sorry, nothing simpler (if you can't re-generate csv dump, as suggested)...
Related
When using league/csv to read a csv file, how could I know which csv controls have been used to parse the file ?
I made different csv files :
delimiter_colon.csv {exported from excel with colon delimiter}
delimiter_tab.csv {exported from excel with tab delimiter}
delimiter_semicolon.csv {exported from excel with semicolon delimiter}
etc...
When using
use League\Csv\Reader;
...
//Read csv from path
$csvReader = Reader::createFromPath( $CSVFile->path );
//get the current delimiter ? Nope always the default one ...
$this->delimiter = $csvReader->getDelimiter();
EDIT: What I want to know is which delimiter has been used by the current reader.
Not the delimiter in the csv file itself.
Whatever the file I use to read the csv, it always gives "," {coma}
So I'm asking here:
How to know which delimiter/enclosure were used to parse the current csv Reader ?
I've also tried using getIterator(). Get default values too.
If you are using the latest version of the library you can simply use Reader::fetchDelimitersOccurrence as explained in the documentation. But be aware that the method will only return information about the suggested delimiters you supply. Because it is not possible for the method to know which delimiter is in use. The result is only a hint that needs to be confirm by the CSV provider.
In PHP, is there a way to force the value "00123" to be inserted into a CSV file as a string?
This way, when you open the CSV file the value will remain 00123 rather than removing the leading zeros and showing 123.
The primary reason I'd like achieve this is for a list of zipcodes, whereas there are multiple zipcodes that have leading zeros and I'd like the values to reflect that.
<?php
if( $fh = fopen('filename.csv','w') ){
$line = ['00123'];
fputcsv($fh,$line);
fclose($fh);
}
CSV does not have types. Values written using the ,"..", syntax merely delimit the value to disambiguate the usage of , within the value itself; it does not mean that the value is "a string".
I suspect your values are mangled when imported into Excel or such. There's no solution to this that CSV can offer; you can only import the file using the import wizard and specify that the column should be used as is and not cast to a number. (This may or may not actually work depending on what effed-up version of Excel you're using.)
If you don't want to go through this every time, you should be producing an XLSX file, which does have types.
I guess there is no way to do it because "CSV" files are just "Comma-Separated Values"
You have to use the editor options for csv import.
I'm writing some PHP code to import a CSV file to a Postgre DB, and I'm getting the error below. Can you help me?
Warning: pg_end_copy(): Query failed: ERROR: literal newline found in data HINT: Use "\n" to represent newline. CONTEXT: COPY t_translation, line 2 in C:\xampp\htdocs\importing_csv\importcsv.php on line 21
<?php
$connString = 'host = localhost dbname= importdb user=postgres password=pgsql';
$db = pg_connect($connString);
$file = file('translation.csv');
//pg_exec($db, "CREATE TABLE t_translation (id numeric, identifier char(100), device char(10), page char(40), english char(100), date_created char(30), date_modified char(30), created_by char(30), modified_by char(30) )");
pg_exec($db, "COPY t_translation FROM stdin");
foreach ($file as $line) {
$tmp = explode(",", $line);
pg_put_line($db, sprintf("%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", $tmp[0], $tmp[1], $tmp[2], $tmp[3], $tmp[4], $tmp[5], $tmp[6], $tmp[7], $tmp[8]));
}
pg_put_line($db, "\\.\n");
pg_end_copy($db);
?>
You need to specify FILE_IGNORE_NEW_LINES flag in file() function as a 2nd parameter which otherwise by default will include the newline char at the end of the each array item. This is likely whats causing the issue here.
So just add this flag FILE_IGNORE_NEW_LINES so that lines extracted from csv file will not have newline char at the end of the each line:
$file = file('translation.csv', FILE_IGNORE_NEW_LINES);
Also I would recommend using fgetcsv() instead to read csv file.
If you're willing to use PDO (necessitates a separate connection call), there's an elegant solution that does not require as much processing of the data by PHP, and that will also work with any combination of fields so long as their names in the CSV header match the names in the database. I'll assume you already have initialized PDO and have the object as $pdo, and the filename is $filename. Then:
$file=fopen($filename,'r');
$lines=explode("\n", fread ($file, filesize($filename)));
if (end($lines)=='') array_pop($lines); // Remove the last line if it empty, as often happens, so it doesn't generate an error with postgres
$fields=array_shift($lines); // Retrieve & remove the field list
$null_as="\\\\N"; // Or whatever your notation for NULL is, if needed
$result=$pdo->pgsqlCopyFromArray('t_translation',$lines,',',$null_as,$fields);
This is pretty minimal, there is no error handling other than $result returning success or failure, but it can be a starting point.
I like this solution better than the approach you are taking though because you don't need to specify the fields at all, it's all handled automatically.
If you don't want to use PDO, there's a similar solution using your setup and syntax, just for the last line replace it with:
pg_copy_from($db,'t_translation',$lines,',',$null_as)
This solution, however, does not dynamically adjust the field names, the fields of the CSV need to exactly match those in the table. However, the names don't need to line up, as the first line of the CSV file is ignored. I haven't tested this last line though because I don't use this type of connection, so there could be an error in it.
I was told that "\t" would 'tell' Excel that it is a new column data.
However, it is just an empty space and all my single line data are in ONE cell instead of seperate columns.
Did i do anything wrong?
while($row =mysqli_fetch_assoc($result))
{
$contents.=$row['idUsers']."\t";
$contents.=$row['First_Name']."\t";
$contents.=$row['Last_Name']."\t";
$contents.=$row['Email_Address']."\t";
$contents.=$row['Verified']."\n";
}
When you're creating a CSV file for MS Excel, you'll find that Excel's separator is locale specific, so what might work for one person won't necessarily work for another.
One way to try and force the issue is to use a sep=<x> line as the very first line of your CSV file; so if you initially define $content using
$contents = "sep=\t\n";
before starting your while loop, you may find that this allows MS Excel to correctly identify what separator you're using when the file you generate is loaded into Excel
And rather than "roll your own" csv format file, why don't you make use of PHP's built-in fputcsv() function, which will also handle quoting strings and generally simplify things for you
Have you tried the CSV format?
Replace your '\t' by a comma..
while($row =mysqli_fetch_assoc($result))
{
$contents.=$row['idUsers'].",";
$contents.=$row['First_Name'].",";
$contents.=$row['Last_Name'].",";
$contents.=$row['Email_Address'].",";
$contents.=$row['Verified']."\n";
}
http://en.wikipedia.org/wiki/Comma-separated_values
I have a series of keywords in an HTML web page - they are comma separated so I could get them to CSV, and would like to know which ones are NOTin another CSV file displayed as an html web page.
How would you do that comparison ? I have ideas for mysql and tables but this is CSV or html sources.
Thanks !
In Python, given 2 csv files, a.csv and b.csv, this script will create (or edit if it already exists) a new file out.csv that contains everything in a.csv that's not found in b.csv.
import urllib
url = 'http://www.website.com/x.csv'
urllib.urlretrieve(url, 'b.csv')
file_a = open('a.csv', 'r')
file_b = open('b.csv', 'r')
file_out = open('out.csv', 'w')
list_a = [x.strip() for x in file_a.read().split(',')]
list_b = [x.strip() for x in file_b.read().split(',')]
list_out = list(set(list_a) - set(list_b)) # Reverse if necessary
file_out.write(','.join(list_out))
file_out.close()
If it is just a list of keywords, you want to do a search and replace (you can use sed) to replace all the commas with carriage returns. So you will end up with a file containing one keyword on each line. Do that to both versions of the list. Then use the "join" command:
join -v 1 leftfile rightfile
This will report all the entries in leftfile that are not in rightfile. Don't forget to sort the files first, or join won't work. There is a bash tool for sorting too (it's called, not surprisingly, "sort").
PHP solution..
Get keywords as strings, convert then in arrays and use array_diff function:
<?php
$csv1 = 'a1, a2, a3, a4';
$csv2 = 'a1, a4';
$csv1_arr = explode(',', $csv1);
$csv2_arr = explode(',', $csv2);
$diff = array_diff($csv1_arr, $csv2_arr);
print_r($diff);
?>