I'm writing php code to parse a csv file. I have to validate the basic data type for all the fields in the csv fileds and should throw an instance of a CsvParserException if there is any broken data in the csv file. I've given a generic exception but I want to give a custom exception which is an instance of CsvParserException.
Below is my code. I'm not sure how to to properly handle this.
<?php
namespace app;
class CsvParser{
public function parse() {
$row = 1;
$flag = true;
if (($handle = fopen("file.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
echo $num;
$row++;
if($flag) { $flag = false; continue; } // this is to skip the first line
for ($c=0; $c < $num; $c++) {
echo "data[0] is = ". $data[0] . " data[c] is = ". $data[$c]. " <br />\n";
if(!((isset($data[0])) && (is_numeric($data[0])))) {
throw new \Exception('Id is not valid');
}
if(!((isset($data[1])) && (is_string($data[1])))) {
throw new \Exception('This is not a valid string. ');
}
if(!((isset($data[2])) && (is_string($data[2])))) {
throw new \Exception('This is not a valid string. ');
}
$date = strtotime($data[3]);
$splitData = explode('|', $data[4]);
foreach($splitData as $value) {
if(!((isset($value)) && (is_numeric($value)))) {
throw new \Exception('Integers is not valid');
}
}
}
}
fclose($handle);
}
}
}
?>
Can someone please help ?
Related
I am importing data from a CSV file and one column has a list of dates. Some of these dates are pre 1970. An example date would be 10/03/1956
I've got a problem in formatting these dates into a format that I can insert into MySQL.
if i use this
$date = DateTime::createFromFormat('d/m/Y', $col[8]);
I get this error message
Call to a member function format() on a non-object
but if I hard code a date like this, it works
$date = DateTime::createFromFormat('d/m/Y', '21/03/1966');
here is the complete code
$inputFileName = 'data.csv';
if (($handle = fopen($inputFileName, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($i = 0; $i < $num; $i++) {
$col[$i] = $data[$i];
}
try {
$date = DateTime::createFromFormat('d/m/Y', trim($col[8]));
}
catch (Exception $e) {
echo $e->getMessage();
exit(1);
}
echo $date->format("Y-m-d");
}
fclose($handle);
}
You have no guarantee that all fields will exist in your CSV file, so you must do a little control
define('DATE_COLUMN', 8);
// #Warning : avoid hard coding file name !
$inputFileName = 'data.csv';
if (($handle = fopen($inputFileName, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($i = 0; $i < $num; $i++) {
$col[$i] = $data[$i];
}
$dateString = NULL;
if (isset($col[DATE_COLUMN]) &&
strlen(trim($col[DATE_COLUMN])) > 0) {
$dateString = trim($col[DATE_COLUMN]);
}
if (is_null($dateString)) {
// print log or throw an exception if you want
}
else {
try {
$date = DateTime::createFromFormat('d/m/Y', dateString);
echo $date->format("Y-m-d");
}
catch (Exception $e) {
echo $e->getMessage();
exit(1);
}
}
}
fclose($handle);
}
When I run the code it is ridiculously slow even if I comment out the second half where I fopen the same file. I have changed the length of stream_get_line. File is 64MB, 73500 lines, 159 columns. Any help to optimize would be much appreciated.
<?php
include "configure.php";
$row = 1;
if(($handle = fopen("ACC_half.txt", "r")) !== false)
{
global $arraySamples;
global $arrayTrans;
global $num;
global $lines;
while (!feof($handle))
{
$data = stream_get_line($handle, 1000000, "\n");
$num = count($data);
$lines = count(file("ACC_half.txt"));
//Get all the names of the samples into array
if($row == 1)
{
for($i=0;$i<$num;$i++)
{
//echo $data[$i];
$arraySamples[]=$data[$i];
}
//Get all the names of the first row(headers) into array for use
}
else
{
$arrayTrans[]=$data[0];
}
$row++;
}
}
fclose($handle);
$row = 1;
$transInc=1;
if(($handle1 = fopen("ACC_half.txt", "r")) !== false)
{
while (($data2 = stream_get_line($handle,1000000,"\n\r")) !== false)
{
if(($row == 2) || ($row==1))
{
$row++;continue;
}
$jnum=count($data2);
for($j=1;$j<$jnum;$j+=2)
{
$g=$j+1;
$h=$j+2;
$import = mysql_query("INSERT into acc (form,sample,raw,scale)
VALUES ('$arrayTrans[$transInc]', '$arraySamples[$g]', '$data2[$j]', '$data2[$g]')")
or die (mysql_error());
}
$transInc++;
}
}
fclose($handle1);
?>
I believe that one of the main problems is that you open the same big file twice. You should open this file once not twice
I think I need a second pair of eyes on this one. For the life of me I can't figure out why my SQL INSERT query is running twice every iteration:
if (($handle = fopen($spreadsheet_url, "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if ($current_row == 1 || $current_row == 2) {
$num = count($data);
//echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
//echo $data[$c] . "<br />\n";
try {
set_time_limit(0);
$stmt = $db_temp_kalio->prepare('INSERT INTO invupdate (sku,prod) VALUES(:sku, :prod)');
$stmt->execute(array(':sku'=> $data[0], ':prod'=> $data[1])); }
catch(PDOException $e) {
echo 'ERROR: ' . $e->getMessage();
exit; }
}
}
$current_row++;
}
fclose($handle);
}
Well, I probably should have put forth a little more effort before asking for assistance. I was able to fix this by adding a row reset counter in the loop:
if (($handle = fopen($spreadsheet_url, "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if ($current_row == 1 || $current_row == 2) {
$row_reset = 0;
$num = count($data);
//echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
//echo $data[$c] . "<br />\n";
if ($row_reset == 0) {
try {
set_time_limit(0);
$stmt = $db_temp_kalio->prepare('INSERT INTO invupdate (sku,prod) VALUES(:sku, :prod)');
$stmt->execute(array(':sku'=> $data[0], ':prod'=> $data[1])); }
catch(PDOException $e) {
echo 'ERROR: ' . $e->getMessage();
exit; }
}
$row_reset++;
}
}
$current_row++;
}
fclose($handle);
}
I'm still curious if there is a better way.
I looks like you were using some code to iterate through your columns, but actually only have two, containing the sku and prod ($data[0] and $data[1].) If that is the case, then you don't need the for loop inside. (BTW, it was that loop that was causing the query to be executed twice, as the query was inside it.) It also looks like you have two counters going for the row (current_row and row) so those can be combined. If you are using the if ($currentRow == 1... statement to ignore the header on row 0 and then only process 2 rows, you will want to switch it to if ($currentRow > 0) when you are ready to run the whole spreadsheet. Using break in the catch clause instead of exit will cause the code to drop out of the while loop and hit the fclose statement:
if (($handle = fopen($spreadsheet_url, "r")) !== FALSE) {
$currentRow = 0;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if ($currentRow == 1 || $currentRow == 2) {
$num = count($data);
//echo "<p> $num fields in line $currentRow: <br /></p>\n";
//echo "<p> The data for this row is: " . $data[0] . " " . $data[1] . "\n";
try {
set_time_limit(0);
$stmt = $db_temp_kalio->prepare('INSERT INTO invupdate (sku,prod) VALUES(:sku, :prod)');
$stmt->execute(array(':sku'=> $data[0], ':prod'=> $data[1]));
}
catch(PDOException $e) {
echo 'ERROR: ' . $e->getMessage();
break;
}
}
$currentRow++;
}
fclose($handle);
}
I want to take a text file formatted like so:
*note I can change the format of the stored messages, basically there are delimiters and different lines
;68.229.164.10:4/5/2013:Hello
;71.73.174.13:4/6/2013:Oh Hey
(;IPADDRESS:TIMESTAMP:MESSAGE)
and put it in a table that looks like so:
IP Time Message
68.229.164.10 4/6/2013 Hello
71.73.174.13 4/6/2013 Oh Hey
I prefer to use something like the following:
http://php.net/manual/en/function.fgetcsv.php
And then format the output accordingly.
From Example 1 on the above referenced page:
<?php
$row = 1;
if (($handle = fopen("test.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
?>
so, you could do something like this...
<?php
$row = 1;
if (($handle = fopen("path_to_your_data_file", "r")) !== FALSE) {
echo '<table>';
echo '<tr><td></td>IP<td>Time</td><td>Message</td></tr>';
while (($data = fgetcsv($handle, 1000, ":")) !== FALSE) {
$num = count($data);
$row++;
if ($num > 2) {
echo '<tr>';
for ($c=0; $c < $num; $c++) {
echo '<td>'.$data[$c].'</td>';
}
echo '</tr>';
}
}
echo '</table>';
fclose($handle);
}
?>
$a=";68.229.164.10:4/5/2013:Hello
;71.73.174.13:4/6/2013:Oh Hey";
preg_match_all('{;(.*?):(.*?):(.*)}',$a,$d);
//set th
$d[0][0]='IP';
$d[0][1]='TIME';
$d[0][2]='Message';
$table = '<table>'.PHP_EOL;
foreach($d AS $tr){
$row = PHP_EOL;
foreach($tr AS $td){
$row .= "<td>{$td}</td>".PHP_EOL;
}
$table .= "<tr>{$row}</tr>".PHP_EOL;
}
$table .= "</table>".PHP_EOL;
echo $table;
I am trying to import a CSV file. Due to the program we use, the first row is basically all headers that I would like to skip since I've already put my own headers in via HTML. How can I get the code to skip the first row of the CSV? (the strpos command is to cut off the first field in all the rows.)
<?php
$row = 1;
if (($handle = fopen("ptt.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++) {
if(strpos($data[$c], 'Finished') !== false) {
$c++;
echo "<TR> <TD nowrap>" . $data[$c] . "</ TD>"; }
Else{
echo "<TD nowrap>" . $data[$c] . "</ TD>";
}
}
}
fclose($handle);
}
?>
Rather than using if condition for checking whether it is the first row, a better solution is to just add an extra line of code before the line from where the while loop starts as shown below :
....
.....
fgetcsv($handle);//Adding this line will skip the reading of th first line from the csv file and the reading process will begin from the second line onwards
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
.......
.......
It is just as simple......
As you are keeping track of the row number anyway, you can use continue to skip the rest of the loop for the first row.
For example, add this at the start of your while loop (just above $num = count($data)):
if($row == 1){ $row++; continue; }
There are other ways to do this, but just make sure that when you continue, $row is still being incremented or you'll get an infinite loop!
Please use the following lines of code
$flag = true;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if($flag) { $flag = false; continue; }
//your code for insert
}
Having the flag variable as true and setting it to false will skip the first line of the CSV file. This is simple and easy to implement.
put this inside your while loop:
if ($row == 1) continue;
Add this in the body of the while loop above the $row++;:
if ($row == 1) {
continue;
}
$count = 0;
while (($fields = fgetcsv($handle, 0, ",")) !== FALSE) {
$count++;
if ($count == 1) { continue; }
this worked for me:
$count = 0;
while(! feof($file))
{
$entry = fgetcsv($file, 0, ';');
if ($count > 0) {
//skip first line, header
}
$count++;
}
use this code
// mysql hostname
$hostname = 'localhost';
// mysql username
$username = 'root';
// mysql password
$password = '';
if (isset($_FILES['file']))
{
// get the csv file and open it up
$file = $_FILES['file']['tmp_name'];
//$handle is a valid file pointer to a file successfully opened by fopen(), popen(), or fsockopen().
$handle = fopen($file, "r");
try {
// Database Connection using PDO
$dbh = new PDO("mysql:host=$hostname;dbname=clasdb", $username, $password);
// prepare for insertion
$STM = $dbh->prepare('INSERT INTO statstrackertemp (ServerName, HiMemUti, AvgMemUti, HiCpuUti, AvgCpuUti, HiIOPerSec, AvgIOPerSec, HiDiskUsage, AvgDsikUsage) VALUES (?, ?, ?, ?, ?,?, ?, ?, ? )');
if ($handle !== FALSE)
{
// fgets() Gets a line from file pointer and read the first line from $handle and ignore it.
fgets($handle);
// created loop here
while (($data = fgetcsv($handle, 1000, ',')) !== FALSE)
{
$STM->execute($data);
}
fclose($handle);
}
}
catch(PDOException $e)
{
die($e->getMessage());
}
echo 'Data imported';
}
else
{
echo 'Could not import Data';
}
?>