I've had no issue with the first line of a CSV being read upon upload and then inserted into the database, but the multi line task is having issues. I can now get it to print an array dump on the web page that shows all lines of the CSV but it's still only inserting the first line into my database. I've worked my code around and came up with this:
if(isset($_POST['submit']))
{
ini_set('auto_detect_line_endings', true);
$file = $_FILES["file"]["tmp_name"];
$handle = fopen($file, "r");
while(!feof($handle)){
$filesop = print_r(fgetcsv($handle, 0, ","));
}
$coldata = array();
I have this code for calling the queries (there are other queries after this, but this is the main one that puts into that staging table):
$tablenames = array("staging"/*,"clients","meters","tests","costs","workorders"*/);
for($tableno = 0;$tableno < sizeof($tablenames);$tableno++){
$q = "";
$q2 = "";
$q3 = "";
$q4 = "";
$q5 = "";
$q6 = "";
$col_list = '`'.str_replace(',','`,`',$table_cols[$tableno]).'`';
$q .= "INSERT INTO ".$tablenames[$tableno]." (".$col_list.") VALUES (";
$last_id = mysqli_insert_id($connect);
$cols = explode(",",$table_cols[$tableno]);
$data = array();
foreach($cols as $key => $fldname) {
$data[] = "'".$coldata[$fldname]."'";
}
/*INSERT INTO STAGING TABLE - INITAL CSV UPLOAD*/
$q .= implode(",",$data).");";
Am I doing something wrong in the queries that's causing it to not insert all lines of the CSV into the db?
Related
I am trying to read CSV file using fgetcsv() php function but It doesn't fetch detail as it supposed to be. I found out that the CSV file is cluttered and has multiple irrelevant commas in starting and in-between. How do I make this CSV cleaner?
I used str_replace() php function to remove triplets of commas but the commas in starting are still giving me a problem. I tried ltrim() also but that didn't work too.
<?php
$file = "grid.csv";
$s = file_get_contents($file);
$s = str_replace(",,,", "", $s);
//$s = ltrim($s,",");
$f = "grid1.csv";
$handle = fopen($f, "w");
fwrite($handle, $s);
?>
I expect the output of this code to be a clean csv file. But I get multiple commas in starting now also in the new file.
This is the Main Code where I was trying to read the file using fgetcsv().
if(isset($_POST["submit"])){
echo "in submit</br>";
if($_FILES['csv_info']['name']){
echo "some file</br>";
$filename = explode(".", $_FILES['csv_info']['name']);
if(end($filename) == 'csv'){
echo "file is csv</br>".$_FILES['csv_info']['tmp_name'];
$handle = fopen($_FILES['csv_info']['tmp_name'],"r");
$sid = 0;
//$query = "select exists(select 1 from tblMarks)";
//$choice = mysqli_query($conn, $query);
while($data = fgetcsv($handle)){
if($sid == 0){
$sid = $sid + 1;
continue;
}
//echo $data;
$name = mysqli_real_escape_string($conn, $data[0]);
$physics = mysqli_real_escape_string($conn, $data[1]);
$maths = mysqli_real_escape_string($conn, $data[2]);
$chemistry = mysqli_real_escape_string($conn, $data[3]);
$bio = mysqli_real_escape_string($conn, $data[4]);
$sst = mysqli_real_escape_string($conn, $data[5]);
echo "</br>inserting sid".$sid." name=".$name." physics=".$physics." maths=".$maths." chemistry=".$chemistry." bio=".$bio." sst=".$sst."</br>";
//$query = "insert into tblMarks (sid, name, physics, maths, chemistry, bio, sst) values ('$sid', '$name', '$physics', '$maths','$chemistry','$bio','$sst') on duplicate key update name = '$name', physics = '$physics',maths = '$maths', chemistry = '$chemistry', bio = '$bio', sst ='$sst'";
//mysqli_query($conn, $query);
$sid = $sid + 1;
}
fclose($handle);
}
else{
$message = '<label class="text-danger">Please Select CSV File Only</lable>';
}
}
else{
$message = '<label class="text-danger">Please Select File</label>';
}
}
The output was this:
OUTPUT
here is the correct method to read CSV file row by row. There are many rows in the CSV files which have blank values, to remove theme array_filter has been used.
$temp = array();
if (($h = fopen("grid.csv", "r")) !== FALSE)
{
// Convert each line into the local $data variable
while (($data = fgetcsv($h, 1000, ",")) !== FALSE)
{
$data = array_filter($data);
if(count($data) > 0){
$temp[] = $data;
}
}
fclose($h);
}
//Write csv file
$fp = fopen('grid1.csv', 'w');
foreach ($temp as $fields) {
fputcsv($fp, $fields);
}
fclose($fp);
I wrote a php script to extract data from a csv script and insert it into an mysql database. It does the job but does not extract data and insert into database for large csv files.(I tried extracting and inserting from a csv file with 40,000 rows. Did not work but worked when i tested it out with a far smaller (60 rows) size csv file). I get no error message. It just doesn't work. During debugging i realised even echoing a string after selecting a large csv file and clicking my upload button does not work. I've searched and tried all the possible configurations in php.ini as well as LimitRequestBody for apache config as well as setting configs directly in my code to no success. Some help would be greatly appreciated. My code is below.
<?php
include ("connection.php");
if(isset($_POST["submit"]))
{
ini_set('auto_detect_line_endings',TRUE);
$file_name = $_FILES['file']['tmp_name'];
$sql10 = "insert into upload (filename) values ('$file_name')";
if(mysqli_query($conn, $sql10)){
$last_id = mysqli_insert_id($conn);
echo "New record created successfully. Last inserted ID is: " .$last_id;
}
//mysqli_close($conn);
$file = $_FILES['file']['tmp_name'];
//ini_set("auto_detect_line_endings", true);
$handle = fopen($file, "r");
$c = 0;
while(($filesop = fgetcsv($handle, ",")) !== false)
{
if ($c > 0) {
$event_id = $filesop[0];
$conf_session_id = $filesop[1];
$service_provider_id = $filesop[2];
$service_provider_id2 = $filesop[3];
$billed_flag = $filesop[4];
$offering_id = $filesop[5];
$time_start = $filesop[6];
$time_ended = $filesop[7];
$duration = $filesop[8];
$orig_calling_code = $filesop[9];
$orig_area_code = $filesop[10];
$termination_reason = $filesop[11];
$rtp_encoding = $filesop[12];
$rtp_clock_rate = $filesop[13];
$sip_from = $filesop[14];
$sip_to = $filesop[15];
$sip_call_id = $filesop[16];
$orig_nbr = $filesop[17];
$dest_nbr = $filesop[18];
$popd_account_number = $filesop[19];
$intl_dest_flag = $filesop[20];
$sip_status = $filesop[21];
$gw_ip_ingress = $filesop[22];
$gw_port_egress = $filesop[23];
$phone_number = $filesop[24];
$orig_flag = $filesop[25];
$subscriber_sip_caller_id = $filesop[26];
$orig_route_type = $filesop[27];
$term_route_type = $filesop[28];
$call_type = $filesop[29];
$mny_BasePrice = $filesop[30];
$call_cost = $filesop[31];
$uploadID = $last_id;
$sql11 = "insert into cdr (event_id, conf_session_id, service_provider_id, service_provider_id2, billed_flag, offering_id, time_start, time_ended, duration, orig_calling_code, orig_area_code, termination_reason, rtp_encoding, rtp_clock_rate, sip_from, sip_to, sip_call_id, orig_nbr, dest_nbr, popd_account_number, intl_dest_flag, sip_status, gw_ip_ingress, gw_port_egress, phone_number, orig_flag, subscriber_sip_caller_id, orig_route_type, term_route_type, call_type, mny_BasePrice, call_cost, uploadID, date_uploaded) values ('$event_id', '$conf_session_id', '$service_provider_id', '$service_provider_id2', '$billed_flag', '$offering_id', '$time_start', '$time_ended', '$duration', '$orig_calling_code', '$orig_area_code', '$termination_reason', '$rtp_encoding', '$rtp_clock_rate', '$sip_from', '$sip_to', '$sip_call_id', '$orig_nbr', '$dest_nbr', '$popd_account_number', '$intl_dest_flag', '$sip_status', '$gw_ip_ingress', '$gw_port_egress', '$phone_number', '$orig_flag', '$subscriber_sip_caller_id', '$orig_route_type', '$term_route_type', '$call_type', '$mny_BasePrice', '$call_cost', '$uploadID', CURRENT_DATE)";
if(mysqli_query($conn, $sql11)){
// $last_id = mysqli_insert_id($conn);
// echo "New record created successfully. Last inserted ID is: " .$last_id;
$updatequery = "UPDATE cdr JOIN client ON cdr.orig_nbr = client.phonenumber SET cdr.clientname = client.clientname";
mysqli_query($conn, $updatequery);
} else{
echo "error: ".mysqli_error($conn);
}
}
$c++;
}
fclose($handle) ;
}
?>
I am curently trying to upload a CSV file into a sql table, but I don't want to upload the first line which contains the name of every column value that will be inserted. I used a variable c to count the number of rows that will be inserted. For the first line c=0 so I used a condition to upload the values from the row only if c!==0, but it stil uploades the values from the first line.
My code looks like this:
$file = $_FILES['file']['tmp_name'];
$handle = fopen($file, "r");
$c = 0;
if ($_FILES["file"]["type"]=='application/vnd.ms-excel')
{
while(($filesop = fgetcsv($handle, 3000, ",")) !== false)
{
$tid = trim($filesop[0]);
$beneficiar = ucwords(strtolower(trim($filesop[1])));
$locatie = ucwords(strtolower(trim($filesop[2])));
$localitate = ucwords(strtolower(trim($filesop[3])));
$judet = ucwords(strtolower(trim($filesop[4])));
$adresa = ucwords(strtolower(trim($filesop[5])));
$model = trim($filesop[6]);
$qry = mysql_query("SELECT id FROM beneficiari WHERE `nume` = '".$beneficiar."'");
while ($row = mysql_fetch_assoc($qry)){
$id_client=$row['id'];
}
$qry_id_model=mysqli_query("SELECT id FROM modele WHERE `model` = '".$model."'");
while ($row = mysql_fetch_assoc($qry_id_model)){
$id_model=$row['id'];
}
$adresa1 = $adresa.",".$localitate;
if ($c!==0){
$sql = mysql_query(
"INSERT INTO pos_equipments_other
(id_client, model, tid, beneficiar, adresa, agentie, judet)
VALUES
('$id_client','$id_model','$tid','$beneficiar','$adresa1',
'$locatie','$judet')"
);
}
$c = $c + 1;
}
}
I am using the following to upload CSV to a table in MYSQL database.
Question: How do I bypass the 1st field? - as in ignore the header - which is currently being saved in my table?
Code:
<?php
/********************************/
/* Code at http://legend.ws/blog/tips-tricks/csv-php-mysql-import/
/* Edit the entries below to reflect the appropriate values
/********************************/
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername ="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
/********************************/
/* Would you like to add an ampty field at the beginning of these records?
/* This is useful if you have a table with the first field being an auto_increment integer
/* and the csv file does not have such as empty field before the records.
/* Set 1 for yes and 0 for no. ATTENTION: don't set to 1 if you are not sure.
/* This can dump data in the wrong fields if this extra field does not exist in the table
/********************************/
$addauto = 0;
/********************************/
/* Would you like to save the mysql queries in a file? If yes set $save to 1.
/* Permission on the file should be set to 777. Either upload a sample file through ftp and
/* change the permissions, or execute at the prompt: touch output.sql && chmod 777 output.sql
/********************************/
$save = 1;
$outputfile = "output.sql";
/********************************/
if(!file_exists($csvfile)) {
echo "File not found. Make sure you specified the correct path.\n";
exit;
}
$file = fopen($csvfile,"r");
if(!$file) {
echo "Error opening data file.\n";
exit;
}
$size = filesize($csvfile);
if(!$size) {
echo "File is empty.\n";
exit;
}
$csvcontent = fread($file,$size);
fclose($file);
$con = #mysql_connect($databasehost,$databaseusername,$databasepassword) or die(mysql_error());
#mysql_select_db($databasename) or die(mysql_error());
$lines = 0;
$queries = "";
$linearray = array();
foreach(split($lineseparator,$csvcontent) as $line) {
$lines++;
$line = trim($line," \t");
$line = str_replace("\r","",$line);
/************************************
This line escapes the special character. remove it if entries are already escaped in the csv file
************************************/
$line = str_replace("'","\'",$line);
/*************************************/
$linearray = explode($fieldseparator,$line);
$linemysql = implode("','",$linearray);
if($addauto)
$query = "insert into $databasetable values('','$linemysql');";
else
$query = "insert into $databasetable values('$linemysql');";
$queries .= $query . "\n";
#mysql_query($query);
}
#mysql_close($con);
if($save) {
if(!is_writable($outputfile)) {
echo "File is not writable, check permissions.\n";
}
else {
$file2 = fopen($outputfile,"w");
if(!$file2) {
echo "Error writing to the output file.\n";
}
else {
fwrite($file2,$queries);
fclose($file2);
}
}
}
echo "Found a total of $lines records in this csv file.\n";
?>
I has been did the same purpose at last week. Please refer this code. It may be help you.
if (isset($_POST['submit'])) {
//Path of Uploading file
$target = "upload_files/results/";
$filename = $_FILES['upload_result']['name'];
if(file_exists($target.$filename)) {
unlink($target.$filename);
}
move_uploaded_file($_FILES['upload_result']['tmp_name'], $target.$filename);
//Store the Detail into the Master table...[class_master]
$new_master = mysql_query("INSERT INTO result_master(classname,sectionname,examname,filename) Values('$class','$section','$examname','$filename')");
$filename1 = explode(".",$filename);
$file = preg_replace("/[^a-zA-Z0-9]+/", "", $filename1[0]);
// get structure from csv and insert db
ini_set('auto_detect_line_endings',TRUE);
$handle = fopen($target.$filename,'r');
// first row, structure
if ( ($data = fgetcsv($handle) ) === FALSE ) {
echo "Cannot read from csv $file";die();
}
$fields = array();
$field_count = 0;
for($i=0;$i<count($data); $i++) {
$f = trim($data[$i]);
if ($f) {
// normalize the field name, strip to 20 chars if too long
$f = substr(preg_replace ('/[^0-9a-z]/', '_', $f), 0, 20);
$field_count++;
$fields[] = $f.' VARCHAR(50)';
}
}
$drop = mysql_query("Drop table IF EXISTS $file;");
$sql = "CREATE TABLE $file (" . implode(', ', $fields) . ')';
echo $sql . "<br /><br />";
$db = mysql_query($sql);
while ( ($data = fgetcsv($handle) ) !== FALSE ) {
$fields = array();
for($i=0;$i<$field_count; $i++) {
$fields[] = '\''.addslashes($data[$i]).'\'';
}
$sql = "Insert into $file values(" . implode(', ', $fields) . ')';
echo $sql;
$db = mysql_query($sql);
}
fclose($handle);
ini_set('auto_detect_line_endings',FALSE);
}
You either assume that there is ALWAYS a header, or that any header will be commented by some character, I will show you the first. You can just wrap the insertion lines in a conditional block to check for this.
if( $lines != 0 ) // or $linemysql[0][0] == '#' (assuming # is a "comment")
{
if($addauto)
$query = "insert into $databasetable values('','$linemysql');";
else
$query = "insert into $databasetable values('$linemysql');";
}
That being said, PLEASE! Do not ever use the code you posted in any internet facing application, you are putting user-provided data directly into the database, so it would be trivial to make a csv file containing SQL injection attack and change your mysql password, steal your data, kill your cat or delete everything. You should also check the number of fields and such, what happens if the csv contains lines without the correct number of fields ?
Read up on http://en.wikipedia.org/wiki/SQL_injection
and also http://php.net/PDO
Use this code in removing the header in the csv file and upload the data in the database:
$header= 1;
if (($handle = fopen("data.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if($header== 1)
{
$header++;
continue;
}
//do your code here saving into the database
}
fclose($handle);
}
I am working in symfony 1.4 and Doctrine using a migration to upload data into a database.
I have a CSV file that I am trying to batch load into the sfGuardUser table, and also match the user id with the sfGuardUserGroup group id, and load that into the sf_guard_user table.
When I run the migration, it is only loading the first record from the CSV. The record that loads looks formatted correctly(building the username and dates).
I tested the code before putting it into the migration, and it worked correctly.
Do I have a syntax eror here, or is it not set up to loop correctly?
Maybe I have just bungled it up entirely.
PHP
public function up()
{
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "web/uploads/batchUpload.csv";
$file = fopen($csvfile,"r");
$size = filesize($csvfile);
$csvcontent = fread($file,$size);
fclose($file);
$lines = 0;
$queries = "";
$linearray = array();
foreach(explode($lineseparator,$csvcontent) as $line) {
//Clean it up
$lines++;
$line = trim($line," \t");
$line = str_replace("\r","",$line);
$line = str_replace("'","\'",$line);
$linearray = explode($fieldseparator,$line);
//build username
if (!$linearray[3]) {
$username = $linearray[0];
$username = substr($linearray[0], 0, 1);
$username .= $linearray[1];
$username = strtolower($username);
$linearray[3] = $username;
}
//add sha1 algorithm
if (!$linearray[4]) {
$linearray[4] = "sha1";
}
//add is_active
if(!$linearray[7]) {
$linearray[7] = "1";
}
//add current date to login, created and updated
$now = getdate();
if ($now['mday'] < 10) {
$now['mday'] = str_pad($now['mday'], 2, "0", STR_PAD_LEFT);
}
if ($now['hours'] < 10) {
$now['hours'] = str_pad($now['hours'], 2, "0", STR_PAD_LEFT);
}
if ($now['minutes'] < 10) {
$now['minutes'] = str_pad($now['minutes'], 2, "0", STR_PAD_LEFT);
}
if ($now['seconds'] < 10) {
$now['seconds'] = str_pad($now['seconds'], 2, "0", STR_PAD_LEFT);
}
$linearray[10] = $now['year']. "-" .$now['mon'].
"-".$now['mday']." ".$now['hours'].":".$now['minutes'].":"
.$now['seconds'];
$linearray[11] = $now['year']. "-" .$now['mon'].
"-".$now['mday']. " ".$now['hours'].":".$now['minutes'].
":".$now['seconds'];
//load up sql
$linemysql = implode("','",$linearray);
$conn = Doctrine_Manager::getInstance()->connection();
$query = "INSERT INTO sf_guard_user VALUES ('', '$linemysql');";
//$queries .= $query . "\n";
$conn->execute($query);
//Add userID to group table
$conn = Doctrine_Manager::getInstance()->connection();
$getFirstID = Doctrine::getTable('sfGuardUser')->orderBy('id DESC')
->limit(1)->execute();
$id = $getFirstID['id'];
$query = "INSERT INTO sf_guard_user_group ('$id', 5, '$linearray[10]', '$linearray[11]');";
$conn->execute($query);
$query = "INSERT INTO sf_guard_user_group ('$id', 14, '$linearray[10]', '$linearray[11]');";
$conn->execute($query);
}// close foreach split
}
UPDATE
If I remove the second query where I am trying to get the last id loaded and match it with the GuardUserGroup group_id, then it works. So I guess we can narrow it down and say my error is in the //Add userID to group table section
UPdated my query to get the latest id record. I think the main issue was $id was still an array in the query. I broke it out like this:
//Add userID to group table
$getFirstID = Doctrine::getTable('sfGuardUser')->createQuery()->select('MAX(id) as id')->fetchArray();
**$id = $getFirstID[0];
$id = $id['id'];**
$q = "INSERT INTO sf_guard_user_group VALUES ('$id', 5, '$linearray[10]', '$linearray[11]');";
$conn->execute($q);
$query = "INSERT INTO sf_guard_user_group VALUES ('$id', 14, '$linearray[10]', '$linearray[11]');";
$conn->execute($query);