So, I have an issue with loading my CSV into SQL database with PHP and Wordpress. Both are running on local xammp atm.
This is my code for loading the CSV file. It first uploads it via html form to some temp folder (data) and then tries with LOAD DATA to copy the content into the table (wp_data)
<?php
if (isset($_POST['submit'])) {
global $wordpress,$wpdb;
$file = $_FILES['fileToUpload']['tmp_name'];
$target_dir = $_SERVER['DOCUMENT_ROOT']."/data";
$target_file = $target_dir . '/' . basename($_FILES["fileToUpload"]["name"]);
move_uploaded_file($file, $target_file);
$sql="
LOAD DATA INFILE '$target_file' INTO TABLE wp_data FIELDS TERMINATED BY ';' IGNORE 1 ROWS";
$query = $wpdb->query($sql);
}
?>
The uploading part works fine, however when it tries to fill the table with the data, it just fills "0s"/NULL values, like this:
The file as text looks like this:
And visually, it looks like this:
Table structure from SHOW CREATE TABLE wp_data:
CREATE TABLE `wp_data` (
`Unix_time` int(11) NOT NULL,
`Message` text NOT NULL,
`Time` float NOT NULL,
`DF` int(11) NOT NULL,
`Type_code/BDS_number` int(11) NOT NULL,
`Latitude` double NOT NULL,
`Longitude` double NOT NULL,
`Altitude` int(11) NOT NULL,
`Ground_speed` double NOT NULL,
`Baro_diff` int(11) NOT NULL,
`Heading` int(11) NOT NULL,
`TAS` int(11) NOT NULL,
`Mach` float NOT NULL,
`FOM_SOURCE` varchar(40) DEFAULT NULL,
`Wind_speed` varchar(40) DEFAULT NULL,
`Wind_direction` varchar(40) DEFAULT NULL,
`Temperture` text,
`Pressure` varchar(40) DEFAULT NULL,
`Turbulence` varchar(40) DEFAULT NULL,
`Humidity` varchar(40) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
What am I doing wrong here? I am quite new at PHP and got stuck on this
:(...
Thx.
Fixed it by setting the encoding in query to CHARACTER SET UTF8 like this:
LOAD DATA INFILE '$target_file'
INTO TABLE wp_data
CHARACTER SET UTF8
FIELDS TERMINATED BY ';'
IGNORE 1 ROWS;
Related
I have this code and for some reason it won't get inserted into my database. It's basically taking an array, turning it into a string and then submit the values.
(If you need me to edit to show my whole code, I will do so)
Code I am having issues with down below
$array = array($RaceNumber,$Track,$Num,$HorseName,$Odds,$Color,$Jockey,$Trainer,$PostTime,$Course,$RaceDistance,$Win,$Place,$Show);
for ($a=0; $a<$Num; $a++) {
$dataArray=array($RaceNumber[$a],$Track[$a],$Num[$a],$HorseName[$a],$Odds[$a],$Color[$a],$Jockey[$a],$Trainer[$a],$PostTime[$a],$Course[$a],$RaceDistance[$a],$Win[$a],$Place[$a],$Show[$a]);
$dataArray--;
for ($j=0; $j<$Num; $j++) {
$RaceNumber=$dataArray[0];
$Track=$dataArray[1];
$Num=$dataArray[2];
$HorseName=$dataArray[3];
$Odds=$dataArray[4];
$Color=$dataArray[5];
$Jockey=$dataArray[6];
$Trainer=$dataArray[7];
$PostTime=$dataArray[8];
$Course=$dataArray[9];
$RaceDistance=$dataArray[10];
$Win=$dataArray[11];
$Place=$dataArray[12];
$Show=$dataArray[13];
$sql="INSERT INTO `$Date` (RaceNumber,Track,HorseNum,HorseName,Odds,Color,JockeyName,TrainerName,PostTime,Course,RaceDistance,Win,Place,Show) VALUES ('$RaceNumber','$Track','$Num','$HorseName','$Odds','$Color','$Jockey','$Trainer','$PostTime','$Course','$RaceDistance','$Win','$Place','$Show')";
echo $sql;
mysqli_query($query2,$sql);
}
}
when I echo my $sql I get
INSERT INTO 2018-09-20 (RaceNumber,Track,HorseNum,HorseName,Odds,Color,JockeyName,TrainerName,PostTime,Course,RaceDistance,Win,Place,Show) VALUES ('1','FingerLakes','1','','','Red','','','','Dirt','','none','none','none')
But when I do my query, it isn't inserting into database.
Part of my code where I create the datatable
<?php
if(isset($_POST['submit'])) {
$Date = $_POST['date'];
$sql = "CREATE TABLE IF NOT EXISTS `$Date` (
`Id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
`RaceNumber` varchar(255) NOT NULL,
`Track` varchar(255) NOT NULL,
`HorseNum` varchar(255) NOT NULL,
`HorseName` varchar(255) NOT NULL,
`Odds` varchar(255) NOT NULL,
`Color` varchar(255) NOT NULL,
`JockeyName` varchar(255) NOT NULL,
`TrainerName` varchar(255) NOT NULL,
`PostTime` varchar(255) NOT NULL,
`Course` varchar(255) NOT NULL,
`RaceDistance` varchar(255) NOT NULL,
`Win` varchar(255) NOT NULL,
`Place` varchar(255) NOT NULL,
`Show` varchar(255) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8"
;
$query2 = mysqli_connect('localhost','root','','Races');
$z= mysqli_query($query2, $sql) or die("Table already exist.. please try again");
echo "Your Table ".$Date." is successfully created <br/>";
$RaceNum = $_POST['RaceNum'];
$i=1;
I am receiving in my error log of
2018-09-20 16:00:59 9444 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'max_value' at position 4 to have type varbinary(255), found type varchar(255).
You are using a column named Show that's a reserved keyword in mysql, add backquotes to it and the insert query should work.
It's worth noting that you shouldn't name your table with only digits and hyphens.
For reference here is the complete list of the reserved keywords:
https://dev.mysql.com/doc/refman/8.0/en/keywords.html
I have csv file which looks:
0,00018332,2016-08-29 00:00:00,2016-08-29 00:00:00,9999,Sale of Parts only,0,DMS Maritime Pty Ltd,xyz,,00000250,1971-01-01 00:00:00,1971-01-01 00:00:00,218335,Sale of parts only,9999
Then I have php script which is importing this csv into table
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "test.csv";
$dbtable = "res";
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$dbtable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
But when I check it back in db table, values 2016-08-29 00:00:00 (column is datetime) is 0000-00-00 00:00:00 , 218335 and 9999 is set to 0 event columns are set to int(11).Everything else is written correctly. I tried with csv and txt and I have same problem.
Do you know why?
EDIT:
CREATE TABLE `res` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` text,
`start` datetime DEFAULT NULL,
`end` datetime DEFAULT NULL,
`room_id` int(11) DEFAULT NULL,
`status` varchar(30) DEFAULT NULL,
`paid` int(11) DEFAULT NULL,
`customer` varchar(50) DEFAULT NULL,
`name_ship` varchar(50) DEFAULT NULL,
`equipment` text,
`port` varchar(30) DEFAULT NULL,
`ETA` datetime DEFAULT NULL,
`ETD` datetime DEFAULT NULL,
`service_id` int(11) NOT NULL,
`service_classification` varchar(30) DEFAULT NULL,
`partners_id` int(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5392 DEFAULT CHARSET=latin1
Your file is encoded in utf16, mysql will not check that on its own. Either change that in your datasource (since your table encoding indicates that you don't want to use utf16 strings anyway) or inform mysql about the file's encoding.
For your file, the following should work:
LOAD DATA LOCAL INFILE 'test.csv' INTO TABLE res
CHARACTER SET utf16le
FIELDS TERMINATED BY ';\0'
LINES TERMINATED BY '\r\0\n\0';
It sets the characterset of the file and defines the delimiters in unicode. You can use just '\n\0' too.
Why not execute : show warnings in mysql cmd and see what's the exact error.
https://dev.mysql.com/doc/refman/5.5/en/show-warnings.html
I have the following function:
function bulk_insert_file($filename) {
$file_location = 'assets/temp/'.$filename;
$sql = 'LOAD DATA LOCAL INFILE '."'$file_location'".' INTO TABLE p4p.users_csv_import
FIELDS TERMINATED BY \',\'
LINES TERMINATED BY \'\\r\\n\'
IGNORE 1 LINES';
$this->db->query($sql);
}
And the following CSV file:
,2,unique_id,first_name,last_name,email,company,nonprofit,username,password,dimension_data,raw_csv_data,
,2,unique_id,first_name,last_name,email,company,nonprofit,username,password,dimension_data,raw_csv_data,
,2,unique_id,first_name,last_name,email,company,nonprofit,username,password,dimension_data,raw_csv_data,
However, when I run the code no data is imported. If I remove the IGNORE 1 LINES part I will at least get the first row imported.
CREATE TABLE `users_csv_import` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`role_id` int(11) NOT NULL DEFAULT '1',
`unique_id` varchar(255) DEFAULT '',
`first_name` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`last_name` varchar(50) CHARACTER SET latin1 NOT NULL,
`email` varchar(100) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`company` int(11) NOT NULL DEFAULT '0',
`nonprofit` int(11) NOT NULL,
`username` varchar(100) CHARACTER SET latin1 NOT NULL DEFAULT '',
`password` varchar(34) CHARACTER SET latin1 NOT NULL DEFAULT '',
`dimension_data` text,
`raw_csv_data` text,
PRIMARY KEY (`id`)
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
Here is the specs on the table..
I assume the csv file is in unix line terminators format.
You should therefore try importing with LINES TERMINATED BY \'\\n\'.
It might sound a bit confusing, and perhaps what I'm doing isn't right at all, but here goes.
Our users can upload .csv files containing their school's data, exported from an external management system called SIMS. Once they upload that file I want to run a .sql script that updates our database with the information they provide in this file. What I've done, on the upload file page (works like a charm), is create a session variable containing the file path and name of the uploaded file, as shown below (file path is blanked out):
$target_path = "xxx";
$target_path = $target_path . basename( $_FILES['uploadedfile']['name']);
if(move_uploaded_file($_FILES['uploadedfile']['tmp_name'], $target_path))
{
echo "<p>The file ". basename( $_FILES['uploadedfile']['name']). " has been uploaded.</p>" ;
echo "<p><a href='sqltest.php'>Click here</a> to upload your files into the database.</p>" ;
$_SESSION['file'] = $target_path ;
}
Below is a portion of my .sql file operated within PHP containing the session variable. Because most of this populate script uses the same kind of commands throughout it will probably not make sense to show you all 973 lines of code (of course there is a session_start() active at the top of the page).
$filename = $_SESSION['file'] ;
mysqli_query($dbc, 'SET foreign_key_checks = 0') ;
$populate =
"CREATE TEMPORARY TABLE IF NOT EXISTS `mldb`.`TempSchool`
(
`CentreNo` INT UNSIGNED NOT NULL,
`School` VARCHAR(255) NULL,
`Street` VARCHAR(255) NULL,
`Town` VARCHAR(255) NULL,
`County` VARCHAR(255) NULL,
`Postcode` VARCHAR(10) NULL,
`Tel` VARCHAR(45) NULL,
`URL` VARCHAR(512) NULL,
`Email` VARCHAR(255) NULL,
`Headteacher` VARCHAR(255) NULL,
`LEA` VARCHAR(45) NULL,
`LEANo` INT UNSIGNED NULL,
`EstablishmentNo` INT UNSIGNED NULL,
`URN` INT UNSIGNED NULL,
`Governance` VARCHAR(45) NULL,
`Phase` VARCHAR(45) NULL,
PRIMARY KEY (`CentreNo`)
)
ENGINE = InnoDB ;
LOAD DATA INFILE '$filename'
IGNORE INTO TABLE `TempSchool`
FIELDS TERMINATED BY ' , '
OPTIONALLY ENCLOSED BY ' \" '
LINES TERMINATED BY ' \r\n '
IGNORE 1 LINES (etc...)
I'm getting an error on line 25 of the code (LOAD DATA INFILE '$filename') displaying the following:
Invalid query:You have an error in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'LOAD DATA INFILE
'C:/Users/Public/Dropbox/mlwebfiles/Trial/uploads/MarksLive Set' at line 25
Presumably it's got something to do with the fact that, even though it's recognizing the file path and file name stored in the $_SESSION['file'] variable, but it's not actually doing anything with it. Is there a way where this .sql file can operate with through PHP set up variables for the file names? Many thanks!
After searching long and hard, and far and wide we've found a (what we think is the only) solution: create a function to loop through every one of these 'mini-queries' individually and it seems to be a lot more happy now! We actually came across this suggestion before but we rejected it initially because it would (and did) take a lot of time to separate 973 lines of code to split all the individual database additions...
$populate =
"
CREATE TEMPORARY TABLE IF NOT EXISTS `mldb`.`TempSchool`
(
`CentreNo` INT UNSIGNED NOT NULL,
`School` VARCHAR(255) NULL,
`Street` VARCHAR(255) NULL,
`Town` VARCHAR(255) NULL,
`County` VARCHAR(255) NULL,
`Postcode` VARCHAR(10) NULL,
`Tel` VARCHAR(45) NULL,
`URL` VARCHAR(512) NULL,
`Email` VARCHAR(255) NULL,
`Headteacher` VARCHAR(255) NULL,
`LEA` VARCHAR(45) NULL,
`LEANo` INT UNSIGNED NULL,
`EstablishmentNo` INT UNSIGNED NULL,
`URN` INT UNSIGNED NULL,
`Governance` VARCHAR(45) NULL,
`Phase` VARCHAR(45) NULL,
PRIMARY KEY (`CentreNo`)
)
ENGINE = InnoDB ;
" ;
populate ($dbc, $populate);
$populate =
"
LOAD DATA INFILE '$path'
IGNORE INTO TABLE `mldb`.`TempSchool`
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES
(#AdNo, #UPN, #ULN, #UCI, #CandidateNo, #LegalSurname, #LegalForename,
#PreferredSurname, #PreferredForename, #Gender, #DOB, #Email,
#InCare, #EverInCare, #FSM, #FSMEver6, #EAL, #SENStatus, #AMA,
#Attendance, #RegGroup, #YearGroup, #EnteredYearDate,
#Class, #Subject, #Staff, #Initials,
CentreNo, School, Street, Town, County, Postcode, Tel, URL,
Email, Headteacher, LEA, LEANo, EstablishmentNo, Governance, Phase)
" ;
populate ($dbc, $populate);
I have a database setup like
`id` int(11) unsigned NOT NULL auto_increment,
`ad-id` int(11) default NULL,
`advertiser-id` int(11) default NULL,
`advertiser-name` varchar(250) default NULL,
`advertiser-category` varchar(250) default NULL,
`buy-url` varchar(255) default NULL,
`catalog-id` varchar(255) default NULL,
`currency` varchar(3) default NULL,
`description` varchar(255) default NULL,
`image-url` varchar(255) default NULL,
`in-stock` varchar(255) default NULL,
`isbn` varchar(13) default NULL,
`manufacturer-name` varchar(255) default NULL,
`manufacturer-sku` varchar(50) default NULL,
`name` varchar(255) default NULL,
`price` decimal(10,2) default NULL,
`retail-price` decimal(10,2) default NULL,
`sale-price` decimal(10,2) default NULL,
`sku` varchar(50) default NULL,
`upc` varchar(50) default NULL,
I am trying to load the data like this:
LOAD DATA LOW_PRIORITY LOCAL INFILE "/home/datafeed/tempfile.csv" REPLACE INTO TABLE products_import FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\n" (ad-id, advertiser-id, advertiser-name, advertiser-category, buy-url,catalog-id,currency, description,image-url,in-stock,isbn,manufacturer-name,manufacturer-sku,name,price,retail-price,sale-price,sku, upc);
but it doesn't seem to like database names with dashes in them. The field names are pretty much set in stone. If I try to insert without the field names it sets everything in the wrong field (starts with autoinc)
I'm not very good with MySQL any help would be appreciated.
Put back-ticks around the field names: `ad-id`, etc., like in your table structure listing.