I have hit a problem and i cant seem to get over it. I am taking information from a text file(two columns 29.12 23.42 for example separated by a space), using LOAD LOCAL DATA INFILE. Initially this was working with only one column in MySQL, however i have had to add an ID AUTO INCREMENT so i can use the most recent entry, in addition i want to now store the other column in my Database. Here is my code, its probably pretty obvious but I can't seem to find it:
<?PHP
$username = "root";
$password = "";
$hostname = "localhost";
$table = "Received_Data";
// Connect to Database and Table
$dbhandle = mysql_connect($hostname, $username, $password)
or die("Unable to connect to MySQL");
echo "Connected to MySQL<br>";
$selectdatabase = mysql_select_db("IWEMSData",$dbhandle)
or die("Could not select IWEMSData");
echo "Connected to IWEMSData<br>";
// Clear Current Table and ReCreate
$dt = /*"UPDATE Received_Data SET PotVal = ''";*/ "DROP TABLE Received_Data";
mysql_query($dt);
$ctb = "CREATE TABLE Received_Data
(
Current DECIMAL(30,2) NOT NULL,
PotVal DECIMAL(30,2) NOT NULL,
ID BIGINT NOT NULL AUTO_INCREMENT,
PRIMARY KEY(ID,PotVal,Current)
)";
mysql_query($ctb, $dbhandle);
echo "Received_Data Table has been created";
// Text File to read in
$Textfile = 'IWEMSData.txt';
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY ""
LINES TERMINATED BY "\\r\\n
(PotVal, Current)
";')
or die('Error Loading Data File.<br>' . mysql_error());
// Close Database connection when complete
mysql_close($dbhandle);
?>
So what i want is Column 1 is ID which will AUTO INCREMENT upon each entry. The second row is PotVal and the third row is Current. Then i want to only store in the second and third column.
The problem i am getting is that ID and Current are displaying the incorrect values and i can only seem to get one row.
Thanks in advance!
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY "" /*<- terminated by nothing? Shouldn't there be a space in it?*/
LINES TERMINATED BY "\\r\\n /*<- the closing " is missing*/
(PotVal, Current)
";') /*<- ah, here's the missing " What does it do here?*/
Write it like this:
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY " "
LINES TERMINATED BY "\\r\\n"
(PotVal, Current)
;')
Related
I'm trying to finish a script that connects to two databases, each on a different server, and preforms an update. Basically, the tables being selected from and inserted to are identical: I did a dump/import the other day. The script needs to keep my local table up to date from the remote once since there will be daily records inserted into the remote one and I need to keep it up to date locally.
The key here is that I'm determining the new rows on the remote server by looking at the Auto-incremented Primary key that the tables share, SESSIONID . I'm trying to get my loop below to say, if the id exists in remote server and not local, then insert those records in local server.
I run the below script in powershell by typing php 'filename', and I get both of my successful connection messages, but then it hangs. After about 10 minutes, it had a memory error so I added ini_set('memory_limit', '256M');. After this it would still hang for about 10 minutes, and then say that MySQL server has gone away and result header couldn't be found, both errors occurring on the line where I check to see if the $rowCount failed.
Note: Replication and large dump/import/table recreations are not an option for us in this situation. We have several similar scripts to this running and we want to keep the same process here. I'm merely looking to resolve these errors or have someone give me a more efficient way of coding this script, perhaps using a max id or something along those lines.
Here's the script:
ini_set('memory_limit', '256M');
// Create connection
$conn = new mysqli($servername, $username, $password);
$conn2 = new mysqli($servername2, $username2, $password2);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
echo "Connected successfully";
// Check connection2
if ($conn2->connect_error) {
die("Connection failed: " . $conn2->connect_error);
}
echo "Connected successfully";
//Start queries
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "select * from cdrdb.session");
// Loop on the results
while($source_item = $source_data->fetch_assoc()) {
// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM ambition.session WHERE SESSIONID = '".$source_item['SESSIONID']."' ");
//if query returns false, rows don't exist with that new ID.
if (!$row_exists){
//Insert new rows into ambition.session
$conn2->query("INSERT INTO ambition.session (SESSIONID,SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,RESPONSIBLEUSEREXTENSIONID,ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)
VALUES ('".$source['SESSIONID']."' ,
'".$source['SESSIONTYPE']."' ,
'".$source['CALLINGPARTYNO']."' ,
'".$source['FINALLYCALLEDPARTYNO']."',
'".$source['DIALPLANNAME']."',
'".$source['TERMINATIONREASONCODE']."',
'".$source['ISCLEARINGLEGORIGINATING']."',
'".$source['CREATIONTIMESTAMP']."',
'".$source['ALERTINGTIMESTAMP']."',
'".$source['CONNECTTIMESTAMP']."',
'".$source['DISCONNECTTIMESTAMP']."',
'".$source['HOLDTIMESECS']."',
'".$source['LEGTYPE1']."',
'".$source['LEGTYPE2']."',
'".$source['INTERNALPARTYTYPE1']."',
'".$source['INTERNALPARTYTYPE2']."',
'".$source['SERVICETYPEID1']."',
'".$source['SERVICETYPEID2']."',
'".$source['EXTENSIONID1']."',
'".$source['EXTENSIONID2']."',
'".$source['LOCATION1']."',
'".$source['LOCATION2']."',
'".$source['TRUNKGROUPNAME1']."',
'".$source['TRUNKGROUPNAME2']."',
'".$source['SESSIONIDTRANSFEREDFROM']."',
'".$source['SESSIONIDTRANSFEREDTO']."',
'".$source['ISTRANSFERINITIATEDBYLEG1']."',
'".$source['SERVICEEXTENSION1']."',
'".$source['SERVICEEXTENSION2']."',
'".$source['SERVICENAME1']."',
'".$source['SERVICENAME2']."',
'".$source['MISSEDUSERID2']."',
'".$source['ISEMERGENCYCALL']."',
'".$source['NOTABLECALLID']."',
'".$source['RESPONSIBLEUSEREXTENSIONID']."',
'".$source['ORIGINALLYCALLEDPARTYNO']."',
'".$source['ACCOUNTCODE']."',
'".$source['ACCOUNTCLIENT']."',
'".$source['ORIGINATINGLEGID']."',
'".$source['SYSTEMRESTARTNO']."',
'".$source['PATTERN']."',
'".$source['HOLDCOUNT']."',
'".$source['AUXSESSIONTYPE']."',
'".$source['DEVICEID1']."',
'".$source['DEVICEID2']."',
'".$source['ISLEG1ORIGINATING']."',
'".$source['ISLEG2ORIGINATING']."',
'".$source['GLOBALCALLID']."',
'".$source['CADTEMPLATEID']."',
'".$source['CADTEMPLATEID2']."',
'".$source['ts']."',
'".$source['INITIATOR']."',
'".$source['ACCOUNTNAME']."',
'".$source['APPNAME']."',
'".$source['CALLID']."',
'".$source['CHRTYPE']."',
'".$source['CALLERNAME']."',
'".$source['serviceid1']."',
'".$source['serviceid2']."')");
}
}
// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM ambition.session WHERE SESSIONID = '".$source_item['SESSIONID']."' ");
//if query returns false, rows don't exist with that new ID.
if (!$row_exists){
This is incorrect: if query returns false, the query failed to execute. You need to check if ($row_exists->num_rows == 0) instead. The way your code is now, it will always insert every record, again and again. Since you're not checking for errors on the INSERT query you're not noticing the failures you're getting on the duplicate entries for your SESSIONID column (I assume that's the primary key column in your local database as well).
Additionally, it would probably be a lot faster if you only SELECT the sessions you don't already have. Since you're working with an auto increment column, you can pretty much assume that anything in the remote database with a SESSIONID that came after the latest SESSIONID from your local database is new:
//Start queries
$latest_result = $conn2->query("SELECT MAX(`SESSIONID`) FROM `ambition`.`session`");
$latest_row = $latest->fetch_row();
$latest_session_id = $latest_row[0];
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "SELECT * FROM `cdrdb`.`session` WHERE `SESSIONID` > $latest_session_id");
Tables in MySQL are usually sorted by the primary key column by default, but if you're concerned about the order in which you're inserting data in your local database in case your script gets interrupted for some reason, you could add an explicit ORDER BY `SESSIONID` ASC to the query.
Sorry if this has been asked before, but I couldnt find anything that would relate to my case here on SE.
I am trying to import a CSV file into my Mysql database table with both the table the CSV having the exact same amount and order of columns, except that the table's column ID is not missing in the CSV file.
What I want to achieve is to import the CSV into the table while generating an ID number that automatically increases with each record. This does not seem possible as the CSV always seem to want to insert its data into the first colum in the table, but in my case I would need it to be the 2nd column.
How do I approach this and is there any reference code I can study? I currently am working off this PDO approach but am having the above mentioned difficulties.
PHP
<?php
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
Thank you
You can have MySQL set values for certain columns during import. If your id field is set to auto increment, you can set it to null during import and MySQL will then assign incrementing values to it.
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
SET id=null;
EDIT - In case the ID column is not present in CSV
The col1, col2, col3,... are names of actual columns in the DB table (without id column)
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
(col1, col2, col3,...)
SET id=null;
The AUTO_INCREMENT attribute can be used to generate a unique identity for new rows. Most version of mysql and engin support this. You need not worry about the ID and can use cron job to insert the needed field and AUTO_INCREMENT will take care of the id itself.
No value was specified for the AUTO_INCREMENT column, so MySQL assigned sequence numbers automatically. You can also explicitly assign 0 to the column to generate sequence numbers, unless the NO_AUTO_VALUE_ON_ZERO SQL mode is enabled. If the column is declared NOT NULL, it is also possible to assign NULL to the column to generate sequence numbers. When you insert any other value into an AUTO_INCREMENT column, the column is set to that value and the sequence is reset so that the next automatically generated value follows sequentially from the largest column value.
You can retrieve the most recent automatically generated AUTO_INCREMENT value with the LAST_INSERT_ID() SQL function or the mysql_insert_id() C API function. These functions are connection-specific, so their return values are not affected by another connection which is also performing inserts.
See example from official link :
[https://dev.mysql.com/doc/refman/5.7/en/example-auto-increment.html]
As you want to recreate the table over and over and want to manipulate the Data from the CSV, try this:
// You have to create the TABLE if not exists
$pdo->exec("TRUNCATE TABLE sample"); // No need to drop the table if columns don't change.
$csvContent = file_get_contents($csvfile); // Raw Data from file
$lines = explode("
", $csvContent); // The standard line separator is an ENTER
// Now you have each line separated
for($i = 0; $i < coount($lines); $i++) {
$col = explode(";", $lines[$i]); // Would be a comma
// Now you have each column separated
$pdo->exec("INSERT INTO sample (id, col1, col2, col3 ... coln) VALUES (NULL, '".$col[0]."', '".$col[1]."', '".$col[2]."' ... '".$col[n]."')");
}
This way you can dig into your Data and, besides setting an AUTO_INCREMENT ID, you can validate what is coming from the CSV and can correct/prevent importation errors.
I'm really stuck here I have this PHP script:
<?php
$databasehost = "localhost";
$databasename = "";
$databasetable = "";
$databaseusername="";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$enclosedbyquote = '"';
$csvfile = "db-core/feed/csv/csv.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$pdo->exec("TRUNCATE TABLE `$databasetable`");
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." REPLACE INTO TABLE `$databasetable`
FIELDS OPTIONALLY ENCLOSED BY ".$pdo->quote($enclosedbyquote)."
TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."
IGNORE 1 LINES");
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
So as you can see that script is truncating my table with a CSV file. It replaces all the data currently in MySQL table.
This data is used to create a listing page that includes vehicles that are in stock, each row contains the data for one vehicle. This script is ran once a day to replace vehicles that are no longer in stock.
I now want to give each of my SQL rows it's own page, I've been told I will need to use a 'NOT NULL AUTO_INCREMENT' attribute on an 'id' key.
However seen as my table truncates wont the ID removed each time my script is ran?
Could I add anything to my script to combat this problem?
Thanks
Yes. Truncating a table is the equivalent of delete from table and nuking all the records. But the auto_increment won't reset to 0 again. The last used auto_inc value is part of the table's metadata and is not affected by a truncate operation.
If you had records 0->100, truncated the table, then added new records, they would start at #101 and climb from there.
If you want the auto_increment value to be reset after the truncation, you'd have to do:
TRUNCATE TABLE foo; // delete all records
ALTER TABLE foo SET auto_increment=1; // reset auto-increment to 1
INSERT ...
So yep... turns out truncate does reset the auto_increment value. Definitely seems wrong, since it also apparently won't process any cascade deletes while truncating. Repopulating a truncated table that is used as a foreign key elsewhere will undoubtedly lead to incorrect record joinings.
I cant quite think about how to do this with mysql and php. Basically I want to be able to submit data into a mysql database but before it is inserted, it will check to see if that entry already exists.
$guid=$_POST['guid'];
$name=$_POST['name'];
//Username
$user="webhost";
//Password
$pass="*******";
//IP To Host
$ip="***********";
//Database
$db="dayz2";
//Table
$table="whitelist";
//Database Connection
$con=#mysql_connect("$ip", "$user", "$pass")
or die(mysql_error());
//Select Database
$dbcon=#mysql_select_db($db, $con)
or die(mysql_error());
$dupesql = "SELECT * FROM $table where (name = '$name' AND guid = '$guid')";
$duperaw = mysql_query($dupesql);
if (mysql_num_rows($duberaw) > 0) {
echo "Entry Already Exists";
}
else {
//Query Data Into Whitelist Table
$sql="INSERT INTO $table (name, guid) VALUES ('$name', '$guid')";
//Submit Data into Whitelist Table
$result=#mysql_query($sql, $con) or die(mysql_error());
}
?>
You can do it in another way, instead of:
submit data into a mysql database but before it is inserted, it will
check to see if that entry already exists.
You can do:
INSERT data into a mysql database if not existed, else ignore them
Something like :
INSERT IGNORE INTO table
INSERT IGNORE INTO yourtablename
SET fieldname = 'blah'
,..
It depends what you are trying to do - what is the exact criteria for your query?
You have several options:
use INSERT IGNORE ... if you only want to insert new rows that don't have a duplicate primary key. See http://dev.mysql.com/doc/refman/5.5/en/insert.html.
use INSERT ... ON DUPLICATE KEY UPDATE to insert new rows and update rows where there is a primary key match.
See http://dev.mysql.com/doc/refman/5.5/en/insert-on-duplicate.html.
use a normal SQL SELECT ... to pull the results first before performing business logic on the results before deciding which to INSERT ... or UPDATE ... depending on your requirements.
It depends how you want to handle case when the entry exists.
I you want to throw some error then you can create table trigger for insert event and put some checks there, but it will be slow because every insert will do this check.
I have the following PHP code that allows me to read in from a text file line by line and write to a MySQL database. The text file consists of 13 values seperated by a space on each line. I have created the table separately in SQL, (with all the relevant Fields needed: Time, WIMU_ID, Ax, Ay etc) and then run the PHP below to INSERT the values into the table. My question is: Is there any way to create the table within the PHP code below and not have to CREATE the table in SQL first? Any help or suggestions appreciated.
Hugh.
<?php
$connection = mysql_connect('localhost','root','bonzo123');
mysql_query('create database gameDB');
mysql_select_db('gameDB');
$wx = array_map('trim',file("Thu-Apr-01-09_41_01-2010.txt_calibrated_120Hz.txt"));
$newwx = array();
foreach($wx as $i => $line)
{
if ($i > 1)
{
$tmp = array_filter(explode(' ',$line));
$q = "insert into test1 (Time, WIMU_ID, Ax, Ay, Az, Gx, Gy, Gz, Mx, My, Mz, Ax70, Ay37) values ('" . implode("','",$tmp) . "')";
$rw = mysql_query($q) or die("Problem with the query: $q<br>" . mysql_error());
}
}
?>
I think what you really want is this:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
which you can invoke through PHP but this would be a lot better unless you have specific reasons it cannot be used.
Sure, you can execute a command like:
CREATE TABLE IF NOT EXISTS `test1` (
...
);
To get your table creation query, issue this command after you've already created it in SQL:
SHOW CREATE TABLE `test1`;
Edit:
Note that you don't have to create the table in SQL directly, but since you already have, issuing the above command will give you exactly what you need to include in PHP to have it create the table on a future installation should the table not exist.
Note also that as #Jeremy suggests, you don't have to "roll your own" code to import a data file to a table. Once you've created the table, you can use LOAD DATA INFILE... to populate the table.
Edit 2:
Here is an example based on your comment:
<?php
// Connect to the database server
$connection = mysql_connect('localhost', 'root', 'bonzo123');
if (!$connection)
die('Could not connect: ' . mysql_error());
// Create database gameDB
if (mysql_query('create database gameDB', $connection)) {
echo "Database gameDB created successfully.\n";
} else {
echo 'Error creating database: ' . mysql_error() . "\n:";
}
// select which database to use
mysql_select_db('gameDB', $connection);
// create table test1 if it doesn't already exist
mysql_query('CREATE TABLE IF NOT EXISTS test1 (
v1 varchar(100),
v2 varchar(100),
v3 varchar(100),
v4 varchar(100),
v5 varchar(100),
v6 varchar(100),
v7 varchar(100),
v8 varchar(100),
v9 varchar(100),
v10 varchar(100)', $connection);
?>