I have written the following code on PHP and i am getting the error:
Data truncated for column 'datetime_gmt' at row 1;
Here is the code:
$lines = new SplFileObject('/home/file.txt');
$x = 0;
while(!$lines->eof()) {
$lines->next();
if($x == 0){
$lines->next();
}
$row = explode(',',$lines);
for($i = 0; $i<4; $i++){
if(!isset($row[$i])){
$row[$i] = null;
}
}
$y = (float) $row[1];
$z = (float) $row[2];
$load_query = "INSERT IGNORE INTO new (datetime_gmt,field2,field3)
VALUES ('".$row[0]."','".$y."','".$z."');";
$x++;
}
$lines = null;
The column is of type 'datetime' and has '0000-00-00 00:00:00' as DEFAULT, and it is the PRI of the table. If you are wondering about the "x" variable, it's for skipping the first 2 lines.
EDIT 1
Here is a sample data:
2013-12-11 8:22:00, 1.462E+12, 3.33E+11
2013-12-12 4:10:00, 1.462E+12, 3.33E+11
2013-12-13 11:52:00, 1.462E+12, 3.33E+11
And here is the description of the table "new":
Field | Type | Null | Key | Default | Extra
datetime_gmt | datetime |No | PRI |0000-00-00 00:00:00 |
field2 | bigint(20)|YES | |NULL |
field3 | bigint(20)|YES | |NULL |
using:
SELECT sum(char_length(COLUMN_NAME))
FROM TABLE_NAME;
I get 19 as the size of the column.
Please run this query directly into your database: INSERT IGNORE INTO new (datetime_gmt,field2,field3) VALUES ('2013-12-11 8:22:00','1462000000000','333000000000');
If that does not add a row, remove the default on the datetime_gmt column and re-try.
Note: You have a syntax error with your code.
Change this:
$load_query = "INSERT IGNORE INTO new (datetime_gmt,field2,field3)
VALUES ('".$row[0]"','".$y."','".$z."');";
To this:
$load_query = "INSERT IGNORE INTO new (datetime_gmt,field2,field3)
VALUES ('".$row[0]."','".$y."','".$z."');";
If the aforementioned doesn't work, try to have just engine substitution in your SQL Modes:
set ##sql_mode='no_engine_substitution';
Then make sure that it shows NO_ENGINE_SUBSTITUTION by running the following:
select ##sql_mode;
Then attempt to run your code again. The set ##sql_mode might not be server wide and it may only work for your current session.
Related
I am trying to make a csv upload page for an application that Im building. It needs to be able to upload thousands of rows of data in seconds each row including a first name, last name, and phone number. The data is being uploaded to a vm that is running ubuntu server. When I run the script to upload the data it takes almost 2 minutes to upload 1500 rows. The script is using PDO, I have also made a test script in python to see if It was a php issue and the python script is just as slow. I have made csv upload scripts in the past that are exactly the same that would upload thousands of rows in seconds. We have narrowed the issue down to the script as we have tested it on other vms that are hosted closer to us and the issue still persist. Is there an obvious issue with the script or PDO that could be slowing it down? Below is the code for the script.
<?php
$servername =[Redacted];
$username = [Redacted];
$password = [Redacted];
try {
$conn = new PDO("mysql:host=$servername;dbname=test", $username, $password);
// set the PDO error mode to exception
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "Connected successfully";
} catch(PDOException $e) {
echo "Connection failed: " . $e->getMessage();
}
echo print_r($_FILES);
$fileTmpPath = $_FILES['fileToUpload']['tmp_name'];
$fileName = $_FILES['fileToUpload']['name'];
$fileSize = $_FILES['fileToUpload']['size'];
$fileType = $_FILES['fileToUpload']['type'];
$CSVfp = fopen($fileTmpPath, "r");
$final_array = [];
while (($data = fgetcsv($CSVfp)) !== false) {
$recived = [];
foreach ($data as $i) {
array_push($recived, $i );
}
array_push($final_array, $recived);
}
echo print_r($final_array);
fclose($CSVfp);
$non_compliant_rows = [];
foreach ($final_array as $key => $row){
$fname = preg_replace('/[^A-Za-z0-9]/', "", $row[0]);
$lname = preg_replace('/[^A-Za-z0-9]/', "", $row[1]);
$mobileNumber = preg_replace( '/[^0-9]/i', '', $row[2]);
$sanatized_row = array($fname, $lname, $mobileNumber);
$recived[$key] = $sanatized_row;
if (strlen($mobileNumber) > 10 or strlen($mobileNumber) < 9){
array_push($non_compliant_rows, $final_array[$key]);
unset($final_array[$key]);
}
}
$final_array = array_values($final_array);
echo print_r($final_array);
foreach($final_array as $item){
try{
$stmt = $conn->prepare("INSERT INTO bulk_sms_list(fname, lname, pn, message, send) VALUES (?, ?, ?, 'EMPTY', 1) ;");
$stmt->execute($item);
}catch(PDOException $e){
echo $e;
}
}
echo "done";
The phone numbers column has a UNIQUE constraint to prevent us from having duplicate phone numbers. We have tried to use batch inserting but if one row doesn't comply with the constraints then all of the insertions fail.
below is the schema of the table:
+---------+------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| fname | text | NO | | NULL | |
| lname | text | NO | | NULL | |
| pn | text | NO | UNI | NULL | |
| message | text | YES | | NULL | |
| send | int | NO | | 1 | |
+---------+------+------+-----+---------+----------------
EDIT:
I have timed the clean up portion of the script at the request of #aynber. the time of the clean up was 0.24208784103394 Seconds. The time it took to do the sql portion is 108.2597219944 Seconds
The fastest solution should be to use LOAD DATA LOCAL INFILE. Since you answered in a comment that duplicate phone numbers should result in skipping a row, you can use the IGNORE option.
Load directly from the file, instead of processing it with PHP.
You can do some of your transformations in the LOAD DATA statement.
For example:
LOAD DATA INFILE ? IGNORE INTO TABLE bulk_sms_list
FIELDS TERMINATED BY ','
(#fname, #lname, #mobile)
SET fname = REGEXP_REPLACE(#fname, '[^A-Za-z0-9]', ''),
lname = REGEXP_REPLACE(#lname, '[^A-Za-z0-9]', ''),
pn = IF(LENGTH(#mobile) BETWEEN 9 AND 10, #mobile, NULL),
message = 'EMPTY',
send = 1;
Then follow the import with some cleanup to get rid of any rows with invalid phone numbers:
DELETE FROM bulk_sms_list WHERE pn IS NULL;
Read https://dev.mysql.com/doc/refman/8.0/en/load-data.html for more information.
I am a newbie to PHP and I am stuck at a certain point. I tried looking up a solution for it however, I didn't find exactly what I need.
My goal is to create a leaderboard, in which the values are displayed in descending order plus the rank and score are displayed. Furthermore, it should also display whether or not a tie is present.
The database should look like this:
+---------+------+----------------+-------+------+
| user_id | name | email | score | tied |
+---------+------+----------------+-------+------+
| 1 | SB | sb#gmail.com | 1 | 0 |
+---------+------+----------------+-------+------+
| 2 | AS | as#web.de | 2 | 0 |
+---------+------+----------------+-------+------+
| 3 | BR | br#yahoo.com | 5 | 1 |
+---------+------+----------------+-------+------+
| 4 | PJ | pj#gmail.com | 5 | 1 |
+---------+------+----------------+-------+------+
And the outputted table should look something like this:
+------+-------------+-------+------+
| rank | participant | score | tied |
+------+-------------+-------+------+
| 1 | BR | 5 | Yes |
+------+-------------+-------+------+
| 2 | PJ | 5 | Yes |
+------+-------------+-------+------+
| 3 | AS | 2 | No |
+------+-------------+-------+------+
| 4 | SB | 1 | No |
+------+-------------+-------+------+
I managed to display the rank, participant and the score in the right order. However, I can't bring the tied column to work in the way I want it to. It should change the value, whenever two rows (don't) have the same value.
The table is constructed by creating the <table> and the <thead> in usual html but the <tbody> is created by requiring a php file that creates the table content dynamically.
As one can see in the createTable code I tried to solve this problem by comparing the current row to the previous one. However, this approach only ended in me getting a syntax error. My thought on that would be that I cannot use a php variable in a SQL Query, moreover my knowledge doesn't exceed far enough to fix the problem myself. I didn't find a solution for that by researching as well.
My other concern with that approach would be that it doesn't check all values against all values. It only checks one to the previous one, so it doesn't compare the first one with the third one for example.
My question would be how I could accomplish the task with my approach or, if my approach was completely wrong, how I could come to a solution on another route.
index.php
<table class="table table-hover" id="test">
<thead>
<tr>
<th>Rank</th>
<th>Participant</th>
<th>Score</th>
<th>Tied</th>
</tr>
</thead>
<tbody>
<?php
require("./php/createTable.php");
?>
</tbody>
</table>
createTable.php
<?php
// Connection
$conn = new mysqli('localhost', 'root', '', 'ax');
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
// SQL Query
$sql = "SELECT * FROM names ORDER BY score DESC";
$result = $conn->query("$sql");
// Initalizing of variables
$count = 1;
$previous = '';
while($row = mysqli_fetch_array($result)) {
$current = $row['score'];
$index = $result['user_id']
if ($current == $previous) {
$update = "UPDATE names SET tied=0 WHERE user_id=$index";
$conn->query($update);
}
$previous = $current;
?>
<tr>
<td>
<?php
echo $count;
$count++;
?>
</td>
<td><?php echo $row['name'];?></td>
<td><?php echo $row['score'];?></td>
<td>
<?php
if ($row['tied'] == 0) {
echo 'No';
} else{
echo 'Yes';
}
?>
</td>
</tr>
<?php
}
?>
I think the problem is here
$index = $result['user_id'];
it should be
$index = $row['user_id'];
after updating tied you should retrieve it again from database
So I solved my question by myself, by coming up with a different approach.
First of all I deleted this part:
$current = $row['score'];
$index = $result['user_id']
if ($current == $previous) {
$update = "UPDATE names SET tied=0 WHERE user_id=$index";
$conn->query($update);
}
$previous = $current;
and the previous variable.
My new approach saves the whole table in a new array, gets the duplicate values with the array_count_values() method, proceeds to get the keys with the array_keys() method and updates the database via a SQL Query.
This is the code for the changed part:
// SQL Query
$sql = "SELECT * FROM names ORDER BY score DESC";
$result = $conn->query("$sql");
$query = "SELECT * FROM names ORDER BY score DESC";
$sol = $conn->query("$query");
// initalizing of variables
$count = 1;
$data = array();
// inputs table into an array
while($rows = mysqli_fetch_array($sol)) {
$data[$rows['user_id']] = $rows['score'];
}
// -- Tied Column Sort --
// counts duplicates
$cnt_array = array_count_values($data);
// sets true (1) or false (0) in helper-array ($dup)
$dup = array();
foreach($cnt_array as $key=>$val){
if($val == 1){
$dup[$key] = 0;
}
else{
$dup[$key] = 1;
}
}
// gets keys of duplicates (array_keys()) and updates database accordingly ($update query)
foreach($dup as $key => $val){
if ($val == 1) {
$temp = array_keys($data, $key);
foreach($temp as $k => $v){
$update = "UPDATE names SET tied=1 WHERE user_id=$v";
$conn->query($update);
}
} else{
$temp = array_keys($data, $k);
foreach($temp as $k => $v){
$update = "UPDATE names SET tied=0 WHERE user_id=$v";
$conn->query($update);
}
}
}
Thank you all for answering and helping me get to the solution.
instead of the update code you've got use something simular
$query = "select score, count(*) as c from names group by score having c > 1";
then you will have the scores which have a tie, update the records with these scores and your done. Make sure to set tie to 0 at first for all rows and then run this solution
UPDATE for an even faster solution sql based:
First reset the database:
$update = "UPDATE names SET tied=0";
$conn->query($update);
All records have a tied = 0 value now. Next update all the records which have a tie
$update = "update docs set tied = 1 where score IN (
select score from docs
group by score having count(*) > 1)";
$conn->query($update);
All records with a tie now have tied = 1 as we select all scores which have two or more records and update all the records with those scores.
This question already has an answer here:
Post form and update multiple rows with mysql
(1 answer)
Closed last year.
For example, I have two data on my table:
| ID | Name | Age |
| 1 | Steve | 25 |
| 2 | Bob | 28 |
When i updating one value (for example: change "Bob" to "George"), it changes all value. This is the result:
| ID | Name | Age |
| 1 | George | 28 |
| 2 | George | 28 |
How to updating multiple rows in one query? To collect values, I use for loop like this:
<?php
...
$id_array = $_POST['id'];
$name_array = $_POST['name'];
$age_array = $_POST['age'];
$id = array();
$name = array();
$age = array();
for ($i = 0; $i < count($id_array); $i++) {
//count($id_array) --> if I input 4 fields, count($id_array) = 4)
$id[] = mysql_real_escape_string($id_array[$i]);
$name[] = mysql_real_escape_string($name_array[$i]);
$age[] = mysql_real_escape_string($age_array[$i]);
}
mysql_query("UPDATE member SET name = '$name', age = '$age' WHERE id = '$id'");
}
...
?>
Can you help me? Thank you.
Construct your query within the loop:
<?php
...
$id_array = $_POST['id'];
$name_array = $_POST['name'];
$age_array = $_POST['age'];
for ($i = 0; $i < count($id_array); $i++) {
//count($id_array) --> if I input 4 fields, count($id_array) = 4)
$id = mysql_real_escape_string($id_array[$i]);
$name = mysql_real_escape_string($name_array[$i]);
$age = mysql_real_escape_string($age_array[$i]);
$query .= "UPDATE member SET name = '$name', age = '$age' WHERE id = '$id';";
}
mysql_query($query);
}
...
?>
Hope that helps..!
Answer to your Question
MySQL updates all rows matching the WHERE clause, so to update multiple rows with the same value, you should use a condition matching all rows. To update all rows, dont set any where clause.
To update multiple rows with different values, you can't, use several queries.
Answer to your issue
In your code, $id, $name and $age are arrays so you can not use it in a string, this will not work. You should do the update in your FOR loop.
I advise you to try to respect resource oriented principe that all properties are assigned to their item (with associative array or object).
If you dont check the result, you could do all queries in one using a semi-colon.
I have a dynamic Matrix box where users can unlimited drugs, the data comes through like this:
[addindividuals] => [{"Drug":"Calpol","Strength":"100mg","Form":"Liquid","Quantity":"1"},{"Drug":"Paracetamol","Strength":"200mg","Form":"Tablet","Quantity":"16"}]
What I'm trying to achieve is to have each line inserted into a new row (MySQL) and inserts into their relevant columns like so:
Columns: | Drug | Strength | Form | Quantity
Row1 | Calpol | 100mg | Liquid | 1
Row2 |Paracetamol | 200mg | Tablet | 16
I'm guessing its using the exploded function> (I'm a novice) and then sql to insert the strings?
If you have the values as a json string collection, First you need to explode then the string then use a for each to loop through each string then use another for each to make single row. Please have a below code this may help you.
$addindividuals = '{"Drug":"Calpol","Strength":"100mg","Form":"Liquid","Quantity":"1"},{"Drug":"Paracetamol","Strength":"200mg","Form":"Tablet","Quantity":"16"}';
$exploded_array = explode('},',$addindividuals);
$final_query = "INSERT INTO `table_name` (`Drug`,`Strength`,`Form`,`Quantity`) VALUES ";
$exploded_array[0] = $exploded_array[0].'}';
foreach($exploded_array as $exploded_element)
{
$single_row = '(';
$json_decode = json_decode($exploded_element,true);
foreach($json_decode as $key => $value)
{
$single_row .= "'$value',";
}
$single_row = substr($single_row,0,-1);
$single_row .= '),';
$final_query .= $single_row;
}
$final_query = substr($final_query,0,-1);
echo $final_query;
I have a table which looks something like the following (first row is columns):
|section | col1 | col2 |
|----------------------|
|bananas | val | val2 |
|----------------------|
|peaches | val | val2 |
With some code to check if a section value matches up with one of the values in an array:
$sectionscope = Array('bananas', 'apples');
$sections = mysql_query("SELECT section FROM table WHERE col1=val AND col2=val2");
if (mysql_num_rows($sections)) {
$i = 0;
while ($row = mysql_fetch_array($sections)) {
if (in_array($row[$i], $sectionscope)) {
$section = $sectionscope[array_search($row[$i], $sectionscope)];
$section_is_valid = 1;
}
$i++;
}
}
If I echo the output from the while loop using echo $row[$i], it gives me: bananas
Doing the select from PHPmyadmin works fine
Can you tell me what I'm doing wrong here?
Thanks,
SystemError
Why are you incrementing $i in the while loop. Your query will return two rows and when you loop through the results, you can access the section value using $row[0] each time.
while ($row = mysql_fetch_array($sections)) {
echo $row[0];
}
This will print bananas and peaches.
In your code, when you enter the while loop second time, you are trying to retrieve section value using $row[1] (since your $i has incremented to 1), which will be null since your query result only contains one column.