So i've been trying to sort this issue out for a while but been unable to figure out if i have to do this trough SQL side or the PHP side.
My code i have so far:
<?php
$servername="";
$username = "";
$password = "";
$dbname = "alphacre_kingsland";
$conn = mysqli_connect($servername, $username, $password, $dbname);
if (!$conn) {
die("Connection failed: " . mysqli_connect_error());
}
$sql = "SELECT * FROM `bm_player_bans` ORDER BY id";
$result = mysqli_query($conn, $sql);
if (mysqli_num_rows($result)>0){
while($row = mysqli_fetch_assoc($result)){
$playerId = bin2hex($row['player_id']);
$storedname= getPlayerName($playerId);
$sql = "INSERT INTO bm_onlinedata (uuid, playername) VALUES ('".$playerId."', '".$storedname."')";
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
}else{
echo "0 results";
}
mysqli_close($conn);
function getPlayerName($uuid){
$json_response = file_get_contents("https://sessionserver.mojang.com/session/minecraft/profile/".$uuid);
$data = json_decode($json_response);
return $data->name;
}
?>
So this stores "UUID" and playername into a seperate table. But i want to check if the UUID is already in there before it adds a new one. Basicly updating it.
This is because playernames can change. And i don't want multiple rows of the same data.
But thats ONLY if its there ofc.
Background information:
I'm fetching the username from a user trough Mojangs API, wich only allows a single lookup of that UUID every minute. So i want to get the data every 5 minutes or so using a scheduled task running this script and store it in a table.
You can easily check with select
$sql = "SELECT * FROM `bm_player_bans` ORDER BY id";
$result = mysqli_query($conn, $sql);
if (mysqli_num_rows($result)>0){
while($row = mysqli_fetch_assoc($result)){
$playerId = bin2hex($row['player_id']);
$storedname= getPlayerName($playerId);
$sql = "SELECT * FROM `bm_onlinedata` where uuid = '" . $playerId ."';";
$result = mysqli_query($link, $sql);
$num_rows = mysqli_num_rows($result);
if ($num_rows> 0) {
$sql = "UPDATE bm_onlinedata
set playername = '". $storedname."'
where uuid = '" .$playerId ."';"; )";
} else {
$sql = "INSERT INTO bm_onlinedata (uuid, playername) VALUES ('".$playerId."', '".$storedname."')";
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
}
}else{
echo "0 results";
}
Below I have Php code that loops through an array and for each it checks if the value already exists in the database and if not, create it. The code itself is working but the loop itself can be insanely big, maximum of a couple tens thousand iterations.
How can I optimize this code? What to use and how to use. There should be a better way to insert this many times without looping through each individual.
foreach($arr as $value){
$checkID = mysqli_query($cenn, "SELECT item_id from items WHERE item_id = '$value'");
if (!$checkID) {
die("Query '$checkID' failed to execute for some reason");
}else{
if (mysqli_num_rows($checkID) > 0) {
$user = mysqli_fetch_array($checkID);
echo "item_id" . checkID . "exists already";
}
else{
echo "item_id: '$user_id' doesn't exist<br>";
$gw2Api = file_get_contents("https://api.guildwars2.com/v2/items/" . $user_id); //12452 30704
$gw2Api_result = json_decode($gw2Api,true);
/*Here would be some code to determine values that are being inserted*/
if (!array_key_exists("description",$gw2Api_result)) {
$description = 'No description available...';
} else{
if($gw2Api_result['description'] === ''){
$description = "No description available...";
} else {
$description = $gw2Api_result['description'];
}
}
$insertItem = "INSERT INTO items
(item_id, name, description,
AccountBindOnUse, AccountBound,
last_update
)
VALUES ('$user_id', '$gw2Api_result[name]', '$description',
'$AccountBindOnUse', '$AccountBound', CURRENT_TIMESTAMP)";
if ($cenn->query($insertItem) === true) {
echo "New record '$user_id' created successfully";
} else {
echo "Error: " . $sql . "<br>" . $cenn->error;
}
}
}
} // end foreach
The question: How to insert many values, new rows, into mysqli database as fast as possible.
Just use bulk insert.
Collect all the rows for insertion and pass it in one query.
echo 'hi';
if (!empty($arr)) {
echo 'ok';
$values = "'" . implode("', '", $arr) . "'";
$qExistingItemIds = mysqli_query($cenn, "SELECT item_id from items WHERE item_id IN($values)");
$existingItemIds = [];
while ($existingItemId = mysqli_fetch_array($qExistingItemIds)) {
$existingItemIds[] = $existingItemId['item_id'];
}
$arr = array_diff($arr, $existingItemIds);
$inserts = array();
$i = 0;
$ic = count($arr);
foreach ($arr as $value) {
$i++;
echo "item_id: $value doesn't exist<br>";
$gw2Api = file_get_contents("https://api.guildwars2.com/v2/items/" . $value); //12452 30704
$gw2Api_result = json_decode($gw2Api,true);
/*Here would be some code to determine values that are being inserted*/
if (!array_key_exists("description", $gw2Api_result)) {
$description = 'No description available...';
} else {
if ($gw2Api_result['description'] === '') {
$description = "No description available...";
} else {
$description = $gw2Api_result['description'];
}
}
$inserts[] = "
('$value', '$gw2Api_result[name]', '$description', '$AccountBindOnUse', '$AccountBound', CURRENT_TIMESTAMP)
";
if ($i == 50 OR $i == $ic) {
$inserts = implode(",", $inserts);
$insert = "
INSERT INTO items
(item_id, name, description, AccountBindOnUse, AccountBound, last_update)
VALUES
$inserts
";
if ($cenn->query($insert) === true) {
echo 'great';
echo "New records created successfully";
} else {
echo "Error: " . $sql . "<br>" . $cenn->error;
}
$ic -= 50;
$i = 0;
$inserts = array();
}
}
}
so now we have only 2 queries. not thousands
details about bulk insert:
http://www.geeksengine.com/database/data-manipulation/bulk-insert.php
If you use prepared statement you should reduce the round trips to the database server and only compile and optimise each query once instead of Number_of_inputs * 2 queries. This should reduce the workload.
I would be very interested to know by how much.
$sql = "SELECT item_id from items WHERE item_id = ?";
$db_select = $cenn->prepare($sql);
if ( ! $db_select ) {
echo $cenn->error;
exit;
}
$sql_insert = "INSERT INTO items
(item_id, name, description,
AccountBindOnUse, AccountBound, last_update)
VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP)";
$db_insert = $cenn->prepare($sql);
if ( ! $db_insert ) {
echo $cenn->error;
exit;
}
foreach($arr as $value){
$db_select->bind_param('i', $value);
$res = $db_select->execute()
if ( $res === FALSE ) {
echo $cenn->error;
exit;
}
if ($db_select->num_rows > 0) {
// dont bother fetching data we already know all we need to
$user = $db_select->free();
echo "item_id $value exists already";
} else {
echo "item_id: $value doesn't exist<br>";
$gw2Api = file_get_contents("https://api.guildwars2.com/v2/items/" . $value);
$gw2Api_result = json_decode($gw2Api,true);
if ( ! array_key_exists("description",$gw2Api_result)
|| $gw2Api_result['description'] === '') {
$description = 'No description available...';
} else{
$description = $gw2Api_result['description'];
}
$db_insert->bind_param('issss', $value, $gw2Api_result[name],
$description, $AccountBindOnUse,
$AccountBound)
if ($cenn->query($insertItem) === true) {
echo "New record $value' created successfully";
} else {
echo "Error: " . $sql_insert . "<br>" . $cenn->error;
}
}
} // end foreach
I need to perform a batch insert in MySQL/MariaDB but since data is dynamic I need to build the proper SQL query. In a few steps:
I should find whether the current row exists or not in table - this is the first SELECT inside the loop
Right now I have 1454 but have to insert around 150k later, is better a batch query than 150k INSERT per item on the loop
If record already exists I should update it if doesn't then I should insert ,I just not care about UPDATE yet and the code you're seeing is only for INSERT
So here is what I am doing:
// Get values from Csv file as an array of values
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
// VALUES should be added only if row doesn't exists
if ($rows_returned === 0) {
// VALUES should be append until they reach 1000
while ($i % 1000 !== 0) {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW())";
++$i;;
}
// QUERY should be output to console to see if it's right or something is wrong
echo "DEBUG: ", $sqlInsert, "\n";
// QUERY should be executed if there are 1000 VALUES ready to add as a batch
/*$rs = $conn->query($sqlInsert);
if ($rs === false) {
echo 'Wrong SQL: '.$sqlInsert.' Error: '.$conn->error, E_USER_ERROR;*/
}
} else {
// UPDATE
echo "UPDATE";
}
}
}
But this line of code: echo "DEBUG: ", $sql, "\n"; is not outputting nothing to console. I must be doing something wrong but I can't find what. Can any help me to build the proper batch query and to execute it each 1000 values append?
Proper output should be:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008ReolAAC'
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='005800000039SIWAA2'
....
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES(...), VALUES(...), VALUES(...)
Obtained result:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RGg6AAG'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RQ4CAAW'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
.... // until reach 1454 results
The table is empty so it should never goes through ELSE condition (UPDATE one).
EDIT
With help from the answer this is how the code looks now:
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 1;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
if ($rows_returned === 0) {
if ($i % 1000 === 0) {
file_put_contents("output.log", $sqlInsert."\n", FILE_APPEND);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
} else {
$sqlInsert .= "($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW()), ";
}
$i++;
} else {
echo "UPDATE";
}
}
}
But still buggy because:
I have got a first empty INSERT query: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES
I have got a second INSERT query with 1000 VALUES() append, but what happened with the rest? The remaining 454?
Can any give me another tip? Help?
Since it looks like you are trying to load data from a CSV file, you might want to consider using LOAD DATA INFILE functionality which is designed specifically for this purpose.
Here is link to documentation: https://dev.mysql.com/doc/refman/5.6/en/load-data.html
consider using INSERT IGNORE INTO table to check if the record already exists. How to 'insert if not exists' in MySQL?
if you haven't already done so, make veeva_rep_id a PRIMARY key so the INSERT IGNORE will work
also check out using PDO for transactions, prepared statements and dynamically generating queries using PDO
PDO Prepared Inserts multiple rows in single query
<?php
$sql = 'INSERT IGNORE INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES ';
$insertQuery = array();
$insertData = array();
/*
assuming the array from the csv is like this
$data = array(
0 => array('name' => 'Robert', 'value' => 'some value'),
1 => array('name' => 'Louise', 'value' => 'another value')
);
*/
foreach ($data as $row) {
$insertQuery[] = '(:veeva_rep_id' . $n . ', :first' . $n . ', :last' . $n . ', :email' . $n . ', :username' . $n . ', :lastLoginAt' . $n . ', :lastSyncAt' . $n . ', :display_name' . $n . ', :rep_type' . $n . ', :avatar_url' . $n . ', :createdAt' . $n . ', :updatedAt' . $n . ')';
$insertData['veeva_rep_id' . $n] = $row['name'];
$insertData['first' . $n] = $row['value'];
$insertData['last' . $n] = $row['name'];
$insertData['email' . $n] = $row['value'];
$insertData['username' . $n] = $row['name'];
$insertData['lastLoginAt' . $n] = $row['value'];
$insertData['lastSyncAt' . $n] = $row['value'];
$insertData['display_name' . $n] = $row['name'];
$insertData['rep_type' . $n] = $row['value'];
$insertData['avatar_url' . $n] = $row['value'];
$insertData['createdAt' . $n] = $row['name'];
$insertData['updatedAt' . $n] = $row['value'];
$n++;
}
$db->beginTransaction();
if (!empty($insertQuery) and count($insertQuery)>1000) {
$sql .= implode(', ', $insertQuery);
$stmt = $db->prepare($sql);
$stmt->execute($insertData);
}
$db->commit();
print $sql . PHP_EOL;
let me know if it helps.
You should have something like:
// Try fetching data from table 1
// If there is no record available, then fetch some data from table 2
// and insert that data inito table 1
You just wrote
$sql = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
But from an insert no data is selected and second...you didn't run a select, where comes $data from?
update Use if ($i % 1000 === 0) { instead of while ($i % 1000 !== 0) {
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$veeva_rep_id = ...;
$first = ...;
$last = ...;
$email = ...;
// ...
// VALUES should be added only if row doesn't exists
if($rs->num_rows == 0) {
// Insert some data
$i++;
if ($i % 1000 === 0) {
echo "DEBUG: ", $sqlInsert, "\n";
// execSql($sqlInsert);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) "; // reset
} else {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,...) ";
}
} else {
echo "UPDATE";
}
}
}
I am working on a php script that stores message ids (Msg_ID, Ref_ID) in their corresponding user account tables.
What I've is, the Msg_ID is properly written, but the Ref_ID is always blank.
How ever when I run the query separately it works, but doesn't work in the script for some odd reason.
Here is the code :
$qry = "SELECT Ref_ID FROM Chat WHERE Msg_ID = " .$MsgID. ")";
$resp = mysqli_query($con, $qry);
$xx = mysqli_fetch_array($resp);
$ref_id = $xx['Ref_ID'];
foreach ($Array as $user){
$query = "Insert into ".$user."(POST_ID, REF_ID) values ('". $MsgID . "', '" .$ref_id. "')";
mysqli_query($con, $query);
}
The $ref_id is always blank and as a result, the blank value is written to the respective database.
Some help with what is wrong will be helpful.
Here is the full code :
<?php
function PostMainThread($Heading, $Message, $Author, $MarkedList){
$con=mysqli_connect("mysql.serversfree.com", "u521497173_root", "123456", "u521497123_mydb");
$Array = explode(',', $MarkedList);
if (mysqli_connect_errno()){
$response["success"] = 0;
$response["message"] = "Connection Failed.";
echo json_encode($response);
}else{
here:$MsgID = rand(1, 9999999);
$query = "Insert into Chat(Msg_ID, Header, MsgBody, Author) values (". $MsgID . "," . "'" . $Heading . "' ," .
"'" . $Message . "', '". $Author . "')";
$result=mysqli_query($con, $query);
if (!$result){
goto here;
}else{
//Put the MsgID in the respective user tables.
$qry = "SELECT Ref_ID FROM Chat WHERE Msg_ID = " .$MsgID. ")";
$resp = mysqli_query($con, $qry);
$xx = mysqli_fetch_array($resp);
$ref_id = $xx['Ref_ID'];
foreach ($Array as $user){
$query = "Insert into ".$user."(POST_ID, REF_ID) values ('". $MsgID . "', '" .$ref_id. "')";
mysqli_query($con, $query);
}
$response["success"] = 1;
$response["message"] = "Submission successful.";
mysqli_close($con);
echo json_encode($response);
}
}
}
function PostReplyToThread($PostID, $Author, $Reply){
$con=mysqli_connect("mysql.serversfree.com", "u521497123_root", "123456", "u521497123_mydb");
if (mysqli_connect_errno()){
echo 2;
}else{
$query = "Insert into Chat(Msg_ID, Header, MsgBody, Author) values (". $PostID . "," . "'" . " " . "' ," .
"'" . $Reply . "', '". $Author . "')";
$result=mysqli_query($con, $query);
if ($result){
echo 3;
}else{
echo 4;
}
mysqli_close($con);
}
}
if (isset($_POST['what_to_do'])){
if ($_POST['what_to_do'] == 0){
if ((isset($_POST['Title'])) &&(isset($_POST['Body']))&&(isset($_POST['Marked']))&&(isset($_POST['_Author']))){
PostMainThread($_POST['Title'], $_POST['Body'], $_POST['_Author'], $_POST['Marked']);
}
}else if ($_POST['what_to_do'] == 1){
if ((isset($_POST['Thread_ID'])) &&(isset($_POST['Answer']))&&(isset($_POST['_Author']))){
PostReplyToThread($_POST['Thread_ID'], $_POST['_Author'], $_POST['Answer']);
}
}
}else{
$response["success"] = 0;
$response["message"] = "Unspecified action";
echo json_encode($response);
}
Definition of the Chat table :
Create table Chat(Ref_ID INT Auto_Increment, Msg_ID INT, Header varchar(50), MsgBody varchar(500
), Author varchar(30), Primary Key(Ref_ID, Msg_ID));
$xx = mysqli_fetch_array($resp);
Will only return a numerically indexed array, as in $xx[0], $xx[1].
To use the column names, you need to use:
$xx = mysqli_fetch_array($resp, MYSQLI_ASSOC);
Or the shorter version:
$xx = mysqli_fetch_assoc($resp);
As a side note, don't forget security, when inserting data that comes from outside the function and could possibly have a quotes or SQL, it needs to be escaped!
$Heading = mysqli_real_escape_string($con, $Heading);
Otherwise it will come back to bite you.