Still having duplicate vaues in mysql table - php

I already took advises on the web regarding to prevent duplicate values from saving to the mysql table but it still saves it and I already placed echo statements and exit to ensure this doesn't happen but it still has. Here is the code that I used and edited to my preferences:
$line = array();
$file = fopen("D:/Documents/Documents/LOGS.txt", "r");
while (! feof($file))
{
$line[] = fgetcsv($file, 4096);
foreach ($line as $value)
{
$newline = explode(" ", $value[1]);
$date = trim($newline[0]);
$time = trim($newline[1]);
$check = mysqli_query($cxn, "select * from temprec where EmpID = '$value[0]'
and ValidDate = '$date' and ValidTime = '$time'") or die("Couldn't execute
query");
$checkrows = mysqli_num_rows($check);
if ($checkrows > 0) {
echo "data already saved";
exit;
}
else
{
$result = mysqli_query($cxn, "insert ignore into
temprec(EmpID,ValidDate,ValidTime) values('$value[0]','$date','$time')") or
die("Couldn't execute query");
}
}
}
fclose($file);
Note: the logs.txt contains data from a biometrics machine and it contains too much duplicate values.

You have a mistake in the check query. In string
"select * from temprec where EmpID = '$value[0]' and ValidDate = '$date' and ValidTime = '$time'" variable $value[0] parces as just $value and you get query like this "select * from temprec where EmpID = 'array[0]' and ValidDate = '$date' and ValidTime = '$time'".
Using '{}' should fix it: "select * from temprec where EmpID = '${value[0]}' and ValidDate = '$date' and ValidTime = '$time'".

have you tried echoing your "insert ignore into temprec(EmpID,ValidDate,ValidTime) values('$value[0]','$date','$time')") or
die("Couldn't execute query" for you to be sure if your query shows the right query?

Related

select all records and the rest

here is a simple mysqli query to select specific records from my mysql database:
foreach ($getData as $data) { {
$sql = "SELECT * FROM `myTable` WHERE `bookid` = '".$data['ID']."' ";
$result = $db->query( $sql );
while ($zeile = $result->fetch_object()) {
// DO SOMETHING
}
}
The question is:
How can I get (best practices) all the other records, which will not be selected with this query filter?
It can be like
foreach ($getData as $data) { {
$sql = "SELECT * FROM `myTable` WHERE `bookid` = '".$data['ID']."' ";
$result = $db->query( $sql );
while ($zeile = $result->fetch_object()) {
// DO SOMETHING
$query2 = "select * from 'myTable' WHERE 'bookid' != ".$zeile['ID']."'";
$result2 = $db -> query($query2);
// do something...
}
}
OR
foreach ($getData as $data) { {
$sql = "SELECT * FROM `myTable` WHERE `bookid` = '".$data['ID']."' ";
$result = $db->query( $sql );
while ($zeile = $result->fetch_object()) {
// DO SOMETHING
}
$query2 = "select * from 'myTable' WHERE 'bookid' != '".$data['ID']."'";
$result2 = $db -> query($query2);
while($zeile2 = $result2-> fetch_object()){
// do something...
}
}
Running successive, more or less identical, queries in a loop is fundamentally a bad way to do things. Create a list of $data['ID'] values you want to work with, then use one query to retrieve all the rows IN that list, and a second query to retrieve everything NOT IN that list:
Important: This code assumes that the values in $getData[]['ID'] can be trusted. i.e. they have been validated before entry to this code, or they come from a trusted source.
// Create a list:
$inList = '('.implode(',', array_column($getData, 'ID')).')';
$sqlIn = "SELECT * FROM `myTable` WHERE `bookid` IN $inList";
// run the query. Check for errors
if (($result = $db->query( $sqlIn )) === false) {
throw new Exception($db->error);
}
while ($zeile = $result->fetch_object()) {
// DO SOMETHING
}
// Now use the same list to exclude those rows
$sqlOut = "SELECT * FROM `myTable` WHERE `bookid` NOT IN $inList";
// run the query. Check for errors
if (($result = $db->query( $sqlOut )) === false) {
throw new Exception($db->error);
}
while ($zeile = $result->fetch_object()) {
// DO SOMETHING
}

More efficient way of storing MikroTik IP accounting data?

I had a previous question on how to do this script and have figured out a way but I am not too sure if I should use this method.
The other post is available here.
Can someone point me in the right direction if there is a more faster/efficient way of running this script as my previous way I was doing it was sometimes taking over an hour to finish.
<?php
//Require admin
require_once("inc/admin.php");
require_once ("../includes/routeros_api.class.php");
//SET
$ip = "10.100.1.1";
//Connect to MikroTik API
$API = new RouterosAPI();
$API->debug = $config['api']['debug'];
if (!$API->connect($ip, $config['api']['username'], $config['api']['password'])) {
echo "Could not connect to RouterOS API";
} else {
$API->write('/ip/accounting/snapshot/take',true);
$READ = $API->read(false);
$ARRAY = $API->parseResponse($READ);
$API->write('/ip/accounting/snapshot/print',true);
$READ = $API->read(false);
$ARRAY = $API->parseResponse($READ);
foreach($ARRAY as $ACCOUNTING) {
$ip_src = $ACCOUNTING['src-address'];
$ip_dst = $ACCOUNTING['dst-address'];
$bytes = $ACCOUNTING['bytes'];
//Check if ip in use UPLOAD
$query = "SELECT id, ipv4 FROM services WHERE ipv4='$ip_src' AND deleted !='1'";
$result = mysqli_query($conn, $query);
$row = mysqli_fetch_array($result);
if(mysqli_num_rows($result) > 0) {
$service_id = $row['id'];
//Update Download Traffic
$check_if_exist_query = "SELECT * FROM traffic_counters WHERE service_id='$service_id' AND date=CURRENT_DATE()";
$check_result = mysqli_query($conn, $check_if_exist_query);
$check_num_rows = mysqli_num_rows($check_result);
if($check_num_rows == 0) {
$add_query = "INSERT INTO traffic_counters (service_id, upload_bytes, date) VALUES ('$service_id', '$bytes', CURRENT_DATE());";
$add_result = mysqli_query($conn, $add_query);
} else {
$update_query = "UPDATE traffic_counters SET
upload_bytes = upload_bytes + $bytes
WHERE service_id='$service_id' AND date=CURRENT_DATE();
";
$update_result = mysqli_query($conn, $update_query);
}
}
//Check if ip in use DOWNLOAD
$query = "SELECT id, ipv4 FROM services WHERE ipv4='$ip_dst' AND deleted !='1'";
$result = mysqli_query($conn, $query);
$row = mysqli_fetch_array($result);
if(mysqli_num_rows($result) > 0) {
$service_id = $row['id'];
//Update Download Traffic
$check_if_exist_query = "SELECT * FROM traffic_counters WHERE service_id='$service_id' AND date=CURRENT_DATE()";
$check_result = mysqli_query($conn, $check_if_exist_query);
$check_num_rows = mysqli_num_rows($check_result);
if($check_num_rows == 0) {
$add_query = "INSERT INTO traffic_counters (service_id, download_bytes, date) VALUES ('$service_id', '$bytes', CURRENT_DATE());";
$add_result = mysqli_query($conn, $add_query);
} else {
$update_query = "UPDATE traffic_counters SET
download_bytes = download_bytes + $bytes
WHERE service_id='$service_id' AND date=CURRENT_DATE();
";
$update_result = mysqli_query($conn, $update_query);
}
}
}
$API->disconnect();
}
?>
1.) You can start with "simplify" insert queries:
Just have some variable $INSERT_VALUES and do this in loops
$INSERT_VALUES.=", ('$service_id', '$bytes', CURRENT_DATE())";
After Loop, you can do only one Insert (only one query) to database:
$add_query = "INSERT INTO traffic_counters (service_id, upload_bytes, date) VALUES INSERT_VALUES";
2.) if you have LARGE amount of rows, using UPDATE is very bad idea, but at this type of using that, there is no way to solve this. You can Try call on mysql START TRANSACTION; and after all of updates, call COMMIT;
3.) On large database SELECT ... WHERE ... is slower than load all datas to php array and then find your needed row by array_search or by using good method of creating array (For example $DATA[service_id]["colunm"]=$value)

Import a single CSV file into multiple tables in MySQL

I would like to import a single CSV with 7 columns into 2 tables in MySQL.
Columns 1, 2 and 3 go into a single row in table1. Columns 4 and 5 go as a row in table2. Columns 6 and 7 go as a row again in same table2.
How can this be done using PHP or mysql directly?
First fetch all values from csv and store it in an array. Then maintain your array as per your requirement and then start looping and inserting.
<?php
$efected = 0;
$file_temp = $_FILES["file"]["tmp_name"];
$handle = fopen($file_temp, "r"); // opening CSV file for reading
if ($handle) { // if file successfully opened
$i=0;
while (($CSVrecord = fgets($handle, 4096)) !== false) { // iterating through each line of our CSV
if($i>0)
{
list($name, $gender, $website, $category, $email, $subcat) = explode(',', $CSVrecord); // exploding CSV record (line) to the variables (fields)
//print_r($subcat); sub_cat_ids
if($email!='')
{
$chk_sql = "select * from employees where employee_email='".$email."'";
$chk_qry = mysqli_query($conn, $chk_sql);
$chk_num_row = mysqli_num_rows($chk_qry);
$cat_array = array(trim($category));
$subcat_array = explode( ';', $subcat );
if(count($subcat_array)>0)
{
$subcat_array_new = array();
foreach($subcat_array as $key => $val)
{
$subcat_array_new[] = trim($val);
}
}
$cat_sub_cat_merge = array_merge($cat_array, $subcat_array_new);
$cat_subcat_comma = implode( "','", $cat_sub_cat_merge );
foreach($cat_array as $cat_key => $cat_val)
{
$chk_cat_sql = "select * from employee_cats where cats_name = '".$cat_val."'";
$chk_cat_qry = mysqli_query($conn, $chk_cat_sql);
$chk_cat_num_row = mysqli_num_rows($chk_cat_qry);
//$all_cat_array = array();
if($chk_cat_num_row == 0)
{
$new_cat_ins = "insert into employee_cats set cats_name = '".$cat_val."', parent_cat_id = '0' ";
$new_cat_ins_qry = mysqli_query($conn, $new_cat_ins);
}
}
foreach($subcat_array_new as $subcat_key => $subcat_val)
{
$chk_subcat_sql = "select * from employee_cats where cats_name = '".$subcat_val."'";
$chk_subcat_qry = mysqli_query($conn, $chk_subcat_sql);
$chk_subcat_num_row = mysqli_num_rows($chk_subcat_qry);
//$all_cat_array = array();
if($chk_subcat_num_row == 0 && trim($subcat_val)!='')
{
//$category
$get_catid_sql = "select * from employee_cats where cats_name = '".trim($category)."'";
$chk_catid_qry = mysqli_query($conn, $get_catid_sql);
$fetch_cat_info = mysqli_fetch_array($chk_catid_qry);
$fetch_cat_id = $fetch_cat_info['cats_id'];
$new_subcat_ins = "insert into employee_cats set cats_name = '".$subcat_val."', parent_cat_id = '".$fetch_cat_id."' ";
$new_subcat_ins_qry = mysqli_query($conn, $new_subcat_ins);
}
}
$get_cat_sql = "select * from employee_cats where cats_name in ('".$cat_subcat_comma."')";
$get_cat_qry = mysqli_query($conn, $get_cat_sql);
$get_cat_num_row = mysqli_num_rows($get_cat_qry);
$sub_category_ids = array();
if($get_cat_num_row>0)
{
while($fetch_cat_id = mysqli_fetch_array($get_cat_qry))
{
if($fetch_cat_id['parent_cat_id']==0)
{
$category_id = $fetch_cat_id['cats_id'];
}
else
{
$sub_category_ids[] = $fetch_cat_id['cats_id'];
}
}
$sub_cat_id_vals_comma = implode(",", $sub_category_ids);
}
else
{
$category_id = 0;
$sub_cat_id_vals_comma = "";
}
if($chk_num_row>0)
{
// and here you can easily compose SQL queries and map you data to the tables you need using simple variables
$update_sql = "update employees set
employee_name='".$name."',
employee_gender='".$gender."',
employees_website='".$website."',
employees_cat_id='".$category_id."',
sub_cat_ids='".$sub_cat_id_vals_comma."'
where employee_email='".$email."'";
$mysqli_qry = mysqli_query($conn, $update_sql);
}
else
{
// and here you can easily compose SQL queries and map you data to the tables you need using simple variables
$insert_sql = "insert into employees set
employee_name='".$name."',
employee_gender='".$gender."',
employees_website='".$website."',
employees_cat_id='".$category_id."',
sub_cat_ids='".$sub_cat_id_vals_comma."',
employee_email='".$email."'";
$mysqli_qry = mysqli_query($conn, $insert_sql);
}
$efected = 1;
}
}
$i++;
}
fclose($handle); // closing file handler
}
if($efected==1)
{
header('location:'.$site_url.'?import=success');
}
else
{
header('location:'.$site_url.'?import=failed');
}
?>

php mysql trying to use empty to only add a new record in table if a primary key exists in a different table

I am trying to only allow a submission via the form only if a party_id exists in a table using empty, here is my code at the moment it is still allowing everything through even if there is no party_id.
Any help would be great.
if($_SERVER["REQUEST_METHOD"]== "POST") {
$party_id = (int)$_POST["partyid"];
$name = $_POST["name"];
$date = $_POST["date"];
$length = (int)$_POST["length"];
$sql = "SELECT * FROM `party` WHERE `party_id`='" . $party_id . "'";
$res = mysqli_query($link, $sql);
if(empty($party_id)) { #Were any records found?
print '<p>No Parties with that ID found! please press the back button to select another party</p>';
} else {
$record = mysqli_fetch_assoc($res);
$party_name = $record["party_name"];
$price = $record["price"];
$cost = $price * $length;
$bookable = true;
$sql2 = "SELECT * FROM `reservations`" or die("Unable to connect to database");
A simpler way might be to just check if the query returned any results like this.
if($_SERVER["REQUEST_METHOD"]== "POST") {
$party_id = (int)$_POST["partyid"];
$name = $_POST["name"];
$date = $_POST["date"];
$length = (int)$_POST["length"];
$sql = "SELECT * FROM `party` WHERE `party_id`='$party_id'";
$res = mysqli_query($link, $sql);
if ( mysqli_num_rows($res ) == 0 ) {
print '<p>No Parties with that ID found! please press the back button to select another party</p>';
} else {
$record = mysqli_fetch_assoc($res);
$party_name = $record["party_name"];
$price = $record["price"];
$cost = $price * $length;
$bookable = true;
$sql2 = "SELECT * FROM `reservations`" or die("Unable to connect to database");

Upload csv and insert or update the results failing

Can anyone see what I am doing wrong here? Basically what is supposed to happen is you upload the csv, it then pulls the rows out and checks with the database if that row exists, if it does it updates it, if it doesn't it inserts it. If the row is blank it ignores it.
What is actually happening is when I upload a csv I just get 'updated' for each of the 4 test lines whether it is in the database or not.
Also if anyone can suggest a better way of doing this, or trimming down the code please let me know, as I know by coding isn't the greatest by any stretch of the imagination.
if(isset($_GET['uploadfile'])) {
$file = fopen($_FILES['csvfile']['tmp_name'], 'r+');
while(! feof($file))
{
$line = fgetcsv($file, 0, ',');
list($productcode, $v9cm, $v1litre, $v2litre, $v3litre, $v5litre, $v7litre) = $line;
$rowcheck = "SELECT * FROM `stock` WHERE `productcode` = '$productcode'";
if (#mysql_num_rows(mysql_query($rowcheck))!=1) {
if ($productcode == NULL OR $productcode == ''){}
else {
$datecreated = date('Y-m-d');
// Insert Posted Data
mysqli_query($db,"INSERT INTO stock (`datecreated`, `productcode`, `9cm`, `1litre`, `2litre`, `3litre`, `5litre`, `7litre`) VALUES ('$datecreated', '$productcode', '$v9cm', '$v1litre', '$v2litre', '$v3litre', '$v5litre', '$v7litre')") or die ('Unable to execute query. '. mysqli_error());
echo $productcode.' - Added<br/ >';
}
}
else{
if ($productcode == NULL OR $productcode == ''){}
else {
$stmt = $db->prepare("UPDATE stock SET 9cm = '$v9cm',1litre = '$v1litre',2litre = '$v2litre',3litre = '$v3litre',5litre = '$v5litre',7litre = '$v7litre' WHERE productcode = '$productcode'");
if ($stmt === FALSE) { echo "an error has occured"; }
$stmt->execute();
$stmt->close();
echo $productcode.' - Updated<br/ >';
}
}
}
fclose($file);
}
}
There seemed to be a brace missing which I think would indeed have made it try to update whatever - indented statements to help make it a bit clearer and added mysqli instead of mysql where it appeared to be wrong - assuming everything should be mysqli. Hope that may help.
You will need to add clean-up/sanitizing/escape code but I assume you know that
if(isset($_GET['uploadfile'])){
$file = fopen($_FILES['csvfile']['tmp_name'], 'r+');
while(! feof($file)){
$line = fgetcsv($file, 0, ',');
list($productcode, $v9cm, $v1litre, $v2litre, $v3litre, $v5litre, $v7litre) = $line;
$rowcheck = "SELECT * FROM `stock` WHERE `productcode` = '$productcode'";
if (#mysqli_num_rows(mysqli_query($rowcheck))!=1){
if ($productcode == NULL OR $productcode == ''){
echo "Product code missing and rowcheck is not 1<br/ >"; // for testing
}else{
$datecreated = date('Y-m-d');
// Insert Posted Data
mysqli_query($db,"INSERT INTO stock (`datecreated`, `productcode`, `9cm`, `1litre`, `2litre`, `3litre`, `5litre`, `7litre`) VALUES ('$datecreated', '$productcode', '$v9cm', '$v1litre', '$v2litre', '$v3litre', '$v5litre', '$v7litre')") or die ('Unable to execute query. '. mysqli_error());
echo $productcode.' - Added<br/ >';
}
}elseif($productcode == NULL OR $productcode == ''){
echo "Product code missing<br/ >"; // for testing
}else{
$stmt = $db->prepare("UPDATE stock SET 9cm = '$v9cm',1litre = '$v1litre',2litre = '$v2litre',3litre = '$v3litre',5litre = '$v5litre',7litre = '$v7litre' WHERE productcode = '$productcode'");
if ($stmt === FALSE){
echo "an error has occured<br/ >";
}else{
$stmt->execute();
$stmt->close();
echo $productcode.' - Updated<br/ >';
}
}
}
}
You might also like to look at the security implications of using uploaded files directly by their temp name http://www.acunetix.com/websitesecurity/php-security-4/ among many others.
This is what worked in the end :
if(isset($_GET['uploadfile'])){
$file = fopen($_FILES['csvfile']['tmp_name'], 'r+');
while(! feof($file)){
$line = fgetcsv($file, 0, ',');
list($productcode, $v9cm, $v1litre, $v2litre, $v3litre, $v5litre, $v7litre) = $line;
$query = mysqli_query($db, "SELECT * FROM stock WHERE productcode='".$productcode."'");
if(mysqli_num_rows($query) > 0){
$stmt = $db->prepare("UPDATE stock SET 9cm = '$v9cm',1litre = '$v1litre',2litre = '$v2litre',3litre = '$v3litre',5litre = '$v5litre',7litre = '$v7litre'
WHERE productcode = '$productcode'");
if ($stmt === FALSE) { echo "an error has occured"; }
$stmt->execute();
$stmt->close();
echo $productcode.' - Updated<br/ >';
}else{
// do something
if (!mysqli_query($db,$query))
{
$datecreated = date('Y-m-d');
// Insert Posted Data
mysqli_query($db,"INSERT INTO stock (`image`,`genusid`,`datecreated`, `productcode`, `9cm`, `1litre`, `2litre`, `3litre`, `5litre`, `7litre`) VALUES ('None.gif','5','$datecreated', '$productcode', '$v9cm', '$v1litre', '$v2litre', '$v3litre', '$v5litre', '$v7litre')") or die ('Unable to execute query. '. mysqli_error());
echo $productcode.' - Added<br/ >';
}
}
}}}
It may be easier to use mysql's ON DUPLICATE KEY UPDATE clause (http://dev.mysql.com/doc/refman/5.7/en/insert-on-duplicate.html).
See hjpotter92's answer here: On Duplicate Key Update same as insert. Your query would need to be modified to something like this:
INSERT INTO table (id,a,b,c,d,e,f,g)
VALUES (1,2,3,4,5,6,7,8)
ON DUPLICATE KEY
UPDATE a=a, b=b, c=c, d=d, e=e, f=f, g=g;
This should allow you to remove most of the conditional logic and also cut down on the number of queries you run.

Categories