In the first foreach statement, I have 4 attendeeid's in the attendees table.
In the second foreach, I have 1 attendeeid in the attend_date_temp table.
I'm trying to load a select box with names from the attendees table, less the one in the attend_date_temp table.
I thought that, since the first foreach would loop 4 times, the second foreach would also loop 4 times. But it doesn't. It loops one time, causing the code in the second foreach to not execute and load the select box with names.
How can this be written so that the second foreach loops 4 times like the first foreach so the select box will have the names loaded to it?
// Load Button Clicked
if(isset($_POST['loadnames'])) {
/* Read the history file and get the last record for each attendee for a particular group
and a particular member and write them to the attend_date_temp table if attend_date = CURDATE().*/
$stmt = $db->prepare('SELECT historyid, attend_date, attendeeid, groupid, memberid
FROM history
WHERE groupid = :groupid
AND memberid = :memberid
AND attend_date = CURDATE()
ORDER BY historyid
DESC LIMIT 1');
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->bindValue(':memberid', $_SESSION['memberid'], PDO::PARAM_INT);
$stmt->execute();
$result = $stmt->fetchAll();
foreach($result as $row) {
$aid = $row[2]; // set the attendeeid
$stmt = $db->prepare('INSERT INTO attend_date_temp (attendeeid, groupid, memberid)
VALUES(:aid, :gid, :mid)');
$stmt->bindValue(':aid', $aid, PDO::PARAM_INT);
$stmt->bindValue(':gid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->bindValue(':mid', $_SESSION['memberid'], PDO::PARAM_INT);
$stmt->execute();
}
$aaa = 0; // used to set the first select box entry to "Select"
/* Load the Select Box with names, less the ones found in attend_date_temp Table. */
$stmt = $db->prepare('SELECT a.attendeeid, fname, lname, a.groupid, a.memberid, s.attendeeid, suspend
FROM attendees AS a
JOIN suspended AS s ON a.attendeeid = s.attendeeid
WHERE a.memberid = :memberid
AND suspend = "N"
AND a.groupid = :groupid
ORDER BY lname');
$stmt->bindValue(':memberid', $_SESSION["memberid"], PDO::PARAM_INT);
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->execute();
$result = $stmt->fetchAll();
foreach($result as $row){
echo '<script type="text/javascript">alert("In the first loop"); </script>';
$aid = $row[0];
$lname = $row[2];
$fname = $row[1];
$stmt = $db->prepare('SELECT attendeeid, memberid
FROM attend_date_temp
WHERE groupid = :groupid
AND attendeeid = :aid');
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->bindValue(':aid', $aid, PDO::PARAM_INT);
$stmt->execute();
$result2 = $stmt->fetchAll();
foreach ($result2 as $row2) {
echo '<script type="text/javascript">alert("In the second loop"); </script>';
// evaluate attendees attendeeid against attend_date_temp attendeeid
if($row2['attendeeid'] != $aid){
// Load the flush Table with the IDs from the selected group
if($_SESSION['flush'] == 0) {
$stmt = $db->prepare('INSERT INTO flush (attendeeid, memberid)
VALUES(:attendeeid, :memberid)');
$stmt->bindValue(':attendeeid', $aid, PDO::PARAM_INT);
$stmt->bindValue(':memberid', $_SESSION['memberid'], PDO::PARAM_INT);
$stmt->execute();
}
if($aaa == 0) {
echo "<option value='Select'>Select</option>";
echo "<option value=".$aid.">".$lname.", ". $fname."</option>";
$aaa = 1;
} else {
echo "<option value=".$aid.">".$lname.", ". $fname."</option>";
}
}
}
}
$_SESSION['flush'] = 1;
exit();
} // last brace: loadnames
The attend_date_temp table:
DROP TABLE IF EXISTS `attend_date_temp`;
CREATE TABLE `attend_date_temp` (
`attendeeid` int(10) unsigned NOT NULL,
`groupid` int(10) unsigned NOT NULL,
`memberid` int(10) unsigned NOT NULL,
KEY `attendeeid` (`attendeeid`),
KEY `memberid` (`memberid`),
CONSTRAINT `attend_date_temp_ibfk_1` FOREIGN KEY (`attendeeid`) REFERENCES `attendees` (`attendeeid`) ON DELETE CASCADE,
CONSTRAINT `attend_date_temp_ibfk_2` FOREIGN KEY (`memberid`) REFERENCES `members` (`memberid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The history table:
DROP TABLE IF EXISTS `history`;
CREATE TABLE `history` (
`historyid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`amount` float NOT NULL,
`subsidy` char(1) NOT NULL,
`last_payment` date NOT NULL,
`amount_paid` float NOT NULL,
`balance` float NOT NULL,
`attend` char(1) NOT NULL DEFAULT 'N',
`attend_date` date NOT NULL,
`groupid` char(1) NOT NULL,
`attendeeid` int(10) unsigned NOT NULL,
`memberid` int(10) unsigned NOT NULL,
PRIMARY KEY (`historyid`),
KEY `attendeeid` (`attendeeid`),
CONSTRAINT `history_ibfk_15` FOREIGN KEY (`attendeeid`) REFERENCES `attendees` (`attendeeid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
UPDATE:
This is a small part of a payment posting page. Names are loaded in the select box based on the group selected, then payments are posted by the name selected. This not only posts their payments but also their attendance. Once all the money has been collected, the remaining names not selected are marked as absent.
However, there are group members that attend groups that are not their own. When they make a payment, their money is posted and attendance recorded. BUT, and that's what this is all about, when that same person's group gets selected for payments, I don't want that person's name to get loaded. He's already paid, and his attendance has already been updated. To have him load again and be processed wld corrupt the history table. So I have to keep from double loading the same person. That's why I'm trying to use this attend_date_temp table.
Simple fix, use different variable names for the inner and outer loops:-
/* Load the Select Box with names, less the ones found in attend_date_temp Table. */
$stmt = $db->prepare('SELECT a.attendeeid, fname, lname, a.groupid, a.memberid, s.attendeeid, suspend
FROM attendees AS a
JOIN suspended AS s ON a.attendeeid = s.attendeeid
WHERE a.memberid = :memberid
AND suspend = "N"
AND a.groupid = :groupid
ORDER BY lname');
$stmt->bindValue(':memberid', $_SESSION["memberid"], PDO::PARAM_INT);
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->execute();
$result = $stmt->fetchAll();
foreach($result as $row){
echo '<script type="text/javascript">alert("In the first loop"); </script>';
$aid = $row[0];
$lname = $row[2];
$fname = $row[1];
$stmt = $db->prepare('SELECT attendeeid, memberid
FROM attend_date_temp
WHERE groupid = :groupid
AND attendeeid = :aid');
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->bindValue(':aid', $aid, PDO::PARAM_INT);
$stmt->execute();
$result2 = $stmt->fetchAll();
foreach ($result2 as $row2) {
echo '<script type="text/javascript">alert("In the second loop"); </script>';
// evaluate attendees attendeeid against attend_date_temp attendeeid
if($row2['attendeeid'] != $aid){
// Load the flush Table with the IDs from the selected group
if($_SESSION['flush'] == 0) {
$stmt = $db->prepare('INSERT INTO flush (attendeeid, memberid)
VALUES(:attendeeid, :memberid)');
$stmt->bindValue(':attendeeid', $aid, PDO::PARAM_INT);
$stmt->bindValue(':memberid', $_SESSION['memberid'], PDO::PARAM_INT);
$stmt->execute();
}
if($aaa == 0) {
echo "<option value='Select'>Select</option>";
echo "<option value=".$aid.">".$lname.", ". $fname."</option>";
$aaa = 1;
} else {
echo "<option value=".$aid.">".$lname.", ". $fname."</option>";
}
}
}
}
To do a join you would do something like this:-
$stmt = $db->prepare('SELECT a.attendeeid, fname, lname, a.groupid, a.memberid, s.attendeeid, suspend, adt.attendeeid AS adt_attendeeid, adt.memberid AS adt_memberid
FROM attendees AS a
INNER JOIN suspended AS s ON a.attendeeid = s.attendeeid
LEFT OUTER JOIN attend_date_temp adt ON adt.groupid = a.groupid AND adt.attendeeid = a.attendeeid
WHERE a.memberid = :memberid
AND suspend = "N"
AND a.groupid = :groupid
AND adt.groupid IS NULL
ORDER BY lname');
$stmt->bindValue(':memberid', $_SESSION["memberid"], PDO::PARAM_INT);
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->execute();
EDIT
Think it can be more simply done like this (not tested so please excuse any typos)
<?php
$first = true;
/* Load the Select Box with names, less the ones found in attend_date_temp Table. */
$stmt = $db->prepare('SELECT a.attendeeid, fname, lname
FROM attendees AS a
INNER JOIN suspended AS s ON a.attendeeid = s.attendeeid
LEFT OUTER JOIN attend_date_temp adt ON adt.groupid = a.groupid AND adt.attendeeid = a.attendeeid
WHERE a.memberid = :memberid
AND suspend = "N"
AND a.groupid = :groupid
AND adt.groupid IS NULL
ORDER BY lname');
$stmt->bindValue(':memberid', $_SESSION["memberid"], PDO::PARAM_INT);
$stmt->bindValue(':groupid', $_POST['groupid'], PDO::PARAM_INT);
$stmt->execute();
$result = $stmt->fetchAll();
foreach($result as $row)
{
$aid = $row[0];
$lname = $row[2];
$fname = $row[1];
// Load the flush Table with the IDs from the selected group
if($_SESSION['flush'] == 0)
{
$stmt = $db->prepare('INSERT INTO flush (attendeeid, memberid)
VALUES(:attendeeid, :memberid)');
$stmt->bindValue(':attendeeid', $aid, PDO::PARAM_INT);
$stmt->bindValue(':memberid', $_SESSION['memberid'], PDO::PARAM_INT);
$stmt->execute();
}
if($first)
{
echo "<option value='Select'>Select</option>";
echo "<option value='".$aid."'>".$lname.", ". $fname."</option>";
$first = false;
}
else
{
echo "<option value='".$aid."'>".$lname.", ". $fname."</option>";
}
}
Related
I want to show the users the amount of posts, comments etc they have created in their profile section, just like any other forums that exist. I am doing this with php and MYSQLI. Post table:
1 post_id Primary int(11) AUTO_INCREMENT
2 title varchar(255)
3 users_id int(11)
4 content varchar(500)
5 type int(11)
6 imagepath varchar(50)
7 date_created datetime
I have tried to add another column to the post table and increment its value by 1 each time the user makes a post in php by an INSERT statement but it's value only stays at 1, even though the user continues to create more posts. Here is what I tried:
function createPost($conn, $content, $title, $users_id, $date_created, $type, $total_post){
$sql = "INSERT INTO post (title, users_id, content, date_created, type, total_post) VALUES (?,?,?,?,?,?);";
$stmt = mysqli_stmt_init($conn);
if (!mysqli_stmt_prepare($stmt, $sql)){
header("location: ../home.php?error=stmtfailed");
exit();
}
$mysqltime = date ('Y-m-d H:i:s');
$total_post++;
$type;
mysqli_stmt_bind_param($stmt, "ssssss", $title, $users_id, $content, $mysqltime, $type, $total_post);
mysqli_stmt_execute($stmt);
mysqli_stmt_close($stmt);
header("location: ../home.php?error=noerroronpost");
exit();
}
This is in profile.php where im trying to display the information to the user
$id = $_SESSION["userid"];
$stmt = $conn->prepare('SELECT * from post LEFT JOIN users on users.users_id = ? order by post_id DESC;');
$stmt->bind_param('s', $id);
$stmt->execute();
$result = $stmt->get_result();
while($row = $result->fetch_assoc()){
echo "<div class='userinfo'>";
echo "<h5 id='usernameprofile'>" ."Username: " .$row["users_username"] ."</h5>";
echo "<h5 id='usernameprofile'>" ."Registration date: " .$row["create_datetime"] ."</h5>";
echo "<h5 id='usernameprofile'>" ."Posts: " .$row["post_id"] ."</h5>"; echo "<br>";
echo "</div>";
}
$stmt->close();
EDITED: Thanks to ADyson the query he provided worked perfectly. I tweaked it a bit to just show the amount of posts.
$id = $_SESSION["userid"];
mysqli_report(MYSQLI_REPORT_ERROR | MYSQLI_REPORT_STRICT);
$stmt = $conn->prepare('SELECT COUNT(p.post_id) as total_posts from post p INNER JOIN users ON users.users_id = p.users_id WHERE p.users_id = ?;');
$stmt->bind_param('s', $id);
$stmt->execute();
$result = $stmt->get_result();
while($row = $result->fetch_assoc()){
$count = $row['total_posts'];
echo "<div class='usertotalpost'>";
echo "<h5 id='totalposts'>" ."Posts: " .$count ."</h5>"; echo "<br>";
echo "</div>";
}
$stmt->close();
You'll still need to count and group if you want to see totals per user. And your join style is all wrong too - you're supposed to link columns in each table together to make the join, not just restrict on an input field.
Try it like this:
SELECT
u.users_username,
u.create_datetime,
COUNT(p.post_id) AS total_posts
FROM
post p
INNER JOIN users
ON users.users_id = p.users_id
WHERE
p.users_id = ?
GROUP BY
u.users_username,
u.create_datetime
(Obviously the WHERE clause is optional - if you want to a see a list of totals for all users then remove it.)
How do I change the row value of a column in a table when I add another table?
How do you ask for help?
I have two tables in the database
The first table is called Drug
It consists of three columns:
Sample Table I
// TABLE Drug
DROP TABLE IF EXISTS `Drug`;
CREATE TABLE IF NOT EXISTS `Drug` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`brId` text NOT NULL,
`nameDrug` text NOT NULL,
PRIMARY KEY (`id`)
)
The second table is named brand
Sample Table II
// TABLE brand
DROP TABLE IF EXISTS `brand`;
CREATE TABLE IF NOT EXISTS `brand` (
`idBrand` int(11) NOT NULL AUTO_INCREMENT,
`brandName` text NOT NULL,
`theUse` text NOT NULL,
PRIMARY KEY (`idBrand`)
)
What I need is when you add a row in the brand table, brId is updated in the Drug table to the new idBrand by id in the drug table that was sent
I've done the following code because it does not work
<?php
require_once('include/config.php');
$id = $_POST['id'];
$brandName = $_POST['brandName'];
$theUse = $_POST['theUse'];
$query = "INSERT INTO brand
(brandName,theUse)VALUES('".$brandName."','".$theUse."');";
$insertBrand = mysqli_query($con,$query);
if($insertBrand)
{
$updatDrug = "UPDATE `drug` SET `brId` = new.idBrand WHERE `id` = '".$id."' ;";
$resultEnd = mysqli_query($con,$updatDrug);
if($resultEnd){
$result = 'OK';
echo json_encode($result);
}else{
$resultno = 'NO';
echo json_encode($resultno);
}
}
mysqli_close($con);
?>
After the INSERT, use mysqli_insert_id as the value for brId.
$br = mysqli_insert_id($con);
$updatDrug = "UPDATE drug SET brId = :brid WHERE id = :id";
$stmt = $con->prepare($updatDrug);
$stmt->bind_param('ii', $br, $id);
$stmt->execute();
$stmt->close();
And please avoid SQL INJECTION
Try transaction commit, here is an example
<?php
$db = new mysqli("localhost","root","","test"); //连接数据库
$db->autocommit(false); //设置为非自动提交——事务处理
$sql1 = "INSERT INTO `test`.`test1` (`name` )VALUES ('1' )";
$result1 = $db->query($sql1);
$sql2 = "INSERT INTO `test`.`test2` (`a` )VALUES ('1')";
$result2 = $db->query($sql2);
if ($result1 && $result2) {
$db->commit(); //全部成功,提交执行结果
echo '提交';
} else {
$db->rollback(); //有任何错误发生,回滚并取消执行结果
echo '回滚';
}
$db->autocommit(true);
$db->close();
?>
Ive been trying to figure this out but im not sure how to do this. I have a select statement that querys the database and returns an array. That array is then used to query the database again. I am not sure how to save the second array.
$stmt = $dbh->query("SELECT user_id FROM users WHERE username = '" . $user_name . "'");
$row = $stmt->fetch(PDO::FETCH_ASSOC);
$user_id = $row['user_id'];
$stmt = $dbh->query("SELECT friend_id FROM friends WHERE user_id ='" . $user_id . "'");
$ret_array = $stmt->fetchAll(PDO::FETCH_ASSOC);
for($i = 0; $i<count($ret_array); $i++) {
$stmt = $dbh->query("SELECT profile_path, username FROM users WHERE user_id=" .$ret_array[$i]['friend_id']);
}
$show_all_friends = $stmt->fetchAll(PDO::FETCH_ASSOC);
return $show_all_friends;
This is my code and i want the values that are obtained in the forloop to be saved into $show_all_friends. How can i do this?
You don't have to make a loop, just 2 queries with binding method for more security and performance.
$sth = $dbh->prepare("SELECT user_id FROM users WHERE username = ?");
$sth->bindParam(1, $user_name, PDO::PARAM_STR);
$sth->execute();
$row = $sth->fetch(PDO::FETCH_ASSOC);
$user_id = $row['user_id'];
$sth = $dbh->prepare('SELECT u.profile_path, u.username
FROM users u JOIN friends f on f.friend_id = u.user_id
WHERE f.user_id = ?');
$sth->bindParam(1, $user_id, PDO::PARAM_INT);
$sth->execute();
return $sth->fetchAll(PDO::FETCH_ASSOC);
Use a single query using two JOINS.
You can "use" the same table twice by giving it unique aliases within the query so any reference to that table is unambigious.
As mentioned in the comments to your question, this won't work for temporary tables with MySQL. Therefore the following example does not use CREATE TEMPORARY TABLE.... in the setup() function. I usually try to avoid that. So, keep in mind that this example might overwrite exsting tables (though I hope the table name is quite unique) and it persists data on your database server.
<?php
$pdo = new PDO('mysql:host=localhost;dbname=test;charset=utf8', 'localonly', 'localonly');
//echo 'client version: ', $pdo->getAttribute(PDO::ATTR_CLIENT_VERSION), "\n";
//echo 'server version: ', $pdo->getAttribute(PDO::ATTR_SERVER_VERSION), "\n";
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
setup($pdo);
$stmt = $pdo->prepare('
SELECT
u2.user_id
FROM
soNOTTEMPUsers as u1
JOIN
soNOTTEMPFriends as f
ON
u1.user_id=f.user_id
JOIN
soNOTTEMPUsers as u2
ON
f.friend_id=u2.user_id
WHERE
u1.username=:username
');
$stmt->bindParam('username', $username);
foreach( array('userA', 'userB', 'userC', 'userD') as $username ) {
echo 'querying friends of ', $username, "\r\n";
$stmt->execute();
foreach( $stmt as $row ) {
echo ' ', join(', ', $row), "\r\n";
}
}
function setup($pdo) {
$queries = array(
'
CREATE TABLE IF NOT EXISTS soNOTTEMPUsers (
user_id INT,
username varchar(32),
primary key(user_id)
)
',
'
CREATE TABLE IF NOT EXISTS soNOTTEMPFriends (
user_id INT,
friend_id INT,
primary key(user_id, friend_id)
)
',
"REPLACE soNOTTEMPUsers (user_id, username) VALUES
(1, 'userA'), (2, 'userB'), (3, 'userC'), (4, 'userD')
",
"REPLACE INTO soNOTTEMPFriends (user_id, friend_id) VALUES
(1,2),(1,3),(1,4),
(2,1),(2,3),(2,4),
(3,1),(3,4),
(4,1)
",
);
foreach( $queries as $q ) {
$pdo->exec($q);
}
}
prints
querying friends of userA
2, 2
3, 3
4, 4
querying friends of userB
1, 1
3, 3
4, 4
querying friends of userC
1, 1
4, 4
querying friends of userD
1, 1
as expected (though not tested for consistency like a friendof b <=> b friendof a)
PLease change your code with following code
$stmt = $dbh->query("SELECT user_id FROM users WHERE username = '" . $user_name . "'");
$row = $stmt->fetch(PDO::FETCH_ASSOC);
$user_id = $row['user_id'];
$stmt = $dbh->query("SELECT friend_id FROM friends WHERE user_id ='" . $user_id . "'");
$ret_array = $stmt->fetchAll(PDO::FETCH_ASSOC);
for($i = 0; $i<count($ret_array); $i++) {
$stmt = $dbh->query("SELECT profile_path, username FROM users WHERE user_id=" .$ret_array[$i]['friend_id']);
$show_all_friends[] = $stmt->fetchAll(PDO::FETCH_ASSOC);
}
return $show_all_friends;
Hope this helps
i have make its here before. mySQL - how to show all records from the messages table, not just one
Hello
This is how I'm going to build a messaging system which make the user 1 and user 2 has a conversation somewhere.
That itself, I go to the site so come all the conversations appear on the page.
the problem is such that it does only one message from the database. Therefore, I would like it to display all messages from the database.
Database
CREATE TABLE IF NOT EXISTS `fms_opslagpm` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`fra_id` int(11) NOT NULL,
`til_id` int(11) NOT NULL,
`title` varchar(30) NOT NULL,
`besked` longtext NOT NULL,
`datotid` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ;
INSERT INTO `fms_opslagpm` (`id`, `fra_id`, `til_id`, `title`, `besked`, `datotid`) VALUES
(1, 2, 1, 'hrerherhe', 'hello world ', '2014-04-01 22:25:29'),
(2, 2, 1, 'hrerherhe', 'hej', '2014-04-01 23:51:49');
mysqli/php here.
$sql = "
SELECT fms_bruger.fornavn, fms_bruger.efternavn, fms_opslagpm.id, fms_opslagpm.fra_id, fms_opslagpm.til_id, fms_opslagpm.title, fms_opslagpm.besked
FROM fms_bruger INNER JOIN fms_opslagpm ON fms_bruger.id=fms_opslagpm.fra_id
WHERE fms_opslagpm.id = ? and fms_opslagpm.fra_id = ? OR fms_opslagpm.til_id = ?
GROUP BY fms_opslagpm.title ORDER BY fms_opslagpm.datotid DESC
";
if ($stmt = $this->mysqli->prepare($sql)) {
$stmt->bind_param('iii', $id, $fra_id, $til_id);
$id = $_GET["id"];
$fra_id = $_SESSION["id"];
$til_id = $_SESSION["id"];
$stmt->execute();
$stmt->store_result();
$stmt->bind_result($fornavn, $efternavn, $id, $fra_id, $til_id, $title, $besked);
while ($stmt->fetch()) {
?>
<tr class="postbox">
<td class="beskedinfoBOX">
<p>
<?php
echo $fornavn . " " . $efternavn;
?>
</p>
</td>
<td>
<?php
//beskeden.
echo $besked;
?>
</td>
</tr>
<?php
}
$stmt->close();
}
else
{
echo 'Der opstod en fejl i erklæringen: ' . $this->mysqli->error;
}
This is how when I write fra_id = 2 and til_id = 1, then shows it is still only the content of the page. So samtidigvæk a message on the page.
fms_opslagpm = fra_id - is he / she who sends the message
fms_opslagpm = til_id - is he / she who receives the message
Your issue is that you are selecting only
WHERE fms_opslagpm.id = ?
So it will only return 1 row where there is an exact match on the id. It looks like you where trying to also select the rows that have the same title as the row with the id
GROUP BY fms_opslagpm.title
but even if you returned more than 1 row, this would have collapsed the results into 1 row again.
You need to change your query to get the title of the row WHERE fms_opslagpm.id = ?, and using OR select all the other rows with the same title.
Try -
SELECT
fms_bruger.fornavn,
fms_bruger.efternavn,
fms_opslagpm.id,
fms_opslagpm.title,
fms_opslagpm.besked
FROM fms_bruger
INNER JOIN fms_opslagpm ON fms_bruger.id=fms_opslagpm.fra_id
WHERE (
fms_opslagpm.id = ?
OR fms_opslagpm.title = (
SELECT fms_opslagpm.title
FROM fms_opslagpm
WHERE fms_opslagpm.id = ?
)
)
AND
(
fms_opslagpm.fra_id = ?
OR
fms_opslagpm.til_id = ?
)
ORDER BY fms_opslagpm.datotid DESC
See this SQLFiddle example - http://sqlfiddle.com/#!2/36d534/6
You will also need to include 1 more param to your bind_param
$stmt->bind_param('iiii', $id, $id1, $fra_id, $til_id);
$id = $_GET["id"];
$id1 = $_GET["id"];
$fra_id = $_SESSION["id"];
$til_id = $_SESSION["id"];
I have analyzed 3 million images using libpuzzle. 2 million from my main server and 1 million from another. I would like to combine the information into 1 MySQL database.
I need to take records in test_images_pending database and insert them into test_images but I have to do it in a way where there isn't duplicated data.
test_images has 115 million records total across all tables, words having 110 million by itself. Size ~4.4 GB
test_images_pending has 69 million and 65 million respectfully. Size ~2.6 GB
I have 8GB ram on my computer, and I am willing to load everything (or try) in memory if I have to, to speed things up.
I hoping with some optimizations to my code and or techniques to make MySQL faster I can improve the rate from about 2 pictures per second (from test_images_pending.picture table) to something more manageable. The very least would be something like 100 pictures per second.
Here is the table setup for both test_images and test_images_pending:
--
-- Table structure for table `errors`
--
CREATE TABLE IF NOT EXISTS `errors` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`url` varchar(255) NOT NULL,
`num` int(11) NOT NULL,
`pid` bigint(20) unsigned NOT NULL,
`error` varchar(512) NOT NULL,
`datetime` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=245688 ;
-- --------------------------------------------------------
--
-- Table structure for table `pictures`
--
CREATE TABLE IF NOT EXISTS `pictures` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`digest` char(32) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_digest` (`digest`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1107725 ;
-- --------------------------------------------------------
--
-- Table structure for table `signatures`
--
CREATE TABLE IF NOT EXISTS `signatures` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`compressed_signature` varchar(338) NOT NULL,
`picture_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `picture_id` (`picture_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1107725 ;
-- --------------------------------------------------------
--
-- Table structure for table `stored_pictures`
--
CREATE TABLE IF NOT EXISTS `stored_pictures` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`url` varchar(255) NOT NULL,
`pid` bigint(20) unsigned NOT NULL,
`num` int(11) NOT NULL,
`updated_at` datetime DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`picture_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_url` (`url`),
KEY `idx_picture_id` (`picture_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=2773867 ;
-- --------------------------------------------------------
--
-- Table structure for table `words`
--
CREATE TABLE IF NOT EXISTS `words` (
`pos_and_word` char(5) NOT NULL,
`signature_id` int(11) NOT NULL,
KEY `idx_pos_and_word` (`pos_and_word`),
KEY `signature_id` (`signature_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
--
Here is my php PDO code I am running:
<html>
<head>
<link href="../css/print.css" rel="stylesheet" type="text/css" media="print" /> <!-- siehe screen.css -->
<link href="../css/screen.css" rel="stylesheet" type="text/css" media="screen, projection" />
<!--[if lte IE 6]><link rel="stylesheet" href="../css/ielte6.css" type="text/css" media="screen" /><![endif]-->
</head>
<body>
<?php
ini_set('max_execution_time', 0);
$dbh = new PDO("mysql:host=127.0.0.1;port=3306;dbname=test_images_pending;charset=utf-8", "root", "");
$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );
$dbh->setAttribute(PDO::ATTR_AUTOCOMMIT, FALSE);
try {
$query = "select id,digest from test_images_pending.pictures";
$sth = $dbh->prepare($query);
$sth->execute();
while ($pending_pictures_rows = $sth->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
// Print out what id it's on.
print $pending_pictures_rows['id']."<br>";
buffer_flush();
try {
$dbh->beginTransaction();
$query = "SELECT COUNT(id) from test_images.pictures WHERE digest = :digest";
$sth1 = $dbh->prepare($query);
$sth1->bindParam(':digest', $pending_pictures_rows['digest']);
$sth1->execute();
$count = $sth1->fetchColumn();
if ($count == 1) {
$query = "SELECT id from test_images.pictures WHERE digest = :digest";
$sth2 = $dbh->prepare($query);
$sth2->bindParam(':digest', $pending_pictures_rows['digest']);
$sth2->execute();
$correct_pic_id = $sth2->fetchColumn();
if(!isset($correct_pic_id) or empty($correct_pic_id)) {
throw new PDOException('correct_pic_id was empty');
}
$query = "select * from test_images_pending.stored_pictures WHERE picture_id = :picture_id";
$sth3 = $dbh->prepare($query);
$sth3->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth3->execute();
while ($row = $sth3->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
$query = "INSERT INTO test_images.stored_pictures
(id, url, pid, num, updated_at, created_at, picture_id)
VALUES
(default, :url, :pid, :num, :updated_at, :created_at, :picture_id);";
$sth4 = $dbh->prepare($query);
$sth4->bindParam(':url', $row['url']);
$sth4->bindParam(':pid', $row['pid']);
$sth4->bindParam(':num', $row['num']);
$sth4->bindParam(':updated_at', $row['updated_at']);
$sth4->bindParam(':created_at', $row['created_at']);
$sth4->bindParam(':picture_id', $correct_pic_id);
$sth4->execute();
}
$query = "DELETE FROM test_images_pending.stored_pictures WHERE picture_id = :picture_id;";
$sth5 = $dbh->prepare($query);
$sth5->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth5->execute();
$query = "select id from test_images_pending.signatures WHERE picture_id = :picture_id;";
$sth6 = $dbh->prepare($query);
$sth6->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth6->execute();
$signature_id = $sth6->fetchColumn();
if(!isset($signature_id) or empty($signature_id)) {
throw new PDOException('signature_id was empty');
}
$query = "DELETE FROM test_images_pending.words WHERE signature_id = :signature_id;";
$sth7 = $dbh->prepare($query);
$sth7->bindParam(':signature_id', $signature_id);
$sth7->execute();
$query = "DELETE FROM test_images_pending.signatures WHERE picture_id = :picture_id";
$sth8 = $dbh->prepare($query);
$sth8->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth8->execute();
$query = "DELETE FROM test_images_pending.pictures WHERE digest = :digest";
$sth9 = $dbh->prepare($query);
$sth9->bindParam(':digest', $pending_pictures_rows['digest']);
$sth9->execute();
} else if ($count == 0){
$query = "INSERT INTO test_images.pictures
(id, digest)
VALUES
(default, :digest);";
$sth2 = $dbh->prepare($query);
$sth2->bindParam(':digest', $pending_pictures_rows['digest']);
$sth2->execute();
$new_pic_id = $dbh->lastInsertId();
$query = "select * from test_images_pending.stored_pictures WHERE picture_id = :picture_id";
$sth3 = $dbh->prepare($query);
$sth3->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth3->execute();
while ($row = $sth3->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
$query = "INSERT INTO test_images.stored_pictures
(id, url, pid, num, updated_at, created_at, picture_id)
VALUES
(default, :url, :pid, :num, :updated_at, :created_at, :picture_id);";
$sth4 = $dbh->prepare($query);
$sth4->bindParam(':url', $row['url']);
$sth4->bindParam(':pid', $row['pid']);
$sth4->bindParam(':num', $row['num']);
$sth4->bindParam(':updated_at', $row['updated_at']);
$sth4->bindParam(':created_at', $row['created_at']);
$sth4->bindParam(':picture_id', $new_pic_id);
$sth4->execute();
}
$query = "DELETE FROM test_images_pending.stored_pictures WHERE picture_id = :picture_id;";
$sth5 = $dbh->prepare($query);
$sth5->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth5->execute();
$query = "select id,compressed_signature from test_images_pending.signatures WHERE picture_id = :picture_id;";
$sth6 = $dbh->prepare($query);
$sth6->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth6->execute();
$fetched = $sth6->fetch(PDO::FETCH_ASSOC);
$signature_id = $fetched['id'];
if(!isset($signature_id) or empty($signature_id)) {
print_r($sth6->fetch(PDO::FETCH_ASSOC));
throw new PDOException('signature_id was empty');
}
$compressed_signature = $fetched['compressed_signature'];
if(!isset($compressed_signature) or empty($compressed_signature)) {
print_r($sth6->fetch(PDO::FETCH_ASSOC));
throw new PDOException('compressed_signature was empty');
}
$query = "INSERT INTO test_images.signatures
(id, compressed_signature, picture_id)
VALUES
(default, :compressed_signature, :picture_id);";
$sth7 = $dbh->prepare($query);
$sth7->bindParam(':picture_id', $new_pic_id);
$sth7->bindParam(':compressed_signature', $compressed_signature);
$sth7->execute();
$new_sig_id = $dbh->lastInsertId();
$query = "SELECT pos_and_word FROM test_images_pending.words WHERE signature_id = :signature_id";
$sth8 = $dbh->prepare($query);
$sth8->bindParam(':signature_id', $signature_id);
$sth8->execute();
while ($row = $sth8->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
$query = "INSERT INTO test_images.words
(pos_and_word, signature_id)
VALUES
(:pos_and_word, :signature_id);";
$sth9 = $dbh->prepare($query);
$sth9->bindParam(':pos_and_word', $row['pos_and_word']);
$sth9->bindParam(':signature_id', $new_sig_id);
$sth9->execute();
}
$query = "DELETE FROM test_images_pending.words WHERE signature_id = :signature_id;";
$sth10 = $dbh->prepare($query);
$sth10->bindParam(':signature_id', $signature_id);
$sth10->execute();
$query = "DELETE FROM test_images_pending.signatures WHERE picture_id = :picture_id";
$sth11 = $dbh->prepare($query);
$sth11->bindParam(':picture_id', $pending_pictures_rows['id']);
$sth11->execute();
$query = "DELETE FROM test_images_pending.pictures WHERE digest = :digest";
$sth12 = $dbh->prepare($query);
$sth12->bindParam(':digest', $pending_pictures_rows['digest']);
$sth12->execute();
} else {
throw new PDOException("Found more than 1 match for the digest '{$pending_pictures_rows['digest']}' in 'test_images.pictures' ", $query);
}
$dbh->commit();
} catch (PDOException $e) {
$dbh->rollback();
print "<pre>"; print_r($e); print "</pre>"; exit;
}
}
try {
$dbh->beginTransaction();
$query = "SELECT * FROM test_images_pending.errors";
$sth13 = $dbh->prepare($query);
$sth13->execute();
while ($row = $sth13->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
$query = "INSERT INTO test_images.errors
(id, url, num, pid, error, datetime)
VALUES
(default, :url, :num, :pid, :error, :datetime);";
$sth14 = $dbh->prepare($query);
$sth14->bindParam(':url', $row['url']);
$sth14->bindParam(':num', $row['num']);
$sth14->bindParam(':pid', $row['pid']);
$sth14->bindParam(':error', $row['error']);
$sth14->bindParam(':datetime', $row['datetime']);
$sth14->execute();
}
$query = "DELETE FROM test_images_pending.errors WHERE 1";
$sth15 = $dbh->prepare($query);
$sth15->execute();
$dbh->commit();
} catch (PDOException $e) {
$dbh->rollback();
print "<pre>"; print_r($e); print "</pre>"; exit;
}
} catch (PDOException $e) {
print "<pre>"; print_r($e); print "</pre>"; exit;
}
function buffer_flush(){
echo str_pad('', 512);
echo '<!-- -->';
if(ob_get_length()){
#ob_flush();
#flush();
#ob_end_flush();
}
#ob_start();
}
?>
</body>
</html>
Edit:
Some profiling:
This INSERT gets ran 100 times each non-similar picture (~5 every 6 thus far). It normally takes 0.5 to 0.9 seconds to finish the while loop with an average of 0.007 per INSERT.
$query = "INSERT INTO test_images.words
(pos_and_word, signature_id)
VALUES
(:pos_and_word, :signature_id);";
$sth9 = $dbh->prepare($query);
$sth9->bindParam(':pos_and_word', $row['pos_and_word']);
$sth9->bindParam(':signature_id', $new_sig_id);
$sth9->execute();
DELETE FROM test_images_pending.stored_pictures WHERE picture_id = :picture_id;
select * from test_images_pending.stored_pictures WHERE picture_id = :picture_id
DELETE FROM test_images_pending.stored_pictures WHERE picture_id = :picture_id;
all take an average of 0.15 seconds or so per similar picture (~1 out of 6).
Edit 2:
Going by this benchmarking: http://we-love-php.blogspot.com/2012/08/mass-inserts-updates-sqlite-vs-mysql.html
Replacing the slow while loop previously mentioned in Edit 1 with just simple writing to text file such as:
$inserts = array();
while ($row = $sth8->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
$inserts[] = "(".$dbh->quote($row['pos_and_word']).", ".$dbh->quote($new_sig_id).")";
}
$query = "INSERT INTO imvu_images.words (pos_and_word, signature_id) VALUES " . implode(',',$inserts) . ";";
file_put_contents("inserts.sql", $query."\n", FILE_APPEND);
Makes it a l ot faster. Not 100 per second though, more like 10-20. I can then just execute the SQL later on and it runs instantly without delay. (That's why I think there is an issue with my code). The reason why I want 100 per second is because I can analyze images and insert them into 1 database at 30 per second. At this rate, it's faster for me to analyze 2 million images and have it insert each one by one than it is to mass insert the rows. This doesn't seem right, that the server can download 30 images, analyze 30 images, and then do 30 inserts in 1 second yet just doing these various SQL statements cannot even match that.
Edit 3:
Updated my.ini with:
key_buffer_size=4000M
read_buffer_size=32M
read_rnd_buffer_size=200M
bulk_insert_buffer_size=1000M
myisam_max_sort_file_size=10000M
myisam_repair_threads=1
tmp_table_size = 1024M
max_heap_table_size = 1024M
join_buffer_size=8M
sort_buffer_size=8M
max_allowed_packet=32M
max_connect_errors=10
myisam_sort_buffer_size=256M
query_cache_limit=12M
query_cache_size=256M
query_cache_type=1
Which seems to have improved performance 2 fold without using the file_put_contents hack. Still though, 5 records a second isn't cutting it.
The reason this process is so slow is not because the individual queries are slow - in fact, I'm surprised at how fast it's all going - but because you're processing millions of records, one at a time, by looping through each record in your outer resultset. What SQL is good at is processing millions of records all in one go.
There's too much business logic in your code for me to want to re-write the whole thing for you, but I think you want to have a re-write the code along the lines of
INSERT INTO test_images.pictures
(id, digest)
SELECT id, digest
from test_images_pending.pictures
where id not in
(select id from test_images.pictures)
Do the same for the other tables. This should run pretty fast - if you've got a good indexing scheme, you'll almost certainly be I/O bound. You should definitely reach more than 2 records per second!
Why can't you use Mysql stored procedures? They execute in Mysql server directly and Faster than Query execution from php.
http://dev.mysql.com/doc/refman/5.0/en/create-procedure.html
Call the stored procedure from php like this:
$res = mysql_query('call sp_sel_test()');
if ($res === FALSE) {
die(mysql_error());
}
You need to set client flags while connecting for using stored procedures with php. Use this:
mysql_connect($this->h,$this->u,$this->p,false,65536);
See MySQL Client Flags for more details.
Edit: The main issue was indices on the source tables being INSERTing into. It is recommended to drop any non needed indices before doing mass inserts, then rebuild afterwords.
With a combination of tweaking the mysql settings and the following code, I was able to get the duplicate images (The join portion) to do 50,000 in 30 seconds or so, 25 seconds being just the JOIN operation.
The 2nd part I am using NOT IN and this is where most of the time occurs but it inserts at a rate of 800 records per second, so it exceeds my goal.
I am going to leave the question open for a bit longer to see if it can be optimized more, because I have 39 million records to process.
<html>
<head>
<link href="../css/print.css" rel="stylesheet" type="text/css" media="print" /> <!-- siehe screen.css -->
<link href="../css/screen.css" rel="stylesheet" type="text/css" media="screen, projection" />
<!--[if lte IE 6]><link rel="stylesheet" href="../css/ielte6.css" type="text/css" media="screen" /><![endif]-->
</head>
<body>
<?php
ini_set('max_execution_time', 0);
$benchmark = false;
$delete = false;
$dbh = new PDO("mysql:host=127.0.0.1;port=3306;dbname=test_images_pending;charset=utf-8", "root", "");
$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );
$dbh->setAttribute(PDO::ATTR_AUTOCOMMIT, FALSE);
$timers = array();
try {
$query = "SELECT * FROM test_images.pictures
INNER JOIN test_images_pending.pictures
USING ( digest )";
$sth = $dbh->prepare($query);
$sth->execute();
while ($join_rows = $sth->fetch(PDO::FETCH_NUM, PDO::FETCH_ORI_NEXT)) {
$digest = $join_rows[0];
$correct_pic_id = $join_rows[1];
$wrong_pic_id = $join_rows[2];
try {
$dbh->beginTransaction();
$query = "INSERT INTO test_images.stored_pictures
(url, pid, num, updated_at, created_at, picture_id)
SELECT
url, pid, num, updated_at, created_at, :correct_pic_id FROM test_images_pending.stored_pictures WHERE picture_id = :wrong_pic_id;";
$sth4 = $dbh->prepare($query);
$sth4->bindParam(':correct_pic_id', $correct_pic_id);
$sth4->bindParam(':wrong_pic_id', $wrong_pic_id);
$sth4->execute();
$dbh->commit();
} catch (PDOException $e) {
$dbh->rollback();
print "<pre>"; print_r($e); print "</pre>"; exit;
}
}
} catch (PDOException $e) {
print "<pre>"; print_r($e); print "</pre>"; exit;
}
try {
$query = "SELECT COUNT(id) FROM `signatures` WHERE (`id` - `picture_id` !=0) ";
$sth = $dbh->prepare($query);
$sth->execute();
$count = $sth->fetchColumn();
if($count > 0) {
die("we got a sig that aint matching its pic_id, we cant assume sig_id = pic_id. Back to drawing board");
}
$sth = null;
$query = " SELECT digest, id
FROM test_images_pending.pictures
WHERE digest NOT IN
(
SELECT digest
FROM test_images.pictures
)";
$sth = $dbh->prepare($query);
$sth->execute();
while ($not_in_rows = $sth->fetch(PDO::FETCH_NUM, PDO::FETCH_ORI_NEXT)) {
$digest = $not_in_rows[0];
$wrong_pic_id = $not_in_rows[1];
try {
$dbh->beginTransaction();
$query = "INSERT INTO test_images.pictures
(id, digest)
VALUES
(default, :digest);";
$sth2 = $dbh->prepare($query);
$sth2->bindParam(':digest', $digest);
$sth2->execute();
$new_pic_id = $dbh->lastInsertId();
$query = "INSERT INTO test_images.stored_pictures
(url, pid, num, updated_at, created_at, picture_id)
SELECT
url, pid, num, updated_at, created_at, :new_pic_id FROM test_images_pending.stored_pictures WHERE picture_id = :wrong_pic_id;";
$sth3 = $dbh->prepare($query);
$sth3->bindParam(':new_pic_id', $new_pic_id);
$sth3->bindParam(':wrong_pic_id', $wrong_pic_id);
$sth3->execute();
$query = "INSERT INTO test_images.signatures
(compressed_signature, picture_id)
SELECT
compressed_signature, :new_pic_id FROM test_images_pending.signatures WHERE picture_id = :wrong_pic_id;";
$sth4 = $dbh->prepare($query);
$sth4->bindParam(':new_pic_id', $new_pic_id);
$sth4->bindParam(':wrong_pic_id', $wrong_pic_id);
$sth4->execute();
$new_sig_id = $dbh->lastInsertId();
$query = "INSERT INTO test_images.words
(pos_and_word, signature_id)
SELECT
pos_and_word, :new_sig_id FROM test_images_pending.words WHERE signature_id = :old_sig_id
";
$sth9 = $dbh->prepare($query);
$sth9->bindParam(':old_sig_id', $wrong_pic_id);
$sth9->bindParam(':new_sig_id', $new_sig_id);
$sth9->execute();
$dbh->commit();
} catch (PDOException $e) {
$dbh->rollback();
print "<pre>"; print_r($e); print "</pre>"; exit;
}
}
} catch (PDOException $e) {
print "<pre>"; print_r($e); print "</pre>"; exit;
}
function buffer_flush(){
echo str_pad('', 512);
echo '<!-- -->';
if(ob_get_length()){
#ob_flush();
#flush();
#ob_end_flush();
}
#ob_start();
}
?>
</body>
</html>