I need to read and process (for example add a "2" at the end of IDs + some other things...) and re-insert at least 2000 records form MySQL with PHP.
It's not possible to do it with a SQL Query...
The problem is when we click on the Go button, it processes almost 500 records but then we see a "Server Internal 500" error! but in localhost we don't have any problem...
Is there a way to do this with these limited resources in our customer websites?
Another question: What causes this problem? What resource needs to be more? RAM? CPU?...?
Here is the code:
(We should read all courses of a semester and all course selections and copy them to the new semester)
foreach ($courseList as $courseInfo)
{
$ccode=$courseInfo['ccode'];
$lid=$courseInfo['lid'];
$pid=$courseInfo['pid'];
$clid=$courseInfo['clid'];
$cgender=$courseInfo['cgender'];
$descriptive_score=$courseInfo['descriptive_score'];
$final_capacity=$courseInfo['final_capacity'];
$days_times=$courseInfo['days_times'];
$exam_date=$courseInfo['exam_date'];
$exam_place=$courseInfo['exam_place'];
$ccourse_start_date=date("Y-m-d H:i:s");
$ccomment=$courseInfo['ccomment'];
$cid=$courseInfo['cid'];
$majors=$course->GetCourseMajorsList($cid);
$scList=$course->GetCourseScoreColumnsList($cid);
$courseScoreColums=array();
foreach ($scList as $scProp)
{
$courseScoreColums[$scProp['scid']]=$scProp['scid'].','.$scProp['cscfactor'];
}
$tid = $term->LastTermID();
$counts = $course->AddCourse($ccode.'2',$tid,$lid,$pid,$clid,$majors,$courseScoreColums,$cgender,$descriptive_score,$final_capacity,$days_times,NULL,$exam_place,$ccourse_start_date,$ccomment,$aid);
if ($counts==1)
{
$new_cid = $course->LastCourseID();
$cs = new ManageCourseStudents();
$query = " WHERE `".$table_prefix."courses`.`cid`=$cid ";
$courseStudentList = $cs->GetCourseStudentList($query,'');
foreach ($courseStudentList as $csInfo)
$cs->AddCourseStudent($csInfo['uid'],$new_cid,$csInfo['lvid'],$aid);
}
}
$str = "select * from tableName"; //replacing * by table columns is a good practice
$result = $conn->query($str);
while($arr = $result->fetch_array(MYSQLI_ASSOC);)
{
$strr1 = " INSERT INTO tableName ( `id` , `title` , `des` ) //add more
VALUES ( '".$arr['id']."2', '".$arr['something']."' //add as you require
";
if($conn->query($strr1) === false) {
trigger_error('Wrong SQL: ' . $sql . ' Error: ' . $conn->error, E_USER_ERROR);
}
}
this could duplicate every data in your database
Related
I'm trying to run a SQL query, and it is working correctly in phpMyAdmin, but whe running it in PHP, the query comes back very wonky. The following query yields two different results:
SELECT `stock_ticker`, `stock_simpleName`, `stock_logo`
FROM `stocks`
WHERE stock_simpleName REGEXP'^c'
I get the following results in phpMyAdmin (Which is correct):
stock_simpleName
----------------------
Coca-Cola
Campbell's
ConAgra Foods
However, in PHP it comes out really weird:
stock_simpleName
-----------------------
Coca-Cola
MasterCard
Campbell's
Microsoft
The Walt Disney Company
PepsiCo
The Hershey Company
Proctor & Gamble
ConAgra Foods
...etc...
Why is this happening? This doesn't make any sense. Is it due to a server setting in PHP or some form of encoding or whatnot?
EDIT:
Here is my PHP Code:
The sub-model class (the creator of the pieces):
public function allOtherSearchResults($query, $dontQuery = null) {
$name = "stocks";
$where = "stock_simpleName REGEXP'^" . $query . "'";
$cols = array("stock_ticker", "stock_simpleName", "stock_logo");
$limit = 5;
return $this->select($name, $cols, $where, $limit);
}
The main-model class (this runs the query):
public function select($tableName, $columns, $where = null, $limit = null) {
global $purifier;
// Make columns SQL friendly
$cols = "`";
$cols .= implode("`, `", $columns);
$cols .= "`";
$table = "`" . $tableName . "`";
if (!empty($where)) {
$where = " WHERE " . $where;
}
// Check limit
if (!empty($limit)) {
$limit = " LIMIT $limit";
}
// SQL CODE
$sql = "SELECT " . $cols . " FROM " . $table . $where . $limit;
// SQL DEBUGGING IF CODE RETURNS BOOLEAN ERROR
echo $sql . "<br>";
$query = $this->conn->query($sql);
// Store the value in a variable called table with an array of that table's name followed by it's values
// EX: $model->table["bands"]["band_name"]
//
// Accessible by the individual page/directory's controller's
while($row = $query->fetch_assoc()){
// Store values as $model->table["tableName"]["columnName"]["index (usually 0)"]
foreach ($row as $key => $val) {
$this->data[$tableName][$key][] = $row[$key];
}
}
// Loop through results to clean them
// Foreach loops through each column
// Make sure the table isn't empty (i.e. login returns an error)
if (!empty($this->data[$tableName])) {
foreach ($this->data[$tableName] as $key => $tableArray) {
// For loop goes through each value in a certain row
for ($i = 0; $i < count($tableArray); $i++) {
// Convert from data variable to table after HTML PURIFIER
$this->table[$tableName][$key][$i] = $purifier->purify($tableArray[$i]);
}
}
}
// Declare the array after loop has finished for use in view
$this->table;
if (!empty($this->table)) {
return true;
}
}
And it gives me the same SQL output as above. I am not sure if there is a different interpretation of certain characters in PHP versus the standard MySQL in phpMyAdmin. Has anyone even had this problem before?
I'm guessing, that there is a problem wiht ^ character.
Try to set proper connection & result encoding, eq.
$this->conn->query("MYSQL SET NAMES utf8");
$this->conn->query("MYSQL SET CHARACTER SET utf8");
Also, check if your php script file is saved in UTF-8 encoding.
Moreover, you should consider of using prepared statement (even to prevent SQL Injection):
$this->conn->prepare("SELECT * FROM `stocks` WHERE `stock_simpleName` REGEXP ?");
$this->conn->bind_param("s", "^c");
$this->conn->execute();
$query = $this->conn->get_result();
I have an array that I am passing through a $_POST variable. In the foreach loop, I would like to check to see if "delete" has been checked. If so, I will delete the record. If delete is not checked, I will update the record. When I run this the update works perfect. If I have delete checked - it will delete the first record in the array - not the record with the corresponding id of the record with the checked box. If I have 2 checked - it will take the first 2. If I have them all checked - it will obviously then delete all the records. I have been wrestling with this one all day - looking for a little help. Thanks.
if (isset($_POST['update'])) {
$slo_id = $_POST['slo_id'];
foreach ($_POST['score_id'] as $key=>$score_id) {
$del = $_POST['del'][$key];
if ($del == 'checked') {
$sql2 = " DELETE FROM slo_score WHERE score_id = $score_id ";
#execute SQL statement
$result = mysql_query($sql2);
# check for error
if (mysql_error()) { print "Database ERROR in SQL Update Statement: " . mysql_error(); }
}
else {
$growth_target = $_POST['growth_target'][$key];
$final_score = $_POST['final_score'][$key];
$meets = $_POST['meets'][$key];
//update table
$sql1 = "UPDATE slo_score SET growth_target='$growth_target',final_score='$final_score',meets='$meets' WHERE score_id = $score_id";
#execute SQL statement
$result = mysql_query($sql1);
# check for error
if (mysql_error()) { print "Database ERROR in SQL Update Statement: " . mysql_error(); }
}
}
header("location:...);
}
try this
if (isset($_POST['update'])) {
$slo_id = $_POST['slo_id'];
foreach ($_POST['score_id'] as $key=>$score_id) {
if (isset($_POST['del'][$key])) {
//use mysqli to delete the record
}
else {
$growth_target = $_POST['growth_target'][$key];
$final_score = $_POST['final_score'][$key];
$meets = $_POST['meets'][$key];
//update table
use mysqli to update the record
}
}
header("location:...);
}
You should avoid looping deletes to MySQL at all costs.
If you are receiving an array of IDs to delete, why not just send that in one (or a few, if this is thousands of IDs we're talking about; chunk them) query:
$score_ids = isset($_POST['score_id']) ? $_POST['score_id'] : array();
$score_ids_in = implode("', '", $score_ids);
$sql = sprintf("DELETE FROM slo_score WHERE score_id IN ('%s')", $score_ids_in);
$result = mysql_query($sql);
Of course, this won't give you line by line feedback but it will keep your database from being choked to death.
Also, you already know you should use Mysqli to keep people from destroying your database, extracting sensitive data from it, or both. It's not a huge change from what you're doing now, except it's a lot more secure.
I'm querying my SQL database in a PHP file from up to three optional search fields (passed through by jQuery). Any one, two or three of these fields can be used at any time to make the query as expansive or as narrow as the user likes. If nothing is in a search field nothing will be returned.
I've written the code so far to handle very basic one search queries and have just begun to add in the multiple parameters - this is where it's starting to get tricky. I can query two fields together without too much bother but adding a third LOCATION parameter is beginning to take up too much code for all of the querying possibilities a user might make.
Here's how my PHP file is set up for two parameters:
if (!empty($_POST['title']) && (!empty($_POST['name'])))
{
require '../db/connect.php';
$sql = "SELECT
....
FROM
....
WHERE
`table 3`.`TRACKTITLE` = '" . mysql_real_escape_string(trim($_POST['title'])) . "' AND `table 3`.`ARTIST` = '" . mysql_real_escape_string(trim($_POST['name'])) . "'";
}
if (!empty($_POST['name']))
{
require '../db/connect.php';
$sql = "SELECT
...
FROM
...
WHERE
`table 3`.`ARTIST` = '" . mysql_real_escape_string(trim($_POST['name'])) . "'";
}
if (!empty($_POST['title'])) {
require '../db/connect.php';
$sql = "SELECT
...
FROM
...
WHERE
`table 3`.`TRACKTITLE` = '" . mysql_real_escape_string(trim($_POST['title'])) . "'";
}
$result = mysql_query($sql);
$data = array();
while ($array = mysql_fetch_assoc($result)) {
$data[] = $array;
Which is the simplest way to build a query with multiple optional parameters in PHP, accounting for any additional parameters that might be added on at a later date? I've read up on isnull values but do they perform a similar function to !emtpy?
Do something along this line:
$whereclauses = array();
$subsets = false;
// for every field
if(!empty($_POST['name']))
{
$subsets = true;
$whereclauses[] = " artist = ". mysql_real_escape_string(trim($_POST['name']));
}
if($subsets)
{
$whereclauses = implode(", ". $whereclauses);
}
else
{
$whereclauses ="";
}
// now build your query
I have a form that is going to be used to search through a table of support tickets.
the user can search from a few difficult optional fields.
Date (to/from)
Ticket Status
Engineer
Ticket Contact
I'm wondering what is the best way to deal with optional search filters. So I have a query that takes in parameters from the user. So if the user searches using both the from and to dates then the query would want to include BETWEEN. So do I have to write a different query for if the user searches for only from. or another query when the user has not added any date parameters? Then what if the status dropdown is blank? Is that another query?
Any help to clear this up would be great!
Jonesy
Build your query in parts. Start with whatever is constant in your query, and add on more SQL depending on what extra conditions:
$query = "SELECT ...
FROM ...
WHERE [where conditions that are always going to be present]";
if (isset($_POST['date_from']) && isset($_POST['date_to']))
{
$query .= ... // query code for dealing with dates
}
if (isset($_POST['status']))
{
$query .= ... // deal with status
}
// etc.
// Once you have your query fully built, execute it
$result_set = mysql_query($query);
This code is obviously just a skeleton, but that's how I would construct my query.
Hard to say without knowing what sort of DB abstraction you're using, but assuming you're hand-writing the SQL, it's fairly simple, just build up sections of your where clause individually for each variable. (Assuming here that your vars are already escaped/quoted.)
$where_clause = array();
if (!empty($date_from)) {
$where_clause[] = "table.date >= $date_from";
}
if (!empty($date_to)) {
$where_clause[] = "table.date <= $date_to";
}
if (!empty($status)) {
$where_clause[] = "status = $status";
}
$query = 'select * from table where ' . join(' and ', $where_clause);
This is an elegant way that I use alot and wish will help you too:
$q = 'SELECT * FROM Users';
$buildQ = array();
if (empty($idOrName) === false) {
$buildQ[] = '(userid = "' . $idOrName . '" OR username LIKE "%' . $idOrName. '%")';
}
if (empty($nickname) === false) {
$buildQ[] = 'nickname="' . $nickname . '"';
}
if (empty($salary) === false) {
$buildQ[] = 'salary="' . $salary . '"';
}
// ... any other criterias like above if statements
if (count($buildQ) === 1) {
$q .= ' WHERE ' . $buildQ[0];
} else if (count($buildQ) > 1) {
$count = 0;
foreach ($buildQ as $query) {
if ($count === 0) {
$q .= ' WHERE ' . $query;
} else {
$q .= ' AND ' . $query;
}
$count++;
}
}
I think it would be better if You generate query dynamically at runtime based on which fields are filled. So You could make some helper which appends specific query fragments if only one date is passed and the other one is null, or when both are passed and so on.
Long before I knew anything - not that I know much even now - I desgined a web app in php which inserted data in my mysql database after running the values through htmlentities(). I eventually came to my senses and removed this step and stuck it in the output rather than input and went on my merry way.
However I've since had to revisit some of this old data and unfortunately I have an issue, when it's displayed on the screen I'm getting values displayed which are effectively htmlentitied twice.
So, is there a mysql or phpmyadmin way of changing all the older, affected rows back into their relevant characters or will I have to write a script to read each row, decode and update all 17 million rows in 12 tables?
EDIT:
Thanks for the help everyone, I wrote my own answer down below with some code in, it's not pretty but it worked on the test data earlier so barring someone pointing out a glaring error in my code while I'm in bed I'll be running it on a backup DB tomorrow and then on the live one if that works out alright.
I ended up using this, not pretty, but I'm tired, it's 2am and it did its job! (Edit: on test data)
$tables = array('users', 'users_more', 'users_extra', 'forum_posts', 'posts_edits', 'forum_threads', 'orders', 'product_comments', 'products', 'favourites', 'blocked', 'notes');
foreach($tables as $table)
{
$sql = "SELECT * FROM {$table} WHERE data_date_ts < '{$encode_cutoff}'";
$rows = $database->query($sql);
while($row = mysql_fetch_assoc($rows))
{
$new = array();
foreach($row as $key => $data)
{
$new[$key] = $database->escape_value(html_entity_decode($data, ENT_QUOTES, 'UTF-8'));
}
array_shift($new);
$new_string = "";
$i = 0;
foreach($new as $new_key => $new_data)
{
if($i > 0) { $new_string.= ", "; }
$new_string.= $new_key . "='" . $new_data . "'";
$i++;
}
$sql = "UPDATE {$table} SET " . $new_string . " WHERE id='" . $row['id'] . "'";
$database->query($sql);
// plus some code to check that all out
}
}
Since PHP was the method of encoding, you'll want to use it to decode. You can use html_entity_decode to convert them back to their original characters. Gotta loop!
Just be careful not to decode rows that don't need it. Not sure how you'll determine that.
I think writing a php script is good thing to do in this situation. You can use, as Dave said, the html_entity_decode() function to convert your texts back.
Try your script on a table with few entries first. This will make you save a lot of testing time. Of course, remember to backup your table(s) before running the php script.
I'm afraid there is no shorter possibility. The computation for millions of rows remains quite expensive, no matter how you convert the datasets back. So go for a php script... it's the easiest way
This is my bullet proof version. It iterates over all Tables and String columns in a database, determines primary key(s) and performs updates.
It is intended to run the php-file from command line to get progress information.
<?php
$DBC = new mysqli("localhost", "user", "dbpass", "dbname");
$DBC->set_charset("utf8");
$tables = $DBC->query("SHOW FULL TABLES WHERE Table_type='BASE TABLE'");
while($table = $tables->fetch_array()) {
$table = $table[0];
$columns = $DBC->query("DESCRIBE `{$table}`");
$textFields = array();
$primaryKeys = array();
while($column = $columns->fetch_assoc()) {
// check for char, varchar, text, mediumtext and so on
if ($column["Key"] == "PRI") {
$primaryKeys[] = $column['Field'];
} else if (strpos( $column["Type"], "char") !== false || strpos($column["Type"], "text") !== false ) {
$textFields[] = $column['Field'];
}
}
if (!count($primaryKeys)) {
echo "Cannot convert table without primary key: '$table'\n";
continue;
}
foreach ($textFields as $textField) {
$sql = "SELECT `".implode("`,`", $primaryKeys)."`,`$textField` from `$table` WHERE `$textField` like '%&%'";
$candidates = $DBC->query($sql);
$tmp = $DBC->query("SELECT FOUND_ROWS()");
$rowCount = $tmp->fetch_array()[0];
$tmp->free();
echo "Updating $rowCount in $table.$textField\n";
$count=0;
while($candidate = $candidates->fetch_assoc()) {
$oldValue = $candidate[$textField];
$newValue = html_entity_decode($candidate[$textField], ENT_QUOTES | ENT_XML1, 'UTF-8');
if ($oldValue != $newValue) {
$sql = "UPDATE `$table` SET `$textField` = '"
. $DBC->real_escape_string($newValue)
. "' WHERE ";
foreach ($primaryKeys as $pk) {
$sql .= "`$pk` = '" . $DBC->real_escape_string($candidate[$pk]) . "' AND ";
}
$sql .= "1";
$DBC->query($sql);
}
$count++;
echo "$count / $rowCount\r";
}
}
}
?>
cheers
Roland
It's a bit kludgy but I think the mass update is the only way to go...
$Query = "SELECT row_id, html_entitied_column FROM table";
$result = mysql_query($Query, $connection);
while($row = mysql_fetch_array($result)){
$updatedValue = html_entity_decode($row['html_entitied_column']);
$Query = "UPDATE table SET html_entitied_column = '" . $updatedValue . "' ";
$Query .= "WHERE row_id = " . $row['row_id'];
mysql_query($Query, $connection);
}
This is simplified, no error handling etc.
Not sure what the processing time would be on millions of rows so you might need to break it up into chunks to avoid script timeouts.
I had the exact same problem. Since I had multiple clients running the application in production, I wanted to avoid running a PHP script to clean the database for every one of them.
I came up with a solution that is far from perfect, but does the job painlessly.
Track all the spots in your code where you use htmlentities() before inserting data, and remove that.
Change your "display data as HTML" method to something like this :
return html_entity_decode(htmlentities($chaine, ENT_NOQUOTES), ENT_NOQUOTES);
The undo-redo process is kind of ridiculous, but it does the job. And your database will slowly clean itself everytime users update the incorrect data.