I'm trying to insert multiple rows into a truncated table and while the process works it takes a long time to finish. With around ~800 rows it's already taking me 60+ seconds to finish, which is already problematic considering I have to do the same process for a table with ~70k rows (I did the math and it would have took around 1 hour and 45 mins to finish). Here is the code:
set_time_limit(5000); //bypass 30s limit
$serverName = "<servername>";
$connectionInfo_mssql = array("Database"=>"<dbname>", "CharacterSet"=>"UTF-8");
try
{
$conn_mssql = new PDO("sqlsrv:Server=$serverName;Database=<dbname>");
$conn_mssql->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$conn_mysql = new PDO("mysql:host=localhost;dbname=<dbname>", "<user>", "<pass>");
$conn_mysql->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
//SELECT FROM SQL SERVER DB
$mssql_array = array();
$mssql_query = $conn_mssql->prepare("SELECT * FROM Awards");
$mssql_query->execute();
while($row = $mssql_query->fetch(PDO::FETCH_BOTH))
{
$mssql_array[] = array('id' => $row['id'],
'IDNO' => $row["IDNO"],
'title' => $row['title'],
'place' => $row['place'],
'awarded' => $row['awarded'],
'datee' => $row['datee']);
}
//empties out the MySQL Table
$mysql_query = $conn_mysql->prepare("TRUNCATE TABLE Awards");
$mysql_query->execute();
if($mysql_query)
{
echo "Truncate Success"."<br>";
}
else
{
echo "Truncate Error";
}
//inserts values from SQL Server DB to the empty MySQL Table
$sql = $conn_mysql->prepare("INSERT INTO Awards VALUES (:id, :IDNO, :title, :place, :awarded, :datee)");
foreach($mssql_array as $key => $value)
{
$params_insert = array(':id' => $value['id'],
':IDNO' => $value['IDNO'],
':title' => $value['title'],
':place' => $value['place'],
':awarded' => $value['awarded'],
':datee' => $value['datee']);
$sql->execute($params_insert);
}
echo 'Table Awards from MS SQL DB and table Awards from MySQL DB are now synced!'."<br>";
echo "<a href='table_updater.php'>Go back to updater</a>";
}
catch(PDOException $e)
{
echo "Error: " . $e->getMessage();
}
?>
I searched for answers from past questions about this particular problem but almost all of them deals with variables with fixed values. I have no idea how to apply that to my code considering I'm using an array to store all the values from the main table. What should I change/add to my code to make it faster?
in this example.. how can i simplified and fasten up the update and insert of data in the database?
//if count($arr_list['sample_element1'] is 500+++ or more records
$i =0;
while ($i < count($arr_list['sample_element1'])) {
$update_db= array('column1' => $arr_list['sample_element1'][$i],'column2' => $arr_list['sample_element2'][$i]);
$this->db->update('sample_table',$update_db);
$i++;
}
if the record to be save is over 1000 data...it will take about sometime to finish the process.. thank in advance.. =)
use batch_update
$this->db->update(table_name,array,where_key_word);
here table_name = your table name
and array = an array containing multiple associative array
and where_key_word = the column which should be used in where condition
also read this question thread for more information
batch updating thousands of element together may create performance issue.Better split them up and update by smaller batch.Also use transactions so that if any of the update fails the whole update procedure can be rolled back
Taking #AL-zami's answer you're looking for this
$data = [];
$this->db->trans_start();
foreach($arr_list as $item)
{
$data[] = [
'column1' => $item['element1'],
'column2' => $item['element2'],
'column3' => $item['element3']
];
}
$this->db->update_batch('mytable', $data, 'column1');
$this->db->trans_complete();
Where 'column1' is the column used for the "WHERE" condition.
I found on the PHP documentation the function "array_multisort" that is meant to sort an array "of columns". They provide one example where the user has an array of rows and then the user has to transform this array of rows into an array of columns. In my case the array is already set by columns, such as:
tablearray
{
['Employee_ID'] = {0 => row1, 1 => row2, 2 => row3, 3 => row4}
['First_Name'] = {0 => row1, 1 => row2, 2 => row3, 3 => row4}
['LastName'] = {0 => row1, 1 => row2, 2 => row3, 3 =>row4}
}
I want to sort by Employee_ID and I need all the other columns to follow the same order. I tried:
array_multisort($tablearray['Employee_ID'], SORT_ASC);
But it only sorts the first column (which becomes a mess). The array has more than 10 columns and it changes the column names depending on the search (the columns names are its keys).
On PHP's documentation for this function, the example provided shows that the after transforming the rows array into a columns array, we should use the original array as a third parameter to match the keys - I don't have the "original" array to do the match since I didn't transform anything.
Thank you.
Desired output, as suggested by one user:
Original:
array
{
['Employee_ID'] = (1002, 4508, 0002, 1112)
['Business_Unit'] = ('UER', 'ABC', 'XYZ', 'EER')
['LastName'] = ('Smith', 'Vicente', 'Simpson', 'Thompson')
}
Sorted by Employee ID:
array
{
['Employee_ID'] = (0002, 1002, 1112, 4508)
['Business_Unit'] = ('XYZ', 'UER', 'EER', 'ABC')
['LastName'] = ('Simpson','Smith', 'Thompson', 'Vicente')
}
--
My original array is a database query output:
Array
(
[0] => Array
(
[Employee_ID] => 0000
[Supervisor_ID] => 00000
[Location_Descr] => somewhere
[Start_Date] => 06/03/2002
[Service_Date] => 06/03/2002
[Rehire_Date] => 00/00/00
[Business_Unit] => YYYY
[Job_Title] => Manager
[Email] => email#example.com
[Dept_Desc] => bla bla bla
[Employee_Name_LF] => Last, First
[Supervisor_Name_LF] => Last, First
[Term_Date] => 00/00/00
[Preferred_Name] => Someone
[Source] => Sheet2
)
)
There a several more rows.
The main purpose is to show the results as an HTML table and to generate a CSV file. I already made those functions using the modified structure (the first that I posted). I thought it would be easier to deal with that structure... Indeed it was, but not for sorting unfortunately.
The array_multisort documentation (http://php.net/manual/en/function.array-multisort.php) suggests separating each column as an individual array.. However, as you can see I have several columns (and the user can select more or less to be shown before performing the query.. So I can't just list all of them on the statement).
I a willing to change everything just to make the code better to be worked with.
Ugly - would be a lot easier if you formatted the input tables.
$arr = array(
'Employee_ID' => array('1002', '4508', '0002', '1112'),
'Business_Unit' => array('UER', 'ABC', 'XYZ', 'EER'),
'LastName' => array('Smith', 'Vicente', 'Simpson', 'Thompson')
);
$employees = array();
foreach (range(0, sizeof($arr[current(array_keys($arr))]) - 1) as $k) {
$emp = array();
foreach ($arr as $col => $vals) {
$emp[$col] = $arr[$col][$k];
}
$employees[] = $emp;
}
$sort = array();
foreach ($employees as $k => $v) {
$sort[$k] = $v['Employee_ID'];
}
array_multisort($sort, SORT_ASC, $employees);
print_r($employees);
And to put back in the original format:
$arr_sorted = array();
foreach (array_keys($arr) as $col) {
$arr_sorted[$col] = array();
foreach ($employees as $emp) {
$arr_sorted[$col][] = $emp[$col];
}
}
print_r($arr_sorted);
Thank you for posting the extra details in your question, as they did help in understanding the intent of your question.Now, you didn't tell us how that table should look; If you want the employees one per row, or one per column. Which is kind of crucial to know. Normally one would have one employee per line, especially if this is to be exported to CVS. However, I have a suspicion that it's the latter you want. Otherwise you've gone about this in a very overly complicated manner.Point in case: Normal one-per-row layout:
<?php
$db = new PDO();
// Defining the fields we need here, to avoid having too long a string for the query.
$fields = "e.employee_id, e.first_name, e.lastname, u.business_unit, s.email";
// Do the sorting in the database itself. Not only is this faster, but it
// is also a lot easier to sort it exactly as you'd like.
// Note that I don't use prepared statements here, as there is no user-input.
$query = <<<outSQL
SELECT {$Fields} FROM `unit` AS u
INNER JOIN `employee` AS e ON e.employee_id = u.unit_id
INNER JOIN `employee` AS s ON s.employee_id = u.supervisor_id
ORDER BY e.`employee_id`
outSQL;
$data = $db->query($query);
// Creating a printf() template for the output, to make the code easier to maintain.
$rowTemplate = <<<outHTML
<tr>
<td>%1\$d</td>
<td>%2\$s</td>
<td>%3\$s</td>
</tr>
outHTML;
// Generate the table template, using placeholders for where the data will be added..
$tableTemplate = <<<outHTML
<table>
<thead>
<tr>
<th>ID</th>
<th>First name</th>
<th>Last name</th>
</tr>
</thead>
<tbody>
%s
</tbody>
</table>
outHTML;
// Regular table output, one employee per line.
$temp = '';
foreach ($data as $e) {
// hs() is a shortcut function to htmlspecialchars (), to prevent against XSS.
$temp .= sprintf($rowTemplate, $e['employee_id'], hs($e['first_name']), hs($e['lastname']));
}
// Add the rows to the table, so that you can echo the completed product wherever you need.
$employeeTable = sprintf($tableTemplate, $temp);
If you want to do it one per column, it becomes a bit more intricate. Though, still a bit easier than your first attempt. :)
Namely, something like this:
<?php
$db = new PDO();
// Defining the fields we need here, to avoid having too long a string for the query.
$fields = "employee_id, first_name, lastname";
// Do the sorting in the database itself. Not only is this faster, but it
// is also a lot easier to sort it exactly as you'd like.
// Note that I don't use prepared statements here, as there is no user-input.
$data = $db->query("SELECT {$Fields} FROM `employees` ORDER BY `employee_id`");
// We need to know how many columns we'll have. One per employee.
$columns = count ($data);
// Rows have a header in front of each line, and one td tag for each employee.
$rowTemplate = "\t\t<th>%s</th>\n".str_repeat("\t\t\t<td>%s</td>\n", $columns);
// Generate the table template, using placeholders for where the data will be added..
$tableTemplate = <<<outHTML
<table>
<tbody>
%s
</tbody>
</table>
outHTML;
// Reformat the array to give us the data per-column.
$temp = array ();
foreach ($data as $field => $e) {
// Since we've already sorted the data in the database we don't need to do any further sorting here.
// Also note that I'm doing the escaping here, seeing as this array will only be used for output.
$temp['Employee ID'][] = intval($e['employee_id']);
$temp['First name'][] = hs($e['first_name']);
$temp['Last name'][] = hs($e['lastname']);
}
// Now we do the same as in the above example.
$rows = '';
foreach ($temp as $label => $l) {
// We have the label as the first template variable to be added, so put it as the first element.
array_unshift($l, $label);
// Add the current row of items to the output, using the previously established template.
$rows = vprintf($rowTemplate, $l);
}
// Add the rows to the table, so that you can echo the completed product wherever you need.
$employeeTable = sprintf($tableTemplate, $temp);
PS: Haven't tested the code, but it should work.
I ran into his problem and after much angst found a really nice solution in the notes on the php manual page - I now have the following function which i use whenever I need to solve this type of problem.
function fnArrayOrderby(){
//function to sort a database type array of rows by the values in one or more column
//source http://php.net/manual/en/function.array-multisort.php - user notes
//example of use -> $sorted = fnArrayOrderby($data, 'volume', SORT_DESC, 'edition', SORT_ASC);
$args = func_get_args(); //Gets an array of the function's argument list (which can vary in length)
//echo "sorting ".$args[0]."<br>";
if (!isset($args[0])) { return;}
$data = array_shift($args); //Shift an element off the beginning of array
foreach ($args as $n => $field) {
if (is_string($field)) {
$tmp = array();
foreach ($data as $key => $row)
$tmp[$key] = $row[$field];
$args[$n] = $tmp;
}
}
$args[] = &$data;
call_user_func_array('array_multisort', $args);
return array_pop($args);
}
i'm making a while loop in php and it all goes well but the problem is that
I don't only want to get the id of the user but also some other stuff that is inside another table, so when I go ahead and make a query inside this while loop and select everything from that second table (where the id is equal to the id of the result from the first query), it only returns 1 result...
So this is the code that I currently have:
public function getFriends($id)
{
global $params;
$get = $this->db->select("{$this->DB['data']['friends']['tbl']}", "*",
array(
"{$this->DB['data']['friends']['one']}" => $id
)
);
if($get)
{
while($key = $get->fetch())
{
$query = $this->db->query("SELECT * FROM {$this->DB['data']['users']['tbl']}
WHERE {$this->DB['data']['users']['id']} = :id",
array(
"id" => $key->{$this->DB['data']['friends']['two']}
)
);
while($row = $query->fetch())
{
$params["user_friends"][] = [
"id" => $key->{$this->DB['data']['friends']['two']},
"name" => $row->{$this->DB['data']['users']['username']},
"look" => $row->{$this->DB['data']['users']['figure']}
];
}
}
}
else
{
$params["update_error"] = $params["lang_no_friends"];
}
}
Thanks in advance!
Please help me out!
In the absence of answers, I don't know what db framework you are using behind the scenese...PDO, mysqli_, or (hopefully not) mysql_. But, in any case, the problem might be that your second query stops the first from continuing. I would use PDO->fetchAll() to get them all...but you say you can't do that...so, looping the first and loading those results into an array is the first thing I would do to see if this is the problem:
public function getFriends($id)
{
global $params;
$get = $this->db->select("{$this->DB['data']['friends']['tbl']}", "*",
array(
"{$this->DB['data']['friends']['one']}" => $id
)
);
$firstResults = array();
if( $get ) {
while( $key = $get->fetch() ) {
$firstResults[] = $key;
}
}
else
{
$params["update_error"] = $params["lang_no_friends"];
}
foreach( $firstResults AS $key )
{
$query = $this->db->query("SELECT * FROM {$this->DB['data']['users']['tbl']}
WHERE {$this->DB['data']['users']['id']} = :id",
array(
"id" => $key->{$this->DB['data']['friends']['two']}
)
);
while($row = $query->fetch())
{
$params["user_friends"][] = [
"id" => $key->{$this->DB['data']['friends']['two']},
"name" => $row->{$this->DB['data']['users']['username']},
"look" => $row->{$this->DB['data']['users']['figure']}
];
}
}
}
If this doesn't work, then we need more data...e.g. what is the query generated? When you run it manually does it return more than one result? If you get rid of the inner-query, does this fix it? etc.
The first step when diagnosing PHP and Mysql issues is to add lines to your code that tell you what each line is doing (declare each time a loop is entered; when each mysql query is run, spit out the query string) so you can narrow down where the problem is. Often this makes you feel stupid in retrospect: "Duh, this query didn't return anything because I formatted the record ID wrong" and so forth.
The code snippet you've provided above isn't super helpful to me. I'm a troubleshooter (not a parser) so I need diagnostic data (not straight code) to be of any more help than this.
I'm posting data from a web page to a php page with Ajax. It's working fine.
I get my data from the $_POST, loop through the values and create an array called checklist.
if($_POST != ''):
$dataset = $_POST['data'];
$checklist = array();
$eventid='';
foreach ($dataset as $i => $row)
{
$uid = $row['box-id'];
$state = $row['box-state'] ;
$eventid = $row['e_id'];
$checklist[] = array('uid'=>$uid,
'state'=> $state);
}
Checklist has two fields, a uid and a state.
I then run a script that generates another array, called $updates. It loops through a different set of objects and outputs the data to populate the variables for $updates. The structure of $updates is as such.
$updates[] = array('uid'=>$uid,
'state'=> $state,
'class' => $class,
'container' => $button_cont,
'closer' => $button_closer);
What I would like to do is to compare $updates with $checklist.
I'd like to know the most efficient way to match the records by the uid and compare the state. If the state matches, I'd like to do nothing.
I've read a few of the articles on looping and search, but I'm thinking I've been looking at this for too long because it's Greek to me. Thanks for assistance.
save the checklist like -
$checklist[$uid] = $state;
same for updates
$updates[$uid] = array('state'=> $state,
'class' => $class,
'container' => $button_cont,
'closer' => $button_closer);
then start the loop
foreach ($updates as $key => $update) {
if ($update['state'] == $checklist[$key]) {
//your action
}//compare the values
}
$key will be the uid.hope it will help you