I have an array that can hold up to several thousands of items.(usually around 5000 items).
I need to split this array into hundreds and process them and then continue with the rest of items.
So far i handle the whole array which is slow.
My code is
foreach($result as $p){
$sqlQuery = mysql_query("INSERT INTO message_details(contact, message_id)VALUES('$p', $message_last_id)");
$last_id = mysql_insert_id();
$xmlString .= "<gsm messageId=\"$last_id\">".$p."</gsm>";
$cnt++;
}
How can i process the items in the array in hundreds? Eg. 100, then 200, then 300 etc
Best Regards,
Nicos
Maybe you might try it that way:
First select the last ID of message_details.
$sqlQuery = mysql_query("SELECT %last_id_col_name% FROM message_details ORDER BY %last_id_col_name% DESC LIMIT 1".$sInserts);
then:
$sInserts = '';
foreach($result as $p){
$sInserts .= "('{$p}', {$message_last_id}),";
}
//Remove last "," from Insert-String
$sInserts = substr($sInserts,0,-1);
//Insert all with one query
$sqlQuery = mysql_query("INSERT INTO message_details(contact, message_id)VALUES".$sInserts);
then select all entries from that table, where id is greater than the one you've selected first
and write to
$xmlString .= '<gsm messageId="'.$last_id.'">'.$p.'</gsm>';
If you are doing it that way, you only need 3 DB-Queries, instead of thousands.
foreach($result as $p){
$sqlQuery = mysql_query("INSERT INTO message_details(contact, message_id)VALUES('$p', $message_last_id)");
$last_id = mysql_insert_id();
$xmlString .= "<gsm messageId=\"$last_id\">".$p."</gsm>";
$cnt++;
if ($cnt % 1000 == 0) { usleep(100) }
}
You can use usleep or sleep depending on what you want. I've already used this. You gain performance. Try it.
If you have to do it in the code the you can use the php function array_chunk to chunk your array into arrays of 100 elements each. Here's the docs on array_chunk: link. But as pointed out this is unlikely to be the bottle neck.
Related
I have an array $table and it contains 6000 items.
When I want to convert this array to json and store it into a file, my memory crashes.
So I had the idea to break the array into chunks of parts with 500 items:
$table = array_chunk($table, 500);
$table_json = $serializer->serialize($table[0], 'json', $context);
$myfile = fopen($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", "w") or die("Unable to open file!");
file_put_contents($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", $table_json);
This runs fast now, but of course now only 500 items are stored. Is there a way to add the other parts of the $table array without memory crash?
You could do something like this as you mentioned you know how to use array_chunk();
Let's look into simplifying the process with a built-in PHP function called array_chunk();
We'll be using HTML tables for design which isn't recommended. The task is better accomplished with CSS, you can follow same way without HTML and CSS.
Our table :
id datePosted firstName lastName pictureName anotherColumn
1 2013-07-01 John Smith SmithJohn.jpg anotherValue
2 2013-05-06 Elroy Johnson JohnsonElroy.jpg anotherValue
3 2013-06-18 Jake Bible BibleJake.jpg anotherValue
4 2013-07-17 Steve Stevenson StevensonSteve.jpg anotherValue
5 2013-04-08 Bill Smith SmithBill2.jpg anotherValue
Building HTML Tables
PDO query is used to grab the database information with prepared statements, Note that the loop only generates the code to display the columns.
The tests to detect where the rows begin and end are unnecessary.
//INITIALIZE VARIABLES
$colsToDisplay = 3;
$htmlOutput = array();
//GET PICTURE LIST
$sql = "SELECT datePosted, firstName, lastName, pictureName FROM pictureList ORDER BY datePosted DESC";
$stmt = $pdo->prepare($sql);
$stmt->execute();
while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
$htmlOutput[] = "<td><img src='images/{$row['pictureName']}' alt='' /><br />{$row['firstName']} {$row['lastName']}</td>";
}
Once the loop is done, the array containing the column information can be broken into groups of three... or whatever value was assigned to $colsToDisplay, What we did here ? we tooks 3 columns from table, So divide table in two parts.
while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
//...
}
//BREAK THE COLUMNS INTO GROUPS
$htmlOutput = array_chunk($htmlOutput, $colsToDisplay);
All that's left is to display the table information. Note that array_chunk() creates a multi-dimensional array. The foreach loop is used to process the groups of columns. Each group is assigned to $currRow which contains an array of columns for the current row. The implode() function is used to quickly display the columns as a string.
Lets continue :
//BREAK THE COLUMNS INTO GROUPS
$htmlOutput = array_chunk($htmlOutput, $colsToDisplay);
//DISPLAY TABLE
print '<table>';
foreach($htmlOutput as $currRow) {
print '<tr>' . implode('', $currRow) . '</tr>';
}
print '</table>';
Checking For Missing Column Tags
One thing you may have noticed is the code to add missing columns was left out.
In other words, this example results in an HTML table where the last row only has two columns.
Apparently, the missing columns aren't needed according to the W3C Markup Validation Service… so they weren't included. However, they can be added by running the following code right before array_chunk() is called.
$colsDifference = count($htmlOutput) % $colsToDisplay;
if($colsDifference) {
while($colsDifference < $colsToDisplay) {
$htmlOutput[] = '<td></td>';
$colsDifference++;
}
}
Final Code :
//INITIALIZE VARIABLES
$colsToDisplay = 3;
$htmlOutput = array();
//GET PICTURE LIST
$sql = "SELECT datePosted, firstName, lastName, pictureName FROM pictureList ORDER BY datePosted DESC";
$stmt = $pdo->prepare($sql);
$stmt->execute();
while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
$htmlOutput[] = "<td><img src='images/{$row['pictureName']}' alt='' /><br />{$row['firstName']} {$row['lastName']}</td>";
}
//OPTIONAL CODE
//IF NEEDED, ADD MISSING COLUMNS
$colsDifference = count($htmlOutput) % $colsToDisplay;
if($colsDifference) {
while($colsDifference < $colsToDisplay) {
$htmlOutput[] = '<td></td>';
$colsDifference++;
}
}
//END: OPTIONAL CODE
//BREAK THE COLUMNS INTO GROUPS
$htmlOutput = array_chunk($htmlOutput, $colsToDisplay);
//DISPLAY TABLE print '<table>';
foreach($htmlOutput as $currRow) {
print '<tr>' . implode('', $currRow) . '</tr>';
}
print '</table>';
Idea Behind This Tutorial :
You dont need to create a file and write arrays into that file to display in datatable.
If you still wants that you can (divide table in two parts) to create two arrays and than write into file, and than use array_push(); or array_merge(); to get both arrays into one.
I might have bad explanition please forgive for mistakes, Hope this help you in your coding life :)
Sticking to your solution, you can append the other chunks to your exisiting file. This works by setting the flag FILE_APPEND, e.g.:
$table_json = $serializer->serialize($table[1], 'json', $context);
$myfile = fopen($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", "w") or die("Unable to
open file!");
file_put_contents($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", $table_json, FILE_APPEND | LOCK_EX);
(LOCK_EX flag to prevent anyone else writing to the file at the same time)
So I have 2 tables. One table has all the actual campaign_id information.
The second table has the impression/statistic information on the campaign_id's
I have a table on the page (i use ajax, but that's besides the point). I want to "sort" a column, but all the rows are generated by the campaign_id table, I run all of the statistics for every campaign first, and then link them up to each row. then after all the info/data is up, it then sorts all of it. this uses a MASSIVE amount of memory and resources. Is this efficient at all? is there a better solution to sorting huge amounts of data?
// I have to increase memory because the sorting takes a lot of resource
ini_set('memory_limit','1028M');
// the column I want to sort
$sortcolumn = $this->input->post('sortcolumn');
// direction of the sort ASC/DESC
$sortby = $this->input->post('sortby');
// millions of impression data that is linked with campaign_id
$cdata= array();
$s = "SELECT report_campaign_id,";
$s .= "SUM(report_imps) AS imps ";
$s .= "FROM Campaign_Impressions";
$s .= "GROUP BY report_campaign_id ";
$r = $this->Qcache->result($s,0,'campaignsql');
foreach($r as $c) {
$cdata[$c->report_campaign_id]['imps'] = ($c->imps) ? $c->imps : 0;
}
// 500,000+ thousand campaigns
// I draw my table from these campaigns
$rows = array();
$s = "SELECT * FROM Campaigns ";
$r = $this->db->query($s)->result();
foreach($r as $c)
{
$row= array();
$row['campaign_id'] = $c->campaign_id;
// other campaign info here...
// campaign statistics here...
$row['campaign_imps'] = $cdata[$c->campaign_id]['imps'];
// table row
$rows[] = $row;
}
// prepare the columns i want to sort
$sortc = array();
foreach($rows as $sortarray) {
if (!isset($sortarray[ $sortcolumn ])) continue;
$sortc[] = str_replace(array('$',','),'',$sortarray[ $sortcolumn ]);
}
// sort columns and direction
array_multisort($sortc,(($sortby==='asc')?SORT_ASC:SORT_DESC),SORT_NATURAL,$rows);
As you can see, the "campaign_impressions" table is running data on "every" campaign, and doesn't seem so efficient, but more effective instead of running a query per row to know the data.
(I dont display all the campaigns, but I need to run every one of them to know the sorting of all)
You should let MySQL do the job my using order by
if this still takes a lot of time on the MySQL side consider using sorted indexes on the columns
Here First of all I fetched all tags from database. (tags are string like marketting,jobs etc).
$POS is array of POS(parts of speech words). For each words in this array, for given tag, I am calculating probability using writtten equation.
As it is appearing, two nested loops are there. To make decision, I need highest value from array $prob and tag belonging to that value.
Here in this code I will get all values in array $prob and from that I can get highest value, but how can I persist tag value too? is there any other data strcture in PHP to manage this short of scenario?
$selectTag = mysqli_query($con,"SELECT tag from koove_tag");
$prob = array();
$i=0;
while ($row1 = #mysqli_fetch_array($selectTag))
{
foreach($POS as $p)
{
//Calculate total pos for given tag
$selectPOS = mysqli_query($con,"SELECT * from koove_post where tag = '".$row1[tag]."'");
while ($row2 = #mysqli_fetch_array($selectTag))
{
$totalPOSs.=$row2[pos].",";
}
$totalPOS_count = str_word_count($totalPOSs);
//calculate how many times particular 'pos' appears for given tag
$selectPOS = mysqli_query($con,"SELECT * from koove_post where tag = '".$row1[tag]."'");
while ($row3 = #mysqli_fetch_array($selectTag))
{
$POSs.=$row3[pos].",";
}
$pos_Count = echo substr_count($string, $p);
//calculate distinct POS in all POSs for all tags
$selectPOS = mysqli_query($con,"SELECT pos from koove_post");
while ($row4 = #mysqli_fetch_array($selectTag))
{
$POSs.=$row4[pos].",";
}
$distinct_pos_Count = echo substr_count($string, $p);
$prob[$i] = ($pos_Count + 1)/ ($totalPOS_count + $distinct_pos_Count);
$i++;
}
}
There are large numbers(thousands) of posts, if is there any better approach to faster the processing that also welcome.
Certainly. Use the stdClass() object to hold the values that you need. Then write method to sort them. For example:
$prbo[$i] = new stdClass();
$prob[$i]->value = ($pos_Count + 1)/ ($totalPOS_count + $distinct_pos_Count);
$prob[$i]->tag = $row1[tag];
After you build that array, then write a custom sort function.
http://www.php.net/manual/en/array.sorting.php
1.You can use join query that will eliminate one of cycles.
2.You can get highest value with end(array) , or custom sort function.
I'm trying to make a simple alphabetical list to order items in my database. The thing I can't figure out is how to actually list it.
I would like it to be the same format as you have on miniclip.com
Here's an image
I looked around, but couldnt find an answer really.
(I would like it to finish even at the end of each vertical column, except the last one for sure)
Any help would be welcome!
In MySQL:
SELECT * FROM table ORDER BY name ASC
In PHP:
$fruits = array("lemon", "orange", "banana", "apple");
sort($fruits);
foreach ($fruits as $key => $val) {
echo "fruits[" . $key . "] = " . $val . "\n";
}
fruits[0] = apple
fruits[1] = banana
fruits[2] = lemon
fruits[3] = orange
Assuming that your result set already is sorted by using the ORDER BY clause, to group the results by their first character you just need to remember the first character of the previous entry and print out the first character of the current entry if they are different. So:
$prevLabel = null;
while ($row = mysql_fetch_assoc($result)) {
$currLabel = strtoupper(substr($row['name'], 0, 1));
if ($currLabel !== $prevLabel) {
echo $currLabel;
$prevLabel = $currLabel;
}
echo $row['name'];
}
This will print the first character as a label for each group that’s members have the same first character.
He doesn't seem to have an issue with the storting, but doing the column format and headers for each new letter.
Suppose $arr contains your alphabetically sorted list with numeric keys. each element has indexes 'name' and 'link'. This should be pretty safe assumption for data from a SQL query.
$firstLetter = -1;
$desiredColumns = 4; //you can change this!
$columnCount = (count($arr)+27)/$desiredColumns+1;
echo "<table><tr><td>";
foreach($arr as $key => $cur)
{
if ($key != 0 && $key % desiredColumns == 0) echo "</td><td>";
if ($cur['name'][0] !== $firstLetter)
{
echo "<strong>$firstLetter</strong> <br />"; $firstLetter = $cur['name'][0];
}
echo "".$cur['name']."<br />";
}
echo "</td><tr></table>";
You'll have to treat numbers as a special case, but this is the idea. If you are using a template engine there are obviously better ways of doing this, but I figure you would have mentioned that. This is a rough sketch, making pretty HTML isn't my thing.
--Query-- get table into $arr. I can't see your tables obviously, Im making assumptions if names nad stuff so you'll need to verify or change them
$sql = "SELECT * FROM table T ORDER BY name";
$conn = //you should have this
$res = mysql_query($sql, $conn);
$arr = array();
while($row = mysql_fetch_assc($res)
$arr[] = $row;
// start above code here. This isn't safe for empty query responses or other error but it works
I presume you're using MySQL (or another SQL) database, in which case you should simply retrieve the data in the required order using a SORT BY clause on the lookup SELECT. (Sorting this PHP is trivial via the sort function, but it makes sense to get the database to do this - that's pretty much what it's for.)
In terms of balancing the output of each of the columns, you could get a COUNT of the required rows in your database (or simply use the count of the resulting PHP array of data) and use this to ensure that the output is balanced.
As a final thought, if this is going to be output on a per-page basis, I'd highly recommend generating it into a static file when the structure changes and simply including this static file as a part of the output - generating this on the fly is needlessly resource inefficient.
The mysql option mentioned above is definitely the best bet. If the data comes out of the DM in order, that's the simplest way to go.
Your next option might be to look at the
asort and ksort functions in PHP to find the exact one you're looking for.
http://www.php.net/manual/en/array.sorting.php
How are you pulling the data?
<?php
$result = mysql_query("SELECT titles FROM gamelist ORDER BY title ASC");
while ($row = mysql_fetch_assoc($result)) {
echo "{$result['title']}<br/>";
}
?>
There are two ways to do it.
You could use your database and use the 'order' clause to pull them by a specific field alphabetically.
You could also use either a key sort or value sort on a PHP array.
The PHP functions are sort($array) and ksort($array).
http://php.net/manual/en/function.sort.php
http://php.net/manual/en/function.ksort.php
<?php
$list = $your_list_array_from_database
//if you need info on how to do this, just let me know
sort($list);
foreach($list as $item) {
echo $item;
}
?>
I found this post and had the same problem. I used the code below to output a list by category name with a header equal to the first letter. In my database table (category) I have name and category_letter. So, name = football and category_list = 'F'.
<section>
<?php
try {
$cats_sql = $dbo->prepare("SELECT name, category_list, FROM category WHERE category_list REGEXP '^[A-Z#]' GROUP BY category_list ASC");
$cats_sql->execute();
$results_cats = $cats_sql->fetchAll();
} catch(PDOException $e) {
include('basehttp/error');
}
$array_cats = $results_cats;
if(is_array($array_cats)) {
foreach($array_cats as $row_cats) {
$cat_var = $row_cats[category_list]; // Each Category list title
?>
<aside>
<h1><a name=""><? echo $cat_var ?></a></h1>
<?php
try {
$search_sql = $dbo->prepare("SELECT name, category_list FROM category WHERE category_list=:cat_var ORDER BY name ASC"); // Pulling a list of names for the category list
$search_sql->bindParam(":cat_var",$cat_var,PDO::PARAM_STR);
$search_sql->execute();
$results_search = $search_sql->fetchAll();
} catch(PDOException $e) {
include('basehttp/error');
}
$array_search = $results_search;
if(is_array($array_search)) { // Output list of names which match category
foreach($array_search as $row_search) {
?>
<h2><?php echo $row_search[name]; ?></h2>
<br class="clear">
<?php
}
}
?>
</aside>
<br class="clear">
<?php
}
}
?>
</section>
Its actually Simple....I did similar thing for my project once. I had to pull out all music albums name and categorize them in alphabetical order.
In my table, "album_name" is the column where names are stored.
$sql= "select * from album_table order by album_name ASC";
$temp_char= ""; // temporary variable, initially blank;
using while loop, iterate through records;
while($row= $rs->fetch_assoc())
{
$album_name= $row['album_name'];
$first_char_of_albm= $album_name[0]; // this will store first alphabet;
$first_char_of_albm= strtoupper($first_char_of_albm); // make uppercase or lower as per your needs
if($temp_char!=$first_char_of_albm)
{
echo $first_char_of_albm;
$temp_char= $first_char_of_albm; // update $temp_char variable
}
}
That's it....
I am posting my answer to this old question for 3 reasons:
You don't always get to write your queries to MySQL or another DBMS, as with a web service / API. None of the other answers address PHP sorting without query manipulation, while also addressing the vertical alphabetical sort
Sometimes you have to deal with associative arrays, and only a couple other answers deal with assoc. arrays. BTW, my answer will work for both associative and indexed arrays.
I didn't want an overly complex solution.
Actually, the solution I came up with was pretty simple--use multiple tags with style="float:left", inside of a giant table. While I was sceptical that having multiple tbody tags in a single table would pass HTML validation, it in fact did pass without errors.
Some things to note:
$numCols is your desired number of columns.
Since we are floating items, you may need to set the width and min-width of parent elements and/or add some <br style="clear: both" />, based on your situation.
for alternative sorting methods, see http://php.net/manual/en/array.sorting.php
Here's my full answer:
function sortVertically( $data = array() )
{
/* PREPARE data for printing */
ksort( $data ); // Sort array by key.
$numCols = 4; // Desired number of columns
$numCells = is_array($data) ? count($data) : 1 ;
$numRows = ceil($numCells / $numCols);
$extraCells = $numCells % $numCols; // Store num of tbody's with extra cell
$i = 0; // iterator
$cCell = 0; // num of Cells printed
$output = NULL; // initialize
/* START table printing */
$output .= '<div>';
$output .= '<table>';
foreach( $data as $key => $value )
{
if( $i % $numRows === 0 ) // Start a new tbody
{
if( $i !== 0 ) // Close prev tbody
{
$extraCells--;
if ($extraCells === 0 )
{
$numRows--; // No more tbody's with an extra cell
$extraCells--; // Avoid re-reducing numRows
}
$output .= '</tbody>';
}
$output .= '<tbody style="float: left;">';
$i = 0; // Reset iterator to 0
}
$output .= '<tr>';
$output .= '<th>'.$key.'</th>';
$output .= '<td>'.$value.'</td>';
$output .= '</tr>';
$cCell++; // increase cells printed count
if($cCell == $numCells){ // last cell, close tbody
$output .= '</tbody>';
}
$i++;
}
$output .= '</table>';
$output .= '</div>';
return $output;
}
I hope that this code will be useful to you all.
I'm trying to generate a tree structure from a table in a database. The table is stored flat, with each record either having a parent_id or 0. The ultimate goal is to have a select box generated, and an array of nodes.
The code I have so far is :
function init($table, $parent_id = 0)
{
$sql = "SELECT id, {$this->parent_id_field}, {$this->name_field} FROM $table WHERE {$this->parent_id_field}=$parent_id ORDER BY display_order";
$result = mysql_query($sql);
$this->get_tree($result, 0);
print_r($this->nodes);
print_r($this->select);
exit;
}
function get_tree($query, $depth = 0, $parent_obj = null)
{
while($row = mysql_fetch_object($query))
{
/* Get node */
$this->nodes[$row->parent_category_id][$row->id] = $row;
/* Get select item */
$text = "";
if($row->parent_category_id != 0) {
$text .= " ";
}
$text .= "$row->name";
$this->select[$row->id] = $text;
echo "$depth $text\n";
$sql = "SELECT id, parent_category_id, name FROM product_categories WHERE parent_category_id=".$row->id." ORDER BY display_order";
$nextQuery = mysql_query($sql);
$rows = mysql_num_rows($nextQuery);
if($rows > 0) {
$this->get_tree($nextQuery, ++$depth, $row);
}
}
}
It's almost working, but not quite. Can anybody help me finish it off?
You almost certainly, should not continue down your current path. The recursive method you are trying to use will almost certainly kill your performance if your tree ever gets even slightly larger. You probably should be looking at a nested set structure instead of an adjacency list if you plan on reading the tree frequently.
With a nested set, you can easily retrieve the entire tree nested properly with a single query.
Please see these questions for a a discussion of trees.
Is it possible to query a tree structure table in MySQL in a single query, to any depth?
Implementing a hierarchical data structure in a database
What is the most efficient/elegant way to parse a flat table into a tree?
$this->nodes[$row->parent_category_id][$row->id] = $row;
This line is destroying your ORDER BY display_order. Change it to
$this->nodes[$row->parent_category_id][] = $row;
My next issue is the $row->parent_category_id part of that. Shouldn't it just be $row->parent_id?
EDIT: Oh, I didn't read your source closely enough. Get rid of the WHERE clause. Read the whole table at once. You need to post process the tree a second time. First you read the database into a list of arrays. Then you process the array recursively to do your output.
Your array should look like this:
Array(0 => Array(1 => $obj, 5 => $obj),
1 => Array(2 => $obj),
2 => Array(3 => $obj, 4 => $obj),
5 => Array(6 => $obj) );
function display_tree() {
// all the stuff above
output_tree($this->nodes[0], 0); // pass all the parent_id = 0 arrays.
}
function output_tree($nodes, $depth = 0) {
foreach($nodes as $k => $v) {
echo str_repeat(' ', $depth*2) . $v->print_me();
// print my sub trees
output_tree($this->nodes[$k], $depth + 1);
}
}
output:
object 1
object 2
object 3
object 4
object 5
object 6
I think it's this line here:
if($row->parent_category_id != 0) {
$text .= " ";
}
should be:
while ($depth-- > 0) {
$text .= " ";
}
You are only indenting it once, not the number of times it should be indented.
And this line:
$this->get_tree($nextQuery, ++$depth, $row);
should be:
$this->get_tree($nextQuery, $depth + 1, $row);
Note that you should probably follow the advice in the other answer though, and grab the entire table at once, and then process it at once, because in general you want to minimize round-trips to the database (there are a few use cases where the way you are doing it is more optimal, such as if you have a very large tree, and are selecting a small portion of it, but I doubt that is the case here)