I've a job to do some migration on wordpress sites, although I don't think wordpress has anything to do with my problem.
So I'm getting a set of data from a table (id, value) which is serialized.
I want to put the serialized data into arrays, according to the function. The algorithm doing everything fine for the first 86 records.
If I run the algorithm on just 86 records I get all the right data instantly. But when I run it 87 or more times its just goes into loading. Doesn't matter how long the max execution time limit is it's still gonna reach it apparently.
I checked the 87th record if theres something odd with it, but nothing i can see. Is there any idea what could couse this?
Here's the code:
global $wpdb;
$houses_built_years = $wpdb->get_results('SELECT post_id, meta_value FROM bp100_postmeta WHERE meta_key = "epites_eve"');
foreach ($houses_built_years as $house_built_years) {
if ($house_built_years->meta_value) {
$years = unserialize($house_built_years->meta_value);
} else {
continue;
}
$build = array(array());
$series = 0; $year = 0;
while (count($years)) {
if ($build[$series][$year] && (min($years) - $build[$series][$year] == 1)) {
$year++;
$build[$series][$year] = min($years);
unset($years[array_search(min($years), $years)]);
} else if ($build[$series][$year]) {
$year = 0;
$series++;
$build[$series][$year] = min($years);
unset($years[array_search(min($years), $years)]);
} else {
$build[$series][$year] = min($years);
unset($years[array_search(min($years), $years)]);
}
}
$repeater = array();
foreach ($build as $series) {
if (count($series) > 1) {
$repeater[] = $series[0] . ' - ' . $series[count($series) - 1];
} else {
$repeater[] = $series[0];
}
}
}
Here's some example of records (algoithm breaking at id = 30517):
30279
1201
epites_eve
a:1:{i:0;s:4:"1898";}
30338
1202
epites_eve
a:1:{i:0;s:4:"1891";}
30397
1203
epites_eve
a:1:{i:0;s:4:"1894";}
30517
1205
epites_eve
a:1:{i:0;s:4:"1897";}
30577
1206
epites_eve
a:1:{i:0;s:4:"1891";}
Related
I need to update a table with more then 12000 row using php Codeigniter and a txt file.. reading the file and the foreach loop are fine but when updating line by line it takes like 30 mins, I guess the problem is I'm searching by name because I have no id in the txt file...
Here is my code:
controller:
$fn = fopen($this->upload->data('full_path'),"r");
$update = true;
while(! feof($fn) && $update) {
$pieces = explode("|", fgets($fn));
if(sizeof($pieces) == 9 && is_numeric(trim($pieces[1]))) {
$update = $this->model_products->update3s($pieces);
}
}
fclose($fn);
Model:
public function update3s($product) {
if ($product) {
$product[2] = trim(str_replace("'","''",$product[2]));
$product[1] = trim($product[1]);
$product[6] = trim($product[6]);
$product[3] = trim($product[3]);
$sql = "UPDATE products set qty = $product[3], price_vente = $product[6] where (name = '$product[2]')";
echo $sql.'<br>';
$update = $query = $this->db->query($sql);
return $update;
}
return false;
}
You can use transaction and add index for column name in database table.
$fn = fopen($this->upload->data('full_path'),"r");
$update = true;
$updatedCount = 0;
while(! feof($fn) && $update) {
$pieces = explode("|", fgets($fn));
if(sizeof($pieces) == 9 && is_numeric(trim($pieces[1]))) {
if ($updatedCount == 0) {
$databaseInstance->beginTransaction();
}
$update = $this->model_products->update3s($pieces);
++$updatedCount;
if ($updatedCount > 500) { //in one transaction update 500 rows
$databaseInstance->commit();
$updatedCount = 0;
}
}
}
if ($updatedCount > 0) { // if we have not commited transaction
$databaseInstance->commit();
}
fclose($fn);
Some tips
Add index to field name
Use prepared statements
Disable the MySQL forgeign key check Read more
writing sql function can do that even in much lesser time .
using feature like :
REPLACE()
cursors
SPLIT_STRING(custom)
in a mysql user defined function
CREATE FUNCTION update3s(hole_file_content LONGTEXT) RETURNS Boolean
BEGIN
-----Your implementation(same logic in sql ) ------
END
then coll it just by if it is CI 3
$this->db->call_function('update3s', file_get_contents($this->upload->data('full_path')));
else
$this->db->query("select update3s(".file_get_contents($this->upload->data('full_path')).")");
I have a code that runs a query on an external (slow) API and loops through a lot of variables and inserts data, sends email, ect'.
This is the main function as an example:
// MAIN: Cron Job Function
public function kas_alert() {
// 0. Deletes all the saved data from the `data` table 1 month+ ago.
// $this->kas_model->clean_old_rows();
// 1. Get 'prod' table
$data['table'] = $this->kas_model->prod_table();
// 2. Go through each row -
foreach ( $data['table'] as $row ) {
// 2.2. Gets all vars from the first query.
$last_row_query = $this->kas_model->get_last_row_of_tag($row->tag_id);
$last_row = $last_row_query[0];
$l_aaa_id = $last_row->prod_aaa_id;
$l_and_id = $last_row->prod_bbb_id;
$l_r_aaa = $last_row->dat_data1_aaa;
$l_r_and = $last_row->dat_data1_bbb;
$l_t_aaa = $last_row->dat_data2_aaa;
$l_t_and = $last_row->dat_data2_bbb;
$tagword = $last_row->tag_word;
$tag_id = $last_row->tag_id;
$country = $last_row->kay_country;
$email = $last_row->u_email;
$prod_name = $last_row->prod_name;
// For the Weekly report:
$prod_id = $last_row->prod_id;
$today = date('Y-m-d');
// 2.3. Run the tagword query again for today on each one of the tags and insert to DB.
if ( ($l_aaa_id != 0) || ( !empty($l_aaa_id) ) ) {
$aaa_data_today = $this->get_data1_aaa_by_id_and_kw($l_aaa_id, $tagword, $country);
} else{
$aaa_data_today['data1'] = 0;
$aaa_data_today['data2'] = 0;
$aaa_data_today['data3'] = 0;
}
if ( ($l_and_id != 0) || ( !empty($l_and_id) ) ) {
$bbb_data_today = $this->get_data1_bbb_by_id_and_kw($l_and_id, $tagword, $country);
} else {
$bbb_data_today['data1'] = 0;
$bbb_data_today['data2'] = 0;
$bbb_data_today['data3'] = 0;
}
// 2.4. Insert the new variables to the "data" table.
if ($this->kas_model->insert_new_tag_to_db( $tag_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $aaa_data_today['data2'], $bbb_data_today['data2'], $aaa_data_today['data3'], $bbb_data_today['data3']) ){
}
// Kas Alert Outputs ($SEND is echoed in it's original function)
echo "<h1>prod Name: $prod_id</h1>";
echo "<h2>tag id: $tag_id</h2>";
var_dump($aaa_data_today);
echo "aaa old: ";
echo $l_r_aaa;
echo "<br> aaa new: ";
echo $aaa_data_today['data1'];
var_dump($bbb_data_today);
echo "<br> bbb old: ";
echo $l_r_and;
echo "<br> bbb new: ";
echo $bbb_data_today['data1'];
// 2.5. Check if there is a need to send something
$send = $this->check_if_send($l_aaa_id, $l_and_id, $l_r_aaa, $aaa_data_today['data1'], $l_r_and, $bbb_data_today['data1']);
// 2.6. If there is a trigger, send the email!
if ($send) {
$this->send_mail($l_aaa_id, $l_and_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $l_r_aaa, $l_r_and, $tagword, $email, $prod_name);
}
}
}
This CodeIgniter controller is runs every day and I get this running (Cronjob) for too long using almost nothing from the CPU and RAM (RAM is at 400M/4.25G and CPU at ONLY 0.7%-1.3%).
I wonder if there's an option to split all foreach loops to smaller threads (and I'm not sure if forking will do here) and run all the foreach loops in parallel but in a way that it wont get my server crashing.
I'm no DevOps and really interested learning - What should I do in this case?
i am working on a school room booking system but i am having some trouble outputting the bookings.
Public function displayRoomByDay() {
$query = DB::GetInstance()->query("SELECT B.roomid, B.period, R.roomname, B.bookingdate, B.bookingid FROM booking B INNER JOIN room R on b.roomid = r.roomid WHERE bookingdate = '2016-11-03' and b.roomid=1 ORDER BY b.period"); //Inner join
$count = $query->count();
$freeCount = 0;
for($periods=1;$periods<=6;$periods++) {
for($x=0;$x<$count-1;$x++) {
$outputted=false;
$freeCount=0;
do {
if($query->results()[$x]->period == $periods) {
echo $query->results()[$x]->period . '<br>';
$outputted=true;
} else {
echo 'FREE' . '<br>';
$freeCount = 1;
}
} while($outputted = false and $freeCount=0);
}
}
}
This is the function that i use to output my data. My SQL query returns two items, a booking in period 5 and a booking in period 1 (i have tested this through PHPMyAdmin). I am trying to use nested for loops and a do while loop to loop through the periods available in a day (6). From there i loop through the two bookings that my sql query returns which is this code:
for($x=0;$x<$count-1;$x++) {
$outputted=false;
$freeCount=0;
do {
if($query->results()[$x]->period == $periods) {
echo $query->results()[$x]->period . '<br>';
$outputted=true;
} else {
echo 'FREE' . '<br>';
$freeCount = 1;
}
} while($outputted = false and $freeCount=0);
}
However when i run my page, i get 1 FREE FREE FREE FREE FREE, when i am trying to get 1 FREE FREE FREE FREE 5 as they are when the bookings are.
Your logic is very complicated for this task. You can simplify it like:
public function displayRoomByDay()
{
$query = DB::GetInstance()->query("
SELECT B.roomid,
B.period,
R.roomname,
B.bookingdate,
B.bookingid
FROM booking B
INNER JOIN room R
ON b.roomid = r.roomid
WHERE bookingdate = '2016-11-03'
AND b.roomid=1 ORDER BY b.period"); //Inner join
$results = array_reduce($query->results(), function ($carry, $item) {
$carry[$item->period] = $item;
return $carry;
}, []);
$periods = range(1, 6);
foreach ($periods as $period) {
if (isset($results[$period]) {
echo $period . '<br />';
} else {
echo 'FREE' . '<br />';
}
}
}
I think that's all you need.
I usually work with web hosting companies but I decided to start learning working with servers to expand my knowledge.
I'll better give a real example to explain my question the best:
I have a web application that gathers data from a slow API that returns JSON data of products.
I have a function running every 1AM running a lot of queries on "id"s in my database.
Crontab:
0 1 * * * cd /var/www/html/tools; php index.php aso Cli_kas kas_alert
So this creates a process for the app (please correct me here if I'm wrong) and each process creates threads, and just to be more accurate, they are multi-threads since they do more than one thing: like pulling data from the DB to get the right variables and string them to the API queries, getting the data from the API, organizing it, searching the relevant data, and then inserting new data to the database.
The main PHP functions:
// MAIN: Cron Job Function
public function kas_alert() {
// 0. Deletes all the saved data from the `data` table 1 month+ ago.
// $this->kas_model->clean_old_rows();
// 1. Get 'prod' table
$data['table'] = $this->kas_model->prod_table();
// 2. Go through each row -
foreach ( $data['table'] as $row ) {
// 2.2. Gets all vars from the first query.
$last_row_query = $this->kas_model->get_last_row_of_tag($row->tag_id);
$last_row = $last_row_query[0];
$l_aaa_id = $last_row->prod_aaa_id;
$l_and_id = $last_row->prod_bbb_id;
$l_r_aaa = $last_row->dat_data1_aaa;
$l_r_and = $last_row->dat_data1_bbb;
$l_t_aaa = $last_row->dat_data2_aaa;
$l_t_and = $last_row->dat_data2_bbb;
$tagword = $last_row->tag_word;
$tag_id = $last_row->tag_id;
$country = $last_row->kay_country;
$email = $last_row->u_email;
$prod_name = $last_row->prod_name;
// For the Weekly report:
$prod_id = $last_row->prod_id;
$today = date('Y-m-d');
// 2.3. Run the tagword query again for today on each one of the tags and insert to DB.
if ( ($l_aaa_id != 0) || ( !empty($l_aaa_id) ) ) {
$aaa_data_today = $this->get_data1_aaa_by_id_and_kw($l_aaa_id, $tagword, $country);
} else{
$aaa_data_today['data1'] = 0;
$aaa_data_today['data2'] = 0;
$aaa_data_today['data3'] = 0;
}
if ( ($l_and_id != 0) || ( !empty($l_and_id) ) ) {
$bbb_data_today = $this->get_data1_bbb_by_id_and_kw($l_and_id, $tagword, $country);
} else {
$bbb_data_today['data1'] = 0;
$bbb_data_today['data2'] = 0;
$bbb_data_today['data3'] = 0;
}
// 2.4. Insert the new variables to the "data" table.
if ($this->kas_model->insert_new_tag_to_db( $tag_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $aaa_data_today['data2'], $bbb_data_today['data2'], $aaa_data_today['data3'], $bbb_data_today['data3']) ){
}
// Kas Alert Outputs ($SEND is echoed in it's original function)
echo "<h1>prod Name: $prod_id</h1>";
echo "<h2>tag id: $tag_id</h2>";
var_dump($aaa_data_today);
echo "aaa old: ";
echo $l_r_aaa;
echo "<br> aaa new: ";
echo $aaa_data_today['data1'];
var_dump($bbb_data_today);
echo "<br> bbb old: ";
echo $l_r_and;
echo "<br> bbb new: ";
echo $bbb_data_today['data1'];
// 2.5. Check if there is a need to send something
$send = $this->check_if_send($l_aaa_id, $l_and_id, $l_r_aaa, $aaa_data_today['data1'], $l_r_and, $bbb_data_today['data1']);
// 2.6. If there is a trigger, send the email!
if ($send) {
$this->send_mail($l_aaa_id, $l_and_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $l_r_aaa, $l_r_and, $tagword, $email, $prod_name);
}
}
}
For #Raptor, this is the function that get's the API data:
// aaa tag Query
// Gets aaa prod dataing by ID.
public function get_data_aaa_by_id_and_tg($id, $tag, $query_country){
$tag_for_url = rawurlencode($tag);
$found = FALSE;
$i = 0;
$data = array();
// Create a stream for Json. That's how the code knows what to expect to get.
$context_opts = array(
'http' => array(
'method' => "GET",
'header' => "Accepts: application/json\r\n"
));
$context = stream_context_create($context_opts);
while ($found == FALSE) {
// aaa Query
$json_query_aaa = "https://api.example.com:443/aaa/ajax/research_tag?app_id=$id&term=$tag_for_url&page_index=$i&country=$query_country&auth_token=666";
// Get the Json
$json_query_aaa = file_get_contents($json_query_aaa, false, $context);
// Turn Json to a PHP array
$json_query_aaa = json_decode($json_query_aaa, true);
// Get the data2
$data2 = $json_query_aaa['tag']['data2'];
if (is_null($data2)){ $data2 = 0; }
// Get data3
$data3 = $json_query_aaa['tag']['phone_prod']['data3'];
if (is_null($data3)){ $data3 = 0; }
// Finally, the main prod array.
$json_query_aaa = $json_query_aaa['tag']['phone_prod']['app_list'];
if ( count($json_query_aaa) > 2 ) {
for ( $j=0; $j<count($json_query_aaa); $j++ ) {
if ( $json_query_aaa[$j]['id'] == $id ) {
$found = TRUE;
$data = $json_query_aaa[$j]['data'] + 1;
break;
}
if ($found == TRUE){
break;
}
}
$i++;
} else {
$data = 0;
break;
}
}
$data['data1'] = $data;
$data['data2'] = $data2;
$data['data3'] = $data3;
return $data;
}
All threads are stacked one after an other, and when one thread is done, only then - the second thread can proceed, ect'.
And in technical view on this, all threads wait in the RAM until the one before them is done working "inside" the CPU. (correct me if I'm wrong again :] )
This doesn't even "tickle" the servers RAM or CPU when looking at it in the process manager (I use "htop"). RAM is at 400M/4.25G and CPU at ONLY 0.7%-1.3%.
Making me feel this isn't the best I can get from my current server, and getting slow results from my web app.
How do I get things done in a way that all threads work in parallel, but not to a point that my app crashes due to lacks of CPU or RAM?
I have two queries sent to a database bring back posts (op_ideas 16 cols) followed by another which holds the votes per post (op_idea_vote cols 4) with matching idea_id's
Example of Data:
Query: op_ideas:
[{"idea_id":"2211","author_id":"100000", "date":"2012-09-06
10:02:28","idea_title":"Another test","4" etc etc
Query: op_idea_votes:
idea_id = 2211, agree=3, disagree=1, abstain=0
The code below ought to look at op_ideas, and then cycle over op_ideas_vote until it finds a match under 'idea_id'. Then it goes to the next record under op_ideas, and again using that idea_id search for it within the op_idea_vote list, find a match, and add it to the array.
This works for only the first record, not for the other three. I am testing, so I have 3 rows in each that match idea_id with different results in the op_idea_vote.
$votes = mysql_query($commentVotes);
$result = mysql_query($gl_query);
while ($gce_result = mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
while($allvotes= mysql_fetch_array($votes)) {
if($voteid = $allvotes['idea_id'])
{
//echo $voteid . " main idea and the votes: " . $allvotes;
$gce_result["agree"] = $allvotes['agree'];
$gce_result["disagree"] = $allvotes['disagree'];
$gce_result["abstain"] = $allvotes['obstain'];
}
else
{
$gce_result["agree"] = 0;
$gce_result["disagree"] = 0;
$gce_result["abstain"] = 0;
}
//print_r($gce_result);
}
$data_result[] = $gce_result;
}
echo json_encode($data_result);
If I use print_f(&gce_result) it works fine in phpfiddle. But when i use the code above, it works for the first record, but it's complete missing the second two. It seems to be missing the second while, as it does not even give me the 0 0 0 results.
Query for op_ideas:
$gl_query = "SELECT DISTINCT * FROM heroku_056eb661631f253.op_ideas INNER JOIN op_organs ORDER BY date ASC;";
if (!mysql_query($gl_query)) {
die('Error: ' . $gl_query . " " . mysql_error());
}
$result = mysql_query($gl_query);
Query For op_idea_vote :
$commentVotes = "SELECT v.idea_id, COUNT(v.agree = 1 or null) as agree, COUNT(v.disagree = 1 or null) as disagree, COUNT(v.obstain = 1 or null) as obstain FROM op_idea_vote v GROUP BY v.idea_id";
if (!mysql_query($commentVotes)) {
die('Error: ' . $commentVotes . " " . mysql_error());
}
$votes = mysql_query($commentVotes);
You can scan a resource only once.
So the inner while will be run only one time.
use == instead of = for checking condition of if & while
in the while loop ,you have to assign the value of $allvotes ,but you never assigned,
while ($gce_result == mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
while($allvotes== mysql_fetch_array($votes)) {
if($voteid == $allvotes['idea_id'])
{
//echo $voteid . " main idea and the votes: " . $allvotes;
$gce_result["agree"] = $allvotes['agree'];
$gce_result["disagree"] = $allvotes['disagree'];
$gce_result["abstain"] = $allvotes['obstain'];
}
else
{
$gce_result["agree"] = 0;
$gce_result["disagree"] = 0;
$gce_result["abstain"] = 0;
}
$data_result[] = $gce_result;
}
}
Your problem is trying to scan over the $votes result more than once.
You should store the result of that query first.
Eg.
while ($vote = mysql_fetch_array($votes)) {
$allvotes['idea_id'] = $vote;
}
while ($gce_result = mysql_fetch_array($result)) {
$voteid = $gce_result['idea_id'];
if (array_key_exists[$voteid, $allvotes]) {
//assign results
} else {
//default
}
}
Another option would be to do the query with a join, so you can do everything in one query. Then just loop over that result.