I have a three PHP script, from which two are running as a separate cron job each day in server.
First script(get_products.php), make curl call to external server to get all products data and stored that data in database. Note that there are around 10000 products and increasing each day.
In Second script(find_related.php), selects products from database stored by first script, perform some operations and store operational data in another database. Each product have 10 rows so in this database there is around 100000 rows. This script is running as cron. Sometimes the script is not executed fully and that's why, the actual and expecting results are not stored in database. I included this line of code in script: ini_set('max_execution_time', '3600');
But it not works.
Here is the process done in this script:
Normally task is to find 10 related products based on tags. I have around 10300 products stored in my DB. Each time query take one product and their tags and try to randomly find one product tagged with same tag as main product and store the related product data into another DB for third script. Only one product per tag is allowed. If it will not find total of 10 related products then randomly gets products from another DB named bestseller_products.
Here is my code:
$get_all_products = mysql_query('SELECT * FROM store_products');
while($get_products_sql_res = mysql_fetch_array($get_all_products)){
$related_products = array();
$tags = explode(",",$get_products_sql_res['product_tags']);
$product_id = $get_products_sql_res['product_id'];
$product_handle = $get_products_sql_res['product_handle'];
$get_products_sql = mysql_query('SELECT * FROM related_products WHERE product_handle="'.$product_handle.'"');
if (mysql_num_rows($get_products_sql)==0)
{
$count = 0;
foreach($tags as $t){
$get_related_products_sql = mysql_query("SELECT product_handle, product_title, product_image FROM store_products WHERE product_tags like '%".$t."%' AND product_id != '".$product_id."' ORDER BY RAND()");
if(!$get_related_products_sql){
continue;
}
while($get_related_products = mysql_fetch_array($get_related_products_sql) ){
$related_product_title = mysql_real_escape_string($get_related_products['product_title']);
$found = false;
foreach($related_products as $r){
if($r['handle'] == $get_related_products['product_handle']){
$found = true;
break;
}
}
if($found == false){
$related_products[$count]['handle'] = $get_related_products['product_handle'];
mysql_query("INSERT INTO related_products (product_handle, product_id, related_product_title, related_product_image, related_product_handle) VALUES ('$product_handle','$product_id','$related_product_title', '$get_related_products[2]', '$get_related_products[0]')");
$count = $count + 1;
break;
}
}
}
if($count < 10){
$bestseller_products = mysql_query("SELECT product_handle, product_title, product_image FROM bestseller_products WHERE product_id != '".$product_id."' ORDER BY RAND() LIMIT 10");
while($bestseller_products_sql_res = mysql_fetch_array($bestseller_products)){
if($count < 10){
$found = false;
$related_product_title = mysql_real_escape_string($bestseller_products_sql_res['product_title']);
$related_product_handle = $bestseller_products_sql_res['product_handle'];
foreach($related_products as $r){
if($r['handle'] == $related_product_handle){
$found = true;
break;
}
}
if($found == false){
$related_product_image = $bestseller_products_sql_res['product_image'];
mysql_query("INSERT INTO related_products (product_handle, product_id, related_product_title, related_product_image, related_product_handle) VALUES ('$product_handle','$product_id','$related_product_title', '$related_product_image', '$related_product_handle')");
$count = $count + 1;
}
}
}
}
}
}
Third script(create_metafields.php), created metafields in external server using data created by second script. And same problem arises as in second script.
So i want to execute the second script into parts. I mean, not to process all 10000 products in one call but want to run unto parts(1-500,501-1000,1001-1500,..) like it. But dont want to create separate cron jobs. Please suggest if someone has solution. I really need to figure it out.
Thanks in advance!
Related
I'm creating a basic main menu for a stock market simulator where the price of a company will be updated periodically. For testing purposes, I need to make a loop to display the price of a share on the website five times (with the website automatically updating without refreshing) and to update the database at the same time.
I have successfully wrote some code which will both update the database with the current share price and will also update the website as well. However, when I have tried to include a loop I have come to a problem. I have included a loop to iterate five times but the problem that I am having is that the code continues to iterate even after five tries.
PHP:
<?php
$conn = mysqli_connect("localhost", "root", "", "prices");
if ($conn->connect_error)
{
die("Connection error: ". $conn->connect_error);
}
$result = $conn->query("SELECT `price` FROM `priceTable` WHERE `company` = 'Bawden'");
$x = 0;
if ($result->num_rows > 0)
{
while ($row = $result->fetch_assoc())
{
echo $row['price'];
echo '<br><br>';
echo $x;
if ($x < 5)
{
$random = (rand(3300, 3700) / 100);
$sql = $conn->query("UPDATE `priceTABLE` SET `price` = '$random' WHERE `company` = 'Bawden'");
$x++;
}
}
}
?>
The above code will be displayed in a separate document with Javascript code and I can post this if required in the original post however I originally chose not to as I believe this is a PHP only problem. I have chosen to display $x to see if the value will increment. However, when running, the value of $x will stay at 0.
My expected result is that, on the website, there will only be five different updates and in the database, the database will only be updated five times.
However, my actual result is that the website and database are both continuously being updated, not stopping after five times.
I'm trying to limit the update command to only 5 updates yes. At the
moment, for testing purposes, there is only one company in my database
with one price only. So I'm updating this one company's price five
times
If you need to do the update 5 times for each row returned from the database, change your if statement to a for loop. Change this :-
if ($x < 5)
{
$random = (rand(3300, 3700) / 100);
$sql = $conn->query("UPDATE `priceTABLE` SET `price` = '$random' WHERE `company` = 'Bawden'");
$x++;
}
to this
for ($x = 0, $x < 5, $x++)
{
$random = (rand(3300, 3700) / 100);
$sql = $conn->query("UPDATE `priceTABLE` SET `price` = '$random' WHERE `company` = 'Bawden'");
}
This will repeat the process exactly 5 times and not rely on a separate counter (remove the other references to $x). Not sure why you would want to update the same record 5 times with different random values though.
The else will break the first loop, the second one will stop on the first while loop.
while ($row = $result->fetch_assoc())
{
echo $row['price'];
echo '<br><br>';
echo $x;
if ($x < 5)
{
$random = (rand(3300, 3700) / 100);
$sql = $conn->query("UPDATE `priceTABLE` SET `price` = '$random' WHERE `company` = 'Bawden'");
$x++;
}else{
break;
}
break;
}
What makes you think the loop should stop after 5 iterations?
You need to add the condition $x<5 in the while ($row = $result->fetch_assoc())
Edit following your comment
What you initially wrote is something like loop hundreds of times if need be and do something in the first 5 occurrences (starting loop 6, keep looping but do nothing).
Now for the 2nd half of your comment, I'm not sure what you mean.
What I see in your code is:
Select all prices for company = 'Bawden'
Update all the prices for company = 'Bawden' 5 times (loop) with a random value, the same one, on all the records.
Not enough information to tell for sure but I don't think it makes sense: on one hand, you except to have several records under company = 'Bawden (= reason why you created a loop), on the other hand, your update feels like it is written under the assumption there would be 1 record only...
Are you missing something like a price date from your table? What is the primary key of priceTable?
Try to post more technical details about your table (definition, sample of data) or it will be complicated to help further.
Problem
I have a magento store, this store run an script that sync intranet products and store, this script run each 1 hour, my server administrator block the site access because this script is getting slow queries.
Question
My script just read a CSV file and for each row check if the SKU already registered, if already update weight, stock and price, have a better way to do this?
this is my script.
$produto = Mage::getModel('catalog/product');
$productId = $produto->getIdBySku($sku);
$produto->load($productId);
if ($produto->getId()) {
$stock = $produto->getStockData();
$stock['qty'] = $quantidade;
$stock['is_in_stock'] = $this->initStock($quantidade);
$stock['manage_stock'] = 1;
$stock['use_config_manage_stock'] = 1;
$produto->setData('price', $preco);
if ($altura > 0)
$produto->setData('volume_altura', $altura);
if ($comprimento > 0)
$produto->setData('volume_comprimento', $comprimento);
if ($largura > 0)
$produto->setData('volume_largura', $largura);
$produto->setData('weight', $this->initWeight($peso));
$produto->setStockData($stock);
$produto->save();
} else {
$produto = Mage::getModel('catalog/product');
$produto->setTypeId('simple');
$produto->setAttributeSetId(4);
$produto->setData('tax_class_id', 0);
$produto->setVisibility(1);
$produto->setStatus(1);
$produto->setData('sku', $sku);
$produto->setData('color', $cor);
$produto->setData('name', utf8_encode($descricao));
$produto->setData('marca', $this->initAttribute(148, $marca));
$produto->setData('codigo_barra', $codBarra);
$produto->setData('price', $preco);
if ($altura > 0)
$produto->setData('volume_altura', $altura);
if ($comprimento > 0)
$produto->setData('volume_comprimento', $comprimento);
if ($largura > 0)
$produto->setData('volume_largura', $largura);
$produto->setData('cost', $custo);
$stock['qty'] = $quantidade;
$stock['is_in_stock'] = $this->initStock($quantidade);
$stock['manage_stock'] = 1;
$stock['use_config_manage_stock'] = 1;
$produto->setStockData($stock);
$produto->setData('weight', $this->initWeight($peso));
$produto->save();
}
Is there any way to reduce the number of items in the CSV file? I mean, if you send all items every time and for each one of them you make a query, the number of queries you make and the time that will take the script to finish will be too high.
I recommend you to only sync with the store the new items or the items with updates values, it will decrease the number of rows in the CSV and the number of queries too, also, if you include in your CSV file a field with a flag Updated/New or something like this, you can avoid the first query to check if the product exists in the case of the "New" flag is found.
I hope it helps.
I have a game script thing set up, and when it creates a new character I want it to find an empty address for that players house.
The two relevant table fields it inserts are 'city' and 'number'. The 'city' is a random number out of 10, and the 'number' can be 1-250.
What it needs to do though is make sure there's not already an entry with the 2 random numbers it finds in the 'HOUSES' table, and if there is, then change the numbers. Repeat until it finds an 'address' not in use, then insert it.
I have a method set up to do this, but I know it's shoddy- there's probably some more logical and easier way. Any ideas?
UPDATE
Here's my current code:
$found = 0;
while ($found == 0) {
$num = (rand()%250)+1; $city = (rand()%10)+1;
$sql_result2 = mysql_query("SELECT * FROM houses WHERE city='$city' AND number='$num'", $db);
if (mysql_num_rows($sql_result2) == 0) { $found = 1; }
}
You can either do this in PHP as you do or by using a MySQL trigger.
If you stick to the PHP way, then instead of generating a number every time, do something like this
$found = 0;
$cityarr = array();
$numberarr = array();
//create the cityarr
for($i=1; $i<=10;$i++)
$cityarr[] = i;
//create the numberarr
for($i=1; $i<=250;$i++)
$numberarr[] = i;
//shuffle the arrays
shuffle($cityarr);
shuffle($numberarr);
//iterate until you find n unused one
foreach($cityarr as $city) {
foreach($numberarr as $num) {
$sql_result2 = mysql_query("SELECT * FROM houses
WHERE city='$city' AND number='$num'", $db);
if (mysql_num_rows($sql_result2) == 0) {
$found = 1;
break;
}
}
if($found) break;
}
this way you don't check the same value more than once, and you still check randomly.
But you should really consider fetching all your records before the loops, so you only have one query. That would also increase the performance a lot.
like
$taken = array();
for($i=1; $i<=10;$i++)
$taken[i] = array();
$records = mysql_query("SELECT * FROM houses", $db);
while($rec = mysql_fetch_assoc($records)) {
$taken[$rec['city']][] = $rec['number'];
}
for($i=1; $i<=10;$i++)
$cityarr[] = i;
for($i=1; $i<=250;$i++)
$numberarr[] = i;
foreach($cityarr as $city) {
foreach($numberarr as $num) {
if(in_array($num, $taken[]) {
$cityNotTaken = $city;
$numberNotTaken = $number;
$found = 1;
break;
}
}
if($found) break;
}
echo 'City ' . $cityNotTaken . ' number ' . $numberNotTaken . ' is not taken!';
I would go with this method :-)
Doing it the way you say can cause problems when there is only a couple (or even 1 left). It could take ages for the script to find an empty house.
What I recommend doing is insert all 2500 records in the database (combo 1-10 with 1-250) and mark with it if it's empty or not (or create a combo table with user <> house) and match it on that.
With MySQL you can select a random entry from the database witch is empty within no-time!
Because it's only 2500 records, you can do ORDER BY RAND() LIMIT 1 to get a random row. I don't recommend this when you have much more records.
This is part of code from my backoffice page. ( is an edit.php page for a user to edit / modify )
// first query to get cats from user table
$query = "select * from user where name='".$_SESSION['username']."' order by id ASC limit 1";
$result=mysql_query($query);
if (mysql_num_rows($result)) {
while($row=mysql_fetch_array($result)){
$cat1 = $row['cat1'];
$cat2 = $row['cat2'];
$cat3 = $row['cat3'];
$cat4 = $row['cat4'];
$cat5 = $row['cat5'];
$cat6 = $row['cat6'];
$cat7 = $row['cat7'];
$cat8 = $row['cat8'];
$cat9 = $row['cat9'];
$cat10 = $row['cat10'];
}
}
// now i want to build 10 select boxes with selected according the user table $cats
// below is what i can build to first box $cat1
// is there a way i can produce this for the 10 select boxes whitout having to make 10 cycles of bellow code
<select name="theme" id="theme">
<?php
$q1 = 'SELECT * FROM cats ORDER BY title ASC';
$r1 = mysql_query($q1);
while( $row = mysql_fetch_array($r1)) {
if ( $cat1 == $row['id'] ) {
print "<option class=\"cssoption\" selected=\"selected\" value=\"".$row['id']."\">".htmlentities($row['title'])."</option>";
}
else {
print "<option class=\"cssoption\" value=\"".$row['id']."\">".htmlentities($row['title'])."</option>";
}
}
?>
</select>
I am not a coder so this might not be effective code.
Hope someone can help me here and understands what i am trying to do.
Many Thanks.
The code is fine. This 10 cycles as you name it is a almost zero cost.
This is the usual way we do it, we fetch sequentialy the records one by one.
In addition it makes no sense to ask not to do the 10 cycles because you are applying an if else condition in the same time, this means that you check every record if the cat id is the same with the row so you need the loop.
On the other hand if for some reason you want to skip some records, you can use the mysql seek function to start fetching from the desired record.
for($i=0;$i<99999;$i++)
(9999*9999);
echo 'This time the cost of this loop was:',(microtime()-$start),' seconds.';
?>
So I've been working on an IRC game for one year now, written in PHP and using a PHP to IRC framework.
Recently, I've added the ability to archive scores (they're being reseted every couple hundreds of games) which forced me to update various admin functions.
I've just updated a function that allows me to merge two players (some users don't bother looking for their old password etc...) in order to merge archived scores too (in case a reset has occurred before I find the duplicated accounts).
The score-merging part (below) works has intended, but I'm wondering if I can optimize the process because I find it rather heavy (but can't think of something better) :
$from_stats = $this->db->query("SELECT `games`, `wins`, `points`, `date_archive` FROM ".$this->dbprefix."score WHERE `id`=".$id1." AND `channel`='".$gamechan."' GROUP BY `date_archive`"); // get scores for the original account
$to_stats = $this->db->query("SELECT `games`, `wins`, `points`, `date_archive` FROM ".$this->dbprefix."score WHERE `id`=".$id2." AND `channel`='".$gamechan."' GROUP BY `date_archive`"); // get scores for the duplicated account
$from_games = array();
$from_wins = array();
$from_points = array();
$from_date = array();
while (list($fromstats_games,$fromstats_wins,$fromstats_points,$fromstats_date) = $this->db->fetchRow($from_stats)) { // build score arrays for the original account
$from_games[count($from_games)] = $fromstats_games;
$from_wins[count($from_wins)] = $fromstats_wins;
$from_points[count($from_points)] = $fromstats_points;
$from_date[count($from_date)] = $fromstats_date;
}
$to_games = array();
$to_wins = array();
$to_points = array();
$to_date = array();
while (list($tostats_games,$tostats_wins,$tostats_points,$tostats_date) = $this->db->fetchRow($to_stats)) { // build score arrays for the duplicated account
$to_games[count($to_games)] = $tostats_games;
$to_wins[count($to_wins)] = $tostats_wins;
$to_points[count($to_points)] = $tostats_points;
$to_date[count($to_date)] = $tostats_date;
}
foreach ($from_date as $key1 => $id1_date) {
foreach ($to_date as $key2 => $id2_date) {
if ($id1_date == $id2_date) { // merge scores if dates match
$from_games[$key1] += $to_games[$key2];
$from_wins[$key1] += $to_wins[$key2];
$from_points[$key1] += $to_points[$key2];
$this->db->query("UPDATE ".$this->dbprefix."score SET `games`=".$from_games[$key1].", `wins`=".$from_wins[$key1].", `points`=".$from_points[$key1]." WHERE `id`=".$id1." AND `channel`='".$gamechan."' AND `date_archive`='".$id1_date."'");
break;
}
}
}
$this->db->query("DELETE FROM ".$this->dbprefix."score WHERE `id`=".$id2); // delete all entries for the duplicated account
Just one tip: after all use this query (if You have appropriate privilages)
$this->db->query("OPTIMIZE TABLE ".$this->dbprefix."score");
This should cause all indexes in this table to be recalculated. You'll notice the index file size has changed to 1kb (or few bytes)