This is more of a logic question than language question, though the approach might vary depending on the language. In this instance I'm using Actionscript and PHP.
I have a flash graphic that is getting data stored in a mysql database served from a PHP script. This part is working fine. It cycles through database entries every time it is fired.
The graphic is not on a website, but is being used at 5 locations, set to load and run at regular intervals (all 5 locations fire at the same time, or at least within <500ms of each-other). This is real-time info, so time is of the essence, currently the script loads and parses at all 5 locations between 30ms-300ms (depending on the distance from the server)
I was originally having a pagination problem, where each of the 5 locations would pull a different database entry since i was moving to the next entry every time the script runs. I solved this by setting the script to only move to the next entry after a certain amount of time passed, solving the problem.
However, I also need the script to send an email every time it displays a new entry, I only want it to send one email. I've attempted to solve this by adding a "has been emailed" boolean to the database. But, since all the scripts run at the same time, this rarely works (it does sometimes). Most of the time I get 5 emails sent. The timeliness of sending this email doesn't have to be as fast as the graphic gets info from the script, 5-10 second delay is fine.
I've been trying to come up with a solution for this. Currently I'm thinking of spawning a python script through PHP, that has a random delay (between 2 and 5 seconds) hopefully alleviating the problem. However, I'm not quite sure how to run exec() command from php without the script waiting for the command to finish. Or, is there a better way to accomplish this?
UPDATE: here is my current logic (relevant code only):
//get the top "unread" information from the database
$query="SELECT * FROM database WHERE Read = '0' ORDER BY Entry ASC LIMIT 1";
//DATA
$emailed = $row["emailed"];
$Entry = $row["databaseEntryID"];
if($emailed == 0)
{
**CODE TO SEND EMAIL**
$EmailSent="UPDATE database SET emailed = '1' WHERE databaseEntryID = '$Entry'";
$mysqli->query($EmailSent);
}
Thanks!
You need to use some kind of locking. E.g. database locking
function send_email_sync($message)
{
sql_query("UPDATE email_table SET email_sent=1 WHERE email_sent=0");
$result = FALSE;
if(number_of_affacted_rows() == 1) {
send_email_now($message);
$result = TRUE;
}
return $result;
}
The functions sql_query and number_of_affected_rows need to be adapted to your particular database.
Old answer:
Use file-based locking: (only works if the script only runs on a single server)
function send_email_sync($message)
{
$fd = fopen(__FILE__, "r");
if(!$fd) {
die("something bad happened in ".__FILE__.":".__LINE__);
}
$result = FALSE;
if(flock($fd, LOCK_EX | LOCK_NB)) {
if(!email_has_already_been_sent()) {
actually_send_email($message);
mark_email_as_sent();
$result = TRUE; //email has been sent
}
flock($fd, LOCK_UN);
}
fclose($fd);
return $result;
}
You will need to lock the row in your database by using a transaction.
psuedo code:
Start transaction
select row .. for update
update row
commit
if (mysqli_affected_rows ( $connection )) >1
send_email();
Related
I have a file that has the function of importing data into a sql database from an api. A problem I encountered was that the api can only retrieve a max dataset size of 1000, even though sometimes I need to retrieve large amounts of data, ranging from 10-200,000. My first thought was to create a while loop in which inside I make calls to the api until all of the data is properly retrieved, and afterwards, can I enter it into the database.
$moreDataToImport = true;
$lastId = null;
$query = '';
while ($moreDataToImport) {
$result = json_decode(callToApi($lastId));
$query .= formatResult($result);
$moreDataToImport = !empty($result['dataNotExported']);
$lastId = getLastId($result['customers']);
}
mysqli_multi_query($con, $query);
The issue I encountered with this is that I was quickly reaching memory limits. The easy solution to this is to simply increase the memory limit until it was suffice. How much memory I needed, however, was undeclared, because there is always a possibility that I need to import very large datasets, and can theoretically always run out of memory. I don't want to set an infinite memory limit, as the problems with that are unimaginable.
My second solution to this was instead of looping through the imported data, I could instead send it to my database, and then do a page refresh, with a get request specifying the last Id I left off on.
if (isset($_GET['lastId'])
$lastId = $_GET['lastId'];
else
$lastId = null;
$result = json_decode(callToApi($lastId));
$query .= formatResult($result);
mysqli_multi_query($con, $query);
if (!empty($result['dataNotExported'])) {
header('Location: ./page.php?lastId='.getLastId($result['customers']));
}
This solution solves my memory limit issue, however now I have another issue, being that browsers, after 20 redirects (depends on the browser), will automatically kill the program to stop a potential redirect loop, then shortly refresh the page. The solution to this would be to kill the program yourself at the 20th redirect and allow it to do a page refresh, continuing the process.
if (isset($_GET['redirects'])) {
$redirects = $_GET['redirects'];
if ($redirects == '20') {
if ($lastId == null) {
header("Location: ./page.php?redirects=2");
}
else {
header("Location: ./page.php?lastId=$lastId&redirects=2");
}
exit;
}
}
else
$redirects = '1';
Though this solves my issues, I am afraid this is more impractical than other solutions, as there must be a better way to do this. Is this, or the issue of possibly running out of memory my only two choices? And if so, is one more efficient/orthodox than the other?
Do the insert query inside the loop that fetches each page from the API, rather than concatenating all the queries.
$moreDataToImport = true;
$lastId = null;
$query = '';
while ($moreDataToImport) {
$result = json_decode(callToApi($lastId));
$query = formatResult($result);
mysqli_query($con, $query);
$moreDataToImport = !empty($result['dataNotExported']);
$lastId = getLastId($result['customers']);
}
Page your work. Break it up into smaller chunks that will be below your memory limit.
If the API only returns 1000 at a time, then only process 1000 at a time in a loop. In each iteration of the loop you'll query the API, process the data, and store it. Then, on the next iteration, you'll be using the same variables so your memory won't skyrocket.
A couple things to consider:
If this becomes a long running script, you may hit the default script running time limit - so you'll have to extend that with set_time_limit().
Some browsers will consider scripts that run too long to be timed out and will show the appropriate error message.
For processing upwards of 200,000 pieces of data from an API, I think the best solution is to not make this work dependant on a page load. If possible, I'd put this in a cron job to be run by the server on a regular schedule.
If the dataset is dependant on the request (for example, if you're processing temperatures from one of 1000s of weather stations - the specific station ID to be set by the user), then consider creating a secondary script that does the work. Calling and forking the secondary script from your primary script will enable your primary script to finish execution while your secondary script executes in the background on your server. Something like:
exec('php path/to/secondary-script.php > /dev/null &');
I usually update my site "by hand", entering in one page called "enterheretoupdate.php". This page refreshes every minute to do all the job I need, so while this page is "open", my site keeps refreshing every minute.
What does "enterheretoupdate.php" do? It makes things related to mysql: create tables, selects from tables, add rows to tables, etc. Apart from that, it also make calculations on php and updates .json files.
I would like to create a cron job, so that it is not necessary for me to visit "enterheretoupdate.php" on my computer for updating my site every minute.
I am quite new on this, but I have learned how to create a cron job (I use 1and1). The example cron job I have created, consisting on sending an email every minute, works fine.
But then, I tried to save "enterheretoupdate.php" as a cron job and it does not work. Is there a "limitation" on the things a cron job can do? How should I "translate" my php file to make it work as a cron job?
Any help is really welcome.
This is how my .php file looks like:
<?php
$page = $_SERVER['PHP_SELF'];
$sec = "10";
//Change 1 to reload, 0 to not to reload;
$reload=1;
$gamecode=7;
$cmp="EL";
$year=2017;
if ($reload==1) echo"<head><meta http-equiv='refresh'content=".$sec.";URL='".$page."?gamecode=".$gamecode."&cmp=".$cmp."&year=".$year."'></head>";
include("../newcon.php");
include("../formulas.php");
include_once("funLightCreateTables.php");
include_once("funLightFirstFive.php");
include_once("funLightChanges.php");
include_once("funLightLiveJsons.php");
if ($cmp=="EC") {$l="U";}
if ($cmp=="EL") {$l="E";}
//Check
$q="SELECT * FROM LightLiveSchedule WHERE year=".$year." and cmp=".$cmp." and gamecode=".$gamecode."";
$res=mysqli_query($link,$q);
while ($r=mysqli_fetch_assoc($res)){
$started=$r['started'];
}
if ($started==0){
LightCreateTables($cmp,$year,$gamecode);
$q="UPDATE LightLiveSchedule SET started=1 WHERE year=".$year." and cmp=".$cmp." and gamecode=".$gamecode."";
mysqli_query($link,$q);
}
//Read
$pbp=file_get_contents("http://thesite.com/data.json?gamecode=".$gamecode."&seasoncode=".$l.$year."");
$pbp = json_decode($pbp,true);
//Insert
mysqli_query($link,"Truncate P_Live_Temp_".$cmp."_".$year."_".$gamecode."");
$lres=0;
$vres=0;
$n=0;
for ($i=0;$i<=4;$i++){
$nplays[$i]=count($pbp[$qtitle[$i]]);
$ii=0;
for ($j=0;$j<=$nplays[$i];$j++){
//change results
if ($pbp[$qtitle[$i]][$ii]['PUNTOS_A']!=null) {
$lres=$pbp[$qtitle[$i]][$ii]['PUNTOS_A'];
}
if ($pbp[$qtitle[$i]][$ii]['PUNTOS_B']!=null) {
$vres=$pbp[$qtitle[$i]][$ii]['PUNTOS_B'];
}
//clean
if (strpos($pbp[$qtitle[$i]][$ii]['CSDESCWEB'],"(")==0) {$play=$pbp[$qtitle[$i]][$ii]['CSDESCWEB'];}
if (strpos($pbp[$qtitle[$i]][$ii]['CSDESCWEB'],"(")>0) {$play=substr($pbp[$qtitle[$i]][$ii]['CSDESCWEB'],0,strpos($pbp[$qtitle[$i]][$ii]['CSDESCWEB'],"(")-1);}
//count
$points=0;
if ($play=="Three Pointer") {$points=3;}
if ($play=="Two Pointer" or $play=="Lay Up" or $play=="Dunk") {$points=2;}
if ($play=="Free Throw In") {$points=1;}
//ntconsole=00:00 at End Game
if ($play=="End Game") {$pbp[$qtitle[$i]][$ii]['NTCONSOLA']="00:00";}
//insert
$q="INSERT INTO P_Live_temp_".$cmp."_".$year."_".$gamecode."
(orden,shteam,shloc,shvis,quarter,minute,ntconsole,pcode,play,locres,visres,points)
VALUES
(".$n.",'".$pbp[$qtitle[$i]][$ii]['NTEQUIPO']."','".$pbp['ca']."','".$pbp['cb']."',".($i+1).",
".$pbp[$qtitle[$i]][$ii]['MINUTO'].",'".$pbp[$qtitle[$i]][$ii]['NTCONSOLA']."',
'".str_replace(" ","",substr($pbp[$qtitle[$i]][$ii]['NTJUGD'],1,10))."','".$play."',".$lres.",".$vres.",".$points.")";
mysqli_query($link,$q);
$ii++;
$n++;
}
}
Do you think it is suitable for a cron job? How should I proceed? Thanks a lot!
I had similar issues but the following worked for me.
See the link to change default mysql permission
How to allow remote connection to mysql
Now change your db_server value in the sql connection file from
localhost to 127.0.0.1
In your case the you need to edit the file ../newcon.php it seems.
I am creating a panel where the user would select a contact group and then schedule mail.
In order to keep a track of mails send, I am storing it into the database.
But this storing of data in tables takes a long time as loop generates near about 1000+ insert queries which makes the page unresponsive to get the data to be populated into the table.
This is just for single user, what if at a time there are 10 users performing the same action, that would cause low performance on application.
I need to know whether is there any mechanism using 'mysql' and 'php' to que my queries so that it could be executed later and user should not wait for the query execution to be completed?
my code is as below,
$recordsCounter++;
//insert into EMAIL_RECIPIENTS
foreach($this->recipients as $val){
$query="INSERT INTO bas_email_recipients SET
recipient_type='".$val['recipientType']."',
email_dump_id='".$emailDumpId."', ";
if(isset($val['contactDetailsId'])){
$query.="contact_details_id='".$val['contactDetailsId']."', ";
}
if(isset($val['contactName'])){
$query.="recipient_name='".$val['contactName']."', ";
}
$query.="email_address='".$val['emailAddress']."',
mail_to='".$val['mailTo']."'";
if($db->query($query)>0){
$recordsCounter++;
}
}
For example, you can put that queries into cache and retrieve/execute/remove them on cron
Instead of
if($db->query($query)>0){ // putting to database
$recordsCounter++;
}
Put query to array
$queryList[] = $query;
and then put this array to cache with some unique key:
$cache->put('key', $queryList);
Install cron job for say every minute to execute:
$queryList = $cache->get('key');
if ($queryList) {
foreach ($queryList as $query) {
$db->query($query);
}
}
Heavy database operations will be executed in cron in background, so you will get much better performance with your web
I am developing a desktop software where it charge user per execution the main action. For example say it will charge user 0.1$ for per PDF print.
and my software provide multithreading. .
so, if it run single thread it works fine :)
but the problem is if user run multiple thread at one (say 10/20 threads)
it (php) also continues user to allow the server/execution even balance get below zero..
though my php script check whether balance is positive ..
but after user run multiple threads balance become like -5.95$ or -25.75$ etc
and that is a big security/financial issue..
here is the code I am using:
<?php
$strSQL = "Select * from users where Email = '$strUser'";
$return = mysql_query($strSQL, $strDBConn);
$strDBData = mysql_fetch_array($return, MYSQL_ASSOC);
//checking balance
$strBalance = $strDBData['Balance'];
if($strBalance < 0)
{
// if balance 0 then exit so, my software/thread will not process further
mysql_close($strDBConn);
exit('Balance Exceed');
}
//rest of the codes that realted to service executaion
// code that substract the balnce
$dblCost = 0.25;
$strSQL = "Update users set Balance = Balance - '$dblCost' where Email = '$strUser'";
$return = mysql_query($strSQL, $strDBConn);
//rest finising codes
?>
any help/suggestion would be highly appreciated..
thanks in advance.
best regards
I think, this is a quite similar question:
What is equivalent of the C# lock statement in PHP?
First, try to switch away from the old "mysql" to somethin new, maybe some PDO like DB access ;).
Then, for getting around with multi-thread in php, it can be a good idea, to write a file for every userid (!) and lock this file, when there's a request. When file is locked in another thread, wait for x seconds for the file to be unlocked by the locker-thread. If it is not unlocked within time, something went wrong. When in locked-thread all went good, unlock the file after every operation needed.
Theoraticaly you will be good with then till there's a multi-thread soloution in PHP ;)
Hey guys i'm making a website where you submit a server for advertising. When the user goes to the index page of my website it grabs the ip's of all the servers submitted and then tests to see if it is online using fsockopen() like so:
while($row = mysql_fetch_assoc($rs)) {
$ip = $row['ip'];
$info = #fsockopen($ip, 25565, $errno, $errstr, 0.5);
if($info) {
$status = "<div><img width='32px' height='32px'
title='$name is online!' src='images/online.png'/></div>";
$online = true;
} else {
$status = "<div><img width='32px' height='32px'
title='$name is offline!' src='images/offline.png'/></div>";
$online = false;
}
}
}
This way works fine, but the only downside is when you load the site it takes a good 2-4 seconds to start loading the website due to the fsockopen() methods being called. I want to know if there is a better way to do this that will reduce the amount of wait time before the website loads.
Any information will be appreciated, thanks.
Store the online status and last check time in a database, if the last check time is longer than 15 minutes for example, update it. I am pretty sure you don't need to get the status on EVERY pageload? It's the time it takes to connect to each server that slows down the website.
Then again, you would probably wanna move the update process to a cronjob instead of relying on someone visiting your website to update the server statuses.
Looking at your example, I'd make all the $status bits be javascript calls to another php page that checks that individual server.
However, the idea to move the status checks to cron job or use some kind of status caching is very good too. Maybe store statuses in a database only only check the ones that have expired (time limit set by you).