Check for new entries in a DB [closed] - php

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am trying to build an application that will query a db, and send data somewhere as it comes into the database.
I can get the data I want from the database using this code:
$sql="SELECT * FROM `signals` order by `time` DESC LIMIT 100";
$result = mysqli_query($DatabasePointer,$sql)
or die(mysqli_error($DatabasePointer));
$row_cnt = mysqli_num_rows($result);
if($row_cnt>0)
{
$array = array();
$i=0;
while($row = mysqli_fetch_array($result))
{
$i++;
//$a = array();
$epoch = $row['time'];
// convert UNIX timestamp to PHP DateTime
$dt = new DateTime("#$epoch");
if(
($row['symbol'] === 'USDJPYecn')
|| ($row['symbol'] === 'USDCADecn')
|| ($row['symbol'] === 'EURUSDecn')
)
{
if(
($row['timeframe'] === 'M5')
|| ($row['timeframe'] === 'M15')
)
{
$a = array(
"time" => $dt->format('Y-m-d H:i:s'),
"signal" => $row['signal'],
"symbol" => $row['symbol'],
"price" => $row['price'],
"timeframe" => $row['timeframe'],
"epoch" => $row['time'],
"candel" => $row['candel']
);
$array[] = $a;
}
}
} // while
echo json_encode($array, JSON_UNESCAPED_SLASHES);
}
However, I am not sure how to revise the code to check to see if the data is new, or has already been sent out to another source. I am also unsure how to revise the code to only send new data as it hits the db, not an entire array of data like I am calling now.
Can somebody please point me in the right direction?
EDIT:
$sql="SELECT * FROM `tdisignals` order by `time` DESC LIMIT 100";
$result = mysqli_query($DatabasePointer,$sql)
or die(mysqli_error($DatabasePointer));
$row_cnt = mysqli_num_rows($result);
if($row_cnt>0)
{
$array = array();
$i=0;
while($row = mysqli_fetch_array($result))
{
$i++;
//$a = array();
$epoch = $row['time'];
// convert UNIX timestamp to PHP DateTime
$dt = new DateTime("#$epoch");
if(
$row['symbol'] === 'USDJPYecn'
|| ($row['symbol'] === 'USDCADecn')
|| ($row['symbol'] === 'GBPUSDecn'))
{
if(
$row['timeframe'] === 'M5')
|| ($row['timeframe'] === 'M15'))
{
$a = array(
"time" => $dt->format('Y-m-d H:i:s'),
"signal" => $row['signal'],
"symbol" => $row['symbol'],
"price" => $row['price'],
"timeframe" => $row['timeframe'],
"epoch" => $row['time'],
"candel" => $row['candel'],
);
$array[] = $a;
}
}
}
// echo json_encode($array, JSON_UNESCAPED_SLASHES);
$fuegostore = json_encode($array, JSON_UNESCAPED_SLASHES);
// $sql2 = "INSERT INTO fuegosync (time, lastsync) ".
// "VALUES ('$date', '$fuegostore')";
// $result2 = mysqli_query($DatabasePointer,$sql2)
// or die(mysqli_error($DatabasePointer));
$sql3="SELECT lastsync, MAX(CAST(time AS CHAR)) FROM `fuegosync`";
$result3 = mysqli_query($DatabasePointer,$sql3)
or die(mysqli_error($DatabasePointer));
$row2 = mysqli_fetch_row($result3);
if($row2[0] === $fuegostore)
echo 'No New Signals';
else
echo 'New Signals';
///OPTION 1:
//print_r (json_encode($array[0], JSON_UNESCAPED_SLASHES));
//OPTION 2:
foreach($array as $x) {
if(strtotime($array[0]['time']) >= $row2[1]) {
echo '<br /><span>'.$x['signal'].' - '.$x['symbol'].' - '.$x['price'].'<br />';
} else {
echo 'No New Signals';
}
}
echo $row2[0];
}
This code is successfully detecting a new data hitting the database. What I am struggling with now, is revising the code to only display the newly detected data piece, not the entire array as you see it.
NEW EDIT:
I have gotten the code to display only the newest piece of data, but now I have a conundrum.
If I poll the database say every minute, and a new piece of data hits the db, the script will pick it up -- however if another new piece of data hits the db seconds after the first new piece was sent to target, the second new piece will totally get ignored because the poll would be every minute. I would basically have to poll the database every few seconds... and that sounds like a performance nightmare...
The grayed out OPTION 1 is what displays the newest data, but would skip a newer piece before a minute based poll unless the db was polled every second.
OPTION 2 works, but displays the entire array... so I am unsure how to revise the code to only display the newest pieces, not the entire thing.
Any suggestions?

Answer 1

Secured
One of the most secure way to do this kind of thing is :
having an incremented field on source
allowing a query on target
Step-by-step, source driven
you add some data in source, with their auto-incremented id
you query your target from source and ask for the last id know
with this id, from source, you get all new record, and query with these data an insert page on target
Alternate, target driven
you add some data in source, with their auto-incremented id
your target get his bigger id and ask source for new data
target update himself
And you can go to step one again. If you are careful on your insert (use of roll-back, break full batch on one fail), you should have a perfect target whenever the fail on the source / target link, and the bandwidth is as low as it could.
One-sided and check-less
This allow to send batch of data from source without answer nor action of target.
This don’t care if the data is lost on the way, it only send once.
having a three states field send on source
Step-by-step
you add some data in source, with their send by default on 0
you set every send ==0 on send = -1
select every -1 and send them to the target
update -1 to 1
Go back to step one.
This allow you big batch without having to put a lock write waiting the send script, and be sure you can't have some line that drop between the send one.
Timestamp
This look a lot like the previous one, but instead of a field on every row, we just use a new table to keep the last sync :
Step-by-step
you add some data in source, with their timestamp
you get the current timestamp - 1 (one second before now)
you get the last sync timestamp (or 0 if it's your first sync)
select and send lines where timestampOfPost <= timestamp-1 and timestampOfPost > timestampLastSync
update your last sync timestamp with the timestamp - 1 of point 2.
Timestamp can be tricky if you don't use the "go 1seconde back in time" and keep it in a variable, as you can lose some update :
If you send at ***754(.1)s, every line from ***754(.2)s to (.9)s will be see as send as we have done the timestamp ***754 and will start the next send at ***755.