I have the following data in php:
ID & Timestamps for each blog
Then I have a table which contains:
| id | blog | timestamp |
I need to find out which of my php timestamp data does not equal the timestamp in the mysql database.
Sure I could run a hundred queries for each and every timestamp I have in php to look if it is changed, but that seems inefficient.
Is there any way to put a single query in Mysql to get the data?
To clarify, my php ID & timestamps are in an array like
Array ( [3] => Array ( [timestamp] => 1389414084 ) )
I need to check in mysql if the record with ID=3 has the timestamp of 1389414084.
Problem is, with a lot of data to check this would end up being a huge number of queries. And I only need the ones that do not match.
I have no idea how to go about this so any help would be very appreciated.
To clarify - the first data is json. That's how it has to be. I decode it and then have to match it up to the blogs timestamp.
What I would suggest is make hash of ID, Blog & Timestamp and store it in the table itself. And instead of checking id & timestamp, you may check for the hash match (select * from my BlogTimeStamp where hash NOT IN ('hash1', 'hash2', ....).
I don't know your exact requirement. maybe there would be a better solution altogether
SELECT *
FROM myBlogTable
WHERE timestamp NOT IN (1389414084);
If you use a select statement to get the 1389414084 then insert that in the parentheses instead of hard coding in the timestamp so that the query is dynamic if you ever need it to be.
Here's what I did in the end. I just broke down the query and constructed it so that I had everything in one query.
SELECT * FROM title WHERE id IN((SELECT id FROM title WHERE id = '75' AND date ='1392501995'),(SELECT id FROM title WHERE id = '74' AND date ='1392401481'))
So you just create it in php.
$i = count($fav);
$sql = 'SELECT * FROM title WHERE id IN(';
foreach($fav as $key => $val){
$sql .= "(SELECT id FROM title WHERE id = '".$key."' AND date ='".$val["date"]."')";
if($i > 1) {
$sql .= ",";
}
$i = $i-1;
}
$sql .= ")";
Related
I have run into a delayed processing time for a PHP program,
I have a MySQL record with over 1000 tables;
Each table is created once a new device is added, e.g assets_data_imeixx - to assets_data_imeixx1000th table
Each table contains about 45,000 rows of records inserted every 10 seconds,
Below is my PHP code to query the database and fetch all these records based on datetime.
Issue: The program executes without error but it takes about 1.3minutes to 4mins for very large records.
PHP Code:
$ms = mysqli connection string in config.php //$ms is OKAY
$user_id = '5';
$q = "SELECT * FROM `user_assets` WHERE `user`='".$user_id ."' ORDER BY `imei` ASC";
$r = mysqli_query($ms,$q);
$result = array(); //$result array to contain all data
while($row =mysqli_fetch_array($r)){
//fetch 7 days record
for ($i=1; $i < 7; $i++) {
$date = "-" . $i . " days";
$days_ago = date('Y-m-d', strtotime($date, strtotime('today')));
$sql1 = "SELECT * FROM assets_data_" . $row["imei"] . " WHERE dt_time LIKE '" . $days_ago . "%' LIMIT 1"; // its correct
//$result1 = $conn->query($sql1);
$result1 = mysqli_query($ms,$sql1);
$row2 = mysqli_fetch_array($result1);
echo $row['imei']." ".$row2['dt_server']."<br/>";
}
}
Above code fetches over 1000 devices from user_assets table, These IMEI each has its own table that contains over 45,000 records in each table of location data.
The for loop iterates over each IMEI table and records.
Above code runs without error but take so much time to complete, I want to find a solution to optimize and have code execute in a very short time max 5 seconds.
I need help and suggestions on optimizing and running this large scale of data and iteration.
(from Comment)
CREATE TABLE gs_object_data_863844052008346 (
dt_server datetime NOT NULL,
dt_tracker datetime NOT NULL,
lat double DEFAULT NULL,
lng double DEFAULT NULL,
altitude double DEFAULT NULL,
angle double DEFAULT NULL,
speed double...
(From Comment)
gs_object_data_072101424612
gs_object_data_072101425049
gs_object_data_072101425486
gs_object_data_072101445153
gs_object_data_111111111111111
gs_object_data_1234567894
gs_object_data_222222222222222
gs_object_data_2716325849
gs_object_data_2716345818
gs_object_data_30090515907
gs_object_data_3009072323
gs_object_data_3009073758
gs_object_data_352093088838221
gs_object_data_352093088839310
gs_object_data_352093088840045
gs_object_data_352121088128697
gs_object_data_352121088132681
gs_object_data_352621109438959
gs_object_data_352621109440203
gs_object_data_352625694095355
gs_object_data_352672102822186
gs_object_data_352672103490900
gs_object_data_352672103490975
gs_object_data_352672103490991
gs_object_data_352887074794052
gs_object_data_352887074794102
gs_object_data_352887074794193
gs_object_data_352887074794417
gs_object_data_352887074794425
gs_object_data_352887074794433
gs_object_data_352887074794441
gs_object_data_352887074794458
gs_object_data_352887074794474
gs_object_data_352887074813696
gs_object_data_352887074813712
gs_object_data_352887074813720
gs_object_data_352887074813753
gs_object_data_352887074813761
gs_object_data_352887074813803
900+ tables each having different location data.
Requirement: Loop through each table, fetch data for selected date range say:
"SELECT dt_server FROM gs_object_data_" . $row["imei"] . " WHERE dt_server BETWEEN '2022-02-05 00:00:00' AND '2022-02-12 00:00:00'";
Expected Result: Return result set containing data from each table containing information for the selected date range. That means having 1000 tables will have to be looped through each table and also fetch data in each table.
I agree with KIKO -- 1 table not 1000. But, if I understand the rest, there are really 2 or 3 main tables.
Looking at your PHP -- It is often inefficient to look up one list, then go into a loop to find more. The better way (perhaps 10 times as fast) is to have a single SELECT with a JOIN to do both selects at once.
Consider some variation of this MySQL syntax; it may avoid most of the PHP code relating to $days_ago:
CURDATE() - INTERVAL 3 DAY
After also merging the Selects, this gives you the rows for the last 7 days:
WHERE date >= CURDATE() - INTERVAL 7 DAY
(I did not understand the need for LIMIT 1; please explain.)
Yes, you can use DATETIME values as strings, but try not to. Usually DateTime functions are more efficient.
Consider "composite" indexes:
INDEX(imei, dt)
which will be very efficient for
WHERE imei = $imei
AND dt >= CURDATE() - INTERVAL 7 DAY
I would ponder ways to have less redundancy in the output; but that should mostly be done after fetching the raw data from the table(s).
Turn on the SlowLog with a low value of long_query_time; it will help you locate the worst query; then we can focus on it.
An IMEI is up to 17 characters, always digits? If you are not already using this, I suggest BIGINT since it will occupy only 8 bytes.
For further discussion, please provide SHOW CREATE TABLE for each of the main tables.
Since all those 1000 tables are the same it would make sense to put all that data into 1 table. Then partition that table on date, use proper indexes, and optimize the query.
See: Normalization of Database
Since you limit results to one user, and one row per device, it should be possible to execute a query in well below one second.
Well guys i have this query
$mysql = "select * from xxx where active = 1 order by Rand() limit $start,12";
mysql_query($mysql);
Everything works great so far.
I want: when i am pressing the next button (page 2 or three etc) to see the next 12 random records but do not display the first 12 random records that i had in my previus page!
Thank you all!
p.s Sorry guys for my bad english!
Just try to retrieve the data you need in an array, randomize it with shuffle() in PHP, and paginate the result with some JQuery, it will be awesome, just one query and no refresh. ;)
You need to keep one array (e.g $arrRecordIds) to track all the id's of records shown on previous pages.
When you are on first page:
$arrRecordIds=array(); // Empty array
When you are on second page:
$arrRecordIds=array_merge($arrRecordIds, $arrNewRecordIds);array_unique( $arrRecordIds );
If your select query simply concat- where id NOT IN ( implode(',', $arrRecordIds ) )
Here $arrNewRecordIds should contains id's of the records on the page.
You can keep track of the previously shown records' ids and put them in an array.
In your query use id NOT IN (array)
Apply the concept of Systematic Random Sampling,
Number the records N, decide on the n (pagination size, eg: 10, 20)
(sample size) that you want or need k = N/n = the interval size
Randomly select an integer between 1 to k then take every k th unit
Refer: http://www.socialresearchmethods.net/kb/sampprob.php
Try using the following script in your showdata.php file
$per_page = 12;
$sqlc = "show columns from coupons";
$rsdc = mysql_query($sqlc);
$cols = mysql_num_rows($rsdc);
$page = $_REQUEST['page'];
$start = ($page-1)*12;
$N = 1000; //Total rows in your table (query to get it dynamically)
$n = $per_page;
$k = ceil($N/$n);
$range[] = $page;
for($i=1;$i<$n;$i++) {
$range[] = ($page+$k)*$i;
}
$sqln = "SELECT * FROM ( SELECT #rownum:= #rownum+1 AS rindex, n.* FROM xxx n, (SELECT #rownum := 0) r ) AS rows WHERE rindex IN (".implode(',',$range).")";
$rsd = mysql_query($sqln);
SOLUTION - that works a treat.
do a select random search of all required records
generate a random user-id eg. "smith".rand(1000,10000)
form a string of all random keys upto required no of records per page
insert above in a table/field containing a corresponding page no.
repeat/loop above until no more pages/recs remaining - use array_splice(str,from,to) - then use $notscreen = print_r($splice, true) for string storage to table -> randompages:
tb.rec-no | user-id | pageno | string ( with keys of recs upto recs/page)
122 | aj7894 | p1 | [0]=>100[1]=>400[2]=>056[3]=>129
123 | aj7894 | p2 | [x]99=>[x]240=>[x]7895[x]458=>320
... upto whole array of pages /no of records / all pages - no duplication of data - only 1-column of key of recs stored in random as retrieved
use user-id & pageno with WHERE to pull out random keys for that individual user & page
convert string back to array and pull out matching key recs for specific pages using the array in a SELECT WHERE query with implode
re-circ [ user-id & pageno ] using $_GET/POST for duration of search/view - reinitialise when new view or new search commences
notes:
-better to use list for search - but requires more work to format string - should give random page results as originally stored
problem with array matching is it orders records per page; lowest being first - not so random for the page display
temp table no good - because cannot be accessed when script is thrown back to server for 2nd and more time - it's lost from memory by mysql
php rules - no flimsy cookies or java-script !
BIG PROBLEM - SOLVED.
re-compsense for help received from your posts / answers.
Happy Days!
I try to build a variable that integrates some other variable.
one of that will be the number of an auto-increment-field where later on an insert-query will happens.
I tried to use:
$get_num = $db/*=>mysqli*/->query("SELECT COUNT (*) auto_increment_column FROM table1");
$num = $query->fetch_assoc($get_num);
$end = $num + 1;
I don't have any update/insert query before that so I can't use
$end = $db->insert_id;
that's why i thought i can just count the numbers of the auto_increment rows and have my last variable that is necessary to build my new variable.
for a reason this wonT count the entries and outputs 0. i dont understand why this happens.
i really would appreciate if there is someone who could tell me what am i doing wrong. thanks a lot.
UPDATE
For everyone who likes to know about what's the goal:
I like to create a specific name or id for a file that later on will be created by the input of the fields from the insert query. I like to have an unique key. this key consists of an user_id and a timestamp. at the end of this generated variable it should be placed the auto_increment nr. of the query that will be placed in the table. so the problem is, that I create an variable before the insert query happens so that this variable will be part of the insert query like:
$get_num = $db->query("SELECT COUNT (*) FROM tableA");
$num = $query->fetch_assoc();
$end = $num + 1;
$file_id = $id .".". time() .".". $end;
$insert = $db->query("INSERT INTO tableA ( file_id, a, b, c) VALUES('".$file_id."','".$a."','".$b."','".c."')");{
hope now, it will be clear what I like to approach.
If you need an auto-incrementing column in MySQL then you should use AUTO_INCREMENT. It implements it all for you and avoids race conditions. The manual way you are trying to implement it has a couple of flaws, namely
If two scripts are trying to insert concurrently they might both get the same COUNT (say 10) and hence both try to insert with ID 11. One will then fail (or else you will have duplicates!)
If you add 10 items but then delete item 1, the COUNT will return 9 but id 10 will already exist.
try
SELECT COUNT(*) FROM table1
I have a PHP script which users submit a post and saves in to the database; I'm storing the date it was posted using time(). I'm trying to figure out a way to filter my data by day. I have a while loop which is returning all the posts that user made which is ordered by date, however what I'm looking to do is have headings of a date for e.g. 10 December 2011 and only have posts that was submitted on that day. I want it only to display the heading if there is data for that day.
Thanks a lot.
EDIT:
Here is my while loop at the moment:
$get_posts = mysql_query( "SELECT * FROM `posts` WHERE `userid` = '$name[id]' ORDER BY `date` DESC" );
while( $userpost = mysql_fetch_assoc( $get_posts ) )
{
echo $userpost[ 'post' ];
}
But somehow before the loop I need to display the date heading the post or posts was submitted. I'm thinking it would need to be a loop outside of the current loop but I have no idea how to go about it.
As you have mentioned in one of your comments to the question that the DB column is of type integer. And so I assume you are storing the UNIX timestamp in the column. From what I understand, you are looking to display something like:
DATE HEADING 1
- data
- data
DATE HEADING 2
- data
- data
And if that is correct, this code snippet might help:
$rs = mysql_query('SELECT `column1`, `column2`, FROM_UNIXTIME(`the_time_column`, \'%d %M %Y\') AS `date_value`
FROM `your_table`
ORDER BY `the_time_column`');
$old = '';
while ($row = mysql_fetch_assoc($rs)) {
$new = $row['date_value'];
if ($new != $old) { // marks the beginning of a new heading
echo 'heading = ' . $new;
}
// start displaying data under the heading
echo $row['column1'];
echo $row['column2'];
// finish displaying data
$old = $new;
}
Please note that I've used dummy column and table names, so please replace them with the actuals. Also, the above snippet will display heading in ascending order of date. To show in reverse, add DESC at the end of the query.
Hope the above helps!
You get that right from the database, i.e.:
SELECT * FROM YourTable
WHERE
DAY(yourDateField)='15' AND
MONTH(yourDateField)='10' AND
YEAR(yourDateField)='2011'
;
or better yet, directly, like this:
SELECT * FROM YourTable
WHERE yourDateFieldld=2011-12-10;
Depends how you store the data in your DB.
In any case, this should be done in the database, not PHP
You could use BETWEEN.
Have a look at The MySQL Reference
I have a course page, this page is setup to display the details (easy), then who is teaching it first, second, third, and forth period of each semester. The problem with that is, my teachers data is handled in one giant chain.
Bobby: 1-1-1-1-1-1-1-1
Tina: 20-20-20-10-1-1-1-1
Joey: 20-1-1-1-49-432-10-19
What I want to do is find a course: 20, when all of the information is shown a search through teachers would be made to find out who is teaching course 20 and what period?
Ex.
Course id: 20
Period 1: Joey, Tina
Period 2: Tina
Period 3: Tina
I want to get every teacher teaching that course in one search but given the obstacle of the capact data which may be a problem.
Teacher Table:
id / name / link / course (1-1-1-1-1-1-1-1)
Course Table:
id / name / code / grade / level
Teachers Course contains the id's from course Table
While re-structuring your db might be the best answer, I thought I'd post a straight php solution that works with your current structure and presumptively the rest of your code.
//I set up variables to contain your expected search results so I could test
$search_course = '20';
$search_results = array(
'Bobby' => '1-1-1-1-1-1-1-1',
'Tina' => '20-20-20-10-1-1-1-1',
'Joey' => '20-1-1-1-49-432-10-19'
);
//explode the course strings into arrays and store with teacher names so
//you can loop through them later
foreach($search_results as $teacher=>$string_courses){
$array_courses = explode('-',$string_courses);
$search_results[$teacher] = $array_courses;
}
//Match course you are searching for to the elements in your array
//Create a result array with period and matching teachers
foreach($search_results as $teacher=>$courses){
foreach($courses as $period => $course){
if($course == $search_course){
$results[$period][] = $teacher;
}
}
}
//Loop through your result array and show the results
//I expect you'll have different html for this
foreach($results as $period => $teachers){
echo 'Period: ';
echo $period+1;
echo implode(',',$teachers);
echo '<br>';
}
The printed results match the list you wanted in your OP
Create a new table
Something like this:
CREATE TABLE `TeacherToPeriod` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`TeacherID` INT NOT NULL ,
`CourseID` INT NOT NULL ,
`Period` INT NOT NULL
) ENGINE = MYISAM ;
Insert the data
Here is some PHP-Code:
foreach($lines as $line){
$line_data = split($line, ': ');
$teacher = $line_data[0];
// SELECT your $teacher_id from the database
$courses = split($line_data[0], '-');
$i = 0;
foreach($courses as $course_id){
$i++;
$sql = "INSERT INTO `TeacherToPeriod` (`TeacherID` ,`CourseID` ,`Period`) ";
$sql.= "VALUES ($teacher_id, $course_id, $i);"
mysql_query($sql);
}
}
Select the data you want
SELECT * FROM `TeacherToPeriod` WHERE `CourseID` = 20 ORDER BY `Period` ASC;
You should change the structure of your db, instead of storing a string of all the periods, you should have an additional table with three columns: teacher,course,period and have a separate row in this table for each course that a teacher is teaching. Then determining who is teaching what course would simply be a matter of querying that table by course id and then sorting by period. e.g:
SELECT teacher_id, course_id, period FROM course_info WHERE course_id = 20
ORDER BY period;