Is there a way to cache results of a mysql query manually to a txt file?
Ex:
$a=1;
$b=9;
$c=0;
$cache_filename = 'cached_results/'.md5("$a,$b,$c").'.txt';
if(!file_exists($cache_filename)){
$result = mysql_query("SELECT * FROM abc,def WHERE a=$a AND b=$b AND c=$c");
while($row = mysql_fetch_array($result)){
echo $row['name'];
}
// Write results on $row to the txt file for re-use
}else{
// Load results just like $row = mysql_fetch_array($result); from the txt file
}
The original query contains more WHEREs and joins that uses multiple tables.
So, Is this possible? If so, please explain.
Thank you,
pnm123
If you're sure that your data has a long time-to-live, you can certainly cache data by saving it temporarily to a text file.
if (!file_exists($cachefile)) {
// Save to cache
$query=mysql_query('SELECT * FROM ...');
while ($row=mysql_fetch_array($query))
$result[]=$row;
file_put_contents($cachefile,serialize($result),LOCK_EX);
}
else
// Retrieve from cache
$result=unserialize(file_get_contents($cachefile));
foreach ($result as $row)
echo $row['name'];
Although using APC, MemCache, or XCache would be a better alternative if you consider performance.
Related
I have a .csv file that is about 5mb (~45,000 rows). What I need to do is run through each row of the file and check if the ID in each line is already in a table in my database. If it is, I can delete that row from the file.
I did a good amount of research on the most memory efficient way to do this, so I've been using a method of writing lines that don't need to get deleted to a temporary file and then renaming that file as the original. Code below:
$file= fopen($filename, 'r');
$temp = fopen($tempFilename, 'w');
while(($row = fgetcsv($file)) != FALSE){
// id is the 7th value in the row
$id = $row[6];
// check table to see if id exists
$sql = "SELECT id FROM table WHERE id = $id";
$result = mysqli_query($conn, $sql);
// if id is in the database, skip to next row
if(mysqli_num_rows($result) > 0){
continue;
}
// else write line to temp file
fputcsv($temp, $row);
}
fclose($file);
fclose($temp);
// overwrite original file
rename($tempFilename, $filename);
Problem is, I'm running into a timeout while executing this bit of code. Anything I can do to make the code more efficient?
You fire a database query per line, aka 45.000 queries... that takes too much time.
Better you do a query before the loop and read the existing id into a lookup array, then only check this array in the loop.
Pseudo code:
$st = query('SELECT id FROM table');
while ($row = $st->fetch()) {
$lookup[ $row['id'] ] = $row['id'];
}
// now read CSV
while($row = fgetcsv($h)) {
$id = $row[6];
if (isset($lookup[ $id ])) {
// exist...
continue;
}
// write the non-existing id to different file...
}
edit:
Assume memory isn't sufficient to hold 1 million integer from the database. How can it still be done efficiently?
Collect ids from CSV into an array. Write a single query to find all those ids in the database and collect (it can be maximal so many as in the CSV). Now array_diff() the ids from file with the ids from database - those ids remaining exist in CSV but not in database.
Pseudo code:
$ids_csv = [];
while($row = fgetcsv($h)) {
$id = row[6];
$ids_csv[] = intval($id);
}
$sql = sprintf('SELECT id FROM table WHERE id IN(%s)', implode(',', $ids_csv));
$ids_db = [];
$st = query($sql);
while ($row = $st->fetch()) {
$ids_db[] = $row['id'];
}
$missing_in_db = array_diff($ids_csv, $ids_db);
I would use LOAD DATA INFILE: https://dev.mysql.com/doc/refman/8.0/en/load-data.html
Your database user needs to have FILE priveleges on the database to use.
to read the csv file into a separate table.
Then you can run one query to delete id's already exist (delete from join ...)
And export the rows that were left intact.
Other option is use your loop to insert your csv file into a seperate table, and then proceed with step 2.
Update: I use LOAD DATA INFILE with csv files up to 2 million rows (at the moment) and do some bulk data manipulation with big queries, it's blazingly fast and I would recommend this route for files containing > 100k lines.
I need to select multiple comments (if there are any) based on the photo_id. As I understand it you can use the WHERE clause but I'm not exactly sure how to select multiple ones and store them in some kind of array?
e.g.
$result = mysqli_query($conn,"SELECT * FROM comments WHERE photo_id='$photo1id'");
$row = $result->fetch_assoc(); // but there's more than 1 row
If for example $photo1id == 21, how do I get all the comments (2 in this case)? Some kind of while loop?
At the end of the PHP file I have this:
echo json_encode(array('photo1id'=>$photo1id));
I need to store each row in that array somehow because I need to retrieve the data in another PHP file using $.getJSON. Or perhaps there is a better solution to this.
Loop through it and generate an array -
while($row = $result->fetch_assoc()) {
$comments[] = $row;
}
After that you can send the array as json.
echo json_encode($comments);
Is there is more rows, you need to use a loop.
while ($row = $result->fetch_assoc()) {
// your code here
}
Try the code below:
//Run query
$result = mysqli_query($conn,"SELECT * FROM comments WHERE photo_id='$photo1id'");
//While there is a result, fetch it
while($row = $result->fetch_assoc()) {
//Do what you need to do with the comment
}
If you don't want to print the code straight away you can just create an array:
$x=0;
while($row = $result->fetch_assoc()) {
$comment[$x]=$row['comment'];
$x++;
}
There are different ways, how to fetch and print MySQL data with PHP.
For example, you can fetch data row by row with PHP loop:
$result = $mysqli->query("SELECT * FROM `table`");
while($data = $result->fetch_assoc())
{
echo '<div>'. $data["field"] .'</div>';
}
Also, you can store all selected data into array, and then go through it:
$result = $mysqli->query("SELECT * FROM `table`");
$data = $result->fetch_all(MYSQLI_ASSOC);
foreach($data as $i => $array)
{
echo '<div>'. $array["field"] .'</div>';
}
Is there any serious reason, why I should use one method instead of the other? And what about performance in case of enermous databases?
In the first example while($data = $result->fetch_assoc()) you call the fetch_assoc function for each time you perform a loop. Second one just calls the fetch_all once, stores the data in an array and later just use it. So, in theory the second approach should be faster, however you'd better to do a simple benchmark to make sure.
I have slow query problem, may be i am wrong, here is what i want,
i have to display more than 40 drop down lists at a single page with same fields , fetched by db, but i feel that the query takes much time to execute and also use more resources..
here is an example...
$sql_query = "SELECT * FROM tbl_name";
$rows = mysql_query($sql_query);
now i use while loop to print all records in that query in drop down list,
but i have to reprint same record in next drop down list up to 40 lists, so i use
mysql_data_seek() to move to first record and then reprint the next list and so on till 40 lists.
but this was seems slow to me so i use the second method like this same query for all 40 lists
$sql_query2 = "SELECT * FROM tbl_name";
$rows2 = mysql_query($sql_query2);
do you think that i wrong about the speed of query, or do you suggest me the another way that is faster than these methods....
Try putting the rows into an array like so:
<?php
$rows = array();
$fetch_rows = mysql_query("SELECT * FROM table");
while ($row = mysql_fetch_assoc($fetch_rows)) {
$rows[] = $row;
}
Then just use the $rows array in a foreach ($rows as $row) loop.
There is considerable processing overhead associated with fetching rows from a MySQL result resource. Typically it would be quite a bit faster to store the results as an array in PHP rather than to query and fetch the same rowset again from the RDBMS.
$rowset = array();
$result = mysql_query(...);
if ($result) {
while ($row = mysql_fetch_assoc($result)) {
// Append each fetched row onto $rowset
$rowset[] = $row;
}
}
If your query returns lots of rows (thousands or tens of thousands or millions) and you need to use all of them, you may reach memory limitations by storing all rows into an array in PHP. In that case it may be more memory-conservative to fetch rows individually from MySQL, but it will still probably be more CPU intensive.
Instead of printing the records, going back, and printing them again, put the records in one big string variable, then echo it for each dropdown.
$str = "";
while($row = mysql_fetch_assoc($rows)) {
// instead of echo...
$str .= [...];
}
// now for each dropdown
echo $str;
// will print all the rows.
I'm fairly new to PHP and MySQL (experienced with other languages). Basically I want to load data from a row relative to its id stated in the URL. Example, load the 3rd row when "index.php?id=3"
Heres what I've managed to do so far: http://pastie.org/1436865
I pretty sure this question has been asked a million times over, but I don't know what term to search far and have not been able to find anything so far :S
Thanks guys
The part id=3... is called query string and you can access it using the $_GET array (see PHP: Predefined Variables).
BTW: It does not make sense to load the 3rd row from a database table, because the rows in the database table do not have an ordering (so instead of a list of rows, a set of records is a more appropriate analogy in this case). Records are usually identified by a so called primary key and you have to make sure yourself, that your database tables have a primary key.
Or if you prefer array's instead of objects try something like this:
$query="SELECT * FROM table WHERE id=".(int)$_GET['id']." LIMIT 1";
$result=mysql_query($query);
while ($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
echo $row['column'];
}
or if you like indexed columns
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
echo $row[0];
}
or both (column names and indexes):
while ($row = mysql_fetch_array($result, MYSQL_BOTH)) {
echo $row[0].'-'.$row['column'];
}
You can even use this if you need only one row:
$result = mysql_query("SELECT * FROM table WHERE id=".(int)$_GET['id']);
$row = mysql_fetch_row($result);
echo $row[0];
Based on all these not-so-clear but working examples, you can make a function:
function getRow($query){
$res = mysql_query($query);
if (!$res) {
trigger_error(mysql_error." in ".$query);
return array();
}
$row = mysql_fetch_assoc($res));
if ($row) return $row;
return array();
}
then store it into some library file, then include this file into your script and call with just single line
$data = getRow("SELECT * FROM tablename WHERE id=".intval($_GET['id']));
Also note Oswald's note, it's very important. You can't and you shouldn't rely on the row's relative position as there is no position at all. A DB table is a heap, not ordered list.
Use certain unique field value to address certain row. That's the way to go
The simplest way is
$id = (int) $_GET['id'];;
$result = mysql_query("SELECT * FROM tablename WHERE id=".$id);
while($res = mysql_fetch_object($result)){
echo $res->columnname;
}
Of course, you should check input parameters, etc. This is only approach.