I am working on a project. I need to query a DB and write the result to a csv file. The result is going to be over 15,000 entries, (thats what the user wants). I am breaking up the results using the LIMIT because if I don't the DB will time out. I divide the query in what I call total_pages. Here is my code.
The big for loop, loops 19 times. The problems is that the code will go through the nested loop one time ( only 500 entries) then It does not go back in. I tried using null on the $results but no luck. Please help.
// using this for my LIMIT
$start_from = 0;
$sql = "select * from someplace where gender = '".$gender."'";
$rs_result = mysql_query($sql);
$total_records = mysql_num_rows($rs_result);
$total_pages = ceil($total_records / 500);
// open a file
//write x in file
file = fopen("./DataFile/gdownload.csv","w");
//write header to the file
$x = "Last Name,First Name,Primary_Name, ........ etc...... \n";
fwrite($file, $x);
for($count = 0; $count <= $total_pages; $count++)
{
$query = "SELECT *
FROM ptable
JOIN person_name ON ptable.Primary_Name = person_name.Primary_Name
WHERE gender = '$gender'
ORDER BY person_name.Lname ASC
LIMIT ".$start_from.", 500";
$result = mysql_query($query) or die(mysql_error());
$num_row = mysql_num_rows($result);
//print tables in rows
while($row = mysql_fetch_array($result))
{
$x="";
$x=$x.$row['Lname'].",";
$x=$x.$row['Fname'].",";
$x=$x.$row['Primary_Name'].",";
$x=$x.$row['asdf#'].",";
$x=$x.$row['qwer'].",";
$x=$x.$row['hjkl'].",";
$x=$x.$row['bnm,'].",";
$x=$x.$row['yui'].",";
$x=$x.$row['aaa'].",";
.....
fwrite($file, $x);
}// end nested while
$start_from+=500;
}// end for loop
fclose($file);
It could be a problem with your LIMIT condition. What you have seems OK to me but the fact that the rows are only being written on the first pass through makes me think $result is empty after the first pass.
Try changing
LIMIT ".$start_from.", 500";
to
LIMIT 500 OFFSET ".$start_from;
Also make sure the query actually returns more than 500 results.
On a different note, it's odd that the request would timeout on just 15,000 records.
Related
I'm developing one API in php to display data on android app from my database using JSON.
In my app I want to display 20 records first, after display again 20 records once user scroll to top.
I'm requesting the last id of the record from app to show next 20 records from last id.
Here is my code
<?php
$last_movie = 0;
$genre = $_REQUEST['genre'];
$last_movie = $_REQUEST['lastid'];
require_once("connect.php");
$myArray = array();
if($last_movie == 0)
{
$result = $conn->query("SELECT * FROM my_movies WHERE genre = '$genre' ORDER BY year DESC LIMIT 20");
}
else
{
$result = $conn->query("SELECT * FROM my_movies WHERE genre = '$genre' ORDER BY year LIMIT ".$last_movie.",20");
}
if ($result) {
while($row = $result->fetch_array(MYSQL_ASSOC)) {
$myArray[] = $row;
}
echo json_encode($myArray);
}
$result->close();
$conn->close();
?>
I'm getting values in some genres, but sometimes it show empty JSON.
I tried with this url
http://freemodedapk.com/bobmovies/by_genre.php?genre=Action
its working , whenever I try from last id
http://freemodedapk.com/bobmovies/by_genre.php?genre=Action&lastid=4714
It returns empty JSON. I have values in database.
But some genres working fine
http://freemodedapk.com/bobmovies/by_genre.php?genre=Drama
http://freemodedapk.com/bobmovies/by_genre.php?genre=Drama&lastid=865
I have total 4858 records in the database with all genres.
Anybody can help me to fix empty JSON problems in some of genres ?
Your main issue is in the wrong LIMIT usage: when you utilize 2 parameters for LIMIT keyword, the first one is for OFFSET value (not for IDs), the second one is to limit your result.
In short words: you should use LIMIT 0,20 to get the first 20 results, LIMIT 20,20 to show next 20 results and so on.
Also, your code is insecure - you have SQL injection. Try to not post direct urls on your sites with the source code which includes injection because some bad guys may drop your database or do some other harmful things.
Sample code is listed below (minor changes may be required):
<?php
require_once('connect.php');
$response = [];
$items_per_page = 20;
$page = (int) (($_REQUEST['page'] - 1) * $items_per_page);
$genre = $conn->escape_string($genre); # replace escape_string with proper method if necessary
$result = $conn->query("SELECT * FROM my_movies WHERE genre = '$genre' ORDER BY year DESC LIMIT $page,$items_per_page");
if ($result)
{
$response = $conn->fetch_all($result, MYSQLI_ASSOC); # replace fetch_all with proper method if necessary
}
echo json_encode($response);
$result->close();
$conn->close();
if you want to get last ID to ASC then use to
SELECT * FROM my_movies WHERE genre = '$genre' id<".$last_movie." ORDER BY year LIMIT 0,20
or if you want to get for pagination then your OFFSET value wrong,
you should be use LIMIT 0,20 to get the first 20 results, LIMIT 20,20 to next 20 , LIMIT 40,20
please check your SQL injection
look like Code
require_once('connect.php');
$result = [];
$limit = 20;
$pageNumber = (int) (($_REQUEST['pageNumber'] - 1) * $limit);
$genre = $conn->escape_string($genre);
$getDta = $conn->query("SELECT id,title,stream,trailer,directors,actors,quality,year,genre,length,translation,rating,description,poster FROM my_movies WHERE genre = '".$genre."' ORDER BY year DESC LIMIT $pageNumber,$limit");
if ($result)
$result =$conn->fetch_all($result, MYSQLI_ASSOC);
echo json_encode($result);
$result->close();
$conn->close();
I know how to encode all the data from an SQL query into JSON using PHP, but don't know How to encode a partial set.
In Android (the client), I send an HTTP request which goes something like this:
public interface DataAPI {
#GET("/top500")
public void getResult(#Query("start") int start, #Query("count") int count,
Callback<Top500> response);
}
As you can see, I am using start and count in Android code.
I have around 500 records in my table and I have successfully encoded all the data into JSON, using below PHP Script:
<?php
$con = mysqli_connect(HOST,USER,PASS,DB);
$sql = "select * from Persons";
$res = mysqli_query($con,$sql);
$result = array();
while($row = mysqli_fetch_array($res)){
array_push($result,
array(
'id'=>$row[0],
'name'=>$row[1]
));
}
echo json_encode(array("result"=>$result));
mysqli_close($con);
?>
JSON Result:
{"result": [{ ... }]}
But I need to encode mysql data 20 by 20 like this:
{"count": 20, "start": 0, "total": 500, "result": [{ ...}]}
What's missing is how to use the start and count parameters to get a partial set in order to get slices of the data.
You need to use LIMIT and OFFSET in your query. Applying some nasty logic in PHP is a bad solution. This is an operation that MySQL should do instead because you don't want to fetch all the rows if you don't need all of them.
If you run a query like this:
SELECT *
FROM Persons
ORDER BY Id
LIMIT 10 OFFSET 20
you will get a subset of the matching rows starting from the 20th and long 10 rows. You can then loop over the full results set.
Better to explicitly order by a field to ensure consistency across different pages.
You're final code (using PDO rather than mysqli):
$pdo = new PDO('mysql:dbname=db;host=127.0.0.1', $user, $password);
$count = $_GET['count'];
$start = $_GET['start'];
$stmt = $pdo->prepare('SELECT * FROM Persons ORDER BY Id LIMIT ? OFFSET ?');
$stmt->execute(array($count, $start));
$rows = $stmt->fetchAll();
foreach ($rows as $row) {
// your logic that was inside the while
}
Use below code.
$con = mysqli_connect(HOST,USER,PASS,DB);
$count = "select count(*) as total from Person"; //query to get row count of your table
$count_res = mysqli_query($con,$count);
$count_row = mysqli_fetch_array($count_res);
$total = $count_row['total']; // get total no of records
$start = YOUR_START_VALUE; // from which offset you want to get record
$end = YOUR_VALUE // how many record want to fetch
$sql = "select * from Persons limit $start, $end";
$res = mysqli_query($con,$sql);
$result = array();
while($row = mysqli_fetch_array($res)){
array_push($result,
array(
'id'=>$row[0],
'name'=>$row[1]
));
}
echo json_encode(array("count"=> $end, "start"=> $start, "total"=> $total,"result"=>$result));
mysqli_close($con);
?>
How about something like:
$start = 7;
$size = 3;
$count = 0;
while($row = mysqli_fetch_array($res)){
if($count>=$start && $count <$start+$size) {
array_push($result,array('id'=>$row[0],'name'=>$row[1]));
}
$count++;
}
The code should be self explanatory, but if you have any questions, feel free to comment.
Note: This is more of a PHP pseudo code, as I don't have an environment and haven't coded in PHP in a while.
I can think of 2 ways to solve this problem.
The first is using just MySQL:
SELECT SQL_CALC_FOUND_ROWS * FROM Persons LIMIT 0,20;
SELECT FOUND_ROWS();
With the first query you retrieve the batch of rows you want. With the second you get the total row count in the result set.
The second option is to use array_slice:
$total = count($result);
$batch = array_slice($result, 0, 20);
echo json_encode(array("total" => $total, "start" => 0, "count" => count($batch), "result"=>$batch));
I have a table with around 100000 records ,the structure is shown below
id | name | desc | length | breadth | -------------|remark //upt o 56 fields
1 FRT-100 -desc- 10 10 remarking
----------------------------------------------------------------
----------------------------------------------------------------
what iam doing this is using a cronjob(Gearman) to write all these data to a csv ,my code is given below
<?php
set_time_limit (0);
ini_set("memory_limit","2048M");
//get the total count of records,so that we can loop it in small chunks
$query = "SELECT COUNT(*) AS cnt FROM tablename WHERE company_id = $companyid";
$result = $link->query($query);
$count = 0;
while ($row = mysqli_fetch_array($result)) {
$count = $row["cnt"];
}
if ($count > 1000) {
$loop = ceil($count / 1000);
} else {
$loop = 1;
}
// im going to write it in small chunks of 1000's each time to avoid time out
for ($ii = 1; $ii <= $loop; $ii++) {
if ($ii == 1) {
$s = 1;
} else {
$s = floatval(($ii * 1000) - 1000);
}
$q = "SELECT * FROM datas WHERE group_company_id = $companyid LIMIT 1000 OFFSET $s";
$r = $link->query($q);
while ($row2 = mysqli_fetch_array($r)) {
//my csv writing will be done here and its working fine for records up to 10,000 ~ 12,000 after than memory exhaustion occours
}
}
?>
I strongly suspects something can be optimised in the offset function of mysql .Can someone show me a better way to optimise it ? open to any suggestions (CRON,third party libraries ..etc)
Try and avoid storing everything in memory at once, instead load each row, then write out result one row at a time.
<?php
$q = "SELECT * FROM datas";
$r = $link->query($q);
$fp = fopen("out.csv","w+");
// Or you could just set the headers for content type, and echo the output
while ($row2 = mysqli_fetch_array($r)) {
fwrite($fp, implode(",",$row2)."\n");
}
fclose($fp);
This should solve the issue, nothing is being stored in memory.
I am writing a web application in PHP which will store employee data and generate employee ID cards to PDF. I am using FPDF for creation of PDFs and that works fine. I am having a problem with showing results from MySQL database.
I have to generate PDF with 4 employee ID cards and I am not sure how to get them from the database. So far I am using LIMIT option in the query to get only 4 results and i will have an if statement based on mysql.php?id=1 id which will define the limit. It is a little messy but there are not going to be more than 80 employees.
This is my code:
$id = $_GET['id'];
if ($id == 1) {
$limit_start = 0;
$limit_end = 4;
}
$result=mysql_query("SELECT users.tajemnik, users.dateCreated, users.showmeID,
users.workerName, users.dateCreated, users.workerPlace, users.workerSID, uploads.userID, uploads.data, uploads.filetype
FROM users INNER JOIN uploads ON users.showmeID = uploads.userID ORDER BY workerName DESC LIMIT $limit_start, $limit_end") or die (mysql_error());
mysql_query("SET NAMES 'utf8'") or die('Spojení se nezdařilo');
while($row = mysql_fetch_array($result)){
$workerName = $row["workerName"];
$workerPlace = $row["workerPlace"];
$workerSID = $row["workerSID"];
$tajemnik = $row["tajemnik"];
$showmeID = $row["showmeID"];
$mysqldatetime = strtotime($row['dateCreated']);
$image = $row["data"];
$phpdatetime = date("d.m.Y",$mysqldatetime);
}
This will get me the first result from the query. I need to get information from all 4 rows and have them stored in variables like $workerName1, $workerName2 etc. I hope it makes sense what I am trying to do.
Thank you for your replies!
V.
I need to get variables like $workerName1, $workerName2 etc
Nope, you don't.
You actually need an array.
So, first, get yourself a function
function sqlArr($sql){
$ret = array();
$res = mysql_query($sql) or trigger_error(mysql_error()." ".$sql);
if ($res) {
while($row = mysql_fetch_array($res)){
$ret[] = $row;
}
}
return $ret;
}
then write a code
mysql_query("SET NAMES 'utf8'") or die('Spojení se nezdařilo');
$sql = "SELECT users.tajemnik, users.dateCreated, users.showmeID, users.workerName,
users.dateCreated, users.workerPlace, users.workerSID, uploads.userID,
uploads.data, uploads.filetype
FROM users
INNER JOIN uploads ON users.showmeID = uploads.userID
ORDER BY workerName DESC LIMIT $limit_start, $limit_end";
$data = sqlArr($sql);
Now you have all your data in the $data array.
So, you can loop over it or access single values like
echo $data[0]['tajemnik'];
or
foreach($data as $row) {
//do whatever you want with database row
echo $row['tajemnik'];
}
First, I coded this, which looks inside a table, gets the last 10 entries, and displays them. The output is as expected, a list of the 10 last entries in the database.
$query = "SELECT dfid FROM downloads_downloads ORDER BY did DESC limit 10";
$dlresult = mysql_query( $query );
$i=0;
$num = mysql_num_rows ($dlresult);
while ($i < $num) {
$dfid= mysql_result($dlresult,$i,"dfid");
echo "<b>filenumber:</b> $dfid <br>";
++$i;
}
But I don't just need the filenumber. I need the actual filename and url from another table. So I added a select statement inside the while statement, using the file number.
But for some reason, this code only displays one filename instead of 10. I know, from the above code, it's getting all 10 file numbers.
$query = "SELECT dfid FROM downloads_downloads ORDER BY did DESC limit 10";
$dlresult = mysql_query( $query );
$i=0;
$num = mysql_num_rows ($dlresult);
while ($i < $num) {
$dfid= mysql_result($dlresult,$i,"dfid");
$query2 = "SELECT file_name, file_name_furl FROM downloads_files WHERE file_id = '$dfid'";
$dlresult2 = mysql_query( $query2 );
$dlfile_name= mysql_result($dlresult2,$i,"file_name");
$dlfile_name_furl= mysql_result($dlresult2,$i,"file_name_furl");
echo "filenumber: $dfid <br>"; //Shows 10, as expected.
echo "filename: $dlfile_name - $dlfile_name_furl <br>"; //Shows only 1?
++$i;
}
I can manually execute the sql statement and retrieve the file_name and file_name_furl from the table. So the data is right. PHP isn't liking the select within the while statement?
Looks like you're only going to ever have 1 row, in your 2nd select statement, because you are just selecting one row, with your where statement.
So, you're only going to ever have row 0 in the 2nd statement, so its only finding row 0 on the first loop. try instead:
$dlfile_name= mysql_result($dlresult2,0,"file_name");
$dlfile_name_furl= mysql_result($dlresult2,0,"file_name_furl");
However, insrtead of making 11 separate queries, try using just one:
$link = new mysqli(1,2,3,4);
$query = "SELECT downloads_downloads.dfid, downloads_files.file_name, downloads_files.file_name_furl FROM downloads_downloads LEFT OUTER JOIN downloads_files ON downloads_files.file_id = downloads_downloads.dfid ORDER BY downloads_downloads.dfid DESC limit 10;
$result = $link->query($query) ;
if((isset($result->num_rows)) && ($result->num_rows != '')) {
while ($row = $result->fetch_assoc()) {
echo "filenumber: $row['dfid'] <br>";
echo "filename: $row['file_name'] - $row['file_name_furl'] <br>";
}
Read up on mysql joins http://www.keithjbrown.co.uk/vworks/mysql/mysql_p5.php
I'm not sure if this is syntax correct, but it gives you the right idea =)