I'm making a query over a database with over 20MM entries, that means im breaking the query into several smaller queries.
The problem is if I try to fetch the 20MM entries the page does not load, and displays a blank screen, with no title and content. However, if I fetch 5MM entries, the page does load correctly, and displays the content:
Here's my code
for($n=0; $n<20000000; $n=$n+500000){
$m=500000;
$query = "SELECT * FROM user_likes LIMIT ". $n .",". $m;
//echo $query;
$result = mysql_query($query) or die(mysql_error());
// craete arrays
while($row = mysql_fetch_array($result)){
set_time_limit(0);
$like[$row['name']]=$like[$row['name']]+1;
if($like[$row['name']]==375) $likes375 ++;
}
}
// print the size
echo count($like)."<br>";
echo "375: ".$likes375;
I would appreciate if someone can help me with this.
Thanks
EDIT:
after adding error_reporting(E_ALL); it display this notice: MySQL server has gone away
The reason for that is because your page will be blank while MySQL is crunching through its index looking for your results. After a while, this will time out (depending on your settings). That is why this doesn't happen on smaller queries.
This is due to time limit of execution of php scripts, by default php script will run for a total of 45 seconds before timing out...
Just use this after <?php
set_time_limit(0); // 0 means unlimited
but it is not recommended to use it, better improve your logic at the both database and php code logic.
EDIT:
try to set error log for this file specificaly by doing this:
ini_set("log_errors" , "1");
ini_set("error_log" , "eorrors.txt");
ini_set("display_errors" , "0");
Related
I have a file that has the function of importing data into a sql database from an api. A problem I encountered was that the api can only retrieve a max dataset size of 1000, even though sometimes I need to retrieve large amounts of data, ranging from 10-200,000. My first thought was to create a while loop in which inside I make calls to the api until all of the data is properly retrieved, and afterwards, can I enter it into the database.
$moreDataToImport = true;
$lastId = null;
$query = '';
while ($moreDataToImport) {
$result = json_decode(callToApi($lastId));
$query .= formatResult($result);
$moreDataToImport = !empty($result['dataNotExported']);
$lastId = getLastId($result['customers']);
}
mysqli_multi_query($con, $query);
The issue I encountered with this is that I was quickly reaching memory limits. The easy solution to this is to simply increase the memory limit until it was suffice. How much memory I needed, however, was undeclared, because there is always a possibility that I need to import very large datasets, and can theoretically always run out of memory. I don't want to set an infinite memory limit, as the problems with that are unimaginable.
My second solution to this was instead of looping through the imported data, I could instead send it to my database, and then do a page refresh, with a get request specifying the last Id I left off on.
if (isset($_GET['lastId'])
$lastId = $_GET['lastId'];
else
$lastId = null;
$result = json_decode(callToApi($lastId));
$query .= formatResult($result);
mysqli_multi_query($con, $query);
if (!empty($result['dataNotExported'])) {
header('Location: ./page.php?lastId='.getLastId($result['customers']));
}
This solution solves my memory limit issue, however now I have another issue, being that browsers, after 20 redirects (depends on the browser), will automatically kill the program to stop a potential redirect loop, then shortly refresh the page. The solution to this would be to kill the program yourself at the 20th redirect and allow it to do a page refresh, continuing the process.
if (isset($_GET['redirects'])) {
$redirects = $_GET['redirects'];
if ($redirects == '20') {
if ($lastId == null) {
header("Location: ./page.php?redirects=2");
}
else {
header("Location: ./page.php?lastId=$lastId&redirects=2");
}
exit;
}
}
else
$redirects = '1';
Though this solves my issues, I am afraid this is more impractical than other solutions, as there must be a better way to do this. Is this, or the issue of possibly running out of memory my only two choices? And if so, is one more efficient/orthodox than the other?
Do the insert query inside the loop that fetches each page from the API, rather than concatenating all the queries.
$moreDataToImport = true;
$lastId = null;
$query = '';
while ($moreDataToImport) {
$result = json_decode(callToApi($lastId));
$query = formatResult($result);
mysqli_query($con, $query);
$moreDataToImport = !empty($result['dataNotExported']);
$lastId = getLastId($result['customers']);
}
Page your work. Break it up into smaller chunks that will be below your memory limit.
If the API only returns 1000 at a time, then only process 1000 at a time in a loop. In each iteration of the loop you'll query the API, process the data, and store it. Then, on the next iteration, you'll be using the same variables so your memory won't skyrocket.
A couple things to consider:
If this becomes a long running script, you may hit the default script running time limit - so you'll have to extend that with set_time_limit().
Some browsers will consider scripts that run too long to be timed out and will show the appropriate error message.
For processing upwards of 200,000 pieces of data from an API, I think the best solution is to not make this work dependant on a page load. If possible, I'd put this in a cron job to be run by the server on a regular schedule.
If the dataset is dependant on the request (for example, if you're processing temperatures from one of 1000s of weather stations - the specific station ID to be set by the user), then consider creating a secondary script that does the work. Calling and forking the secondary script from your primary script will enable your primary script to finish execution while your secondary script executes in the background on your server. Something like:
exec('php path/to/secondary-script.php > /dev/null &');
I've faced a problem that I can't solve. Thing I want to do, to "save" all output SQL data to file (which should renew at timed interval ex. 5 mins) and print the file out on the website.
This is my example script, which shows data... but it makes connection each time web is loaded.
<?php
mysql_connect($dbserver, $dblogin, $dbpass);
mysql_select_db($dbname);
mysql_query("SET NAMES 'utf8'");
$rektanrekt = mysql_query("SELECT * FROM characters WHERE (accesslevel < '1') order by pkkills desc LIMIT 10");
$i = 1;
echo '<table id="top_table"><tr id="table_title"><td></td><td> </td><td>Nick</td><td></td><td>Kills</td></tr>';
while($row = mysql_fetch_array($rektanrekt))
{
echo '<tr><td id="skaiciai">' . $i . '.</td><td></td><td id="nickas"><font>';
echo $row["char_name"];
echo '</font></td><td> </td><td id="kills"><font>';
echo $row["pkkills"];
echo '</font></td></tr>';
$i++;
}
echo '</table>';
?>
And it does the job done -> http://prntscr.com/a3hpmx
But is it possible to make "backup" of this file, and show it if SQL is offline ... or even better - if SQL is ON update the file each time interval, if SQL is OFF just show latest one?
Saving output data to file:
The simplest way is to output the results in a text file using code similar to the following:
$timeStamp = time();
$pathName = "/gameLog/".$timeStamp.".txt";
$myFile = fopen($pathName, "w");
//+++++++++++++++++++++++++++++++++++
/*here you place your previous code, and inside the while
loop you use the fwrite() function to include the information
you want to save. For example:
fwrite($myFile, ($info."\n"));
where $info is the data you want to include in the file, and
"\n" is the line break.
*/
//+++++++++++++++++++++++++++++++++++
fclose($myFile);
Note: I used $timeStamp = time(); as the name of the file you are saving with the information you wanted because it will consistently generate a new value, hence you won't be overwriting your log files. Also it can help track at what time this log was saved. You could make this cleaner of course.
Doing this in intervals:
As jeroen mentioned, a cron job is probably your best solution to doing so. If you are using CPanel (linux), when you go to the CPanel homepage of your webhosting account (I am assuming your have an account with godaddy or another provider), there should be an option called "Cron Jobs" or something similar. What a Cron Job does is that it is a "Scheduled Event" that is told to execute some file in an interval you specify (you also specify the directory of your file). I found an example on youtube for you, however the video is a bit out of date, though the process should be similar:
https://www.youtube.com/watch?v=bmBjg1nD5yA
Edited: I noticed you are working on L2Aria, if you need any help, let me know! :)
These two pull out data from two different data bases. It's all good undtil I repeat the first one for a second time (to pull out the count of comments of more than one article), the second script (pulls out the data about vistis to the articles and arranges them by desc. order) stops working (no error, nothing, just doesn't provide an output). I'm no expert (yet) in the PHP so I can't seem to figure what is wrong in this sutuation. Maybe some of you will notice some obvious flaw which makes 'em interfere like that?
1st script (comment count). Just to make it clear: I don't use "define", the second time I use it for a diff. article. It's needed just in the first one to work.):
<?php
$id = "1"; //The ID of the page. You can get this from Manage -> Pages.
define('IN_COMMENTICS', '1');
require ($_SERVER['DOCUMENT_ROOT'].'comments/includes/db/connect.php');
$query = mysql_query("SELECT * FROM `".$cmtx_mysql_table_prefix."comments` WHERE is_approved = '1' AND page_id = '$id'");
$total = mysql_num_rows($query);
echo $total;
?>
Second script (counts visits):
<?php
$sql = "SELECT pagename, hits, title FROM counts ORDER BY hits DESC LIMIT 10";
$res = mysql_query($sql);
if(!$res) {
// oops - exit?
}
while(list($page,$hits,$title) = mysql_fetch_row($res)) {
echo "<li><a href='$page'>$title</a> $hits</li>";
}
?>
The only thing I could see is that you are using the function require to include you database initializing file, AND its executed two times which my create problems to causing this issue you are facing.
Perhaps, consider using require_once function which will take care to not load the database initializing file more than one time.
To conclude, I would suggest to do the following
Replace this:
require ($_SERVER['DOCUMENT_ROOT'].'comments/includes/db/connect.php');
by this:
require_once ($_SERVER['DOCUMENT_ROOT'].'comments/includes/db/connect.php');
I'm experiencing a strange problem. I'm caching the output of a query using memcache functions in a file named count.php. This file is called by an ajax every second when a user is viewing a particular page. The output is cached for 5 seconds, so within this time if there will be 5 hits to this file i expect the cached result to be returned 3-4 times atleast. However this is not happening, instead everytime a query is going to db as evidenced from a echo statement, but if the file is called from the browser directly by typing the url (like http://example.com/help/count.php) repeatedly many times within 5 seconds data is returned from cache (again evidenced from the echo statement). Below is the relevant code of count.php
mysql_connect(c_dbhost, c_dbuname, c_dbpsw) or die(mysql_error());
mysql_select_db(c_dbname) or die("Coud Not Find Database");
$product_id=$_POST['product_id'];
echo func_total_bids_count($product_id);
function func_total_bids_count($product_id)
{
$qry="select count(*) as bid_count from tbl_userbid where userbid_auction_id=".$product_id;
$row_count=func_row_count_only($qry);
return $row_count["bid_count"];
}
function func_row_count_only($qry)
{
if($_SERVER["HTTP_HOST"]!="localhost")
{
$o_cache = new Memcache;
$o_cache->connect('localhost', 11211) or die ("Could not connect to memcache");
//$key="total_bids" . md5($product_id);
$key = "KEY" . md5($qry);
$result = $o_cache->get($key);
if (!$result)
{
$qry_result = mysql_query($qry);
while($row=mysql_fetch_array($qry_result))
{
$row_count = $row;
$result = $row;
$o_cache->set($key, $result, 0, 5);
}
echo "From DB <br/>";
}
else
{
echo "From Cache <br/>";
}
$o_cache->close();
return $row_count;
}
}
I'm confused as to why when an ajax calls this file, DB is hit every second, but when the URL is typed in the browser cached data is returned. To try the URL method i just replaced $product_id with a valid number (Eg: $product_id=426 in my case). I'm not understanding whats wrong here as i expect data to be returned from cache within 5 seconds after the 1st hit. I want the data to be returned from cache. Can some one please help me understand whats happening ?
If you're using the address bar, you're doing a GET, but your code is looking for $_POST['...'], so you will end up with an invalid query. So for a start, the results using the address bar won't be what you're expecting. Is your Ajax call actually doing a POST?
Please also note that you've got a SQL injection vulnerability there. Make sure $product_id is an integer.
There are many problems with your code, first of all you always connect to the database and select a table, even if you don't need it. Second, you should check $result with !empty($result) which is more reliable as just !$result, because it's also covers empty objects.
As above noted, if the 'product_id' is not in the $_POST array, you could use $_REQUEST to also cover $_GET (but you shouldn't, if you are certain it's coming via $_POST).
I need to write a php script to load 1 million record and make insert statements of them.
i make each select to return only 100,000 record ,but the script just do nothing
ini_set('memory_limit', '200M');
ini_set('max_execution_time', '0');
include_once("/var/www/adodb/adodb.inc.php");
$DB_Conn = NewADOConnection("mssql");
$DB_Conn->PConnect("*.*.*.*", "***","***", "**");
echo 'Start'."\n";
$rs = $DB_Conn->_Execute("SELECT * FROM *** where id >=1 and id <=100000 order by id asc ");
while (!$rs->EOF) {
//print the result to a file as insert statements,and i try print to the console
$rs->MoveNext();
}
but the script just print start ,i wait it very long but nothing printed.
i try to set ini_set('memory_limit', '200M');
but it has no effect, i appreciate your help.
Added
Can i export records from mssql 2000 to mysql in any other way.(under linux without gui)
Morning,
i wouldnt fetch so many records in one step, why not 50.000 per run or something like this? Further you can try to set the timelimit to 0 ?