I need to fetch, convert and then insert 4 000 000 rows into another table.
Needles to say, a memory exhausted error kicks in after 300 000~ entries. After every loop allocated memory increases by exactly 160 bytes.
I know that using mysql_unbuffered_query() would be possible, although it forces me to fetch all result rows before I can execute another query, which ends up in memory exhausted error again. So, what's the best way to do this in a single run?
mysql_connect($host, $user, $password);
mysql_select_db($db);
$getOldData = mysql_query("
SELECT *
FROM players_online
ORDER by id ASC
");
$numRows = mysql_num_rows($getOldData);
for ($i=0; $i < $numRows; $i++) {
$oldData = mysql_fetch_assoc($getOldData);
$hour = explode(':', $oldData['hour']);
$quarters = $hour[0] * 4 + $hour[1] / 15;
$update = mysql_query("
INSERT INTO players_online_2 (world_id, players_online, quarters_past_midnight, date)
VALUES (
'".$oldData['world_id']."',
'".$oldData['players_online']."',
'".$quarters."',
'".$oldData['date']."'
)
ON DUPLICATE KEY UPDATE
world_id='".$oldData['world_id']."',
players_online='".$oldData['players_online']."',
quarters_past_midnight='".$quarters."',
date='".$oldData['date']."'
");
if (mysql_error()) {
echo mysql_error();
die();
}
echo memory_get_usage().PHP_EOL;
}
MySQL Workbench will let you export the old database and import it into the new location.
With that said though, if you want to do this in PHP, you should probably not return the entire table in one shot. You could use LIMIT and OFFSET to split the query into pages, and then execute one page at a time. Something like this:
for ($i = 0; $i < $Limit; $i++)
{
// SELECT * FROM players_online ORDER by id ASC LIMIT 1000 OFFSET ' . $i
}
You can do this in a single query, something like:
INSERT INTO players_online_2 (world_id, players_online, quarters_past_midnight, date)
select world_id, players_online, quarters_past_midnight, date
FROM players_online po
ORDER by id ASC
ON DUPLICATE KEY UPDATE
world_id = po.world_id,
players_online = po.players_online,
quarters_past_midnight = po.quarters_past_midnight,
date = po.date;
Related
I have 1,00,000 record in my table A and I need some data of A insert B table through Corn with some other data. So In implement below code and set pagination rule but every new page it taking time.
Every 10000 record (it's a page limit) page change and it's increase process time. how to reduse process time.
$DB = new myclassi();
$resource_new = "SELECT COUNT(`gi`.`igems_item_id`) as total
FROM `gems_item` AS `gi`
WHERE NOT EXISTS (SELECT `gicd`.`id` FROM `gems_item_cron_data` As `gicd` WHERE `gi`.`old_sku` = `gicd`.`gems_old_sku` AND `gi`.`igems_item_id` = `gicd`.`gems_itemid`)
AND `gi`.`diamond_video_url` IS NOT NULL
AND (`gi`.`diamond_video_url` LIKE '%http%' OR `gi`.`diamond_video_url` LIKE '%https%')";
$res = $DB->select_assoc($resource_new);
$total_rows = $res[0]['total'];
$no_of_records_per_page = 10000;
$total_pages = ceil ($total_rows / $no_of_records_per_page);
$limit = "";
$pageno = 1;
for($i=0;$i < $total_pages; $i++){
$offset = ($pageno) * $no_of_records_per_page;
if ($pageno > 1) {
$prev_offset = ($offset - $no_of_records_per_page) + 1;
$limit = $prev_offset.", ".$no_of_records_per_page;
}else{
$limit = "0, ".$no_of_records_per_page;
}
$sqlr="SELECT `gi`.`igems_item_id`, `gi`.`old_sku`, `gi`.`diamond_video_url`
FROM `gems_item` AS `gi`
WHERE NOT EXISTS (SELECT `gicd`.`id` FROM `gems_item_cron_data` As `gicd` WHERE `gi`.`old_sku` = `gicd`.`gems_old_sku` AND `gi`.`igems_item_id` = `gicd`.`gems_itemid`)
AND `gi`.`diamond_video_url` IS NOT NULL
AND (`gi`.`diamond_video_url` LIKE '%http%' OR `gi`.`diamond_video_url` LIKE '%https%') LIMIT ". $limit;
$results = $DB->select_assoc($sqlr);
if (isset($results) && !empty($results)) {
foreach ($results as $row) {
$url = urldecode($row['diamond_video_url']);
$k = $row['igems_item_id'];
$video_url = 'xxxfdafas';
$old_sku = $row['old_sku'];
$diamond_video_url = $row['diamond_video_url'];
$diamond_final_video_url = urlencode($video_url);
$is_url_found = '';
$is_processed = 0;
$created_at = date('Y-m-d H:i:s');
$sql = "INSERT INTO gems_item_cron_data (`gems_itemid`,`gems_old_sku`, `diamond_final_video_url`,`is_processed`,`created_at`)
VALUES ($k, '$old_sku','$diamond_final_video_url', '$is_processed', '$created_at')";
$DB->insert($sql);
}
}
$pageno++;
}
$DB->close();
Please let me know how to make faster insert process.
Simplify test
This
AND `gi`.`diamond_video_url` IS NOT NULL
AND (`gi`.`diamond_video_url` LIKE '%http%'
OR `gi`.`diamond_video_url` LIKE '%https%'
)";
can be simplified, and sped up significantly, to this
AND `gi`.`diamond_video_url` LIKE 'http%'
especially if the URL necessarily starts with "http".
Eliminate COUNT
I suggest there is no need to first count the rows. Simply do the SELECT to fetch the rows, then notice whether you got any rows. That, alone, will (probably) more than double the speed of the program.
Batch INSERT
If practical, replace the loop and single-row inserts with
INSERT INTO ... (...)
SELECT ...
That might give another factor of 10 in speedup.
If the SELECT returns no rows, there will be nothing to INSERT. Real cheap.
After you have done those, there may be further optimizations, but then I would need to see the new query, plus SHOW CREATE TABLE.
I would like get number of records in a table then divide them by 4, after dividing them by 4 i want to create sql statements with limit ranges based on my result. For example I have a table with 8 records I divide by 4, I will create 2 sql statements with a limit range like limit 0,4 and limit 4,8
Final results will look like
Select * from prop where id=123 LIMIT 0,4
Select * from prop where id=123 LIMIT 4,8
My approach was to have for loop which will count the number of sql statements to be made.
Then in the loop: first circle 0-4 and second will be 4-8
Am struggling on the limit 0-4 and limit 4-8
PHP script
include('connect.php');
$query_1 = "Select COUNT(*) as Total from prop where ref = 'SB2004'";
$results_query_1 = mysql_query($query_1);
while($row_query_1 = mysql_fetch_array($results_query_1))
{
$cnt = $row_query_1['Total'];
}
echo $cnt;
echo "<br>";
$num_grps = 0;
if ($cnt % 4 == 0 )
{
echo $num_grps = $cnt / 4 ;
}
$count_chk= $num_grps * 4;
for ($i=1;$i<=$num_grps;$i++)
{
//for loop for range
for()
{
$range = '0,4';
echo "SELECT prop_ref from prop limit".$range;
}
}
Either you've not understood the problem or haven't explained it very well.
The most immediate problem here is that you have misunderstood the syntax for the LIMIT clause. The first argument specifies the offset to start at and the second defines the number of rows to return, hence LIMIT 4,8 will return 8 rows (assuming there are 12 or more rows in the dataset).
The next issue is that you've not said if the results need to be reproducible - e.g. if you have rows with primary keys 1 and 2, should these always be returned in the same query. In the absence of an explicit ORDER BY clause, the rows will be returned based on the order in which they are found by the query.
The next issue is that you've not explained how you want to deal with the last case when the total number of rows is not an even multiple of 4.
The code you've provided counts the number of rows where ref = 'SB2004' but then creates queries which are not filtered - why?
The code you've provided does not change the limit in the queries - why?
The next issue is that there is never a good reason for running SELECT queries inside a loop like this. You didn't exlpain what you intend doing with the select queries. But based on the subsequent update....
include('connect.php');
$query_1 = "Select COUNT(*) as Total from prop where ref = 'SB2004'";
$cnt = mysql_fetch_assoc(mysql_query($query_1));
$blocks=$cnt['Total']/4 + (0 == $cnt['Total'] % 4 ? 0 : 1);
$qry2="SELECT * FROM prop where ref='SB2004' ORDER BY primary_key";
$res=mysql_fetch_assoc($qry2);
for ($x=0; $x<$blocks; $x++) {
print "<div>\n$block<br />\n";
for ($y=0; $y<4; $y++) {
print implode(",", #mysql_fetch_assoc($res)). "\n";
}
print "</div>\n";
}
It's trivial to refine this further to only issue a single query to the database.
If you really must generate individual SELECTs....
include('connect.php');
$query_1 = "Select COUNT(*) as Total from prop where ref = 'SB2004'";
$cnt = mysql_fetch_assoc(mysql_query($query_1));
$blocks=$cnt['Total']/4 + (0 == $cnt['Total'] % 4 ? 0 : 1);
for ($x=0; $x<$blocks; $x++) {
$y=$x*4;
print "SELECT * FROM prop where ref='SB2004'
ORDER BY primary_key LIMIT $y,4<br />\n"
}
sorry about the title, i really did not know what I should call it, but hopefully you will be able to aid me with my script.
What I am trying to achieve (with my less than 5 hour total experience with any sort of "programming", hence the horrid coding) is to send one query X times, and then put a new query into those newly created rows.
if(isset($_SESSION['email'])) { // IF LOGGED IN
$sql = mysql_query("SELECT max(ordrenr) FROM antalstabel") or die(mysql_error());
$maxordrenr = mysql_query($sql);
$nextnumber = $maxordrenr + 1;
$maxplusantal = $maxordrenr + $antal;
$antal = count($items); // COUNTS DIFFERENT ITEMS IN CART.
for ($i = $maxordrenr; $i <= $maxplusantal; $i++) {
$sql = mysql_query("INSERT INTO antalstabel (ordrenr) VALUES ('$nextnumber')") or die(mysql_error());
}
}
This is my first query, what this does (or what I want it to do) is to get the max ID of the table "antalstabel" add +1 and then count a certain amount up which is defined as $items untill it has executed X rows.
My first issue here, is the fact that my table consists of two key primaries, so returning a query like this would result in an error since after one return the two rows would be identical and will not execute.
The second issue is the fact that the next value in the table should not be inserted X times after each other, but rather be certain IDs added in afterwards.
What I am trying to achieve ultimately (not only by this script, but this is the current issue) is something like this:
ordrenr(key)varenr(key) antal
1 3 1
1 2 2
2 1 4
3 1 1
3 2 1
3 3 1
Does this make any sense whatsoever for anyone and can anyone tell me whether my method of doing this is jsut hopeless or have some better ideas for me to use as execution for ending up with something like this?
Should I not use primary keys or how does this work?
Thank you for even taking the time to read this :)
-Victor
EDIT for future:
changed script to this for it to work:
if(isset($_SESSION['email'])) { // IF LOGGED IN
$sql = mysql_query("SELECT * FROM antalstabel ORDER BY ordrenr DESC LIMIT 1") or die(mysql_error());
$row = mysql_fetch_assoc($sql);
$maxordrenr = $row['ordrenr'];
$nextnumber = $maxordrenr + 1;
$maxplusantal = $maxordrenr + $antal;
$antal = count($items); // COUNTS DIFFERENT ITEMS IN CART.
for ($i = $maxordrenr; $i <= $maxplusantal; $i++) {
$sql = mysql_query("INSERT INTO antalstabel (ordrenr, varenr) VALUES ('$nextnumber','1236')") or die(mysql_error());
}
$maxordrenr = mysql_query($sql);
should be
$maxordrenr = mysql_result($sql,0);
I have this MySQL query :
while ($x <= 9) {
$data_1 = "SELECT scene FROM star WHERE star LIKE '%".$star."%' ORDER BY RAND() LIMIT 1";
$result_1 = mysql_query ($data_1) OR die("Error: $data_1 </br>".mysql_error());
while($row_1 = mysql_fetch_object($result_1)) {
$scene = $row_1->scene;
$x = $x + 1;
}
}
I want to get everytime a new scene for each execution, but I always get the same scene. Whats the issue? Can someone make me a few pointers in which direction I have to search ?
What you need to do is tie a random seed to each row, and tie a new one each time. Do this by assigning a random value as an aliased transient column to your table, and then select from it.
SELECT s.scene as scene FROM (
SELECT stars.scene as scene, RAND() as seed
FROM stars
WHERE star LIKE '%".$star."%'
ORDER BY seed
) s
LIMIT 1;
Using PDO it's going to look something like this:
function getScene() {
$sql = 'SELECT s.scene as scene FROM ( SELECT stars.scene as scene, RAND() as seed FROM stars WHERE star LIKE '%:star%' ORDER BY seed ) s LIMIT 1;';
$query = $this->db->prepare($sql);//'db' is the PDO connection
$query->execute(array(':star' => "Allison Brie"));
foreach ($conn->query($sql) as $row) {
print $row['scene'] . "\t";
}
}
I'm not sure what you were trying to accomplish with the rest of your code, but it mostly looks like cruft.
Faily new to php and mysql, this will probably seem very messy.
This is what I came up with:
$query = "show tables like 'whatever%'";
$result = mysql_query($query);
$num_results = mysql_num_rows($result);
for ($i = 0; $i < $num_results; $i++)
{
$row = mysql_fetch_array($result);
$sql=mysql_query("SELECT * FROM ". $row[0] ." WHERE a=(SELECT MAX(a)) AND b=(SELECT MAX(b)) AND c LIKE 'd%' ORDER BY date DESC LIMIT 1");
while($info=mysql_fetch_array($sql)){
echo "...";
}
}
I get the desired value from each table, so x results depending on the amount of tables. What I would like is to have the results of the queried tables but only show the top 10-5 ordered by date/time.
Is this possible with the current script? Is there an easier way (while, number of tables changing constantly)? Is this query method database intensif?
Cheers!
I call constantly changing number of tables having similar structure a design error. also query switch to
$sql=mysql_query("SELECT * FROM $tbl WHERE c LIKE 'd%' ORDER BY a DESC, b DESC, date DESC LIMIT 1");
is a little relief to database.