I need to write a php script to load 1 million record and make insert statements of them.
i make each select to return only 100,000 record ,but the script just do nothing
ini_set('memory_limit', '200M');
ini_set('max_execution_time', '0');
include_once("/var/www/adodb/adodb.inc.php");
$DB_Conn = NewADOConnection("mssql");
$DB_Conn->PConnect("*.*.*.*", "***","***", "**");
echo 'Start'."\n";
$rs = $DB_Conn->_Execute("SELECT * FROM *** where id >=1 and id <=100000 order by id asc ");
while (!$rs->EOF) {
//print the result to a file as insert statements,and i try print to the console
$rs->MoveNext();
}
but the script just print start ,i wait it very long but nothing printed.
i try to set ini_set('memory_limit', '200M');
but it has no effect, i appreciate your help.
Added
Can i export records from mssql 2000 to mysql in any other way.(under linux without gui)
Morning,
i wouldnt fetch so many records in one step, why not 50.000 per run or something like this? Further you can try to set the timelimit to 0 ?
Related
I have a curious issue that is happening on my system when I run a PHP program I created. I am taking data from two tables and doing a comparison of data from Table A with Table B. If the record is not in Table A, then write that record to Table B. I discovered that the program works fine but, at least one record has to be in Table B first. After my first successful run Table A will have 16000+ records in it, now Table B has 15000+ or so records. I understand that there is going to be a bit of time for this to process. The curious thing is I noticed my hard drive is losing free space as the program runs. I have tried manually running the garbage collection. I also looked where the session files are being stored, only to find a few files that are rather small in size. I also tried adjusting the length of time that session files are stored from 1440 seconds to 30 secs. When I say that I am "losing" free space, there is something that is filling up my hard drive. I have gone from having 6GB to 5.75GB, if I allow the program to run longer, I only lose more space. I have also tried just simply restarting my system and I only regain a small portion of the space I lost. At this point I am unsure what I need to do to stop this from happening. Here is a sample of my code below:
<?php
include('./connect_local_pdo.php'); //Includes DB Connection Script
ini_set('max_execution_time', 5400); //5400 seconds = 90 minutes
gc_disable();
try {
$tbl_a_data = $conn->prepare('SELECT col_a, col_b, col_c from table_a');
$tbl_a_data->execute();
$tbl_b_data = $conn->prepare('SELECT col_a, col_b, col_c from table_b');
$tbl_b_data->execute();
$tbl_b_array = $tbl_b_data->fetchAll(PDO::FETCH_ASSOC);
while($tbl_a_array = $tbl_a_data->fetch(PDO::FETCH_ASSOC)){
foreach ($tbl_b_array as $tbl_b_array2){
if ($tbl_a_array['col_a'] !== $tbl_b_array2['col_a']){
$stmt = $conn->prepare("INSERT INTO table_b
(col_a, col_b, col_c)
VALUES
(:col_a, :col_b, :col_c)");
$stmt->bindParam(':col_a', $tbl_a_array['col_a']);
$stmt->bindParam(':col_b', $tbl_a_array['col_b']);
$stmt->bindParam(':col_c', $tbl_a_array['col_c']);
$stmt->execute();
} else {
$stmt = $conn->prepare("update table_b
set
col_b = table_a.col_b,
col_c = table_a.col_c
from table_a
where table_b.col_a = table_a.col_a ");
$stmt -> execute();
}
}
}
gc_collect_cycles();
gc_mem_caches();
clearstatcache();
} catch (PDOException $a) {
echo $a->getMessage();//Remove or change message in production code
}
Any assistance with this will be greatly appreciated! As of this post I have lost 2 gigs of space running this program.
With your line
include('./connect_local_pdo.php'); //Includes DB Connection Script
I assume it is a Database on localhost.
The Database is growing with your entries. It will take space to add rows.
If two users execute the same php file, will it be executed parallel or sequential? Example:
If I have a database data which only has one column id would it be possible that the following code produces for two different users the same outcome?
1. $db=startConnection();
2. $query="SELECT id FROM data";
3. $result=$db->query($query)or die($db->error);
4. $zeile=mysqli_fetch_row($result);
5. $number=$zeile['id'];
6. $newnumber=$number+1;
7. echo $number;
8. $update = "UPDATE data Set id = '$newnumber' WHERE id = '$number'";
9. $db->query($query)or die($db->error);
10. mysqli_close($db);
If it is not executed parallel, does it mean when 100 people are loading a php file that has a loading time of 1 second, then one of them has to wait 99 seconds?
Edit: In the comments it is stated that I could messup my database, I guess this is how it could mess up:
User A executes the file from 1.-7. in this moment user B executes the file from 1.-7. then A loads 8.-10. and B loads 8.-10. In this scenario both users would have the same number on the screen.
Now lets take the following example:
1. $db=startConnection();
2. $query=" INSERT INTO data VALUES ()";
3. $result=$db->query($query)or die($db->error);
4. echo $db->insert_id;
5. mysqli_close($db);
Lets say A executes the file from 1.-3. in this moment user B executes the file from 1.-5., after that user A loads the file from 4.-5. I guess in this scenario also both would have the same number on the screen right? Does transaction prevent both scenarios?
You can say that php files executed parallel (for most cases it is so, but this depends on web server).
Yes, it is possible that the following code produces for two different users the same outcome.
How to avoid this possibility?
1) If you are using MySQL, you can use transactions and "SELECT ... UPDATE FOR" to avoid this possibility. Just using transaction wouldn't help!
2) Be sure that you are using InnoDB or any other database engine that support transactions. For example MyISAM doesn't support transactions. Also you can have problems if any form of snapshotting is enabled in the database to handle reading locked records.
3) Example of using "SELECT ... UPDATE FOR":
$db = startConnection();
// Start transaction
$db->query("START TRANSACTION") or die($db->error);
// Your SELECT request but with "FOR UPDATE" lock
$query = "SELECT id FROM data FOR UPDATE";
$result = $db->query($query);
// Rollback changes if there is error
if (!$result)
{
mysql_query("ROLLBACK");
die($db->error);
}
$zeile = mysqli_fetch_row($result);
$number = $zeile['id'];
$newnumber = $number + 1;
echo $number;
$update = "UPDATE data Set id = '$newnumber' WHERE id = '$number'";
$result = $db->query($query);
// Rollback changes if there is error
if (!$result)
{
mysql_query("ROLLBACK");
die($db->error);
}
// Commit changes in database after requests sucessfully executed
mysql_query("COMMIT");
mysqli_close($db);
Why just using transaction wouldn't help?
Just transaction will lock only for write. You can test examples bellow by running two mysql console clients in two separate terminal windows. I did so and that's how it works.
We have client#1 and client#2 that executed parallel.
Example #1. Without "SELECT ... FOR UPDATE":
client#1: BEGIN
client#2: BEGIN
client#1: SELECT id FROM data // fetched id = 3
client#2: SELECT id FROM data // fetched id = 3
client#1: UPDATE data Set id = 4 WHERE id = 3
client#2: UPDATE data Set id = 4 WHERE id = 3
client#1: COMMIT
client#2: COMMIT
Both clients fetched the same id (3).
Example #2. With "SELECT ... FOR UPDATE":
client#1: BEGIN
client#2: BEGIN
client#1: SELECT id FROM data FOR UPDATE // fetched id = 3
client#2: SELECT id FROM data FOR UPDATE // here! client#2 will wait for end of transaction started by client#1
client#1: UPDATE data Set id = 4 WHERE id = 3
client#1: COMMIT
client#2: client#1 ended transaction and client#2 fetched id = 4
client#1: UPDATE data Set id = 5 WHERE id = 4
client#2: COMMIT
Hey, I think such read-locks reduce performance!
"SELECT ... FOR UPDATE" do read-lock only for clients that use "SELECT ... FOR UPDATE". That's good, cause it means that such read-lock wouldn't affect on standart "SELECT" requests without "FOR UPDATE".
Links
MySQL documentation: "SELECT ... FOR UPDATE" and other read-locks
Parallel or Sequential?
Part of your question was about PHP running either parallel or sequential. As I have read everything and its opposite about that topic, I decided to test it myself.
Field testing:
On a LAMP stack running PHP 5.5 w/ Apache 2, I made a script with a very expensive loop:
function fibo($n)
{
return ($n > 1) ? fibo($n - 1) + fibo($n - 2) : 1;
}
$start = microtime(true);
print "result: ".fibo(38);
$end = microtime(true);
print " - took ".round(($end - $start), 3).' s';
Result with 1 script running:
result: 63245986 - took 19.871 s
Result with 2 scripts running at the same time in two different browser windows:
result: 63245986 - took 20.753 s
result: 63245986 - took 20.847 s
Result with 3 scripts running at the same time in three different browser windows:
result: 63245986 - took 26.172 s
result: 63245986 - took 28.302 s
result: 63245986 - took 28.422 s
CPU usage while running 2 instances of the script:
CPU usage while running 3 instances of the script:
So, it's parallel!
Althoug inside a PHP script, you can't easily use multithreading (while it's possible), Apache takes benefit from your servers having multiple cores to dispatch the load.
So if your 1-second script is run by 100 users at the same time, well if you have 100 CPU cores the 100th user will hardly notice anything. If you have 8 CPU cores (which is more common), then the 100th user will theoritically have to wait something like 100 / 8 = 12.5 seconds for his instance of the script to begin. In practice, as the "benchmark" puts in evidence, each thread's performance diminishes when other threads are running at the same time on other cores. So it could be a lot more. But not 100 seconds more.
I'm making a query over a database with over 20MM entries, that means im breaking the query into several smaller queries.
The problem is if I try to fetch the 20MM entries the page does not load, and displays a blank screen, with no title and content. However, if I fetch 5MM entries, the page does load correctly, and displays the content:
Here's my code
for($n=0; $n<20000000; $n=$n+500000){
$m=500000;
$query = "SELECT * FROM user_likes LIMIT ". $n .",". $m;
//echo $query;
$result = mysql_query($query) or die(mysql_error());
// craete arrays
while($row = mysql_fetch_array($result)){
set_time_limit(0);
$like[$row['name']]=$like[$row['name']]+1;
if($like[$row['name']]==375) $likes375 ++;
}
}
// print the size
echo count($like)."<br>";
echo "375: ".$likes375;
I would appreciate if someone can help me with this.
Thanks
EDIT:
after adding error_reporting(E_ALL); it display this notice: MySQL server has gone away
The reason for that is because your page will be blank while MySQL is crunching through its index looking for your results. After a while, this will time out (depending on your settings). That is why this doesn't happen on smaller queries.
This is due to time limit of execution of php scripts, by default php script will run for a total of 45 seconds before timing out...
Just use this after <?php
set_time_limit(0); // 0 means unlimited
but it is not recommended to use it, better improve your logic at the both database and php code logic.
EDIT:
try to set error log for this file specificaly by doing this:
ini_set("log_errors" , "1");
ini_set("error_log" , "eorrors.txt");
ini_set("display_errors" , "0");
I am running 10 PHP scripts at the same time and it processing at the background on Linux.
For Example:
while ($i <=10) {
exec("/usr/bin/php-cli run-process.php > /dev/null 2>&1 & echo $!");
sleep(10);
$i++;
}
In the run-process.php, I am having problem with database loop. One of the process might already updated the status field to 1, it seem other php script processes is not seeing it. For Example:
$SQL = "SELECT * FROM data WHERE status = 0";
$query = $db->prepare($SQL);
$query->execute();
while ($row = $query->fetch(PDO::FETCH_ASSOC)) {
$SQL2 = "SELECT status from data WHERE number = " . $row['number'];
$qCheckAgain = $db->prepare($SQL2);
$qCheckAgain->execute();
$tempRow = $qCheckAgain->fetch(PDO::FETCH_ASSOC);
//already updated from other processs?
if ($tempRow['status'] == 1) {
continue;
}
doCheck($row)
sleep(2)
}
How do I ensure processes is not re-doing same data again?
When you have multiple processes, you need to have each process take "ownership" of a certain set of records. Usually you do this by doing an update with a limit clause, then selecting the records that were just "owned" by the script.
For example, have a field that specifies if the record is available for processing (i.e. a value of 0 means it is available). Then your update would set the value of the field to the scripts process ID, or some other unique number to the process. Then you select on the process ID. When your done processing, you can set it to a "finished" number, like 1. Update, Select, Update, repeat.
The reason why your script executeds the same query multiple times is because of the parallelisation you are creating. Process 1 reads from the database, Process 2 reads from the database and both start to process their data.
Databases provide transactions in order to get rid of such race conditions. Have a look at what PDO provides for handling database transactions.
i am not entirely sure of how/what you are processing.
You can introduce limit clause and pass that as a parameter. So first process does first 10, the second does the next 10 and so on.
you need lock such as "SELECT ... FOR UPDATE".
innodb support row level lock.
see http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html for details.
I have a big problem:
A user logs in, and the session opens.
After that he clicks a button (in a form) which action is "example.php".
This example.php should increase one record in MySQL database by 1.
How to write this "example.php"?
Please help.
EDIT:
Sorry, I haven't asked what i was supposed to ask.
Main problem:
After a user clicks a button, the example.php script should execute after a specified time, for example, a 600 sec.
The other problem is that the user can click the button and log out, and despite that, the example.php should execute 600 sec later.
EDIT (18:48):
OK, I've read all your suggestions, but dont't have an idea how to make all of this things work together.
I made a form.php, with input type="submit" and action="example.php".
I want to:
1. start a javascript timer;
2. increase a value in database after a time, specified earlier in a variable.
Please, if it is possible, give me an example, how to do this.
In example.php execute SQL:
UPDATE table SET field = field + 1;
Do you need more info to do that ?
EDIT:
I think that there is no other way like jobs mechanism. When user calls example.php, you add to the database new job with a current timestamp + 600 seconds.
Parallel there should be running some job executor that will gather from database all jobs that have timestamp set to timestamp <= NOW().
If it will some records, call specified piece of code and remove/mark as done that jobs.
You may delay the execution of code using sleep
Example:
<?php
echo 'Script Start: '.date('h:i:s') . '<br>';
sleep(5); // delay in seconds (here 5)
echo 'Script Ende: '.date('h:i:s') . '<br>';
?>
I do not know if what you are trying to do makes perfect sense, but your exampl.php would look something like:
<?php
sleep(600); // delay in seconds
// the code to be executed delayed here
?>
There are several ways you could go about doing this...
As mentioned there is the sleep method.
You could have a job - i.e. Add something to a jobs list and have a cronjob check it every so often to see if it is due.
You could use a javascript timer to execute after x number of seconds.
<?php
if( isset($_POST['submit_button']) ) {
$link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
$sql = 'UPDATE table SET counter_column = counter_column + 1'; // WHERE table_name_id =' . intval($id);
mysql_query($sql, $link);
}
?>
This example.php will only update the value if the submit_button is clicked on and not if you just type in the URL example.php