2300 request to mysql using php takes 2 hours , how to optimize - php

Question

am working on a news paper project but without Rss Feed, so i was Compelled to make it's feed programatically using php , so i have 2300 process of processing pages and inserting in Mysql the results of the processing ,
so the technique i used is to process every single page and then insert it's contents in mysql , it's working good but some times i got "MySQL server gone" ,
i tried to process 30 page and insert them in one request but it stop's after some time
so i am asking about any way to optimize this processing to reduce the time used in ?!
thanks alot

Answer 1

Your batch insert approach is correct and likely to help. You need to find out why it stops after some time like you say.
It is likely the php script timing out. Look for max_execution_time on your php.ini file to make sure it's high enough to allow for the script to finish.
Also, make sure your mysql config allows for a large enough packet because you're sending large batches which may be large.
Hope that helps!

Answer 2

There are plenties of reasons why "MySQL server has gone away". Take a look.
Anyway, it is strange that you load the WHOLE pages. Usually RSS feed implies that you put there just a subject and some text snippet. I'd create RSS feed as simple XML file so it is not necessary to load data from MySQL on EVERY hit users do. You create news -> regenerate RSS XML file, you wrote new article -> regenerate RSS XML file.
If you still want to prepare your data to be inserted, just create a file with ALL inserts and then load data from this file.
$ mysql -u root
mysql> \. /tmp/data_to_load.sql
Yes! all 2300 at a time ;)
$generated_sql = 'insert into Table (c1,c1,c3) values (data1,data2,data3);insert into Table (c1,c1,c3) values (data4,data5,data6);';
$sql_file = '/tmp/somefile';
$fp = fopen($sql_file, 'w');
fwrite($fp, $generated_sql, strlen($generated_sql)); // wrote sql script
fclose($fp);
`mysql -u $mysql_username --password=$mysql_password < $sql_file`;
Backticks are necessary in the last line!
$ mysql -u root test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 171
Server version: 5.1.37-1ubuntu5.5 (Ubuntu)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create table test (id int(11) unsigned not null primary key);
Query OK, 0 rows affected (0.12 sec)
mysql> exit
Bye
$ echo 'insert into test.test values (1); insert into test.test values (2);' > file
$ php -a
Interactive shell
php > `mysql -u root < /home/nemoden/file`;
php > exit
$ mysql -u root test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 180
Server version: 5.1.37-1ubuntu5.5 (Ubuntu)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select * from test
-> ;
+----+
| id |
+----+
| 1 |
| 2 |
+----+
2 rows in set (0.00 sec)
So, as you can see, it worked perfectly.