I have a .txt file with a list of 60,000 English words. I wanted to insert those to my database, so I simply just did as show here.
$file = new SplFileObject('list.txt');
foreach ($file as $line => $word) {
$p = new PDO('mysql:host=localhost; dbname=test_dictionary', 'root', 'test');
$p->query("INSERT INTO words (english) VALUES('$word') ");
}
Now, after I run this script, I get the following error:
Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [1040] Too many connections' in /var/www/skillz/test/curl/index.php:17 Stack trace: #0 /var/www/test/index.php(17): PDO->__construct('mysql:host=loca...', 'root', 'test') #1 {main} thrown in /var/www/test/index.php on line 4
That line 4 is where the new PDO('mysql:') is located. So, I tried to search this error, and found this answer that seemed a solution. And I edited mysql accordingly, as
$ vi /etc/my.cnf
max_connections=250
But I still get the same error, I have MySql 5.5.38 running PHP-FPM, NGINX in CentOS 6.5
Don't open a new Connection for every word. You only need one connection open for the lifetime of your inserts. I'm not sure about the true lifetime of a PDO object, I know they're cleaned up when they're not used, but garbage collection might not do that for a couple of minutes, and for 60,000 words, you're going to hit your limit of connections to the database faster than it can clean them up.
$file = new SplFileObject('list.txt');
$p = new PDO('mysql:host=localhost; dbname=test_dictionary', 'root', 'test');
foreach ($file as $line => $word) {
$p->query("INSERT INTO words (english) VALUES('$word') ");
}
You should declare the SQL connection outside the foreach.
Because it will make 60.000 connections.
$file = new SplFileObject('list.txt');
$p = new PDO('mysql:host=localhost; dbname=test_dictionary', 'root', 'test');
foreach ($file as $line => $word) {
$p->query("INSERT INTO words (english) VALUES('$word') ");
}
You only need to declare the SQL connection once and you can use it anytime as you want.
If you put it in foreach it will make a SQL connection every word, that's why you got that message.
Solution 1
Use batch insert statement :
INSERT INTO words (col1, col2) VALUES ('val1', 'val2'), ('val3', 'val4'), ...('val3n', 'val4n');
This fails if you also want to check if some rows failed or not. So, below is another solution.
Solution 2
Create a persistent database connection. This will use the same connection in all iterations of the loop.
$file = new SplFileObject('list.txt');
$p = new PDO('mysql:host=localhost; dbname=test_dictionary', 'root', 'test', array(
PDO::ATTR_PERSISTENT => true)); //Persistent Database Connection
foreach ($file as $line => $word) {
$p->query("INSERT INTO words (english) VALUES('$word') ");
}
$p = null; //Destroy Connection
Related
I'm currently looping to create a MBTiles map, and add information to my database each time.
Here's how I configured my connection and execute actions during the loop:
if ($pdo_mbtiles == null) {
echo "Opening new database connection".PHP_EOL;
$pdo_mbtiles = new PDO('sqlite:'.$filename,
'',
'',
array(
PDO::ATTR_PERSISTENT => true
)
);
$pdo_mbtiles->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
$pdo_mbtiles->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
}
$q = $pdo_mbtiles->prepare("INSERT INTO tiles (zoom_level, tile_column, tile_row,tile_data) VALUES (:zoom_level, :tile_column, :tile_rowTMS, :tile_data)");
$q->bindParam(':zoom_level', $zoom_level);
$q->bindParam(':tile_column', $tile_column);
$q->bindParam(':tile_rowTMS', $tile_rowTMS);
$q->bindParam(':tile_data', $tile_data, PDO::PARAM_LOB);
$q->execute();
After 1018 times looping (this number doesn't change no matter how many times I try), I get this error message:
SQLSTATE[HY000]: General error: 14 unable to open database file
I checked the solution written here:
How to prevent SQLITE SQLSTATE[HY000] [14]?
but the echoed message only appears at the first time of the loop, so I assume the PDO connection isn't closed.
I didn't find other documentation relevant to this error code.
What could possibly go wrong here?
I tried to move the prepare and bind commands in a condition as follows. The exception isn't raised, but only the first tile is saved (or every tile is saved on top of the first one, not sure):
if ($pdo_mbtiles == null) {
echo "Opening new database connection".PHP_EOL;
$pdo_mbtiles = new PDO('sqlite:'.$filename,
'',
'',
array(
PDO::ATTR_PERSISTENT => true
)
);
$pdo_mbtiles->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
$pdo_mbtiles->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
}
if ($q == null) {
$q = $pdo_mbtiles->prepare("INSERT INTO tiles (zoom_level, tile_column, tile_row,tile_data) VALUES (:zoom_level, :tile_column, :tile_rowTMS, :tile_data)");
$q->bindParam(':zoom_level', $zoom_level);
$q->bindParam(':tile_column', $tile_column);
$q->bindParam(':tile_rowTMS', $tile_rowTMS);
$q->bindParam(':tile_data', $tile_data, PDO::PARAM_LOB);
}
$q->execute();
Here are the files during the generation:
And here after the exception is raised:
Also, when the exception is raised, I do a var_dump of my $pdo_mbtiles, and here's the result (exactly the same as when I do it with a success):
object(PDO)#116 (0) {
}
Edit: Still trying to solve this problem, I simplified the code to create the MBTiles file. No success yet, but here's a sample if anyone want's to reproduce the issue. You can download it from https://www.dropbox.com/s/33vqamc9tn4c3ux/sample_PHP_MBTiles_generation_bug.zip?dl=0
I suggest you to reuse the connection if it is open.
Create a property: private $pdo;
And check if it's null before creating a new object:
function opendatabase(){
try{
if($this->pdo==null){
$this->pdo =new PDO("sqlite:database/database.db","","",array(
PDO::ATTR_PERSISTENT => true
));
}
return $this->pdo;
}catch(PDOException $e){
logerror($e->getMessage(), "opendatabase");
print "Error in openhrsedb ".$e->getMessage();
}
}
The error message was misleading. After many hours of debugging, I found that it was completely unrelated to my database connection.
It's juste that I used fopen() to get tiles data, and didn't fclose() after registration, thus reaching the limit of 1024.
1024 is because I used six require or require_once statements, so 1018 tile requests + 6 require = 1024 opened connections.
For some reason this code is causing odbc_execute(); to attempt to open a file...
$file = fopen('somefile.csv', 'r');
fgetcsv($file); // Skip the first line
$data = [];
while (($line = fgetcsv($file)) != false) {
$data[] = $line;
}
fclose($file);
try {
$conn = odbc_connect("Teradata", "User", "Pass");
odbc_autocommit($conn, false);
odbc_exec($conn, 'DELETE FROM table');
foreach ($data as &$test) {
$stmt = odbc_prepare($conn, 'INSERT INTO table (experiment_code, experiment_name, variant_code, variant_name, version_number, report_start_date, report_end_date, status, transaction_date, experiment_test_id, test_manager, product_manager, pod, created_date, last_updated_date) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)');
odbc_execute($stmt, $test);
}
odbc_commit($conn);
$result = odbc_exec($conn, 'SELECT * FROM table');
odbc_result_all($result);
} catch (Exception $e) {
odbc_rollback($conn);
echo $e->getMessage();
}
Here is a snip-it of the CSV file...
H1225,Some random text,H1225:001.000,Control,3,02/06/2014,03/31/2014,Completed,,HMVT-1225,Some random name,Some random name,Checkout,03/31/2014 16:54,02/06/2014 16:38
H1225,Some random text,H1225:001.000,Control,3,02/06/2014,03/31/2014,Completed,,HMVT-1225,Some random name,Some random name,Checkout,03/31/2014 16:54,02/06/2014 16:38
And here is the type of error I am getting...
Warning: odbc_execute(): Can't open file Control in C:\wamp\www\HEXinput\assets\php\dumpCSV.php on line 19
I get multiple version of the same error just with a different file name. The file name seems to be coming from column 3 (0 based). Another weird thing is that it actually does insert some lines correctly.
The final error I get is...
Fatal error: Maximum execution time of 120 seconds exceeded in C:\wamp\www\HEXinput\assets\php\dumpCSV.php on line 27
I am using Teradatas ODBC Drivers for version 15 on windows 7 64bit.
What could be causing this?
Turns out that some of the fields in the CSV file had single quotes in them which broke the query.
Simple but annoying oversight.
I have this test code:
session_start();
session_write_close();
set_time_limit(0);
for ($i = 1; $i < 1000; $i++)
{
if (rand(1,5) == 1)
{
$file_db = new PDO('sqlite:x.db');
$file_db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$file_db->exec('DROP TABLE IF EXISTS messages');
$file_db = null;
}
$file_db = new PDO('sqlite:x.db');
$file_db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$file_db->exec('CREATE TABLE IF NOT EXISTS messages (id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT,message TEXT,time INTEGER)');
$file_db->query('INSERT INTO messages VALUES (null, "'.rand(1,9999).'", "'.rand(1,99).'", date("now"))');
$file_db = null;
}
$file_db = new PDO('sqlite:x.db');
$file_db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$i = 0;
foreach ($file_db->query('SELECT * FROM messages') as $m)
{
$i++;
echo $i.' -> '.$m['id'].'<br>';
}
with this code Im testing if the PDO 3 works with multiple concurrencies. If this code runs in one thread, all is ok. But as soon as I start two instance of this, I get this message:
Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 17 database schema has changed' in code.php:* Stack trace: #0 code.php(*): PDO->query('INSERT INTO mes...') #1 {main} thrown in code.php on line *
the error line is where the INSERT INTO is. Why does this happen and how to dodge it? (Im creating a cache-system where there is do a possibility that tables gets created and dropped. I know there is TRUNCATE, I just want to know why this happens)
[...] as soon as I start two instance of this, I get this message: [...] SQLSTATE[HY000]: General error: 17 database schema has changed [...] Why does this happen and how to dodge it?
Sqlite is a single file database. You open it in parallel and you then change the database structure. Inserting data while you just dropped the table to insert into just does not work because the table is gone. This explains why it happens.
As you hopefully now have understood that inserting into a non-existing table is not possible, the solution might be clearly visible: Only insert into an existing table.
To ensure that you create a table after you've dropped it, it might be helpful to do such operations in transactions: https://www.sqlite.org/lang_transaction.html
Try$file_db = new PDO(realpath('sqlite:x.db'));
for
$file_db = new PDO('sqlite:x.db');
I've got an sql dump generated from mysqldump. This file includes mysql-version specific comments (/*!{MySQL version number} {Code} */).
If I insert an sql syntax error after this block, PDO doesn't trigger an exception.
php code
$sql = file_get_contents('FooBar.sql');
$pdo = new \PDO('mysql:host=localhost;dbname=FooBar', 'root');
$pdo->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_EXCEPTION);
$pdo->exec($sql);
FooBar.sql
/*!40101 SET #Foo='Bar' */;
ERROR
INSERT INTO Foo VALUES ('Bar');
This executes without causing any exceptions or errors. If I either remove the /*!40101 SET #Foo='Bar' */; statement, or move the error on line up an PDOException is thrown.
Thanks to the hek2mgl's for putting me on the right path.
PDO doesn't support multiple queries. If you execute a statement containing multiple queries, they get executed, but it seems PDO stops behaving after the first query is executed. The /*!{MySQL version number} {Code} */ style comment gets executed as a regular query by MySql and anything after this gets ignored by PDO, even though it gets executed by MySql.
The same error indicated would trigger by the following query:
SET #Foo='Bar';
ERROR
INSERT INTO Foo VALUES ('Bar');
To make this work using PDO, I need to split up the statements.
$sql = file_get_contents('FooBar.sql');
$lines = explode(';', $sql);
$pdo = new \PDO('mysql:host=localhost;dbname=FooBar', 'root');
$pdo->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_EXCEPTION);
foreach ($lines as $line) {
$trimmedLine = trim($line, " \t\n\r\0\x0B");
if (strlen($trimmedLine) > 0) {
$pdo->exec($trimmedLine.';');
}
}
EDIT:
An alternative solution is to use pdo prepared statements.
$sql = file_get_contents('FooBar.sql');
$pdo = new \PDO('mysql:host=localhost;dbname=FooBar', 'root');
$stmt = $pdo->prepare($sql);
$stmt->execute();
do {
... Do stuff ...
} while ($stmt->nextRowset());
if ($stmt->errorCode() != '00000') {
... Handle error ...
}
It's because of the ; at the end of the comment. I don't know why at the moment. Will investigate further...
Found this bug report. But don't expect the char encoding to be a problem as I've investigated the network traffic using wireshark and the MySQL returns a Syntax error (as expected.) Still don't know why PDO doesn't handle this correctly.
A workaround would be to use Mysqli which seems to handle this properly. The following example demonstrates this:
$pdo = new PDO('mysql:host=localhost;dbname=test', 'user', 'secret');
$result = $pdo->query('
SELECT 1 AS num;
ERROR
SELECT 1 AS num;
');
if(!$result) {
var_dump($pdo->errorInfo); // silence ...
}
$mysqli = new Mysqli('localhost', 'root', 'user', 'secret');
$result = $mysqli->query('
SELECT 1 AS num;
ERROR
SELECT 1 AS num;
');
if(!$result) {
print( $mysqli->error);
// Output: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ERROR SELECT 1 AS num' at line 2
}
Allrighty, it's the first time I ask a question here. My problem is as awkward as it is difficult to get to the bottom of.
Story goes like this: I have this little system, which sends alot of e-mail invitations(not spam). So, being sensible, I don't use the PHP function mail(),
I use PEAR classes like Mail, Mail_Queue, Net_SMTP, etc.
Only problem is, my error logs fill up with tons of errors like this:
PHP Notice: Error in sending mail:
Mail Queue Error: Cannot initialize container in /usr/lib/php/PEAR.php on line 873
And then, of course:
[18-Feb-2011 17:38:44] PHP Fatal error:
Allowed memory size of 33554432 bytes exhausted
(tried to allocate 3 bytes) in /usr/lib/php/PEAR.php on line 844
Here's the code which inits the mail queue(inside a class called Newsletter)
//I know passing the DSN as the string is kind of deprecated,
//but it;'s the only way it works on my system
$dsn ="mysql://$db_user:$db_pass#$db_host/$db_name";
$db_options = array();
$db_options['type'] = 'db';
$db_options['dsn'] = $dsn;
$db_options['mail_table'] = TABLE_QUEUE;
$this->host = '-- valid host here --';//data in these strings has been obscured
$this->username = '-- valid username here --';
$this->password = '-- valid password here --';
//optionally, a 'dns' string can be provided instead of db parameters.
//look at DB::connect() method or at DB or MDB docs for details.
//you could also use mdb container instead db
//$server = isset($_SERVER['SERVER_NAME'])?$_SERVER['SERVER_NAME']:'localhost';
$mail_options = array(
'driver' => 'smtp',
'host' => $this->host,
'port' => 25,
'auth' => true,
'username' => $this->username,
'password' => $this->password,
);
$this->mail_queue = new Mail_Queue($db_options, $mail_options);
Some more code down the line,
public function sendChunk($start, $count)
{
global $db;
$ids = $db->get_results("SELECT DISTINCT id_user as id FROM ".TABLE_QUEUE);
$ret = array();
foreach ($ids as $id)
$ret[] = $id->id;
unset($ids);
$this->mail_queue->sendMailsInQueue($count, $start, 1);
return true;
}
Problem is, I double checked every line of code I wrote, and it's doing it's job. Only problem is that sometimes it refuses to send any mails.
Thanks in advance for replies.
I switched to MDB2 instead of DB
$db_options['type'] = 'db';
to
$db_options['type'] = 'mdb2';
this helped in taking care of memory exhaust problem, I am still looking to take care of initialize container in /usr/lib/php/PEAR.php problem
Ok found the solution for container errors:
Apply this patch
http://svn.php.net/viewvc/pear/packages/Mail_Queue/trunk/Mail/Queue.php?r1=303876&r2=309126
Try to limit result. using limit in your select statement.
Try to flush the old main queye.