Improve sqlite performance in yii - php

I saw some discussions similar obviously but couldn't found a solution(if there is one). I have a project running on linux machine.
The problem is that working with the Database takes forever. For example 1000 inserts takes approximately 10 seconds.
I tried reducing the time in a different ways with little success so I thought I just place here a part of my code and maybe there is something critical I'm not doing right.
First of all, in main.php the database is configured like this:
'db'=>array(
'pdoClass' => 'NestedPDO',
'connectionString' => 'sqlite:/tmp/mydb.db',
'class' => 'CDbConnection',
'schemaCachingDuration' => 100)
I work with the database in the following way:
$connection=Yii::app()->db;
$transaction = $connection->beginTransaction();
try
{
Some Code..
$transaction->commit();
}
catch (Exception $ex)
{
$transaction->rollback();
}
Inside Some Code there could be calls to different functions that the connection variable is passed to.
Finally each sqlite command (for example, 1000 inserts) is written like:
$statement= 'insert into my_tbl (id, name) VALUES(:id, :name)';
$command=$connection->createCommand($statement);
$command->bindParam(":id", $id);
$command->bindParam(":name", $name);
$command->execute();

First make sure where the bottleneck is.
try
{
$t_start = time();
$transaction->commit();
$elapsed = time() - $t_start;
}
I have add somewhat good success by committing more often.

Related

rollback transactions on multiple databases

I have multiple PDO objects, referring to different databases on different servers. I need to be able to rollback all transactions in case of error at any step.
In fact, I want to have something like this:
try {
$db1Connector->beginTransaction();
$db2Connector->beginTransaction();
//some functionality
$db1Connector->commit();
$db2Connector->commit();
} catch (\Exception $exception) {
$db1Connector->rollback();
$db2Connector->rollback();
}
I have this functionality in my code, but it seems it does not work correctly. I noticed some cases that only one of the transactions is rollbacked. I guess this problem occurs when $db1Connector->commit() or $db2Connector->commit() fails.
Yes, it is possible. You need to use a distributed transaction. See https://stackoverflow.com/questions/17772363/distributed-transaction-on-mysql#:~:text=so%20server%20A%20sends%20a,server%20A%20and%20server%20B.

PHPunit: Problems with testing

I have run into a major problem while writing tests. I am using Laravel 5.6.0 as framework and PHPUnit 7.0 for testing. As a testing DB I have used sqlite with storage in memory. This is from my database.php:
'sqlite_testing' => [
'driver' => 'sqlite',
'database' => ':memory:',
'prefix' => '',
],
But my problem is that I have several places where I use whereRaw, for example $query->whereRaw('STR_TO_DATE(CONCAT(date, " ",from), "%Y-%m-%d %k") < ?', [$before]);. The problem here is that sqlite does not have the STR_TO_DATE or CONCAT functions that MySQL has. So PHPUnut throws a bunch of errors because of that.
My solution was instead using a MySQL DB as testing DB. But this doesn't seem to work since I get several different errors, mostly I have several tests where foreign key constraint fails.
One exaple for this is that I have the following in my Base TestCase setUp method:
if (Schema::hasTable('gym_schedule') && !empty(GymSchedule::createGymSchedules())) {
$this->artisan('db:seed', ['--class' => 'GymScheduleTableSeeder']);
}
This fails every time except the first because it says that a schedùle with id 1 already exists (id is my primary key). I did try to truncate all tables between each test class using tearDown, but that did not help at all, and also the testing became reeeeally slow, like 3 seconds for each test.
So basically that approach does not work either.
I am at a loss. I have tried googling this and searching through StackOverflow. I am open to any suggestion that is not too complicated (either remove all MySQL functions somehow, solve usage of MySQL or anything else really).
Does anyone have a good idea?
Regarding the first part of the question, unit testing a Laravel app when using whereRaw(), this is how I wrote the methods:
$raw_condition = "CONCAT(`country_code`, `number`) = ?";
if (env('DB_CONNECTION') === 'sqlite') {
$raw_condition = "(`country_code` || `number`) = ?";
}
return Model::whereRaw($raw_condition, [$number])->firstOrFail();

What is the best way to begin transactions?

I have installed cron on server who runs each 3 minutes some functions.
this are functions:
$xmldb->sendOddsToDb();
$xmldb->copyHAtoHandicap();
$xmldb->sendFixturesToDb();
$xmldb->fillBaby();
Each function has:
try{
$this->conn->connect(); //connect to database
$this->PDO->beginTransaction(); // begin
$stmt = $this->PDO->prepare($this->insTLS);
//some params not important
$this->PDO->commit(); //SAVE
$this->conn->close(); //CLOSE
}
catch(Exception $e){
$this->PDO->rollBack();
}
Now my question, is better to use transactions like this, for each function new transaction or is better to start just once, and commit on end of all functions?
For example:
try{
$this->conn->connect(); //connect to database
$this->PDO->beginTransaction(); // begin
$xmldb->sendOddsToDb();
$xmldb->copyHAtoHandicap();
$xmldb->sendFixturesToDb();
$xmldb->fillBaby();
$this->PDO->commit(); //SAVE
$this->conn->close(); //CLOSE
}
catch(Exception $e){
$this->PDO->rollBack();
}
I need to insert as fastest possible data to db, because i get data from feed where more than 100 000 rows each 3 minutes.
I'd suggest reading this PHP PDO Transactions Documentation.
First off, there's no difference if you explicitly begin a transaction, execute a PDOStatement, and commit that transaction or just simply execute the transaction.
Second, if the four database functions are dependent on each other, wrap them all in a transaction.
Third, regardless of whether or not the functions are related, wrapping them in a single transaction will definitely be faster.

Database connection issue with long time running shell in CakePHP

I have following database configuration in database.php file from my CakePHP app:
public $default = array(
'datasource' => 'Database/Mysql',
'persistent' => false,
'host' => 'localhost',
'login' => 'root',
'password' => '',
'database' => 'database',
'prefix' => '',
);
All is working fine, except of one queue shell script. The script is looping and waiting for commands to run (for example to update some reports). After a while 1-2 days database data is changing, but the script will still "see" the old data, and results of command is wrong. If I restart shell script, the results are OK... for few days.
I have to mention that I had "lost database connection" issue before in the script and I have solved it by runing every 10-15 min:
$user = $this->User->find('first');
Now I am affraid this is making the connection persistent somehow...
How can I reset the database connection ?
EDIT:
I was just refactoring the code to check if I can set $cacheQueries to false on the Model. But in few parts of the code I am using ConnectionManager directly, and only then I have "cache" problem. If I query database from Model->find results are ok. I need direct queries for performance reasons in few places...
$query = "SELECT COUNT(1) as result
FROM
......
";
$db = ConnectionManager::getDataSource('default');
$result = $db->query($query);
The property $cacheQueries which #burzum mentioned, seems not to be in use in any cake model method.
But I found another interesting fact in the source of the DboSource.
You need to use the second parameter of the DboSource::query() method to turn off the caching. Or the third if you want to provide additional parameters for the DboSource::fetchAll() method.
Eventhough this will fix your problem, you should write your queries with the Model::find() method that CakePHP offers.
You should only not use them if they are seriously impacting your performance.
func0der
Try to set these two model properties to false:
$cacheQuery http://api.cakephp.org/2.4/source-class-Model.html#265
$cacheSources http://api.cakephp.org/2.4/source-class-Model.html#499

Call Multiple Stored Procedures with the Zend Framework

I'm using Zend Framework 1.7.2, MySQL and the MySQLi PDO adapter. I would like to call multiple stored procedures during a given action. I've found that on Windows there is a problem calling multiple stored procedures. If you try it you get the following error message:
SQLSTATE[HY000]: General error: 2014
Cannot execute queries while other
unbuffered queries are active.
Consider using
PDOStatement::fetchAll().
Alternatively, if your code is only
ever going to run against mysql, you
may enable query buffering by setting
the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY
attribute.
I found that to work around this issue I could just close the connection to the database after each call to a stored procedure:
if (strtoupper(substr(PHP_OS, 0, 3)) === 'WIN') {
//If on windows close the connection
$db->closeConnection();
}
This has worked well for me, however, now I want to call multiple stored procedures wrapped in a transaction. Of course, closing the connection isn't an option in this situation, since it causes a rollback of the open transaction. Any ideas, how to fix this problem and/or work around the issue.
More info about the work around
Bug report about the problem
I has same errors when called queries like this(variables to use in next query)
$db->query("SET #curr = 0.0;");
To fix this I've changed my config file to
'database' => array(
'adapter' => 'mysqli',
This pattern of preparing, executing and then closing each $sql statement that calls a stored procedure does work.
public function processTeams($leagueid,$raceid,$gender)
{
$db = Zend_Db_Table::getDefaultAdapter();
$sql = $db->prepare(sprintf("CALL addScoringTeams(%d,%d,'%s')",$leagueid,$raceid,$gender));
$sql->execute();
$sql->closeCursor();
$this->logger->info(sprintf("CALL addScoringTeams(%d,%d,'%s')",$leagueid,$raceid,$gender));
$sql1 = $db->prepare(sprintf("CALL updateScoringTeamTotals(%d)",$raceid));
$sql1->execute();
$sql1->closeCursor();
$this->logger->info(sprintf("CALL updateScoringTeamTotals(%d)",$raceid));
$sql2 = $db->prepare(sprintf("CALL updateScoringTeamClasses(%d,'%s')",$raceid,$gender));
$sql2->execute();
$sql2->closeCursor();
$this->logger->info(sprintf("CALL updateScoringTeamClasses(%d,'%s')",$raceid,$gender));
}
You can use prepare statement . No need to change driver
$sql = "CALL procedure()";
$stmt = $this->db->createStatement();
$stmt->prepare($sql);
$result = $stmt->execute();

Categories