I'm trying to use the Laravel Redis session driver,
Problem is - It seems to always assume usage of the "default" connection. Does anyone know if its possible to use a different connection ?
For example "session" ? So all my session would be on one server?
Also, whenever I don't have a "default" key in the redis array, i get exceptions all over the place. Must I declare a "default" connection ? What's wrong with having my own names?
e.g.
It doesn't look like there's an easy way of doing what you want. The Redis based session driver uses the Redis based cache driver:
http://laravel.com/api/source-class-Illuminate.Cache.RedisStore.html#5-155
Unfortunately that doesn't provide any way to specify which connection to use - it just uses the default connection:
http://laravel.com/api/source-class-Illuminate.Redis.Database.html#3-96
(magic method calls to the above class go straight to the default connection)
A possible way to get what you want might be to write your own class that extends RedisStore with a constructor that takes in a connection name and sets $this->redis to whatever Redis::connection($name) returns. You'd also need to figure out how to get the session handler to use your own cache driver though. I don't know how easy it would be or how you'd do that - it might be difficult to try and wire it all up.
There are a few ways to go about this:
I will outline why you were getting errors and how to fix this the simple way (incase someone else has the same issue):
open at vendor/laravel/framework/src/illuminate/Redis/Database.php
create a $name variable for your connection at the top of the class
class Database {
/**
* The host address of the database.
*
* #var array
*/
protected $clients;
protected $name;
change the connection function to expect your own 'default' fallback and 'set the name' of your current connection.
on the command method you will need to provide your 'new connection name' ($this->name) instead of the default one.
that's it!
now from anywhere in the application you can use the 'Redis' facade to access your connections just like this:
REDIS::CONNECTION('small');
REDIS::CONNECTION('large');
REDIS::CONNECTION('session');
REDIS::CONNECTION('etc');
Change the session driver in Config\session.php from native to redis
Related
I've managed to get a stable load balanced front end servers that can scale horizontally quite well however the next bottle neck would be the db. There was a blog post discussing scaling dbs horizontally however very little detail on it. I'm currently using PostgreSQL and so the only plugin I've found wouldn't work.
Are my only options creating my own HAProxy or rewriting the PostgreSQL plugin to allow connections with read replicas?
I'm using AWS for all my hosting
Firstly - I'd love to be corrected on this!
Having only had a quick look through some of the ORM classes in a SilverStripe 3.5 site, it looks like while the ORM does support multiple database connections (see DB::get_conn with argument for name) it is designed for specific use cases in mind. That is to say, you may have a module that needs to write to a specific database, so this would allow it to.
What you want is native and automatic support for this within the framework, so that all reads go to your slave(s) and writes go to your master. Unfortunately, it doesn't look like this comes out of the box. You might be able to achieve it by overloading a couple of the core SQL classes using the injector.
If you were to try it, this answer outlines how you could separate select statements out from the rest and run them through a different database connector.
As a quick example of how you might go at achieving this with SQLSelect, you will notice that it is injectable, which means you can easily overload it.
File: mysite/_config/injector.yml
Injector:
SQLSelect:
class: ReadOnlySQLSelect
You need to register a new database connection with the DB class:
File: mysite/_config.php
$readDatabaseConfig = array(/** define your DB credentials here, as with the default $databaseConfig **/);
if (!DB::connect($readDatabaseConfig, 'default_read')) {
user_error('Failed to connect to read replica DB!', E_USER_ERROR);
}
Now, overload the SQLSelect class and replace the parts of it that call the DB class methods. This class inherits from SQLExpression which is the class the contains the methods you actually care about in this instance:
File: mysite/code/ReadOnlySQLSelect.php
class ReadOnlySQLSelect extends SQLSelect
{
public function sql(&$parameters = array())
{
// Changed from SQLExpression: third parameter passed as connection name
$sql = DB::build_sql($this, $parameters, 'default_read');
if (empty($sql)) {
return null;
}
if ($this->replacementsOld) {
$sql = str_replace($this->replacementsOld, $this->replacementsNew, $sql);
}
return $sql;
}
public function execute()
{
$sql = $this->sql($parameters);
// Changed from SQLExpression: skip DB::prepared_query since it doesn't allow
// you to provide the connection name - replace it with its contents instead.
$conn = DB::get_conn('default_read');
return $conn->preparedQuery($sql, $parameters);
}
}
Note: SQLSelect::unlimitedRowCount should technically be replaced where it calls DB::prepared_query, since the prepared query method calls DB::get_conn with no arguments, so will always return the default connection. You could replace the DB::prepared_query line the same as used above:
$conn = DB::get_conn('default_read');
$result = $conn->preparedQuery($sql, $innerParameters);
If you implement the above method, also change new SQLSelect() to SQLSelect::create(), otherwise you'll end up with some queries that still hit the master server because it'll bypass your class by not using the injector.
There's also an instance in SQLConditionalExpression that you should replace too (::toSelect) but that is likely to affect query transformations from other child implementations of that class, and you won't be able to do much about it without either (A) PRing a fix to the framework or (B) overloading all the other SQL* classes.
At this point you should have everything you need to route select queries to your default_read connection.
Infrastructure
On the infrastructure side, you should be able to set up read replicas through the RDS console. When you do so it will provide you with a DNS endpoint for your replica node(s), which you can use in your _config.php to configure the connection to the read replica database.
If this works for you, you should create a module for it and put it up on GitHub - this would definitely be useful for others in future!
You may also consider making pull requests to the framework to add additional arguments to methods like DB::prepared_query to accept a connection name.
Also worth noting is that if you're using the mysqlnd database adapter you may be able to take advantage of read/write splitting, implemented with some sort of injector overloading but all handled at a lower level than the application layer.
I need to modify the existing 'default' connection.
I tried to write it like this:-
ConnectionManager::config('default',$consoptions);
It gives an error telling me that we cannot reconfigure existing key "default".
Is there any way to achieve what I want?
The example shown in the CakePHP Cookbook is wrong and causes the error
Cannot reconfigure existing key "default"
You actually need to drop the default database connection configuration before replacing it with a new one.
In addition, the ConnectionManager::config() methos is deprecated in favour of ConnectionManager::setConfig()
This is a working example that modifies the default database connection:
use Cake\Datasource\ConnectionManager;
$dsn = 'mysql://root:password#localhost/my_database';
ConnectionManager::drop('default');
ConnectionManager::config('default', ['url' => $dsn]);
It gives an error telling me that we cannot reconfigure existing key "default" Is there any way to achieve what I want?
Drop the config then configure it again.
Spend some time checking the API, it's almost all there.
I have developed a project under which several sql query have been used. Now I want to monitor some query in order increase security. So I want every query to be passed through a function first. As there are too many queries so I can not go back and edit every file and query. Is there a way that I can trap into queries before they are sent to mysql server?
There are four ways to accomplish this depending on what you are using, the last being the much more reliable.
The General Query Log
MySQL provides a mechanism to log just about everything that the mysqld process is doing, via the general query log. As you described in your question you probably do not have persistent connections, so you will need to either:
Enable the MySQL general query log when the mysqld process is started, with the --log[=file_name]
Set a global/session variable with SET GLOBAL general_log = 'ON'.
Fore more information about the general query log, see the MySQL 5.1 reference manual.
Using sed (or manually!)
This technique involves creating a a new function, and renaming all of the mysqli_* function calls to call another function.
Presuming your newly created function is named proxy_query(), you can use sed to traverse through all files and change them automatically:
sed i '.bck' 's/mysqli_query/proxy_query/'
The -i paramater specifies that the file should be edited in place, and that a copy should be made of the original file and have a .bck extension appended.
The runkit extension
I must admit that I'm being naive here, and that I haven't used this particular extension before - but it is possible to rename functions with this PECL extension.
The requirements for this extension can be found here, and note that it is not bundled with PHP.
As with above, you can create a proxy function where all calls will go through. Let's assume it's also called proxy_query. Usage would go something like this:
// rename the function (a very bad idea, really!)
runkit_function_renam('mysqli_connect', 'proxy_super');
function mysqli_query($query, $resultmode = MYSQLI_STORE_RESULT)
{
// do something with the SQL in $query
// .. and call mysqli_query, now proxy_super
return proxy_super($query, $resultmode);
}
I have to note here that this method is highly discouraged. You shouldn't ever need to set default PHP functions.
Using Pdo/OO-mysqli
This is the simplest technique, and probably the most reliable as well. If you're using Pdo already, you can simply extend the \Pdo class. A similar approach could be used with MySQL Improved(mysqli):
class MyPdo extends \Pdo
{
public function query($query [, ... ])
{
// do something with $query
return parent::query($query [, ... ]);
}
}
Also note here, that this will only work if you are using Pdo, and if you are able to change the instantiation of the Pdo object, to overwrite it to your own class: MyPdo.
For more information about the \Pdo class, and it's children, see the manual.
If you want to monitor incoming queries using SQL profiler can be an excellent way to gather information on what's going on inside SQL without passing it all through a single function or procedure.
PHP's Mongo driver lacks a renameCommand function. There is reference to do this through the admin database. But it seems more recent versions of the Mongo driver don't let you just "use" the admin database if do don't have login privileges on that database. So this method no longer works. I've also read this doesn't work in sharded environments although this isn't a concern for me currently.
The other suggestion people seem to have is to iterate through the "from" collection and insert into the "to" collection. With the proper WriteConcern (fire and forget) this could be fairly fast. But it still means pulling down each record over the network into the PHP process and then uploading it back over the network back into the database.
I ideally want a way to do it all server-side. Sort of like an INSERT INTO ... SELECT ... in SQL. This way it is fast, network efficient and a low load on PHP.
I have just tested this, it works as designed ( http://docs.mongodb.org/manual/reference/command/renameCollection/ ):
$mongo->admin->command(array('renameCollection'=>'ns.user','to'=>'ns.e'));
That is how you rename an unsharded collection. One problem with MR is that it will change the shape of the output from the original collection. As such it is not very good at copying a collection. You would be better off copying it manually if your collection is sharded.
As an added note I upgraded to 1.4.2 (which for some reason comes out from the pecl channel into phpinfo() as 1.4.3dev :S) and it still works.
Updates:
Removed my old map/reduce method since I found out (and Sammaye pointed out) that this changes the structure
Made my exec version secondary since I found out how to do it with renameCollection.
I believe I have found a solution. It appears some versions of the PHP driver will auth against the admin database even though it doesn't need to. But there is a workaround where the authSource connection param is used to change this behavior so it doesn't auth against the admin database but instead the database of your choice. So now my renameCollection function is just a wrapper around the renameCollection command again.
The key is to add authSource when connecting. In the below code $_ENV['MONGO_URI'] holds my connection string and default_database_name() returns the name of the database I want to auth against.
$class = 'MongoClient';
if( !class_exists($class) ) $class = 'Mongo';
$db_server = new $class($_ENV['MONGO_URI'].'?authSource='.default_database_name());
Here is my older version that used eval which should also work although some environments don't allow you to eval (MongoLab gives you a crippled setup unless you have a dedicated system). But if you are running in a sharded environment this seems like a reasonable solution.
function renameCollection($old_name, $new_name) {
db()->$new_name->drop();
$copy = "function() {db.$old_name.find().forEach(function(d) {db.$new_name.insert(d)})}";
db()->execute($copy);
db()->$old_name->drop();
}
you can use this. "dropTarget" flag is true then delete exist database.
$mongo = new MongoClient('_MONGODB_HOST_URL_');
$query = array("renameCollection" => "Database.OldName", "to" => "Database.NewName", "dropTarget" => "true");
$mongo->admin->command($query);
I am using Zend Framework for my PHP developments and here is a small function I used to execute a query. This is not about an error. The code and everything works fine. But I want to know some concept behind this.
/**
* Get dataset by executing sql statement
*
* #param string $sql - SQL Statement to be executed
*
* #return bool
*/
public function executeQuery($sql)
{
$this->sqlStatement = $sql;
if ($this->isDebug)
{
echo $sql;
exit;
}
$objSQL = $this->objDB->getAdapter()->prepare($sql);
try
{
return $objSQL->execute();
}
catch(Exception $error)
{
$this->logMessage($error->getMessage() . " SQL : " .$sql);
return false;
}
return false;
}
Bellow are unclear areas for me.
How Zend_Db_Table_Abstract Maintain database connections?
Is it creating new connection all the time when I call this function or Does it have some connection pooling?
I didn't write any coding to open or close database connection. So will zend framework automatically close connections?
If this open and close connection works all the time if I execute this function, Is there any performance issue?
Thank you and appreciate your suggestions and opinion on this.
Creating Connection
Creating an instance of an Adapter class does not immediately connect to the RDBMS server. The Adapter saves the connection parameters, and makes the actual connection on demand, the first time you need to execute a query. This ensures that creating an Adapter object is quick and inexpensive. You can create an instance of an Adapter even if you are not certain that you need to run any database queries during the current request your application is serving.
If you need to force the Adapter to connect to the RDBMS, use the getConnection() method. This method returns an object for the connection as represented by the respective PHP database extension. For example, if you use any of the Adapter classes for PDO drivers, then getConnection() returns the PDO object, after initiating it as a live connection to the specific database.
It can be useful to force the connection if you want to catch any exceptions it throws as a result of invalid account credentials, or other failure to connect to the RDBMS server. These exceptions are not thrown until the connection is made, so it can help simplify your application code if you handle the exceptions in one place, instead of at the time of the first query against the database.
Additionally, an adapter can get serialized to store it, for example, in a session variable. This can be very useful not only for the adapter itself, but for other objects that aggregate it, like a Zend_Db_Select object. By default, adapters are allowed to be serialized, if you don't want it, you should consider passing the Zend_Db::ALLOW_SERIALIZATION option with FALSE, see the example above. To respect lazy connections principle, the adapter won't reconnect itself after being unserialized. You must then call getConnection() yourself. You can make the adapter auto-reconnect by passing the Zend_Db::AUTO_RECONNECT_ON_UNSERIALIZE with TRUE as an adapter option.
Closing a Connection
Normally it is not necessary to close a database connection. PHP automatically cleans up all resources and the end of a request. Database extensions are designed to close the connection as the reference to the resource object is cleaned up.
However, if you have a long-duration PHP script that initiates many database connections, you might need to close the connection, to avoid exhausting the capacity of your RDBMS server. You can use the Adapter's closeConnection() method to explicitly close the underlying database connection.
Since release 1.7.2, you could check you are currently connected to the RDBMS server with the method isConnected(). This means that a connection resource has been initiated and wasn't closed. This function is not currently able to test for example a server side closing of the connection. This is internally use to close the connection. It allow you to close the connection multiple times without errors. It was already the case before 1.7.2 for PDO adapters but not for the others.
More information