I would like to test a process that queries across multiple schemas using PHPUnit's PHPUnit_Extensions_Database_TestCase class. I've dug through the documentation, SO and the source code, but it seems like I have to set up my database using something like:
protected function getConnection()
{
$this->pdo = new PDO('mysql:host=localhost;dbname=unit_test_schema', 'xxxx', 'yyyy');
return $this->createDefaultDBConnection($this->pdo, 'unit_test_schema');
}
protected function getDataSet()
{
return $this->createXMLDataSet(DB_SETUP_DIR.'/schema.xml');
}
Is there a way to somehow use more than one schema so I can test a query like:
SELECT *
FROM schema1.tableA
JOIN schema2.tableB
USING (id)
EDIT: To be clear, the issue I'm trying to resolve is that I can only figure out how to pass schema setup files to one database. I'd like to find a way to say "create tables1.xml in schema1 and tables2.xml in schema2." The tables referenced in the different xml files would be filled-and-killed for each test.
One thing I've done is created a definition for each schema, then overrode the definition when unit testing
define('schema1', 'schema1_prod');
define('schema2', 'schema2_prod');
Then for unit testing
define('schema1', 'unit_tests');
define('schema2', 'unit_tests');
If you have like table names across multiple schemas this will still break, but it should help if you don't.
In my symfony project, I have a "complex" query that looks like:
$d = Doctrine_Core::getTable('MAIN_TABLE')
// Create the base query with some keywords
->luceneSearch($keywords)
->innerJoin('w.T1 ws')
->innerJoin('ws.T2 s')
->innerJoin('w.T3 piv')
->innerJoin('piv.T4 per')
->innerJoin('w.T5 st')
...
->innerJoin('doc.T12 docT')
->innerJoin('w.Lang lng')
->execute();
I added all those innerJoin to reduce the number of query due to my data model. Actually all data are recovered with this only query.... but the query took from 2 to 20 sec. depends on keywords.
I decided to use memcache because data are not changing all the time.
What I've done is configuring memcache and adding
...
->useResultCache(true)
->execute();
to my query.
What is strange is that :
The first time (when the cache is empty/flushed), only one query is execute
The second time, ~130 ares executed and it take more time than the first...
Those "new" queries are retrieving data from "inner join" for each record.
What I don't undestand is why "innerjoined" data are not saved in the cache?
I tried to change the hydrate mode but it seems not to be influent.
Someone has an idea?
After a whole day to googlise, to analyse doctrine and become desperate, I found an article that explain the solution:
class User extends BaseUser{
public function serializeReferences($bool=null)
{
return true;
}
}
The problem was the profile object was not getting stored in the result cache and thus causing a query each time it was called from the user object. After much hunting around, a long time in #doctrine, and a few leads from a couple of people, it turns out, by default, Doctrine will only serialize the immediate relation to the main object. However, you can make it so that it will serialize objects further down the line by overriding the function serializeReferences to return true in the class you want to serialize references from. In my example this is the User class. Since our application will never only need the ‘User’ class to be serialized on a result cache I completely overrode the function and made it always return true
http://shout.setfive.com/2010/04/28/using-doctrine-result-cache-with-two-deep-relations/
I'm kind of proud not to accept concepts if there is no good reason. But I'm in doubt about using active record pattern. Currently I'm using zend but say code Igniter has active record.
I dont use.Because
sql is sql it has own syntax.
you can copy to sql editor and it works (if it is working!)
you dont learn another syntax
you dont need to kill your script to gather if active record is writing sql the way you expected
but active record has
you pretend writing like objective php.
When you need to move another db(oracle>mysql :p), you dont need to change rand function to random, active record can make it for you.
does active record have much more capability that I am missing? Can you give some example cases where active record could be a life saver?
An ActiveRecord is
An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.
This is not what CodeIgniter uses. CI's AR is a basic query object.
The main benefit of an ActiveRecord is it's simplicity. If your application is mainly doing simple CRUD operations and your Table structure matches the ActiveRecord very closely, then it's a good choice. It's easy to abstract CRUD in that case. And you can still add handcrafted SQL to it for certain more complex row manipulations.
class User
{
protected static $dbAdapter;
protected $username;
…
public static function findById($id)
{
$result = self::$dbAdapter->query(
sprintf('SELECT * FROM users WHERE id = %d', $id)
);
if($result) {
return new User($result);
}
}
public function create()
{
try {
return self::$dbAdapter->query(
sprintf(
'INSERT into users …',
$this->username,
…
);
);
} catch …
}
…
}
You'd use that in your application like this:
$john = User::findById(123);
echo $user->username; // John
$jane = new User(array(
'username' => 'Jane'
));
$jane->create();
You definitely don't want to use ActiveRecord if the your rows and the AR don't match closely. AR is an object representing a database row. AR couples the object design to the database design. AR is not an ORM. Trying to put that into it is not practical. If you find you are in need of more juicy Domain Models, you won't be happy with it, because it will ultimately hamper your development due to object-relational impedance mismatch.
Additional readings:
Kore Nordmann: Why Active Record sucks
Bill Karwin: ActiveRecord does not suck
Well this is a simple design question I've wondered about many times and never found a satisfying solution for. My example is with php-sql, but this certainly applies to other languages too.
I have a small database table containing only very few entries, and that almost never needs updating. eg this usertype table:
usertype_id (primary key) | name | description
---------------------------+------------+-------------------
1 | 'admin' | 'Administrator'
2 | 'reguser' | 'Registered user'
3 | 'guest' | 'Guest'
Now in the php code, I often have to check or compare the type of user I'm dealing with. Since the user types are stored in the database, I can either:
1) Select * from the usertype table at class instantiation, and store it in an array.
Then all the ids are available to the code, and I can do a simple select to get the rows I need. This solution requires an array and a db query every time the class is instantiated.
$query = "SELECT info, foo FROM user WHERE usertype_id = ".$usertypes['admin'];
2) Use the name column to select the correct usertype_id, so we can effectively join with other tables. This is more or less equivalent to 1) but without needing to cache the whole usertype table in the php object:
$query = "SELECT info, foo FROM user JOIN usertype USING (usertype_id) WHERE usertype.name = 'admin' ";
3) Define constants that match the keys in the usertype table:
// As defines
define("USERTYPE_ADMIN",1);
define("USERTYPE_REGUSER",2);
//Or as class constants
const USERTYPE_ADMIN = 1;
const USERTYPE_REGUSER = 2;
And then do a simple select.
$query = "SELECT info, foo FROM user WHERE usertype_id = " . USERTYPE_ADMIN;
This is probably the most resource-efficient solution, but it is bad to maintain, as you have to update both the table and the code if you need to modify something in the usertype table..
4) Scrap the usertype table and only keep the types in the php code. I don't really like this because it lets any value get into the database and get assigned to the type of user. But maybe, all things considered, it isn't so bad and i'm just complicating something that should be simple..
Anyways, to sum it up the solution I like most is #2 because it's coherent and with an index on usertype.name, it can't be that bad. But what I've often ended up using is #3, for efficiency.
How would you do it? Any better solutions?
(edit: fixed query in #2)
I would suggest #3 to avoid useless queries, and prevent risk of behavior changes if existing DB table rows are incidentally modified:
Adding the necessary constants in the model class:
class Role // + use namespaces if possible
{
// A good ORM could be able to generate it (see #wimvds answer)
const ADMIN = 1;
const USER = 2;
const GUEST = 3;
//...
}
Then querying like this makes sense:
$query = "SELECT info, foo FROM user WHERE role_id = ".Role::ADMIN;
With an ORM (e.g. Propel in the example below) you'll end up doing:
$isAdminResults = UserQuery::create()->filterByRoleId(Role::ADMIN);
I almost always go for option 3). You could generate the code needed automatically based on what is available in the DB. The only thing you have to remember then is that you have to run the script to update/rewrite that info when you add another role (but if you're using phing or a similar build tool to deploy your apps, just add a build rule for it to your deploy script and it will always be run whenever you deploy your code :p).
Why not denormalize the DB table so instead of having usertype_id, you'd have usertype with the string type (admin). Then in PHP you can just do define('USERTYPE_ADMIN', 'admin');. It saves you from having to modify two places if you want to add a user type...
And if you're really worried about any value getting in, you could always make the column an ENUM data type, so it would self manage...
For tables that will contain "type" values especially when is expected such table to change over time I tend to use simple approach:
Add Varchar column named hid (comes from "human readable id") with unique key. Then I fill it with id meaningful to humans like:
usertype_id (primary key) | name | description | hid (unique key)
---------------------------+------------+-------------------+---------------
1 | 'admin' | 'Administrator' | 'admin'
2 | 'reguser' | 'Registered user' | 'user'
3 | 'guest' | 'Guest' | 'guest'
When you need the actual id you will have to do select based on hid column, i.e.
select usertype_id from tablename where hid = "admin"
This is not an efficient approach but it will ensure compatibility of your application among different deployments (i.e. one client may have 1.admin, 2. guest; other client 1.admin, 2. user, etc.). For your case I think #3 is pretty suitable but if you expect to have more than 10 different user roles - try the "hid" approach.
Are you using any kind of framework here? Could these values be stored in a single source - a config file - which both creates a list of the objects in PHP and also populates the table when you bootstrap the database? I'm thinking from a Rails perspective, as it's been a while since I've written any PHP. Solution there would probably be fixtures.
Why not to make it just
foreach (getdbarr("SELECT * FROM usertype") as $row) {
define($row['name'],$row['id']);
}
You shouldn't need a JOIN in every query to fetch the information about types/roles. You can keep your 'user' model and 'role' models separate in the data access objects (DAO) -- especially since there are so few records for user types.
In most cases where I have a limited number of options that I'd otherwise be joining against a large table, I cache them in memcached as an associative array. In the event I need some information about a particular relationship (like a role) I just lazy load it.
$user = DAO_User::get(1); // this pulls a JOIN-less record
$role = $user->getRole(); // lazy-load
The code for $user->getRole() can be something like:
public function getRole() {
// This comes from a cache that may be called multiple
// times per request with no penalty (i.e. store in a registry)
$roles = DAO_UserRoles::getAll();
if(isset($roles[$this->role_id]))
return $roles[$this->role_id];
return null; // or: new Model_UserRole();
}
This also works if you want to display a list with 1000 users on it. You can simply render values for that column from a single $roles associative array.
This is a major performance improvement on the SQL end, and it goes a long way to reducing complexity in your code base. If you have several other foreign keys on the user table you can still use this approach to grab the necessary information when you need it. It also means you can have dependable Model_* classes without having to create hybrids for every possible combination of tables you might JOIN -- which is much better than simply getting a result set, iterating it, and freeing it.
Even with more than 100 rows on both sides of your JOIN, you can still use the lazy load approach for infrequent or highly redundant information. With a reasonable caching service in your code, there's no penalty for calling DAO_UserRole::get(1500) multiple times because subsequent calls during the same request shouldn't hit the database twice. In most cases you're only going to be displaying 10-25 rows per page out of 1000s, and lazy loading will save your database engine from having to JOIN all the extraneous rows before you actually need them.
The main reason to do a JOIN is if your WHERE logic requires it, or if you need to ORDER BY data from a foreign key. Treating JOINs as prohibitively expensive is a good habit to be in.
For basicly static lookup tables, I generally make static constant files (such as your #3). I generally use classes such as:
namespace Constants;
class UserTypes {
const ADMIN = 1;
const USER = 2;
const GUEST = 3;
}
$id = Constants\UserTypes::ADMIN;
When I'm using lookup takes that are a bit more variable, then I'll pull it into a object and then cache it for 24 hours. That way it only gets updated once a day. That will save you from making database round trips, but allow you to deal with things in code easily.
Yeah, you're right about avoiding #3 and sticking with #2. As much as possible, look-ups like when you use a usertype table to contain the roles and then relate them to the user table using the id values should stay in the database. If you use constants, then the data must always rely on your php code to be interpreted. Also, you can enforce data integrity by using foreign keys (where servers allow) and it will allow you to port the reporting from your php code to other reporting tools. Maintenance also becomes easier. Database administrators won't need to know php in order to derive the meanings of the numbers if you used #3, should they ever be asked to aid in reports development. It may not seem too relevant, but in terms of maintenance, using stored procedures than embedded sql in your php code would also be maintenance-friendly in several ways, and will also be advantageous to DBAs.
I'd go for option #2 and use the join as it is intended to be used. You never know what the future will throw up, it's always better to be prepared today!
With regards to leaving the database alone as much as possible for such operations, there is also the possibility of caching in the long term. For this route, within PHP an option is to use a file cache, one that will only get updated when time calls for it. For the framework I have created, here's an example; I'd be interested to know what people think:
Note:
(LStore, LFetch, GetFileName) belong to a Cache object which gets called statically.
(Blobify and Unblobify) belong to a SystemComponent object which is always alive
Each piece of cache data has a key. this is the only thing you ever have to remember
public function LStore($key,$data, $blnBlobify=true) {
/* Opening the file in read/write mode */
$h = fopen(self::GetFileName($key, 'longstore'),'a+');
if (!$h) throw new Exception('Could not write to cache');
flock($h,LOCK_EX); // exclusive lock, will get released when the file is closed
fseek($h,0); // go to the start of the file
/* truncate the file */
ftruncate($h,0);
if($blnBlobify==true) { $data = SystemComponent::Blobify(array($data)); }
If (fwrite($h,$data)===false) {
throw new Exception('Could not write to cache');
}
fclose($h);
}
public function LFetch($key) {
$filename = self::GetFileName($key, 'longstore');
if (!file_exists($filename)){ return false;}
$h = fopen($filename,'r');
if (!$h){ return false;}
/* Getting a shared lock */
flock($h,LOCK_SH);
$data = file_get_contents($filename);
fclose($h);
$data = SystemComponent::Unblobify($data);
if (!$data) {
/* If unserializing somehow didn't work out, we'll delete the file */
unlink($filename);
return false;
}
return $data;
}
/* This function is necessary as the framework scales different directories */
private function GetFileName($key, $strCacheDirectory='') {
if(!empty($strCacheDirectory)){
return SystemComponent::GetCacheAdd() . $strCacheDirectory.'/' . md5($key);
} else {
return SystemComponent::GetCacheAdd() . md5($key);
}
}
public function Blobify($Source){
if(is_array($Source)) { $Source = serialize($Source); }
$strSerialized = base64_encode($Source);
return $strSerialized;
}
public function Unblobify($strSerialized){
$Decoded = base64_decode($strSerialized);
if(self::CheckSerialized($Decoded)) { $Decoded = unserialize($Decoded); }
return $Decoded;
}
function CheckSerialized($Source){
$Data = #unserialize($Source);
if ($Source === 'b:0;' || $Data !== false) {
return true;
} else {
return false;
}
}
Now when it comes to accessing the actual data, I just call a fetch. For making sure it is up to date, I tell it to store. In your case, this would be after updating the usertype table.
I have a sqlite3 database on my harddrive (file.db) with 5 tables.
I'd like to copy 3 of these tables to an in-memory database (:memory:).
Is there a simple way to do so using PHP5's PDO format?
Not a pdo-specific solution that may or may not be sufficient in your case:
create a :memory: database
Attach the existing database file
CREATE TABLE ... AS SELECT * FROM ...
Detach the database file
edit: an example
First an example database stored in mydb.sq3
<?php
$pdo = new PDO('sqlite:mydb.sq3');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec('CREATE TABLE foo(x INTEGER PRIMARY KEY ASC, y, z)');
$stmt = $pdo->prepare("INSERT INTO foo (x,y,z) VALUES (:x,:y,:z)");
$stmt->bindParam(':x', $x);
$stmt->bindParam(':y', $y);
$stmt->bindParam(':z', $z);
for($x=0; $x<100; $x++) {
$y = $x*2;
$z = $x*2+1;
$stmt->execute();
}
Now we have a :memory: database and want to transfer the table foo
<?php
$pdo = new PDO('sqlite::memory:');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec('ATTACH "mydb.sq3" as filedb');
$pdo->exec('CREATE TABLE bar AS SELECT * FROM filedb.foo');
$pdo->exec('DETACH filedb');
Done. But let's take a look at the sqlite_master table
foreach($pdo->query('SELECT sql FROM sqlite_master') as $row) {
echo $row['sql'];
}
this prints
CREATE TABLE bar(x INT,y,z)
The INTEGER PRIMARY KEY ASC declaration is lost. Might be sufficient though....
If that's what you need to do, then VolkerK's answer is the one I'd provide, but I feel that I have to point out that you're going to read the contents of those tables into memory each time you run that code (every time that page loads?), so it might be better just to query the data files from disk.
Note that one could always use some kind of shared memory mechanism (e.g. APC, memcache, etc..) to keep sqlite's in-memory databases persistent across connections.
You can dump the database at the end of the connection, save it as apc variable and then load and run again from apc at the beginning of the next execution.
Using the method outlined by VolkerK roughly doubled the performance of my code when using a ~150Mb sqlite database.
Loading the database into an in-memory sqlite db didn't require any other changes to my existing code.
My use case was batch processing data so I didn't have to deal with the problems Wez Furlong highlights.
Reading the data into memory was surprisingly fast. The whole 150Mb was loaded into memory from SSD in less than two seconds. It seemed too good to be true so I checked and rechecked the data.
Many thanks to VolkerK for a magic solution!
I also found that when I indexed the in-memory tables my queries executed three times faster
//adapting VolkerK's example...
//loop through tables from the local sqlite db
$tables = array('companies', 'directors', 'previous_names');
foreach($tables as $table){
//load each table into memory
$pdo->exec("CREATE TABLE $table AS SELECT * FROM filedb.$table");
//index each table on the relevant columns
$pdo->exec("CREATE INDEX IF NOT EXISTS `".$table."_company_number`
ON $table (`company_number`);");
}
If you need a small database for tests, you may export your databse to an SQL file and then execute it as a single query in PHP:
class EphemeralPDO extends \PDO {
public function __construct() {
parent::__construct('sqlite::memory:', null, null, [PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]);
$queries = file_get_contents(__DIR__ . '/database.export.sql');
$this->exec($queries);
}
}