I'm writing some testing code on a Drupal 6 project, and I can't believe how slow these tests seem to be running, after working with other languages and frameworks like Ruby on Rails or Django.
Drupal.org thinks this question is spam, and won't give me a way to prove I'm human, so I figured SO is the next base place to ask a question like this, and get a sanity check on my approach to testing.
The following test code in this gist is relatively trivial.
http://gist.github.com/498656
In short I am:
creating a couple of content types,
create some roles,
creating users,
creating content as the users,
checking if the content can be edited by them
checking if it's visible to anonymous users
And here's the output when I run these tests from the command line:
Drupal test run
---------------
Tests to be run:
- (ClientProjectTestCase)
Test run started: Thu, 29/07/2010 - 19:29
Test summary:
-------------
ClientProject feature 52 passes, 0 fails, and 0 exceptions
Test run duration: 2 min 9 sec
I'm trying to run tests like this before I push code to a central repo everytime, but if it's taking this long this early on the project, I dread to think about it further down the line when we have ever more test cases.
What can I do to speed this up?
I'm using a MacbookPro with:
4gb of ram,
2.2ghz Core 2 Duo processor,
PHP 5.2,
Apache 2.2.14, without any opcode caching,
Mysql 5.1.42 (Innodb tables are my default)
A 5400 RPM laptop hard drive
I understand that in the examples above I'm bootstrapping Drupal each time, and this is a very expensive operation, but this isn't unheard with other frameworks, like Ruby on Rails, or Django, and I don't understand why it's averaging out at a little over a minute per testcase on this project.
There's a decent list of tricks here for speeding up Drupal 7, many of which look like they'd apply to Drupal 6 as well, but I haven't yet had a chance to try them yet, and it would be great to hear how these have worked out for others by I blunder down further blind alleys,
What has worked for you when you've been working with Drupal 6 in this situation, and where are the quick wins for this?
One minute per test case when I'm expecting to easily more than a hundred test cases feels insane.
It looks like the biggest increase in speed will come from running the test database in a ram disk, based on this post here on Performance tuning tips for Drupal 7 testing on qa.drupal.org
DamZ wrote a modified mysql init.d script for /etc/init.d/mysql on Debian 5 that runs MySQL databases entirely out of tmpfs. It's at http://drupal.org/files/mysql-tmpfs.txt, attached to http://drupal.org/node/466972.
It allowed the dual quad core machine donated to move from a 50 minute test and huge disk I/O with InnoDB to somewhere under 3 minutes per test. It's live as #32 on PIFR v1 for testing.d.o right now. It is certainly the only way to go.
I have not and won't be trying it on InnoDB anytime soon if anyone wants to omit the step on skip-innodb below and try it on tmpfs.
Also there some instructions here for creating a ram disk on OS X, although this is for moving your entire stock of mysql databases into a ram disk, instead of just a single database:
Update - I've tried this approach now with OS X, and documented what I've found
I've been able to cut 30-50% from the test times by switching to a ram disk. Here are the steps I've taken:
Create a ram disk
I've chosen a gigabyte mainly because I've got 4gb of RAM, and I'm not sure how much space I might need, so I'm playing it safe:
diskutil erasevolume HFS+ "ramdisk" `hdiutil attach -nomount ram://2048000`
Setup mysql
Next I ran the mysql install script to get mysql installed on the new ramdisk
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/Volumes/ramdisk
Then, I took the following steps: I made sure the previous mysqld was no longer running, and then started the mysql daemon, making sure we tell it to use ram disk as our data directory, rather than the default location.
/usr/local/mysql/bin/mysqld \
--basedir=/usr/local/mysql \
--datadir=/Volumes/ramdisk \
--log-error=/Volumes/ramdisk/mysql.ramdisk.err \
--pid-file=/Volumes/ramdisk/mysql.ramdisk.pid \
--port=3306 \
--socket=/tmp/mysql_ram.sock
Add the database for testing
I then pulled down the latest database dump on our staging site with drush, before updating where settings.php points to it:
drush sql-dump > staging.project.database.dump.sql
Next was to get this data into the local testing setup on the ram disk. This involved creating a symlink to the ramdisk database socket, and creating the database, granting rights to the mysql user specified in the drupal installation, then loading the database in to start running tests. Step by step:
Creating the symlink - this because the mysql command by default looks for /tmp/mysql.sock, and symlinking that to our short term ram disk was simpler than constantly changing php.ini files
ln -s /tmp/mysql_ram.sock /tmp/mysql.sock
Creating the database (from the comamnd line at the mysql prompt)
CREATE DATABASE project_name;
GRANT ALL PRIVILEGES ON project_name.* to db_user#localhost IDENTIFIED BY 'db_password';
Loading the content into the new database...
mysql project_database < staging.project.database.dump.sql
Run the tests on the command line
...and finally running the test from the command line, and using growlnotify to tell me when tests have finished
php ./scripts/run-tests.sh --verbose --class ClientFeatureTestCase testFeaturesCreateNewsItem ; growlnotify -w -m "Tests have finished."
Two test cases takes around a minute and half still, is still unusably slow - orders of magnitude slower than other frameworks I might have used before.
What am I doing wrong here?
This can't be the standard way of running tests with Drupal, but I haven't been able to find any stats on how long I should expect a test suite to take with Drupal, to tell me otherwise,
The biggest issue with Drupal SimpleTests is it takes a long time to install Drupal, and that's done for every test case.
So use simpletest_clone -- basically, dump your database fresh after installation and it lets you use that dump as the starting point for each test case rather than running the entire installer.
I feel your pain, and your observations are spot on. A suite that takes minutes to run is a suite that inhibits TDD. I've resorted to plain PHPUnit tests run on the command line which run as fast as you'd expect coming from a Rails environment. The real key is to get away from hitting the database at all; use mocks and stubs.
Related
I am in the process of upgrading our MongoDBs from 3.4 (using MMAPv1 storage engine) to 4.2 (using WiredTiger). One thing I have encountered that is pretty much a blocker at this point is a serious slowdown of our tests.
Long story short (more details below) - MongoDB 4.2 WiredTiger is taking much longer to process repeated database setup/teardown in tests. The slowdown is in the ballpark of a factor of 10. The tests used to run about 10 minutes, with 4.2 they run almost 90 minutes. This slowdown reproduces even with just a fraction of tests and seems to come from the setup/teardown stage of the testing.
Environment
A few words about our environment -- we are using PHP with Doctrine ODM to talk to MongoDB. We have about 3000 tests, some pure unit tests, some (many) functional, actually using the database. The tests are running in a Dockerized environment - we spin up a fresh MongoDB Docker container for each pipeline, but I have confirmed that the same slowdown occurs even in a production-like baremetal setting. The experiments below were done on bare metal, to limit problems coming from somewhere else.
Each functional test first drops the database, then loads fixtures into it (+ creates indices) and then the actual test scenario is executed.
Profiling PHP
Running a small subset of the tests and measuring the timing, I get these results:
3.4:
real 0m12.478s
user 0m7.054s
sys 0m2.247s
4.2:
real 0m56.669s
user 0m7.488s
sys 0m2.334s
As you can see, the actual CPU time taken by the tests is about the same, no significant difference there. The real time is very different though, which suggests a lot of waiting (for I/O in this case?).
I have further profiled the PHP code and I can see from the results that there is 9-10x increase in the time spent in this function:
MongoDB\Driver\Manager::executeWriteCommand()
The documentation for that function says:
This method will apply logic that is specific to commands that write (e.g. ยป drop)
That makes me think that the amount of setup/teardown (i.e. dropping collection, creating indexes) will be at play here.
Profiling MongoDB
Profiling PHP pointed at a slowdown in MongoDB so I profiled that as well. The subset of tests that I ran resulted in
1366 profiling documents for 3.4 MMAPv1
2092 profiling documents for 4.2 WiredTiger
Most of the disparity between those numbers can be attributed to the fact that in 4.2 there are no documents for createIndexes (maybe they were added to profiling post-3.4? I don't know).
I filtered the profiling documents to only show those which took at least 1 millisecond (>0). There were:
2 such documents for MongoDB 3.4 (two drop commands)
950+ such documents for MongoDB 4.2 (209x drop, 715x createIndexes, 4x insert, 23x query)
As I mentioned earlier, Mongo 3.4 does not seem to report createIndexes in the profiling. But let's assume all of those commands would take as long as they do in 4.2 (they will probably take shorter, based on the rest of the profiling results though).
Then there are all those drop commands that take up to 15 milliseconds per operation in 4.2. In 3.4 there are also 209 drop commands, but almost all of them are reported to have lasted 0 milliseconds.
There is only a minimal amount of inserting and querying and the size of the collections when those are happening is only a handful of documents (less than 10 per collection, less than 5 collections actually queried and inserted into). This slowdown is not a result of missing caches or indices. Even full scans would be fast in this setting.
Memory and hardware
Most of the discussion I have found regarding this have been around setting an appropriate cache size for the working sets. I ran the tests on a small server with a single core and 4GB RAM with the default cache size (which should be 50% of available memory, i.e. 2GB). That is definitely large enough for all the data the tests could have created. They were truly trivial and most of the time spent on them was on setup/teardown of the database state.
Conclusion
This is the first time I have profiled our tests and their interaction with the database. The ratio of drop-and-index-creation to actual work can definitely be improved, but it has worked so far with MMAPv1 and MongoDB 3.4. Is this type of a slowdown something that is expected with WiredTiger? Is there something I can do to mitigate this?
I am now afraid of upgrading the production MongoDB instances because I don't know how those will behave. If this is related mostly to index creation and database dropping, then I suppose production workload should be fine, but I don't want to take chances. Sadly we are a fairly small company and do not have any performance/stress tests of the production environment.
Edits
Using tmpfs
Since I'm running the tests in Docker and Docker supports tmpfs volumes out-of-the-box, I gave that a try. When using RAM-backed tmpfs as the mount for the MongoDB data, I managed to cut down the test time to about half:
4.2:
real 0m56.669s
user 0m7.488s
sys 0m2.334s
4.2 - tmpfs:
real 0m30.951s
user 0m7.697s
sys 0m2.279s
This is better, but still a far cry from the 12 seconds it takes to run on MMAPv1. Interestingly, using tmpfs with MMAPv1 did not yield a significantly different result.
The real cause of test slowdown - indices
Turns out that our testing framework and fixture loader created indices for all managed collections with each database purge. This resulted in about 100 index creations per test case and this was what caused the slowdown. I did not find a concrete proof directly from Mongo but it seems that index creation with WiredTiger is significantly slower than with MMAPv1. Removing the index creation from the test setup code sped up the tests significantly, getting us back to the pre-upgrade times.
The vast majority of our tests do not need the indexes and their creation takes much longer than the speedup in queries they provide. I implemented an option to enforce index creation for test cases where the developer knows they will need them. That is an acceptable solution for us.
Put the database's data into memory. On Linux, I recommend zram.
In my experience zram is 2x as fast as top of the line nvme ssd (samsung 860 pro I think) in raid 0 and is I think almost 10x as fast as a single consumer-grade laptop ssd. The difference should be even bigger for spinning disk or storage accessed over network.
MongoDB has various other storage engines (there's one called "ephemeral for test" I believe) but they don't support transactions, so you need to use WT if your application makes use of 4.2 (or even 4.0 I think) functionality.
In production you are most likely not dropping collections every request so actual performance difference between 3.x and 4.2 should be smaller.
Use ephemeralForTest engine!
Even though #d-sm mentioned this in their answer, I missed it so let me emphatise it for future readers.
If you simply need a quick storage engine to run your unit tests against, and you need to update MongoDB to v4.2+ (so mmapv1 engine is not an option anymore), you can use the ephemeralForTest engine instead.
This is NOT TO BE CONFUSED with the Enterprise-only In-Memory engine, and it was silently added in v3.2 (see changelog).
This engine is not officially supported in production and has some limitations (e.g. the lack of transactions support), but it's pretty close to mmapv1 in terms of performance for unit tests (which also lacked these features).
So maybe it won't fit all use-cases, but I'm sure it will be just enough for most, so give it a try before trying tmpfs or other solutions, since those will still not grant the same performances.
I'm working on a Laravel 5.3.30 app that is performing unacceptably slow, and it seems like there is some issue beyond what any level of optimization can solve. I'm developing the app on the following environment:
Wampserver 3.0.6 64 bit including PHP 7, MySQL 5.7.14, and Apache 2.4.23
Windows 10 Pro 64 bit
ORM: Doctrine 2
Hardware: ZBook 17 G2, i7-4710MQ Processor, 8gb Memory, Quadro K2200M, SSD
I've read about Laravel performance and the advice seems to be to not pay too much attention to speed as the ample performance of production servers will handle the bloat of the framework as I plan to put it on a VPS server running on SSD drives. But my hardware itself isn't all that slow and is also running on SSD, so I'm not so sure. I feel that there is something fundamentally wrong with the setup and have attached the below results from debugbar for loading a very basic page which calls a single SQL query (for selecting the user to verify the session), along with the full SQL log with "general-log" enabled for the request.
I've run the usual optimizations with minimal effect:
php artisan optimize --force
php artisan config:cache
php artisan route:cache
php artisan clear-compiled
caching
I've seen many posts with people stating that achieving <100ms for basic requests on Laravel 5 like the one I benchmarked for this post is a non-issue, so I assume there is something else that's going on here, and I'm hoping someone can point me to what is causing this slow down.
Thanks!
Update
It just occurred to me as soon as I posted this, that I'm using a symlink inside the webroot to connect to the public folder in the Laravel app (which is outside the webroot), and I wonder if this symlink is what is causing the slowdown? I will update if I get the chance to benchmark this.
Update 2
As per the suggestion to test the speed for an Ajax request, the results from the debugbar below show it's just as slow, if not slower.
So having used PHP's get_included_files(), it turns out that Laravel was including some 700 files, and I thought this was the culprit. But having asked a couple of folks on slack, they had similar include numbers, so I realized that I was caching wrong. First off, I should have been calling the optimize query after calling the config cache query, as per:
php artisan config:cache
php artisan optimize --force
And sure enough, this reduced the number of files being included to under 500 files. And as per advice from slack, I took the remaining files and cached as many of them as possible using obcache, which helped as well. So now it's running at acceptable performance, and I imagine using a more fine-tuned strategy with obcache would speed things up even more.
I took over an app based on a heavily modified version of Drupal. The last developer didn't do a good job. At least at first sight I saw a lot of monkey-patching and things you should never ever do (password files, called password.txt, in a public webfolder, etc.).
The app is incredibly slow. I guess it is caused by bad code (since it doesn't load large assets) and I guess it must be MySQL related - but I don't know for sure.
Since the app is not documented, what would be the best and fastest approach to look for bottlenecks? Are there any tools that could help me? Where should I start?
I pulled a version to run it locally on a Mac, maybe that makes things easier.
If you suspect mysql queries, turn on log-slow-queries in mysql and set long_query_time to low value, eg 0.010 seconds. Or you can turn query log for all queries:
general_log=1
general_log_file="query.log"
Then xdebug has profiling abilities.
since I'm using MAMP I thought I share what I did to check the MySQL queries. Beware: Only do that for testing purposes. Or clean your log files regularly, they'll build up.
I edited the my.cnf for MySQL in MAMP and added the following lines in the Mysqld Section
# The MySQL server
[mysqld]
# Some other configuration will be here
# ADD THE FOLLOWING LINES
slow-query-log = 1
slow-query-log-file = /Applications/MAMP/logs/mysql_slow.log
long_query_time = 0.001
log-queries-not-using-indexes
general_log=1
general_log_file= /Applications/MAMP/logs/mysql_query.log
log = /Applications/MAMP/logs/mysql_sql.log
Then just cd into the /Applications/MAMP/logs/ folder and watch with
tail -f ./mysql_slow.log
what is going down.
I am running PHPUnit to test a CodeIgniter application using CIUnit (a third party interface between the two). A number of the tests select data from an empty MySQL database that is populated with 5-10 records in setUp(). On Windows and the web server (Ubuntu 10.04/Apache 2.2/MySQL 5.1/PHP 5.3), the 105 tests run in 2-3 seconds with a memory usage of around 30mb. On my local (Ubuntu 12.04/Apache 2.2/MySQL 5.5/PHP 5.3), the 105 tests run with the same memory usage, but take approx 45 seconds.
I have narrowed down the slowness to tests which utilise the database; are there any configuration settings that I am possibly missing that are making the tests run 15 times slower? If not, would my best bet be to try downgrading MySQL, or maybe even Ubuntu (I have already tried downgrading from 12.10 to 12.04)?
Any answers much appreciated.
You are most likely running into the performance hit created by barriers being default on in the ext4 filesystem. Read more of them here:
Here's what they do from the docs:
barrier=<0|1(*)>
the jbd code. barrier=0 disables, barrier=1 enables. This also
requires an IO stack which can support barriers, and if jbd gets an
error on a barrier write, it will disable again with a warning. Write
barriers enforce proper on-disk ordering of journal commits, making
volatile disk write caches safe to use, at some performance penalty.
If your disks are battery-backed in one way or another, disabling
barriers may safely improve performance.
You can try to remount your filesystem without them like this (use the mount point where the mysql data files are living)
mount -o remount,nobarrier /
In my environment this makes a Tests: 83, Assertions: 194 suite's runtime reduce from 48 seconds to 6.
I was able to significantly speed up my PHPUnit tests by following this suggestion [1]:
If you are using the InnoDB engine (the default on Fedora), try putting this in your my.cnf database configuration:
[mysqld]
...
innodb_flush_log_at_trx_commit=2
...
then restart your server.
It does seem to reduce the reliability of your database, IF you get a power loss WHILE writing etc. It was definitely an acceptable rick for my development machine, but I wouldn't recommend it for production. See more details about the reliability at [2].
[1] http://aventinesolutions.nl/mediawiki2/index.php/PHPUnit:_a_Quick_Way_to_Speed_Up_Test_Suites?goback=.gde_1685627_member_107087295
[2] http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
I'm trying to speed up my builds some and was looking for some thoughts on how to do so. I currently use Hudson as a continuous integration server for a PHP project.
I use an Ant build.xml file to do the build, using a file similar to Sebastian Bergmann's php-hudson-template. At the moment, though (due to some weird problems with Hudson crashing otherwise), I'm only running phpDocumentor, phpcpd, and phpUnit. phpUnit does generate Clover code-coverage reports, too.
Here are some possible bottlenecks:
phpDocumentor: Takes 180 seconds. There are some large included libraries in my project, such as awsninja, DirectedEdge, oauthsimple, and phpMailer. I'm not sure that I really need to be developing documentation for these. I'm also not sure how to ignore whole subdirectories using my build.xml file.
phpUnit: Takes 120 seconds. This is the only portion of the build that's not run as a parallelTask. The more tests that get written, the longer this time will increase. Really not sure what to do about this, aside from maybe running multiple Hudson build slaves and doling out separate test suites to each slave. But I also have no idea how to go about that, either.
phpcpd: Takes 97 seconds. I'm sure that I can eliminate some parsing and conversion time by ignoring those included libraries. Not sure how to do this in my build.xml file.
My server: Right now I'm using a single Linode server. It seems to get pretty taxed by the whole process.
Any other possible bottlenecks you can think of I'll add to the list.
What are some solutions for reducing my build time?
I'm not a PHP expert at all, but you ought to be able to split your PHPUnit tests onto multiple Hudson slaves if you need to. I would just split your test suite up and run each subset as a separate, parallel Hudson job. If you have a machine with multiple CPUs / cores you can run multiple slaves on it.
One obvious thing you didn't mention - how about just upgrading your hardware, or taking a look at what else is running on the Hudson host and possibly taking up resources ?
phpDocumenter: phpdoc -h reveals the -i option which allows you to specify a comma separated list of files/directories to ignore. This can be added to the arguments tag of your phpdoc build.xml tag
phpUnit: I noticed it can be laggy if I am running tests against a database, but I am not aware of anyway to improve this.
One possible thing that might help would be to not run documenter every time and only run it as part of a build that only happens once a day (or something similar)
I just recently started using these tools and these are few things I discovered.
When we had a similar problem, we resorted to running the documentation in a separate overnight build (along with our functional test scripts in Selenium, as this is also pretty slow). This way, our main CI build wasn't slowed down by generating our API documentation.
However, I note that PHP Documentor has now been updated to version 2, which has significant speed improvements over the slow old version 1. It looks like it's in the region of two to three times faster than v1. This will make a big difference to your CI process. See http://phpdoc.org/ for more info.
Alternatively, you could take a look at apiGen and phpDox, both of which are alternatives to PHPDoc. They are both definitely faster than PHPDoc v1; I haven't compared them with v2 yet.