Doctrine2 has problems with more than 1000 rows at select - php

Question

I have this query which returns me all POIs in the given area:
$query = $em
->createQuery(
'SELECT f FROM MyApplication\MyBundle\Entity\POI p
WHERE (p.latitude BETWEEN :southEastLatitude AND :norhtWestLatitude) AND
(p.longitude BETWEEN :southEastLongitude AND :norhtWestLongitude)
ORDER BY p.name
');
$query->setParameter(":norhtWestLatitude", $northWestLat);
$query->setParameter(":norhtWestLongitude", $northWestLng);
$query->setParameter(":southEastLatitude", $southEastLat);
$query->setParameter(":southEastLongitude", $southEastLng);
If i try to access with a small area (with params with small differences), I successfully get the result. I think I get the result up to 1000 rows... I'm not quite sure.
If i try to access with a bigger area, I get an empty resultset...
Somehow firing the same query with the params of the bigger area, I'm getting the correct resultset (~1060 rows).
So I'm wondering about limitations of doctrine (or even Symfony??? I'm using doctrine inside my Symfony2 Project), are there any? I also tried the $query->setMaxResults(999999999); but it didn't help...
Anybody had the same problem?
Edit: Maybe the php memory usage is to high? I added these lines before and after the getresult:
echo "Memory usage before: " . (memory_get_usage() / 1024) . " KB" . PHP_EOL;
echo "Memory usage after: " . (memory_get_usage() / 1024) . " KB" . PHP_EOL;
The output was:
Memory usage after: 5964.0625 KB
Memory usage after: 10019.421875 KB
Edit:
Strange tests:
1) If I test with these parameters:
WHERE
(f.latitude BETWEEN 45.64273082966722 AND 47.29965978937995) AND
(f.longitude BETWEEN 4.93262593696295 AND 9.99999999999999)
I'm getting 923 rows (normal behaviour)
2) If I change the parameter 9.999999999999 to 10.000000000 (or some number bigger than 9.9999999), I'm getting an empty resultset in my application.
The database still returns 923 rows (for 10.000000000).
EDIT:
I could fix the problem, discussed here: https://groups.google.com/d/msg/doctrine-user/qLSon6m4nM4/y5vLztHcbDgJ

Answer 1

From the Doctrine2 mailing list I learned that you mapped the properties as type "string":
/** #ORM\Column(name="latitude", type="string", length=255, nullable=false) */
private $latitude;
/** #ORM\Column(name="longitude", type="string", length=255, nullable=false) */
private $longitude;
This means that your db will have VARCHAR columns for these properties. So when the db runs your query, it will perform a string-compare.
A string-compare is different from a (normal) number-compare. See these results:
$a = 1234567890;
$b = 987654321;
echo $a == $b ? '0' : ($a < $b ? '-1' : '1'); // output: 1 (means $a is bigger than $b)
echo strcmp( $a, $b ); // output: -8 (means $a is smaller than $b)
So it's probably best to map the properties as something that represents numbers in your db (DECIMAL would be a good choice):
/** #ORM\Column(name="latitude", type="decimal", nullable=false) */
private $latitude;
/** #ORM\Column(name="longitude", type="decimal", nullable=false) */
private $longitude;
PS: You mention you get valid results when you perform the query yourself. This is probably because you did:
... latitude BETWEEN 45.64273082966722 AND 47.29965978937995 ...
But Doctrine (because you mapped the properties as strings) will do:
... latitude BETWEEN '45.64273082966722' AND '47.29965978937995' ...
Note the quotes here. There is a big difference in how these two statements are treated, like my test-case shows ;)

Answer 2

It's probably related to memory usage. Please check your apache log for memory errors.
When selecting large amounts of data it may be helpful to use $query->iterate() instead of $query->getResult(), as pointed out in this article.
To check if your query generates the right SQL, you could log or otherwise output the result of calling $query->getSQL() (after setting the parameters). Feed the resulting SQL to the database client of your choice to see if the desired result comes back.