I have a website with around half a million geocoded locations in a database. I want people to be able to search for these via a map. Obviously, that's far too many for a standard Google (or, for that matter, Bing) map display, even when using something like MarkerClusterer.
What I want to do, therefore, is dynamically load the map data as people scroll around on the map so that there are never too many icons, or too much data, loaded at once. Here's an example of a site which already does something like this:
http://www.globrix.com/property/buy/wr11%203dl?ns=true&rd=1&hits=10&br=buy&qt=wr11+3dl&keyword_field=
Unfortunately, I'm not a skilled enough javascript programmer to reverse engineer that code! So I was hoping that there might be an open source project which I can use or adapt instead.
I've mostly used Google maps in the past (and the site currently uses Google maps for small-area search), but I'd be equally happy with Bing if that's easier. The backend is all in PHP.
Any suggestions?
Listen for the 'idle' event on the Map.
You'll want to do some sort of spatial query, using the bounds of the Map.
Also, consider using Fusion Tables:
http://google.com/fusiontables
Related
I have a whitelist of cities. Let's say, Seattle, Portland, Salem. Using GeoIP, I'd detect user city. Let's call it $user_city. Based on $user_city, I want to display classified-listings from nearest city from my whitelist (Seattle || Portland || Salem) with in 140 miles. If city is not listed in 140 miles, I'd just show a drop-down and ask user to manually select a city.
There are a few ways of doing this:
calculate this on the fly (I found an algorithm in one of SO answers)
with help of DB (let me explain):
create a table called regions
regions will have
city 1 | city 2 | distance (upto 140 miles)
city 1= cities from whitelist
city 2= any city within 140 miles from city 1
This would create a reasonable sized table. If my whitelist has 200 cities, and there are 40 cities (or towns) within 140 miles of each city. This would create 8000 rows.
Now, when a user comes to my site:
1) I check if user is from whitelist city already (city 1 column). If so, display that city
2). If not, check if $user_city is in "city 2" column
2a) if it is, get whitelist city with lowest distance
2b) if it is not, display drop-down for manual input
Final constraint: whichever method we select, it has to work from within iFrame. I mean, can I create this page on my mysite1.com and embed this page inside someothersite2.com inside an iframe? Will it still be able to get user_city and find nearest whitelisted city? I know there are some cross-domain scripting rules so I am not sure if iFrame would be able to get user-ip address, pass it to GeoIP, and resolve it to $user_city
So, my question:
How best to do this? If a lot of people embed my page in their page (using iframe) then my server would get pounded 10000s of times per second (wishful thinking, but let's assume that's the case). I don't know if a DB would be able to handle so much pounding. I don't want to have to pay for more DB servers or web-servers. I want to minimize resource-requirement at my end. So, I don't mind offloading a bit of work to user's browser via JavaScript.
EDIT:
Some answers have recommended storing lat, long and then doing the Math. The reason I suggested creating a 'regions' table is that this way all math is precomputed. If I have a "whitelist" of cities, and if I precompute all possible nearby city for each whitelisted city. Then I don't have to compute distance (using Haversine algorithm for eg) everytime.
Is it possible to offload all of this to user's browser via some crafty use of Java Script? I don't want to overload my server for a free service. It might make money but I am very close to broke and I am afraid my server would go down before I make enough money to pay for the upgrades.
So, the three constraints of this problem are 1) should work from inside iframe (I am hoping this will go viral and every blogger would want to embed my site into their page's iframe. 2) should be very fast 3) should minimize load on my server
Use one table City and do a mysql math-calculation for every query, with the addition of a cache layer eg memcache. Fair performance and very flexible!
Use two tables City (id,lat,lng,name) and Distance (city_id1,city_id2,dist), get your result by a traditional JOIN. (Could use a cache layer too.) Not very flexible.
Custom data structure: CityObj (id,lat,lng,data[blob]) just serialize and compress a php-array of the cities and store it. This might rise your eyebrows but as we know the bottleneck is never CPU or memory, it's disc IO. This is one read from an index of an INT as apposed to the JOIN which uses a tmp-table. This is not very flexible but will be fast and scalable. Easy to shard and cluster.
Is it possible to offload all of this to user's browser via some crafty use of Java Script? I don't want to overload my server for a free service. It might make money but I am very close to broke and I am afraid my server would go down before I make enough money to pay for the upgrades.
Yes, it is possible...using Google Maps API and the geometry library. The function you are looking for is google.maps.geometry.spherical.computeDistanceBetween. Here is an example that I made a while ago that might help get you started. I use jQuery here. Take a look at the source to see what's happening and modify as needed. Briefly:
supplierZips is an Array of zip codes comparable to your city whitelist.
The first thing I do on page load is geocode the whitelist locations. You can actually do this ahead of time and cache the results, if your city whitelist is constant. This'll speed up your app.
When the user enters a zip code, I first check if it's a valid zip from a json dataset of all valid zip codes in the U.S.( http://ampersand.no.de/maps/validUSpostalCodes.json, 352 kb, data generated from zip code data at http://www.geonames.org).
If the zip is valid, I compute the location between that zip and each location in the whitelist, using the aforementioned computeDistanceBetween in the Google Maps API.
Hope this helps get you started.
You just have to get the lat and the long of each city and add it to the database.
So every city only has 1 record. No distances are stored on the position on the globe.
Once you have that you can easily do a query with using haversine formula ( http://en.wikipedia.org/wiki/Haversine_formula ) to get the nearest cities within a range.
know there are some cross-domain scripting rules so I am not sure if iFrame would be able to get user-ip address
It will be possible to get the user ip or whatever if you just get the info from the embedded page.
I don't know if a DB would be able to handle so much pounding
If you have that many requests you should have by then found a way to make a buck with it :-) which you can use for upgrades :D
Your algorithm seems generally correct. What I would do is use PostGIS (a postgresql plugin, and easier to set up than it looks :-D). I believe the additional learning curve is totally worth it, it is THE standard for geodata.
If you put the whitelist cities in as POINTs, with latitudes and longitudes, you can actually ask PostGIS to sort by distance to a given lat/lon. It should be much more efficient than doing it yourself (PostGIS is very optimized).
You could get lats and longs of your user cities (and the whitelist cities) by using a geocoding API like Yahoo Placefinder or Google Maps. What I would do would be to have a table (either the same as the whitelist cities or not) that stores city name, lat, and lon, and do lookups on that. If the city name isn't found though, hit the API you are using, and cache the result in the table. This way you'll quickly not need to hit the API except for obscure places. The API is fast too.
If you're really going to be seeing that kind of server load, you may want to look into using something besides PHP though (such as node.js). Incidentally you shouldn't have any trouble geocoding from an iframe, from the Point of View of the server, its just like the browser is going to that page "normally".
is there any offline map libraries that can be used from PHP? I want to make an application which displays all data position of stations (with latitude and longitude) on a map.
But the user doesn't want this application connect to internet, so I can't use Google Maps as a solution. Do you know any offline map libraries that can mark position based latitute and longitude information?
Thanks in advance
If you need to render graphical maps of anywhere on the globe, you're gonna need a lot of geographical data and a lot of storage (unless you're keeping the resolution low). For geographical data of the U.S., you can purchase bulk data sets (or obtain smaller free data sets) from certain commercial organizations, or try GeoData.gov.
Once you've obtained your GIS data set, you can use Image_GIS to visualize the data. I haven't used the library, so I don't know how the results look, but that's one solution.
I'd like to find a way to take a piece of user supplied text and determine what addresses on the map are mentioned within the text. I'd be happy to use a free web service if it exists or use a script which will not consume too many resources.
One way I can imagine doing this is taking a gigantic database of addressing and searching for each of them individually in the text, but this does not seem efficient. Is there a better algorithm or technique one can suggest?
My basic idea is to take the location information and turn it into markers on a Google Map. If it is too difficult or CPU intensive to determine the locations automatically, I could require users to add information in a location field if necessary but I would prefer not to do this as some of the users are going to be quite young students.
This needs to be done in PHP as that is the scripting language available on my school hosted server.
Note this whole set-up will happen within the context of a Drupal node, and I plan on using a filter to collect the necessary location information from the individual node, so this parsing would only happen once (when the new text enters the database).
You could get something like opencalais to tag your text. One of the catigories which it returns is "city" you coud then use another third party module to show the location of the city.
If you did have a gigantic list of locations in a relational database, and you're only concerned about 500 to 1000 words, then you could definitely just pass the SQL command to find matches for the 500-1000 words and it would be quite efficient.
But even if you did have to call a slow API, you could feasibly request for 500 words one by one. If you kept a cache of the matches, then the cache would probably quickly fill up with all the stop words (you know, like "the", "if", "and") and then using the cache, it'd be likely that you would be searching much less than 500 words each time.
I think you might be surprised at how fast the brute force approach would work.
For future reference I would just like to mention the Yahoo API called Placemaker and the service GeoMaker that is built on top of it.
Those tools can be used to parse out locations from a text as requested here. Unfortunately no Drupal module seems to exists right now- but a custom solution seems easy to code.
I'm stuck here again. I have a database with over 120 000 coordinates that I need to be displayed on a google maps integrated in my application. The thing is that and I've found out the hard way simply looping through all of the coordinates and creating an individual marker for each and adding it using the addOverlay function is killing the browser. So that definitely has to be the wrong way to do this- I've read a bit on clustering or Zoom level bunching - I do understand that theres no point in rendering all of the markers especially if most of them won't be seen in non rendered parts of the map except I have no idea how to get this to work.
How do I fix this here. Please guys I need some help here :(
There is a good comparison of various techniques here http://www.svennerberg.com/2009/01/handling-large-amounts-of-markers-in-google-maps/
However, given your volume of markers, you definitely want a technique that only renders the markers that should be seen in the current view (assuming that number is modest - if not there are techniques in the link for doing sensible things)
If you really have more than 120,000 items, there is no way that any of the client-side clusterers or managers will work. You will need to handle the markers server-side.
There is a good discussion here with some options that may help you.
Update: I've posted this on SO before, but this tutorial describes a server-side clustering method in PHP. It's meant to be used with the Static Maps API, but I've built it so that it will return clustered markers whenever the view changes. It works pretty well, though there is a delay in transferring the markers whenever the view changes. Unfortunately I haven't tried it with more than 3,000 markers - I don't know how well it would handle 120,000. Good luck!
I've not done any work with Google maps specifically but many moons ago, I was involved in a project which managed a mobile workforce for a large Telco.
They had similar functionality in that they had maps which they could zoom in on for their allocated jobs (local to the machine rather than over the network) and we solved a similar problem which sounds very similar like yours. Points of interest on the maps were called landmarks and were indicated by small markers on the map called landmark pointers, which the worker could select to get a textual description..
At the minimum zoom, there would have been a plethora of landmark pointers, making the map useless. We made a command decision to limit the landmark pointers to a smaller number (400). In order to do that, the map was divided into a 20x20 matrix no matter what the zoom level, which gave us 400 matrix elements.
Then, if a landmark shared the same matrix element as another, the application combined them and generated a single landmark pointer with the descriptive text containing the text of all the landmarks in that matrix element.
That way there were never more than 400 landmark pointers. As the minion zoomed in, the landmark pointers were regenerated and landmarks could end up in different matrix elements - in that case, they were no longer combined with other landmarks.
Similarly, zooming out sometimes merged two or more landmarks into a single landmark pointer.
That sounds like what you're trying to achieve with "clustering or zoom level bunching" although, as I said, I have little experience with Google Maps itself so I'm not sure this is possible. But given Google's reputation, I suspect it is.
I suggest that you use a marker manager class such as this one along with your existing code. Marker manager class allows you to manage thousands of markers and optimizes memory usage. There is a variety of marker managers (there is not just one) and I suggest you Google a bit.
Here is a non-cluster solution if you want to display hundreds or even thousands of markers very quickly. You can use a combination of OverlayView and DocumentFragment.
http://nickjohnson.com/b/google-maps-v3-how-to-quickly-add-many-markers
If only there is something more powerful than JS for this...
Ok enough sarcasm : ).
Have you used the Flash Maps API? Kevin Macdonald has successfully used it to cluster not 120K markers but 1,000,000 markers. Check out the Million Marker Map:
http://www.spatialdatabox.com/million-marker-map/million-marker-map.html
Map responsiveness is pretty much un-affected in this solution. If you are interested you can contact him here: http://www.spatialdatabox.com/sdb-contact-sales.html
Try this one :
http://googlegeodevelopers.blogspot.com/2009/04/markerclusterer-solution-to-too-many.html
Its an old question and already got many answers , but Stackoverflow is more as reference i hope this will help anyone who searches for the same problem .
There is a fairly simple solution- Use HTML5 canvas, though it sounds strange , its the fastest way to load upto 10,000 markers as well as a labels, which am sure no browser can handle if its a normal marker. Not conventional markers but light markers.
I'm developing a store location application.
Looking up a store, it currently shows the location in googlemaps based on address and zip code.
Now I want to build a function which also shows other shops within 500 meter radius.
To do this, I have to do a proximity search / calculation.
My biggest question, is how I should approach this.
I did find this link, which has some example code. But I'm unsure if I can use the code (and which of the codes I should use). Does anyone have better examples?
Also I'm thinking of adding a new table to the database, which stores the geo code for each store. Do I need more fields than 'id', 'latitude' and 'longitude' ?
UPDATE
I just found this link at phpro.org. It looks like it's just what I need! Has anyone used their examples and can comment upon it?
You cant radius search directly with the google maps API, however, you should know (or can figure them out via geocoding their addresses) the latitude and longitude of each point of interest (POI) you want to include in the search.
After this you can use the Great Circle equation to search for proximity, and it turns out to be very fast. We have implemented this as a stored procedure for the locator service at my work and use it to search through >3500 locations with response times under 0.1 seconds.
Some SQL implementations contain geospatial extensions. In those implementations you can directly write a WHERE clause that filters the results by distance from a specified point.
Check the documentation of your SQL implementation. If it has geospatial POINT type, then it makes sense to enter the coordinates as POINTs rather than a lat/lng pair, and consider using that field as a SPATIAL KEY if you're going to be accessing the table mainly by location.