SO I have jQuery DataTables setup and running fine.
My eventual goal, is to allow the user to use google places autocomplete, to update their location, and then to have a 'sortable' distance column added to my DataTables table, upon refresh.
The plan is is to have a button next to the text input of google autocomplete, that says use this address, once user searches for and finds an address using autocomplete, and then pushes the button , I want to (probably)send them to a loading page for a few seconds whilst we
1)read the text string, format it for use with google geo coding api, aka
"Cranbourne, Victoria, Australia" ----> "Cranbourne+Victoria+Australia"
2)insert it in a geocode request
3)extract the lat and lon values from the resulting json, and assign them to vars
Now comes the hard part, I was thinking using some php, to calculate the distance between the users(lat,long), and every entry that was in the DataTable table, which was was populated with sql data using a foreign key (all sql rows have lat lon data)
Then I was thinking of sending said entries to an xml, only this time every entry will have an extra value "DISTANCE".
Of coarse with some clever code this whole thing wont take a second, the new DataTable table is then populated with the xml, only this time it contains a distance column with distance values for every row, which of coarse will be sortable, without any DataTables hacks what so ever.
So am I stupid, or is this somewhat of a good idea, is there an easier way?
I would really like some answers to this thank you.
That actually could be difficult (depending on your math expertise), because of the fact that it requires using spherical geometry. You might consider using an API:
https://developers.google.com/maps/documentation/distancematrix/
If you want to try doing it yourself, you might want to use a class written for that, such as:
http://www.phpclasses.org/browse/file/36294.html
The second option is probably better, because Google has limits on its API. Once you have a way of calculating distance, it's a simple matter to query the database and create the XML. I don't know why you want to use XML instead of json, but the PHP function json_encode() might make it simpler, and it's supported by DataTables.
I know this is a really old question but I just came across it and wanted to recommend using the Haversine Formula to calculate the distances between lat/long values
I've used this implementation and found it worked quite wonderfully
Related
I have a table with over 2 million rows. One of the values is an address. Some rows have a common address. I am using php.
On my website, I want the user to put in their zip code, and I will return all results within that zip code. I would then use Google to Geolocate them on a map. The problem is that since Google charges by the query, I can't be wasting time and money requesting coordinates for an address I already have. Here is what I believe to be the correct approach:
Ask user for zip code
Run "Select * with 'Zip Code' = $user_zip" (paraphrasing)
Run a Geolocate on first address and plot on map
Check for matching addresses in result and group with the mapped result
Find next new address
Repeat 3-6 until complete
Is there a better way to approach this? I am looking for efficiency, easy way to manipulate all matching results at once, and the least amount of queries. If my way is correct, can someone please help me with the logic for numbers 3-5?
If I understand this right what you are trying to do is to render a map with markers for each record in your database that is within a certain zip area. And your challenge is that you need coordinates to render each marker. The biggest issue with your approach in terms of wasting resources is that you do not store the coordinates of each address in your database. I would suggest you to:
1 - Alter the endpoint (or script or whatever) that creates these records in your db to fetch the coordinates and store them in the database.
2 - Run a one time migration to fetch coordinates for each record. While I understand that doing this for 2 milion rows could be "costly" with Google's Geocoding (Estimate is 1000$ for 2 milion api calls). To save the costs you could look into some of the opensource map tools.
Either way fetching coordinates during the request lifecycle is both a waste of resource and it will significantly affect speeds.
I'm programming a search engine for my website in PHP, SQL and JQuery. I have experience in adding autocomplete with existing data in the database (i.e. searching article titles). But what about if I want to use the most common search queries that the users type, something similar to the one Google has, without having so much users to contribute to the creation of the data (most common queries)? Is there some kind of open-source SQL table with autocomplete data in it or something similar?
As of now use the static data that you have for auto complete.
Create another table in your database to store the actual user queries. The schema of the table can be <queryID, query, count> where count is incremented each time same query is supplied by some other user [Kind of Rank]. N-Gram Index (so that you could also auto-complete something like "Manchester United" when person just types "United", i.e. not just with the starting string) the queries and simply return the top N after sorting using count.
The above table will gradually keep on improving as and when your user base starts increasing.
One more thing, the Algorithm for accomplishing your task is pretty simple. However the real challenge lies in returning the data to be displayed in fraction of seconds. So when your query database/store size increases then you can use a search engine like Solr/Sphinx to search for you which will be pretty fast in returning back the results to be rendered.
You can use Lucene Search Engiine for this functionality.Refer this link
or you may also give look to Lucene Solr Autocomplete...
Google has (and having) thousands of entries which are arranged according to (day, time, geolocation, language....) and it is increasing by the entries of users, whenever user types a word the system checks the table of "mostly used words belonged to that location+day+time" + (if no answer) then "general words". So for that you should categorize every word entered by users, or make general word-relation table of you database, where the most suitable searched answer will be referenced to.
Yesterday I stumbled on something that answered my question. Google draws autocomplete suggestions from this XML file, so it is wise to use it if you have little users to create your own database with keywords:
http://google.com/complete/search?q=[keyword]&output=toolbar
Just replacing [keyword] with some word will give suggestions about that word then the taks is just to parse the returned xml and format the output to suit your needs.
I have a mysql table of postcodes with 200000 Records in it. I want to select just one field post_code from it and fetch the postcodes for auto suggestion in a textbox. I will do this using jquery but I want to know which method I should use to create code in php and select records faster?
Without specifics it's hard to give an answer more specific than "Write a faster query".
I'm going to make a huge assumption here and assume you're working on a web application that's populating an autocomplete form.
You don't need all the post-codes! Not even close.
Defer populating the autocomplete until the user starts typing into the postcode field. When that happens, do an AJAX load to get the postcodes from the database.
Why is doing it this way better than just fetching all the post codes?
Because you now know what letter the user's post code starts with.
Knowing the first letter of the post code means you can eliminate any post code in your dataset that doesn't start with the same letter. You'll go from having to fetch 20000 postcodes to having to fetch less than 2000, improving performance by over an order of magnitude. If you wait until the user has entered two letters, you can get the amount of data to be fetched down even further.
Also, I'm assuming you're using an index on your postcode column, right?
I would think you simply want to create a MySql index on "post_code"?
Q: How many postal codes are you planning on fetching at a time (and passing back to the browser)? One? Five? 200,000?
You can use Sphinx-Php Api to deal with big database
But you will need to indexer file for your search for the first time to use this way.
Or can go for Zend Lucene Search
you probably want to filter the records that matches the current input of the user and your query should be as follows:
select postal_code from table_name where postal_code like '<userinput>%' LIMIT 50 eg: '60%'
and send back this result set to your jquery callback function. remember to set a limit on the postal codes retrieved as you dont want to send back too much data which not only hampers the performance but doesnt help the user in a direct way.
I have a whitelist of cities. Let's say, Seattle, Portland, Salem. Using GeoIP, I'd detect user city. Let's call it $user_city. Based on $user_city, I want to display classified-listings from nearest city from my whitelist (Seattle || Portland || Salem) with in 140 miles. If city is not listed in 140 miles, I'd just show a drop-down and ask user to manually select a city.
There are a few ways of doing this:
calculate this on the fly (I found an algorithm in one of SO answers)
with help of DB (let me explain):
create a table called regions
regions will have
city 1 | city 2 | distance (upto 140 miles)
city 1= cities from whitelist
city 2= any city within 140 miles from city 1
This would create a reasonable sized table. If my whitelist has 200 cities, and there are 40 cities (or towns) within 140 miles of each city. This would create 8000 rows.
Now, when a user comes to my site:
1) I check if user is from whitelist city already (city 1 column). If so, display that city
2). If not, check if $user_city is in "city 2" column
2a) if it is, get whitelist city with lowest distance
2b) if it is not, display drop-down for manual input
Final constraint: whichever method we select, it has to work from within iFrame. I mean, can I create this page on my mysite1.com and embed this page inside someothersite2.com inside an iframe? Will it still be able to get user_city and find nearest whitelisted city? I know there are some cross-domain scripting rules so I am not sure if iFrame would be able to get user-ip address, pass it to GeoIP, and resolve it to $user_city
So, my question:
How best to do this? If a lot of people embed my page in their page (using iframe) then my server would get pounded 10000s of times per second (wishful thinking, but let's assume that's the case). I don't know if a DB would be able to handle so much pounding. I don't want to have to pay for more DB servers or web-servers. I want to minimize resource-requirement at my end. So, I don't mind offloading a bit of work to user's browser via JavaScript.
EDIT:
Some answers have recommended storing lat, long and then doing the Math. The reason I suggested creating a 'regions' table is that this way all math is precomputed. If I have a "whitelist" of cities, and if I precompute all possible nearby city for each whitelisted city. Then I don't have to compute distance (using Haversine algorithm for eg) everytime.
Is it possible to offload all of this to user's browser via some crafty use of Java Script? I don't want to overload my server for a free service. It might make money but I am very close to broke and I am afraid my server would go down before I make enough money to pay for the upgrades.
So, the three constraints of this problem are 1) should work from inside iframe (I am hoping this will go viral and every blogger would want to embed my site into their page's iframe. 2) should be very fast 3) should minimize load on my server
Use one table City and do a mysql math-calculation for every query, with the addition of a cache layer eg memcache. Fair performance and very flexible!
Use two tables City (id,lat,lng,name) and Distance (city_id1,city_id2,dist), get your result by a traditional JOIN. (Could use a cache layer too.) Not very flexible.
Custom data structure: CityObj (id,lat,lng,data[blob]) just serialize and compress a php-array of the cities and store it. This might rise your eyebrows but as we know the bottleneck is never CPU or memory, it's disc IO. This is one read from an index of an INT as apposed to the JOIN which uses a tmp-table. This is not very flexible but will be fast and scalable. Easy to shard and cluster.
Is it possible to offload all of this to user's browser via some crafty use of Java Script? I don't want to overload my server for a free service. It might make money but I am very close to broke and I am afraid my server would go down before I make enough money to pay for the upgrades.
Yes, it is possible...using Google Maps API and the geometry library. The function you are looking for is google.maps.geometry.spherical.computeDistanceBetween. Here is an example that I made a while ago that might help get you started. I use jQuery here. Take a look at the source to see what's happening and modify as needed. Briefly:
supplierZips is an Array of zip codes comparable to your city whitelist.
The first thing I do on page load is geocode the whitelist locations. You can actually do this ahead of time and cache the results, if your city whitelist is constant. This'll speed up your app.
When the user enters a zip code, I first check if it's a valid zip from a json dataset of all valid zip codes in the U.S.( http://ampersand.no.de/maps/validUSpostalCodes.json, 352 kb, data generated from zip code data at http://www.geonames.org).
If the zip is valid, I compute the location between that zip and each location in the whitelist, using the aforementioned computeDistanceBetween in the Google Maps API.
Hope this helps get you started.
You just have to get the lat and the long of each city and add it to the database.
So every city only has 1 record. No distances are stored on the position on the globe.
Once you have that you can easily do a query with using haversine formula ( http://en.wikipedia.org/wiki/Haversine_formula ) to get the nearest cities within a range.
know there are some cross-domain scripting rules so I am not sure if iFrame would be able to get user-ip address
It will be possible to get the user ip or whatever if you just get the info from the embedded page.
I don't know if a DB would be able to handle so much pounding
If you have that many requests you should have by then found a way to make a buck with it :-) which you can use for upgrades :D
Your algorithm seems generally correct. What I would do is use PostGIS (a postgresql plugin, and easier to set up than it looks :-D). I believe the additional learning curve is totally worth it, it is THE standard for geodata.
If you put the whitelist cities in as POINTs, with latitudes and longitudes, you can actually ask PostGIS to sort by distance to a given lat/lon. It should be much more efficient than doing it yourself (PostGIS is very optimized).
You could get lats and longs of your user cities (and the whitelist cities) by using a geocoding API like Yahoo Placefinder or Google Maps. What I would do would be to have a table (either the same as the whitelist cities or not) that stores city name, lat, and lon, and do lookups on that. If the city name isn't found though, hit the API you are using, and cache the result in the table. This way you'll quickly not need to hit the API except for obscure places. The API is fast too.
If you're really going to be seeing that kind of server load, you may want to look into using something besides PHP though (such as node.js). Incidentally you shouldn't have any trouble geocoding from an iframe, from the Point of View of the server, its just like the browser is going to that page "normally".
I'm developing a store location application.
Looking up a store, it currently shows the location in googlemaps based on address and zip code.
Now I want to build a function which also shows other shops within 500 meter radius.
To do this, I have to do a proximity search / calculation.
My biggest question, is how I should approach this.
I did find this link, which has some example code. But I'm unsure if I can use the code (and which of the codes I should use). Does anyone have better examples?
Also I'm thinking of adding a new table to the database, which stores the geo code for each store. Do I need more fields than 'id', 'latitude' and 'longitude' ?
UPDATE
I just found this link at phpro.org. It looks like it's just what I need! Has anyone used their examples and can comment upon it?
You cant radius search directly with the google maps API, however, you should know (or can figure them out via geocoding their addresses) the latitude and longitude of each point of interest (POI) you want to include in the search.
After this you can use the Great Circle equation to search for proximity, and it turns out to be very fast. We have implemented this as a stored procedure for the locator service at my work and use it to search through >3500 locations with response times under 0.1 seconds.
Some SQL implementations contain geospatial extensions. In those implementations you can directly write a WHERE clause that filters the results by distance from a specified point.
Check the documentation of your SQL implementation. If it has geospatial POINT type, then it makes sense to enter the coordinates as POINTs rather than a lat/lng pair, and consider using that field as a SPATIAL KEY if you're going to be accessing the table mainly by location.