I have a "users" table with columns:
Id | user_id | name | ip_address | lat | lng
The id column, user_id, name and ip_address have been populated, and now have 100k users.
The lat (latitude) and lng (longitude) columns have now been created, and I need them populated from the IP (you can not get the geolocation of the user).
I do not want precise location, just the approximate location, so what I need is a script that picks up the IP of each user, somehow extracts the latitude and longitude, and inserts into the database.
I have looked at "http://www.geoplugin.net/xml.gp?ip=" and "http://ip-api.com/json/" but I have no idea how to use api for 100k of users, nor PHP or mysql code, or shell to do this.
I am using only PHP, MySQLi with MySQL database. The application will display a list of users "nearby" (within a 100miles radius).
Excuse me, I'm very new to programming and I really do not know how to solve this.
Edit:
I just created an api that returns me all user_id and their respective IPs in an array.
Just being 105k of records, for being json, it was light but I did not want to share json publicly here because it would be the same as explaining my users' data.
the result of api is something like this:
{
"api_status": 200,
"ips": [
{
"user_id": "1",
"ip_address": "177.198.86.7x"
},
{
"user_id": "21",
"ip_address": "177.18.246.9x"
},
{
"user_id": "52",
"ip_address": "177.36.60.1x"
}
]
}
I am using the following function to return my data in php
function List_ip() {
global $sqlConnect, $db;
if ($db['loggedin'] == false) {
return false;
}
$ips = mysqli_query($sqlConnect, "SELECT `user_id` , `ip_address` FROM " . T_USERS . " WHERE `active` = '1'");
while ($fetched_data = mysqli_fetch_assoc($ips)) {
$list_ip[] = array_map('utf8_encode', $fetched_data);
}
return $list_ip;
}
I don't have much idea about limiting the number of requests, but try the below code to get the details from ip-api.
$url = "http://ip-api.com/batch";
$headers = array('Content-Type: application/json');
$fields = '[{"query": "208.80.152.201"}, // You can add multiple IP address in the same format.
{"query": "8.8.8.8"},
{"query": "24.48.0.1"}]';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_POSTFIELDS, $fields);
$httpCode = curl_getinfo($curl , CURLINFO_HTTP_CODE);
$response = curl_exec($curl);
if ($response === false)
$response = curl_error($curl);
foreach(json_decode($response) as $value){
echo $value->lat; //Use the result to store the information in DB
}
curl_close($curl);
Refer their documentation to get an idea about the data returned by their API.
Also, for limiting, try this solution
The maxmind/GeoIP2-php should get you want you need.
Install it and download the GeoIP2-City.mmdb file.
Than just a matter of initiating it:
$record = new Reader('path/to/GeoLite2-City.mmdb');
and reading the data from it, such as:
$city = $record->city->name;
$latitude = $record->location->latitude;
$longitude = $record->location->longitude;
Doing it this way will mean zero dependency on third-party API limitations.
What you are asking for requires intermediate skills with PHP and MySQL etc.. I have done many things like this, and will tell you that firstly to get accurate GEO IP information you will have to use javascript or pay for a service (from my findings). For example www.maxmind.com has a free and pay, the pay is a bit better.
What I would do is try writing a script that reads all of the users out of the database in a loop, then download geopip or use an api to take the IP fields and return the Lat/Lon. Then finally add the step that writes it back to the DB .
A very rough example:
// build a class or function that looks up IP addresses (can be local)
$geoip2 = new GeoIp2Lookup(true);
// get all users from DB
$allUsersArray = function_results_db_lookup_users();
// get the users lat/lon
for ($i=0; $i<count($allUsersArray); $i++) {
$record = $geoip2->getGeoDataArr($allUsersArray[$i]['ip_address']);
$allUsersArray[$i]['lat'] = $record['lat'];
$allUsersArray[$i]['lon'] = $record['lon'];
}
// now write a function to place in DB the new data
update_users_db($allUsersArray);
You can easily use the IP2Location PHP Module at https://www.ip2location.com/development-libraries/ip2location/php which is free.
To install, add the below line into your composer.json file.
{
"require": {
"ip2location/ip2location-php": "8.*"
}
}
Run the command:
composer install
Note: You just need to include the below code in order to use the library.
require 'vendor/autoload.php';
You'll need the DB5.LITE database which you can download for free at https://lite.ip2location.com/
Example usage of the PHP module
<?php
require 'IP2Location.php';
$myip = '8.8.8.8';
$db = new \IP2Location\Database('./database/IP-COUNTRY-SAMPLE.BIN', \IP2Location\Database::FILE_IO);
$records = $db->lookup($myip, \IP2Location\Database::ALL);
echo '<pre>';
echo 'IP Address : ' . $records['ipAddress'] . "\n";
echo 'Country Code : ' . $records['countryCode'] . "\n";
echo 'Country Name : ' . $records['countryName'] . "\n";
echo 'Region Name : ' . $records['regionName'] . "\n";
echo 'City Name : ' . $records['cityName'] . "\n";
echo 'Latitude : ' . $records['latitude'] . "\n";
echo 'Longitude : ' . $records['longitude'] . "\n";
echo '</pre>';
?>
Related
I have a CouchDB view in my _user database, where I can find users based on their email addresses (user.email is part of their user doc).
{
"_id": "_design/find-user",
"views": {
"email->doc": {
"map": "function (doc) {\n emit(doc.email, doc);\n}"
}
},
"language": "javascript"
}
I am using the following curl url via PHP to access matching users:
curl_setopt($ch, CURLOPT_URL, $url . '/_users/_design/find-user/_view/email->doc?key="' . $email . '"');
This works, but is case sensitive, which I don't want. For example, I need $email = TesT#example.com to find tESt#example.com in my database.
I have tried the following, but that only transforms the output.
function (doc) {
emit(doc.email.toLowerCase(), doc);
}
You're on the right track with toLowerCase(). The only piece you're missing is that you must also transform your search term.
Don't search for "TEST#example.com", instead search for strtolower("TEST#example.com"):
curl_setopt($ch, CURLOPT_URL, $url . '/_users/_design/find-user/_view/email->doc?key="' . strtolower($email) . '"');
OK, I have this script and it works but:
<?php
SESSION_START();
// access_token received from /oauth2/token call
// Assign it the properties of id (optional - id of target HTML element, from URL)
// and 'message' - the feedback message we want the user to see.
if (isset($_GET['wxurl'])) {
$wxURL = $_GET['wxurl'];
}
if ($wxURL === "autoip") {
//Get the IP of the location
$uri = curl_init("http://api.wunderground.com/api/<APIKEY>/geolookup/q/autoip.json");
} else if ($wxURL === "wxcity") {
if (isset($_GET['city']) && isset($_GET['state'])) {
$wxCity = $_GET['city'];
$wxState = $_GET['state'];
}
//Get the locations weather data
$uri = curl_init("http://api.wunderground.com/api/<APIKEY>/conditions/q/" . $wxState . "/" . $wxCity . ".json");
} else if ($wxURL === "freeJSON") {
//This gets the ACTUAL USERS location NOT that of the server **<--- WELL it's SUPPOSED to get the LOCATION of the user NOT the server**
$uri = curl_init("http://freegeoip.net/json/");
}
$ch = $uri; // URL of the call
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Authorization: Bearer $access_token"));
// execute the api call
$result = curl_exec($ch);
echo $result;
PS - I blocked my API key with weather underground because i'll get enormous hits. It's FREE so fill in your own key.
My problem: When I poll use my LOCALHOST, I get weather for my own CITY, but, when use the site on the server, it PULLS the data from the SERVER's LOCATION... So if I'm here on the west coast, the server is in Temple, PA.
How can I get the locale data for a customer when the code pulls the server data???
UPDATE: I found this: http://diveintohtml5.info/geolocation.html but what I get is: UNABLE TO GET YOUR LOCATION. That's because I have GEO LOC turned off in my browser but when I use the http://freegeoip.net/json this is what I get:
{
ip: "11.22.333.444",
country_code: "US",
country_name: "United States",
region_code: "WA",
region_name: "Washington",
city: "MT Vernon",
zip_code: "98273",
time_zone: "America/Los_Angeles",
latitude: 47.5614,
longitude: -123.7705,
metro_code: 985
}
So, Geolocation.js baffles me...
Thoughts?
I've been trying to do this for a couple of days through trial and error etc, but getting absolutely nowhere. PHP isn't my strong point, but I'm generally comfortable with it that I can learn as I go when I need to do specific things.
What I'm trying to do, is take the API from one platform that is used, and input it into another platform that is used. I can get the data from the API easily enough via a URL, and it runs fine on a different server so I'm pretty sure everything is fine on that side of things.
The issue is, that when I do manage to get it from a URL, it comes out looking quite messy. Nothing I've tried so far will display it as a nice tidy block. Furthermore, I'd like to be able to pull specific data from the result and display just that. The data comes out as follows when visited via the URL (have changed values for privacy etc, but the integrity should remain):
{"Data":[{"DeviceID":"1","DeviceName":"Phone 1","Platform":"Phone OS","Edition":"Deluxe","State":"0","Time":"2016-03-16T13:47:44+01:00"}]}
Essentially, what I'm trying to do is:
Display the data in a block list, as opposed to long lines
Allow selection of a specific device through "Device Name", and then display the information relevant to that device
I've tried the following scripts so far:
1:
<?php
$json = file_get_contents('URLHERE');
$obj = json_decode($json);
echo $obj->DeviceID;
?>
2:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, 'URLHERE');
$result = curl_exec($ch);
curl_close($ch);
$obj = json_decode($result);
echo $result->DeviceName;
?>
3:
<?php
$url = 'URLHERE';
$obj = json_decode(file_get_contents($url), true);
echo $obj['DeviceID'];
?>
4:
<?php
$url = "URLHERE";
$json = file_get_contents($url);
$json_data = json_decode($json, true);
echo "Device: ". $json_data["DeviceID"];
?>
5:
<?php
$json = file_get_contents('URLHERE');
$encodeJ = utf8_encode($json);
$obj = json_decode($encodeJ);
var_dump($obj-> DeviceID);
?>
The fourth one is the closest I've managed to get to it displaying data using these methods, but rather than any information I just get "Device: NULL"
Any help would be appreciated. Starting to pull my hair out here!
UPDATE:
Have managed to make some progress with the following:
<?php
$data = file_get_contents('URLHERE');
$response = json_decode($data, true);
echo 'Device: ' . $response['Data']['0']['DeviceName'];
echo 'Device: ' . $response['Data']['1']['DeviceName'];
?>
This is displaying the device names from the array for value 0 and 1. So now I need to figure out how to iterate through the array and display each one in sequence, as opposed to hard coding each one.
Your DeviceID is in Data & it's an array so you can't access directly. When you
$data = json_decode('{"Data":[{"DeviceID":"1","DeviceName":"Phone 1","Platform":"Phone OS","Edition":"Deluxe","State":"0","Time":"2016-03-16T13:47:44+01:00"}]}', true);//I am using array so second parameter is true to easily demonstrate
Your structure is
[
"Data" => [
[
"DeviceID" => "1",
"DeviceName" => "Phone 1",
"Platform" => "Phone OS",
"Edition" => "Deluxe",
"State" => "0",
"Time" => "2016-03-16T13:47:44+01:00",
],
],
]
So to get only first DeviceID if you want then
$deviceID = isset($data['Data'][0]['DeviceID']) ? $data['Data'][0]['DeviceID'] : null;
or if you want all the DeviceIDs then
$deviceIds = [];
if (isset($data['Data']) && is_array($data['Data']))
{
foreach ($data['Data'] as $row)
{
if (isset($row['DeviceID']))
{
$deviceIds[] = $row['DeviceID'];
}
}
}
or you can use array_column if your php version is >= 5.5.0 or php 7
$deviceIds = [];
if (isset($data['Data']) && is_array($data['Data']))
{
$deviceIds = array_column($data['Data'], 'DeviceID');
}
To get the data, use:
$json = file_get_contents( $url );
Then get it into an array, as:
$arr = json_decode( $json, TRUE );
To "Display the data in a block list, as opposed to long lines", use:
foreach ( $arr AS $element ) {
foreach ( $element AS $e ) {
echo $e['DeviceName'] . '<br>';
}
}
To "Allow selection of a specific device through "Device Name", and then display the information relevant to that device", use:
$deviceName = "Phone 1"; // depending upon your use case, you'll need to decide how you want to set this variable; it's hard coded here for the sake of example
foreach ( $arr AS $element ) {
foreach ( $element AS $e ) {
if ( $e['DeviceName'] = $deviceName ) {
echo '<pre>';
print_r( $e );
echo '</pre>';
}
}
}
While it's not entirely clear what you mean by "Allow selection of a specific device through "Device Name"", I'm inclined to believe you're looking for a way to let a user select a device from the list of device names. That's not a task you can accomplish with PHP alone. You'll need to build something for the front end in HTML or Javascript that interacts with your PHP on the back end.
I'm new to PHP, and I want to get latitude and longitude of a place and then add them to MySQL database.
I'm using Google Geo-code API to get them, this is what I do right-row
for ($i = 0; $i<1000; $i++) {
$sql = mysql_query("SELECT place_address FROM place_locator WHERE place_id =".$i, $this->db) or die('invalide request : ' . mysql_error());
if (mysql_num_rows($sql)) {
while ($place = mysql_fetch_assoc($sql)) {
//Encode the place string I got, to get rid of space
$encodePlace = str_replace(" ", "%20", $place["place_address"]);
//Use Google API
$url = 'http://maps.googleapis.com/maps/api/geocode/json?address='.$encodePlace.'&sensor=false';
//Use Curl to send the request
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
$obj = json_decode($response, true);
$updateSql = mysql_query("UPDATE `place_locator`.`place_locator` SET
`latitude` = '".$obj["results"][0]["geometry"]["location"]["lat"]."',
`longitude` = '".$obj["results"][0]["geometry"]["location"]["lng"]."' WHERE `place_locator`.`place_id` = ".$i, $this->db) or die('Invalide : ' . mysql_error());
curl_close($ch);
}
}
It works for a loop of 10,when going to 1000, it will take a lot of time and many results didn't updated to the database.
I think may be multi thread should help, but I don't really know how it works, please help me.
Thanks in advance
I had the same problem. Google limits the frequency of the requests! Try a sleep(1); in the loop and it will work but need much more time.
I have following problem. I'm using google maps in php to get the address of latitude and longitude and a map snapshot of this location.
To obtain address I use following code:
// INITIALIZING CURL
$returnValue = NULL;
$ch = curl_init();
// SERVICE CALL
$url = "http://maps.googleapis.com/maps/api/geocode/json?latlng=".$lat.",".$lon."&sensor=false";
// SETTING PARAMS OF CURL
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
// GETTING AND RESULTING RESULT
$result_part = curl_exec($ch);
$json = json_decode($result_part, TRUE);
And I parse obtained JSON as follows:
// PARSING RESULTS FROM JSON
if (isset($json['results'])) {
foreach ($json['results'] as $result_part) {
foreach ($result_part['address_components'] as $address_component) {
$types = $address_component['types'];
// GETTING STREET
if (in_array('route', $types)) {
$addr = $address_component['long_name'];
}
// GETTING STREET NUMBER
if (in_array('street_number', $types)) {
$number = $address_component['long_name'];
}
// GETTING COUNTRY
if (in_array('country', $types)) {
$country = $address_component['long_name'];
}
// GETTING POSTAL CODE
if (in_array('postal_code', $types)) {
$postal_code = $address_component['long_name'];
}
// GETTING CITY
if (in_array('locality', $types)) {
$city = $address_component['long_name'];
}
}
}
}
It works fine but sometimes the address is not obtained. It looks like some overload of requests but I dont understand why because the site that I'm programming is not accessible yet for other people.
Other problem connected to this is the map snapshots. Here is the code:
<? echo "<a href = \"https://maps.google.com/maps?q=".$lat.",".$lon."\" target=\"_blank\">" ?>
<? echo "<img src=\"http://maps.googleapis.com/maps/api/staticmap?center=" . $lat . "," . $lon . "&zoom=16&size=200x200&markers=color:blue%7Clabel:I%7C" . $lat . "," . $lon . "&sensor=false\" alt=\"google maps\" width=\"200\" height=\"200\" /></a>" ?>
This works also fine but sometimes I obtain image like this:
I doubt that I exceeded the limit.
Any ideas ? Thank you for your answers.
As #geocodezip has suggested it could be because of not using a key.
Also as per the reverse geocoding documentation on :
https://developers.google.com/maps/documentation/geocoding/#ReverseGeocoding
Note: Reverse geocoding is an estimate. The geocoder will attempt to find the closest addressable location within a certain tolerance; if no match is found, the geocoder will return zero results.