I have a CouchDB view in my _user database, where I can find users based on their email addresses (user.email is part of their user doc).
{
"_id": "_design/find-user",
"views": {
"email->doc": {
"map": "function (doc) {\n emit(doc.email, doc);\n}"
}
},
"language": "javascript"
}
I am using the following curl url via PHP to access matching users:
curl_setopt($ch, CURLOPT_URL, $url . '/_users/_design/find-user/_view/email->doc?key="' . $email . '"');
This works, but is case sensitive, which I don't want. For example, I need $email = TesT#example.com to find tESt#example.com in my database.
I have tried the following, but that only transforms the output.
function (doc) {
emit(doc.email.toLowerCase(), doc);
}
You're on the right track with toLowerCase(). The only piece you're missing is that you must also transform your search term.
Don't search for "TEST#example.com", instead search for strtolower("TEST#example.com"):
curl_setopt($ch, CURLOPT_URL, $url . '/_users/_design/find-user/_view/email->doc?key="' . strtolower($email) . '"');
Related
I have a "users" table with columns:
Id | user_id | name | ip_address | lat | lng
The id column, user_id, name and ip_address have been populated, and now have 100k users.
The lat (latitude) and lng (longitude) columns have now been created, and I need them populated from the IP (you can not get the geolocation of the user).
I do not want precise location, just the approximate location, so what I need is a script that picks up the IP of each user, somehow extracts the latitude and longitude, and inserts into the database.
I have looked at "http://www.geoplugin.net/xml.gp?ip=" and "http://ip-api.com/json/" but I have no idea how to use api for 100k of users, nor PHP or mysql code, or shell to do this.
I am using only PHP, MySQLi with MySQL database. The application will display a list of users "nearby" (within a 100miles radius).
Excuse me, I'm very new to programming and I really do not know how to solve this.
Edit:
I just created an api that returns me all user_id and their respective IPs in an array.
Just being 105k of records, for being json, it was light but I did not want to share json publicly here because it would be the same as explaining my users' data.
the result of api is something like this:
{
"api_status": 200,
"ips": [
{
"user_id": "1",
"ip_address": "177.198.86.7x"
},
{
"user_id": "21",
"ip_address": "177.18.246.9x"
},
{
"user_id": "52",
"ip_address": "177.36.60.1x"
}
]
}
I am using the following function to return my data in php
function List_ip() {
global $sqlConnect, $db;
if ($db['loggedin'] == false) {
return false;
}
$ips = mysqli_query($sqlConnect, "SELECT `user_id` , `ip_address` FROM " . T_USERS . " WHERE `active` = '1'");
while ($fetched_data = mysqli_fetch_assoc($ips)) {
$list_ip[] = array_map('utf8_encode', $fetched_data);
}
return $list_ip;
}
I don't have much idea about limiting the number of requests, but try the below code to get the details from ip-api.
$url = "http://ip-api.com/batch";
$headers = array('Content-Type: application/json');
$fields = '[{"query": "208.80.152.201"}, // You can add multiple IP address in the same format.
{"query": "8.8.8.8"},
{"query": "24.48.0.1"}]';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_POSTFIELDS, $fields);
$httpCode = curl_getinfo($curl , CURLINFO_HTTP_CODE);
$response = curl_exec($curl);
if ($response === false)
$response = curl_error($curl);
foreach(json_decode($response) as $value){
echo $value->lat; //Use the result to store the information in DB
}
curl_close($curl);
Refer their documentation to get an idea about the data returned by their API.
Also, for limiting, try this solution
The maxmind/GeoIP2-php should get you want you need.
Install it and download the GeoIP2-City.mmdb file.
Than just a matter of initiating it:
$record = new Reader('path/to/GeoLite2-City.mmdb');
and reading the data from it, such as:
$city = $record->city->name;
$latitude = $record->location->latitude;
$longitude = $record->location->longitude;
Doing it this way will mean zero dependency on third-party API limitations.
What you are asking for requires intermediate skills with PHP and MySQL etc.. I have done many things like this, and will tell you that firstly to get accurate GEO IP information you will have to use javascript or pay for a service (from my findings). For example www.maxmind.com has a free and pay, the pay is a bit better.
What I would do is try writing a script that reads all of the users out of the database in a loop, then download geopip or use an api to take the IP fields and return the Lat/Lon. Then finally add the step that writes it back to the DB .
A very rough example:
// build a class or function that looks up IP addresses (can be local)
$geoip2 = new GeoIp2Lookup(true);
// get all users from DB
$allUsersArray = function_results_db_lookup_users();
// get the users lat/lon
for ($i=0; $i<count($allUsersArray); $i++) {
$record = $geoip2->getGeoDataArr($allUsersArray[$i]['ip_address']);
$allUsersArray[$i]['lat'] = $record['lat'];
$allUsersArray[$i]['lon'] = $record['lon'];
}
// now write a function to place in DB the new data
update_users_db($allUsersArray);
You can easily use the IP2Location PHP Module at https://www.ip2location.com/development-libraries/ip2location/php which is free.
To install, add the below line into your composer.json file.
{
"require": {
"ip2location/ip2location-php": "8.*"
}
}
Run the command:
composer install
Note: You just need to include the below code in order to use the library.
require 'vendor/autoload.php';
You'll need the DB5.LITE database which you can download for free at https://lite.ip2location.com/
Example usage of the PHP module
<?php
require 'IP2Location.php';
$myip = '8.8.8.8';
$db = new \IP2Location\Database('./database/IP-COUNTRY-SAMPLE.BIN', \IP2Location\Database::FILE_IO);
$records = $db->lookup($myip, \IP2Location\Database::ALL);
echo '<pre>';
echo 'IP Address : ' . $records['ipAddress'] . "\n";
echo 'Country Code : ' . $records['countryCode'] . "\n";
echo 'Country Name : ' . $records['countryName'] . "\n";
echo 'Region Name : ' . $records['regionName'] . "\n";
echo 'City Name : ' . $records['cityName'] . "\n";
echo 'Latitude : ' . $records['latitude'] . "\n";
echo 'Longitude : ' . $records['longitude'] . "\n";
echo '</pre>';
?>
What I need
I need to automatically find & download profile picture for user knowing his email address only. Originally, I focused on Facebook considering the amount of people actively using it. However, there seem to be no direct support from their API anymore.
There was similar question here:
How to get a facebook user id from the login email address which is quite outdated and current answers there are "it's deprecated" / "it's not possible"...
EDIT: I've found even better question: Find Facebook user (url to profile page) by known email address (where it is actually explained why and since when this feature isn't supported)
There must be a way...
What makes me think that this should be possible is that Spokeo is somehow doing it:
http://www.spokeo.com/email-search/search?e=beb090303%40hotmail.com
There are some services / APIs offering this kind of feature:
Clearbit
Pipl
...but I haven't found anything free.
Alternatives
If there is some workaround or different approach than using Facebook's API to achieve this, I would like to know. If Facebook is really completely hopeless here, then combination of these: Google+, Linkedin and/or Gravatar could do.
My first (original) attempt:
Once you have Facebook's username or user ID, it's easy to build URL to download the picture. So I was trying to look for Facebook's user IDs using emails with the /search Graph API:
https://graph.facebook.com/search?q=beb090303#hotmail.com&type=user&access_token=TOKEN
which unfortunatelly always ends with "A user access token is required to request this resource."
Using FB PHP API + FB App ID & Secret
I've also tried this: at first I retrieve access_token using app ID and secret and then I'm trying to use it as a part of /search request with curl:
function post_query_url($url, $data) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$res = curl_exec($ch);
curl_close($ch);
return $res;
}
function get_query_url($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$ret = curl_exec($ch);
curl_close($ch);
return $ret;
}
function get_retrieve_app_access_token($app_id, $secret) {
$url = 'https://graph.facebook.com/oauth/access_token?client_id='.$app_id.'&client_secret='.$secret.'&grant_type=client_credentials';
$res = get_query_url($url);
if (!empty($res)) {
$tokens = explode('=', $res);
if (count($tokens) == 2)
return $tokens[1];
}
return null;
}
function post_retrieve_app_access_token($app_id, $secret) {
$url = 'https://graph.facebook.com/oauth/access_token';
$data = 'client_id='.$app_id.'&client_secret='.$secret.'&grant_type=client_credentials';
$res = post_query_url($url, $data);
if (!empty($res)) {
$tokens = explode('=', $res);
if (count($tokens) == 2)
return $tokens[1];
}
return null;
}
function get_id_from_email($email, $accessToken) {
$url = 'https://graph.facebook.com/search?q='.urlencode($email).'&type=user&access_token='.$accessToken;
$res = get_query_url($url);
if (!empty($res)) {
return $res;
}
return null;
}
echo 'Retrieving token...<br>';
$token = post_retrieve_app_access_token('MY_APP_ID', 'SECRET');
echo 'Retrieved token: ' . $token . '<br>';
echo 'Retrieving user ID...<br>';
$id = get_id_from_email('beb090303#hotmail.com', $token);
echo 'Retrieved ID: ' . $id . '<br>';
outputs something like:
Retrieving token...
Retrieved token: 367458621954635|DHfdjCnvO243Hbe1AFE3fhyhrtg
Retrieving user ID...
Retrieved ID: {"error":{"message":"A user access token is required to request this resource.","type":"OAuthException","code":102}}
Other info
Since it's asking for "user access token", I've also tried to go to Facebook's Graph Explorer: https://developers.facebook.com/tools/explorer/
let it generate access token for me and queried:
search?q=beb090303#hotmail.com&type=user&debug=all
That one ends with:
{
"error": {
"message": "(#200) Must have a valid access_token to access this endpoint",
"type": "OAuthException",
"code": 200
}
}
...so Facebook seems kinda hopeless here.
That's exactly why Gravatar exists and why people use Gravatar, users know which public profile image they bind to which e-mail address and they know where to change it.
Your app can have the possibility for users to upload their own profile image and fallback to Gravatar.
If you just try to extract an image from Facebook or Google+, it might freak your users out and it will also be harder for them to know where your service got the profile image from.
Using Gravatar in PHP it is as simple as this:
<?php
$email = "email#server.com";
$default = ""; // absolute url to default image goes here or leave empty for default gravatar image
$size = 200;
$grav_url = "http://www.gravatar.com/avatar/" . md5(strtolower(trim($email))) . "?d=" . urlencode($default) . "&s=" . $size;
header("content-type: image/jpeg");
echo file_get_contents($grav_url);
?>
Apart from that, you can also use Facebook and/or Google+ as external login providers where users can grant your application access to their profile information.
There was a bug: Can't search for user by email after July 2013 Breaking Changes that has been closed as "By Design" with official response:
"The ability to pass in an e-mail address into the "user" search type was removed on July 10, 2013. This search type only returns results that match a user's name (including alternate name)" ~ Joseph Tuấn Anh Phan (Facebook Team)
so probably no direct support from Graph API.
I've tried Graph API Explorer where you can try to play with some FQL too (just need to select version 2.0 as newer versions are not supported anymore), unfortunately query like:
SELECT uid, name FROM user where email = 'some.email#gmail.com'
gives:
"error": {
"message": "(#604) Your statement is not indexable. The WHERE clause must contain
an indexable column. Such columns are marked with * in the tables linked from
http://developers.facebook.com/docs/reference/fql ",
"type": "OAuthException",
"code": 604
}
and reference for table user shows that only uid and third_party_id can be used in WHERE.
You should need access token as well as Facebook id of the user. without knowing them cannot get their profile pic
I think Spokeo might have an agreement with Facebook to access the data? I would not be surprised.
Anyway, if you are on a profile you can maybe search for profile_id in the HTML. It's a hack, not sure if it works.
You could always allow people to comment by logging in with their g+/facebook/whatever account (requires you to do something OpenID-like, though); if they've logged in, you should be able to get the facebook uid.
Also, there's something called libravatar, which allows people to associate pictures with their OpenID or email address (and which falls back to gravatar if they haven't configured anything specifically for libravatar); using that should give you more photos than if you stick to "just" gravatar.
I'm trying to call the DocuSign REST login information within a CodeIgniter application. The Docusign sample code shows:
// Input your info here:
$integratorKey = '...';
$email = '...#....com';
$password = '...';
$name = 'John Doe';
// construct the authentication header:
$header = "<DocuSignCredentials><Username>" . $email . "</Username><Password>" . $password . "</Password><IntegratorKey>" . $integratorKey . "</IntegratorKey></DocuSignCredentials>";
/////////////////////////////////////////////////////////////////////////////////////////////////
// STEP 1 - Login (to retrieve baseUrl and accountId)
/////////////////////////////////////////////////////////////////////////////////////////////////
$url = "https://demo.docusign.net/restapi/v2/login_information";
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, array("X-DocuSign-Authentication: $header"));
$json_response = curl_exec($curl);
$status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
if ( $status != 200 ) {
echo "error calling webservice, status is:" . $status;
exit(-1);
}
I keep getting a status response of 0. When I echo out the curl all the xml tags have been converted to lowercase (which won't work with Docusign). Has anyone done this call within CodeIgniter? How did you accomplish it? I know my credentials are good because I can do a command line curl and get a response.
I'm not sure what CodeIgniter is doing and why the tags would be converted to lower case but you're right in that DocuSign expects xml tags to begin with a capital letter so that might not work. What you can do, though, is use JSON headers and request bodies instead.
For instance, instead of an XML formatted authentication header like
<DocuSignCredentials>
<Username>username</Username>
<Password>password</Password>
<IntegratorKey>integrator_key</IntegratorKey>
</DocuSignCredentials>
You could use the corresponding JSON auth header like
{
"Username": "username",
"Password": "password",
"IntegratorKey": "integrator_key"
}
The "nodes" still start with a capital but since it's not xml I'm wondering if CodeIgniter maybe leaves it alone. Examples of this can be found at the DocuSign Developer Center on this page.
I have been trying to use MongoLabs api to simplify my life, and for the most part it was working until I tried to push updates to the db using php and curl, anyway no dice. My code is similar to this:
$data_string = json_encode('{"user.userEmail": "USER_EMAIL", "user.pass":"USER_PASS"}');
try {
$ch = curl_init();
//need to create temp file to pass to curl to use PUT
$tempFile = fopen('php://temp/maxmemory:256000', 'w');
if (!$tempFile) {
die('could not open temp memory data');
}
fwrite($tempFile, $data_string);
fseek($tempFile, 0);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
//curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
//curl_setopt($ch, CURLOPT_INFILE, $tempFile); // file pointer
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, DB_API_REQUEST_TIMEOUT);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($data_string),
)
);
$cache = curl_exec($ch);
curl_close($ch);
} catch (Exception $e) {
return FALSE;
}
My problem seems to be with MongoLab's api. The code bit works perfect except for the fact that labs tells me that the data I am passing is an 'Invalid object{ "user.firstName" :"Pablo","user.newsletter":"true"}: fields stored in the db can't have . in them.'. I have tried passing a file and using the postfields, but neither worked.
When I test it on firefox's Poster plugin the value work fine. If someone out there has a better understanding of MongoLabs stuff I would love some enlightenment. Thanks in advance!
You will need to remove the dots from your field names. You might try going to a schema like this:
{ "user": { "userEmail": "USER_EMAIL", "pass": "USER_PASS" } }
Unfortunately, MongoDB doesn't support using dots in field names. This is because its query language uses the dot as an operator to chain nested field names. If MongoDB were to allow dots in field names dotted queries would become ambiguous without some kind of escaping mechanism.
If this document were legal:
{ "bow.ties": "uncool", "bow": { "ties": "cool" } }
This query would be ambiguous:
{ "bow.ties": "cool" }
Not clear if the document would match or not. Did you mean the field "bow.ties" or the field "ties" nested within the value of field "bow"?
Here's a capture of a mongo shell session demonstrating these ideas.
% mongo
MongoDB shell version: 2.1.1
connecting to: test
> db.stuff.save({"bow.ties":"uncool"})
Wed Jul 18 11:17:59 uncaught exception: can't have . in field names [bow.ties]
> db.stuff.save({"bow":{"ties":"cool"}})
> db.stuff.find({"bow.ties":"cool"})
{ "_id" : ObjectId("5006ff3f1348197bacb458f7"), "bow" : { "ties" : "cool" } }
After sometime working with some other functionality of the project I realized my mistake, and ultimately the source of the confusion.
The curl PUT was intended to send modifier operations to MongoDB. I was sending all my data as JSON and was interrupting decoding it to use in PHP then re-encoding part of it to send back. So the orignal data received looks something like this:
{"userEmail":"p#g.com","pass":"****", "$oid":"5555", "$set":{"user.firstName":"Pablo","user.newsletter":"true"}}
The problem was that I was grabbing the value of "$set" object (in php) and reencoding only the value, {"user.firstName":"Pablo","user.newsletter":"true"} without the operator "$set" and was sending it giving the error. In this case the proper string to send would have been {"$set":{"user.firstName":"Pablo","user.newsletter":"true"}}
While this is a simple mistake I hope that the next time someone does something like this and gets an invalid object error that they are luck enough to find this.
how to find the total no.of inbound and outbound links of a website using php?
To count outbound links
parse html for webpage
parse all links using regex
filter links which starts with your domain or "/"
To inbound link
Grab google results page
http://www.google.ca/search?sourceid=chrome&ie=UTF-8&q=site:
parse similarly
For outbound links, you will have to parse the HTML code of the website as some here have suggested.
For inbound links, I suggest using the Google Custom Search API, sending a direct request to google can get your ip banned. You can view the search api here. Here is a function I use in my code for this api:
function doGoogleSearch($searchTerm)
{
$referer = 'http://your-site.com';
$args['q'] = $searchTerm;
$endpoint = 'web';
$url = "http://ajax.googleapis.com/ajax/services/search/".$endpoint;
$args['v'] = '1.0';
$key= 'your-api-key';
$url .= '?'.http_build_query($args, '', '&');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $referer);
$body = curl_exec($ch);
curl_close($ch);
//decode and return the response
return json_decode($body);
}
After calling this function as: $result = doGoogleSearch('link:site.com'), the variable $result->cursor->estimatedResultCount will have the number of results returned.
PHP can't determine the inbound links of a page through some trivial action. You either have to monitor all incoming visitors and check what their referrer is, or parse the entire internet for links that point to that site. The first method will miss links not getting used, and the second method is best left to Google.
On the other hand, the outbound links from a site is doable. You can read in a page and analyze the text for links with a regular expression, counting up the total.
function getGoogleLinks($host)
{
$request = "http://www.google.com/search?q=" . urlencode("link:" . $host) ."&hl=en";
$data = getPageData($request);
preg_match('/<div id=resultStats>(About )?([\d,]+) result/si', $data, $l);
$value = ($l[2]) ? $l[2] : "n/a";
$string = "" . $value . "";
return $string;
}
//$host means the domain name