Is there a service that lets me check a URL to see if it may possibly be a dangerous site?
When a user exits our application by clicking on an untrusted link, we sent them through a "are you sure you want to leave" redirection screen. It'd be a nice touch to do a quick check to see if we should warn the user as well.
Try with Google Safe Browsing API.
The Google Safe Browsing Lookup API is an experimental API that allows applications to check URLs against Google's constantly-updated lists of suspected phishing and malware pages.
You can use the Google Safe Browsing API to check if a URL is safe according to what they know about it. (API documentation)
How about z-protect.com?
Example report: http://www.z-protect.com/report/stackoverflow.com/
this system is using a dns server to block the pages ... maybe interessting?
An this service has a api! but with a limit of 10000 requests per day:
http://www.z-protect.com/api
Related
I know I can disallow robots using robots.txt but few search engines does not follow this. Hence I have a API where my users sends transactional info to insert/update/delete etc., using my API Request Parameters. But when I look at my logs, huge hits have been made to my .php page, Hence I google to use it in my php API page and found nothing.
Hence I landed on SO to get help from experts, is there any way I can block/disallow SE robots to access my base API URL?
The main approaches that I know of for dealing with bots that are ignoring robots.txt are to either:
Blacklist them via your firewall or server
Only allow whitelisted users to access your API
However, you should ask yourself whether they're having any impact on your website. If they're not spamming you with requests (which would be a DDoS attack) then you can probably safely ignore them and filter them out of your logs if you need to analyse real traffic.
If you're running a service that people use and you don't want it to be wide open to spam then here's a few more options on how to limit usage:
Restrict access to your API just to your users by assigning them an API token
Rate limit your API (either via the server and/or via your application)
Read the User Agent (UA) of your visitors, a lot of bots will mention they're bots or have fake UAs, the malicious ones will pretend to be users
Implement more advanced measures such as limiting access to a region if a lot of requests suddenly come from there in a short period of time
Use DDoS protection services such as CloudFlare
There's no perfect solution and each option involves trade-offs. If you're worried about DDoS then you could start by looking into your server's capabilities, for example here's an introduction into how NGINX can control traffic: https://www.nginx.com/blog/rate-limiting-nginx/
In a nutshell, any IP hitting your site can be a bot so you should defend by imposing limits and analysing behaviour, since there's no way to know for sure who is a malicious visitor and who isn't until they start using your service.
I'm building application that uses components from Facebook API, and there are certain requirements so app could go to review. I solved other requirements and I'm not quite sure do I need a HTTPS or it could work with HTTP just fine?
You only need https if you create a Page/Tab App or a Canvas App. Check out the App Settings, those platforms specifically ask for a https link.
Working with http is fine, Token security is done best with appsecret_proof. It does not really matter if you use http or https for that.
That being said, having https is better than not having https. But the question was "do i need it", so...see my very first sentence :)
I don't know specifically about facebook rules but:
If the front end (javascript) has access to token to identification the user, maybe you should use https to protect that token.
If you manipulate personal data of the user, then in a lot of country (in European countries for example) you have an obligation of protection.
As https protect your website against unwanted modifications (ISP injecting Ads) and protect your users, in 2016 the question is less "should I use https?" and more "do I have a good reason not to?"
optimizepress is a wordpress plugin. I own a copy and use it and am wondering how they use the licensing to secure the product.
I would like to consider this for securing my own php script if it's viable.
Here's what they do to secure their product:
On there server where you download the script you have to enter your domain url in a text box to license the plugin for that url.
They have 2 textboxs to enter your domains in: 1. if it's first time licensing sites 2. adding more sites to your account
Then you click a submit button and a serial code is sent back
After you install the plugin in wordpress, you must goto the settings area where it asks you to enter the serial code for verification otherwise you can't use the script
How is this done? Could this be used with my php script I'll be distributing?
thanks for your thoughts
I'm not specifically familier with this script however it is very possible it is a script using cURL, the serial number will be used to authenticate your account for verification to access your account.
Next it will most likely pull the URL the script is being run from and verify that the URL is listed on your account.
From there if the URL is not listed it will probably send a fail command killing the script, if the URL is listed it will authenticate.
If it is listed then the script will check the status of the license and either run or kill the script.
There is a division in using this type of model whereby some developers believe if there is no response from the auth server then the script should assume it is allowed to run to not interrupt the user experience, others believe if the auth server doesn't reply its very possible it could be an unauthorised usage so they kill the activity.
There are other technologies available to do the same thing and depending on your application this may not be suitable, just have a look at various licensing models.
As a note, for this style of distribution you may want to use a PHP encoder such as ionCube so the script cannot be hacked to bypass authentication.
In addition when writing a script to use an API, I also recommend having the user assign the IP address, this way the first query to the API for auth is am I allowed to talk to this server. This provides a lot of protection to your API script that will then move on to verify the license is valid or not.
I hope this will be helpful for you.
I have a demo server where I put samples of my apps, I send potential customers links to those apps. Is it possible to use htaccess to track visitors, without adding tracking capability to the apps themselves? The data I'm interested in are:
date and time of page visit
ip of visitor
url of the page visited
referrer
post and get (query string) data if any
That entirely depends on your webserver, what options it provides for htaccess overrides.
For Apache, the access log logs what you are looking for
http://httpd.apache.org/docs/current/logs.html#accesslog
but is not configurable via htaccess.
no, that's impossible to use .htaccess file, because it's merely a configuration file, not executable one.
However you can use another web-server capability - log files.
Everything you asking for is already stored in the access log, almost in the same format you listed here.
An important note: unlike google analytics or any other third-party or scripting solution, web-server logs is the only reliable and exact source of tracking data, contains very request been made to your site.
Best way it to use google analytics.
You will get all what you need and much much more.
I know this thread has been quiet for a while, but i it not possible to use the prepend?? directive that prepends a script to all visits to track site/page visits ?
I have not got the code (tried something similarthough was not successfull) but I used the prepend directive to prepend a script that "switches" on gzip for all site visits. I am sure the same can be implemented for logs (for those of us with cheap shared servers!) Come on coders, do us all a favour and reveal the secret!
I have a website where most of the traffic comes from the API (http://untiny.com/api/). I use Google Analytics to collect traffic data, however, the statistics do not include the API traffic because I couldn't include the Google Analytics javascript code into the API pages, and including it will affect the API results. (example: http://untiny.com/api/1.0/extract/?url=tinyurl.com/123).
The solution might be executing the javascript using a javascript engine. I searched stackoverflow and found javascript engines/interpreters for Java and C, but I couldn't find one for PHP except an old one "J4P5" http://j4p5.sourceforge.net/index.php
The question: is using a javascript engine will solve the problem? or is there another why to include the API traffic to Google Analytics?
A simple problem with this in general is that any data you get could be very misleading.
A lot of the time it is probably other servers making calls to your server. When this is true the location of the server in no way represents to location of the people using it, the user agent will be fake, and you can't tell how many different individuals are actually using the service. There's no referrers and if there is they're probably fake... etc. Not many stats in this case are useful at all.
Perhaps make a PHP back end that logs IP and other header information, that's really all you can do to. You'll at least be able to track total calls to the API, and where they're made from (although again, probably from servers but you can tell which servers).
I spent ages researching this and finally found an open source project that seems perfect, though totally under the radar.
http://code.google.com/p/serversidegoogleanalytics/
Will report back on results.
you would likely have to emulate all http calls on the server side with whatever programming language you are using..... This will not give you information on who is using it though, unless untiny is providing client info through some kind of header.
if you want to include it purely for statistical purposes, you could try using curl (if using php) to access the gif file if you detect untiny on the server side
http://code.google.com/apis/analytics/docs/tracking/gaTrackingTroubleshooting.html#gifParameters
You can't easily do this as the Javascript based Google Analytics script will not be run by the end user (unless of course, they are including your API output exactly on their display to the end user: which would negate the need for a fully fledged API [you could just offer an iframable code], pose possible security risks and possibly run foul of browser cross-domain javascript checks).
Your best solution would be either to use server side analytics (such as Apache or IIS's server logs with Analog, Webalizer or Awstats) or - since the most information you would be getting from an API call would be useragent, request and IP address - just log that information in a database when the API is called.