I'm currently trying to grab a file from an external url that has an authorization box that pops up (like the default one asking for a username and password)
How can I have a script get the contents of the page (it's a video), save it to a directory and handle the authorization (i have a username and password)
Thanks :)
file_put_contents('where to put it', file_get_contents('http://username:password#domain.com/video'));
In a word, look at curl: http://php.net/curl, for all you posting/logging in/cookies/session needs in HTTP country.
You don't need to download the page, just check what is being submitted to the web server. Chances are it's just a POST. It may have some additional checks (i.e. checksum) which may need to be scraped from the page.
You can use the HTTP Headers plugin for Firefox to see how the browser is communicating with the server. You then just need to emulate that transaction. It is likely a POST, which is easy to do with CURL.
I don't think file_put_contents will work since it doesn't do an http POST.
Related
I don't know the proper way to say this, so here is my issue.
I am dealing with an API that sends information to the server, from either web form or whatever in this format:
http://server/non_agent_api.php?source=test&user=6666&pass=1234....
with a bunch more parameters.
Normally, I have dealt with API's that just send it with SOAP or REST, not in a URL like that. My question is how do I send that information using php or something. So if I wanted to take in a username and password from a webform, how do I send that link to the server without clicking on the URL itself.
I hope that makes sense. Thanks for the help.
You can use curl for that like,
Read curl(), here you can find a class which can be easily used.
In PHP use
header("Location: TARGETURL");
Create the TARGETURL using the information sent from the form.
The Location Header makes the server to generate 302 Moved temporarily HTTP return code. The browser then sends the user to the TARGETURL transparently without any further interaction.
I have looked at some examples and they use Curl in php etc but I just want a simple command that passes my credentials to a site so it can login and give me access to download a file. Current problem is when I try to use get_contents I get stuck at login page because it needs a login before it can allow a download so isnt there a way to send my login info before get_contents in php? Thanks
for example we can assume the website is located at www.confluence.com
You're out of luck. file_get_contents() can only get content, hence the name. You can try to authenticate via the get syntax for standard authentication, like http://username:password#example.com.
If you have to post your credentials via HTTP POST, you'll have to use curl.
The problem is that when you log in the server send to your browser a cookies that your browser automatically stores.
With file_get_content() you can actually pass cookies ( the third context parameter of file_get_content() can handle this).
Have a look at stream context create.
By the way, you need to first send your login info to the login page (with curl), when you recive cookies back, pass them as option to file_get_content() and the trick is done.
If the server is using a login system different from cookies let us know, so we can help you
there is a page that i need to post a password to it and then i get a file to download.
the post goes to the same page address its loads again and pop up the download manager (download starts automatically).
now i want to do the same but in curl, i posted the data to the url and then its sends me the file back but i don't want my script to download the whole file i want only to get a link to download it by myself.
how can i do that?
Actually, you most probably can't. Such password protected download system usually checks either cookies or browser / environment based variables. Getting the link itself shouldn't be problem, however you could not use it outside this generator's scope anyway.
firstly you need to post that password with curl assuming "on specific form. the form will take you to the downloading page" now you need to use regex (regular expressions).
filter the data you want then save it on other variable to re-use it.
There is for sure a redirection after you hit 1st page with POST. Look for that redirection with curl and read http response headers: Content-Location or Location or even Refresh
To prevent the automatic download you have to set the curl opt to not follow redirects. I can't remember the exact command but curl by default will follow auto refreshes and URL redirects, which happen in split seconds so humans don't actually see it happening.
I kinda don't understand what you really want to do, but if you just want a link then have the php script perform the entire curl post and everything when they click it. Doesn't matter what the web server will require a password before access to a file, you can't skip that step.
I have a site that is using frames. Is it still possible from the browser for someone to craft post data for one of the frames using the address bar? 2 of the frames are static and the other frame has php pages that communicate using post. And it doesn't appear to be possible but I wanted to be sure.
No, it is not possible to POST data from the address bar. You can only initiate GET requests from there by adding params to the URL. The POST Body cannot be attached this way.
Regardless of this, it is very much possible to send POST requests to your webserver for the pages in a frame. HTTP is just the protocol with which your browser and webserver talk to each other. HTTP knows nothing about frames or HTML. The page in the frame has a URI, just like any other page. When you click a link, your browser asks the server if it has something for that URI. The server will check if it has something for that URI and respond accordingly. It does not know what it will return though.
With tools like TamperData for Firefox or Fiddler for IE anyone can tinker with HTTP Requests send to your server easily.
Any data in the $_REQUEST array should be considered equally armed and dangerous regardless of the source and/or environment. This includes $_GET, $_POST, and $_COOKIE.
POST data can not be added in the address bar.
You should always check & sanitize all data you get in your PHP code, because anyone could post data to all of your pages.
Don't trust data from outside of your page. Clean it & check it.
Maybe not from the browser, but they can still catch the request (tinker with it) and forward it to the provided destination, with a tool like burp proxy.
To answer your question: No, it is not possible to send post data using the addressbar.
BUT it is possible to send post data to any url in a snap. For example using cURL, or a Firefox extension. So be sure to verify and sanitize all the data you receive no matter if POST or GET or UPDATE or whatever.
This is not iFrame or php specific, so it should be considered in every webapplication. Never ever rely on data send by anyone being correct, valid or secure - especially when send by users.
Yes, they absolutely can, with tools like Firebug, and apparently more specialized tools like the ones listed by Gordon. Additionally, even if they couldn't do it in the browser from your site, they could always create their own form, or submit the post data through scripting or commandline tools.
You absolutely cannot rely on the client for security.
I need to make a proxy script that can access a page hidden behind a login screen. I do not need the proxy to "simulate" logging in, instead the login page HTML should be displayed to the user normally, and all the cookies and HTTP GET/POST data to flow through the proxy to the server, so the login should be authentic.
I don't want the login/password, I only need access to the HTML source code of the pages generated after logging in.
Does anybody here know how this can be accomplished? Is it easy?
If not, where do I begin?* (I'm currently using PHP)*
Have your PHP script request the URL you want, and rewrite all links and form actions to point back to your php script. When receiving requests to the script that have a URL parameter, forward that to the remote server and repeat.
You won't be able to catch all JavaScript requests, (unless you implemented a JavaScript portion of your "proxy")
Eg: User types http://example.com/login.php into your proxy form.
send the user to http://yoursite.com/proxy.php?url=http://example.com/login.php
make sure to urlencode the parameter "http://example.com/login.php"
In http://yoursite.com/proxy.php, you make an HTTP request to http://example.com/login.php
$url = $_REQUEST['url'];
// make sure we have a valid URL and not file path
if (!preg_match("`https?\://`i", $url)) {
die('Not a URL');
}
// make the HTTP request to the requested URL
$content = file_get_contents($url);
// parse all links and forms actions and redirect back to this script
$content = preg_replace("/some-smart-regex-here/i", "$1 or $2 smart replaces", $content);
echo $content;
Note that /some-smart-regex-here/i is actually a regex expression you should write to parse links, and such.
The example just proxies the HTTP Body, you may want to proxy the HTTP Headers. You can use fsockopen() or PHP stream functions in PHP5+ (stream_socket_client() etc.)
You could check out http://code.google.com/p/php-transparent-proxy/ , I made it because I was asking myself that exact same question and I decided to make one. It's under BSD license, so have fun :)
What you are talking about is accessing pages for which you need to authenticate yourself.
Here are a few things that must be laid down:
you can't view those pages without authenticating yourself.
if the website (whose HTML code you want to see) only supports web login as an authentication method, you will need to simulate login by sending a (username,password) via POST/GET, as the case may be
if the website will let you authenticate yourself in other ways (like LDAP, Kerberos etc), then you should do that
The key point is that you cannot gain access without authenticating yourself first.
As for language, it is pretty doable in PHP. And as the tags on the question suggest, you are using the right tools to do that job already.
One thing I would like to know is, why are you calling it a "proxy"? do you want to serve the content to other users?
EDIT: [update after comment]
In that case, use phproxy. It does what you want, along with a host of other features.
I would recommand using Curl (php library that you might need to activate in your php.ini)
It's used to manipulate remote websites, handling cookies and every http parameters you need.
You'll have to write your proxy based on the web pages you're hitting, but it'll make the job.