I'm using PHP to data scrape another website. However, on certain occasions I need to confirm a variable (due to have two very alike possibilities).
The button I'm supposed to click to confirm my variable is:
<input type="submit" class="buttonEmphasized confirm_nl" name="start" value="Bevestig" accesskey="s" />
However, adding &start=Bevestig to the url doesn't seem to solve the problem, and I'm receiving the same page. What's more, is that the website is using sessions and every http_post_data seems to be starting a new session.
Is there a way to let PHP "click" a button if a certain output is missing?
This is a train time table data scraping system (using the HAFAS system).
Cheers
there is no generalized solution for this problem. every site is different in some way. your best bet is to analyze http message being sent by the original page. you can do it with firefox+firebug+live http headers for example. this way you're going to see all the parameters(required or not) and then replicate this message with your script.
it might(will, most likely) require faking session/cookie data. you might need to use curl for that.
If the post seems to be starting a new session, I would suspect that you are not respecting the cookies that were provided by the other side.
You need to send the session cookies back in the POST request.
That's also where you should be sending your start field. While many pages will accept parameters in URL or posted, they are not equivalent concepts.
"Is there a way to let PHP "click" a button if a certain output is missing?"
Nop, PHP is server-sided. Use Javascript
Related
I am currently diving into php and html and working on a simple redirect just for the purpose of showing database content through an url.
I know you can generate an URL in 2 ways, probably more but these two are the reason why I started this question:
php:
<?php
header('Location: example.php?parameter');
?>
html:
<form action="example.php" method="post">
<!--input fields etc -->
<input type="submit">
But now I was wondering "What is the best practice regarding these two options". Is it just a personal opinion with what you like the best and what is the best suitable way in a situation or is there something else I am overlooking.
I am not trying to start a discussion here, just interested in what is 'normally/commonly' used.
Thanks in advance!
These two pieces of code do fundamentally different things, even though in some cases the user-observed behavior may be very similar.
This is a server-initiated redirect:
<?php
header('Location: example.php?parameter');
?>
Basically this is the server's way of telling the browser that it should browse to another location. (The browser can ignore it, but doesn't really have a reason to ignore it.) Additional details can be added to the response to tell the browser if this redirection is temporary or permanent, or has other conditions regarding it. But at its simplest this is just the server saying "I don't really have anything for you here, go over there for your information."
This is a client-initiated form POST:
<form action="example.php" method="post">
Well, "client-initiated" in that the actual action of POSTing the form comes from the browser. The server probably gave that HTML tag to the client to tell it to do that, but the client is free to change it if it wants. (There's no reason to do so, though.) The point here is that this is a means by which the client sends data to example.php. It has nothing to do with redirects, it's just sending data to a specific resource on the server.
The server can respond to that data with a redirect, or a rendered page, or any other response.
These might be used in conjunction in a number of ways. Let's say you have page1.php and page2.php. On page1 there is a form, and after that form is submitted you want the user to see page2. This is where the user-observed result might be indistinguishable.
page1 can post to page2 and page2 can handle the submitted data and then display. Or page1 can post back to page1, handle the submitted data, and redirect to page2. To the end user, there's essentially no difference. The main difference is in how you organize your code. In that regard, sure, personal preference comes into play. But this isn't the only scenario in which either of these tools are employed. For example, you might want to submit values to a completely different page for a completely different reason, or redirect on a page request for some server-side reason completely unknown to the client.
As you develop more complex web applications you'll find certain patterns work well in certain situations, and personal preference will begin to conform to those patterns. In the end, these are just tools to perform actions (redirect the client to another location, send data to the server) and your overarching patterns and practices simply make use of the tools.
The HTTP location header and a HTML form are not really comparable.
The header should be used if you want to create a redirect during the execution of PHP. The form should be used if you want to submit user input from the client side (browser) to the server side.
HTML anchors are the best way to provide links on a web page:
Click
In most cases you use html forms or links. header() is used mainly if you want to redirect an user after the code is executed (e.g. after a successful login, or when is not authorized to access a restricted page)
I want to implement a system where I want to know where a POST request cam from.
For example: I have a form with a submit button in it. When the User clicks on the submit button it goes to the page. But I want to know the source from where the post request came.
This is the code till now:
<form action="profile.php?id=<?php echo $user->id; ?>" method="post" name="formAdd"><input name="btnAddUser" type="submit" value="Add User"></form>
Should I use a hidden input element? Would that be a good way OR maybe something else?
First of all, there is no reliable way - users can tamper with requests if they want to.
Besides that, there are two ways to get the kind of information you want:
The referer, available via $_SERVER['HTTP_REFERER']: It contains the full URL from which the request came, but some people use extensions/firewalls/etc. that block or even spoof referers
As you suggested, a hidden form element. This always works unless the user actively wants to tamper with the data sent. So that's the preferred way.
The $_SERVER['HTTP_REFERER'] will let you know where the request came from.
More info:
http://php.net/manual/en/reserved.variables.server.php
It really depends on how secure and reliable you need it to be. A hidden form field would work although it means you'd need to add it to every form that points to your processing script. It's also easy to fake if someone wanted to. Alternatively you could use $_SERVER['HTTP_REFERER']. This isn't always reliable - I believe it does depend on what browser you're using but should be good enough in most simple scenarios. Another alternative would be to store something in the session and use that. That's probably the most secure as it's all server-side and can't be tampered with, but it is probably the hardest to implement (not that it's rocket science).
You could save the page in a session variable ($_SESSION["something"] = "page.php"), that is the most secure way, I think, because a hidden input in a form could be changed by the user, and $_SERVER['HTTP_REFERER'] is not always avaliable.
I would use a hidden field where the value="name_of_referring_page".
This way, no matter what the user's settings, firewall, browser, etc you get the info that you want.
I'm preparing my paypal system and have a separate page that forwards the user to paypal. This page currently creates a form with all the needed hidden fields and then submits itself using
<body onload="form1.submit()">
However, when Javascript is not activated, the user gets stuck at this page.
What other method could I possibly use directly in PHP to solve this problem?
Just do the relevant request in PHP, for example using the curl binding.
PHP uses the header() directive, in which you can forward someone to another url. Not sure about your other information. If PayPal allows that to be sent in the GET string, this could work for you. If it has to be POST, then you're probably out of luck.
Or, you can use the cURL library if PayPal returns a url for you to forward the user to.
Another option may be to allow the user to physically click the submit button for the form, and use JS to hide the form itself or something.
PHP runs on your server, so without an intermediate language (like JavaScript), you are out of luck.
When you view a PHP page, the PHP engine runs the code, gets the output, and serves a plain ol' HTML page to the user. The user never interacts directly with the PHP code, only with the output.
As indicated before, you can fall back on a header() redirect with GET parameters.
header('Location: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick&business=youremail#example.com&amount=1¤cy_code=USD');
Just append the URL with any parameters you need in name=value pair format, a list of which you can find here: https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/e_howto_html_Appx_websitestandard_htmlvariables
Why not just provide a more manual solution for users with Javascript disabled?
E.g. if the user has Javascript disabled, just show a submit button where they can manually move themselves along to PayPal?
Presumably, users with Javascript disabled are somewhat accustomed to a lower quality of service across the web.
I need to cause a user's browser to post data using PHP to another site.
Example: You go to start.com/auto-login-hack (via GET)... then PHP sets the right headers etc. and causes the browser to, via POST, go to 3rdparty.com/login.php with login credentials.
I have done this is the past by having an HTML form and an onload script that submits the form to the destination.
I don't know enough about headers and etc. Is this possible? Can anyone link an example? My search skills just turned up how to use $_POST.
Thanks.
Yes, you can submit POST requests from PHP.
One of your choices is to use curl as shown in this SO question.
However, you cannot do redirects.
You cannot redirect to a POST; this is a limitation of HTTP. You'd have to use JavaScript to cause the browser to post a form.
I have an application that supplies long list of parameters to a web page, so I have to use POST instead of GET. The problem is that when page gets displayed and user clicks the Back button, Firefox shows up a warning:
To display this page, Firefox must send information that will repeat any action (such as a search or order confirmation) that was performed earlier.
Since application is built in such way that going Back is a quite common operation, this is really annoying to end users.
Basically, I would like to do it the way this page does:
http://www.pikanya.net/testcache/
Enter something, submit, and click Back button. No warning, it just goes back.
Googling I found out that this might be a bug in Firefox 3, but I'd like to somehow get this behavior even after they "fix" it.
I guess it could be doable with some HTTP headers, but which exactly?
See my golden rule of web programming here:
Stop data inserting into a database twice
It says: “Never ever respond with a body to a POST-request. Always do the work, and then respond with a Location: header to redirect to the updated page so that browser requests it with GET”
If browser ever asks user about re-POST, your web app is broken. User should not ever see this question.
One way round it is to redirect the POST to a page which redirects to a GET - see Post/Redirect/Get on wikipedia.
Say your POST is 4K of form data. Presumably your server does something with that data rather than just displaying it once and throwing it away, such as saving it in a database. Keep doing that, or if it's a huge search form create a temporary copy of it in a database that gets purged after a few days or on a LRU basis when a space limit is used. Now create a representation of the data which can be accessed using GET. If it's temporary, generate an ID for it and use that as the URL; if it's a permanent set of data it probably has an ID or something that can be used for the URL. At the worst case, an algorithm like tiny url uses can collapse a big URL to a much smaller one. Redirect the POST to GET the representation of the data.
As a historical note, this technique was established practice in 1995.
One way to avoid that warning/behavior is to do the POST via AJAX, then send the user to another page (or not) separately.
I have been using the Session variable to help in this situation. Here's the method I use that has been working great for me for years:
//If there's something in the POST, move it to the session and then redirect right back to where we are
if ($_POST) {
$_SESSION['POST']=$_POST;
redirect($_SERVER["REQUEST_URI"]);
}
//If there's something in the SESSION POST, move it back to the POST and clear the SESSION POST
if ($_SESSION['POST']) {
$_POST=$_SESSION['POST'];
unset($_SESSION['POST']);
}
Technically you don't even need to put it back into a variable called $_POST. But it helps me in keeping track of what data has come from where.
I have an application that supplies long list of parameters to a web page, so I have to use POST instead of GET. The problem is that when page gets displayed and user clicks the Back button, Firefox shows up a warning:
Your reasoning is wrong. If the request is without side effects, it should be GET. If it has side effects, it should be POST. The choice should not be based on the number of parameters you need to pass.
As another solution you may stop to use redirecting at all.
You may process and render the processing result at once with no POST confirmation alert. You should just manipulate the browser history object:
history.replaceState("", "", "/the/result/page")
See full or short answers