I want to POST an URL using CURL and php.
There is a big form on webpage and I don't want to manually copy all the variables and put it in my POST request.
I am guessing there has to be a way to serialize the form automatically (using DOM or something) and then just change whatever values I need.
I could not google my way out of this one so I was wondering would anyone be kind enough to help.
So, is there anyway to automatically serialize a form which is buried in a bunch of html content I just pulled from a URL?
Thanks for any help,
Andrew
$http = new HttpQueryString();
$http->set($_POST);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$http->get());
Requires PECL pecl_http >= 0.22.0
Its not to clear to me if you are asking how to get the form in the browser to the server or how to place the posted form in the curl request.
From php, assuming the form posted over, it would be as simple as:
curl_setopt($handle, CURLOPT_POSTFIELDS, $_POST);
though no data is validated that way.
From the web side, not sure why you would serialize the form using the DOM/Javascript, as opposed to just submitting it via a normal post?
Not sure what the question really is, but you're either wanting to do something like this:
$fields_string = http_build_query($data_to_send);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$fields_string);
or you need to look into this:
$html = file_get_html('http://www.thesite.com/thepage.html');
foreach($html->find('input') as $element)
echo $element->name . '<br>';
I don't understand the question. It sounds like you want to screen-scrape a form, fill it in, and then POST it back to the page you got it from. Is that right?
Edit in response to comment:
I'd recommend scraping the CURL'd HTML with a tool like Simple HTML DOM (that's what I use for scraping with PHP). The documentation for your library of choice will help you figure out how to identify the form fields. After that, you'll want to curl the form's action page, with the CURL_POST_FIELDS attribute set to the values you want to pass to the form, urlencode()'d of course.
Related
There is a PHP form which queries a massive database. The URL for the form is https://db.slickbox.net/venues.php. It takes up to 10 minutes after the form is sent for results to be returned, and the results are returned inline on the same page. I've tried using Requests, URLLib2, LXML, and Selenium but I cannot come up with a solution using any of these libraries. Does anyone know of a way to retrieve the page source of the results after submitting this form?
If you know of a solution for this, for the sake of testing just fill out the name field ("vname") with the name of any store/gas station that comes to mind. Ultimately, I need to also set the checkboxes with the "checked" attribute but that's a subsequent goal after I get this working. Thank you!
I usually rely on Curl to do these kind of thing.
Instead of sending the form with the button to retrieve the source, call directly the response page (giving it your request).
As i work under PHP, it's quite easy to do this. With python, you will need pycURL to manage the same thing.
So the only thing to do is to call venues.php with the good arguments values thrown using POST method with Curl.
This way, you will need to prepare your request (country code, cat name), but you won't need to check the checkbox nor load the website page on your browser.
set_ini(max_execution_time,1200) // wait 20 minutes before quitting
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "https://db.slickbox.net/venues.php");
curl_setopt($ch, CURLOPT_HEADER, 0);
// prepare arguments for the form
$data = array('adlock ' => 1, 'age' => 0,'country' => 145,'imgcnt'=>0, 'lock'=>0,'regex'=>1,'submit'=>'Search','vname'=>'test');
//add arguments to our request
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
//launch request
if( ! $result = curl_exec($ch))
{
trigger_error(curl_error($ch));
}
echo $result;
How about ghost?
from ghost import Ghost
ghost = Ghost()
with ghost.start() as session:
page, extra_resources = session.open("https://db.slickbox.net/venues.php", wait_onload_event=True)
ghost.set_field_value("input[name=vname]", "....")
# Any other values
page.fire_on('form', 'submit')
page, resources = ghost.wait_for_page_loaded()
content = session.content # or page.content I forgot which
After you can use beautifulsoup to parse the HTML or Ghost may have some rudimentary utilities to do that.
i need to send raw multipart data with a php POST but without an html form... im starting the process with jquery $.post() instead (the objective is to change a twitter account's background).
How can i achieve that? This is my current (and still incomplete) code:
1) Image filename is inserted in this hidden input field:
<input type="hidden" id="profile_background_image_url" value="oats.jpg" />
2) when clicking on the submit button, a javascript function is triggered... and it calls:
$.post('helper.php',{
profile_background_image_url:$('#profile_background_image_url').val()
});
3) helper.php has
$param = array();
$param['image'] = '/www/uploads/'.$_POST['profile_use_background_image'];
$status = $connection->post('account/update_profile_background_image',$param);
Notes:
all the background files are inside the /www/uploads local directory.
im using Abraham Williams' twitteroauth library 0.2
Bottom line, in step three i need to send $param['image'] in raw multipart data to the $connection object (twitter library).
Any ideas?
Some references: http://dev.twitter.com/doc/post/account/update_profile_background_image
Yeah i see now that hes building the post fields array into a query string which means you have to manually set the content type and that the # key in the image fields wont do its magic since that only works with an array argument. More importantly i dont see a way to modify the headers without hacking the library or extending it and replacing certain functions.
I would try would be prepending # to the file path of the image param like:
$param['image'] = '#/www/uploads/'.$_POST['profile_use_background_image'];
That is the convenient way to do it with cURL, and it looks like the libray basically uses cURL to make the request, so that should work.
solved!
curl_setopt($ci, CURLOPT_POST, TRUE);
if(is_array($files)){
$post_file_array = array();
foreach($files as $key=>$value){
$post_file_array[$key] = "#{$value}";
}
curl_setopt($ci, CURLOPT_POSTFIELDS, $post_file_array);
if (!empty($postfields)) {
$url = "{$url}?{$postfields}";
}
}
else if (!empty($postfields)) {
curl_setopt($ci, CURLOPT_POSTFIELDS, $postfields);
}
I have a website in which I have a several php forms that I would like to fill out with auto generated content (for the purposes of exercising different content the user could submit). I would like to write a client side application that enables me to do so.
Is there any way either using webtoolkit, java script etc of doing this?
If you are already familiar with php, why not use php on the "client side" as well? You can use the Client URL package to submit POST data to a web form. Example:
<?php
$ch = curl_init();
$data = array('name' => 'phpnoob', 'address' => 'somewhere');
curl_setopt($ch, CURLOPT_URL, 'http://localhost/url/to/your/php/form.php'); // use the URL that shows up in your <form action="...url..."> tag
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_exec($ch);
?>
It would probably be better, more stable and more efficient to mock a submission by sending data directly into your application. PHPUnit is a great framework for unit testing PHP applications.
But yes, it would be possible to write a client side submission too. You could also write Selenium tests, which use JavaScript to interact with your page.
if you name your form using the id attribute, you can call the javascript function
document.myform.submit();
where myform is the name of that form.
You could have an onload event attached to the body element which would submit the forms automatically.
<body onload="document.form1.submit();document.form2.submit();">
<form id="form1" action="url" method="post">
</form>
<form id="form2" action="url" method="post">
</form>
</body>
Of course this would be completed better using jQuery or another API.
IF you are php programmer then you might not want any javascript answer. the best solution to this is
1.USE A PROXY like paros or HTTP analyser they will give u the insight how site's form is structued
2.notE down the forms POST OR GET VALUE and their SYNTAX FROM THE HTTP analyzer or paros proxy.
read this tutorial it is the best tutorial out there
http://www.html-form-guide.com/php-form/php-form-submit.html
4.change the content of $_post or $_get according to their structure which you have noted down in
paros or HTTP analyser
I'm currently trying to get a script to submit a form to a page that is external to my site but will also e-mail the answers given by the customer to me. The mail() function has worked fine for the mail... but how do I then take these values and also submit them to the external page?
Thanks for your help!
If you get the form to submit to your script, can could first send the email and then use cURL to make a HTTP request to the external page, POSTing the values you want to send. This won't work though if the external site is relying on any cookies the user has, because the request is made from your web server.
e.g.
<?php
//data to post
$data = array( 'name' => 'tom', 'another_form_field'=>'a' );
//external site url (this should be the 'action' of the remote form you are submitting to)
$url = "http://example.com/some/url";
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $data);
//make curl return the content returned rather than printing it straight out
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);
if ($result === false) {
//curl error
}
curl_close($curl);
//this is what the webserver sent back when you submitted the form
echo $result;
You're going to have to dig through the source of the external form to determine the HTML name's of the relevant fields and whether the form is submitted using GET or POST.
If the form uses the GET method, you can easily generate a query-string that follows the same form as the actual form: http://example.com/form.php?name1=value1&name2=value2 ...
If, on the other hand, the form uses the POST method, you'll have to generate a HTTP POST request using something like the cURL library (http://us2.php.net/curl).
You could send a custom HTTP POST request from the script that you're using to send the email. Try fsockopen to establish the connection and then send your own HTTP request containing the data you just received from the form.
Edit:
A bit more specific. There's this example that shows you how to send a simple HTTP POST request. Just seed it with your $_POST array like this:
do_post_request(your_url, $_POST);
and that should do the trick. Afterwards, you could optionally evaluate the response to check whether everything went OK.
For POST, you'll need to set the external page as the processing action:
<form action="http://external-page.com/processor.php" method="POST">
<!-- Form fields go here --->
</form>
If it's GET, you can either change the form method to GET, or create a custom query string:
submit
Edit: I just realized you probably want to send these from within your PHP processing class. In that case, you could use set the location header with the custom query string:
header("Location: http://external-page.com/processor.php?field1=value1&field2=value2");
I have a form on my site which sends data to some remote site - simple html form.
What I want to do is to use data user enters into form for statistical purposes.
So I instead of sending data to the remote page I send it first to my script which resends it the remote site.
The thing is I need it to behave in exact way the usual form would behave taking user to the remote site and displaying resources.
When I use this code it kinda works but not in the way I want it to:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $action);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close($ch);
Problem is that it displays response in the same script. For example if $action is for example:
somesite.com/processform.php and my script name is mysqcript.php it would display the response of "somesite.com/processform.php" inside "mysqcript.php" so all the relative links are not working.
How do I make it to send the user to "somesite.com/processform.php"? Same thing that pressing the button would do?
Leonti
I think you will have to do this on your end, as translating relative paths is the client's job. It should be simple: Just take the base directory of the request you made
http://otherdomain.com/my/request/path.php
and add it in front of every outgoing link that does not begin with "/" or a protocol ("http://", "ftp://").
Detecting all the outgoing links is hard, but I am 100% sure there are ready-made PHP classes that do that. Check for example this article and the getLinks() function in the user comments. I am not 100% sure whether this is what you need but it certainly goes to the right direction.
Here are a couple of possible solutions, which I post separately so they don't get mixed up with the one I recommend:
1 - keep using cURL, parse the response and add a <base/> tag to it. It should work for pretty much everything on that page.
<base href="http://realsite.com/form_url.php" />
2 - do not alter the submit URL. Submit the form to the real URL, but capture its content using some Javascript library (YUI does that) and send it to your script via XHR. It's still kind of hacky though.
There are several ways to do that. Here's one of the easiest: just use a 307 redirect.
header('Location: http://realsite.com/form_url.php', true, 307');
You can do your logging and stuff either before or after header() but if you do it after calling header() you will need to start your script with
ignore_user_abort(true);
Note that browsers are supposed to notify the user that their form is being redirected.