i need to know if is possible to load an html page and submit the form inside this page using php. so something like:
<?php
$html = fopen("http://www.mysite.com","r");
//get position of form...
...
//submit it
?>
is possible? can someone help me? thanks!!!
EDIT:
i have to submit this form
https://annunci.ebay.it/pubblica-annuncio
my problem is that in this page there is an image upload and i don't know how to do that using php( scraping it )
You can also use curl to POST to any URL, for instance the form's action url.
$ch = curl_init('http://example.com/form_action.php');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('your' => 'data', 'goes' => 'here');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
This will call the URL http://example.com/form_action.php as if it was called from a form with 'your' set to value 'data' and 'goes' set to value 'here'.
To find out the URL you need to POST, you can inspect source code. When doing that, check the "name" atribute on the <input> tags you want to send.
EDIT: If the POST url and the fields can change, you should check #Adan's answer.
Basically this is what you need to do
1- Get the content of the HTML page using file_get_contents() (bearing in mind the security risks)
2- Parse the HTML using DOMDocument
3- Get the form's attributes, most importantly (ACTION, METHOD) using DOMDocument
4- Get the form's fields' names using DOMDocument
5- Then send to the ACTION url using the method METHOD a request with the data you want replacing the fields using cURL
you can use curl for getting page in php. as mentioned in answer #Lumbendil. For parsing the HTML you can use libraries like
http://simplehtmldom.sourceforge.net/
Or you can use
http://code.google.com/p/phpquery/
As another option, which would be more clean, you could use the eBay API. It provides methods to add new items, and it probably has already built libraries for php, such as the PHP Accelerator toolkit for eBay.
I am providing a code that I got from net to get the contents of a page. After that you can use jquery(maybe) to force the submit function.
$url = "URL OF YOUR PAGE"; // I have tested page from same server
$lines = file( $url );
foreach( $lines as $line_num => $line ) {
$line = htmlspecialchars( $line );
$line = str_replace( "<", '<span><', $line );
$line = str_replace( ">", '></span>', $line );
$line = str_replace( "<!–", '<em><!–', $line );
$line = str_replace( "–>", '–></em>', $line );
echo "<span class=\"linenumber\">Line <strong>$line_num </strong></span> : " . $line . "<br/>\n";
}
The above code gave me contents from another page on same server. Now you have to find a way around to check if a form exist and then ; force submit that form.
Related
I want to fill a database table with certain items from the Steam Marketplace, specifically at the moment, guns from CSGO. I can't seem to find any database or list already of all the gun names, skin names and skin qualities, which is what I want.
One way I thought of to do it is to get to the list of items I want, EG "Shotguns", and save each item on the page into the database, and go through each page of that search. EG:
http://steamcommunity.com/market/search?appid=730&q=shotgun#p1_default_desc
http://steamcommunity.com/market/search?appid=730&q=shotgun#p2_default_desc
Ect..
Firstly, I'm not exactly sure how I would do that, and secondly, I wanted to know if there would be an easier way.
I plan on using the names of items to later get the prices by substituting the names into this: http://steamcommunity.com/market/priceoverview/?currency=3&appid=730&market_hash_name=StatTrak%E2%84%A2%20P250%20%7C%20Steel%20Disruption%20%28Factory%20New%29
And updating the prices every hour or so by running that check for every item. (probably at least a few thousand..)
The general gist of what you need to do boils down to:
Identify the urls you need to parse. In your case you'll notice that the results are loaded via ajax. Right-click the page, click 'inspect element' and go to the network tab. You'll see that the actual url is: http://steamcommunity.com/market/search/render/?query=&start=<STARTVALUE>&count=<NUMBEROFRESULTS>&search_descriptions=0&sort_column=quantity&sort_dir=desc&appid=730&category_730_ItemSet%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Type%5B%5D=tag_CSGO_Type_Pistol&category_730_Type%5B%5D=tag_CSGO_Type_SMG&category_730_Type%5B%5D=tag_CSGO_Type_Rifle&category_730_Type%5B%5D=tag_CSGO_Type_SniperRifle&category_730_Type%5B%5D=tag_CSGO_Type_Shotgun&category_730_Type%5B%5D=tag_CSGO_Type_Machinegun&category_730_Type%5B%5D=tag_CSGO_Type_Knife
Identify what the response type is. In this case it is json, and the data we want is inside a html-snippet
Find the framework required to parse it. You can use json_decode(...) to decode the json string. This question will give more information how to parse html.
You can now feed these urls to a function that loads the page. You can use file_get_contents(...) or the curl library.
Enter the values you parse from the response into your database. Make sure that the script does not get killed when it runs for too long. This question will give you more information about that.
You can use the following as a framework. You'll have to figure the structure of the html yourself, and lookup a tutorial of the html parser and mysql library you want to use.
<?php
//Prevent this script from being killed. Please note that if this script never
//ends, you'll have to kill it manually
set_time_limit( 0 );
//The api does not allow for more than 100 results at a time
$start = 0;
$count = 100;
$maxresults = PHP_INT_MAX;
$baseurl = "http://steamcommunity.com/market/search/render/?query=&start=$1&count=$2&search_descriptions=0&sort_column=quantity&sort_dir=desc&appid=730&category_730_ItemSet%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Type%5B%5D=tag_CSGO_Type_Pistol&category_730_Type%5B%5D=tag_CSGO_Type_SMG&category_730_Type%5B%5D=tag_CSGO_Type_Rifle&category_730_Type%5B%5D=tag_CSGO_Type_SniperRifle&category_730_Type%5B%5D=tag_CSGO_Type_Shotgun&category_730_Type%5B%5D=tag_CSGO_Type_Machinegun&category_730_Type%5B%5D=tag_CSGO_Type_Knife";
while( $start < $maxresults ) {
//Constructing the next url
$url = str_replace( "$1", $start, $baseurl );
$url = str_replace( "$2", $count, $url );
//Doing the request
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
$result = json_decode( curl_exec( $ch ), TRUE );
curl_close( $ch );
//Doing things with the result
//
//First let's see if everything went according to plan
if( $result == NULL || $result["success"] !== TRUE ) {
echo "Something went horribly wrong. Please edit the script to take this error into account and rerun it.";
exit( -1 );
}
//Bookkeeping for the next url we have to fetch
$count = $result["pagesize"];
$start += $count;
$maxresults = $result["total_count"];
//This is the html we have to parse
$html = $result["results_html"];
//Look up an example how to parse html, and how to get data from it
//Look up how to make a database connection and how to insert data into
//your database
}
echo "And we are done!";
So checked via a phpinfo() and Safe Mode on my server is off, Curl is activated and there are no reasons for it not to work.
I also made sure Sharrre.php is in my root directory. Even included the Curlurl to the php file. Tried both absolute and relative linking. The google button with the counter shows as soon it is uploaded but not as expected because the counter shows 0 the entire time.
The culprit seems to be: $json = array('url'=>'','count'=>0);
After a few lines of other code we got this:
if(filter_var($_GET['url'], FILTER_VALIDATE_URL)){
if($type == 'googlePlus'){ //source http://www.helmutgranda.com/2011/11/01/get-a-url-google-count-via-php/
$contents = parse('https://clients6.google.com/rpc?key=AIzaSyCKSbrvQasunBoV16zDH9R33D88CeLr9gQurl=' . $url . '&count=true');
preg_match( '/window\.__SSR = {c: ([\d]+)/', $contents, $matches );
if(isset($matches[0])){
$json['count'] = (int)str_replace('window.__SSR = {c: ', '', $matches[0]);
}
}
So either the google url code is not valid anymore or... well maybe there is something wrong with the suspected culprit because:
when changed to a value higher than 0 $json = array('url'=>'','count'=>15);
It shows 15 counts as you can see. I want it to be dynamic though and get the counts I already have and update those per click.
What can be done to solve this?
In my particular case the problem was in the asignement of the URL to the Curl object.
The original script sharrre.php sets the URL by asigning it to an array element of the curl object, but this is not working and causes Google counter not retrieve any amount.
Instead, the URL must be asigned by the curl_setopt() function.
This resolved this problem in my case:
sharrre.php:
//...
$ch = curl_init();
//$options[CURLOPT_URL] = $encUrl; // <<<--- not working! comment this line.
curl_setopt_array($ch, $options);
curl_setopt($ch, CURLOPT_URL, $encUrl ); // <<<--- Yeeaa, working! Add this line.
//...
Hope this help.
I'm a newbie with drupal.
I was trying to send data from drupal to node.js function and save the data from node into mongo.
I wrote a function in php
function sample_working_submit($form, &$form_state) { //Function to connect with node and save into Mongo using node
$Name = check_plain($form_state['values']['Name']);
$Age = check_plain($form_state['values']['Age']);
$request_url = "http://10.20.5.112:3001/save/$Name/$Age ";
$response = drupal_http_request($request_url);
}
This is working as long as there is no 'space' between the names(input). How can save the input with spaces.Why does this issue came?
How can i rewrite the function as post?
<?php
$url = http://localhost:8080/myservlet;
$data = array (
'name' => check_plain($form_state['values']['Name']),
'age' => check_plain($form_state['values']['Age'])
);
$response = drupal_http_request($url, $headers, 'POST', json_encode($data));
?>
This would POST data to URL. Note that you need to change logic in server side as well to receive POST data instead of GET data.
Refer here for more info
If the space between the names is the issue, try using urlencode().
The code will be something like:
$Name = urlencode(check_plain($form_state['values']['Name']));
For POST requests I use SimpleBrowser instead of drupal_http_request(). It's easy to use and you'll be able to pass strings in any form.
I want to fill a form field and submit it using CURL.The webpage has many form fields which are already filled so i do not want to touch these fields.
So is it possible to use curl and fill only the required field and submit it along with all other fields which are already filled ?
A full walkthrough (given the supplied HTML scrape text)
To customize you would simply add field names to the $fields_i_want array to specify all of the text field values you wanted to pull from the downloaded source text and change the URLs for retrieval and submission locations.
Also, a better alternative to file_get_contents() is curl. You can use the instructions on this SO post for how to retrieve remote text via curl.
// First, retrieve the remote source using file_get_contents()
$str = file_get_contents('http://www.example.com/');
$my_field_val = 'my_field_value';
$fields_i_want = array('audien', 'unifor');
$field_vals = array();
$field_string = '';
// Use DOM to parse the values you want from the form
$dom = new DOMDocument;
$dom->loadHTML($str);
// Get all the input field nodes
$inputs = $dom->getElementsByTagName('input');
// Iterate over the input fields and save the values we want to an array
foreach ($inputs as $input) {
$name = $input->getAttribute('name');
if (in_array($name, $fields_i_want)) {
$val = $input->getAttribute('value');
$field_vals[$name] = $val;
}
}
// append the field value we set ourselves to the list
$field_vals['my_field'] = $my_field_val;
foreach ($field_vals as $key => $val) {
$field_vals[$key] = urlencode($val)
}
// url-ify the data for the POST
foreach($fields as $key=>$value) {
$fields_string .= $key.'='.$value.'&';
}
rtrim($fields_string, '&');
// open connection
$ch = curl_init();
// POST the data to the form submission url
$submit_url = 'http://www.submitform.com';
// set the url, number of POST vars, POST data
curl_setopt($ch, CURLOPT_URL, $submit_url);
curl_setopt($ch, CURLOPT_POST, count($fields));
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields_string);
// execute post (returns TRUE on success or FALSE on failure)
$result = curl_exec($ch);
// close connection
curl_close($ch);
Depends how the form is initially populated.
If the pre-populated elements all use value="foo" then just grab the page (using curl), load it up in to a DOMDocument, fetch the <form> you're after and populate the field(s) you need, then pass it off as another request using a new cURL request (taking in to account the form's action and method attributes, as well as form data being sent off).
However, if they're populated with JS and you don't plan on writing the cURL request to mimic what you're doing on your browser, I don't see an easy way of mimicking JS actions, then populating, then sending it off.
Also, this doesn't take in to account any cookies that may be present. If you need those, you're going to have to store them from the first request and make sure to send them off in the actual submission call.
Not sure if it fits, but in order to check what are the required fields you need to parse the HTML and check for each input file and look for any required mention.
To parse the HTML, you can use some tools like:
http://framework.zend.com/manual/en/zend.dom.query.html
http://simplehtmldom.sourceforge.net/
With that tools, you can open the page, look for any required tags on the fields, and then decide with form field to submit.
Can i send for example a string or another piece of information to another .php file without it being exposed [thus not by GET but by POST conform to what i know] without using a form?
If you don't want your data to be seen by the user, use a PHP session.
Data in a post request is still accessible (and manipulable) by the user.
Checkout this tutorial on PHP Sessions.
You could use AJAX to send a POST request if you don't want forms.
Using jquery $.post method it is pretty simple:
$.post('/foo.php', { key1: 'value1', key2: 'value2' }, function(result) {
alert('successfully posted key1=value1&key2=value2 to foo.php');
});
Send your data with SESSION rather than post.
session_start();
$_SESSION['foo'] = "bar";
On the page where you recieve the request, if you absolutely need POST data (some weird logic), you can do this somwhere at the beginning:
$_POST['foo'] = $_SESSION['foo'];
The post data will be valid just the same as if it was sent with POST.
Then destroy the session (or just unset the fields if you need the session for other purposes).
It is important to destroy a session or unset the fields, because unlike POST, SESSION will remain valid until you explicitely destroy it or until the end of browser session. If you don't do it, you can observe some strange results. For example: you use sesson for filtering some data. The user switches the filter on and gets filtered data. After a while, he returns to the page and expects the filter to be reset, but it's not: he still sees filtered data.
Simply use: file_get_contents()
// building array of variables
$content = http_build_query(array(
'username' => 'value',
'password' => 'value'
));
// creating the context change POST to GET if that is relevant
$context = stream_context_create(array(
'http' => array(
'method' => 'POST',
'content' => $content, )));
$result = file_get_contents('http://www.example.com/page.php', null, $context);
//dumping the reuslt
var_dump($result);
Reference: my answer to a similar question:
have a look at the php documentation for theese functions you can send post reqeust using them.
fsockopen()
fputs()
or simply use a class like Zend_Http_Client which is also based on socket-conenctions.
also found a neat example using google...
function redir(data) {
document.getElementById('redirect').innerHTML = '<form style="display:none;" position="absolute" method="post" action="location.php"><input id="redirbtn" type="submit" name="value" value=' + data + '></form>';
document.getElementById('redirbtn').click();
}
<button onclick="redir('dataToBeSent');">Next Page</button>
<div id="redirect"></div>
You can use this method which creates a new hidden form whose "data" is sent by "post" to "location.php" when a button[Next Page] is clicked.
I would highly recommend using curl in such situation, file_get_content() does work but not at all times, and it could be troublesome to use it in some applications.
Though curl comes in different variations depending on what you want to send and in what method, here is the most common method of posting your data without HTML form using curl.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://example.com/request_uri');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
$post = array(
'data1' => 'value1',
'data2' => $value2
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close($ch);
//do what you want with the responce
var_dump($result)