I'm trying to retrieve data from www.dotabuff.com for a list of steam ID, but I only get in return "We couldn't find what you are looking for!".
The goal is to go to the website, search a player with his steam ID and extract his win rate.
(This is intended for very small lists, which will not bother dotabuff).
Here is my code (example with a static steam ID):
//create array of data to be posted
$post_data['utf8'] = '✓';
$post_data['q'] = '76561198055615656';
$post_data['commit'] = 'Search';
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection
$curl_connection =
curl_init('https://www.dotabuff.com//');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
echo $result;
//close the connection
curl_close($curl_connection);
You are using POST with curl to do the search, but looking at the Dotabuff website the search form is using GET, so I suspect the website is looking for GET vars instead of POST vars. I haven't tested this but I would suggest doing a curl request to https://www.dotabuff.com/search?q=76561198055615656 (without the POSTFIELDS) and see what that gives you.
Init curl without the URL, or assemble a GET version of the URL and post it to the site.
For example, if you want info on user 'USH!' (I reandomly selected one). the URL you'd synthesize would look like this:
https://www.dotabuff.com/search?utf8=%E2%9C%93&q=USH%21&button=
I's suggest using urlencode to convert the player ID to a GET URI safe string
ex:
$safe_player_id = urlencode($player_id)
$URI = sprintf("https://www.dotabuff.com/search?q=%s&button=",$safe_player_id);
Or, if you know their player ID code (int) then this would work to get the page of stats (this works):
https://www.dotabuff.com/players/76561198055615656
Then you just need to parse the page for what you want.
Related
This question already has answers here:
Using curl to submit/retrieve a forms results
(2 answers)
Closed 8 years ago.
The form on the website is...
<form action="" method="POST">
<input style="width:30%;background-color:#e2e2e2;border:#000;color:#000;" type="text" name="userName" placeholder="Enter a username" required="">
<br>
<input type="submit" name="userBtn" value="get Username">
</form>
Once you fill out a value in name="userName" and click name="userBtn" the page refreshes and changes the value of name="userName" to the information that I want.
How would I go about submitting a form and then retrieving the data that it writes?
See this detailed example, Begin by creating a new connection.
$curl_connection =
curl_init('http://www.domainname.com/target_url.php');
A new connection is created using curl_init() function, which takes the target URL as parameter (The URL where we want to post our data). The target URL is same as the “action” parameters of a normal form, which would look like this:
<form method="post" action="http://www.domainname.com/target_url.php">
Now let’s set some options for our connection. We can do this using the curl_setopt() function. Go to curl_setopt() reference page for more information on curl_setopt() and a complete list of options.
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
What options do we set here?
First, we set the connection timeout to 30 seconds, so we don’t have our script waiting indefinitely if the remote server fails to respond.
Then we set how cURL will identify itself to the remote server. Some servers will return different content for different browsers (or agents, such as spiders of the search engines), so we want our request to look like it is coming from a popular browser.
CURLOPT_RETURNTRANSFER set to true forces cURL not to display the output of the request, but return it as a string.
Then we set CURLOPT_SSL_VERIFYPEER option to false, so the request will not trigger an error in case of an invalid, expired or not signed SSL certificate.
Finally, we set CURLOPT_FOLLOWLOCATION to 1 to instruct cURL to follow “Location: ” redirects found in the headers sent by the remote site.
Now we must prepare the data that we want to post. We can first store this in an array, with the key of an element being the same as the input name of a regular form, and the value being the value that we want to post for that field.
For example,if in a regular form we would have:
<input type="text" name="firstName" value="Name">
<input type="hidden" name="action" value="Register">
we add this to our array like this:
$post_data['firstName'] = 'Name';
$post_data['action'] = 'Register'
Do the same for every form field.
Data will be posted in the following format:
key1=value1&key2=value2
In order to format the data like this, we are going to create strings for each key-value pair (for example key1=value1), put them in another array ($post_items) then combine them in one string using PHP function implode() .
foreach ( $post_data as $key => $value)
{
$post_items[] = $key . '=' . $value;
}
$post_string = implode ('&', $post_items);
Next, we need to tell cURL which string we want to post. For this, we use the CURLOPT_POSTFIELDS option.
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
Finally, we execute the post, then close the connection.
$result = curl_exec($curl_connection);
curl_close($curl_connection);
By now, the data should have been posted to the remote URL. Go check this, and if it did not work properly, use curl_getinfo() function to see any errors that might have occurred.
print_r(curl_getinfo($curl_connection));
This line displays an array of information regarding the transfer. This must be used before closing the connection with curl_close();
You can also see number and description of the error by outputting curl_errno($curl_connection) and curl_error($curl_connection).
So let’s put everything together. Here is our code:
<?php
//create array of data to be posted
$post_data['firstName'] = 'Name';
$post_data['action'] = 'Register';
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection
$curl_connection =
curl_init('http://www.domainname.com/target_url.php');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
print_r(curl_getinfo($curl_connection));
echo curl_errno($curl_connection) . '-' .
curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
?>
I want to auto fill an html form and submit the form and display the result. I used below code got from here.
<?php
//create array of data to be posted
$post_data['email'] = 'myemail';
$post_data['pass'] = 'mypassword';
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection
$curl_connection =
curl_init('http://m.facebook.com/');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
print $result;
//show information regarding the request
echo curl_errno($curl_connection) . '-' .
curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
?>
and I used Facebook mobile site and gmail to test this code.
I placed the url of login page of those sites in curl_init function, gave the value of name attributes of username and password fields of login page into keys of $post_data array and saved this code as my.php file and placed it in xampp htdocs directory in local machine.
when I browse the my.php, it displays the login page with username field is filled and password field is not filled. according to the code, The expected result is, It should return the successfully logged page because I have provided the correct username and password. also the curl_errno returns 0. that means no error occurred. Then why I can't get the expected result? and Why password field is not filled although username field is filled?
Inspecting the code at http://m.facebook.com/ i see there are some hidden fields that you may (should) try to send. Usually those are there to prevent automated POST.
First get http://m.facebook.com/ and get the hidden fields using some DOM parser and build the query to post them to the action url.
I am collecting form posts via jQuery via the.val() method, validating those posts and passing errors back to the form inputs, or passing true to the .$post method which invokes a PHP cURL script.
A typical var will look like this:
var industs_servedVal = $('#industs_served').val();
In this case it is a select multiple form field. I understand that the .val() method in jQuery passes an array, so that seems reasonable, and am I right in saying that the var will also collect the array.
I then pass var industs_servedVal to the $.post method like this ( then slide up a thank you note):
$.post('../inc/form_sendsf_modified.php', {
othervars: othervarsVal;
industs_served: industs_servedVal,
}, function(data) {
$('#sendEmail').slideUp('slow', 'swing', function() {
$('#sendEmail').replaceWith('<h3>Thank you!</h3><p>Your message was sent to us. We\'ll get back to you as soon as we can.</p>');
});
});
}
return false;
});
The file "form_sendSF_modified.php" handles those posts and sends to the Sales Force Cloud using cURL. This works; however, the problem is that Sales Force shows "array" as the values received for the multiple field array, not the array values themselves. Is there a problem in the way I am collecting the array and passing it to sales force. Is the foreach loop capable of sending the multiple field array values as well as the other values as an array as shown in the code.
$post_data['00N70000002U2fA'] = $_POST['industs_served']; //Array
//$otherpost data
//cURL CODE for post
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection to SFDC
$curl_connection = curl_init('https://www.salesforce.com/servlet/servlet.WebToLead?encoding=UTF-8');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
//print_r(curl_getinfo($curl_connection));
//echo curl_errno($curl_connection) . '-' .
curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
//End cURL
You can use array itself (but it will change Content-Type header to multipart/form-data).
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_data);
Or you can use build string with http_build_query function, look example #3.
use this
$post = "ac=on&p=1&pr[]=0&pr[]=1&a[]=3&a[]=4&pl=on&sp[]=3&ct[]=3&s=1&o=0&pp=3&sortBy=date";
parse_str($post,$fields);
$url = 'http://example.com/';
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);
)
i want the smallest possible script to send current URL via email without a send button
but as a result of an if { } else {} condition..
ideas anyone?
ok.. i have gotten so far.. got a script on one end reciving and mailing
the desired data to the desired email.. then i need a way to send the current
page URL without a send button but as a result of an if (!notsomething) {} else{ send!!}
so i have this so far AND it works!! but i am getting partial URL
instead of "http://www.example.com/subfile.php" i am getting just "/subfile.php"
$websiteurl = $_SERVER['REQUEST_URI'];
$curl_connection = curl_init('http://www.sagive.co.il/downloads/wordpress/linkRemovedNotification.php');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $websiteurl);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
print_r(curl_getinfo($curl_connection));
echo curl_errno($curl_connection) . '-' . curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
For some weird reason it wont send when i use:
$websiteurl = $_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
.
SO.. thats the last problem.. ideas anyone?
Look into jQuery for this probably
<script rel="javascript" src="jquery.js"></script>
<script>
$.post('ajax/test.html?url='+window.location.pathname)
</script>
Ok! solved it :) - thanks for your help and your time.
here is the code i used and it works (Collected from sites across the web)
//extract data from the post
$currentw = $_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
$currentth = SpaceJunkie;
//set POST variables
$url = 'http://www.example.com';
$fields = array(
'siteurl'=>urlencode($currentwebsite),
'themename'=>urlencode($currentheme),
);
//url-ify the data for the POST
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string,'&');
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);
I am trying to call the __doPostback javascript function in a asp.net page from php using curl.
I learnt that this can be done by making a post request to the asp.net page with the appropriate parameters.
So in curl,
I make a get request / just use file_get_contents to retrieve the initial page.
From this, I extract the values for __VIEWSTATE and __EVENTVALIDATION.
So far everything seems ok.
Now, I understand that we need to make a post request using cURL with __VIEWSTATE and other parameters required. ( values for the fields present in the asp.net form )
I am unable to construct the CURLOPT_POSTFIELDS correctly.
For instance, I am trying this out,
$postoptions1='__EVENTTARGET='.('ctl00$ContentPlaceHolder1$gRef').'&__EVENTARGUMENT='.('$2');
$postoptions2 = '&__VIEWSTATE='.urlencode($viewState) ;
$otherparams = '&ctl00$ContentPlaceHolder1$ddlName=Abc';
And before using setopt for CURLOPT_POSTFIELDS, I am doing,
urlencode ($postoptions1.$postoptions2.$otherparams)
This does not work. The submit results are not shown, which means, the required parameter __VIEWSTATE was not found in my post request.
If I change the order of the parameters and place __VIEWSTATE as the first parameter, the results page is shown but the other parameter values are not honoured.
I think there is some problem with the way I am encoding the parameters.
Please tell me how to construct the parameters for the post request to a asp.net page.
Thanks.
--Edited--
Here is the complete code:
$resultsPerPage='10';
$url = "www.example.com"; // url changed
$curl_connection = curl_init($url);
function sendCurl($curl_connection,$url,$params,$isPost=false) {
//$post_string = $params;
$post_string = http_build_query($params);
//$post_string = build_query_string($params);
//$post_string = urlencode($params);
echo 'After Encode'.$post_string;
$cookie="/cookie.txt";
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 300);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_HEADER, 0); // don't return headers
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl_connection,CURLOPT_REFERER, $url);
if($isPost) {
curl_setopt ($curl_connection, CURLOPT_POST, true);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
curl_setopt($curl_connection,CURLOPT_COOKIEJAR,$cookie);
}
else {
curl_setopt($curl_connection,CURLOPT_COOKIEFILE,$cookie);
}
$response1 = curl_exec($curl_connection);
if($response1 === false)
{
echo 'Curl error: ' . curl_error($curl_connection);
}
else
{
echo 'Operation completed without any errors';
}
return $response1;
} **// First time, get request to asp.net page
$response1 = sendCurl($curl_connection,$url,'',false);
$viewState=getVStateContent($response1);
$eventValidation =getEventValidationContent($response1);
$simpleParams = '&__VIEWSTATE='.$viewState.'&ctl00$ContentPlaceHolder1$ddlManuf=&ctl00$ContentPlaceHolder1$ddlCrossType=&ctl00$ContentPlaceHolder1$ddlPageSize='.$resultsPerPage.'&ctl00$ContentPlaceHolder1$btnSearch=Search&ctl00_ToolkitScriptManager1_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=';
// Second post - for submitting the search form
$response2= sendCurl($curl_connection,$url,$simpleParams,true);
----**
What you want is http_build_query, which will format an array as proper HTTP parameters.
Edit: To clarify what this should probably look like:
$params = array(
'__EVENTTARGET' => 'ctl00$ContentPlaceHolder1$gRef',
'__EVENTARGUMENT' => '$2',
'__VIEWSTATE' => $viewState,
'ctl00$ContentPlaceHolder1$ddlName' => 'Abc'
);
curl_setopt($curlHandler, CURLOPT_POSTFIELDS, http_build_query($params));
Also, what's ctl00$ContentPlaceHolder1$ddlName supposed to be?
Don't urlencode() the ampersands (&) linking the parameters together, just the keys & values (the stuff on either side of the ampersands).