getting results from a site using curl - php

I have been trying to post some variables to a site using POST method, using curl to get some results. I am posting to this link.
http://www.rasta.pk/Lhr/Lhr_Traffic.aspx
At this page you will see a drop down menu .. onchange some values are returned in "Yellow" colored table.
I have monitored this site and trying to get those results by making a post request to that link. But, I am getting "Bad Header" error. I have tried changing things but ubable to find a solution.
Here is my code:
"Canal Bank Rd",
"ScriptManager1 " => "UpdatePanelDDLRoads|DropDownRoads",
"__EVENTARGUMENT" => "",
"__EVENTTARGET" => "DropDownRoads"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headerz);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://www.rasta.pk/Lhr/Lhr_Traffic.aspx");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
?>

There's too little details presented for us to know for sure.
A guess is that you're doing the wrong kind of post, since when you pass in a hash array to CURLOPT_POSTFIELDS it will do a multipart formpost which might not be what the server expects. Pass in a string instead to make a "normal" POST.
If that is not enough, use LiveHTTPHeaders or similar in a browser to figure out exactly what is sent in a "manual" session and then you make sure that your curl program mimics that operation as closely as possible.

Related

How to make a call to .aspx https from php script from my localhost with xamp?

I am trying to send SMS from my localhost with xamp installed.
Requested page is on https and an .aspx page.
I am getting error: "HTTP Error 400. The request is badly formed." or blank page only in some cases.
Detaisl is as follows :
$url = 'https://www.ismartsms.net/iBulkSMS/HttpWS/SMSDynamicAPI.aspx';
$postArgs = 'UserId='.$username.
'&Password='.$password.
'&MobileNo='.$destination.
'&Message='.$text.
'&PushDateTime='.$PushDateTime.
'&Lang='.$Lang;
function getSslPage($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$response = getSslPage($all);
echo "<pre>";
print_r($response); exit;
I tried every possible solution/combination found on internet but could not resolve that. The API developers do not have a example for php script.
I tried httpful php library and file_get_contents function but getting empty page. Also tried every combination with curl_setup.
I need to call this url without any post data and see the response from it.
Instead getting a blank page.
Please note that when I execute the url with all details in browser it works fine.
Can anybody help me in this regard.
Thank you,
Usman
First do urlencode over your data as follows:
$postArgs = 'UserId='. urlencode($username.
'&Password='.urlencode($password).
'&MobileNo='.urlencode($destination).
'&Message='.urlencode($text).
'&PushDateTime='.urlencode($PushDateTime).
'&Lang='.urlencode($Lang);
After that two possible solutions. One is using GET.
curl_setopt($ch, CURLOPT_URL, $url . "?" . $postArgs);
Second option is using POST method.
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postArgs);

Using curl to bring search results from external site

I have 2 sites, one main, one external. On the main site, I am using Lucene to search through it. The problem is, I am trying to also search through the external site.
The Form action for the external site:
<form action="https://secure.bcchf.ca/SuperheroPages/searchResults.cfm?Event=WOT" method="post" name="search_tribute" >
I've tried to use curl, but it only brings up the search form without actually doing the search (the field is empty as well).
<?php
$ch = curl_init("https://secure.bcchf.ca/SuperheroPages/searchResults.cfm?Event=WOT");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, tname='hello');
$output = curl_exec($ch);
echo $output;
curl_close($ch);
?>
Any tips?
I don't have access to the form action since it's on an external site. All i have is a form that links to it when I submit it.
<?php
$ch = curl_init("https://secure.bcchf.ca/SuperheroPages/searchResults.cfm?Event=WOT");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array("teamName" => "hello", "searchType" => "team"));
$output = curl_exec($ch);
echo $output;
curl_close($ch);
?>
Can you try this?
I'm pretty sure it's supposed to be teamName instead of tName
Most search engine use GET and not POST .. you can try
// asumption
$_POST['search'] = "hello";
// Return goole Search Result
echo curlGoogle($_POST['search']);
function curlGoogle($keyword) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.google.com/search?hl=en&q=' . urlencode($keyword) . '&btnG=Google+Search&meta=');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FILETIME, true);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Or if you want post then
curl_setopt($ch, CURLOPT_POSTFIELDS, array("search"=>"hello"));
Your php code is not valid syntax, it does not compile.
So if this is really what you have, your problem is that your file generates a fatal error.
That being said, this question is hard to answer since we don't know the site you want to grab your search results from.
Try modifying your line like this:
curl_setopt($ch, CURLOPT_POSTFIELDS, "search=hello");
or alternatively
curl_setopt($ch, CURLOPT_POSTFIELDS, array("search" => "hello");
Maby it will work, however it may be that more post data is required or that the element name is not correct.
You have to look at the form or try making a request and look at it with chromes developer tools or firebug.
Also there are a number of ways for external sites to prevent what you are doing, altough evertything can be worked around somehow.
Assuming that is not the case, I hope i could help you.
Try just putting it into an array.
as that will be the variable the $_POST checks on the other side
and just checked your link, its teamName for the field
$fields = array("teamName"=>"julia");
Then..
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
So your complete code is...
<?php
$ch = curl_init("https://secure.bcchf.ca/SuperheroPages/searchResults.cfm?Event=WOT");
$fields = array("teamName"=>"julia");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
$output = curl_exec($ch);
var_dump($output);
curl_close($ch);
?>

Can't see XML response from PHP cURL api Call

I'm using a remote API with cURL and querystrings. The issue I'm having is that I can't see the data being returned to me (which is in XML format). To start, here's my code:
$url = 'https://api.somehost.com/GetRequest.do';
$query = 'User=JohnDoe&Password=ABC123&Function=auth';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$curlresponse = curl_exec($ch);
I know the cURL request is working because if I change the password to something incorrect, I get a response back (can see it using print_r($curlresponse) telling me the password is incorrect.
However, when I enter the correct password, I see nothing when I print_r($curlresponse) yet I know I'm receiving the data because a print_r(curl_getinfo($ch)) shows a larghe download_content_length which is the data I'm expecting. The data I'm expecting back is in XML format. I'm just not sure what I'm missing.
Thank you!

JIRA Update custom fields using REST, PHP and cURL

I am trying to update some custom fields using the REST API and PHP/cURL.
I'm wondering if I might have edited something without realizing it, while what I have below "worked" yesterday (I think), it does not work now.
I get varying responses using the different "methods", from:
I get this one using the POST method, as it is uncommented below.
HTTP 405 - The specified HTTP method is not allowed for the requested
resource ().
I get this one if I use the commented-out PUT method, with POST commented out.
{"status-code":500,"message":"Read timed out"}
And this one mixing and matching PUT and POST.
{"errorMessages":["No content to map to Object due to end of input"]}
What am I missing/doing wrong? I am using the following code:
<?php
$username = 'username';
$password = 'password';
$url = "https://example.com/rest/api/2/issue/PROJ-827";
$ch = curl_init();
$headers = array(
'Accept: application/json',
'Content-Type: application/json'
);
$test = "This is the content of the custom field.";
$data = <<<JSON
{
"fields": {
"customfield_11334" : ["$test"]
}
}
JSON;
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
// Also tried, with the above two lines commented out...
// curl_setopt($ch, CURLOPT_PUT, 1);
// curl_setopt($ch, CURLOPT_INFILE, $data);
// curl_setopt($ch, CURLOPT_INFILESIZE, strlen($data));
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
$result = curl_exec($ch);
$ch_error = curl_error($ch);
if ($ch_error) {
echo "cURL Error: $ch_error";
} else {
echo $result;
}
curl_close($ch);
?>
The problem here is that PHP's cURL API is not particularly intuitive.
You might think that because a POST request body is sent using the following option
that a PUT request would be done the same way:
// works for sending a POST request
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
// DOES NOT work to send a PUT request
curl_setopt($ch, CURLOPT_PUT, 1);
curl_setopt($ch, CURLOPT_PUTFIELDS, $data);
Instead, to send a PUT request (with associated body data), you need the following:
// The correct way to send a PUT request
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
Note that even though you're sending a PUT request, you still have to use the CURLOPT_POSTFIELDS
option to send your PUT request body. It's a confusing and inconsistent process, but it's what you've
got if you want to use the PHP cURL bindings.
According to the relevant manual entrydocs, the CURLOPT_PUT option seems to only work for PUTting a file directly:
TRUE to HTTP PUT a file. The file to PUT must be set with CURLOPT_INFILE and CURLOPT_INFILESIZE.
A better option IMHO is to use a custom stream wrapper for HTTP client operations. This carries the
added benefit of not making your application reliant on the underlying libcurl library. Such an
implementation is beyond the scope of this question, though. Google is your friend if you're interested
in developing a stream wrapper solution.

Using CURL with Google

I want to CURL to Google to see how many results it returns for a certain search.
I've tried this:
$url = "http://www.google.com/search?q=".$strSearch."&hl=en&start=0&sa=N";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
$response = curl_exec($ch);
curl_close($ch);
But it just returns a 405 Method Allowed google error.
Any ideas?
Thanks
Use a GET request instead of a POST request. That is, get rid of
curl_setopt($ch, CURLOPT_POST, true);
Or even better, use their well defined search API instead of screen-scraping.
Scrapping Google is a very easy thing to do. However, if you don't require more than the first 30 results, then the search API is preferable (as others have suggested). Otherwise, here's some sample code. I've ripped this out of a couple of classes that I'm using so it might not be totally functional as is, but you should get the idea.
function queryToUrl($query, $start=null, $perPage=100, $country="US") {
return "http://www.google.com/search?" . $this->_helpers->url->buildQuery(array(
// Query
"q" => urlencode($query),
// Country (geolocation presumably)
"gl" => $country,
// Start offset
"start" => $start,
// Number of result to a page
"num" => $perPage
), true);
}
// Find first 100 result for "pizza" in Canada
$ch = curl_init(queryToUrl("pizza", 0, 100, "CA"));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $this->getUserAgent(/*$proxyIp*/));
curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
Note: $this->_helpers->url->buildQuery() is identical to http_build_query except that it will drop empty parameters.
Use the Google Ajax API.
http://code.google.com/apis/ajaxsearch/
See this thread for how to get the number of results. While it refers to c# libraries, it might give you some pointers.
Before scrapping data please read https://support.google.com/websearch/answer/86640?rd=1
Against google terms
Automated traffic includes:
Sending searches from a robot, computer program, automated service, or search scraper
Using software that sends searches to Google to see how a website or webpage ranks on Google
CURLOPT_CUSTOMREQUEST => ($post)? "POST" : "GET"

Categories