I have read many similar title questions, but none of them worked for me...
The problem is that when I'm sending an cURL query to a website all I get is a blank page.
Here is my code:
<?php
$action = "http://www.website.com/index.php?section=login&do=process";
$fields = array(
'username' => $user,
'rememberMe' => '1'
);
$login = curl_post($action, $fields);
var_dump($login);
function curl_post($url, array $post = NULL, array $options = array())
{
$defaults = array(
CURLOPT_POST => 1,
CURLOPT_HEADER => 0,
CURLOPT_HTTPHEADER => array('Accept-Language: pl,en-us;q=0.7,en;q=0.3', 'Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7'),
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3",
CURLOPT_URL => $url,
CURLOPT_FRESH_CONNECT => 1,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FORBID_REUSE => 1,
CURLOPT_TIMEOUT => 4,
CURLOPT_NOBODY => false,
CURLOPT_POSTFIELDS => http_build_query($post)
);
$ch = curl_init();
curl_setopt_array($ch, ($options + $defaults));
if( !$result = curl_exec($ch))
{
return(curl_error($ch));
}
curl_close($ch);
return $result;
}
?>
Of course I have my cURL enabled in PHP so its not about that.
If you have any ideas, please share!
Update
I have also been trying adding the following lines at the top of my PHP file:
ini_set("display_errors", 1);
error_reporting(E_ALL);
But the problem still appears - result is a blank page. When I use file_get_contents("http://website.com/"); I can see the page content, so it doesnt work with cURL only.
Running this locally and pointing it at Google, I see two things immediately:
PHP Notice: Undefined variable: user
Google returns a 'Error 405 (Method Not Allowed)!!1' error page but probably because I'm trying to post to it
What happens when you define $user and try again?
Related
I tried to use use the Steam Web API to get Skin Prices for PUBG Skins. Yesterday it worked well but today the request returns nothing. Not even the "Too many Requests" Error. Just nothing. I tried everything.
With "file_get_contents" and with a curl request
function getRequest($url, $refer = "", $timeout = 10)
{
$ssl = stripos($url,'https://') === 0 ? true : false;
$curlObj = curl_init();
$options = [
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_AUTOREFERER => 1,
CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; MSIE 5.01; Windows NT 5.0)',
CURLOPT_TIMEOUT => $timeout,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_0,
CURLOPT_HTTPHEADER => ['Expect:'],
CURLOPT_IPRESOLVE => CURL_IPRESOLVE_V4,
];
if ($refer) {
$options[CURLOPT_REFERER] = $refer;
}
if ($ssl) {
//support https
$options[CURLOPT_SSL_VERIFYHOST] = false;
$options[CURLOPT_SSL_VERIFYPEER] = false;
}
curl_setopt_array($curlObj, $options);
$returnData = curl_exec($curlObj);
if (curl_errno($curlObj)) {
//error message
$returnData = curl_error($curlObj);
}
curl_close($curlObj);
return $returnData;
}
I know that it is really hart to work with the API but somehow it has to work. BTW when I request the URL from my local PC it is working well. Maybe it is something of a IP ban. But shouldn't it return at least a error message?
I am using this code to get the contents of a post request url using php curl
Code looks as below:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://www1.ptt.gov.tr/tr/interaktif/sonuc-yd.php',
CURLOPT_USERAGENT => 'Codular Sample cURL Request',
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array(
'barcode' => 'CP021325078TR',
'security_code' => $capcha2
)
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
echo "<pre>";
var_dump($resp);
echo "</pre>";
The result doesn’t seem to return anything at all.
What is wrong with this code?
Try this:
$url = 'http://www1.ptt.gov.tr/tr/interaktif/sonuc-yd.php';
$postvals = array(
'barcode' => 'CP021325078TR',
'security_code' => $capcha2
);
$resp = Request($url,$postvals);
echo "<pre>"; var_dump($resp); exit;
...
function Request($url,$params=array()){
$ch = curl_init();
$curlOpts = array(
CURLOPT_URL => $url,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true
);
if(!empty($params)){
$curlOpts[CURLOPT_POST] = true;
$curlOpts[CURLOPT_POSTFIELDS] = $params;
}
curl_setopt_array($ch,$curlOpts);
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch); exit;
}
curl_close($ch);
return $answer;
}
EDIT:
I tested this and got:
Could not resolve host: www1.ptt.gov.tr
So make sure you're calling the right endpoint.
Actually you need to set this variable
$captcha2
To use it here -
'security_code' => $capcha2
I am trying to scrape a website using PHP, CURL and POST method in order to submit a form before web scraping the page. The problem I am experiencing is that there is connected with POST method: no data is submitted to the server, so the scraped webpage doesn't contain what I am looking for.
I quit sure the problem is connected with the form type: enctype="multipart/form-data".
How can I manage this POST request, considering that the form is multipart/form-data?
Do I have to encode the post_string in a special way?
Here's the code I'm using:
function curl($url) {
//POST string
$post_string="XXXX";
$options = Array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_AUTOREFERER => TRUE,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8",
CURLOPT_URL => $url,
CURLOPT_CAINFO => dirname(__FILE__)."/cacert.pem",
CURLOPT_POSTFIELDS => $post_string,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
curl_error($ch);
curl_close($ch);
return $data;
}
$scraped_page = curl("XXXURLXXX");
echo $scraped_page;
Thank you!
Set the CURLOPT_POST to true:
CURLOPT_POST = true
Then fill your post fields like this 'setup':
$postfields = array();
$postfields['field1'] = 'value1';
$postfields['field2'] = 'value2';
CURLOPT_POSTFIELDS => $postfields
If value is an array, the Content-Type header will be set to multipart/form-data.
The PHP manual
Yes, $post_string needs to be an array.
Also set CURLOPT_POST to true.
I wish to mimic, using CURL with PHP, the operation of a website that retrieves data using an AJAX POST.
Normally when I'm viewing POST requests using Firebug you will see variable/value pairs, but in this case all you see is a single JSON string. E.g.
{"refId":"14536"}
Is there a way to mimic this request using CURL? I've looked at CURL but as far as I can see the CURLOPT_POSTFIELDS parameter has to be a query string made up of one or more name/value.
Here is my test code with a normal POST request using a single name/value pair. I'd like to modify it to do the above.
$curlOptions = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3",
CURLOPT_CONNECTTIMEOUT => 600, // timeout on connect
CURLOPT_TIMEOUT => 600, // timeout on response
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => 'var1=113',
CURLOPT_URL => "http://localhost/t4.php"
);
$curlCh = curl_init();
curl_setopt_array( $curlCh, $curlOptions );
$fileContents = curl_exec( $curlCh );
$curlErr = curl_errno( $curlCh );
$curlErrmsg = curl_error( $curlCh );
if( $curlErr ) echo "CURL ERROR:</b> $curlErr $curlErrmsg";
echo $fileContents; //check worked
curl_close( $curlCh );
How about something like:
$postData = json_encode(array('refId' => '14536'));
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
I have a class function to interface with the RESTful API for Last.FM - its purpose is to grab the most recent tracks for my user. Here it is:
private static $base_url = 'http://ws.audioscrobbler.com/2.0/';
public static function getTopTracks($options = array())
{
$options = array_merge(array(
'user' => 'bachya',
'period' => NULL,
'api_key' => 'xxxxx...', // obfuscated, obviously
), $options);
$options['method'] = 'user.getTopTracks';
// Initialize cURL request and set parameters
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => self::$base_url,
CURLOPT_POST => TRUE,
CURLOPT_POSTFIELDS => $options,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_TIMEOUT => 30,
CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)'
));
$results = curl_exec($ch);
return $results;
}
This returns "Empty reply from server". I know that some have suggested that this error comes from some fault in network infrastructure; I do not believe this to be true in my case. If I run a cURL request through the command line, I get my data; the Last.FM service is up and accessible.
Before I go to those folks and see if anything has changed, I wanted to check with you fine folks and see if there's some issue in my code that would be causing this.
Thanks!
ANSWER: #Jan Kuboschek helped me stumble onto what is (maybe) going on here. By giving CURLOPT_POSTFIELDS an associative array, a particular content-type is specified that may not work with certain RESTful services. A smarter solution is to manually create a URL-encoded version of that data and pass that as the CURLOPT_POSTFIELDS.
For more info, check out: http://www.brandonchecketts.com/archives/array-versus-string-in-curlopt_postfields
A common issue are spaces in the URL - beginning, in the middle, or trailing. Did you check that out?
Edit - per comments below, spacing is not the issue.
I ran your code and had the same problem - no output whatsoever. I tried the URL and with a GET request, the server talks to me. I would do the following:
Use the following as $base_url: $base_url = 'http://ws.audioscrobbler.com/2.0/?user=bachya&period=&api_key=xxx&method=user.getTopTracks';
Remove the post fields from your request.
Edit
I moved your code out of the class since I didn't have the rest and modified it. The following code runs perfect for me. If these changes don't work for you, I suggest that your error is in a different function.
<?php
function getTopTracks()
{
$base_url = 'http://ws.audioscrobbler.com/2.0/?user=bachya&period=&api_key=8066d2ebfbf1e1a8d1c32c84cf65c91c&method=user.getTopTracks';
$options = array_merge(array(
'user' => 'bachya',
'period' => NULL,
'api_key' => 'xxxxx...', // obfuscated, obviously
));
$options['method'] = 'user.getTopTracks';
// Initialize cURL request and set parameters
$ch = curl_init($base_url);
curl_setopt_array($ch, array(
CURLOPT_URL => $base_url,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_TIMEOUT => 30,
CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)'
));
$results = curl_exec($ch);
return $results;
}
echo getTopTracks();
?>
The server received your request, but sent an empty response. Check the result of curl_getinfo($ch, CURLINFO_HTTP_CODE) to find out if the server responded with an HTTP error code.
Update: Ok so the server responds with the 100 Continue HTTP status code. In that case, this should solve your problem:
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));
I found this here: PHP and cURL: Disabling 100-continue header. Hope it works!
I came acorss the same issue. My Http_code returned 200 but my response was empty. There could be many reasons for this as i experienced.
--Your hedaers might be incorrect
CURLOPT_HTTPHEADER => array('Content-Type:application/json', 'Expect:')
--You might need to send data as post fields in culr and not attached to the URl like url?p1=a1&p2=a2
$data = array (p1=>a1, p2=>a2)
CURLOPT_POSTFIELDS => $data
So your options array would be similar to the below
array(
CURLOPT_URL => $url,
CURLOPT_FAILONERROR => TRUE, // FALSE if in debug mode
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_TIMEOUT => 4,
CURLOPT_HTTPHEADER => array('Content-Type:application/json', 'Expect:'),
CURLOPT_POST => TRUE,
CURLOPT_POSTFIELDS => $data,
);
According to Last.FM API documentation you should use GET method instead of POST to pass parameters. When I've changed POST to GET I've received the answer about incorrect key.
And here's the code for get Album Info from Laft.FM even if return error:
The Function:
function getAlbum($xml,$artist,$album)
{
$base_url = $xml;
$options = array_merge(array(
'user' => 'YOUR_USERNAME',
'artist'=>$artist,
'album'=>$album,
'period' => NULL,
'api_key' => 'xYxOxUxRxxAxPxIxxKxExYxx',
));
$options['method'] = 'album.getinfo';
// Initialize cURL request and set parameters
$ch = curl_init($base_url);
curl_setopt_array($ch, array(
CURLOPT_URL => 'http://ws.audioscrobbler.com/2.0/',
CURLOPT_POST => TRUE,
CURLOPT_POSTFIELDS => $options,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTPHEADER => array( 'Expect:' ) ,
CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)'
));
$results = curl_exec($ch);
unset ($options);
return $results;
}
Usage:
// Get the XML
$xml_error = getAlbum($xml,$artist,$album);
// Show XML error
if (preg_match("/error/i", $xml_error)) {
echo " <strong>ERRO:</strong> ".trim(strip_tags($xml_error));
}