I've made a simple PHP browser to navigate on remote websites. It works perfectly when I run the script on my local server (It stores cookies and I can retrieve a session later using the sid). But when I push my files to my remote server, it doesn't save the cookies. It looks to create a new session everytime.
Did I make something wrong or do I have to enable a special option on my remote server ?
Here is my code:
<?php
class Browser
{
private $session_id = '';
private $user_agent = '';
private $curl = null;
function __construct($session_id, $user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2')
{
$this->session_id = $session_id;
$this->user_agent = $user_agent;
// Create a new instance of Curl
$this->curl = curl_init();
if (empty($this->curl)) throw new Exception('CURL doesn\'t look to be available');
}
function __destruct()
{
// Close session
curl_close($this->curl);
unset($this->curl);
}
private function request($url, $reset_cookies, $post_data)
{
$options = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HEADER => 0,
CURLOPT_FAILONERROR => 1,
CURLOPT_USERAGENT => $this->user_agent,
CURLOPT_CONNECTTIMEOUT => 30,
CURLOPT_TIMEOUT => 30,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_MAXREDIRS => 10,
CURLOPT_AUTOREFERER => 1,
CURLOPT_COOKIESESSION => $reset_cookies ? 1 : 0,
CURLOPT_COOKIEJAR => $this->session_id,
CURLOPT_COOKIEFILE => $this->session_id,
);
// Add POST data
if (isset($post_data))
{
$options[CURLOPT_POST] = 1;
$options[CURLOPT_POSTFIELDS] = http_build_query($post_data);
}
// Attach options
curl_setopt_array($this->curl, $options);
// Execute the request and read the response
$content = curl_exec($this->curl);
// Handle any error
if (curl_errno($this->curl)) throw new Exception(curl_error($this->curl));
return $content;
}
public function get($url, $reset_cookies = false)
{
return $this->request($url, $reset_cookies, null);
}
public function post($url, $post_data, $reset_cookies = false)
{
return $this->request($url, $reset_cookies, $post_data);
}
}
Thanks for your help.
You need to make sure your cookies folder is outside of the public_html folder and that the user that is running PHP has access to write to the folder.
Set the folder permissions to 755, this will be enough. Any extra permissions are a security hole and can let other people write to the folder on shared hosting..
example path...
/home/my_awesome_website/cookies
Related
I need to scrape an ASP website using cURL. My hosting does not allow me to turn off safe_mode or open_basedir. That's why CURLOPT_FOLLOWLOCATION cannot be activated (it throws an error "CURLOPT_FOLLOWLOCATION cannot be activated when an open_basedir is set").
I tried to implement some workaround but after several unlucky days starting to be desperate. I am wondering how to change the code below to contain manual redirection instead of CURLOPT_FOLLOWLOCATION:
include_once __DIR__.'/simple_html_dom.php';
define('COOKIE_FILE', __DIR__.'/cookie.txt');
#unlink(COOKIE_FILE); //clear cookies before we start
define('CURL_LOG_FILE', __DIR__.'/request.txt');
#unlink(CURL_LOG_FILE);//clear curl log
class ASPBrowser {
public $exclude = array();
public $lastUrl = '';
public $dom = false;
/**Get simplehtmldom object from url
* #param $url
* #param $post
* #return bool|simple_html_dom
*/
public function getDom($url, $post = false) {
$f = fopen(CURL_LOG_FILE, 'a+'); // curl session log file
if($this->lastUrl) $header[] = "Referer: {$this->lastUrl}";
$curlOptions = array(
CURLOPT_ENCODING => 'gzip,deflate',
CURLOPT_AUTOREFERER => 1,
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_URL => $url,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_MAXREDIRS => 9,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HEADER => 0,
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36",
CURLOPT_COOKIEFILE => COOKIE_FILE,
CURLOPT_COOKIEJAR => COOKIE_FILE,
CURLOPT_STDERR => $f, // log session
CURLOPT_VERBOSE => true,
);
if($post) { // add post options
$curlOptions[CURLOPT_POSTFIELDS] = $post;
$curlOptions[CURLOPT_POST] = true;
}
$curl = curl_init();
curl_setopt_array($curl, $curlOptions);
$data = curl_exec($curl);
$this->lastUrl = curl_getinfo($curl, CURLINFO_EFFECTIVE_URL); // get url we've been redirected to
curl_close($curl);
if($this->dom) {
$this->dom->clear();
$this->dom = false;
}
$dom = $this->dom = str_get_html($data);
fwrite($f, "{$post}\n\n");
fwrite($f, "-----------------------------------------------------------\n\n");
fclose($f);
return $dom;
}
function createASPPostParams($dom, array $params) {
$postData = $dom->find('input,select,textarea');
$postFields = array();
foreach($postData as $d) {
$name = $d->name;
if(trim($name) == '' || in_array($name, $this->exclude)) continue;
$value = isset($params[$name]) ? $params[$name] : $d->value;
$postFields[] = rawurlencode($name).'='.rawurlencode($value);
}
$postFields = implode('&', $postFields);
return $postFields;
}
function doPostRequest($url, array $params) {
$post = $this->createASPPostParams($this->dom, $params);
return $this->getDom($url, $post);
}
function doPostBack($url, $eventTarget, $eventArgument = '') {
return $this->doPostRequest($url, array(
'__EVENTTARGET' => $eventTarget,
'__EVENTARGUMENT' => $eventArgument
));
}
function doGetRequest($url) {
return $this->getDom($url);
}
}
(Credits: Andrey http://256cats.com/scraping-asp-websites-php-dopostback-ajax-emulation/)
You're probably looking for the CURLINFO_REDIRECT_URL info variable, as that returns the URL that it would otherwise had redirected to if you'd allowed it. Added in PHP 5.3.7.
Note that the exact response code 3xx also affects how the HTTP request method is supposed to change or not change when you follow a redirect. See details in the HTTP spec, RFC 7231 section 6.4.
The libcurl docs for CURLINFO_REDIRECT_URL.
I'm creating a function to untilize NameCheap's API for registering domain names. The registration process worked out smoothly, now I'm looking to set the proper DNS Hosts.
When I create a pure POST request with something like POSTMAN this works fine and returns the expected XML response. However when I try to pass the data through PHP's CURL functions it breaks. I've narrowed the problem the the '#' symbol that needs to be passed to the DNS Host. If i put anything else there the request goes through. I've tried to url_encode the symbol but the API does not accept that.
Any suggestions?
public function setDNSHost($name, $server){
list($domain,$tld) = explode('.',$name,2);
$request = $this->request_URL;
$curl = curl_init();
$args['ApiUser'] = $this->API_User;
$args['ApiKey'] = $this->API_Key;
$args['UserName'] = $this->API_User;
$args['Command'] = 'namecheap.domains.dns.setHosts';
$args['ClientIP'] = $this->Client_IP;
$args['SLD'] = $domain;
$args['TLD'] = $tld;
$args['HostName1'] = utf8_encode('#');
$args['RecordType1'] = 'A';
$args['Address1'] = $server;
$args['HostName2'] = 'www';
$args['RecordType2'] = 'CNAME';
$args['Address2'] = $name;
$args['HostName3'] = '*';
$args['RecordType3'] = 'CNAME';
$args['Address3'] = $name;
curl_setopt_array($curl, array(
CURLOPT_URL => $request,
CURLOPT_USERAGENT => 'API',
// CURLOPT_FAILONERROR => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => $args,
CURLOPT_TIMEOUT => 15
));
$response = curl_exec($curl);
curl_close($curl);
// $oXML = new SimpleXMLElement($response);
return $response;
}
Some caracters are not allowed to be in the string. To avoid such problems you could use http_build_query on your data before you use the curl function.
I'm trying to use curl instead of the http request 2 pear module in PHP to query the plivo api. They have an existing library for easily making calls to their API but it uses a pear module called http request2. I don't really know how to install a pear module on a server so I thought of just rewriting some parts of their library to just use curl.
Here's the part of their code that I specifically want to modify:
function __construct($auth_id, $auth_token, $url="https://api.plivo.com", $version="v1") {
if ((!isset($auth_id)) || (!$auth_id)) {
throw new PlivoError("no auth_id");
}
if ((!isset($auth_token)) || (!$auth_token)) {
throw new PlivoError("no auth_token");
}
$this->version = $version;
$this->api = $url."/".$this->version."/Account/".$auth_id;
$this->auth_id = $auth_id;
$this->auth_token = $auth_token;
}
private function request($method, $path, $params=array()) {
$url = $this->api.rtrim($path, '/').'/';
if (!strcmp($method, "POST")) {
$req = new HTTP_Request2($url, HTTP_Request2::METHOD_POST);
$req->setHeader('Content-type: application/json');
if ($params) {
$req->setBody(json_encode($params));
}
} else if (!strcmp($method, "GET")) {
$req = new HTTP_Request2($url, HTTP_Request2::METHOD_GET);
$url = $req->getUrl();
$url->setQueryVariables($params);
} else if (!strcmp($method, "DELETE")) {
$req = new HTTP_Request2($url, HTTP_Request2::METHOD_DELETE);
$url = $req->getUrl();
$url->setQueryVariables($params);
}
$req->setAdapter('curl');
$req->setConfig(array(
'timeout' => 30,
'ssl_verify_peer' => FALSE,
));
$req->setAuth($this->auth_id, $this->auth_token, HTTP_Request2::AUTH_BASIC);
$req->setHeader(array(
'Connection' => 'close',
'User-Agent' => 'PHPPlivo',
));
$r = $req->send();
$status = $r->getStatus();
$body = $r->getbody();
$response = json_decode($body, true);
return array("status" => $status, "response" => $response);
}
public function get_account($params=array()) {
return $this->request('GET', '', $params);
}
And here's the code that I have so far:
<?php
$curl = curl_init();
$curl_options = array(
CURLOPT_URL => 'https://api.plivo.com/v1/Account/',
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_USERPWD => 'auth_id:auth_token',
CURLOPT_HTTPHEADER => array("Connection: close", "User-Agent: PHPPlivo"),
CURLOPT_TIMEOUT => 30
);
curl_setopt_array($curl, $curl_options);
$response = curl_exec($curl);
curl_close($curl);
?>
I don't really know what's going on behind the scenes but this specific code is telling me that its using basic authentication using the values for the auth id and auth token:
$req->setAuth($this->auth_id, $this->auth_token, HTTP_Request2::AUTH_BASIC);
So I also set it using curl:
CURLOPT_USERPWD => 'auth_id:auth_token',
I'm pretty much stuck. All I get as a respose is the following:
{
"error": "not found"
}
It doesn't really make much sense into what I have missed or done wrong. Please help. Thank you in advance!
Below are the things you need to handle to sync your new code with old one:
If you are using GET method
CURLOPT_URL => 'https://api.plivo.com/v1/Account/'.http_build_query($params),
CURLOPT_HTTPHEADER => array("User-Agent: PHPPlivo"),
If you are using POST method
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query($params),
CURLOPT_HTTPHEADER => array("Content-type: application/json", "User-Agent: PHPPlivo"),
Yea... The PEAR dependency is definitely overkill for Plivo's wrapper. So that was one of the first modifications I made to the code.
Check out:
https://github.com/ashbeats/Plivo-Curl-Based-Wrapper/
Only difference is the RestAPI::request() method.
I need to connect with the eBay motors site using a function in openbay which is an opencart extension. The site id for eBay motors is 100, but for the life of me I cannot get it to change with the way this function is written, am I missing something here???
API function call
public function openbay_call($call, array $post = NULL, array $options = array(), $content_type = 'json', $statusOverride = false){
if(defined("HTTPS_CATALOG")){
$domain = HTTPS_CATALOG;
}else{
$domain = HTTPS_SERVER;
}
$data = array(
'token' => $this->token,
'language' => $this->config->get('openbay_language'),
'secret' => $this->secret,
'server' => $this->server,
'domain' => $domain,
'openbay_version' => (int)$this->config->get('openbay_version'),
'data' => $post,
'content_type' => $content_type
);
$defaults = array(
CURLOPT_POST => 1,
CURLOPT_HEADER => 0,
CURLOPT_URL => $this->url.$call,
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1",
CURLOPT_FRESH_CONNECT => 1,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FORBID_REUSE => 1,
CURLOPT_TIMEOUT => 0,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_POSTFIELDS => http_build_query($data, '', "&")
);
$ch = curl_init();
curl_setopt_array($ch, ($options + $defaults));
if( ! $result = curl_exec($ch)){
$this->log('openbay_call() - Curl Failed '.curl_error($ch).' '.curl_errno($ch));
}
curl_close($ch);
/* There may be some calls we just dont want to log */
if(!in_array($call, $this->noLog)){
$this->log('openbay_call() - Result of : "'.$result.'"');
}
/* JSON RESPONSE */
if($content_type == 'json'){
$encoding = mb_detect_encoding($result);
/* some json data may have BOM due to php not handling types correctly */
if($encoding == 'UTF-8') {
$result = preg_replace('/[^(\x20-\x7F)]*/','', $result);
}
$result = json_decode($result, 1);
$this->lasterror = $result['error'];
$this->lastmsg = $result['msg'];
if(!empty($result['data'])){
return $result['data'];
}else{
return false;
}
/* XML RESPONSE */
}elseif($content_type == 'xml'){
$result = simplexml_load_string($result);
$this->lasterror = $result->error;
$this->lastmsg = $result->msg;
if(!empty($result->data)){
return $result->data;
}else{
return false;
}
}
}else{
$this->log('openbay_call() - OpenBay not active');
$this->log('openbay_call() - Data: '.serialize($post));
}
}
predefined parameters within the class - probably don't help but included anyways.
public function __construct($registry) {
$this->registry = $registry;
$this->token = $this->config->get('openbaypro_token');
$this->secret = $this->config->get('openbaypro_secret');
$this->logging = $this->config->get('openbaypro_logging');
$this->tax = $this->config->get('tax');
$this->server = 1;
$this->lasterror = '';
$this->lastmsg = '';
}
the function call
$this->data['test_category_features'] = $this->ebay->openbay_call('listing/getCategoryFeatures/', array('id' => 35618));
Everything works but how would i get this to change siteid to 100, the only way I can figure it out is to re-write my own API call class, but the client is paying for the subscription to openbay and wants to use the API calls through them, so I have to use there function. Im trying to return eBay motors category features so he can list them the same way he has been for years "used parts". If you don't switch to the eBay motors site id "100" then it will not return the category variations needed or more less accept the categories when trying to add products to eBay through the opencart extension.
Any advice would be greatly appreciated, really stuck here!!! Thanks in advance :)
according to this page: http://developer.ebay.com/DevZone/merchandising/docs/Concepts/SiteIDToGlobalID.html you need to add "... X-EBAY-SOA-GLOBAL-ID HTTP header for each API call" so add that to the curl options.
I´m trying to upload a file from PHP via Box-API v2 and I only get a boolean false response.
I think this is caused by CURL, not Box-API but I was fighting the last five hours, and I can´t find the solution. Any idea??
The implicated code is that:
note: the file exists and is accessible from code and the token is ok (other calls to API work fine)
const CONTENT_ENDPOINT = 'https://api.box.com/2.0/';
$file = "unexeceles.xlsx";
private $defaultOptions = array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_VERBOSE => true,
CURLOPT_HEADER => true,
CURLINFO_HEADER_OUT => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => false,
);
public function putFile($file) {
$options = $this->defaultOptions;
$options[CURLOPT_HTTPHEADER] = array ("Authorization: Bearer ".$this->token);
$options[CURLOPT_POST] = true;
$postfields = array();
$postfields["filename"] = '#'.$file;
$postfields["parent_id"] = 0;
$options[CURLOPT_POSTFIELDS] = $postfields;
$handle = curl_init(BoxConfig::CONTENT_ENDPOINT."files/content");
curl_setopt_array($handle, $options);
$response = curl_exec($handle);
curl_close($handle);
if (is_string($response)) {
$response = $this->parse($response);
}
return $response;
}
Finally I've found the solution.
The problem was the relative path to the file, the file exists and it´s accessible form code, but CURL seems to need the entire path to the file.
Very helpful the function curl_errno($handle)
if(curl_errno($handle)) {
echo 'Curl error: ' . curl_error($handle);
}