I have problem with a WebService function for moodle callen "mod_scorm_insert_scorm_tracks"
This function is used for inserting track information (i.e. star time) of a user in his SCORM progress.
Part of the estructure of this function is
scoid= int
attempt= int
tracks[0][element]= string
tracks[0][value]= string
NEW
PHP structe has to look like this
[tracks] =>
Array
(
[0] =>
Array
(
[element] => string
[value] => string
)
)
I have used one of the examples they had in his website everything was fine until I got this error
<b>Notice</b>: Array to string conversion in <b>C:\xampp\htdocs\otros\PHP-REST\curl.php</b> on line <b>247</b><br />
<?xml version="1.0" encoding="UTF-8" ?>
<EXCEPTION class="invalid_parameter_exception">
<ERRORCODE>invalidparameter</ERRORCODE>
<MESSAGE>Invalid parameter value detected</MESSAGE>
<DEBUGINFO>tracks => Invalid parameter value detected: Only arrays accepted. The bad value is: 'Array'</DEBUGINFO>
</EXCEPTION>
And the problem seems to be here:
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(
array(
array(
'element' => 'x.start.time',
'value' => '1473102672'
),
),
array(
array(
'element' => 'x.start.time',
'value' => '1473102680'
),
),
);
I tried in many ways
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array('element' => 'x.start.time','value' => '1473102672');
or
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(array ('element' => 'x.start.time','value' => '1473102672'));
And still getting the same message, I'm pretty that is problema with my wyntax but I have tried in many ways and still not working I hope yo can help me.
Complete Code:
/// SETUP - NEED TO BE CHANGED
$token = '481bf3d85a7eb539e37eabc88feccb3c';
$domainname = 'http://localhost/moodle';
//$functionname = 'mod_scorm_launch_sco';
$functionname = 'mod_scorm_insert_scorm_tracks';
//$functionname ='mod_scorm_view_scorm';
// REST RETURNED VALUES FORMAT
$restformat = 'xml'; //Also possible in Moodle 2.2 and later: 'json'
//Setting it to 'json' will fail all calls on earlier Moodle version
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(
array(
array(
'element' => 'x.start.time',
'value' => 1473102672
),
),
array(
array(
'element' => 'x.start.time',
'value' => 1473102680
),
),
);
$params = $item1;
/// REST CALL
header('Content-Type: text/plain');
$serverurl = $domainname . '/webservice/rest/server.php'. '?wstoken=' . $token . '&wsfunction='.$functionname;
require_once('./curl.php');
$curl = new curl;
//if rest format == 'xml', then we do not add the param for backward compatibility with Moodle < 2.2
$restformat = ($restformat == 'json')?'&moodlewsrestformat=' . $restformat:'';
$resp = $curl->post($serverurl . $restformat, $params);
print_r($resp);
curl.php
<?php
/**
* cURL class
*
* This is a wrapper class for curl, it is quite easy to use:
* <code>
* $c = new curl;
* // enable cache
* $c = new curl(array('cache'=>true));
* // enable cookie
* $c = new curl(array('cookie'=>true));
* // enable proxy
* $c = new curl(array('proxy'=>true));
*
* // HTTP GET Method
* $html = $c->get('http://example.com');
* // HTTP POST Method
* $html = $c->post('http://example.com/', array('q'=>'words', 'name'=>'moodle'));
* // HTTP PUT Method
* $html = $c->put('http://example.com/', array('file'=>'/var/www/test.txt');
* </code>
*
* #author Dongsheng Cai <dongsheng#moodle.com> - https://github.com/dongsheng/cURL
* #license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class curl {
/** #var bool */
public $cache = false;
public $proxy = false;
/** #var array */
public $response = array();
public $header = array();
/** #var string */
public $info;
public $error;
/** #var array */
private $options;
/** #var string */
private $proxy_host = '';
private $proxy_auth = '';
private $proxy_type = '';
/** #var bool */
private $debug = false;
private $cookie = false;
private $count = 0;
/**
* #param array $options
*/
public function __construct($options = array()){
if (!function_exists('curl_init')) {
$this->error = 'cURL module must be enabled!';
trigger_error($this->error, E_USER_ERROR);
return false;
}
// the options of curl should be init here.
$this->resetopt();
if (!empty($options['debug'])) {
$this->debug = true;
}
if(!empty($options['cookie'])) {
if($options['cookie'] === true) {
$this->cookie = 'curl_cookie.txt';
} else {
$this->cookie = $options['cookie'];
}
}
if (!empty($options['cache'])) {
if (class_exists('curl_cache')) {
$this->cache = new curl_cache();
}
}
}
/**
* Resets the CURL options that have already been set
*/
public function resetopt(){
$this->options = array();
$this->options['CURLOPT_USERAGENT'] = 'MoodleBot/1.0';
// True to include the header in the output
$this->options['CURLOPT_HEADER'] = 0;
// True to Exclude the body from the output
$this->options['CURLOPT_NOBODY'] = 0;
// TRUE to follow any "Location: " header that the server
// sends as part of the HTTP header (note this is recursive,
// PHP will follow as many "Location: " headers that it is sent,
// unless CURLOPT_MAXREDIRS is set).
//$this->options['CURLOPT_FOLLOWLOCATION'] = 1;
$this->options['CURLOPT_MAXREDIRS'] = 10;
$this->options['CURLOPT_ENCODING'] = '';
// TRUE to return the transfer as a string of the return
// value of curl_exec() instead of outputting it out directly.
$this->options['CURLOPT_RETURNTRANSFER'] = 1;
$this->options['CURLOPT_BINARYTRANSFER'] = 0;
$this->options['CURLOPT_SSL_VERIFYPEER'] = 0;
$this->options['CURLOPT_SSL_VERIFYHOST'] = 2;
$this->options['CURLOPT_CONNECTTIMEOUT'] = 30;
}
/**
* Reset Cookie
*/
public function resetcookie() {
if (!empty($this->cookie)) {
if (is_file($this->cookie)) {
$fp = fopen($this->cookie, 'w');
if (!empty($fp)) {
fwrite($fp, '');
fclose($fp);
}
}
}
}
/**
* Set curl options
*
* #param array $options If array is null, this function will
* reset the options to default value.
*
*/
public function setopt($options = array()) {
if (is_array($options)) {
foreach($options as $name => $val){
if (stripos($name, 'CURLOPT_') === false) {
$name = strtoupper('CURLOPT_'.$name);
}
$this->options[$name] = $val;
}
}
}
/**
* Reset http method
*
*/
public function cleanopt(){
unset($this->options['CURLOPT_HTTPGET']);
unset($this->options['CURLOPT_POST']);
unset($this->options['CURLOPT_POSTFIELDS']);
unset($this->options['CURLOPT_PUT']);
unset($this->options['CURLOPT_INFILE']);
unset($this->options['CURLOPT_INFILESIZE']);
unset($this->options['CURLOPT_CUSTOMREQUEST']);
}
/**
* Set HTTP Request Header
*
* #param array $headers
*
*/
public function setHeader($header) {
if (is_array($header)){
foreach ($header as $v) {
$this->setHeader($v);
}
} else {
$this->header[] = $header;
}
}
/**
* Set HTTP Response Header
*
*/
public function getResponse(){
return $this->response;
}
/**
* private callback function
* Formatting HTTP Response Header
*
* #param mixed $ch Apparently not used
* #param string $header
* #return int The strlen of the header
*/
private function formatHeader($ch, $header)
{
$this->count++;
if (strlen($header) > 2) {
list($key, $value) = explode(" ", rtrim($header, "\r\n"), 2);
$key = rtrim($key, ':');
if (!empty($this->response[$key])) {
if (is_array($this->response[$key])){
$this->response[$key][] = $value;
} else {
$tmp = $this->response[$key];
$this->response[$key] = array();
$this->response[$key][] = $tmp;
$this->response[$key][] = $value;
}
} else {
$this->response[$key] = $value;
}
}
return strlen($header);
}
/**
* Set options for individual curl instance
*
* #param object $curl A curl handle
* #param array $options
* #return object The curl handle
*/
private function apply_opt($curl, $options) {
// Clean up
$this->cleanopt();
// set cookie
if (!empty($this->cookie) || !empty($options['cookie'])) {
$this->setopt(array('cookiejar'=>$this->cookie,
'cookiefile'=>$this->cookie
));
}
// set proxy
if (!empty($this->proxy) || !empty($options['proxy'])) {
$this->setopt($this->proxy);
}
$this->setopt($options);
// reset before set options
curl_setopt($curl, CURLOPT_HEADERFUNCTION, array(&$this,'formatHeader'));
// set headers
if (empty($this->header)){
$this->setHeader(array(
'User-Agent: MoodleBot/1.0',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Connection: keep-alive'
));
}
curl_setopt($curl, CURLOPT_HTTPHEADER, $this->header);
if ($this->debug){
echo '<h1>Options</h1>';
var_dump($this->options);
echo '<h1>Header</h1>';
var_dump($this->header);
}
// set options
foreach($this->options as $name => $val) {
if (is_string($name)) {
$name = constant(strtoupper($name));
}
curl_setopt($curl, $name, $val);
}
return $curl;
}
/**
* Download multiple files in parallel
*
* Calls {#link multi()} with specific download headers
*
* <code>
* $c = new curl;
* $c->download(array(
* array('url'=>'http://localhost/', 'file'=>fopen('a', 'wb')),
* array('url'=>'http://localhost/20/', 'file'=>fopen('b', 'wb'))
* ));
* </code>
*
* #param array $requests An array of files to request
* #param array $options An array of options to set
* #return array An array of results
*/
public function download($requests, $options = array()) {
$options['CURLOPT_BINARYTRANSFER'] = 1;
$options['RETURNTRANSFER'] = false;
return $this->multi($requests, $options);
}
/*
* Mulit HTTP Requests
* This function could run multi-requests in parallel.
*
* #param array $requests An array of files to request
* #param array $options An array of options to set
* #return array An array of results
*/
protected function multi($requests, $options = array()) {
$count = count($requests);
$handles = array();
$results = array();
$main = curl_multi_init();
for ($i = 0; $i < $count; $i++) {
$url = $requests[$i];
foreach($url as $n=>$v){
$options[$n] = $url[$n];
}
$handles[$i] = curl_init($url['url']);
$this->apply_opt($handles[$i], $options);
curl_multi_add_handle($main, $handles[$i]);
}
$running = 0;
do {
curl_multi_exec($main, $running);
} while($running > 0);
for ($i = 0; $i < $count; $i++) {
if (!empty($options['CURLOPT_RETURNTRANSFER'])) {
$results[] = true;
} else {
$results[] = curl_multi_getcontent($handles[$i]);
}
curl_multi_remove_handle($main, $handles[$i]);
}
curl_multi_close($main);
return $results;
}
/**
* Single HTTP Request
*
* #param string $url The URL to request
* #param array $options
* #return bool
*/
protected function request($url, $options = array()){
// create curl instance
$curl = curl_init($url);
$options['url'] = $url;
$this->apply_opt($curl, $options);
if ($this->cache && $ret = $this->cache->get($this->options)) {
return $ret;
} else {
$ret = curl_exec($curl);
if ($this->cache) {
$this->cache->set($this->options, $ret);
}
}
$this->info = curl_getinfo($curl);
$this->error = curl_error($curl);
if ($this->debug){
echo '<h1>Return Data</h1>';
var_dump($ret);
echo '<h1>Info</h1>';
var_dump($this->info);
echo '<h1>Error</h1>';
var_dump($this->error);
}
curl_close($curl);
if (empty($this->error)){
return $ret;
} else {
return $this->error;
// exception is not ajax friendly
//throw new moodle_exception($this->error, 'curl');
}
}
/**
* HTTP HEAD method
*
* #see request()
*
* #param string $url
* #param array $options
* #return bool
*/
public function head($url, $options = array()){
$options['CURLOPT_HTTPGET'] = 0;
$options['CURLOPT_HEADER'] = 1;
$options['CURLOPT_NOBODY'] = 1;
return $this->request($url, $options);
}
/**
* Recursive function formating an array in POST parameter
* #param array $arraydata - the array that we are going to format and add into &$data array
* #param string $currentdata - a row of the final postdata array at instant T
* when finish, it's assign to $data under this format: name[keyname][][]...[]='value'
* #param array $data - the final data array containing all POST parameters : 1 row = 1 parameter
*/
function format_array_postdata_for_curlcall($arraydata, $currentdata, &$data) {
foreach ($arraydata as $k=>$v) {
$newcurrentdata = $currentdata;
if (is_object($v)) {
$v = (array) $v;
}
if (is_array($v)) { //the value is an array, call the function recursively
$newcurrentdata = $newcurrentdata.'['.urlencode($k).']';
$this->format_array_postdata_for_curlcall($v, $newcurrentdata, $data);
} else { //add the POST parameter to the $data array
$data[] = $newcurrentdata.'['.urlencode($k).']='.urlencode($v);
}
}
}
/**
* Transform a PHP array into POST parameter
* (see the recursive function format_array_postdata_for_curlcall)
* #param array $postdata
* #return array containing all POST parameters (1 row = 1 POST parameter)
*/
function format_postdata_for_curlcall($postdata) {
if (is_object($postdata)) {
$postdata = (array) $postdata;
}
$data = array();
foreach ($postdata as $k=>$v) {
if (is_object($v)) {
$v = (array) $v;
}
if (is_array($v)) {
$currentdata = urlencode($k);
$this->format_array_postdata_for_curlcall($v, $currentdata, $data);
} else {
$data[] = urlencode($k).'='.urlencode($v);
}
}
$convertedpostdata = implode('&', $data);
return $convertedpostdata;
}
/**
* HTTP POST method
*
* #param string $url
* #param array|string $params
* #param array $options
* #return bool
*/
public function post($url, $params = '', $options = array()){
$options['CURLOPT_POST'] = 1;
if (is_array($params)) {
$params = $this->format_postdata_for_curlcall($params);
}
$options['CURLOPT_POSTFIELDS'] = $params;
return $this->request($url, $options);
}
/**
* HTTP GET method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function get($url, $params = array(), $options = array()){
$options['CURLOPT_HTTPGET'] = 1;
if (!empty($params)){
$url .= (stripos($url, '?') !== false) ? '&' : '?';
$url .= http_build_query($params, '', '&');
}
return $this->request($url, $options);
}
/**
* HTTP PUT method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function put($url, $params = array(), $options = array()){
$file = $params['file'];
if (!is_file($file)){
return null;
}
$fp = fopen($file, 'r');
$size = filesize($file);
$options['CURLOPT_PUT'] = 1;
$options['CURLOPT_INFILESIZE'] = $size;
$options['CURLOPT_INFILE'] = $fp;
if (!isset($this->options['CURLOPT_USERPWD'])){
$this->setopt(array('CURLOPT_USERPWD'=>'anonymous: noreply#moodle.org'));
}
$ret = $this->request($url, $options);
fclose($fp);
return $ret;
}
/**
* HTTP DELETE method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function delete($url, $param = array(), $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'DELETE';
if (!isset($options['CURLOPT_USERPWD'])) {
$options['CURLOPT_USERPWD'] = 'anonymous: noreply#moodle.org';
}
$ret = $this->request($url, $options);
return $ret;
}
/**
* HTTP TRACE method
*
* #param string $url
* #param array $options
* #return bool
*/
public function trace($url, $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'TRACE';
$ret = $this->request($url, $options);
return $ret;
}
/**
* HTTP OPTIONS method
*
* #param string $url
* #param array $options
* #return bool
*/
public function options($url, $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'OPTIONS';
$ret = $this->request($url, $options);
return $ret;
}
public function get_info() {
return $this->info;
}
}
/**
* This class is used by cURL class, use case:
*
* <code>
*
* $c = new curl(array('cache'=>true), 'module_cache'=>'repository');
* $ret = $c->get('http://www.google.com');
* </code>
*
* #package core
* #subpackage file
* #copyright 1999 onwards Martin Dougiamas {#link http://moodle.com}
* #license http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
*/
class curl_cache {
/** #var string */
public $dir = '';
/**
*
* #param string #module which module is using curl_cache
*
*/
function __construct() {
$this->dir = '/tmp/';
if (!file_exists($this->dir)) {
mkdir($this->dir, 0700, true);
}
$this->ttl = 1200;
}
/**
* Get cached value
*
* #param mixed $param
* #return bool|string
*/
public function get($param){
$this->cleanup($this->ttl);
$filename = 'u_'.md5(serialize($param));
if(file_exists($this->dir.$filename)) {
$lasttime = filemtime($this->dir.$filename);
if(time()-$lasttime > $this->ttl)
{
return false;
} else {
$fp = fopen($this->dir.$filename, 'r');
$size = filesize($this->dir.$filename);
$content = fread($fp, $size);
return unserialize($content);
}
}
return false;
}
/**
* Set cache value
*
* #param mixed $param
* #param mixed $val
*/
public function set($param, $val){
$filename = 'u_'.md5(serialize($param));
$fp = fopen($this->dir.$filename, 'w');
fwrite($fp, serialize($val));
fclose($fp);
}
/**
* Remove cache files
*
* #param int $expire The number os seconds before expiry
*/
public function cleanup($expire){
if($dir = opendir($this->dir)){
while (false !== ($file = readdir($dir))) {
if(!is_dir($file) && $file != '.' && $file != '..') {
$lasttime = #filemtime($this->dir.$file);
if(time() - $lasttime > $expire){
#unlink($this->dir.$file);
}
}
}
}
}
/**
* delete current user's cache file
*
*/
public function refresh(){
if($dir = opendir($this->dir)){
while (false !== ($file = readdir($dir))) {
if(!is_dir($file) && $file != '.' && $file != '..') {
if(strpos($file, 'u_')!==false){
#unlink($this->dir.$file);
}
}
}
}
}
}
Thanks!
Well after some research I finally took plan B
I wrote tracks in a different variable:
$tracks = array();
$tracks[] = array(
'element' => 'cmi.core.lesson_status',
'value' => 'completed'
);
And I followed curl.php array set option:
$arrayName = array('' => , );
Then when I inserted scoid and attemps as single variables in the array:
$params = array('scoid' => '2', 'attempt' => '1', 'tracks' => $tracks);
and boala!the record is on my table:
Related
Ok a few weeks ago my script stopped working and today I notice I'm getting an error: [05-Mar-2017 06:31:32 America/Denver] PHP Notice: Undefined index: transfer_encoding in /home2/website/public_html/maps_apps/EasyWebFetch.php on line 105.
Here is what line 105 is:
if ($this->_resp_headers['transfer_encoding'] == 'chunked') {
Can someone point me in the right direction on getting this fixed?
Here is the main code:
<?php
require_once '/home2/website/public_html/maps_apps/EasyWebFetch.php';
$callback = isset($_GET['callback']) ? $_GET['callback'] : 'mymap.weatherhandler';
$station = isset($_GET['rid']) ? $_GET['rid'] : 'FWS';
$product = isset($_GET['product']) ? $_GET['product'] : 'NCR';
$nframes = isset($_GET['frames']) ? $_GET['frames'] : 10;
if (strlen($product) != 3 || strlen($station) != 3) { exit; }
// fetch directory listing
$wf = new EasyWebFetch;
if (!$wf->get("https://radar.weather.gov/ridge/RadarImg/$product/$station/")) {
print $wf->getErrorMessage();
exit;
}
$page = $wf->getContents();
echo $page."\n\n";
$size = preg_match_all( "/href=\"({$station}[\d_]+{$product}\.gif)\"/" , $page, $match);
$files = $match[1];
if ($nframes == 'all') { $nframes = count($files); }
$nframes = min(count($files), $nframes);
$files = array_slice($files, -$nframes);
echo $callback."\n(\n{\ndirectory:\n[\n";
for ($i=0; $i < $nframes; $i++) {
echo "\"ridge/RadarImg/$product/$station/$files[$i]\",\n";
}
echo "]\n}\n)\n;"
?>
and here is EasyWebFetch.php
<?php
/*
* EasyWebFetch - Fetch a page by opening socket connection, no dependencies
*
* PHP version 5
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* #author Nashruddin Amin <me#nashruddin.com>
* #copyright Nashruddin Amin 2008
* #license GNU General Public License 3.0
* #package EasyWebFetch
* #version 1.1
*/
class EasyWebFetch
{
private $_request_url;
private $_host;
private $_path;
private $_query;
private $_fragment;
private $_headers_only;
private $_portnum = 80;
private $_user_agent = "SimpleHttpClient/3.0";
private $_req_timeout = 30;
private $_maxredirs = 5;
private $_use_proxy = false;
private $_proxy_host;
private $_proxy_port;
private $_proxy_user;
private $_proxy_pass;
private $_status;
private $_resp_headers;
private $_resp_body;
private $_is_error;
private $_errmsg;
/**
* class constructor
*/
public function __construct()
{
$this->_resp_headers = array();
$this->_resp_body = "";
}
/**
* get the requested page
*
* #param string $url URL of the requested page
* #param boolean $headers_only true to return headers only,
* false to return headers and body
*
* #return boolean true on success, false on failure
*/
public function get($url = '', $headers_only = false)
{
$this->_request_url = $url;
$this->_headers_only = $headers_only;
$redir = 0;
while(($redir++) <= $this->_maxredirs) {
$this->parseUrl($this->_request_url);
if (($response = $this->makeRequest()) == false) {
return(false);
}
/* split head and body */
$neck = strpos($response, "\r\n\r\n");
$head = substr($response, 0, $neck);
$body = substr($response, $neck+2);
/* read response headers */
$this->_resp_headers = $this->parseHeaders($head);
/* check for redirects */
if ($this->getStatus() == 301 || $this->getStatus() == 302) {
$follow = $this->_resp_headers['location'];
$this->_request_url = $this->setFullPath($follow, $this->_request_url);
continue;
} else {
/* no redirects, start reading response body */
break;
}
}
/* read the body part */
if ($this->_resp_headers['transfer_encoding'] == 'chunked') {
$this->_resp_body = $this->joinChunks($body);
} else {
$this->_resp_body = $body;
}
return(true);
}
/**
* build HTTP header and perform HTTP request
*
* #return mixed HTTP response on success, false on failure
*/
private function makeRequest()
{
$method = ($this->_headers_only == true) ? "HEAD" : "GET";
$proxy_auth = base64_encode("$this->_proxy_user:$this->_proxy_pass");
$response = "";
if ($this->_use_proxy) {
$headers = "$method $this->_request_url HTTP/1.1\r\n"
. "Host: $this->_host\r\n"
. "Proxy-Authorization: Basic $proxy_auth\r\n"
. "User-Agent: $this->_user_agent\r\n"
. "Connection: Close\r\n\r\n";
$fp = fsockopen($this->_proxy_host, $this->_proxy_port, $errno, $errmsg, $this->_req_timeout);
} else {
$headers = "$method $this->_path$this->_query$this->_fragment HTTP/1.1\r\n"
. "Host: $this->_host\r\n"
. "User-Agent: $this->_user_agent\r\n"
. "Connection: Close\r\n\r\n";
$fp = fsockopen($this->_host, $this->_portnum, $errno, $errmsg, $this->_req_timeout);
}
if (!$fp) {
$this->_is_error = true;
$this->_errmsg = "Unknown error";
return(false);
}
fwrite($fp, $headers);
while(!feof($fp)) {
$response .= fgets($fp, 4096);
}
fclose($fp);
return($response);
}
/**
* parse the requested URL to its host, path, query and fragment
*
* #return void
*/
private function parseUrl($url)
{
$this->_host = parse_url($url, PHP_URL_HOST);
$this->_path = parse_url($url, PHP_URL_PATH);
$this->_query = parse_url($url, PHP_URL_QUERY);
$this->_fragment = parse_url($url, PHP_URL_FRAGMENT);
if (empty($this->_path)) {
$this->_path = '/';
}
}
/**
* get the full path of the page to redirect. if the requested page is
* http://www.example.com and it redirects to redirpage.html, then the
* new request is http://www.example.com/redirpage.html
*
* #param string $loc new location from the HTTP response headers
* #param string $parent_url the parent's URL
*
* #return string full path of the page to redirect
*/
private function setFullPath($loc, $parent_url)
{
$parent_url = preg_replace("/\/[^\/]*$/", "", $parent_url);
if (strpos($loc, 'http://') !== false) {
return($loc);
}
if (strpos($loc, '../') === false) {
return("$parent_url/$loc");
}
while (strpos($loc, '../') !== false) {
$loc = preg_replace("/^\.\.\//", "", $loc);
$parent_url = preg_replace("/\/[^\/]+$/", "", $parent_url);
}
return("$parent_url/$loc");
}
/**
* parse HTTP response headers to array
*
* #param string $string HTTP response headers
*
* #return array
*/
private function parseHeaders($string)
{
$string = trim($string);
$headers = array();
$lines = explode("\r\n", $string);
$headers['http_status'] = $lines[0];
/* read HTTP _status in first line */
preg_match('/HTTP\/(\\d\\.\\d)\\s*(\\d+)\\s*(.*)/', $lines[0], $m);
$this->_status = $m[2];
array_splice($lines, 0, 1); /* remove first line */
foreach ($lines as $line) {
list($key, $val) = explode(': ', $line);
$key = str_replace("-", "_", $key);
$key = strtolower($key);
$val = trim($val);
$headers[$key] = $val;
}
return($headers);
}
/**
* join parts of the HTTP response body with chunked transfer-encoding
*
* #param string $chunks HTTP response body
*
* #return string full body
*/
private function joinChunks($chunks)
{
preg_match("/\r\n([0-9a-z]+)(;?.*)\r\n/", $chunks, $match);
$size = hexdec($match[1]);
$body = "";
while($size > 0) {
/* remove line with chunk size */
$chunks = preg_replace("/\r\n.+\r\n/m", "", $chunks, 1);
$part = substr($chunks, 0, $size);
$chunks = substr($chunks, $size);
$body .= $part;
/* get next chunk size */
preg_match("/\r\n([0-9a-z]+)(;?.*)\r\n/", $chunks, $match);
$size = hexdec($match[1]);
}
return($body);
}
/**
* set the requested URL
*
* #param string $url URL of the requested page
*/
public function setRequestUrl($url)
{
$this->_request_url = $url;
}
/**
* set to return headers only
*
* #param boolean $headers_only true to return headers only,
* false to return headers and body
*/
public function returnHeadersOnly($headers_only)
{
$this->_headers_only = $headers_only;
}
/**
* set proxy host and port
*
* #param string $hostport proxy host and proxy port in format proxy_host:proxy_port
*/
public function setProxyHost($hostport)
{
list($this->_proxy_host, $this->_proxy_port) = explode(':', $hostport);
$this->_use_proxy = true;
}
/**
* set proxy user and password
*
* #param string $userpass proxy user and password in format proxy_user:proxy_password
*/
public function setProxyUser($userpass)
{
list($this->_proxy_user, $this->_proxy_pass) = explode(':', $userpass);
}
/**
* get the HTTP response status (200, 404, etc)
*
* #return string
*/
public function getStatus()
{
return($this->_status);
}
/**
* get the requested URL
*
* #return string
*/
public function getRequestUrl()
{
return($this->_request_url);
}
/**
* set maximum redirects
*
* #param int $maxredirs
*/
public function setMaxRedirs($maxredirs)
{
$this->_maxredirs = $maxredirs;
}
/**
* get HTTP response headers
*
* #return array
*/
public function getHeaders()
{
return($this->_resp_headers);
}
/**
* get the HTTP response body, usually in HTML
*
* #return string
*/
public function getContents()
{
return($this->_resp_body);
echo $this->_resp_body;
}
/**
* get error message
*
* #return string
*/
public function getErrorMessage()
{
return($this->_errmsg);
}
/**
* print debug information
*/
private function debug($text)
{
print "$text\n";
}
}
?>
The array _resp_headers doesn't have any element with key transfer_encoding.
To fix the notice, you should check if the array has the key transfer_encoding:
if (array_key_exists('transfer_encoding', $this->_resp_headers) && $this->_resp_headers['transfer_encoding'] == 'chunked') {
But I can't tell you why the key is not set and why the script has stopped working if you don't show more code.
difficult to answer with such a small sample of code but you can check for the index's existence before
if (isset($this->_resp_headers['transfer_encoding']) &&
$this->_resp_headers['transfer_encoding'] == 'chunked') {
I would like to take an array and break it up into chunks (closure $r accomplishes this)
/**
* Breaks an array into bits
*
* #param $list
* #param $p
* #return array
*/
$r = function ($list, $p)
{
$ll = count( $list ); $pl = floor( $ll / $p ); $pt = $ll % $p; $r = []; $m = 0;
for ($px = 0; $px < $p; $px++)
{
$inc = ($px < $pt) ? $pl + 1 : $pl; $r[$px] = array_slice( $list, $m, $inc );
$m += $inc;
}
return $r;
};
$hosts = [
'devhost-0',
'devhost-1',
'devhost-2',
'devhost-3',
'devhost-4',
'devhost-5',
'devhost-6',
'devhost-7',
'devhost-8',
'devhost-9',
'devhost-10',
'devhost-11',
'devhost-12',
'devhost-13',
'devhost-14',
'devhost-15',
'devhost-16',
'devhost-17',
'devhost-18',
'devhost-19',
'devhost-20',
'devhost-21',
'devhost-22',
'devhost-23',
'devhost-24',
'devhost-25',
'devhost-26',
'devhost-27',
'devhost-28',
'devhost-29',
'devhost-30',
'devhost-31',
'devhost-32',
'devhost-33',
'devhost-34',
'devhost-35',
'devhost-36',
];
$hosts = $r($hosts, 6); // chunks of six
This will break the above array into chunks of 5-6, from here I would like to run each of the chunks at the same time
with a simple method, lets say this ping closure below.
/**
* Polls host
*
* #param $host
* #param $port
* #param $timeout
* #return bool
*/
$ping = function ($host, $port, $timeout)
{
$errno = $errstr = false; // silence....
return (! #fSockOpen($host, $port, $errno, $errstr, $timeout))
? false
: true;
};
I'm not sure how to accomplish this though? I would assume that I would be using pcntl_fork() or pthreads but am uncertain as to how I should be setting this up? I have read multiple articles on running processes asynchronously but I am having issues finding one using an array as parts in an example.
i think this could work (based off threading example here http://acm.msu.ru/mkoshp/php-chunked-xhtml/pthreads.tutorials.html )
$hosts = [
'devhost-0',
'devhost-1',
'devhost-2',
'devhost-3',
'devhost-4',
'devhost-5',
'devhost-6',
'devhost-7',
'devhost-8',
'devhost-9',
'devhost-10',
'devhost-11',
'devhost-12',
'devhost-13',
'devhost-14',
'devhost-15',
'devhost-16',
'devhost-17',
'devhost-18',
'devhost-19',
'devhost-20',
'devhost-21',
'devhost-22',
'devhost-23',
'devhost-24',
'devhost-25',
'devhost-26',
'devhost-27',
'devhost-28',
'devhost-29',
'devhost-30',
'devhost-31',
'devhost-32',
'devhost-33',
'devhost-34',
'devhost-35',
'devhost-36',
];
class Devhost_worker extends Thread{
public $jobs;
public function __construct($jobs) {
$this->jobs = $jobs;
}
public function run() {
//do your stuff with the $this->jobs here.
}
}
$chunks=array_chunk($hosts,6,true);
$threads=array();
$i=0;
foreach($chunks as $chunk){
$threads[$i]=new Devhost_worker($chunk);
$threads[$i]->start();
++$i;
}
Something that I came up with just a while ago, will not accept but will post as a possible solution to my question. I was able to ping 4k of my servers in about 1 minute or so.
<?php
define('APP_DIR', dirname(__DIR__)); // autloads in non-sample script....
$hosts = explode("\n", file_get_contents(APP_DIR . '/data/test.txt'));
/**
* Processes in Parallel.
*
* Run a function (with no return result) on each item in an array in parallel.
* Note: This function is only useful if order is not important, and you don't
* need any return values from the function (i.e. no inter-process communication).
*
* #param mixed $func A closure function to apply to each item in parallel.
* #param array $arr The array to apply function to.
* #param integer $procs Number of processes to run in parallel.
*
* #return void
*/
function parallelize($func, array $arr, $procs = 4)
{
$chunks = array_chunk($arr, ceil((count($arr) / $procs)));
$pid = -1;
$children = [];
foreach ($chunks AS $items)
{
$pid = pcntl_fork();
switch ($pid)
{
case (-1): die ('Unable to fork');
case (0): // We are the child process. Pass a chunk of items to process.
array_walk($items, $func);
exit(0);
default: // We are the parent.
$children[] = $pid;
break;
}
}
// Wait for children to finish.
foreach ($children AS $pid)
{
// We are still the parent.
pcntl_waitpid($pid, $status);
}
}
/**
* Polls host
*
* #param $host
* #param $port
* #param $timeout
* #return bool
*/
$ping = function ($host, $port, $timeout)
{
$errno = $errstr = false; // silence....
return (! #fSockOpen($host, $port, $errno, $errstr, $timeout))
? false
: true;
};
/**
* Simple true false return on pinging...
*
* #param $host
* #return bool
*/
$tap = function ($host) USE ($ping)
{
if (! $ping($host, 22, 3))
{
echo "[fail] does not exist\n";
return false;
}
else
{
echo "[good] exists\n";
return true;
}
};
parallelize($tap, $hosts, 20); // do work....
I am trying to use the code from #eyecatchup (https://github.com/eyecatchup/php-webmaster-tools-downloads) to get data via the Google Webmaster Tools API. I am able to get "TOP_PAGES" and "TOP QUERIES" but that is it. What I really want is "EXTERNAL_LINKS" or even "LATEST_LINKS".
Here is the gwtdata.php code:
<pre>
<?php
/**
* PHP class for downloading CSV files from Google Webmaster Tools.
*
* This class does NOT require the Zend gdata package be installed
* in order to run.
*
* Copyright 2012 eyecatchUp UG. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* #author: Stephan Schmitz <eyecatchup#gmail.com>
* #link: https://code.google.com/p/php-webmaster-tools-downloads/
*/
class GWTdata
{
const HOST = "https://www.google.com";
const SERVICEURI = "/webmasters/tools/";
public $_language, $_tables, $_daterange, $_downloaded, $_skipped;
private $_auth, $_logged_in;
public function __construct()
{
$this->_auth = false;
$this->_logged_in = false;
$this->_language = "en";
$this->_daterange = array("","");
$this->_tables = array("TOP_PAGES", "TOP_QUERIES",
"CRAWL_ERRORS", "CONTENT_ERRORS", "CONTENT_KEYWORDS",
"INTERNAL_LINKS", "EXTERNAL_LINKS", "SOCIAL_ACTIVITY"
);
$this->_errTablesSort = array(0 => "http",
1 => "not-found", 2 => "restricted-by-robotsTxt",
3 => "unreachable", 4 => "timeout", 5 => "not-followed",
"kAppErrorSoft-404s" => "soft404", "sitemap" => "in-sitemaps"
);
$this->_errTablesType = array(0 => "web-crawl-errors",
1 => "mobile-wml-xhtml-errors", 2 => "mobile-chtml-errors",
3 => "mobile-operator-errors", 4 => "news-crawl-errors"
);
$this->_downloaded = array();
$this->_skipped = array();
}
/**
* Sets content language.
*
* #param $str String Valid ISO 639-1 language code, supported by Google.
*/
public function SetLanguage($str)
{
$this->_language = $str;
}
/**
* Sets features that should be downloaded.
*
* #param $arr Array Valid array values are:
* "TOP_PAGES", "TOP_QUERIES", "CRAWL_ERRORS", "CONTENT_ERRORS",
* "CONTENT_KEYWORDS", "INTERNAL_LINKS", "EXTERNAL_LINKS",
* "SOCIAL_ACTIVITY".
*/
public function SetTables($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) <= 2) {
$valid = array("TOP_PAGES","TOP_QUERIES","CRAWL_ERRORS","CONTENT_ERRORS",
"CONTENT_KEYWORDS","INTERNAL_LINKS","EXTERNAL_LINKS","SOCIAL_ACTIVITY");
$this->_tables = array();
for($i=0; $i < sizeof($arr); $i++) {
if(in_array($arr[$i], $valid)) {
array_push($this->_tables, $arr[$i]);
} else { throw new Exception("Invalid argument given."); }
}
} else { throw new Exception("Invalid argument given."); }
}
/**
* Sets daterange for download data.
*
* #param $arr Array Array containing two ISO 8601 formatted date strings.
*/
public function SetDaterange($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) == 2) {
if(self::IsISO8601($arr[0]) === true &&
self::IsISO8601($arr[1]) === true) {
$this->_daterange = array(str_replace("-", "", $arr[0]),
str_replace("-", "", $arr[1]));
return true;
} else { throw new Exception("Invalid argument given."); }
} else { throw new Exception("Invalid argument given."); }
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetDownloadedFiles()
{
return $this->_downloaded;
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetSkippedFiles()
{
return $this->_skipped;
}
/**
* Checks if client has logged into their Google account yet.
*
* #return Boolean Returns true if logged in, or false if not.
*/
private function IsLoggedIn()
{
return $this->_logged_in;
}
/**
* Attempts to log into the specified Google account.
*
* #param $email String User's Google email address.
* #param $pwd String Password for Google account.
* #return Boolean Returns true when Authentication was successful,
* else false.
*/
public function LogIn($email, $pwd)
{
$url = self::HOST . "/accounts/ClientLogin";
$postRequest = array(
'accountType' => 'HOSTED_OR_GOOGLE',
'Email' => $email,
'Passwd' => $pwd,
'service' => "sitemaps",
'source' => "Google-WMTdownloadscript-0.1-php"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postRequest);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
if($info['http_code'] == 200) {
preg_match('/Auth=(.*)/', $output, $match);
if(isset($match[1])) {
$this->_auth = $match[1];
$this->_logged_in = true;
return true;
} else { return false; }
} else { return false; }
}
/**
* Attempts authenticated GET Request.
*
* #param $url String URL for the GET request.
* #return Mixed Curl result as String,
* or false (Boolean) when Authentication fails.
*/
public function GetData($url)
{
if(self::IsLoggedIn() === true) {
$url = self::HOST . $url;
$head = array("Authorization: GoogleLogin auth=".$this->_auth,
"GData-Version: 2");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $head);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
return ($info['http_code']!=200) ? false : $result;
} else { return false; }
}
/**
* Gets all available sites from Google Webmaster Tools account.
*
* #return Mixed Array with all site URLs registered in GWT account,
* or false (Boolean) if request failed.
*/
public function GetSites()
{
if(self::IsLoggedIn() === true) {
$feed = self::GetData(self::SERVICEURI."feeds/sites/");
if($feed !== false) {
$sites = array();
$doc = new DOMDocument();
$doc->loadXML($feed);
foreach ($doc->getElementsByTagName('entry') as $node) {
array_push($sites,
$node->getElementsByTagName('title')->item(0)->nodeValue);
}
return $sites;
} else { return false; }
} else { return false; }
}
/**
* Gets the download links for an available site
* from the Google Webmaster Tools account.
*
* #param $url String Site URL registered in GWT.
* #return Mixed Array with keys TOP_PAGES and TOP_QUERIES,
* or false (Boolean) when Authentication fails.
*/
public function GetDownloadUrls($url)
{
if(self::IsLoggedIn() === true) {
$_url = sprintf(self::SERVICEURI."downloads-list?hl=%s&siteUrl=%s",
$this->_language,
urlencode($url));
$downloadList = self::GetData($_url);
return json_decode($downloadList, true);
} else { return false; }
}
/**
* Downloads the file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV($site, $savepath=".")
{
if(self::IsLoggedIn() === true) {
$downloadUrls = self::GetDownloadUrls($site);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$tables = $this->_tables;
foreach($tables as $table) {
if($table=="CRAWL_ERRORS") {
self::DownloadCSV_CrawlErrors($site, $savepath);
}
elseif($table=="CONTENT_ERRORS") {
self::DownloadCSV_XTRA($site, $savepath,
"html-suggestions", "\)", "CONTENT_ERRORS", "content-problems-dl");
}
elseif($table=="CONTENT_KEYWORDS") {
self::DownloadCSV_XTRA($site, $savepath,
"keywords", "\)", "CONTENT_KEYWORDS", "content-words-dl");
}
elseif($table=="INTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"internal-links", "\)", "INTERNAL_LINKS", "internal-links-dl");
}
elseif($table=="EXTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"external-links-domain", "\)", "EXTERNAL_LINKS", "external-links-domain-dl");
}
elseif($table=="SOCIAL_ACTIVITY") {
self::DownloadCSV_XTRA($site, $savepath,
"social-activity", "x26", "SOCIAL_ACTIVITY", "social-activity-dl");
}
else {
$finalName = "$savepath/$table-$filename.csv";
$finalUrl = $downloadUrls[$table] ."&prop=ALL&db=%s&de=%s&more=true";
$finalUrl = sprintf($finalUrl, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($finalUrl,$finalName);
}
}
} else { return false; }
}
/**
* Downloads "unofficial" downloads based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV_XTRA($site, $savepath=".", $tokenUri, $tokenDelimiter, $filenamePrefix, $dlUri)
{
if(self::IsLoggedIn() === true) {
$uri = self::SERVICEURI . $tokenUri . "?hl=%s&siteUrl=%s";
$_uri = sprintf($uri, $this->_language, $site);
$token = self::GetToken($_uri, $tokenDelimiter);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$finalName = "$savepath/$filenamePrefix-$filename.csv";
$url = self::SERVICEURI . $dlUri . "?hl=%s&siteUrl=%s&security_token=%s&prop=ALL&db=%s&de=%s&more=true";
$_url = sprintf($url, $this->_language, $site, $token, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($_url,$finalName);
} else { return false; }
}
/**
* Downloads the Crawl Errors file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional: Path to save CSV to (no trailing slash!).
* #param $separated Boolean Optional: If true, the method saves separated CSV files
* for each error type. Default: Merge errors in one file.
*/
public function DownloadCSV_CrawlErrors($site, $savepath=".", $separated=false)
{
if(self::IsLoggedIn() === true) {
$type_param = "we";
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
if($separated) {
foreach($this->_errTablesSort as $sortid => $sortname) {
foreach($this->_errTablesType as $typeid => $typename) {
if($typeid == 1) {
$type_param = "mx";
} else if($typeid == 2) {
$type_param = "mc";
} else {
$type_param = "we";
}
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$typename-$sortname-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=%s&sort=%s";
$_url = sprintf($url, $this->_language, $site, $token, $typeid, $sortid);
self::SaveData($_url,$finalName);
}
}
}
else {
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=0";
$_url = sprintf($url, $this->_language, $site, $token);
self::SaveData($_url,$finalName);
}
} else { return false; }
}
/**
* Saves data to a CSV file based on the given URL.
*
* #param $finalUrl String CSV Download URI.
* #param $finalName String Filepointer to save location.
*/
private function SaveData($finalUrl, $finalName)
{
$data = self::GetData($finalUrl);
if(strlen($data) > 1 && file_put_contents($finalName, utf8_decode($data))) {
array_push($this->_downloaded, realpath($finalName));
return true;
} else {
array_push($this->_skipped, $finalName);
return false;
}
}
/**
* Regular Expression to find the Security Token for a download file.
*
* #param $uri String A Webmaster Tools Desktop Service URI.
* #param $delimiter String Trailing delimiter for the regex.
* #return String Returns a security token.
*/
private function GetToken($uri, $delimiter)
{
$matches = array(); $tmp = self::get_data($uri); preg_match_all("#46security_token(.?)$delimiter#si", $tmp, $matches); return #substr($matches[1][0],3,-1);
}
/**
* Validates ISO 8601 date format.
*
* #param $str String Valid ISO 8601 date string (eg. 2012-01-01).
* #return Boolean Returns true if string has valid format, else false.
*/
private function IsISO8601($str)
{
$stamp = strtotime($str);
return (is_numeric($stamp) && checkdate(date('m', $stamp),
date('d', $stamp), date('Y', $stamp))) ? true : false;
}
}
?>
</pre>
And here is the code I am using to try to extract external links:
<pre>
<?php
include 'gwtdata.php';
try {
$email = "***#gmail.com";
$password = "***";
# If hardcoded, don't forget trailing slash!
$website = "***";
# Valid values are "TOP_PAGES", "TOP_QUERIES", "CRAWL_ERRORS",
# "CONTENT_ERRORS", "CONTENT_KEYWORDS", "INTERNAL_LINKS",
# "EXTERNAL_LINKS" and "SOCIAL_ACTIVITY".
$tables = array("EXTERNAL_LINKS");
$gdata = new GWTdata();
if($gdata->LogIn($email, $password) === true)
{
$gdata->SetTables($tables);
$gdata->DownloadCSV($website, "./csv");
}
$files = $gdata->GetDownloadedFiles();
foreach($files as $file)
{
print "Saved $file\n</a>";
}
} catch (Exception $e) {
die($e->getMessage());
}
?>
</pre>
Part of the author's answer here https://stackoverflow.com/a/16002159/624466, is the answer to your question too.
[..] this code is neither released by Google nor makes use of an official
API, but is rather a custom script processing data from the web
interface.
[..] there were some changes to the Google Webmaster Tools web
interface [..]. Thus,
it broke some functionality of the PHP class GWTdata
I have created application in dropbox developer account.
I am using this component class for dropbox .when i trying to login i am getting this error "Please create your dropbox_token and dropbox_token_secret fields in your user model."
<?php
/**
* CAKEPHP DROPBOX COMPONENT v0.4
* Connects Cakephp to Dropbox using cURL.
*
* Copyright (C) 2010 Kyle Robinson Young
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*
* #author Kyle Robinson Young <kyle at kyletyoung.com>
* #copyright 2010 Kyle Robinson Young
* #license http://www.opensource.org/licenses/mit-license.php The MIT License
* #version 0.4
* #link http://www.kyletyoung.com/code/cakephp_dropbox_component
*
* SETTINGS:
* email/password: To your dropbox account
* cache: Set to name of cache config or false for no cache
*
* When in doubt, clear the cache.
*
* TODO:
* Make sync function smarter (use modified).
*
*/
class DropboxComponent extends Object
{
var $email, $password;
var $loggedin = false;
var $post, $cookie = array();
var $cache = 'default';
var $_wcache = array();
/**
* INITIALIZE
* #param class $controller
* #param array $settings
*/
function initialize(&$controller, $settings=array())
{
if (!extension_loaded('curl'))
{
trigger_error('Dropbox Component: I require the cURL extension to work.');
} // no curl
if (empty($settings['email']) || empty($settings['password']))
{
trigger_error('Dropbox Component: I need your dropbox email and password to login.');
} // email|pass empty
else
{
$this->email = $settings['email'];
$this->password = $settings['password'];
if (isset($settings['cache'])) $this->cache = $settings['cache'];
$this->login();
} // else
} // initialize
/**
* UPLOAD
* Upload a local file to a remote folder.
*
* #param $file
* #param $dir
* #return bool
*/
function upload($from=null, $to='/')
{
if (!file_exists($from)) return false;
$data = $this->request('https://www.dropbox.com/home');
$token = $this->findOnDropbox('token_upload', $data);
if ($token === false) return false;
$this->post = array(
'plain' => 'yes',
'file' => '#'.$from,
'dest' => $to,
't' => $token
);
$data = $this->request('https://dl-web.dropbox.com/upload');
if (strpos($data, 'HTTP/1.1 302 FOUND') === false) return false;
return true;
} // upload
/**
* DOWNLOAD
* Download a remote file to a local folder.
* Both from and to must be a path to a file name.
*
* #param str $from
* #param str $to
* #param str $w
* #return bool
*/
function download($from=null, $to=null, $w=null)
{
$data = $this->file($from, $w);
if (empty($data['data'])) return false;
if (!is_writable(dirname($to))) return false;
if (!$fp = fopen($to, 'w')) return false;
if (fwrite($fp, $data['data']) === false) return false;
fclose($fp);
return true;
} // download
/**
* SYNC
* Compares files from the local and remote folders
* then syncs them.
* Both local and remote must be folders.
*
* TODO:
* Currently only checks if files exists. Doesn't
* check if they are up to date which it should.
*
* #param str $local
* #param str $remote
* #return bool
*/
function sync($local=null, $remote=null)
{
if (!is_dir($local)) return false;
// GET REMOTE FILES
$remote_files = $this->files($remote);
// GET LOCAL FILES
$local_files = array();
$d = dir($local);
while (false !== ($entry = $d->read()))
{
if (substr($entry, 0, 1) == '.') continue;
if (is_dir($local.DS.$entry)) continue;
$local_files[] = $entry;
} // while
$d->close();
// DOWNLOAD FILES
$tmp = array();
foreach ($remote_files as $file)
{
if (empty($file['w'])) continue;
$tmp[] = $file['name'];
if (in_array($file['name'], $local_files)) continue;
$this->download($file['path'].$file['name'], $local.$file['name'], $file['w']);
} // foreach
// UPLOAD FILES
foreach ($local_files as $file)
{
if (in_array($file, $tmp)) continue;
$this->upload($local.$file, $remote);
} // foreach
return true;
} // sync
/**
* FILES
* Returns an array of remote files/folders
* within the given dir param.
*
* #param str $dir
* #return array
*/
function files($dir='/')
{
$dir = $this->escape($dir);
if ($this->cache === false) Cache::delete('dropbox_files_'.$dir, $this->cache);
if (($files = Cache::read('dropbox_files_'.$dir, $this->cache)) === false)
{
$files = array();
$data = $this->request('https://www.dropbox.com/browse_plain/'.$dir.'?no_js=true');
// GET FILES
$matches = $this->findOnDropbox('files', $data);
if ($matches === false) return false;
// GET TYPES
$types = $this->findOnDropbox('file_types', $data);
// GET SIZES
$sizes = $this->findOnDropbox('file_sizes', $data);
// GET MODS
$mods = $this->findOnDropbox('file_modified_dates', $data);
$i = 0;
foreach ($matches as $key => $file)
{
// IF PARENT
if (strpos($file, "Parent folder") !== false) continue;
// GET FILENAME
$found = $this->findOnDropbox('filename', $file);
if ($found === false) continue;
$found = parse_url($found);
$filename = pathinfo($found['path']);
$filename = $filename['basename'];
if (empty($filename)) continue;
// SET DEFAULTS
$path = $dir.$filename;
$type = 'unknown';
$size = 0;
$modified = 0;
// GET TYPE
if (!empty($types[$key])) $type = trim($types[$key]);
// GET SIZE
if (!empty($sizes[$key])) $size = trim($sizes[$key]);
// GET MODIFIED
if (!empty($mods[$key])) $modified = trim($mods[$key]);
// ADD TO FILES
$files[$i] = array(
'path' => urldecode($dir),
'name' => $filename,
'type' => $type,
'size' => $size,
'modified' => $modified
);
// IF FILE OR FOLDER - FILES HAVE W
$w = $this->findOnDropbox('w', $file);
if ($w !== false)
{
$files[$i]['w'] = $w;
// SAVE W FOR LATER
$this->_wcache[$dir.'/'.$filename] = $w;
} // !empty
$i++;
} // foreach
} // Cache::read
if ($this->cache !== false)
{
Cache::write('dropbox_files_'.$dir, $files, $this->cache);
} // if cache
return $files;
} // files
/**
* FILE
* Returns a remote file as an array.
*
* #param str $file
* #param str $w
* #return array
*/
function file($file=null, $w=null)
{
$file = $this->escape($file);
if ($this->cache === false) Cache::delete('dropbox_file_'.$file, $this->cache);
if (($out = Cache::read('dropbox_file_'.$file, $this->cache)) === false)
{
if (empty($w))
{
if (!empty($this->_wcache[$file])) $w = $this->_wcache[$file];
else return false;
} // empty w
$data = $this->request('https://dl-web.dropbox.com/get/'.$file.'?w='.$w);
$type = $this->findOnDropbox('content_type', $data);
$data = substr(stristr($data, "\r\n\r\n"), 4);
if (!empty($type[0])) $type = $type[0];
$out = array(
'path' => $file,
'w' => $w,
'data' => $data,
'content_type' => $type
);
if ($this->cache !== false)
{
Cache::write('dropbox_file_'.$file, $out, $this->cache);
} // if cache
} // Cache::read
return $out;
} // file
/**
* LOGIN
* to dropbox
*
* #return bool
*/
function login()
{
if (!$this->loggedin)
{
if (empty($this->email) || empty($this->password)) return false;
$data = $this->request('https://www.dropbox.com/login');
// GET TOKEN
$token = $this->findOnDropbox('token_login', $data);
if ($token === false) return false;
// LOGIN TO DROPBOX
$this->post = array(
'login_email' => $this->email,
'login_password' => $this->password,
't' => $token
);
$data = $this->request('https://www.dropbox.com/login');
// IF WERE HOME
if (stripos($data, 'location: /home') === false) return false;
$this->loggedin = true;
} // if loggedin
return true;
} // login
/**
* REQUEST
* Returns data from given url and
* saves cookies. Use $this->post and
* $this->cookie to submit params.
*
* #param str $url
* #return str
*/
function request($url=null)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// IF POST
if (!empty($this->post))
{
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $this->post);
$this->post = array();
} // !empty
// IF COOKIES
if (!empty($this->cookie))
{
$cookies = array();
foreach ($this->cookie as $key => $val)
{
$cookies[] = "$key=$val";
} // foreach
$cookies = implode(';', $cookies);
curl_setopt($ch, CURLOPT_COOKIE, $cookies);
} // !empty
// GET DATA
$data = curl_exec($ch);
// SAVE COOKIES
$cookies = $this->findOnDropbox('cookies', $data);
if ($cookies !== false)
{
$this->cookie = array_merge($this->cookie, $cookies);
} // if cookies
curl_close($ch);
return $data;
} // request
/**
* ESCAPE
* Returns a dropbox friendly str
* for a url
*
* #param str $str
* #return str
*/
function escape($str=null)
{
return str_replace(
array('+','_','%2E','-','%2F','%3A'),
array('%20','%5F','.','%2D','/',':'),
urlencode($str)
);
} // escape
/**
* FIND ON DROPBOX
* A single function for parsing data from
* Dropbox. For easy update when/if Dropbox
* updates their html.
*
* #param str $key
* #param str $data
* #return mixed
*/
function findOnDropbox($key=null, $data=null)
{
switch (strtolower($key))
{
// FIND FILES & NAMES
case 'files':
preg_match_all('/<div.*details-filename.*>(.*?)<\/div>/i', $data, $matches);
if (!empty($matches[0])) return $matches[0];
break;
// FIND FILE TYPES
case 'file_types':
preg_match_all('/<div.*details-icon.*>(<img.*class="sprite s_(.*)".*>)<\/div>/i', $data, $matches);
if (!empty($matches[2])) return $matches[2];
break;
// FIND FILE SIZES
case 'file_sizes':
preg_match_all('/<div.*details-size.*>(.*)<\/div>/i', $data, $matches);
if (!empty($matches[1])) return $matches[1];
break;
// FIND FILE MODIFIED DATES
case 'file_modified_dates':
preg_match_all('/<div.*details-modified.*>(.*)<\/div>/i', $data, $matches);
if (!empty($matches[1])) return $matches[1];
break;
// FIND FILE NAME
case 'filename':
preg_match('/href=[("|\')]([^("|\')]+)/i', $data, $match);
if (!empty($match[1])) return $match[1];
break;
// FIND W
case 'w':
preg_match('/\?w=(.[^"]*)/i', $data, $match);
if (!empty($match[1])) return $match[1];
break;
// FIND CONTENT TYPE
case 'content_type':
preg_match('/Content-Type: .+\/.+/i', $data, $type);
if (!empty($type)) return $type;
break;
// FIND COOKIES
case 'cookies':
preg_match_all('/Set-Cookie: ([^=]+)=(.*?);/i', $data, $matches);
$return = array();
foreach ($matches[1] as $key => $val)
{
$return[(string)$val] = $matches[2][$key];
} // foreach
if (!empty($return)) return $return;
break;
// FIND LOGIN FORM TOKEN
case 'token_login':
preg_match('/<form [^>]*\/login[^>]*>.*?<\/form>/si', $data, $match);
if (!empty($match[0]))
{
preg_match('/<input [^>]*name="t" [^>]*value="(.*?)"[^>]*>/si', $match[0], $match);
if (!empty($match[1])) return $match[1];
} // !empty
break;
// FIND UPLOAD FORM TOKEN
case 'token_upload':
preg_match('/<form [^>]*https\:\/\/dl-web\.dropbox\.com\/upload[^>]*>.*?<\/form>/si', $data, $match);
if (!empty($match[0]))
{
preg_match('/<input [^>]*name="t" [^>]*value="(.*?)"[^>]*>/si', $match[0], $match);
if (!empty($match[1])) return $match[1];
} // !empty
break;
} // switch
return false;
} // findOnDropbox
} // DropboxComponent
?>
Controller code
class DropboxWebserverController extends AppController
{
var $name = 'DropboxWebserver';
var $uses = array();
var $autoRender = false;
var $components = array('Dropbox' => array(
'email' => 'email#gmail.com',
'password' => 'password',
//'cache' => false
));
var $root_folder = '/';
var $default_home = array('index.html', 'index.htm', 'index.php');
/**
* INDEX
*/
function index()
{
$args = func_get_args();
$args = implode('/', $args);
$path = pathinfo($args);
if ($path['dirname'] == ".")
{
$folder = $path['basename'];
$file = '';
} // dirname == .
else
{
$folder = $path['dirname'];
$file = $path['basename'];
} // else
$files = $this->Dropbox->files($this->root_folder.$folder);
//debug($files);
// FIND FILE
foreach ($files as $f)
{
if (strpos($f['type'], 'folder') !== false) continue;
if (empty($f['name'])) continue;
if ($f['name'] == $file)
{
$file = $this->Dropbox->file($this->root_folder.$folder.'/'.$file, $f['w']);
$output = $file['data'];
$content_type = $file['content_type'];
break;
} // name == file
// FIND DEFAULT HOME
if (in_array($f['name'], $this->default_home))
{
$default = $f;
} // in_array
} // foreach
if (!empty($output))
{
header('Content-Type: '.$content_type);
echo $output;
} // !empty
elseif (!empty($default))
{
$file = $this->Dropbox->file($this->root_folder.$folder.'/'.$default['name'], $default['w']);
header('Content-Type: '.$file['content_type']);
echo $file['data'];
} // !empty default
else
{
echo 'Error 404: File Not Found';
} // else
} // index
}
How to login and access drop box account using cakephp
A default error logging using vqmod is as follows:
---------- Date: 2012-10-09 19:46:06 ~ IP : 127.0.0.1 ----------
REQUEST URI : /oc/
MOD DETAILS:
modFile : C:\wamp\www\oc\vqmod\xml\templace.xml
id : Template
version : 1.5.2 - 1.5.2.1
vqmver : 1.0.8
author : templace.com
SEARCH NOT FOUND (ABORTING MOD): require_once(foo . 'library/template.php');
----------------------------------------------------------------------
Example vqmod causing the error
<modification>
<id>Templace</id>
<version>1.5.2 - 1.5.2.1</version>
<author>templace.com</author>
<vqmver>1.0.8</vqmver>
<file name="system/startup.php">
<operation>
<search position="before"><![CDATA[
require_once(foo . 'library/template.php');
]]></search>
<add><![CDATA[
require_once(DIR_SYSTEM . 'library/templace.php');
]]></add>
</operation>
</file>
</modification>
To resole this issue I would have to open the vqmod file templace.xml and search for the file[name] this error is referring too.
QUESTION: How could I add the parent file[name] the actual error is referring too?
E.g adding: "system/startup.php" to the error message to make it easier to debug.
vqmod.php
/**
* VQMod
* #description Main Object used
*/
final class VQMod {
private $_vqversion = '2.1.7';
private $_modFileList = array();
private $_mods = array();
private $_filesModded = array();
private $_cwd = '';
private $_doNotMod = array();
private $_virtualMode = true;
public $useCache = false;
public $logFilePath = 'vqmod/vqmod.log';
public $vqCachePath = 'vqmod/vqcache/';
public $protectedFilelist = 'vqmod/vqprotect.txt';
public $logging = true;
public $cacheTime = 5; // local=5secs live=60secs
public $log;
/**
* VQMod::__construct()
*
* #param bool $path File path to use
* #param bool $logging Enable/disabled logging
* #return null
* #description Startup of VQMod
*/
public function __construct($path = false, $logging = true) {
if(!class_exists('DOMDocument')) {
die('ERROR - YOU NEED DOMDocument INSTALLED TO USE VQMod');
}
if(!$path){
$path = dirname(dirname(__FILE__));
}
$this->_setCwd($path);
$this->logging = (bool) $logging;
$this->log = new VQModLog($this);
$this->_getMods();
$this->_loadProtected();
}
/**
* VQMod::modCheck()
*
* #param string $sourceFile path for file
* #return string
* #description Checks if a file has modifications and applies them, returning cache files or the file name
*/
public function modCheck($sourceFile) {
if(!preg_match('%^([a-z]:)?[\\\\/]%i', $sourceFile)) {
$sourcePath = $this->path($sourceFile);
} else {
$sourcePath = realpath($sourceFile);
}
if(!$sourcePath || is_dir($sourcePath) || in_array($sourcePath, $this->_doNotMod)) {
return $sourceFile;
}
$stripped_filename = preg_replace('~^' . preg_quote($this->getCwd(), '~') . '~', '', $sourcePath);
$cacheFile = $this->_cacheName($stripped_filename);
if($this->useCache && file_exists($cacheFile)) {
//return $cacheFile; // useCache being Deprecated in favor of cacheTime
}
if(isset($this->_filesModded[$sourcePath])) {
return $this->_filesModded[$sourcePath]['cached'] ? $cacheFile : $sourceFile;
}
$changed = false;
$fileHash = sha1_file($sourcePath);
$fileData = file_get_contents($sourcePath);
foreach($this->_mods as $modObject) {
foreach($modObject->mods as $path => $mods) {
if($this->_checkMatch($path, $sourcePath)) {
$modObject->applyMod($mods, $fileData);
}
}
}
// START QPHORIA CACHELOCK CODE
//
if (sha1($fileData) != $fileHash) {
$writePath = $cacheFile;
$cacheLock = false;
if(file_exists($writePath) && ((filemtime($writePath) + (float)$this->cacheTime) >= time())) {
$cacheLock = true;
$changed = true;
}
if(!$cacheLock && (!file_exists($writePath) || is_writable($writePath))) {
file_put_contents($writePath, $fileData);
$changed = true;
} else {
//file_put_contents('./cachelock.txt', "$writePath \r\n", FILE_APPEND); // debugging only.
}
//file_put_contents('./cachetotal.txt', "$writePath \r\n", FILE_APPEND);
} // END QPHORIA CACHELOCK CODE
/* Original Code
if(sha1($fileData) != $fileHash) {
$writePath = $this->_virtualMode ? $cacheFile : $sourcePath;
if(!file_exists($writePath) || is_writable($writePath)) {
file_put_contents($writePath, $fileData);
$changed = true;
}
}*/
$this->_filesModded[$sourcePath] = array('cached' => $changed);
return $changed ? $writePath : $sourcePath;
}
/**
* VQMod::path()
*
* #param string $path File path
* #param bool $skip_real If true path is full not relative
* #return bool, string
* #description Returns the full true path of a file if it exists, otherwise false
*/
public function path($path, $skip_real = false) {
$tmp = $this->_cwd . $path;
$realpath = $skip_real ? $tmp : realpath($tmp);
if(!$realpath) {
return false;
}
if(is_dir($realpath)) {
$realpath = rtrim($realpath, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR;
}
return $realpath;
}
/**
* VQMod::getCwd()
*
* #return string
* #description Returns current working directory
*/
public function getCwd() {
return $this->_cwd;
}
/**
* VQMod::_getMods()
*
* #return null
* #description Gets list of XML files in vqmod xml folder for processing
*/
private function _getMods() {
$this->_modFileList = glob($this->path('vqmod/xml/') . '*.xml');
if($this->_modFileList) {
$this->_parseMods();
} else {
$this->log->write('NO MODS IN USE');
}
}
/**
* VQMod::_parseMods()
*
* #return null
* #description Loops through xml files and attempts to load them as VQModObject's
*/
private function _parseMods() {
$dom = new DOMDocument('1.0', 'UTF-8');
foreach($this->_modFileList as $modFileKey => $modFile) {
if(file_exists($modFile)) {
if(#$dom->load($modFile)) {
$mod = $dom->getElementsByTagName('modification')->item(0);
$this->_mods[] = new VQModObject($mod, $modFile, $this);
} else {
$this->log->write('DOM UNABLE TO LOAD: ' . $modFile);
}
} else {
$this->log->write('FILE NOT FOUND: ' . $modFile);
}
}
}
/**
* VQMod::_loadProtected()
*
* #return null
* #description Loads protected list and adds them to _doNotMod array
*/
private function _loadProtected() {
$file = $this->path($this->protectedFilelist);
if($file && is_file($file)) {
$protected = file_get_contents($file);
if(!empty($protected)) {
$protected = preg_replace('~\r?\n~', "\n", $protected);
$paths = explode("\n", $protected);
foreach($paths as $path) {
$fullPath = $this->path($path);
if($fullPath && !in_array($fullPath, $this->_doNotMod)) {
$this->_doNotMod[] = $fullPath;
}
}
}
}
}
/**
* VQMod::_cacheName()
*
* #param string $file Filename to be converted to cache filename
* #return string
* #description Returns cache file name for a path
*/
private function _cacheName($file) {
return $this->path($this->vqCachePath) . 'vq2-' . preg_replace('~[/\\\\]+~', '_', $file);
}
/**
* VQMod::_setCwd()
*
* #param string $path Path to be used as current working directory
* #return null
* #description Sets the current working directory variable
*/
private function _setCwd($path) {
$realpath = realpath($path);
if(!$realpath) {
die('COULDNT RESOLVE CWD REALPATH');
}
$this->_cwd = rtrim($realpath, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR;
}
/**
* VQMod::_checkMatch()
*
* #param string $modFilePath Modification path from a <file> node
* #param string $checkFilePath File path
* #return bool
* #description Checks a modification path against a file path
*/
private function _checkMatch($modFilePath, $checkFilePath) {
$modFilePath = str_replace('\\', '/', $modFilePath);
$checkFilePath = str_replace('\\', '/', $checkFilePath);
$modFilePath = preg_replace('/([^*]+)/e', 'preg_quote("$1", "~")', $modFilePath);
$modFilePath = str_replace('*', '[^/]*', $modFilePath);
$return = (bool) preg_match('~^' . $modFilePath . '$~', $checkFilePath);
return $return;
}
}
/**
* VQModLog
* #description Object to log information to a file
*/
class VQModLog {
private $_sep;
private $_vqmod;
private $_defhash = 'da39a3ee5e6b4b0d3255bfef95601890afd80709';
private $_logs = array();
/**
* VQModLog::__construct()
*
* #param VQMod $vqmod VQMod main class as reference
* #return null
* #description Object instantiation method
*/
public function __construct(VQMod $vqmod) {
$this->_vqmod = $vqmod;
$this->_sep = str_repeat('-', 70);
}
/**
* VQModLog::__destruct()
*
* #return null
* #description Logs any messages to the log file just before object is destroyed
*/
public function __destruct() {
if(empty($this->_logs) || $this->_vqmod->logging == false) {
return;
}
$txt = array();
$txt[] = str_repeat('-', 10) . ' Date: ' . date('Y-m-d H:i:s') . ' ~ IP : ' . (isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : 'N/A') . ' ' . str_repeat('-', 10);
$txt[] = 'REQUEST URI : ' . $_SERVER['REQUEST_URI'];
foreach($this->_logs as $count => $log) {
if($log['obj']) {
$vars = get_object_vars($log['obj']);
$txt[] = 'MOD DETAILS:';
foreach($vars as $k => $v) {
if(is_string($v)) {
$txt[] = ' ' . str_pad($k, 10, ' ', STR_PAD_RIGHT) . ': ' . $v;
}
}
}
foreach($log['log'] as $msg) {
$txt[] = $msg;
}
if ($count > count($this->_logs)-1) {
$txt[] = '';
}
}
$txt[] = $this->_sep;
$txt[] = str_repeat(PHP_EOL, 2);
$logPath = $this->_vqmod->path($this->_vqmod->logFilePath, true);
if(!file_exists($logPath)) {
$res = file_put_contents($logPath, '');
if($res === false) {
die('COULD NOT WRITE TO LOG FILE');
}
}
file_put_contents($logPath, implode(PHP_EOL, $txt), FILE_APPEND);
}
/**
* VQModLog::write()
*
* #param string $data Text to be added to log file
* #param VQModObject $obj Modification the error belongs to
* #return null
* #description Adds error to log object ready to be output
*/
public function write($data, VQModObject $obj = NULL) {
if($obj) {
$hash = sha1($obj->id);
} else {
$hash = $this->_defhash;
}
if(empty($this->_logs[$hash])) {
$this->_logs[$hash] = array(
'obj' => $obj,
'log' => array()
);
}
$this->_logs[$hash]['log'][] = $data;
}
}
/**
* VQModObject
* #description Object for the <modification> that orchestrates each applied modification
*/
class VQModObject {
public $modFile = '';
public $id = '';
public $version = '';
public $vqmver = '';
public $author = '';
public $mods = array();
private $_vqmod;
private $_skip = false;
/**
* VQModObject::__construct()
*
* #param DOMNode $node <modification> node
* #param string $modFile File modification is from
* #param VQMod $vqmod VQMod object as reference
* #return null
* #description Loads modification meta information
*/
public function __construct(DOMNode $node, $modFile, VQmod $vqmod) {
if($node->hasChildNodes()) {
foreach($node->childNodes as $child) {
$name = (string) $child->nodeName;
if(isset($this->$name)) {
$this->$name = (string) $child->nodeValue;
}
}
}
$this->modFile = $modFile;
$this->_vqmod = $vqmod;
$this->_parseMods($node);
}
/**
* VQModObject::skip()
*
* #return bool
* #description Returns the skip status of a modification
*/
public function skip() {
return $this->_skip;
}
/**
* VQModObject::applyMod()
*
* #param array $mods Array of search add nodes
* #param string $data File contents to be altered
* #return null
* #description Applies all modifications to the text data
*/
public function applyMod($mods, &$data) {
if($this->_skip) return;
$tmp = $data;
foreach($mods as $mod) {
$indexCount = 0;
$tmp = $this->_explodeData($tmp);
$lineMax = count($tmp) - 1;
switch($mod['search']->position) {
case 'top':
$tmp[$mod['search']->offset] = $mod['add']->getContent() . $tmp[$mod['search']->offset];
break;
case 'bottom':
$offset = $lineMax - $mod['search']->offset;
if($offset < 0){
$tmp[-1] = $mod['add']->getContent();
} else {
$tmp[$offset] .= $mod['add']->getContent();
}
break;
case 'all':
$tmp = array($mod['add']->getContent());
break;
default:
$changed = false;
foreach($tmp as $lineNum => $line) {
if($mod['search']->regex == 'true') {
$pos = #preg_match($mod['search']->getContent(), $line);
if($pos === false) {
if($mod['error'] == 'log' || $mod['error'] == 'abort' ) {
$this->_vqmod->log->write('INVALID REGEX ERROR - ' . $mod['search']->getContent(), $this);
}
continue 2;
} elseif($pos == 0) {
$pos = false;
}
} else {
$pos = strpos($line, $mod['search']->getContent());
}
if($pos !== false) {
$indexCount++;
$changed = true;
if(!$mod['search']->indexes() || ($mod['search']->indexes() && in_array($indexCount, $mod['search']->indexes()))) {
switch($mod['search']->position) {
case 'before':
$offset = ($lineNum - $mod['search']->offset < 0) ? -1 : $lineNum - $mod['search']->offset;
$tmp[$offset] = empty($tmp[$offset]) ? $mod['add']->getContent() : $mod['add']->getContent() . "\n" . $tmp[$offset];
break;
case 'after':
$offset = ($lineNum + $mod['search']->offset > $lineMax) ? $lineMax : $lineNum + $mod['search']->offset;
$tmp[$offset] = $tmp[$offset] . "\n" . $mod['add']->getContent();
break;
default:
if(!empty($mod['search']->offset)) {
for($i = 1; $i <= $mod['search']->offset; $i++) {
if(isset($tmp[$lineNum + $i])) {
$tmp[$lineNum + $i] = '';
}
}
}
if($mod['search']->regex == 'true') {
$tmp[$lineNum] = preg_replace($mod['search']->getContent(), $mod['add']->getContent(), $line);
} else {
$tmp[$lineNum] = str_replace($mod['search']->getContent(), $mod['add']->getContent(), $line);
}
break;
}
}
}
}
if(!$changed) {
$skip = ($mod['error'] == 'skip' || $mod['error'] == 'log') ? ' (SKIPPED)' : ' (ABORTING MOD)';
if($mod['error'] == 'log' || $mod['error'] == 'abort') {
$this->_vqmod->log->write('SEARCH NOT FOUND' . $skip . ': ' . $mod['search']->getContent(), $this);
}
if($mod['error'] == 'abort') {
$this->_skip = true;
return;
}
}
break;
}
ksort($tmp);
$tmp = $this->_implodeData($tmp);
}
$data = $tmp;
}
/**
* VQModObject::_parseMods()
*
* #param DOMNode $node <modification> node to be parsed
* #return null
* #description Parses modifications in preparation for the applyMod method to work
*/
private function _parseMods(DOMNode $node){
$files = $node->getElementsByTagName('file');
foreach($files as $file) {
$fileToMod = $file->getAttribute('name');
$error = ($file->hasAttribute('error')) ? $file->getAttribute('error') : 'log';
$fullPath = $this->_vqmod->path($fileToMod);
if(!$fullPath){
if(strpos($fileToMod, '*') !== false) {
$fullPath = $this->_vqmod->getCwd() . $fileToMod;
} else {
if ($error == 'log' || $error == 'abort') {
$skip = ($error == 'log') ? ' (SKIPPED)' : ' (ABORTING MOD)';
$this->_vqmod->log->write('Could not resolve path for [' . $fileToMod . ']' . $skip, $this);
}
if ($error == 'log' || $error == 'skip') {
continue;
} elseif ($error == 'abort') {
return false;
}
}
}
$operations = $file->getElementsByTagName('operation');
foreach($operations as $operation) {
$error = ($operation->hasAttribute('error')) ? $operation->getAttribute('error') : 'abort';
$this->mods[$fullPath][] = array(
'search' => new VQSearchNode($operation->getElementsByTagName('search')->item(0)),
'add' => new VQAddNode($operation->getElementsByTagName('add')->item(0)),
'error' => $error
);
}
}
}
/**
* VQModObject::_explodeData()
*
* #param string $data File contents
* #return string
* #description Splits a file into an array of individual lines
*/
private function _explodeData($data) {
return explode("\n", $data);
}
/**
* VQModObject::_implodeData()
*
* #param array $data Array of lines
* #return string
* #description Joins an array of lines back into a text file
*/
private function _implodeData($data) {
return implode("\n", $data);
}
}
/**
* VQNode
* #description Basic node object blueprint
*/
class VQNode {
public $trim = 'false';
private $_content = '';
/**
* VQNode::__construct()
*
* #param DOMNode $node Search/add node
* #return null
* #description Parses the node attributes and sets the node property
*/
public function __construct(DOMNode $node) {
$this->_content = $node->nodeValue;
if($node->hasAttributes()) {
foreach($node->attributes as $attr) {
$name = $attr->nodeName;
if(isset($this->$name)) {
$this->$name = $attr->nodeValue;
}
}
}
}
/**
* VQNode::getContent()
*
* #return string
* #description Returns the content, trimmed if applicable
*/
public function getContent() {
$content = ($this->trim == 'true') ? trim($this->_content) : $this->_content;
return $content;
}
}
/**
* VQSearchNode
* #description Object for the <search> xml tags
*/
class VQSearchNode extends VQNode {
public $position = 'replace';
public $offset = 0;
public $index = 'false';
public $regex = 'false';
public $trim = 'true';
/**
* VQSearchNode::indexes()
*
* #return bool, array
* #description Returns the index values to use the search on, or false if none
*/
public function indexes() {
if($this->index == 'false') {
return false;
}
$tmp = explode(',', $this->index);
foreach($tmp as $k => $v) {
if(!is_int($v)) {
unset($k);
}
}
$tmp = array_unique($tmp);
return empty($tmp) ? false : $tmp;
}
}
/**
* VQAddNode
* #description Object for the <add> xml tags
*/
class VQAddNode extends VQNode {
}
Also couple of other ideas to make debugging even easier:
List any other vqmod files which have previously edited this same file.
This is another common issue where I find when two extensions are editing the same file and the latter is causing the error but it would be useful to know about any other vqmods editing the same file. Yes I suppose I could add error="skip" to everything but dont think this is the best approach to just hide all of the errors, the user should be made aware there is an error...
"Suggested Fix", maybe some smart way you can test what type of error it is.
Contradict what I said above but even at its most basic form you could suggest hiding the error if its not essential. So that anybody can read it and understand how it fix it.
E.g
OPEN: vqmod/xml/templace.xml (line:23)
FIND: <operation>
REPLACE <operation error="skip">
Adding the line number in the XML file the error is coming from. It would be lovely not having to search all of the time and could quickly go to the line number in the vqmod
The issue for the file being edited is certainly one that is way overdue and one I plan on adding in the next release of vQmod. As for the other suggestions
Interesting idea, and one that could certainly be considered. The only problem I see with this is that it would possibly make some log files enormous
This is going to be next to impossible to incorporate
This is impossible without some pretty expensive runtime. The error doesn't lie in the XML as such, so would require re-opening the xml that's been parsed, searching for the line in question line by line and then reporting that. it sounds simple, but you have to remember that xml's can have the same search parameter for multiple operations - so in that situation you'd be no better off than searching the file yourself