Undefined index: transfer_encoding - php

Ok a few weeks ago my script stopped working and today I notice I'm getting an error: [05-Mar-2017 06:31:32 America/Denver] PHP Notice: Undefined index: transfer_encoding in /home2/website/public_html/maps_apps/EasyWebFetch.php on line 105.
Here is what line 105 is:
if ($this->_resp_headers['transfer_encoding'] == 'chunked') {
Can someone point me in the right direction on getting this fixed?
Here is the main code:
<?php
require_once '/home2/website/public_html/maps_apps/EasyWebFetch.php';
$callback = isset($_GET['callback']) ? $_GET['callback'] : 'mymap.weatherhandler';
$station = isset($_GET['rid']) ? $_GET['rid'] : 'FWS';
$product = isset($_GET['product']) ? $_GET['product'] : 'NCR';
$nframes = isset($_GET['frames']) ? $_GET['frames'] : 10;
if (strlen($product) != 3 || strlen($station) != 3) { exit; }
// fetch directory listing
$wf = new EasyWebFetch;
if (!$wf->get("https://radar.weather.gov/ridge/RadarImg/$product/$station/")) {
print $wf->getErrorMessage();
exit;
}
$page = $wf->getContents();
echo $page."\n\n";
$size = preg_match_all( "/href=\"({$station}[\d_]+{$product}\.gif)\"/" , $page, $match);
$files = $match[1];
if ($nframes == 'all') { $nframes = count($files); }
$nframes = min(count($files), $nframes);
$files = array_slice($files, -$nframes);
echo $callback."\n(\n{\ndirectory:\n[\n";
for ($i=0; $i < $nframes; $i++) {
echo "\"ridge/RadarImg/$product/$station/$files[$i]\",\n";
}
echo "]\n}\n)\n;"
?>
and here is EasyWebFetch.php
<?php
/*
* EasyWebFetch - Fetch a page by opening socket connection, no dependencies
*
* PHP version 5
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* #author Nashruddin Amin <me#nashruddin.com>
* #copyright Nashruddin Amin 2008
* #license GNU General Public License 3.0
* #package EasyWebFetch
* #version 1.1
*/
class EasyWebFetch
{
private $_request_url;
private $_host;
private $_path;
private $_query;
private $_fragment;
private $_headers_only;
private $_portnum = 80;
private $_user_agent = "SimpleHttpClient/3.0";
private $_req_timeout = 30;
private $_maxredirs = 5;
private $_use_proxy = false;
private $_proxy_host;
private $_proxy_port;
private $_proxy_user;
private $_proxy_pass;
private $_status;
private $_resp_headers;
private $_resp_body;
private $_is_error;
private $_errmsg;
/**
* class constructor
*/
public function __construct()
{
$this->_resp_headers = array();
$this->_resp_body = "";
}
/**
* get the requested page
*
* #param string $url URL of the requested page
* #param boolean $headers_only true to return headers only,
* false to return headers and body
*
* #return boolean true on success, false on failure
*/
public function get($url = '', $headers_only = false)
{
$this->_request_url = $url;
$this->_headers_only = $headers_only;
$redir = 0;
while(($redir++) <= $this->_maxredirs) {
$this->parseUrl($this->_request_url);
if (($response = $this->makeRequest()) == false) {
return(false);
}
/* split head and body */
$neck = strpos($response, "\r\n\r\n");
$head = substr($response, 0, $neck);
$body = substr($response, $neck+2);
/* read response headers */
$this->_resp_headers = $this->parseHeaders($head);
/* check for redirects */
if ($this->getStatus() == 301 || $this->getStatus() == 302) {
$follow = $this->_resp_headers['location'];
$this->_request_url = $this->setFullPath($follow, $this->_request_url);
continue;
} else {
/* no redirects, start reading response body */
break;
}
}
/* read the body part */
if ($this->_resp_headers['transfer_encoding'] == 'chunked') {
$this->_resp_body = $this->joinChunks($body);
} else {
$this->_resp_body = $body;
}
return(true);
}
/**
* build HTTP header and perform HTTP request
*
* #return mixed HTTP response on success, false on failure
*/
private function makeRequest()
{
$method = ($this->_headers_only == true) ? "HEAD" : "GET";
$proxy_auth = base64_encode("$this->_proxy_user:$this->_proxy_pass");
$response = "";
if ($this->_use_proxy) {
$headers = "$method $this->_request_url HTTP/1.1\r\n"
. "Host: $this->_host\r\n"
. "Proxy-Authorization: Basic $proxy_auth\r\n"
. "User-Agent: $this->_user_agent\r\n"
. "Connection: Close\r\n\r\n";
$fp = fsockopen($this->_proxy_host, $this->_proxy_port, $errno, $errmsg, $this->_req_timeout);
} else {
$headers = "$method $this->_path$this->_query$this->_fragment HTTP/1.1\r\n"
. "Host: $this->_host\r\n"
. "User-Agent: $this->_user_agent\r\n"
. "Connection: Close\r\n\r\n";
$fp = fsockopen($this->_host, $this->_portnum, $errno, $errmsg, $this->_req_timeout);
}
if (!$fp) {
$this->_is_error = true;
$this->_errmsg = "Unknown error";
return(false);
}
fwrite($fp, $headers);
while(!feof($fp)) {
$response .= fgets($fp, 4096);
}
fclose($fp);
return($response);
}
/**
* parse the requested URL to its host, path, query and fragment
*
* #return void
*/
private function parseUrl($url)
{
$this->_host = parse_url($url, PHP_URL_HOST);
$this->_path = parse_url($url, PHP_URL_PATH);
$this->_query = parse_url($url, PHP_URL_QUERY);
$this->_fragment = parse_url($url, PHP_URL_FRAGMENT);
if (empty($this->_path)) {
$this->_path = '/';
}
}
/**
* get the full path of the page to redirect. if the requested page is
* http://www.example.com and it redirects to redirpage.html, then the
* new request is http://www.example.com/redirpage.html
*
* #param string $loc new location from the HTTP response headers
* #param string $parent_url the parent's URL
*
* #return string full path of the page to redirect
*/
private function setFullPath($loc, $parent_url)
{
$parent_url = preg_replace("/\/[^\/]*$/", "", $parent_url);
if (strpos($loc, 'http://') !== false) {
return($loc);
}
if (strpos($loc, '../') === false) {
return("$parent_url/$loc");
}
while (strpos($loc, '../') !== false) {
$loc = preg_replace("/^\.\.\//", "", $loc);
$parent_url = preg_replace("/\/[^\/]+$/", "", $parent_url);
}
return("$parent_url/$loc");
}
/**
* parse HTTP response headers to array
*
* #param string $string HTTP response headers
*
* #return array
*/
private function parseHeaders($string)
{
$string = trim($string);
$headers = array();
$lines = explode("\r\n", $string);
$headers['http_status'] = $lines[0];
/* read HTTP _status in first line */
preg_match('/HTTP\/(\\d\\.\\d)\\s*(\\d+)\\s*(.*)/', $lines[0], $m);
$this->_status = $m[2];
array_splice($lines, 0, 1); /* remove first line */
foreach ($lines as $line) {
list($key, $val) = explode(': ', $line);
$key = str_replace("-", "_", $key);
$key = strtolower($key);
$val = trim($val);
$headers[$key] = $val;
}
return($headers);
}
/**
* join parts of the HTTP response body with chunked transfer-encoding
*
* #param string $chunks HTTP response body
*
* #return string full body
*/
private function joinChunks($chunks)
{
preg_match("/\r\n([0-9a-z]+)(;?.*)\r\n/", $chunks, $match);
$size = hexdec($match[1]);
$body = "";
while($size > 0) {
/* remove line with chunk size */
$chunks = preg_replace("/\r\n.+\r\n/m", "", $chunks, 1);
$part = substr($chunks, 0, $size);
$chunks = substr($chunks, $size);
$body .= $part;
/* get next chunk size */
preg_match("/\r\n([0-9a-z]+)(;?.*)\r\n/", $chunks, $match);
$size = hexdec($match[1]);
}
return($body);
}
/**
* set the requested URL
*
* #param string $url URL of the requested page
*/
public function setRequestUrl($url)
{
$this->_request_url = $url;
}
/**
* set to return headers only
*
* #param boolean $headers_only true to return headers only,
* false to return headers and body
*/
public function returnHeadersOnly($headers_only)
{
$this->_headers_only = $headers_only;
}
/**
* set proxy host and port
*
* #param string $hostport proxy host and proxy port in format proxy_host:proxy_port
*/
public function setProxyHost($hostport)
{
list($this->_proxy_host, $this->_proxy_port) = explode(':', $hostport);
$this->_use_proxy = true;
}
/**
* set proxy user and password
*
* #param string $userpass proxy user and password in format proxy_user:proxy_password
*/
public function setProxyUser($userpass)
{
list($this->_proxy_user, $this->_proxy_pass) = explode(':', $userpass);
}
/**
* get the HTTP response status (200, 404, etc)
*
* #return string
*/
public function getStatus()
{
return($this->_status);
}
/**
* get the requested URL
*
* #return string
*/
public function getRequestUrl()
{
return($this->_request_url);
}
/**
* set maximum redirects
*
* #param int $maxredirs
*/
public function setMaxRedirs($maxredirs)
{
$this->_maxredirs = $maxredirs;
}
/**
* get HTTP response headers
*
* #return array
*/
public function getHeaders()
{
return($this->_resp_headers);
}
/**
* get the HTTP response body, usually in HTML
*
* #return string
*/
public function getContents()
{
return($this->_resp_body);
echo $this->_resp_body;
}
/**
* get error message
*
* #return string
*/
public function getErrorMessage()
{
return($this->_errmsg);
}
/**
* print debug information
*/
private function debug($text)
{
print "$text\n";
}
}
?>

The array _resp_headers doesn't have any element with key transfer_encoding.
To fix the notice, you should check if the array has the key transfer_encoding:
if (array_key_exists('transfer_encoding', $this->_resp_headers) && $this->_resp_headers['transfer_encoding'] == 'chunked') {
But I can't tell you why the key is not set and why the script has stopped working if you don't show more code.

difficult to answer with such a small sample of code but you can check for the index's existence before
if (isset($this->_resp_headers['transfer_encoding']) &&
$this->_resp_headers['transfer_encoding'] == 'chunked') {

Related

Upgrading from reCAPTCHA v1 to v2 - server-side PHP

I have a website that someone else has build. There is a register page that calls PHP which has the below:
require_once('inc/func/recaptchalib.php');
// Get a key from https://www.google.com/recaptcha/admin/create
$publickey = "XXXXXXX";
$privatekey = "XXXXXXX";
# The response from reCAPTCHA
$resp = null;
# The error code from reCAPTCHA, if any
$error = null;
# Was there a reCAPTCHA response?
if ($_POST["recaptcha_response_field"]) {
$resp = recaptcha_check_answer(
$privatekey,
$_SERVER["REMOTE_ADDR"],
$_POST["recaptcha_challenge_field"],
$_POST["recaptcha_response_field"]);
if ($resp->is_valid) {
//echo "You got it!";
}
else {
# Set the error code so that we can display it
//$error = $resp->error;
return "Invalid Verification Code (reenter CAPTCHA code)";
}
}
else {
return "Invalid Verification Code (reenter CAPTCHA code)";
}
As you can see the script is calling recaptchalib.php which is shown below:
<?php
/*
* This is a PHP library that handles calling reCAPTCHA.
* - Documentation and latest version
* http://recaptcha.net/plugins/php/
* - Get a reCAPTCHA API Key
* https://www.google.com/recaptcha/admin/create
* - Discussion group
* http://groups.google.com/group/recaptcha
*
* Copyright (c) 2007 reCAPTCHA -- http://recaptcha.net
* AUTHORS:
* Mike Crawford
* Ben Maurer
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
/**
* The reCAPTCHA server URLs
*/
define("RECAPTCHA_API_SERVER", "http://www.google.com/recaptcha/api");
define("RECAPTCHA_API_SECURE_SERVER", "https://www.google.com/recaptcha/api");
define("RECAPTCHA_VERIFY_SERVER", "www.google.com");
/**
* Encodes the given data into a query string format
* #param $data - array of string elements to be encoded
* #return string - encoded request
*/
function _recaptcha_qsencode ($data) {
$req = "";
foreach ($data as $key => $value)
$req .= $key . '=' . urlencode(stripslashes($value)) . '&';
// Cut the last '&'
$req = substr($req, 0, strlen($req)-1);
return $req;
}
/**
* Submits an HTTP POST to a reCAPTCHA server
* #param string $host
* #param string $path
* #param array $data
* #param int port
* #return array response
*/
function _recaptcha_http_post($host, $path, $data, $port = 80) {
$req = _recaptcha_qsencode ($data);
$http_request = "POST $path HTTP/1.0\r\n";
$http_request .= "Host: $host\r\n";
$http_request .= "Content-Type: application/x-www-form-urlencoded;\r\n";
$http_request .= "Content-Length: " . strlen($req) . "\r\n";
$http_request .= "User-Agent: reCAPTCHA/PHP\r\n";
$http_request .= "\r\n";
$http_request .= $req;
$response = '';
if (false == ( $fs = #fsockopen($host, $port, $errno, $errstr, 10))) {
die ('Could not open socket');
}
fwrite($fs, $http_request);
while (!feof($fs))
$response .= fgets($fs, 1160); // One TCP-IP packet
fclose($fs);
$response = explode("\r\n\r\n", $response, 2);
return $response;
}
/**
* Gets the challenge HTML (JavaScript and non-JavaScript version).
* This is called from the browser, and the resulting reCAPTCHA HTML widget
* is embedded within the HTML form it was called from.
* #param string $pubkey A public key for reCAPTCHA
* #param string $error The error given by reCAPTCHA (optional, default is null)
* #param boolean $use_ssl Should the request be made over ssl? (optional, default is false)
* #return string - The HTML to be embedded in the user's form.
*/
function recaptcha_get_html ($pubkey, $error = null, $use_ssl = false)
{
if ($pubkey == null || $pubkey == '') {
die ("To use reCAPTCHA you must get an API key from <a href='https://www.google.com/recaptcha/admin/create'>https://www.google.com/recaptcha/admin/create</a>");
}
if ($use_ssl) {
$server = RECAPTCHA_API_SECURE_SERVER;
}
else {
$server = RECAPTCHA_API_SERVER;
}
$errorpart = "";
if ($error) {
$errorpart = "&error=" . $error;
}
return '<script type="text/javascript" src="'.
$server .
'/challenge?k=' .
$pubkey .
$errorpart .
'"></script>
<noscript>
<iframe src="'. $server . '/noscript?k=' . $pubkey . $errorpart . '" height="300" width="500" frameborder="0"></iframe><br/>
<textarea name="recaptcha_challenge_field" rows="3" cols="40"></textarea>
<input type="hidden" name="recaptcha_response_field" value="manual_challenge"/>
</noscript>';
}
/**
* A ReCaptchaResponse is returned from recaptcha_check_answer()
*/
class ReCaptchaResponse {
var $is_valid;
var $error;
}
/**
* Calls an HTTP POST function to verify if the user's
* guess was correct
* #param string $privkey
* #param string $remoteip
* #param string $challenge
* #param string $response
* #param array $extra_params an array of extra variables
* to post to the server
* #return ReCaptchaResponse
*/
function recaptcha_check_answer ($privkey, $remoteip, $challenge, $response, $extra_params = array())
{
if ($privkey == null || $privkey == '') {
die ("To use reCAPTCHA you must get an API key from <a href='https://www.google.com/recaptcha/admin/create'>https://www.google.com/recaptcha/admin/create</a>");
}
if ($remoteip == null || $remoteip == '') {
die ("For security reasons, you must pass the remote IP address to reCAPTCHA");
}
// Discard spam submissions
if ($challenge == null ||
strlen($challenge) == 0 ||
$response == null ||
strlen($response) == 0) {
$recaptcha_response = new ReCaptchaResponse();
$recaptcha_response->is_valid = false;
$recaptcha_response->error = 'incorrect-captcha-sol';
return $recaptcha_response;
}
$response = _recaptcha_http_post(
RECAPTCHA_VERIFY_SERVER, "/recaptcha/api/verify",
array (
'privatekey' => $privkey,
'remoteip' => $remoteip,
'challenge' => $challenge,
'response' => $response
) + $extra_params
);
$answers = explode ("\n", $response [1]);
$recaptcha_response = new ReCaptchaResponse();
if (trim ($answers [0]) == 'true') {
$recaptcha_response->is_valid = true;
}
else {
$recaptcha_response->is_valid = false;
$recaptcha_response->error = $answers [1];
}
return $recaptcha_response;
}
/**
* Gets a URL where the user can sign up for reCAPTCHA. If your
* application has a configuration page where you enter a key,
* you should provide a link using this function.
* #param string $domain The domain where the page is hosted
* #param string $appname The name of your application
*/
function recaptcha_get_signup_url ($domain = null, $appname = null) {
return "https://www.google.com/recaptcha/admin/create?" .
_recaptcha_qsencode(array('domains' => $domain,
'app' => $appname));
}
function _recaptcha_aes_pad($val) {
$block_size = 16;
$numpad = $block_size - (strlen ($val) % $block_size);
return str_pad($val, strlen ($val) + $numpad, chr($numpad));
}
/* Mailhide-related code */
function _recaptcha_aes_encrypt($val, $ky) {
if (! function_exists ("mcrypt_encrypt")) {
die ("To use reCAPTCHA Mailhide, you need to have the mcrypt PHP module installed.");
}
$mode = MCRYPT_MODE_CBC;
$enc = MCRYPT_RIJNDAEL_128;
$val = _recaptcha_aes_pad($val);
return mcrypt_encrypt($enc,
$ky,
$val,
$mode,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0");
}
function _recaptcha_mailhide_urlbase64 ($x) {
return strtr(base64_encode ($x), '+/', '-_');
}
/* Gets the reCAPTCHA Mailhide URL for a given email, public
key and private key */
function recaptcha_mailhide_url($pubkey, $privkey, $email) {
if ($pubkey == '' ||
$pubkey == null ||
$privkey == "" ||
$privkey == null) {
die ("To use reCAPTCHA Mailhide, you have to sign up for a public and private key, " .
"you can do so at <a href='http://www.google.com/recaptcha/mailhide/apikey'>http://www.google.com/recaptcha/mailhide/apikey</a>");
}
$ky = pack('H*', $privkey);
$cryptmail = _recaptcha_aes_encrypt ($email, $ky);
return "http://www.google.com/recaptcha/mailhide/d?k=" .
$pubkey .
"&c=" .
_recaptcha_mailhide_urlbase64 ($cryptmail);
}
/**
* Gets the parts of the email to expose to the user.
* for example, given johndoe#example.com return ["john", "example.com"].
* the email is then displayed as john...#example.com
*/
function _recaptcha_mailhide_email_parts ($email) {
$arr = preg_split("/#/", $email);
if (strlen ($arr[0]) <= 4) {
$arr[0] = substr ($arr[0], 0, 1);
}
else if (strlen ($arr[0]) <= 6) {
$arr[0] = substr ($arr[0], 0, 3);
}
else {
$arr[0] = substr ($arr[0], 0, 4);
}
return $arr;
}
/**
* Gets HTML to display an email address given a public
* an private key. To get a key, go to:
*
* http://www.google.com/recaptcha/mailhide/apikey
*/
function recaptcha_mailhide_html($pubkey, $privkey, $email) {
$emailparts = _recaptcha_mailhide_email_parts ($email);
$url = recaptcha_mailhide_url ($pubkey, $privkey, $email);
return htmlentities($emailparts[0]) .
"<a href='" . htmlentities ($url) .
"' onclick=\"window.open('" .
htmlentities ($url) .
"', '', 'toolbar=0,scrollbars=0,location=0,statusbar=0,menubar=0,resizable=0,width=500,height=300'); return false;\" title=\"Reveal this e-mail address\">...</a>#" .
htmlentities ($emailparts [1]);
}
?>
I need assistance upgrading this to V2. I want to do this via PHP if possible, but I am struggling to find out what I need to replace, etc.
How can I do this?

Function mod_scorm_insert_scorm_tracks "Invalid parameter"

I have problem with a WebService function for moodle callen "mod_scorm_insert_scorm_tracks"
This function is used for inserting track information (i.e. star time) of a user in his SCORM progress.
Part of the estructure of this function is
scoid= int
attempt= int
tracks[0][element]= string
tracks[0][value]= string
NEW
PHP structe has to look like this
[tracks] =>
Array
(
[0] =>
Array
(
[element] => string
[value] => string
)
)
I have used one of the examples they had in his website everything was fine until I got this error
<b>Notice</b>: Array to string conversion in <b>C:\xampp\htdocs\otros\PHP-REST\curl.php</b> on line <b>247</b><br />
<?xml version="1.0" encoding="UTF-8" ?>
<EXCEPTION class="invalid_parameter_exception">
<ERRORCODE>invalidparameter</ERRORCODE>
<MESSAGE>Invalid parameter value detected</MESSAGE>
<DEBUGINFO>tracks => Invalid parameter value detected: Only arrays accepted. The bad value is: 'Array'</DEBUGINFO>
</EXCEPTION>
And the problem seems to be here:
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(
array(
array(
'element' => 'x.start.time',
'value' => '1473102672'
),
),
array(
array(
'element' => 'x.start.time',
'value' => '1473102680'
),
),
);
I tried in many ways
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array('element' => 'x.start.time','value' => '1473102672');
or
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(array ('element' => 'x.start.time','value' => '1473102672'));
And still getting the same message, I'm pretty that is problema with my wyntax but I have tried in many ways and still not working I hope yo can help me.
Complete Code:
/// SETUP - NEED TO BE CHANGED
$token = '481bf3d85a7eb539e37eabc88feccb3c';
$domainname = 'http://localhost/moodle';
//$functionname = 'mod_scorm_launch_sco';
$functionname = 'mod_scorm_insert_scorm_tracks';
//$functionname ='mod_scorm_view_scorm';
// REST RETURNED VALUES FORMAT
$restformat = 'xml'; //Also possible in Moodle 2.2 and later: 'json'
//Setting it to 'json' will fail all calls on earlier Moodle version
$item1 = new stdClass();
$item1->scoid = '2';
$item1->attempt = '1';
$item1->tracks = array(
array(
array(
'element' => 'x.start.time',
'value' => 1473102672
),
),
array(
array(
'element' => 'x.start.time',
'value' => 1473102680
),
),
);
$params = $item1;
/// REST CALL
header('Content-Type: text/plain');
$serverurl = $domainname . '/webservice/rest/server.php'. '?wstoken=' . $token . '&wsfunction='.$functionname;
require_once('./curl.php');
$curl = new curl;
//if rest format == 'xml', then we do not add the param for backward compatibility with Moodle < 2.2
$restformat = ($restformat == 'json')?'&moodlewsrestformat=' . $restformat:'';
$resp = $curl->post($serverurl . $restformat, $params);
print_r($resp);
curl.php
<?php
/**
* cURL class
*
* This is a wrapper class for curl, it is quite easy to use:
* <code>
* $c = new curl;
* // enable cache
* $c = new curl(array('cache'=>true));
* // enable cookie
* $c = new curl(array('cookie'=>true));
* // enable proxy
* $c = new curl(array('proxy'=>true));
*
* // HTTP GET Method
* $html = $c->get('http://example.com');
* // HTTP POST Method
* $html = $c->post('http://example.com/', array('q'=>'words', 'name'=>'moodle'));
* // HTTP PUT Method
* $html = $c->put('http://example.com/', array('file'=>'/var/www/test.txt');
* </code>
*
* #author Dongsheng Cai <dongsheng#moodle.com> - https://github.com/dongsheng/cURL
* #license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class curl {
/** #var bool */
public $cache = false;
public $proxy = false;
/** #var array */
public $response = array();
public $header = array();
/** #var string */
public $info;
public $error;
/** #var array */
private $options;
/** #var string */
private $proxy_host = '';
private $proxy_auth = '';
private $proxy_type = '';
/** #var bool */
private $debug = false;
private $cookie = false;
private $count = 0;
/**
* #param array $options
*/
public function __construct($options = array()){
if (!function_exists('curl_init')) {
$this->error = 'cURL module must be enabled!';
trigger_error($this->error, E_USER_ERROR);
return false;
}
// the options of curl should be init here.
$this->resetopt();
if (!empty($options['debug'])) {
$this->debug = true;
}
if(!empty($options['cookie'])) {
if($options['cookie'] === true) {
$this->cookie = 'curl_cookie.txt';
} else {
$this->cookie = $options['cookie'];
}
}
if (!empty($options['cache'])) {
if (class_exists('curl_cache')) {
$this->cache = new curl_cache();
}
}
}
/**
* Resets the CURL options that have already been set
*/
public function resetopt(){
$this->options = array();
$this->options['CURLOPT_USERAGENT'] = 'MoodleBot/1.0';
// True to include the header in the output
$this->options['CURLOPT_HEADER'] = 0;
// True to Exclude the body from the output
$this->options['CURLOPT_NOBODY'] = 0;
// TRUE to follow any "Location: " header that the server
// sends as part of the HTTP header (note this is recursive,
// PHP will follow as many "Location: " headers that it is sent,
// unless CURLOPT_MAXREDIRS is set).
//$this->options['CURLOPT_FOLLOWLOCATION'] = 1;
$this->options['CURLOPT_MAXREDIRS'] = 10;
$this->options['CURLOPT_ENCODING'] = '';
// TRUE to return the transfer as a string of the return
// value of curl_exec() instead of outputting it out directly.
$this->options['CURLOPT_RETURNTRANSFER'] = 1;
$this->options['CURLOPT_BINARYTRANSFER'] = 0;
$this->options['CURLOPT_SSL_VERIFYPEER'] = 0;
$this->options['CURLOPT_SSL_VERIFYHOST'] = 2;
$this->options['CURLOPT_CONNECTTIMEOUT'] = 30;
}
/**
* Reset Cookie
*/
public function resetcookie() {
if (!empty($this->cookie)) {
if (is_file($this->cookie)) {
$fp = fopen($this->cookie, 'w');
if (!empty($fp)) {
fwrite($fp, '');
fclose($fp);
}
}
}
}
/**
* Set curl options
*
* #param array $options If array is null, this function will
* reset the options to default value.
*
*/
public function setopt($options = array()) {
if (is_array($options)) {
foreach($options as $name => $val){
if (stripos($name, 'CURLOPT_') === false) {
$name = strtoupper('CURLOPT_'.$name);
}
$this->options[$name] = $val;
}
}
}
/**
* Reset http method
*
*/
public function cleanopt(){
unset($this->options['CURLOPT_HTTPGET']);
unset($this->options['CURLOPT_POST']);
unset($this->options['CURLOPT_POSTFIELDS']);
unset($this->options['CURLOPT_PUT']);
unset($this->options['CURLOPT_INFILE']);
unset($this->options['CURLOPT_INFILESIZE']);
unset($this->options['CURLOPT_CUSTOMREQUEST']);
}
/**
* Set HTTP Request Header
*
* #param array $headers
*
*/
public function setHeader($header) {
if (is_array($header)){
foreach ($header as $v) {
$this->setHeader($v);
}
} else {
$this->header[] = $header;
}
}
/**
* Set HTTP Response Header
*
*/
public function getResponse(){
return $this->response;
}
/**
* private callback function
* Formatting HTTP Response Header
*
* #param mixed $ch Apparently not used
* #param string $header
* #return int The strlen of the header
*/
private function formatHeader($ch, $header)
{
$this->count++;
if (strlen($header) > 2) {
list($key, $value) = explode(" ", rtrim($header, "\r\n"), 2);
$key = rtrim($key, ':');
if (!empty($this->response[$key])) {
if (is_array($this->response[$key])){
$this->response[$key][] = $value;
} else {
$tmp = $this->response[$key];
$this->response[$key] = array();
$this->response[$key][] = $tmp;
$this->response[$key][] = $value;
}
} else {
$this->response[$key] = $value;
}
}
return strlen($header);
}
/**
* Set options for individual curl instance
*
* #param object $curl A curl handle
* #param array $options
* #return object The curl handle
*/
private function apply_opt($curl, $options) {
// Clean up
$this->cleanopt();
// set cookie
if (!empty($this->cookie) || !empty($options['cookie'])) {
$this->setopt(array('cookiejar'=>$this->cookie,
'cookiefile'=>$this->cookie
));
}
// set proxy
if (!empty($this->proxy) || !empty($options['proxy'])) {
$this->setopt($this->proxy);
}
$this->setopt($options);
// reset before set options
curl_setopt($curl, CURLOPT_HEADERFUNCTION, array(&$this,'formatHeader'));
// set headers
if (empty($this->header)){
$this->setHeader(array(
'User-Agent: MoodleBot/1.0',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Connection: keep-alive'
));
}
curl_setopt($curl, CURLOPT_HTTPHEADER, $this->header);
if ($this->debug){
echo '<h1>Options</h1>';
var_dump($this->options);
echo '<h1>Header</h1>';
var_dump($this->header);
}
// set options
foreach($this->options as $name => $val) {
if (is_string($name)) {
$name = constant(strtoupper($name));
}
curl_setopt($curl, $name, $val);
}
return $curl;
}
/**
* Download multiple files in parallel
*
* Calls {#link multi()} with specific download headers
*
* <code>
* $c = new curl;
* $c->download(array(
* array('url'=>'http://localhost/', 'file'=>fopen('a', 'wb')),
* array('url'=>'http://localhost/20/', 'file'=>fopen('b', 'wb'))
* ));
* </code>
*
* #param array $requests An array of files to request
* #param array $options An array of options to set
* #return array An array of results
*/
public function download($requests, $options = array()) {
$options['CURLOPT_BINARYTRANSFER'] = 1;
$options['RETURNTRANSFER'] = false;
return $this->multi($requests, $options);
}
/*
* Mulit HTTP Requests
* This function could run multi-requests in parallel.
*
* #param array $requests An array of files to request
* #param array $options An array of options to set
* #return array An array of results
*/
protected function multi($requests, $options = array()) {
$count = count($requests);
$handles = array();
$results = array();
$main = curl_multi_init();
for ($i = 0; $i < $count; $i++) {
$url = $requests[$i];
foreach($url as $n=>$v){
$options[$n] = $url[$n];
}
$handles[$i] = curl_init($url['url']);
$this->apply_opt($handles[$i], $options);
curl_multi_add_handle($main, $handles[$i]);
}
$running = 0;
do {
curl_multi_exec($main, $running);
} while($running > 0);
for ($i = 0; $i < $count; $i++) {
if (!empty($options['CURLOPT_RETURNTRANSFER'])) {
$results[] = true;
} else {
$results[] = curl_multi_getcontent($handles[$i]);
}
curl_multi_remove_handle($main, $handles[$i]);
}
curl_multi_close($main);
return $results;
}
/**
* Single HTTP Request
*
* #param string $url The URL to request
* #param array $options
* #return bool
*/
protected function request($url, $options = array()){
// create curl instance
$curl = curl_init($url);
$options['url'] = $url;
$this->apply_opt($curl, $options);
if ($this->cache && $ret = $this->cache->get($this->options)) {
return $ret;
} else {
$ret = curl_exec($curl);
if ($this->cache) {
$this->cache->set($this->options, $ret);
}
}
$this->info = curl_getinfo($curl);
$this->error = curl_error($curl);
if ($this->debug){
echo '<h1>Return Data</h1>';
var_dump($ret);
echo '<h1>Info</h1>';
var_dump($this->info);
echo '<h1>Error</h1>';
var_dump($this->error);
}
curl_close($curl);
if (empty($this->error)){
return $ret;
} else {
return $this->error;
// exception is not ajax friendly
//throw new moodle_exception($this->error, 'curl');
}
}
/**
* HTTP HEAD method
*
* #see request()
*
* #param string $url
* #param array $options
* #return bool
*/
public function head($url, $options = array()){
$options['CURLOPT_HTTPGET'] = 0;
$options['CURLOPT_HEADER'] = 1;
$options['CURLOPT_NOBODY'] = 1;
return $this->request($url, $options);
}
/**
* Recursive function formating an array in POST parameter
* #param array $arraydata - the array that we are going to format and add into &$data array
* #param string $currentdata - a row of the final postdata array at instant T
* when finish, it's assign to $data under this format: name[keyname][][]...[]='value'
* #param array $data - the final data array containing all POST parameters : 1 row = 1 parameter
*/
function format_array_postdata_for_curlcall($arraydata, $currentdata, &$data) {
foreach ($arraydata as $k=>$v) {
$newcurrentdata = $currentdata;
if (is_object($v)) {
$v = (array) $v;
}
if (is_array($v)) { //the value is an array, call the function recursively
$newcurrentdata = $newcurrentdata.'['.urlencode($k).']';
$this->format_array_postdata_for_curlcall($v, $newcurrentdata, $data);
} else { //add the POST parameter to the $data array
$data[] = $newcurrentdata.'['.urlencode($k).']='.urlencode($v);
}
}
}
/**
* Transform a PHP array into POST parameter
* (see the recursive function format_array_postdata_for_curlcall)
* #param array $postdata
* #return array containing all POST parameters (1 row = 1 POST parameter)
*/
function format_postdata_for_curlcall($postdata) {
if (is_object($postdata)) {
$postdata = (array) $postdata;
}
$data = array();
foreach ($postdata as $k=>$v) {
if (is_object($v)) {
$v = (array) $v;
}
if (is_array($v)) {
$currentdata = urlencode($k);
$this->format_array_postdata_for_curlcall($v, $currentdata, $data);
} else {
$data[] = urlencode($k).'='.urlencode($v);
}
}
$convertedpostdata = implode('&', $data);
return $convertedpostdata;
}
/**
* HTTP POST method
*
* #param string $url
* #param array|string $params
* #param array $options
* #return bool
*/
public function post($url, $params = '', $options = array()){
$options['CURLOPT_POST'] = 1;
if (is_array($params)) {
$params = $this->format_postdata_for_curlcall($params);
}
$options['CURLOPT_POSTFIELDS'] = $params;
return $this->request($url, $options);
}
/**
* HTTP GET method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function get($url, $params = array(), $options = array()){
$options['CURLOPT_HTTPGET'] = 1;
if (!empty($params)){
$url .= (stripos($url, '?') !== false) ? '&' : '?';
$url .= http_build_query($params, '', '&');
}
return $this->request($url, $options);
}
/**
* HTTP PUT method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function put($url, $params = array(), $options = array()){
$file = $params['file'];
if (!is_file($file)){
return null;
}
$fp = fopen($file, 'r');
$size = filesize($file);
$options['CURLOPT_PUT'] = 1;
$options['CURLOPT_INFILESIZE'] = $size;
$options['CURLOPT_INFILE'] = $fp;
if (!isset($this->options['CURLOPT_USERPWD'])){
$this->setopt(array('CURLOPT_USERPWD'=>'anonymous: noreply#moodle.org'));
}
$ret = $this->request($url, $options);
fclose($fp);
return $ret;
}
/**
* HTTP DELETE method
*
* #param string $url
* #param array $params
* #param array $options
* #return bool
*/
public function delete($url, $param = array(), $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'DELETE';
if (!isset($options['CURLOPT_USERPWD'])) {
$options['CURLOPT_USERPWD'] = 'anonymous: noreply#moodle.org';
}
$ret = $this->request($url, $options);
return $ret;
}
/**
* HTTP TRACE method
*
* #param string $url
* #param array $options
* #return bool
*/
public function trace($url, $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'TRACE';
$ret = $this->request($url, $options);
return $ret;
}
/**
* HTTP OPTIONS method
*
* #param string $url
* #param array $options
* #return bool
*/
public function options($url, $options = array()){
$options['CURLOPT_CUSTOMREQUEST'] = 'OPTIONS';
$ret = $this->request($url, $options);
return $ret;
}
public function get_info() {
return $this->info;
}
}
/**
* This class is used by cURL class, use case:
*
* <code>
*
* $c = new curl(array('cache'=>true), 'module_cache'=>'repository');
* $ret = $c->get('http://www.google.com');
* </code>
*
* #package core
* #subpackage file
* #copyright 1999 onwards Martin Dougiamas {#link http://moodle.com}
* #license http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
*/
class curl_cache {
/** #var string */
public $dir = '';
/**
*
* #param string #module which module is using curl_cache
*
*/
function __construct() {
$this->dir = '/tmp/';
if (!file_exists($this->dir)) {
mkdir($this->dir, 0700, true);
}
$this->ttl = 1200;
}
/**
* Get cached value
*
* #param mixed $param
* #return bool|string
*/
public function get($param){
$this->cleanup($this->ttl);
$filename = 'u_'.md5(serialize($param));
if(file_exists($this->dir.$filename)) {
$lasttime = filemtime($this->dir.$filename);
if(time()-$lasttime > $this->ttl)
{
return false;
} else {
$fp = fopen($this->dir.$filename, 'r');
$size = filesize($this->dir.$filename);
$content = fread($fp, $size);
return unserialize($content);
}
}
return false;
}
/**
* Set cache value
*
* #param mixed $param
* #param mixed $val
*/
public function set($param, $val){
$filename = 'u_'.md5(serialize($param));
$fp = fopen($this->dir.$filename, 'w');
fwrite($fp, serialize($val));
fclose($fp);
}
/**
* Remove cache files
*
* #param int $expire The number os seconds before expiry
*/
public function cleanup($expire){
if($dir = opendir($this->dir)){
while (false !== ($file = readdir($dir))) {
if(!is_dir($file) && $file != '.' && $file != '..') {
$lasttime = #filemtime($this->dir.$file);
if(time() - $lasttime > $expire){
#unlink($this->dir.$file);
}
}
}
}
}
/**
* delete current user's cache file
*
*/
public function refresh(){
if($dir = opendir($this->dir)){
while (false !== ($file = readdir($dir))) {
if(!is_dir($file) && $file != '.' && $file != '..') {
if(strpos($file, 'u_')!==false){
#unlink($this->dir.$file);
}
}
}
}
}
}
Thanks!
Well after some research I finally took plan B
I wrote tracks in a different variable:
$tracks = array();
$tracks[] = array(
'element' => 'cmi.core.lesson_status',
'value' => 'completed'
);
And I followed curl.php array set option:
$arrayName = array('' => , );
Then when I inserted scoid and attemps as single variables in the array:
$params = array('scoid' => '2', 'attempt' => '1', 'tracks' => $tracks);
and boala!the record is on my table:

Is it possible to get EXTERNAL_LINKS using the GWT API?

I am trying to use the code from #eyecatchup (https://github.com/eyecatchup/php-webmaster-tools-downloads) to get data via the Google Webmaster Tools API. I am able to get "TOP_PAGES" and "TOP QUERIES" but that is it. What I really want is "EXTERNAL_LINKS" or even "LATEST_LINKS".
Here is the gwtdata.php code:
<pre>
<?php
/**
* PHP class for downloading CSV files from Google Webmaster Tools.
*
* This class does NOT require the Zend gdata package be installed
* in order to run.
*
* Copyright 2012 eyecatchUp UG. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* #author: Stephan Schmitz <eyecatchup#gmail.com>
* #link: https://code.google.com/p/php-webmaster-tools-downloads/
*/
class GWTdata
{
const HOST = "https://www.google.com";
const SERVICEURI = "/webmasters/tools/";
public $_language, $_tables, $_daterange, $_downloaded, $_skipped;
private $_auth, $_logged_in;
public function __construct()
{
$this->_auth = false;
$this->_logged_in = false;
$this->_language = "en";
$this->_daterange = array("","");
$this->_tables = array("TOP_PAGES", "TOP_QUERIES",
"CRAWL_ERRORS", "CONTENT_ERRORS", "CONTENT_KEYWORDS",
"INTERNAL_LINKS", "EXTERNAL_LINKS", "SOCIAL_ACTIVITY"
);
$this->_errTablesSort = array(0 => "http",
1 => "not-found", 2 => "restricted-by-robotsTxt",
3 => "unreachable", 4 => "timeout", 5 => "not-followed",
"kAppErrorSoft-404s" => "soft404", "sitemap" => "in-sitemaps"
);
$this->_errTablesType = array(0 => "web-crawl-errors",
1 => "mobile-wml-xhtml-errors", 2 => "mobile-chtml-errors",
3 => "mobile-operator-errors", 4 => "news-crawl-errors"
);
$this->_downloaded = array();
$this->_skipped = array();
}
/**
* Sets content language.
*
* #param $str String Valid ISO 639-1 language code, supported by Google.
*/
public function SetLanguage($str)
{
$this->_language = $str;
}
/**
* Sets features that should be downloaded.
*
* #param $arr Array Valid array values are:
* "TOP_PAGES", "TOP_QUERIES", "CRAWL_ERRORS", "CONTENT_ERRORS",
* "CONTENT_KEYWORDS", "INTERNAL_LINKS", "EXTERNAL_LINKS",
* "SOCIAL_ACTIVITY".
*/
public function SetTables($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) <= 2) {
$valid = array("TOP_PAGES","TOP_QUERIES","CRAWL_ERRORS","CONTENT_ERRORS",
"CONTENT_KEYWORDS","INTERNAL_LINKS","EXTERNAL_LINKS","SOCIAL_ACTIVITY");
$this->_tables = array();
for($i=0; $i < sizeof($arr); $i++) {
if(in_array($arr[$i], $valid)) {
array_push($this->_tables, $arr[$i]);
} else { throw new Exception("Invalid argument given."); }
}
} else { throw new Exception("Invalid argument given."); }
}
/**
* Sets daterange for download data.
*
* #param $arr Array Array containing two ISO 8601 formatted date strings.
*/
public function SetDaterange($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) == 2) {
if(self::IsISO8601($arr[0]) === true &&
self::IsISO8601($arr[1]) === true) {
$this->_daterange = array(str_replace("-", "", $arr[0]),
str_replace("-", "", $arr[1]));
return true;
} else { throw new Exception("Invalid argument given."); }
} else { throw new Exception("Invalid argument given."); }
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetDownloadedFiles()
{
return $this->_downloaded;
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetSkippedFiles()
{
return $this->_skipped;
}
/**
* Checks if client has logged into their Google account yet.
*
* #return Boolean Returns true if logged in, or false if not.
*/
private function IsLoggedIn()
{
return $this->_logged_in;
}
/**
* Attempts to log into the specified Google account.
*
* #param $email String User's Google email address.
* #param $pwd String Password for Google account.
* #return Boolean Returns true when Authentication was successful,
* else false.
*/
public function LogIn($email, $pwd)
{
$url = self::HOST . "/accounts/ClientLogin";
$postRequest = array(
'accountType' => 'HOSTED_OR_GOOGLE',
'Email' => $email,
'Passwd' => $pwd,
'service' => "sitemaps",
'source' => "Google-WMTdownloadscript-0.1-php"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postRequest);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
if($info['http_code'] == 200) {
preg_match('/Auth=(.*)/', $output, $match);
if(isset($match[1])) {
$this->_auth = $match[1];
$this->_logged_in = true;
return true;
} else { return false; }
} else { return false; }
}
/**
* Attempts authenticated GET Request.
*
* #param $url String URL for the GET request.
* #return Mixed Curl result as String,
* or false (Boolean) when Authentication fails.
*/
public function GetData($url)
{
if(self::IsLoggedIn() === true) {
$url = self::HOST . $url;
$head = array("Authorization: GoogleLogin auth=".$this->_auth,
"GData-Version: 2");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $head);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
return ($info['http_code']!=200) ? false : $result;
} else { return false; }
}
/**
* Gets all available sites from Google Webmaster Tools account.
*
* #return Mixed Array with all site URLs registered in GWT account,
* or false (Boolean) if request failed.
*/
public function GetSites()
{
if(self::IsLoggedIn() === true) {
$feed = self::GetData(self::SERVICEURI."feeds/sites/");
if($feed !== false) {
$sites = array();
$doc = new DOMDocument();
$doc->loadXML($feed);
foreach ($doc->getElementsByTagName('entry') as $node) {
array_push($sites,
$node->getElementsByTagName('title')->item(0)->nodeValue);
}
return $sites;
} else { return false; }
} else { return false; }
}
/**
* Gets the download links for an available site
* from the Google Webmaster Tools account.
*
* #param $url String Site URL registered in GWT.
* #return Mixed Array with keys TOP_PAGES and TOP_QUERIES,
* or false (Boolean) when Authentication fails.
*/
public function GetDownloadUrls($url)
{
if(self::IsLoggedIn() === true) {
$_url = sprintf(self::SERVICEURI."downloads-list?hl=%s&siteUrl=%s",
$this->_language,
urlencode($url));
$downloadList = self::GetData($_url);
return json_decode($downloadList, true);
} else { return false; }
}
/**
* Downloads the file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV($site, $savepath=".")
{
if(self::IsLoggedIn() === true) {
$downloadUrls = self::GetDownloadUrls($site);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$tables = $this->_tables;
foreach($tables as $table) {
if($table=="CRAWL_ERRORS") {
self::DownloadCSV_CrawlErrors($site, $savepath);
}
elseif($table=="CONTENT_ERRORS") {
self::DownloadCSV_XTRA($site, $savepath,
"html-suggestions", "\)", "CONTENT_ERRORS", "content-problems-dl");
}
elseif($table=="CONTENT_KEYWORDS") {
self::DownloadCSV_XTRA($site, $savepath,
"keywords", "\)", "CONTENT_KEYWORDS", "content-words-dl");
}
elseif($table=="INTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"internal-links", "\)", "INTERNAL_LINKS", "internal-links-dl");
}
elseif($table=="EXTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"external-links-domain", "\)", "EXTERNAL_LINKS", "external-links-domain-dl");
}
elseif($table=="SOCIAL_ACTIVITY") {
self::DownloadCSV_XTRA($site, $savepath,
"social-activity", "x26", "SOCIAL_ACTIVITY", "social-activity-dl");
}
else {
$finalName = "$savepath/$table-$filename.csv";
$finalUrl = $downloadUrls[$table] ."&prop=ALL&db=%s&de=%s&more=true";
$finalUrl = sprintf($finalUrl, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($finalUrl,$finalName);
}
}
} else { return false; }
}
/**
* Downloads "unofficial" downloads based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV_XTRA($site, $savepath=".", $tokenUri, $tokenDelimiter, $filenamePrefix, $dlUri)
{
if(self::IsLoggedIn() === true) {
$uri = self::SERVICEURI . $tokenUri . "?hl=%s&siteUrl=%s";
$_uri = sprintf($uri, $this->_language, $site);
$token = self::GetToken($_uri, $tokenDelimiter);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$finalName = "$savepath/$filenamePrefix-$filename.csv";
$url = self::SERVICEURI . $dlUri . "?hl=%s&siteUrl=%s&security_token=%s&prop=ALL&db=%s&de=%s&more=true";
$_url = sprintf($url, $this->_language, $site, $token, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($_url,$finalName);
} else { return false; }
}
/**
* Downloads the Crawl Errors file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional: Path to save CSV to (no trailing slash!).
* #param $separated Boolean Optional: If true, the method saves separated CSV files
* for each error type. Default: Merge errors in one file.
*/
public function DownloadCSV_CrawlErrors($site, $savepath=".", $separated=false)
{
if(self::IsLoggedIn() === true) {
$type_param = "we";
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
if($separated) {
foreach($this->_errTablesSort as $sortid => $sortname) {
foreach($this->_errTablesType as $typeid => $typename) {
if($typeid == 1) {
$type_param = "mx";
} else if($typeid == 2) {
$type_param = "mc";
} else {
$type_param = "we";
}
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$typename-$sortname-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=%s&sort=%s";
$_url = sprintf($url, $this->_language, $site, $token, $typeid, $sortid);
self::SaveData($_url,$finalName);
}
}
}
else {
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=0";
$_url = sprintf($url, $this->_language, $site, $token);
self::SaveData($_url,$finalName);
}
} else { return false; }
}
/**
* Saves data to a CSV file based on the given URL.
*
* #param $finalUrl String CSV Download URI.
* #param $finalName String Filepointer to save location.
*/
private function SaveData($finalUrl, $finalName)
{
$data = self::GetData($finalUrl);
if(strlen($data) > 1 && file_put_contents($finalName, utf8_decode($data))) {
array_push($this->_downloaded, realpath($finalName));
return true;
} else {
array_push($this->_skipped, $finalName);
return false;
}
}
/**
* Regular Expression to find the Security Token for a download file.
*
* #param $uri String A Webmaster Tools Desktop Service URI.
* #param $delimiter String Trailing delimiter for the regex.
* #return String Returns a security token.
*/
private function GetToken($uri, $delimiter)
{
$matches = array(); $tmp = self::get_data($uri); preg_match_all("#46security_token(.?)$delimiter#si", $tmp, $matches); return #substr($matches[1][0],3,-1);
}
/**
* Validates ISO 8601 date format.
*
* #param $str String Valid ISO 8601 date string (eg. 2012-01-01).
* #return Boolean Returns true if string has valid format, else false.
*/
private function IsISO8601($str)
{
$stamp = strtotime($str);
return (is_numeric($stamp) && checkdate(date('m', $stamp),
date('d', $stamp), date('Y', $stamp))) ? true : false;
}
}
?>
</pre>
And here is the code I am using to try to extract external links:
<pre>
<?php
include 'gwtdata.php';
try {
$email = "***#gmail.com";
$password = "***";
# If hardcoded, don't forget trailing slash!
$website = "***";
# Valid values are "TOP_PAGES", "TOP_QUERIES", "CRAWL_ERRORS",
# "CONTENT_ERRORS", "CONTENT_KEYWORDS", "INTERNAL_LINKS",
# "EXTERNAL_LINKS" and "SOCIAL_ACTIVITY".
$tables = array("EXTERNAL_LINKS");
$gdata = new GWTdata();
if($gdata->LogIn($email, $password) === true)
{
$gdata->SetTables($tables);
$gdata->DownloadCSV($website, "./csv");
}
$files = $gdata->GetDownloadedFiles();
foreach($files as $file)
{
print "Saved $file\n</a>";
}
} catch (Exception $e) {
die($e->getMessage());
}
?>
</pre>
Part of the author's answer here https://stackoverflow.com/a/16002159/624466, is the answer to your question too.
[..] this code is neither released by Google nor makes use of an official
API, but is rather a custom script processing data from the web
interface.
[..] there were some changes to the Google Webmaster Tools web
interface [..]. Thus,
it broke some functionality of the PHP class GWTdata

Is there any possible to get Google Webmaster Tool (GWT) datas using PHP?

I'm trying to get some datas from Google Webmaster Tool (GWT), I have searched some of the API Documents and Implements, But they are returning few of the datas only from the GWT.
My Needs :
Needs to get the datas of the following from GWT,
(1). TOP_PAGES
(2). TOP_QUERIES
(3). CRAWL_ERRORS
(4). CONTENT_ERRORS
(5). CONTENT_KEYWORDS
(6). INTERNAL_LINKS
(7). EXTERNAL_LINKS
(8). SOCIAL_ACTIVITY
After getting these datas, i need to generate the Excel file for each of them.
Achieved :
I have got few datas from the above and generated into the Excel file.such as,
(1). TOP_PAGES
(2). TOP_QUERIES
(3). INTERNAL_LINKS
(4). EXTERNAL_LINKS
(5). CONTENT_KEYWORDS
Not Achieved :
Still I'm not getting the major parts / datas like,
(1). CRAWL_ERRORS
(2). CONTENT_ERRORS
(3). SOCIAL_ACTIVITY
Code Samples For Your Reference :
I have used two files in PHP for this GWT API,
File #1 : ( gwdata.php )
<?php
/**
* PHP class for downloading CSV files from Google Webmaster Tools.
*
* This class does NOT require the Zend gdata package be installed
* in order to run.
*
* Copyright 2012 eyecatchUp UG. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* #author: Stephan Schmitz <eyecatchup#gmail.com>
* #link: https://code.google.com/p/php-webmaster-tools-downloads/
*/
class GWTdata
{
const HOST = "https://www.google.com";
const SERVICEURI = "/webmasters/tools/";
public $_language, $_tables, $_daterange, $_downloaded, $_skipped;
private $_auth, $_logged_in;
public function __construct()
{
$this->_auth = false;
$this->_logged_in = false;
$this->_language = "en";
$this->_daterange = array("","");
$this->_tables = array("TOP_PAGES", "TOP_QUERIES",
"CRAWL_ERRORS", "CONTENT_ERRORS", "CONTENT_KEYWORDS",
"INTERNAL_LINKS", "EXTERNAL_LINKS", "SOCIAL_ACTIVITY"
);
$this->_errTablesSort = array(0 => "http",
1 => "not-found", 2 => "restricted-by-robotsTxt",
3 => "unreachable", 4 => "timeout", 5 => "not-followed",
"kAppErrorSoft-404s" => "soft404", "sitemap" => "in-sitemaps"
);
$this->_errTablesType = array(0 => "web-crawl-errors",
1 => "mobile-wml-xhtml-errors", 2 => "mobile-chtml-errors",
3 => "mobile-operator-errors", 4 => "news-crawl-errors"
);
$this->_downloaded = array();
$this->_skipped = array();
}
/**
* Sets content language.
*
* #param $str String Valid ISO 639-1 language code, supported by Google.
*/
public function SetLanguage($str)
{
$this->_language = $str;
}
/**
* Sets features that should be downloaded.
*
* #param $arr Array Valid array values are:
* "TOP_PAGES", "TOP_QUERIES", "CRAWL_ERRORS", "CONTENT_ERRORS",
* "CONTENT_KEYWORDS", "INTERNAL_LINKS", "EXTERNAL_LINKS",
* "SOCIAL_ACTIVITY".
*/
public function SetTables($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) <= 2) {
$valid = array("TOP_PAGES","TOP_QUERIES","CRAWL_ERRORS","CONTENT_ERRORS",
"CONTENT_KEYWORDS","INTERNAL_LINKS","EXTERNAL_LINKS","SOCIAL_ACTIVITY");
$this->_tables = array();
for($i=0; $i < sizeof($arr); $i++) {
if(in_array($arr[$i], $valid)) {
array_push($this->_tables, $arr[$i]);
} else { throw new Exception("Invalid argument given."); }
}
} else { throw new Exception("Invalid argument given."); }
}
/**
* Sets daterange for download data.
*
* #param $arr Array Array containing two ISO 8601 formatted date strings.
*/
public function SetDaterange($arr)
{
if(is_array($arr) && !empty($arr) && sizeof($arr) == 2) {
if(self::IsISO8601($arr[0]) === true &&
self::IsISO8601($arr[1]) === true) {
$this->_daterange = array(str_replace("-", "", $arr[0]),
str_replace("-", "", $arr[1]));
return true;
} else { throw new Exception("Invalid argument given."); }
} else { throw new Exception("Invalid argument given."); }
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetDownloadedFiles()
{
return $this->_downloaded;
}
/**
* Returns array of downloaded filenames.
*
* #return Array Array of filenames that have been written to disk.
*/
public function GetSkippedFiles()
{
return $this->_skipped;
}
/**
* Checks if client has logged into their Google account yet.
*
* #return Boolean Returns true if logged in, or false if not.
*/
private function IsLoggedIn()
{
return $this->_logged_in;
}
/**
* Attempts to log into the specified Google account.
*
* #param $email String User's Google email address.
* #param $pwd String Password for Google account.
* #return Boolean Returns true when Authentication was successful,
* else false.
*/
public function LogIn($email, $pwd)
{
$url = self::HOST . "/accounts/ClientLogin";
$postRequest = array(
'accountType' => 'HOSTED_OR_GOOGLE',
'Email' => $email,
'Passwd' => $pwd,
'service' => "sitemaps",
'source' => "Google-WMTdownloadscript-0.1-php"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postRequest);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
if($info['http_code'] == 200) {
preg_match('/Auth=(.*)/', $output, $match);
if(isset($match[1])) {
$this->_auth = $match[1];
$this->_logged_in = true;
return true;
} else { return false; }
} else { return false; }
}
/**
* Attempts authenticated GET Request.
*
* #param $url String URL for the GET request.
* #return Mixed Curl result as String,
* or false (Boolean) when Authentication fails.
*/
public function GetData($url)
{
if(self::IsLoggedIn() === true) {
$url = self::HOST . $url;
$head = array("Authorization: GoogleLogin auth=".$this->_auth,
"GData-Version: 2");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $head);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
return ($info['http_code']!=200) ? false : $result;
} else { return false; }
}
/**
* Gets all available sites from Google Webmaster Tools account.
*
* #return Mixed Array with all site URLs registered in GWT account,
* or false (Boolean) if request failed.
*/
public function GetSites()
{
if(self::IsLoggedIn() === true) {
$feed = self::GetData(self::SERVICEURI."feeds/sites/");
if($feed !== false) {
$sites = array();
$doc = new DOMDocument();
$doc->loadXML($feed);
foreach ($doc->getElementsByTagName('entry') as $node) {
array_push($sites,
$node->getElementsByTagName('title')->item(0)->nodeValue);
}
return $sites;
} else { return false; }
} else { return false; }
}
/**
* Gets the download links for an available site
* from the Google Webmaster Tools account.
*
* #param $url String Site URL registered in GWT.
* #return Mixed Array with keys TOP_PAGES and TOP_QUERIES,
* or false (Boolean) when Authentication fails.
*/
public function GetDownloadUrls($url)
{
if(self::IsLoggedIn() === true) {
$_url = sprintf(self::SERVICEURI."downloads-list?hl=%s&siteUrl=%s",
$this->_language,
urlencode($url));
$downloadList = self::GetData($_url);
return json_decode($downloadList, true);
} else { return false; }
}
/**
* Downloads the file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV($site, $savepath=".")
{
if(self::IsLoggedIn() === true) {
$downloadUrls = self::GetDownloadUrls($site);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$tables = $this->_tables;
foreach($tables as $table) {
if($table=="CRAWL_ERRORS") {
self::DownloadCSV_CrawlErrors($site, $savepath);
}
elseif($table=="CONTENT_ERRORS") {
self::DownloadCSV_XTRA($site, $savepath,
"html-suggestions", "\)", "CONTENT_ERRORS", "content-problems-dl");
}
elseif($table=="CONTENT_KEYWORDS") {
self::DownloadCSV_XTRA($site, $savepath,
"keywords", "\)", "CONTENT_KEYWORDS", "content-words-dl");
}
elseif($table=="INTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"internal-links", "\)", "INTERNAL_LINKS", "internal-links-dl");
}
elseif($table=="EXTERNAL_LINKS") {
self::DownloadCSV_XTRA($site, $savepath,
"external-links-domain", "\)", "EXTERNAL_LINKS", "external-links-domain-dl");
}
elseif($table=="SOCIAL_ACTIVITY") {
self::DownloadCSV_XTRA($site, $savepath,
"social-activity", "x26", "SOCIAL_ACTIVITY", "social-activity-dl");
}
else {
$finalName = "$savepath/$table-$filename.csv";
$finalUrl = $downloadUrls[$table] ."&prop=ALL&db=%s&de=%s&more=true";
$finalUrl = sprintf($finalUrl, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($finalUrl,$finalName);
}
}
} else { return false; }
}
/**
* Downloads "unofficial" downloads based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional path to save CSV to (no trailing slash!).
*/
public function DownloadCSV_XTRA($site, $savepath=".", $tokenUri, $tokenDelimiter, $filenamePrefix, $dlUri)
{
if(self::IsLoggedIn() === true) {
$uri = self::SERVICEURI . $tokenUri . "?hl=%s&siteUrl=%s";
$_uri = sprintf($uri, $this->_language, $site);
$token = self::GetToken($_uri, $tokenDelimiter);
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
$finalName = "$savepath/$filenamePrefix-$filename.csv";
$url = self::SERVICEURI . $dlUri . "?hl=%s&siteUrl=%s&security_token=%s&prop=ALL&db=%s&de=%s&more=true";
$_url = sprintf($url, $this->_language, $site, $token, $this->_daterange[0], $this->_daterange[1]);
self::SaveData($_url,$finalName);
} else { return false; }
}
/**
* Downloads the Crawl Errors file based on the given URL.
*
* #param $site String Site URL available in GWT Account.
* #param $savepath String Optional: Path to save CSV to (no trailing slash!).
* #param $separated Boolean Optional: If true, the method saves separated CSV files
* for each error type. Default: Merge errors in one file.
*/
public function DownloadCSV_CrawlErrors($site, $savepath=".", $separated=false)
{
if(self::IsLoggedIn() === true) {
$type_param = "we";
$filename = parse_url($site, PHP_URL_HOST) ."-". date("Ymd-His");
if($separated) {
foreach($this->_errTablesSort as $sortid => $sortname) {
foreach($this->_errTablesType as $typeid => $typename) {
if($typeid == 1) {
$type_param = "mx";
} else if($typeid == 2) {
$type_param = "mc";
} else {
$type_param = "we";
}
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$typename-$sortname-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=%s&sort=%s";
$_url = sprintf($url, $this->_language, $site, $token, $typeid, $sortid);
self::SaveData($_url,$finalName);
}
}
}
else {
$uri = self::SERVICEURI."crawl-errors?hl=en&siteUrl=$site&tid=$type_param";
$token = self::GetToken($uri,"x26");
$finalName = "$savepath/CRAWL_ERRORS-$filename.csv";
$url = self::SERVICEURI."crawl-errors-dl?hl=%s&siteUrl=%s&security_token=%s&type=0";
$_url = sprintf($url, $this->_language, $site, $token);
self::SaveData($_url,$finalName);
}
} else { return false; }
}
/**
* Saves data to a CSV file based on the given URL.
*
* #param $finalUrl String CSV Download URI.
* #param $finalName String Filepointer to save location.
*/
private function SaveData($finalUrl, $finalName)
{
$data = self::GetData($finalUrl);
if(strlen($data) > 1 && file_put_contents($finalName, utf8_decode($data))) {
array_push($this->_downloaded, realpath($finalName));
return true;
} else {
array_push($this->_skipped, $finalName);
return false;
}
}
/**
* Regular Expression to find the Security Token for a download file.
*
* #param $uri String A Webmaster Tools Desktop Service URI.
* #param $delimiter String Trailing delimiter for the regex.
* #return String Returns a security token.
*/
private function GetToken($uri, $delimiter)
{
$matches = array();
$tmp = self::GetData($uri);
//preg_match_all("#x26security_token(.*?)$delimiter#si", $tmp, $matches);
preg_match_all("#46security_token(.*?)$delimiter#si", $tmp, $matches);
//return substr($matches[1][0],4,-1);
return substr($matches[1][0],3,-1);
}
/**
* Validates ISO 8601 date format.
*
* #param $str String Valid ISO 8601 date string (eg. 2012-01-01).
* #return Boolean Returns true if string has valid format, else false.
*/
private function IsISO8601($str)
{
$stamp = strtotime($str);
return (is_numeric($stamp) && checkdate(date('m', $stamp),
date('d', $stamp), date('Y', $stamp))) ? true : false;
}
}
?>
File #2: ( index.php )
<?php
include 'gwtdata.php';
include 'credentials.php';
try {
$website = "http://www.yourdomain.com/"; /* Add Your Website Url */
$gdata = new GWTdata();
if($gdata->LogIn($email, $password) === true)
{
$gdata->DownloadCSV($website,"Here Add Your Folder Path To Save CSV File With GWT Data");
echo "Datas Are Successfully Downloaded";
}
} catch (Exception $e) {
die($e->getMessage());
}
?>
Can anyone help me in this, to achieve all those datas and make it as excel file to generate using PHP.
[..] I have searched some of the API Documents and Implements, [..]
[..] I have used two files in PHP for this GWT API, [..]
I am the author of the code that you quote (GWTdata PHP class) and first off want to make clear that this code is neither released by Google nor makes use of an official API, but is rather a custom script processing data from the web interface.
[..] returning few of the datas only from the GWT. [..]
A couple of weeks ago, there were some changes to the Google Webmaster Tools web interface (which, again, was/is used to process data requests). Thus, it broke some functionality of the PHP class GWTdata - such as downloading the crawl errors.
[..] Can anyone help me in this, to achieve all those datas and make it as excel file to generate using PHP. [..]
Unfortunately, for the most data there is nothing I/we can do about it (since the data is just not accessable any longer).
[..] Still I'm not getting the major parts / datas like,
1. Crawl errors [..]
Anyway, you can use this followup project to get the crawl errors.
GwtCrawlErrors (Download website crawl errors from Google Webmaster Tools as CSV):
https://github.com/eyecatchup/GWT_CrawlErrors-php
The Google API Client for PHP now supports the Webmasters API. Documentation is (as per usual) scarce for the PHP library, but it maps reasonably cleanly on to the methods described in the Webmasters API reference and there are some examples in the code so it's not too hard to get a hold on.

Vqmod improve error logging, by adding the parent file name the error is referring too

A default error logging using vqmod is as follows:
---------- Date: 2012-10-09 19:46:06 ~ IP : 127.0.0.1 ----------
REQUEST URI : /oc/
MOD DETAILS:
modFile : C:\wamp\www\oc\vqmod\xml\templace.xml
id : Template
version : 1.5.2 - 1.5.2.1
vqmver : 1.0.8
author : templace.com
SEARCH NOT FOUND (ABORTING MOD): require_once(foo . 'library/template.php');
----------------------------------------------------------------------
Example vqmod causing the error
<modification>
<id>Templace</id>
<version>1.5.2 - 1.5.2.1</version>
<author>templace.com</author>
<vqmver>1.0.8</vqmver>
<file name="system/startup.php">
<operation>
<search position="before"><![CDATA[
require_once(foo . 'library/template.php');
]]></search>
<add><![CDATA[
require_once(DIR_SYSTEM . 'library/templace.php');
]]></add>
</operation>
</file>
</modification>
To resole this issue I would have to open the vqmod file templace.xml and search for the file[name] this error is referring too.
QUESTION: How could I add the parent file[name] the actual error is referring too?
E.g adding: "system/startup.php" to the error message to make it easier to debug.
vqmod.php
/**
* VQMod
* #description Main Object used
*/
final class VQMod {
private $_vqversion = '2.1.7';
private $_modFileList = array();
private $_mods = array();
private $_filesModded = array();
private $_cwd = '';
private $_doNotMod = array();
private $_virtualMode = true;
public $useCache = false;
public $logFilePath = 'vqmod/vqmod.log';
public $vqCachePath = 'vqmod/vqcache/';
public $protectedFilelist = 'vqmod/vqprotect.txt';
public $logging = true;
public $cacheTime = 5; // local=5secs live=60secs
public $log;
/**
* VQMod::__construct()
*
* #param bool $path File path to use
* #param bool $logging Enable/disabled logging
* #return null
* #description Startup of VQMod
*/
public function __construct($path = false, $logging = true) {
if(!class_exists('DOMDocument')) {
die('ERROR - YOU NEED DOMDocument INSTALLED TO USE VQMod');
}
if(!$path){
$path = dirname(dirname(__FILE__));
}
$this->_setCwd($path);
$this->logging = (bool) $logging;
$this->log = new VQModLog($this);
$this->_getMods();
$this->_loadProtected();
}
/**
* VQMod::modCheck()
*
* #param string $sourceFile path for file
* #return string
* #description Checks if a file has modifications and applies them, returning cache files or the file name
*/
public function modCheck($sourceFile) {
if(!preg_match('%^([a-z]:)?[\\\\/]%i', $sourceFile)) {
$sourcePath = $this->path($sourceFile);
} else {
$sourcePath = realpath($sourceFile);
}
if(!$sourcePath || is_dir($sourcePath) || in_array($sourcePath, $this->_doNotMod)) {
return $sourceFile;
}
$stripped_filename = preg_replace('~^' . preg_quote($this->getCwd(), '~') . '~', '', $sourcePath);
$cacheFile = $this->_cacheName($stripped_filename);
if($this->useCache && file_exists($cacheFile)) {
//return $cacheFile; // useCache being Deprecated in favor of cacheTime
}
if(isset($this->_filesModded[$sourcePath])) {
return $this->_filesModded[$sourcePath]['cached'] ? $cacheFile : $sourceFile;
}
$changed = false;
$fileHash = sha1_file($sourcePath);
$fileData = file_get_contents($sourcePath);
foreach($this->_mods as $modObject) {
foreach($modObject->mods as $path => $mods) {
if($this->_checkMatch($path, $sourcePath)) {
$modObject->applyMod($mods, $fileData);
}
}
}
// START QPHORIA CACHELOCK CODE
//
if (sha1($fileData) != $fileHash) {
$writePath = $cacheFile;
$cacheLock = false;
if(file_exists($writePath) && ((filemtime($writePath) + (float)$this->cacheTime) >= time())) {
$cacheLock = true;
$changed = true;
}
if(!$cacheLock && (!file_exists($writePath) || is_writable($writePath))) {
file_put_contents($writePath, $fileData);
$changed = true;
} else {
//file_put_contents('./cachelock.txt', "$writePath \r\n", FILE_APPEND); // debugging only.
}
//file_put_contents('./cachetotal.txt', "$writePath \r\n", FILE_APPEND);
} // END QPHORIA CACHELOCK CODE
/* Original Code
if(sha1($fileData) != $fileHash) {
$writePath = $this->_virtualMode ? $cacheFile : $sourcePath;
if(!file_exists($writePath) || is_writable($writePath)) {
file_put_contents($writePath, $fileData);
$changed = true;
}
}*/
$this->_filesModded[$sourcePath] = array('cached' => $changed);
return $changed ? $writePath : $sourcePath;
}
/**
* VQMod::path()
*
* #param string $path File path
* #param bool $skip_real If true path is full not relative
* #return bool, string
* #description Returns the full true path of a file if it exists, otherwise false
*/
public function path($path, $skip_real = false) {
$tmp = $this->_cwd . $path;
$realpath = $skip_real ? $tmp : realpath($tmp);
if(!$realpath) {
return false;
}
if(is_dir($realpath)) {
$realpath = rtrim($realpath, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR;
}
return $realpath;
}
/**
* VQMod::getCwd()
*
* #return string
* #description Returns current working directory
*/
public function getCwd() {
return $this->_cwd;
}
/**
* VQMod::_getMods()
*
* #return null
* #description Gets list of XML files in vqmod xml folder for processing
*/
private function _getMods() {
$this->_modFileList = glob($this->path('vqmod/xml/') . '*.xml');
if($this->_modFileList) {
$this->_parseMods();
} else {
$this->log->write('NO MODS IN USE');
}
}
/**
* VQMod::_parseMods()
*
* #return null
* #description Loops through xml files and attempts to load them as VQModObject's
*/
private function _parseMods() {
$dom = new DOMDocument('1.0', 'UTF-8');
foreach($this->_modFileList as $modFileKey => $modFile) {
if(file_exists($modFile)) {
if(#$dom->load($modFile)) {
$mod = $dom->getElementsByTagName('modification')->item(0);
$this->_mods[] = new VQModObject($mod, $modFile, $this);
} else {
$this->log->write('DOM UNABLE TO LOAD: ' . $modFile);
}
} else {
$this->log->write('FILE NOT FOUND: ' . $modFile);
}
}
}
/**
* VQMod::_loadProtected()
*
* #return null
* #description Loads protected list and adds them to _doNotMod array
*/
private function _loadProtected() {
$file = $this->path($this->protectedFilelist);
if($file && is_file($file)) {
$protected = file_get_contents($file);
if(!empty($protected)) {
$protected = preg_replace('~\r?\n~', "\n", $protected);
$paths = explode("\n", $protected);
foreach($paths as $path) {
$fullPath = $this->path($path);
if($fullPath && !in_array($fullPath, $this->_doNotMod)) {
$this->_doNotMod[] = $fullPath;
}
}
}
}
}
/**
* VQMod::_cacheName()
*
* #param string $file Filename to be converted to cache filename
* #return string
* #description Returns cache file name for a path
*/
private function _cacheName($file) {
return $this->path($this->vqCachePath) . 'vq2-' . preg_replace('~[/\\\\]+~', '_', $file);
}
/**
* VQMod::_setCwd()
*
* #param string $path Path to be used as current working directory
* #return null
* #description Sets the current working directory variable
*/
private function _setCwd($path) {
$realpath = realpath($path);
if(!$realpath) {
die('COULDNT RESOLVE CWD REALPATH');
}
$this->_cwd = rtrim($realpath, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR;
}
/**
* VQMod::_checkMatch()
*
* #param string $modFilePath Modification path from a <file> node
* #param string $checkFilePath File path
* #return bool
* #description Checks a modification path against a file path
*/
private function _checkMatch($modFilePath, $checkFilePath) {
$modFilePath = str_replace('\\', '/', $modFilePath);
$checkFilePath = str_replace('\\', '/', $checkFilePath);
$modFilePath = preg_replace('/([^*]+)/e', 'preg_quote("$1", "~")', $modFilePath);
$modFilePath = str_replace('*', '[^/]*', $modFilePath);
$return = (bool) preg_match('~^' . $modFilePath . '$~', $checkFilePath);
return $return;
}
}
/**
* VQModLog
* #description Object to log information to a file
*/
class VQModLog {
private $_sep;
private $_vqmod;
private $_defhash = 'da39a3ee5e6b4b0d3255bfef95601890afd80709';
private $_logs = array();
/**
* VQModLog::__construct()
*
* #param VQMod $vqmod VQMod main class as reference
* #return null
* #description Object instantiation method
*/
public function __construct(VQMod $vqmod) {
$this->_vqmod = $vqmod;
$this->_sep = str_repeat('-', 70);
}
/**
* VQModLog::__destruct()
*
* #return null
* #description Logs any messages to the log file just before object is destroyed
*/
public function __destruct() {
if(empty($this->_logs) || $this->_vqmod->logging == false) {
return;
}
$txt = array();
$txt[] = str_repeat('-', 10) . ' Date: ' . date('Y-m-d H:i:s') . ' ~ IP : ' . (isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : 'N/A') . ' ' . str_repeat('-', 10);
$txt[] = 'REQUEST URI : ' . $_SERVER['REQUEST_URI'];
foreach($this->_logs as $count => $log) {
if($log['obj']) {
$vars = get_object_vars($log['obj']);
$txt[] = 'MOD DETAILS:';
foreach($vars as $k => $v) {
if(is_string($v)) {
$txt[] = ' ' . str_pad($k, 10, ' ', STR_PAD_RIGHT) . ': ' . $v;
}
}
}
foreach($log['log'] as $msg) {
$txt[] = $msg;
}
if ($count > count($this->_logs)-1) {
$txt[] = '';
}
}
$txt[] = $this->_sep;
$txt[] = str_repeat(PHP_EOL, 2);
$logPath = $this->_vqmod->path($this->_vqmod->logFilePath, true);
if(!file_exists($logPath)) {
$res = file_put_contents($logPath, '');
if($res === false) {
die('COULD NOT WRITE TO LOG FILE');
}
}
file_put_contents($logPath, implode(PHP_EOL, $txt), FILE_APPEND);
}
/**
* VQModLog::write()
*
* #param string $data Text to be added to log file
* #param VQModObject $obj Modification the error belongs to
* #return null
* #description Adds error to log object ready to be output
*/
public function write($data, VQModObject $obj = NULL) {
if($obj) {
$hash = sha1($obj->id);
} else {
$hash = $this->_defhash;
}
if(empty($this->_logs[$hash])) {
$this->_logs[$hash] = array(
'obj' => $obj,
'log' => array()
);
}
$this->_logs[$hash]['log'][] = $data;
}
}
/**
* VQModObject
* #description Object for the <modification> that orchestrates each applied modification
*/
class VQModObject {
public $modFile = '';
public $id = '';
public $version = '';
public $vqmver = '';
public $author = '';
public $mods = array();
private $_vqmod;
private $_skip = false;
/**
* VQModObject::__construct()
*
* #param DOMNode $node <modification> node
* #param string $modFile File modification is from
* #param VQMod $vqmod VQMod object as reference
* #return null
* #description Loads modification meta information
*/
public function __construct(DOMNode $node, $modFile, VQmod $vqmod) {
if($node->hasChildNodes()) {
foreach($node->childNodes as $child) {
$name = (string) $child->nodeName;
if(isset($this->$name)) {
$this->$name = (string) $child->nodeValue;
}
}
}
$this->modFile = $modFile;
$this->_vqmod = $vqmod;
$this->_parseMods($node);
}
/**
* VQModObject::skip()
*
* #return bool
* #description Returns the skip status of a modification
*/
public function skip() {
return $this->_skip;
}
/**
* VQModObject::applyMod()
*
* #param array $mods Array of search add nodes
* #param string $data File contents to be altered
* #return null
* #description Applies all modifications to the text data
*/
public function applyMod($mods, &$data) {
if($this->_skip) return;
$tmp = $data;
foreach($mods as $mod) {
$indexCount = 0;
$tmp = $this->_explodeData($tmp);
$lineMax = count($tmp) - 1;
switch($mod['search']->position) {
case 'top':
$tmp[$mod['search']->offset] = $mod['add']->getContent() . $tmp[$mod['search']->offset];
break;
case 'bottom':
$offset = $lineMax - $mod['search']->offset;
if($offset < 0){
$tmp[-1] = $mod['add']->getContent();
} else {
$tmp[$offset] .= $mod['add']->getContent();
}
break;
case 'all':
$tmp = array($mod['add']->getContent());
break;
default:
$changed = false;
foreach($tmp as $lineNum => $line) {
if($mod['search']->regex == 'true') {
$pos = #preg_match($mod['search']->getContent(), $line);
if($pos === false) {
if($mod['error'] == 'log' || $mod['error'] == 'abort' ) {
$this->_vqmod->log->write('INVALID REGEX ERROR - ' . $mod['search']->getContent(), $this);
}
continue 2;
} elseif($pos == 0) {
$pos = false;
}
} else {
$pos = strpos($line, $mod['search']->getContent());
}
if($pos !== false) {
$indexCount++;
$changed = true;
if(!$mod['search']->indexes() || ($mod['search']->indexes() && in_array($indexCount, $mod['search']->indexes()))) {
switch($mod['search']->position) {
case 'before':
$offset = ($lineNum - $mod['search']->offset < 0) ? -1 : $lineNum - $mod['search']->offset;
$tmp[$offset] = empty($tmp[$offset]) ? $mod['add']->getContent() : $mod['add']->getContent() . "\n" . $tmp[$offset];
break;
case 'after':
$offset = ($lineNum + $mod['search']->offset > $lineMax) ? $lineMax : $lineNum + $mod['search']->offset;
$tmp[$offset] = $tmp[$offset] . "\n" . $mod['add']->getContent();
break;
default:
if(!empty($mod['search']->offset)) {
for($i = 1; $i <= $mod['search']->offset; $i++) {
if(isset($tmp[$lineNum + $i])) {
$tmp[$lineNum + $i] = '';
}
}
}
if($mod['search']->regex == 'true') {
$tmp[$lineNum] = preg_replace($mod['search']->getContent(), $mod['add']->getContent(), $line);
} else {
$tmp[$lineNum] = str_replace($mod['search']->getContent(), $mod['add']->getContent(), $line);
}
break;
}
}
}
}
if(!$changed) {
$skip = ($mod['error'] == 'skip' || $mod['error'] == 'log') ? ' (SKIPPED)' : ' (ABORTING MOD)';
if($mod['error'] == 'log' || $mod['error'] == 'abort') {
$this->_vqmod->log->write('SEARCH NOT FOUND' . $skip . ': ' . $mod['search']->getContent(), $this);
}
if($mod['error'] == 'abort') {
$this->_skip = true;
return;
}
}
break;
}
ksort($tmp);
$tmp = $this->_implodeData($tmp);
}
$data = $tmp;
}
/**
* VQModObject::_parseMods()
*
* #param DOMNode $node <modification> node to be parsed
* #return null
* #description Parses modifications in preparation for the applyMod method to work
*/
private function _parseMods(DOMNode $node){
$files = $node->getElementsByTagName('file');
foreach($files as $file) {
$fileToMod = $file->getAttribute('name');
$error = ($file->hasAttribute('error')) ? $file->getAttribute('error') : 'log';
$fullPath = $this->_vqmod->path($fileToMod);
if(!$fullPath){
if(strpos($fileToMod, '*') !== false) {
$fullPath = $this->_vqmod->getCwd() . $fileToMod;
} else {
if ($error == 'log' || $error == 'abort') {
$skip = ($error == 'log') ? ' (SKIPPED)' : ' (ABORTING MOD)';
$this->_vqmod->log->write('Could not resolve path for [' . $fileToMod . ']' . $skip, $this);
}
if ($error == 'log' || $error == 'skip') {
continue;
} elseif ($error == 'abort') {
return false;
}
}
}
$operations = $file->getElementsByTagName('operation');
foreach($operations as $operation) {
$error = ($operation->hasAttribute('error')) ? $operation->getAttribute('error') : 'abort';
$this->mods[$fullPath][] = array(
'search' => new VQSearchNode($operation->getElementsByTagName('search')->item(0)),
'add' => new VQAddNode($operation->getElementsByTagName('add')->item(0)),
'error' => $error
);
}
}
}
/**
* VQModObject::_explodeData()
*
* #param string $data File contents
* #return string
* #description Splits a file into an array of individual lines
*/
private function _explodeData($data) {
return explode("\n", $data);
}
/**
* VQModObject::_implodeData()
*
* #param array $data Array of lines
* #return string
* #description Joins an array of lines back into a text file
*/
private function _implodeData($data) {
return implode("\n", $data);
}
}
/**
* VQNode
* #description Basic node object blueprint
*/
class VQNode {
public $trim = 'false';
private $_content = '';
/**
* VQNode::__construct()
*
* #param DOMNode $node Search/add node
* #return null
* #description Parses the node attributes and sets the node property
*/
public function __construct(DOMNode $node) {
$this->_content = $node->nodeValue;
if($node->hasAttributes()) {
foreach($node->attributes as $attr) {
$name = $attr->nodeName;
if(isset($this->$name)) {
$this->$name = $attr->nodeValue;
}
}
}
}
/**
* VQNode::getContent()
*
* #return string
* #description Returns the content, trimmed if applicable
*/
public function getContent() {
$content = ($this->trim == 'true') ? trim($this->_content) : $this->_content;
return $content;
}
}
/**
* VQSearchNode
* #description Object for the <search> xml tags
*/
class VQSearchNode extends VQNode {
public $position = 'replace';
public $offset = 0;
public $index = 'false';
public $regex = 'false';
public $trim = 'true';
/**
* VQSearchNode::indexes()
*
* #return bool, array
* #description Returns the index values to use the search on, or false if none
*/
public function indexes() {
if($this->index == 'false') {
return false;
}
$tmp = explode(',', $this->index);
foreach($tmp as $k => $v) {
if(!is_int($v)) {
unset($k);
}
}
$tmp = array_unique($tmp);
return empty($tmp) ? false : $tmp;
}
}
/**
* VQAddNode
* #description Object for the <add> xml tags
*/
class VQAddNode extends VQNode {
}
Also couple of other ideas to make debugging even easier:
List any other vqmod files which have previously edited this same file.
This is another common issue where I find when two extensions are editing the same file and the latter is causing the error but it would be useful to know about any other vqmods editing the same file. Yes I suppose I could add error="skip" to everything but dont think this is the best approach to just hide all of the errors, the user should be made aware there is an error...
"Suggested Fix", maybe some smart way you can test what type of error it is.
Contradict what I said above but even at its most basic form you could suggest hiding the error if its not essential. So that anybody can read it and understand how it fix it.
E.g
OPEN: vqmod/xml/templace.xml (line:23)
FIND: <operation>
REPLACE <operation error="skip">
Adding the line number in the XML file the error is coming from. It would be lovely not having to search all of the time and could quickly go to the line number in the vqmod
The issue for the file being edited is certainly one that is way overdue and one I plan on adding in the next release of vQmod. As for the other suggestions
Interesting idea, and one that could certainly be considered. The only problem I see with this is that it would possibly make some log files enormous
This is going to be next to impossible to incorporate
This is impossible without some pretty expensive runtime. The error doesn't lie in the XML as such, so would require re-opening the xml that's been parsed, searching for the line in question line by line and then reporting that. it sounds simple, but you have to remember that xml's can have the same search parameter for multiple operations - so in that situation you'd be no better off than searching the file yourself

Categories