Recently tasked to monitor external webpage response/loading time via CACTI. I found some PHP scripts that were working (pageload-agent.php and class.pageload.php) using cURL. All was working fine until they requested it to be transferred from LINUX to Windows 2012R2 server. I'm having a very hard time modifying the scripts to work for windows. Already installed PHP and cURL and both working as tested. Here are the scripts taken from askaboutphp.
class.pageload.php
<?php
class PageLoad {
var $siteURL = "";
var $pageInfo = "";
/*
* sets the URLs to check for loadtime into an array $siteURLs
*/
function setURL($url) {
if (!empty($url)) {
$this->siteURL = $url;
return true;
}
return false;
}
/*
* extract the header information of the url
*/
function doPageLoad() {
$u = $this->siteURL;
if(function_exists('curl_init') && !empty($u)) {
$ch = curl_init($u);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_ENCODING, "gzip");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, false);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)");
$pageBody = curl_exec($ch);
$this->pageInfo = curl_getinfo($ch);
curl_close ($ch);
return true;
}
return false;
}
/*
* compile the page load statistics only
*/
function getPageLoadStats() {
$info = $this->pageInfo;
//stats from info
$s['dest_url'] = $info['url'];
$s['content_type'] = $info['content_type'];
$s['http_code'] = $info['http_code'];
$s['total_time'] = $info['total_time'];
$s['size_download'] = $info['size_download'];
$s['speed_download'] = $info['speed_download'];
$s['redirect_count'] = $info['redirect_count'];
$s['namelookup_time'] = $info['namelookup_time'];
$s['connect_time'] = $info['connect_time'];
$s['pretransfer_time'] = $info['pretransfer_time'];
$s['starttransfer_time'] = $info['starttransfer_time'];
return $s;
}
}
?>
pageload-agent.php
#! /usr/bin/php -q
<?php
//include the class
include_once 'class.pageload.php';
// read in an argument - must make sure there's an argument to use
if ($argc==2) {
//read in the arg.
$url_argv = $argv[1];
if (!eregi('^http://', $url_argv)) {
$url_argv = "http://$url_argv";
}
// check that the arg is not empty
if ($url_argv!="") {
//initiate the results array
$results = array();
//initiate the class
$lt = new PageLoad();
//set the page to check the loadtime
$lt->setURL($url_argv);
//load the page
if ($lt->doPageLoad()) {
//load the page stats into the results array
$results = $lt->getPageLoadStats();
} else {
//do nothing
print "";
}
//print out the results
if (is_array($results)) {
//expecting only one record as we only passed in 1 page.
$output = $results;
print "dns:".$output['namelookup_time'];
print " con:".$output['connect_time'];
print " pre:".$output['pretransfer_time'];
print " str:".$output['starttransfer_time'];
print " ttl:".$output['total_time'];
print " sze:".$output['size_download'];
print " spd:".$output['speed_download'];
} else {
//do nothing
print "";
}
}
} else {
//do nothing
print "";
}
?>
Thank you. any type of assistance is greatly appreciated.
Related
I have been working on a clients WP site which lists deals from Groupon. I am using the Groupon's official XML feed, importing via WP All Import. This works without much hassle. Now the issue is Groupon doesn't update that feed frequently but some of their deals get sold out or off the market often. So to get this resolved what I am trying is using a CURL script to crawl the links and check if the deal is available or not then turn the unavailable deals to draft posts (Once a day only).
The custom script is working almost perfectly, only after the first 14/24 requests the server starts responding with 502/503 HTTP status codes. To overcome the issue I have used the below precautions -
Using the proper header (captured from the requests made by the browser)
Parsing cookies from response header and sending back.
Using proper referrer and user agent.
Using proxies.
Trying to send request after a set interval. PHP - sleep(5);
Unfortunately, none of this got me the solution I wanted. I am attaching my code and I would like to request your expert insights on the issue, please.
Thanks in advance for your time.
Shahriar
PHP SCRIPT - https://pastebin.com/FF2cNm5q
<?php
// Error supressing and extend maximum execution time
error_reporting(0);
ini_set('max_execution_time', 50000);
// Sitemap URL List
$all_activity_urls = array();
$sitemap_url = array(
'https://www.groupon.de/sitemaps/deals-local0.xml.gz'
);
$cookies = Array();
// looping through sitemap url for scraping activity urls
for ($u = 0; $u < count($sitemap_url); $u++)
{
$ch1 = curl_init();
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch1, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0');
curl_setopt($ch1, CURLOPT_REFERER, "https://www.groupon.de/");
curl_setopt($ch1, CURLOPT_TIMEOUT, 40);
// curl_setopt($ch1, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch1, CURLOPT_URL, $sitemap_url[$u]);
curl_setopt($ch1, CURLOPT_SSL_VERIFYPEER, FALSE);
// Parsing Cookie from the response header
curl_setopt($ch1, CURLOPT_HEADERFUNCTION, "curlResponseHeaderCallback");
$activity_url_source = curl_exec($ch1);
$status_code = curl_getinfo($ch1, CURLINFO_HTTP_CODE);
curl_close($ch1);
if ($status_code === 200)
{
// Parsing XML sitemap for activity urls
$activity_url_list = json_decode(json_encode(simplexml_load_string($activity_url_source)));
for ($a = 0; $a < count($activity_url_list->url); $a++)
{
array_push($all_activity_urls, $activity_url_list->url[$a]->loc);
}
}
}
if (count($all_activity_urls) > 0)
{
// URL Loop count
$loop_from = 0;
$loop_to = (count($all_activity_urls) > 0) ? 100 : 0;
// $loop_to = count($all_activity_urls);
$final_data = array();
echo 'script start - ' . date('h:i:s') . "<br>";
for ($u = $loop_from; $u < $loop_to; $u++)
{
//Pull source from webpage
$headers = array(
'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'accept-language: en-US,en;q=0.9,bn-BD;q=0.8,bn;q=0.7,it;q=0.6',
'cache-control: max-age=0',
'cookie: ' . implode('; ', $cookies),
'upgrade-insecure-requests: 1',
'user-agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
);
$site = $all_activity_urls[$u];
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_REFERER, "https://www.groupon.de/");
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $site);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
// Parsing Cookie from the response header
curl_setopt($ch, CURLOPT_HEADERFUNCTION, "curlResponseHeaderCallback");
$data = curl_exec($ch);
$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($status_code === 200)
{
// Ready data for parsing
$document = new DOMDocument();
$document->loadHTML('<meta http-equiv="content-type" content="text/html; charset=utf-8">' . $data);
$xpath = new DOMXpath($document);
$title = '';
$availability = '';
$price = '';
$base_price = '';
$link = '';
$image = '';
$link = $all_activity_urls[$u];
// Scraping Availability
$raw_availability = $xpath->query('//div[#data-bhw="DealHighlights"]/div[0]/div/div');
$availability = $raw_availability->item(0)->nodeValue;
// Scraping Title
$raw_title = $xpath->query('//h1[#id="deal-title"]');
$title = $raw_title->item(0)->nodeValue;
// Scraping Price
$raw_price = $xpath->query('//div[#class="price-discount-wrapper"]');
$price = trim(str_replace(array("$", "€", "US", " "), array("", "", "", ""), $raw_price->item(0)->nodeValue));
// Scraping Old Price
$raw_base_price = $xpath->query('//div[contains(#class, "value-source-wrapper")]');
$base_price = trim(str_replace(array("$", "€", "US", " "), array("", "", "", ""), $raw_base_price->item(0)->nodeValue));
// Creating Final Data Array
array_push($final_data, array(
'link' => $link,
'availability' => $availability,
'name' => $title,
'price' => $price,
'baseprice' => $base_price,
'img' => $image,
));
}
else
{
$link = $all_activity_urls[$u];
if ($status_code === 429)
{
$status_msg = ' - Too Many Requests';
}
else
{
$status_msg = '';
}
array_push($final_data, array(
'link' => $link,
'status' => $status_code . $status_msg,
));
}
echo 'before break - ' . date('h:i:s') . "<br>";
sleep(5);
echo 'after break - ' . date('h:i:s') . "<br>";
flush();
}
echo 'script end - ' . date('h:i:s') . "<br>";
// Converting data to XML
$activities = new SimpleXMLElement("<?xml version=\"1.0\"?><activities></activities>");
array_to_xml($final_data, $activities);
$xml_file = $activities->asXML('activities.xml');
if ($xml_file)
{
echo 'XML file have been generated successfully.';
}
else
{
echo 'XML file generation error.';
}
}
else
{
$activities = new SimpleXMLElement("<?xml version=\"1.0\"?><activities></activities>");
$activities->addChild("error", htmlspecialchars("No URL scraped from sitemap. Stoping script."));
$xml_file = $activities->asXML('activities.xml');
if ($xml_file)
{
echo 'XML file have been generated successfully.';
}
else
{
echo 'XML file generation error.';
}
}
// Recursive Function for creating XML Nodes
function array_to_xml($array, &$activities)
{
foreach ($array as $key => $value)
{
if (is_array($value))
{
if (!is_numeric($key))
{
$subnode = $activities->addChild("$key");
array_to_xml($value, $subnode);
}
else
{
$subnode = $activities->addChild("activity");
array_to_xml($value, $subnode);
}
}
else
{
$activities->addChild("$key", htmlspecialchars("$value"));
}
}
}
// Cookie Parsing Function
function curlResponseHeaderCallback($ch, $headerLine)
{
global $cookies;
if (preg_match('/^Set-Cookie:\s*([^;]*)/mi', $headerLine, $cookie) == 1)
{
$cookies[] = $cookie[1];
}
return strlen($headerLine); // Needed by curl
}
There is a mess of cookies in your snippet. The callback function just appends cookies to the array regardingless of whether they already exist or not. Here is a new version which at least seems to work in this case since there are no semicolon-seperated multiple cookie definitions. Usually the cookie string should be even parsed. If you have installed the http extension you can use http_parse_cookie.
// Cookie Parsing Function
function curlResponseHeaderCallback($ch, $headerLine)
{
global $cookies;
if (preg_match('/^Set-Cookie:\s*([^;]+)/mi', $headerLine, $match) == 1)
{
if(false !== ($p = strpos($match[1], '=')))
{
$replaced = false;
$cname = substr($match[1], 0, $p+1);
foreach ($cookies as &$cookie)
if(0 === strpos($cookie, $cname))
{
$cookie = $match[1];
$replaced = true;
break;
}
if(!$replaced)
$cookies[] = $match[1];
}
var_dump($cookies);
}
return strlen($headerLine); // Needed by curl
}
<?php
if(isset($_POST["submit"]))
{
$adm=$_POST["admno"];
$phn=$_POST["phn1"];
include("model.php");
$db = new database;
$r=$db->register($adm);
while($row=mysql_fetch_array($r))
{
if($row["phn_no1"]==$phn || $row["phn_no2"]==$phn || $row["phn_no3"]==$phn)
{
$formatted = "".substr($phn,6,10)." ";
$password = $formatted + $adm;
echo $password;
$db->setpassword($adm,$password);
$pre = 'PREFIX';
$suf = '%20ThankYou';
$sms = $pre.$password.$suf;
session_start();
$ch = curl_init("http://www.perfectbulksms.in/Sendsmsapi.aspx? USERID=ID&PASSWORD=PASS&SENDERID=SID&TO=$phn&MESSAGE=$sms");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
$result = curl_exec($ch);
curl_close($ch);
header("Location:password.php?msg=new");
}
else
{
header("Location:register.php?msg=invalid");
}
}
}
?>
this code is working perfect on my local host .. but when i put it on server ... it takes lots of time but the code in curl command is not working it only refers to next page ... i checked that curl is enabled .. if i use only sms api without curl command it sends sms immidiately.... but i want to run both header and also want to hide my sms api.... is there any alternate of this ???
I am trying to search (filter) for files in a Dropbox folder, but no files are being found when there are files that match the filter. I am not using the PHP library provided by Dropbox.
Here is an extract of the code:
class Dropbox {
private $headers = array();
private $authQueryString = "";
public $SubFolders = array();
public $Files = array();
function __construct() {
$this->headers = array('Authorization: OAuth oauth_version="1.0", oauth_signature_method="PLAINTEXT", oauth_consumer_key="'.DROPBOX_APP_KEY.'", oauth_token="'.DROPBOX_OAUTH_ACCESS_TOKEN.'", oauth_signature="'.DROPBOX_APP_SECRET.'&'.DROPBOX_OAUTH_ACCESS_SECRET.'"');
$this->authQueryString = "oauth_consumer_key=".DROPBOX_APP_KEY."&oauth_token=".DROPBOX_OAUTH_ACCESS_TOKEN."&oauth_signature_method=PLAINTEXT&oauth_signature=".DROPBOX_APP_SECRET."%26".DROPBOX_OAUTH_ACCESS_SECRET."&oauth_version=1.0";
}
public function GetFolder($folder, $fileFilter = "") {
//Add the required folder to the end of the base path for folder call
if ($fileFilter == "")
$subPath = "metadata/sandbox";
else
$subPath = "search/sandbox";
if (strlen($folder) > 1) {
$subPath .= (substr($folder, 0, 1) != "/" ? "/" : "")
.$folder;
}
//Set up the post parameters for the call
$params = null;
if ($fileFilter != "") {
$params = array(
"query" => $fileFilter
);
}
//Clear the sub folders and files logged
$this->SubFolders = array();
$this->Files = array();
//Make the call
$content = $this->doCall($subPath, $params);
//Log the files and folders
for ($i = 0; $i < sizeof($content->contents); $i++) {
$f = $content->contents[$i];
if ($f->is_dir == "1") {
array_push($this->SubFolders, $f->path);
} else {
array_push($this->Files, $f->path);
}
}
//Return the content
return $content;
}
private function doCall($urlSubPath, $params = null, $filePathName = null, $useAPIContentPath = false) {
//Create the full URL for the call
$url = "https://api".($useAPIContentPath ? "-content" : "").".dropbox.com/1/".$urlSubPath;
//Initialise the curl call
$ch = curl_init();
//Set up the curl call
curl_setopt($ch, CURLOPT_HTTPHEADER, $this->headers);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
if ($params != null)
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
$fh = null;
if ($filePathName != null) {
$fh = fopen($filePathName, "rb");
curl_setopt($context, CURLOPT_BINARYTRANSFER, true);
curl_setopt($context, CURLOPT_INFILE, $fh);
curl_setopt($context, CURLOPT_INFILESIZE, filesize($filePathName));
}
//Excecute and get the response
$api_response = curl_exec($ch);
if ($fh != null)
fclose($fh);
//Process the response into an array
$json_response = json_decode($api_response);
//Has there been an error
if (isset($json_response->error )) {
throw new Exception($json_response["error"]);
}
//Send the response back
return $json_response;
}
}
I then call the GetFolder method of Dropbox as such:
$dbx = new Dropbox();
$filter = "MyFilter";
$dbx->GetFolder("MyFolder", $filter);
print "Num files: ".sizeof($dbx->Files);
As I am passing $filter into GetFolder, it uses the search/sandbox path and creates a parameter array ($params) with the required query parameter in it.
The process works fine if I don't provide the $fileFilter parameter to GetFolder and all files in the folder are returned (uses the metadata/sandbox path).
Other methods (that are not in the extract for brevity) of the Dropbox class use the $params feature and they to work fine.
I have been using the Dropbpox API reference for guidance (https://www.dropbox.com/developers/core/docs#search)
At first glance, it looks like you're making a GET request to /search but passing parameters via CURLOPT_POSTFIELDS. Try using a POST or encoding the search query as a query string parameter.
EDIT
Below is some code that works for me (usage: php search.php <term>). Note that I'm using OAuth 2 instead of OAuth 1, so my Authorization header looks different from yours.
<?php
$access_token = '<REDACTED>';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.dropbox.com/1/search/auto');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Authorization:Bearer ' . $access_token));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('query' => $argv[1]));
$api_response = curl_exec($ch);
echo "Matching files:\n\t" . join("\n\t",
array_map(function ($file) {
return $file['path'];
}, json_decode($api_response, true)))."\n";
?>
Thanks guys and gals got it working
//create a function
function get_stock_data($symbol){
//set up the url to be called
$revenue_url = "http://finance.yahoo.com/q/is?s=".$symbol;
//curl call:
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $revenue_url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// grab URL and pass it to the browser
$result = curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
//finish by returning the result
return $result;
}
//REQUEST WILL BE POPULATED IF EITHER GET OR POST IS SET!
$data = null; // this will hold our data, declared here for accessibility
if(isset($_REQUEST['symbol']) && $_REQUEST['symbol'] != ''){
//call our get_data function
$data = get_stock_data($_REQUEST['symbol']);
}
// data returned from our get_stock_data() call.
$ppe = $data['ppe'];
$revenue = $data['revenue'];
$income = $data['income'];
$market_cap = $data['market_cap'];
$depreciation = $data['depreciation'];
$rate_of_return = $data['rate_of_return'];
$rate_of_return_w_ppe = $data['rate_of_return_w_ppe'];
$debt = $data['debt'];
}
Add following code in your update button(page) script at last
<script type="text/javascript">
var php_var = "<?php echo $symbol; ?>";
locationInfo="stock_next.php?symbol="+php_var;
setTimeout(function(){
location =locationInfo
},2000)
</script>
Your page will be automatically updated after some seconds
I'm writing an application to get all the relevant media based to a user or a tag.
I was able to the media but the resolution of the user's profile picture found under data/user/profile_picture is quite poor (around 150*150px).
So my question is : is there anyway to get a user profile's picture in a bigger size ? Here are the queries I use to retrieve the media :
https://api.instagram.com/v1/users/3/media/recent/?access_token=ACCESS-TOKEN
https://api.instagram.com/v1/tags/snow/media/recent?access_token=ACCESS-TOKEN
This gets the 600x600 profile picture:
function Request($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
function get_value($username, $att, $accesstoken) {
$url = "https://api.instagram.com/v1/users/search?q=" . $username . "&access_token=" . $accesstoken;
if($result = json_decode(Request($url), true)) {
if ($att == "full_name") {
return preg_replace("/[^A-Za-z0-9 ]/", '', $result['data'][0][$att]);
} elseif ($att == "profile_picture") {
$res = str_replace("s480x480", "s600x600", $result['data'][0][$att]);
$res = str_replace("s320x320", "s600x600", $res);
$res = str_replace("s150x150", "s600x600", $res);
return $res;
} else {
return $result['data'][0][$att];
}
}
}
Example Usage:
$profile_picture = get_value("USERNAME","profile_picture", "ACCESS_TOKEN");