php RRD graph separation - php

I am trying to create RRD graphs with the help of PHP in order to keep track of the inoctets,outoctets and counter of a server.
So far the script is operating as expected but my problems comes when I am trying to produce 2 or more separate graphs. I am trying to produce (hourly, weekly , etc) graphs. I thought by creating a loop would solve my problem, since I have split the RRA in hours and days. Unfortunately I end up having 2 graphs that updating simultaneously as expected but plotting the same thing. Has any one encounter similar problem? I have applied the same program in perl with RRD::Simple,where is extremely easy and everything is adjusted almost automatically.
I have supplied under a working example of my code with the minimum possible data because the code is a bit long:
<?php
$file = "snmp-2";
$rrdFile = dirname(__FILE__) . "/snmp-2.rrd";
$in = "ifInOctets";
$out = "ifOutOctets";
$count = "sysUpTime";
$step = 5;
$rounds = 1;
$output = array("Hourly","Daily");
while (1) {
sleep (6);
$options = array(
"--start","now -15s", // Now -10 seconds (default)
"--step", "".$step."",
"DS:".$in.":GAUGE:10:U:U",
"DS:".$out.":GAUGE:10:U:U",
"DS:".$count.":ABSOLUTE:10:0:4294967295",
"RRA:MIN:0.5:12:60",
"RRA:MAX:0.5:12:60",
"RRA:LAST:0.5:12:60",
"RRA:AVERAGE:0.5:12:60",
"RRA:MIN:0.5:300:60",
"RRA:MAX:0.5:300:60",
"RRA:LAST:0.5:300:60",
"RRA:AVERAGE:0.5:300:60",
);
if ( !isset( $create ) ) {
$create = rrd_create(
"".$rrdFile."",
$options
);
if ( $create === FALSE ) {
echo "Creation error: ".rrd_error()."\n";
}
}
$t = time();
$ifInOctets = rand(0, 4294967295);
$ifOutOctets = rand(0, 4294967295);
$sysUpTime = rand(0, 4294967295);
$update = rrd_update(
"".$rrdFile."",
array(
"".$t.":".$ifInOctets.":".$ifOutOctets.":".$sysUpTime.""
)
);
if ($update === FALSE) {
echo "Update error: ".rrd_error()."\n";
}
$start = $t - ($step * $rounds);
foreach ($output as $test) {
$final = array(
"--start","".$start." -15s",
"--end", "".$t."",
"--step","".$step."",
"--title=".$file." RRD::Graph",
"--vertical-label=Byte(s)/sec",
"--right-axis-label=latency(min.)",
"--alt-y-grid", "--rigid",
"--width", "800", "--height", "500",
"--lower-limit=0",
"--alt-autoscale-max",
"--no-gridfit",
"--slope-mode",
"DEF:".$in."_def=".$file.".rrd:".$in.":AVERAGE",
"DEF:".$out."_def=".$file.".rrd:".$out.":AVERAGE",
"DEF:".$count."_def=".$file.".rrd:".$count.":AVERAGE",
"CDEF:inbytes=".$in."_def,8,/",
"CDEF:outbytes=".$out."_def,8,/",
"CDEF:counter=".$count."_def,8,/",
"COMMENT:\\n",
"LINE2:".$in."_def#FF0000:".$in."",
"COMMENT:\\n",
"LINE2:".$out."_def#0000FF:".$out."",
"COMMENT:\\n",
"LINE2:".$count."_def#FFFF00:".$count."",
);
$outputPngFile = rrd_graph(
"".$test.".png",
$final
);
if ($outputPngFile === FALSE) {
echo "<b>Graph error: </b>".rrd_error()."\n";
}
} /* End of foreach function */
$debug = rrd_lastupdate (
"".$rrdFile.""
);
if ($debug === FALSE) {
echo "<b>Graph result error: </b>".rrd_error()."\n";
}
var_dump ($debug);
$rounds++;
} /* End of while loop */
?>

A couple of issues.
Firstly, your definition of the RRD has a step of 5seconds and RRAs with steps of 12x5s=1min and 300x5s=25min. They also have a length of only 60 rows, so 1hr and 25hr respectively. You'll never get a weekly graph this way! You need to add more rows; also the step seems rather short, and you might need a smaller-step RRA for hourly graphs and a larger-step one for weekly graphs.
Secondly, it is not clear how you're calling the graph function. You seem to be specifying:
"--start","".$start." -15s",
"--end", "".$t."",
"--step","".$step."",
... which would force it to use the 5s interval (unavailable, so the 1min one would always get used) and for the graph to be only for the time window from the start to the last update, not a 'hourly' or 'daily' as you were asking.
Note that the RRA you have defined do not define the time window of the graph you are asking for. Also, just because you have more than one RRA defined, it doesnt mean you'll get more than one graph unless oyu call the graph function twice with different arguments.
If you want a daily graph, use
"--start","end - 1 hour",
"--end",$t,
Do not specify a step as the most appropriate available will be used anyway. For a daily graph, use
"--start","end - 1 day"
"--end",$t,
Similarly, no need to specify a step.
Hopefully this will make it a little clearer. Most of the RRD graph options have sensible defaults, and RRDTool is pretty good at picking the correct RRA to use based on the graph size, time window, and DEF statements.

Related

Solution for calling a function doing lots of stuff in it by Cron?

function cronProcess() {
# > 100,000 users
$users = $this->UserModel->getUsers();
foreach ($users as $user) {
# Do lots of database Insert/Update/Delete, HTTP request stuff
}
}
The problem happens when the number of users reaches ~ 100,000.
I called the function by CURL via CronTab.
So what is the best solution for this?
I do a lot of bulk tasks in CakePHP, some processing millions of records. It's certainly possible to do, the key as others suggested is small batches in a loop.
If this is something you're calling from Cron, it's probably easier to use a Shell (< v3.5) or the newer Command class (v3.6+) than cURL.
Here's generally how I paginate large batches, including some helpful optional things like a progress bar, turning off hydration to speed things up slightly, and showing how many users/second the script was able to process:
<?php
namespace App\Command;
use Cake\Console\Arguments;
use Cake\Console\Command;
use Cake\Console\ConsoleIo;
class UsersCommand extends Command
{
public function execute(Arguments $args, ConsoleIo $io)
{
// I'd guess a Finder would be a more Cake-y way of getting users than a custom "getUsers" function:
// See https://book.cakephp.org/3.0/en/orm/retrieving-data-and-resultsets.html#custom-finder-methods
$usersQuery = $this->UserModel->find('users');
// Get a total so we know how many we're gonna have to process (optional)
$total = $usersQuery->count();
if ($total === 0) {
$this->abort("No users found, stopping..");
}
// Hydration takes extra processing time & memory, which can add up in bulk. Optionally if able, skip it & work with $user as an array not an object:
$usersQuery->enableHydration(false);
$this->info("Grabbing $total users for processing");
// Optionally show the progress so we can visually see how far we are in the process
$progress = $io->helper('Progress')->init([
'total' => 10
]);
// Tune this page value to a size that solves your problem:
$limit = 1000;
$offset = 0;
// Simply drawing the progress bar every loop can slow things down, optionally draw it only every n-loops,
// this sets it to 1/5th the page size:
$progressInterval = $limit / 5;
// Optionally track the rate so we can evaluate the speed of the process, helpful tuning limit and evaluating enableHydration effects
$startTime = microtime(true);
do {
$users = $usersQuery->offset($offset)->toArray();
$count = count($users);
$index = 0;
foreach ($users as $user) {
$progress->increment(1);
// Only draw occasionally, for speed
if ($index % $progressInterval === 0) {
$progress->draw();
}
### WORK TIME
# Do your lots of database Insert/Update/Delete, HTTP request stuff etc. here
###
}
$progress->draw();
$offset += $limit; // Increment your offset to the next page
} while ($count > 0);
$totalTime = microtime(true) - $startTime;
$this->out("\nProcessed an average " . ($total / $totalTime) . " Users/sec\n");
}
}
Checkout these sections in the CakePHP Docs:
Console Commands
Command Helpers
Using Finders & Disabling Hydration
Hope this helps!

How to prin RRD Graph data with PHP

Thanks in advance for your time and effort. This is my first script with PHP and RRD's. While I was writing a short program for SNMP I came across with RRD a powerful tool for graphical output. I tried to wright a short operating script to imitate the output of a graph. I read as much as possible documentations regarding RRD from the official page and tried to add them to my PHP code. I have found some functions while I was trying to debug my code which are showing that my data are coming in normally and cased on reference as expected. Sample provided under:
["last_update"]=>
int(1396917542)
["ds_cnt"]=>
int(3)
["ds_navm"]=>
array(3) {
[0]=>
string(10) "ifInOctets"
[1]=>
string(11) "ifOutOctets"
[2]=>
string(9) "sysUpTime"
}
["data"]=>
array(3) {
[0]=>
string(4) "1405"
[1]=>
string(4) "1219"
[2]=>
string(4) "1893"
}
}
Based on function:
"$debug = rrd_lastupdate (
"".$rrdFile.""
);"
I am having difficulties to understand since the input is coming correctly and there is not print error displayed while compiling my code why I do not get any output?
I have included my working code as example for possible reproduction and better understanding of my error.
<?php
// rrdtool info /var/www/snmp.rrd Debugging command
while (1) {
sleep (1);
$file = "snmp";
$rrdFile = dirname(__FILE__) . "/".$file.".rrd";
$in = "ifInOctets";
$out = "ifOutOctets";
$count = "sysUpTime";
$options = array(
"--start","now -10s", // Now -10 seconds (default)
"--step", "10", // Step size of 300 seconds 5 minutes
"DS:".$in.":COUNTER:20:U:U",
"DS:".$out.":COUNTER:20:U:U",
"DS:".$count.":COUNTER:20:U:U",
/* DS:ds-name:DST:dst arguments
(DST: GAUGE, COUNTER, DERIVE, and ABSOLUTE):heartbeat:min:max
heartbeat: in case that there is not input up to 600 seconds
then the input will characterised as undefined (blank)
Based on Nyquist rate (Fs >= 2 * Fmax) 300 (step) 600 (heartbeat)
32-bit = 2^32-1 = 4294967295 (counter ticks)
64-bit = 2^64-1 = 18446744073709551615 (counter ticks)
64-bit counter (caution!!!) different oid's
*/
"RRA:MIN:0.5:10:300",
"RRA:MIN:0.5:20:600",
"RRA:MAX:0.5:10:300",
"RRA:MAX:0.5:20:600",
"RRA:AVERAGE:0.5:10:300",
"RRA:AVERAGE:0.5:20:600",
/* RRA:AVERAGE | MIN | MAX | LAST:xff:steps:rows
xff range: 0-1 (exclusive) defines the allowed number of unknown
*UNKNOWN* PDPs to the number of PDPs in the interval. Step defines
how many of those data points are used to build consolidated data.
rows defines how many data values are kept in an RRA.
*/
);
//create rrd file
$create = rrd_create(
"".$rrdFile."",
$options
);
if ($create === FALSE) {
echo "Creation error: ".rrd_error()."\n";
}
$ifInOctets = rand(0, 1500); // ifInOctets (OID: 1.3.6.1.2.1.2.2.1.10)
$ifOutOctets = rand(0, 2500); // ifOutOctets (OID: 1.3.6.1.2.1.2.2.1.16)
$sysUpTime = rand(0, 2000); // sysUpTime (OID: 1.3.6.1.2.1.1.3)
$t = time();
//update rrd file
$update = rrd_update(
"".$rrdFile."",
array(
/* Update database with 3 values
based on time now (N:timestamp) */
"".$t.":".$ifInOctets.":".$ifOutOctets.":".$sysUpTime.""
)
);
if ($update === FALSE) {
echo "Update error: ".rrd_error()."\n";
}
$start = "now";
$title = "Hourly Server Data";
$final = array(
"--start","".$start." -10s",
"--step","10",
"--title=".$title."",
"--vertical-label=Bytes/sec",
"--lower-limit=0",
//"--no-gridfit",
"--slope-mode",
//"--imgformat","EPS",
"DEF:".$in."_def=".$file.".rrd:".$in.":AVERAGE",
"DEF:".$out."_def=".$file.".rrd:".$out.":AVERAGE",
"DEF:".$count."_def=".$file.".rrd:".$count.":AVERAGE",
"CDEF:inbits=".$in."_def,8,*",
"CDEF:outbits=".$out."_def,8,*",
"CDEF:counter=".$count."_def,8,*",
/* "VDEF:".$in_min."=inbits,MINIMUM",
"VDEF:".$out_min."=outbits,MINIMUM",
"VDEF:".$in_max."=inbits,MAXIMUM",
"VDEF:".$out_max."=outbits,MAXIMUM",
"VDEF:".$in_av."=inbits,AVERAGE",
"VDEF:".$out_av."=outbits,AVERAGE", */
"COMMENT:\\n",
"LINE:".$in."_def#FF00FF:".$in."",
"GPRINT:inbits:LAST:last \: %6.2lf %SBps",
"COMMENT:\\n",
"LINE:".$out."_def#0000FF:".$out."",
"GPRINT:outbits:LAST:last \: %6.2lf %SBps",
"COMMENT:\\n",
"LINE:".$count."_def#FFFF00:".$count."",
"GPRINT:counter:LAST:last\: %6.2lf %SBps",
"COMMENT:\\n",
);
// graph output
$outputPngFile = rrd_graph(
"".$file.".png",
$final
);
if ($outputPngFile === FALSE) {
echo "<b>Graph error: </b>".rrd_error()."\n";
}
/* Returns the first data sample from the
specified RRA of the RRD file. */
$result = rrd_first (
$rrdFile,
$raaindex = 0
);
if ($result === FALSE) {
echo "<b>Graph result error: </b>".rrd_error()."\n";
}
/* Returns the UNIX timestamp of the most
recent update of the RRD database. */
$last = rrd_last (
$rrdFile
);
if ($last === FALSE) {
echo "<b>Graph result error: </b>".rrd_error()."\n";
}
$info = rrd_info (
"".$rrdFile.""
);
if ($info === FALSE) {
echo "<b>Graph result error: </b>".rrd_error()."\n";
}
/* Gets array of the UNIX timestamp and the values stored
for each date in the most recent update of the RRD database file. */
$debug = rrd_lastupdate (
"".$rrdFile.""
);
if ($debug === FALSE) {
echo "<b>Graph result error: </b>".rrd_error()."\n";
}
var_dump ($debug);
/* $version = rrd_version ( );
echo "This is the version ".$version."\n"; */
} // End of while condition
?>
There are a lot of issues in your code.
Firstly, your RRD file is apparently re-created in every iteration of your While loop! This will overwrite the previous loop's update.
Secondly, although you create the RRD with a step size of 10, you are not doing a sleep(10) at the end of each update loop. You cannot update the RRD more frequently than the Step size.
Thirdly, you have used a COUNTER type for the DS, which assumes an constantly increasing count. However your test data are random numbers and so not increasing. A decrease may be taken as a counter wraparound, so a huge number, which is outside the valid range for your DS - and hence an Unknown is stored.
Fourthly, you need to have two consecutive updates for a Counter to have a valid rate; you are overwriting your RRD file each iteration and so never manage to get this.
Fifthly, your smallest RRA being defined has a COunt of 10; this means you need 10 data points to make a single consolidated pointin the RRA. With a Step of 10sec, this means you need to run for 110sec (11 updates) before you even have a single data point to plot. You should probably try adding a Count 1 RRA.
And finally, your graph is being requested for a 10 second time window. This is less than one RRA sample. Remember, you r Step is 10s and your smallest RRA is count=10, so a consolidated step of 100s.
I suggest you fix the loop so that the create comes outside; make the sleep equivalent to the RRD step; add a Count 1 AVG RRA; and make your graph request for a longer time window.

Why is this PHP foreach loop only executing once?

Trying to debug some issues with WordPress' notorious pseudo-cron. Thought about putting this on WordPressAnswers but I think this is less about WP and more a basic PHP question on a foreach that seems to never get past its first run. (Yes, definitely looked at a bunch of similar questions, but no love there, sorry.)
Here's the code, with 3 checkpoints where I added output for testing. I'm including the entire code of everything in the loop, just to be thorough, though most of it's probably beside the point. Checkpoints 2 and 3 never display for me, and checkpoint 1 only shows up once (i.e. on first iteration of loop). Not shown is where I'm outputting $crons, and count($crons) for good measure, to confirm that, yes, it absolutely has multiple elements.
foreach ( $crons as $timestamp => $cronhooks ) {
echo("<br>loop for timestamp " . $timestamp); // checkpoint 1
if ( $timestamp > $gmt_time ) {
break;
}
foreach ( $cronhooks as $hook => $keys ) {
echo("<br>loop for hook " . $hook); // checkpoint 2
foreach ( $keys as $k => $v ) {
$schedule = $v['schedule'];
if ( $schedule != false ) {
$new_args = array($timestamp, $schedule, $hook, $v['args']);
call_user_func_array('wp_reschedule_event', $new_args);
}
wp_unschedule_event( $timestamp, $hook, $v['args'] );
do_action_ref_array( $hook, $v['args'] );
// If the hook ran too long and another cron process stole the lock, quit.
if ( _get_cron_lock() != $doing_wp_cron ) {
echo("quitting!"); // checkpoint 3
return;
}
}
}
}
All the output I'm getting from this, despite my 5 elements, is:
loop for timestamp 1382968401
I'll be grateful for extra eyes on whatever I'm missing. I'm 95% sure this is WP core code, except my checkpoints, although for various reasons it's not out of the question that another programmer was mucking about in here in an end-run around our version control system. A whole other problem, obviously!
Assuming $gmt_time is current UTC time, then the timestamp 1382968401 will be greater (as of this writing):
$d = new DateTime('#'.'1382968401', new DateTimeZone('UTC'));
echo $d->format('Y-m-d H:i:s'); // output 2013-10-28 13:53:21
Which would explain why your loop breaks on the first iteration.
Well, foreach can't be wrong.
The only way your loop could do only one iteration is that it breaks. And that could only mean that the first $timestamp is greater than $gmt_time.
(There's also the return possibility near the bottom, but you already said you receive one line of text, so it couldn't have gone that way...)
I'm guessing that you want to just skip the iteration if the scheduled task is in the future. To do that, just change break with continue.

determining proper gearman task function to retrieve real-time job status

Very simply, I have a program that needs to perform a large process (anywhere from 5 seconds to several minutes) and I don't want to make my page wait for the process to finish to load.
I understand that I need to run this gearman job as a background process but I'm struggling to identify the proper solution to get real-time status updates as to when the worker actually finishes the process. I've used the following code snippet from the PHP examples:
do {
sleep(3);
$stat = $gmclient->jobStatus($job_handle);
if (!$stat[0]) // the job is known so it is not done
$done = true;
echo "Running: " . ($stat[1] ? "true" : "false") . ", numerator: " . $stat[2] . ", denomintor: " . $stat[3] . "\n";
} while(!$done);
echo "done!\n";
and this works, however it appears that it just returns data to the client when the worker finished telling the job what to do. Instead I want to know when the literal process of the job finished.
My real-life example:
Pull several data feeds from an API (some feeds take longer than others)
Load a couple of the ones that always load fast, place a "Waiting/Loading" animation on the section that was sent off to a worker queue
When the work is done and the results have been completely retrieved, replace the animation with the results
This is a bit late, but I stumbled across this question looking for the same answer. I was able to get a solution together, so maybe it will help someone else.
For starters, refer to the documentation on GearmanClient::jobStatus. This will be called from the client, and the function accepts a single argument: $job_handle. You retrieve this handle when you dispatch the request:
$client = new GearmanClient( );
$client->addServer( '127.0.0.1', 4730 );
$handle = $client->doBackground( 'serviceRequest', $data );
Later on, you can retrieve the status by calling the jobStatus function on the same $client object:
$status = $client->jobStatus( $handle );
This is only meaningful, though, if you actually change the status from within your worker with the sendStatus method:
$worker = new GearmanWorker( );
$worker->addFunction( 'serviceRequest', function( $job ) {
$max = 10;
// Set initial status - numerator / denominator
$job->sendStatus( 0, $max );
for( $i = 1; $i <= $max; $i++ ) {
sleep( 2 ); // Simulate a long running task
$job->sendStatus( $i, $max );
}
return GEARMAN_SUCCESS;
} );
while( $worker->work( ) ) {
$worker->wait( );
}
In versions of Gearman prior to 0.5, you would use the GearmanJob::status method to set the status of a job. Versions 0.6 to current (1.1) use the methods above.
See also this question: Problem With Gearman Job Status

facebook api: count attendees for event (limit problem)

Okay normally I'm all fine about the facebook API but I'm having a problem which just keeps me wondering. (I think it's a bug (Check ticket http://bugs.developers.facebook.net/show_bug.cgi?id=13694) but I wanted to throw it here if somebody has an idea).
I'm usng the facebook PHP library to count all attendees for a specific event
$attending = $facebook->api('/'.$fbparams['eventId'].'/attending');
this works without a problem it correctly returns an array with all attendees...
now heres the problem:
This event has about 18.000 attendees right now.
The api call returns a max number of 992 attendees (and not 18000 as it should).
I tried
$attending = $facebook->api('/'.$fbparams['eventId'].'/attending?limit=20000');
for testing but it doesn't change anything.
So my actual question is:
If I can't get it to work by using the graph api what would be a good alternative? (Parsing the html of the event page maybe?) Right now I'm changing the value by hand every few hours which is tedious and unnecessary.
Actually there are two parameters, limit and offset. I think that you will have to play with both and continue making calls until one returns less than the max. limit.
Something like this, but in a recursive approach (I'm writting pseudo-code):
offset = 0;
maxLimit = 992;
totalAttendees = count(result)
if (totalAttendees >= maxLimit)
{
// do your stuff with each attendee
offset += totalAttendees;
// make a new call with the updated offset
// and check again
}
I've searched a lot and this is how I fixed it:
The requested URL should look something like this.
Here is where you can test it and here is the code I used:
function events_get_facebook_data($event_id) {
if (!$event_id) {
return false;
}
$token = klicango_friends_facebook_token();
if ($token) {
$parameters['access_token'] = $token;
$parameters['fields']= 'attending_count,invited_count';
$graph_url = url('https://graph.facebook.com/v2.2/' . $event_id , array('absolute' => TRUE, 'query' => $parameters));
$graph_result = drupal_http_request($graph_url, array(), 'GET');
if(is_object($graph_result) && !empty($graph_result->data)) {
$data = json_decode($graph_result->data);
$going = $data->attending_count;
$invited = $data->invited_count;
return array('going' => $going, 'invited' => $invited);
}
return false;
}
return false;
}
Try
SELECT eid , attending_count, unsure_count,all_members_count FROM event WHERE eid ="event"

Categories