I am in the process of coding a cloud monitoring application and coudnt find useful logic of getting performance counters from AZURE php SDK documentation(such as CPU utilization, disk utilization, ram usage).
can anybody help ??
define('PRODUCTION_SITE', false); // Controls connections to cloud or local storage
define('AZURE_STORAGE_KEY', '<your_storage_key>');
define('AZURE_SERVICE', '<your_domain_extension>');
define('ROLE_ID', $_SERVER['RoleDeploymentID'] . '/' . $_SERVER['RoleName'] . '/' . $_SERVER['RoleInstanceID']);
define('PERF_IN_SEC', 30); // How many seconds between times dumping performance metrics to table storage
/** Microsoft_WindowsAzure_Storage_Blob */
require_once 'Microsoft/WindowsAzure/Storage/Blob.php';
/** Microsoft_WindowsAzure_Diagnostics_Manager **/
require_once 'Microsoft/WindowsAzure/Diagnostics/Manager.php';
/** Microsoft_WindowsAzure_Storage_Table */
require_once 'Microsoft/WindowsAzure/Storage/Table.php';
if(PRODUCTION_SITE) {
$blob = new Microsoft_WindowsAzure_Storage_Blob(
'blob.core.windows.net',
AZURE_SERVICE,
AZURE_STORAGE_KEY
);
$table = new Microsoft_WindowsAzure_Storage_Table(
'table.core.windows.net',
AZURE_SERVICE,
AZURE_STORAGE_KEY
);
} else {
// Connect to local Storage Emulator
$blob = new Microsoft_WindowsAzure_Storage_Blob();
$table = new Microsoft_WindowsAzure_Storage_Table();
}
$manager = new Microsoft_WindowsAzure_Diagnostics_Manager($blob);
//////////////////////////////
// Bring in global include file
require_once('setup.php');
// Performance counters to subscribe to
$counters = array(
'\Processor(_Total)\% Processor Time',
'\TCPv4\Connections Established',
);
// Retrieve the current configuration information for the running role
$configuration = $manager->getConfigurationForRoleInstance(ROLE_ID);
// Add each subscription counter to the configuration
foreach($counters as $c) {
$configuration->DataSources->PerformanceCounters->addSubscription($c, PERF_IN_SEC);
}
// These settings are required by the diagnostics manager to know when to transfer the metrics to the storage table
$configuration->DataSources->OverallQuotaInMB = 10;
$configuration->DataSources->PerformanceCounters->BufferQuotaInMB = 10;
$configuration->DataSources->PerformanceCounters->ScheduledTransferPeriodInMinutes = 1;
// Update the configuration for the current running role
$manager->setConfigurationForRoleInstance(ROLE_ID,$configuration);
///////////////////////////////////////
// Bring in global include file
//require_once('setup.php');
// Grab all entities from the metrics table
$metrics = $table->retrieveEntities('WADPerformanceCountersTable');
// Loop through metric entities and display results
foreach($metrics AS $m) {
echo $m->RoleInstance . " - " . $m->CounterName . ": " . $m->CounterValue . "<br/>";
}
this is the code I crafted to extract processor info ...
UPDATE
Do take a look at the following blog post: http://blog.maartenballiauw.be/post/2010/09/23/Windows-Azure-Diagnostics-in-PHP.aspx. I realize that it's an old post but I think this should give you some idea about implementing diagnostics in your role running PHP. The blog post makes use of PHP SDK for Windows Azure on CodePlex which I think is quite old and has been retired in favor of the new SDK on Github but I think the SDK code on Github doesn't have diagnostics implemented (and that's a shame).
ORIGINAL RESPONSE
Since performance counters data is stored in Windows Azure Table Storage, you could simply use Windows Azure SDK for PHP to query WADPerformanceCountersTable in your storage account to fetch this data.
I have written a blog post about efficiently fetching diagnostics data sometime back which you can read here: http://gauravmantri.com/2012/02/17/effective-way-of-fetching-diagnostics-data-from-windows-azure-diagnostics-table-hint-use-partitionkey/.
Update
Looking at your code above and source code for TableRestProxy.php, you could include a query as the 2nd parameter to your retrieveEntities call. You could something like:
$query = "(CounterName eq '\Processor(_Total)\% Processor Time` or CounterName eq '\TCPv4\Connections Established')
$metrics = $table->retrieveEntities('WADPerformanceCountersTable', $query);
Please note that my knowledge about PHP is limited to none so the code above may not work. Also, please ensure to include PartitionKey in your query to avoid full table scan.
Storage Analytics Metrics aggregates transaction data and capacity data for a storage account. Transactions metrics are recorded for the Blob, Table, and Queue services. Currently, capacity metrics are only recorded for the Blob service. Transaction data and capacity data is stored in the following tables:
$MetricsCapacityBlob
$MetricsTransactionsBlob
$MetricsTransactionsTable
$MetricsTransactionsQueue
The above tables are not displayed when a listing operation is performed, such as the ListTables method. Each table must be accessed directly.
When you retrieve metrics,use these tables.
Ex:
$metrics = $table->retrieveEntities('$MetricsCapacityBlob');
URL:
http://msdn.microsoft.com/en-us/library/windowsazure/hh343264.aspx
Related
As i checked on https://www.php.net/manual/en/ref.com.php#19688 page, i can see some examples of how to use COM object function but I don't know where to see the full documentation of these COM function? For example:
<?php
/***
* Grouping Rows optically in Excel Using a COM Object
*
* That was not easy, I have spent several hours of trial and error to get
* this thing to work!!!
*
* #author Kulikov Alexey <a.kulikov#gmail.com>
* #since 13.03.2006
*
* #see Open Excel, Hit Alt+F11, thne Hit F2 -- this is your COM bible
***/
//starting excel
$excel = new COM("excel.application") or die("Unable to instanciate excel");
print "Loaded excel, version {$excel->Version}\n";
//bring it to front
#$excel->Visible = 1;//NOT
//dont want alerts ... run silent
$excel->DisplayAlerts = 0;
//create a new workbook
$wkb = $excel->Workbooks->Add();
//select the default sheet
$sheet=$wkb->Worksheets(1);
//make it the active sheet
$sheet->activate;
//fill it with some bogus data
for($row=1;$row<=7;$row++){
for ($col=1;$col<=5;$col++){
$sheet->activate;
$cell=$sheet->Cells($row,$col);
$cell->Activate;
$cell->value = 'pool4tool 4eva ' . $row . ' ' . $col . ' ak';
}//end of colcount for loop
}
///////////
// Select Rows 2 to 5
$r = $sheet->Range("2:5")->Rows;
// group them baby, yeah
$r->Cells->Group;
// save the new file
$strPath = 'tfile.xls';
if (file_exists($strPath)) {unlink($strPath);}
$wkb->SaveAs($strPath);
//close the book
$wkb->Close(false);
$excel->Workbooks->Close();
//free up the RAM
unset($sheet);
//closing excel
$excel->Quit();
//free the object
$excel = null;
?>
In the script above, I can not find documentation for Visible, DisplayAlerts, Worksheets, activate, SaveAs, Workbooks->Add().... property, function in php website
Thank in advance
The Excel object model is described in MSDN.
Be aware that Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.
If you are building a solution that runs in a server-side context, you should try to use components that have been made safe for unattended execution. Or, you should try to find alternatives that allow at least part of the code to run client-side. If you use an Office application from a server-side solution, the application will lack many of the necessary capabilities to run successfully. Additionally, you will be taking risks with the stability of your overall solution. Read more about that in the Considerations for server-side Automation of Office article.
If you deal with open XML documents only you may consider using the Open XML SDK, see Welcome to the Open XML SDK 2.5 for Office for more information.
I am integrating a bigQuery in my Google Cloud project. I have settle up all the requirements which required to integrate the big query. Now I want to perform the update operation through my PHP file. I have created a dataset and table in bigQuery.
Dataset Name - count
Table name - companies
I want to update in this table through my PHP file. Before this, here is my code for updating the values in cloud datastore:
$dataset = $bigQuery->dataset('count');
$table = $dataset->table('companies');
if ($check) {
$updateResponse = $table->update(['name' => 'A friendly name.']);
if ($updateResponse->isSuccessful()) {
print('Data updated into BigQuery successfully' . PHP_EOL);
}
}
But this code is not working ?
The code seems to be ok, but name is not a valid metadata. friendlyName is a valid one, check other table metadata resources. I'm not quite sure if isSuccessful exists, so, to verify whether the operation was successful or not, I would apply:
$info = $table->info();
echo $info['friendlyName'];
If you want to update data in the BigQuery table, you can execute DML statements just as you would a SELECT statement:
$bigQuery = new BigQueryClient(['projectId' => $projectId,]);
$query = 'UPDATE statement';
$jobConfig = $bigQuery->query($query);
$job = $bigQuery->startQuery($jobConfig);
In this case you can ask the progress of the DML operation with $job->isComplete(), see the complete example here.
You mentioned that "... updating the values in cloud datastore"; however, your code is for BigQuery, if there is something related to datastore, please clarify.
I have a very large table in cassandra (~500mil) and I want to export all rows for some columns to a file. I tried this using the COPY command with:
COPY keyspace.table (id, value) TO 'filepath' WITH DELIMITER=',';
but it took ~12 hours to complete the export. Is there any option this could be done faster?
If it is a problem to just export some columns it wouldn't be a problem to export all data. The important thing is that I need a way to get all entries which I can proceed afterwards.
The other question is, is it possible to process this export in PHP just with the DataStax PHP driver?
COPY ... TO ... not a good idea to use on a big amount of data.
is it possible to process this export in PHP just with the DataStax PHP driver
I did CSV export from the Cassandra with the help of Datastax Java driver, but PHP must have the same algorithm. According to documentation you can easily do a request and print output. Take in to attention pagination as well.
You can convert array to CSV with the help of fputcsv funciton
So, the simplest example would be:
<?php
$cluster = Cassandra::cluster() // connects to localhost by default
->build();
$keyspace = 'system';
$session = $cluster->connect($keyspace); // create session, optionally scoped to a keyspace
$statement = new Cassandra\SimpleStatement( // also supports prepared and batch statements
'SELECT keyspace_name, columnfamily_name FROM schema_columnfamilies'
);
$future = $session->executeAsync($statement); // fully asynchronous and easy parallel execution
$result = $future->get(); // wait for the result, with an optional timeout
// Here you can print CSV headers.
foreach ($result as $row) { // results and rows implement Iterator, Countable and ArrayAccess
// Here you can print CSV values
// printf("The keyspace %s has a table called %s\n", $row['keyspace_name'], $row['columnfamily_name']);
}
The short answer is yes, there are faster ways to do this.
The how is a longer answer, if you are going to be saving these rows to file on a regular basis - you might want to use Apache Spark. Depending how much memory is on your Cassandra nodes, you can bring a simple 500 million row table scan => write to file down to < 1 hour.
There are some of options which can give you fast & reliable turn-around:
Hive [ My Preferred One, run SQL like Query ]
Shark/Beeline [ run SQL like Query ]
Spark [ Fast for data related computation but not the best option for your use-case]
For PHP[Hive PHP Client]:
<?php
// set THRIFT_ROOT to php directory of the hive distribution
$GLOBALS['THRIFT_ROOT'] = '/lib/php/';
// load the required files for connecting to Hive
require_once $GLOBALS['THRIFT_ROOT'] . 'packages/hive_service/ThriftHive.php';
require_once $GLOBALS['THRIFT_ROOT'] . 'transport/TSocket.php';
require_once $GLOBALS['THRIFT_ROOT'] . 'protocol/TBinaryProtocol.php';
// Set up the transport/protocol/client
$transport = new TSocket('localhost', 10000);
$protocol = new TBinaryProtocol($transport);
$client = new ThriftHiveClient($protocol);
$transport->open();
// run queries, metadata calls etc
$client->execute('SELECT * from src');
var_dump($client->fetchAll());
$transport->close();
Ref: https://cwiki.apache.org/confluence/display/Hive/HiveClient#HiveClient-PHP
I am connecting to an API, and getting a report in a TSV format. I am needing to upload the report to Google BigQuery, but all the documentation I have found so far loads data from Google Cloud Storage. Is there a way to load data from a seperate URL?
Here is the code I have thus far:
$service = new Google_BigqueryService($client);
// Your project number, from the developers.google.com/console project you created
// when signing up for BigQuery
$project_number = '*******';
// Information about the destination table
$destination_table = new Google_TableReference();
$destination_table->setProjectId($project_number);
$destination_table->setDatasetId('php_test');
$destination_table->setTableId('my_new_table');
// Information about the schema for your new table
$schema_fields = array();
$schema_fields[0] = new Google_TableFieldSchema();
$schema_fields[0]->setName('Date');
$schema_fields[0]->setType('string');
$schema_fields[1] = new Google_TableFieldSchema();
$schema_fields[1]->setName('PartnerId');
$schema_fields[1]->setType('string');
....
$destination_table_schema = new Google_TableSchema();
$destination_table_schema->setFields($schema_fields);
// Set the load configuration, including source file(s) and schema
$load_configuration = new Google_JobConfigurationLoad();
$load_configuration->setSourceUris(array('EXTERNAL URL WITH TSV'));
$load_configuration->setDestinationTable($destination_table);
$load_configuration->setSchema($destination_table_schema);
$job_configuration = new Google_JobConfiguration();
$job_configuration->setLoad($load_configuration);
$load_job = new Google_Job();
$load_job->setKind('load');
$load_job->setConfiguration($job_configuration);
$jobs = $service->jobs;
$response = $jobs->insert($project_number, $load_job);
I realize that this is meant for Google Cloud Storage, but I do not want to use it, if I am just going to pass data through it and delete it within the hour.
Is there PHP code that I can use that will allow me to use external URLs and load data from them?
As Pentiuum10 mentioned above, BigQuery doesn't support reading from non-Google Cloud Storage URLs. The logistics involved would be tricky ... we'd need credentials to access the data, which we don't really want to have to be responsible for. If this is a popular request, we might end up supporting external paths that are either unrestricted or support oauth2. That said, we haven't had a lot of users asking for this so far.
Feel free to file a feature request via the public issue tracker here: https://code.google.com/p/google-bigquery/issues/.
Awhile ago I came across a script that basically fetched a list of countries/states from a web resource if it wasn't located in a database, and this script would then populate the database with those contents and from then on, rely on them from then on.
Since I'm working on a localization class of my own, I'll be using the same locale data Zend is using, in the form of around ~60 or so xml files which contain localised data such as countries, languages for locales.
I figure since the framework I'm working on will rely on these files from now on ( where it isn't now ), and none of the servers now have this data, should I:
Setup my web application to download these files from a central server where all the content is stored in a .tar.gz, unpack them, store them on the server and then rely on them
Create a separate script to do this, and not actually do this within the application.
Pseudo code:
if ( !data ) {
resource = getFile( 'http://central-server.com/tar.gz' );
if ( resource ) {
resource = unpack( directory, resource )
return true
}
throw Exception('could not download files.')
}
I would go for the first option iff the data needs to be contantly updated, otherwise I would choose your second option.
Here is a method I developed some years ago, that was part of a GeoIP class:
function Update()
{
$result = false;
$databases = glob(HIVE_DIR . 'application/repository/GeoIP/GeoIP_*.dat');
foreach ($databases as $key => $value)
{
$databases[$key] = basename($value);
}
$databases[] = 'GeoIP.dat.gz';
$date = date('ym');
if ((!in_array('GeoIP_' . $date . '.dat', $databases)) && (date('j') >= 2))
{
if ($this->Hive->Filesystem->Write(HIVE_DIR . 'application/repository/GeoIP/GeoIP.dat.gz', file_get_contents('http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz'), false) === true)
{
$handler = gzopen(HIVE_DIR . 'application/repository/GeoIP/GeoIP.dat.gz', 'rb');
$result = $this->Hive->Filesystem->Write(HIVE_DIR . 'application/repository/GeoIP/GeoIP_' . $date . '.dat', gzread($handler, 2 * 1024 * 1024), false);
gzclose($handler);
foreach ($databases as $database)
{
$this->Hive->Filesystem->Delete(HIVE_DIR . 'application/repository/GeoIP/' . $database);
}
}
}
return $result;
}
Basically the Update() was executed every single time, it would then check if the day of the month equal or higher than 2 (MaxMind releases GeoIP databases on the first day of the month) and if a a database for that month didn't existed already. Only if both these conditions where true the method would download, unpack, rename the database and remove all the old databases from previous months.
In your case, since you're dealing with locales, doing a periodical check similar to this once in a while might not be a bad idea, since countries change stuff (names, currencies, calling codes, etc...) a lot.
if this is a library, i would probably have this be part of the setup steps. an error can be printed if data isn't there.
Have an install script do the downloading, or throw an error if its not available. Downloading as requested from the server could lead to timeouts and would likely turn away users. fsockopen is the easiest way to do this and deal with sockets by hand if you don't have CURL setup and can't fopen/fread remote files.