I have a script in python receiving continuously data from sensors.
I need to publish the last data at request on a webpage using php.
Apache, php and python are all on the same linux install (actually on an raspberry).
I'm not interested in storing previous data and I'm a bit concerned about data corruption on writing on SD. I would prefer to reduce complexity and increase speed refresh (avoid sql).
Could a text file written in ramfs / tmpfs solve the problem? or there is a method to share memory like memcache to share also global variables?
Any practical example or howto will be really well-accepted.
I think you may try System V shared memory.
As example:
In Python side:
python -m pip install sysv_ipc
then somewhere in the python script:
import sysv_ipc
...
KEY=20171220
sysv_memory=sysv_ipc.SharedMemory(KEY, sysv_ipc.IPC_CREAT, 0666, 256)
...
sysv_memory.write('1234'+'\0')
Then in the PHP side:
$SHARED_MEMORY_KEY = 20171220;
...
$shmId = shmop_open($SHARED_MEMORY_KEY, 'a', 0666, 256);
...
$test_string = shmop_read($shmId, 0, 0);
I can get the $test_string as '1234' successfully in PHP.
Here is a solution using memcached that works on Raspbian 10 (buster), PHP 7.3.19, and Python 3.7.3:
1. Install memcached
apt-get install memcached php-memcached
pip install pymemcache
These commands install memchached server and client modules for PHP and Python.
2. PHP code
$m = new Memcached();
// connect
$m->addServer('127.0.0.1', 11211);
// get a value
$value = $m->get('key');
// set a value
$m->set('key', 'value');
// clean up
$m->quit();
3. Python code
from pymemcache.client import base
# connect
m = base.Client(('127.0.0.1', 11211))
# get a value
value = m.get('key')
# set a value
m.set('key', 'value')
# clean up
m.close()
Note:
I used default memcached settings here. If you need to change them edit sudo nano /etc/memcached.conf and restart the daemon: sudo /etc/init.d/memcached restart.
You can use any interoperable format like json or msgpack.
Whenever you generate data in python, add it to a caching layer like memcache/redis using json format ( and preferably a gzip compressed version), then your PHP script can unserialize the json data and display it in the app.
Clearly memcache as a means to share data different application will work.
You will not have any corrupted data for sure as all the memcache operation are atomic. memcache atomic discussion could be useful.
On memcached's FAQ:
Is memcached atomic? Aside from any bugs you may come across, yes all commands are internally atomic. Issuing multiple sets at the same time has no ill effect, aside from the last one in being the one that sticks.
Note: Running memcache service might consume considerable amount of memory.
Hope it helps!
Related
using sys_getloadavg() we can get server load,
using memory_get_usage() we get MEM asigned to THIS_script.php
however:
is possible some similar to this program using PURE code PHP (not shell, not bash):
<?php
function get_ALL_process_PHP_running_just_now(){
...
...
... get memory of ALL process PHP
return array_process_number();
}
then obtain some similar to:
total scripts running: 35
users running process: 6
process with more of 5 minutes: 2
memory GLOBAL asigned to all process PHP: 8GB
etc...
is possible obtain that info with "admin.php" ?
As far as i know there is no build-in function that pieces together that data, however the functions you refer to (sys_getloadavg, memory_get_usage) are just wrappers around the /proc filesystem (on linux anyway, i don't think too many of them have windows counterparts).
The ordinary filesystem functions, which you use to read files, can be used to read the /proc filesystem, which in turn contains all the information you might want.
I'm building a web app that retrieves dynamic generated content through puppeteer. I have set up (apache + php) docker containers, one for the p5js project that generates an svg based on a (large, 2MB) json file, and one container with PHP that retrieves that svg. Dockers runs in an Nginx config (nginx for routing, apache for quicker PHP handling). I'm using the cheapest CENTOS server available on digitalocean. So upgrading would definitley help.
I don't want the javascript in the p5js project to be exposed to the public, so I thought a nodejs solution would be best in this scenario.
The PHP page does a shell_exec("node pup.js"). It basically runs in approx 1-3 seconds which is perfect.
Problem is when I try to test a multi user scenario and open 5 tabs to run this PHP page, the loadtime drops to even 10+ seconds, which is killing for my app.
So the question would be how to set up this architecture (php calling a node command) for a multi user environment.
===
I've tried several frameworks like x-ray, nightmare, jsdom, cheerio, axios, zombie, phantom just trying to replace puppeteer. Some of the frameworks returned nothing, some just didn't work out for me. I think I just need a headless browser solution, to be able to execute the p5js. Eventually puppeteer gets the job done, only not in a multi-user environment (I think due to my current php shell_exec puppeteer architecture).
Maybe my shell_exec workflow was the bottleneck, so I ended up building a simple node example.js which waits 5 seconds before finish (not using puppeteer), and I ran this with several tabs simultaneously, works like a charm. All tabs load in about 5-6 seconds.
I've also tried pm2 to test if my node command was the bottleneck, I did some testing on the commandline, with no major results and I couldn't get PHP to run a pm2 command, so I dropped this test.
I've tried setting PuPHPeteer up, but couldn't get it to run.
At some time I thought it had something to do with multiple puppeteer browsers launched, but I've read that this should be no problem.
The PHP looks like:
<?php
$puppeteer_command = "node /var/www/pup.js >&1";
$result = shell_exec($puppeteer_command);
echo $result;
?>
My puppeteer code:
const puppeteer = require('puppeteer');
let url = "http://the-other-dockercontainer/";
let time = Date.now();
let scrape = async () => {
const browser = await puppeteer.launch({
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto(url);
await page.waitForSelector('svg', { timeout: 5000 });
let svgImage = await page.$('svg');
await svgImage.screenshot({
path: `${time}.png`,
omitBackground: true,
});
await browser.close();
return time;
}
scrape().then((value) => {
console.log(value); // Success!
});
I was thinking about building the entire app in nodejs if that is the best solution, but I've put so many hours in this PHP infrastructure, I'm at the point of really like getting some advice :)
Since I have full control over the target and destination site one brainfart would be to have node to serve a server which accepts a json file and return the svg based on a local p5js site, but don't now (yet) if this would be any different.
UPDATE
So thanks to some comments, I've tried a new approach: not using p5js, but native processing code (java). I've exported the processing code to a linux 64bit application and created this little nodejs example:
var exec = require('child_process').exec;
var cmd = '/var/www/application.linux64/minimal';
exec(cmd, processing);
// Callback for command line process
function processing(error, stdout, stderr) {
// I could do some error checking here
console.log(stdout);
};
When I call this node example.js within a shell_exec in PHP, I get this:
First call takes about 2 seconds. But when I hit a lot of refreshes, time is again building up by a lot of seconds. So, clearly, my understanding of multithreading is not that good, or am I missing something crucial in my testing?
I have a setup similar to you. The biggest problem is your droplet. You should use at minimum the cheapest CPU optimized droplet ($40 per month from memory). The cheapest generic droplets on DO are in a shared environment (noisy neighbours create performance fluctuations). You can easily test this out by making a snapshot of your server and cloning your drive.
Next as someone else suggested is to reduce the cold starts. On my server a cold start adds around 2 extra seconds. I take 10 screenshots before opening a new browser. Anything more than that and you may run into issues with memory.
If you are trying to use puppeteer with php you should use make sure you have composer installed then while inside the project folder open terminal windows and type composer require nesk/puphpeteer
also install nesk/rialto, then require autoload everything should work. One thing when working with puppeteer setting const use $ and skip using await command plus replace "." with ->
<?php
require 'vendor/autoload.php';
use Nesk\Puphpeteer\Puppeteer;
use Nesk\Rialto\Data\JsFunction;
$puppeteer = new Puppeteer;
$browser = $puppeteer->launch(['headless'=>false,'--proxy-server=000.00.0.0:80']);// Type Proxy
$bot = $browser->newPage();
$data = $bot->evaluate(JsFunction::createWithBody('return document.documentElement.innerHTML'));
$urlPath = $bot->url();
$bot->goto('https://google.com');//goes to google.com
$bot->waitForTimeout(3000);// waits
$bot->type('div', '"cellphonemega"');//searches
$bot->keyboard->press('Enter');//presses enter
$bot->waitForTimeout(8000);//waits
$bot->click('h3', 'https:');//clicks webpage
$bot->waitForTimeout(8000);//waits while site loads
$bot->screenshot(['path' => 'screenshot.png']);// TAKES SCREENSHOT!
$browser->close();//Shuts Down
I've been all over the internet looking for an answer to my problem. Here is the setup, I am running embedded Linux (created with Yocto) which is running the Lighttpd web server with PHP5. In my C++ code I have the following:
shared = shm_open(SHARED_FILE_NAME, O_RDWR | O_CREAT | O_TRUNC, 0666);
ftruncate(shared, FILE_SIZE);
map = mmap(...);
// shm_unlink() isn't called until my C++ thread ends.
Everything works well and I do not get any errors and other C++ processes and threads are also able to access the shared memory and map without any problems (I have one writer thread and all other threads and processes do a read only on the memory). The memory is used as a ring buffer where the writing thread is updating data very quickly. The problems start to occur when trying to access that same memory in PHP. In PHP I do (need read only):
<?php
$shm_key = ftok("/dev/shm/shared_file.shm", 'c');
$shm_id = shm_open($shm_key, "a", 0, 0);
...
?>
When looking at the value from ftok() it returns a non -1 number which means it did not fail. I do get a fail on the PHP's shm_open() call which reads:
Warning: shmop_open(): unable to attach or create shared memory segment in /www/pages/shared.php on line 9
I've changed the permission of the file with chmod 777 /dev/shm/shared.shm just to rule out any file permission issues. Also when I run ipcs -m I do not get any listings for shared memory segments, yet my C++ code is running just fine. I've also looked for SELinux and tried entering setenforce 0 but I get a response of -sh: setenforce: command not found so I figure this isn't an issue. I've also tried running wget <local ip address>/shared.php to see if running locally would return the correct data but when looking at the file which was returned it had the same error messages.
I am looking to be able to have a web page on my embedded system read this shared memory and stream back chunks of binary to feed a graph when a request comes in (not interested in web sockets at the time). I am able to get named pipes to work across PHP and C++ just fine but I need shared memory for this application and the shared memory access seems to be troublesome. Any help is appreciated.
I'm developing PHP functions that need to use C Shared Memory. As your code, my C functions use shm_open, mmap, etc.. and I guess to use PHP ftok(), shmop_open() to access the C's shared memory but this PHP functions don't work.
The two area are not compatible. I found different properties of the two areas in this documents http://menehune.opt.wfu.edu/Kokua/More_SGI/007-2478-008/sgi_html/ch03.html:
C (with shm_open, mmap, like the Straton source code) use “POSIX Shared Memory”
PHP (with shmop_* functions) use “System V Shared Memory”
I suggest you to try with Sync http://php.net/manual/en/book.sync.php: you need the PECL sync extension.
I've recently started using APC cache on our servers. One of the most important parts of our product is a CLI (Cron/scheduled) process, whose performance is critical. Typically the batchjob consists of running some 16-32 processes in parallel for about an hour (they "restart" every few minutes).
By default, using APC cache in CLI is a waste of time due to the opcode cache not being retained between individual calls. But APC also contains apc_bin_dumpfile() and apc_load_dumpfile() functions.
I was thinking these two function might be used to make APC efficient in CLI mode by having it all compiled sometime outside the batchjob, stored in a single dumpfile and having the individual processes load the dumpfile.
Does anybody have any experience with such a scenario or can you give good reasons why it will or will not work? If any significant gains could reasonably be had, either in memory use or performance? What pitfalls are lurking in the shadows?
Disclaimer: As awesome as APC is when it works in CLI, and it is awesome, it can equally be as frustrating. Use with a healthy load of patience, be thorough, step away from the problem if you're spinning, keep in mind you are working with cache that is why it seems like its doing nothing, it is actually doing nothing. Delete dump file, start with just the basics, if that doesn't work forget it try a new machine, new OS, if it is working make a copy, piece by piece expand functionality - there are loads of things that won't work, if it is working commit or make a copy, add another piece and test again, for sanity-check recheck the copies that were working before, cliches or not; if at first you don't succeed try try again, you can't keep doing the same thing expecting new results.
Ready? This is what you've been waiting for:
Enable apc for cli
apc.enable-cli=1
it is not ideal to create, populate and destroy the APC cache on every CLI request
- previous answer by unknown poster since removed.
You're absolutely right that sucks, lets fix it shall we?
If you try and use APC under CLI and it is not enabled you will get warnings.
something like:
PHP Warning: apc_bin_loadfile(): APC is not enabled,
apc_bin_loadfile not available.
PHP Warning: apc_bin_dumpfile(): APC is not enabled,
apc_bin_dumpfile not available.
Warning: I suggest you don't enable cli in php.ini, it is not worth the frustration, you are going to forget you did it and have numerous other headaches with other scripts, trust me its not worth it, use a launcher script instead. (see below)
apc_loadfile and apc_dumpfile in cli
As per the comment by mightye php we need to disable apc.stat or you will get a warnings
something like:
PHP Warning: apc_bin_dumpfile(): Excluding some files from apc_bin_dump[file].
Cached files must be included using full path with apc.stat=0.
launcher script - php-apc.sh
We will use this script to launch our apc enabled scripts (ex. ./php-apc.sh apc-cli.php) instead of changing the properties in php.ini directly.
#/bin/sh
php -d apc.enable_cli=1 -d apc.stat=0 $1
Ready for the basic functionality? Sure you are =)
basic APC persisted - apc-cli.php
<?php
/** check if dump file exists, you don't want to use file_exists */
if (false !== $dump_file = stream_resolve_include_path('apc.dump'))
/** so where were we lets have a look see shall we */
if (false !== apc_bin_loadfile($dump_file))
/** fetch what was stored last run just for fun */
if (false !== $value = apc_fetch('my.awesome.apc.store'))
echo "$value from apc\n";
/** store what gets fetched the next run just for fun */
apc_store('my.awesome.apc.store', 'awesome in cli');
/** what a shlep lets not do that all over again shall we */
apc_bin_dumpfile(array(),null,'apc.dump');
Notice: Why not use file_exists? Because file_exists == stat you see and we want to reap the reward that is apc.stat=0 so; work within the include path; use absolute and not relative paths - as returned by stream_resolve_include_path(); avoid include_once, require_once use the non *_once counterparts; check your stat usage, when not using APC(Muchos important senor), with the help of a StreamWrapper echo for calls to method url_stat; Oops: Fatal scope over-run error! aborting notice thread. see url_stat
message: Error caused by StreamWrapper outside the scope of this discussion.
The smoke test
Using the launcher execute the basic script
./php-apc.sh apc-cli.php
A whole bunch of nothing happened that's what we want right, why else do you want to use cache? If it did output anything then it didn't work, sorry.
There should be a dump file called apc.dump see if you can find it? If you can't find it then it didn't work, sorry.
Good we have the dump file there were no errors lets run it again.
./php-apc.sh apc-cli.php
What you want to see:
awesome in cli from apc
Success! =)
There are few in PHP as satisfying as a working APC implementation.
nJoy!
I would definitely not use it in the CLI as when you restart it, it's almost as if it was never running in the first place!
The better way of using APC is to have it running on the webserver itself all the time, this way with it being active it will actually do what it's supposed to do!
I tryed with curl and APC.it works
use these commands in CLI
curl --data "param1=value2" http://testsite.com/test.php
so it will post data to test.php and you writes the code in it.
I need to clear all APC cache entries when I deploy a new version of the site.
APC.php has a button for clearing all opcode caches, but I don't see buttons for clearing all User Entries, or all System Entries, or all Per-Directory Entries.
Is it possible to clear all cache entries via the command-line, or some other way?
You can use the PHP function apc_clear_cache.
Calling apc_clear_cache() will clear the system cache and calling apc_clear_cache('user') will clear the user cache.
I don't believe any of these answers actually work for clearing the APC cache from the command line. As Frank Farmer commented above, the CLI runs in a process separate from Apache.
My solution for clearing from the command line was to write a script that copies an APC clearing script to the web directory and accesses it and then deletes it. The script is restricted to being accessed from the localhost.
apc_clear.php
This is the file that the script copies to the web directory, accesses, and deletes.
<?php
if (in_array(#$_SERVER['REMOTE_ADDR'], array('127.0.0.1', '::1')))
{
apc_clear_cache();
apc_clear_cache('user');
apc_clear_cache('opcode');
echo json_encode(array('success' => true));
}
else
{
die('SUPER TOP SECRET');
}
Cache clearing script
This script copies apc_clear.php to the web directory, accesses it, then deletes it. This is based off of a Symfony task. In the Symfony version, calls are made to the Symfony form of copy and unlink, which handle errors. You may want to add checks that they succeed.
copy($apcPaths['data'], $apcPaths['web']); //'data' is a non web accessable directory
$url = 'http://localhost/apc_clear.php'; //use domain name as necessary
$result = json_decode(file_get_contents($url));
if (isset($result['success']) && $result['success'])
{
//handle success
}
else
{
//handle failure
}
unlink($apcPaths['web']);
I know it's not for everyone but: why not to do a graceful Apache restart?
For e.g. in case of Centos/RedHat Linux:
sudo service httpd graceful
Ubuntu:
sudo service apache2 graceful
This is not stated in the documentation, but to clear the opcode cache you must do:
apc_clear_cache('opcode');
EDIT: This seems to only apply to some older versions of APC..
No matter what version you are using you can't clear mod_php or fastcgi APC cache from a php cli script since the cli script will run from a different process as mod_php or fastcgi. You must call apc_clear_cache() from within the process (or child process) which you want to clear the cache for. Using curl to run a simple php script is one such approach.
If you are running on an NGINX / PHP-FPM stack, your best bet is to probably just reload php-fpm
service php-fpm reload (or whatever your reload command may be on your system)
If you want to clear apc cache in command : (use sudo if you need it)
APCu
php -r "apcu_clear_cache();"
APC
php -r "apc_clear_cache(); apc_clear_cache('user'); apc_clear_cache('opcode');"
Another possibility for command-line usage, not yet mentioned, is to use curl.
This doesn't solve your problem for all cache entries if you're using the stock apc.php script, but it could call an adapted script or another one you've put in place.
This clears the opcode cache:
curl --user apc:$PASSWORD "http://www.example.com/apc.php?CC=1&OB=1&`date +%s`"
Change the OB parameter to 3 to clear the user cache:
curl --user apc:$PASSWORD "http://www.example.com/apc.php?CC=1&OB=3&`date +%s`"
Put both lines in a script and call it with $PASSWORD in your env.
apc_clear_cache() only works on the same php SAPI that you want you cache cleared. If you have PHP-FPM and want to clear apc cache, you have do do it through one of php scripts, NOT the command line, because the two caches are separated.
I have written CacheTool, a command line tool that solves exactly this problem and with one command you can clear your PHP-FPM APC cache from the commandline (it connects to php-fpm for you, and executes apc functions)
It also works for opcache.
See how it works here: http://gordalina.github.io/cachetool/
As defined in APC Document:
To clear the cache run:
php -r 'function_exists("apc_clear_cache") ? apc_clear_cache() : null;'
If you want to monitor the results via json, you can use this kind of script:
<?php
$result1 = apc_clear_cache();
$result2 = apc_clear_cache('user');
$result3 = apc_clear_cache('opcode');
$infos = apc_cache_info();
$infos['apc_clear_cache'] = $result1;
$infos["apc_clear_cache('user')"] = $result2;
$infos["apc_clear_cache('opcode')"] = $result3;
$infos["success"] = $result1 && $result2 && $result3;
header('Content-type: application/json');
echo json_encode($infos);
As mentioned in other answers, this script will have to be called via http or curl and you will have to be secured if it is exposed in the web root of your application. (by ip, token...)
if you run fpm under ubuntu, need to run the code below (checked on 12 and 14)
service php5-fpm reload
The stable of APC is having option to clear a cache in its interface itself. To clear those entries you must login to apc interface.
APC is having option to set username and password in apc.php file.
apc.ini
apc.stat = "1" will force APC to stat (check) the script on each request to determine if it has been modified. If it has been modified it will recompile and cache the new version.
If this setting is off, APC will not check, which usually means that to force APC to recheck files, the web server will have to be restarted or the cache will have to be manually cleared. Note that FastCGI web server configurations may not clear the cache on restart. On a production server where the script files rarely change, a significant performance boost can be achieved by disabled stats.
New APC Admin interface have options to add/clear user cache and opcode cache, One interesting functionality is to add/refresh/delete directory's from opCode Cache
APC Admin Documentation
A good solution for me was to simply not using the outdated user cache any more after deploy.
If you add prefix to each of you keys you can change the prefix on changing the data structure of cache entries. This will help you to get the following behavior on deploy:
Don't use outdated cache entries after deploy of only updated structures
Don't clean the whole cache on deploy to not slow down your page
Some old cached entries can be reused after reverting your deploy (If the entries wasn't automatically removed already)
APC will remove old cache entries after expire OR on missing cache space
This is possible for user cache only.
Create APC.php file
foreach(array('user','opcode','') as $v ){
apc_clear_cache($v);
}
Run it from your browser.
My work-around for Symfony build having loot of instances at the same server:
Step 1.
Create trigger or something to set a file flag (eg. Symfony command) then create marker file ..
file_put_contents('clearAPCU','yes sir i can buggy')
Step 2.
On index file at start add clearing code and remove marker file.
if(file_exists('clearAPCU')){
apcu_clear_cache();
unlink('clearAPCU');
}
Step 2.
Run app.
TL;DR: delete cache files at /storage/framework/cache/data/
I enabled APC but it wasn't installed (and also couldn't be installed), so it threw Call to undefined function Illuminate\Cache\apc_store().
"Ok, I'd just disable it and it should work".
Well, nope. Then I got stuck with Call to undefined function Illuminate\Cache\apc_delete()
We had a problem with APC and symlinks to symlinks to files -- it seems to ignore changes in files itself. Somehow performing touch on the file itself helped. I can not tell what's the difference between modifing a file and touching it, but somehow it was necessary...