How can I get the CPU % consumed by a Linux process with PHP or by bash? I tried to find any utility for this but couldn't. All I found was always getting the same result for same reason.
I found this amazing answer here: https://unix.stackexchange.com/questions/554/how-to-monitor-cpu-memory-usage-of-a-single-process
To use that information on a script you can do this:
calcPercCpu.sh
#!/bin/bash
nPid=$1;
nTimes=10; # customize it
delay=0.1; # customize it
strCalc=`top -d $delay -b -n $nTimes -p $nPid \
|grep $nPid \
|sed -r -e "s;\s\s*; ;g" -e "s;^ *;;" \
|cut -d' ' -f9 \
|tr '\n' '+' \
|sed -r -e "s;(.*)[+]$;\1;" -e "s/.*/scale=2;(&)\/$nTimes/"`;
nPercCpu=`echo "$strCalc" |bc -l`
echo $nPercCpu
use like: calcPercCpu.sh 1234 where 1234 is the pid
For the specified $nPid, it will measure the average of 10 snapshots of the cpu usage in a whole of 1 second (delay of 0.1s each * nTimes=10); that provides a good and fast accurate result of what is happening in the very moment.
Tweak the variables to your needs.
Related
I have a simple script used to recreate and format partitions in a disk that runs fine from the command line, but fails complaining about device being busy at the first command that changes the partition table when called from php. I've done a lot of reading in stackoverflow and other sources and most of the times in similar situations the problem is a relative path and/or permissions that don't let the script run but in my case the script runs but isn't able to do the partitioning/formating.
Things I've checked/tried:
Paths: I use absolute paths/chdir and the script executes.
Permissions: all set to rwx to be on the safe side while testing.
Sudoers file: granted all permissions for testing.
Relevant (imho) php.ini directives: (also for testing purposes)
disable_functions =
safe_mode =
fuser command reports nothing.
I've used parted/fdisk/sfdisk and a mix of them just in case, same results.
The script runs as expected with sudo from the command line.
The php code is this:
$path="/usr/share/test";
$logFile=$path . "/partition.log";
chdir(path);
if($_GET['run']) {
$command="sudo ./partition.sh 2>&1 | tee $logFile";
shell_exec($command);
}
<a href=?run=true>Run script</a>
And the partitioning script goes like this:
umount -v -f -l /dev/sda1
umount -v -f -l /dev/sda2
umount -v -f -l /dev/sda3
sfdisk --delete /dev/sda
sleep 5
/sbin/partprobe
/sbin/parted -a optimal -s /dev/sda mkpart primary ext4 32.3kB 409600MB
/sbin/parted -a optimal -s /dev/sda mkpart primary ext4 409600MB 614400MB
/sbin/parted -a optimal -s /dev/sda mkpart primary ext4 614400MB 1024GB
sleep 5
/sbin/partprobe
umount -v -f -l /dev/sda1
umount -v -f -l /dev/sda2
umount -v -f -l /dev/sda3
echo "***formating partitions"
/sbin/mkfs.ext4 -F /dev/sda1
/sbin/mkfs.ext4 -F /dev/sda2
/sbin/mkfs.ext4 -F /dev/sda3
echo "***mounting"
mount /dev/sda1 /media/vms
mount /dev/sda2 /media/apps
mount /dev/sda3 /media/novos
I'm unmounting my partitions twice because I discovered the hard way parted mount them automatically and occasionally they've been reported mounted by umount (ref: https://unix.stackexchange.com/questions/432850/is-parted-auto-mounting-new-partitions)
Sample output: (please omit the warning messages, optimization comes later!)
+ umount -v -l /dev/sda1
umount: /media/vms (/dev/sda1) unmounted
+ umount -v -l /dev/sda2
umount: /media/apps (/dev/sda2) unmounted
+ umount -v -l /dev/sda3
umount: /media/novos (/dev/sda3) unmounted
+ sfdisk --delete /dev/sda
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
Syncing disks.
+ sleep 5
+ /sbin/partprobe
Error: Partition(s) 1, 2, 3 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
+ /sbin/parted -a optimal -s /dev/sda mkpart primary ext4 32.3kB 409600MB
Warning: The resulting partition is not properly aligned for best performance.
Error: Partition(s) 2, 3 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
+ /sbin/parted -a optimal -s /dev/sda mkpart primary ext4 409600MB 614400MB
Warning: The resulting partition is not properly aligned for best performance.
Error: Partition(s) 3 on /dev/sda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
+ /sbin/parted -a optimal -s /dev/sda mkpart primary ext4 614400MB 1024GB
Warning: The resulting partition is not properly aligned for best performance.
+ sleep 5
+ /sbin/partprobe
+ umount -v -l /dev/sda1
umount: /dev/sda1: not mounted
+ umount -v -l /dev/sda2
umount: /dev/sda2: not mounted
+ umount -v -l /dev/sda3
umount: /dev/sda3: not mounted
+ echo '***formating partitions'
***formating partitions
+ /sbin/mkfs.ext4 -F /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
/dev/sda1 is apparently in use by the system; will not make a filesystem here!
+ /sbin/mkfs.ext4 -F /dev/sda2
mke2fs 1.43.4 (31-Jan-2017)
/dev/sda2 is apparently in use by the system; will not make a filesystem here!
+ /sbin/mkfs.ext4 -F /dev/sda3
mke2fs 1.43.4 (31-Jan-2017)
/dev/sda3 is apparently in use by the system; will not make a filesystem here!
+ echo '***mounting'
***mounting
+ mount /dev/sda1 /media/vms
+ mount /dev/sda2 /media/apps
+ mount /dev/sda3 /media/novos
My system runs from a CFast card mounted on /dev/sdc and /dev/sda is used for logging. All tests where made ssh to the box.
Linux test 4.9.0-12-rt-amd64 #1 SMP PREEMPT RT Debian
4.9.210-1 (2020-01-20) x86_64 GNU/Linux
PHP 7.0.33-0+deb9u7 (cli) (built: Feb 16 2020 15:11:40) ( NTS )
parted (GNU parted) 3.2
util-linux 2.29.2 (fdisk, sfdisk)
Any pointers/comments will be very appreciated.
Thanks in advance.
I can use both ini_set('memory_limit', '512M'); in the file and php -d memory_limit=512M from the command line, but can also trace memory usage from terminal?
I know I can use memory_get_usage() inside a PHP file, but how to trace it from the command line?
Try:
$ watch -n 5 'php -r "var_dump(memory_get_usage());"'
This will watch every 5 seconds the memory state
Or may be you can use the 'ps' tool:
$ ps -F -C php-cgi
Output:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
http 10794 10786 0 4073 228 0 Jun09 ? 00:00:00 /usr/bin/php-cgi
RSS is the Real-memory (resident set) size in kilobytes of the process.
The solution I was looking for with a simple output is
watch -n 5 'php -r "echo (string) memory_get_usage(true)/pow(10, 6);"'
Will return how many MB the PHP process is using.
2.097152
NOTICE: Solution like ps -F -C php-cgi on macOS machines will fail with
ps: illegal option -- F
This is pretty weird, and I searched and tried everything, but I think I'm just making a dumb syntax error here.
I'm trying to run a stress test on the CPU , then immediately limit it's cpu usage to 30% , all this via PHP. The test is also run under another user and with a specified name so it can be limited. The stress test starts fine, but I can see the PHP file still loading, and it ends the second the stress test ends.
Here's some of the ways I tried doing it
$output = exec('sudo runuser -l test -c "exec -a MyUniqueProcessName stress -c 1 -t 60s & cpulimit -e MyUniqueProcessName -l 30"');
$output = exec('sudo runuser -l test -c "exec -a MyUniqueProcessName stress -c 1 -t 60s > /dev/null & cpulimit -e MyUniqueProcessName -l 30"');
The whole purpose of this is because I am writing a script for a game hosting website, and I want to limit the resource consumption of each server to improve quality and not let someone hog all the resources.
Basically, instead of the stress test, a game server will run.
edit::
here's what I have now:
I need to run the stress under "test" , but the cpulimit under either sudo apache or root, because it requires special permissions. The stress still starts fine but it eats 99.9%
passthru('sudo runuser -l test -c "exec -a MyUniqueProcessName stress -c 1 -t 60s &" & sudo cpulimit -e MyUniqueProcessName -l 30 -i -z');
I can't see the cpulimit in the process list after doing this http://i.imgur.com/iK2nL43.png
Unfortunately, the && does more or less the opposite of what you want. :-) When you do A && B in Bash, that means, "Run command A and wait until it's done; if it succeeded, then run command B."
By contrast, A & B means, "Run command A and then immediately run command B."
So you're close to right in your command, but just getting tripped up by using two bash commands (should only need one) and the &&.
Also, did you try running each command separately, outside PHP, in two terminals? I just downloaded and built both stress and cpulimit (I assume these are the ones you're using?), ran the commands separately, and spotted a problem: cpulimit still isn't limiting the percentage.
Looking at the docs for stress, I see it works by forking child processes, so the one you're trying to CPU-limit is the parent, but that's not the one using the CPU. cpulimit --help reveals there's option -i, which includes child processes in what is limited.
This gets me to the point of being able to enter this in one terminal (first line shows input at the prompt; subsequent show output):
$> exec -a MyUniqueProcessName stress -c 1 -t 60s & cpulimit -e MyUniqueProcessName -l 30 -i
[1] 20229
MyUniqueProcessName: info: [20229] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
Process 20229 found
Then, in another terminal running top, I see:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20237 stackov+ 20 0 7164 100 0 R 30.2 0.0 0:04.38 stress
Much better. (Notice that outside the Bash shell where you aliased it with exec -a, you will see the process name as stress.) Unfortunately, I also see another issue, which is cpulimit remaining "listening" for more processes with that name. Back to cpulimit --help, which reveals the -z option.
Just to reduce the complexity a bit, you could leave the alias off and use the PID of the stress process, via the special Bash variable $!, which refers to the PID of the last process launched. Running the following in a terminal seems to do everything you want:
stress -c 1 -t 60s & cpulimit -p $! -l 30 -i -z
So now, just change the PHP script with what we've learned:
exec('bash -c "exec -a MyUniqueProcessName stress -c 1 -t 60s & cpulimit -e MyUniqueProcessName -l 30 -i -z"');
...or, simpler version:
exec('bash -c "stress -c 1 -t 60s & cpulimit -p \$! -l 30 -i -z"');
(Notice the $ in the $! had to be escaped with a backslash, \$!, because of the way it's quoted when passed to bash -c.)
Final Answer:
Based on the last example you amended to your question, you'll want something like this:
passthru('bash -c "sudo -u test stress -c 1 -t 60s & sudo -u root cpulimit -p \$! -l 30 -i -z"');
When I run this with php stackoverflow-question.php, it outputs the following:
stress: info: [3374] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [3374] successful run completed in 60s
Process 3371 found
(The second two lines only appear after the PHP script finishes, so don't be mislead. Use top to check.)
Running top in another terminal during the 60 seconds the PHP script is running, I see:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3472 test 20 0 7160 92 0 R 29.5 0.0 0:07.50 stress
3470 root 9 -11 4236 712 580 S 9.0 0.0 0:02.28 cpulimit
This is exactly what you've described wanting: stress is running under the user test, and cpulimit is running under the user root (both of which you can change in the command, if desired). stress is limited to around 30%.
I'm not familiar with runuser and don't see the applicability, since sudo is the standard way to run a process as another user. To get this to work, you may have to adjust your settings in /etc/sudoers (which will require root access, but you obviously already have that). That's entirely outside the scope of this discussion, but as an example, I added the following rules:
my-user ALL=(test) NOPASSWD:NOEXEC: /home/my-user/development/stackoverflow/stress
my-user ALL=(root) NOPASSWD:NOEXEC: /home/my-user/development/stackoverflow/cpulimit
Ok, so I have a ssh connection open to a remote server. I'm running a tail on the logs and if an ID shows up in the logs I need to do an insert into the database.
So I have my ssh tail working and I have it piping into my grep function which is giving me the IDs I need. The next step is that as those IDs are found it needs to immediately kick off a php script.
What I thought it would look like is:
ssh -t <user>#<host> "tail -f /APP/logs/foo.log" | grep -oh "'[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'" | php myscript.php <grep result>
And yes my regex is horrible, wanted to use [0-9]{8}, but what I have gets the job done.
Any suggestions? Tried looking at -exec, awk, or any other tool. I could write the result to its own file and then read the new file, but that doesn't catch the streaming ids.
-=-=-=-=-EDIT-=-=-=-=-=-
So here is what I'm using:
ssh -t <user>#<host> "tail -f /APP/logs/foo.log" |grep "^javax.ejb.ObjectNotFoundException" |awk '/[0-9]/ { system("php myscript.php "$6) }'
And if I use tail -l #lines it works, or if after a while I ctrl-c, it then works. The behavior I wanted though was to as the tail got a bad ID to kick off the script to fix the bad ID. Not wait until an EOF or some tail buffer...
I'm having similar problem. There's something funny with tail -f and grep -o combination when ssh.
So on local server, if you do
tail -f myfile.log |grep -o keyword
It grep just fine.
But if you do it from remote server....
ssh user#server 'tail -f myfile.log |grep -o keyword'
doesn't work
But if you remove -f from tail or -o from grep, work just fine... weird :-/
This is my ffmpeg process:
exec("/usr/local/bin/ffmpeg -y -i source.avi dest.mp4 >/dev/null 2>/dev/null &
Now, I wish to execute a PHP file after the conversion is complete. Logically, this is what I have:
exec("/usr/local/bin/ffmpeg -y -i source.avi dest.mp4 >/dev/null 2>/dev/null ; php proceed.php &
This doesn't work though, since then PHP will hold up the process to wait till the ffmpeg conversion is complete. What I want is basically to call proceed.php after the conversion completes, both of which are done in the background.
If anyone can provide the Windows server solution, that will be awesome too.
Write an external (bash/php) script that executes both the ffmpeg and php process, and tack & after that.
For windows, please open a new question on SO.
To add on to what Evert had posted, here is an example of what I use for my FFMPEG bash script... it's far from done (it doesn't alert if the program crashes, for instance) but it's somewhere to start:
#!/bin/sh
## Set our paths
FFMPEG_PATH=/usr/local/bin
SITE_PATH=path_to_file
VIDEO_PATH=$SITE_PATH/public_html/videos
## Make sure we have permissions to do this stuff
chown -R wwwrun:www $VIDEO_PATH/$2
chmod -R 765 $VIDEO_PATH/$2
## Set the options for mp4 compression
options="-vcodec libx264 -b 512k -ar 22050 -flags +loop+mv4 -cmp 256 \
-partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 \
-me_method hex -subq 7 -trellis 1 -refs 5 -bf 3 \
-flags2 +bpyramid+wpred+mixed_refs+dct8x8 -coder 1 -me_range 16 \
-g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qmin 10\
-qmax 51 -qdiff 4"
## Start the conversion.
$FFMPEG_PATH/ffmpeg -y -i $VIDEO_PATH/$2/original/$1 -an -pass 1 -threads 2 $options $VIDEO_PATH/$2/$2.mp4 2> $VIDEO_PATH/$2/pass_one.log
$FFMPEG_PATH/ffmpeg -y -i $VIDEO_PATH/$2/original/$1 -acodec libfaac -ab 96k -pass 2 -threads 2 $options $VIDEO_PATH/$2/$2.mp4 2> $VIDEO_PATH/$2/pass_two.log
## Create the thumbnail for the video
. $SITE_PATH/bin/create_thumbnail $2 00:00:15 2> $VIDEO_PATH/$2/generate_thumbnails.log
## Clean up the log files that were created
## find /log_path/ -name *log* -exec rm {} \;
## Update datbase and send email that we're done here.
php $SITE_PATH/public_html/admin/includes/video_status.php converting_finished $2
And this all gets called from a PHP file that does (along with some other code):
proc_close(proc_open(server_path.'/bin/convert_video_mp4 '.mysql_result($next_video, 0, "uid").'.'.mysql_result($next_video, 0, "original_ext").' '.mysql_result($next_video, 0, "uid").' &', array(), $foo));
PS - I know mysql extension are on their way out, I haven't been using or updating this code in a while, so please update to your specifications