I'd use the following code:
$SERVER_PATH = dirname(__FILE__);
shell_exec($PHP_LOCATION.' '.$SERVER_PATH."/script.php?k1=v1&k2=v2 > /dev/null 2>/dev/null &");
Where:
$PHP_LOCATION should contain the path to PHP,
$SERVER_PATH - is current working directory (fortunately the script to run is in the same directory),
> /dev/null 2>/dev/null & added to make this call asynchronous (taken from Asynchronous shell exec in PHP question)
This code has two problems:
As far as I remember ?k1=v1&k2=v2 will work for web-call only, so in this particular case parameters will not be passed to the script.
I don't really know how to init the $PHP_LOCATION variable to be flexible and to work on the most hosts.
I conducted some research regarding both problems:
To solve 1 suggested to use -- 'parameters_string' but it is also recommended to modify the script to parse parameters string which looks a bit clumsy. Is there a better solution?
To solve 2 I found a solution to use PHP_BINARY but this is a PHP 5.4+ case (I'm using 5.3). But the original question was about to run PHP of the same version as the original script version. So for me (as I use PHP 5.3 only) is there probably a solution?
EDIT 0
Let me do some explanation why I stuck to this weird (for PHP) approach:
Those PHP scripts should be separate from each other:
one of those will analyze the data and
the second will generate PNG graphs as a final result.
Those scripts aren't intended to run simultaneously, this means that the second can run at it's own schedule it is only needed that the run should be upon its data will be ready (which done by the first script). So no data should be passed back from second script (child) to the first (parent).
EDIT 1
As seeing from most of the comments the main discussion goes to forking direction. However I'd like to make stress on the point 1 and 2 asked in the original questions. I have some reasons to solve the task in the way I pointed out and I tried to point all that reason. If some of my points looks weird, please post a comment - I will make it more clear or I will change the main question.
Thank you in advance!
How to get executable
Assuming you're using Linux, you can use:
function getBinaryRunner($binary)
{
return trim(shell_exec('which '.$binary));
}
For instance, same may be used for checking if needed stuff is installed:
function checkIfCommandExists($command)
{
$result = shell_exec('which '.$command);
return !empty($result);
}
Some points:
Yes, it will work only for Linux
You should be careful with user input if it is allowed to be passed to shell commands: escapeshellarg() and company
Indeed, normally PHP should not be used for stuff like this as if it is about asynchronous requests, better to either implement forking or run commands from external workers.
How to pass parameters
With doing shell_exec() via file system path you're accessing the file and, obviously, all "GET" parameters are becoming just part of file name, it is no longer "URI" as there is no web-server to process that. So you have two options:
Invoke call via accessing your web-server. So it will be like:
//Yes, you will use wget or, better, curl to make web-request from CLI
shell_exec('wget http://your.web-server.domain/script.php?foo=bar');
Downside here: if you'll access your web-server via public DNS, it will cause network gap and all processing overheads. Benefit - obviously, you will not have to expect anything else in your script and make no distinction between CLI and non-CLI calls
Use $_SERVER array in your script and pass parameters as it should be with CLI:
shell_exec('/usr/bin/php /path/to/script.php foo bar');
//inside your script.php you will see:
//$_SERVER['argv'][0] is "script.php"
//$_SERVER['argv'][1] is "foo"
//$_SERVER['argv'][2] is "bar"
Yes, it will require modification in the script, and, probably, some logic of how to map "regular" web-requests and CLI ones. I would suggest even to think of separating CLI-related stuff to different scripts bundle so not to mess that logic.
More about "asynchronous run"
When you do php script.php & you just run it in background mode. That, however, still keeps parent-child relation for your process. That means - if parent process dies, it's childs will also be removed. To be precise, SIGHUP will be triggered and to avoid this situation you should use nohup command. It will allow to emulate "detaching" of a process and therefore making it's run reliable and independent of circumstances happening to parent process.
Related
I have a web server running Apache 2 on Raspbian Stretch. It is going to be a programming contest website, where users can send code via a HTML form, that sends their source code to PHP via a POST request. PHP then runs (using exec()) a Python script with arguments such as the submitted code path. The script then executes the code (using subprocess.run()) with a custom input and compares it to an expected output. All of that works just fine.
However, I want to make sure no one is going to send malicious code to overwrite files such as index.php, or read the expected outputs, for example. I'd like to know if there is any way to prevent an application that is being executed by subprocess.run() from reading, creating and writing to files other than stdin, stdout and stderr.
I have tried using Docker but didn't have success, as when I build and run the Dockerfile using PHP's exec() it reaches step 2/4 and just stops. My Dockerfile should copy the script, the code and the expected outputs to an image, cd to the new location and execute the code, but that is not very relevant since I want to avoid Docker as it isn't working properly.
I am considering using a chroot jail, but I am still looking for other less-complicated ways of doing that.
This is the PHP code I'm using. It calls the Python 3 code verifier (variables are retrieved from a HTML form and from a SQL query, those are not relevant):
$cmd = "python3 verify.py $uploadedFile $questionID $uploadedFileExtension $questionTimeLimit 2>&1";
And this is the Python 3 code that executes the submitted code:
def runCmd(args, vStdin, timelimit = 10):
p = subprocess.run(args, stdout = subprocess.PIPE, stderr = subprocess.PIPE, input = vStdin, encoding = 'utf-8', timeout=timelimit)
vStdout = p.stdout
vStderr = p.stderr
if vStdout.endswith('\n'):
vStdout = vStdout[:-1]
if vStderr.endswith('\n'):
vStderr = vStderr[:-1]
return vStdout, vStderr
...
# Assuming it is a .py file
# Its path is given by PHP's exec.
runCmd(['python3', sys.argv[1], 'simulatedinput.in', int(sys.argv[4]))
The combination of both programs works just fine. It runs the code with a simulated input, compares the stdout with the expected output and returns a status string to the PHP code. However, if the code sent has a malicious bit of code, such as
open('/var/www/html/contest/index.php', 'w').write('oops!')
the index.php file will be overwritten.
All I need is a way of executing the user-sent code in a way that its attempts to read or write to files (other than stdin, stdout and stderr) are denied.
Any thoughts?
doing this securely, to put it simply, is difficult. it's relatively easy to escape even a chroot jail if you're not really careful about how you set it up. basically the Unix security model isn't built to make this sort of thing easy and it's assumed that things are mostly cooperative
docker would probably be my suggestion, but there are other lighter weight solutions like chroot (but they'd probably still have the ability to do naughty things with the web server's network connection) or maybe something like firejail
with docker you'd probably want to create a single minimal docker image/container containing Python and whatever libraries are appropriate. you'd then use volumes to make the user supplied code appear inside the VM at runtime. you don't want to be creating containers all the time, that would entail lots of cleanup work
see https://security.stackexchange.com/q/107850/36536 for some more info on using docker as a sandbox, basically there are still lots ways out of it unless you're careful
I have a PHP script which sends messages to a list of users, which I hosted in Heroku. Now I wanted to add a delay in between those messages. Say like, 6 mins gap between each message. So if there are 90 users, that script should run for 9 hours, in the background.
I tried calling this script using ajax, so it runs in the background and adding sleep(360); inside the for loop, to get 6 mins delay. But it only works for approx 10 to 20 users, after that it stops.
foreach ($users_list["users"] as $key => $value) {
try{
....
code for sending message
.....
}catch(Exception $e){
continue;
}
sleep(360);
}
So I would like to know, what is the optimal way to achieve this, in Heroku.
Calling a script using AJAX doesn't call it in the "background" it just runs it Asynchronously from the page you are on. In other words it's still running in Apache, has any session data, and still bound by the timeout settings of PHP and Apache.
To run it truly in the background you can use something like CRON
Or if you are allowed to on your server you can call it by command line like with exec or shell_exec, there are a few other similar functions too, such as popen, system etc. They all do things in a slightly different way.
Some environmental stuff will be different and this may have a big impact on your code. For example a lot of stuff in $_SERVER is not set or has different information. Such as the servers IP address may not be in there, you won't have any session stuff. You won't be able to use $_GET or $_POST but can get the input data (form the command line call) from the $argv array, the first item being the files path... etc...
Basically you need to call it like this:
exec('php -f "path/to/php/file.php" "arg1" "arg2"');
Calling it this way it will still be blocking, meaning it waits for execution of the called script.
To go one step further and make it non-blockin you can add (on Linux)
exec('php -f "path/to/php/file.php" "arg1" "arg2" > /dev/null &');
The & at the end is the most important bit.
Now on Windows it's a bit of a different ballgame. I've had success using this
$WshShell = new \COM('WScript.Shell');
$cmd = 'cmd /C php "path/to/php/file.php" "arg1" "arg2"';
$WshShell->Run($cmd, 0, false);
Also on windows to run PHP with just php you have to add the path to the php.exe yhou want to use to the path environmental variable. Otherwise you have to use the full path to the exe instead of just php
In either case you should be very careful about putting end user data in any command line call. There are 2 functions to sanitize it, but I try to just not put it in.
escapeshellarg
escapeshellcmd
I wrote a wrapper class for this you can find on my GitHub
Hope it helps.
I am trying to build a small custom task scheduler. Basically the idea is I have cron run my process script, which looks in the database and finds any scheduled tasks that are ready to run, and runs them. So I think the best way to do this would be to try to launch the tasks "in the background" by way of shell_exec and using > /dev/null, which I understand makes it so the initial script (the process script) doesn't wait for the task scripts to complete.
So first, if there is a better way to achieve this, I'm open to suggestions. Though note I am on php 5.3 so there may be some options in 5.4 and up that I don't have access to :(
However here's the question at hand:
I am testing on WAMP on my windows machine and I am trying to make a call that looks like this:
shell_exec("php $path$base_url$querystring > output_test.txt 2>&1 &");
$path is the full windows path to the script
$base_url is the base url of the script I am calling
$querystring is of course the query string being passed to the task script
I am also outputting to output_test.txt which creates such file in same directory, where I get the following error:
Could not open input file:
C:\xampp\htdocs\email\batch_email_send_u2u.php?dealer=7
Yes I realize the path references an xampp installation, but that is not the issue - all the wamp files are executing from there and everything else has worked like this for years - it was just set up this way to support a legacy setup.
It seems to me shell_exec is locating and running php, it's just that it can't open the referenced script. Can't figure out why.
Also I need to eventually get this working on a real linux server so any advice on how to make that happen would be greatly appreciated!
Found a solution! Special thanks to dan08 for getting me set on the right path.
Ultimately I found the answer in this thread: Pass variable to php script running from command line
I ended up using the argv[] array as described in that post and with a little tweak to the script I'm calling it works like a champ now.
READ FURTHER BELOW at CLI, FOR THE CLI QUESTION, WHICH JUST ADDED TO THE CONVERSATION! THX!
I have written a script which processes an xml file of around 160'000 entries with 48.1MB and a text file of 150'000 entries with 31.1MB, including some directory searches for external files, heavy interlinking and recursive checks and the result formatted and all saved into html files.
Surely, I did review the program couple times and ended up with the most efficient code I could think of. This is a local program and the generator doesn't need to run regularly. One could argue that I should use an other language than PHP, but PHP with simplexml, etc. just works the best for me and for this purpose. Also a set_time_limit('70000') doesn't bother me.
Although, here my question, is it possible to make the apache2 on my linux system, use my 4 CPU cores running my PHP script?
Even if I split the process and make several request's simultaneously, the CPU usage can't go above 1 CPU at a time.
I googled this topic but couldn't find a solution, so I may have to just run it over night, even though, I would appreciate some help to boost that thing!!!
ADDED INFO - And here a picture of my processes:
CLI:
I need to call my index.php in the linux terminal to execute. But I also wanna send four post variables ($_POST['example']) to the script. On top of that, I am looking for having my echos presented in some output file. Could anyone help quickly with the terminal command and the php command to track those 4 post variables inside:
if (PHP_SAPI === 'cli')
{
// ...
}
? ...sorry but this is my first php-cli interaction. Thx!
No, a single PHP script will never use multiple threads and thus always run on single core.
Depending on how much the things you do depend on each other you couldn't easily split them on multiple threads anyway.
EDIT: Author's response
This is not a solution but a nice workaround. I clone my virtual machine with the linux/apache2 install to kick in the same process but different parts of the file/process on different vm's, which lets the host system apply one core for each virtual system, that way I could break down the process time by around the factor 4. Thanks for your posts!
===============
If it's local, and you want to run it every now and then, you should probably just invoke it from a cron job. That way, you can spawn a process for each task you are doing. If you really do want to use PHP for it, you can even invoke PHP to do it from the cron line.
None the less, it sounds like you're doing an inherently single-threaded process anyway, and if you want it faster, should probably use something that isn't PHP for this.
Maybe you can use Spork! It's a php lib allowing you to fork the php process into multiple ones.
<?php
use Spork\Deferred\DeferredFactory;
use Spork\ProcessManager;
$manager = new ProcessManager(new DeferredFactory());
$manager->fork(function() {
// do something in another process!
})->then(function($output, $status) {
// do something in the parent process when it's done!
});
https://github.com/kriswallsmith/spork
SOLUTION, THX TO ThiefMaster and Zebediah49 recommending cli and my friend who supported me with the links: http://ch.php.net/manual/en/reserved.variables.argv.php / http://ch.php.net/manual/en/function.getopt.php
and here how I call the php through cli:
//whenRunFromCLI
//callCLI
//php index.php './data/xyfullFile1.xml' './data/xxfullFile2.utf' 0 60000
//php index.php './data/xyfullFile1.xml' './data/xxfullFile2.utf' 60000 120000
//php index.php './data/xyfullFile1.xml' './data/xxfullFile2.utf' 120000 all
if (PHP_SAPI === 'cli'){
$_POST['xml'] = $argv[1];
$_POST['example'] = $argv[2];
#$_POST['rangeFrom'] = $argv[3];
#$_POST['rangeTo'] = $argv[4];
}
and the Result of calling the php file in three terminals:
I know, I must give some more RAM to my virtual machine, lucky that I still have 8GB spare ;-)
Cheers and peace!
I've got a script that calls two functions, A and B, from the same class. A creates an Amazon virtual server and B destroys one, both via shell_exec()'s of Amazon's command line tools. The script, doActions.php, pulls actions from a queue. If the action is "create" it creates an instance; when the action is "destroy" it kills one.
The script works fine to execute both A and B when I execute it from the command line: php script.php.
When I put it on a cron, it runs but only successfully runs the B function. It deletes destroys instances but won't create them.
The point of failure is clearly function B. It chokes at the first and most important shell_exec, returning and echoing nothing.
echo $string = shell_exec('/home/user/public_html/domain.com/private/ec2-api-tools/bin/ec2-run-instances ami-23b6534a -k gsg-keypair -z us-east-1a');
Unless you know something specific about the way Amazon's command line tools work, please suggest to me reasons why a shell_exec might work in one case and not the other.
Another shell_exec in the same place behaves as expected:
echo $string = shell_exec ('echo overflow');
My guess is that it has to do something with permissions. But when I have it run shell_exec('whoami') it return "root," and when I su and run the command it works fine. I'm having a hard time thinking of creative ways to troubleshoot why my PHP script won't work in cron when it does from the command line. Can you suggest some?
When something runs from the command line but refuses to do so within cron, it's often an environment issue (path or some other environment variable that's needed by the code you're running).
For a start you should modify the script to output the current environment (shell_exec('env')?) at the very top and examine the output from the command line and cron.
Hopefully, there will be something obvious such as AMAZON_EC2_VITAL_VAR but, if not, you should move the cron environment towards your command line one, one variable at a time, until it starts working.
A quick test to ascertain this. From your command line, do:
env >/tmp/pax_env.sh
Then run your PHP script from a shell script which first executes:
. /tmp/pax_env.sh
so that the environments are identical.
And keep in mind that su on its own doesn't give you the same environment as you'd get from logging in directly as a specific user (su - does, I think). You may want to check the behaviour for when you log in as root directly.
Re your comment:
Yes, I do believe you've got it. I'm likely going to mark your answer as correct but need you to suffer through a few addendums about your clever solution. First of all, what's the best way to execute the pax_env.sh script? Does shell_exec() work?
Never let it be said I didn't work for my money :-) No. The shell_exec will almost certainly be running a sub-shell so the variables would be set in that sub-shell but would not affect the PHP parent process.
My advice, if you wanted all those variables set, would be to create a shell-script consisting of all the commands in /tmp/pax_env.sh (probably prefixing each with export) followed by the command you currently have running in cron, something along the lines of:
export PATH=.:/usr/bin
export PS1=Urk:
export PS2=MoreUrk:
/home/user/pax/scriptB.php
Then run that script from cron rather than /home/user/pax/scriptB.php directly. That will ensure the environment is set up before your PHP code is called.
Astute readers will have noticed the phrase "if you wanted all those variables set" above. I don't personally think it's a good idea to dump all your command line variables into the shell script for the cron job. I'd prefer to actually find out which ones are needed and only include those. That lessens the pollution your cron job has to run under. For example, it's unlikely that the PS1/PS2 prompt variables will be required for your PHP script.
If it works, you can set all the environment variables - I just prefer the absolute minimum so I don't have to worry too much when things change.
A way of finding out what's needed is to comment out one export at a time until your script breaks again. Then you know that variable is needed. Once it works with the maximum amount of export statements commented out, you can just delete those commented export statements altogether and what remains, however improbable, must be okay (with apologies to Sir Arthur Conan Doyle).