Accessing python nltk with php fails - php

Question

Im trying to call a python file containing a sentence/word tokenizer from my php file like this:
$output = shell_exec('python tokenizer.py $sentence')
I've tried single exec, full paths to python and tokenizer.py, wrapping $sentence in double quotes. But logically, It should not be the problem because calling print(1) at the beginning of python the python code before actually using any nltk packages makes $output equal to '1'. So I came to conclusion that the problem here is the nltk itself, like the path to the modules is not correct or something...
But, calling python from the shell using the same command as above gives me fully tokenized output! To conclude: looks like when calling python from php magically 'turns off' nltk, while it fully works when executed from the shell.
Here's the part of the python code I am using:
import sys
import nltk
from nltk.tokenize import sent_tokenize
sample_text2 = sys.argv[1]
gust = sent_tokenize(sample_text2)
#print(1) here doesn't work, but everywhere above (before calling sent_tokenize) it does.
The server's running on CentOS (Linux), I am accessing it via SSH.
Obvious question: What am I doing wrong here with PHP? Or generally? Any alternatives?
EDIT
As visible in dvhh's answer and its comments, the situation happened because there were two versions installed on the server (2.6 and 2.7), while the www user had access to 2.6 and through console, the default version was 2.7. The solution was to change the default python to 2.7 for both cases and to put the nltk modules to one of the dependency folders. (Or append the dependency directory using sys.path.append)

Answer 1

Your php script is executed by the www user.
You could check if the python script interpreter is correctly called, it is usually in one of the directory in the PATH environment variable (like /usr/bin/python), but the www user don't have a PATH environment variable set.
Solution specify the whole path to your python interpreter in your shell_exec call ( also specify the full path to your script when you're at it )
What about the path the nltk library is installed, you could check if the python interpreter would correctly look for it by looking at the sys.path while running python with the www user.
Diagnostic : use the shell_exec call to run a python script to print the sys.path values
Solution : append the library path to the sys.path in your python script before the import nltk
These would be the most obvious solutions considering the information provided in the question.
Update :
As there is 2 version version of python installed (on that haven't got the library installed ), it is recommended to specify the path to the desired interpreter. The first solution help correct the issue.
In unix like system I would recommend using which python command to determine the path of your default python interpreter.

Answer 2

Option 1
Setup a simple python httpserver listening on localhost. This old answer might help but there are plenty of howtos out there. The advantage is that you don't have the overhead of starting the python interpreter each time the ntlk stuff needs to be executed and you don't have to worry about shell script executions, permissions etc. Disadvantage is a little of extra work and a little overhead.
Option 2
Using a task queue. Whatever said and done it's not safe to execute commands from your web facing PHP scripts. If you are already using RabbitMQ or something similar you can use that here. Or else if you are using redis you can use the lpush, rpop methods to make redis behave like a queue. Disadvantage: the result is not immidiately available.
Option 3
Anbother strategy for your php script to enter the data into a table and setup your python script to run as a cron job to check the table once a minute. Disadvantage: the result is not immidiately available.
Option 4
Your current choice but please make sure that you have escaped the data properly by #lafor if this option is chosen #dvhh 's answer ought to work.

Answer 3

In my case it wasn't problem of python version but problem (as I soon found) of the nltk_data folder. First I thought it was as permission problem, so I changed the permissions to 777, but that didn't work.
I had to copy the folder.
First you have to find where was the nltk folder installed. You will find it by running python3 command from your bash and then put following lines:
import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
It will say something like:
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Package punkt is already up-to-date!
Now you know you have it installed eg in /root folder (i had it there).
Then put in your python script which is called by php this:
import nltk
print(nltk.data.path)
It will output something like: (note the folders are another if you run it from php script or if you call it from command line)
['/var/www/nltk_data', '/usr/nltk_data', '/usr/share/nltk_data', '/usr/lib/nltk_data', '/usr/share/nltk_data', '/usr/local/share/nltk_data', '/usr/lib/nltk_data', '/usr/local/lib/nltk_data']
Now just copy your original folder to any of the above folders. I did:
cp /root/nltk_data /var/www/nltk_data -r
Voila. It works now.