I have a scraper which scrape one site (Written in python). While scraping the site, that print lines which are about to write in CSV. Scraper has been written in Python and now I want to execute it via PHP code. My question is
how can I print each line which is getting printed by python code.
I have used exec function but it is none of my use and gives output after executing all the program. So;
Is it possible to get python output printed while it is getting executed via PHP.
If i understand it well, your python scraper output to a file and you want to "live" display the output via php. What about doing and loop in php in which you use filemtime to know whether or not the file has been updated? You might add a little sleep in order not to overload your server.
If your are using a web page, you may use AJAX to reload only the concerned part of the page at a regular interval.
Hoping this helps you.
Simple case
Assuming execution of scraper is limited to php runtime, run it via popen: http://php.net/manual/en/function.popen.php
More involved case
If you want scraper to run on background and only connect to it vis php from time to time, you can either use some pub/sub toolkit or implement a small web server in the scraper that you can fetch result updates with fopen("https://localhost:port/...") or curl. Many other rpc mechanisms are possible, domain sockets, watching a file, sysv rpc...
I'd communicate using stdout instead of a file. Meaning the python script writes to stdout and the php script reads that.
Using proc_open you can control the python process from php and also read it's output.
Instead of using exec you could use passthru, that will output the data directly to the browser. http://php.net/manual/en/function.passthru.php
That should be enough to get the println from your script.
I think I have a fair idea of what you are saying put I am not too sure what you mean.
If you mean to say that everytime the python script does a print, you want the php code to output what was print?
If that is the case you could pass it as a POST DATA via HTTP. That is instead of printing in Python, you could send it to the PHP Script, which on receiving the data would print it.
I am not too sure if this is what you want though.
For proper communication you need to setup any medium, so you can use fifo, through it you can write string in python and read it with php.
For PHP fifo
http://php.net/manual/en/function.posix-mkfifo.php
For Python
http://www.doughellmann.com/PyMOTW/Queue/
Simply use system() instead of exec(). exec() saves all lines of stdout output of the external program into an array, but system() flushes stdout output "as it happens".
Related
I am using php to run a python script and fetching its output using json.dump and showing on my php page. I feel its slower than when I run it through python idle.
If I understand your question correctly it's no surprise that it feels slower because calling your Python script from PHP, instead of calling it from CLI, increases the operations your PC has to execute. Consider this: first PHP has to create a shell to call your script, then wait for it to finish (e.g. wait for an exit code appearing in the buffer), grab everything from the buffer and then push it into the output buffer and then flush the output buffer, so the data is actually displayed on your page. And besides all of that your output data is transported twice, first from Python to PHP and then from PHP to your browser.
Furthermore, the processing speed depends on the method you use to call your Python script - there are a couple of ways to achieve this and some have more overhead than others.
I have written a python which used to take values from command line and process the same. 5 variables are taken using raw_input() and in each case something is returned to screen.
I want this whole interaction to happen via php program which calls my python program. They are supposed to exchange variables more than one time.
I have read the possible solutions like running the python through php via shell and passing arguments. But I am not sure how once I start my python program I can simply keep on sending variables to it so that my python program reaches its logical end by getting variables from php.
you have to use an IPC mechanism like file, pipe, named pipe, shared memory,...
http://en.wikipedia.org/wiki/Inter-process_communication
You can generally communicate between languages by using common
language formats, and using stdin and stdout [pipe] to communicate
the data.
from: http://www.stackoverflow.com/questions/14047979/executing-python-script-in-php-and-exchanging-data-between-the-two
I am running an external script in PHP using its exec() function. I was looking for various options to create a progress bar. I am able to create a plain rotating loader through AJAX but I couldn't achieve the percentage progress bar. Is there any way to do that?
Depending on the program you want to execute, you could use proc_open() instead of exec(), so you can proccess the output, calculate a percentage and throw it back to your ajax script using ob_flush() and flush().
It's not true that you need to wait for the execution to finish before sending any output, but there are some caveats with your server cache configuration and browsers rendering engines, so it is not so reliable.
If you're not using Websockets (the clean and modern option, which can be achieved with PHP using Ratchet or with nodejs using various options), the most reliable way of doing what you want is with polling.
Briefly, instead of calling your script once, you do a first ajax request to init the proccess and then start poking the server again and again to ask the execution status of your script.
For more information, take a look at those answers:
Run process with realtime output in PHP
PHP - Flushing While Loop Data with Ajax
Grab results from a php exec() while the command is still running?
PHP runs on the server, thus can not achieve this (to my knowledge),
here are some answered questions that might be able to help you.
How to show loading status in percentage for ajax response?
Jquery:: Ajax powered progress bar?
I am trying to run a program using PHP and keep sending output to it. I've tried using exec() but as the documentation page says, it hangs, waiting for the process to return.
Is there something like exec() that would allow me to keep sending commands to a CLI application?
Please note that since the application is closed-source, I don't have the option of changing the application to look for a lock file or any other thing suggested as answers to similar questions.
You are looking for the proc_open command. This command runs a process connecting its standard input and output to file descriptors opened as pipes in your program. What you write the 0 descriptor is taken as input on stdin by the process, and what it outputs can be read by your program as of from a file. The example code in the linked php documentation should get you going.
However, if you are trying to communicate with mysql specifically, rather use the built in mysql functionality of php
I have php file that is to be executed as cronjob, this php file contains some javascript.
I will explain the flow :
the Php is used to retrive some data(A LIST OF URLS) from DB.
For each URL Obtained, a Java script API is used.
THe result Obj returned from API contains data for each url.
The data is then sent back to as an AJAX Call for each url to a php file .
Can this be implemented Via CRON JOBS ?
OR
Is there any method to schedule javascript to run periodically, like cron for php?
UPDATE: i could manage the javascript call to API with PHP curl ,And the cron Job is getting executed perfectly. But i dont think it is the correct solution to this question may be Node.Js is the solution(i didnt test it yet).
You can't run Javascript in Cronjobs because Javascript is ran by browsers. I think you should take a look at curl in php to call an api instead.
http://www.php.net/manual/en/book.curl.php
You have to split the work: Cron the JS, Cron the PHP. In the middle, deliver one's results to another. Agree with phantomjs usage for JS execution (or casperJS-I prefer). Execute the JS, output to JSON as a file, read from the file using file_get_contents from PHP. And define these actions in two different cron jobs.
You can run Javascript via cron in a Javascript runtime like node.js: http://nodejs.org/
phantomjs is one possibility, see this thread wget + JavaScript?
Otherwise you could run Node.js on your server to execute JavaScript in a CLI type environment but mixing node.js and PHP could become complicated.
you can schedule javascript with cron by using Node.js