youtube-dl and php exec - php

I have installed youtube-dl on my CentOS 6 / Plesk 10 Dedicated server, and via SSH, everything works like a charm !
The thing is I'm trying to write a php script which takes a POST parameter in containing the URL of the video I want to copy to the server, copy it and then process it with ffmpeg.
I thought I would use the exec() function.
I can get an output if I echo the results of youtube-dl --help, but everytime I ask php to execute a command which actually does something with the video, it returns a status of '1', and outputs nothing.
Any ideas on what I'm doing wrong ?
Here is my php code:
<?php
$result = array();
$status;
$url = $_POST['src'];
$string = 'youtube-dl "'.$url.'" -f 18 -o "/var/www/vhosts/my.virtual.host.net/httpdocs/downloader/downloads/%(id)s.%(ext)s"';
$string2 = 'youtube-dl --help';
exec($string, $result, $status);
echo json_encode(array('status' => $status, 'url_orginal'=>$url, 'url' => $result));
?>
When I execute $string2, I get status: "0" and "url": [youtube-dl help text lines]
But when I execute $string, nothing happens, I get "status": "1" and nothing else, no video is downloaded. I've tried also a simulation with the "-g" parameter, and variants but as soon as youtube-dl has to fetch the video, it breaks.
Thank you in advance !
EDIT
I edited my code so it looks like this :
<?php
$result = array();
$status;
$url = $_POST['src'];
$string = 'youtube-dl "'.$url.'" -f 18 -o "/var/www/vhosts/my.virtual.host.net/httpdocs/downloader/downloads/%(id)s.%(ext)s"';
$string2 = 'youtube-dl --help';
exec($string, $result, $status);
echo json_encode(array('status' => $status, 'url_orginal'=>$url, 'url' => $result, 'command' => $string));
?>
and the result, which I didn't get yesterday now is :
command: "youtube-dl "http://www.youtube.com/watch?v=coq9klG41R8" -f 18 -o "/var/www/vhosts/my.virtual.host.net/httpdocs/downloader/downloads/%(id)s.%(ext)s""
status: 1
url:
0: "[youtube] Setting language"
1: "[youtube] coq9klG41R8: Downloading video info webpage"
2: "[youtube] coq9klG41R8: Extracting video information"
url_orginal: "http://www.youtube.com/watch?v=coq9klG41R8"
Which is weird, considering a) that yesterday I got an empty url[], and that b) even if now I get what apparently looks like a normal youtube-dl return, it only contains the 3 first lines, and I cannot see any video file in the specified path... Any ideas ?

exec reads only stdout, so you're missing the error message on stderr, whatever that may be.
Use the following code to get stderr as well:
$url = 'http://www.youtube.com/watch?v=coq9klG41R8';
$template = '/var/www/vhosts/my.virtual.host.net/httpdocs/downloader/' .
'downloads/%(id)s.%(ext)s';
$string = ('youtube-dl ' . escapeshellarg($url) . ' -f 18 -o ' .
escapeshellarg($template));
$descriptorspec = array(
0 => array("pipe", "r"), // stdin
1 => array("pipe", "w"), // stdout
2 => array("pipe", "w"), // stderr
);
$process = proc_open($string, $descriptorspec, $pipes);
$stdout = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$stderr = stream_get_contents($pipes[2]);
fclose($pipes[2]);
$ret = proc_close($process);
echo json_encode(array('status' => $ret, 'errors' => $stderr,
'url_orginal'=>$url, 'output' => $stdout,
'command' => $string));
Most likely, you don't have permission to write to that directory. To test that theory, check whether
touch /var/www/vhosts/my.virtual.host.net/httpdocs/downloader/downloads/test
works. You can use chmod and chown to change permissions.

Related

Python ImportError while using youtube-dl by php

Traceback (most recent call last): File "/usr/bin/youtube-dl", line
3, in import youtube_dl File
"/usr/lib/python2.7/dist-packages/youtube_dl/init.py", line 65, in
from .utils import ( File
"/usr/lib/python2.7/dist-packages/youtube_dl/utils.py", line 18, in
import ssl File "/usr/lib/python2.7/ssl.py", line 61, in import _ssl
#if we can't import it, let the error propagate ImportError: /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so: symbol
GENERAL_NAME_free, version OPENSSL_1.0.0 not defined in file
libcrypto.so.1.0.0 with link time reference
The Command i used
youtube-dl --max-quality 2180 --write-thumbnail -x --audio-format mp3
-c -o "/home/bahaa/%(id)s.%(ext)s" https://www.youtube.com/watch?v=xFQFMSNZW08&list=RDV-jLo0Ovems
The php code i am using
<?php
$url = 'https://www.youtube.com/watch?v=xFQFMSNZW08&list=RDV-jLo0Ovems';
//escapeshellarg
$string = ('youtube-dl --max-quality 2180 --write-thumbnail -x --audio-format mp3 -c -o "/home/bahaa/%(id)s.%(ext)s" https://www.youtube.com/watch?v=xFQFMSNZW08&list=RDV-jLo0Ovems');
$descriptorspec = array(
0 => array("pipe", "r"), // stdin
1 => array("pipe", "w"), // stdout
2 => array("pipe", "w"), // stderr
);
$process = proc_open($string, $descriptorspec, $pipes);
$stdout = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$stderr = stream_get_contents($pipes[2]);
fclose($pipes[2]);
$ret = proc_close($process);
echo json_encode(
array(
'status' => $ret,
'errors' => str_replace("\n", "<br />", $stderr."<hr />"),
'output' => $stdout,
)
);
?>
Hi Finally I got the solution!
Firstly, would you please let me know your server configuration? I'm using XAMPP, in my case, the "Python" and its libs used by Apache are not the same as used by my ubuntu, so I build and install Python3 into /opt/lampp/bin and build openssl by myself. You can download openssl1.0.1 from its website and configure it by "./config shared" to build a .so file.
Next copy the libssl.so* and libcrypto.so* (totally 4 files) into /opt/lampp/lib (this is where LD_LIBRARY_PATH points to) and restart apache now
It works for me, Hope it will help you

Call a program via shell_exec with utf-8 text input

Perquisites: hunspell and php5.
Test code from bash:
user#host ~/ $ echo 'sagadījās' | hunspell -d lv_LV,en_US
Hunspell 1.2.14
+ sagadīties
- works properly.
Test code (test.php):
$encoding = "lv_LV.utf-8";
setlocale(LC_CTYPE, $encoding); // test
putenv('LANG='.$encoding); // and another test
$raw_response = shell_exec("LANG=$encoding; echo 'sagadījās' | hunspell -d lv_LV,en_US");
echo $raw_response;
returns
Hunspell 1.2.14
& sagad 5 0: tagad, sagad?ties, sagaudo, sagand?, sagar?o
*
*
Screenshot (could not post code with invalid characters):
It seems that shell_exec cannot handle utf-8 correctly, or maybe some additional encoding/decoding is needed?
EDIT: I had to use en_US.utf-8 to get valid data.
Try this code:
<?php
// The word we are checking
$subject = 'sagadījās';
// We want file pointers for all 3 std streams
$descriptors = array (
0 => array("pipe", "r"), // STDIN
1 => array("pipe", "w"), // STDOUT
2 => array("pipe", "w") // STDERR
);
// An environment variable
$env = array(
'LANG' => 'lv_LV.utf-8'
);
// Try and start the process
if (!is_resource($process = proc_open('hunspell -d lv_LV,en_US', $descriptors, $pipes, NULL, $env))) {
die("Could not start Hunspell!");
}
// Put pipes into sensibly named variables
$stdIn = &$pipes[0];
$stdOut = &$pipes[1];
$stdErr = &$pipes[2];
unset($pipes);
// Write the data to the process and close the pipe
fwrite($stdIn, $subject);
fclose($stdIn);
// Display raw output
echo "STDOUT:\n";
while (!feof($stdOut)) echo fgets($stdOut);
fclose($stdOut);
// Display raw errors
echo "\n\nSTDERR:\n";
while (!feof($stdErr)) echo fgets($stdErr);
fclose($stdErr);
// Close the process pointer
proc_close($process);
?>
Don't forget to verify that the encoding of the file (and therefore the encoding of the data you are passing) actually is UTF-8 ;-)

How to gpg encrypt via PHP on a windows platform, under a webserver?

I'm trying to do GPG encryption on a Windows platform, in PHP, running XAMPP.
The webserver is Apache and is running PHP 5.2.9.
I'm using GPG4Win 2.0.4.
I've had success running the encrypt command from the command line. I've changed the recipient and host names.
C:\>C:\PROGRA~1\GNU\GnuPG\pub\gpg.exe --encrypt --homedir C:\DOCUME~1\reubenh.AD\APPLIC~1\gnupg --recipient name#host.com --armor < test.txt > test.enc.txt
In PHP, I'm using proc_open() so I can pipe the content to be encrypted directly to the process, and use the stdout pipe to grab the output.
Following is a snippet of the code:
$cmd = Configure::read('Legacy.GPG.gpg_bin').' --encrypt '.
'--homedir '.Configure::read('Legacy.GPG.gpg_home').' '.
'--recipient '.Configure::read('Legacy.MO.gnugp_keyname').' '.
'--local-user '.'me#host.com'.' '.
'--armor --no-tty --batch --debug-all';
error_log('Encrypting Command line is '.$cmd);
$descriptors = array(
0 => array('pipe', 'r'),
1 => array('pipe', 'w'),
2 => array('file', LOGS.'gpg.log', 'a')
);
$process = proc_open($cmd, $descriptors, $pipes);
if (is_resource($process)) {
error_log(print_r($pipes, true));
list($stdin, $stdout) = $pipes;
error_log('Have pipes');
error_log('body length is '.strlen($this->request['body']));
$ret = fwrite($stdin, $this->request['body'], strlen($this->request['body']));
error_log($ret.' written');
error_log('before fclose()');
fclose($stdin);
error_log('Done with input');
$encryptedData = '';
while(!feof($stdout)) {
$line = fgets($stdout);
error_log('Line read:'.error_log(print_r($line, true)));
$encryptedData .= $line;
}
fclose($stdout);
$return = proc_close($process);
error_log('Encryption process returned '.print_r($return, true));
if ($return == '0') { // ... next step is to sign
The generated command from the first error_log() statement is:
C:\PROGRA~1\GNU\GnuPG\pub\gpg.exe --encrypt --homedir C:\DOCUME~1\reubenh.AD\APPLIC~1\gnupg --recipient name#host.com --local-user me#host.com --armor --no-tty --batch --debug-all
The actual running seems to get as far as "Have pipes". After that, it just stops.
I can also see in the Process Explorer, that the gpg.exe also spawns a gpg2.exe. I suspect that it is this gpg2.exe that I do not have a handle to, is waiting for the input, not the original gpg.exe that I invoked.
I've tried invoking gpg2.exe directly, but a child gpg2.exe is still spawned.
I'd rather use proc_open(), and avoid using disk I/O to provide the content and grab the output, since this will be run on a webserver, and proc_open() will allow me to not bother generating unique files, and then having to clean them up.
I've ended up compromising, to get the solution initially "work", although I'm not very happy about the way it's been done.
The problem seemed to be in two parts.
The first part was the process would hang when trying to sign, and use the --passwd-fd option. If I left this option out, I would get a prompt through the interactive nature of the webserver, enter it manually, and everything would be ok. The workaround, for an unattended application, is to simply have no passphrase. I've seen recommendations in various GnuPG forums to the effect that if your passphrase is going to stored as plain text on the same machine as the private key, then you may as well dispense with the pretence and don't have one. No passphrase is working for the moment.
The second part was that the input was too large. The magic number seemed to be 72kb. Any payload to be encrypted larger than that, using proc_open and a standard pipe just didn't seem to work. As a result, I've opted for temporarily writing the payload to a file, to be read by the proc_open. See as follows:
$tmpfile = tmpfile();
fwrite($tmpfile, $this->request['body']);
fseek($tmpfile, 0);
$cmd = '...'; // similar to question command, but with --sign --encrypt and no --passphrase-fd
$descriptors = array(
0 => $tmpfile,
1 => array('pipe', 'w'),
2 => array('file', LOGS.'gpg.log', 'a')
);
$options = array('bypass_shell' => true);
$process = proc_open($cmd, $descriptors, $pipes, null, null, $options);
if (is_resource($process)) {
stream_set_blocking($pipes[1], 0);
fclose($tmpfile);
$encryptedData = '';
$line = fgets($pipes[1]);
while (!feof($pipes[1])) {
$encryptedData .= $line;
$line =fgets($pipes[1]);
}
fclose($pipes[1]);
$return = proc_close($process);
if ($return = '0') {
// success processing
}
}
I elected not to use list() = $pipes, because only the stdout pipe would actually return in the array.
If anyone had has experience with GPG and PHP in a Windows environment, I'd be more than welcome to hear something, even if it's years down the track.

Using linux 'dialog' command from PHP

I'm trying to write a helper script for doing various admin tasks on a server which can only be done from the command line and trying to use the 'dialog' command to display message boxes, inputs, password prompts etc, however, the needs of this task call for me to process the data in PHP.
I'm having problems getting the dialog command to work in this way and can't figure out what i'm doing wrong.
There's an example here
Unfortunately it doesn't work.
When you run PHP and exec/backtick/system to an external application, the IO doesn't appear to work how you'd expect.
The nearest I can get is using the passthru() command:
<?php
$CMD = "dialog --menu \"Please select\" 10 40 3 backup \"Backup Files\" restore \"Restore Files\"";
passthru($CMD);
?>
This is the only way that PHP will let dialog use the STDOUT properly, anything else results in no display but you can press return to select an option.
I've tried backticks, exec() and system() but nothing seems to work.
What I wondered was how to read STDERR properly from within PHP to get the return value into a variable called $result.
I'm sure some other sysadmins have had to do this before.
My reasons for not using bash for this are that one command I have to execute as a result of a selection produces XML output only and I can't parse that effectively in bash.
Just in case someone else is searching for this:
function dialog ($args) {
$pipes = array (NULL, NULL, NULL);
// Allow user to interact with dialog
$in = fopen ('php://stdin', 'r');
$out = fopen ('php://stdout', 'w');
// But tell PHP to redirect stderr so we can read it
$p = proc_open ('dialog '.$args, array (
0 => $in,
1 => $out,
2 => array ('pipe', 'w')
), $pipes);
// Wait for and read result
$result = stream_get_contents ($pipes[2]);
// Close all handles
fclose ($pipes[2]);
fclose ($out);
fclose ($in);
proc_close ($p);
// Return result
return $result;
}
It requires dialog (apt-get install dialog) and proc_xxx (PHP 4.3.0, PHP 5)
It works, at least for me. :)
You can use proc_open() but not as show above... All dialog boxes do not work in same way. I provides a concrete sample below:
#!/usr/bin/env php
<?php
$pipes = array();
$process = null;
$output = '';
$ret = -1;
/**
* Start process
*
* #param string $cmd Command to execute
* #param bool $wantinputfd Whether or not input fd (pipe) is required
* #retun void
*/
function processStart($cmd, $wantinputfd = false)
{
global $process, $pipes;
$process = proc_open(
$cmd,
array(
0 => ($wantinputfd) ? array('pipe', 'r') : STDIN, // pipe/fd from which child will read
1 => STDOUT,
2 => array('pipe', 'w'), // pipe to which child will write any errors
3 => array('pipe', 'w') // pipe to which child will write any output
),
$pipes
);
}
/**
* Stop process
*
* #return void
*/
function processStop()
{
global $output, $pipes, $process, $ret;
if (isset($pipes[0]) {
fclose($pipes[0]);
usleep(2000);
}
$output = '';
while ($_ = fgets($pipes[3])) {
$output .= $_;
}
$errors = '';
while ($_ = fgets($pipes[2])) {
fwrite(STDERR, $_);
$errors++;
}
if ($errors) {
fwrite(STDERR, "dialog output the above errors, giving up!\n");
exit(1);
}
fclose($pipes[2]);
fclose($pipes[3]);
do {
usleep(2000);
$status = proc_get_status($process);
} while ($status['running']);
proc_close($process);
$ret = $status['exitcode'];
}
// Test for yesno dialog box
processStart("dialog --backtitle 'dialog test' --title 'Little test' --output-fd 3 --yesno 'yesno dialog box' 0 70");
processStop();
echo "Exit code is $ret\n";
// Test for gauge dialog box
processStart("dialog --backtitle 'dialog test' --title 'Little test' --output-fd 3 --gauge 'Gauge dialog box' 0 70 0", true);
sleep(1);
fwrite($pipes[0], "XXX\n0\nFirst step\nXXX\n20\n");
sleep(1);
fwrite($pipes[0], "XXX\n20\nSecond step\nXXX\n50\n");
sleep(1);
fwrite($pipes[0], "XXX\n50\nThird step\nXXX\n80\n");
sleep(1);
fwrite($pipes[0], "XXX\n80\nFourth step\nXXX\n100\n");
sleep(1);
processStop();
echo "Exit code is $ret\n";
// Test for input dialog box
processStart("dialog --backtitle 'dialog test' --title 'Little test' --output-fd 3 --inputbox 'input dialog box' 0 70");
processStop();
echo "Output is $output\n";
echo "Exit code is $ret\n";
// Test for errors output
processStart("dialog --backtitle 'dialog test' --title 'Little test' --output-fd 3 --dummy 'my input box' 0 70");
processStop();
exit(0);
You can use proc_open() to run a command and interact on all pipes, STDIN, STDOUT and STDERR:
$pipes = array(NULL, NULL, NULL);
$proc = proc_open(
"dialog --gauge ..",
array(
0 => array('pipe', 'r'),
1 => array('pipe', 'w'),
2 => array('pipe', 'w'),
),
$pipes
);
print fgets($pipes[2]);
See the manual for more examples.
PHP-GTK looks like a solution for this problem
http://gtk.php.net/
I think you can't run ncurses application through PHP like this - maybe you should take a look at this: http://php.net/manual/de/book.ncurses.php

PHP StdErr after Exec()

In PHP I am executing a command with exec(), and it returns if successful an URL;
$url = exec('report');
However, I want to check stderr, if something went wrong. How would I read the stream?
I want to use php://stderr, but I am not sure how to use it.
If you want to execute a command, and get both stderr and stdout, not "merged", a solution would probably to use proc_open, which provides a great level of control over the command that's being executed -- including a way to pipe stdin/stdout/stderr.
And here is an example : let's consider we have this shell-script, in test.sh, which writes to both stderr and stdout :
#!/bin/bash
echo 'this is on stdout';
echo 'this is on stdout too';
echo 'this is on stderr' >&2;
echo 'this is on stderr too' >&2;
Now, let's code some PHP, in temp.php -- first, we initialize the i/o descriptors :
$descriptorspec = array(
0 => array("pipe", "r"), // stdin
1 => array("pipe", "w"), // stdout
2 => array("pipe", "w"), // stderr
);
And, then, execute the test.sh command, using those descriptors, in the current directory, and saying the i/o should be from/to $pipes :
$process = proc_open('./test.sh', $descriptorspec, $pipes, dirname(__FILE__), null);
We can now read from the two output pipes :
$stdout = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$stderr = stream_get_contents($pipes[2]);
fclose($pipes[2]);
And, if we output the content of those two variables :
echo "stdout : \n";
var_dump($stdout);
echo "stderr :\n";
var_dump($stderr);
We get the following output when executing the temp.php script :
$ php ./temp.php
stdout :
string(40) "this is on stdout
this is on stdout too
"
stderr :
string(40) "this is on stderr
this is on stderr too
"
A little function that might be helpful:
function my_shell_exec($cmd, &$stdout=null, &$stderr=null) {
$proc = proc_open($cmd,[
1 => ['pipe','w'],
2 => ['pipe','w'],
],$pipes);
$stdout = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$stderr = stream_get_contents($pipes[2]);
fclose($pipes[2]);
return proc_close($proc);
}
The exit code is returned and STDOUT and STDERR are reference params if you need them.
The short way to do such a things with exec is to return the exit code ( status of the command )
Note that I am trying to list a non-exists directory /non-dir/
exec('ls /non-dir/', $out, $retval);
var_dump($retval);
Output
ls: cannot access '/non-dir/': No such file or directory
int(2)
Normally in unix-based system most of successful statuses codes is ( 0 ) so you can check your $retval to know the status of the command.
to dismiss the error from listing an invalid path ls: cannot access '/non-dir/': No such file or directory you can redirect your stderr to null
exec('ls /non-dir/ 2>/dev/null', $out, $retval);
var_dump($retval);
this will output :
int(2)
also if you need the error string to use it in any scenario you may redirect your stderr to the stdout.
exec('ls /non-dir/ 2>&1', $out, $retval);
print_r($out);
var_dump($retval);
this will output the following:
Array
(
[0] => ls: cannot access '/non-dir/': No such file or directory
)
int(2)
Another way to get unmerged stdout/stderr.
$pp_name = "/tmp/pp_test";
#unlink($pp_name);
posix_mkfifo($pp_name, 0777);
$pp = fopen($pp_name, "r+");
stream_set_blocking($pp, FALSE);
exec("wget -O - http://www.youtube.com 2>$pp_name", $r_stdout);
$r_stderr = stream_get_contents($pp);
var_dump($r_stderr);
fclose($pp);
unlink($pp_name);
If you want to ignore stdout and get only stderr, you can try this:
exec("wget -O - http://www.youtube.com 2>&1 >/dev/null", $r_stderr);
exec("{$command} 2>&1"
,$output
,$exitCode
);
2>&1 redirects stderr to stdout for consistent success / fail behaviour.
$exitCode determines $command completion status.
$output contains all output associated with $exitCode.
Slightly ugly but good enough. Put the stderr into a temp file and read it back.
$tmp = tempnam("/tmp", "ERR_");
exec('report 2> ' . escapeshellarg($tmp), $stdout, $retcode);
$stderr = file_get_contents($tmp);
unlink($tmp);
if ($retcode == 0)
{
// good
$url = $stdout[0];
} else {
// bad
error_log("FAIL: $stderr");
}

Categories