I have 2 servers serv1 and serv2 and need to compare the images in those 2 servers to detect which files are missing or has been modified.
So far I have 3 options:
- Create an API using PHP
I created an API file that will return all the images in serv1/www/app/images/
get the modification time of each images
return the result as json
output is something like this: { 'path/to/file' : 123232433422 }
I fetch that in serv2, decode then merge the array to the images in serv2/www/app/images
get the array_diff, works fine
cons:
- takes a lot of time (fetching, decoding, merging, looping, comparison... )
- Use rsync
Dry run to get the list of images that is existing in serv1 but is missing or modified in serv2 (very fast :))
cons:
apache can't run ssh because it's not authorized to access ~/.ssh/
would need to give apache permission but my client doesn't want it
so in short, i cannot use anything that would require permission
- maybe I could use some library or vendor but I doubt my client would allow me. If it can be shell script or a php built in function, I'll do it as long as it's possible.
So my question is if there is another way to fetch the images and modification date of those images without requiring authentication? My first solution is okay if it can be optimized cause if the array is too large, it takes a lot of time.
I hope the solution can be done in PHP, or Shell script.
Please help give me more options. Thanks
Install utility md5deep (or sha1deep) on both servers.
Execute md5deep on first server and save result to text file:
user#server1> md5deep -l -r mydir > server1.txt
Result file would look like this:
e7c3fcf5ad7583012379ec49e9a47b28 .\a\file1.php
2ef76c2ecaefba21b395c6b0c6af7314 .\b\file2.txt
45e19bb4b38d529d6310946966f4df12 .\c\file3.bin
...
Then, copy file server1.txt to second server and run md5deep in negative matching mode:
md5deep -l -r -X server1.txt mydir
This will print checksums and names of all files on second server which are different from first server.
Alternatively, you can compare text files created by md5deep -l -r dir yourself using diff or similar utility.
Last note - it may be easier to simply run md5deep -l -r mydir | gzip > md5deep.txt.gz in cron on each server, such that you have ready to compare filelist with checksums on each server (gzipped so it is fast to fetch).
Related
Opencart generates its sitemap on the fly and this is a problem in a big catalogs over 10.000 products. So I have modified the function to generate a static sitemap in an XML file.
When I access to my http://localhost/index.php?route=extension/feed/google_sitemap I generate a sitemap-google.xml file without problems and with a unlimited execution time.
I tried to add it in a cron in the development server each 4 hours
0 0,4,8,12,16,20 /usr/bin/php /path/to/index.php?route=extension/feed/google_sitemap
But I'm receiving a "command not found".
Can I execute on cli the "?params/etc"?
You cannot do that, as the URL parameters are only evaluated this way when calling the script over a server. But a quick solution could be to use wget: keep a copy of that sitemap script anywhere under some kind of "secret URL", call it using wget and put the result on your disk.
If you cannot use wget, you could use a PHP script containing file_get_contents. In the same way, it could request the data over a HTTP request and save it in the cached sitemap file.
As a note: if you know which logic should be present to generate that sitemap, you could also write all that logic directly to a PHP script. Running it from shell helps to avoid a blocked server thread, but might be more work
https://stackoverflow.com/a/62145786/4843247
You can use following command to generate sitemap.xml:
cd /PATH_TO_YOUR_WWW_ROOT && QUERY_STRING="route=extension/feed/google_sitemap" php -r 'parse_str($_SERVER["QUERY_STRING"],$_GET);include "index.php";' >sitemap.xml.new && mv sitemap.xml.new sitemap.xml
I have an unusual problem. Using PHP script, I have to traverse through a folder with around 1 million small text files (size ranges from 1KB to 1MB), and pick only those with ctime in certain interval AND content containing particular search string.
First part (picking files that have time of creation in certain range) I managed using readdir but checking the file content for search string proves to be a challenge. Using file_get_contents (and then stripos) simply won't do. It's slow, it brings my PHP script to its knees.
I'm sure I'm not the first one with this kind of a problem, but I'm not a PHP developer. This code has been inherited from previous dev. I'm not sure which alternative I should be using and what code would spare my server RAM and CPU.
I would try shell_exec combined with find and grep:
$output = shell_exec("find . -type f -ctime $MyCtime -exec grep -H -m 1 $MySearchString {} +;");
-H to show filename
-m 1 to stop searching at first ocurrence in file
PHP won't handle it easily(it will take lots of time + will overload the CPU), consider to use bash and regular expressions to solve the problem
Simply saying, PHP is not the right tool in the situation
I have a command like this one:
php myscript.php |
curl -X POST --data-binary #- "http://localhost/webservice.php/$$/output/"
I use | to send the output of myscript.php to the curl command. I use $$ to get the PID, but this returns the PID of the curl command, not the php command.
In fact, I will use this command line for all my crontab's jobs. Instead of receiving thousands mails, I would like to send every information to my webservice.
So in myscript.php, I call (1 is cronjob ID) :
file_get_contents('http://localhost/webservice.php/1/' . getmypid() . '/start/');
...
file_get_contents('http://localhost/webservice.php/1/' . getmypid() . '/end/');
I try to send the output of myscript.php to a webservice I'm creating.
So I send start / end commands to my webservice. There's only one thing I miss : the output of the PHP if there are warnings or some "echo".
The goal is to redirect the output of the script (which could be a bash script also) to the webservice, using the PID of the PHP command (which I use to identify the process in my webservice).
How can I get the PID of PHP command ? I tried $$ as I found in Stackoverflow, but as you answer, it seems to be the current shell PID...
Thanks by advance !
TL;DR
The $$ variable is a special variable. It returns the PID of the invoking shell, which may not be the same as the current shell's PID as stored in $BASHPID. Neither variable has anything to do with PIDs from processes in your pipeline. The Bash manual says:
($$) Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the invoking shell, not the subshell.
You need to refactor your PHP script and your pipeline to do something different.
Create Unique Directories Properly
You haven't explained what your PHP script is actually doing, so this seems like an X/Y problem. In general, you should be using mktemp or its PHP equivalents:
PHP's tempnam
PHP's tmpfile
PEAR's System::mktemp
to create temporary or unique filenames. Since you seem to want to use unique directories, I give that as an example below. However, it might be cleaner to have a single directory containing filenames that are guaranteed to be unique. Your mileage may vary.
Rather than trying to use a PID, you should be using the mktemp utility to create unique directories or filenames without race conditions. If you do this in Bash, you can export the variable to the environment and then use the same directory inside your PHP script. For example:
#!/usr/bin/env bash
# Create a unique temp directory safely.
export DIR=$(mktemp -d)
# Create your output directory for whatever reason.
mkdir "$DIR/output"
# Ensure PHP uses $_ENV["DIR"] to do whatever it's doing.
php myscript.php | curl -X POST --data-binary #- "file://$DIR/output/"
# Clean up the temp directory at some point.
echo "I did something with '$DIR' so clean it up later." > /dev/stderr
Just Use Unique Files with Prefixes
Rather than using directories, you might also consider using unique filenames with a defined prefix. For example:
# Create files like "/tmp/myscript.D6147vts"
export TEMPFILE=$(mktemp -t myscript)
Then, like before, use $_ENV["TEMPFILE"] in your PHP script. The benefit of this approach is that your filenames are guaranteed to be unique but they're all in one place and have a common prefix. That can make cleanup or finding the right file much easier later on.
If you really need unique directories, use them. However, if you don't really need the directory structure, don't create extra work for yourself: just use unique filenames.
Try
ps aux | grep "process id" and the pid if the second column. Parse it with pnp_exec; meaning, store it in an array and extract PID number.
I run Bash-Scripts from PHP (5.4) with root permissions through a binary wrapper (see [Execute root commands via PHP), which works perfectly fine except for the following example. Additionally, I am using zfs-on-linux on CentOS7.
I prepared 2 simple example Bash-Scripts:
test_zfsadd:
#!/bin/bash
#ARGS=1
err="$(zfs create $1 2>&1 > /dev/null)"
if [ $? -ne 0 ]
then
echo $err
exit 1
fi
echo "OK"
exit 0
test_zfspart:
#!/bin/bash
#ARGS=1
msg="$(zfs get mounted $1 -H | awk '{print $3}')"
echo $msg
exit 0
When I call the according binaries from PHP with e. g.
<?php
$partition = 'raid1/testpart';
$ret = shell_exec("/path/test_zfsadd_bin $partition");
echo "<p>Return value: $ret</p>\n";
$ret = shell_exec("/path/test_zfspart_bin $partition");
echo "<p>Is mounted: $ret</p>\n";
the output is:
Return value: OK
Is mounted: yes
This looks good, but when I call 'test_zfspart_bin raid1/testpart' directly from console, I get the correct result which is
no
(means that the partition is NOT mounted, checked in /proc/mounts). So I get 2 different answers from the same script depending somehow on the context. I first thought it has something to do with the SUID-Bit, but calling the script in console with an unprivileged user works fine. If I try (as root)
zfs mount raid1/testpart
in console I get
filesystem 'raid1/testpart' is already mounted
cannot mount 'raid1/testpart': mountpoint or dataset is busy
which is weird. I also can't destroy the 'partition' from console, this works only from PHP. On the other hand, if I create a partition as root directly from bash and try to delete it via PHP, it doesn't work either. Looks like the partitions are somehow separated from each other by context. Everything gets synchronized again if I do
systemctl restart httpd
I think apache or PHP is keeping the zfs system busy to some extend, but I have absolutely no clue why and how. Any explanation or some workaround is much appreciated.
I figured it out myself in the meantime.. The problem was not the apache process itself, it was how it is started by systemd. There is an option called 'PrivateTmp', which is set to 'true' in the httpd service file by default (at least in CentOS7). The man page says
PrivateTmp=
Takes a boolean argument. If true sets up a new file system
namespace for the executed processes and mounts a private /tmp
directory inside it, that is not shared by processes outside of
the namespace. This is useful to secure access to temporary files
of the process, but makes sharing between processes via /tmp
impossible. Defaults to false.
This explains it all I think. The newly created zfs partition is mounted in this 'virtual' file system and is therefore invisible to the rest of the system, what is not desired in this case. The apache process is not able to mount or unmount file systems outside its namespace. After disabling the option everything worked as expected.
I need a tool to join 2 mp3 files in 1.
The first file ( background ) will be some sound ( music, submitted by user ),
second file will be a machine (google) speaking text, which was submitted ( by user ).
In output I need a single mp3 file, with background music and speaking robot text playing together.
I can't really find any PHP standalone solution like some library or something like, only shell commands atc.
So are there some libraries?
or there's some unique shell command which works on all OS to combine files?
Or how do I complete the task?
Based off of this question, you should be able to install FFMPEG onto your server (hopefully not a shared host?) and use
//Reduce background volume, assuming it's input2.mp3
shell_exec('ffmpeg -i input2.mp3 -af "volume=0.3" backround.wav');
//Merge the two
shell_exec('ffmpeg -y -i input1.mp3 -i background.wav -filter_complex amerge -c:a libmp3lame -q:a 4 output.mp3');
Then you can simply serve up the output.mp3 file. If this is being performed by PHP running under apache (or other web host, instead of CLI) you'll need to make sure your www-data user has read access to the input files, and write access to the output directory.
All your temporary output, like the background, should be saved as .wav, and only converted back to .mp3 for the final output. Recompressing the audio at each step may result in a very poor final output.
I am making assumptions about how FFMPEG works here, so you may need to consult the documentation for it in order to build a functioning or more efficient set of commands.
You can simply do this
file_put_contents('combined.mp3',
file_get_contents('file1.mp3') .
file_get_contents('file2.mp3'));