I have run the examples scripts from phirehose through terminal and seen the live/active ptint_r of my test tweets. I've loaded up the ghetto queue files and they execute their logging with resounding success. But I can't seem to find where the data went. What file is it actually writing to? The two example files ghetto-queue-collect.php and ghetto-queue-consume.php lead me to believe it is the /tmp directory relative to where the scripts are executing but I see nothing. Any suggestions?
Here are some of the lines from the logs
[03-Mar-2014 02:44:43 America/New_York] Phirehose: Opening new active status stream: /tmp/.phirehose-ghettoqueue.current
[03-Mar-2014 02:45:12 America/New_York] Phirehose: Successfully rotated active stream to queue file: /tmp/phirehose-ghettoqueue.20140303-024512.queue
----- and ------
[03-Mar-2014 03:41:58 America/New_York] Processing file: /tmp/phirehose-ghettoqueue.20140303-024102.queue
[03-Mar-2014 03:41:59 America/New_York] Successfully processed 1 tweets from /tmp/phirehose-ghettoqueue.20140303-024102.queue - deleting.
-- The bits of code in question, I think --
/**
* Subclass specific constants
*/
const QUEUE_FILE_PREFIX = 'phirehose-ghettoqueue';
const QUEUE_FILE_ACTIVE = '.phirehose-ghettoqueue.current';
public function __construct($username, $password, $queueDir = '/tmp', $rotateInterval = 10)
// Set subclass parameters
$this->queueDir = $queueDir;
// Construct stream file name, log and open
$this->streamFile = $this->queueDir . '/' . self::QUEUE_FILE_ACTIVE;
$this->log('Opening new active status stream: ' . $this->streamFile);
$this->statusStream = fopen($this->streamFile, 'a'); // Append if present (crash recovery)
My post to the author's github revealed that the files are written to the actual /tmp directory relative to your hosting or webserver? I'm not sure exactly but, in addition, the files are hidden through Unix because of a precursory '.'
The data is literally being written to /tmp - if you're on a mac or linux machine, you can see these files by opening terminal and running:
ls -la /tmp/
Alternately, you can set $queueDir to whatever you want in the script.
I was able to find them by using putty.
Related
I have an application written in PHP 8. I've added some Unit Tests for it using PHPUnit.
In one of the tests I am using PHP's copy function to move a file from one location to another. This is done to test an endpoint which downloads the file by moving a "dummy" file to the "real" location that the file would be in, in the production application.
My test looks like this:
// tests/TestCase/Controller/DocsControllerTest.php
public function testDownload()
{
$testFile = '75e57e4a-2149-4270-9d76-c7c8f0298c2c.pdf';
copy('/full/path/to/testFiles/' . $testFile, '/webroot/docs/');
// Download the file from the endpoint
$id = 9; // File ID to download
$this->get('/download/' . $id);
// This should return a HTTP 200 response containing the PDF
$this->assertResponseCode(200, 'Downloading a valid PDF should produce a 200 response.');
}
To explain the function above:
We have a test file called 75e57e4a-2149-4270-9d76-c7c8f0298c2c.pdf. This is a real PDF file with appropriate encoding.
We move the file, using copy(), from a directory where we hold some test files, into the full directory path where the production web application will really store the files (/webroot/docs/).
The remainder of the logic deals with downloading the file from the endpoint. The $this->get makes a HTTP GET request to an endpoint (/download/) which also passes in the appropriate file ID. The location of the file is looked up from a MySQL database and then is streamed to the browser, thus generating a HTTP 200 response containing the PDF.
This works both when I run phpunit locally by executing vendor/bin/phpunit --filter testDownload:
PHPUnit 9.5.10 by Sebastian Bergmann and contributors.
Time: 00:05.053, Memory: 20.00 MB
OK (1 test, 16 assertions)
It also works in a browser, i.e. if I make a request to /download/9 I am served the appropriate PDF.
The problem I'm having is on GitHub. When I run the unit test there it fails the CI with this error:
Warning Error: copy(/home/runner/work/my-app/webroot/docs/75e57e4a-2149-4270-9d76-c7c8f0298c2c.pdf): Failed to open stream: No such file or directory
In [/home/runner/work/my-app/tests/TestCase/Controller/DocsControllerTest.php, line 745]
Given that this works locally I can't understand why this error is occurring. Is there some restriction with using copy() in GitHub's CI?
The directory and files at /full/path/to/testFiles/ are not .gitignore'd so they are committed with the rest of the repo code. So the test file, 75e57e4a-2149-4270-9d76-c7c8f0298c2c.pdf, exists within the codebase on GitHub.
I am using PHPUnit 9.5.10, PHP 8.0 on a Mac running macOS Monterey (12.2).
It clearly reads:
Failed to open stream: No such file or directory
Maybe add .gitkeep into target directory docs? It may also be, that the source file is not there. One usually can take this error message literal. While it's entirely unclear what $this->get() even is or why your Mac would have anything to do with running GitHub Action?
Objective: Use PHP to call a vbs that converts an xls/xlsx file to a csv.
Question: How can I pass a source file path and a destination file path to a vbs that converts xls/xlsx to csv and run that vbs in a PHP web application?
Details: I have a working vbs that takes a source file path and a destination file path and converts the xls/xlsx at source file path into a csv. I can execute it from the Windows cmd line and it does exactly what I want it to do. I can also put the execution command into a bat file and run the bat file to achieve the same results. However, when I use exec()/shell_exec()/system() in PHP to execute the same command no csv is created. (If I try to run the bat from PHP using system() the contents of the bat file show up on the page, in fact, echo Conversion complete! prints "echo Conversion complete! Conversion complete.") I haven't seen any errors yet.
Note: I know about PHPExcel, I'd prefer not to use it.
excelToCsv.vbs
On Error Resume Next
if WScript.Arguments.Count < 2 Then WScript.Echo "Please specify the source and the destination files. Usage: ExcelToCsv <xls/xlsx source file> <csv destination file>"
Wscript.Quit
End If
csv_format = 6
Set objFSO = CreateObject("Scripting.FileSystemObject")
src_file = objFSO.GetAbsolutePathName(Wscript.Arguments.Item(0))
dest_file = objFSO.GetAbsolutePathName(WScript.Arguments.Item(1))
Dim oExcel
Set oExcel = CreateObject("Excel.Application")
Dim oBook
Set oBook = oExcel.Workbooks.Open(src_file)
oBook.SaveAs dest_file, csv_format
oBook.Close False
oExcel.Quit
batConverter.bat
excelToCsv.vbs conversionTestSourceMS2003.xls batTest.csv
echo Conversion Complete!
index.phtml
<?php
system("cmd /c batConvert.bat")
?>
Note: All of the above files (along with conversionTestSourceMS2003.xls) are in the same directory. I have not implemented any way to pass the parameters (since I can't get it to work even if it's all hard coded...)
Set Up: PHP5, Zend Framework, WAMP, Windows 7 (localhost).
For the sake of simplicity, I merged everything into a single ASP page. This will allow me to hopefully see a similar problem in IIS, and since it is in a single ASP script, I will be able to see the error more directly. My test machine is running on Windows Vista SP2 on IIS7 with Excel 2007 SP3.
excelToCsv.asp
<%
Option Explicit
Dim csv_format, src_file, dest_file, strPath, objFSO
csv_format = 6
src_file = "conversionTestSourceMS2003.xls"
dest_file = "testbat.csv"
strPath = "[HARDCODED PATH HERE]\"
src_file = strPath & src_file
dest_file = strPath & dest_file
Dim objExcel, objBook
Set objExcel = CreateObject("Excel.Application")
Set objBook = objExcel.Workbooks.Open(src_file)
objBook.SaveAs dest_file, csv_format
objBook.Close False
Response.Write "Conversion Complete!"
objExcel.Quit
%>
When running this code, I got a generic ASP error. So, I enabled detailed error messages in ASP and I get this following error...
Microsoft Office Excel error '800a03ec'
Microsoft Office Excel cannot access the file '[HARDCODED PATH
HERE]\conversionTestSourceMS2003.xls'. There are several possible
reasons: • The file name or path does not exist. • The file is being
used by another program. • The workbook you are trying to save has the
same name as a currently open workbook.
/temp/ExcelToCsv.asp, line 18
Now, this is not Apache, but I do believe the problem is related to yours. This error implies there is a security/permission problem where Excel cannot do what it needs to do to access or read the file. In fact, I encountered similar errors when I was executing the VBScript (and passing the error up the chain) from PHP (in IIS).
I believe it can be resolved by changing the Windows User being used to create the process. This can be configured in Services.msc by editing the Apache service and changing the Log On tab to an actual Windows user instead of a Service Account. I have not tested it yet, though, since setting up Apache is not something I can do right now.
I am trying to send data from an Android app to a PHP file on my server (school server) but I'm running into some problems. I have my own hosting space through SimpleHelix, and I was able to send the data just fine, but when I try to use my school's server, the PHP program returns the following error:
Notice: Undefined index: message in /home/alespurg/test_good.php on line 4
Warning: file_put_contents(androidmessages.html) [function.file-put-contents]: failed to open stream: Permission denied in /home/alespurg/test_good.php on line 9
Warning: file_get_contents(androidmessages.html) [function.file-get-contents]: failed to open stream: No such file or directory in /home/alespurg/test_good.php on line 11
All I have changed in my Java file is the URL that I need it to post to. Could there be restrictions on my school's server that prevent the post to go through? I've checked my permissions on the folders and files, they're all 775. Again, I know the program works, I'm just having problems with the server. I did not use the IP address of the domain for either one. I couldn't find it for my school's server.
EDIT:
<?php
// get the "message" variable from the post request
// this is the data coming from the Android app
$message=$_POST["message"];
// specify the file where we will save the contents of the variable message
$filename="androidmessages.html";
// write (append) the data to the file
file_put_contents($filename,$message."<br />",FILE_APPEND);
// load the contents of the file to a variable
$androidmessages=file_get_contents($filename);
// display the contents of the variable (which has the contents of the file)
echo $androidmessages;
?>
If the permissions are 775 that means that if the web server process isn't either (1) the user who owns /home/alespurg/ (not likely) or (2) a member of the group that owns /home/alespurg/ (also not likely) then it won't be able write to the directory.
Are you sure the web server process is the owner or a member of the ownership group for /home/alespurg/?
I'll assume your web server is apache, which usually runs as apache:apache or nobody:nobody. Judging from the directory structure, it's likely that /home/alespurg/ is owned by alespurg:alespurg or some derivation.
ADDENDUM
If you need to do things like this an apache process is not really the appropriate place. You may not know it but you can write php scripts and execute them from the shell environment on your server where you're logged in as the appropriate user to write to that directory. Further, you can have CRON run them for you if necessary. The web server process is an unnecessary middleman.
I've been trying to debug this error for over three hours now, changing filenames, trying to use GeoIP Lite instead of GeoCity (the latter has a 27mb file to be included, so did this thinking fopen() had a max), etc.
Here's my structure file structure: index.php -> include("configuration/config.php") - config.php -> include("inc/geo_text.php") -> geo_text.php
The contents of geo_text.php is:
$ip = $_SERVER['REMOTE_ADDR'];
include("GeoIP/geoip.inc");
$gi = geoip_open("GeoIP/GeoIP.dat",GEOIP_STANDARD);
$count_name = geoip_country_name_by_addr($gi, $ip);
geoip_close($gi);
echo($count_name);
Now, if I access geo_text.php no errors are given, and just to make sure I placed echo($count_name) in geo_text.php and it returned, as it should, my country.
However, when I run config.php it returns the error:
Warning: fopen(GeoIP/GeoIP.dat) [function.fopen]: failed to open stream: No such file or directory in /nfs/c09/h02/mnt/177978/domains/domain.com/html/labs/final/configuration/inc/GeoIP/geoip.inc on line 399
Can not open GeoIP/GeoIP.dat
Has anyone got any ideas why this could be?
SSH into your server and run the following command (assuming it's a Linux server):
cd /nfs/c09/h02/mnt/177978/domains/domain.com/html/labs/final/configuration/inc/GeoIP/
ls -lah
Then paste the output here for us to see. My guess is that that path doesn't exist.
That's very strange. As a test, try moving both geo files into the same directory as your code files and then alter the paths in your code accordingly.
It's checking for the files in domain.com, but you just cd'd into themeplated.com, that's the problem. Your code needs to point to the themeplated.com directory.
/nfs/c09/h02/mnt/177978/domains/domain.com/html/labs/final/configuration/inc/GeoIP/
/nfs/c09/h02/mnt/127878/domains/themeplated.com/html/labs/final/configuration/inc/GeoIP/
It's a path issue.
geoip_open("/absolute/path/to/GeoIP/GeoIP.dat",GEOIP_STANDARD);
should work.
I need to write a script that is run as a cron job every night which transfers some report files via sftp to another server.
The report files are created every night using another cron in the format 'support_[date].csv' & 'download_[date].csv'.
I'm wondering if you had any pointers on how to do the following:
Find the 2 files created on latest [date]
Copy these files to another server using SFTP
I've tried several PHP scripts utilising the ssh2 extension, but to no avail. Is there a way to do it using a shell script? It's not something I am hugely familiar with to be honest (hence going down the PHP route initially)
This was one of my PHP scripts which didn't work:
$src = 'test.csv';
$filename = 'test.csv';
$dest = '/destination_directory_on_server/'.$filename;
$connection = ssh2_connect('example.com', 22);
ssh2_auth_password($connection, 'username', 'password');
// Create SFTP session
$sftp = ssh2_sftp($connection);
$sftpStream = fopen('ssh2.sftp://'.$sftp.$dest, 'w');
try {
if (!$sftpStream) {
throw new Exception("Could not open remote file: $dest<br>");
}
$data_to_send = file_get_contents($src);
if ($data_to_send === false) {
throw new Exception("Could not open local file: $src.<br>");
}
if (fwrite($sftpStream, $data_to_send) === false) {
throw new Exception("Could not send data from file: $src.<br>");
} else {
//Upload was successful, post-upload actions go here...
}
fclose($sftpStream);
} catch (Exception $e) {
//error_log('Exception: ' . $e->getMessage());
echo 'Exception: ' . $e->getMessage();
if($sftpStream) {fclose($sftpStream);}
}
This were the error messages I got:
Warning: fopen() [function.fopen]: URL
file-access is disabled in the server
configuration in
/path_to_script/sftp-test.php on line
17
Warning: fopen(ssh2.sftp://Resource id
3/destination_directory_on_server/test.csv)
[function.fopen]: failed to open
stream: no suitable wrapper could be
found in /path_to_script/sftp-test.php
on line 17 Exception: Could not open
remote file:
/destination_directory_on_server/test.csv
using the terminal to find latest date of your file, you can use ls -1tr . Then use scp (not sftp) to copy/transfer files over
example,
#!/bin/bash
latest_download=$(ls -1tr download*csv | tail -1)
latest_support=$(ls -1tr support*csv | tail -1)
scp $latest_download user#somehost.com:somedir # syntax from memory, check man page for correct syntax
scp $latest_support user#somehost.com:somedir
check the man page of scp for usage
Muchos kudos to ghostdog74! Managed to get this working, but with sftp.
First I managed to set up key authentication, then partly using ghostdog74's script I did this and it worked perfectly!
cd /directorywithfilesin
latest_download=$(ls -1tr download* | tail -1)
latest_support=$(ls -1tr support* | tail -1)
sftp username#example.com <<EOF
cd /dir_to_copy_to
put $latest_download
put $latest_support
EOF
Thanks!
Among other problems with ghostdog74's method is that it's non-portable. My recommendation would be to use phpseclib, a pure PHP SFTP implementation.
This will not work from PHP from your server because your php.ini has disabled remote wrappers
allow_url_fopen boolean
This option enables the URL-aware fopen wrappers that enable accessing URL object like files. Default wrappers are provided for the access of remote files using the ftp or http protocol, some extensions like zlib may register additional wrappers.
Note: This setting can only be set in php.ini due to security reasons.
However, you could simply let your cron job call a shell script that that uses sftp or rsync directly. You don't have to do this with PHP.
I'm voting to move this to ServerFault to get better support for shell scripting.
The answer is right there, in the error message:
Warning: fopen() [function.fopen]: URL file-access is disabled in the server configuration
means that file-access through URL wrappers is disabled in the server configuration.
Check your PHP config, especially allow_url_fopen. PHP documentation says "This setting can only be set in php.ini due to security reasons", so check it there.
See also fopen: "If PHP has decided that filename specifies a registered protocol, and that protocol is registered as a network URL, PHP will check to make sure that allow_url_fopen is enabled. If it is switched off, PHP will emit a warning and the fopen call will fail." As far as I can tell, that's exactly what is happening there.
If you can't or won't enable allow_url_fopen, you still have some options:
call sftp directly
mount a share with sshfs and then use it as a normal folder
Try as follows (Shell)
SFTP=<sftp path>
KEY_FILE=<your key>
USERNAME=<remote username>
SERVER =<remote server>
REMOTE_DIR=<remote location>
APP_HOME =<App location>
FILENAME=<file name>
${SFTP} -o IdentityFile=${KEY_FILE} ${USERNAME}#${SERVER} <<_COMMAND
lcd ${APP_HOME}
cd ${REMOTE_DIR}
put ${FILENAME}
bye
_COMMAND