I am trying to program a Cron Job that fetches some data from multiple websites. The script uses phantomjs that runs a js file responsible for saving each website's content to a file locally.
My cron job command:
/usr/local/bin/php /home/user/public_html/fetch.php >> /home/user/public_html/fetch-log.txt
fetch.php
ini_set('max_execution_time', 300);
require_once 'Utils.php';
$sources = Utils::get_sources();
foreach ($sources as $source) {
$cmd = "/home/user/bin/phantomjs get-website.js '$source->url' $source->id";
exec($cmd);
}
echo date("Y-m-d H:i:s") . ">> Done fetching sources\n";
get-website.js
var system = require('system');
var webPage = require('webpage');
var fs = require('fs');
var page = webPage.create();
var url = system.args[1];
var file = system.args[2];
var requestsArray = [];
page.onResourceRequested = function (requestData, networkRequest) {
requestsArray.push(requestData.id);
};
page.onResourceReceived = function (response) {
var index = requestsArray.indexOf(response.id);
requestsArray.splice(index, 1);
};
page.open(url, function (status) {
var interval = setInterval(function () {
if (requestsArray.length === 0) {
clearInterval(interval);
try {
fs.write("/home/user/public_html/rss_contents/" + file + ".txt", page.content, 'w');
} catch (e) {
console.log(e);
}
phantom.exit();
}
}, 500);
});
The script works normally when running from browser, and the cron job also works normally when run with a simple echo("done"); php script, but when I try to run the cron job with the fetch.php script, the last line is echoed to the file, but nothing has happen.
I am using shared hosting.
EDIT:
When trying to run phantomjs from ssh console directly, I get this error
2019-06-27T11:34:12 [WARNING] QThread::start: Thread creation error:
Resource temporarily unavailable
And the console just hangs without any response.
I couldn't find any posts talking about this warning on Google.
Related
I have a div section. I want to reload this section every 5 seconds. How do I do this. Here is my code:
<script>
$("#send_parent_general_chat").submit(function()
{
var rec = $("#data").val();
var msg = $("#msg").val();
var dataString = 'rec='+ rec + '&msg='+ msg;
$.ajax({
type: "POST",
url: "<?php echo base_url(); ?>" + "Client/send_general_parent_chat_msg/<?php echo $per_job->id;?>",
data: dataString,
cache: false,
success: function(result){
$('#display_general_msg').html(result);
$('#send_parent_general_chat')[0].reset(); //form reset
}
});
return false;
});
</script>
<script>
$(document).ready(function(){
setInterval(function(){
// alert("===111==");
$("#display_general_msg").load('<?php echo base_url(); ?>" + "Client/refresh_general_parent_chat_msg/<?php echo $per_job->id;?>')
}, 5000);
});
</script>
I have created one more controller for refreshing the div I have used the time interval function but it is not loading, it shows this error:
Access forbidden!
You don't have permission to access the requested object. It is either read-protected or not readable by the server.
If you think this is a server error, please contact the webmaster.
Error 403
I need to refresh only the div content not the whole page.
How do I achieve this?
You can Use :
setTimeout(function()
{
Your_Function(); //this will send request again and again;
}, 5000);
Replace Your_Function with your Function Name.
Hope this will help !!
Below is an example which will update the contents in every 5 seconds using php websockets. This is a simple example, but you can use it to modify to fit for your application needs. You don't need the timeout functions on the client side here we use server sleep
Install the Workerman socket library
composer require workerman/workerman
The client side code
<!DOCTYPE HTML>
<html>
<head>
<script type = "text/javascript">
function WebSocketTest() {
if ("WebSocket" in window) {
//alert("WebSocket is supported by your Browser!");
// Let us open a web socket
var ws = new WebSocket("ws://localhost:2346");
ws.onopen = function() {
// Web Socket is connected, send data using send()
ws.send("Message to send");
//alert("Message is sent...");
};
ws.onmessage = function (evt) {
var received_msg = evt.data;
//alert("Message is received..." + received_msg);
document.getElementById("demo").innerHTML = "Timestamp is updated every 5 sec " +received_msg;
};
ws.onclose = function() {
// websocket is closed.
alert("Connection is closed...");
};
} else {
// The browser doesn't support WebSocket
alert("WebSocket NOT supported by your Browser!");
}
}
</script>
</head>
<body>
<div id = "sse">
Run WebSocket
</div>
<div id="demo" style="font-size: 64px; color: red;"></div>
</body>
</html>
The Server side code
<?php
require_once __DIR__ . '/vendor/autoload.php';
use Workerman\Worker;
// Create a Websocket server
$ws_worker = new Worker("websocket://0.0.0.0:2346");
// 4 processes
$ws_worker->count = 4;
// Emitted when new connection come
$ws_worker->onConnect = function($connection)
{
echo "New connection\n";
};
// Emitted when data received
$ws_worker->onMessage = function($connection, $data)
{
// Send hello $data
while(true) {
$connection->send(time());
sleep(5); //Sleep for 5 seconds to send another message.
}
};
// Emitted when connection closed
$ws_worker->onClose = function($connection)
{
echo "Connection closed\n";
};
// Run worker
Worker::runAll();
The backend service can be started with the following command from the terminal or you can autostart on boot if you want.
$php index.php start
Here index.php is our backendnd file name.
Just start the service and load the page then you can see the timestamp is updated every 5 seconds which comes from the server side. This is a working example tested on my local machine. Try and let me know if you need any other help.
The output
you can also try below one:
setInterval(function(){
loadlink() // this will run after every 5 seconds
}, 5000);
setInterval approach will be more accurate than the setTimeout approach
// or
$(function(){ // document.ready function...
setTimeout(function(){
$('form').submit();
},5000);
});
I was try to create a web screenshot with phantomjs,But alwaise it result a blank image.I am using Fedora OS.Here is my javascript.
screen.js
var WebPage = require('webpage');
page = WebPage.create();
page.open('http://google.com');
page.onLoadFinished = function() {
page.render('googleScreenShot' + '.png');
phantom.exit();}
And here is my php code
shot.php
$exec = '/var/www/html/test/bin/phantomjs /var/www/html/test/screen.js';
$escaped_command = escapeshellcmd($exec);
exec($escaped_command);
But the created image was blank.phantomjs have executre permission.Please help me
I am not sure but this may work,try it
var WebPage = require('webpage');
page = WebPage.create();
page.open('http://google.com');
page.onLoadFinished = function() {
window.setTimeout(function () {
page.render('googleScreenShot' + '.png');
phantom.exit();
}, 2000);
}
You have a problem because the time your code execute the next line of code is more faster than phantomjs logging that page.onLoadStarted is True.
easy and clean way to fix this is by adding a time interval of 100 ms between each function and make all your code in functions, check Vijay Boyapati method for more clarification : https://stackoverflow.com/a/9256079/1738230
In dealing with this problem I completed a code module, as follows:
link
You can download it locally, and then invoke it by command
phantomjs rasterize.js "http://www.google.com" 800px*800px > wait.html
Okay so i have a casperjs script which does a specific task for me then echoes a result back to me.
i normally use the command
Casperjs script.js "variable1" "variable2"
after a few moments it will echo a results saying either
True
or
False
how can i use php on my server instead of using the command line
i would like the php script to execute the command above and then echo the result back to me on my php page
You can use PHP's shell_exec which uses php to call your command, like so:
.php
$casperjs = "casperjs";
$script = "script.js";
$arg0 = $variable1;
$arg1 = $variable2;
$command = "$casperjs $script $arg0 $arg1";
$result = shell_exec($command);
echo $result;
script.js
var casper = require('casper').create();
var arg0 = casper.cli.get(0);
var arg1 = casper.cli.get(1);
casper.start('http://example.com/', function() {
if (this.getTitle() == 'blahblah') {
this.echo('True');
} else {
this.echo('False');
}
});
casper.run();
I'm using jQuery + PHP on my website and I want to do something like this:
To simplify things I've got a page with 2 buttons () and according to which I click I want to start a script in background (php script) so I do a simple thing with jquery:
$("#button1").click(function(){
$.post("script.php", {val:"but1"}, function(result){
alert(result); // I want something with the result
});
});
$("#button2").click(function(){
$.post("script.php", {val:"but2"}, function(result){
alert(result);
});
});
and in the script.php I've got a simple if statement deciding what I should do.
In the php script I'm downloading a file and I want to create a progress bar of the file download. The php script would return me values (echo percentage; flush();) in some time interval.
And here comes the problem - when I echo those percentage values and flush it refreshes it "just in php" but the jquery waits until the script is finished anyway. Is there a way to get those values as they appear (after flush) or is there a completely other approach to this? I can't think of anything else right now, maybe I shouldn't be using jquery at all.
Thanks for the help.
You can store the progress in a text file or DB while running the script and then have another file get the result via AJAX calls.
JS/HTML
<script>
$(function() {
var id;
var timeoutLength = 1000;
$("#button1").click(function() {
$("#result").html("");
// Create unique identifier
id = (new Date().getTime()) + "-" + Math.floor(Math.random()*100);
$.get("script.php", {id: id}, function(result){
$("#result").html(result);
});
setTimeout(checkProgress, timeoutLength);
});
function checkProgress() {
$.get("script.php", {progress: true, id: id}, function(data) {
prog = parseInt(data);
// Display progress bar
if(prog < 100) {
$("#progress").css({ display: 'block', width: prog });
$("#progress").html(prog + "%");
setTimeout(checkProgress, timeoutLength);
} else {
$("#progress").css('display', 'none');
}
});
}
})
</script>
<button id="button1">Click</button>
<div id="progress" style="background: green"></div>
<div id="result"></div>
script.php
<?php
function getProgress($file) {
return (file_exists($file)) ? file_get_contents($file) : 0;
}
function setProgress($file, $progress) {
file_put_contents($file, $progress);
}
// Remove invalid characters
$id = str_replace('/[^a-zA-Z0-9\-_]/', '', $_GET['id']);
$file = "progress-" . $id . ".txt";
if(isset($_GET['progress'])) {
echo getProgress($file);
} else {
// Do something and update progress
for($i = 10; $i <= 100; $i += 10) {
// Do something
setProgress($file, $i);
sleep(1);
}
unlink($file);
echo "Result: " . rand(1, 100);
}
Edit:
Added support for multiple requests and simple progress bar.
i believe that what all you need is the pecl uploadprogress package
it is not very easy to install and its not sure it works, i have find hard to make it work under certain server configurations, but i think its a good shot
EDIT: sorry i didnt see you mention download so have a look here
Below script will work for the progress-bar.
But, what I would do is, trigger an AJAX call to start the download and trigger another AJAX call to start receiving the input (using How to show file download progress in PHP Shell Scripting?).
PHP:
<?php
echo "wget '$feedURL'\n";
$execute = "wget -O ".$filePath." '$feedURL'\n";
$systemOutput = shell_exec($execute);
$systemOutput = str_replace( "\n", "\n\t", $systemOutput);
echo "\t$systemOutput\n";
?>
GetProgress:
function getProgress(){
GDownloadUrl("getprogress.php?progress_key=<?php echo($unique_id)?>",
function(percent, responseCode) {
document.getElementById("progressinner").style.width = percent+"%";
if (percent < 100){
setTimeout("getProgress()", 100);
}
});
}
Here is nice implementation with the help PHP's APC to get a nice progress-bar during UPLOAD of a file - http://www.haughin.com/2007/10/23/php-upload-progress-with-php-52-apc/
Alternative solution:
You can store the progress of downloading file into $_SESSION array inside PHP.
Besides that, you can write a dedicated PHP to only retrieve this percentage from session. This PHP will be called from your client's AJAX. This way the proccess won't take so much resources from server to calculate or to echo the download's percentage.
Try to avoid I/O from files if you are going to handle high rates of reading/writing onto them. Session variables are less expensive to achieve this functionality.
Provided that your PHP has some way to know the progress percentage, you should do multiple requests for progress updates:
function receiveProgress(data) {
// update progres indicator here
if (parseFloat(data) < 100) {
setTimeout(checkProgress, 100);
} else {
pressedButton = '';
}
}
function checkProgress() {
$.post("script.php", {val:pressedButton}, receiveProgress);
}
var pressedButton = '';
$("#button1").click(function(){
pressedButton = 'but1';
checkProgress();
});
This will ask server for progress updates every 100ms
I am running a shell script in the background and redirecting the output to a log file in php. I want to display the contents from the log file on the page. I am able to do that using the code below.
<?php
$logfile = "hello";
?>
function displayOutput()
{
var html = <?php
echo filesize($logfile)
? json_encode(file_get_contents($logfile))
: '"Log file is getting generated"';
?>;
document.form.text1.value = html;
}
However, the log file keeps updating till the script completes executing. How can i reload the updated contents from the file on the same page?
The technique that I developed + discuss here may be useful:
https://web.archive.org/web/20150206001444/http://commavee.com/2007/04/13/ajax-logfile-tailer-viewer/
It's been around for a while + works well.
You need to set an interval timer to call your function every n seconds. Have a look at this answer to help you out - how to schedule ajax calls every N seconds?.
setInterval(displayOutput, (10 * 1000));
// reload log contents every 10 seconds
Maybe what you want basic XMLHttpRequest usage.
I am not really php guy, neiter javascript guru, just trying to give you an idea
function refreshText()
{
if (window.XMLHttpRequest)
{
xhttp = new XMLHttpRequest();
}
else // for IE 5/6, just in case
{
xhttp = new ActiveXObject("Microsoft.XMLHTTP");
}
xhttp.open("GET","/page.php?action=download_log_file", false);
xhttp.send();
document.form.text1.value = xhttp.responseXML;
}
setInterval(refreshText, (10 * 1000)); // refresh text1.value every 10 seconds
same thing using jQuery
setInterval(function {
$.get('/page.php?action=download_log_file', function(data) {
$('#text1').val(data);
});
}, (10 * 1000));
The handler script at the server just prints file data, see
http://www.w3schools.com/xml/xml_server.asp for example
I've implemented "COMET-like" functionality recently to do just this. The way it works is to have an AJAX poll with a long timeout:
var lines = 0
function getLog(file, lines) {
$.ajax({
type: 'POST',
url: 'http://thissite.com/getLogFile.php?File=' + file + '&Lines=' + lines,
dataType: 'json',
timeout: 400000,
error:
function() {
return false;
},
success:
function(data) {
if (data.Error) {
alert(data.Message)
} else {
if (data.Lines > lines) {
// do something with data.LogLines, e.g. add to a textarea
}
getLogFile(file, data.Lines)
}
}
})
}
The back end script then simply loops like this:
Count the number of lines in the log file
If it's the same as lines, sleep (say for a second), then go back to 1
If the number of lines is greater, return the new lines, and the new line count, and exit
After some number of iterations (I use 100), exit and return the existing line count
The data structure returned by the back end script is JSON:
{
Error: // 0 or 1,
Lines: // Number of lines
Text: // New lines from log file
}
This works just like 'tail -f' in UNIX, but in a browser!