I am trying to run a script from within a script and have the below which does not seem to work. I am basically getting a load of data from a database, looping through each row and running a script that will generate XML based on that data.
set_time_limit(0);
//connect to database
$msSqlDB = new mySqlConnect('store');
$select = "SELECT * FROM FStores";
$run = mysql_query($select);
while($row = mysql_fetch_array($run)){
exec('/var/www/web/shop_xml/index.php?shopKeeper=$row[SKID]&shop=1');
}
Whats the best possible way to do this? Will the 2nd row wait until the 1st row is successfully executed or can multiple run at the same time? Many thanks for any advice with this.
You need to do something like this:
exec('php /var/www/web/shop_xml/index.php "'.escapeshellarg($row['SKID']).'" "1" > /dev/null 2>&1 &');
Then, in your index.php script:
<?php
$shopKeeper = $argv[1];
$shop = $argv[2];
// ... do stuff
What you have attempted to do is use a HTTP query string in a file system invoke, which will not work. You need to pass the data as command-line arguments, like you would in a terminal. Then, you can get the data from $argv.
You need to start the command with php otherwise the kernel will (most likely, unless you add a hashbang and set permissions) not know how to execute the script, or have permissions to do it.
If you add > /dev/null 2>&1 & the commands will be run asynchronously, i.e. you will not have to wait for the last command to finish before you can invoke another. Be carefull with this though, you could end up with many processes if your query returns many rows.
To avoid this, you could do somthing like:
<?php
// Number of records to process at a time
$perBatch = 5;
set_time_limit(0);
//connect to database
$msSqlDB = new mySqlConnect('Freewebstore');
// Get the number of records in the table
$query = "SELECT count(*) FROM FacebookStores";
$result = mysql_fetch_row(mysql_query($select));
$count = $result[0];
for ($i = 0; $i < $count; $i += $perBatch) {
// Get $perBatch records from the DB
$query = "SELECT * FROM FacebookStores LIMIT $i,$perBatch";
$result = mysql_query($select);
for ($j = 1; $row = mysql_fetch_array($result); $j++) {
// Base command
$command = 'php /var/www/web/shop_xml/index.php "'.escapeshellarg($row['SKID']).'" "1"';
// Run all except the last asynchronously
if ($j < $perBatch) {
$command .= ' > /dev/null 2>&1 &';
}
exec($command);
}
}
This would get $perBatch records from the DB at a time, and process all but the last asynchronously. This would result in them being processed roughly $perBatch records at a time, and help to avoid a large number of processes eating server resources.
As a side note, you seem to be using an odd mix of OO DB code and procedural DB code - you should stick to one or the other to avoid confusion.
The answer by George P answers your question literally; how to run the script as-is from the command line. Let me in turn offer some alternatives.
First of all, your shop_xml/index.php script could check to see if it is being called from command line and read arguments accordingly. Thus:
if( php_sapi_name() == 'cli' ) {
// you would, of course, want to escape these for malicious code!
$shopKeeper = $argv[1];
$shop = $argv[2];
}
else {
$shopKeeper = $_GET['shopKeeper'];
$shop = $_GET['shop'];
}
Then you would call your command as follows:
$arg = escapeshellarg($row['SKID']);
exec( "/var/www/web/shop_xml/index.php \"$arg\" 1" );
However, a far better solution would be to take the code in index.php which is being called from the CLI and move it to an include (like functions.php), placing it inside a function like so:
function processShopRow($showKeeper, $shop) {
// stuff that used to be in index.php goes here
}
Now, in both index.php and in the code run from the CLI, you will include and run the code as follows:
include('functions.php');
set_time_limit(0);
//connect to database
$msSqlDB = new mySqlConnect('Freewebstore');
$select = "SELECT * FROM FacebookStores";
$run = mysql_query($select);
while($row = mysql_fetch_array($run)){
processShopRow($row['SKID'], 1);
}
your script is not working becuase
exec('/var/www/web/shop_xml/index.php?shopKeeper=$row[SKID]&shop=1');
make it
exec("/var/www/web/shop_xml/index.php?shopKeeper=$row[SKID]&shop=1");
single quotes can't have variables inside them.
Have you tried using this:
exec("/var/www/web/shop_xml/index.php?shopKeeper=$row[SKID]&shop=1");
instead of your original? The single quotes ' don't allow you to evaluate embedded variables or php objects in it. You'll have to use the double quotes " --Could you try that out?
In your exec command you need to either run the script using lynx (command line browser) or the PHP CLI. You also should reference the domain, not the full path:
exec("php http://domain.com/index.php?shopKeeper=$row[SKID]&shop=1");
exec() attempts to invoke a program on your server's file system, exactly as if you'd typed in the exec() parameter at a shell prompt. Shells do not know what to do with a URL, so you'll get a "no matching files" error, as the ? in the url will be seen as a shell wildcard.
If you want to request a url, you'd have to at minimum do
$output = file_get_contents("/var/www/etc....");
and PHP will do a full HTTP request for you.
Related
I have a website on an Ubuntu LAMP Server - that has a form which gets variables and then they get submitted to a function that handles them. The function calls other functions in the controller that "explodes" the variables, order them in an array and run a "for" loop on each variable, gets new data from slow APIs, and inserts the new data to the relevant tables in the database.
Whenever I submit a form, the whole website gets stuck (only for my IP, on other desktops the website continue working regularly), and I get redirected until I get to the requested "redirect("new/url);".
I have been researching this issue for a while and found this post as an example:
Continue PHP execution after sending HTTP response
After studding how this works in the server side, which is explained really good in this video: https://www.youtube.com/watch?v=xVSPv-9x3gk
I wanted to start learning how to write it's syntax and found out that this only work on CLI and not from APACHE, but I wasn't sure.
I opened this post a few days ago: PHP+fork(): How to run a fork in a PHP code
and after getting everything working from the server side, installing fork and figuring out the differences of the php.ini files in a server (I edited the apache2 php.ini, don't get mistaked), I stopped getting the errors I used to get for the "fork", but the processes don't run in the background, and I didn't get redirected.
This is the controller after adding fork:
<?php
// Registers a new keyword for prod to the DB.
public function add_keyword() {
$keyword_p = $this->input->post('key_word');
$prod = $this->input->post('prod_name');
$prod = $this->kas_model->search_prod_name($prod);
$prod = $prod[0]->prod_id;
$country = $this->input->post('key_country');
$keyword = explode(", ", $keyword_p);
var_dump($keyword);
$keyword_count = count($keyword);
echo "the keyword count: $keyword_count";
for ($i=0; $i < $keyword_count ; $i++) {
// create your next fork
$pid = pcntl_fork();
if(!$pid){
//*** get new vars from $keyword_count
//*** run API functions to get new data_arrays
//*** inserts new data for each $keyword_count to the DB
print "In child $i\n";
exit($i);
// end child
}
}
// we are the parent (main), check child's (optional)
while(pcntl_waitpid(0, $status) != -1){
$status = pcntl_wexitstatus($status);
echo "Child $status completed\n";
}
// your other main code: Redirect to main page.
redirect('banana/kas');
}
?>
And this is the controller without the fork:
// Registers a new keyword for prod to the DB.
public function add_keyword() {
$keyword_p = $this->input->post('key_word');
$prod = $this->input->post('prod_name');
$prod = $this->kas_model->search_prod_name($prod);
$prod = $prod[0]->prod_id;
$country = $this->input->post('key_country');
$keyword = explode(", ", $keyword_p);
var_dump($keyword);
$keyword_count = count($keyword);
echo "the keyword count: $keyword_count";
// problematic part that needs forking
for ($i=0; $i < $keyword_count ; $i++) {
// get new vars from $keyword_count
// run API functions to get new data_arrays
// inserts new data for each $keyword_count to the DB
}
// Redirect to main page.
redirect('banana/kas');
}
The for ($i=0; $i < $keyword_count ; $i++) { is the part that I want to get running in the background because it's taking too much time.
So now:
How can I get this working the way I explained? Because from what I see, fork isn't what I'm looking for, or I might be doing this wrong.
I will be happy to learn new techniques, so I will be happy to get suggestions about how I can do this in different ways. I am a self learner, and I found out the great advantages of Node.js for exmaple, which could have worked perfectly in this case if I would have learnt it. I will consider to learn working with Node.js in the future. sending server requests and getting back responses is awesome ;).
***** If there is a need to add more information about something, please tell me in comments and I will add more information to my post if you think it's relevant and I missed it.
What you're really after is a queue or a job system. There's one script running all the time, waiting for something to do. Once your original PHP script runs, it just adds a job to the list, and it can continue it's process as normal.
There's a few implementations of this - take a look at something like https://laravel.com/docs/5.1/queues
i have a kind of problems here and i don't know how to solve it... google said nothing and the search function here didn't shown me anything...
i use this code
$GAME_DB = new mysqli($__CONFIG['MySQL']['HOST'], $__CONFIG['MySQL']['USER'], $__CONFIG['MySQL']['PASS'], $__CONFIG['MySQL']['DB']['GAME']);
if($GAME_DB->connect_errno) { echo $GAME_DB->connect_error; exit; }
$VILLAGE_DATA['villageID'] = $mysqli->real_escape_string($VILLAGE_DATA['villageID']);
$query = "SELECT name, level, time FROM actions WHERE type='build' AND villageID='".$VILLAGE_DATA['villageID']."'";
if($result = $GAME_DB->query($query))
{
// table header
while($row = $result->fetch_row())
{
// some rows in here
}
// table footer
}
there aren't any syntax errors in that query and the results are there!
sometimes the query is successful and i see my table but sometimes (for example: when i reload) i receive the error message Commands out of sync; you can't run this command now
how can this just work "sometimes"? where is my problem?
http://dev.mysql.com/doc/refman/5.6/en/commands-out-of-sync.html
If you get Commands out of sync; you can't run this command now in
your client code, you are calling client functions in the wrong order.
This can happen, for example, if you are using mysql_use_result() and
try to execute a new query before you have called mysql_free_result().
It can also happen if you try to execute two queries that return data
without calling mysql_use_result() or mysql_store_result() in between.
But from the code you've posted I don't see how this could happen. Either there's more code than what you've posted and something there is happening out of order, or your MySQL connections are being improperly established/pooled/persisted somewhere outside of PHP.
I wrote a PHP script to pull tweets from the Twitter firehose and store them into a database. Ideally I want to just let it run so that it collects tweets over time, thus, it's wrapped in a while(1) loop.
This seems to be problematic because it's timing out. If I just run it in a browser, it won't run for more than 30 seconds before timing out and giving me a 324 Error.
Question: Is there a way that I can have it run for a certain amount of time (20 seconds), auto kill itself, then restart? All in a cron job (PS...I don't know how to write a cron job)?
Background: Site hosted on Godaddy. Would ideally like to run this on my hosting server there.
The Script:
<?php
$start = time();
$expAddress = "HOSTNAME";
$expUser = "USERNAME";
$expPwd = "PASSWORD";
$database = "DBNAME";
$opts = array(
'http' => array(
'method' => "POST",
'content' => 'keywords,go,here',
)
);
// Open connection to stream
$db = mysql_connect($expAddress, $expUser, $expPwd);
mysql_select_db($database, $db);
$context = stream_context_create($opts);
while (1) {
$instream = fopen('https://USERNAME:PASSWORD#stream.twitter.com/1/statuses/filter.json','r' ,false, $context);
while(! feof($instream)) {
if(time() - $start > 5) { // break after 5 seconds
break;
}
if(! ($line = stream_get_line($instream, 100000, "\n"))) {
continue;
}
else {
$tweet = json_decode($line);
// Clean before storing
// LOTS OF VARIABLES FOR BELOW...REMOVED FOR READABILITY
// Send to database
$ok = mysql_query("INSERT INTO tweets
(created_at, from_user, from_user_id, latitude, longitude, tweet_id, language_code,
place_name, profile_img_url, source, text, retweet_count, followers_count,
friends_count, listed_count, favorites_count)
VALUES
(NOW(), '$from_user', '$from_user_id', '$latitude', '$longitude', '$tweet_id', '$language_code',
'$place_name', '$profile_img_url', '$source', '$text', '$retweet_count', '$followers_count',
'$friends_count', '$listed_count', '$favorites_count')");
if (!$ok) { echo "Mysql Error: ".mysql_error(); }
flush();
}
}
}
?>
You can have cron jobs run once a minute.
To do this follow these steps:
Make a script that runs your PHP code, for example:
#!/bin/bash
wget myurl.com/blah > /dev/null
Save it as my-cron.sh in some folder (like /var)
Add it to cron. Run crontab -e See Cron Format and Crontab usage.
This for example, will run it once a minute.
# Minute Hour Day of Month Month Day of Week Command
* * * * * /var/my-cron.sh
If I get well your need, the best thing for you is to use cron job making a script run indefinitely will not be a good idea.
As specifier in one of you comments you are using a hosting server Godaddy so probably you will not be able to have shell access, BUT depending on your cPanel version you may be able to create and define cron job.
see this link and this google search
Perhaps, if you don't have this option and you are wiling to let a browser opened I would suggest the following
create an html page as a client which would make an ajax request every hours to your PHP script, like this you emulate a cron job function
the ajax request code might look like (using jQuery)
function makeRequest(){
$.ajax({
url: "http://yourhost/url-to-your-script.php",
complete: function(data){
setTimeout(function(){
makeRequest();
}, 60 * 60 * 1000); // Minutes * Seconds * MS
}
});
}
makeRequest();
I hope this helps
EDIT
this link might help too
IMPORTANT DO NOT FORGET TO REMOVE THE INFINITE LOOP
I just had same issue.
Only cron job can do if you want run script off browser. You can set up cron job with free providers or you can set up cron job in windows's Scheduled tasks.
If your site has a good traffic then you can follow the option below that your users does the work for you.
In php you can find time in hour and seconds
$time= date(' H:i:s');
create a table to track if the code was run.
eg; table column name check with option 0 and 1;
select check from table.
enter code here
if ($minute > 59)
{
if($check==0)
{
run your code
then update the table each time when it was run
eg; update table set check='1'
}
}
then another if condition to reset your code
if(minute>0 && minute <1)
{
select check from your table.
if(check==1)
{
update table set check='0'
}
}
I have used a multi curl library for PHP that facilitates fetching multiple pages in parallel (basically an easy to use API).
My Scenario: Fetch user data from API , process it and store results. All those users whose data have to be fetched are place in queue. This whole fetching , processing & storing result will take almost 8 - 10 min. And its really costly if I process it synchronously. So I have used php curl library for multi-threading. Its works fine if I run it in browser but since its cron-job so I have to run same script using command line. When I do so ; it will not work. Can anybody help me? Thanks in advance.
Psuedo Code:
$query = " Fetch users based on certain criteria LIMIT 200" ;
$result = execute-query ;
$curl_handle = curl_multi_init();
$i = 0;
$curl = array();
while ($row = mysql_fetch_assoc($result)) {
$curl[$i] = add_handle($curl_handle, API_CALL);
}
exec_handle($curl_handle);
for ($j = 0; $j < count($curl); $j++)//remove the handles
curl_multi_remove_handle($curl_handle, $curl[$i]);
curl_multi_close($curl_handle);
// Reference url
http://codestips.com/php-multithreading-using-curl/
I have two databases. Below is the code that I am using to get information from the first database.
$myrow = mysql_query("SELECT SUM(uploaded) FROM peers",$db);
$sum = mysql_fetch_array($myrow);
$c = $sum[0] / 1000000;
$d = $c / 1000000;
$l = round($d,3);
echo "<p>UP: $l TB</p>";
$myrow1 = mysql_query("SELECT SUM(downloaded) FROM peers",$db);
$sum1 = mysql_fetch_array($myrow1);
$a = $sum1[0] / 1000000;
$b = $a / 1000000;
$k = round($b,3);
echo "<p>DW: $k TB</p>";
I need to add this information to my second database and update it every 10 min with new fresh information from first database. I am using phpmyadmin.
Your question is very generic so I will try to answer for all scenario
You shoud create a process that run every 10 minutes (cron if you use Linux, scheduled task if you use windows)
If you use Linux you can
If you really want to use PHP, create a PHP script and call it using php command line or (much worst) create a php page that do what you want and have CRON to call it every 10 mins using LYNX browser.
Create a program in c/python/etc. that connects to first DB, query info, and writes to second.
Create a bash script that use mysql commandline to connect to DBs and do the same. (This has the advantage of not having to program)
If you use Windows you can:
Create a scheduled task in C# or vb.net or similar
Create a scheduled task using powershell
Use cron jobs for updating your information in DB.
cron jobs or PHP scheduler
http://net.tutsplus.com/tutorials/other/scheduling-tasks-with-cron-jobs