Im pulling in search query results using CURL and then iterating through a database to load additional queries then storing the results back in the database. Im running into hassles with php maximum time and have tried setting the maximum time variable to a higher amount which i think isnt working on my host using this:
ini_set('max_execution_time', 600);
in the file that is run by cron so it only changes the max time for the importing process.
The question is, would it be more effecient to store the result of each CURL connection in the database and then having a secondary function that pulls the dataabase results and sorts into the relevant tables and run the secondary function every 10 minutes hypothetically OR is it more effecient to pull the file and insert the sorted records in one go??
You can always find out whether your host is allowing you to modify the ini_set function by using ini_get('max_execution_time') right after your call to ini_set().
Instead of storing the results in the database, I would put them into a directory. Name the files by using the function microtime(true) (which makes it easy to pull the most or least recently written file). Then have a separate script that checks to see if there are files in the directory, and if so, processes one of them. Have the scripts run on a 1 minute interval.
I will note that there is a possible race condition on processing the file if it takes more than one minute, however, even if it takes longer than one minute, it is unlikely to ever occur.
Related
I have a database that has 20 rows each row I had set a Boolean value to it, so it is by default zero and when a row gets viewed its value changes to 1
I want the database to send any kind of signal that when 10 rows their value change from zero to 1, a certain PHP file fires up and starts a process that will affect only these 10 rows
How can I do that?
Thanks in advance
I would say, query from the php file every set amount of time to your database
The other way, database to execute a php file is almost impossible.
If you are using mySQL as database, a trigger could invoke the sys_exec() UDF available here: https://github.com/mysqludf/lib_mysqludf_sys#readme
So, there might be a possibility, actually, via an UDF function that would launch the php executable/script; not that easy, but seems possible ;-)
Invoking php from mysql is impossible, all you can do is set cron jobs for it. Cron job check mysql after certain interval of time and run the respected code
Every database is only a storage and it is its purpose in the system. Don't try to trigger any external process by the storage. The communication with the storage should be only a one way.
Rather think how to trigger your process from outside. Generally, there are two approaches:
a script that will check your database data in some interval like 1s, 10s, 1min or whatever would fit for a particular process
the current process that is updating your data can check your data and trigger another process if needed.
You can not trigger external file/script from mysql.
What you can do is create a cron job which run after certain interval of time which check database and perform certain operations.
Here's what I'm trying to accomplish in high-level pseudocode:
query db for a list of names (~100)
for each name (using php) {
query a 3rd party site for xml based on the name
parse/trim the data received
update my db with this data
Wait 15 seconds (the 3rd party site has restrictions and I can only make 4 queries / minute)
}
So this was running fine. The whole script took ~25 minutes (99% of the time was spent waiting 15 seconds after every iteration). My web host then made a change so that scripts will timeout after 70 seconds (understandable). This completely breaks my script.
I assume I need to use cronjobs or command line to accomplish this. I only understand the basic us of cronjobs. Any high level advice on how to split up this work in a cronjob? I am not sure how a cronjob could parse through a dynamic list.
cron itself has no idea of your list and what is done already, but you can use two kinds of cron-jobs.
The first cron-job - that runs for example once a day - could add your 100 items to a job queue.
The second cron-job - that runs for example once every minute in a certain period - can check if there are items in the queue, execute one (or a few) and remove it from the queue.
Note that both cron-jobs are just triggers to start a php script in this case and you have two different scripts, one to set the queue and one to process part of a queue so almost everything is still done in php.
In short, there is not much that is different. Instead of executing the script via modphp or fcgi, you are going to execute it via command line php /path/to/script.php.
Because this is a different environment than http, some things obviously don't work. Sessions, cookies, get and post variables. Output gets send to stdout instead of the browser.
You can pass arguments to your script by using $argv.
i have a big script written in php, which should import a lot of informations in a prestashop installation, using webservices, this script is written in "sections" I mean, there is a function that import the categories, another one that import products, then manufacturers, and so on, there are about 7 - 10 functions called in the main script. Basically I assume that this script must run for about an hour, passing from a function to the next one and so on since it arrives at the last function, then return some values and stops until the next night.
i would like to understand if it could be better :
1) impose a time limit of 30 minutes everytime i enter a new function (this will prevent the timeout)
2) make a chain of pages, each one with a single function call (and of course the time limit)
or any other idea... i would like to :
know if a function has been called (maybe using a global variable?)
be sure that the server will execute the function in order (so the pages chain)...
i hope to have beeen clear, otherwise i'll update the question.
edits:
the script is executed by another server that will call a page, the other server is "unkown" from me, so I simply know only that this page is called (they could also call the function by going on the page) but anyway i have no controll on it.
For any long running scripts, I would run it through the commandline, probably with a cronjob to kick it off. If it's triggered from the outside, I would create a job queue (for example in the database) where you insert a new row to signify that it should run, along with any variable input params. Then the background job would run - say - every 5 minutes, check if there's a new job in the queue. If there's not, just exit. If there is, mark that it has begun work and start processing. When done, mark that it's done.
1 hour of work is a looooooooong time though. Nothing you can do to optimise that?
You can increase the time limit for execution of a script as much as you want using :
set_time_limit(seconds);
And also for long running scripts you need a more memory. you can increase the memory limit using :
ini_set('memory_limit','20M');
And second other thing you have to make sure is that you are running your script on a dedicated server because if you are using a shared server you server will kill automatically long running scripts.
I'm created a Joomla extension in which i'm storing records from table A to table B. My script is working fine if table A contains less data.
If table A contains large amout of data. While inserting this huge data execution is getting exceed & showing this error 'Fatal error: Maximum execution time of 30 seconds exceeded in
/mysite/libraries/joomla/database/database/mysqli.php on line 382'.
I can overcome this problem by making change in ini file, but its Joomla extension which people gonna use it in their site so i can't tell them to make change in ini file infact i don't wanna tell them.
take a look into this
http://davidwalsh.name/increase-php-script-execution-time-limit-ini_set
ini_set('max_execution_time', 300);
use this way or
set_time_limit(0);
Use the below codes at the start of the page where you wrote the query codes
set_time_limit(0);
Technically, you can increase the maximum execution time using set_time_limit. Personally, I wouldn't mess with limits other people set on their servers, assuming they put them in for a reason (performance, security - especially in a shared hosting context, where software like Joomla! is often found). Also, set_time_limit won't work if PHP is run in safe mode.
So what you're left with is splitting the task into multiple steps. For example, if your table has 100000 records and you measure that you can process about 5000 records in a reasonable amount of time, then do the operation in 20 individual steps.
Execution time for each step should be a good deal less than 30 seconds on an average system. Note that the number of steps is dynamic, you programmatically divide the number of records by a constant (figure out a useful value during testing) to get the number of steps during runtime.
You need to split your script into two parts, one that finds out the number of steps required, displays them to the user and sequentially runs one step after another, by sending AJAX requests to the second script (like: "process records 5001 to 10000"), and marking steps as done (for the user to see) when the appropriate server respone arrives (i.e. request complete).
The second part is entirely server-sided and accepts AJAX requests. This script does the actual work on the server. It must receive some kind of parameters (the "process records 5001 to 10000" request) to understand which step it's supposed to process. When it's done with its step, it returns a "success" (or possibly "failure") code to the client script, so that it can notify the user.
There are variations on this theme, for instance you can build a script which redirects the user to itself, but with different parameters, so it's aware where it left off and can pick up from there with the next step. In general, you'd want the solution that gives the user the most information and control possible.
I have a simple question. I‘d like to write a php function to check the database rows and if the number of rows are affected by the last ran query, execute an internal php file. The catch is, that I want it to check the rows, and check the timestamp at the same time so if the time stamp is different and the row count is different, it executes the php file.
The file in question is a sql database backup, so I need it to only execute if there was a change in the database and if the time stamp is older than 43200 seconds (half a day). This would backup the database if there was activities on the site (one activity would back once, two activity would back up twice and anything more than that would be ignored), and if not, it would not do anything. I hope I’m explaining it right.
Cron job is out of question, since it’s dependant on the database changes not just the time.
The code I’m using is like this (without checking the database rows) and is only accessed when a customer access the shopping cart checkout or account page:
<?php
$dbbackuplog = '/path/to/backuptime.log';
if (file_exists($dbbackuplog)) {
$lastRun = file_get_contents($dbbackuplog);
if (time() - $lastRun >= 43200) {
//Its been more than 12 hours so run the backup
$cron = file_get_contents('/file.php');
//update backuptime.log with current time
file_put_contents($dbbackuplog, time());
}
}
?>
I appreciate any input or suggestions.
First of all, you cannot run anything with file_get_contents. That function simply reads the bare contents of the file you ask for and under no circumstances will it run any code. If you want to run the code, you want include or require instead.
Second, your idea about not just triggering but also fully executing backups while a customer is performing an action is, well, I 'm not going to pull any punches, terrible. There's a reason why people use cron for backups (actually more than one reason) and you should follow that example. That's not to say that you are not allowed to affect the behavior of the cron script based on dynamic factors, but rather that the act of taking a backup should always be performed behind the scenes.