Firstly, I am working on a windows xp 64 machine with 4gb ram and 2.29 ghz x4
I am indexing 220,000 lines of text that are more or less the same length. These are divided into 15 equally sized files. File 1/15 takes 1 minute to index. As the script indexes more files, it seems to take much longer with file 15/15 taking 40 minutes.
My understanding is that the more I put in memory, the faster the script is. The dictionary is indexed in a hash, so fetch operations should be O(1). I am not sure where the script would be hanging the CPU.
I have the script here.
You can try to monitor your machine to see if you're running out of memory. If so, you may want to look for memory leaks in your code.
Related
Recently my app started throwing fatal errors regarding exhaustion of the max allowed memory. After some research, I found out that the limit is set in the .htaccess file and it sets to 64m.
The tried allocation is of around 80MB, and we are able to provide these resources, but I wanted to ask the community if increasing the value of this variable is a good solution to the problem.
Thank you very much
The answer depends on what your script is doing :) It sounds like the increase in your case was quite small (from 64 to 80 MB, I'd recommend sticking to the powers of two though and upping it to 128MB btw.) so it shouldn't make much of a difference for modern machines. But to know if it was the right thing to do you need to find out why you needed more memory.
If your processing simply requires more memory (eg. you're processing uploaded files in memory, or decoding big json or xml structures or doing something else which is memory intensive) upping your limit is ok and common.
If however your application has memory leaks or is written in an inefficient way then upping the memory limit is just masking the problem and not solving it. You'll likely keep running into this issue and end up upping the memory all the time which is not feasible.
If you don't know what caused the sudden increase in memory consumption I'd recommend profiling your application using e.g. xhprof. You can also look at the last few changes to your app and see what might have caused it. If you can justify it then give your script more memory, otherwise try optimising your code first.
PHP is typically deployed in a way that a PHP process serves multiple request after another. During a single request a script can now allocate memory. At the end this memory will be free'd. So far so good. Now most operating systems are built in a way to keep memory, which was allocated bound to a process, even when freed. The assumption there is that a programmed which required memory once will need the amount again and it's cheaper to keep it available to the process than taking it back. Thus in a PHP deployment it might happen, that one request takes a lot of memory and then the memory is bound to the process and not available to the system anymore. Additionally it's a possible indication for a bug if some process takes a lot more memory than anticipated. So for those two things the memory_limit serves as safety net.
If your application needs more memory it's generally fine to increase the limit. The absolute maximum value is dependant on the system (available RAM / number of worker processes might be a rough formula, rough as it doesn't include other memory needed) Typically you should only increase by an amount needed.
Of course when changing this you have to remember when moving to other systems. Also typically less memory usage means faster execution thus you should try to see if you can optimise your code.
Side note: I purposely simplified the memory model above, ignoring virtual memory pages and how operating systems optimize there
How I can Limit cpu usage of apache2 php scirpts using
RLimitCPU seconds|max [seconds|max]
Please show me an example.
e.g RLimitCPU 2 2 ? whats that mean ?
I know its cpu seconds but question is how to convert GHz to seconds.
One php for video streaming script sometimes is taking 100% CPU usage on 2 cores.
http://httpd.apache.org/docs/2.2/mod/core.html#rlimitcpu
1 GHz is 1,000,000,000 CPU cycles per second - so a 2.6 GHz CPU is going to go through 2,600,000,000 cycles in one second. How many instructions actually get executed in a cycle is going to vary with the CPU - they'll all take a certain number of cycles to actually complete an instruction.
2 CPU seconds is "the CPU is completely maxed out for two full seconds or the equivalent". So if your program uses the CPU at half capacity for 4 full seconds that's 2 CPU seconds.
For your app, if you have a 2.6 GHz CPU and you run at 2 CPU seconds, you'll have executed 5,200,000,000 CPU cycles. How many instructions that is harder to work out, and how many instructions you actually need for your "video streaming script" is going to be incredibly hard to work out (and is going to vary with the length of the video).
I'd advise just running the script for the biggest video you'd ever send, seeing how many CPU seconds you use (top -u apache-usr will let you see the PHP process running, "TIME+" column is CPU time) and then tripling that as your RLimitCPU.
Bear in mind that RLimitCPU is just going to kill your PHP script when it takes more CPU time than the limit. It's not some magical tool that means your script will take less CPU time, it's just a limit on the maximum time the script can take.
Apache Reference: http_core, RLimitCPU
RLimitCPU
Resource Limit on CPU Usage
Syntax: RLimitCPU soft-seconds [hard-seconds]
Example: RLimitCPU 60 120
Since: Apache 1.2
This directive sets the soft and hard limits for maximum CPU usage of a process in seconds. It takes one or two parameters. The first parameter, soft-seconds, sets the soft resource limit for all processes. The second parameter, hard-seconds, sets the maximum resource limit. Either parameter can be a number, or max'', which indicates to the server that the limit should match the maximum allowed by the operating system configuration. Raising the maximum resource limit requires the server to be running as the userroot'', or in the initial start-up phase.
http://www.apacheref.com/ref/http_core/RLimitCPU.html
Hi guys I have a question about server's RAM and PHP/MySQL/Jquery script.
Can scripts kills RAM when script doesn't take extra RAM? (I know it could happen when RAM grow up to maximum or because of memory limit. But it isn't this case.)
I'm testing script but everytime when I do that RAM goes quickly down.
Script doesn't show error for memory limit and it's correctly loading all data. When I don't test script RAM is still down.
In database is a couple records - maybe 350 records in 9 tables (the bigges tables has 147 records).
(I haven't any logs just simply (really simple) graph for running server.)
Thank for your time.
If you're not getting errors in your PHP error log about failing to allocate memory, and you're not seeing other problems with your server running out of RAM (such as extreme performance degradation due to memory pages being written to disk for demand paging) you probably don't need to really worry about it. Any use case where a web server uses up that much memory in a single request is going to be pretty rare.
As for trying to profile the actual memory usage, trying to profile it by watching something like the task manager is going to be pretty unreliable. Most PHP scripts are going to complete in milliseconds, which isn't enough time for the memory allocations to really even register in the task manager.
Even if you have a more reliable method of profiling the memory usage (I don't recall if PHP has built in functions for this, but probably does), bear in mind that memory usage is going to flucuate tremendously for reasons that may be hard to understand. PHP in particular is very high level: you can open a database connection, which involves everything down to the OS opening network sockets, creating internal datastructures, caching things, and much more all in a single line of code. The script may allocate many megabytes of memory for such a thing for a single database row, but may then deallocate it a millisecond later.
Those database sizes are pretty neglibible. Depending on the row sizes it's possibly under a megabyte of data which is a tiny drop in a bucket for memory on anything remotely modern. Don't worry about memory usage for something like that. Only if you see your scripts failing and your error log reports running out of memory should you really worry about it.
I created a crawler that will operate as a cron job. The object of the crawler is to go through posts on my site and pull keywords from them.
Currently, I am optimizing the script for both speed and server load - but I am curious one what types of benchmarks for each are considered "good"?
For example, here are some configurations I have tested, running through 5,000 posts each time (you'll notice the trade off between speed and memory):
Test 1 - script optimized for memory conservation:
Run time: 52 seconds
Avg. memory load: ~6mb
Peak memory load: ~7mb
Test 2 - script optimized for speed
Run time: 30 seconds
Avg. memory load: ~40mb
Peak memory load: ~48mb
Clearly the decision here is speed vs. server load. I am curious what your reactions are to these numbers. Is 40mb an expensive number, if it increases speed so drastically (and also minimizes MySQL connections?)
Or is it better to run the script slower with more MySQL connections, and keep the overhead memory low?
This is a really subjective question given that what is "tolerable" depends on many factors such as how many concurrent processes will be running, the specs of the hardware it'll be running on, and how long you expect it to take.
I'm running a few PHP job which fetches 100th thousands of data from a webservice and insert them to database. These jobs take up the CPU usage of the server.
My question is, how much is it considered high?
When i do a "top" command on linux server,
it seems like 77% .. It will go up to more than 100% if i run more jobs simultaneously. It seems high to me, (does more than 100% means it is running on the 2nd CPU ?)
28908 mysql 15 0 152m 43m 5556 S 77.6 4.3 2099:25 mysqld
7227 apache 15 0 104m 79m 5964 S 2.3 7.8 4:54.81 httpd
This server is also has also webpages/projects hosted in it. The hourly job since to be affecting the server as well as the other web project's loading time.
If high, is there any way of making it more efficient on the CPU?
Anyone can enlighten?
A better indicator is the load average, if I simplify, it is the amount of waiting tasks because of insufficient resources.
You can access it in the uptime command, for example: 13:05:31 up 6 days, 22:54, 5 users, load average: 0.01, 0.04, 0.06. The 3 numbers at the end are the load averages for the last minute, the last 5 minutes and the last 15 minutes. If it reaches 1.00, (no matter of the number of cores) it is that something it waiting.
I'd say 77% is definitly high.
There are probably many ways to make the job more efficient, (recursive import), but not much info given.
A quick fix would be invoking the script with the nice cmd,
and add a few sleeps to stretch the load over time.
I guess you also saturate the network during import, so can you split up the job it would prevent your site from stalling.
regards,
/t
You can always nice your tasks
http://unixhelp.ed.ac.uk/CGI/man-cgi?nice
With the command nice you can give proccesses more or less priority
These jobs take up the CPU usage of the server.
My question is, how much is it considered high?
That is entirely subjective. On computing nodes, the CPU usage is pretty much 100% per core all the time. Is that high? No, not at all, it is proper use of hardware that has been bought for money.
Nice won't help much, since it's mysql that's occupying your cpu,
putting nice on a php-client as in
nice -10 php /home/me/myjob.php
won't make any significant difference.
Better to split up the job so smaller parts, call your php-script
from cron and build it like
<?
ini_set("max_execution_time", "600")
//
//1. get the file from remote server, in chunks to avoid net saturation
$fp = fopen('http://example.org/list.txt');
$fp2 = fopen('local.txt','w');
while(!feof($fp)) {
fwrite($fp2,fread($fp,10000));
sleep(5);
}
fclose($fp/fp2);
while(!eof(file) {
//read 1000 lines
//do insert..
sleep(10);
}
//finished, now rename to .bak, log success or whatever...