I am trying to be make a online gcc compiler which can be accessed by the browser. For all this I have use php,cygwin on window XP. Actually I am giving the code window on the browser to the user.
The general process is as:
$source write in .c file
.c file compile by gcc compiler and .exe file created
the output of .exe file is shown to the user.
It's supposed that there is no read function in c program for testing I am use only a single printf statement.
The problem is:
If there are about to 30 simultaneous request means 30 users compiler the program at the same point of time then it will produce output in about 15 seconds, which is too long.
Can some please one help me to reduce this time. Each suggestions are welcomed
Can I just read the output of C program with out making a .exe file.
A starting point could be exploring distributed build systems.
On Windows I have known a (non-free) solution is Incredibuild. We used it for one of our projects around 8 years ago, and it reduced clean and build time from 25 minutes to around 5 minutes. There is a review of it here.
Right now, as I just searched for other alternatives and non-windows solutions I have also found distcc.
There is also a discussion (seemingly old) here about alternatives to Incredibuild.
If the C source code are almost the same for each compilation request, you can use compiler cache enabled building system. E. g. waf and cmake. They can utilized the copy of the previous building to speed up the compilation.
Related
If I would like to distribute PHP application with installer(package system of OS) how should I proceed? I don't want PHP files to be there, just working application, so when I type 'app' into console, it ends up being launching application, without need to install PHP on system(no php installation on host required). I would also like the application to have patch-able byte-code, so it's in parts, loaded when needed and only part needs to be replaced on update.
What I would do now is following:
->Compile PHP with extensions for specific platform.
->Make binary application which launches '/full/php app' when app is launched.
->Pack it in installer in a way, that there would be binary added to path when added, launching specific installation of PHP which is alongside the app with argument of start point->App would be running.
Problem is:
Maybe I don't want my PHP files to be exposed(in application, there will be available source anyway) is there some ready made stuff to do this? Is there some better way than I proposed?
Alternative: Modifying OP Cache to work with "packing" application to deliver byte codes to modified OP Cache which just reads the cache.
My suggestion would be a tiny tool I just finished, for almost exactly the same problem. (Oh yes I tried all the others but they're old and rusty, sometimes they're stuck with 4.x syntax, have no support, have no proper documentation, etc)
So here's RapidEXE:
http://deneskellner.com/sw/rapidexe
In the classical way, it's not a really-real compiler, just a glorified packer, but does exactly what you need: the output exe will be standalone, carrying everything with it and transparently building an ad-hoc runtime environment. Don't worry, it all happens very fast.
It uses PHP 7.2 / Win64 by default but has 5.x too, for XP compatibility.
It's freeware, obviously. (MIT License.)
(Just telling this because I don't want anyone to think I'm advertising or something. I just took a few minutes to read the guidelines about own-product answers and I'm trying to stay within the Code of the Jedi here.)
However...
I would also like the application to have patch-able byte-code, so it's in parts, loaded when needed and only part needs to be replaced on update.
It's easier to recompile the exe. You can extract the payload pieces of course but the source pack is one big zip; there seems to be no real advantage of handling it separately. Recompiling a project is just one command.
Maybe I don't want my PHP files to be exposed(in application, there will be available source anyway)
In this case, the exe contains your source compressed but eventually they get extracted into a temp folder. They're deleted immediately after run but, well, this is no protection whatsoever. Obfuscation seems to be the only viable option.
If something goes wrong, feel free to comment or drop me a line on developer-at-deneskellner-dot-com. (I mean, I just finished it, it's brand new, it may misbehave so consider it something like a beta for now.)
Happy compiling!
PHP doesn't do that natively, but here are a few ideas:
Self-extracting archive
Many archival programs allow you to create a self-extracting archive and some even allow to run a program after extraction. Configure it so that it extracts php.exe and all your code to a temp folder and then runs ir from there; deleting after the script has complete.
Transpilers/compilers
There's the old HPHC which translates PHP code to C++, and its wikipedia age also contains links to other, similar projects. Perhaps you can take advantage of those.
Modified PHP
PHP itself is opensource. You should be able to modify it withot too much difficulty to take the source code from another location, like some resource compiled directly inside the php.exe.
Use Zend Guard tool that compiles and converts the plain-text PHP scripts into a platform-independent binary format known as a 'Zend Intermediate Code' file. These encoded binary files can then be distributed instead of the plain text PHP. Zend Guard loaders are available for Windows and Linux platform that enables PHP to run the scripts encoded by Zend Guard.
Refer to http://www.zend.com/en/products/zend-guard
I would like to add another answer for anyone who might be Googling for answers.
Peach Pie compiler/runtime
There is an alternative method to run (and build apps from) .php source codes, without using the standard php.exe runtime. The solution is based on C#/.NET and is actually able to compile php source files to .NET bytecode.
This allows you to distribute your program without exposing its source code.
You can learn more about the project at:
https://www.peachpie.io/
You've got 3 overlapping questions.
1. Can I create a stand-alone executable from a PHP application?
Answered in this question. TL;DR: yes, but it's tricky, and many of the tools you might use are semi-abandoned.
2. Can I package my executable for distribution on client machines?
Yes, though it depends on how you answer question 1. If you use the .Net compiler, your options are different to the C++ option.
3. Can I protect my source code once I've created the application?
Again, depends on how you answer question 1. Many compilers include an "obfuscator" option which makes it hard to make sense of any information you get from decompiling the app. However, a determined attacker can probably get through that (this is why software piracy is possible).
I would like xdebug to trace only "jumps" of over X ms or of over Y KB RAM, for instance, every execution that took longer than 100ms or increased memory use by more than 100KB. This would let me ignore thousands of trace lines I don't need to see and would make optimisation much easier - as it is, in ZF2, the bloated framework takes 1 second just to start with the composer autoloader on our enterprise project, which results in thousands of lines I really have no use for. Somewhere along the line I do find the bigger jumps in execution time, but not after a long bout of scrolling.
Is there a default option to disable logging of "faster than X" executions, or if not, do you know of a helper shell/python script that could grep just the slower ones out?
For clarification, I am talking about the contents of the .xt file I get by running xdebug_start_trace() in my app.
I know nothing about such options, but what I may suggest is to use profile instead of trace.
Here is an article how you can use it.
If short, place these lines to your php.ini file:
xdebug.profiler_enable = 0
xdebug.profiler_enable_trigger = 1
xdebug.profiler_output_dir="c:\Projects"
and when you want to start profiler, run url with query parameter ?XDEBUG_PROFILE=1
This will produce a file with name like cachegrind.out.* and place it into profiler_output_dir.
That file could be viewed with CacheGrind viewer for your OS. Link above has a list of apps to view those files for different platforms. I were using wincachegrind (for Windows) to profile ZendFramework app. Very useful tool, as for me. And interface allow to see call tree, execution time, number of calls etc.
Well, but I see no option to measure memory usage with it.
The following PHP code does return me a runtime of about 3.5 seconds (measured multiple times and averaged):
$starttime = microtime(true);
exec('/usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');
$endtime = microtime(true);
$time_taken = $endtime-$starttime;
When i run the same command over a ssh terminal, the runtime is reduced to about 0.6 seconds (measured with the command line tool time).
The version of the imagemagick library is
Version: ImageMagick 6.7.0-10 2012-12-18 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP
What could be the reason for this time difference?
One answer to a similar question here on stackoverflow was that the overhead comes from the Webserver having to start a thread/shell. Could this be really the reason? I thought threads are leightweight and don't take long at all to start/terminate.
Prior to calling exec i set the number of threads used by imagemagick (because this was/is a bug in OpenMP?, Reference) to 1 with exec('env MAGICK_THREAD_LIMIT=1');. The runtime from PHP does not change much, no matter what value i set for MAGICK_THREAD_LIMIT. Anyway there does not seem to be a bug in OpenMP on in this version because the runtime of the command line execution is ok.
Any suggestions of how i could improve the runtime of the above command would be greatly appreciated.
Thank you very much for your help.
When you log in to a Unix machine, either at the keyboard, or over ssh, you create a new instance of a shell. The shell is usually something like /bin/sh or /bin/bash. The shell allows you to execute commands.
When you use exec(), it also creates a new instance of a shell. That instance executes the commands you sent to it, and then exits.
When you create a new instance of a shell command, it has it's own environment variables. So if you do this:
exec('env MAGICK_THREAD_LIMIT=1');
exec('/usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');
Then you create two shells, and the setting in the first shell never gets to the second shell. To get the environment variable into in the second shell, you need something like this:
exec('env MAGICK_THREAD_LIMIT=1; /usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');
Now, if you think that the shell itself may be the problem, because it takes too long to make a shell, test it with something that you know takes almost no time:
$starttime = microtime(true);
exec('echo hi');
$endtime = microtime(true);
$time_taken = $endtime-$starttime;
At that point you know to try and find some way to make the shell instantiate faster.
Hope this helps!
I've been programming computers for over 56 years, but this is the first time I've encountered a bug like this. So I spent nearly a week trying to understand the 7X worse execution speed when executing a perl program from php via exec versus executing the perl program directly at the command line. As part of this effort, I also pored over all the times this issue was raised on the web. Here's what I found:
(1) This is a bug that was first reported in 2002 and has not been fixed in the subsequent 11 years.
(2) The bug is related to the way apache interacts with php, so both of those organizations pass the buck to the other.
(3) The bug is the same on exec, system, or any of the alternatives.
(4) The bug doesn't depend on whether the exec-ed program is perl, exe, or whatever.
(5) The bug is the same on UNIX and Windows.
(6) The bug has nothing to do with imagemagick or with images in general. I encountered the bug in a completely different setting.
(7) The bug has nothing to do with startup times for fork, shell, bash, whatever.
(8) The bug is not fixed by changing the owner of the apache service.
(9) I'm not sure, but I think it relates to vastly increased overhead in calling subroutines.
When I encountered this problem I had a perl program that would execute in 40 seconds but through exec took 304 seconds. My ultimate solution was to figure out how to optimize my program so that it would execute in 0.5 seconds directly or in 3.5 seconds through exec. So I never did solve the problem.
#Philipp since you have SSH and since your server allows access to the exec() I will assume that you also have full root access to the machine.
Recommended for single file processing
Having root access to the machine means that you can change the /etc/php5/php.ini memory limit settings.
Even without direct access to /etc/php5/php.ini you could check if your server supports overriding php.ini directives by creating a new php.ini file in you projects directory.
Even if the overrides are not permitted you can change your memory settings from .htaccess if AllowOverride is All.
Yet another means of changing the memory limit is by setting it during the PHP runtime using ini_set('memory_limit', 256);.
Recommended for batch file processing
The only good thing about running the convert via exec() is if you don't plan on getting a result back from exec() and allowing it to run asynchronously:
exec('convert --your-convert-options > /dev/null 2>/dev/null &');
The above approach is usually helpful if you're trying to batch process many files, you don't want to wait for them to finish processing and don't need confirmation regarding each having been processed.
Performance notes
Using the code above to make exec run async for processing a single file will cost more processor time and more memory than using GD/Imagick within PHP. The time/memory will be used by a different process that does not affect the PHP process (making the visitors feel the site moving faster), but the memory consumption exists and when it comes to handling many connections that will count.
This is not a PHP bug and has nothing to do with Apache/Nginx or any webserver.
I had the same issue lately and had a look at the PHP source code to check the exec() implementation.
Essentially, PHP's exec() calls Libc's popen() function.
The offender here is C's popen() which seems to be very slow. A quick google search on "c popen slow" will show you lots of questions such as yours.
I also found that someone implemented a function called popen_noshell() in C to overcome this performance problem:
https://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/
Here is a screenshot showing the speed difference vs popen() and popen_noshell():
PHP's exec() uses the regular popen() - the one at the right of the screenshot above. The CPU used by the system when executing C's popen() is very high as you can see.
I see 2 solutions to this problem:
Create a PHP extension that implements popen_noshell
Request from the PHP team to create a new set of functions popen_noshell(), exec_noshell(), etc... which is unlikely to happen I guess...
Additional note:
While searching for this, I discovered the PHP function of the same name as the C's one: popen()
It is interesting because one can execute an external command asynchronously with: pclose(popen('your command', 'r'));
Which essentially has the same effect as exec('your command &');
I have experienced this problem, a graphic processing command that when run via the command line would take about .025 seconds was taking about .3 seconds when called via exec() in PHP. Upon much research, it seems that most people believe this to be a problem with apache or PHP. I then tried running the command via a CGI script, bypassing PHP altogether, and i got the same result.
It seemed therefore the problem must be apache, so i installed lighttpd, and got the same result!
After some thought and experimentation i realized this must be a problem with processor priority. So if you want your commands to run with a similar speed as the command line it must be executed as follows.
exec('echo "password" | sudo -S nice -n -20 command')
PLEASE NOTE: I know there will be all sorts of security objections to this. I just wanted to focus simply in the answer that all you have to do is add nice before your command.
When you call exec php does not create a thread, It creats a new child process. Creating a new process is big overhead.
However when you connect with ssh you are just passing a command to execute. You are not owner of that program so it executes as the user whom you connected with. For exec its the user who runs PHP.
I am building a web based interface where people can type in simple C code for solving algorithmic programming questions. I am using Ace editor where people can type in code and when the press the run button, the C code is sent to server, compiled and output sent back.
How do the accomplish the second part in a secure way. I mean given a C code file, compile it and execute it. I can't trust the code so how do i make sure its not malicious and will not harm my system. Also how to impose memory and time limits.
Is there any already existing system open source system available which I can modify to suit my needs? I didn't find anything in my search. Or some pointers on how i should proceed next?
edit: Found http://cs.sru.edu/~contest/rocktest/ and trying to understand their code but still looking for better options, preferably in php
Allow me to plug AppArmor, a simple mandatory access control mechanism that can make creating these sorts of sandboxes simple. Here is a profile I have in place to confine my xpdf PDF viewer:
#include <tunables/global>
/usr/bin/xpdf {
#include <abstractions/base>
#include <abstractions/bash>
#include <abstractions/X>
#include <abstractions/fonts>
/dev/tty rw,
owner /dev/pts/* rw,
/etc/papersize r,
/etc/xpdf/* r,
/bin/bash ix,
/usr/bin/xpdf r,
/usr/bin/xpdf.bin rmix,
/usr/share/xpdf/** r,
/usr/share/icons/** r,
owner /**.pdf r,
owner /tmp/* rw,
}
You could learn the basics of confining applications of your choice in half a day or so, and have profiles written for your server in another half day. (That xpdf profile took me about four minutes to write, but I know what I'm doing. We have deployed AppArmor on a leading online retailer's public-facing servers over the course of an afternoon, with similar results with other deployments.)
AppArmor also gives an easy interface for configuring run-time limits, such as how much memory a process is allowed to allocate:
rlimit as <= 100M, # limit address space to 100 megabytes
AppArmor would be easiest to use on Ubuntu, openSUSE, SLES, PLD, Mandriva, Pardis, or Annvix distributions, as the tools come pre-installed. But the core AppArmor functionality is in stock Linux kernels 2.6.36 and newer, and it is possible to install AppArmor on any Linux distribution.
Other similar tools include SElinux, TOMOYO, or SMACK. I think SMACK would be the next-easiest to deploy, but any of them could prevent malicious code from harming your system.
I recommend the Ideaone API: http://ideone.com/api
You'll have to execute the code in a sandboxed environment. There is a similar question on SO that might help.
You could also run some virtual machines to execute the code, but that's basically an example of sandboxing - just a bit heavy.
Run the code in a sandbox - a virtual machine.
In addition to that I would remove access to any sytem calls and only allow calls to the standard C libraries. Also, replace any unsafe library calls with your own calls that check the input and delegate safe inputs to the real functions (in particular for malloc you would want to put an upper bound on how much each program can allocate).
If you do the above, just one virtual machine should be enough for everyone's code.
I will be using uevalrun:
"The primary use case for uevalrun is evaluation of solution programs submitted by contestants of programming contests: uevalrun compiles the solution, runs it with the test input, compares its output against the expected output, and writes a status report."
I have cron job - php script which is called one time in 5 minutes. I need to be sure that previously called php script has finished execution - do not want to mix data that's being processed.
There are three approaches I used to apply:
Creation of auxiliary text file which contains running-state flag. Executed script analyzes the contents of the file and breaks if flag is set to true. It's the simplest solution, but every time I create such script, I feel that I invented a bike one more time. Is there any well-known patterns or best-practices which would satisfy most of the needs?
Adding UNIX service. This approach is the best for the cron jobs. But it's more time consuming to develop and test UNIX service: good bash scripting knowledge is required.
Tracking processes using database. Good solution, but sometimes database usage is not encouraged and again - do not want to invent a bike, hope there is a good flexible solution already.
Maybe you have other suggestions how to manage single-processing of php scripts? Would be glad to hear your thoughts about this.
I'd recommend using the file locking mechanism. You create a text file, and you make your process lock it exclusively (see php flock function: http://us3.php.net/flock). If it fails to lock, then you exit because there is another instance running.
The advantage of using file locking is that if your PHP scripts dies unexpectedly or gets killed, it will automatically release the lock. This will not happen if you use plain text files for the status (if the script is set to update this file at the end of execution and it terminates unexpectedly, you will be left with untrue data).
http://php.net/flock with LOCK_EX should be enough in your case.
You could check wether or not your script is currently running using the ps command, helped by the grep command. "man ps" and "man grep" will tell you all about these unix/linux commands if you need informations about these.
Let's assume your script is called 'my_script.php'. This unix command :
ps aux | grep my_script.php
...will tell you if your script is running. You can run this command with shell_exec() at the start of your script, and exit() if it's already running.
The main advantage of this method is that it can't be wrong, where the script could have crashed, leaving your flag file in a state that would let you think it's still running.
I'd stick to version number 1. It's simple and works out. As long as you only wan't to check whether the script has finished or not it should be sufficent. If more complex data is to be remembered I'd go for version 3 in order to be able to 'memorize' the relevant data...
hth
K