I have a php spider script that works for taking web site contents, and save the content to my database. It takes about 50 days to finish its job. I have a Virtual dedicated server for that, but as you realize, it takes huge time to finish execution. So, can cloud computing help my task? I searched on cloud computing and php, but I could not find nice guide for starting. Can you help me about how to start cloud computing for running php scripts faster?
You can go for Amazon EC2. Some or should I say many big companies in the world goes for Amazon EC2.
But before upgrading to such high level I suggest you to rewrite your codes multi-threaded/mutli-core using Python or C or Java, whichever seems to be your favorite.
For more information/startup guide/pricing for Amazon EC2 just check out this link:
http://aws.amazon.com/ec2/
Related
I am currently using an AWS micro instance as a web server for a website that allows users to upload photos. Two questions:
1) When looking at my CloudWatch metrics, I have recently noticed CPU spikes, the website receives very little traffic at the moment, but becomes utterly unusable during these spikes. These spikes can last several hours and resetting the server does not eliminate the spikes.
2) Although seemingly unrelated, whenever I post a link of my website on Twitter, the server crashes (i.e.,Error Establishing a Database Connection). Once restarting Apache and MySQL, the website returns to normal functionality.
My only guess would be that the issue is somehow the result of deficiencies with the micro instance. Unfortunately, when I upgraded to the small instance, the site was actually slower due to fact that the micro instances can have two EC2 compute units.
Any suggestions?
If you want to stay in the free tier of AWS (micro instance), you should off load as much as possible away from your EC2 instance.
I would suggest you to upload the images directly to S3 instead of going through your web server (see some example for it here: http://aws.amazon.com/articles/1434).
S3 can also be used to serve most of your web pages (images, js, css...), instead of your weak web server. You can also add these files in S3 as origin to Amazon CloudFront (CDN) distribution to improve your application performance.
Another service that can help you in off loading the work is SQS (Simple Queue Service). Instead of working with online requests from users, you can send some requests (upload done, for example) as a message to SQS and have your reader process these messages on its own pace. This is good way to handel momentary load cause by several users working simultaneously with your service.
Another service is DynamoDB (managed NoSQL DB service). You can put on dynamoDB most of your current MySQL data and queries. Amazon DynamoDB also has a free tier that you can enjoy.
With the combination of the above, you can have your micro instance handling the few remaining dynamic pages until you need to scale your service with your growing success.
Wait… I'm sorry. Did you say you were running both Apache and MySQL Server on a micro instance?
First of all, that's never a good idea. Secondly, as documented, micros have low I/O and can only burst to 2 ECUs.
If you want to continue using a resource-constrained micro instance, you need to (a) put MySQL somewhere else, and (b) use something like Nginx instead of Apache as it requires far fewer resources to run. Otherwise, you should seriously consider sizing up to something larger.
I had the same issue: As far as I understand the problem is that AWS will slow you down when you reach a predefined usage. This means that they allow for a small burst but after that things will become horribly slow.
You can test that by logging in and doing something. If you use the CPU for a couple of seconds then the whole box will become extremely slow. After that you'll have to wait without doing anything at all to get things back to "normal".
That was the main reason I went for VPS instead of AWS.
In our project we have a front-end part with PHP and Apache, a MySQL database and some Mono servers that do some logic (chat, etc...)
It's all completed but before starting and buy the servers i want know how make them scalable and be fault tollerance. We want buy the servers on So You Start.
How can we proceed? What we must buy (possible cheap)? We thinked about a virtualization but we don't know how move.
Thank you in advance.
Amazon has a lot of very good documentation about how to architect their cloud services. I would suggest you start with a free (for 1 year) Usage Tier for learning AWS and testing your application. The free tier It enables you to setup a Unix and Windows instance as well as RDS (Relational Database Server) and email service. You can later add fail-over servers located in geographically dispersed locations. You can also scale easily by adding (and removing) instances as needed.
Start with this overview.
In AWS you can start small and then use AutoScaling with ELB and cloudfront to really scale up as needed.
http://dailytechscape.com
I'm designing a PHP application that works on several upload servers
What I basically want to do, is to use a Cronjob that fetches reports from each sub-server, and gives me:
1- How much space is left on the drive and
2- How much the bandwidth for that server is saturated (e.g. if a server is running at full bandwidth, use another now)
The first problem I solved with df on linux, but how do I get the current bandwidth saturation to be used in my PHP app? Which command should I use in linux?
Thanks
You can capture the output of shell_exec() while executing a command, but I figure you already do that for df.
As for the bandwidth, this is described here. You could read those files directly in PHP.
So unfortunately I'm a newbie to the application (web) development world. While I did graduate with a CS degree its theory funness hasn't helped out much. However I am trying to jump in feet first. My php application I build is simply sitting on a cheap hosting provider. In order to launch it I'd like to it to be able to scale well (fun buzz word).
However, as I said I'm totally new to it and I suck at git.
So as I've read I'm looking to host the mySql database and the php files on phpfog, my css and images on amazon S3, and then the email server as Amazon SES?
Is this the right frame of mind, and/or will this handle a flurry of (hoped for) traffic?
Thanks!
That's a lot of scaling in mind for just starting out. Keep it simple. Google Apps Standard gives you 10 accounts to work with, and you can a nice GMail interface as a bonus.
Keep your hosting simple for now, you don't need Amazon S3 for CSS and images yet. Silver pricing seems decent for just starting out at PHPFog. Remember you can always increase capacity later on.
Before you say "flurry of traffic", make sure you have a solid business plan. How will you market your product/service? Who are your competitors? What differentiates your product/service? Pretend like you're pitching this to a VC firm.
If you don't have a solid plan to work with, you're throwing too much money at infrastructure that you won't be using.
If you want to learn about the technical aspects of the infrastructure you're choosing, you probably should begin by learning about how it works. http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf
will give you a nice overview on how scaling works on AWS and the cloud in general.
I have a php script which requires no web hosting or disk space etc.
The script simply retrieves infromation from a server, does some processing on this information
and than passes it on to a client (an iPhone app in this case).
The only thing is that if traffic gets high than there is a high demand for bandwidth and speed.
Does anyone know of a service with high speed and badwidth (apart from web hosting services) that allows you to host (on a static ip) such a php script?
Thanks.
You may want to try some sort of cloud service where you can set up the environment you actually need. Let's say your script need a lot of RAM but only little CPU power (or the other way around) you can have exactly such a system. Amazon EC2 is one of many cloud computing providers out there.
hmm the performance point you can use something like "Facebook HipHop" to convert your php script into "c++" then you have the performance you need.
Cloud solution is perfect. You can even write shell scripts to increase decrease RAM whenever demand goes up.
Like everyone here mentioned, cloud hosting is your best bet. It's slightly more expensive for resources & bandwidth than a dedicated but is superior in performance/latency/scalability. I have a similar application setup for a current project and I am running on the RackSpace cloud with 100K+ active users on a daily basis and I have had no problems (been running for 6 months).
If your code is simple, don't use Php !
You can consider:
Python
GWan server + C
Java
Php is good for big project because its simple, fast to use/test/debug ...