I currently have a bookmarklet that is used by many people in my department at work. The script takes a list of product IDs or gathers a list from the current page and visits each product detail page and gets various pieces of information. Using queries is not possible as I don't have database access. Anyway, I was thinking of building a fully fledged application for this + other features using either rails or PHP/some other framework. The biggest thing is the ability to send asynchronous HTTP requests as Javascript does.
I found this project: http://code.google.com/p/multirequest/, but haven't looked into it yet. What is the best way to go about this?
See this library https://code.google.com/p/multirequest/
It allows you to send multiple concurrent requests with CURL from PHP v5+
An example script is available which basically boils down to...
Define callback methods for when each request completes
Define what each request will look like (headers, etc)
Push each url to the MultiRequest_Handler class
Start the MultiRequest_Handler
At the most basic level, you can use multi curl in PHP. I've tested up to 150 simultaneous requests. You'd want to set a timeout to make sure a single request doesn't stall the whole process. It's not asynchronous, it's simultaneous, but it's quick.
Related
Question:
Could anyone please let me know at what frequency calling someone's website via cURL is not considered harmful?
Explanation:
I am building a small web app, where I fetch wordpress posts and some of its information from a clients website.
(!) Not as a web scraper , as they have to install mini-plugin that supplies only relevant information using my authkey.
Because the amount of pages can vary from 10 to 1000+. I am not doing it in one call; So I have made a page with Ajax script that pulls max 50 pages per call. This Ajax url calls my fetch.php, verifies the url each time (including header) and then gets the information via cURL. Repeats until finished.
Scenario:
Let's imagine client website has 1000 pages. So I would need to make a call 20 times (without delays, it's likely to happen within 30s).
Also, might need to consider that because I have to verify Domain URL before each call, which also have cURL with get headers only(as faster alternative to get_headers()).
I believe it's effectively doubles the amount of calls to 40 times.
So, ethically do I need to make a delay? or such volume of calls is not considered harmful to the client's website?
Thank you
This is likely to vary a lot, but as long as you make your calls sequentially one at a time I can't see that it could be harmful even for a small site. If you make them run at the same time it is another story.
I am writing an small web page at the moment that will consume a 3rd party API and process the data and display the return processed data in a table, a user will be able to change the data query based via a form input.
A couple of questions I have,
1) PHP seems like a redundant language here I can do ajax requests in vuejs?
1a) However I would like to be able to cache the 3rd party data, so if user selects the same query twice I don't need to go off an fetch it again, this seems like good practice?
1b) Or would it be better to cache the results page, and show that when a repeat request is made?
I am also using this excercise to start writing test for my PHP is is possible to write tests for 3rd party APIs?
The answer depends on wether you need caching or not. Keep in mind that ajax requests are sent by the browser and therefore don't cost you any server resources. Caching is only really necessary if the third party api you are using can't handle a huge amount of requests.
If you've decided you need caching, you'll have to access the api via your backend, in your case that means using php. Of course you could also write your own api dispatcher / cache in something like NodeJS and use this as a microservice, but that sounds overly complicated for a small project.
In my opinion you are best off to just access the api via ajax in vue, it'll save resources and will be the easiest way to go, everything else seems just redundant.
Testing a third party api can be tricky and in your case is probably redundant. What you rather want to test is how your application integrates with the api. You also probably want to write a mock for that api so that you can run your tests without depending on the api.
I need some advice on website design.
Lets take example of twitter for my question. Lets say I am making twitter. Now on the home_page.php ,I need both, Data about tweets (Tweet id , who tweeted , tweet time etc. etc) and Data about the user( userId , username , user profile pic).
Now to display all this, I have two option in mind..
1) Making separate php files like tweets.php and userDetails.php. By using AJAX queries, I can get the data on the home_page.php.
2) Adding all the php code (connecting to db, fetching data ) in the home_page.php itself.
In option one, I need to make many HTTP requests, which (i think) will be load to the network. So it might slow down the website.
But option two, I will have a defined REST API. Which will be good of adding more features in the future.
Please give me some advice on picking the best. Also I am still a learner, so if there are more options of implementing this, please share.
In number 1 you're reliant on java-script which doesn't follow progressive enhancement or graceful degradation; if a user doesn't have JS they will see zero content which is obviously bad.
Split your code into manageable php files to make it easier to read and require them all in one main php file; this wont take any extra http requests because all the includes are done server side and 1 page is sent back.
You can add additional javascript to grab more "tweets" like twitter does, but dont make the main functionality rely on javascript.
Don't think of PHP applications as a collection of PHP files that map to different URLs. A single PHP file should handle all your requests and include functionality as needed.
In network programming, it's usually good to minimize the number of network requests, because each request introduces an overhead beyond the time it takes for the raw data to be transmitted (due to protocol-specific information being transmitted and the time it takes to establish a connection for example).
Don't rely on JavaScript. JavaScript can be used for usability enhancements, but must not be used to provide essential functionality of your application.
Adding to Kiee's answer:
It can also depend on the size of your content. If your tweets and user info is very large, the response the single PHP file will take considerable time to prepare and deliver. Then you should go for a "minimal viable response" (i.e. last 10 tweets + 10 most popular users, or similar).
But what you definitely will have to do: create an API to bring your page to life. No matter which approach you will use...
I want to program some sort of notification system. What would be the be the best way to achieve this?
By calling Ajax request call to database on page load? But the problem with this is that it only checks on page load. It would be better if it was even realtime, I guess that depends on the priority of the message. How would I go about on achieving this?
By using cookies?
I am using PHP and jquery.
Like you said, it depends on message priority.
If notifications are made upon the visitor performing an action, it would be easy to tie in session-based notifications.
If notifications are made upon other users performing an action (e.g. "John made an edit to this page. Click here to view."), you could use long-polling AJAX to wait for a notification.
You might wanna look into COMET programming.
This would most likely require you to implement this as a distributed solution depending on your traffic.
Basically you'll have a java script function that goes to a server to check for notifications and if it finds any, calls another script that fetches the notifications back to the client. This should happen every so often, and WILL create a higher traffic and leave more connections open.
Take a look at this thread: Using comet with PHP?
For something small, I would consider using setInterval in combination with jQuery .load to load JSON.
Well AJAX and Jquery can be used to develop basic notification system. I've implemented it completely here: http://www.cloudways.com/blog/real-time-php-notification-system/
You can follow it step by step and also scale it as per your requirements.
Soon I'm going to have 3 identical scripts on 3 different VPS's and I want to have a fourth VPS which will control them all..
So for the sake of this question what I need to do is insert SQL rows and create files on the sub-servers, and the sub-servers also need to send statistical data back to the mother server. What is the most efficient way of doing this?
I was thinking of making scripts on the servers to do the jobs I need and using cURL to send requests to these scripts making use of URL parameters for the data which needs to be transferred, but perhaps there is a better way? Ideally I want it to be as fast as possible because they will most likely be sending requests to each other every second.
You could use XML-RPC, which exists in many manifestations:
http://us3.php.net/manual/en/book.xmlrpc.php
If you want dead-simple, just use plain HTTP(S) requests, provided you're careful about implementing it.
To perform a simple request, use cURL, file_get_contents, or fopen. This website is packed full of usage examples.
For simple comminication (ie. a script on server A triggers a script on server B), plain and simple HTTP queries works great. You can add basic authentication (htaccess) to avoid unauthorized people to trigger your script, and stronger security by using HTTPS.