I have just started using Amazon SQS.
In order to make a request I must first call $sqs->GetQueueURL(...) to retreive the full URL of my queue.
ie: https://sqs.ap-southeast-1.amazonaws.com/XXXXXXX/QUEUENAME
As far as I can tell, I need to make this request every time before calling an operation on the queue (ie: ReceiveMessage or SendMessage).
Is there an inbuilt support for AWS PHP SDK to cache these URLs? Does it do it automatically?
I can't seem to find any detail on what (if any) caching is happening by default.
you do dont have to pull the queue url each time as it does not change.
go to sqs consule, and check all your queues. then copy the url you need, and use it.
Related
I'm currently developing a website. I want to cache results from an API but the API is slow, so I must handle concurrent PHP requests for same result.
When a PHP is collecting API results for a certain ID, I want next PHP requests for same ID to wait the first to finish and just read the cached value.
My current solution is to add an empty value in cache (and if value is empty new PHP requests will just sleep and recheck) but it sometimes doesn't work.
Is there another or a better way ?
Consider using synchronization provided by PECL Sync library and SyncReaderWriter class, or just by simple File Locking System.
If You want working solution, I've created PHP Cache library that synchronize read/write process especially for task like You described above.
I think you may find it useful, check it here: https://github.com/tztztztz/php-no-slam-cache
I've been working on a project for a while that fetches data from an API and processes that data locally for various uses. Currently, a consumer picks up JSON objects from the message queue that it uses to trigger a matching Symfony command. The rate limiting is built in to this one consumer, is fairly simple and adjusts itself automatically to status responses from the API. The problem is, the way it is set up, it cannot run in parallel and if there is a major update to the versioned static data on the API, all processing halts while it caches the new static data.
I looked at using the rabbitmq-bundle Symfony bundle and converting the commands into separate consumers with their own channels so that they can be run in parallel and no longer block each other, however this comes with a couple of issues I'm stuck with how to handle.
The first is that I still need to manage limiting the API calls across all the consumers. I have a wrapper for Guzzle that could, in theory, use a simple file to manage to number of calls across all instances of it. I looked at an existing token bucket library but setting it up to work in Symfony looks problematic as each consumer could potentially reset the number of tokens if the consumer is restarted, so... Not sure where to go with that.
The second is that some consumers may hit data from the main API that we're still do not have the matching version of the static data for. If this happens, it needs to trigger the related consumers but only if there isn't already a trigger in each queue... Possible solution I can see for this is record the latest requested version in a file at the time a message is published to update it and have the consumer wait for the data to be available locally. Again, kind of lost about how best to handle this.
I am trying to set up an API system that synchronously communicates with a number of workers in Laravel. I use Laravel 5.4 and, if possible, would like to use its functionality whenever possible without too many plugins.
What I had in mind are two servers. The first one with a Laravel instance – let’s call it APP – receiving and answering requests from and to a user. The second one runs different workers, each a Laravel instance. This is how I see the workflow:
APP receives a request from user
APP puts request on a queue
Workers look for jobs on the queue and eventually finds one.
Worker resolves job
Worker responses to APP OR APP finds out somehow that job is resolved
APP sends response to user
My first idea was to work with queues and beanstalkd. The problem is that this all seem to work asynchronously. Is there a way for the APP to wait for the result of one of the workers?
After some more research I stumbled upon Guzzle. Would this be a way to go?
EDIT: Some extra info on the project.
I am talking about a Restful API. E.g. a user sends a request in the form of "https://our.domain/article/1" and their API token in the header. What the user receives is a JSON formatted string like {"id":1,"name":"article_name",etc.}
The reason for using two sides is twofold. At one hand there is the use of different workers. On the other hand we want all the logic of the API as secure as possible. When a hack attack is made, only the APP side would be compromised.
Perhaps I am making things all to difficult with the queues and all that? If you have a better approach to meet the same ends, that would of course also help.
I know your question was how you could run this synchronously, I think that the problem that you are facing is that you are not able to update the first server after the worker is done. The way you could achieve this is with broadcasting.
I have done something similar with uploads in our application. We use a Redis queue but beanstalk will do the same job. On top of that we use pusher which the uses sockets that the user can subscribe to and it looks great.
User loads the web app, connecting to the pusher server
User uploads file (at this point you could show something to tell the user that the file is processing)
Worker sees that there is a file
Worker processes file
Worker triggers and event when done or on fail
This event is broadcasted to the pusher server
Since the user is listening to the pusher server the event is received via javascript
You can now show a popup or update the table with javascript (works even if the user has navigated away)
We used pusher for this but you could use redis, beanstalk and many other solutions to do this. Read about Event Broadcasting in the Laravel documentation.
I have a need for part of my application to make calls to Reddit asynchronously from my core application's workflow. I have implemented a semi-workable solution by using a Reddit API library I have built here. For those that are unaware, Reddit manages authentication via OAuth and returns a bearer and a token for a particular user that expires in 60 minutes after generation.
I have opted to use cookies to store this authorization information for the mentioned time period, as seen in the requestRedditToken() method here. If a cookie is not found (i.e. it has expired) when another request to Reddit needs to be made, another reddit token is generated. This seems like it would work just fine.
What I am having trouble with is conceptualizing how cookies are handled when integrated with a daemonized queue worker, furthermore, I need to understand why these calls are failing periodically.
The application I'm working with, as mentioned, makes calls to Reddit. These calls are created by a job class being handled: UpdateRedditLiveThreadJob, which you can see here.
These jobs are processed by a daemonized Artisan queue worker using Laravel Forge, you can see the details of the worker here. The queue driver in this case is Redis, and the workers are monitored by Supervisor.
Here is the intended workflow of my app:
An UpdateRedditLiveThreadJob is created and thrown into the queue to be handled.
The handle() method of the job is called.
A Reddit client is instantiated, and a reddit token is requested if a cookie doesn't exist.
My Reddit client successfully communicates with Reddit.
The Job is considered complete.
What is actually happening:
The job is created.
Handle is called.
Reddit client is instantiated, something odd happens here generally.
Reddit client tries to communicate, but gets a 401 response which produces an Exception. This is indicative of a failed authorization.
The task is considered 'failed' and loops back to step 2.
Here are my questions:
Why does this flow work for the first hour, and then collapse as described above, after presumably, the cookie has expired?
I've tried my best to understand how Laravel Queues work, but I fundamentally am having a hard time of conceptualizing the different types of queue management options available: queue:listen, queue:work, a daemonized queue:work running on Supervisor, etc. Is my current queue infrastructure compatible with using cookies to manage tokens?
What adjustments do I need to make to my codebase to make the app function as intended?
How will my workflow handle multiple users, who each potentially have multiple cookies?
Why does the workflow magically start working again if I restart my queue worker?
Please let me know if I'm incorrectly describing anything here or need clarification, I've tried my best to explain the problem succinctly.
Your logic is incorrect. A queue job is in fact a cli running php script. It has no interaction with a browser. Cookies are set in a browser, see this related thread for reference.
Seeing you're interacting with an API it would make more sense to set the token as a simple variable in the Job (or better yet in that wrapper) and then re-use that within that job.
TL:DR: your wrapper is not an API client.
I know this is not a complete answer to all your questions, but it's a push in the right direction. Because would I have answered all your questions - in the end - might not have given any solution to your issues ;)
We do a lot of processing for images and a lot of the time this processing kills all our CPU and causes our site to crash. What we want to do is put the image processing on another server, so that we can scale that server as nessicary and not have our current server crash.
I'm wondering how to go about this though. Our current process is:
1) User's make AJAX request to our Image Processing Script.
2) We construct a string based on the user's input. This string contains the commands to perform an ImageMagick process.
3) We run the string through PHP's system() command.
4) We then send headers to the page and use PHP's imagecreatefrompng() functions on the file to output the image to the user.
So what I'd like to know, what's the best way to now transfer the ImageMagick process. I thought of remote connecting to the other server via SSH, but I'm sure there is a limit on the number of connections that can be made via SSH. We have 100s of users online at a time so we need to be able to do that many connections at a time.
Anyone with any ideas on how best to transfer our image processing to another server would be greatly welcomed.
Thanks!
SSH would not be an appropriate protocol to distribute a work request to another server. A popular trend is to leverage a messaging queue to dispatch tasks to "worker" nodes. The implementation can very greatly depending on design, needs, and resource constraints. Here's a quick bare-bone outline...
A web server receives a new image item.
Writes image to CDN, or network mount &etc.
Publishes a task to a messaging queue, like RabbitMQ
A worker node listens for new tasks.
Consumes and performs request.
Writes result output next to source on CDN
Notifies tasks complete by either updating a record in DB, or publish back to MQ.
Checkout RabbitMQ/PHP "Hello World", and "Work Queues" articles for detailed examples.