I'm using RabbitMq for submitting data for registered web hooks.
base info:
If a contact is created in the system, the message is put in the queue and the consumer is sending later the hook data to the registered url.
To my question:
Its possible, that a contact is updated in 5 seconds twice and that both messages are still in the queue.
But I'd like, if the second message is queued, that the first message will be removed.
I know that i can't delete the message manually. But is it somehow possible to set an id on the message and if two messages with the same id are in the same queue, that the first one is automatically removed/replaced?
That only one request is sent to the url. I know you can set a message id on the message self. But I have nothing found to replace the old one.
My PHP Code (simplified):
$connection = new AMQPConnection('localhost', 5672, 'test', 'test');
$channel = $connection->channel();
$channel->queue_declare(self::QUEUE_NAME, false, true, false, false);
$data = array(
'model' => get_class($subject),
'id' => $subject->getId(),
'event' => $event->getName()
);
$messageProperties = array(
'message_id' => get_class($subject) . '-' . $subject->getId()
);
$channel->basic_publish(new AMQPMessage(json_encode($data), $messageProperties), '', self::QUEUE_NAME);
$channel->close();
$connection->close();
Btw i'm using the the php amqplib https://github.com/videlalvaro/php-amqplib.
Thanks for the help
Flo
RabbitMQ doesn't delete/filter messages that way. You have to do that at the app level, probably using something like a bloom filter.
You could potentially tag each message with a unique message ID. A consuming application should keep an optimised list of inbound message IDs that it has already processed (in a thread-safe HashMap (Java), or Dictionary (.NET) implementation.
Should a message arrive, that has already been processed (the message ID is present in the stored list of handled message IDs), it will be ignored (or a polite “please wait” style response should be issued), preserving idempotency.
Related
I'm working on trace logger of sorts that pushes log message requests onto a Queue on a Service Bus, to later be picked off by a worker role which would insert them into the table store. While running on my machine, this works just fine (since I'm the only one using it), but once I put it up on a server to test, it produced the following error:
HTTP_Request2_MessageException: Malformed response: in D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 1013
0 HTTP_Request2_Response->__construct('', true, Object(Net_URL2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:1013
1 HTTP_Request2_Adapter_Socket->readResponse() D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:139
2 HTTP_Request2_Adapter_Socket->sendRequest(Object(HTTP_Request2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2.php:939
3 HTTP_Request2->send() D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\Http\HttpClient.php:262
4 WindowsAzure\Common\Internal\Http\HttpClient->send(Array, Object(WindowsAzure\Common\Internal\Http\Url)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\RestProxy.php:141
5 WindowsAzure\Common\Internal\RestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\ServiceRestProxy.php:86
6 WindowsAzure\Common\Internal\ServiceRestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:139
7 WindowsAzure\ServiceBus\ServiceBusRestProxy->sendMessage('<queuename>/mes…', Object(WindowsAzure\ServiceBus\Models\BrokeredMessage)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:155
⋮
I've seen previous posts that describe similar issues; Namely:
Windows Azure PHP Queue REST Proxy Limit (Stack Overflow)
Operations on HTTPS do not work correctly (GitHub)
That imply that this is a known issue regarding the PHP Azure Storage libraries, where there are a limited amount of HTTPS connections allowed. Before requirements were changed, I was accessing the table store directly, and ran into this same issue, and fixed it in the way the first link describes.
The problem is that the Service Bus endpoint in the connection string, unlike Table Store (etc.) connection string endpoints, MUST be 'HTTPS'. Trying to use it with 'HTTP' will return a 400 - Bad Request error.
I was wondering if anyone had any ideas on a potential workaround. Any advice would be greatly appreciated.
Thanks!
EDIT (After Gary Liu's Comment):
Here's the code I use to add items to the queue:
private function logToAzureSB($source, $msg, $severity, $machine)
{
// Gather all relevant information
$msgInfo = array(
"Severity" => $severity,
"Message" => $msg,
"Machine" => $machine,
"Source" => $source
);
// Encode it to a JSON string, and add it to a Brokered message.
$encoded = json_encode($msgInfo);
$message = new BrokeredMessage($encoded);
$message->setContentType("application/json");
// Attempt to push the message onto the Queue
try
{
$this->sbRestProxy->sendQueueMessage($this->azureQueueName, $message);
}
catch(ServiceException $e)
{
throw new \DatabaseException($e->getMessage, $e->getCode, $e->getPrevious);
}
}
Here, $this->sbRestProxy is a Service Bus REST Proxy, set up when the logging class initializes.
On the recieving end of things, here's the code on the Worker role side of this:
public override void Run()
{
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
Client.OnMessage((receivedMessage) =>
{
try
{
// Pull the Message from the recieved object.
Stream stream = receivedMessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string message = reader.ReadToEnd();
LoggingMessage mMsg = JsonConvert.DeserializeObject<LoggingMessage>(message);
// Create an entry with the information given.
LogEntry entry = new LogEntry(mMsg);
// Set the Logger to the appropriate table store, and insert the entry into the table.
Logger.InsertIntoLog(entry, mMsg.Service);
}
catch
{
// Handle any message processing specific exceptions here
}
});
CompletedEvent.WaitOne();
}
Where Logging Message is a simple object that basically contains the same fields as the Message Logged in PHP (Used for JSON Deserialization), LogEntry is a TableEntity which contains these fields as well, and Logger is an instance of a Table Store Logger, set up during the worker role's OnStart method.
This was a known issue with the Windows Azure PHP, which hasn't been looked at in a long time, nor has it been fixed. In the time between when I posted this and now, We ended up writing a separate API web service for logging, and had our PHP Code send JSON strings to it over cURL, which works well enough as a temporary work around. We're moving off of PHP now, so this wont be an issue for much longer anyways.
I'm using codeigniter-gcm library on top of codeigniter to send messages to Google Cloud Messaging service. It sends the message and the message is received at the mobile device, but if I send multiple messages, only the latest message appears on the device (as if it is overriding the previous messages).
I'm seeing that I might need to create a unique notification ID, but I'm not seeing how it's done anywhere on the codeigniter-gcm documentation or Google's documentation for downstream messages.
Any idea how this should be done?
Here's my code in the codeigniter controller. It is worth mentioning that Google's response contains a different message_id for each time I send a push...
public function index() {
$this->load->library("gcm");
$this->gcm->setMessage("Test message sent on " . date("d.m.Y H:i:s"));
$this->gcm->addRecepient("*****************");
$this->gcm->setData(array(
'title' => 'my title',
'some_key' => 'some_val'
));
$this->gcm->setTtl(false);
$this->gcm->setGroup(false);
if ($this->gcm->send())
echo 'Success for all messages';
else
echo 'Some messages have errors';
print_r($this->gcm->status);
print_r($this->gcm->messagesStatuses);
}
After three exhausting days I found the solution. I'm posting it here in hope of saving someone else's time...
I had to add a parameter to the data object inside the greater JSON object, named "notId" with a unique integer value (which I chose to use a random integer from a wide range). Now why Google didn't include this in their docs? Beats me...
Here's how my JSON looks now, when it creates separate notifications instead of overriding:
{
"data": {
"some_key":"some_val",
"title":"test title",
"message":"Test message from 30.09.2015 12:57:44",
"notId":14243
},
"registration_ids":["*******"]
}
Edit:
I'm now thinking that the notId parameter is not really determined by Google, but by a plugin I use on the mobile app side.
To extend further on my environment, my mobile app is developed using Phonegap, so to get push notification I use phonegap-plugin-push which I now see in its docs that parameter name.
I'm kinda' lost now as far as explaining the situation - but happy it is no longer a problem for me :-)
You need to pass a unique ID to each notification. Once you have clicked on the notification you use that ID to remove it.
...
mNotificationManager = (NotificationManager) getSystemService(NOTIFICATION_SERVICE);
mNotificationManager.cancel(SIMPLE_NOTFICATION_ID_A);
...
But I'm sure you shouldn't have so much of notifications for user at once. You should show a single notification that consolidates info about group of events like for example Gmail client does. Use Notification.Builder for that purpose.
NotificationCompat.Builder b = new NotificationCompat.Builder(c);
b.setNumber(g_push.Counter)
.setLargeIcon(BitmapFactory.decodeResource(c.getResources(), R.drawable.list_avatar))
.setSmallIcon(R.drawable.ic_stat_example)
.setAutoCancel(true)
.setContentTitle(pushCount > 1 ? c.getString(R.string.stat_messages_title) + pushCount : title)
.setContentText(pushCount > 1 ? push.ProfileID : mess)
.setWhen(g_push.Timestamp)
.setContentIntent(PendingIntent.getActivity(c, 0, it, PendingIntent.FLAG_UPDATE_CURRENT))
.setDeleteIntent(PendingIntent.getBroadcast(c, 0, new Intent(ACTION_CLEAR_NOTIFICATION), PendingIntent.FLAG_CANCEL_CURRENT))
.setDefaults(Notification.DEFAULT_LIGHTS | Notification.DEFAULT_VIBRATE)
.setSound(Uri.parse(prefs.getString(
SharedPreferencesID.PREFERENCE_ID_PUSH_SOUND_URI,
"android.resource://ru.mail.mailapp/raw/new_message_bells")));
I'm using php to send the message to rabbitmq and a python consumer to process it.
Here is what I did.
This part send a json to rabbitmq.
$data = array(
'id' => 123,
'url' => 'baidu.com',
);
$msg = new AMQPMessage(json_encode($data));
$channel->basic_publish($msg, $exchange);
And this part receive the message and process it (using celery).
#app.task
def mytask(json_obj):
print(json_obj)
data = json.loads(json_obj)
thread_id = data['id']
url = data['url']
return py_read(thread_id, url)
Here is what I get from the console:
[2014-09-29 15:51:34,564: WARNING/MainProcess] celery#mickey-Thurley ready.
[2014-09-29 15:51:37,395: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?
The full contents of the message body was: body: '{"id":123,"url":"baidu.com"}' (28b)
{content_type:None content_encoding:None
delivery_info:{'redelivered': False, 'routing_key': '', 'exchange': 'celery', 'delivery_tag': 1, 'consumer_tag': '4'} headers={}}
I'm sure that the consumer received the message, but why the message didn't be processd? and what should I do to deal with it?
A Celery task is not simply data. You also need to have something that tells the worker what task you are actually calling, and that's missing from your message.
Rather than trying to implement this yourself, you should probably use one of the Celery PHP implementations that are out there, such as this one.
I am building a notification system using Redis and predis (not mobile push notification, just a visual warning on a website).
For example, when user1 sends a message to user2, I create an entry in Redis.
On that type of event, I create a hash with all the information about the notification (date, content, sender...).
Then I add the hash key to a list that is specific to the user.
//creating the notification
notificationId = uniqid();
$this->redis->hmset($notificationId, array(
"sender" => "sender's Name",
"type" => "message",
"user_id" => "recipient's id",
"content" => "message content",
"date" => new \Datetime()
)
);
//adding the id of the notification in the user's message notifications list
$this->redis->lpush("messageList".$userId, $notificationId);
Then when I want to retrieve all the message notifications for the user I do :
$listName = "messageList:".$userId;
$arrNotifications = $this->redis->pipeline(function ($pipe) use ($listName) {
foreach ($pipe->getClient()->lrange($listName, 0, -1) as $key => $id) {
$arrNotif[] = $pipe->hgetall($id);
}
});
I get all the desired results with this method, but if the message list contains a couple thousands entries, the operation takes 0.5 seconds. It looks kinda slow, Redis is well known for being super fast.
So I am wondering if I am doing things correctly.
Any advice ?
Thanks
You may be much more satisfied with performance if you use phpredis.
It's a php C extension you have to compile in your server.
I started with predis too, and was glad to benchmark phpredis a better solution.
I'm attempting to use pecl-amqp for a project of mine. I'm having difficulties though with the ACK process. I need to manually ACK each message I receive off a queue, but the messages appear to be auto-ACKing when the message is retrieved.
I've set my queue to AMQP_NOACK and am using AMQPQueue->get(AMQP_NOACK) but none of it seems to have any affect, the messages are still removed from the queue without me sending AMQPQueue->ack().
If anyone has any experience with the pecl-amqp I would appreciate the help.
I have been having the same problem but have managed to get the acknowledge mechanism to work using the consume method. Using rabbitmqctl to list the entries in the queue it appears to work OK, though I seems to be getting the messages off the queue in a different order to that in which they were sent - which kind of defeats the object of a queue. My code is as follows:
// Create the queue to be:
// AMQP_DURABLE - messages will withstand a broker restart (i.e. they are written to disk).
// AMQP_NOACK - when consumed messages will not be marked as delivered until an explicit ACK is received.
$q->declare($queueName, AMQP_DURABLE | AMQP_NOACK );
// Bind it on the exchange to routing key.
$q->bind($exchangeName, $routingKey);
// Set the options for our consumption of the messages:
// Get a minimum of 0 msg.
// Get a maximum of 1 msg.
// Don't ACK the message on consumption i.e. explicitly acknoledge later.
$options = array(
'min' => 0,
'max' => 1,
'ack' => false
);
// Get the messages
$results_array = $q->consume($options);
// show the message
print_r($results_array);
$delivery_tag = $results_array[0]['delivery_tag'];
echo 'delivery_tag: [' . $delivery_tag . "].\r\n";
// Acknowledge receipt of the message.
$q->ack($delivery_tag);
For reference, the AMQP extension did not support the AMQP_NOACK correctly. You can get it to do what you want, but it isnt pretty. I am working on fixing that now. FYI, AMQP_NOACK means that you will not have to come back to acknowledge the message later, i.e. as soon as the server sends the message to the client, the server marks the message as ack'ed. There has been some confusion around this, so I wanted to clarify.