skip to Main Content
Solving Complex Ordering Challenges With Amazon SQS FIFO Queues

Solving Complex Ordering Challenges with Amazon SQS FIFO Queues

Contributed by Shea Lutton, AWS Cloud Infrastructure Architect

Amazon Simple Queue Service (Amazon SQS) is a fully managed queuing service that helps decouple applications, distributed systems, and microservices to increase fault tolerance. SQS queues come in two distinct types:

  • Standard SQS queues are able to scale to enormous throughput with at-least-once delivery.
  • FIFO queues are designed to guarantee that messages are processed exactly once in the exact order that they are received and have a default rate of 300 transactions per second.

As customers explore SQS FIFO queues, they often have questions about how the behavior works when messages arrive and are consumed. This post walks through some common situations to identify the exact behavior that you can expect. It also covers the behavior of message groups in depth and explains why message groups are key to understanding how FIFO queues work.

The simple case

Suppose that you run a major auction platform where people buy and sell a wide range of products. Your platform requires that transactions from buyers and sellers get processed in exactly the order received. Here’s how a FIFO queue helps you keep all your transactions in one straight flow.

A seller currently is holding an auction for a laptop, and three different bids are received for the same price. Ties are awarded to the first bidder at that price so it is important to track which arrived first. Your auction platform receives the three bids and sends them to a FIFO queue before they are processed.

Now observe how messages leave the queue. When your consumer asks for a batch of up to 10 messages, SQS starts filling the batch with the oldest message (bid A1). It keeps filling until either the batch is full or the queue is empty. In this case, the batch contains the three messages and the queue is now empty. After a batch has left the queue, SQS considers that batch of messages to be “in-flight” until the consumer either deletes them or the batch’s visibility timer expires.

 

When you have a single consumer, this is easy to envision. The consumer gets a batch of messages (now in-flight), does its processing, and deletes the messages. That consumer is then ready to ask for the next batch of messages.

The critical thing to keep in mind is that SQS won’t release the next batch of messages until the first batch has been deleted. By adding more messages to the queue, you can see more interesting behaviors. Imagine that a burst of 11 bids is sent to your FIFO queue, with two bids for Auction A arriving last.

The FIFO queue now has at least two batches of messages in it. When your single consumer requests the first batch of 10 messages, it receives a batch starting with B1 and ending with A1. Later, after the first batch has been deleted, the consumer can get the second batch of messages containing the final A2 message from the queue.

Adding complexity with multiple message groups

A new challenge arises. Your auction platform is getting busier and your dev team added a number of new features. The combination of increased messages and extra processing time for the new features means that a single consumer is too slow. The solution is to scale to have more consumers and process messages in parallel.

To work in parallel, your team realized that only the messages related to a single auction must be kept in order. All transactions for Auction A need to be kept in order and so do all transactions for Auction B. But the two auctions are independent and it does not matter which auctions transactions are processed first.

FIFO can handle that case with a feature called message groups. Each transaction related to Auction A is placed by your producer into message group A, and so on. In the diagram below, Auction A and Auction B each received three bid transactions, with bid B1 arriving first. The FIFO queue always keeps transactions within a message group in the order in which they arrived.

How is this any different than earlier examples? The consumer now gets the messages ordered by message groups, all the B group messages followed by all the A group messages. Multiple message groups create the possibility of using multiple consumers, which I explain in a moment. If FIFO can’t fill up a batch of messages with a single message group, FIFO can place more than one message group in a batch of messages. But whenever possible, the queue gives you a full batch of messages from the same group.

The order of messages leaving a FIFO queue is governed by three rules:

  1. Return the oldest message where no other message in the same message group is currently in-flight.
  2. Return as many messages from the same message group as possible.
  3. If a message batch is still not full, go back to rule 1.

To see this behavior, add a second consumer and insert many more messages into the queue. For simplicity, the delete message action has been omitted in these diagrams but it is assumed that all messages in a batch are processed successfully by the consumer and the batch is properly deleted immediately after.

In this example, there are 11 Group A and 11 Group B transactions arriving in interleaved order and a second consumer has been added. Consumer 1 asks for a group of 10 messages and receives 10 Group A messages. Consumer 2 then asks for 10 messages but SQS knows that Group A is in flight, so it releases 10 Group B messages. The two consumers are now processing two batches of messages in parallel, speeding up throughput and then deleting their batches. When Consumer 1 requests the next batch of messages, it receives the remaining two messages, one from Group A and one from Group B.

Consider this nuanced detail from the example above. What would happen if Consumer 1 was on a faster server and processed its first batch of messages before Consumer 2 could mark its messages for deletion? See if you can predict the behavior before looking at the answer.

If Consumer 2 has not deleted its Group B messages yet when Consumer 1 asks for the next batch, then the FIFO queue considers Group B to still be in flight. It does not release any more Group B messages. Consumer 1 gets only the remaining Group A message. Later, after Consumer 2 has deleted its first batch, the remaining Group B message is released.

Conclusion

I hope this post answered your questions about how Amazon SQS FIFO queues work and why message groups are helpful. If you’re interested in exploring SQS FIFO queues further, here are a few ideas to get you started:

credits: aws.amazon.com

This Post Has 132 Comments
  1. Good day! I just want to give you a huge thumbs up for the excellent information you have right here on this post.
    I will be returning to your web site for more soon.

  2. Howdy, i read your blog from time to time and i own a similar one
    and i was just curious if you get a lot of spam responses?
    If so how do you stop it, any plugin or anything you can recommend?
    I get so much lately it’s driving me insane so any assistance is very much appreciated.

  3. It’s in reality a nice and helpful piece
    of info. I am satisfied that you shared this useful information with us.
    Please stay us up to date like this. Thanks for sharing.

  4. Hey there! Someone in my Facebook group shared
    this site with us so I came to take a look.
    I’m definitely loving the information. I’m bookmarking and will be tweeting this
    to my followers! Fantastic blog and amazing style annd design.

  5. I’m not sure exactly why but this blog is loading extremely slow for
    me. Is anyone else having this issue or is it a issue on my end?
    I’ll check back later and see if the problem still exists.

  6. hello!,I really like your writing so a lot! proportion we keep in touch extra approximately your post on AOL?
    I require a specialist on this house to solve
    my problem. May be that is you! Having a look ahead to look you.

  7. What you posted made a lot of sense. But, what
    about this? suppose you added a little content?

    I am not suggesting your information is not solid., but suppose you added something
    that grabbed folk’s attention? I mean Solving Complex Ordering Challenges with Amazon SQS FIFO Queues – ANJ Webtech
    is a little boring. You might look at Yahoo’s front page and note how they
    create post titles to grab people to open the links.

    You might try adding a video or a related pic or two to get readers excited about everything’ve written. Just my opinion, it could bring your posts a little livelier.

  8. Good day! Do you use Twitter? I’d like to follow you if
    that would be ok. I’m absolutely enjoying your blog and look forward to new posts.

  9. Hey there! This is my first visit to your blog! We are a team of volunteers and starting
    a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done
    a wonderful job!

  10. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make
    your point. You obviously know what youre talking about, why throw away your intelligence on just
    posting videos to your site when you could be giving us
    something informative to read?

  11. After exploring a number of the blog articles
    on your web site, I really appreciate your
    way of writing a blog. I saved as a favorite it to my bookmark
    website list and will be checking back soon. Please visit my
    website too and let me know your opinion.

  12. hello there and thank you for your information – I’ve
    definitely picked up anything neew from right here. I did howewver
    expertise several technical issues using this website,
    since I experienced to reload the web site mazny times previous to I could
    get it to load correctly. I had been wondering if your
    hosting iss OK? Nott hat I’m complaining, but sluggish loading
    instances times will often affect your placement in google and could damage your high quality score iif advertising and
    marketing wigh Adwords. Well I amm adding this RSS to my email and could look
    out foor a lot more of your respective fascinatinng content.
    Make sure you update this again soon.

  13. Hello! I know this is kinda off topicc but I was wondering iif you knew where I could get a capptcha plugin for my comment form?
    I’m ueing the same blog plastform as yours and I’m having trouble finhding one?
    Thanks a lot!

  14. It’s in point of fact a great and helpful piece of information. I’m satisfied that you just shared this useful info with us.

    Please stay us informed like this. Thanks for sharing.

  15. Today, I went to the beach front with my children. I found a sea
    shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and
    screamed. There was a hermit crab inside and it pinched
    her ear. She never wants to go back! LoL I know this is
    totally off topic but I had to tell someone!

  16. We stumbled over here from a different web page and thought I might
    as well check things out. I like what I see
    so now i’m following you. Look forward to exploring your web
    page for a second time.

  17. I have read a few just right stuff here. Definitely value bookmarking for revisiting.
    I wonder how so much effort you set to make this type of
    great informative web site.

  18. Great post. I used to be checking continuously
    this weblog and I am inspired! Very helpful information specifically the closing phase :
    ) I maintain such information a lot. I used to be seeking this certain information for a long
    time. Thanks and good luck.

  19. Can I simply just say what a comfort to find somebody who
    genuinely understands what they’re discussing on the net.
    You actually understand how to bring an issue to light and make
    it important. More people have to check this out and understand this side
    of your story. I can’t believe you are not more
    popular since you surely possess the gift.

  20. My spouse and I stumbled over here coming from a
    different page and thought I might as well check things out.

    I like what I see so now i’m following you. Look forward
    to finding out about your web page for a second time.

  21. Nice post. I learn something totally new and challenging on blogs I stumbleupon everyday.

    It will always be useful to read through articles from
    other writers and practice a little something from other sites.

  22. I don’t even know the way I stopped up here, however I assumed this post was good.
    I do not understand who you’re however certainly you are going to a famous blogger if you are not
    already. Cheers!

  23. Your style is very unique in comparison to other people I’ve read
    stuff from. Thanks for posting when you’ve got the opportunity, Guess I will
    just bookmark this page.

  24. I’m curious to find out what blog platform you happen to
    be working with? I’m having some minor security issues with my latest blog and I would like to find something more safe.
    Do you have any solutions?

  25. I’m not sure exactly why but this site is loading very slow for
    me. Is anyone else having this problem or is it a issue on my end?
    I’ll check back later and see if the problem still exists.

  26. Great goods from you, man. I have understand your stuff previous to and you are just too excellent.
    I really like what you’ve acquired here, certainly like what you’re saying and the way in which you
    say it. You make it enjoyable and you still care for to keep
    it wise. I cant wait to read much more from you.
    This is really a wonderful site.

  27. Hello, There’s no doubt that your blog could possibly be having internet browser compatibility issues.
    Whenever I take a look at your blog in Safari, it looks fine however, when opening in IE, it has some overlapping issues.
    I simply wanted to give you a quick heads up! Aside from that, excellent blog!

  28. Do you have a spam problem on this site; I also am
    a blogger, and I was wanting to know your situation; many of us have created some nice methods and
    we are looking to trade solutions with other folks, be sure to shoot me an e-mail if interested.

  29. We stumbled over here coming from a different page and thought I might as
    well check things out. I like what I see so i am just following you.
    Look forward to looking over your web page
    repeatedly.

  30. Hi there i am kavin, its my first time to commenting anyplace, when i read this piece of writing i thought i could also create comment due to this sensible paragraph.

  31. Hello would you mind stating which blog platform you’re using?
    I’m planning to start my own blog in the near future but I’m having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design seems different then most blogs
    and I’m looking for something completely unique.
    P.S My apologies for getting off-topic but I had to ask!

  32. It’s really very complicated in this full of activity life to listen news on TV, therefore I just use the
    web for that purpose, and take the most up-to-date information.

  33. Amazing blog! Do you have any tips and hints for aspiring writers?
    I’m hoping to start my own site soon but I’m a little lost on everything.
    Would you advise starting with a free platform like WordPress or go
    for a paid option? There are so many choices out there that I’m completely confused ..
    Any suggestions? Kudos!

  34. Having read this I believed it was extremely informative.

    I appreciate you taking the time and effort to put this
    article together. I once again find myself personally
    spending a lot of time both reading and leaving comments.
    But so what, it was still worthwhile!

  35. Hello, i think that i saw you visited my website thus
    i came to “return the favor”.I am attempting to find things to improve my site!I
    suppose its ok to use some of your ideas!!

  36. This is very interesting, You are a very skilled blogger.
    I’ve joined your rss feed and look forward to seeking more of your fantastic post.
    Also, I’ve shared your site in my social networks!

  37. My brother suggested I would possibly like this blog. He
    used to be entirely right. This submit actually made my day.

    You cann’t believe just how a lot time I had spent for this info!

    Thanks!

  38. Thanks for your marvelous posting! I actually enjoyed reading it, you
    will be a great author. I will always bookmark your blog and will come back in the future.
    I want to encourage you to ultimately continue your great posts, have a nice day!

  39. Hi, Neat post. There’s an issue with your site in internet explorer, may test this?

    IE nonetheless is the marketplace chief and
    a large element of folks will omit your great writing due to this problem.

  40. hey there and thank you for your info – I’ve definitely picked up anything new from right
    here. I did however expertise a few technical points using this site,
    as I experienced to reload the web site many times previous to I could get it
    to load correctly. I had been wondering if your web hosting is OK?
    Not that I am complaining, but sluggish loading instances times will
    often affect your placement in google and could damage your quality score if advertising and marketing with Adwords.
    Well I am adding this RSS to my email and could look out for a lot more of your respective interesting content.
    Make sure you update this again soon.

  41. An outstanding share! I’ve just forwarded this onto
    a co-worker who had been conducting a little research on this.
    And he actually ordered me lunch due to the fact that I stumbled upon it for him…
    lol. So allow me to reword this…. Thanks for the meal!!
    But yeah, thanks for spending some time to talk about this
    issue here on your web page.

  42. Hi! I could have sworn I’ve been to this site before but after checking
    through some of the post I realized it’s new to me. Nonetheless, I’m definitely delighted
    I found it and I’ll be bookmarking and checking back frequently!

  43. Hi, I think your website might be having browser compatibility issues.
    When I look at your blog in Ie, it looks fine but when opening in Internet Explorer,
    it has some overlapping. I just wanted to give you
    a quick heads up! Other then that, fantastic blog!

  44. Howdy would you mind sharing which blog platform
    you’re using? I’m going to start my own blog in the near future but I’m having a tough time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design and style seems different then most blogs
    and I’m looking for something unique.
    P.S Apologies for getting off-topic but I had to ask!

  45. Good day! I could have sworn I’ve been to this web site before but after going through
    a few of the posts I realized it’s new to me.
    Anyhow, I’m definitely delighted I found it and I’ll be bookmarking it
    and checking back regularly!

  46. Great goods from you, man. I’ve understand your
    stuff previous to and you are just extremely
    great. I really like what you have acquired here, certainly like what you’re stating and the way
    in which you say it. You make it enjoyable and you still care for to keep it wise.
    I cant wait to read far more from you. This is actually a wonderful web
    site.

  47. Thank you for another fantastic article. Where else may anyone get that kind of information in such an ideal way of writing?

    I have a presentation next week, and I’m at the search for such info.

  48. Hmm is anyone else encountering problems with the pictures on this blog
    loading? I’m trying to determine if its a problem on my end or if it’s the
    blog. Any responses would be greatly appreciated.

  49. Hi, i think that i noticed you visited my site so i came to return the favor?.I am attempting to find issues to enhance
    my web site!I suppose its ok to use some of your ideas!!

  50. you are in reality a good webmaster. The web site loading pace is incredible.

    It seems that you are doing any distinctive trick.
    In addition, The contents are masterwork. you have done a magnificent task in this subject!

  51. What i don’t realize is actually how you are no longer really
    much more smartly-favored than you may be right now. You’re very intelligent.
    You understand therefore significantly with regards to
    this topic, made me individually consider it from a lot of
    various angles. Its like women and men don’t seem to
    be fascinated until it is something to accomplish with Lady gaga!

    Your individual stuffs outstanding. Always handle it up!

  52. Hey I know this is off topic but I was wondering if you knew of any
    widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would
    have some experience with something like this. Please let me know if you run into anything.
    I truly enjoy reading your blog and I look forward
    to your new updates.

  53. Just desire to say your article is as amazing. The clarity in your submit is
    just nice and that i can think you are knowledgeable in this
    subject. Fine along with your permission let me to grab your feed
    to stay up to date with forthcoming post. Thank you 1,000,000 and please continue the rewarding work.

  54. Your style is really unique compared to other folks I’ve read stuff from.

    I appreciate you for posting when you have the opportunity, Guess I will just book mark
    this web site.

  55. This is the perfect webpage for anybody who hopes to understand this topic.

    You understand a whole lot its almost hard to
    argue with you (not that I actually will need to…HaHa).
    You certainly put a new spin on a topic that’s been written about for many years.
    Wonderful stuff, just excellent!

  56. It’s the best time to make some plans for the future and it’s time to be happy.
    I’ve read this post and if I could I wish to suggest you few interesting things or suggestions.

    Perhaps you could write next articles referring to this article.
    I want to read more things about it!

  57. Fantastic items from you, man. I’ve remember your stuff prior to and
    you are just extremely wonderful. I actually like what you’ve got right here, really like what you are saying and the best
    way through which you say it. You make it enjoyable and you still take care of
    to keep it smart. I cant wait to learn much more from you.

    This is actually a wonderful web site.

  58. It is the best time to make some plans for the future and it’s time to be happy.

    I’ve read this post and if I could I desire to suggest you few interesting things or tips.
    Maybe you can write next articles referring to this article.
    I want to read more things about it!

  59. Can I simply say what a relief to find somebody who genuinely knows what they are discussing on the internet.
    You actually understand how to bring a problem to
    light and make it important. A lot more people must read this and understand this side of the story.

    I can’t believe you aren’t more popular since you definitely have the gift.

  60. Thanks for the good writeup. It in truth was once a amusement account it.
    Glance complicated to more brought agreeable from you! However, how can we communicate?

  61. Greetings from Carolina! I’m bored to tears at work so
    I decided to check out your site on my iphone during lunch break.
    I enjoy the info you present here and can’t wait to take a
    look when I get home. I’m surprised at how fast your blog loaded on my phone ..

    I’m not even using WIFI, just 3G .. Anyways, awesome site!

  62. I think that everything published was actually very logical.
    But, what about this? what if you were to create a
    killer title? I ain’t suggesting your information is not good, but suppose
    you added something that grabbed people’s attention? I mean Solving
    Complex Ordering Challenges with Amazon SQS FIFO
    Queues – ANJ Webtech is a little plain. You should look at Yahoo’s front page and watch how they create news
    headlines to get viewers to open the links. You might try adding a video or
    a pic or two to get readers excited about everything’ve written. Just my opinion, it would
    bring your blog a little bit more interesting.

  63. That is really fascinating, You’re an excessively professional blogger.
    I’ve joined your rss feed and look forward to searching for more of your excellent post.
    Also, I have shared your web site in my social networks

  64. Can I just say what a relief to find somebody that truly knows what they are discussing on the web.
    You certainly know how to bring an issue to light and make it important.
    More people have to look at this and understand
    this side of the story. It’s surprising you aren’t more popular
    since you certainly possess the gift.

  65. An outstanding share! I have just forwarded this onto a colleague who had been conducting a little
    research on this. And he in fact bought me lunch because I discovered it
    for him… lol. So allow me to reword this…. Thanks for the
    meal!! But yeah, thanks for spending the time to talk about this issue here on your
    blog.

  66. Its like you read my thoughts! You appear to know so much about this,
    like you wrote the e-book in it or something. I believe
    that you could do with some p.c. to pressure the message house
    a little bit, but other than that, this is great blog. A great read.

    I will certainly be back.

  67. My developer is trying to persuade me to move to .net from
    PHP. I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using WordPress on numerous websites for about a year and am anxious
    about switching to another platform. I have heard very
    good things about blogengine.net. Is there a way I can import all my wordpress posts into
    it? Any kind of help would be greatly appreciated!

  68. Hello, I think your site could possibly be having browser
    compatibility issues. When I look at your web site
    in Safari, it looks fine however when opening in Internet Explorer, it has some overlapping issues.
    I just wanted to provide you with a quick heads up!
    Apart from that, wonderful site!

  69. Howdy just wanted to give you a quick heads up. The text
    in your content seem to be running off the screen in Chrome.
    I’m not sure if this is a format issue or something to do with internet browser compatibility
    but I thought I’d post to let you know. The design look great though!
    Hope you get the issue resolved soon. Thanks

  70. I think this is one of the most vital information for me.
    And i am glad reading your article. But wanna remark on some general things,
    The web site style is great, the articles is really great
    : D. Good job, cheers

  71. My partner and I stumbled over here from a different web address and thought I may
    as well check things out. I like what I see so now i
    am following you. Look forward to looking into your web page repeatedly.

  72. When someone writes an article he/she keeps the image
    of a user in his/her brain that how a user can know it.
    Therefore that’s why this article is perfect.

    Thanks!

Leave a Reply

Your email address will not be published.

Back To Top