skip to Main Content
Solving Complex Ordering Challenges With Amazon SQS FIFO Queues

Solving Complex Ordering Challenges with Amazon SQS FIFO Queues

Contributed by Shea Lutton, AWS Cloud Infrastructure Architect

Amazon Simple Queue Service (Amazon SQS) is a fully managed queuing service that helps decouple applications, distributed systems, and microservices to increase fault tolerance. SQS queues come in two distinct types:

  • Standard SQS queues are able to scale to enormous throughput with at-least-once delivery.
  • FIFO queues are designed to guarantee that messages are processed exactly once in the exact order that they are received and have a default rate of 300 transactions per second.

As customers explore SQS FIFO queues, they often have questions about how the behavior works when messages arrive and are consumed. This post walks through some common situations to identify the exact behavior that you can expect. It also covers the behavior of message groups in depth and explains why message groups are key to understanding how FIFO queues work.

The simple case

Suppose that you run a major auction platform where people buy and sell a wide range of products. Your platform requires that transactions from buyers and sellers get processed in exactly the order received. Here’s how a FIFO queue helps you keep all your transactions in one straight flow.

A seller currently is holding an auction for a laptop, and three different bids are received for the same price. Ties are awarded to the first bidder at that price so it is important to track which arrived first. Your auction platform receives the three bids and sends them to a FIFO queue before they are processed.

Now observe how messages leave the queue. When your consumer asks for a batch of up to 10 messages, SQS starts filling the batch with the oldest message (bid A1). It keeps filling until either the batch is full or the queue is empty. In this case, the batch contains the three messages and the queue is now empty. After a batch has left the queue, SQS considers that batch of messages to be “in-flight” until the consumer either deletes them or the batch’s visibility timer expires.

 

When you have a single consumer, this is easy to envision. The consumer gets a batch of messages (now in-flight), does its processing, and deletes the messages. That consumer is then ready to ask for the next batch of messages.

The critical thing to keep in mind is that SQS won’t release the next batch of messages until the first batch has been deleted. By adding more messages to the queue, you can see more interesting behaviors. Imagine that a burst of 11 bids is sent to your FIFO queue, with two bids for Auction A arriving last.

The FIFO queue now has at least two batches of messages in it. When your single consumer requests the first batch of 10 messages, it receives a batch starting with B1 and ending with A1. Later, after the first batch has been deleted, the consumer can get the second batch of messages containing the final A2 message from the queue.

Adding complexity with multiple message groups

A new challenge arises. Your auction platform is getting busier and your dev team added a number of new features. The combination of increased messages and extra processing time for the new features means that a single consumer is too slow. The solution is to scale to have more consumers and process messages in parallel.

To work in parallel, your team realized that only the messages related to a single auction must be kept in order. All transactions for Auction A need to be kept in order and so do all transactions for Auction B. But the two auctions are independent and it does not matter which auctions transactions are processed first.

FIFO can handle that case with a feature called message groups. Each transaction related to Auction A is placed by your producer into message group A, and so on. In the diagram below, Auction A and Auction B each received three bid transactions, with bid B1 arriving first. The FIFO queue always keeps transactions within a message group in the order in which they arrived.

How is this any different than earlier examples? The consumer now gets the messages ordered by message groups, all the B group messages followed by all the A group messages. Multiple message groups create the possibility of using multiple consumers, which I explain in a moment. If FIFO can’t fill up a batch of messages with a single message group, FIFO can place more than one message group in a batch of messages. But whenever possible, the queue gives you a full batch of messages from the same group.

The order of messages leaving a FIFO queue is governed by three rules:

  1. Return the oldest message where no other message in the same message group is currently in-flight.
  2. Return as many messages from the same message group as possible.
  3. If a message batch is still not full, go back to rule 1.

To see this behavior, add a second consumer and insert many more messages into the queue. For simplicity, the delete message action has been omitted in these diagrams but it is assumed that all messages in a batch are processed successfully by the consumer and the batch is properly deleted immediately after.

In this example, there are 11 Group A and 11 Group B transactions arriving in interleaved order and a second consumer has been added. Consumer 1 asks for a group of 10 messages and receives 10 Group A messages. Consumer 2 then asks for 10 messages but SQS knows that Group A is in flight, so it releases 10 Group B messages. The two consumers are now processing two batches of messages in parallel, speeding up throughput and then deleting their batches. When Consumer 1 requests the next batch of messages, it receives the remaining two messages, one from Group A and one from Group B.

Consider this nuanced detail from the example above. What would happen if Consumer 1 was on a faster server and processed its first batch of messages before Consumer 2 could mark its messages for deletion? See if you can predict the behavior before looking at the answer.

If Consumer 2 has not deleted its Group B messages yet when Consumer 1 asks for the next batch, then the FIFO queue considers Group B to still be in flight. It does not release any more Group B messages. Consumer 1 gets only the remaining Group A message. Later, after Consumer 2 has deleted its first batch, the remaining Group B message is released.

Conclusion

I hope this post answered your questions about how Amazon SQS FIFO queues work and why message groups are helpful. If you’re interested in exploring SQS FIFO queues further, here are a few ideas to get you started:

credits: aws.amazon.com

This Post Has 225 Comments
  1. Thanks for your marvelous posting! I genuinely enjoyed reading it, you can be a great author.I will always bookmark your
    blog and will often come back in the foreseeable future. I want
    to encourage one to continue your great work, have a nice evening!

  2. Big Game Toys~LITE Brite Magic Screen Light Bright (Premium Pack) …. compatible), and the paper is too thick to punch through and impossible to line up. … I had one of these as a kid, about 15 years ago? and they were really cool and fun.

  3. I was very ppleased too discover this page. I wanted to thank you foor your
    time just for this wonderful read!! I definitely appreciated
    everry part of it and i also have you book-marked to see nnew information oon your site.

  4. Thank you for some other informative website. The place else may just I
    am getting that kind of information written in such an ideal way?

    I have a project that I’m simply now working on, and I have been at the
    look out for such info.

  5. First of all I want to say excellent blog! I had a
    quick question in which I’d like to ask if you don’t mind.
    I was curious to find out how you center yourself and clear
    your thoughts prior to writing. I’ve had a hard time clearing my thoughts in getting
    my ideas out there. I do take pleasure in writing however it just
    seems like the first 10 to 15 minutes tend to
    be lost just trying to figure out how to begin. Any recommendations or tips?
    Kudos!

  6. I was recommended this website by my cousin. I’m not sure whether this post is written by him as nobody else know such detailed about my trouble. You are incredible! Thanks!

  7. Hey are using WordPress for your blog platform? I’m new to the blog world but I’m trying to get started
    and set up my own. Do you require any html coding expertise to
    make your own blog? Any help would be greatly appreciated!

  8. Hi! I’ve been following your blog for a while now and finally got the courage to go ahead
    and give you a shout out from Huffman Texas! Just wanted to tell you keep up the great
    work!

  9. Thanks for one’s marvelous posting! I seriously enjoyed reading it, you could be a great author.

    I will be sure to bookmark your blog and will eventually come
    back from now on. I want to encourage you to ultimately continue your great posts, have a nice evening!

  10. I think this is among the most important information for me.
    And i am glad reading your article. But wanna remark on few general
    things, The site style is great, the articles
    is really excellent : D. Good job, cheers

  11. Excellent blog here! Also your website rather a lot up very fast!

    What host are you the usage of? Can I get your associate link on your host?

    I wish my website loaded up as fast as yours lol

  12. This articlepostpiece of writingparagraph providesoffersgivespresents clear idea fordesigned forin favor ofin support of the new userspeopleviewersvisitors of blogging, that reallyactuallyin facttrulygenuinely how to do bloggingblogging and site-buildingrunning a blog.

  13. Hmm it looks like your blog ate my first comment (it was extremely long) so I guess I’ll just sum it
    up what I had written and say, I’m thoroughly enjoying
    your blog. I as well am an aspiring blog blogger but I’m
    still new to everything. Do you have any tips for inexperienced
    blog writers? I’d really appreciate it.

  14. GreatWonderfulFantasticMagnificentExcellent goods from you, man. I’veI have understand your stuff previous to and you’reyou are just tooextremely greatwonderfulfantasticmagnificentexcellent. I reallyactually like what you’veyou have acquired here, reallycertainly like what you’reyou are statingsaying and the way in which you say it. You make it entertainingenjoyable and you still take care ofcare for to keep it smartsensiblewise. I cantcan notcan’t wait to read far moremuch more from you. This is actuallyreally a terrificgreatwonderfultremendous websitesiteweb site.

  15. You are so interesting! I don’t believe I have read through a single thing like that before.

    So nice to discover somebody with a few unique thoughts on this topic.
    Really.. thank you for starting this up. This website is something that’s needed on the web, someone with a bit of
    originality!

  16. Admiring the persistence you put into your website and detailed information you offer. It’s great to come across a blog every once in a while that isn’t the same outdated rehashed information. Great read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.

  17. [url=https://tadacip2019.com/]tadacip[/url] [url=https://celebrex200.com/]celebrex 200 mg[/url] [url=https://albendazoletablets.com/]albendazole 200mg[/url] [url=https://antabuse250.com/]order antabuse no prescription[/url] [url=https://buyzithromaxonline.com/]zithromax[/url]

  18. Howdy! Do you know if they make any plugins to protect against
    hackers? I’m kinda paranoid about losing everything I’ve worked hard on.
    Any recommendations?

  19. Fantastic beat ! I wish to apprengice at the ame time as you amend your site, how can i subscribe for a
    blog web site? The account aided mee a appropriate deal. I had been tiny bit
    acquainted of thiis your broadcast offered bright clear idea

  20. [url=https://tadacip2019.com/]tadacip 20[/url] [url=https://celebrex200.com/]celebrex generic[/url] [url=https://albendazoletablets.com/]albendazole 200mg[/url] [url=https://buyzithromaxonline.com/]zithromax[/url] [url=https://antabuse250.com/]antabuse tablets[/url]

  21. [url=http://atenololmedication.com/]buy atenolol[/url] [url=http://allopurinol100.com/]allopurinol drug[/url] [url=http://furosemide20.com/]furosemide[/url] [url=http://tadalafil5.com/]generic tadalafil[/url] [url=http://cafergotmedication.com/]cafergot pills[/url]

  22. I amI’m extremelyreally impressed with your writing skills and alsoas well as with the layout on your blogweblog. Is this a paid theme or did you customizemodify it yourself? Either wayAnyway keep up the niceexcellent quality writing, it’sit is rare to see a nicegreat blog like this one these daysnowadaystoday.

  23. I’m really impressed with your writing skills and also with the format in your weblog.
    Is that this a paid theme or did you customize it your self?
    Anyway keep up the nice quality writing, it’s uncommon to look a great weblog
    like this one today..

  24. Hi there! I could have sworn I’ve been to your blog before but after going through many of the articles I realized it’s new to me.
    Anyways, I’m certainly happy I stumbled upon it and I’ll be book-marking
    it and checking back regularly!

Leave a Reply

Your email address will not be published.

Back To Top