AWS Integration & Messaginng

SQS, SNS & Kinesis

• When we start deploying multiple applications, they will inevitably need to communicate with one another
• There are two patterns of application communication
  • Synchromous communication (app to app)
  • Asynchronous / Evenbased (app to queue to app)

    Introdution

• Synchronnous between applications can be problematic if there are sudden spikes of traffic
• What if you need to suddenly encode 1000 videos but usually it’s 10?
• in that case, it better to decouple your applications,
  • SQS: queue model
  • SNS: pub/sub model
  • Kinesis: real-time streaming model
• These services can scale independently from our application!

  Amazon SQS, What’s a queue?

  

  SQS - Standard queue

• Oldest offering (over 10 years old)
• Fully managed service, used to decouple applications
• Attributes:
  • unlimited throughput, unlimited number of messages in queue
  • Default retention of message: 4 days, maxximum of 14 days
  • Low latenccy (< 10ms on publish and receive)
  • Limitation of 256kb per message sent
• Can have duplicate messages (at least once delivery, occasionally)
• Can have out of order messages (best effort ordering)

  SQS - product messages

• Produced to SQS using the SDK (SendMessage API)
• The message is persited in SQS until consumer deletes it
• Message retention: default 4 days, upto 14days
• Example:
  • order id
  • customer id
  • any attributes you want
• SQS standard: unlimited throughput

  SQS - Consuming messages

• Consumes (running on EC2 instances, Servers, or AWS lambda)…
• Poll SQS for messages (receive up to 10 messages at a time)
• Process the messages (example: insert the message inyo an RDS database)
• Delete the messages using thhe deleteMessage API

  SQS - Mutiple EC2 Innstances Consumers

• Consumers receive and process messages in parallel
• At least once delivery
• Best effort message ordering
• Consumers delete messages after processig them
• We can scale consumers horzontally to improve throughput of processing

  SQS with auto scaling group (ASG)

  

  SQS to decouple between appliccation tiers

  

  Amazon SQS - Security

• Encryption
  • In-flight encryption using HTTPS API
  • At rest encryption using KMS keys
  • Client side encrytion if the client wants to perform encryptioon/decrytion itself
• Access Controls: IAM policies too regulate access to SQS API
• SQS Access Policies (similiar to S3 bucket policies)
  • Useful for cross account access to SQS queues
  • Useful for allowinng other services (SNS, S3…) to write to an SQS queue

    SQS Queue Access Policy

• Cross Acccount Access
• Publish S3 Event Notifications to SQS Queue

  SQS Message Visiblity Timeout

• After a message is polled by a consumer, it becomes invisible to other consumers
• By default, the “message visibility timeout” is 30 seconds
• That means the message has 30 seconnds to be processed
• After the message visibility timeout is over, the message is “visible” in SQS
• If a message is not processed within the visibility timeout, it will be processed twice
• A consumer could call the ChangelMessageVisibility API tto get more time
• If visibility timeout is high (hours), and consumer crashes, re-processing will take ttime
• If visiblity ttimeouut is too low (seconds), we may get duplicates

  Amazon SQS - Dead Letter Queue

• If a consumer fails to process a message within the visibility timeout… the message goes back to the queue!
• We can set a threshold of how manny times a message can go back to the queue
• After tthe mmaximumreceivves tthreshold is exceeded, the message goes intto a dead lettter queue (DLQ)
• Userful for debugginng!
• Make sure to process the messages in the DLQ before they expire:
  • Good to sett a rettenttion oof 14days in the DLQ

    SQS DLQ - Redrive to Source

• Feature to help consume messages in the DLQ to understand what is wrong with them.
• When our code is fixed, we can redrive the messages from the DLQ backk into the source queue (or any otther queue) inn batches without writing custtom code

  Amazon SQS - Delay Queue

• Delay a message (consummers don’t see it immediately) up to 15 minuttes
• Defaultt is 0 secconnds (message is available rightt away)
• Can set a defaultt at queue level
• Can override tthe defaultt o sennd using the DelaySecccond parameters

  Amazon SQS - Long Polling

• When a consumer requests messages from the queue, it can optioally “wait” for messages to arrive if tthere are nonne in the queue
• This is called long pollinng
• LongPolling dereases the nuumber of API calls made tto SQS while increasign tthe efficiency and reduinng lattenccy of your applicationn
• The wait time can be between 1 se tto 20 sec (20 sec preferable)
• Lonng polling is preferable to short polling
• long polling level using waittimeseconnds