AWS API Gateway

• AWS Lambda + API Gateway: No infrastructure to manage
• Support for the WebSocket Protocol
• Handle API versioning (v1, v2…)
• Handle different environments (dev, test, prod…)
• Handle security (Authentication and Authorization)
• Create API keys, handle request throttling
• Swagger / Open API import to quickly define APIs
• Transform and validate requests and responses
• Generate SDK and API specifications
• Cache API responses

API Gateway – Integrations High Level

• Lambda Function
  • Invoke Lambda function
  • Easy way to expose REST API backed by AWS Lambda
• HTTP
  • Expose HTTP endpoint in the backend
  • Example: internal HTTP API on premise, Application Load Balancer…
  • Why? Add rate limiting, caching, user authentications, API keys, etc…
• AWS Service
  • Expose any AWS API through the API Gateway?
  • Example: start an AWS Step Function workflow, post a message to SQS
  • Why? Add authentication, deploy publicly, rate control…

API Gateway - Endpoint Types

• Edge-Optimized (default): For global clients
  • Requests are routed through the CloudFront Edge locations (improves latency)
  • The API Gateway still lives in nly one region
• Regional:
  • For client within the same region
  • Cloud manually combine with CloudFront (more control over the caching strategies and the distribution)
• Private:
  • Can only be accessed from your VPC using an interface VPC endpint (ENI)
  • Use a resource policy to define access

API Gateway – Security: IAM Permissions

• Create an IAM policy authorization and attach to User / Role
• API Gateway verifies IAM permissions passed by the calling application
• Good to provide access within your own infrastructure
• Leverages “Sig v4” capability where IAM credential are in headers

API Gateway – Security: Lambda Authorizer (formerly Custom Authorizers)

• Uses AWS Lambda to validate the token in header being passed
• Option to cache result of authentication
• Help to use Oauth / SAML / 3rd party type of authentication
• Lambda must return an IAM policy for the user

API Gateway – Security: Cognito User Pools

• Cognito fully manages user lifecycle
• API gateway verifies identity automatically from AWS Cognito
• No custom implementation required
• Cognito only helps with authentication, not authorization

API Gateway – Security – Summary

1. IAM
   • Greate for users/ roles already within your AWS account
   • Handle authentication + authorization
   • Leverages Sig v4
2. Custom Authorizer:
   • Greate for 3rd party tokens
   • Very flexible in terms of what IAM policy is returned
   • Handle Authentication + Authorization
   • Pay per Lambda invocation
3. Cognito User Pool
   • you manage your user pool (can be backed by Facebookm Google login etc…)
   • No need to write any custom code
   • Must implement authorization in the backend

AWS Integration & Messaginng

SQS, SNS & Kinesis

• When we start deploying multiple applications, they will inevitably need to communicate with one another
• There are two patterns of application communication
  • Synchromous communication (app to app)
  • Asynchronous / Evenbased (app to queue to app)

Introdution

• Synchronnous between applications can be problematic if there are sudden spikes of traffic
• What if you need to suddenly encode 1000 videos but usually it’s 10?
• in that case, it better to decouple your applications,
  • SQS: queue model
  • SNS: pub/sub model
  • Kinesis: real-time streaming model
• These services can scale independently from our application!

Amazon SQS, What’s a queue?

SQS - Standard queue

• Oldest offering (over 10 years old)
• Fully managed service, used to decouple applications
• Attributes:
  • unlimited throughput, unlimited number of messages in queue
  • Default retention of message: 4 days, maxximum of 14 days
  • Low latenccy (< 10ms on publish and receive)
  • Limitation of 256kb per message sent
• Can have duplicate messages (at least once delivery, occasionally)
• Can have out of order messages (best effort ordering)

SQS - product messages

• Produced to SQS using the SDK (SendMessage API)
• The message is persited in SQS until consumer deletes it
• Message retention: default 4 days, upto 14days
• Example:
  • order id
  • customer id
  • any attributes you want
• SQS standard: unlimited throughput

SQS - Consuming messages

• Consumes (running on EC2 instances, Servers, or AWS lambda)…
• Poll SQS for messages (receive up to 10 messages at a time)
• Process the messages (example: insert the message inyo an RDS database)
• Delete the messages using thhe deleteMessage API

SQS - Mutiple EC2 Innstances Consumers

• Consumers receive and process messages in parallel
• At least once delivery
• Best effort message ordering
• Consumers delete messages after processig them
• We can scale consumers horzontally to improve throughput of processing

SQS with auto scaling group (ASG)

SQS to decouple between appliccation tiers

Amazon SQS - Security

• Encryption
  • In-flight encryption using HTTPS API
  • At rest encryption using KMS keys
  • Client side encrytion if the client wants to perform encryptioon/decrytion itself
• Access Controls: IAM policies too regulate access to SQS API
• SQS Access Policies (similiar to S3 bucket policies)
  • Useful for cross account access to SQS queues
  • Useful for allowinng other services (SNS, S3…) to write to an SQS queue

SQS Queue Access Policy

• Cross Acccount Access
• Publish S3 Event Notifications to SQS Queue

SQS Message Visiblity Timeout

• After a message is polled by a consumer, it becomes invisible to other consumers
• By default, the “message visibility timeout” is 30 seconds
• That means the message has 30 seconnds to be processed
• After the message visibility timeout is over, the message is “visible” in SQS
• If a message is not processed within the visibility timeout, it will be processed twice
• A consumer could call the ChangelMessageVisibility API tto get more time
• If visibility timeout is high (hours), and consumer crashes, re-processing will take ttime
• If visiblity ttimeouut is too low (seconds), we may get duplicates

Amazon SQS - Dead Letter Queue

• If a consumer fails to process a message within the visibility timeout… the message goes back to the queue!
• We can set a threshold of how manny times a message can go back to the queue
• After tthe mmaximumreceivves tthreshold is exceeded, the message goes intto a dead lettter queue (DLQ)
• Userful for debugginng!
• Make sure to process the messages in the DLQ before they expire:
  • Good to sett a rettenttion oof 14days in the DLQ

SQS DLQ - Redrive to Source

• Feature to help consume messages in the DLQ to understand what is wrong with them.
• When our code is fixed, we can redrive the messages from the DLQ backk into the source queue (or any otther queue) inn batches without writing custtom code

Amazon SQS - Delay Queue

• Delay a message (consummers don’t see it immediately) up to 15 minuttes
• Defaultt is 0 secconnds (message is available rightt away)
• Can set a defaultt at queue level
• Can override tthe defaultt o sennd using the DelaySecccond parameters

Amazon SQS - Long Polling

• When a consumer requests messages from the queue, it can optioally “wait” for messages to arrive if tthere are nonne in the queue
• This is called long pollinng
• LongPolling dereases the nuumber of API calls made tto SQS while increasign tthe efficiency and reduinng lattenccy of your applicationn
• The wait time can be between 1 se tto 20 sec (20 sec preferable)
• Lonng polling is preferable to short polling
• long polling level using waittimeseconnds

Amazon SQS – Dead Letter Queue

• If a consumer fails to process a message within the Visibility Timeout…
• We can set a threshold of how many times a message can go back to the queue
• After the maximumReceives threshold is exceeded, the message goes into a dead letter queue (DLQ)
• Useful for debugging!
• Make sure to process the message in the DLQ before they expire:
  • Good to set a retention of 14days in the DLQ

SQS DLQ – Redrive to Source

• Redrive to Source
• Feature to help consume messages in the DLQ to understand what is wrong with them
• When our code is fixed, we can redrive the messages from the DLQ back into the source queue (or any other queue) in batches without writing custom code

Amazon SQS – Delay Queue

• Delay a message up to 15minutes
• Default is 0 seconds
• Can set a default at queue level
• Can override the default on send using the delayseconds parameter

Amazon SQS - Long Polling

• When a consumer requests messages from the queue, it can optionally “wait” for messages to arrive if there are none in the queue
• This is called long polling
• Longpolling decrease the number of API calls mafe to SQS while increasing the
• The wait time can be between 1 sc to 20 sec
• Long polling is preferable to short polling
• Long polling can be enabled at the queue level or at the API level using WaitTimeSeconds

SQS – Request-Response Systems

• To implement this pattern: use the SQS Temporary Queue Client
• It leverages virtual queues instead of creating / deleting SQS queues (cost-effective)

Amazon SQS – FIFO Queue

• FIFO = First In First Out (ordering of messages in the queue)
• Limited throughput: 300 msg/s without batching, 3000 msg/s with
• Exactly-once send capability (by removing duplicates)
• Messages are processed in order by the consumer

Kinesis Overview

• Makes it easy to collect, process, and analyze streaming data in real-time
• Ingest real-time data such as: Application logs, Metrics, Website clickstreams, IoT telemetry data…
• Kinesis Data Streams: capture, process, and store data streams
• Kinesis Data Firehose: load data streams into AWS data stores
• Kinesis Data Analytics: analyze data streams with SQL or Apache Flink
• Kinesis Video Streams: capture, process, and store video streams

Kinesis Data Streams

• Retention between 1 day to 365 days
• Ability to reprocess (replay) data
• Once data is inserted in Kinesis, it can’t be deleted (immutability)
• Data that shares the same partition goes to the same shard (ordering)
• Producers: AWS SDK, Kinesis Producer Library(KPL), Kinesis Agent
• Consumers:
  • Write your own: Kinesis Client Library (KCL), AWS SDK
  • Managed: AWS Lambda, Kinesis Data Firehose, Kinesis Data Analytics

Kinesis Data Streams – Capacity Modes

• Provisioned mode:
  • You choose the number of shards provisioned, scale manually or using API
  • Each shard get 1MB/s in (or 1000 records per second)
  • Each shard get 2MB/s out (classic or enhanced fan-out consumer)
  • You pay per shard provisioned per hour
• On-demand mode:
  • No need to provision or manage the capacity
  • Default capacity provisioned (4 MB/s in or 4000 records per second)
  • Scales automatically based on observed throughput peak during the last 30days
  • Pay per stream per hour & data in/out per GB

Kinesis Data Firehose

• Fully managed service, no administration, automatic scaling, serverless
  • AWS: Redshift / Amazon S3 / ElasticSearch
  • 3rd party partner: Splunk / MongoDB / DataDog / NewRelic / …
  • Custom: send to any HTTP endpoint
• Pay for data going through Firehose
• Near Real Time
  • 60 seconds latency minimum for non full batches
  • Or minimum 32 MB of data at a time
• Supports many data formats, conversions, transformations, compression
• Supports custom data transformations using AWS Lambda
• Can send failed or all data to a backup S3 bucket

Kinesis Data Streams vs Firehose

Kinesis Data Streams

• Streaming service for ingest at scale
• Write custom code (producer / consumer)
• Realtime (~200ms)
• Manage scaling (shard splitting / merging)
• Data storage for 1 to 365 days
• Suports replay capability

Kinesis Data Firehose

• Load streaming data into S3 / Redshift / ES / 3 rd party / custom HTTP
• Fully managed
• Near real-time (buffer time min. 60 sec)
• Automatic scaling
• No data storage
• Doesn’t support replay capability

Kinesis Data Analytics (SQL application)

• Perform real-time analytics on Kinesis Streams using SQL
• Fully managed, no servers to provision
• Automatic scaling
• Real-time analytics
• Pay for actual consumption rate
• Can create streams out of the real-time queries
• Use cases:
  • Time-series analytics
  • Real-time dashboards
  • Real-time metrics

Ordering data into Kinesis

• Imagine you have 100 trucks (truck_1, truck_2,.., truck_100) on the road sending their GPS positions regularly into AWS
• You want to consume the data in order for each truck, so that you can track their movment accurately
• How should you send that data into Kinesis?
• Answer: send using a “Partition Key” value of the “truck_id”
• The same key will always go to the same shard

Ordering data into SQS

• For SQS standard, there is no ordering
• For SQS FIFO, if you don’t use a Group ID, messages are consumed in the order they are sent, with only one consumer
• You want to scale the number of consumers, but you want messages to be “grouped” when they are related to each other
• Then you use a Group ID (similar to Partition Key in Kinesis)

Kinesis vs SQS ordering

• Let’s assume 100 trucks, 5 kinesis shards, 1 SQS FIFO
• Kinesis Data Streams: (500 consumer)
  • On average you’ll have 20 trucks per shard
  • Trucks will have their data ordered within each shard
  • The maximum amount of consumers in parallel we can have is 5
  • Can receive up to 5 MB/s of data
• SQS FIFO
  • You only have one SQS FIFO queue
  • You will have 100 Group ID
  • You can have up to 100 Consumers (due to the 100 Group ID)
  • You have up to 300 messages per second (or 3000 if using batching)

Amazon SNS

• What if you want to send one message to many receivers?
• The “event producer” only sends message to one SNS topic
• As many “event receivers” (subscriptions) as we want to listen to the SNS topic notifications
• Each subscriber to the topic will get all the messages (note: new feature to filter messages)
• Up to 12,500,000 subscriptions per topic
• 100,000 topics limit

SNS integrates with a lot of AWS services

• Many AWS services can send data directly to SNS for notifications

Amazon SNS – How to publish

• Topic Publish (using the SDK)
  • Create a topic
  • Create a subscription (or many)
  • Publish to the topic
• Direct Publish (for mobile apps SDK)
  • Create a platform application
  • Create a platform endpoint
  • Publish to the platform endpoint
  • Works with Google GCM, Apple APNS, Amazon ADM…

Amazon SNS – Security

• Encryption:
  • In-flight encryption using HTTPS API
  • At-rest encryption using KMS keys
  • Client-side encryption if the client wants to perform encryption/decryption itself
• Access Controls: IAM policies to regulate access to the SNS API
• SNS Access Policies (similar to S3 bucket policies)
  • Useful for cross-account access to SNS topics
  • Useful for allowing other services ( S3…) to write to an SNS topic

SNS + SQS: Fan Out

• Push once in SNS, receive in all SQS queues that are subscribers
• Fully decoupled, no data loss
• SQS allows for: data persistence, delayed processing and retries of work
• Ability to add more SQS subscribers over time
• Make sure your SQS queue accesss policy allows for SNS to write

Application: S3 Events to multiple queues

• For the same combination of: event type (e.g. object create) and prefix (e.g. images/) you can only have one S3 Event rule
• If you want to send the same S3 event to many SQS queues, use fan-out

Application: SNS to Amazon S3 through Kinesis Data Firehose

• SNS can send to Kinesis and therefore we can have the following solutions architecture:

Amazon SNS – FIFO Topic

• FIFO = First In First Out (ordering of messages in the topic)
• Similar features as SQS FIFO:
  • Ordering by message group ID (all messages in the same group are ordred)
  • Deduplication using a Deduplication ID or Content Based Deduplication
• Can only have SQS FIFO queues as subscribers
• Limited throughput (same throughput as SQS FIFO)

SNS FIFO + SQS FIFO: Fan Out

• In case you need fan out + ordering + deduplication

SNS – Message Filtering

• JSON policy used to filter messages sent to SNS topic’s subscriptions
• If a subscription doesn’t have a filter policy, it receives every message

SQS vs SNS vs Kinesis

SQS

• Consumer “pull data”
• Data is deleted after being consumed
• Can have as many workers (consumers) as we want
• No need to provision throughput
• Ordering guarantes only on FIFO queues
• Individual message delay capability

SNS

• Push data to many subscribers
• Up to 12500000 subcribers
• Data is not persisted (lost if not delivered)
• Pub/sub
• Up to 100000 topics
• No need to provision throughput
• Integrates with SQS for fan-out architecture pattern
• FIFO capability for SQS FIFO

  Kinesis

• Standard: Pull data
  • 2Mb per shard
• Enhanced-fan out: push data
  • 2MB per shard per consumer
• Possibility to replay data
• Meant for real-time big data, analytics and ETL
• Ordering at the shard level
• Data expires aftr X days
• Provisioned mode or on-demand capacity mode

Amazon MQ

• SQS, SNS are “cloud-native” service, and they’re using proprietary protocols from AWS
• Traditional applications running from on-premises may use open protocols such as: MQTT, AMQP, STOMP, Openwire, WSS
• When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ
• Amazon MQ = managed Apache ActiveMQ
• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on a dedicated machine, can run in HA with failover
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)

Amazon MQ – High Availability

SERVERLESS AND APPLICATION SERVICES

• Tiered Architecture
• When we start deploying multiple applications, they will inevitably need to communicate with one another.
• There are two patterns of application communication.
  • Evolving with queues
  • Event-driven architecture

Amazon SQS – Standard Queue

• Oldest offering (over 10 years old)
• Fully managed service, used to decouple applications
• Attributes:
  • Unlimited throughput, unlimited number of messages in queue
  • Default retention of messages: 4 days, maximum of 14 days
  • Low latency (<10 ms on publish and receive)
  • Limitation of 256KB per message sent
• Can have duplicate messages (at least once delivery, occasionally)
• Can have out of order messages (best effort ordering)

SQS – Producing Messages

• Produced to SQS using the SDK (SendMessage API)
• The message is persisted in SQS until a consumer deletes it
• Message retention: default 4 days, up to 14 days
• Example: send an order to be processed
  • Order id
  • Customer id
  • Any attributes you want
• SQS standard: unlimited throughput

SQS – Consuming Messages

• Consumers (running on EC2 instances, servers, or AWS Lambda)…
• Poll SQS for messages (receive up to 10 messages at a time)
• Process the messages (example: insert the message into an RDS database)
• Delete the messages using the DeleteMessage API

Lambda: function as a service

1
2
3
4
5
6
7
8
9
- You are billed for the duration that a function runs
- The environment has a direct memory (indirect CPU) allocation
- Deployment a package with 50MB zipped and 250MB unzipped
- 512 MB storage available as /tmp
- Serverless application (S3, API Gateway, Lambda)
- File Processing (S3, S3 event, lamba)
- Database Triggers (DynamoDB, Streams, Lambda)
- Serverless CROn (EventBrige/CWEvents + Lambda)
- Realtime Stream Data Processing (kinesis + Lambda)

Lambda in depth:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
- Public Lambda:
![](https://data.terabox.com/thumbnail/9fb8e7f5d4deb514c210c436daa33b68?fid=4401547290288-250528-754626766169061&rt=pr&sign=FDTAER-DCb740ccc5511e5e8fedcff06b081203-AyXAee6ddgI%2bVvNASA%2fyuIDC100%3d&expires=8h&chkbd=0&chkv=0&dp-logid=125173180413113238&dp-callid=0&time=1667199600&size=c1920_u1080&quality=90&vuk=4401547290288&ft=image&autopolicy=1)
- Private Lambda:
![](https://data.terabox.com/thumbnail/33d179bf393395941a01511d2a744ce3?fid=4401547290288-250528-1097958099823614&rt=pr&sign=FDTAER-DCb740ccc5511e5e8fedcff06b081203-45Ti2FPZBOQcIaJFI2fcCqPuJCU%3d&expires=8h&chkbd=0&chkv=0&dp-logid=125190523939933141&dp-callid=0&time=1667199600&size=c1920_u1080&quality=90&vuk=4401547290288&ft=image&autopolicy=1)
    - Lambda run in VPC obey all VPC networking rules
- Lambda security:
    - Lambda execution roles are IAM roles attached to lambda functions which control the permissions the lambda function receives ...
    - Lambda resource policy controls what services and accounts can invoke lambda functions
- Lambda Logging:
    - Lambda uses Cloudwatch, cloudwatch logs & x-ray
    - Logs from lambda executions - cloudwatchlogs
    - Metrics - invocation success/failure, retries, latency ... stored in cloudwatch
    - lambda can be integrated with x-ray for distributed tracing
    - Cloudwatch logs requires permissions via execution role

Lambda in depth: Invocation

• Invocation:
• Synchronous
  • cli / api, client communicates with APIGW , proxied to lambda function
• Asynchronous (typically used when AWS services invoke lambda functions)
  • S3 isn’t waitting for any kind of response. The event is generated and S3 stops tracking
  • If processing of the event fails, lambda will retry between 0 and 2 time (configurable). Lambda handles the retry logic
  • The lambda function needs to be idempotent reprocessing a result should have the same end state
• Kinesis data stream: producers(telemetry)
• An execution context is the environment a lambda function run in. A cold start is a full creation and configuration including function code dowload.

Cloudwatch events and eventbridge

• If X happen, or at Y times .. do Z
• Eventbrigde is cloudwatch api v2
• a default event bus for account
• cloudwatch event has only one bus
• eventbrigde have additional bus
• rules match event
• route the events to +1 targets as Lambda CloudWatch Events and EventBridge have visibility over events generated by supported AWS services within an account. They can monitor the default account event bus - and pattern match events flowing through and deliver these events to multiple targets. They are also the source of scheduled events which can perform certain actions at certain times of day, days of the week, or multiple combinations of both - using the Unix CRON time expression format. Both services are one way how event driven architectures can be implemented within AWS.

SQS – Multiple EC2 Instances Consumers

• Consumers receive and process messages in parallel
• At least once delivery
• Best-effort message ordering
• Consumers delete messages after processing them
• We can scale consumers horizontally to improve throughput of processing