AWS SQS

Configure an AWS SQS destination for the GoldRush Pipeline API to deliver structured blockchain data to standard or FIFO queues.

The AWS SQS destination delivers structured pipeline output as messages to an Amazon SQS queue. It supports both standard and FIFO queues with built-in deduplication.

Configuration

destination:
  type: "sqs"
  queue_url: "https://sqs.us-east-1.amazonaws.com/123456789/my-queue.fifo"
  region: "auto"
  access_key_id: "${AWS_ACCESS_KEY_ID}"
  secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  batch_size: 10
  message_group_id: ""

Fields

Field	Type	Required	Default	Description
queue_url	string	yes	—	SQS queue URL
region	string	yes	—	AWS region
access_key_id	string	yes	—	AWS access key
secret_access_key	string	yes	—	AWS secret key
batch_size	int	no	10	Messages per batch (max 10)
message_group_id	string	no	auto-derived	FIFO message group ID

FIFO Queue Detection

FIFO mode is automatically activated when either condition is met:

• The queue_url ends with .fifo
• A message_group_id value is explicitly set

When FIFO mode is active, the destination sets the MessageGroupId and MessageDeduplicationId on every message.

Message Format

Each message is a JSON object containing the project, stream name, and the record payload:

{
  "project": "analytics-prod",
  "stream": "hl_fills",
  "record": {
    "block_number": 12345,
    "...": "..."
  }
}

Message Attributes

Each SQS message includes the following message attributes:

Attribute	Type	Description
project	String	Project name
stream	String	Stream name
block_number	String	Block number, if present in the record

Deduplication

For FIFO queues, the deduplication ID is deterministically derived from source metadata using the format:

{topic}-{partition}-{offset}

This ensures that retried deliveries do not produce duplicate messages in the queue.