Skip to main content
Use S3 Connector to ingest documents from S3 object storage (AWS S3, MinIO, B2, R2, etc).

​
What It Does

  • reads files from one or more buckets
  • converts content to Markdown for indexing
  • runs ingestion on configurable schedules

​
Required Environment Variables

Set these in .env.rag:
  • S3_ACCOUNT1_ENDPOINT: S3 API endpoint URL
  • S3_ACCOUNT1_ACCESS_KEY: access key ID
  • S3_ACCOUNT1_SECRET_KEY: secret key
  • S3_ACCOUNT1_REGION: bucket region
  • S3_ACCOUNT1_USE_SSL: True or False
  • S3_ACCOUNT1_BUCKETS: bucket name(s), comma-separated for multiple
  • S3_ACCOUNT1_SCHEDULES: ingestion interval(s) in seconds (default is 3600)

​
config.yaml Example

sources:
  - type: "s3"
    name: "account1"
    config:
      endpoint: "${S3_ACCOUNT1_ENDPOINT}"
      access_key: "${S3_ACCOUNT1_ACCESS_KEY}"
      secret_key: "${S3_ACCOUNT1_SECRET_KEY}"
      region: "${S3_ACCOUNT1_REGION}"
      use_ssl: "${S3_ACCOUNT1_USE_SSL}"
      buckets: "${S3_ACCOUNT1_BUCKETS}"
      schedules: "${S3_ACCOUNT1_SCHEDULES}"

​
Configuration Reference

FieldRequiredDefaultDescription
endpointyes—S3 API endpoint URL
access_keyyes—Access key ID
secret_keyyes—Secret access key
regionyes—Bucket region
use_sslnoTrueWhether to use HTTPS
bucketsyes—Bucket name(s), comma-separated
schedulesno3600Ingestion interval in seconds
request_delayno0Seconds to wait between API requests. Increase to avoid rate-limiting (e.g. 0.1)

​
Multiple S3 Accounts

Add more sources entries (account2, account3, etc) with separate env vars per account.