Working With S3 and Compatible Services
The S3 API has become the universal language for object storage. Learn it once, and you can work with AWS S3, Cloudflare R2, DigitalOcean Spaces, Backblaze B2, and many other services. This portability makes S3 knowledge incredibly valuable.
Basic Operations With Python
The boto3 library is the standard way to interact with S3-compatible services in Python. Here's how to set up a client:
import boto3
s3 = boto3.client('s3',
endpoint_url='https://s3.amazonaws.com', # Change for other providers
aws_access_key_id='YOUR_ACCESS_KEY',
aws_secret_access_key='YOUR_SECRET_KEY'
)
For Cloudflare R2, you'd change the endpoint to your R2 URL. For DigitalOcean Spaces, use their regional endpoint. The rest of your code stays the same.
Uploading and Downloading
Uploading a file is straightforward:
# Simple upload
s3.upload_file('local-photo.jpg', 'my-bucket', 'images/photo.jpg')
# Upload with metadata and permissions
s3.upload_file('local-photo.jpg', 'my-bucket', 'images/photo.jpg',
ExtraArgs={
'ContentType': 'image/jpeg',
'ACL': 'public-read'
}
)
Setting the correct ContentType matters — browsers use it to display files properly. The ACL (Access Control List) determines who can access the object.
Downloading works similarly:
s3.download_file('my-bucket', 'images/photo.jpg', 'downloaded.jpg')
Bucket Policies for Access Control
While ACLs control individual objects, bucket policies define rules for entire buckets. This JSON policy makes everything in the public/ prefix publicly readable:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-bucket/public/*"
}]
}
This pattern is common for static websites — public assets are readable, but uploads and private files remain protected.
Generating Presigned URLs
Sometimes you want to give temporary access without making objects public:
url = s3.generate_presigned_url('get_object',
Params={'Bucket': 'my-bucket', 'Key': 'images/photo.jpg'},
ExpiresIn=3600 # URL valid for 1 hour
)
The returned URL includes a signature that grants temporary access. After expiration, the URL stops working. This is perfect for sharing private files securely.