Incomplete S3 Multipart Uploads

Why This Matters

Incomplete multipart uploads create invisible waste - you're paying Standard S3 storage costs for data that doesn't exist as accessible files. This is especially common in CI/CD environments where network issues cause upload failures, leaving behind orphaned storage parts that accumulate costs indefinitely.

Common causes:

• CI/CD pipelines with unstable network connections
• Backup systems with inadequate error handling
• Applications uploading large files without proper retry mechanisms
• Network timeouts during large file transfers

How to Identify Incomplete Uploads

AWS Saver flags buckets with incomplete multipart uploads when they contain:

1. Active multipart uploads with orphaned parts
2. Storage costs accumulating from incomplete upload parts
3. Failed upload attempts from CI/CD or application failures
4. No lifecycle policies to automatically clean up incomplete uploads

How to Fix Incomplete S3 Uploads

Step 1: List all buckets to check

aws s3api list-buckets --query 'Buckets[].Name'

Step 2: Check each bucket for incomplete uploads

aws s3api list-multipart-uploads --bucket your-bucket-name

Step 3: Abort specific incomplete upload

aws s3api abort-multipart-upload \
  --bucket your-bucket-name \
  --key your-file-key \
  --upload-id your-upload-id

Step 4: Abort all incomplete uploads for a bucket (batch cleanup)

aws s3api list-multipart-uploads --bucket your-bucket-name \
  --query 'Uploads[].{Key:Key,UploadId:UploadId}' | \
jq -r '.[] | "aws s3api abort-multipart-upload --bucket your-bucket-name --key \"" + .Key + "\" --upload-id " + .UploadId'

Step 5: Create lifecycle policy for automatic cleanup

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-bucket-name \
  --lifecycle-configuration file://cleanup-policy.json

Example cleanup-policy.json:

{
  "Rules": [{
    "ID": "cleanup-incomplete-uploads",
    "Status": "Enabled",
    "AbortIncompleteMultipartUpload": {
      "DaysAfterInitiation": 7
    }
  }]
}

Prevention Tips

Implement lifecycle policies: Set up automatic cleanup of incomplete uploads after 7 days on all buckets.

Add retry logic: Implement proper retry mechanisms in applications doing multipart uploads.

Monitor upload failures: Set up CloudWatch alarms for high multipart upload failure rates.

CI/CD best practices: Use AWS CLI with retry configurations and timeout handling for large file uploads.

Automation Available

Skip the manual work. AWS Saver automatically scans S3 buckets for incomplete multipart uploads with the following limitations:

✅ Limited bucket scanning - Checks up to 50 buckets for incomplete upload parts

✅ Cost impact estimation - Estimates monthly waste from orphaned uploads using 2.3GB average size

✅ Upload count detection - Counts incomplete multipart uploads per bucket

⚠️ Basic recommendations - Suggests manual cleanup (automatic lifecycle policy setup not included)