Useful AWS CLI Scripts that use JQ

Ever found yourself in the need to quickly print details or execute commands on multiple of the same type of AWS resource? Like, get the size of all your s3 buckets, all list/update all tags on your EC2 machines? AWS CLI is more than enough!

The following are a set of useful AWS CLI scripts that make use of either command querying or parse the json output using JQ to for loop a list of resources and execute modify commands or get additional details on them.

AWS CLI diagram
AWS CLI is a unified tool to manage your AWS services

All these assume you have a local install of both AWS CLI and JQ, and you have CLI configured with your account and region.

Code Completion for AWS CLI on your bash_profile

First, a useful script to allow aws cli code completion on your bash profile. Useful for when you don’t remember all the commands (Which are a lot)

#!/bin/bash
# Adds a tag to all kinesis streams on the default region.
# depends on AWS CLI and JQ
TAG_KEY="Hello"
TAG_VALUE="World"
for amiId in `aws ec2 describe-images –region us-east-1 –owners self –output json –query 'Images' | jq '.[] | .ImageId' -r`; do
instances=$(aws ec2 describe-instances –region us-east-1 –filters "Name=image-id,Values=${amiId}")
echo ${amiId} '|' ${instances}
done
echo "Done"

List all EMR Clusters along with the associated EC2 Instance Ids

This one will list all EMR Clusters along with the associated EC2 Instance Ids.
It will print the result in the format: # "cluster_id | [$ec2_instance-id]... "

#!/bin/bash
# lists all EMR Clusters along with the associated EC2 Instance Ids
# Use this directly on your command shell. It will print the result in the format:
# "cluster_id | [$ec2_instance-id]… "
# depends on AWS CLI and JQ
for cluster in `aws emr list-clusters –active –query 'Clusters[].Id' –output text`; do
instances=$(aws emr list-instances –cluster-id ${cluster} –query 'Instances[?Status.State==`RUNNING`].[InstanceGroupId, InstanceType]' | jq -r -c '.[] | @tsv')
echo ${cluster} '|' ${instances//$'\n'/ }
done

List all AMIs along with any associated instances

At work, we recently had the need to list all our unused AMI’s, making sure they are not being used by any instance or running resource. This is how you can get such a list

#!/bin/bash
# Adds a tag to all kinesis streams on the default region.
# depends on AWS CLI and JQ
TAG_KEY="Hello"
TAG_VALUE="World"
for amiId in `aws ec2 describe-images –region us-east-1 –owners self –output json –query 'Images' | jq '.[] | .ImageId' -r`; do
instances=$(aws ec2 describe-instances –region us-east-1 –filters "Name=image-id,Values=${amiId}")
echo ${amiId} '|' ${instances}
done
echo "Done"

Add Tags to a Kinesis Stream

Adding tags is a useful feature for organizing your resources. At my work, we use it for cost allocation across teams. This is a simple way to do so. Notice you can customize this one to work for any kind of resource that allow tagging.

#!/bin/bash
# Adds a tag to all kinesis streams on the default region.
# depends on AWS CLI and JQ
TAG_KEY="Hello"
TAG_VALUE="World"
for amiId in `aws ec2 describe-images –region us-east-1 –owners self –output json –query 'Images' | jq '.[] | .ImageId' -r`; do
instances=$(aws ec2 describe-instances –region us-east-1 –filters "Name=image-id,Values=${amiId}")
echo ${amiId} '|' ${instances}
done
echo "Done"

Get S3 Buckets Tags

Tags, or any details/Information, really. Just change or add the appropriate field on the query, and have it printed.

#!/bin/bash
# lists all buckets along with their tags in the following format:
# bucket_name | { tag_name: tag_value }
# depends on AWS CLI and JQ
for bucket in `aws s3api list-buckets | jq .Buckets[].Name -r`; do
tags=$(aws s3api get-bucket-tagging –bucket $bucket | jq -c '.[][] | {(.Key): .Value}' | tr '\n' '\t')
echo $bucket '|' $tags
done

If you are curious, there’s a lot more Gists on my Github profile.