Browsed by
Author: akinnard

AWS get the current users ARN – BASH

AWS get the current users ARN – BASH

I have already posted on how to get the AWS username from the users arn for both Ruby and Python. But what about BASH?

I love to use Bash to quickly whip something together and the awscli makes it super easy. However the get-caller-identity method was not introduced until version 1.10 of the cli. So you may need to upgrade your cli first. On a Mac/Linux desktop/server this is easy.

pip install --upgrade awscli

This should upgrade you to the latest version. At the time of this article it is/was 1.11.2.

Now the sts get-caller-identity should be working

aws sts get-caller-identity

It will return something like this:

{
"Account": "123456789012",
"UserId": "Abcdefg123456789XYZ01",
"Arn": "arn:aws:iam::123456789012:user/bob"
}

Now we can parse that, we can either use jq, or –query. I will show both.

First for jq, which is a favorite tool around my shop:

aws sts get-caller-identity --output json | jq -r '.Arn' | cut -f 2 -d '/'

It looks a little messy but works as long as you have jq installed, but what about –query:

aws sts get-caller-identity --output text --query 'Arn' | cut -f 2 -d '/'

I use –output text to eliminate the double quotes .

As you can see bash, IMHO, is a much easier tool to work with using the aws cli to quickly build small shell scripts.

 

AWS get the current users ARN – Ruby

AWS get the current users ARN – Ruby

I already wrote a post on how to do this using Python. But here is how to do the same thing in Ruby:

I write a lot of automation scripts. I switch back and forth using both Ruby and Python. So when using the aws sdk in a ruby script I want to know who is running the script. I like to use the aws profile(s) for key rather than having the keys stored in yet another place on the users machine. I normally either ask the user to enter the profile name or have it passed on the command line. The sdk offers a couple of ways to get at the AWS user associated with they kepair in the profile.

First there is the IAM get-user class, however this class requires that the profile have IAM access which most profiles should not have. So this is not a good way to get at this information.

The second way is using STS or Security Token Service . This API offers a method called GetCallerIdentity.  This method returns the Account, ARN and UserId for the aws credentials used to make the request. So let’s see how to do this in Ruby.

First we use the SharedCredentials API to create a new SharedCredentials object for the profile information:

myCredentials = Aws::SharedCredentials.new(profile_name: myProfile)

Next we create a new STS Client object using those credentials:

myStsClient = Aws::STS::Client.new(credentials: myCredentials)

And finally call the get_caller_identity method:

mySts = myStsClient.get_caller_identity()

That object then returns the following elements:

puts "My Account #{myStsClient.account}"
puts "My ARN #{myStsClient.arn"
puts "My User id #{myStsClient.user_id}"

Now if you use this you will see that the UserID is not what you were expecting. It returns the unique identifier for the profile. That is well and good, but I want to get the username. Fortunately it is part of the ARN. so we can split it out like so:

puts "My User #{myStsClient.arn.split('/')[-1]}"

Now we have something we can use.

Here it is all put together:

#!/usr/bin/env ruby
require 'aws-sdk'
require 'optparse'

options = {:myProfile => nil }

parser = OptionParser.new do|opts|
  opts.banner = "Sample STS Script [options]"
  opts.on('-m', '--my-profile myProfile', 'myProfile') do |myProfile|
    options[:myProfile] = myProfile;
  end

  opts.on('-h', '--help', 'Displays Help') do
    puts opts
    exit
  end
end

parser.parse!

myProfile = options[:myProfile]

myCredentials = Aws::SharedCredentials.new(profile_name: myProfile)
myStsClient = Aws::STS::Client.new(credentials: myCredentials)
mySts = myStsClient.get_caller_identity()

puts "My Account #{mySts.account}"
puts "My ARN #{mySts.arn}"
puts "My User id #{mySts.user_id}"
puts "My User #{mySts.arn.split('/')[-1]}"
AWS get the current users ARN – Python

AWS get the current users ARN – Python

In writing scripts it is good to know who is running them. I create a lot of AWS python scripts. These examples are using python3 and boto3. I prefer to use the aws config profile for creating the sessions. It is easier on users and allows for multiple key pairs to be used and switched out easily.

devAWSSession = boto3.Session(profile_name=args.devProfile)

That works great for setting up a session, but what if you want to log the user behind that profile? Sometimes the profile names are not helpful. (I am looking at you “default”) The aws iam command has a get_user function that works great, if you have iam access. What if you don’t want all of your profiles to have IAM access, and you shouldn’t.

AWS provides another set of classes called “sts” or “Security Token Service”. With this service you can call get_caller_identity. This will give you back the account number, arn, and userid. However that is not the userid is not the user name you would expect, it actually returns the unique AWS user id. But the username is part of the arn. First let’s get the data, using the session from above:

mySts = myAWSSession.client('sts').get_caller_identity()
myArn = mysts["Arn"]

Now we have the complete arn “arn:aws:iam::123456789012:user/Bob”. So now we can do a normal split and get Bob from the arn:

myUser = myArn.split('/')[-1]

Now myUser = Bob

Super simple and easy

import boto3
import argparse

parser = argparse.ArgumentParser()

parser.add_argument("-m", "--my-profile", dest = "myProfile", default = "default", help="My AWS Profile")

args = parser.parse_args()

myAWSSession = boto3.Session(profile_name=args.myProfile)

mySts = myAWSSession.client('sts').get_caller_identity()
myArn = mySts["Arn"]
myAccount = mySts["Account"]
myUser = myArn.split('/')[-1]

print("My profile user: {}".format(myUser))
How to increase the root volume size using the aws cli

How to increase the root volume size using the aws cli

I love using the AWS CLI. It makes everything completely recreatable. I can add the commands to a run book, and presto, next time we need to spin something up we have the commands. No fumbling through the GUI, wondering which security groups or subnets to use. When the gremlins attack you will be in a lot better place with the cli commands documented.

Recently when spinning up a server I immediately got a warning about disk space on the root partition. That is odd, it did not come up in dev, but then again we don’t have Datadog running for our dev instances. I checked it out and it turns out that we install a lot of packages in this recipe. Just enough to put us over the 80% mark. My standard root partition is 8G. So now what? I couldn’t just leave it so I needed to figure out how to increase the root volume. I terminated the instance, and went back into our dev environment.

I am used to doing block mappings, and add them all the time:

aws ec2 run-instances \
--subnet-id subnet-99999999 \
--security-group-ids sg-99999999 \
--image-id ami-99999999 \
--key-name mykey \
--iam-instance-profile Name=my-default-ec2-role \
--instance-type t2.micro \
--block-device-mappings '[{"DeviceName":"/dev/sdb","Ebs":{"VolumeSize":5}}]' \
--output json

With the key there being the block device mappings. All root volumes are ebs volumes now, so it should be easy enough to change that, right?

Not sure if this would work I tested this in our dev account. It turns out it really is that simple, Just add another block mapping  for xvda and boom. It works like a champ

--block-device-mappings '[{"DeviceName":"/dev/sdb","Ebs":{"VolumeSize":5}}, {"DeviceName": "/dev/xvda", "Ebs": { "VolumeSize": 12 }}]'

When the instance booted it took the new setting and made the root partition 12G instead of the 8G that is default.

One week in on my new standing desk

One week in on my new standing desk

I blogged last week about my $50 standing desk I built from Ikea. Today I will review how my first week has been,

My feet hurt. 

Not unexpected but after a week of standing for 8+ hours a day my feet are sore. Mostly on my heels. The pain is not unbearable. I have been able to push through it. The first day was the roughest. When I went to bed I could really feel it. I have found that wearing socks help. I have a chair mat under my desk which I also think is helping give some extra padding. I find myself shifting from side to side a lot. I think the soreness will subside over the next few weeks. But we will see.

fitbitWhere did all my steps go?

First off I love my fitbit. It has pushed me hard to stay more active. I had assumed that using a standing desk would mean more steps. But I was very wrong. On Monday after lunch I checked my step count and boy was I low. I find standing I take less stretching breaks so I got a lot less steps. I do get a lot more work done, but this meant extending my gym time on the treadmill just to get my steps in.

Forcing myself to stand. 

This has not been a problem for me. I did not buy a drafting chair on purpose. So if I do sit down it becomes very awkward. I do have my gaming computer next to my standing desk so I do sit to play games. If something happens at work I find I have to stand up or it just seems like I am a little child trying to use my parents computer. Barely able to see the keyboard and the bad angle for the monitor, if I don’t stand it puts a strain on my neck.

TL:DR
After one week I like it. I get more done. My feet are a little sore. But other wise it is all good. I am glad I made the switch.

Ikea $50 Standing Desk

Ikea $50 Standing Desk

With working from home for the past year and a half I have found I sit way too much. This is the second time in less than 6 months where I have thrown my back out. So I decided I would try a standing desk. I started with a Tripod mounted desk. My feet hurt after just a few hours, but I slowly adjusted. So I decided to do it with a real desk. How ever my office is too small to accommodate another desk. I love my current desk, It has held up well for the past 2 years. And that is saying something as I am usually very hard on desks. So I started looking at risers I could put on my desk. Those were either super expensive or not very good. Then I stumbled on an old blog post that described a $22 dollar Ikea desk . I liked the idea, but I thought I could tweak it a bit to fit my desk better.

First I wanted a longer and narrower table for my monitor. I own a 34″ monitor, and it would not look right on a small table top, Plus coming out 22″ from the back would take up most of my desk top. What I found instead was the LACK tv stand. Same height as the LACK side table, only longer and narrower. The TV stand was $15.

Next with the longer table I would need a longer shelf for my keyboard. This was also important to me as I keep a work journal and write in it all the time. Having room for it on my shelf was important to me. With the table 35″ wide I needed to get a longer shelf than was in the original post, I went with the next size up, an EKBY JÄRPEN. It is also $15.

This shelf is only designed to have brackets 31″ apart, so I knew I would need an additional bracket for the middle to support the shelf. The EKBY VALTER are perfect. This puts the keyboard at exactly the right height for my posture. So I bought 3 of them at $5 each. This brought my total to $45. I already had wood screws, so no extra costs there. I did pick up a pack of felt feet for $2, just so it did not scratch my desk

From there it was time to assemble. The LACK tv stand was super simple to put together. Once that was done I added two of the brackets. One to each of the legs.2016-08-21 12.12.56 Then I centered the shelf and started to mount it.

2016-08-21 12.13.18First thing I noticed was that it was very front heavy. It would be very easy to topple over. This is where I was glad I bought an extra bracket. I spun it around so it would not only support the extra length, but also prevent the whole thing from falling over on me.

The finished product sits perfectly for me.

2016-08-20 16.47.11

How to find unused RDS instances in AWS

How to find unused RDS instances in AWS

It is not uncommon to have developers run their own development servers in AWS. Sometimes things are too large to run locally. The problem is that those servers sit idle a lot and cost money doing so. This becomes especially true if you have a large database running on RDS.

I have scripted the deployment of a dev RDS database. With our snapshots that can take up to ten minutes to deploy. IMHO that is fast enough that you can spin one up when you need it. The argument that I am facing is that they “always” need the RDS database. I disagree and set out to prove this.

I believe in public shaming to change behavior. So I started sending slack messages when I would see a database that was up for an extended period of time. But that approach did not work and took a lot of my time. The argument that came back was that they were “using” the database during that time. This was a daily manual process which was not fun for me. So I set out to automate this and get a little creative as well.

The bash script I am going to explain here can be found on my public gist. First off why is this bash? I am using the aws cli for all the heavy lifting so it made sense. Plus I love me some bash.

First get all the databases that are “available”:

aws rds describe-db-instances --output text --query 'DBInstances[*].{DBInstanceIdentifier:DBInstanceIdentifier,InstanceCreateTime:InstanceCreateTime,DBInstanceStatus:DBInstanceStatus}' | grep available

Now that we have the list, let’s loop through it and use some cloudwatch metrics. Not many people realize the wealth of data in cloudwatch. In this script we are going to grab the “DatabaseConnections” metric. In my gist we are looking for how many database connections in the last hour. In my cron script I am looking over a 12 hour period. Kinda hard to say you needed that server when you have not even connected to it in the past 12 hours. You can change that as you need.

for j in $servers
do
 server=$(echo "$j" | cut -f1 -d$'\t')
 update=$(echo "$j" | cut -f3 -d$'\t')
 connections=$(aws cloudwatch get-metric-statistics --metric-name DatabaseConnections --start-time $STARTDATE --end-time $UTCDATE --period $period --namespace AWS/RDS --statistics Maximum --dimensions Name=DBInstanceIdentifier,Value=$server --output text --query 'Datapoints[0].{Maximum:Maximum}')
 if [ "$connections" == "0.0" ]
 then
 echo "Server $server has been up since $update"
 echo "There have been $connections maximun connections in the last hour"
 echo "To terminate this instance run one of the following commands:"
 echo "aws rds delete-db-instance --db-instance-identifier $server --final-db-snapshot-identifier ${server}-final-${MYDATE}"
 echo "aws rds delete-db-instance --db-instance-identifier $server --skip-final-snapshot"
 echo "---------------------------------------------------------------------------------"
 fi
done

In my production script I send the message to slack, but you can change that as you need. Now when this gets put in a public channel it is hard to defend.

 

 

Use your runlist as an attribute in a Chef recipe

Use your runlist as an attribute in a Chef recipe

Here is a corner case that came up recently for us in Chef. How do we set an attribute in a template to a value if a specific cookbook/role was used.

A little background. We use role cookbooks in Chef. This allows us to do all kinds of overrides and versioning. All of this works great. We gain all the benefits of  overriding attributes and versioning of the roles. So normally if we have a specific value needed in a “role” it is very easy. But we came up with an edge case that required a little deeper thinking.

With a lot of distributed systems one of ours requires an advertised hostname. Normally we just set this to the fqdn

advertised.host.name=<%= node['fqdn'] %>

But we came up to an instance where we want this set to “localhost”. The first idea was to create an additional attribute to store a state and but an if in the template checking for it. But that seemed like a lot. This is needed for a specific role, so can we just use the runlist to see if the role is in the runlist?

Turns out the answer is yes: node[‘recipes’] returns a list of the recipes in the expanded ran list. By expanded, I mean that if you use a traditional role and it contains 5 recipes, all five will be listed in this attribute. This list can be rather expansive. This is where a proper naming convention in your recipes and roles can help you. Because we do I was able to simply use the .include method on the string.

<% if node['recipes'].include? "my-role-cookbook" %>
advertised.host.name=localhost
<% else %>
advertised.host.name=<%= node['fqdn'] %>
<% end %>

This works great. If you are using traditional roles, you can do the same thing just change node[‘recipes’] to node[‘roles’].

<% if node['roles'].include? "my-role" %>
advertised.host.name=localhost
<% else %>
advertised.host.name=<%= node['fqdn'] %>
<% end %>