In the past I’ve written about how bastion hosts are trash and that we don’t need them anymore, but I overlooked RDS with that statement. Outside of using AWS CloudShell we do need a bastion host to connect to our RDS instances.
When using RDS on a private network we need to carry out tasks like connecting to a database tool such as MySQL Workbench on our local machine or deploying migrations to an RDS table in our CI/CD pipelines, and this connection has a dependency on using a Bastion Host. Whether we’re using Systems Manager Session Manager with port forwarding, Direct Connect, or AWS VPN, we cannot log directly into an RDS instance.
Why can’t we simply connect directly to an RDS instance?
If your RDS database is publicly accessible, you can connect to it directly, but in the majority of architectures your database will be in a private subnet with no internet access - or at least it probably should be. And when our RDS database is in a private subnet we need a bastion host to connect to and proxy requests to the RDS instance.
We cannot connect directly to a private RDS database due to the Shared Responsibility Model and the fact that RDS is an AWS managed service. There is no user we can access on an RDS instance, we depend on the APIs that AWS provide us with.
What is the current process for using a bastion host to connect to RDS?
There are other similar options available to us, but I believe the best and most secure option at the time of writing is to have a private EC2 instance as a bastion host in the same private subnet as our RDS instance. This means we do not need to expose our virtual machines publicly for either of our instances.
To connect to the bastion host we can leverage Systems Manager Session Manager with port-forwarding. The EC2 should have the SSM agent running, which is available by default on most modern Amazon Machine Images (AMIs). The SSM agent will establish a connection to AWS Systems Manager. We can then make requests to Systems Manager Session Manager which are then port-forwarded onto the bastion host, which subsequently allows us to establish a connection with the RDS database.
The process is impressive from a security perspective, as it allows us to avoid the use of key pairs and it doesn’t require public access to our virtual machines; that being said it is quite a complex process which isn’t documented particularly well. Luckily, Rami McCarthy has carried out much more research than I have and he has put together a great post showing the steps involved in much more detail, with some great links to additional resources, which can be found here: https://ramimac.me/rds-iam-auth
What existing tools can make our life easier?
I mentioned that the process can be complex to establish by ourselves, especially across a large organisation, but there is good news, there are third-party open source tools available we can leverage to make our life easier.
One of which is Basti, which can be found here: https://github.com/basti-app/basti
I was introduced to Basti by Rami, the author of the guide I linked to earlier, which is a tool that is useful in automating large parts of the current process. I will write a follow up post specifically around using Basti, as it will allow me to go into more detail.
How can this process be made easier?
I would love to see the AWS Systems Manager team who own Session Manager introduce a managed bastion host option, or maybe it can be achieved by the RDS team introducing a sidecar container to act as an ephemeral bastion host for connecting to our RDS databases in a private subnet.
Whichever option it is, it would be a great quality of life improvement for engineers, as the current solution requires a lot of configuration and architectural decisions to get it working successfully within an organisation. It’s even complicated and time consuming to achieve as an individual.
Luckily Basti facilitates bastion hosts being deployed for us and shutdown through the cleanup command, as configuring this manually is a challenge - we can then leverage SSM to carry out automatic patching on these instances when they’re shutdown.
By introducing a managed bastion host option teams can focus on the Identity and Access Management configuration to ensure only the right people have access to RDS databases when they need it, and not have to worry as much about the underlying networking.
For that reason it’s going on my wish list for AWS, hopefully one day a time will come when we don’t need to worry about bastion hosts anymore 🙏
