Recovering From A Botched bashrc Change on DigitalOcean

So I made a slight error today that caused my personal blog to go down for a hot minute. Scared the crap out of me, and I wasn't completely sure how I'd recover from my mistake but I figured it out and all is well. Let's go through the journey together.

All of my sites that use my web app are built using Node (Next, specifically). As such, each time I need to update my site, I need to do the following:

1. ssh into my server
2. pull the updated code using git
3. re-build the site using npm
4. Reload the app using pm2

I've been doing this a lot over the last couple of weeks, and I'm at least moderately knowledgeable with bash, so I figured "hey, why not make a command that makes this a little easier?" I know I could make a workflow through GitHub, but eh, let's just start by making a quick little command that will do these four steps for me automatically.

So I went on my merry way, building the script and doing my thing. After building the script, I updated my .bashrc file to include the command in my list of commands, and apparently I did it wrong because doing so immediately caused me to get booted off the SSH connection, and any subsequent attempt instantly booted me as well.

Shit. Now what?

I started digging around, trying a few different things like accessing the recovery console, to no avail. I couldn't seem to find the actual bashrc file that I changed to revert what I did.

Then it occurred to me...what if I run a command over ssh without entering the interactive shell? If I do that, maybe it won't source the user's bacshrc, and won't break my connection. If so, Maybe I can use that to wrestle control back.

So, I stuck my toe in the water, and with the `-t` flag, I tried to run a basic echo call.

ssh -t user@site.com 'echo hi'

And to my delight, I got:

hi
connection closed

Which told me that my command ran before that connection error happened. A few commands later, I had figured out the directory of the bad script I wrote that was causing the problems and was able to remove it using rm -rf. Once that file was gone, I was able to connect to the server with the interactive shell. It complained that the file didn't exist anymore, but it let me in. One quick revert on the bashrc file and I was good to go. whew.

In hindsight, my error became clear. The script I wrote, and then linked in bash was just that - a script. I didn't put the script into a function or anything that was callable, and as a result, that script simply ran as soon as I logged in. Naturally, that script has some fail-safes in place for arguments, and if something went wrong, it bailed early using exit. So, I inadvertently created a script that would instantly close my SSH session when I started it up????.

Guess I'm just gonna have to do the GitHub flow instead, after all 'cause I'm probably not messing with that again. Although...at least I know how to fix it now ????, perhaps it would be safe to try again but, ya know, test my script before sourcing it.