tl;dr: I’m not impressed with Thrust::VPS.

Greetings, introductions, background, etc

A virtual private server is pretty cool thing to have.  Something that is always connected is very nice to have for a lot of reasons, and it seems everyone has one these days; I’ve had one for a few years.  After hopping from a few different providers, my current vps host is Thrust::VPS, which appears to be owned by Damn::VPS.  I have the “Metropolis” package, which can be seen here.

Let’s talk about security

I normally ssh into my vps with keys, I do not permit root logins, however I did have my ssh-server on standard port 22.  fail2ban is an awesome install-it-and-forget-it security application that scans logs like /var/log/pwdfail, and /var/log/auth.log, and bans any IP that makes too many password failures. It updates firewall rules to reject the offending IP address.  Pretty sweet, and a good idea for anyone running an ssh server.

Noticing oddities

Occasionally, I’ve been in the middle of an ssh session and had it hung up.  No typing, no cursor movements, just…a stuck terminal.  The first time this happened, I gave it 5 minutes,tried to ssh in again, and was able to get in, no problem.  Cool, maybe it was a hiccup in the network on the server side, my side, or anything inbetween.  I didn’t notice any more issues that day.

The next day, it happened again.  I hadn’t restarted my vps in a while, I figured maybe that’d clear things up (I know, I can restart services, etc, but if I remember correctly, I had some security update for Debian Wheezy that needed me to restart. No big deal).  I logged into the control panel, and gave it a restart.  As soon as it was back up, I was back in.

It happened twice again before the end of the day, and I was quite frustrated, and had a hint of nervousness about security.  Time to investigate.

Wtf

A who showed no odd users.  A look at the process tree showed nothing out of the ordinary.  No odd users were created, my bash history seemed normal, authorized keys were all mine.  I planned on doing a chkrootkit, but I wanted to look at a few other things first.

Wtf: Part 2 – Here’s where it gets fun

A simple glance at my /var/log/auth.conf showed all logins are coming from the same source IP on the network, both valid and invalid.  They also happen to be on the same network as my vps.  I am .167, the source for all traffic is .130.  A quick check of the apache log shows all http traffic also originating from that IP.  sudo iptables -L shows nothing odd on my side.  I’ve basically concluded it isn’t my issue, but for some reason, my hosting provider is funneling all of my traffic through this .130 address.  When someone tries to bruteforce my ssh and fail2ban kicks in, it’s adding a ban to .130, which is exactly what is happening. Apparently that’s me, and that’s why I’m getting DoS’d from my own server.

Contacting the provider: Attempt 1

Not cool, Thrust::VPS, not cool.  Time to email them on what the hell this is.

Greetings,

I have a security concern with my VPS. I am:

Hostname: mc.[redacted].com
IP Address: 109.xxx.xxx.162

I’ve been having SSH drop my connections sporadically, and I was thinking it had to do with fail2ban putting connections in jails (as designed, if there’s bad login attempts).

After consulting my /var/log/fail2ban.log, I see http://pastebin.com/[redacted]

In addition, I see in /var/log/auth.log, I see http://pastebin.com/[redacted]

In doing testing, *EVERY* login attempt appears to be coming from the same IP, 109.xxx.xxx.130, regardless of where I’m logging in from (home, work, etc), and what that actual IP is. I had a friend try from across the country (from a different IP, ISP), and it was coming from the same 109.xxx.xxx.130 address in the logs. This effectively puts this IP address as the source of *ALL* traffic to my VPS, which is odd.

In doing a lookup on that IP address, it appears to belong to ThrustVPS http://www.ip-address.org/tracer/ip-whois.php.

I am wondering why *every connection* do my VPS appears to be going through this .130 address, and I am concerned for the security of my machine.  I have the machine powered off, for safety.  Is this a load-balancer on your side?  If it is, it effectively provides an easy to DoS any host on your network, which is unacceptable, as well as bad practice.

I am awaiting your response as fast as possible.

– 
- Anthony Hook

Provider’s Response 1

Hi,

Please be informed that its 109.xxx.xxx.130  the IP address of our host node. If you face any abuse from this IP, I hope some of the abuser may attack thru this gateway. I highly recommend you to disable the direct root login and if possible please add any user and then login as root.

Else the best way is to use ssh keys to login to your vps. Also please do update all the softwares that is running on your vps to upto date to avoid the security holes.
–
Get in touch: Twitter @thrustvps

If you have any concerns or comments please feel free to ask for your ticket to be esculated to management.

Ticket Details
===================
Ticket ID: ZEN-681952
Department: Support
Priority: Low
Status: Awaiting Customer Reply

Contacting the provder: Attempt 2

Praveen,

Be assured I have disabled root login, as well as I use keys to log in.

My concern is all of the traffic goes through the host, and appears to only come from this address (please see the auth logs).

This completely makes fail2ban ineffective, as well as kills ability to actually see who is trying to ‘attack’ my VPS, because *all* traffic looks like it’s coming from the .130 node, instead of where it is actually coming from.  I have never seen another virtual machine, or hosting provider do things this way.

Essentially, it looks like this ‘host node’ is trying to get into my sever, which locks out traffic from .130, which disconnects any sessions I have open.  This is unacceptable, and makes a DoS extremely effective, and easy, from anywhere.  I cannot tell what the source is that is actually trying to log in.

–

- Anthony Hook

Resolution

They ‘adjusted a setting,’ contacted me, and asked if everything was okay.  The issue was finally resolved, however I have since moved hosting providers.