nick-web.co.uk

The blog of nick

Category: Hosting

  • Replacing Uptime Kuma with Gatus on a Tiny VPS for $1.20/Year

    I wanted a self-hosted, off-site uptime monitoring and alerting system. I ended up building one for around $1.20 (£0.90) per year.

    I was checking Uptime Kuma on my self-hosted home server and noticed the RAM usage was hovering around 160 MB, which for 8 checks seemed a bit excessive. After looking around I found Gatus. With identical checks configured and both actively polling, Gatus used ~19 MB. It covers everything I personally needed with HTTP(S) checks, pings, notifications, SSL expiry and a dashboard. Gatus provides a number of notification services, and I use it with ntfy.sh for instant alerts that meet the criteria I set for being notified. They have pre-built services for Slack, Discord, Telegram, E-Mail and many many more.

    Seeing how well Gatus worked, I started looking for the tiniest VPS I could run it on so that I could deploy it away from my homelab.

    I found one. 128 MB RAM. “Economy” CPU. 1 GB SSD disk space, however it uses shared NAT IPv4. This comes in at approx. 10¢ per month from TierHive.com

    This is an ultra-budget VPS and performance expectations should match the price. For my use it’s acceptable. At this price point, expect oversold nodes, noisy neighbours, weak disk performance, and occasional instability.

    I initially thought about running docker – and the Gatus image. I found the TierHive VPS after a trip to lowendtalk.com and checking the site I picked a Micro Server.

    I played around with the settings and selected

    Low CPU Priority
    Low Disk IOPS
    Low Network throughput

    and my reasons were pretty simple; it’s a low memory single application that does not tax the CPU. Waiting a few milliseconds for a slice of CPU time is not world ending. I don’t need fast network throughput – Gatus sends a HTTP(S) request and gets a reply, and logs it to a SQLite database. There is nothing in this stack that needs an ultra-mega-fast of any of those options. I installed the only viable (and allowable) OS – Alpine.

    I’ll save the next few paragraphs for another day but Docker repeatedly failed to start, likely due to the 128MB RAM limit. I didn’t spend too much time and energy trying to diagnose it – so I had to change my plan.

    The easiest documented route on the Gatus GitHub repo was Docker, so I built a static binary instead. I downloaded the source and cross compiled on my Mac. I confirmed the target architecture on the VPS with

    ~# cat /proc/cpuinfo
    processor : 0
    vendor_id : GenuineIntel
    cpu family : 6
    model : 158
    model name : Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
    stepping : 9
    microcode : 0xf8
    cpu MHz : 3503.998
    cache size : 16384 KB

    and then cross compiled with

    GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o gatus .

    and that gave me a single static binary. After uploading that to the VPS and creating the required config file, I was off to the races.

    NAT IPv4 is one of those things that sometimes, you need to deal with when delving into cheap, low end VPS’s. You don’t get a unique IPv4 linked straight to the VPS. Public IPv4 is shared via NAT amongst a few customers at once. This usually means you don’t get ports 80 and 443 directly assigned to you. You are assigned a few ports that only you can use with that IP. Some providers give you a couple of ports, others more. 20 ports is around the average and they will generally be all within a certain range – think high, non standard port numbers (30234 to 30254 etc.). This means you have to set up whatever service to listen on either the non-standard port or forward the assigned external port to the application’s internal port – and then you still have to call the IP with the port attached (i.e 203.0.113.10:30234). A bit ugly.

    TierHive (and this might be new as I hadn’t seen it before this) has a web interface allowing you to map a provided port to a local port on the VPS, essentially meaning I didn’t have to change the Gatus application. It fired up on it’s usual port (Port 8080), and I set up a forward so that port 30234 sent traffic to the Gatus port transparently.

    The final piece of the puzzle came with TierHive also providing a shared HAProxy frontend IP that routes requests by hostname allowing me to set up a custom A record at my DNS provider, pointing to the HAProxy IP. The HAProxy then pointed at the IP and port at the VPS, meaning I can now visit status.website.tld and see some uptime stats, but also get alerting if anything goes down.

    I’ve made this setup redundant with Gatus also running on my homelab. If the TierHive server goes down and can’t send notifications the homelab Gatus instance will. The chances of both Gatus instances going down at the same time are low. Gatus is very extensible and the documentation on GitHub is very thorough.

    The VPS chosen at TierHive comes in a 10¢ per month, roughly working out to $1.20 a year.

    TierHive.com looks relatively new so the prices may change – but https://serverdeals.cc/category/ovz also lists a few alternatives if the company raises prices or disappears. For the small amount of time it took to set up and deploy and with the secondary Gatus instance on my homelab, I’m happy with this setup!

    With TierHive being so cheap I’ll elect to not post any affiliate links!

  • New Domains!

    Because everyone needs more domains, right?

    During a conversation the other week I was lamenting about how much of a pain it is to have to explain my domain when giving it out —

    Yeah, its hyphen web — the take-away sign then the word web. W for Whisky, E for Echo, B for Bravo.

    A typical reply when asked for my email address

    I was asked — why don’t you just buy it without the hyphen? Good advice I suppose. Except that it had already been purchased. I checked on it previously and had done so at random intervals since registering nick-web.co.uk. The last update the wayback machine was on the 10th January 2018 – where it simply proclaimed

    Welcome to nickweb.co.uk

Staging Server

    and had done so since May 2013. Prior to this is looked like a personal server for a Nick B.

    I love that about the Wayback Machine — pop in a domain and see how its changed over the years. I’m a bit annoyed at myself as a number of years ago I requested that I be excluded from it and I can’t figure out how to re-enable it, but I digress.

    Checking the domain I seen that it was now available for purchase! So I jumped on it, meaning nick-web.co.uk is now improved, with one character less! 14 to 13 characters – a whopping 7.1% decrease!

    But Wait – Theres More!

    Nominet have also introduced the .uk domain – dropping the .co section. Checking with OVH they had a deal and I ended up grabbing that domain for less than £2 for the first year. A bargain! So that means that the domain now has 10 characters, translating to a near 30% reduction in characters!

    The domains will all redirect to the nick-web.co.uk domain as I’ve had it since 2002. I can’t get rid of it that easily!

    Why? Why not…

  • What to expect when your expecting… to be hacked

    A bit of a disappointing one this. Theres nothing worse (well, I’m sure there is but in this context lets leave it as is) than receiving that dreaded e-mail from your host. It starts with a subject line similar to

    Security Incident Concerning

    and a body of text along the lines of

    A security risk has been detected on your server.
We have been informed that your server contains or redirects to harmful or malicious content, such as malware or phishing sites.

    Not at all ideal. In summary — the server in question began to host malicious content. It was from a domain that I don’t use and have a holding page only up, and a user at some point has reported the URL as a malware/phishing attempt. This then gets reported to my host, who then reports it to me.

    After getting the email I was a bit perplexed as to how this site had been flagged as a security risk. I checked the URL given and sure enough, a redirect was in place taking it away from my server to some other (compromised) server. I thought it might have been a coding issue that allowed my domain to freely redirect pages (meaning any attacker could mask their own server with mine). I logged in and checked a few things. It wasn’t my code that was doing anything. I checked and seen a few other files that shouldn’t have been there, all with recent creation dates. A quick

    find . -maxdepth 20 -mtime -20

    netted me a few files that had been created 2 days prior. These were in a variety of directories, and as I spread my domains across a couple of servers these files also appeared in those directories. The suspect files all were all Base 64 encoded, and executed php scripts – given that they all started with 

    <?php eval(base64_decode('BASE64ENCODEDSTRINGHERE')) ?>

    These files either redirected pages, contained a mass emailer (LeafPHPMailer) or opened up a (pretty feature rich but visually poor) file manager. I’m not going to go into too much detail but the reason for the infection came from one WordPress installation that I had completely forgot about after transferring to the new host. Its a site thats very seldom accessed, and to be honest, doesn’t require a WordPress Installation, but it was an easy CMS solution for someone.

    Using an out-of-date plugin the attacker managed to place obfuscated PHP file on the server. This file was then accessed via a web browser which ran the PHP code, and allowed other files to be placed in different locations on the server.

    When I deployed these servers I began hardening them against attacks like this. Unfortunately, I didn’t finish it. Some of my actions stopped the potential full-scale destruction of the server which I’m thankful for, but I’m a bit annoyed I didn’t finish my hardening steps.

    Having separate users for different tasks on the server helped. This meant that any file modifications were only able to be done at the root web-directory level. Config files, where appropriate were hosted out-with the directory, and permissions meant that other files could not be modified. There were a few other steps that I’m not going to go into detail about however cashing up on a couple of guides on how to harden or secure your server should help.

    After figuring out what happened, and how it happened, I stopped any public access – essentially shutting down the HTTP Daemon, removed any newly created files matching the time scales above, pretty deleted any WordPress installations and re-downloaded fresh copies of the MD5-checked files from wordpress.org, then manually checked all the database tables for Indicators of Compromise (IoC’s) line by line, entry by entry, and slowly reloaded everything.

    I then finished my hardening that I should have done before.

    I think it’s important to vent that “hacking” isn’t hacking any more. It’s what used to be known as “Script Kiddies” who are now essentially Serious and Organised Crime Groups that use these phishing and malware scams to extract money. Theres no hacking in the traditional sense – just the unauthorised access to computers that wreck havoc on people who are caught by it. It’s the same as a teenager using a Low Orbit Ion Cannon.

    Am I embarrassed this happened to me? Of course. Annoyed? Yep. But I’m also relieved that it did — it means I was able to stop it before it became much, much worse.

  • And Done

    Thats all my domains now moved over to the new host.

    7 years ago I started hosting with Digital Ocean. At exactly 0218Hrs on Friday 23/08/2013 I made my first $5 droplet – and that $5 droplet worked exceptionally well. I had automated backups and with VAT it came to around $7.20 every month. Not a lot by any stretch of the imagination.

    The existing VPS was destroyed at 1655Hrs, Monday 30/03/2020 – meaning that droplet ran for exactly 2411 days, 14 hours and 37 minutes. Or 6 years, 7 months, 7 days, 14 hours and 37 minutes – give or take a few seconds. Its a good thing I waited until today to destroy the droplet otherwise there would be an extra calculation due to the DST difference!

    But anyway. Onwards and upwards. My new host is £1.20 per month for each server, with a £10 setup fee. I currently have 2 separate servers running – for both reliability and to ease strain as a new customer. So far I’ve spent £16.80 per server for 3 months. Thats £5.60 per month each, so £11.20 per month. Making it a tad more expensive.

    Just for information. I just spent the last hour dealing with Digital Ocean’s Billing API to obtain all the following figures – then attempting to use jq to parse this data – and finally excel to do some quick calculations. I also used [1],[2] and [3] in my quest to save time.

    Im going use the rounded up value of 6 years and 8 months for the following math – exactly 80 months.

    I’ve had a total of $110 in Digital Ocean credit over those 18 months — this includes welcome credit and referrals. Over the 80 months I spent exactly $427.04. Minus that credit brings the total cost to $317.04, so per month that equates to $3.96. Not bad, and still under the $5 cost it should have been! Converting that into UK Pounds equates to roughly £3.15 per month.

    As I said my new server is only £1.20 per month, but there’s two of them, so £2.40. A whopping 75p per month saving.

    However – I still have to factor in that £10 (each) per server set upon cost – and if I take the average of 80 months £20 becomes £0.25 per month.

    All totalled, It looks like i’ll be saving around 50p per month.

    Damm. I wish I had done the math before the moves.

  • 🤦🏼‍♂️

    There comes a time when everyone makes a mistake with computers. Garbage In Garbage Out and all that jazz. I seem to just like shovelling a large amount of Garbage into my servers. Take this evening for example.

    I have one domain left to transfer to my new servers. The third last one was this blog and that went almost without a hitch. There were some wordpress-related things I hadn’t accounted for but nothing I couldn’t handle. The penultimate one was another wordpress installation that went much smoother after I transferred this blog.

    And then I started preparing the move for the last domain. I’m being careful with this one because I know things will go wrong given half a chance. I’ve been checking and re-checking everything and decided to start to the process this evening.

    First up was to update the existing code on that domain so that I could transfer everything fresh. That has a small hiccup which ended with me erasing (thanks to an errant FTP command) almost every folder and leaving but a handful of files behind.

    Panicking, I managed to copy a new instance of the software over however in my haste this overwrote the main configuration file. Thank goodness I managed to remember some very old username/password/database combinations.

    That ended up being rescued and is now working fine on the old host.

    Whilst running about I then tried to login to my new host and managed to type with my ham-fists the incorrect password a few times and put my own IP address in a fail2ban jail.

    I managed to get a remote KVM Console running via my hosting provider, and took myself out of the fail2ban jail. Now, I’m not entirely sure why I thought this was a good idea but I seen this button and figured I should click it.

    CTRL+ALT+DEL

    Now I’m no new entrant to linux and its ilk. I’m also not a stranger to Windows – and my organisation requires the use of Ctrl+Alt+Del to initiate a login. Completely forgetting that Ctrl+Alt+Del on Linux restarts the server I click the button. No confirmation. No warning. Just a graceful restart.

    Remember how I didn’t want any downtime? 🤦🏼‍♂️

    I am not a smart man.

    On the plus side, I today confirmed that a reboot would bring all services back online in a sane manner!

    But the last site move has been postponed.

  • Easy root MySQL Password reset

    I’ve recently found myself in that awkward situation where I’ve chosen a (very) secure password for a root MySQL user and I’ve forgotten it. I suppose its the most secure password I’ve had then because even I cant use it to login to the MySQL database – not that you should be using a root account under any circumstances – this was debugging.

    I’ve had this happen before and I’ve used the

    --skip-grant-tables

    trick that stops the MySQL server, starts it up with no password, allows you to change the password, then stop and restart the server with the new password in force.

    Ideally I wanted no downtime (even for those few precious seconds). As usual, google ended up pointing (to me) to a unknown way to reset the root password using another account that has full root privileges in a standard Ubuntu install.

    Full credit to https://www.configserverfirewall.com/ubuntu-linux/reset-mysql-root-password-ubuntu/, and the main parts are only reproduced here in case the original site goes offline.

    The debian-sys-maint user account is an administrative user account automatically created when installing MySQL Server on Ubuntu and this user have full access to all databases.

    Quote from https://www.configserverfirewall.com/ubuntu-linux/reset-mysql-root-password-ubuntu/https://www.configserverfirewall.com/ubuntu-linux/reset-mysql-root-password-ubuntu/

    This means that viewing the debian.cnf file using cat

    sudo cat /etc/mysql/debian.cnf

    nets us directly the login details including password for this account. Looking under

    user = debian-sys-maint

    is a

    password = LongAndRandomPassword

    so, using these details, a a quick

    mysql -udebian-sys-maint -p

    and

    FLUSH PRIVILEGES;
 ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'NewVerySecurePassword';
 FLUSH PRIVILEGES;

    nets you with an uninterrupted uptime MySQL password reset.

    Now. To read about this account that has MySQL Root Privileges that I’ve never noticed before.

  • Time for an update.

    Update 1 below, posted at 12/12/2019 @ 1910Hrs

    It’s Annual Leave time. Seeing as I’m not going away anywhere that means I have to fill my time with other things. This time, I’ve decided to migrate hosts. Oh the joys.

    I have approximately 10 domains to move, database to migrate, files to transfer and DNS changes to propagate. This is going to be fun.

    Why? Cost. I currently pay around £9 a month for my current hosting, which is to a company based in the US. I recon I can (safely) get it down to around £2 per month, maximum. Thats roughly 77% less. The hosting for the sites I do isn’t mission critical. Its still just a hobby. The £2 a month hosting should be more than adequate for my needs.

    Because I’m lazy I figure a script will help with some of the automation. I’m using Apache VirtualHosts, and because each domain requires a separate VirtualHosts configuration file, I created this script. It doesn’t have the full configuration in it, but its enough to help you get started. I also output the command required to enable each configuration into a separate file, which can be viewed here.

    Update 1: Most of my sites have been transferred over with no issues. All DNS records have been updated and appear to be working fine, apart from Virgin Media dragging its heels and apparently (still) not adhering to TTL values.

    This one, and my busiest have yet to be moved. Im contemplating moving these two to a separate server, and at £1 per month, I don’t see why not.