Jason Taylor - Developer Extraordinaire

Scaling Elasticsearch for Fun and Profit

Tue, 13 Feb 2024 00:00:00 +0000

I have been managing an Elasticsearch cluster at work for over 3 years. This time has been a constant learning experience to say the least. When I started working on this cluster, we were using a cloud provider that gave us exactly one tool for scaling up: a slider to provision more nodes and larger nodes. The standard practice at the time was to slide the slider up whenever the cluster started acting up¹. This would give us “more capacity” for a couple months and then the acting up process would start all over again. This method of scaling also had the notable side effect of generating massive bills.

This post is a list of everything I tried that gave me positive results when the cluster was handling load or a concept that was important in helping me understand more about how Elasticsearch runs. The list is in no particular order, but I tried to group it by topic to make it easier to read. The most important takeaway you can have from this post is to make sure you are monitoring enough to see what effect each change has on performance as you apply it. This will help you determine what the cluster is struggling with the most and help you prioritize what to try next.

At the time of my applying the techniques described below we were using Elasticsearch 6, but I think a lot of this advice is applicable on Elasticsearch 7 as well. After version 7.10, Elastic changed licenses for Elasticsearch and it is no longer open source, so buyer beware. :)

General Advice

Start with 3 large generic nodes and go as far as you can with this.

Specialized nodes increase complexity and cost quite a bit but don’t necessarily speed up writes or searches. Best to stay on the simplest possible cluster configuration and focus on optimizing your indexes and data.
Pay someone else to run your cluster.

We have a vendor who runs the actual hardware for our cluster. They manage the nodes and provisioning; we manage the data in the cluster. I highly recommend this if you have the budget, running a cluster yourself would 10x the difficulty of managing Elasticsearch in your system unless you have someone on your team with this specific expertise. It’s worth it to pay someone else for this.
Learn the terminology.

Spend the time reading the docs to learn the terminology specific to Elasticsearch because this will make life much easier when you need to find answers on the Internet. Not to mention you will pick up a ton of incidental knowledge about how Elasticsearch works which makes working on it a lot easier.
Be aware of the default settings of the elasticsearch-transport library.

For example, the Ruby version retries a request up to 3 times by default if it fails on a connection issue or a timeout. For us, this led to compounding issues on the cluster whenever we had enough load that the cluster started to reject some requests because they would all immediately retry 3 times instead of going through our exponential backoff retry cycle (more on this in a later section).
Monitor the Write and Search queue size and rejections.

This is the best “canary in the coal mine” indicator for when your cluster is starting to struggle. Each node in your cluster has as Write and Search queue. Incoming requests are automatically queued into these depending on whether they are a write requests or a search requests and then handled in order.

If these queues fill up the cluster will immediately start rejecting requests with an error. When this happens, you will see a spike in the rejections stats that the cluster emits in the metrics. A few of these here and there are normal and should get handled by your retry logic, however if these start to add up, you need intervention or you’re going to have a bad time.
Don’t put Elasticsearch in the hot path.

In my experience, having Elasticsearch in the hot path does not go well as soon as you scale to the point of having a decent amount of load on the server. If your site requires Elasticsearch to be instantly responsive to a request to load a main page, you’re going to have a bad time.

Index Management

Use aliases.

Without aliases, managing your indexes will become very difficult without requiring scheduled downtime for your cluster. Aliases give you a way to perform a lot of index maintenance tasks, such as re-indexing, while always having a live index in use by users.
Some settings can only be changed on index creation.

Be aware of certain settings that can only be changed at index creation time like number_of_shards for example. Changing these after the index is created requires creating a whole new index and migrating the data to it. If you have large indexes, this can take several hours. Switching indexes is when your aliases will come in handy.
Avoid using anything other than default index settings.

The defaults will take you a long way and by the time you might need to fiddle with a lot of these, you will have learned enough to know what they do and how they might affect the system. After 4 years of running this cluster, the only setting I have needed to change on an index is number_of_shards.
Indexes that have different workloads on the same cluster is difficult.

Let’s say you have an index that is a mostly static dataset but gets searched all the time, and you have a second index that receives tons of writes but doesn’t get searched that much. It can be difficult to tune the cluster well for opposite workloads like this. The reason for this is that you essentially have one shared pool of resources on each node, so if you optimize too much for writes, then search performance will become resource constrained and begin to suffer and vice versa. It’s good to be aware of this and always maintain a good balance when tuning.
Primary shard vs Replica shards.

You should set number_of_shards (aka primary shards) to the number of data nodes in your cluster and number_of_replicas to 1 to ensure that the index has an even distribution of shards across the cluster. This will result in the index having a primary shard and a replica shard on each node in the cluster which is ideal. For larger indexes you can set number_of_shards higher but it should always be a multiple of the number of data nodes in the cluster to ensure that you have even distribution. Having an uneven distribution of shards can unbalance the load between nodes and lead to some nodes running hot while others are under-utilized.
~50GB per shard is a good rule of thumb.

Aim for ~50GB per shard as a maximum shard size. Shards that are larger than this will become slow to search and write to and other operational tasks the cluster performs in the background (i.e. merges) will become less performant. If your index starts to exceed 50GB per shard, you may want to increase the number_of_shards on that index.

Handling Writes Gracefully

Do not make write requests inline with a web request.

Elasticsearch by its nature is slow at writes compared to a relational database like Postgres. This is because to write a new object, Elasticsearch isn’t just inserting the data in the index, it’s also processing it by whatever rules you have set up to index it for searching. This takes a lot of time, especially if you have more complex search requirements. The best solution for this is to farm this out to an asynchronous process, like a job queue or data stream for processing later.
Implement exponential backoff for retries.

Under conditions where the cluster is having trouble keeping up with the amount of load, you definitely don’t want your jobs to keep slamming it with the same write request over and over. If it tells you it needs help, give it some help in the form of backing off the retries exponentially. Basically instead of retrying a failed job every second, you would instead retry after 1 second, 2 seconds, 4 seconds, 16 seconds, etc.
Use the Bulk API and actually do writes in batches.

The Bulk API was added by Elasticsearch to handle indexing and deleting operations in bulk. This is much faster than sending requests through the Index or Delete APIs for individual documents. Every blog post you can find will tell you this.
Build indexes with a document structure that requires as few updates as possible.

When Elasticsearch updates a document in the index with new values, it doesn’t update the existing document in place. Instead, it creates a new document, indexes that document from scratch, then deletes the old document. Not only is this slow, but it also leads to bloated indexes because the deleted documents do not get removed from the index, they just get marked as deleted. I have seen indexes with more than 100GB of bloat due to this issue, to the point where we have to periodically run manual re-indexing operations on our larger indexes to keep them from becoming too bloated. This takes several hours to run on a large index.

A better path forward here is to construct your index in a way where you can basically append new data to it if possible. For example, if we have an index of containing documents that represent text message conversations that might look something like this:
```
{
  "_source": {
    "id": <conversation_id>,
    "messages": [
      { "id": <message_id>, "body": "message 1" },
      { "id": <message_id>, "body": "message 2" }
    ]
  }
}
```
Each time we receive a new message for a conversation, we update the conversation in the index to add the new message body in the messages list for the conversation. This requires Elasticsearch to reindex the entire conversation document for search even though most of the data is unchanged. This means we’re using cluster resources to index the same data repeatedly, while leaving deleted versions of the conversation laying around and filling up the storage space.

However, since we know that once a text message is sent, the body of the message doesn’t change, we could refactor this index to be an index of messages instead of conversations:
```
{
  "_source": {
    "id": <message_id>,
    "conversation_id": <conversation_id>,
    "body": "message 1",
  }
},
{
  "_source": {
    "id": <message_id>,
    "conversation_id": <conversation_id>,
    "body": "message 2",
  }
}
```
This enables several benefits that we didn’t have using the old structure.
- Small, concise documents for quicker indexing on write.
- No updates, just add new messages as new documents.
- Ability to use the Rollover Index API to manage size of the index.
- No bloating of deleted documents over time which removes the need to Reindex periodically
- More CPU resources available for search requests since they aren’t being spent on writing whole conversations repeatedly.
Documents in the index should have a flat structure.

Documents in the index should have as flat a structure as possible. Having a deeply nested structure doesn’t perform well because Elasticsearch separately indexes each node in the document as a separate document which takes a lot of processing power on write and leads to large, bloated indexes which need more shards, which then makes search slower.

Monitoring

Here is the list of metrics I’ve found helpful to monitor over the years.

ES Node Stats

Searches and writes
Write queue size
Write queue rejections
Search queue size
Search queue rejections

ES Index Stats

Searches and writes
Data size
Document count
Shard count

Job Stats

Queue lag
Time to process

Conclusions

So if you’ve made it this far into reading this, good luck with your cluster! Elasticsearch can run quite well if you are thoughtful about how you structure your data and set up your interactions between the application and cluster well. Our cluster has gone from being a consistent source of outages to stable enough that it “Just Runs” and reaching that point feels like real success. It can be done!

Footnotes:

Acting up, in this case, being defined as the cluster becoming unresponsive due to being unable to process the volume of requests it was receiving. This ultimately would cause the site to go down because the web servers would be saturated with requests that were waiting on responses from the cluster. ↩

Building An Online Resume With Jekyll & pdfmake

Tue, 02 Jan 2018 00:00:00 +0000

UPDATE 2024-01-23: Since writing this I have removed this version of my resume from the site. If you’re still interested in how this was working, you can find all the resume related code in this commit from the repo. GL;HF!

ORIGINAL POST:

One day it dawned on me, I am a web developer but I was still manually maintaining my resume in a Word document. This was not ideal because it was difficult to keep versions and did little to show off any of my relevant skills in web development. To solve this problem I decided to build an online version of my resume which I could use to show prospective employers that I can, in fact, build things on the Internet.

My first step was to grab all the existing data from my Word based resume and manually build it into HTML so that I could put it on my site. This worked great but when I had to update my resume there were now two copies of it to maintain, one in Word and one in HTML.

At this point I took a step back and thought about what my requirements really were. I needed the following:

One source of data to rule them all
Support essential formats (HTML, PDF)
Generate PDF automatically so it’s always up to date
Generate PDF on client side since my site is on GitHub Pages so it can’t be done on the server

Enter Jekyll Data Files and pdfmake to save the day!

Jekyll Data Files

Built into Jekyll is a super handy way to create your own sets of data and make them available to the site as a whole.

To do this you simply add a directory called _data to your Jekyll project, then add a file in this directory with the name of the data set you want to represent. In my case this file is called resume.yml. This can be referenced elsewhere in the project with site.data.resume. Jekyll supports JSON, CSV, or YAML formats for these data files. I used YAML because, for me, it’s the most readable.

At this point it was easy to use this data to generate my resume because I could simply loop through the data to build various sections of my resume. As an example, here is the code that builds the jobs section:

<!-- _includes/jobs_section.html -->
{% for job in resume.jobs %}
  <div>
    <h3>
      <a href={{ job.company.website }} target="_blank">{{ job.company.name }}</a>
    </h3>

    {% for position in job.positions %}
      <div>
        <span>{{ position.title }}</span>
        <span>({{ position.duration }})</span>
      </div>
    {% endfor %}

    <p>{{ job.description }}</p>

    <ul>
      {% for accomplishment in job.accomplishments %}
        <li>{{ accomplishment }}</li>
      {% endfor %}
    </ul>
  </div>
{% endfor %}

At this point I had a working web page for my resume of but I still needed to make the PDF.

pdfmake

Since I run my Jekyll site on GitHub Pages, I am restricted in terms of the amount of server side processing that I can do so I had to find a library that was capable of generating the PDF on the client side. I tried out several JavaScript libraries capable of converting JSON into PDF in the browser. The winner ended up being pdfmake because it has good documentation and was the easiest for me to get up and running.

In order for pdfmake to do it’s magic though, I needed to get the data for my resume onto the client side in JSON format. Thankfully this is trivially easy in Jekyll. I just put it into a hidden div on the same page as the HTML resume and converted the data hash to JSON with Liquid’s very handy jsonify utility.

<!-- _layouts/resume.html -->
<div hidden id="resumeJson">
  {{ site.data.resume | jsonify }}
</div>

I then wrote a simple JS function that processes the JSON and builds document definition object that includes styles and formatting that pdfmake needs to be able to generate the PDF. That function can be found here on GitHub.

And that’s pretty much it!

Authentication with Devise & Turbolinks for iOS

Mon, 03 Oct 2016 00:00:00 +0000

I previously published this article with the first version of how I got Devise 4.2 working with Turbolinks 5 through an iOS application. The issue with that implementation was that, since Turbolinks for iOS manages a single shared web view, the authentication page shared the same web view in the iOS app with the rest of the pages. This resulted in some weird navigation behaviors and required altogether too much JavaScript in the Rails side of my app to handle cases like errors on login.

After doing more research and really reading through the code in this Turbolinks iOS demo app I realized that the much better implementation for this flow is to have the authentication screen be in it’s own web view and to have it share a session with the rest of the app via a WKProcessPool instance that would be available to any other view that will need to be rendered after the user has been authenticated.

That is a what I’m trying to do here in this post. If you have any questions please submit a GitHub issue to this repo because this is very much a work in progress for me and maybe we can work through similar issues together. :)

Here is my current implementation for the authentication process that works on web and in iOS. It should be noted that most, if not all, of the Swift code in this post was taken from this Turbolinks iOS demo app so thank you very much to the people who put that together.

The Rails Side

On the Rails side of this application the basic setup is to move the Devise views into app/views/users and set up a custom controller for sessions that will know how to respond to JSON formatted requests. Devise has an install generator that will copy the views for you.

In this implementation, the login form’s form_for does not need to have the remote: true flag because it will not need to send an XHR request when POSTing the form. I consider this to be a feature of this implementation because it simplifies some of the cases where errors need to be handled.

# config/devise.rb
config.scoped_views = true

# config/routes.rb
devise_for :users, controllers: { sessions: 'sessions', registrations: 'registrations' }

# app/controllers/sessions_controller.rb
class SessionsController < Devise::SessionsController
  respond_to :json
end

Normally with Devise, when an unauthenticated user tries to navigate to a page Devise will respond with a 302 that redirects the user to the login page. This will not work in iOS because a 302 means the same web view will be used to show the login page which pushes it onto the stack of navigable views. This is bad because once the user logs in they can hit a back button in the top navigation bar and end up back on the login page even though they are already authenticated.

Instead, the server should recognize when the user is using the iOS app and change the format of the request to JSON so that Devise will respond with a raw 401 to the client. Then on the iOS side we can detect the 401 error and display a separate authentication view.

# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
  # ...
  before_action :detect_browser

  def detect_browser
    case request.user_agent
    when /iPhone/i
      request.variant = :phone
      request.format = :json unless request.path.include?(Devise.mappings[:user].fullpath) || user_signed_in?
    else
      request.variant = :desktop
    end
  end
  # ...
end

The iOS Side

On the iOS side of this I basically have 3 view controllers; ApplicationController, AuthenticationController, and MyViewController.

ApplicationController is the main controller for the application (and well named!). It holds the host URL for the server and manages the WKProcessPool instance that is used to share a session between the AuthenticationController and MyViewController, which is important if you want your users to stay logged in after they log in.

ApplicationController acts as a SessionDelegate, so that it can catch errors that may happen when loading Visitable views. It also acts as an AuthenticationControllerDelegate so once the user has authenticated successfully, it can dismiss the AuthenticationController view and load the first page of the app.

In the SessionDelegate part of ApplicationController there is a method named session:didFailRequestForVisitable:withError that catches any HTTP failures and matches the response code to the case statement to see how it should handle that particular error. This gives us a place to respond to the 401 and call presentAuthenticationController() which will render the AuthenticationController view.

// ApplicationController.swift
import UIKit
import WebKit
import Turbolinks

class ApplicationController: UINavigationController {
    private let URL = NSURL(string: "<Host URL>")!
    private let webViewProcessPool = WKProcessPool()

    private var application: UIApplication {
        return UIApplication.sharedApplication()
    }

    private lazy var webViewConfiguration: WKWebViewConfiguration = {
        let configuration = WKWebViewConfiguration()
        configuration.processPool = self.webViewProcessPool
        configuration.applicationNameForUserAgent = "myApp"
        return configuration
    }()

    private lazy var session: Session = {
        let session = Session(webViewConfiguration: self.webViewConfiguration)
        session.delegate = self
        return session
    }()

    override func viewDidLoad() {
        super.viewDidLoad()
        presentVisitableForSession(session, URL: URL)
    }

    private func presentVisitableForSession(session: Session, URL: NSURL, action: Action = .Advance) {
        let visitable = MyViewController(URL: URL)

        if action == .Advance {
            pushViewController(visitable, animated: true)
        } else if action == .Replace {
            popViewControllerAnimated(false)
            pushViewController(visitable, animated: false)
        }

        session.visit(visitable)
    }

    private func presentAuthenticationController() {
        let authenticationController = AuthenticationController()
        authenticationController.delegate = self
        authenticationController.webViewConfiguration = webViewConfiguration
        authenticationController.URL = URL.URLByAppendingPathComponent("/users/sign_in")
        authenticationController.title = "Log In"

        let authNavigationController = UINavigationController(rootViewController: authenticationController)
        presentViewController(authNavigationController, animated: true, completion: nil)
    }
}

extension ApplicationController: SessionDelegate {
    func session(session: Session, didProposeVisitToURL URL: NSURL, withAction action: Action) {
        presentVisitableForSession(session, URL: URL, action: action)
    }

    func session(session: Session, didFailRequestForVisitable visitable: Visitable, withError error: NSError) {
        guard let myViewController = visitable as? MyViewController, errorCode = ErrorCode(rawValue: error.code) else { return }

        switch errorCode {
        case .HTTPFailure:
            let statusCode = error.userInfo["statusCode"] as! Int
            switch statusCode {
            case 401:
                presentAuthenticationController()
            default:
                myViewController.presentError(Error(HTTPStatusCode: statusCode))
            }
        case .NetworkFailure:
            myViewController.presentError(.NetworkError)
        }
    }
}

extension ApplicationController: AuthenticationControllerDelegate {
    func authenticationControllerDidAuthenticate(authenticationController: AuthenticationController) {
        session.reload()
        dismissViewControllerAnimated(true, completion: nil)
    }
}

// AuthenticationController.swift
import UIKit
import WebKit

protocol AuthenticationControllerDelegate: class {
    func authenticationControllerDidAuthenticate(authenticationController: AuthenticationController)
}

class AuthenticationController: UIViewController {
    var URL: NSURL?
    var webViewConfiguration: WKWebViewConfiguration?
    weak var delegate: AuthenticationControllerDelegate?

    lazy var webView: WKWebView = {
        let configuration = self.webViewConfiguration ?? WKWebViewConfiguration()
        let webView = WKWebView(frame: CGRectZero, configuration: configuration)
        webView.translatesAutoresizingMaskIntoConstraints = false
        webView.navigationDelegate = self
        return webView
    }()

    override func viewDidLoad() {
        super.viewDidLoad()

        view.addSubview(webView)
        view.addConstraints(NSLayoutConstraint.constraintsWithVisualFormat("H:|[view]|", options: [], metrics: nil, views: [ "view": webView ]))
        view.addConstraints(NSLayoutConstraint.constraintsWithVisualFormat("V:|[view]|", options: [], metrics: nil, views: [ "view": webView ]))

        if let URL = self.URL {
            webView.loadRequest(NSURLRequest(URL: URL))
        }
    }
}

extension AuthenticationController: WKNavigationDelegate {
    func webView(webView: WKWebView, decidePolicyForNavigationAction navigationAction: WKNavigationAction, decisionHandler: (WKNavigationActionPolicy) -> Void) {
        if let URL = navigationAction.request.URL where URL != self.URL {
            decisionHandler(.Cancel)
            delegate?.authenticationControllerDidAuthenticate(self)
            return
        }

        decisionHandler(.Allow)
    }
}

Conclusions

Never publish a blog post until you’ve lived with your implementation for a bit because you may have to completely rewrite it as a result finding a better implementation. :)

How I Built This Site

Mon, 29 Feb 2016 00:00:00 +0000

Once upon a time, I decided to start a blog. I’ve tried this several times in the past but the project always seems to die when it comes to figuring out what technology to use or how to host it.

I’ve tried several platforms; Wordpress, Blogger, and even building the Classic Rails Blog™️ more than once. All of those solutions seem to require a ton of work just to maintain the site. What I really want is something that is easy to set up, host, and will allow me to spend time on more interesting problems.

GitHub Pages to the rescue!

Github Pages came to my attention and at first glance it seemed easy to use so I decided to try it out. GitHub offers an automatic page generator for Pages sites that you can access in Settings > Options > GitHub Pages but this doesn’t seem to offer as much flexibility for the theme and layout of the site as I wanted.

Fortunately, GitHub Pages has support for Jekyll which is a static site generator with support for writing blogs in Markdown.

Here’s what I had to do to make this work.

Setting up a GitHub Page

The first thing I did to set this up was to create a git repo named <your-github-username>.github.io.

In my case this was thetizzo.github.io. It is important to make sure that your username is spelled correctly and the name of the repo matches this format or GitHub Pages will not display the site correctly.

For personal GitHub Pages, the site will automatically be served using the master branch of this repository. It’s shocking how quickly changes will show up on the actual site once you push them to GitHub. It often doesn’t take more than a few seconds.

Setting up Jekyll

To get Jekyll set up I ran these 3 steps:

gem install jekyll
jekyll new thetizzo.github.io
jekyll serve

At this point you can go to localhost:4000 and see the default Jekyll site.

Integrating a theme

If you want, you can start writing blogs posts right away, but I wanted to change the theme of the site make it my own. Jekyll lets you build anything so you could use Sass to design the site any way you want or you can download a pre-made theme, which are available from several places.

I decided to use a pre-made theme because I figured that working through integrating that theme with my site would be a great way to gain a better understanding of how to work with Jekyll.

I found this theme on jekyllthemes.org and thought it was a nice, clean look that would be a good starting place for my site.

This theme is set up to be it’s own standalone site so in order to integrate it I started by setting up a new page that would use a separate layout from the rest of the site. To add a new page to the site you just need to add an HTML file to the project root. In my case this was resume.html which looks like this:

---
layout: resume
---

That’s really it. In Jekyll, anything at the top of a file between the dashes is called Front Matter and can be used to pass variables and defaults into a page.

Since I’m using a theme, I just need the front matter for resume.html to point to the _layouts/resume.html layout file which will end up being the place where the bulk of the page structure is.

I then copied over the rest of the theme to the appropriate places, making sure to namespace all the files to keep the two different themes (default and resume) separate until I was ready to combine them. It looked something like this:

resume-template -> my_project

_config.yml contents -> _config.yml
_layouts/resume.html -> _layouts/resume.html
css/main.scss -> css/resume-main.scss
_sass/*.scss -> _sass/resume-*.scss
_includes/*.html -> _includes/resume-*.html
_includes/*.html -> _includes/resume-*.html

I was then able to build a resume page using this theme while leaving the rest of the existing site alone. Once that was finished, I merged the styles from the resume page into the default layout by removing the default theme styles and removing the namespace for the resume styles and files so they would apply everywhere on the site.

Using Font Awesome with Jekyll

I personally love Font Awesome. If you haven’t heard of Font Awesome, it’s a set of icons that you can easily include in your site. They are included as a font to your site so you can easily customize them with CSS.

Font Awesome offers a CDN to serve the assets so including them in a project is super easy. I simply added the following line to _includes/head.html:

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">

Setting up a custom domain

Another nice thing about GitHub Pages is that they offer a super easy way to put your own custom domain on the site. This is nice because it gives you the performance benefits of GitHub’s CDN but also let’s the site have the feel of a completely custom website.

I will let GitHub’s own instructions speak for themselves on this matter but basically all I had to do was add a CNAME record through my DNS provider for thetizzo.com to point at thetizzo.github.io and add a file called CNAME to the root of the project that looks like this.

Adding Google Analytics

To add Google Analytics to the site I created a file, _includes/google_analytics.html, and put in the code snippet that Google gives you when you sign up for Analytics which looks something like this:

<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', '<your tracking ID number>', 'auto');
  ga('send', 'pageview');
</script>

Then included that file in _includes/head.html:

<!-- some head content -->
{% include google_analytics.html %}
<!-- the rest of the head content -->

Have fun!

Hopefully this was helpful. Good luck and have fun with your new, highly advanced GitHub page.