How to train your Goroutine

Introduction

Golang’s Goroutines run on top of OS threads and OS threads run on logical cores. Goroutines switch between threads and threads switch between cores. In this article I will talk about how we can “pin” a goroutine to a logical core.

I would not advice anyone do this, because Go scheduler does a great job in running goroutines efficiently, by respecting OS scheduler.

There have been some discussion about this topic, so I will add the links here:

is it possible to force a go routine to be run on a specific CPU?

OS and Go schedulers

First, we need to understand how OS scheduler works. There is a great article which can help you get started on this topic without going too deep.
Main take away points regarding OS threads are:

• Threads switch between cores (context switch)
• Thread context switches are expensive

created by gifmaker.me

Go scheduler works on top of OS scheduler, and developers of Go took into account how OS scheduler works. In particular, they tried to minimize number of OS thread context switches. More on this is written in another great article.

Let’s do this

After searching about this topic on the Internet, I have found some tools that could help me and I started experimenting with them.

LockOSThread

The runtime package has this cool function LockOSThread. According to the documentation it gives the ability for Go developers to use non-Go libraries which utilize thread local storage (TLS).

However, we will use this function for a different purpose. By calling this function, we make sure that a goroutine will stay on the same OS thread and will not be scheduled to run on other threads.

As you may have already guessed, this is not enough. OS threads may still switch between cores. A goroutine will also switch to a different core, together with its locked OS thread in case if OS scheduler decides to do so.

SchedSetaffinity

After searching for some more, I have stumbled upon this function from golang.org/x/sys/unix package. From it’s documentation we can see that this function can help us lock an OS thread to a logical core. Note, that this can be tricky on different types of OS’s.

Final recipe and tricky points

The steps to achieve our goal now look like this:

• Call runtime.LockOSThread from the target goroutine
• Call unix.SchedSetaffinity afterwards

This might be enough for a simple use case. However, if you want almost no interruption of your goroutine’s work, then you might also want to look at isolcpus and taskset utilities. But still, there are no guarantees that Go scheduler will not schedule some of it’s internal jobs (like garbage collection) on one of the locked OS threads.

Note that SchedSetaffinity does not respect taskset and can run your goroutine on any core you specify. Therefore, you might first want to checkout SchedGetaffinity to see which cores you can “pin” to.

Another important point is that we should be careful and leave at least one core to other goroutines, which will not be locked and need some CPU resources to run.

Some coding

I have written a simple Go package - gofine, which can help you manage locking goroutines to logical cores easily. Feedback is very welcome.

P.S.: gofine = Go + Core affine and lgore = Go + logical core

Summary and Credits

Once again, you might not need to do this and there are no guarantees that it will work as intended for everyone. I encourage readers (and the author) of this post to read more about Go scheduler to fully understand how it works.

Thanks to people who wrote this:
https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part1.html
https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part2.html
https://stackoverflow.com/a/19759235
https://codywu2010.wordpress.com/2015/09/27/isolcpus-numactl-and-taskset/