Channel your inner communication, little goroutines


Below are some patterns and ideas I’ve come across wrt channels and goroutines. Just remember…

...if you reach into your toolbox and decide to use goroutines always ask: how and when will those goroutines stop

This section will likely be split off into more specific posts in the future.


limit number of working goroutines with buffered channels

Often times you don’t need thousands of goroutines working at once, especially true when communicating with external resources, e.g., sockets, files, network connections, etc.

In the example below we’re using buffered channels to limit the number of working goroutines at any one time.

  • careful with loop iterator variables inside goroutines (i := i in the example below). A concise explanation here.

EDIT(2017-08-18): After seeing Dave’s Concurrency made easy video, I’ve adapted the example below.

An interesting point raised is the location of the semaphore block. I previously placed workerChan <- 1 outside the anonymous function so that n number of goroutine workers would start and run at any one time.

But, by placing workerChan <- 1 inside the anonymous function right before actual work starts, enables us to fire off all goroutines almost immediately and, subsequently, goroutines will begin work as they become unblocked.

acquire semaphores when you're ready to use them

EDIT(2018-10-06): Although it seemed like a good idea to put the semaphore inside the goroutine, I no longer adopt that pattern.

There are some goroutines working, and potentially many goroutines “idling”. Blocking outside the goroutine as apposed to inside ensures only the working goroutines are consuming resources. More often than not you’ll care about resource consumption instead of the time it takes to “fire up” the next goroutine.

func main() {
    rand.Seed(time.Now().UnixNano()) // ignore this

    workerChan := make(chan int, 2) // run up to two goroutines at once
    errChan := make(chan error)

    n := 10 // amount of work to do; URLs to fetch, SSH connections, files or sockets to open, etc.

    var wg sync.WaitGroup
    wg.Add(n)

    for i := 0; i < n; i++ {
        i := i // capture range variable

        // put val on chan if there is space, otherwise block until buffered chan is freed
        workerChan <- 1

        go func() {
            defer wg.Done()


            if err := doWork(i); err != nil {
                errChan <- err
            }

            <-workerChan // take val off the chan; freeing up space
        }()
    }

    go func() {
        wg.Wait()

        // if errChan is not closed the main goroutine will block forever
        close(errChan)
    }()

    // grab errors off the channel and print them
    for err := range errChan {
        fmt.Println(err)
    }
}

func doWork(i int) error {
    // generate random-ish number of seconds between 0-4, to simulate work
    sec := time.Duration(rand.Intn(5)) * time.Second
    <-time.After(sec)

    if sec > 3*time.Second {
        return fmt.Errorf("goroutine [%d] took too long: %v", i, sec)
    }

    fmt.Printf("goroutine [%d] worked for %v\n", i, sec)

    return nil
}