Demystifying Modules

Under the hood of the Go toolchain

31 May 2019

Modules

Go 1.11 introduced the concept of modules:

• adds support for Go package versioning
• in turn enables reproducible builds
• a large, ongoing change (build system, tools, community)

In my opinion, the conceptual model is simpler than it may initially appear.
Need to get to the underlying ideas.

Scope

I won't have time to cover:

• uncooperative dependencies
• vendoring
• new Go services (Go Module Mirror, Checksum Database, Module Index)
• more

Modules

Different approach to version selection:

• Uses semver
• Relies on "import compatibility rule" and "semantic import versioning"

Those constraints make Minimal Version Selection algorithm viable.

Arguably, much simpler.

Import compatibility rule

"If an old package and a new package have the same import path, the new package must be backwards compatible with the old package."

• github.com/russross/blackfriday@v1.5.1
• github.com/russross/blackfriday@v1.5.2 - backwards compatible

Semantic import versioning

Including the major version number in the module path and import paths is "semantic import versioning".

• github.com/russross/blackfriday
• github.com/russross/blackfriday/v2 - not backwards compatible

Minimal version selection

The set of modules providing packages to builds is called the "build list".

If multiple versions of a particular module are in the list, then at the end only the latest version (according to semantic version ordering) is kept for use in the build.

Minimal version selection

Version v1.5.2 of blackfriday gets chosen.

Minimal version selection

Both are used.

GOPATH mode pop quiz

Suppose you have:

package main

import (
    // ...

    _ "image/png"
)

func main() {
    // ...
}

GOPATH mode pop quiz

And then:

package main

import (
    // ...
    "image/png"

    _ "image/png" // (forgot to remove)
)

func main() {
    // ...
    err := png.Encode(w, m)
    // ...
}

Is image/png being imported twice?

GOPATH mode pop quiz

Maybe you had two separate .go files:

// a.go
package main

import (
    // ...
    "path"
    // ...
)

// ...

// b.go
package main

import (
    // ...
    pathpkg "path"
    // ...
)

// ...

GOPATH mode pop quiz

Then refactored into one:

package main

import (
    // ...
    "path"
    pathpkg "path"
    // ...
)

// ...

Is path being imported twice?

GOPATH mode pop quiz

No:

$ go list -deps
...
image/png
...

$ go list -deps
...
path
...

Module mode example

Modules are easy to get started with. Example:

# Take a medium-sized GOPATH mode project.
gocd github.com/shurcooL/home
go test ./...

# (To simulate a "first time" experience...)
export GOPATH=$(mktemp -d)
export GOPROXY=file://$HOME/GoConCanada/homecache

# And build it in module-aware mode.
export GO111MODULE=on
go mod init
go test ./...

What is a module?

Package

• constructed from one or more source files
• smallest unit that can be built

Module

• a collection of related Go packages
• unit of source code interchange and versioning

Package resolution

• go get golang.org/x/text/currency@v0.3.0

Gets resolved to a module version by go command:

• module path golang.org/x/text
• module version v0.3.0
• provides package with import path golang.org/x/text/currency

Module version

• a specific version of a module
• unambiguous
• immutable

package module

// A Version (for clients, a module.Version) is defined by a module path and version pair.
type Version struct {
    // Path is a module path, like "golang.org/x/text" or "rsc.io/quote/v2".
    Path string

    // Version is a module version.
    Version string `json:",omitempty"`
}

Modules

There's a mapping from modules to packages.

The key is {ModulePath, Version},
and the value is []Package:

module version ➡ collection of Go packages

golang.org/x/text:
    v0.1.0 → [packages...]
    v0.2.0 → [packages...]
    v0.3.0 → [currency, other packages...]
    v0.3.1 → [packages...]
    v0.3.2 → [packages...]

Module path

Module path is a URL (without scheme), like package import path.

You can create module versions at:

github.com/you/project
github.com/you/project/sub/directory
github.com/you/project/v2

bitbucket.org/you/project
bitbucket.org/you/project/sub/directory
bitbucket.org/you/project/v2

you.example
you.example/sub/directory
you.example/v2

Module queries

The go command accepts a "module query" in place of a module version both on the command line and in the main module's go.mod file.

The string "latest" matches the latest available tagged version, or else the underlying source repository's latest untagged commit.

go get github.com/gorilla/mux
go get github.com/gorilla/mux@latest    # same (@latest is default for 'go get')
go get github.com/gorilla/mux@v1.6.2    # records v1.6.2
go get github.com/gorilla/mux@e3702bed2 # records v1.6.2
go get github.com/gorilla/mux@c856192   # records v0.0.0-20180517173623-c85619274f5d
go get github.com/gorilla/mux@master    # records current meaning of master

Module version

If there are no tagged versions at all, go get chooses the latest known commit:

v0.0.0-20180609043247-fa215029cf59
v0.0.0-20180621172110-9453c974ce53
v0.0.0-20180930192442-2dc491213fbb
v0.0.0-20180930193832-d0ebfa270e3e
v0.0.0-20180930203227-0cf138a823b3
v0.0.0-20181021170909-7a718d85c543
v0.0.0-20181022071201-c4eb07ba2d71
v0.0.0-20181027033722-d15030e56f3b
v0.0.0-20190408044430-c9d1ccdd732c
v0.0.0-20190408044501-666a987793e9 ← latest

Module version

If there are only tagged pre-release versions, go get chooses the latest of those:

v0.0.0-20180609043247-fa215029cf59
v0.0.0-20180621172110-9453c974ce53
v0.0.0-20180930192442-2dc491213fbb
v0.0.1-pre1
v0.0.1-pre1.0-20180930203227-0cf138a823b3
v0.0.1-pre1.0-20181021170909-7a718d85c543
v0.0.1-pre1.0-20181022071201-c4eb07ba2d71
v0.0.1-pre2                               ← latest
v0.0.1-pre2.0-20190408044430-c9d1ccdd732c
v0.0.1-pre2.0-20190408044501-666a987793e9

Module version

If there are tagged release versions, go get chooses the latest of those:

v0.0.0-20180609043247-fa215029cf59
v0.0.0-20180621172110-9453c974ce53
v0.4.5
v0.4.6-0.20180930193832-d0ebfa270e3e
v0.4.6-0.20180930203227-0cf138a823b3
v0.4.6-0.20181021170909-7a718d85c543
v1.2.3                               ← latest
v1.2.4-0.20181027033722-d15030e56f3b
v1.2.4-0.20190408044430-c9d1ccdd732c
v1.2.4-0.20190408044501-666a987793e9

Downloading packages

Module proxy protocol

GET $GOPROXY/<module>/@v/list
GET $GOPROXY/<module>/@v/<version>.info
GET $GOPROXY/<module>/@v/<version>.mod
GET $GOPROXY/<module>/@v/<version>.zip

Note, the <module> and <version> elements are case-encoded.

Module zip archive

The zip archive for a specific version of a given module is a standard zip file that contains the file tree corresponding to the module's source code and related files.

The archive uses slash-separated paths, and every file path in the archive must begin with <module>@<version>/, where the module and version are substituted directly, not case-encoded.

The root of the module file tree corresponds to the <module>@<version>/ prefix in the archive.

example.com/Abc@v1.0.0
├── go.mod
├── a.go
├── b.go
└── dir
    └── c.go

Handcrafting a module

Handcrafting a module

# Zip a module.
@v $ zip -r -D v1.0.2.zip 'dmitri.shuralyov.com/test/modtest1@v1.0.2'
  adding: dmitri.shuralyov.com/test/modtest1@v1.0.2/go.mod (deflated 2%)
  adding: dmitri.shuralyov.com/test/modtest1@v1.0.2/inner/p/p.go (deflated 38%)

Conclusion

A module version is defined by a module path and version pair.
It's unambiguous and immutable.

You have fine control over the module versions you publish,
and the module versions you use.

Modules are a foundational building block that new workflows
and tools can be built on.
Module support in the go command will continue to improve.

Thank you