Go Module Internals

Inside the Zip Files

24 September 2019

Intro

• I've been working with Go's module support, and want to share some of the fun stuff I found.

• Talk should be less serious than title may suggest.

Go Module Services

• Module Mirror, Checksum Database, and Module Index are launched!

https://proxy.golang.org

https://sum.golang.org

https://index.golang.org

• Highly recommend Katie Hockman's GopherCon 2019 talk that explains how they work and why they're useful.

Go 1.13

• If you just want to get going with Go, now's the best time! Go 1.13 is the best release yet.

• The defaults provide a great high-level experience. Mind GOPRIVATE for non-public modules.

Quick Demo

GOPATH mode:

$ export GO111MODULE=off
$ go get honnef.co/go/tools/cmd/staticcheck

Module mode:

$ export GO111MODULE=on
$ go get honnef.co/go/tools/cmd/staticcheck

In Go 1.13, can now do:

$ go version ./bin/staticcheck

Programs can use debug.ReadBuildInfo() for introspection.

Caution

• I'm going to do bad things and adjust knobs to show internal details. You shouldn't need to do any of this (unless you're developing Go itself).

• Please be responsible: internal details that are not a part of a public API may change.

Personal website

I have one of those!

dmitri.shuralyov.com

What are module zip files

Previously in Go...

GOPATH mode:

• Development likely use git, hg, ...
• Releasing must use git, hg, ... - must push to branch
• Downloading must use git, hg, ...

go get [-u] import/path

Previously in Go...

Previously in Go...

• Imagine developing with git, having a dependency that uses hg, which in turn has a dependency on svn...

golang.org/issue/19614 - proposal: cmd/go: go get should support source archives

What are module zip files

"[...] when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, [...]"

Example

Let's start simple and work our way up.

package main

import (
    // We'll create this next:
    "github.com/dmitshur-test/nyc2019sep/hi"

    "fmt"
)

func main() {
    fmt.Println(hi.Text)
}

$ cd ~/go
$ mkdir -p src/nyc.example
$ cd src/nyc.example
...

Step 1

Module zip archive

The zip archive for a specific version of a given module is a standard zip file that contains the file tree corresponding to the module's source code and related files.

The archive uses slash-separated paths, and every file path in the archive must begin with <module>@<version>/, where the module and version are substituted directly, not case-encoded.

The root of the module file tree corresponds to the <module>@<version>/ prefix in the archive.

example.com/Abc@v1.0.0
├── go.mod
├── a.go
├── b.go
└── dir
    └── c.go

Zip files

Look at a zip:

$ unzip -Z1 v1.0.0.zip

Create a zip:

# -r   recurse into directories
# -D   do not add directory entries

$ zip -r -D v1.0.0.zip 'example.com/project@v1.0.0'

(Maybe don't push to production right away though!)

Checksums

The "h1:" directory hash function uses SHA-256 and is documented at godoc.org/golang.org/x/mod/sumdb/dirhash#Hash1.

You can try it on a zip:

$ goexec 'dirhash.HashZip("v1.0.0.zip", dirhash.Hash1)'
(string)("h1:qbmjkFj7Mx6ih7qhHhptxkar8UII0/4eV6ErleN5PRM=")
(interface{})(nil)

Or a go.mod file:

$ goexec 'dirhash.Hash1(
    []string{"go.mod"},
    func (string) (io.ReadCloser, error) {
        return os.Open("v1.0.0.mod")
    })'
(string)("h1:sq+2TcA+fKd4qCU0iVD1kEphBJLlgD/0mS5W+vKb7FE=")
(interface{})(nil)

Step 2

Module proxy protocol

• A more efficient way to get just the go.mod of a module.

• Uniform access to Go packages (zip) and module metadata (info, mod).

GET $GOPROXY/<module>/@v/list
GET $GOPROXY/<module>/@v/<version>.info
GET $GOPROXY/<module>/@v/<version>.mod
GET $GOPROXY/<module>/@v/<version>.zip

• Note, the <module> and <version> elements are case-encoded.

Escaped paths

package module // import "golang.org/x/mod/module"

func EscapePath(path string) (escaped string, err error)
func EscapeVersion(v string) (escaped string, err error)

func UnescapePath(escaped string) (path string, err error)
func UnescapeVersion(escaped string) (v string, err error)

For example:

github.com/Azure/azure-sdk-for-go             <-> github.com/!azure/azure-sdk-for-go
github.com/GoogleCloudPlatform/cloudsql-proxy <-> github.com/!google!cloud!platform/cloudsql-proxy
github.com/Sirupsen/logrus                    <-> github.com/!sirupsen/logrus
github.com/shurcooL/githubv4                  <-> github.com/shurcoo!l/githubv4

Step 3

Simple dynamic proxy

Can play with:

// List endpoint.
http.HandleFunc("/modproxy/nyc.example/@v/list", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "v1.0.0\n")
})

// Serve the .info file.
http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.info", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "{\n\t\"Version\": \"v1.0.0\",\n\t\"Time\": \"2019-05-04T15:44:36Z\"\n}\n")
})

// Serve the .mod file.
http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.mod", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "module nyc.example\n")
})

Simple dynamic proxy

http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.zip", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "application/zip")
    z := zip.NewWriter(w)
    for _, file := range []struct {
        Name, Body string
    }{
        {"nyc.example@v1.0.0/go.mod", "module nyc.example\n"},
        {"nyc.example@v1.0.0/p.go", "package p\n\n// Life is the answer.\nconst Life = 42\n"},
    } {
        f, err := z.Create(file.Name)
        if err != nil {
            panic(err)
        }
        _, err = f.Write([]byte(file.Body))
        if err != nil {
            panic(err)
        }
    }
    err := z.Close()
    if err != nil {
        panic(err)
    }
})

Previously in Go... (reminder)

GOPATH mode:

• Development likely use git, hg, ...
• Releasing must use git, hg, ... - must push to branch
• Downloading must use git, hg, ...

go get [-u] import/path

Now in Go...

Module mode:

• Development likely use git, hg, ...
• Releasing likely use git, hg, ... - can push to branch, or make tag or maybe mod (zip, mod, info) - custom release setup # rarely done, but possible
• Downloading go get import/path@version

can use git, hg, ...         - direct from VCS
can use mod (zip, mod, info) - via module proxy  # very common thanks to Module Mirror!

Observation

• Releasing workflow no longer has to == development.

• Now's your chance to create that new Version Control System!

Step 4

Look, no git!

export PATH="$HOME/bin/nogit:$PATH"
git version

Try to download a module:

mkmoddirect
go get -d dmitri.shuralyov.com/text/kebabcase

Again with a proxy:

export GOPROXY=https://proxy.golang.org,direct
go get -d dmitri.shuralyov.com/text/kebabcase

Some thoughts

Reading module zips can be useful for:

• Accessing a large corpus of Go code in a uniform way.

• More? (To be discovered by you!)

Some thoughts

Writing module zips can be useful for:

• Don't do it! (Or take full responsibility for it if you do.)

• Performance? (If you're serving git repositories yourself...)

• More? (To be discovered by you!)

Right now, go mod download is the only officially supported way to create module zip files.

One day...

One day...

One day...

One day...

One day...

Conclusion

• Module zips are great for reading, offer a unified viewport. No need for git, hg, svn clients.

• Possible to write zips, but you assume full responsibility; most likely no need.

• Follow me on twitter.com/dmitshur for more quirky Go stuff!

Thank you