SFTP is widely considered to be a standard and secure protocol through which parties can safely transfer files and data. However, the matter of actually engaging with an SFTP server can be quite troublesome, especially if you’re using the always under-documented Go. Which brings us to this post right here: By the end of the following tutorial, you will be able to utilize and connect to SFTP with your hands tied behind your back!

Requirements

As always, preparation comes first.

You're going to need an SFTP server to connect to. If you don't have one, you can set up an SFTP endpoint on SFTP To Go in less than 30 seconds.

The libraries github.com/pkg/sftp and  golang.org/x/crypto/ are required in order to connect and interact with an SFTP server. When you are ready to install them, manually run:

$ go get github.com/pkg/sftp
$ go get golang.org/x/crypto/ssh

Or create a go.mod file and declare your dependencies in it. Go.mod files define Go modules, which, amongst other things, is used to add dependencies to other Go modules. Save the following as go.mod:

module sftptogo.com/examples/go

go 1.13

require (
	github.com/pkg/sftp v1.12.0
	golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897
)
mod.go

Connecting to SFTP and Disconnecting

In this post, we’ll be using an environment variable named SFTPTOGO_URL that contains all the information required to connect to an SFTP server in a URI format: sftp://user:password@host. When you use SFTP To Go as an Heroku add-on, this variable is automatically created in your app and contains all required information. In the code below, the variable is parsed to extract the URI parts, and the remote server’s host key is fetched from the known_hosts file to identify the remote host.

Once the connection is established, the SFTP client object will be assigned to the variable: sc.

package main

import (
    "fmt"
    "io"
    "os"
    "net"
    "net/url"
    "bufio"
    "strings"
    "path/filepath"

    "golang.org/x/crypto/ssh"
    "golang.org/x/crypto/ssh/agent"

    "github.com/pkg/sftp"
)

func main() {
    // Get SFTP To Go URL from environment
    rawurl := os.Getenv("SFTPTOGO_URL")

    parsedUrl, err := url.Parse(rawurl)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to parse SFTP To Go URL: %s\n", err)
        os.Exit(1)
    }

    // Get user name and pass
    user := parsedUrl.User.Username()
    pass, _ := parsedUrl.User.Password()

    // Parse Host and Port
    host := parsedUrl.Host
    // Default SFTP port
    port := 22

    hostKey := getHostKey(host)

    fmt.Fprintf(os.Stdout, "Connecting to %s ...\n", host)

    var auths []ssh.AuthMethod

    // Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses 
    // for communication with other processes.
    if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
        auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
    }

    // Use password authentication if provided
    if pass != "" {
        auths = append(auths, ssh.Password(pass))
    }
    
    // Initialize client configuration
    config := ssh.ClientConfig{
        User: user,
        Auth: auths,
        // Uncomment to ignore host key check
        //HostKeyCallback: ssh.InsecureIgnoreHostKey(),
        HostKeyCallback: ssh.FixedHostKey(hostKey),
    }

    addr := fmt.Sprintf("%s:%d", host, port)

    // Connect to server
    conn, err := ssh.Dial("tcp", addr, &config)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to connecto to [%s]: %v\n", addr, err)
        os.Exit(1)
    }

    defer conn.Close()

    // Create new SFTP client
    sc, err := sftp.NewClient(conn)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to start SFTP subsystem: %v\n", err)
        os.Exit(1)
    }
    defer sc.Close()
}

// Get host key from local known hosts
func getHostKey(host string) ssh.PublicKey {
    // parse OpenSSH known_hosts file
    // ssh or use ssh-keyscan to get initial key
    file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to read known_hosts file: %v\n", err)
        os.Exit(1)
    }
    defer file.Close()
 
    scanner := bufio.NewScanner(file)
    var hostKey ssh.PublicKey
    for scanner.Scan() {
        fields := strings.Split(scanner.Text(), " ")
        if len(fields) != 3 {
            continue
        }
        if strings.Contains(fields[0], host) {
            var err error
            hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
            if err != nil {
                fmt.Fprintf(os.Stderr, "Error parsing %q: %v\n", fields[2], err)
                os.Exit(1)
            }
            break
        }
    }
 
    if hostKey == nil {
        fmt.Fprintf(os.Stderr, "No hostkey found for %s", host)
        os.Exit(1)
    }
 
    return hostKey
}
connect.go

Listing Files

Now that we have set up a connection, we can use it to list files on the remote SFTP server. This is done by calling the listFiles function, and passing both the SFTP client (sc) and the remote directory path to the listfiles function as arguments. An example call would look like this: listFiles(*sc, "."). The function prints out the name, modification timestamp, and size of the files in the SFTP server.

func listFiles(sc sftp.Client, remoteDir string) (err error) {
    fmt.Fprintf(os.Stdout, "Listing [%s] ...\n\n", remoteDir)
    
    files, err := sc.ReadDir(remoteDir)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to list remote dir: %v\n", err)
        return
    }

    for _, f := range files {
        var name, modTime, size string

        name = f.Name()
        modTime = f.ModTime().Format("2006-01-02 15:04:05")
        size = fmt.Sprintf("%12d", f.Size())

        if f.IsDir() {
            name = name + "/"
            modTime = ""
            size = "PRE"
        }
        // Output each file name and size in bytes
        fmt.Fprintf(os.Stdout, "%19s %12s %s\n", modTime, size, name)
    }

    return
}
listfiles.go

Upload File

The next step is to upload a file. Use the uploadFile function and pass the following arguments: the SFTP client, the path to the local file, and the remote path (which is where the file should end up after we upload). A function call would look like this: uploadFile(*sc, "./local.txt", "./remote.txt")

// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
    fmt.Fprintf(os.Stdout, "Uploading [%s] to [%s] ...\n", localFile, remoteFile)

    srcFile, err := os.Open(localFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
        return
    }
    defer srcFile.Close()

    // Make remote directories recursion
    parent := filepath.Dir(remoteFile)
    path := string(filepath.Separator)
    dirs := strings.Split(parent, path)
    for _, dir := range dirs {
        path = filepath.Join(path, dir)
        sc.Mkdir(path)
    }

    // Note: SFTP To Go doesn't support O_RDWR mode
    dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY|os.O_CREATE|os.O_TRUNC))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
        return
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to upload local file: %v\n", err)
        os.Exit(1)
    }
    fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
    
    return
}
uploadfile.go

Download File

We’re almost done! We just need to download our files. Use the downloadFile function, and pass the SFTP client, the path to the remote file, and the local path in which to store the downloaded file to the function. You would call the function like this: downloadFile(*sc, "./remote.txt", "./download.txt")

// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {

    fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)
    // Note: SFTP To Go doesn't support O_RDWR mode
    srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
        return
    }
    defer srcFile.Close()

    dstFile, err := os.Create(localFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
        return
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)
        os.Exit(1)
    }
    fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
    
    return
}
downloadfile.go

The Whole Thing

So we’ve made it to the end! If you would like to run the entire program from start to finish, copy the following code and save it as main.go:

package main

import (
    "fmt"
    "io"
    "os"
    "net"
    "net/url"
    "bufio"
    "strings"
    "path/filepath"

    "golang.org/x/crypto/ssh"
    "golang.org/x/crypto/ssh/agent"

    "github.com/pkg/sftp"
)

func main() {
    // Get SFTP To Go URL from environment
    rawurl := os.Getenv("SFTPTOGO_URL")

    parsedUrl, err := url.Parse(rawurl)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to parse SFTP To Go URL: %s\n", err)
        os.Exit(1)
    }

    // Get user name and pass
    user := parsedUrl.User.Username()
    pass, _ := parsedUrl.User.Password()

    // Parse Host and Port
    host := parsedUrl.Host
    // Default SFTP port
    port := 22

    hostKey := getHostKey(host)

    fmt.Fprintf(os.Stdout, "Connecting to %s ...\n", host)

    var auths []ssh.AuthMethod

    // Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses 
    // for communication with other processes.
    if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
        auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
    }

    // Use password authentication if provided
    if pass != "" {
        auths = append(auths, ssh.Password(pass))
    }
    
    // Initialize client configuration
    config := ssh.ClientConfig{
        User: user,
        Auth: auths,
        // Uncomment to ignore host key check
        //HostKeyCallback: ssh.InsecureIgnoreHostKey(),
        HostKeyCallback: ssh.FixedHostKey(hostKey),
    }

    addr := fmt.Sprintf("%s:%d", host, port)

    // Connect to server
    conn, err := ssh.Dial("tcp", addr, &config)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to connecto to [%s]: %v\n", addr, err)
        os.Exit(1)
    }

    defer conn.Close()

    // Create new SFTP client
    sc, err := sftp.NewClient(conn)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to start SFTP subsystem: %v\n", err)
        os.Exit(1)
    }
    defer sc.Close()

    //*
    //* List working directory files
    //*
    listFiles(*sc, ".")
    fmt.Fprintf(os.Stdout, "\n")

    //*
    //* Upload local file to remote file
    //*
    uploadFile(*sc, "./local.txt", "./remote.txt")
    fmt.Fprintf(os.Stdout, "\n")

    //*
    //* Download remote file to local file
    //*
    downloadFile(*sc, "./remote.txt", "./download.txt")
    fmt.Fprintf(os.Stdout, "\n")
}

func listFiles(sc sftp.Client, remoteDir string) (err error) {
    fmt.Fprintf(os.Stdout, "Listing [%s] ...\n\n", remoteDir)
    
    files, err := sc.ReadDir(remoteDir)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to list remote dir: %v\n", err)
        return
    }

    for _, f := range files {
        var name, modTime, size string

        name = f.Name()
        modTime = f.ModTime().Format("2006-01-02 15:04:05")
        size = fmt.Sprintf("%12d", f.Size())

        if f.IsDir() {
            name = name + "/"
            modTime = ""
            size = "PRE"
        }
        // Output each file name and size in bytes
        fmt.Fprintf(os.Stdout, "%19s %12s %s\n", modTime, size, name)
    }

    return
}


// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
    fmt.Fprintf(os.Stdout, "Uploading [%s] to [%s] ...\n", localFile, remoteFile)

    srcFile, err := os.Open(localFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
        return
    }
    defer srcFile.Close()

    // Make remote directories recursion
    parent := filepath.Dir(remoteFile)
    path := string(filepath.Separator)
    dirs := strings.Split(parent, path)
    for _, dir := range dirs {
        path = filepath.Join(path, dir)
        sc.Mkdir(path)
    }

    // Note: SFTP To Go doesn't support O_RDWR mode
    dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY|os.O_CREATE|os.O_TRUNC))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
        return
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to upload local file: %v\n", err)
        os.Exit(1)
    }
    fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
    
    return
}

// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {

    fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)
    // Note: SFTP To Go doesn't support O_RDWR mode
    srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
        return
    }
    defer srcFile.Close()

    dstFile, err := os.Create(localFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
        return
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)
        os.Exit(1)
    }
    fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
    
    return
}

// Get host key from local known hosts
func getHostKey(host string) ssh.PublicKey {
    // parse OpenSSH known_hosts file
    // ssh or use ssh-keyscan to get initial key
    file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to read known_hosts file: %v\n", err)
        os.Exit(1)
    }
    defer file.Close()
 
    scanner := bufio.NewScanner(file)
    var hostKey ssh.PublicKey
    for scanner.Scan() {
        fields := strings.Split(scanner.Text(), " ")
        if len(fields) != 3 {
            continue
        }
        if strings.Contains(fields[0], host) {
            var err error
            hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
            if err != nil {
                fmt.Fprintf(os.Stderr, "Error parsing %q: %v\n", fields[2], err)
                os.Exit(1)
            }
            break
        }
    }
 
    if hostKey == nil {
        fmt.Fprintf(os.Stderr, "No hostkey found for %s", host)
        os.Exit(1)
    }
 
    return hostKey
}
main.go

Finally, run it using the command:

go main.go

All done! Congratulations on connecting to SFTP using Go!

Cloud FTP with maximum security and reliability
SFTP To Go offers managed cloud storage service - highly available, reliable and secure. Great for companies of any size, any scale.
Try SFTP To Go for free!

Level-up your skills with some more code samples on Github.