SFTP is widely considered to be a standard and secure protocol through which parties can safely transfer files and data. However, the matter of actually engaging with an SFTP server can be quite troublesome, especially if you’re using the always under-documented Go. Which brings us to this post right here: By the end of the following tutorial, you will be able to utilize and connect to SFTP with your hands tied behind your back!
Requirements
As always, preparation comes first.
You're going to need an SFTP server to connect to. If you don't have one, you can set up an SFTP endpoint on SFTP To Go in less than 30 seconds.
The libraries github.com/pkg/sftp and golang.org/x/crypto/ are required in order to connect and interact with an SFTP server. When you are ready to install them, manually run:
$ go get github.com/pkg/sftp
$ go get golang.org/x/crypto/sshOr create a go.mod file and declare your dependencies in it. Go.mod files define Go modules, which, amongst other things, is used to add dependencies to other Go modules. Save the following as go.mod:
module sftptogo.com/examples/go
go 1.13
require (
github.com/pkg/sftp v1.12.0
golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897
)Connecting to SFTP and Disconnecting
In this post, we’ll be using an environment variable named SFTPTOGO_URL that contains all the information required to connect to an SFTP server in a URI format: sftp://user:password@host. When you use SFTP To Go as an Heroku add-on, this variable is automatically created in your app and contains all required information. In the code below, the variable is parsed to extract the URI parts, and the remote server’s host key is fetched from the known_hosts file to identify the remote host.
Once the connection is established, the SFTP client object will be assigned to the variable: sc.
package main
import (
"fmt"
"io"
"os"
"net"
"net/url"
"bufio"
"strings"
"path/filepath"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/agent"
"github.com/pkg/sftp"
)
func main() {
// Get SFTP To Go URL from environment
rawurl := os.Getenv("SFTPTOGO_URL")
parsedUrl, err := url.Parse(rawurl)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to parse SFTP To Go URL: %s\n", err)
os.Exit(1)
}
// Get user name and pass
user := parsedUrl.User.Username()
pass, _ := parsedUrl.User.Password()
// Parse Host and Port
host := parsedUrl.Host
// Default SFTP port
port := 22
hostKey := getHostKey(host)
fmt.Fprintf(os.Stdout, "Connecting to %s ...\n", host)
var auths []ssh.AuthMethod
// Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses
// for communication with other processes.
if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
}
// Use password authentication if provided
if pass != "" {
auths = append(auths, ssh.Password(pass))
}
// Initialize client configuration
config := ssh.ClientConfig{
User: user,
Auth: auths,
// Uncomment to ignore host key check
//HostKeyCallback: ssh.InsecureIgnoreHostKey(),
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
addr := fmt.Sprintf("%s:%d", host, port)
// Connect to server
conn, err := ssh.Dial("tcp", addr, &config)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to connecto to [%s]: %v\n", addr, err)
os.Exit(1)
}
defer conn.Close()
// Create new SFTP client
sc, err := sftp.NewClient(conn)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to start SFTP subsystem: %v\n", err)
os.Exit(1)
}
defer sc.Close()
}
// Get host key from local known hosts
func getHostKey(host string) ssh.PublicKey {
// parse OpenSSH known_hosts file
// ssh or use ssh-keyscan to get initial key
file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to read known_hosts file: %v\n", err)
os.Exit(1)
}
defer file.Close()
scanner := bufio.NewScanner(file)
var hostKey ssh.PublicKey
for scanner.Scan() {
fields := strings.Split(scanner.Text(), " ")
if len(fields) != 3 {
continue
}
if strings.Contains(fields[0], host) {
var err error
hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
if err != nil {
fmt.Fprintf(os.Stderr, "Error parsing %q: %v\n", fields[2], err)
os.Exit(1)
}
break
}
}
if hostKey == nil {
fmt.Fprintf(os.Stderr, "No hostkey found for %s", host)
os.Exit(1)
}
return hostKey
}Listing Files
Now that we have set up a connection, we can use it to list files on the remote SFTP server. This is done by calling the listFiles function, and passing both the SFTP client (sc) and the remote directory path to the listfiles function as arguments. An example call would look like this: listFiles(*sc, "."). The function prints out the name, modification timestamp, and size of the files in the SFTP server.
func listFiles(sc sftp.Client, remoteDir string) (err error) {
fmt.Fprintf(os.Stdout, "Listing [%s] ...\n\n", remoteDir)
files, err := sc.ReadDir(remoteDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to list remote dir: %v\n", err)
return
}
for _, f := range files {
var name, modTime, size string
name = f.Name()
modTime = f.ModTime().Format("2006-01-02 15:04:05")
size = fmt.Sprintf("%12d", f.Size())
if f.IsDir() {
name = name + "/"
modTime = ""
size = "PRE"
}
// Output each file name and size in bytes
fmt.Fprintf(os.Stdout, "%19s %12s %s\n", modTime, size, name)
}
return
}Upload File
The next step is to upload a file. Use the uploadFile function and pass the following arguments: the SFTP client, the path to the local file, and the remote path (which is where the file should end up after we upload). A function call would look like this: uploadFile(*sc, "./local.txt", "./remote.txt")
// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
fmt.Fprintf(os.Stdout, "Uploading [%s] to [%s] ...\n", localFile, remoteFile)
srcFile, err := os.Open(localFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
return
}
defer srcFile.Close()
// Make remote directories recursion
parent := filepath.Dir(remoteFile)
path := string(filepath.Separator)
dirs := strings.Split(parent, path)
for _, dir := range dirs {
path = filepath.Join(path, dir)
sc.Mkdir(path)
}
// Note: SFTP To Go doesn't support O_RDWR mode
dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY|os.O_CREATE|os.O_TRUNC))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
return
}
defer dstFile.Close()
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to upload local file: %v\n", err)
os.Exit(1)
}
fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
return
}Download File
We’re almost done! We just need to download our files. Use the downloadFile function, and pass the SFTP client, the path to the remote file, and the local path in which to store the downloaded file to the function. You would call the function like this: downloadFile(*sc, "./remote.txt", "./download.txt")
// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {
fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)
// Note: SFTP To Go doesn't support O_RDWR mode
srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
return
}
defer srcFile.Close()
dstFile, err := os.Create(localFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
return
}
defer dstFile.Close()
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)
os.Exit(1)
}
fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
return
}The Whole Thing
So we’ve made it to the end! If you would like to run the entire program from start to finish, copy the following code and save it as main.go:
package main
import (
"fmt"
"io"
"os"
"net"
"net/url"
"bufio"
"strings"
"path/filepath"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/agent"
"github.com/pkg/sftp"
)
func main() {
// Get SFTP To Go URL from environment
rawurl := os.Getenv("SFTPTOGO_URL")
parsedUrl, err := url.Parse(rawurl)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to parse SFTP To Go URL: %s\n", err)
os.Exit(1)
}
// Get user name and pass
user := parsedUrl.User.Username()
pass, _ := parsedUrl.User.Password()
// Parse Host and Port
host := parsedUrl.Host
// Default SFTP port
port := 22
hostKey := getHostKey(host)
fmt.Fprintf(os.Stdout, "Connecting to %s ...\n", host)
var auths []ssh.AuthMethod
// Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses
// for communication with other processes.
if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
}
// Use password authentication if provided
if pass != "" {
auths = append(auths, ssh.Password(pass))
}
// Initialize client configuration
config := ssh.ClientConfig{
User: user,
Auth: auths,
// Uncomment to ignore host key check
//HostKeyCallback: ssh.InsecureIgnoreHostKey(),
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
addr := fmt.Sprintf("%s:%d", host, port)
// Connect to server
conn, err := ssh.Dial("tcp", addr, &config)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to connecto to [%s]: %v\n", addr, err)
os.Exit(1)
}
defer conn.Close()
// Create new SFTP client
sc, err := sftp.NewClient(conn)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to start SFTP subsystem: %v\n", err)
os.Exit(1)
}
defer sc.Close()
//*
//* List working directory files
//*
listFiles(*sc, ".")
fmt.Fprintf(os.Stdout, "\n")
//*
//* Upload local file to remote file
//*
uploadFile(*sc, "./local.txt", "./remote.txt")
fmt.Fprintf(os.Stdout, "\n")
//*
//* Download remote file to local file
//*
downloadFile(*sc, "./remote.txt", "./download.txt")
fmt.Fprintf(os.Stdout, "\n")
}
func listFiles(sc sftp.Client, remoteDir string) (err error) {
fmt.Fprintf(os.Stdout, "Listing [%s] ...\n\n", remoteDir)
files, err := sc.ReadDir(remoteDir)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to list remote dir: %v\n", err)
return
}
for _, f := range files {
var name, modTime, size string
name = f.Name()
modTime = f.ModTime().Format("2006-01-02 15:04:05")
size = fmt.Sprintf("%12d", f.Size())
if f.IsDir() {
name = name + "/"
modTime = ""
size = "PRE"
}
// Output each file name and size in bytes
fmt.Fprintf(os.Stdout, "%19s %12s %s\n", modTime, size, name)
}
return
}
// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
fmt.Fprintf(os.Stdout, "Uploading [%s] to [%s] ...\n", localFile, remoteFile)
srcFile, err := os.Open(localFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
return
}
defer srcFile.Close()
// Make remote directories recursion
parent := filepath.Dir(remoteFile)
path := string(filepath.Separator)
dirs := strings.Split(parent, path)
for _, dir := range dirs {
path = filepath.Join(path, dir)
sc.Mkdir(path)
}
// Note: SFTP To Go doesn't support O_RDWR mode
dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY|os.O_CREATE|os.O_TRUNC))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
return
}
defer dstFile.Close()
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to upload local file: %v\n", err)
os.Exit(1)
}
fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
return
}
// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {
fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)
// Note: SFTP To Go doesn't support O_RDWR mode
srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)
return
}
defer srcFile.Close()
dstFile, err := os.Create(localFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)
return
}
defer dstFile.Close()
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)
os.Exit(1)
}
fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)
return
}
// Get host key from local known hosts
func getHostKey(host string) ssh.PublicKey {
// parse OpenSSH known_hosts file
// ssh or use ssh-keyscan to get initial key
file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to read known_hosts file: %v\n", err)
os.Exit(1)
}
defer file.Close()
scanner := bufio.NewScanner(file)
var hostKey ssh.PublicKey
for scanner.Scan() {
fields := strings.Split(scanner.Text(), " ")
if len(fields) != 3 {
continue
}
if strings.Contains(fields[0], host) {
var err error
hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
if err != nil {
fmt.Fprintf(os.Stderr, "Error parsing %q: %v\n", fields[2], err)
os.Exit(1)
}
break
}
}
if hostKey == nil {
fmt.Fprintf(os.Stderr, "No hostkey found for %s", host)
os.Exit(1)
}
return hostKey
}Finally, run it using the command:
go main.goAll done! Congratulations on connecting to SFTP using Go!
Level-up your skills with some more code samples on Github.