I'm working on a project where I need to process a directory from someone's file system. The flow is as follows:

{client request containing directory and metadata} --> {api http handler that processes client directory}

Pretty straightforward as a flow since it's just passing data from the client to server, but I realized I've never actually sent streaming data with Go before (or ever for that matter). I didn't want to store anything in S3 or use any external APIs as a proxy for getting that data, so we are now on the verge of learning something new!

The purpose of this post is to show how to successfully take a directory from a client's local machine and send it to our server using Go. Luckily, the client request is also being sent using Go as it's a CLI, so we don't have to deal with any multi-language complexity here. Let's begin.

The Client

Tar the Directory to stream

Here's the code for tarring our directory (our input for the request):

// tarCwd creates a tar.gz of the current working directory when the push command is run
func tarCwd() error {
    // get all filenames and ignore dep-related directories
    var fileNames []string
    root, err := os.Getwd()
    if err != nil {
        return err
    }

    files, err := ioutil.ReadDir(root)
    if err != nil {
        return err
    }

    for _, file := range files {
        if !(file.IsDir() && file.Name() == "vendor" || file.Name() == "node_modules") {
            fileNames = append(fileNames, file.Name())
        }
    }

    err = archiver.Archive(fileNames, "deployDir.tar.gz")
    if err != nil {
        return err
    }

    return nil
}

Commentary

We first get the path of the current working directory

root, err := os.Getwd()

Then, we get the list of files we want within our directory (filter out any dependency sub-directories as they're unnecessary and potentially large)

files, err := ioutil.ReadDir(root)
if err != nil {
    return err
}

for _, file := range files {
    if !(file.IsDir() && file.Name() == "vendor" || file.Name() == "node_modules") {
        fileNames = append(fileNames, file.Name())
    }
}

We use a library called archiver to tar our directory (which is an amazing library on its own btw).

err = archiver.Archive(fileNames, "deployDir.tar.gz")
if err != nil {
    return err
}

That's our client code! Simple and straightforward. Get the directory of interest's path, get its list of files minus the ones we don't want and then tar that using a well-supported community library.

The harder part on the client-side comes next: including it within the HTTP request.

The HTTP Request

Here's the code

// sendHTTPRequestWithFile sends a POST request to the specified targetURL with the requested file as the payload
func sendHTTPRequestWithFile(targetURL, fileName string) (*http.Response, error) {
    bodyBuf := &bytes.Buffer{}
    bodyWriter := multipart.NewWriter(bodyBuf)

    fileWriter, err := bodyWriter.CreateFormFile("uploadfile", fileName)
    if err != nil {
        return nil, err
    }

    fh, err := os.Open(fileName)
    if err != nil {
        return nil, err
    }

    defer fh.Close()

    _, err = ioutil.ReadAll(fileWriter, fh)
    if err != nil {
        return nil, err
    }

    contentType := bodyWriter.FormDataContentType()
    bodyWriter.Close()

    resp, err := http.Post(targetURL, contentType, bodyBuf)
    if err != nil {
        return nil, err
    }

    return resp, nil
}