Implementing the SSE Protocol | Real-time Apps with Go and ReactJS

About the Series

Welcome to the second part of a series of tutorials called Real-time Applications with Go and ReactJS. We will build a live dashboard that monitors servers and receives webhooks via GitlabCI (or any CI really), rendering this data live to every client that is connected without any delay.

After the series, this is what the app will look like:

We are going to cover every step from an empty text editor to configuring your CI for the dashboard and deploying the docker container we build.

Technologies covered

• GoLang for the server
• HTTP/2 and Server-Sent Events (SSE)
• ReactJS for the client
• Docker for building the app and deploying it

Series Index

• Part 1: Building a real-time HTTP Server
• Part 2: Implementing SSE Protocol Standards (you are here)
• Part 3: Creating a Basic UI with ReactJS
• Part 4: Visualizing Real-time Data with ReactJS (not yet released)
• Part 5: Getting Ready for Production (not yet released)

Additional Information: Definition of real-time in Computing

After reviewing the tutorial with my colleague @schaeferthomas, he stated that "real-time" could be understood in different ways. For this tutorial I use it in the context of Public Networking Applications using the following definition:

[adjective] (real-time)Computation of or relating to a system in which input data is processed within milliseconds so that it is available virtually immediately as feedback, e.g., in missile guidance or airline booking system.

• Oxford Pocket Dictionary of Current English

Introduction

We will extend the server we've built in the last part to send correct data according to the SSE Standard. The working code from the last part is available here: https://gist.github.com/fdrechsler/a20e8d2b8ff656db3bff9533e957be0c

Step 1: The SSE Protocol

Protocol Definition

The ServerSentEvents (or EventSource) protocol is a W3C standard which allows the server to push data to the client. In the past, there have been several approaches to achieve that in web applications, e.g. long-polling or WebSockets. I think SSE is what most of the Real-World-Applications actually should use instead of Websockets. (Hence, it's compatible with HTTP/2)

If you want to read the whole standard, feel free: https://html.spec.whatwg.org/multipage/server-sent-events.html

I will cover the very essentials here.

1. The protocol knows 4 keywords:

• id: {int}\n (optional)
• retry: {int}\n (optional)
• event: {string}\n (optional, defaults to 'message')
• data: {string}\n

2. Multiline data has to begin with data:
3. Every event has to end with a double endline \n\n

Example of a message with JSON data:

retry: 100
event: newmessage
data: {"author": "Someone",
data: "message": "Something"}\n
\n

Added the double \n for you to see how double endlines end the event.

Step 2: Creating a Function to form valid SSE Push Data

We create a function formatSSE that takes two arguments:

1. a string that represents the event-name. event
2. a string for the payload in UTF-8 data

The return value will be a []byte array because that's what our Write function processes.

formatSSE(event string, data string) []byte

In the body of the function we go through the protocol, step by step:

Write the Event-Name

Initializing a payload variable with the event name, closing with a new line.

payload := "event: " + event + "\n"

Split Payload by Line

Multi-line data needs to begin with data: on every line. First, we split the lines by their line-breaks into an array. **

dataLines := strings.Split(data, "\n")

Then we loop over the array and add the data: entries one by one to our payload.

for _, line := range dataLines {
 eventPayload = eventPayload + "data: " + line + "\n"
 }

Terminate the EventStream Chunk

Before converting the eventPayload string to a []byte type, we append one more newline.

return []byte(eventPayload + "\n")

Adding the extra \n is the Standard in the protocol to say that this is the end of the event.

Code of the Function

func formatSSE(event string, data string) []byte {
    eventPayload := "event: " + event + "\n"
    dataLines := strings.Split(data, "\n")
    for _, line := range dataLines {
        eventPayload = eventPayload + "data: " + line + "\n"
    }
    return []byte(eventPayload + "\n")
}

Step 3: Encoding JSON in the /say Handler

We are dealing with JavaScript on the client side this time, so it is a good idea to send JSON formatted data. The core module encoding/json will provide us a "Marshal" function that exerts multiple interfaces and returns a []byte.

import "encoding/json"

In the sayHandler we raise a JSON structure out of the name and message we receive and write it to the message channel instead of the plain text we had before:

func sayHandler(w http.ResponseWriter, r *http.Request) {

    name := r.FormValue("name")
    message := r.FormValue("message")

    jsonStructure, _ := json.Marshal(map[string]string{
        "name": name,
        "message": message})
//......
    messageChannel <- []byte(jsonStructure)

Step 4: Using the formatSSE function in the /listen handler

Since we already implemented all the logic needed for a simple SSE server, we only need to change the Write in the listenHandler method to take the return value from our formatSSE function instead of plain data.

        case _msg := <-_messageChannel:
            w.Write(formatSSE("message", string(_msg)))
            w.(http.Flusher).Flush()

You can try it out with the same curl commands we used earlier in the series:

curl localhost:4000/listen

connects a listener

curl http://localhost:4000/say -d "name=Florian&message=A new Message"

sends a message to all listeners

You should see an output like this on your listener:

event: message
data: {"name":"Florian","message":"A new Message"}

Step 5: Getting the Browser involved

The real browser side is covered in the following two parts, but to test an HTML5 spec protocol we probably won't get around involving the browser early.

touch test.html

Moreover, add the following example script to any HTML page:

    let eventListener = new EventSource("http://localhost:4000/listen")
            eventListener.onmessage = (event) => {
                let {type, data} = event
                alert(`received event: ${type} with data: ${data}`)
            }

Attention: depending on your local setup you might need cors headers to work correctly. The simple fix is to add w.Header().Set("Access-Control-Allow-Origin", "*") at the beginning of your listenFunc.

Try it out

Open the test.html with the added JavaScript from above in your browser.

Emit a new event by:

curl http://localhost:4000/say -d "name=Florian&message=A new Message"

You will see something similar to this in your browser:

Conclusion

We implemented a working SSE protocol from scratch. You should now understand what SSE is and how it works. That's a great start to do something with!

Here is the full working code example:

// main.go
package main

import (
    "encoding/json"
    "log"
    "net/http"
    "strings"
)

func formatSSE(event string, data string) []byte {
    eventPayload := "event: " + event + "\n"
    dataLines := strings.Split(data, "\n")
    for _, line := range dataLines {
        eventPayload = eventPayload + "data: " + line + "\n"
    }
    return []byte(eventPayload + "\n")
}

var messageChannels = make(map[chan []byte]bool)

func sayHandler(w http.ResponseWriter, r *http.Request) {
    name := r.FormValue("name")
    message := r.FormValue("message")

    jsonStructure, _ := json.Marshal(map[string]string{
        "name":    name,
        "message": message})

    go func() {
        for messageChannel := range messageChannels {
            messageChannel <- []byte(jsonStructure)
        }
    }()

    w.Write([]byte("ok."))
}

func listenHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Access-Control-Allow-Origin", "*")

    _messageChannel := make(chan []byte)
    messageChannels[_messageChannel] = true

    for {
        select {
        case _msg := <-_messageChannel:
            w.Write(formatSSE("message", string(_msg)))
            w.(http.Flusher).Flush()
        case <-r.Context().Done():
            delete(messageChannels, _messageChannel)
            return
        }
    }
}

func main() {
    http.HandleFunc("/say", sayHandler)
    http.HandleFunc("/listen", listenHandler)

    log.Println("Running at :4000")
    log.Fatal(http.ListenAndServe(":4000", nil))
}

<!-- test.html -->
<script type="text/javascript">
    let eventListener = new EventSource("http://localhost:4000/listen")
            eventListener.onmessage = (event) => {
                let {type, data} = event
                alert(`received event: ${type} with data: ${data}`)
            }
</script>

Thanks for Reading