About the Series
Welcome to the first part of a series of tutorials called Real-time Applications with Go and ReactJS. We will build a live dashboard that monitors servers and receives webhooks via GitlabCI (or any CI really), rendering this data live to every client that is connected without any delay.
After the series, this is what the app will look like:
We are going to cover every step from an empty text editor to configuring your CI for the dashboard and deploying the docker container we build.
Technologies covered
- GoLang for the server
- HTTP/2 and Server-Sent Events (SSE)
- ReactJS for the client
- Docker for building the app and deploying it
Series Index
- Part 1: Building a real-time HTTP Server (you are here)
- Part 2: Implementing SSE Protocol Standards
- Part 3: Creating a Basic UI with ReactJS
- Part 4: Visualizing Real-time Data with ReactJS (not yet released)
- Part 5: Getting Ready for Production (not yet released)
Additional Information: Definition of real-time
in Computing
After reviewing the tutorial with my colleague @schaeferthomas, he stated that "real-time" could be understood in different ways. For this tutorial I use it in the context of Public Networking Applications using the following definition:
[adjective] (real-time)Computation of or relating to a system in which input data is processed within milliseconds so that it is available virtually immediately as feedback, e.g., in missile guidance or airline booking system.
- Oxford Pocket Dictionary of Current English
Introduction: Building a real-time HTTP Server
Prerequisites
- Basic networking knowledge
- Basic knowledge of HTTP
- Some experience in parallel or concurrent programming
- Minimal knowledge of Go
If you have not yet had any contact with golang, don't worry, I am not going to dive deep into the mechanics of golang. You should be able to follow along even if you haven't used golang yet. However, wouldn't this be a great time to make the first contact with Go?
To follow along without any knowledge of golang, I recommend using gobyexample.com as a reference.
What we are going to build in this part
We are going to use the core net/http
package to build a very basic real-time server that keeps connections to an endpoint /listen
alive and takes input at /say
.
Results at over 9000 Requests per Second
Requests [total, rate] 45005, 9001.05
Duration [total, attack, wait] 5.000096946s, 4.99997s, 126.946µs
Latencies [mean, 50, 95, 99, max] 132.54µs, 126.556µs, 174.755µs, 255.119µs, 3.755665ms
Success [ratio] 100.00%
Conclusion: Go is tremendously fast at doing stuff not necessarily in parallel but in concurrency.
Compiler
Code goes to main.go
, create it:
touch main.go
Then add the template for our application in your favourite editor:
package main
import "log"
func main() {
log.Println("Starting with Go")
}
You need a golang compiler. For development, I would recommend installing golang and using the builtin development compiler.
go run main.go
Alternative: Build with Docker (NOT recommended, go run is faster in dev)
docker run --rm -v "$PWD":/app -w /app -e GOOS=$(uname -s | tr '[A-Z]' '[a-z]') golang:1.12-alpine go build main.go
Then execute with
./main
Step 1 - Implementing the /say
Handler
Starting an HTTP server in Go is very straight forward. The core package net/http
provides the ListenAndServe(address string, handler Handler)
function. The function runs till it may receive an unrecoverable error, returning the error message. Since it is blocking, you should add the statement at the end of func main
.
log.Fatal( http.ListenAndServe(":4000", nil) )
We implement HTTP handlers with the http.HandleFunc(urlPattern string, handlerFunction Handler)
function. It takes a pattern that describes the URL, in our example /say
and a callback function that is going to execute on any request to that URL.
The callback function receives a ResponseWriter
interface which has a Write([]byte])
function.
The write method takes a byte array. That's great for HTTP/2 which is a binary protocol, unlike HTTP.
In our case, we want to return a UTF-8 string. Gladly, this isn't C (even if it looks like it is) and the byte array type has a very convenient interface for converting our string to a byte array: []byte("string here")
.
Now we stick the parts together:
package main
import "net/http"
func sayHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hi"))
}
func main() {
http.HandleFunc("/say", sayHandler)
http.ListenAndServe(":4000", nil)
}
Testing it with curl
:
$ curl localhost:4000/say
Hi%
Step 2 - Processing Input-Data
The Web and HTTP(S)(/2) is a core construct in Go; actually, golang was made for web development and networking.
Of course, it comes with parsing functions for URL and POST/PATCH/PUT body.
Request.FormValue(key String)
returns a string with the value of the key.
We exchange the static "Hi" with the string we read from a requests URL or body.
func sayHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(r.FormValue("name")))
}
Test: curl
(or open it in any web browser)
$ curl localhost:4000/say -d 'name=Florian'
Florian%+
For our application, we need another parameter message
.
Usually, in golang, you would create a
struct
now. However, this is not a golang tutorial, let's keep it simple.
func sayHandler(w http.ResponseWriter, r *http.Request) {
name := r.FormValue("name")
message := r.FormValue("message")
w.Write([]byte(name + " " + message))
}
Step 3 - Implementing the /listen
Handler
For the listenHandler
we do the same as we did for the sayHandler
, but without parsing any input.
We instead tell the client that the connection should be kept alive.
We create a new handler listenHandler
and set the HTTP Header "Connection" to "keep-alive" to tell the client not to terminate the connection. Also, we set HTTP Header "Content-Type" to "text/event-stream".
To make sure that we are not terminating the connection from our side early, we wait for the close event of the client.
//......
func listenHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Content-Type", "text/event-stream")
select {
case <-r.Context().Done():
return;
}
}
//......
func main() {
http.HandleFunc("/listen", listenHandler)
//......
}
The arrow syntax <-
belongs to one of the core concepts of concurrency in golang: channel
s, it blocks the routine until it receives data from a channel
.
A
channel
in go is a typed conduit that can receive datachannel <- data
and data can be read fromdata <- channel
. writing to or reading from a channel BLOCKS the subroutine. Example
Step 4 - Connecting the Handlers
We have a /say
endpoint, receiving data from the client. And a /listen
endpoint supposed to send the data we receive on /say
to connected clients.
Now let us combine those. To do that, we need a new channel
for every listener connected to send the data; we list them in a global map of channels like so:
var messageChannels = make(map[chan []byte]bool)
(FAQ) I use a map because later in the series we will have multiple event types.
listenerHandler
Now every new listener should create his messageChannel:
_messageChannel := make(chan []byte)
And then, list it to the messageChannels map:
messageChannels[_messageChannel] = true
In the select
statement of the listenHandler
, we are already waiting for data coming from the requests close channel
before we return the function and end the connection.
Now, we create another case
in the select, which will be waiting for data from the messageChannel
and write the data into the ResponseWriter stream.
func listenHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Connection", "keep-alive")
_messageChannel := make(chan []byte)
messageChannels[_messageChannel] = true
for {
select {
case _msg := <- _messageChannel:
w.Write(append(_msg,[]byte("\r\n")...))
w.(http.Flusher).Flush()
case <-r.Context().Done():
delete(messageChannels, _messageChannel)
return;
}
}
}
w.(http.Flusher).Flush()
flushs buffered data to the client explicitly. (be aware of proxy handling here if in a real world app)
sayHandler
In the sayHandler we write to the messageChannels
the listeners added. We do this in a dedicated thread, so we don't let the client wait till we channelled and processed all the data.
Since concurrency is the core concept of golang, the keyword for creating a new thread is go
.
// sayHandler Function
// ...
//old: w.Write([]byte(name + " " + message))
go func() {
for messageChannel := range messageChannels {
messageChannel <- []byte(name + " " + message)
}
}()
w.Write([]byte("ok"))
pay attention to the
}()
: We are creating an instantly invoking the function.
Conclusion
We just built a real-time chat app in 45 lines of Go.
The /say
endpoint processes name
and message
.
The /listen
endpoint keep-alives connections and forwards input from /say
Test it with curl
or visit localhost:4000/listen in your favourite web browser and send events with curl /say in terminal!
Disclaimer: This is not production ready code, for reasons of simplicity we omitted error checking and input sanitization.
package main
import (
"log"
"net/http"
)
var messageChannels = make(map[chan []byte]bool)
func sayHandler(w http.ResponseWriter, r *http.Request) {
name := r.FormValue("name")
message := r.FormValue("message")
go func() {
for messageChannel := range messageChannels {
messageChannel <- []byte(name + " " + message + "\r\n")
}
}()
w.Write([]byte("ok."))
}
func listenHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Content-Type", "text/event-stream")
_messageChannel := make(chan []byte)
messageChannels[_messageChannel] = true
for {
select {
case _msg := <-_messageChannel:
w.Write(_msg)
w.(http.Flusher).Flush()
case <-r.Context().Done():
delete(messageChannels, _messageChannel)
return
}
}
}
func main() {
http.HandleFunc("/say", sayHandler)
http.HandleFunc("/listen", listenHandler)
log.Println("Running at :4000")
log.Fatal(http.ListenAndServe(":4000", nil))
}
In the next part, we will build the Server-Sent-Event protocol ourself and connect it to a JavaScript client for real-time browser action. Go to the next part