The state of software engineering

11 August 2019

The field of software engineering is in a strange place today. A lot of the mainstream tools and concepts look less like deliberate choices made by intelligent people anticipating change and more like ad-hoc reuse of things some people were already familiar with, despite the problems this may cause at scale. Things I consider to be concrete examples of this are all the major programming languages (Java, C#, Python, C++ that don’t even have algebraic data types that are around since the 70ies), docker containers (where the usual way of constructing them leads to linear dependencies which lead to poor composability), YAML as a configuration format (with its many pitfalls and high complexity), Microservices as almost a default architecture choice - the list goes on. This is not to say that I don’t consider these things useful in certain situations - it just strikes me as odd that the majority of our industry sees little problem with the status quo of defaulting to these options and that there are so few attempt at improving things.

Maybe my experiences with Elm and F# have spoiled me - I was lucky enough to use both on my job for the last three years. I feel like so many of the problems I regularly encountered in the previous ten years of doing mostly C#, some Ruby, Python, Javascript and C++ - are just gone. In Elm (and with some caveats in F#) you don’t need to worry about null values or exceptions; you don’t spend your time discussing pointless ontological questions and creating strange inheritance hierarchies that couple logic depending solely on the answer to the question “Can you say A IS-A B?”.

Customer and Employee inherit from Person Should the code for them be tightly coupled? Customer IS-A Person! Employee IS-A Person!

In Elm, F# and similar languages, you write functions that deal with data, you model data in different forms accurately with algebraic data types and you let the compiler use the types to check if the shape of your data matches in all places. Because values are immutable, state changes only in a few well defined places. Elaborate “magic” like two-way databinding is replaced with simpler patterns like the Elm architecture. Such patterns look a bit more verbose in the beginning, but as the application that uses it grows it always behaves predictably. This is something I have never seen with two-way data binding UI frameworks at non-trivial scale.

Of course types are not everything and using Elm or Haskell etc. to implement some software will not magically make it bug free. But what they afford you to do is to concentrate on the interesting parts of your code and test more interesting properties since the technicalities are taken care of by a compiler. Refactoring complex pieces of software becomes a mundane tasks of following compiler errors instead of the Russian roulette it is in languages without static typing or with a poor type system. Since software tends to grow in size and requirements tend to change, I consider this to be a big advantage.

All that is not so say that languages like Elm, F#, Haskell and Purescript cannot improve further. By listening to the excellent Future of coding podcast I found Pharo, a language strongly inspired by Smalltalk, and the Glamorous toolkit. The Glamorous toolkit places a high value on an interactive canvas that can be used to understand your code. How great would it be if there were a standard way for an ML family language to ship interactive debugging UIs with standard libraries that aid in your debugging and REPL experience? Think e.g. a monadic probability library that visualizes distributions while you work with it in the REPL.

The power of Erlangs VM with hot code swapping and supervision trees is very desirable, even more so if messages could be typed with something like session types. Algebraic effects like in the Eff language look like a very promising way for handling side effects (I/O, concurrency, exceptions, …), so they can be separated from the pure code but without the pain of monad transformer stacks. Datomic which is built in Clojure is a really interesting way of dealing with data queries and between that and Prolog I think there is a lot of unexplored scope of embedding reasoning/analytics engines in ML family languages.

I find it strange that not more programmers see the value of ML like languages for larger projects. Do so few know these languages in enough detail? Is the inertia so powerful, the desire to stay with what you already know? Many of the excuses of a few years ago, like bad editor tooling or small ecosystems apply less and less (and some languages have a strong FFI that lets you tap into other languages’ ecosystems).

Maybe one of the reasons is that evaluating technologies is often done on small examples where ease of setup and the speed of getting started are most important. Problems that only arise in bigger projects show up too late and by this time get confused as “this is just what programming in a large project looks like”.

At my job we have one relatively complex backend written in Python - because at the time GraphQL support in ML languages was not developed far enough. To give praise where it is due, the Django admin interface was also a huge time saver - oh how I would love something like it to exist in F# for the SAFE stack! We were very fast initially in getting the first version out but now every non-trivial change is a pain and we are just so slow in that codebase compared to the F# backends (and we make more mistakes). If I hadn’t had the F# experience, I might’ve mistaken this slowing down for an inevitable fact of refactoring code.

That's a lot of coffee It's a lot of python

Maybe none of this is so surprising given how young our field is. The first people who worked with textual instructions for computers did so from around 1950 - if you assume that the working lifetime of a person is around 40 years then this is not even two generations ago. In that time we scaled from a few hundred to maybe around 20 million programmers. Pretty much all other engineering professions or crafts developed over much longer timespans and with an only moderate change in the number of practitioners from decade to decade. They had time to try ideas and evaluate their outcome over long timespans and to develop cultural institutions like guilds or engineering societies. To complicate matters, computers kept getting faster at an insane pace for most of the last 70 years - so received wisdom from a generation ago is often no longer applicable. Maybe these are just growing pains of a still relatively new field then?

I would be more relaxed about this if it were not for the fact that the stakes are getting higher all the time. We are putting software into ever more areas of human life and into ever more critical situations - but the tools that are mainstream in our profession are error prone and limit our expressivity. Software development is still so often at the same time intellectually challenging and incredibly boring even in the best of cases.

Some things give me hope, e.g. Rust is a great new language that improves many things for system programming from the previous status quo - and it seems to gain traction lately (Microsoft Security recommended it recently).

But when it comes to mainstream adoption, we are still so far behind what is easily possible today. So much software is buggy and needs crazy amounts of development time and is just so brittle all around. Metaphors are always problematic, but it feels a bit like as an industry we insist on building skyscrapers with clay and neglect to learn how to work with steel and concrete for no good reason.

What are you building? Skyscraper. With clay bricks? Sure. Don't you think steel would be better? Don't have time learn that.

I try to spread the word about these ideas with things like the Friendly Functional Programming Meetup Berlin that I run together with Michael, but I often feel like such events mostly reach those that already see the value of functional programming. I think our best bet is to work on creating entire toolchains that work well for pragmatic applications and then showcasing those. I think Elm did a good job on the web frontend side; the SAFE stack does so quite well on the backend as well as the frontend; and Servant does a great job at showing how useful full type level representations of HTTP APIs are because they let you type check both the client and server of a REST API at compile time from the same type definitions.

I think as enthusiasts of these languages we should talk more about the pragmatic ways these powerful tools can enable us to do more work in less time to a higher quality standard. We should try less to explain Kleisli composition to outsiders (not that this is not useful, but the focus should be on the former IMHO). Maybe then we will find ourselves in a future where software is built on top of good abstractions, at high speed, and can be safely changed and modified. Let’s work towards that future.

Using F# on both the frontend and the backend

10 December 2016

Note: This post was written in 2016. Since then, the introduction of the SAFE stack has significantly improved the sitation in F#, but the basic ideas of this blog post are still valid - they are just a lot nicer now :)

When the call for the F# Advent Calendar came up I thought that this would be a great opportunity to finally check out a technology stack that intrigued me for quite a while now: Using F# on the backend (which by now works well on linux/osx as well as windows) and also on the frontend, via Fable, the F# to Javascript compiler backend. I have played a bit with F# in the past but I wanted to see how well this particular combination works in practice.

The same language on the backend and the frontend

Using the same language on the server and the client has many benefits in my opinion, chief among them the ability to reuse data definitions and business logic. Even though many modern web sites shoulder a lot of complexity on the client to guide the user and give immediate feedback, the backend usually has to replicate most of this logic to verify the client submissions and guard against malicious clients. Discount rules are a prime example of such code that is often replicated in both the frontend and the backend.

The first language that will probably come to mind for this job is Javascript. Since the advent of Node, Javascript is available on the backend as well as in the browser, and, with Electron, even for desktop apps with a native feel. Heavy performance tuning has made Javascript reasonably fast. However, I think that its dynamically typed nature makes Javascript a very problematic choice for non-trivial software projects. Static type checking can catch many bugs before they ever hit production and it can free your developers to concentrate on testing the properties of their code that are actually interesting.

A very desirable property for a type system is for it to have Algebraic Data Types. This means that, in addition to Product types (think classes or records) where ALL stated fields exist for every instance, they also support Sum types (Discriminated Unions in F#) where every value adheres to exactly one of several strictly predefined layouts. Together, Product and Sum types allow a much richer modelling where you can make illegal states irrepresentable (see this recent article by Marc Seemann on the benefits of ADTs). Having Sum types also allows languages to get rid of nulls - a very desirable trait since a big fraction of errors in dynamically typed languages or such with weak type systems are NullReferenceExceptions or the infamous “undefined is not a function”.

Possible languages

Ok, now that I have outlined desirable feature, let us look for languages that meet these two core requirements (a strong, static type system with ADTs and being usable via cross-compilation to Javascript on both the browser frontend and the backend). Not a lot of options remain. The ones I know are:

All of these are interesting languages. Here is my very personal reasoning for choosing among these languages: Typescript is a superset of Javascript with type annotations that, since version 2.0, has support for union types. It is a big improvement over raw Javascript, but I think that one still feels a lot of the odd language design decisions underneath it, so I will disregard it for now.

The javascript compiler backends for OCaml and Haskell have a relatively small set of Javascript library bindings available and are IMHO harder to interface with native Javascript libraries - for now I don’t think that they are immediately usable if you want to write user interfaces with them.

This leaves me with F#/Fable and Purescript. Purescript was designed from the beginning with Javascript as the compile target in mind. Quite a few important Javascript libraries have been annotated with type definitions for Purescript and are thus readily usable, and if they are not, the Purescript FFI was built for Javascript and is very straightforward to use. Additionally, the type system of Purescript is very powerful and feels a bit like a modern refresh of Haskell’s type system.

So what about F#/Fable? The basic story is similar to OCaml/Haskell in that you have to create FFI definitions and these are only available for a few libraries so far. What is really neat about Fable though is that there is a way to bootstrap Fable FFI files from Typescript definitions via ts2fable - and those are available for a huge number of important Javascript libraries. I think that this lowers the pain of interfacing with native Javascript libraries enough for F#/Fable to be a reasonable choice.

F# versus Purescript

Purescript is definitely an interesting contender that I want to study more in the future, but this blog post is about F#, so, let us ask the obvious question: why use F# instead of Purescript? YMMV but I see F# to have an interesting advantage on the server side since IMO on the backend the .NET platform is a more solid runtime than Node. Since F# compiles to IL and can interface with any .NET library, it has access to a huge amount of libraries that interface with all sorts of systems, parse files etc. It also supports strong multi-threading primitives and F# even has support to cross compile to the GPU - this makes it much more interesting for numerically intense workloads than Node.

This means that if you either already have a big chunk of business logic written in a .NET language or if you need to do high performance computations on the server, F# is IMO a more attractive choice than Purescript. Additionally, because F# is more in the tradition of pragmatic functional languages like OCaml than the Haskell tradition of that Purescript adopted, it offers the escape hatches of mutability and side effects within the normal control flow (this is arguably also dangerous, but debating this is for another post).

The downside of F#/Fable is that the same language is bolted onto two different runtimes with slightly different semantics. One example of such a mismatch is that the fixed-point arithmetic datatype Decimal of the .NET platform does not exist in Javascript of course and is thus silently converted to a double. For many cases this is fine but it is definitely something to be aware of (a longer list is here )

Alright, so this is why I think it makes sense to use the same language on the backend and frontend, some of my criteria for choosing a language and why F# is the candidate I want to use.

The proof of concept idea

Let’s finally get to some code! Be warned that some of this is probably not exactly idiomatic F# code since I haven’t used the language much until now :)

The basic idea for my little proof of concept was:

  • Write a frontend in F# that is the most primitive shopping cart you can imagine - it just has one product and the user can use buttons to increase/decrease the quantity.
  • Have a type that defines a discount, consisting of: a name, a function that checks whether the discount applies and one that modifies the price. Reuse these data types and functions between the frontend and the backend
  • On the frontend, use Fable-Arch to bring the clarity of the Elm Architecture to F#
  • Bundle up the generated Javascript into an Electron app to see how well this works
  • Since the Electron Desktop app is based on the Chromium engine, use the power of CSS and icon fonts to pep up the UI a little by just dropping in some CSS from a third party.
  • Try Suave as a http server with a functional design. Reuse the type definitions and discount calculation code. Handle Json-Serialization and CORS headers.

The implementation

The entire setup is on github at https://github.com/danyx23/FableSuaveSharedCode where you can just clone it and run both the Suave server and the electron based client.

Since this is a proof of concept, the actual code is maybe less interesting than the setup around it, but for completness sake, what follows are the three main file that are the essence of the two programs. First, the shared code that defines the Cart and Discount types, a concrete instance of the Discount type that gives a 90% discount at twelve items and the function to calculate the price for a cart given a list of potentially applicable discounts

namespace DiscountSample

module Shared =

  let ItemPrice = 20.0

  type Cart =
    { Quantity: int
    }

  type Discount =
    { Name: string
      Applies: (Cart -> bool)
      ApplyDiscount: (double -> double)
    }

  let doesTwelveItemsDiscountApply cart =
    match cart.Quantity with
    | 12 -> true
    | _ -> false


  let twelveItemsDiscount =
    { Name = "12 Items - 90% off!"
      Applies = doesTwelveItemsDiscountApply
      ApplyDiscount = (fun price -> price * 0.1)
    }

  let calculatePrice (discounts : Discount list) (cart : Cart) : double =
    let discountsThatApply = List.filter (fun discount -> discount.Applies cart) discounts
    let cartCalculationFunction =
      match discountsThatApply with
      | [] -> id
      | x :: xs -> x.ApplyDiscount // use the first discount that applies only, discard the others
    let undiscountedPrice = (double cart.Quantity) * ItemPrice
    cartCalculationFunction undiscountedPrice

This shared code is then used in the frontend:

#r "../node_modules/fable-core/Fable.Core.dll"
#load "../node_modules/fable-arch/Fable.Arch.Html.fs"
#load "../node_modules/fable-arch/Fable.Arch.App.fs"
#load "../node_modules/fable-arch/Fable.Arch.Virtualdom.fs"
#load "../../backend/src/ClientServerShared.fs"
#r "../node_modules/fable-powerpack/Fable.PowerPack.dll"


open Fable.Core
open Fable.Core.JsInterop
open Fable.Import.Browser

open Fable.Arch
open Fable.Arch.App
open Fable.Arch.Html
open DiscountSample.Shared
open Fable.PowerPack

type Status =
  | Shopping
  | TransactionPending
  | BuyingFailed of string
  | BuyingSucceeded of double

type Model =
  { Cart: Cart
    Status: Status
  }

let initModel =
  { Cart = { Quantity = 0 }
    Status = Shopping
  }

type Actions =
  | IncreaseQuantity
  | DecreaseQuantity
  | Buy
  | ServerResponseOk of double
  | ServerResponseError of string
  | SetStatus of Status


// Update
let update model action =
  let model' =
    match action with
    | IncreaseQuantity ->
      { model with Cart = { model.Cart with Quantity = model.Cart.Quantity + 1 } }
    | DecreaseQuantity ->
      { model with Cart = { model.Cart with Quantity = System.Math.Max(0, model.Cart.Quantity - 1) } }
    | SetStatus status ->
      { model with Status = status }
    | Buy -> model
    | ServerResponseOk price -> { model with Status = BuyingSucceeded price }
    | ServerResponseError error -> { model with Status = BuyingFailed error }

  let delayedCall h =
    match action with
    | Buy ->
        let promise = Fable.PowerPack.Fetch.postRecord "http://127.0.0.1:8083/test" model.Cart []
                      |> Fable.PowerPack.Promise.bind
                          (fun response ->
                            response.json<double>()
                          )
                      |> Fable.PowerPack.Promise.map
                          (fun price ->
                            h (ServerResponseOk price)
                          )
                      |> Fable.PowerPack.Promise.catch
                          (fun err ->
                            h (ServerResponseError err.Message)
                          )

        h (SetStatus TransactionPending)
     | _ -> ()

  // We return the model, and a list of Actions to execute
  model', delayedCall |> toActionList

let availableDiscounts =
  [ twelveItemsDiscount ]

let view model =
  let infoText =
    match model.Status with
    | Shopping -> "Please make your selection"
    | TransactionPending -> "Waiting for confirmation from server"
    | BuyingFailed msg -> "Sorry, your purchase failed! The reason was: " + msg
    | BuyingSucceeded price -> "The purchase worked, with a final price of " + (price.ToString())

  let counterView quantity =
    div []
      [ label [] [ text ("How many items do you want?") ]
        button [ onMouseClick (fun _ -> IncreaseQuantity) ] [ text "+" ]
        label [] [ text (quantity.ToString()) ]
        button [ onMouseClick (fun _ -> DecreaseQuantity) ] [ text "-" ]
      ]

  let counters =
    [ counterView (model.Cart.Quantity) ]

  let applicableDiscounts =
    List.filter (fun item -> item.Applies model.Cart ) availableDiscounts

  let discountView (discount : Discount) =
    li []
      [ label [] [ text discount.Name ] ]

  let discountsView (discounts : Discount list) =
    ul [ classy "discounts" ]
      (List.map discountView discounts)

  let totalPrice =
    div [] [ text <| "The total price is: " + ((calculatePrice availableDiscounts (model.Cart)).ToString()) ]

  div
    []
    [
      label
        []
        [text "This is your shopping cart: "]
      div
        []
        [ text infoText ]
      br []
      div [] counters
      discountsView applicableDiscounts
      totalPrice
      button
        [ onMouseClick (fun _ -> Buy)
          classy "buy"
        ]
        [ i [ classy "fa fa-bolt" ] []
          text " Buy!"
        ]
    ]

/// Create our application
createApp initModel view update Virtualdom.createRender
|> withStartNodeSelector "#echo"
|> start

And finally the server code using Suave:

open Suave                 // always open suave
open Suave.Web             // for config
open Suave.Writers
open Suave.Filters
open Suave.Successful
open Suave.Operators
open Newtonsoft.Json
open FSharp.Core
open DiscountSample.Shared


type Error =
  { Error : string
  }

let discounts = [ twelveItemsDiscount ]

let serializeJson obj =
  let converter = new Fable.JsonConverter()
  JsonConvert.SerializeObject(obj, converter)

let deserializeCart str =
  let converter = new Fable.JsonConverter()
  let deserialized = JsonConvert.DeserializeObject<Cart>(str, converter)
  let boxed = box deserialized
  match boxed with
    | null ->
      None
    | other ->
      Some deserialized


let deserializeCalculateSerialize json =
  let deserialized =
    json
    |> deserializeCart

  match deserialized with
    | Some cart ->
      cart
        |> calculatePrice discounts
        |> serializeJson

    | None ->
      { Error = "Could not deserialize" }
      |> serializeJson

let mapJson f =
  request(fun r ->
    System.Text.Encoding.UTF8.GetString(r.rawForm)
    |> f
    |> Successful.OK)

let setCORSHeaders =
    setHeader  "Access-Control-Allow-Origin" "*"
    >=> setHeader "Access-Control-Allow-Headers" "content-type"

let allow_cors : WebPart =
    choose [
        OPTIONS >=>
            fun context ->
                context |> (
                    setCORSHeaders
                    >=> OK "CORS approved" )
    ]

let app =
  choose [
    allow_cors
    POST >=> path "/test" >=> (mapJson deserializeCalculateSerialize) >=> Writers.setMimeType "application/json"
  ]

startWebServer defaultConfig app

Final thoughts

I think that the ability to use F# on the backend with the full power of the .NET ecosystem but now also share code that does not depend on .NET libraries (like business logic and the core problem domain code) with the frontend is a very exiting development. I’m looking forward to exploring this in more detail in the future :).

I’d also be interested in hearing about people who already use this stack or are exploring one of the other options on using the same language on the frontend and the backend.

Finally I’d like to thank the numerous great people in the F# universe who keep improving this ecosystem and create examples and blogposts about them - to name just a few that I learned from the most about F#: Tomas Petricek, Scott Wlaschin, Alfonso Garcia Caro, Steffen Forkmann and Sergey Tihon!