Skip to main content

Library Design

· 24 min read
Kevin Frey
Kevin Frey
Maintainer of Siren

Siren consists of one main code base written in F#. This code base is then transpiled to JavaScript, TypeScript, Python and made accessible from C#.

Fable.Multiverse

To make it as easy as possible to create a Fable library that publishes in multiple languages, I created a template called Fable.Multiverse 🎉

Idea 💡​

During a hackathon for our research data management consortium, we were discussing ideas for visualizing graph like structures in a way that allows easy gitlab integration and can be understood and used by any level of user. The idea we pursued was to add .md files with mermaid graphs for an easy overview. So why not write a domain specific language for mermaid graphs to make creation of such graphs easier and more error proof. Good idea, but what programming language should we use? In our consortium we have several groups, some using JavaScript, some Python, some (us) .NET or more specifically F#. Because i already have quite some experience using Fable, i did the mental checklist to see if it would be a good fit for this project.

  • Does not need any dependencies. Mermaid graphs, are mostly YAML, so no complex syntax.
  • Does not require IO interaction. We can simply focus on writing our mermaid graph as string and allow the user to do whatever they want with it.
  • Only Fable compatible languages needed. We are already very happy offering such a tool in Python, JavaScript and F#.

.. and thats it 🎉 So we can start developing a libary with one codebase for 4 [5] languages.

What is Fable?​

Fable is a F# to X transpiler. It started out targeting only JavaScript, using a naming reference to the popular Babel JavaScript transpiler. Now Fable aims to support multiple languages, all in different states. At the time of writing, the offical Fable docs state the following:

LanguageStatus
JavaScriptStable
TypeScriptStable
DartBeta
PythonBeta
RustAlpha
PHPExperimental

Benefits​

  • Type Safety. F# is a statically typed language, which means that the compiler can catch many errors before they even happen.
  • Lightweight Syntax. F# has a very lightweight syntax, which makes it easy to read and write. It does not require a lot of boilerplate code and you can get right into the meat of your program.
  • Testing(?). Main codebase is written in F# as well as most tests. This allows us to also transpile the tests to other languages and run them there. This is a big advantage, as we can be sure that the tests are the same in all languages.
But why the question mark behind testing?

Because we can recycle the tests, to ensure correct functionality, but we still must test if the library can be used from all supported languages without hurdles.

API Design​

To make the code look and feel as native as possible in all languages, there are some things we need to consider. But first let us have a look at fable transpiled code.

info

The following code will use the Fable REPL to transpile code for easy showcasing!

let helloWorld = printfn "Hello World"

We can already notice some things:

  • Fable tries to transpile into native syntax, so for example snake_case in Python and camelCase in JavaScript
  • Fable has some wrappers for functions which might have native equivalents. In F# printfn is used to print to the console, in JavaScript console.log and in Python print. But Fable uses their own printf function to ensure 100% correct f# transpilation.

Next we will have a look at a class with some member functions.

type MyClass =
    static member add (x: int) (y: int) = x + y

Oh no, this does not look good. Fable does a thing called name mangling. Have a look at the offical docs for a deeper view on this topic. For now its enought to know, this is done to allow overloading functions in F#.

Edit- Tree-Shaking

ncave pointed also out to me that tree-shaking in JavaScript was facilitated in this way. As it was easier for bundlers to detect import of single functions, rather then unused class members.

Using [<AttachMembers>]​

But we can tell Fable that we know what we are doing and ignore name mangling.

open Fable.Core

[<AttachMembers>]
type MyClass =
    static member add (x: int) (y: int) = x + y

That looks better and allows us to do the following in all 3 languages: MyClass.add. This is the basic design i chose to use for most user facing api. All F#/Fable code not easily usable from other languages is hidden behind a facade like this.

C# Compatibility​

Strangely enough allowing C# users the same ease of use as Python and JavaScript users is the hardest. This is because C# has some issues with F# optional parameters and F# tuples.

In F# we can define a function like this:

[<AttachMembers>]
type flowchart =
    static member raw (txt: string) = FlowchartElement txt
    static member id (txt: string) = FlowchartElement txt
    static member node (id: string, ?name: string) : FlowchartElement = ...

flowchart.node("My id")
flowchart.node("My id", "My name")

Using the F# function in C# will result in an error, when you try to do flowchart.node("My id"), as ?name is a Microsoft.FSharp.Core.FSharpOption<string> without any default information.

By creating a C# access layer we can avoid this issue for C# users:

info

The C# extensions for optional parameters and tuples are taken from Plotly.NET with the help from my dear colleague Kevin Schneider.

public static class flowchart
{
    public static FlowchartElement raw(string txt) => Siren.flowchart.raw(txt);

    public static FlowchartElement id(string txt) => Siren.flowchart.raw(txt);

    public static FlowchartElement node(string id, Optional<string> name = default) =>
        Siren.flowchart.node(id, name.ToOption());
    //...
}

A similar issue arises with F# tuples:

[<AttachMembers>]
type flowchart =
    static member stylesNode (nodeId: string, styles:#seq<string*string>) = Flowchart.formatNodeStyles [nodeId] (List.ofSeq styles) |> FlowchartElement

Code generator helper​

These incompatibilities are not only annoying but providing a consistently native C# experience, requires a wrapping for all apis. To make this easier, i created a code generator that takes a F# file and generates the C# wrapper for it. .. Or at least 95% of it. The rest is done by hand.

Code Generator
open System.Reflection

[<LiteralAttribute>]
let FSharpOptionDefault = "Optional<string>"

let transformParameterTypeName (paramTypeName: string)=
    match paramTypeName with
    | "String" -> "string"
    | "Int32" -> "int"
    | "Double" -> "double"
    | "FSharpOption`1" -> FSharpOptionDefault // this is not always true but a good approximation
    | "Tuple`2" -> "(string,string)" // this is not always true but a good approximation
    | "Boolean" -> "bool"
    | _ -> paramTypeName

type ParameterInfo = {
    Type: string
    Name: string
} with
    member this.FSharpParam =
        match this with
        | {Type = FSharpOptionDefault} -> this.Name + ".ToOption()"
        | _ -> this.Name
    member this.CSharpParam =
       match this with
       | {Type = FSharpOptionDefault} -> sprintf "%s %s = default" this.Type this.Name
       | _ -> sprintf "%s %s" this.Type this.Name

    static member create(typeName: string, name: string) =
        {Type = transformParameterTypeName typeName; Name = name}

let generateCSharpCode<'A>() =

    let t = typeof<'A>
    let members = t.GetMethods(BindingFlags.Static ||| BindingFlags.Public)

    let mutable csharpCode = sprintf "public static class %s\n{\n" t.Name
    for m in members do
        let methodName = 
            let name0 = m.Name
            if name0.StartsWith("get_") then
                name0.Substring(4)
            else
                name0
        let returnType = m.ReturnType.Name
        let params0 = 
            m.GetParameters() 
            |> Array.map (fun p -> ParameterInfo.create(p.ParameterType.Name, p.Name))
        let csharpParameters =
            if params0.Length = 0 then
                ""
            else    
                params0
                |> Array.map _.CSharpParam
                |> String.concat(", ")
                |> fun s -> "(" + s + ")"
        
        let fsharpParameters = 
            if params0.Length = 0 then
                ""
            else    
                params0 
                |> Array.map _.FSharpParam
                |> String.concat(", ")
                |> fun s -> "(" + s + ")"
        
        let methodSignature = $"public static {transformParameterTypeName returnType} {methodName}{csharpParameters}"
        let methodBody = 
            if methodName.StartsWith("get_") then
                let withoutGet = methodName.Substring(4)
                $"return Siren.{t.Name}.{withoutGet};"
            else
                $" => Siren.{t.Name}.{methodName}{fsharpParameters};"
        csharpCode <- csharpCode + $"    {methodSignature}\n        {methodBody}\n"

    csharpCode <- csharpCode + "}\n"
    csharpCode

let test() = 
    generateCSharpCode<Siren.classDiagram>() // Here you can pass any type you want to generate C# code for
    |> printfn "%A"

This is something i did not want to spend a lot of time on, so when i quickly wrote this and noticed it was able to create most of the C# code correctly. I think improving this code to create everything perfectly would be awesome, but would have taken me longer thant fixing the few mistakes it makes by hand.

Maintainability​

This is a big issue. Whenever i update my f# api i must also update the c# wrapper. Changes are mostly catched by the compiler but missings functions are not.

That is why i added unit tests to check the count and name of a c# and f# class and compare it for equality:

public static class Utils
{
    public static int GetMemberCount(Type type)
    {
        var members = type.GetMembers();
        return members.Length;
    }

    public static List<string> GetMemberNameDifferences(Type type1, Type type2)
    {
        List<string> differences = new List<string>();
        //transform string to lower
        var type1Members = type1.GetMembers().Select(m => m.Name.ToLower());
        var type2Members = type2.GetMembers().Select(m => m.Name.ToLower());
        differences.AddRange(type1Members.Except(type2Members));
        differences.AddRange(type2Members.Except(type1Members));

        return differences;
    }

    public static void CompareClasses(Type csharpType, Type fsharpType)
    {
        int csharpMemberCount = GetMemberCount(csharpType);
        int fsharpMemberCount = GetMemberCount(fsharpType);
        List<string> differences = GetMemberNameDifferences(fsharpType, csharpType);

        Assert.Empty(differences);
        Assert.Equal(fsharpMemberCount, csharpMemberCount);
    }
}

This at least helps me to catch missing functions in the c# wrapper, even if i still have to write them by hand.

Python/JavaScript import and Index files​

F# and C# use namespaces to organize code. I can have multiple files with the same namespace and access all functions simply by writing open Siren/using Siren.Sea;.

In Python and JavaScript this is not possible. Here imports happen on a file basis. So i needed to have a single file that imports all other files and exports them.

Luckily this file can be created automatically!

Index File
module Index.Util

open System
open System.IO
open System.Text.RegularExpressions

type FileInformation = {
    FilePath : string
    Lines : string []
} with
    static member create(filePath: string, lines: string []) = {
        FilePath = filePath
        Lines = lines
    }

let getAllFiles(path: string, extension: string) = 
    let options = EnumerationOptions()
    options.RecurseSubdirectories <- true
    IO.Directory.EnumerateFiles(path,extension,options)
    |> Seq.filter (fun s -> s.Contains("fable_modules") |> not)
    |> Array.ofSeq

let findClasses (rootPath: string) (cois: string []) (regexPattern: string -> string) (filePaths: seq<string>) = 
    let files = [|
        for fp in filePaths do
            yield FileInformation.create(fp, System.IO.File.ReadAllLines (fp))
    |]
    let importStatements = ResizeArray()
    let findClass (className: string) = 
        /// maybe set this as default if you do not want any whitelist
        let classNameDefault = @"[a-zA-Z_0-9]"
        let pattern = Regex.Escape(className) |> regexPattern
        let regex = Regex(pattern)
        let mutable found = false
        let mutable result = None
        let enum = files.GetEnumerator()
        while not found && enum.MoveNext() do
            let fileInfo = enum.Current :?> FileInformation
            for line in fileInfo.Lines do
                let m = regex.Match(line)
                match m.Success with
                | true -> 
                    found <- true
                    result <- Some <| (className, IO.Path.GetRelativePath(rootPath,fileInfo.FilePath))
                | false ->
                    ()
        match result with
        | None ->
            failwithf "Unable to find %s" className
        | Some r ->
            importStatements.Add r
    for coi in cois do findClass coi
    importStatements
    |> Array.ofSeq

let writeIndexFile (path: string) (fileName: string) (content: string) =
    let filePath = Path.Combine(path, fileName)
    File.WriteAllText(filePath, content)

The resulting files look like this:

export { SirenElement } from "./SirenTypes.js";
export { themeVariable, quadrantTheme, gitTheme, timelineTheme, xyChartTheme, pieTheme } from "./ThemeVariables.js";
export { graphConfig, flowchartConfig, sequenceConfig, ganttConfig, journeyConfig, timelineConfig, classConfig, stateConfig, erConfig, quadrantChartConfig, pieConfig, sankeyConfig, xyChartConfig, mindmapConfig, gitGraphConfig, requirementConfig } from "./Config.js";
export { formatting, direction, flowchart, notePosition, sequence, classMemberVisibility, classMemberClassifier, classDirection, classCardinality, classRltsType, classDiagram, stateDiagram, erKey, erCardinality, erDiagram, journey, ganttTime, ganttTags, ganttUnit, gantt, pieChart, quadrant, rqRisk, rqMethod, requirement, gitType, git, mindmap, timeline, sankey, xyChart, block, theme, siren } from "./Siren.js";
info

Python packages will default import whatever is in the __init__.py file. So we must simply name the index file for python as such (at least for publishing).

Publish​

As soon as we get to the publishing step everything is back to standard for the respective languages.

.NET​

The easiest. We already have the required project files, no need to transpile. So we can simply use dotnet pack to create .nupkg files. Then dotnet nuget push to push to nuget.org.

warning

These are not the exact CLI args used. Details can be found in the build project in the GitHub repository.

Pypi​

Also quite easy. Transpile f# code to python code, copy pyproject.toml and README.md into the dist folder and create index.py file. Run python -m poetry build to create a publishing files. Then publish files with python -m poetry publish

warning

These are not the exact CLI args used. Details can be found in the build project in the GitHub repository.

JavaScript (+Types)​

For this i followed the Example of Better Typed than Sorry by Alfonso García-Caro.

Transpile F# to TypeScript, then use tsc to transpile TypeScript to JavaScript and type information files (*.d.ts). Add the index.js file to the dist folder Then publish to npm with npm publish.

warning

These are not the exact CLI args used. Details can be found in the build project in the GitHub repository.

Deep Dive​

From here on are some additional issues i encountered during development.

Overloads​

The concept of allowing different inputs for the same function exists in f#, as well as python and javascript. By using the [<AttachMembers>] attribute we are not longer allowed to use standard f# overloads. Because JavaScript does not have the same kind of type interference it is unable to recognice which function should be invoked:

open Fable.Core

[<AttachMembers>]
type MyClass =
    static member add (x: int, y: int) = x * y // this should be invoked
    static member add (x: string, y: string) = x + y

let result = MyClass.add(10, 20)

printfn "%A" result

Erased Unions​

It is possible to imitate js overload behaviour by using erased unions. Here we use a Fable provides discriminate union called U2(U3, U4...). After transpilation it is replaced by a js type check.

open Fable.Core
open Fable.Core.JsInterop

[<AttachMembers>]
type MyClass =
    static member test (arg: U2<string, int>) = 
        match arg with
        | U2.Case1 s -> printfn "This is a string: %s" s
        | U2.Case2 i -> printfn "This is a integer: %i" i

let result = MyClass.test(U2.Case2 10) // or MyClass.test(!^10)

printfn "%A" result

As you can see, this looks really nice in JavaScript, but is cumbersome to use in F#.

It would be possible to do do both and use f# overloads and shadow them with the erased union. But this would add more additional maintainance work.

Compiler Statements

We can use #if FABLE_COMPILER ... #else ... #endif syntax to include code only in certain compiler states.

In the following example the erased union is only used (and accessible!) when the code is transpiled by Fable to JavaScript.

[<AttachMembers>]
type MyClass =
#if FABLE_COMPILER_JAVASCRIPT
    static member test (arg: U2<string, int>) = 
        match arg with
        | U2.Case1 s -> printfn "This is a string: %s" s
        | U2.Case2 i -> printfn "This is a integer: %i" i
#else
    static member test(arg: int) =
        printfn "This is a integer: %i" arg
    static member test(arg: string) =
        printfn "This is a string: %s" arg
#endif

let result = MyClass.test(10)

This would allow us to use the erased union only in JavaScript and use the F# overloads in F#. But i have not investigated how this would work for python 😅.

Due to the additional workload i decided to avoid using overloads in the api. Instead i tried finding the core functions and functions, which allow additional inputs with a different name:

type sequence =
    static member note(id: string, text: string, ?notePosition: NotePosition) = //..
    static member noteSpanning(id1: string, id2, text: string, ?notePosition: NotePosition) = //..

JavaScript optional parameters​

Using functions with multiple optional parameters is easily done in F#, C# and Python, but can get quite annoying in JavaScript:

// tripple null ...
requirement.requirement("my id", null, null, null, rqMethod.test)

The JavaScript native approach would be using an object with only the values you want to set. There is even a way to tell Fable to transpile parameters as object using the [ParamObject] attribute.

open Fable.Core
open Fable.Core.JsInterop

[<AttachMembers>]
type MyClass =
    [<ParamObject(1)>] // Start creating obj from params at index 1
    static member test (name: int, ?id: string, ?text: string, ?rqRisk: string, ?rqMethod: string) =
        0


MyClass.test(10, rqRisk = "Hello")
MyClass.test(10)

As you can see adding no optional parameters requires an empty object, as Fable checks if the value at object key xyz is null and not if the object is null. Without the empty object the function would throw an error, whenever any of the optional parameters is referenced in the function. (Bad example as i just return 0).

Member Names​

Different languages have different expectations for member names. Aside from styling best practises, there are some things that are not possible in all languages.

info

F# typically uses PascalCase for class names and camelCase for member names. For easier usage i ignore this rule and used camelCase for everything.

F# reserved keywords​

The first issue i encountered where reserved keywords in F#.

For example classDiagram.``class``. "class" is a reserved keyword which is not allowed in F#. The standard solution is wrapping the name in backticks. But at least for me on VisualStudio Community this resulted in issues with my auto complete. This resulted in me handling this issue inconsistently. The issues i encountered were mostly in (optional) parameters, which is why i changed their names to PascalCase:

For members i mostly stayed true to the backtick syntax.

[<AttachMembers>]
type classMemberClassifier =
    // abstract is a reserved keyword
    static member Abstract = ClassMemberClassifier.Abstract
    // static is a reserved keyword
    static member Static = ClassMemberClassifier.Static
    static member custom str = ClassMemberClassifier.Custom str

[<AttachMembers>]
type classDiagram =
    static member ``class`` (id: string, members: #seq<ClassDiagramElement>) =

The C# wrapper used the C# best practise syntax @class, which worked fine for me.

C# - Member name = enclosing type​

I encountered this issue first for block.block. In C# member names are not allowed to be the same as the enclosing type.

As i am not a very experienced C# developer, i am still rather undecided on how to handle this issue. So far I have been using Block.block, as I am thinking about using PascalCase for all classes in C#. And if only to mute the warnings in VS Community.

If you have a strong opinion about this topic, please let me know! I am interested in hearing your thoughts.

Transpiled names​

And back to classDiagram.``class`` . While JavaScript does not seem to care about this topic to much, Python does.

JavaScript gives us the best result, the F# backtick syntax is transpiled to a simple camelCase name classDiagram.class.

Python on the other hand has class also as reserved keyword. Fable transpiles it to classDiagram.class_. Which raises the question if i should simply apply this syntax to all cases with naming problems.

Docs/Native Test Maintainance​

The core library + C# wrapper were done rather quickly. I can also recycle my F# unit tests to check if transpilation works as expected, using Fable.Pyxpecto.

But testing correct native accessibillity and writing docs (showcasing the test cases) was the most time consuming part and something i am not happy with.

Here are some ideas on how to improve this:

  • Theoretically, i could use the transpiled tests for docs. But i still have to remove the Fable specific helper functions and replace them with native ones.
  • Kevin S. had an idea, repurposing jupyter notebooks for docs and testing. To at least unify the testing and docs.

If you have any ideas on how to improve this, please let me know!