Making Http Requests in F#

This is the third post in Simple things in F#.

In the F# ecosystem, there are multiple F# specific libraries. Additionally, there is a large number of libraries that use C# as the code base.

What do they have in common?

They are .NET libraries
They can be used in any of the .NET languages (C#, F#, VB)

Sometimes this means that the library is not idiomatic for the language you're using and there may surface interoperability issues between languages, but don't let that stop you from trying libraries here and there, as well as consuming a C# library in F#, or in the reverse order.

There are three ways to do Http Requests with different libraries. We'll first explore the BCL's (Base Class Library) System.Net.Http, then proceed to use Flurl, and finally check FsHttp,

System.Net.Http

System.Net.Http
This is part of the BCL, so it's very likely that there’s plenty of code out there using HttpClient. This HttpClient provides some constructs that allow users to create Http Requests in any shape or form. Though, since this is part of the BCL it might be clunky in some aspects, but it is very useful to build either libraries or services on top of it.

The exercise will be: “Download a web page”. Please notice that the HttpClient class implements the IDisposable interface, meaning that this class can release any resources it's using, but having too many instances of an HttpClient can lead to port exhaustion, in production code please try to use a single instance to make your requests.


open System.Net.Http
open System.IO

task {
    /// note the ***use*** instead of ***let***
    use client = new HttpClient()
    let! response = 
        client.GetStringAsync("https://dev.to/tunaxor/doing-some-io-in-f-4agg")
    do! File.WriteAllTextAsync("./response.html", response)
    // after the client goes out of scope
    // it will get disposed automatically thanks to the ***use*** keyword
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

Retrieving the contents as strings and storing in memory might not be efficient if the string size is high or the content is a binary file that can't be represented with strings such as a PDF or an Excel file. The one way to handle this is by using the streams themselves


open System.Net.Http
open System.IO

task {
    /// open the file and note the ***use*** keyword in the file and the client
    use file = File.OpenWrite("./dummy.pdf")
    use client = new HttpClient()
    let! response = client.GetStreamAsync("https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf")
    /// copy the response contents to the file asynchronously
    do! response.CopyToAsync(file)
    // both file and client will be disposed automatically
    // after they get out of scope
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

This should download a PDF file.

Posing JSON is very similar, the following example uses JsonPlaceholder


open System.Net.Http
open System.Net.Http.Json
// model of a "Post" from the jsonplaceholder website
type Post =
    { userId: int
      id: int
      title: string
      body: string }

task {
    use client = new HttpClient()
    // use an anonymous record to create a partial post
    let partialPost =
        {| userId = 1
           title = "Sample"
           body = "Content" |}

    let! response =
        let url = "https://jsonplaceholder.typicode.com/posts"

        client.PostAsJsonAsync(url, partialPost)

    let! createdPost = response.Content.ReadFromJsonAsync()
    printfn $"Id: {createdPost.id} - Title: {createdPost.title}"
    // Id: 101 - Title: Sample
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy the content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

We're using a disposable HttpClient instance as well as using an anonymous record of a partial Post as the payload of the request. Thanks to the System.Net.Http.Json namespace, we can also read the content as JSON and we don't need to parse it or handle the streams ourselves.

In case we just need to do a simple GET that has a JSON response, we could run the following


open System.Net.Http
open System.Net.Http.Json

type Post =
    { userId: int
      id: int
      title: string
      body: string }

task {
    use client = new HttpClient()
    let url = "https://jsonplaceholder.typicode.com/posts"

    let! posts = client.GetFromJsonAsync<post[]>(url)

    printfn $"%A{posts}" // prints the 100 post array to the console
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy the content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

This shows basic usage of the HttpClient, but further operations may be a little cumbersome and that's why I want to write about other libraries as well.

Let's move on to a relatively popular (2.5k GitHub stars at the time of writing) library for HTTP requests

Flurl

This is a nice library with an API designed to be consumed from C#. Besides the fluent style it uses, this library has more ergonomic methods that may drive productivity in the long run when compared to crafting http Requests with BCL's HttpClient.

Let's try to replicate the first example of downloading an html page


open System.IO
open Flurl.Http

task {
    let! content =
        let url = "https://dev.to/tunaxor/doing-some-io-in-f-4agg"

        url.GetStringAsync()

    do! File.WriteAllTextAsync("./response.html", content)
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

After executing it, the html page should have downloaded. We can open it and it should work in the same way as the HttpClient downloaded page.
Notice that we didn't need to handle the lifecycle of any HttpClient. That is done by the library itself.

It is worth mentioning that the same approach as with large strings is also needed regardless of the library you use: “If the content is heavy, you should use streams rather than buffering the strings in memory”

let's try the stream example:


#r "nuget: Flurl.Http" 
open System.IO
open Flurl.Http

task {
    use file = File.OpenWrite("./response.html")

    let! content =
        "https://dev.to/tunaxor/doing-some-io-in-f-4agg"
            .GetStreamAsync()

    do! content.CopyToAsync(file)
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

One of the benefits of using libraries is that they are meant to ease some of the pains/productivity issues that may be encountered when using base class library elements.

To further illustrate the point, you could skip all of the System.IO.File operations and let the library handle it.


#r "nuget: Flurl.Http"

open System.IO
open Flurl.Http

task {
    let path = Path.GetFullPath("./Downloads")

    let! result =
        "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
            .DownloadFileAsync(path, "dummy.pdf")

    printfn "%s" result
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

Let's continue to the JSON example.


#r "nuget: Flurl.Http"

open Flurl.Http

type Post =
    { userId: int
      id: int
      title: string
      body: string }

task {
    let! postResult =
        "https://jsonplaceholder.typicode.com/posts"
            .WithHeaders(
                {| Accept = "application/json"
                   X_MY_HEADER = "my-header-value" |},
                true // replace _ with -
            )
            .PostJsonAsync(
                {| userId = 1
                   title = "Sample"
                   body = "Content" |}
            )
            .ReceiveJson()

    printfn "%A" postResult
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

Even with a headers object (including custom headers), the code to create an HTTP request is fairly simple and short. Flurl also has methods for PATCH, PUT, OPTIONS among others and can cover most needs.

Uploading files is slightly different given that the send-request generally has a multipart form with the filename as the last item in the request. Here’s an example using Flurl:


#r "nuget: Flurl.Http"

open System.IO
open Flurl.Http

task {
    let path = Path.GetFullPath("./dummy.pdf")

    try
        let! response =
            // I don't have a jsonplaceholder endpoint for files
            // so I'll put a fake one here
            "https://sampleurl.nox"
                .PostMultipartAsync(fun content ->
                    content
                        .AddString("firstName", "Jane")
                        .AddString("lastName", "Smith")
                        .AddString("email", "jane@smith.lol")
                        .AddFile("pdfresume", path, "application/pdf")
                    // remember in F# functions always return
                    // so let's ignore the content since we've already set what we need
                    |> ignore)

        printfn "Status code: %i" response.StatusCode
    with ex -> printfn "%s" ex.Message
    // Call failed. No such host is known. (sampleurl.nox:443): POST https://sampleurl.nox
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

When using a C# library in these F# samples. It's worth mentioning that not every library behaves as well when you use them in another .NET language, their design is often influenced by the language they are written in and they favor certain patterns that might not be a good match for your current language.

In case there are no other alternatives, writing a slim wrapper helps ease pain points and is the path forward. Some examples of these wrappers are

Even Flurl itself can have a slim wrapper.

FsHttp

This library offers a more streamlined experience for F# developers.

For these examples I'll be using the Domain Specific Language (DSL) flavor of this library which exposes Computation Expressions (CE). For our purposes let's think of these as some sort of a Functional Builder Pattern.

Downloading the html page once again


#r "nuget: FsHttp"

open System.IO
open FsHttp
open FsHttp.DslCE
open FsHttp.Response

task {
    let! response =
        httpAsync {
            GET "https://dev.to/tunaxor/doing-some-io-in-f-4agg" 
        }
    let! content = response |> toTextAsync
    do! File.WriteAllTextAsync("./response.html", content)
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

The httpAsync CE exposes some methods that are known as custom operations. In this case GET is a method exposed by the `EagerAsyncHttpBuilder` that takes a string as the parameter. Once this CE runs, it returns an `Async<Response>` value but let! (which is the equivalent to `let result = await httpAsync { GET "" })` is binding the result of the `Async<Response>` to the response variable which at that point it's just a Response object.

Let's continue with the stream version knowing that it will not keep the string in memory as the other example did.


#r "nuget: FsHttp"

open System.IO
open FsHttp
open FsHttp.DslCE
open FsHttp.Response

task {
    use file = File.OpenWrite("./response.html")
    let! response = 
        httpAsync { 
            GET "https://dev.to/tunaxor/doing-some-io-in-f-4agg" 
        }
    let! content = response |> toStreamAsync
    do! content.CopyToAsync(file)
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

Let's move on to the JSON requests. In F# JSON Serialization/Deserialization is an interesting topic. Option, Result, Discriminated Unions and other F# specific constructs sometimes don't translate well to or from plain JSON and that's why often F# libraries leave that aspect to other libraries such as Thoth.Json or FSharp.SystemTextJson. For now we'll fall back to the BCL and use the System.Text.Json namespace since we are only using primitive types in our model


#r "nuget: FsHttp"

open FsHttp
open FsHttp.DslCE
open FsHttp.Response
open System.Text.Json

type Post =
    { userId: int
      id: int
      title: string
      body: string }

let content =
    JsonSerializer.Serialize(
        {| userId = 1
           title = "Sample"
           body = "Content" |}
    )

task {
    let! response =
        httpAsync {
            POST "https://jsonplaceholder.typicode.com/posts"
            body
            json content
        }

    let! responseStream = response |> toStreamAsync
    let! responseValue = JsonSerializer.DeserializeAsync(responseStream)
    printfn "%A" responseValue
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

FsHttp also has a JSON extensions module that can be used to dynamically access the parsed JSON data but I'll skip that for now since it returns a JSON object that must be accessed via the dynamic operator it provides (`json?page.AsInteger()`).

In the case of the multipart request, here's the code:


#r "nuget: FsHttp"

open FsHttp
open FsHttp.DslCE
open System.IO

task {
    let path = Path.GetFullPath("./dummy.pdf")

    try
        let! response =
            httpAsync {
                // feel free to use your endpoint in case you have one
                POST "https://sampleurl.nox"
                multipart
                valuePart "firstName" "Jane"
                valuePart "lastName" "Smith"
                valuePart "email" "jane@smith.lol"
                filePartWithName "pdfresume" path
            }

        printfn "StatusCode %A" response.statusCode
    with ex -> printfn "%s" ex.Message
    // --- FsHttp: running in FSI - Try registering printer...
    // --- FsHttp: Printer successfully registered.
    // One or more errors occurred. (No such host is known. (sampleurl.nox:443))
}
|> Async.AwaitTask
// we run synchronously
// to allow the fsi to finish the pending tasks
|> Async.RunSynchronously

To Run this, copy this content into a file named script.fsx (or whatever name you prefer) and type:

dotnet fsi script.fsx

There you have it, a few ways to create http requests with three different libraries.

________________________________________

One of the cool things of developing F# programs is that you have access to the whole .NET ecosystem so you don't have to re-invent the wheel. Often, it’s possible to re-use what may be battle-tested code that works perfectly irrespective of C#, F#, or VB.

About Encora

Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.

Making Http Requests in F#

System.Net.Http

Flurl

FsHttp

About Encora

About Encora

Accelerate Your Path
to Market Leadership

Speak With an Expert

Making Http Requests in F#

System.Net.Http

Flurl

FsHttp

About Encora

About Encora

Accelerate Your Path to Market Leadership

Speak With an Expert

Accelerate Your Path
to Market Leadership