Streaming File Data | Effect Http Client

For large files, loading the entire content into memory isn’t practical. HttpClientRequest.bodyStream() takes a Stream<Uint8Array> and sends it as the request body, reading and transmitting chunks incrementally.

1
import {
2
  FetchHttpClient,
3
  HttpClient,
4
  HttpClientRequest,
5
} from "effect/unstable/http";
6
import { Effect, Stream } from "effect";
7

8
function uploadLargeData() {
9
  return Effect.gen(function* () {
10
    const client = yield* HttpClient.HttpClient;
11

12
    // Create a stream that produces chunks of data
13
    // In practice, this might come from reading a large file
14
    const encoder = new TextEncoder();
15
    const dataStream = Stream.fromIterable([
16
      encoder.encode("chunk 1: Hello "),
17
      encoder.encode("chunk 2: from "),
18
      encoder.encode("chunk 3: a stream!"),
19
    ]);
20

21
    const request = HttpClientRequest.post("https://httpbin.org/post").pipe(
22
      HttpClientRequest.bodyStream(dataStream, {
23
        contentType: "text/plain",
24
        contentLength: 47,
25
      }),
26
    );
27

28
    const response = yield* client.execute(request);
29
    const data = yield* response.json;
30

31
    return data;
32
  }).pipe(Effect.provide(FetchHttpClient.layer));
33
}
34

35
// Test it
36
Effect.runPromise(uploadLargeData()).then((data) => {
37
  console.log("Body received:", data.data);
38
});

Output:

1
Body received: chunk 1: Hello chunk 2: from chunk 3: a stream!

bodyStream takes a Stream<Uint8Array> and an optional options object with contentType and contentLength. The content length is optional but recommended. Some servers reject requests without a Content-Length header, and others use it to show upload progress.

The stream is consumed lazily. Each chunk is sent over the network as it’s produced, so only one chunk needs to be in memory at a time. This makes it suitable for uploading files of any size.

Stay in the loop