Streams

A stream reads or writes data a piece at a time, which is how you handle files too large to hold in memory.

Syntaxusing StreamReader reader = new StreamReader(path);

A stream is a one-piece-at-a-time view of data. You ask for the next chunk, use it, then ask for the next. Streams exist because data is often larger than memory: a twenty gigabyte log file cannot be one string, but it can be read one line at a time. A stream is also the only possible shape when the data has not all arrived yet, as with a network connection.

Two layers

FileStream moves raw bytes to and from a file. Text is not bytes until an encoding turns it into bytes, so two wrappers do that step. StreamReader turns bytes into characters, and StreamWriter turns characters into bytes. reader.ReadLine() returns the next line, or null when there is nothing left.

Disposing is the whole discipline

A stream holds a file handle from the operating system. Until that handle is released the file can stay locked, and a StreamWriter keeps the most recent characters in a memory buffer rather than on disk. Dispose() flushes the buffer and releases the handle. Do not call it by hand: write using, and the compiler calls it for you, including when an exception is thrown.

// longhand: the braces mark the lifetime
using (StreamReader reader = new StreamReader(path))
{
    Console.WriteLine(reader.ReadLine());
}

// shorthand (C# 8): disposed at the end of the enclosing block
using StreamReader shorter = new StreamReader(path);
Console.WriteLine(shorter.ReadLine());

Those two forms mean the same thing. File.ReadAllText does all of this internally. Open a stream yourself when the data is too big to hold at once, or when some other code hands you a Stream instead of a path.

Example

using System;
using System.IO;

string path = Path.Combine("data", "log.txt");
Directory.CreateDirectory("data");

// StreamWriter turns characters into bytes and sends them through the stream.
using (StreamWriter writer = new StreamWriter(path))
{
    writer.WriteLine("start");
    writer.WriteLine("work");
    writer.WriteLine("done");
}   // Dispose runs here: the buffer is flushed, the handle released

// A using declaration: same meaning, disposed at the end of this block.
using StreamReader reader = new StreamReader(path);

int count = 0;
string? line = reader.ReadLine();
while (line is not null)   // ReadLine returns null at the end of the file
{
    count++;
    Console.WriteLine($"{count}: {line}");
    line = reader.ReadLine();
}

Console.WriteLine($"lines: {count}");

// Output:
// 1: start
// 2: work
// 3: done
// lines: 3

When to use it

Scan a multi-gigabyte log file for matching lines without loading it.
Copy an uploaded file to disk in fixed-size chunks.
Write a long report line by line instead of building one huge string.
Accept a Stream parameter so the same code reads a file or an in-memory buffer.

More examples

Copying bytes with FileStream

This is the raw byte level, with no encoding involved. The important detail is that Read is allowed to return fewer bytes than the buffer holds, so you must write exactly read bytes and not buffer.Length. Getting that wrong pads the copy with stale bytes. You write loops like this when handling uploads, images or any data that is not text.

using System;
using System.IO;

File.WriteAllText("source.bin", "0123456789");

using (FileStream input = File.OpenRead("source.bin"))
using (FileStream output = File.Create("copy.bin"))
{
    byte[] buffer = new byte[4];   // deliberately tiny, to show the loop
    int total = 0;
    int read;

    // Read returns how many bytes it actually got, and 0 at the end.
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write(buffer, 0, read);   // write only the bytes just read
        total += read;
    }

    Console.WriteLine($"copied: {total}");
}

Console.WriteLine($"same: {File.ReadAllText("copy.bin") == File.ReadAllText("source.bin")}");

// Output:
// copied: 10
// same: True

A stream that is not a file

FileStream and MemoryStream are both streams, so StreamReader works over either. That is the practical reason to write methods that accept a Stream rather than a file path: the same method can then read a file in production and a MemoryStream in a test, with no temporary files to clean up.

using System;
using System.IO;
using System.Text;

byte[] bytes = Encoding.UTF8.GetBytes("alpha\nbeta\ngamma");

// A MemoryStream is a stream over a byte array. No disk is touched.
using MemoryStream stream = new MemoryStream(bytes);
using StreamReader reader = new StreamReader(stream);

string? line = reader.ReadLine();
while (line is not null)
{
    Console.WriteLine(line.ToUpperInvariant());
    line = reader.ReadLine();
}

// Output:
// ALPHA
// BETA
// GAMMA

Appending with StreamWriter

The second constructor argument decides between replacing the file and adding to it, and passing it by name makes the call readable at a glance. Opening a writer once and writing many lines is far cheaper than calling File.AppendAllText per line, because each of those calls opens and closes the file again.

using System;
using System.IO;

string path = "audit.log";
File.WriteAllText(path, "");   // start from an empty file

// append: true keeps what is already there and writes after it.
using (StreamWriter writer = new StreamWriter(path, append: true))
{
    writer.WriteLine("opened");
    writer.WriteLine("checked");
}

using (StreamWriter writer = new StreamWriter(path, append: true))
{
    writer.WriteLine("closed");
}

Console.Write(File.ReadAllText(path));

// Output:
// opened
// checked
// closed

Discussion

Be the first to comment on this lesson.

Two layers

Disposing is the whole discipline

Example

When to use it

More examples

Copying bytes with FileStream

A stream that is not a file

Appending with StreamWriter

Related

Discussion