Other Formats and Encoding

CSV by hand versus with a library, why UTF-8 is the only encoding you should choose, and Base64 for bytes in a text field.

Syntaxstring encoded = Convert.ToBase64String(bytes);

JSON is not the only text you will handle. Two other jobs come up constantly: reading and writing CSV, and getting raw bytes through a field that only accepts text.

CSV

A CSV file holds one record per line, with fields separated by commas. The format looks trivial and is not. A field may be wrapped in double quotes; a quoted field may contain a comma, a doubled quote meaning one literal quote character, or even a line break. So line.Split(',') is correct only for data you produced and have checked yourself. For files arriving from elsewhere, use a library: dotnet add package CsvHelper gives you CsvReader and CsvWriter, which handle quoting and header rows. There is no CSV support in the base class library.

Encodings

A file holds bytes. Text is characters. An encoding is the table that maps between the two. UTF-8 can represent every character, spends one byte on each plain ASCII character, and is the default throughout .NET: File.ReadAllText and File.WriteAllText use it when you pass no Encoding. Choose UTF-8 and stop thinking about it. Older single-byte encodings such as Windows-1252 cannot represent most characters, and Encoding replaces what it cannot encode with a question mark, which destroys the data permanently.

Base64

Some fields accept text only, such as a JSON string or an HTTP header. To carry bytes there, call Convert.ToBase64String, which maps every three bytes onto four safe ASCII characters, and Convert.FromBase64String to reverse it. It is neither encryption nor compression: the result is about a third larger. For bytes going into a URL, Base64Url in System.Buffers.Text (.NET 9 and later) uses - and _ in place of + and /.

Example

using System;
using System.Collections.Generic;
using System.Text;

// These two are the same string. The raw literal (C# 11) needs no backslashes.
string line = """1,"Ada, Countess of Lovelace",1815""";
string escaped = "1,\"Ada, Countess of Lovelace\",1815";
Console.WriteLine($"same: {line == escaped}");

// Splitting on every comma is wrong here: one field contains a comma.
Console.WriteLine($"naive: {line.Split(',').Length}");

List<string> fields = SplitCsvLine(line);
Console.WriteLine($"fields: {fields.Count}");
Console.WriteLine($"name: {fields[1]}");
Console.WriteLine($"utf8 bytes: {Encoding.UTF8.GetBytes(fields[1]).Length}");
Console.WriteLine($"base64: {Convert.ToBase64String(Encoding.UTF8.GetBytes("ok"))}");

static List<string> SplitCsvLine(string text)
{
    List<string> result = new List<string>();
    StringBuilder field = new StringBuilder();
    bool inQuotes = false;

    foreach (char c in text)
    {
        if (c == '"')
        {
            inQuotes = !inQuotes;            // a quote opens or closes a field
        }
        else if (c == ',' && !inQuotes)
        {
            result.Add(field.ToString());     // end of a field
            field.Clear();
        }
        else
        {
            field.Append(c);
        }
    }

    result.Add(field.ToString());             // the last field has no comma after it
    return result;
}

// Output:
// same: True
// naive: 4
// fields: 3
// name: Ada, Countess of Lovelace
// utf8 bytes: 25
// base64: b2s=

When to use it

Import a spreadsheet export where one column contains commas inside quotes.
Fix a file that arrived in a legacy encoding and shows question marks.
Put a small image or signature inside a JSON field.
Produce a CSV that opens correctly for colleagues in other countries.

More examples

Bytes, characters and the byte order mark

Character count and byte count are different numbers, and code that assumes they are equal breaks on the first accented name. The last two lines explain a bug people hit repeatedly: Encoding.UTF8 is a UTF8Encoding configured to emit a byte order mark, while the default used by File.WriteAllText and StreamWriter is one that does not. If another tool reports a stray character before your first column, this is why.

using System;
using System.IO;
using System.Text;

string text = "héllo";   // five characters, one of them not ASCII

byte[] utf8 = Encoding.UTF8.GetBytes(text);
byte[] utf16 = Encoding.Unicode.GetBytes(text);   // UTF-16: two bytes per character here
Console.WriteLine($"utf8: {utf8.Length}  utf16: {utf16.Length}");
Console.WriteLine(Encoding.UTF8.GetString(utf8) == text);

// No encoding argument: UTF-8 with no byte order mark.
File.WriteAllText("plain.txt", text);

// Passing Encoding.UTF8 adds a three-byte mark at the front.
File.WriteAllText("marked.txt", text, Encoding.UTF8);

Console.WriteLine($"plain: {new FileInfo("plain.txt").Length}  marked: {new FileInfo("marked.txt").Length}");

// Output:
// utf8: 6  utf16: 10
// True
// plain: 6  marked: 9

Base64 round trip

Reach for this when a byte array has to sit inside a JSON string, an HTTP header or a database text column: a thumbnail, a signature, an encrypted blob. The trailing equals signs are padding, not corruption. Remember the size cost before putting a large file in a JSON field, and remember that anyone can decode it, so Base64 hides nothing.

using System;
using System.Text;

byte[] bytes = new byte[] { 0, 1, 2, 250 };   // not printable text at all

string encoded = Convert.ToBase64String(bytes);
Console.WriteLine(encoded);

byte[] decoded = Convert.FromBase64String(encoded);
Console.WriteLine(decoded.Length);
Console.WriteLine(decoded[3]);

// Base64 grows the data by about a third.
byte[] payload = Encoding.UTF8.GetBytes("a short message");
Console.WriteLine($"{payload.Length} -> {Convert.ToBase64String(payload).Length}");

// Output:
// AAEC+g==
// 4
// 250
// 15 -> 20

CSV with a library

The library matches the header row to property names, handles quoted fields, and writes a header when saving. Two details matter. GetRecords yields records one at a time, so calling ToList inside the using block is what stops an ObjectDisposedException later. And CultureInfo.InvariantCulture fixes the number and date formats, so a machine configured for a comma decimal separator does not produce a file nobody else can read.

// dotnet add package CsvHelper
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using CsvHelper;

namespace Importer;

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; } = "";
}

public static class CustomerFile
{
    public static List<Customer> Read(string path)
    {
        using StreamReader reader = new StreamReader(path);
        using CsvReader csv = new CsvReader(reader, CultureInfo.InvariantCulture);

        // GetRecords reads lazily, so finish the list before the reader is disposed.
        return csv.GetRecords<Customer>().ToList();
    }

    public static void Write(string path, IEnumerable<Customer> customers)
    {
        using StreamWriter writer = new StreamWriter(path);
        using CsvWriter csv = new CsvWriter(writer, CultureInfo.InvariantCulture);
        csv.WriteRecords(customers);
    }
}

Discussion

Be the first to comment on this lesson.

CSV

Encodings

Base64

Example

When to use it

More examples

Bytes, characters and the byte order mark

Base64 round trip

CSV with a library

Related

Discussion