CSV and Tabular Files
Read and write CSV with DictReader and DictWriter, why newline='' is required, and what quoting and types really do.
csv.DictReader(f, fieldnames=None, restkey=None, restval=None, dialect='excel')CSV has no type system, no schema and no single specification, which is why the csv module exists: it handles the quoting, embedded commas and embedded newlines that a line.split(",") gets wrong on the first address field.
Dicts instead of index positions
csv.DictReader(f) reads the first row as the header and yields one dict per row. The longhand is csv.reader(f) plus dict(zip(header, row)) after a next() for the header — identical result, more moving parts. If you pass fieldnames yourself the first row is treated as data instead. Short rows are padded with restval; extra fields land in a list under restkey. Writing is the mirror: csv.DictWriter(f, fieldnames), then writeheader() and writerow() or writerows().
newline='' is not optional
The docs state that a file object passed to the csv module should be opened with newline=''. Without it, newlines inside quoted fields are not interpreted correctly, and on platforms using \r\n an extra \r is written. The module does its own newline handling, so it is always safe to pass. Set encoding explicitly in the same call.
Everything arrives as a string
Every value from a reader is str, including the column you think of as a number. Convert once, as rows come in, not wherever the value is first used. quoting controls output only: QUOTE_MINIMAL is the default, QUOTE_ALL quotes every field, and QUOTE_NONNUMERIC on a reader converts unquoted fields to float.
Example
When to use it
- Load a supplier price list where one description field contains commas and quotes.
- Export a report that opens correctly in Excel on both Windows and macOS.
- Convert a column of numeric strings to int or Decimal once, as the rows arrive.
- Count or aggregate rows in a log export too large to load into a list.
More examples
Real files, both ends
The shape every CSV task takes: open with newline='' and an explicit encoding, read into dicts, write through a DictWriter whose fieldnames fix the column order. QUOTE_ALL is worth it when the consumer is a spreadsheet that would otherwise guess at types.
import csv
from pathlib import Path
src = Path("orders.csv")
# newline="" on BOTH sides: it lets the csv module do its own newline handling
# on read, and stops a second \r being added on write.
with src.open(newline="", encoding="utf-8") as f:
rows = list(csv.DictReader(f))
with Path("report.csv").open("w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["sku", "units"], quoting=csv.QUOTE_ALL)
writer.writeheader()
# writerows takes any iterable, so a generator expression works here.
writer.writerows({"sku": r["sku"], "units": int(r["qty"])} for r in rows)Ragged rows: restkey and restval
Vendor exports are rarely rectangular. Without restval a short row gives qty=None, which crashes int() three functions later; without restkey the surplus fields vanish silently. Setting both turns a malformed row into something you can detect and reject at the boundary.
import csv, io
RAW = "sku,qty\nA-100,2\nB-200\nC-300,1,extra,junk\n"
reader = csv.DictReader(io.StringIO(RAW), restkey="_extra", restval="0")
for row in reader:
print(row)
# {'sku': 'A-100', 'qty': '2'}
# {'sku': 'B-200', 'qty': '0'} <- restval filled the short row
# {'sku': 'C-300', 'qty': '1', '_extra': ['extra', 'junk']}Streaming a file too big to hold
Use this whenever the file is bigger than you want in RAM, or when the job is an aggregate rather than a transformation. The reader must be consumed inside the with block: once the file is closed, iterating further raises ValueError on a closed file.
import csv
from collections import Counter
from pathlib import Path
# DictReader is an iterator: it reads one row at a time, so a multi-gigabyte
# file costs one row of memory. list() is what makes it expensive, not the reader.
counts = Counter()
with Path("events.csv").open(newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
counts[row["kind"]] += 1
print(counts.most_common(3))
Discussion