Python Scrapy Export to CSV and JSON

When you run a Scrapy spider, the items it yields are the only thing the crawler produces unless you direct them somewhere. The standard way to get scraped data into a file is Scrapy's feed export system, which writes items to CSV, JSON, JSON Lines, or other formats as they are scraped. For the common case of python scrapy export csv json, you have two paths: the -o command line flag for quick output, and the FEEDS setting for repeatable, configurable exports.

Quick Output with the -o Flag

The simplest export is a single command line flag:

scrapy crawl quotes -o quotes.csv

Scrapy infers the format from the file extension. The same works for JSON:

scrapy crawl quotes -o quotes.json

If the extension is ambiguous or you want to force a format, use -t:

scrapy crawl quotes -t csv -o quotes.data

The -o flag appends to an existing file if it already exists, which can cause duplicate rows when you rerun a spider. For a clean run, remove the output file first or use the FEEDS setting with overwrite: true.

Configuring Exports with FEEDS

For anything beyond a one-off run, put the export configuration in settings.py. The FEEDS setting maps output locations to format options:

FEEDS = {
    "output/quotes.csv": {
        "format": "csv",
        "encoding": "utf-8",
        "overwrite": True,
    },
    "output/quotes.json": {
        "format": "json",
        "encoding": "utf-8",
        "overwrite": True,
    },
}

This writes both formats in a single crawl. The overwrite key controls whether an existing file is truncated or appended. Leaving it False (the default) appends, which is useful when you want to accumulate results across multiple runs but dangerous when you expect a fresh file each time.

You can also scope feeds per spider by defining custom_settings inside the spider class:

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    custom_settings = {
        "FEEDS": {
            "quotes.csv": {"format": "csv", "overwrite": True},
        }
    }

JSON Output Is JSON Lines, Not a JSON Array

A common surprise is that Scrapy's json feed format writes one JSON object per line rather than a single JSON array. The file looks like this:

{"author": "Douglas Adams", "text": "The answer is 42"}
{"author": "Terry Pratchett", "text": "I don't know"}

This is valid JSON Lines (.jsonl) but not a valid JSON document. If you need a proper array, use the jsonlines format for line-delimited output, or post-process the file with a small script that wraps the lines in brackets and joins them with commas.

The reason Scrapy streams items this way is memory: writing each item as it is scraped avoids buffering the entire result set. A CSV or JSON Lines file can grow to gigabytes without holding more than one item in memory at a time. A JSON array, by contrast, requires either building the whole list in memory or writing a closing bracket after the last item, which Scrapy does not do for its json feed.

If you genuinely need an array, write a small post-processing step:

import json

with open("quotes.json", "r") as f:
    items = [json.loads(line) for line in f if line.strip()]

with open("quotes_array.json", "w") as f:
    json.dump(items, f, indent=2)

Controlling Field Order and Selection in CSV

CSV has no schema, so Scrapy decides which columns to write based on the first item it encounters. If your spider yields items with inconsistent fields, the CSV columns will be whatever the first item happened to contain, and later items with additional fields will silently drop those columns.

Fix this by declaring the field list in the feed configuration:

FEEDS = {
    "quotes.csv": {
        "format": "csv",
        "fields": ["author", "text", "tags"],
        "overwrite": True,
    },
}

The fields key fixes both the column order and the set of columns. Any item field not listed is omitted. This is the difference between a CSV that changes shape between runs and one that is stable enough to load into a database or spreadsheet.

When items are Scrapy Item objects with defined fields, the field order in the item class also influences output, but the explicit fields list in FEEDS takes precedence and is the more reliable place to control it.

Serialization of Non-String Values

CSV and JSON both require values to be serializable. Scrapy handles the common Python types, but you will hit edge cases with dates, enums, and custom objects.

For JSON, a datetime object raises a TypeError because the standard json module cannot serialize it. The fix is a custom serializer in the feed configuration:

import json
from datetime import datetime, date

class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, (datetime, date)):
            return obj.isoformat()
        return super().default(obj)

FEEDS = {
    "quotes.json": {
        "format": "json",
        "serializer": DateTimeEncoder,
        "overwrite": True,
    },
}

For CSV, every value is converted to a string by the writer, so dates become their str() representation unless you convert them in the item pipeline first. If you need ISO 8601 dates in the CSV, convert the value in the spider or in a pipeline before the item reaches the feed exporter.

The cleanest place to normalize values is an item pipeline, because it runs before the feed exporter writes the item:

class DateNormalizerPipeline:
    def process_item(self, item, spider):
        if isinstance(item.get("published"), datetime):
            item["published"] = item["published"].isoformat()
        return item

Performance and Memory Behavior

The feed exporter writes items as they are yielded, so the memory footprint stays flat regardless of how many items the spider produces. The main cost is I/O: writing to a local disk is fast, but writing to a remote location or a network filesystem can slow the crawl if the exporter cannot keep up with the spider's yield rate.

For large exports, consider these points:

• The overwrite flag controls truncation, but appending to a large existing file still opens the file in append mode and adds new rows at the end without rewriting existing content.
• If you use a post-processing step to convert JSON Lines into an array, that step loads the entire file into memory. For very large files, stream the conversion instead of using json.loads on every line at once.
• CSV with many columns and long text fields produces larger files than the equivalent JSON because every value is quoted and escaped according to CSV rules. This affects disk usage but not crawl speed in most cases.

There is no built-in compression in the feed exporter. If disk space is a concern, write the feed to a temporary file and compress it after the crawl, or pipe the output through a compression tool at the command line.

Failure Modes and Debugging

The most common failure is a TypeError during serialization, which stops the crawl. The traceback points to the item and the field that failed, so check whether the value is a datetime, Decimal, set, or a custom class before blaming the exporter.

Another frequent issue is empty output. If the spider yields no items, the feed exporter creates an empty file (for CSV) or no file at all (for some formats). This is not an error—it simply means the spider's parsing rules matched nothing. Check the spider's logs for the number of items scraped to distinguish an empty result from a broken export.

When overwrite is False and the output file already exists, Scrapy appends to it. Rerunning a spider without cleaning the file produces duplicate rows. This is the single most common cause of "my CSV has double the data" reports, and it is a configuration issue, not a bug.

For CSV specifically, if the fields list is missing and items vary in shape, the output can have inconsistent columns. Always declare fields for CSV feeds that must be loaded by other tools. When you need both CSV and JSON from the same crawl, configure both in FEEDS so the exporter handles each format independently and you avoid writing custom conversion code between the two.