file_re

file_re is a Rust-backed Python library for running regular expressions over large files. It mirrors the public surface of re and adds a single max_span_lines parameter that controls how much of the file is held in memory, so the same API scales from small configuration files to 50 GB compressed logs.

Highlights

• search, match, findall, finditer, and compile with the same shape as re.

• A max_span_lines knob: None for full-file scans, 1 for line-by-line streaming, and N for a sliding N-line window.

• Transparent .gz and .xz decompression.

• Proper re.Match semantics, including None for non-participating groups.

• GIL is released during IO and regex work, so file_re plays well with multiprocessing and threaded pipelines.

Getting started

• Installation
  • From PyPI
  • Supported platforms
  • Python requirements
  • Building from source
  • Development install
• Quickstart
  • Basic search
  • Capture groups and named groups
  • findall and finditer
  • Compiled patterns
  • Flags
  • Compressed files
  • Large files

Guides

• Large files and max_span_lines
  • The three modes
  • Memory trade-offs
  • Behavior when a match would exceed the window
  • Search returns the first match
  • Multiprocessing over huge files
  • Compressed files
• Flags
  • Supported flags
  • Rust-vs-Python divergences
  • Inline directives
• Migrating from 1.x to 2.0
  • Minimum Python 3.9
  • multiline=True is now the default
  • num_lines=N is now max_span_lines=N
  • search() in window mode now returns the first match
  • Match.groups now contains None for non-participating groups
  • Flag changes
  • New: finditer, match, and compile
  • Streaming \r\n handling

API reference

• Module-level functions
  • file_re_cls
• Pattern
  • Pattern
• Match
  • Match

Indices and tables

• Index

• Module Index

• Search Page