Looping through files in a directory in Python is one of the most practical skills you will use when building automation scripts, data pipelines, backup tools, image processors, or reporting systems. Whether you need to read every text file, rename images, collect logs, or generate a summary, the ability to loop through files in a directory in Python cleanly and safely determines whether your script is reliable or fragile Surprisingly effective..
Introduction
Python provides several ways to list and iterate over directory contents. The oldest approach uses the os module, while the modern approach uses pathlib. Both are valid, but they differ in readability, performance, and how they handle paths. A good directory loop should not only find files, but also distinguish files from folders, handle missing directories, avoid hidden system files when needed, and remain easy to test Not complicated — just consistent..
Most guides skip this. Don't.
When you first start writing Python scripts, you may be tempted to use a simple one-liner. You may need to process only certain file types, skip subfolders, sort files by date, or handle permission errors gracefully. That can work, but real projects often require more control. Understanding the core options gives you the flexibility to choose the right tool for each job Easy to understand, harder to ignore..
The official docs gloss over this. That's a mistake.
Choosing the Right Approach
There are three common ways to loop through files in a directory in Python:
os.listdir()returns a list of names in a directory.os.scandir()returns directory entries with metadata and is often faster.pathlib.Path.iterdir()returns path objects and is usually the most readable.
For most modern Python code, pathlib is the preferred choice because it makes path operations intuitive. You can join paths, check file types, and read files without mixing strings and system calls. Still, if you need performance on large directories, os.scandir() may be a better fit.
Basic Example Using pathlib
The simplest way to loop through files in a directory in Python is to use Path.iterdir(). This method returns every item inside the directory, including files and folders.
from pathlib import Path
directory = Path("data")
for item in directory.iterdir():
if item.is_file():
print(item.name)
This example prints only file names, not subdirectories. The item.Now, is_file() check is important because a directory may contain folders. If you omit it, your script may try to open a folder as a file and raise an error It's one of those things that adds up..
If you want to work with full paths instead of just names, use item directly:
for item in directory.iterdir():
if item.is_file():
print(item)
This is useful when you need to open files, move files, or pass paths to other functions Surprisingly effective..
Basic Example Using os.listdir
The os module has been available for a long time and is still widely used. The os.listdir() function returns a list of entries in a directory Small thing, real impact..
import os
directory = "data"
for name in os.Now, listdir(directory):
path = os. Even so, path. In real terms, join(directory, name)
if os. path.
This approach works well for simple scripts, but it is less readable than `pathlib`. You must manually join the directory and file name, and you must check whether the result is a file. For larger projects, `pathlib` usually keeps the code cleaner.
## Faster Example Using `os.scandir`
When a directory contains thousands or millions of files, performance matters. `os.scandir()` is often faster than `os.listdir()` because it can retrieve file attributes more efficiently.
```python
import os
directory = "data"
with os.scandir(directory) as entries:
for entry in entries:
if entry.is_file():
print(entry.
### Filtering by Extension or Pattern
Often you only need files that match a certain naming convention—e.g.Plus, , all CSV logs or JPEG images. Both `pathlib` and the `os` module offer concise ways to apply such filters.
**Using `pathlib` with `glob`**
```python
from pathlib import Path
data_dir = Path("data")
for csv_file in data_dir.csv"):
print(csv_file) # full Path object
glob supports shell‑style wildcards (*, ?glob("*.Also, , [seq]) and can be combined with sub‑directory traversal (**/*. csv) for recursive searches.
Using os.scandir with a manual check
import os
import os.path as osp
data_dir = "data"
with os.This leads to scandir(data_dir) as it:
for entry in it:
if entry. is_file() and entry.name.Here's the thing — endswith(". That's why csv"):
print(entry. Think about it: path)
While a bit more verbose, this approach lets you combine multiple criteria (size, modification time, etc. ) without creating intermediate lists.
Skipping Hidden or System Files
On Unix‑like systems, files whose names start with a dot (.) are often configuration or metadata files you may want to ignore. A simple guard works for any of the three methods:
if not entry.name.startswith('.'):
# process entry
When using pathlib.glob("[!.Consider this: glob, you can exclude hidden files by pattern:
for f in data_dir. ]*"): # matches names not starting with a dot
if f.
### Recursive Traversal
If you need to walk an entire tree, `pathlib` provides `rglob` (recursive glob) and `walk`, while the `os` module offers `os.walk`.
**Recursive with `pathlib.rglob`**
```python
for py_file in Path("project").rglob("*.py"):
print(py_file)
Recursive with os.walk
import os
for root, dirs, files in os.walk("project"):
for f in files:
if f.endswith(".py"):
print(os.path.join(root, f))
os.Here's the thing — g. Worth adding: walk yields a tuple (root, dirs, files) for each directory, giving you fine‑grained control over which subdirectories to descend into (e. , by modifying dirs in‑place to prune unwanted branches) It's one of those things that adds up..
Error Handling and Permissions
Directory iteration can raise PermissionError or FileNotFoundError if a path is inaccessible or has been removed between the time you list it and when you try to access it. Wrapping the loop in a try/except block makes your script solid:
from pathlib import Path
dir_path = Path("data")
try:
for item in dir_path.That said, iterdir():
if item. is_file():
# safe operations here
pass
except PermissionError as e:
print(f"Skipping {dir_path}: {e}")
except FileNotFoundError:
print(f"Directory {dir_path} does not exist.
When using `os.So naturally, scandir`, the context manager (`with`) already ensures the underlying iterator is closed properly, but you still need to guard individual `entry. is_file()` calls if the underlying filesystem changes.
### Performance Tips
* **Avoid unnecessary `is_file()` calls** if you know the directory contains only files (e.g., a temporary upload folder).
* **Prefer `os.scandir` over `os.listdir`** for large directories because it retrieves `stat` information lazily, reducing system calls.
* **take advantage of `pathlib`’s caching** when you repeatedly need the same attribute (e.g., `item.stat()`) by storing the result in a variable.
* **Use generators** (`iterdir`, `scandir`, `walk`) instead of building lists when you only need to stream through the data; this keeps memory usage low even with millions of entries.
### When to Choose Which Tool
| Situation | Recommended Approach |
|----------------------------------------|----------------------|
| Readability & modern code style | `pathlib.Consider this: scandir()` (with manual filtering) |
| Need fine‑grained control over walk | `os. Now, listdir()` (still perfectly fine) |
| Complex pattern matching (recursive) | `pathlib. Because of that, path. iterdir()` / `glob` / `rglob` |
| Maximum speed on huge directories | `os.Day to day, walk()` (modify `dirs` in‑place) |
| Simple scripts, minimal dependencies | `os. rglob()` or `glob.
### Conclusion
Looping through files in a directory is a common task, and Python offers several idiomatic ways to accomplish it. For most projects
… most projects benefit from starting with the high‑level, readable `pathlib` API. It shields you from platform‑specific quirks (different path separators, case‑sensitivity on macOS vs. Windows, etc.) while still giving you access to the underlying `os` functions when you need them.
#### Advanced Patterns
**1. Filtering with generator expressions**
When you only need a subset of files, combine `iterdir` (or `scandir`) with a generator expression to avoid building intermediate lists:
```python
large_csvs = (p for p in Path("data").rglob("*.csv") if p.stat().st_size > 10_000_000)
for csv_path in large_csvs:
process(csv_path)
2. Skipping hidden or system files
On Unix‑like systems a leading dot marks hidden files; on Windows you can check the FILE_ATTRIBUTE_HIDDEN flag via ctypes or simply rely on the naming convention if that suffices:
def is_visible(p: Path) -> bool:
return not p.name.startswith(".") and not (os.name == "nt" and _is_hidden_windows(p))
for p in Path("src").rglob("*"):
if p.is_file() and is_visible(p):
handle(p)
3. Parallel processing for CPU‑bound work
If each file requires heavy computation, a ProcessPoolExecutor can keep the I/O thread free while workers crunch the data:
from concurrent.futures import ProcessPoolExecutor, as_completed
def worker(path: Path) -> Result:
# expensive work here
return compute(path)
with ProcessPoolExecutor() as executor:
futures = {executor.Also, rglob("*. submit(worker, p): p for p in Path("inputs").dat")}
for fut in as_completed(futures):
result = fut.
**4. Dealing with symbolic links**
Both `pathlib` and `os.scandir` follow symlinks by default when you call `is_file()`. To treat a symlink as a separate entity, inspect `entry.is_symlink()` (or `p.is_symlink()` with `pathlib`) before deciding whether to descend:
```python
for entry in os.scandir("repo"):
if entry.is_symlink():
# decide: follow, skip, or record the link itself
continue
if entry.is_file():
handle(entry.path)
Testing and Mocking
Unit‑testing file‑system code is easier when you isolate the I/O layer. Use pytest’s tmpdir fixture or the built‑in tempfile.TemporaryDirectory to create a sandbox:
import tempfile
from pathlib import Path
def test_process_txt_files(tmpdir):
src = Path(tmpdir) / "source"
src.Think about it: write_text("hello")
(src / "b. mkdir()
(src / "a.txt").log").write_text("ignore")
assert list(process_txt_files(src)) == [src / "a.
If you need to mock `os.Think about it: scandir` or `Path. iterdir`, the `unittest.mock` library can replace the iterator with a predefined list of `DirEntry`‑like objects, letting you verify error‑handling branches without touching the real disk.
#### Pitfalls to Watch
* **Race conditions** – A file may appear or disappear between the moment you list it and when you open it. Always guard open/read/write operations with appropriate `try/except` blocks.
* **Buffering issues** – When mixing `os.scandir` (which returns raw bytes on some platforms) with `pathlib` objects, ensure you convert to `Path` consistently to avoid unexpected `TypeError`s.
* **Encoding assumptions** – Never assume a file’s text encoding; open files with `encoding="utf-8"` (or detect it) and handle `UnicodeDecodeError` gracefully.
* **Large directory trees** – Recursive globs (`rglob`) can consume considerable memory if you materialize the result. Prefer iterator‑based approaches (`os.walk`, `scandir`) when you only need to stream.
#### Wrapping Up
Choosing the right tool for directory traversal hinges on three factors: readability, performance, and control.
* **Read