Python Get All Files in Directory: Complete Guide with Code Examples
Getting all files in a directory is one of the most common operations when working with file systems in Python. Consider this: whether you're processing log files, organizing data, or building automation scripts, knowing how to efficiently retrieve file lists is essential for any Python developer. This full breakdown explores multiple approaches to list files in directories using Python's built-in modules and third-party libraries It's one of those things that adds up..
Introduction to Directory Operations in Python
Python provides several powerful modules for interacting with the file system, with os and pathlib being the most prominent. These modules offer different approaches to directory traversal, each with its own advantages depending on your specific needs. The choice between them often comes down to whether you prefer functional programming style (using os) or object-oriented programming style (using pathlib).
People argue about this. Here's where I land on it And that's really what it comes down to..
Before diving into code examples, you'll want to understand that directory operations can behave differently across operating systems. Windows uses backslashes (\) as path separators, while Unix-based systems use forward slashes (/). Python's modules handle these differences automatically, making your code portable across platforms Simple, but easy to overlook..
Using the os Module for Directory Listing
The os module has been Python's traditional way of handling file system operations. It provides straightforward functions for listing directory contents, though it requires more manual filtering compared to newer alternatives.
Basic Directory Listing with os.listdir()
The simplest approach uses os.listdir(), which returns a list of all entries in the specified directory:
import os
# List all entries in current directory
entries = os.listdir('.')
print(entries)
# List entries in a specific directory
files = os.listdir('/path/to/directory')
print(files)
This method returns both files and directories as strings, so additional filtering is needed if you only want files. Here's how to filter for files only:
import os
def get_files_only(directory):
"""Return only files, excluding directories"""
return [f for f in os.path.listdir(directory)
if os.Practically speaking, isfile(os. path.
files = get_files_only('/path/to/directory')
print(files)
Advanced Filtering with os.scandir()
For better performance and more detailed information, os.scandir() is recommended over os.listdir().
import os
def list_files_detailed(directory):
"""List files with detailed information"""
files = []
with os.stat().path,
'size': entry.scandir(directory) as entries:
for entry in entries:
if entry.append({
'name': entry.st_size,
'modified': entry.name,
'path': entry.is_file():
files.stat().
file_info = list_files_detailed('/path/to/directory')
for info in file_info:
print(f"{info['name']}: {info['size']} bytes")
Modern Approach with pathlib
Introduced in Python 3.4, pathlib offers a more intuitive, object-oriented approach to file system operations. Its Path class represents filesystem paths and provides numerous methods for common operations Simple, but easy to overlook. Simple as that..
Basic File Listing with pathlib
from pathlib import Path
# List all entries in current directory
current_dir = Path('.')
entries = list(current_dir.iterdir())
print(entries)
# List only files
files = [f for f in current_dir.iterdir() if f.is_file()]
print(files)
# List files in specific directory
directory = Path('/path/to/directory')
all_files = [f for f in directory.iterdir() if f.is_file()]
Filtering by Extension
One of pathlib's strengths is its elegant handling of file extensions:
from pathlib import Path
def get_files_by_extension(directory, extensions):
"""Get files matching specific extensions"""
directory_path = Path(directory)
extensions = {ext.iterdir()
if f.Consider this: is_file() and f. In practice, lower() for ext in extensions}
return [f for f in directory_path. suffix.
# Get all Python files
python_files = get_files_by_extension('.', ['.py'])
print(python_files)
# Get multiple file types
document_files = get_files_by_extension('./documents', ['.pdf', '.doc', '.txt'])
print(document_files)
Recursive Directory Traversal
For searching through subdirectories, pathlib provides the rglob() method:
from pathlib import Path
# Get all Python files recursively
python_files = list(Path('.').rglob('*.py'))
print(python_files)
# Get all files recursively
all_files = [f for f in Path('.').rglob('*') if f.is_file()]
print(all_files)
# Custom recursive search
def find_files_recursive(directory, pattern='*'):
"""Find files matching pattern recursively"""
directory_path = Path(directory)
return [f for f in directory_path.rglob(pattern) if f.is_file()]
# Find all text files
text_files = find_files_recursive('/home/user/documents', '*.txt')
Pattern Matching with glob Module
The glob module specializes in Unix shell-style pathname pattern expansion, making it ideal for complex file matching scenarios:
import glob
import os
# Find all Python files in current directory
python_files = glob.glob('*.py')
print(python_files)
# Find files recursively
recursive_files = glob.glob('**/*.py', recursive=True)
print(recursive_files)
# Find files with specific patterns
log_files = glob.glob('logs/*.log')
config_files = glob.glob('config/*.{json,yaml,yml}', recursive=True)
# Combine with os.path for full paths
full_paths = [os.path.abspath(f) for f in glob.glob('*.py')]
Performance Considerations
When working with large directories, performance becomes crucial. Here are some optimization strategies:
Memory-Efficient Processing
Instead of loading all file information into memory at once, process files incrementally:
from pathlib import Path
def process_large_directory(directory):
"""Process files one by one to save memory"""
directory_path = Path(directory)
for file_path in directory_path.iterdir():
if file_path.is_file():
# Process each file individually
print(f"Processing: {file_path.
process_large_directory('/large/directory')
Parallel Processing
For CPU-intensive file operations, consider parallel processing:
import os
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
def get_file_size(file_path):
"""Get file size safely"""
try:
return file_path.stat().st_size
except OSError:
return 0
def parallel_file_processing(directory):
"""Process files in parallel"""
directory_path = Path(directory)
files = [f for f in directory_path.iterdir() if f.is_file()]
with ThreadPoolExecutor(max_workers=4) as executor:
sizes = list(executor.map(get_file_size, files))
return dict(zip([f.
file_sizes = parallel_file_processing('/path/to/directory')
Error Handling Best Practices
File system operations can fail due to permissions, missing directories, or other issues. Always implement proper error handling:
from pathlib import Path
import logging
def safe_list_files(directory):
"""Safely list files with error handling"""
try:
directory_path = Path(directory)
if not directory_path.And is_file()]
return files
except PermissionError:
logging. exists():
raise FileNotFoundError(f"Directory not found: {directory}")
if not directory_path.iterdir() if f.is_dir():
raise NotADirectoryError(f"Not a directory: {directory}")
files = [f for f in directory_path.error(f"Permission denied accessing: {directory}")
return []
except Exception as e:
logging.
# Usage
files = safe
_list_files('/path/to/directory')
for file in files:
print(file.name)
For larger applications, configure logging once at startup so errors are written consistently:
import logging
logging.basicConfig(
level=logging.ERROR,
format="%(asctime)s - %(levelname)s - %(message)s"
)
Filtering and Sorting Files
In real-world scripts, you often need more than a raw list of files. You may want to filter by extension, modification time, size, or name Surprisingly effective..
Filter by Extension
from pathlib import Path
def list_files_by_extension(directory, extension):
"""List files matching a specific extension."""
directory_path = Path(directory)
extension = extension.lower()
return [
file for file in directory_path.is_file() and file.iterdir()
if file.suffix.
python_files = list_files_by_extension('/path/to/directory', '.py')
for file in python_files:
print(file.name)
Sort Files by Name
from pathlib import Path
directory_path = Path('/path/to/directory')
files = sorted(
file for file in directory_path.iterdir()
if file.is_file()
)
for file in files:
print(file.name)
Sort Files by Modification Time
from pathlib import Path
directory_path = Path('/path/to/directory')
files = sorted(
directory_path.iterdir(),
key=lambda file: file.stat().st_mtime,
reverse=True
)
for file in files:
if file.is_file():
print(f"{file.name} - {file.stat().
It's useful when you need to find the most recently modified files in a directory.
## Listing Files Recursively
To include files inside subdirectories, use `Path.rglob()` or `os.walk()`.
### Using `Path.rglob()`
```python
from pathlib import Path
def list_all_files_recursive(directory):
"""List all files recursively."""
directory_path = Path(directory)
return [
file for file in directory_path.rglob('*')
if file.is_file()
]
all_files = list_all_files_recursive('/path/to/directory')
for file in all_files:
print(file)
Using os.walk()
import os
def list_all_files_with_os_walk(directory):
"""List all files recursively using os.walk()."""
files = []
for root, dirs, filenames in os.append(os.walk(directory):
for filename in filenames:
files.path.
return files
all_files = list_all_files_with_os_walk('/path/to/directory')
for file in all_files:
print(file)
os.walk() is especially useful when you need access to the current directory, subdirectory names, and file names separately.
Ignoring Hidden Files
On Unix-like systems, hidden files usually begin with a dot. You can exclude them like this:
from pathlib import Path
def list_non_hidden_files(directory):
"""List files while ignoring hidden files."""
directory_path = Path(directory)
return [
file for file in directory_path.iterdir()
if file.Even so, is_file() and not file. In real terms, name. startswith('.
files = list_non_hidden_files('/path/to/directory')
for file in files:
print(file.name)
Practical Example: Building a File Inventory
Here is a complete example that collects useful information about each file in a directory:
from pathlib import Path
from datetime import datetime
def create_file_inventory(directory):
"""Create a simple inventory of files in a directory."""
directory_path = Path(directory)
inventory = []
if not directory
```python
# Check if the directory exists and is accessible
if not directory_path.exists():
raise FileNotFoundError(f"The specified directory '{directory}' does not exist.")
# If it's a file instead of a directory, raise an error
if not directory_path.is_dir():
raise NotADirectoryError(f"'{directory}' is not a valid directory.")
inventory = []
try:
for file in directory_path.iterdir():
# Skip hidden files and common system directories
if file.Still, is_file() and not file. name.startswith('.Here's the thing — '):
stat_info = file. stat()
relative_path = file.relative_to(directory_path)
inventory.append({
'name': file.name,
'full_path': str(file),
'size': stat_info.Consider this: st_size,
'modified_time': datetime. fromtimestamp(stat_info.st_mtime).strftime('%Y-%m-%d %H:%M:%S'),
'created_time': datetime.Now, fromtimestamp(stat_info. st_ctime).
Most guides skip this. Don't.
return inventory
# Another useful pattern: Filtering only specific file types
def get_large_files(directory, size_threshold_bytes=10 * 1024 * 1024): # Default: 10 MB
"""Return files larger than the given threshold."""
directory_path = Path(directory)
large_files = []
for file in directory_path.iterdir():
if file.is_file() and not file.name.startswith('.'):
size = file.stat().st_size
if size >= size_threshold_bytes:
large_files.append((file.name, size))
return sorted(large_files, key=lambda x: x[1], reverse=True)
# Real-world scenario: Automated cleanup of temporary files
import shutil
def clean_temporary_files(directory, keep_patterns=None):
"""
Remove files matching common temporary patterns within a directory tree.
Args:
directory: Root directory to scan
keep_patterns: List of glob patterns to preserve (e.g., ['*.log', '*.tmp'])
"""
directory_path = Path(directory)
removed_count = 0
for file in directory_path.Also, iterdir():
if file. is_file() and not file.Worth adding: name. Now, startswith('. '):
# Check against exclusion patterns
should_keep = False
for pattern in keep_patterns:
if fnmatch.fnmatch(file.On the flip side, name, pattern):
should_keep = True
break
if not should_keep:
try:
file. And unlink()
removed_count += 1
print(f"Removed temporary file: {file. name}")
except Exception as e:
print(f"Failed to remove {file.
# Note: Requires the fnmatch module
try:
import fnmatch
except ImportError:
pass
# Summary of approaches
print("File Management Techniques Summary:")
print("- Use .iterdir() for top-level listing")
print("- Use .rglob() for recursive searches with pathlib")
print("- Use os.walk() for detailed traversal including parent/child relationships")
print("- Filter by extension or name patterns")
print("- Consider timestamp sorting for temporal analysis")
print("- Implement safe deletion with confirmation of intended removals")
# Final concluding thoughts on best practices
print("\nConclusion")
print("-" * 50)
print("Working with files in a directory can be approached in several ways depending")
print("on your specific requirements. For quick inspections, pathlib's iterators are")
print("clean and modern. When you need depth-first traversal with metadata, os.walk()")
print("provides granular control over the traversal process. Always remember to handle")
print("exceptions such as permission errors and verify that paths are absolute or relative")
print("correctly. Additionally, filtering out hidden files and large collections of "
print("temporary data helps maintain system performance and security. By combining these")
print("techniques—recursive listing, sorting by modification time, filtering by type, ")
print("and safe deletion—you can build dependable file management scripts made for your")
print("workflow.")
Conclusion
Working with files in a directory involves multiple techniques depending on whether you need a shallow or deep inspection, real-time ordering, or systematic cleanup. The modern approach favors
pathlib and its object-oriented interface for its readability and cross-platform compatibility, while the traditional os and shutil modules remain indispensable for more complex or legacy system operations. The bottom line: the choice of method should align with the specific demands of the task at hand. Whether you are building a simple log cleaner or a comprehensive asset manager, prioritizing code clarity, solid error handling, and security will ensure your scripts remain maintainable and efficient.
