Glob find all files in subfolders is a common task for developers, system administrators, and data scientists who need to locate every file matching a certain pattern deep inside a directory tree. Still, whether you are cleaning up temporary files, gathering assets for a build process, or scanning logs for analysis, knowing how to wield glob patterns efficiently saves time and reduces errors. This guide explains what globbing is, shows practical examples in Python, Bash, and PowerShell, highlights common mistakes, and offers performance tips so you can reliably retrieve every file you need—no matter how nested the folder structure Took long enough..
Understanding Glob Patterns
A glob is a simple wildcard syntax used by shells and many programming libraries to match file names. The most familiar symbols are:
*– matches any sequence of characters except a directory separator (/on Unix,\on Windows).?– matches exactly one character.[abc]– matches any single character inside the brackets (e.g.,[0-9]for a digit).{foo,bar}– expands to eitherfooorbar(supported in Bash and some libraries).
When you want to glob find all files in subfolders, you need a pattern that descends into directories. In many environments the double‑asterisk ** serves this purpose: it matches zero or more directories recursively. Day to day, not all glob implementations support ** natively, so you sometimes combine it with other tools (like find or os. walk) to achieve true recursion Which is the point..
Using Glob in Python
Python’s glob module (available since Python 3.5) understands the ** pattern when you pass the recursive=True flag. This makes it straightforward to glob find all files in subfolders with just a few lines of code.
Basic Example
import glob
# Find every .txt file in the current directory and all subdirectories
txt_files = glob.glob('**/*.txt', recursive=True)
print(txt_files)
**/*.txttells glob to match any number of directories (**) followed by any name ending in.txt.- Setting
recursive=Trueactivates the recursive behavior; without it,**is treated as a literal asterisk pair.
Filtering with Multiple Extensions
If you need several file types at once, combine patterns with a list comprehension:
patterns = ['**/*.py', '**/*.ipynb', '**/*.md']
all_files = []
for pat in patterns:
all_files.extend(glob.glob(pat, recursive=True))
# Remove duplicates that may appear if a file matches more than one pattern
unique_files = list(set(all_files))
print(f"Found {len(unique_files)} files")
Using pathlib (Modern Alternative)
Starting with Python 3.4, pathlib.Path offers an object‑oriented approach that also supports recursive globbing:
from pathlib import Path
root = Path('.')
py_files = list(root.rglob('*.
`rglob` automatically adds the `**` prefix, making the intent clear: “search recursively for *.py*”.
### Handling Hidden Files
By default, glob does **not** return entries that start with a dot (`.`). To include hidden files, you must explicitly match them:
```python
hidden_logs = glob.glob('**/.*log', recursive=True)
Be cautious when combining hidden‑file patterns with normal ones; you may inadvertently match parent directory entries like ...
Using Glob in Bash
Bash’s built‑in pathname expansion also supports ** when the globstar option is enabled. This lets you glob find all files in subfolders directly from the command line without invoking external utilities No workaround needed..
Enabling globstar
shopt -s globstar # enable ** recursion
After this, patterns like **/*.log work as expected Easy to understand, harder to ignore. Practical, not theoretical..
Practical Examples
-
List all JPEG images
echo **/*.jpg -
Count files
shopt -s nullglob # prevents literal pattern return when no matches files=(**/*.txt) echo "${#files[@]} text files found" -
Perform an action on each match
for f in **/*.bak; do echo "Removing backup: $f" rm -i "$f" done
Combining with Other Tools
Sometimes you need more complex selection criteria (size, modification time, etc.). In those cases, pipe glob results to while read or use find:
# Find files larger than 10 MB
shopt -s globstar nullglob
for f in **/*; do
[[ -f "$f" && $(stat -c%s "$f") -gt 10485760 ]] && echo "$f"
done
Using Glob in PowerShell
PowerShell treats * and ?So as wildcards in many cmdlets, and the -Recurse parameter provides directory traversal. While PowerShell does not use the ** syntax natively, you can achieve the same effect with Get-ChildItem Not complicated — just consistent..
Simple Recursive Search
# Get all .csv files in the current folder and all subfolders
Get-ChildItem -Path . -Filter *.csv -Recurse -File
-Filterlimits the match to the given extension.-Recursetells PowerShell to descend into subdirectories.-Fileensures only files (not directories) are returned.
Using Wildcards in -Include / -Exclude
If you need multiple patterns, -Include accepts an array:
Get-ChildItem -Path . -Recurse -File -Include *.log, *.txt
Advanced Filtering with Where-Object
For conditions beyond name patterns (e.g., file size), pipe to Where-Object:
Get-ChildItem -Path . -Recurse -File |
Where-Object { $_.Length -gt 5MB } |
Select-Object FullName, Length
Common Pitfalls and How to Avoid Them
| Pitfall | Symptom | Solution |
|---|---|---|
Forgetting to enable globstar in Bash |
**/*.So txt |
Run shopt -s globstar before using the pattern. So naturally, txtreturns the literal string**/*. joinorpathlibin Python; in Bash, enablenocaseglobif needed, or use\as separator. or use -not -name ".glob without recursive=True |
| Path separators mismatch on Windows | Patterns with / fail |
Use `os. |
Using glob.path. entries |
Explicitly filter out names starting with .*" in find. Consider this: or.. Plus, |
|
| Matching hidden files unintentionally | Scripts process `. | |
| Performance degradation on huge trees | Script hangs or consumes excessive memory | Limit scope with -maxdepth (find) or process results in streams (generators) rather than building large lists. |
Performance Considerations for Large Codebases
When working with repositories containing tens of thousands of files, the mechanics of pattern expansion become a bottleneck. Day to day, the shell expands globs before executing the command, meaning rm **/*. Now, log passes every single matching path as a distinct argument to rm. This hits the kernel’s ARG_MAX limit and consumes significant memory Worth keeping that in mind..
Streaming with find + xargs
For bulk operations, find combined with xargs (or find -exec ... +) streams results in manageable batches, avoiding argument limits entirely.
# Safe deletion of millions of .tmp files
find . -type f -name '*.tmp' -print0 | xargs -0 rm -f
-print0/-0handle filenames with spaces, newlines, or quotes safely.xargsbatches arguments automatically (typically 128KB chunks).
Parallel Execution
Modern multi-core systems benefit from parallel processing. GNU parallel or xargs -P can use all cores for CPU-bound tasks like compression or linting Most people skip this — try not to. Turns out it matters..
# Compress all .log files using 4 parallel workers
find . -type f -name '*.log' | parallel -j4 gzip --best {}
Python: Generators over Lists
In Python, glob.glob and pathlib.Path.rglob return lists, loading all paths into memory. For massive trees, os.scandir (used internally by pathlib in 3.11+) or glob.iglob (iterator version) keeps memory usage constant (O(1)) Simple, but easy to overlook..
# Memory-efficient iteration (Python 3.5+)
import glob
for filepath in glob.iglob('**/*.log', recursive=True):
process_log(filepath) # Process one at a time
Cross-Platform Portability Strategies
Writing glob logic that behaves identically on Linux, macOS, and Windows requires abstracting filesystem differences.
Normalize Separators Early
Never hardcode / or \. Use the standard library abstractions It's one of those things that adds up..
| Language | Correct Approach |
|---|---|
| Python | `pathlib.js** |
| Node.Join("src", "", "*.So js', { windowsPathsNoEscape: true })` | |
| Go | filepath. Glob(filepath.go")) (requires doublestar lib for **) |
| Rust | `glob::Pattern::new("**/*. |
Case Sensitivity
- Linux/macOS (APFS default): Case-sensitive.
*.txtmissesREADME.TXT. - Windows/macOS (default): Case-insensitive.
Portable Fix: Explicitly define case handling.
- Bash:
shopt -s nocaseglob(makes current shell case-insensitive). - Python:
glob.glob(pattern, recursive=True, case_sensitive=False)(3.10+). - PowerShell: Inherently case-insensitive; use
-CaseSensitiveswitch on operators if strict matching is needed.
Handling Symlinks and Junctions
Recursive globs follow symlinks by default in Python (pathlib) and find -L, but not in Bash ** or PowerShell Get-ChildItem. Infinite loops occur if a symlink points to a parent directory.
Defensive Pattern:
# Bash: Avoid loops, ignore symlinks to directories
shopt -s globstar dotglob
for f in **/*.conf; do
[[ -L "$f" && -d "$f" ]] && continue # Skip symlinked directories
process "$f"
done
Modern Alternatives: fd and ripgrep
For interactive use and CI pipelines, modern Rust-based tools replace find/grep/glob combinations with faster, smarter defaults (respecting .gitignore, hidden files, parallelism) But it adds up..
| Tool | Use Case | Glob Equivalent |
|---|---|---|
fd |
Finding files by name | fd '\.rs
|
