How to get all files under a specific directory in MATLAB?

Retrieving All Files Under a Specific Directory in MATLAB

Accessing and managing files within directories is a fundamental task in MATLAB, whether you're processing data, organizing scripts, or automating workflows. MATLAB provides several methods to list all files within a specific directory, including its subdirectories. This guide will walk you through the most efficient and optimized ways to achieve this, catering to different MATLAB versions and use cases.

Table of Contents

1. Using dir with Wildcards (**)
2. Filtering Out Directories
3. Full File Paths
4. Handling Specific File Types
5. Recursive Function for Older MATLAB Versions
6. Example: Comprehensive Script
7. Best Practices
8. Additional Tips

Using dir with Wildcards (**)

Starting from MATLAB R2016b, the dir function supports the ** wildcard, which allows for recursive searching through directories and their subdirectories. This feature simplifies the process of listing all files within a directory hierarchy.

Syntax

files = dir(fullfile(directoryPath, '**', '*'));

• directoryPath: The path to the target directory.
• '**': Represents all subdirectories recursively.
• '*': Matches all files and folders.

Example

% Define the target directory
directoryPath = '/path/to/your/directory';

% Retrieve all files and folders recursively
files = dir(fullfile(directoryPath, '**', '*'));

Filtering Out Directories

The dir function returns both files and directories. To obtain only files, you need to filter out the directories from the results.

Method

Use logical indexing to exclude entries where the isdir field is true.

Example

% Retrieve all files and folders recursively
files = dir(fullfile(directoryPath, '**', '*'));

% Exclude directories
files = files(~[files.isdir]);

• [files.isdir]: Creates a logical array indicating which entries are directories.
• ~[files.isdir]: Logical NOT to select only files.

Full File Paths

By default, dir returns file names without their full paths. To work with the files effectively, especially when dealing with multiple directories, it's often necessary to obtain their full paths.

Method

Combine the folder and name fields of the dir output using fullfile.

Example

% Retrieve all files and folders recursively
files = dir(fullfile(directoryPath, '**', '*'));

% Exclude directories
files = files(~[files.isdir]);

% Get full file paths
fullFilePaths = fullfile({files.folder}, {files.name});

• {files.folder}: Cell array of folder paths.
• {files.name}: Cell array of file names.
• fullfile: Concatenates folder and file names into full paths.

Handling Specific File Types

Often, you might want to retrieve only files of specific types (e.g., .txt, .mat, .csv). You can adjust the wildcard pattern to match desired file extensions.

Syntax

files = dir(fullfile(directoryPath, '**', '*.txt'));  % For text files

Example

% Define the target directory and file extension
directoryPath = '/path/to/your/directory';
fileExtension = '*.csv';  % Change to desired extension

% Retrieve all specified files recursively
files = dir(fullfile(directoryPath, '**', fileExtension));

% Exclude directories (optional, as '*.csv' should match files)
files = files(~[files.isdir]);

% Get full file paths
fullFilePaths = fullfile({files.folder}, {files.name});

Recursive Function for Older MATLAB Versions

If you're using a MATLAB version prior to R2016b, the dir function does not support the ** wildcard. In such cases, you can create a recursive function to traverse directories and collect file information.

Example Recursive Function

function fileList = getAllFiles(directoryPath)
    % Initialize an empty array to store file information
    fileList = [];
    
    % Get list of all items in the current directory
    items = dir(directoryPath);
    
    % Exclude '.' and '..' directories
    items = items(~ismember({items.name}, {'.', '..'}));
    
    for i = 1:length(items)
        fullPath = fullfile(directoryPath, items(i).name);
        if items(i).isdir
            % Recursively call the function for subdirectories
            fileList = [fileList; getAllFiles(fullPath)];
        else
            % Append the file information
            fileList = [fileList; items(i)];
        end
    end
end

Usage

% Define the target directory
directoryPath = '/path/to/your/directory';

% Retrieve all files recursively
files = getAllFiles(directoryPath);

% Get full file paths
fullFilePaths = fullfile({files.folder}, {files.name});

Notes

• Performance: Recursive functions can be slower for large directory trees. Preallocating arrays or using more efficient data structures can help improve performance.
• Stack Overflow: Deeply nested directories may lead to stack overflow. Ensure your directory structure is manageable or consider iterative approaches.

Example: Comprehensive Script

Below is a comprehensive script that combines the above methods to retrieve all files under a specific directory, excluding directories, and obtaining their full paths. This script is compatible with MATLAB R2016b and later versions.

% Define the target directory
directoryPath = '/path/to/your/directory';

% Retrieve all files and folders recursively
files = dir(fullfile(directoryPath, '**', '*'));

% Exclude directories
files = files(~[files.isdir]);

% Get full file paths
fullFilePaths = fullfile({files.folder}, {files.name});

% Display the list of files
disp(fullFilePaths');

Explanation

1. Retrieve All Entries: Uses dir with ** to list all files and folders recursively.
2. Filter Out Directories: Excludes directory entries to keep only files.
3. Construct Full Paths: Combines folder paths with file names to get full file paths.
4. Display Files: Prints the list of full file paths.

Output

/path/to/your/directory/file1.txt
/path/to/your/directory/subdir/file2.csv
/path/to/your/directory/subdir/nested/file3.mat
...

Best Practices

1. Use Absolute Paths: When possible, use absolute paths to avoid ambiguity, especially when dealing with multiple directories.

2. Handle Permissions: Ensure that your MATLAB session has the necessary permissions to read the directories and files.

3. Error Handling: Incorporate error handling to manage scenarios where directories may not exist or access is denied.

   if ~isfolder(directoryPath)
       error('The specified directory does not exist.');
   end

4. Optimize Performance: For very large directories, consider processing files in batches or using parallel processing techniques to speed up operations.

5. Avoid Using eval or Dynamic Field Names: Keep your code clean and maintainable by avoiding complex dynamic evaluations when handling file paths and names.

Additional Tips

• Using genpath: While genpath generates a path string with all subdirectories, it does not provide file information. It can be used in combination with other functions if needed.

  pathStr = genpath(directoryPath);
  pathCells = strsplit(pathStr, pathsep);

• Sorting Files: You can sort the files array based on different criteria, such as name, date, or size.

  % Sort files by name
  [~, idx] = sort({files.name});
  sortedFiles = files(idx);

• Using arrayfun for Processing: Apply functions to each file efficiently using arrayfun.

  fileSizes = arrayfun(@(x) x.bytes, files);

• Visualizing Directory Structure: For better understanding, visualize the directory structure using recursive traversal and plotting tools.

Conclusion

Retrieving all files under a specific directory and its subdirectories in MATLAB can be accomplished efficiently using built-in functions like dir with the ** wildcard for recursive searches (MATLAB R2016b and later). For older versions, a custom recursive function serves as an effective alternative. Filtering out directories, obtaining full file paths, and handling specific file types are straightforward once the files are listed.

By leveraging these methods, you can automate file management tasks, streamline data processing workflows, and enhance the robustness of your MATLAB applications.

Happy Coding!