Deleting files from both on-premises and cloud storage is a common task in data integration scenarios. In Azure Data Factory, the delete activity provides a convenient way to delete files and folders. In this article, we will explore the various options available for file deletion using the delete activity.

Delete Files from a Folder

The delete activity allows us to delete all files from a specific folder. By providing a dataset that points to the folder, we can easily delete all the files within it. Additionally, we can specify options such as wildcard file names or filter files by their last modified date.

Delete Contents in a Folder and the Folder Itself

If we want to delete not only the files within a folder but also the folder itself, we can enable the “recursively” option in the delete activity. This ensures that the folder and its contents are completely removed.

Delete Specific File Types in a Folder

In some cases, we may only want to delete files of a specific type within a folder. By using wildcard file names, we can easily specify the file types we want to delete. For example, we can delete all text files within a folder by using the “*.txt” wildcard.

Delete a Single File Using a Wildcard File Name

If we want to delete a single file using a wildcard file name, we can specify the file name in the wildcard file path. This allows us to delete a specific file while keeping the rest of the files intact.

Filter Files Using the Last Modified Date

The delete activity also provides an option to filter files based on their last modified date. By specifying a start date and an end date, we can delete files that fall within the specified date range. This is useful when we want to delete files that are no longer needed or have expired.

Delete a Set of Files in a Folder

If we have a specific set of files that we want to delete, we can create a list of filenames and provide it to the delete activity. This allows us to delete multiple files at once, making the process more efficient and convenient.

Deleting from Subfolders

In addition to deleting files from a folder, the delete activity can also delete files from subfolders. By enabling the “recursively” option, we can ensure that files within subfolders are also deleted. This is useful when we want to clean up an entire directory structure.

Logging in Delete Activity

The delete activity provides an option to log deleted files or folders. By enabling logging and specifying a linked service, we can store the log file in Azure Storage, Azure Data Lake Storage Gen1, or Azure Data Lake Storage Gen2. The log file contains information such as the name, category, status, and error message (if any) of the deleted files or folders.

Conclusion

The delete activity in Azure Data Factory is a powerful tool for deleting files and folders from both on-premises and cloud storage. By understanding the various options available, we can efficiently manage our data and keep our storage locations clean. It is important to note that once a file is deleted, it cannot be recovered unless soft delete is enabled at the storage location.