Delta: DeltaTable - Files Used List

[SriramAvatar]

Is there a way to find the list of files actively used by a delta table? The delta table states that it is using 20k files but we find almost 13 million files in the storage account. To check whats going on and seperate the files that arent used by delta anymore, do we have a command?

The vacuum command is already running with the defailt 7days. What is the best way to clean the unused files?

[Flexron]

use the generate method of the DeltaTable class this will generate a manifest that contain all parquet files path used. May be we should try to delete manualy all other parquet not listed by the manifest.

[SriramAvatar]

use the generate method of the DeltaTable class this will generate a manifest that contain all parquet files path used. May be we should try to delete manualy all other parquet not listed by the manifest.

@Flexron , thanks for the reply. Apologies that I haven't mentioned I am on Azure Databricks. I have tried that one but it isn't supported in Azure databricks. Error: "Generation of manifests for Delta table is currently only supported on AWS S3." So I think I cannot use this option.

[Flexron]

I'm using Azure Databricks too. You Can disable the fileSystem check by set in your sparkSession conf spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled to false.

[SriramAvatar]

I'm using Azure Databricks too. You Can disable the fileSystem check by set in your sparkSession conf spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled to false.

Thanks @Flexron , after changing this setting, am able to generate the manifest. But when I compare the count of files from the manifest and the one displayed in the interface, they are not matching. I tried to run an optimize, vacuum with 0 hours retention and then generate and compare the counts. Still the counts don't match. I have a feeling the manifest generation is also not giving the latest set of files being used. Kindly let me know if I am missing someting.

The following is the code used to generate and get the file list.

# Set the configurataion for AzureDatabricks
spark.conf.set("spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled", False)

# Generate symlink_format_manifest
from delta import *
delta_table = DeltaTable.forPath(spark, "/mnt/datalake/pathofthedeltalocation")
delta_table.generate("symlink_format_manifest")

generatedManifestPath = "/mnt/datalake/pathofthedeltalocation/_symlink_format_manifest/"
tempManifestFilesOnlyPath = "/mnt/datalake/temp/"

#List the manifest Files
def get_dir_content(ls_path):
  dir_paths = dbutils.fs.ls(ls_path)
  subdir_paths = [get_dir_content(p.path) for p in dir_paths if p.isDir() and p.path != ls_path]
  flat_subdir_paths = [p for subdir in subdir_paths for p in subdir]
  return list(map(lambda p: p.path, dir_paths)) + flat_subdir_paths

#Exclude empty folders and copy the manifest files to temp path
paths = get_dir_content(generatedManifestPath)
i=1
for path in paths:
  if "/manifest" in  path:
    dbutils.fs.cp(path, tempManifestFilesOnlyPath + str(i) + ".csv")
    i = i + 1

#Read the Manifest Files and give the count
df = spark.read.csv(tempManifestFilesOnlyPath)
df.count()

I get the count as 4700 while the interface shows 935. Thanks in advance