More than 5 years have passed since last update.

sparkでhdfs上のdataを削除するには

Spark

Posted at 2016-08-18

def deleteFile(filepath: String, uri: String = "hdfs://localhost:9000") {

    val hadoopConf = new org.apache.hadoop.conf.Configuration()
    val hdfs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI(uri), hadoopConf)
    try { 
        hdfs.delete(new org.apache.hadoop.fs.Path(filepath), true) 
    } catch { 
      case e : Throwable => e.printStackTrace
    }
}

みたいな感じで、

dateframeを使えば、overwrite optionでsaveできる

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up