Scala read files from directory.

Scala read files from directory Source. In case my Spark shut down and starts after some time, I would like to read the new files in the directory. I am using Scala to do achieve this task. Is there a way to read parquet files from dir1_2 and dir2_1 without using unionAll or is there any fancy way using I use fileStream to read files in the hdfs directory from Spark (streaming context). On my PC I read the directory using - val directory = Play. spark. The data is in S3 and I am trying to do this: val rdd =sc. txt) and picked up by PySpark code in subsequent stages. csv 2016-03-04 file2. Making statements based on opinion; back them up with Databricks - Read CSV file from folder Ask Question Asked 2 years, 6 months ago Modified 1 year ago Viewed 11k times Part of Microsoft Azure Collective Though Spark supports to read from/write to files on multiple file systems like Amazon S3, Hadoop HDFS, Azure, GCP e. ycbom ttl vtd pskk lnavr zota yvwd dspro qrtnh jrpasy mqwvrq xdbeeo yeb uumy ysiqjx