Dear Rajesh,

Hope you are doing well.

For csv file we need databricks jar to be registered or if you don't want to use this jar then read it as a normal textfile.

For writing a json file

dataset:

kamini,100
chichu,200
kartik,300

Code:

val a =sc.textFile("file:///home/edureka/Desktop/sample").toDF
a.registerTempTable("sampletable")
val data = sqlContext.sql("select * from sampletable").toDF
data.write.json("file:///home/edureka/Desktop/myoutput1112")

Ouput:

image


Reading a CSV file:

dataset:

kamini,100
chichu,200
kartik,300

Code:

val df= sc.textFile("file:///home/edureka/Desktop/sample").toDF
df.write.format("com.databricks.spark.csv").save("file:///home/edureka/Desktop/mycsvoutput.csv")

Output:


image



Please note if you are not happy with the response on this ticket, please escalate it to escalations@edureka.in.
We assure you that we will get back to you within 24 hours
 




Regards,
Kamini at Edureka
edureka! Solution Team
On Tue, 16 May at 8:37 AM , Rk.k1 <rk.k1@outlook.com> wrote:
Hi Kamini,

Thanks you for the timely response. However as part of the original request, I would like to know to know How to save the JSON ,AVRO ,Parquet and CSV files in SPARK.

And also please advise , can't we read the CSV file without data brick packages ?

Regards,
Rajesh..





From: Hadoop at Edureka <hadoop@edureka.in>
Sent: Monday, May 15, 2017 11:49 AM
To: rk.k1@outlook.com
Subject: Re: [#370803] How to load & Save the JSON ,AVRO ,Parquet and CSV files in SPARK shell.
 



Dear Learner,

Hope you are doing well.

Please check the below details.

For reading a CSV file:

==> Open the terminal and type spark-shell --packages com.databricks:spark-csv_2.10:1.5.0. Now when you get the scala shell fire the below code for reading the file.

val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("file:///home/edureka/Desktop/StockPrices.csv")

image

image

 
image


Spark json and avro:

My json file:

{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}
{"name":"Michael","address":{"city":"America","state":"California"}}
My code:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val people = sqlContext.jsonFile("file:///home/edureka/Desktop/file")
people.registerTempTable("people")
people.show()
people.printSchema()
val b=sqlContext.sql("Select * from people")
 b.collect.foreach(println)


Screen Shot of the output:






Please try once on your side and let us know if you face any issue.

We are eagerly waiting for your response.


Please note if you are not happy with the response on this ticket, please escalate it to escalations@edureka.in.
We assure you that we will get back to you within 24 hours
 



Regards,
Kamini at Edureka
edureka! Solution Team
On Mon, 15 May at 4:01 PM , Rk.k1 <rk.k1@outlook.com> wrote:
How to load/read & Save the JSON ,AVRO ,Parquet and CSV files in SPARK shell. Sample code snippets will here.


Please let us know your opinion on our support experience.




Awesome Just Okay Not Good
370803:41453