Spark spark.sql.session.timeZone doesn't work with JSON source
up vote
2
down vote
favorite
Does Spark v2.3.1 depends on local timezone when reading from JSON file?
My src/test/resources/data/tmp.json:
[
"timestamp": "1970-01-01 00:00:00.000"
]
and Spark code:
SparkSession.builder()
.appName("test")
.master("local")
.config("spark.sql.session.timeZone", "UTC")
.getOrCreate()
.read()
.option("multiLine", true).option("mode", "PERMISSIVE")
.schema(new StructType()
.add(new StructField("timestamp", DataTypes.TimestampType, true, Metadata.empty())))
.json("src/test/resources/data/tmp.json")
.show();
Result:
+-------------------+
| timestamp|
+-------------------+
|1969-12-31 22:00:00|
+-------------------+
How to make spark return 1970-01-01 00:00:00.000?
P.S. This question is not a duplicate of Spark Strutured Streaming automatically converts timestamp to local time, because provided there solution not work for me and is already included (see .config("spark.sql.session.timeZone", "UTC")) into my question.
apache-spark apache-spark-sql apache-spark-2.3
edited Nov 11 at 20:55
Community♦
11
asked Nov 11 at 17:44
VB_
18.5k2279159
|
show 1 more comment
up vote
2
down vote
favorite
Does Spark v2.3.1 depends on local timezone when reading from JSON file?
My src/test/resources/data/tmp.json:
[
"timestamp": "1970-01-01 00:00:00.000"
]
and Spark code:
SparkSession.builder()
.appName("test")
.master("local")
.config("spark.sql.session.timeZone", "UTC")
.getOrCreate()
.read()
.option("multiLine", true).option("mode", "PERMISSIVE")
.schema(new StructType()
.add(new StructField("timestamp", DataTypes.TimestampType, true, Metadata.empty())))
.json("src/test/resources/data/tmp.json")
.show();
Result:
+-------------------+
| timestamp|
+-------------------+
|1969-12-31 22:00:00|
+-------------------+
How to make spark return 1970-01-01 00:00:00.000?
P.S. This question is not a duplicate of Spark Strutured Streaming automatically converts timestamp to local time, because provided there solution not work for me and is already included (see .config("spark.sql.session.timeZone", "UTC")) into my question.
apache-spark apache-spark-sql apache-spark-2.3
edited Nov 11 at 20:55
Community♦
11
asked Nov 11 at 17:44
VB_
18.5k2279159
Could you confirm thatorg.apache.spark.sql.RuntimeConfigactually reflects the configuration? Does the other solution fix the problem?
– user6910411
Nov 11 at 18:57
Actually nevermind. Settinguser.timezonedoes solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.
– user6910411
Nov 11 at 19:31
1
@user6910411 solved withTimeZone.setDefault(TimeZone.getTimeZone("UTC"))
– VB_
Nov 11 at 21:13
1
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51
|
show 1 more comment
up vote
2
down vote
favorite
up vote
2
down vote
favorite
Does Spark v2.3.1 depends on local timezone when reading from JSON file?
My src/test/resources/data/tmp.json:
[
"timestamp": "1970-01-01 00:00:00.000"
]
and Spark code:
SparkSession.builder()
.appName("test")
.master("local")
.config("spark.sql.session.timeZone", "UTC")
.getOrCreate()
.read()
.option("multiLine", true).option("mode", "PERMISSIVE")
.schema(new StructType()
.add(new StructField("timestamp", DataTypes.TimestampType, true, Metadata.empty())))
.json("src/test/resources/data/tmp.json")
.show();
Result:
+-------------------+
| timestamp|
+-------------------+
|1969-12-31 22:00:00|
+-------------------+
How to make spark return 1970-01-01 00:00:00.000?
P.S. This question is not a duplicate of Spark Strutured Streaming automatically converts timestamp to local time, because provided there solution not work for me and is already included (see .config("spark.sql.session.timeZone", "UTC")) into my question.
apache-spark apache-spark-sql apache-spark-2.3
edited Nov 11 at 20:55
Community♦
11
asked Nov 11 at 17:44
VB_
18.5k2279159
Does Spark v2.3.1 depends on local timezone when reading from JSON file?
My src/test/resources/data/tmp.json:
[
"timestamp": "1970-01-01 00:00:00.000"
]
and Spark code:
SparkSession.builder()
.appName("test")
.master("local")
.config("spark.sql.session.timeZone", "UTC")
.getOrCreate()
.read()
.option("multiLine", true).option("mode", "PERMISSIVE")
.schema(new StructType()
.add(new StructField("timestamp", DataTypes.TimestampType, true, Metadata.empty())))
.json("src/test/resources/data/tmp.json")
.show();
Result:
+-------------------+
| timestamp|
+-------------------+
|1969-12-31 22:00:00|
+-------------------+
How to make spark return 1970-01-01 00:00:00.000?
P.S. This question is not a duplicate of Spark Strutured Streaming automatically converts timestamp to local time, because provided there solution not work for me and is already included (see .config("spark.sql.session.timeZone", "UTC")) into my question.
apache-spark apache-spark-sql apache-spark-2.3
apache-spark apache-spark-sql apache-spark-2.3
edited Nov 11 at 20:55
Community♦
11
asked Nov 11 at 17:44
VB_
18.5k2279159
edited Nov 11 at 20:55
Community♦
11
asked Nov 11 at 17:44
VB_
18.5k2279159
edited Nov 11 at 20:55
Community♦
11
edited Nov 11 at 20:55
Community♦
11
edited Nov 11 at 20:55
Community♦
11
11
asked Nov 11 at 17:44
VB_
18.5k2279159
asked Nov 11 at 17:44
VB_
18.5k2279159
asked Nov 11 at 17:44
VB_
18.5k2279159
18.5k2279159
Could you confirm thatorg.apache.spark.sql.RuntimeConfigactually reflects the configuration? Does the other solution fix the problem?
– user6910411
Nov 11 at 18:57
Actually nevermind. Settinguser.timezonedoes solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.
– user6910411
Nov 11 at 19:31
1
@user6910411 solved withTimeZone.setDefault(TimeZone.getTimeZone("UTC"))
– VB_
Nov 11 at 21:13
1
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51
|
show 1 more comment
Could you confirm thatorg.apache.spark.sql.RuntimeConfigactually reflects the configuration? Does the other solution fix the problem?
– user6910411
Nov 11 at 18:57
Actually nevermind. Settinguser.timezonedoes solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.
– user6910411
Nov 11 at 19:31
1
@user6910411 solved withTimeZone.setDefault(TimeZone.getTimeZone("UTC"))
– VB_
Nov 11 at 21:13
1
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51
Could you confirm that
org.apache.spark.sql.RuntimeConfig actually reflects the configuration? Does the other solution fix the problem?– user6910411
Nov 11 at 18:57
Could you confirm that
org.apache.spark.sql.RuntimeConfig actually reflects the configuration? Does the other solution fix the problem?– user6910411
Nov 11 at 18:57
Actually nevermind. Setting
user.timezone does solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.– user6910411
Nov 11 at 19:31
Actually nevermind. Setting
user.timezone does solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.– user6910411
Nov 11 at 19:31
1
1
@user6910411 solved with
TimeZone.setDefault(TimeZone.getTimeZone("UTC"))– VB_
Nov 11 at 21:13
@user6910411 solved with
TimeZone.setDefault(TimeZone.getTimeZone("UTC"))– VB_
Nov 11 at 21:13
1
1
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51
|
show 1 more comment
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251473%2fspark-spark-sql-session-timezone-doesnt-work-with-json-source%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Could you confirm that
org.apache.spark.sql.RuntimeConfigactually reflects the configuration? Does the other solution fix the problem?– user6910411
Nov 11 at 18:57
Actually nevermind. Setting
user.timezonedoes solve the problem. As of this it qualifies for JIRA ticket, if there isn't one already. The behavior is not only confusing, but also inconsistent between different data sources.– user6910411
Nov 11 at 19:31
1
@user6910411 solved with
TimeZone.setDefault(TimeZone.getTimeZone("UTC"))– VB_
Nov 11 at 21:13
1
That however, wont' help you in a distributed mode.
– user6910411
Nov 12 at 11:14
@user6910411 yeah, good point
– VB_
Nov 12 at 11:51