Write to S3 parquet with Impala
I would like to write an entire table to s3a in parquet format.
Let's call the table abc_schem.thattable. I would like to use an Impala query to
SELECT * WHERE to_date(create_time) = 'YYYY-MM-DD'
What is the exact syntax for this to write to Parquet S3?
hadoop amazon-s3 hive impala
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
add a comment |
I would like to write an entire table to s3a in parquet format.
Let's call the table abc_schem.thattable. I would like to use an Impala query to
SELECT * WHERE to_date(create_time) = 'YYYY-MM-DD'
What is the exact syntax for this to write to Parquet S3?
hadoop amazon-s3 hive impala
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
add a comment |
I would like to write an entire table to s3a in parquet format.
Let's call the table abc_schem.thattable. I would like to use an Impala query to
SELECT * WHERE to_date(create_time) = 'YYYY-MM-DD'
What is the exact syntax for this to write to Parquet S3?
hadoop amazon-s3 hive impala
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
I would like to write an entire table to s3a in parquet format.
Let's call the table abc_schem.thattable. I would like to use an Impala query to
SELECT * WHERE to_date(create_time) = 'YYYY-MM-DD'
What is the exact syntax for this to write to Parquet S3?
hadoop amazon-s3 hive impala
hadoop amazon-s3 hive impala
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
edited Nov 15 '18 at 2:17
cricket_007
82.6k1144112
82.6k1144112
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
asked Nov 15 '18 at 0:21
PR102012PR102012
2421319
2421319
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
You can create an external table in a specific location and insert into it assuming s3 system is already configured
CREATE EXTERNAL TABLE abc_schem.thattable(
...
)
STORED AS PARQUET
LOCATION 's3a://bucket/path';
Then use some LOAD DATA or INSERT INTO... SELECT... FROM commands to get data there
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
add a comment |
CREATE TABLE schema.temp_c
STORED AS PARQUET LOCATION "s3a://s3highlevel/c/lowlevel" AS
SELECT * FROM schema.table
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53310733%2fwrite-to-s3-parquet-with-impala%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can create an external table in a specific location and insert into it assuming s3 system is already configured
CREATE EXTERNAL TABLE abc_schem.thattable(
...
)
STORED AS PARQUET
LOCATION 's3a://bucket/path';
Then use some LOAD DATA or INSERT INTO... SELECT... FROM commands to get data there
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
add a comment |
You can create an external table in a specific location and insert into it assuming s3 system is already configured
CREATE EXTERNAL TABLE abc_schem.thattable(
...
)
STORED AS PARQUET
LOCATION 's3a://bucket/path';
Then use some LOAD DATA or INSERT INTO... SELECT... FROM commands to get data there
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
add a comment |
You can create an external table in a specific location and insert into it assuming s3 system is already configured
CREATE EXTERNAL TABLE abc_schem.thattable(
...
)
STORED AS PARQUET
LOCATION 's3a://bucket/path';
Then use some LOAD DATA or INSERT INTO... SELECT... FROM commands to get data there
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
You can create an external table in a specific location and insert into it assuming s3 system is already configured
CREATE EXTERNAL TABLE abc_schem.thattable(
...
)
STORED AS PARQUET
LOCATION 's3a://bucket/path';
Then use some LOAD DATA or INSERT INTO... SELECT... FROM commands to get data there
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
answered Nov 15 '18 at 2:17
cricket_007cricket_007
82.6k1144112
82.6k1144112
add a comment |
add a comment |
CREATE TABLE schema.temp_c
STORED AS PARQUET LOCATION "s3a://s3highlevel/c/lowlevel" AS
SELECT * FROM schema.table
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
add a comment |
CREATE TABLE schema.temp_c
STORED AS PARQUET LOCATION "s3a://s3highlevel/c/lowlevel" AS
SELECT * FROM schema.table
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
add a comment |
CREATE TABLE schema.temp_c
STORED AS PARQUET LOCATION "s3a://s3highlevel/c/lowlevel" AS
SELECT * FROM schema.table
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
CREATE TABLE schema.temp_c
STORED AS PARQUET LOCATION "s3a://s3highlevel/c/lowlevel" AS
SELECT * FROM schema.table
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
answered Nov 16 '18 at 17:09
PR102012PR102012
2421319
2421319
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53310733%2fwrite-to-s3-parquet-with-impala%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
