Pushing JSON files to elasticsearch using Python Script: Works but does not show up in Kibana
0
Background Information:
Hi,
I've been trying to push JSON files into Elasticsearch using a custom script, the script runs and finishes - however, when I check Elasticsearch and Kibana the index exists but none of the docs that I'm sending are uploaded. I've done this a countless number of times. Not sure where I am going wrong since this has worked before -
Code
#Change the IP as desiered
es = Elasticsearch(
['localhost'],
timeout=420000,
port=9200
)
print ("Start")
#Append to this list if all files from a folder have been uploaded
done_list =
filenames = os.listdir("/home/directory/file_json")
print(filenames)
for current_folder in filenames:
for f in glob.glob("/home/directory/file_json/"+current_folder+"/*.json"):
print("File Open")
MyFile = open(f).read()
print("Bulking - "+f)
es.bulk(MyFile,"test","_doc")
print("done elasticsearch")
done_list.append(current_folder)
print(current_folder + " done")
print ("Finished")
If anyone could let me know if this has some sort of error or maybe something I'm missing that would be greatly appreciated.
python json file elasticsearch kibana
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
add a comment |
0
Background Information:
Hi,
I've been trying to push JSON files into Elasticsearch using a custom script, the script runs and finishes - however, when I check Elasticsearch and Kibana the index exists but none of the docs that I'm sending are uploaded. I've done this a countless number of times. Not sure where I am going wrong since this has worked before -
Code
#Change the IP as desiered
es = Elasticsearch(
['localhost'],
timeout=420000,
port=9200
)
print ("Start")
#Append to this list if all files from a folder have been uploaded
done_list =
filenames = os.listdir("/home/directory/file_json")
print(filenames)
for current_folder in filenames:
for f in glob.glob("/home/directory/file_json/"+current_folder+"/*.json"):
print("File Open")
MyFile = open(f).read()
print("Bulking - "+f)
es.bulk(MyFile,"test","_doc")
print("done elasticsearch")
done_list.append(current_folder)
print(current_folder + " done")
print ("Finished")
If anyone could let me know if this has some sort of error or maybe something I'm missing that would be greatly appreciated.
python json file elasticsearch kibana
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
1
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value ofes.bulk, or run the script in a debugger and single-step intoes.bulk(and the functions it calls) to inspect every value returned by an api call.
– alexis
Nov 14 '18 at 17:06
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.es.bulk(listOfJsonLines, "indexname", "mapping")
– BarathVutukuri
Nov 14 '18 at 17:09
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00
add a comment |
0
0
0
Background Information:
Hi,
I've been trying to push JSON files into Elasticsearch using a custom script, the script runs and finishes - however, when I check Elasticsearch and Kibana the index exists but none of the docs that I'm sending are uploaded. I've done this a countless number of times. Not sure where I am going wrong since this has worked before -
Code
#Change the IP as desiered
es = Elasticsearch(
['localhost'],
timeout=420000,
port=9200
)
print ("Start")
#Append to this list if all files from a folder have been uploaded
done_list =
filenames = os.listdir("/home/directory/file_json")
print(filenames)
for current_folder in filenames:
for f in glob.glob("/home/directory/file_json/"+current_folder+"/*.json"):
print("File Open")
MyFile = open(f).read()
print("Bulking - "+f)
es.bulk(MyFile,"test","_doc")
print("done elasticsearch")
done_list.append(current_folder)
print(current_folder + " done")
print ("Finished")
If anyone could let me know if this has some sort of error or maybe something I'm missing that would be greatly appreciated.
python json file elasticsearch kibana
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
Background Information:
Hi,
I've been trying to push JSON files into Elasticsearch using a custom script, the script runs and finishes - however, when I check Elasticsearch and Kibana the index exists but none of the docs that I'm sending are uploaded. I've done this a countless number of times. Not sure where I am going wrong since this has worked before -
Code
#Change the IP as desiered
es = Elasticsearch(
['localhost'],
timeout=420000,
port=9200
)
print ("Start")
#Append to this list if all files from a folder have been uploaded
done_list =
filenames = os.listdir("/home/directory/file_json")
print(filenames)
for current_folder in filenames:
for f in glob.glob("/home/directory/file_json/"+current_folder+"/*.json"):
print("File Open")
MyFile = open(f).read()
print("Bulking - "+f)
es.bulk(MyFile,"test","_doc")
print("done elasticsearch")
done_list.append(current_folder)
print(current_folder + " done")
print ("Finished")
If anyone could let me know if this has some sort of error or maybe something I'm missing that would be greatly appreciated.
python json file elasticsearch kibana
python json file elasticsearch kibana
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
asked Nov 14 '18 at 17:03
Jerin RaisaJerin Raisa
154
154
1
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value ofes.bulk, or run the script in a debugger and single-step intoes.bulk(and the functions it calls) to inspect every value returned by an api call.
– alexis
Nov 14 '18 at 17:06
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.es.bulk(listOfJsonLines, "indexname", "mapping")
– BarathVutukuri
Nov 14 '18 at 17:09
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00
add a comment |
1
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value ofes.bulk, or run the script in a debugger and single-step intoes.bulk(and the functions it calls) to inspect every value returned by an api call.
– alexis
Nov 14 '18 at 17:06
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.es.bulk(listOfJsonLines, "indexname", "mapping")
– BarathVutukuri
Nov 14 '18 at 17:09
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00
1
1
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value of
es.bulk, or run the script in a debugger and single-step into es.bulk (and the functions it calls) to inspect every value returned by an api call.– alexis
Nov 14 '18 at 17:06
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value of
es.bulk, or run the script in a debugger and single-step into es.bulk (and the functions it calls) to inspect every value returned by an api call.– alexis
Nov 14 '18 at 17:06
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.
es.bulk(listOfJsonLines, "indexname", "mapping")– BarathVutukuri
Nov 14 '18 at 17:09
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.
es.bulk(listOfJsonLines, "indexname", "mapping")– BarathVutukuri
Nov 14 '18 at 17:09
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53305342%2fpushing-json-files-to-elasticsearch-using-python-script-works-but-does-not-show%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53305342%2fpushing-json-files-to-elasticsearch-using-python-script-works-but-does-not-show%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I don't use this api but elasticsearch probably returns a status message. Grab and inspect the return value of
es.bulk, or run the script in a debugger and single-step intoes.bulk(and the functions it calls) to inspect every value returned by an api call.– alexis
Nov 14 '18 at 17:06
First of all, each line of your file should be a json. You have to iterate the file after opening and add all the lines into a list. Push that list instead of file in the bulk API.
es.bulk(listOfJsonLines, "indexname", "mapping")– BarathVutukuri
Nov 14 '18 at 17:09
I'm pretty new to working with Python, so I'm a bit unsure how to do this? I've seen a couple solutions but I haven't been able to apply it to my problem, also thank you for the suggestions!!
– Jerin Raisa
Dec 10 '18 at 20:00