Scrapy view and requests.get difference for “https://www.mywebsite.com.sg”
I am trying to scrape https://www.mywebsite.com.sg but the following command returns 400 bad request error:
scrapy view https://www.mywebsite.com.sg
If I use:
data=requests.get("https://www.mywebsite.com.sg")
I can get the content of the webpage in data.text and data.content.
However all the xpath operations in my script dotn't work as data.xpath and data.content are both empty.
There seems to be no protection in the webpage as postman software can get result with a simple HTTP GET query.
How do I get the response object to be properly filled?
scrapy python-requests response
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
add a comment |
I am trying to scrape https://www.mywebsite.com.sg but the following command returns 400 bad request error:
scrapy view https://www.mywebsite.com.sg
If I use:
data=requests.get("https://www.mywebsite.com.sg")
I can get the content of the webpage in data.text and data.content.
However all the xpath operations in my script dotn't work as data.xpath and data.content are both empty.
There seems to be no protection in the webpage as postman software can get result with a simple HTTP GET query.
How do I get the response object to be properly filled?
scrapy python-requests response
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
1
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24
add a comment |
I am trying to scrape https://www.mywebsite.com.sg but the following command returns 400 bad request error:
scrapy view https://www.mywebsite.com.sg
If I use:
data=requests.get("https://www.mywebsite.com.sg")
I can get the content of the webpage in data.text and data.content.
However all the xpath operations in my script dotn't work as data.xpath and data.content are both empty.
There seems to be no protection in the webpage as postman software can get result with a simple HTTP GET query.
How do I get the response object to be properly filled?
scrapy python-requests response
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
I am trying to scrape https://www.mywebsite.com.sg but the following command returns 400 bad request error:
scrapy view https://www.mywebsite.com.sg
If I use:
data=requests.get("https://www.mywebsite.com.sg")
I can get the content of the webpage in data.text and data.content.
However all the xpath operations in my script dotn't work as data.xpath and data.content are both empty.
There seems to be no protection in the webpage as postman software can get result with a simple HTTP GET query.
How do I get the response object to be properly filled?
scrapy python-requests response
scrapy python-requests response
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
edited Nov 14 '18 at 8:02
Guillaume
1,1531724
1,1531724
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
asked Nov 13 '18 at 15:23
Patrick ChanPatrick Chan
111
111
1
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24
add a comment |
1
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24
1
1
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53284213%2fscrapy-view-and-requests-get-difference-for-https-www-mywebsite-com-sg%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53284213%2fscrapy-view-and-requests-get-difference-for-https-www-mywebsite-com-sg%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Scraping sites is an art, not a science. Unfortunately, no one is going to investigate tghis for you and give you a solution. Your best bet is to research further and come back when you have a specific question about why this site is resisting being scraped. Always provide a Minimal, Complete, and Verifiable example if you do, along with some sample output.
– jdv
Nov 13 '18 at 15:34
Maybe you should tell us which URL you are trying to scrape?
– Guillaume
Nov 14 '18 at 3:03
Thanks I am trying to scrape watsons.com.sg
– Patrick Chan
Nov 15 '18 at 14:23
Just can't figure out why request is ok but not scrapy view.
– Patrick Chan
Nov 15 '18 at 14:24