Select all tag (and extract text) between two other tags with BeatifulSoup









up vote
2
down vote

favorite












I would like to extract all instances of a given tag that are included between two tags. Currently I am working with BeautifulSoup.
Below you can find an example:






<p class='x' id = '1'> some content 1 <p>
<p class='y' id = 'a'> some content a <p>
<p class='y' id = 'b'> some content b <p>
<p class='y' id = 'c'> some content c <p>
<p class='potentially some other class'> <p>
<p class='x' id = '2'> some content 2 <p>
<p class='y' id = 'd'> some content d <p>
<p class='y' id = 'e'> some content e <p>
<p class='y' id = 'f'> some content f <p>





I am interested in selecting all the instances of class 'y' between the two tags 'x', which also have different id's. With regards to the specific example, I would like to select all p's with class = 'y' to then retrieve the text. My final desired output would be: 'some content a', 'some content b', and 'some content c'.



I tried using the findAllNext method, but that gives me 'some content a', 'some content b', 'some content c' AND 'some content d', 'some content e', 'some content f'.



Below is my code



par = BeautifulSoup(HTML_CODE).content, 'lxml') 
loc = par.find('p', class_ = 'x', id ='1')
desired = loc.findAllNext('p', class_ = 'y')


Is there any way to avoid selecting also the instances of class = 'y' that appear after the tag with class='x' with id = '2' ?



Thank you.










share|improve this question

























    up vote
    2
    down vote

    favorite












    I would like to extract all instances of a given tag that are included between two tags. Currently I am working with BeautifulSoup.
    Below you can find an example:






    <p class='x' id = '1'> some content 1 <p>
    <p class='y' id = 'a'> some content a <p>
    <p class='y' id = 'b'> some content b <p>
    <p class='y' id = 'c'> some content c <p>
    <p class='potentially some other class'> <p>
    <p class='x' id = '2'> some content 2 <p>
    <p class='y' id = 'd'> some content d <p>
    <p class='y' id = 'e'> some content e <p>
    <p class='y' id = 'f'> some content f <p>





    I am interested in selecting all the instances of class 'y' between the two tags 'x', which also have different id's. With regards to the specific example, I would like to select all p's with class = 'y' to then retrieve the text. My final desired output would be: 'some content a', 'some content b', and 'some content c'.



    I tried using the findAllNext method, but that gives me 'some content a', 'some content b', 'some content c' AND 'some content d', 'some content e', 'some content f'.



    Below is my code



    par = BeautifulSoup(HTML_CODE).content, 'lxml') 
    loc = par.find('p', class_ = 'x', id ='1')
    desired = loc.findAllNext('p', class_ = 'y')


    Is there any way to avoid selecting also the instances of class = 'y' that appear after the tag with class='x' with id = '2' ?



    Thank you.










    share|improve this question























      up vote
      2
      down vote

      favorite









      up vote
      2
      down vote

      favorite











      I would like to extract all instances of a given tag that are included between two tags. Currently I am working with BeautifulSoup.
      Below you can find an example:






      <p class='x' id = '1'> some content 1 <p>
      <p class='y' id = 'a'> some content a <p>
      <p class='y' id = 'b'> some content b <p>
      <p class='y' id = 'c'> some content c <p>
      <p class='potentially some other class'> <p>
      <p class='x' id = '2'> some content 2 <p>
      <p class='y' id = 'd'> some content d <p>
      <p class='y' id = 'e'> some content e <p>
      <p class='y' id = 'f'> some content f <p>





      I am interested in selecting all the instances of class 'y' between the two tags 'x', which also have different id's. With regards to the specific example, I would like to select all p's with class = 'y' to then retrieve the text. My final desired output would be: 'some content a', 'some content b', and 'some content c'.



      I tried using the findAllNext method, but that gives me 'some content a', 'some content b', 'some content c' AND 'some content d', 'some content e', 'some content f'.



      Below is my code



      par = BeautifulSoup(HTML_CODE).content, 'lxml') 
      loc = par.find('p', class_ = 'x', id ='1')
      desired = loc.findAllNext('p', class_ = 'y')


      Is there any way to avoid selecting also the instances of class = 'y' that appear after the tag with class='x' with id = '2' ?



      Thank you.










      share|improve this question













      I would like to extract all instances of a given tag that are included between two tags. Currently I am working with BeautifulSoup.
      Below you can find an example:






      <p class='x' id = '1'> some content 1 <p>
      <p class='y' id = 'a'> some content a <p>
      <p class='y' id = 'b'> some content b <p>
      <p class='y' id = 'c'> some content c <p>
      <p class='potentially some other class'> <p>
      <p class='x' id = '2'> some content 2 <p>
      <p class='y' id = 'd'> some content d <p>
      <p class='y' id = 'e'> some content e <p>
      <p class='y' id = 'f'> some content f <p>





      I am interested in selecting all the instances of class 'y' between the two tags 'x', which also have different id's. With regards to the specific example, I would like to select all p's with class = 'y' to then retrieve the text. My final desired output would be: 'some content a', 'some content b', and 'some content c'.



      I tried using the findAllNext method, but that gives me 'some content a', 'some content b', 'some content c' AND 'some content d', 'some content e', 'some content f'.



      Below is my code



      par = BeautifulSoup(HTML_CODE).content, 'lxml') 
      loc = par.find('p', class_ = 'x', id ='1')
      desired = loc.findAllNext('p', class_ = 'y')


      Is there any way to avoid selecting also the instances of class = 'y' that appear after the tag with class='x' with id = '2' ?



      Thank you.






      <p class='x' id = '1'> some content 1 <p>
      <p class='y' id = 'a'> some content a <p>
      <p class='y' id = 'b'> some content b <p>
      <p class='y' id = 'c'> some content c <p>
      <p class='potentially some other class'> <p>
      <p class='x' id = '2'> some content 2 <p>
      <p class='y' id = 'd'> some content d <p>
      <p class='y' id = 'e'> some content e <p>
      <p class='y' id = 'f'> some content f <p>





      <p class='x' id = '1'> some content 1 <p>
      <p class='y' id = 'a'> some content a <p>
      <p class='y' id = 'b'> some content b <p>
      <p class='y' id = 'c'> some content c <p>
      <p class='potentially some other class'> <p>
      <p class='x' id = '2'> some content 2 <p>
      <p class='y' id = 'd'> some content d <p>
      <p class='y' id = 'e'> some content e <p>
      <p class='y' id = 'f'> some content f <p>






      python html beautifulsoup tags






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 11 at 18:53









      mgiom

      226




      226






















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          2
          down vote



          accepted










          You can start iterating from where you want and end it until found something mark finished.



          from bs4 import BeautifulSoup

          html = """

          <p class='x' id = '1'> some content 1 </p>
          <p class='y' id = 'a'> some content a </p>
          <p class='y' id = 'b'> some content b </p>
          <p class='y' id = 'c'> some content c </p>
          <p class='potentially some other class1'> potentially some other class 1 </p>
          <p class='potentially some other class2'> potentially some other class 2</p>
          <p class='potentially some other class3'> potentially some other class 3 </p>
          <p class='x' id = '2'> some content 2 </p>
          <p class='y' id = 'd'> some content d </p>
          <p class='y' id = 'e'> some content e </p>
          <p class='y' id = 'f'> some content f </p>
          """

          soup = BeautifulSoup(html,"lxml")
          start = soup.find("p",class_="y",id="c")
          end = soup.find("p",class_="x",id="2")
          def next_ele(ele,result=):
          row = ele.find_next("p")
          if not row or row == end:
          return result
          result.append(row)
          return next_ele(row,result)

          print(next_ele(start))





          share|improve this answer






















          • This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
            – mgiom
            Nov 12 at 20:41






          • 1




            @mgiom i have edited my code, this only an example. In different situation you need to add more params.
            – kcorlidy
            Nov 13 at 4:55










          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252053%2fselect-all-tag-and-extract-text-between-two-other-tags-with-beatifulsoup%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          2
          down vote



          accepted










          You can start iterating from where you want and end it until found something mark finished.



          from bs4 import BeautifulSoup

          html = """

          <p class='x' id = '1'> some content 1 </p>
          <p class='y' id = 'a'> some content a </p>
          <p class='y' id = 'b'> some content b </p>
          <p class='y' id = 'c'> some content c </p>
          <p class='potentially some other class1'> potentially some other class 1 </p>
          <p class='potentially some other class2'> potentially some other class 2</p>
          <p class='potentially some other class3'> potentially some other class 3 </p>
          <p class='x' id = '2'> some content 2 </p>
          <p class='y' id = 'd'> some content d </p>
          <p class='y' id = 'e'> some content e </p>
          <p class='y' id = 'f'> some content f </p>
          """

          soup = BeautifulSoup(html,"lxml")
          start = soup.find("p",class_="y",id="c")
          end = soup.find("p",class_="x",id="2")
          def next_ele(ele,result=):
          row = ele.find_next("p")
          if not row or row == end:
          return result
          result.append(row)
          return next_ele(row,result)

          print(next_ele(start))





          share|improve this answer






















          • This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
            – mgiom
            Nov 12 at 20:41






          • 1




            @mgiom i have edited my code, this only an example. In different situation you need to add more params.
            – kcorlidy
            Nov 13 at 4:55














          up vote
          2
          down vote



          accepted










          You can start iterating from where you want and end it until found something mark finished.



          from bs4 import BeautifulSoup

          html = """

          <p class='x' id = '1'> some content 1 </p>
          <p class='y' id = 'a'> some content a </p>
          <p class='y' id = 'b'> some content b </p>
          <p class='y' id = 'c'> some content c </p>
          <p class='potentially some other class1'> potentially some other class 1 </p>
          <p class='potentially some other class2'> potentially some other class 2</p>
          <p class='potentially some other class3'> potentially some other class 3 </p>
          <p class='x' id = '2'> some content 2 </p>
          <p class='y' id = 'd'> some content d </p>
          <p class='y' id = 'e'> some content e </p>
          <p class='y' id = 'f'> some content f </p>
          """

          soup = BeautifulSoup(html,"lxml")
          start = soup.find("p",class_="y",id="c")
          end = soup.find("p",class_="x",id="2")
          def next_ele(ele,result=):
          row = ele.find_next("p")
          if not row or row == end:
          return result
          result.append(row)
          return next_ele(row,result)

          print(next_ele(start))





          share|improve this answer






















          • This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
            – mgiom
            Nov 12 at 20:41






          • 1




            @mgiom i have edited my code, this only an example. In different situation you need to add more params.
            – kcorlidy
            Nov 13 at 4:55












          up vote
          2
          down vote



          accepted







          up vote
          2
          down vote



          accepted






          You can start iterating from where you want and end it until found something mark finished.



          from bs4 import BeautifulSoup

          html = """

          <p class='x' id = '1'> some content 1 </p>
          <p class='y' id = 'a'> some content a </p>
          <p class='y' id = 'b'> some content b </p>
          <p class='y' id = 'c'> some content c </p>
          <p class='potentially some other class1'> potentially some other class 1 </p>
          <p class='potentially some other class2'> potentially some other class 2</p>
          <p class='potentially some other class3'> potentially some other class 3 </p>
          <p class='x' id = '2'> some content 2 </p>
          <p class='y' id = 'd'> some content d </p>
          <p class='y' id = 'e'> some content e </p>
          <p class='y' id = 'f'> some content f </p>
          """

          soup = BeautifulSoup(html,"lxml")
          start = soup.find("p",class_="y",id="c")
          end = soup.find("p",class_="x",id="2")
          def next_ele(ele,result=):
          row = ele.find_next("p")
          if not row or row == end:
          return result
          result.append(row)
          return next_ele(row,result)

          print(next_ele(start))





          share|improve this answer














          You can start iterating from where you want and end it until found something mark finished.



          from bs4 import BeautifulSoup

          html = """

          <p class='x' id = '1'> some content 1 </p>
          <p class='y' id = 'a'> some content a </p>
          <p class='y' id = 'b'> some content b </p>
          <p class='y' id = 'c'> some content c </p>
          <p class='potentially some other class1'> potentially some other class 1 </p>
          <p class='potentially some other class2'> potentially some other class 2</p>
          <p class='potentially some other class3'> potentially some other class 3 </p>
          <p class='x' id = '2'> some content 2 </p>
          <p class='y' id = 'd'> some content d </p>
          <p class='y' id = 'e'> some content e </p>
          <p class='y' id = 'f'> some content f </p>
          """

          soup = BeautifulSoup(html,"lxml")
          start = soup.find("p",class_="y",id="c")
          end = soup.find("p",class_="x",id="2")
          def next_ele(ele,result=):
          row = ele.find_next("p")
          if not row or row == end:
          return result
          result.append(row)
          return next_ele(row,result)

          print(next_ele(start))






          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 13 at 4:52

























          answered Nov 12 at 12:31









          kcorlidy

          1,725317




          1,725317











          • This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
            – mgiom
            Nov 12 at 20:41






          • 1




            @mgiom i have edited my code, this only an example. In different situation you need to add more params.
            – kcorlidy
            Nov 13 at 4:55
















          • This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
            – mgiom
            Nov 12 at 20:41






          • 1




            @mgiom i have edited my code, this only an example. In different situation you need to add more params.
            – kcorlidy
            Nov 13 at 4:55















          This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
          – mgiom
          Nov 12 at 20:41




          This is useful, but I don't necessarily know how many instances of class 'y' I have between the two tags with class = 'x'. Is there a way to proceed in the iteration you described, but in such a way that it goes on until it finds again an instance of the class 'x' ? Thank you
          – mgiom
          Nov 12 at 20:41




          1




          1




          @mgiom i have edited my code, this only an example. In different situation you need to add more params.
          – kcorlidy
          Nov 13 at 4:55




          @mgiom i have edited my code, this only an example. In different situation you need to add more params.
          – kcorlidy
          Nov 13 at 4:55

















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252053%2fselect-all-tag-and-extract-text-between-two-other-tags-with-beatifulsoup%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          這個網誌中的熱門文章

          How to read a connectionString WITH PROVIDER in .NET Core?

          In R, how to develop a multiplot heatmap.2 figure showing key labels successfully

          Museum of Modern and Contemporary Art of Trento and Rovereto