Odtnhj

Edit:

I found the links to everything embedded under a div tag in a link tag - not knowing ajax/front end dev, I'm not sure what you call it: looks like this:

<a class="tabs__link js-tabs-ranking" href="it" data-ajax-stack="&quot;itg&quot;:&quot;/en/ajax/ranking/19/itg/18f11c81b4cd83f7b82b47a88d939a9c/none&quot;,&quot;ipg&quot;:&quot;/en/ajax/ranking/19/ipg/b1c62bbc714bc8823f59f3ec1030a3d7/none&quot;,&quot;etg&quot;:&quot;/en/ajax/ranking/19/etg/5b2a3871133c7df8954b81ca884d233f/none&quot;,&quot;img&quot;:&quot;/en/ajax/ranking/19/img/03a4a10eac4baaffa954cebf29c39b1c/none&quot;,&quot;ijg&quot;:&quot;/en/ajax/ranking/19/ijg/ec301eb70c0b7df824159aaa00d79135/none&quot;,&quot;icg&quot;:&quot;/en/ajax/ranking/19/icg/81b5589ac9889472dcda9560dd23683d/none&quot;" data-type="g" data-xtclick="ranking::tab::overall">General classification</a>

I've gotten the following code to get the tables into dataframes for everything EXCEPT pages where there are multiple tables and headings - type ime, ipe - I've made a if-else to try and handle those pages differently.

What I'd like to do is put each table into its own dataframe but I keep getting a bunch of ugly html and an error at the end "TypeError: 'NoneType' object is not callable". I'll keep banging away at this but any suggestions are more than welcome!

import requests
import html5lib
import pandas as pd
from bs4 import BeautifulSoup


#type_dict = 'e':'Stage', 'g':'General Classification'
tab_dict = 'ite':'Stage',
'ipe':'Points',
'ime':'Mountains',
'ije':'Young riders',
'ice':'Combativity',
'ete':'Teams',
'itg':'General Classification',
'ipg':'Points Classification',
'img':'Mountains Classification',
'ijg':'Young Riders Classification',
'icg':'Combativity Classification',
'etg':'Teams Classification'
#Add a user input for the URL
start_url = "https://www.letour.fr/en/rankings/stage-19"
base_url = start_url.split('/')[2]

page = requests.get(start_url)
content = page.content
r_table = pd.read_html(content)

#This worked to get the table out into a DataFrame
df = r_table[0]
#print(df['Rider'])
soup = BeautifulSoup(content, "html5lib")

all_links = soup.find_all(class_="tabs__link js-tabs-ranking")
#grabbing the block of ajax links that give URLs to various stage/GC results
for item in all_links:
 myurl = item['data-ajax-stack']
 myurl = myurl.replace('/', '/').replace('', '').replace('', '').replace('"','')
 myurl = dict(x.split(':') for x in myurl.split(','))
#looping through the lists of links and getting the pages
 for key, value in myurl.items():
 r_type = tab_dict[key]
 print("Getting the data for: " + r_type)
 url = ("http://" + base_url + value)
 try:
 if key == "ipe" or key == "ime":

 page = requests.get(url).content
 print(page)
 soup = BeautifulSoup(page, "html5lib")
 #heading = soup.find_all('div', class_="rankingTables__caption")
 for caption in soup.find_all('div', class_="rankingTables__caption"):
 res_caption = caption.text.title()
 print(res_caption)
 res_table = pd.read_html(caption)
 df = res_table[0]
 print(df) #debugging, test
 else:
 page = requests.get(url).content
 soup = BeautifulSoup(page, "html5lib")
 res_table = pd.read_html(page)
 df = res_table[0]
 print(df) #debugging/test

 except ValueError:
 print("No table found for " + key)
 break

I am relatively new to Python, and am using a web scraping project to learn more. I am stuck on a problem trying to get multiple blocks of tabular data from a dynamic web page. Is there an easy way to get the tables generated from the various clicks on this page?

My code below works on the results that come up on the default load, but I want to be able to loop through the tabs and grab them all into the same dataframe.

Looking at the source code, there is one tag that changes depending on the tab you click:

I thought about making a dictionary
data-current-type 'e':'Stage', 'g':'General Classification'
data-current-tab 'it':'Individual Classification',
'ip':'Points',
'im':'Mountains',
'ij':'Young riders',
'ic':'Combativity',
'et':'Teams'

This design relies upon being able to pass these different tags back to the page and I don't think that's going to work.

Looking at the post, the URLs are dynamically generated:

https://www.letour.fr/en/ajax/ranking/20/itg/8c7d5ddc44042219f544306cab96c718/subtab
https://www.letour.fr/en/ajax/ranking/20/ipg/2d4afa3722c55ad1564caddee00f117f/subtab

Can anyone point me in the direction of the best tool to get the data I want? I've tried searching and searching this forum but I must not be using the right tags...

import requests
import pandas as pd

start_url = "https://www.letour.fr/en/rankings/stage-20"

page = requests.get(start_url)

content = page.content
#get the table
res_table = pd.read_html(content)
#Define the DataFrame
df = res_table[0]

You could do something like the following. Where I loop the links you provided and concat results into a final dataframe.

import pandas as pd
links = ["https://www.letour.fr/en/ajax/ranking/20/itg/8c7d5ddc44042219f544306cab96c718/subtab","https://www.letour.fr/en/ajax/ranking/20/ipg/2d4afa3722c55ad1564caddee00f117f/subtab"]
final = 

for link in links:
 result = pd.read_html(link)
 # print(result)
 header = result[0][0:0]
 final.append(result[0][0:])

df = pd.concat(final, sort=False)
df.drop_duplicates()
df.index = pd.RangeIndex(len(df.index))
print(df)
df.to_csv(r"C:UsersUserDesktoptest.csv", encoding='utf-8')

I am not so sure the links are dynamically generated. Refreshing the page appears that the links are the same.

That being said, what you may want to do is extract the links based on the xpath of the <a> elements of the tabs.

So your xpath to get the links might be a dict:

links_xpath = 
 'climber' : "//a[contains(@class, 'tabs__link') and contains(text(), 'Climber')]/@href",
 'points' : "//a[contains(@class, 'tabs__link') and contains(text(), 'Points')]/@href",
 # etc.

That will extract the link, which you can then concat with the base URL and ensure your scraper works regardless of the underlying link, at least until the page layout might change.