Odtnhj

I'm very new to this, My Task is : Let me tell what exactly I need, I want to scrapy to search and download for some contract which TYPE is EX-10.1, EX-10.2, etc., upto EX-10.99. Contracts available in .htm format and txt format.
1. How to scrap the links and download.
2. How to filter the file types and download.
Scrapy wants to go through this path and download, Path : url->Enter into each CIK links -> search and Download the EX-10 Type files.

My code

import urlparse

from scrapy.http import Request
from scrapy.spiders import BaseSpider
class legco(BaseSpider):
name = "sec_gov"

allowed_domains = ["www.sec.gov", "search.usa.gov", "secsearch.sec.gov"]
start_urls = ["https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany"]


#extract search results
def parse(self, response): 
for link in response.xpath('//div[@id="seriesDiv"]//table[@class="tableFile2"]/a/@href').extract():
 req = Request(url = link, callback = self.parse_page)
 yield req

def parse(self, response):
base_url = 'http://www.sec.gov/cgi-bin/browse-edgar'
for a in response.xpath('//a[@href]/@href'):
 link = a.extract()
 if link.endswith('.htm'):
 link = urlparse.urljoin(base_url, link)
 yield Request(link, callback = self.save_pdf)

def save_pdf(self, response):
path = response.url.split('/')[-1]
with open(path, 'wb') as f:
 f.write(response.body)

What changes I have to do? Can anyone help me with this Issue please. Thank you in Advance.