Browse Source

Make embedly scraper skip inapplicable links

And this is exactly why this needs to be refactored into a common
behavior.
merge-requests/76/head
Deimos 5 years ago
parent
commit
416daf4d7d
  1. 3
      tildes/consumers/topic_embedly_extractor.py

3
tildes/consumers/topic_embedly_extractor.py

@ -43,6 +43,9 @@ class TopicEmbedlyExtractor(PgsqlQueueConsumer):
if not topic.is_link_type:
return
if not self.scraper.is_applicable(topic.link):
return
# see if we already have a recent scrape result from the same url
result = (
self.db_session.query(ScraperResult)

Loading…
Cancel
Save