go-neb

Commit Graph

Author	SHA1	Message	Date
Nikos Filippakis	1e297c50ad	Re-format project as a Go module (#310 ) * Define project as a Go module and update dependency versions Signed-off-by: Nikos Filippakis <me@nfil.dev> * Update docs, configs and dockerfile to use latest Go version Signed-off-by: Nikos Filippakis <me@nfil.dev> * Add postgres database driver Signed-off-by: Nikos Filippakis <me@nfil.dev>	5 years ago
Nikos Filippakis	69e0d1bc39	RSS show author & ability to filter items (#311 ) * Show author of RSS feed item if available closes #298 Signed-off-by: Nikos Filippakis <me@nfil.dev> * Filter RSS feed items by keywords closes #236, closes #296 Signed-off-by: Nikos Filippakis <me@nfil.dev>	5 years ago
Kegan Dougal	d9213d61a4	Make it compile	5 years ago
Ville Ranki	7e0f2f6196	Add spaces between url and parenthesis	6 years ago
Andrew Morgan	5188df5d0e	Don't mutate the passed in attribute	6 years ago
Andrew Morgan	0bf5586881	Use feed's title if item does not have one This prevents a link from not being shown if an item's title could not be extracted. Workaround(?) fix for matrix-org#284.	6 years ago
Andrew Morgan	6f1879f359	Fix mismatching types error (#277 )	6 years ago
thomas-profitt	4febdd6330	Improve formatting of RSS posts (#269 ) Better on clients that don't support formatting and doesn't waste space on those that do.	6 years ago
Michael Kaye	2938b1bd22	Make golint happy	6 years ago
Will Hunt	468cc87b1d	Drop unused dependency	6 years ago
Will Hunt	149d66f995	Remove "url" from RSS metric It's not really that useful	6 years ago
Kegan Dougal	38ef6c591d	Increase the max number of GUIDs we keep per RSS feed Some feeds have >1k items such as: - https://dashboard.downdetector.com/events/qfu1JvvwVwYf1OvgvORuD4HQMt5xMF2Y/217:405-5025.xml If Go-NEB tracks this feed, it will continually send N-1000 items into the matrix room because it doesn't remember all the GUIDs. This is only a hack to fix this problem for now, to avoid DoSing Synapse.	8 years ago
Kegan Dougal	efe04d7917	Keep doing the Inviter hack when joining rooms so non-ops can kick the bot they invited	8 years ago
Kegan Dougal	b98e878227	Use gomatrix	8 years ago
Kegan Dougal	7522ddf1a4	Explain why no ParseURL	8 years ago
Kegan Dougal	c4e98238d8	uniq prior to length checks for accuracy	8 years ago
Kegan Dougal	1722871345	Don't use gofeed.ParseURL It leaks the response because it doesn't close the `resp.Body` on non-2xx.	8 years ago
Kegan Dougal	332b451d77	Redo how recent GUIDs are calculated This is an attempt to fix #133. Previously, we just clobbered the recent GUIDs with the lastest response every single time, assuming that Atom/RSS feeds would consistently return the same items. This appears to not be the case. In the wild, the number of items returned on a single request can vary (sometimes even being 1 or 2 when usually it is 50!). This patch alters how many and which GUIDs we keep between requests, in an attempt to prevent sending old news for buggy RSS feeds.	8 years ago
Kegan Dougal	d128ed2626	Add more logging for RSS feeds	8 years ago
Kegan Dougal	677b556376	Return an error so it is logged	8 years ago
Kegan Dougal	66fb3253f7	Treat RSS feeds with 0 items as errors In the wild it looks like some RSS feeds will occasionally return 0 items to requests but not return an error. This previously meant we would clobber our knowledge of recent GUIDs with the empty set. This meant that the next successful poll would resend the entire RSS feed.	8 years ago
Kegan Dougal	a06ac2d3be	Make it clear that these are example requests	8 years ago
Kegan Dougal	209af79efe	Add example configs for each service	8 years ago
Kegan Dougal	3fec1c6fa5	Add docs for all the remaining services and convert them to use NewDefaultService My hands hurt.	8 years ago
Kegan Dougal	fcd3befb09	HTML decode the RSS title/description fields	8 years ago
Kegan Dougal	e35ea2f27c	Use the NextPollTimestampSecs or else we'll resend on /configureService due to JSON key name change	8 years ago
Kegan Dougal	bc1e6283d8	Make some fields 'public'	8 years ago
Kegan Dougal	fae997fdb7	Add a flag `IsFailing` to let clients know if their RSS feeds are down	8 years ago
Kegan Dougal	299164d525	Fix RSS feeds which do not return GUIDs and do not have published dates We previously relied on the published date and GUIDs to determine which items were new. We now only rely on the GUID and not the published date as a lot of RSS feeds don't have published dates. A lot of feeds don't have GUIDs either, so we now fallback to the HTTP `Link` field, or worst-case, the item `Title`.	8 years ago
Kegan Dougal	5b5de2dc4b	Set "Go-NEB" as the User-Agent on RSS feed polling Some services, notably Reddit, basically force you to set a custom UA in order to use their RSS feeds. All the common default non-browser UAs are HEAVILY rate-limited such that you can't realistically use them.	8 years ago
Kegan Dougal	5c8cc4a81c	strconv to convert int to string	8 years ago
Kegan Dougal	f493a1933e	s/sendMetric/incrementMetrics/	8 years ago
Kegan Dougal	2e5c8e4d79	Only send the domain part of the RSS feed as that's more useful when doing aggregate metrics	8 years ago
Kegan Dougal	b4b0a661c4	Add more metrics	8 years ago
Kegan Dougal	248822a3d6	Fix #86 - set a 20s min threshold before re-polling	8 years ago
Kegan Dougal	8316ac21ab	Maintain a list of GUIDs per feed and use them to de-dupe Some RSS feeds will edit the published time of an item AFTER putting it out, which resulted in RSS Bot sending the same article twice. We now remember the "GUID" field for each item and de-dupe based on that. Normal timestamp algorithm still applies.	8 years ago
Kegan Dougal	5f015e26ee	Join rooms when registering rss service	8 years ago
Kegan Dougal	b903860af9	Bump min to 5min to help not spam feeds with no cache control	8 years ago
Kegan Dougal	e088e11821	Swap to an LRU cache set to 20MB	8 years ago
Kegan Dougal	b0eb284347	Add a caching HTTP client	8 years ago
Kegan Dougal	fd54eb1a55	Move OnPoll into the Service method set This feels a lot better because now `OnPoll` works in a similar way to `OnReceiveWebhook` (called on a `Service`) rather than have this strange global-per-service-type struct.	8 years ago
Kegan Dougal	7518ed1995	Fix NPE if oldService is nil	8 years ago
Kegan Dougal	cd39219113	s/feedreader/rssbot/g Because apparently "Feed Reader" is the name of a thing popular enough that we don't want to step on toes, so let's call it the name of a less popular thing.	8 years ago
Kegan Dougal	185a7d5678	Rejig how feedreaders stores rooms to send updates to	8 years ago
Kegan Dougal	7b4448957b	Review comments	8 years ago
Kegan Dougal	98e7533469	Send feed items in reverse-order for chronological times Also add checks on Register() to make sure the feeds are valid	8 years ago
Kegan Dougal	0b09fcd86d	First working feed reader version	8 years ago
Kegan Dougal	2d6b3cd6e9	Flesh out the feedreader service Just need to send messages into rooms now for a first cut to be done. Notable improvements to make: - We currently do 1 goroutine per service. This could be bad if we have lots of these things running around. - We do not cache the response to RSS feeds. If we have 10 independent services on the same feed URL, we will hit the URL 10 times. This is similar to how we currently do 1 webhook/service, so it's plausible that in the future we will want to have some kind of generic caching layer. - We don't send messages to Matrix yet. We need a `Clients` instance but can't get at one. There's only ever one, so I wonder if we should global it like we do with `GetServiceDB()` for ease of use? - The polling interval is divorced from the actual feed repoll time. Ideally we would schedule the goroutine only when we need it, rather than checking frequently, determining we have nothing to do, and going back to sleep.	8 years ago
Kegan Dougal	a5e515e118	Lint	8 years ago
Kegan Dougal	3e3f71ad92	s/rss/feedreader/ since it does atom feeds too. Add TODO	8 years ago

1 Commits (533451cb2fae0c7d2b6181762f07db15ef543d02)