go-neb

Commit Graph

Author	SHA1	Message	Date
Kegan Dougal	352d741558	Review comments	8 years ago
Kegan Dougal	5c65d4cf95	Add a mock HTTP client for Matrix clients to use. Add example test. This will need rejigging at some point to make the test easier to set up.	8 years ago
Kegan Dougal	dc2fadd450	Factor out startup process to `setup()` and add top-level test - Pass in an `envVars` struct rather than grabbing directly from `os`. - Accept a `ServeMux` rather than always using the default mux. - Add a `_test.go` file to instrument Go-NEB using `TestMain` to call `setup()`	8 years ago
Kegan Dougal	236c97d279	Tests and comments	8 years ago
Kegan Dougal	92ac46e925	Factor out a Protect() function for incoming HTTP requests	8 years ago
Kegan Dougal	7ab9794623	Protect panicking poll loops from taking down the entire process	8 years ago
Kegan Dougal	ce414ea586	Prevent panicking code triggered by matrix events from taking down the entire process Specifically, this protects panicking `!commands` and expansions.	8 years ago
Kegan Dougal	3efb632b31	Log panicking HTTP requests Whilst Go does this by default (does not `exit`), it logs to the std logger which isn't written to the log file. Handle all panicking `OnIncomingRequests`.	8 years ago
Kegan Dougal	9d9934e923	Tidy up how we load the database	8 years ago
Kegan Dougal	79c3145cca	Docs for Check() functions	8 years ago
Kegan Dougal	e4cacb1cd5	Use correct key names for service configs	8 years ago
Kegan Dougal	08d4db20bf	Fix panic if github is down	8 years ago
Kegan Dougal	0fce8aea09	Glue everything together	8 years ago
Kegan Dougal	295c9bbb4b	Add 'api' package and move HTTP API requests there This will make docs easier in addition to allowing the 'database' package not circular-dep.	8 years ago
Kegan Dougal	56e92a9632	Convert YAML to NEB types via roundabout methods	8 years ago
Kegan Dougal	a7de9a7fdd	Add YAML tags to things so we can unmarshal them. Cannot partially unmarshal json.RawMessage-like though...	8 years ago
Kegan Dougal	f4033d1745	Add stub config file loader	8 years ago
Kegan Dougal	e35ea2f27c	Use the NextPollTimestampSecs or else we'll resend on /configureService due to JSON key name change	8 years ago
Kegan Dougal	bc1e6283d8	Make some fields 'public'	8 years ago
Kegan Dougal	fae997fdb7	Add a flag `IsFailing` to let clients know if their RSS feeds are down	8 years ago
Kegan Dougal	299164d525	Fix RSS feeds which do not return GUIDs and do not have published dates We previously relied on the published date and GUIDs to determine which items were new. We now only rely on the GUID and not the published date as a lot of RSS feeds don't have published dates. A lot of feeds don't have GUIDs either, so we now fallback to the HTTP `Link` field, or worst-case, the item `Title`.	8 years ago
Kegan Dougal	5b5de2dc4b	Set "Go-NEB" as the User-Agent on RSS feed polling Some services, notably Reddit, basically force you to set a custom UA in order to use their RSS feeds. All the common default non-browser UAs are HEAVILY rate-limited such that you can't realistically use them.	8 years ago
Kegan Dougal	5c8cc4a81c	strconv to convert int to string	8 years ago
Kegan Dougal	433331ca20	Metrics: Add auth session counter	8 years ago
Kegan Dougal	f493a1933e	s/sendMetric/incrementMetrics/	8 years ago
Kegan Dougal	2e5c8e4d79	Only send the domain part of the RSS feed as that's more useful when doing aggregate metrics	8 years ago
Kegan Dougal	b4b0a661c4	Add more metrics	8 years ago
Kegan Dougal	cdb49fa60b	Add command type as a label	8 years ago
Kegan Dougal	32e8a5b6cc	Docs	8 years ago
Kegan Dougal	19eeb96610	Use enums instead	8 years ago
Kegan Dougal	219da01e08	Add extremely noddy counter metrics for !commands Missing the command type currently. Also, not sure how "nice" this is: we could probably blob the 3 functions into 1 and use an enum but, meh?	8 years ago
Kegan Dougal	729b80c6bc	Set max open conns to 1 on sqlite3	8 years ago
Kegan Dougal	c9de6eaf06	Instrument all HTTP handlers	8 years ago
Kegan Dougal	248822a3d6	Fix #86 - set a 20s min threshold before re-polling	8 years ago
Kegan Dougal	ef6fc0d3bf	Remove useless field	8 years ago
Kegan Dougal	8316ac21ab	Maintain a list of GUIDs per feed and use them to de-dupe Some RSS feeds will edit the published time of an item AFTER putting it out, which resulted in RSS Bot sending the same article twice. We now remember the "GUID" field for each item and de-dupe based on that. Normal timestamp algorithm still applies.	8 years ago
Kegan Dougal	77b729d4ba	Remove spammy logs	8 years ago
Kegan Dougal	5f015e26ee	Join rooms when registering rss service	8 years ago
Kegan Dougal	b903860af9	Bump min to 5min to help not spam feeds with no cache control	8 years ago
Kegan Dougal	e088e11821	Swap to an LRU cache set to 20MB	8 years ago
Kegan Dougal	b0eb284347	Add a caching HTTP client	8 years ago
Kegan Dougal	fd54eb1a55	Move OnPoll into the Service method set This feels a lot better because now `OnPoll` works in a similar way to `OnReceiveWebhook` (called on a `Service`) rather than have this strange global-per-service-type struct.	8 years ago
Kegan Dougal	7518ed1995	Fix NPE if oldService is nil	8 years ago
Kegan Dougal	cd39219113	s/feedreader/rssbot/g Because apparently "Feed Reader" is the name of a thing popular enough that we don't want to step on toes, so let's call it the name of a less popular thing.	8 years ago
Kegan Dougal	185a7d5678	Rejig how feedreaders stores rooms to send updates to	8 years ago
Kegan Dougal	7b4448957b	Review comments	8 years ago
Kegan Dougal	98e7533469	Send feed items in reverse-order for chronological times Also add checks on Register() to make sure the feeds are valid	8 years ago
Kegan Dougal	d92a95ff4e	Linting	8 years ago
Kegan Dougal	0b09fcd86d	First working feed reader version	8 years ago
Kegan Dougal	2d6b3cd6e9	Flesh out the feedreader service Just need to send messages into rooms now for a first cut to be done. Notable improvements to make: - We currently do 1 goroutine per service. This could be bad if we have lots of these things running around. - We do not cache the response to RSS feeds. If we have 10 independent services on the same feed URL, we will hit the URL 10 times. This is similar to how we currently do 1 webhook/service, so it's plausible that in the future we will want to have some kind of generic caching layer. - We don't send messages to Matrix yet. We need a `Clients` instance but can't get at one. There's only ever one, so I wonder if we should global it like we do with `GetServiceDB()` for ease of use? - The polling interval is divorced from the actual feed repoll time. Ideally we would schedule the goroutine only when we need it, rather than checking frequently, determining we have nothing to do, and going back to sleep.	8 years ago

... 2 3 4 5 6 ...

362 Commits (8f635f573dbb465a24d8acc83472a7b40f75fe3e)