Kegan Dougal
38ead2c174
Make ServiceDB an interface called `Storer`
Implement a nop `Storer` called `NopStorage` which tests can make use of.
This was awful to write.
8 years ago
Kegan Dougal
6fd0f20c81
Review comments
8 years ago
Kegan Dougal
fcd3befb09
HTML decode the RSS title/description fields
8 years ago
Kegan Dougal
07e93c5ba2
Add failing TestHTMLEntities test
8 years ago
Kegan Dougal
ba6e5e8862
Rotate logs on a daily basis: Fixes #97
8 years ago
Kegan Dougal
f753e0b669
Move test util functions to dedicated file to clean up test files
Also use verbose mode on tests so you can see which test is running when things
print to stdout.
8 years ago
Kegan Dougal
352d741558
Review comments
8 years ago
Kegan Dougal
5c65d4cf95
Add a mock HTTP client for Matrix clients to use. Add example test.
This will need rejigging at some point to make the test easier to set up.
8 years ago
Kegan Dougal
dc2fadd450
Factor out startup process to `setup()` and add top-level test
- Pass in an `envVars` struct rather than grabbing directly from `os`.
- Accept a `ServeMux` rather than always using the default mux.
- Add a `_test.go` file to instrument Go-NEB using `TestMain` to call `setup()`
8 years ago
Kegan Dougal
236c97d279
Tests and comments
8 years ago
Kegan Dougal
92ac46e925
Factor out a Protect() function for incoming HTTP requests
8 years ago
Kegan Dougal
7ab9794623
Protect panicking poll loops from taking down the entire process
8 years ago
Kegan Dougal
ce414ea586
Prevent panicking code triggered by matrix events from taking down the entire process
Specifically, this protects panicking `!commands` and expansions.
8 years ago
Kegan Dougal
3efb632b31
Log panicking HTTP requests
Whilst Go does this by default (does not `exit`), it logs to the std logger
which isn't written to the log file. Handle all panicking `OnIncomingRequests`.
8 years ago
Kegan Dougal
9d9934e923
Tidy up how we load the database
8 years ago
Kegan Dougal
79c3145cca
Docs for Check() functions
8 years ago
Kegan Dougal
e4cacb1cd5
Use correct key names for service configs
8 years ago
Kegan Dougal
08d4db20bf
Fix panic if github is down
8 years ago
Kegan Dougal
0fce8aea09
Glue everything together
8 years ago
Kegan Dougal
295c9bbb4b
Add 'api' package and move HTTP API requests there
This will make docs easier in addition to allowing the 'database' package
not circular-dep.
8 years ago
Kegan Dougal
56e92a9632
Convert YAML to NEB types via roundabout methods
8 years ago
Kegan Dougal
a7de9a7fdd
Add YAML tags to things so we can unmarshal them. Cannot partially unmarshal json.RawMessage-like though...
8 years ago
Kegan Dougal
f4033d1745
Add stub config file loader
8 years ago
Kegan Dougal
e35ea2f27c
Use the NextPollTimestampSecs or else we'll resend on /configureService due to JSON key name change
8 years ago
Kegan Dougal
bc1e6283d8
Make some fields 'public'
8 years ago
Kegan Dougal
fae997fdb7
Add a flag `IsFailing` to let clients know if their RSS feeds are down
8 years ago
Kegan Dougal
299164d525
Fix RSS feeds which do not return GUIDs and do not have published dates
We previously relied on the published date and GUIDs to determine which items
were new. We now only rely on the GUID and not the published date as a lot of
RSS feeds don't have published dates. A lot of feeds don't have GUIDs either,
so we now fallback to the HTTP `Link` field, or worst-case, the item `Title`.
8 years ago
Kegan Dougal
5b5de2dc4b
Set "Go-NEB" as the User-Agent on RSS feed polling
Some services, notably Reddit, basically force you to set a custom UA in order
to use their RSS feeds. All the common default non-browser UAs are HEAVILY
rate-limited such that you can't realistically use them.
8 years ago
Kegan Dougal
5c8cc4a81c
strconv to convert int to string
8 years ago
Kegan Dougal
433331ca20
Metrics: Add auth session counter
8 years ago
Kegan Dougal
f493a1933e
s/sendMetric/incrementMetrics/
8 years ago
Kegan Dougal
2e5c8e4d79
Only send the domain part of the RSS feed as that's more useful when doing aggregate metrics
8 years ago
Kegan Dougal
b4b0a661c4
Add more metrics
8 years ago
Kegan Dougal
cdb49fa60b
Add command type as a label
8 years ago
Kegan Dougal
32e8a5b6cc
Docs
8 years ago
Kegan Dougal
19eeb96610
Use enums instead
8 years ago
Kegan Dougal
219da01e08
Add extremely noddy counter metrics for !commands
Missing the command type currently. Also, not sure how "nice" this is: we could
probably blob the 3 functions into 1 and use an enum but, meh?
8 years ago
Kegan Dougal
729b80c6bc
Set max open conns to 1 on sqlite3
8 years ago
Kegan Dougal
c9de6eaf06
Instrument all HTTP handlers
8 years ago
Kegan Dougal
248822a3d6
Fix #86 - set a 20s min threshold before re-polling
8 years ago
Kegan Dougal
ef6fc0d3bf
Remove useless field
8 years ago
Kegan Dougal
8316ac21ab
Maintain a list of GUIDs per feed and use them to de-dupe
Some RSS feeds will edit the published time of an item AFTER putting it out,
which resulted in RSS Bot sending the same article twice. We now remember the
"GUID" field for each item and de-dupe based on that. Normal timestamp
algorithm still applies.
8 years ago
Kegan Dougal
77b729d4ba
Remove spammy logs
8 years ago
Kegan Dougal
5f015e26ee
Join rooms when registering rss service
8 years ago
Kegan Dougal
b903860af9
Bump min to 5min to help not spam feeds with no cache control
8 years ago
Kegan Dougal
e088e11821
Swap to an LRU cache set to 20MB
8 years ago
Kegan Dougal
b0eb284347
Add a caching HTTP client
8 years ago
Kegan Dougal
fd54eb1a55
Move OnPoll into the Service method set
This feels a lot better because now `OnPoll` works in a similar way to
`OnReceiveWebhook` (called on a `Service`) rather than have this strange
global-per-service-type struct.
8 years ago
Kegan Dougal
7518ed1995
Fix NPE if oldService is nil
8 years ago
Kegan Dougal
cd39219113
s/feedreader/rssbot/g
Because apparently "Feed Reader" is the name of a thing popular enough that we
don't want to step on toes, so let's call it the name of a less popular thing.
8 years ago