Kegan Dougal
9d9934e923
Tidy up how we load the database
8 years ago
Kegan Dougal
f40803ba6b
Flesh out config file to explain gotchas
8 years ago
Kegan Dougal
79c3145cca
Docs for Check() functions
8 years ago
Kegan Dougal
e4cacb1cd5
Use correct key names for service configs
8 years ago
Kegan Dougal
0fce8aea09
Glue everything together
8 years ago
Kegan Dougal
295c9bbb4b
Add 'api' package and move HTTP API requests there
This will make docs easier in addition to allowing the 'database' package
not circular-dep.
8 years ago
Kegan Dougal
56e92a9632
Convert YAML to NEB types via roundabout methods
8 years ago
Kegan Dougal
a7de9a7fdd
Add YAML tags to things so we can unmarshal them. Cannot partially unmarshal json.RawMessage-like though...
8 years ago
Kegan Dougal
bbf89c13bc
Merge branch 'master' into kegan/config-yaml
8 years ago
Kegan Dougal
36562c6f2c
Add YAML parser
8 years ago
Kegan Dougal
f4033d1745
Add stub config file loader
8 years ago
Kegan Dougal
7beaf57ff7
Merge branch 'master' into kegan/config-yaml
8 years ago
Kegan Dougal
e35ea2f27c
Use the NextPollTimestampSecs or else we'll resend on /configureService due to JSON key name change
8 years ago
Kegsay
1cb7a3b565
Merge pull request #94 from matrix-org/kegan/rss-notify-on-failures
Add a flag `IsFailing` to let clients know if their RSS feeds are down
8 years ago
Kegan Dougal
bc1e6283d8
Make some fields 'public'
8 years ago
Kegan Dougal
fae997fdb7
Add a flag `IsFailing` to let clients know if their RSS feeds are down
8 years ago
Kegan Dougal
3c7cf89995
Add a sample config file
8 years ago
Kegan Dougal
299164d525
Fix RSS feeds which do not return GUIDs and do not have published dates
We previously relied on the published date and GUIDs to determine which items
were new. We now only rely on the GUID and not the published date as a lot of
RSS feeds don't have published dates. A lot of feeds don't have GUIDs either,
so we now fallback to the HTTP `Link` field, or worst-case, the item `Title`.
8 years ago
Kegsay
1f47c5a202
Merge pull request #92 from matrix-org/kegan/rss-user-agent
Set "Go-NEB" as the User-Agent on RSS feed polling
8 years ago
Kegan Dougal
5b5de2dc4b
Set "Go-NEB" as the User-Agent on RSS feed polling
Some services, notably Reddit, basically force you to set a custom UA in order
to use their RSS feeds. All the common default non-browser UAs are HEAVILY
rate-limited such that you can't realistically use them.
8 years ago
Kegan Dougal
5c8cc4a81c
strconv to convert int to string
8 years ago
Kegan Dougal
433331ca20
Metrics: Add auth session counter
8 years ago
Kegsay
3f31f63b19
Merge pull request #91 from matrix-org/kegan/neb-metrics
Add metrics to NEB
8 years ago
Kegan Dougal
f493a1933e
s/sendMetric/incrementMetrics/
8 years ago
Kegan Dougal
2e5c8e4d79
Only send the domain part of the RSS feed as that's more useful when doing aggregate metrics
8 years ago
Kegan Dougal
b4b0a661c4
Add more metrics
8 years ago
Kegan Dougal
cdb49fa60b
Add command type as a label
8 years ago
Kegan Dougal
32e8a5b6cc
Docs
8 years ago
Kegan Dougal
19eeb96610
Use enums instead
8 years ago
Kegan Dougal
219da01e08
Add extremely noddy counter metrics for !commands
Missing the command type currently. Also, not sure how "nice" this is: we could
probably blob the 3 functions into 1 and use an enum but, meh?
8 years ago
Kegan Dougal
729b80c6bc
Set max open conns to 1 on sqlite3
8 years ago
Kegsay
e875b9c251
Merge pull request #87 from matrix-org/kegan/metrics
Instrument all HTTP handlers
8 years ago
Kegan Dougal
c9de6eaf06
Instrument all HTTP handlers
8 years ago
Kegan Dougal
d169b5560f
Add prometheus dep for metrics
8 years ago
Kegan Dougal
248822a3d6
Fix #86 - set a 20s min threshold before re-polling
8 years ago
Kegan Dougal
ef6fc0d3bf
Remove useless field
8 years ago
Kegsay
7bcffa4391
Merge pull request #85 from matrix-org/kegan/database-is-locked
Edit README to put `_busy_timeout` to fix database is locked errors (#63 )
8 years ago
Kegsay
d9da3caac4
Merge pull request #84 from matrix-org/kegan/rss-guids
Maintain a list of GUIDs per feed and use them to de-dupe
8 years ago
Kegan Dougal
95521a08f2
Edit README to put `_busy_timeout` to fix database is locked errors ( #63 )
8 years ago
Kegan Dougal
8316ac21ab
Maintain a list of GUIDs per feed and use them to de-dupe
Some RSS feeds will edit the published time of an item AFTER putting it out,
which resulted in RSS Bot sending the same article twice. We now remember the
"GUID" field for each item and de-dupe based on that. Normal timestamp
algorithm still applies.
8 years ago
Kegan Dougal
77b729d4ba
Remove spammy logs
8 years ago
Kegan Dougal
5f015e26ee
Join rooms when registering rss service
8 years ago
Kegan Dougal
b903860af9
Bump min to 5min to help not spam feeds with no cache control
8 years ago
Kegan Dougal
1640728857
Add lrucache
8 years ago
Kegsay
2f0a2c7f10
Merge pull request #83 from matrix-org/kegan/cache-rss-feeds
Add a caching HTTP client
8 years ago
Kegan Dougal
e088e11821
Swap to an LRU cache set to 20MB
8 years ago
Kegan Dougal
e61356431a
Merge branch 'master' into kegan/cache-rss-feeds
8 years ago
Kegan Dougal
7587dbc9c4
Add lrucache
8 years ago
Kegan Dougal
b0eb284347
Add a caching HTTP client
8 years ago
Kegan Dougal
223ccac851
Add HTTP caching library
8 years ago