Debian derivatives census: subscription system

[Anastasia Tsikoza]

Hi all, 

As part of the Outreachy project we are going to implement a
subscription system for error notifications produced by derivatives. 

The initial idea is that we add a new field to the census wiki pages so
that anyone interested in the error output of the specific derivative
can add their email address. Then the system will from time to time
save the derivative's error output to a file and send it to those who
have signed up. 

For now we have those errors listed in the check-* and .warnings files
in each derivative's directory on deriv.debian.net. We also plan to add
new scripts for checking each area of the census and producing new
files, for example check-logo and check-wiki-text.

If you are going to sign up for a subscription, please tell us, what
error output you consider useful and whether you would like to receive
the emails regularly or after some specific event (like if the new
check-* file differs from the old). 
Also even if you are not going to subscribe but have any ideas how to
improve either the way of signing up, or the error file format, or the
notification frequency, or the very idea of this feature, your feedback
is very welcome!

Thanks,
Anastasia Tsikoza

[Paul Wise]

Post by Anastasia Tsikoza
If you are going to sign up for a subscription, please tell us, what
error output you consider useful and whether you would like to receive
the emails regularly or after some specific event (like if the new
check-* file differs from the old).

I would like to sign up to notifications of changes to the error
output for all derivatives. Previously we handled that by using a
mailing list but there isn't really an appropriate list here, so
perhaps we can hard-code a list of such subscribers initially and find
a different solution if that proves to be problematic.

[intrigeri]

Hi Anastasia,

Post by Anastasia Tsikoza
As part of the Outreachy project we are going to implement a
subscription system for error notifications produced by derivatives. 

Great! Thanks for working on this :)

Post by Anastasia Tsikoza
If you are going to sign up for a subscription, please tell us, what
error output you consider useful and whether you would like to receive
the emails regularly or after some specific event (like if the new
check-* file differs from the old). 

I would happily subscribe to the Tails errors/warnings if:

- They bring enough information to be actionable (for example, I have
no idea what to do with "mismatched source packages in
Sources/Packages"; which are the faulty packages?). Fixing them all
now in not a prerequisite IMO, as long as someone is ready to fix
them incrementally when reported by subscribers once the
notifications are in place.

- The notifications are sent in a way such as I won't quickly and
unconsciously learn to ignore them all.

I would like to be notified when check results change. A basic
home-made implementation might be OK as a proof of concept but on
the long term, I am pretty sure it'll be insufficient: keeping the
signal/noise ratio high for monitoring checks is not a trivial
problem. Thankfully, we have good monitoring systems available with
the needed features to solve it, e.g. flapping detection and
notifying only after N successive failures. So I humbly suggest
plugging the checks into one such existing monitoring system
instead of writing a new one from scratch.

On top of that, I wouldn't mind a reminder sent 1-4 times a year
about the unchanged, still pending issues; but that's not necessary
if the whole thing is exposed to me via a monitoring system's
web dashboard.

An Lintian-like override mechanism and output ("here are the
non-overriden errors, oh and by the way you're also overriding
N other ones") would be nice but probably vastly overkill at
this stage.

Cheers,

[Paul Wise]

To get an idea of the kinds of warning output that happen, you can
peruse the archives of the old dex-census mailing list. Please note
that these mails do not include the check-* files, which were only
communicated to derivatives by humans so far.

https://alioth-lists-archive.debian.net/pipermail/dex-census/

Post by intrigeri
- They bring enough information to be actionable

For the most part that should be the case, although for example when a
Homepage is down due to server issues, that isn't really actionable
and will resolve itself.

Post by intrigeri
I have no idea what to do with "mismatched source packages in
Sources/Packages"; which are the faulty packages?).

This means that the list of source packages listed in the Sources
files in the apt repository is different to the list of source
packages listed in the Packages files in the apt repository. There is
also a corresponding test for binary packages. The message probably
needs to be updated to mention the diff files. For Tails there are two
thunderbird source packages mentioned in Packages files but only one
in Sources files. This appears to be because you have two versions of
thunderbird-dbgsym, probably the older one should be removed from your
repository.

http://deriv.debian.net/Tails/diff_source_packages
http://deriv.debian.net/Tails/packages_source_packages
http://deriv.debian.net/Tails/sources_source_packages

Post by intrigeri
Fixing them all
now in not a prerequisite IMO, as long as someone is ready to fix
them incrementally when reported by subscribers once the
notifications are in place.

Agreed, some of the checks report low-priority issues.

Post by intrigeri
- The notifications are sent in a way such as I won't quickly and
unconsciously learn to ignore them all.

This is a hard problem to solve :)

Post by intrigeri
I would like to be notified when check results change.

This should be doable using diffs initially.

Post by intrigeri
An Lintian-like override mechanism and output ("here are the
non-overriden errors, oh and by the way you're also overriding
N other ones") would be nice but probably vastly overkill at
this stage.

I think that this should be out of scope for the Outreachy project,
there are higher priority tasks to work on.