How to determine Merge confidence for packages which are *intended* to break

[rarkins]

Take prettier for example. From using it for 1-2 years:

1. Quite often the new version doesn't fail/break anything, so we merge
2. Sometimes it breaks things, but I like the change, so we fix it and merge
3. Occasionally I don't like the change, and hold off hoping they'll revert it (only happened once)

Both scenarios (2) and (3) don't mean "bad upgrade!". In scenario (1), I shouldn't really care even if it broke something big like 40% of other repos. In scenario (2), it can be useful to know "it wasn't just me" but it doesn't really change my decision whether it's 2% or 40% of others breaking. For (3), if I see it causes a large percent of repos to fail then it's interesting to know but still doesn't change much.

The same thing goes especially for things like eslint plugins.

[thibaudcolas]

For what it’s worth ESLint has an official semantic versioning policy to document why there is expected breakage and how that fits with semantic versioning, which might be relevant to think of for this.

Prettier is probably the best example still, as when new formatting is introduced it’s likely done as an improvement, even though that means systematically breaking builds 😅 .