first pass at exploring infra flakes #448
Conversation
antter
requested review from
durandom,
MichaelClifford and
tumido
as code owners
sesheta
added
the
do-not-merge/work-in-progress
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
sesheta
added
the
size/XL
antter
force-pushed
the
infra_flakes
branch
2 times, most recently
from
d99c734
to
2a16d7e
Compare
antter
changed the title
sesheta
removed
the
do-not-merge/work-in-progress
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
are we sure that the x-axis here are in days? If iirc they were successive builds. I could be wrong here, we should double check.
Reply via ReviewNB
There was a problem hiding this comment.
No you're right I just copy pasted this graph from some other place
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
can we have a count of the NA values? If it's 1% it wouldn't be a problem, if it's 99% we'd have to think this through.
Reply via ReviewNB
There was a problem hiding this comment.
Should the NA value counting go before the array is fillna'ed with zeros?
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
Change the last line to better explain the example. "So if a column has 2 tests that failed, and the first had a row score of 2/5...."
Reply via ReviewNB
There was a problem hiding this comment.
Could you please describe the intuition behind the approach in a bit more detail? E.g. what does sum of entropies signify? why are we looking at entropies of row scores for failed tests?
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
If these outputs are meant to function as a "progress bar", I'd suggest just using tqdm instead
There was a problem hiding this comment.
My tqdm has never worked on the MOC jupyterhub ):
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
How are you bringing the score into the 0-1 range? What does 0 mean and what does 1 mean?
Reply via ReviewNB
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
Can you print some examples of jobs that have a high unexpected failure rate according to this method? And some that are expected failures?
Reply via ReviewNB
There was a problem hiding this comment.
Quite a creative method @antter 👏
Left some comments on the notebook.
| @@ -0,0 +1,973 @@ | |||
| { | |||
There was a problem hiding this comment.
Related Issues and Dependencies
#447
This introduces a breaking change
This Pull Request implements
Infra flakes are likely not detectable in an honest matter with methods by just looking at grid data; more analysis will be needed with different data, e.g. the logs of the outputs. This is a notebook that works to detect anomalous columns in the testgrid data just using very simple probability calculations.