TableNumber and ImageNumber discrepancy

[niranjchandrasekaran]

@Arkkienkeli asked

I found that TableNumber and ImageNumber columns are used differently across sources,
that might be critical if those are used elsewhere for selection.

All below for one arbitrary plate in the source.

For source 3:
sqlite> select count(distinct TableNumber) from Nuclei;
384
sqlite> select count(distinct ImageNumber) from Nuclei;
9

For source 4:
sqlite> select count(distinct TableNumber) from Nuclei;
3416
sqlite> select count(distinct ImageNumber) from Nuclei;
3416

For source 2:
sqlite> select count(distinct TableNumber) from Nuclei;
1
sqlite> select count(distinct ImageNumber) from Nuclei;
2302

I believe that both TableNumber and ImageNumber should be unique for image as in source 4, though source 2 and 3 don't seem to follow this convention.
I noticed that because I use John Arevalo's script for location extraction (which uses join on TableNumber field), which usually works well on different datasets but fails on source 3.
The safe choice seems to use both TableNumber and ImageNumber in queries (if the distinction per image is needed).

[niranjchandrasekaran]

tagging in @bethac07 and @shntnu

[bethac07]

I'm not surprised to see the source3 setup - it's what you might expect to see in certain ways of setting up CellProfiler (that's WHY we add TableNumber, because if ImageNumber was always unique, we wouldn't need it!) - I can go into more detail if you're curious but it's not critical to understand here.

I'm not sure why source2 looks the way it does, but as long as between the two columns we can always figure out which is which, then I'm fine with it :)

[shntnu]

I've moved this issue to the repo we are using to track our collaboration with Mike Ando + Verily
@Arkkienkeli - you might find this issue useful to skim #5

I'll try to peek back in later

[shntnu]

I'll close this in favor of #5