Dear all,
Clem, thanks for this excellent summary. For those who have not heard
all the calls, I wanted to add my concurrence to this summary and to the
proposed actions which we've discussed extensively now on- and off-line.
But we should solicit full comment from the collaboration at this
point--please weigh in.
As an aside, although Clem describes this situation as unfortunate it
strikes me how much I would rather be in our shoes--having an experiment
that concentrates sensitivity enough to identify and study anomalies at
relatively high S/N when they rise to a level that could impact our
result--than an experiment that spreads sensitivity across more sky
where to reach similar levels we'd have to assure ourselves that we can
rule out effects that are tiny compared to our noise. The recent
efforts to understand the false signal by isolating the inconsistent
modes are encouraging and should continue to be pushed. Until we reach
such understanding, we have an upper-limits-only experiment, so we must
push on.
Regarding our policy on looking at results, some have wanted
clarification so let me lay out how I see our general approach:
We do deliberate, cautious, coordinated analysis. We proceed at each
step of the analysis in a deliberate way, asking as a collaboration
whether looking at the answer from our actual data could prejudice
analysis choices not yet made. We do not do "blind analysis". Some
physicists have very specific ideas about what blind analysis means and
I personally agree with those who say that trying to rigidly map that
framework onto CMB analysis makes you *more* likely to miss real
problems in your data and so screw up. We should probably avoid casual
use of loaded terms like "unblinding" and "opening the
box"--let's all
attempt to be more disciplined about that going forward. When we choose
to calculate a specific statistic on the actual non-jack data, that is
how we should describe it. The key to our ethics is that we are
deliberate in proceeding on those steps that involve looking at answers.
For the present case, I would emphasize that we do fail an internal
consistency test--BICEP2 vs. Keck--a powerful test but a harder one to
work with than our usual jacks. As Clem says, for 4 months now we have
searched without success to find a BICEP2-only jackknife that fails as
obviously. Our motivation for doing this, without first looking at the
BICEP2 B bandpowers, was to preserve prospects for a quick BICEP2-only
path to a result we can believe to be "clean". But after 4 months, I
feel that now even if we had the a-ha insight into the origin of this
false-B next week and could propose a fix, we ourselves probably would
not be convinced any remaining B signal in BICEP2 is celestial without
invoking consistency tests with Keck data.
As we decide now to take the step of directly comparing power in jack0
BICEP2 and Keck B-modes (we have been visually comparing their patterns
for some time), I'd say (1) that the path forward to understanding this
effect requires that we use all the information available to us, (2)
that once we have such understanding, trust in our post-fix data being
clean is probably going to have to involve checking consistency between
BICEP2 and Keck, (3) that we are interested in the short term in seeing
what immediate upper limits on the combination of actual B signal and
correlated noise/systematics the combination of BICEP2 and Keck data can
give us. Any correlation will be a valuable diagnostic; reliable upper
limits will be a valuable result.
John
On 3/26/13 5:29 PM, Clem Pryke wrote:
Dear all,
On the pipeline telecon today we reached some decisions which the whole group
should be given an opportunity to comment on.
Background:
Posting like this one:
http://bicep.caltech.edu/~spuder/analysis_logbook/analysis/20121120_sim1234…
showed that B2 2yr data shows no strong jackknife failures in E and B spectra.
We therefore looked at the real BB 2d power spectrum where we saw "lots of
power in excess of what signal+noise sims show". (One can see this in the
"maps pager" of the above link.)
We would regard this signal as real (cosmology or foreground) except that it
is smaller and/or different in each of the 2012 Keck single rx maps. See this
pager:
http://bicep.caltech.edu/~spuder/analysis_logbook/analysis/20130226_b2_keck…
k
We have searched without success to find a jackknife which fails - for which
the contamination appears to be significantly different in the two halves of
the split. We have also searched for a systematic which might cause this
signal by simulating candidates and making trials of deprojecting them -
without success so far.
We find ourselves in an unfortunate situation. Without understanding where the
false signal comes from any result we can pull out of any of our receivers can
honestly be treated as an upper limit only. We therefore must continue
pursuing such an understanding. However we are also strongly motivated to
publish something asap - at this point we must consider such an upper limit
only publication. There are thorny "political" issues as to which is the least
bad option for such a publication.
Proposed actions:
- We have deliberately held off from answering the question "What is the r
value corresponding to our nominal B2 detection and what is it's
significance?". We propose to now do so. Chris will add the real BB points to
the "chi2 pager" in the first link above and compare to r=0.1 sims.
- We will take simple cross spectra between the B2 3yr and 2012 Keck single rx
maps. In so much as these are unbiased by correlated noise or systematic
contamination they will scatter about zero. This is a speculative first step
which might lead to an alternate path to publication.
- We will run conventional sims identical to those already run for B2 2yr for
2012 Keck rx0 only. This receiver shows no strong nominal signal and can be
expected to yield a non-detection with a sensitivity a few times worse than
B2. This might be a possible path to a "stand alone" although rather lame
paper.
--
___________________________________________________________________
John Kovac jmkovac(a)cfa.harvard.edu
Assistant Professor, Astronomy and Physics, Harvard University
160 Concord Ave rm 310, Cambridge MA 02138, 617-496-0611