Sept 14, 2001 rev. Sept 18, 2001 RMG. Summary of Summer 2001 Debugging of PICNIC Spikes ================================================= Introduction ------------ In June-July 2001 spikes were present in the PICNIC readout, when using the FPGA/VME, of order 100 adu in size, and many per scan of 256 points. The problem appears both with the actual detector presented as input to the ADC and with an external test signal, but not when shorting the ADC input at the ADC card (see RMG data and plots). This problem did not seem to appear in the PICNIC data sampled using the Pentium in April 2001 (see RMG data and plots). It also was never present in the NICMOS3 system. Because of the success of the FPGA/VME system in sampling the grounded ADC with no artifacts, and because of analog weirdness was seen in the ADC board (see WT and EP notes), we shifted from believing that the source of the problem was the FPGA/VME readout to thinking that it could be due to a problem in the PICNIC electronics. Therefore, the power supply and card cage were shipped to Cambridge for debugging over the summer, by EP and RMG. Summer Debugging ---------------- The setup consisted of the PICNIC power supply and card cage containing its three boards (ADC, Supply and Bias) + cables, all exactly same as used on site. We provided a test signal (a few mV) to ADC board input from an external power supply. 1. Analog debugging We discovered that during conversion, the ADC board regulators: (1) spike during conversion, (2) oscillate (tens mV amplitude) between successive conversions, and (3) the DC level output is loaded down (tens mV). These effects may be seen by simply putting a scope probe at VCC and VEE inputs of ADC. However, John Geary was not scandalized by any of these things, particularly the spiking during conversion; apparently these things are normal due to the extra current demands, and ADCs are deigned to deal with them. Nevertheless, we determined that (2) goes away if we eliminate the inductors (L2, L3) in the regulators network, and (3) goes away if we eliminate the 4.7 ohm resistors (R11, R12) in the regulators network. Incidentally, those two are the only components in the regulators network not included in the recommendations on how to wire the ADC (AD676) described in its datasheet. Everybody's opinion is that this is not what is killing us (all our ADC boards do the same, and never bothered the NICMOS3 system), and the connections were left as they were for further debugging, but of course this is something we can still experiment with. 2. Digital debugging We used a logic analyzer to sample the ADC output. ADC conversions were triggered by a TTL from a frequency generator, at 15 KHz. Acquisition of the ADC output by the logic analyzer was triggered by each rising edge of the ADC BUSY signal, and the logic analyzer can store 1000 such 16-bit samples. We discovered that the power supply we were initially using for these tests (which is the same we used in June-July at the site!!, what's its name?) is itself extremely glitchy!! Therefore, many June-July conclusions may be false, and the noise seen in some tests solely due to the test signal itself. We then switched to a good quality HP supply. Eventually our setup was clean enough and we were in a situation such that we had the low noise performance we know the system is capable of: 99.8% of the time (literally) the rms of the digitized samples is in the range 0.5-2 adu (usually about 1 adu). We tested this over the full range of the ADC. In what follows, our method was to do 5 runs of 1000 samples each (5000 samples total), and record every sample above or below about 5 sigma from the mean. 2.1. Small spikes On top of this generally good behaviour there are spikes of order 10 adu, and of frequency about <10 per 5000 samples. By making these measurements using all possible combinations of ADC and ADC board, we determined that the problem seems to travel with the ADC board: spikes are always present when using the board used for the PICNIC system in June-July at the site; and there were exactly 0 spikes every time the board from the NICMOS3 system was used, both facts irrespective of which ADC was used. Note however that these spikes are rare enough, and the procedure tedious enough, that the number of spikes varied greatly from run to run, and it is hard to make definite statements with confidence. We also tested the size of the spikes over the whole range of the ADC, with the result that they were always of similar amplitude. These spikes are much lower in amplitude and frequency than the ones seen at the site in June-July (certainly we would have searched fringes under such circumstances). Therefore, we are treating this as possibly a different problem, the cause of which we still have to figure out. 2.2. Big spikes We tested dependence on humidity, board temperature and running the electronics for long periods of time (all night), finding no effects. However, in the process of using a high-power heat gun to test for temperature, we accidentally discovered that big spikes (of order 100 adu, <10 per 1000 samples) were present if the data was acquired *while* the heat gun was on. The severity of the problem was not related to actual heat or air flow or distance of heat gun from electronics. It does however vary depending exactly on where things are plugged in. We concluded that it is not radiated noise, but noise induced in ground power lines, or something (here we enter the black magic territory of EM compatibility, on which so far nobody has admitted real understanding). Steve Amato was surprised by this, he thought that good power supplies such as the ones we have should not be sensitive to this sort of thing; he also checked and agreed with our wiring of the power supply box & cable shielding); he experimentally demonstrated the immunity of his CCD system to the heat gun. We will also consult with John Geary on this as soon as he returns to CfA. The hypothesis would be that at the IOTA site there is a piece of equipment that is mimicking the heat gun, and inducing noise to which the PICNIC electronics is not immune. Because of the fact that this problem did not exist in the NICMOS3 system, it would have to be a new piece of equipment (the VME rack?) or the combination of a new piece of equipment and some new grounding problem (recall that the spikes were not there in April 2001 either, when I used the PICNIC electronics but sampled using the Pentium). WT procured for us (from central engineering) an isolation transformer and a set of ferrite beads, in order to see whether we could make our system immune to the heat gun noise. The isolation transformer completely eliminated the problem (we were left only with the smaller spikes described above). The ferrite beads inserted on the power lines significantly reduced the magnitude and number of big spikes. Conclusions ----------- 1) It was hard to make the lab setup behave as bad as it did at the site. 2) We should keep an eye on and experiment further with the regulator networks of the ADC board. 3) Perhaps we have 2 separate problems, one intrinsic to the ADC board which causes small spikes and one related to external noise which causes big spikes. 4) It is not 100% clear, unfortunately, that any of the above findings is really what prevented us from operating in June-July 2001, but we do have a few new leads. What Next? ---------- We believe that given what we have found and (mostly) not found in the lab, it would be a waste of time to do any more work on those spikes. We think we are better off working with the real setup at the site. RMG and EP will go to Mt Hopkins ASAP with this as a first priority for the October 2001 engineering run. See RMG notes elsewhere for a detailed plan, but essentially our new weapons will be: (1) make improvements/cleanup of FPGA/VME readout circuit/code (e.g. scan-to-scan display artifacts, modified sync with ADC BUSY signal), (2) use good power supply (borrowed from JG lab) as test signal, (3) use Pentium readout systematically for comparison with FPGA/VME readout, (4) use logic analyzer (same model as that of JG lab is available at IOTA site) to measure noise, which allows to decouple this measurements from the VME rack, (5) experiment with ADC regulators network (inductor and resistor), (6) make detailed schematic and analyze grounding scheme of entire PICNIC system, consult with JG on EM compatibility, (7) experiment with use of isolation transformer (a brand new one is available at IOTA site) and ferrite beads (for high frequency filtering), (8) Mark Shure (CHARA) will measure the noise and spikes on his PICNIC system, which is a replica of ours (so far he has not seen the spike problem we have); he will also compare the behaviour of his ADC regulator networks with ours. Update (Sept. 18, 2001) ------ The fact that we have those big spikes while the noisy heat gun motor is on, and the fact that the problem went away when using the isolation transformer, makes John Geary believe that we have a grounding problem somewhere in the system. RMG and EP made a detailed schematic of the PICNIC system grounding scheme and discussed it carefully with JG. "Unfortunately" he approved all of it. Therefore we next need to verify that the scheme we discussed - based on schematics - actually corresponds to reality (i.e. there is no accidental bad ground connection) which we will do this week at the site. JG also pointed out the most sensitive areas of the system, so we know what/where to look.