Dreams and Nightmares

March 29: My first night actually observing with DECam has the almost unreal quality of a dream come true. While I’ve done equally exciting science with even bigger telescopes in the past, this project is more fully my own than any I have attempted before with such a powerful telescope. Tonight, given good weather and working equipment, I am going to image asteroids fainter than any human has ever measured before. And I have good weather and working equipment.

 Good weather and working equipment.  World-class telescopes such as the Blanco and instruments like DECam are painstakingly maintained by engineers and technicians at least as skilled as the astronomers who use them – but even so, they are complex, one-of-a-kind systems.  A million different things can go wrong, even under a perfect sky.  I have seen many bizarre failures in the twelve years since I did my first astronomical research at a major observatory. I cannot forget the vast number of things that all have to work together for any night’s observations to be a success. Even though the alignment of good weather with perfectly functioning instruments is the norm at a fine observatory such as this one, it feels like a miracle every time. I thank God, and I honor the human genius and the diligent work by many that makes my work possible.

 As I prepare to begin tonight’s observations, I post on Facebook the cryptic status update I decided on weeks ago if this happened: “Tonight is the night when I cup my hands, and God fills them full of asteroids.”

 The data are coming in, one beautiful image after another.  The sky is clear, and the seeing is good.  Pessimistic and skeptical person than I am, I have a hard time believing this is actually happening.  I am still looking for something I might be doing wrong, something that I have to fix in order for this night really to be as good as it seems.  But my plans are meticulously laid, and they are unfolding beautifully.  God is pouring the asteroid-rich data into my lap. The reality is sinking in.  I am smiling a lot.

 While DECam is flawlessly executing my observing scripts, I have time to work on the huge challenge of processing this data. I started weeks ago, downloading publically available example images from DECam and teaching myself how to work with them.  DECam images are huge.  After processing into the format I need for asteroid searches, each image is 3.5 GB.  I have two 4 TB backup hard drives, but even this is barely enough.  I am continually running into problems doing things that are easy with smaller images. So far I have always been able to write more computer code to solve them.  Over the next few weeks, I will be racing against time to accurately measure the brightest asteroids in the data before it is too late for other observers to follow them up. Without follow-up observations, their positions will soon become uncertain.  Future observers will not know where to point their telescopes to see my asteroids, and the asteroids will join the thousands of objects that have been seen and followed over one night or a few nights but then lost.  This is not a disaster scientifically: my main objective is to study the statistics of extremely faint asteroids, and I can do that perfectly well with objects that later become lost.  It is unsatisfying, however, not to make full discoveries of at least a few asteroids when the data are this good.  I am determined to do it if I can.

 But it is also necessary to enjoy the moments of observing.  I look around the control room in leisure moments near midnight.  It is a large, well-lit room with a white linoleum floor. Three sets of doors open out from it at somewhat odd angles, giving it the shape not of rectangle but of a round-ish irregular polygon.  One set leads to the exit from the observatory, another set to the refrigerator and rudimentary kitchen area, and a third to a computer room where mere astronomers like me are not allowed – a room that is unexpectedly about to become extremely important.  The actual observing floor is several stories above us, where the huge telescope moves slowly in the starlight, responding with magnificent precision to our computerized commands.

 There are three major workstations in the control room, each with six to eight widescreen monitors stacked in double rows.  I am at the observer’s station.  My monitors are filled with more status information than I can possibly process mentally.  Fortunately, as long as everything is going smoothly, most of it doesn’t matter at all.  Fifteen feet to my right is the telescope operator’s station, similarly well supplied with data displays I suspect are as complex as those in the cockpit of a Space Shuttle.  He has to understand his displays much better than I do mine.  My job is only to make this telescope do excellent science, provided it is working.  His job is to make it work. There is also a man in the room whose official title I forget, but who is effectively an instrument specialist.  He is the one I can appeal to for immediate rescue if something seems to be going wrong with DECam.  He doesn’t have his own workstation, but will help me with mine if something happens that I don’t understand or can’t fix.  At present, thankfully, he has nothing to do. The telescope operator and instrument specialist are both experts in complex systems of which I have only the most basic understanding.  I have enormous respect for them, as for their counterparts at the other observatories I have used throughout the world.

 There is a third workstation which I have the privilege of using if I want.  It is for quick-look processing of DECam data.  Sometimes I display an image on it, to check the starfield and admire the handful of stunning spiral galaxies I am capturing without even meaning to.  I also use it now and then to evaluate the seeing – that is, the extent to which turbulence in the earth’s atmosphere is blurring my images. I think the seeing is very good tonight.  The images are blurred only a little more than one arcsecond, which is about the size of a bacterium held at arm’s length.  I comment on this aloud.  “We can do better,” says the instrument specialist.  He doesn’t mean that he or anyone has control of the atmospheric turbulence, but rather that this excellent site in the desolate foothills of the Andes sometimes produces even better seeing than we have tonight. I hope that tomorrow night we will indeed do even better.

 At 1:30 AM DECam abruptly stops working. I and the instrument specialist try to fix it. We restart some components of the control software.  Then we restart the whole thing. It doesn’t work.

 We call for reinforcements.  The experts work on the system, trying things I don’t understand, ruling out possible causes of the problem one by one.  3 AM. Outside the  southern Milky Way shines magnificently in a perfect, moonless sky above our useless telescope. I feel physically sick, breathless with the value of what I am losing, moment by moment.

 One does not complain in such situations. It will not help the experts frantically working to solve the problem.  Everyone knows that observing is a gamble that can be lost, and one does not whine about losing however great the stakes.

 One can pray, though.  I would like to pray aloud, as much to affirm to myself that God is in control as to make any request of him. I am afraid of distracting those working on the problem, however, so I kneel silently a few feet behind the observer’s workstation.

 The problem has been traced now to a serious hardware failure: one of the computers behind the forbidden door has crashed.  Restarting it could fix the problem, but would more likely create a cascade of other issues, as the effects of the restart propagated through the interconnected systems.  A senior member of the observatory staff is on the way.  He may give us the go ahead to restart the computer.  He arrives at about 3:45.  Internally, I have already given up the rest of the night for lost.

 But it isn’t! The new system expert finds a workaround for the ailing computer, and we are back online at 4 AM! Chile lies far west in its time zone, so sunrise will be after eight.  There are three more usable hours in the night, and the total time lost was only two and a half hours. Once again, I thank God, and I silently bless the technicians who work around the clock to keep the world’s most powerful telescopes running. 

 The night is no longer perfectly clear, but the clouds stay away from my primary science field until it is getting too low to observe anyway.  The loss of 2.5 hours of perfect weather remains painful, but the night’s data set is excellent.  In the last hour before dawn, with my primary science field setting in the west, I point the telescope instead to the positions of two very small near-earth asteroids that recently made a close pass by Earth and are now receding into the distance without having been observed by anyone.  If the images I am taking do show them, I will have helped save them from being lost.

 No man can hinder the dawn, and it is a fitting end even to the most precious night. The sky brightness in my images rises abruptly, the stars fade into the noise, and I tell the telescope operator I am finished observing and ready to close the dome.

 In my notebook I write down the parameters of the last exposure, including the sky background value that showed me the night’s observations were certainly over. Let there be no doubt that I used this night down to the last dregs. Underneath the data on the last exposure, I write in my notebook, “So ends a stomach-churning nail-biter of a night, which nevertheless turns out to be basically successful. Thank you, my Lord!”