|Crews Work Overtime to Restore Beamtime|
Thanks to the ingenuity and dedication of ALS engineering and accelerator physics staff, a significant chunk of beamtime for user operations was saved this week after a transformer for a major power supply failed. The effort involved all ALS trades and electronic specialties, and the resulting fix required over 400 worker-hours in total to design, install, and test. As a result, nearly 50 users with scheduled beamtime this week will be able to run their experiments, with beam settings the same as those used by the ALS a year ago.
At 2 a.m. on Tuesday, April 22, a transformer supplying power to 18 of the ALS's QFA (quadrupole focusing) magnets failed. Prompt investigation suggested that there had been a short from one of the windings to ground, which eventually perforated a water-cooled winding, causing water to leak out onto the floor. The accelerator was shut down. The damaged transformer was removed and expeditiously shipped off to the manufacturer in Los Angeles. However, rebuilding would take a minimum of 2–3 weeks. A major shutdown was scheduled to begin in less than two weeks; if no replacement or fix could be found, there would be no more user shifts for two and a half months, until well into July.
Electrical engineers, led by Chris Pappas, began exploring the installation of a temporary solution for the failed QFA power supply. They looked at the large transformers that were immediately available in storage and came up with three from a recently replaced storage-ring gradient (bend-magnet) power supply. These transformers were not the same functionally, but the team came up with a configuration using all three to emulate the failed transformer. This involved tailoring the output phasing to match what the QFA power supply was expecting.
Ken Baptiste, who was brought in on Friday afternoon to help coordinate the work, ended up helping tremendously with understanding the phasing of the transformers. Mike Fahmie, who was away from the Lab until Thursday morning, was consulted about the plan and offered several critical observations by e-mail before arriving. Once here, he also found a workable connection of the three transformers to get the phasing correct, and with Ken they were successful in understanding the phase shifts used in the end.
It was tedious work of many hours, figuring out from just six outputs how one transformer was configured inside, with a false start caused by a faulty testing instrument. Once that was done, installation was no less tricky. Some of the wires had very close clearances, and moving the heavy cables around on the transformer was not easy. "In one case, to move the three primary wires took about two hours and some skinned knuckles," said Mike.
After extensive efforts, user operations resumed on Sunday, April 27, at 9:00 p.m., with a slightly "fat beam" (larger horizontal beam size) due to the voltage limitations of the replacement transformers. "We were pretty confident that it was going to work, we just didn't know how well it would work," said Chris.
In addition, to Chris, Mike, and Ken, many people were involved in making this happen, including Ken Berg, Pat Casey, Tim Kuneli, the electronics maintenance and electronics installation crews, electricians, and the riggers and plumbers from the mechanical group. "It was definitely a big, all-hands-on-deck effort," said Mike.
Roger Falcone thanked the team, on behalf of ALS users and himself, for their incredible efforts to get the ALS back online over the last week and weekend. "They were dedicated to making this work, and they were extremely creative in finding a safe and successful solution from the limited set of options available on a short timescale," he said.