The Teamwork and Collaboration at CERN: Machine Operators and Physicists
The collaboration between machine operators and machine physicists at CERN exemplifies a beautiful team effort. The roles, duties, perceptions, and success stories of this partnership are explored through insightful images and discussions. The agenda delves into who machine operators and beam physicists are, their responsibilities, cultural convergence, and recent successful teamwork instances. The diverse sectors and departments at CERN highlight the vital roles played in operations, accelerator physics, technology, finance, HR, research, computing, and international relations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Machine operators and machine physicists (at CERN): a beautiful teamwork Benoit Salvant With input from many members of the Operation (BE-OP) and Accelerators and Beam Physics Groups (BE-ABP)! Abdelouahid Akroh, Gianluigi Arduini, Wolfgang Bartmann, Hannes Bartosik, Stephane Cettour, Karel Cornelis, Paul Collier, Kajetan Fuchsberger, Rossano Giachino, Mike Lamont, Yannick Le Borgne, Anton Lechner, Elias Metral, Mirko Pojer, Stefano Redaelli, Anthony Rey, Giovanni Rumolo, Rende Steerenberg, Rogelio Tomas, Jorg Wenninger. 1
Agenda Who are machine operators and beam physicists? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 2
How operators see themselves How physicists see themselves How physicists (may) see operators How operators (may) see physicists Physicist who wants to change a parameter operator 3
How everyone else sees operators How everyone else sees (accelerator) physicists 4
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 5
CERN sectors Accelerator & technology Finance & HR Research and computing International Relations 6
Departments of the Accelerator & Technology sector Accelerator & technology Finance, HR Research and computing International Relations Technology Engineering Beams 7
Where can one find the operation and beam physics teams at CERN Accelerator & technology Finance, HR Research and computing International Relations Technology Engineering Beams Accelerator & Beam physics Beam Instrumentation Controls Operations RF Industrial Controls and Safety systems 8
Where can one find the operation and beam physics teams at CERN Accelerator & technology Finance, HR Research and computing International Relations Technology Engineering Beams Accelerator & Beam physics 52 staffs 115 students and postdocs Beam Instrumentation Beam physics Controls 83 staffs 12 students and postdocs Operations Operation RF 2 well separated core groups within the same department Industrial Controls and Safety systems 9
Where can one find the operation and beam physics teams at CERN Accelerator & technology Finance, HR Research and computing International Relations Technology Engineering Beams In charge of transfer lines injection, extraction and beam dumping systems 5 staffs + 7 students Beam interaction with matter 5 staffs 10 students Accelerator & Beam physics 52 staffs 115 students/postdocs Engineers with beam physics background Beam Instrumentation Beam physics 2 staffs 9 staffs 2 students Controls Engineers in Charge and physicists with beam physics background Operations 11 staffs 9 students Longitudinal dynamics experts RF Industrial Controls and Safety systems There are beam physicists all over the place! 10
Where can one find the operation and beam physics teams at CERN Accelerator & technology Finance, HR Research and computing International Relations Technology Engineering Beams coordination Coordination Linacs/CTF3/CLEAR operation Accelerator & Beam Physics 6 machine supervisors 21 machine supervisors Coordination Beam Instrumentation 4 machine supervisors Controls Coordination 83 staffs 12 students/postdocs 26 machine supervisors Operations Coordination Linacs/CTF3/CLEAR operation Operation RF 6 machine supervisors Industrial Controls and Safety systems Many groups heavily involved in operations 11
Who are machine operators and physicists at CERN? Strong core teams in well identified groups (ABP and OP) Nevertheless, competencies and roles scattered across many groups Operation thanks to supervision and internal detachments Beam physics thanks to second job for engineer in charge The same person can be an operator at some point and a physicist later that day, depending on who is on shift Direct link preserved between equipment specialists and directly related beam physics RF and injection/extraction Deliberate wish of CERN management to facilitate teamwork and share goals between operation and accelerator physics (as well as with equipment groups). 12
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 13
Main duties Duties of operations team On shift Ensure safety of users (safety first!) Provide steady beams to all users optimization of the performance (through optimization of procedures and operational tools to streamline operation and minimize turn-around and down-time) Responsibility over the hardware protection Duties of machine physicists Not on shift Design particle accelerator Study and understand the beam dynamics in order to optimize performance Conceive operational scenarios and performance estimate Organize and run machine development studies Provide operational support (machine supervision, coordination, commissioning and in case of performance issues) 14
Main duties Duties of operations team On shift Ensure safety of users (safety first!) Provide steady beams to all users optimization of the performance (through optimization of procedures and operational tools to streamline operation and minimize turn-around and down-time) Responsibility over the hardware protection Clear common goal Duties of machine physicists Not on shift Design particle accelerator Study and understand the beam dynamics in order to optimize performance Conceive operational scenarios and performance estimate Organize and run machine development studies Provide operational support (machine supervision, coordination, commissioning and in case of performance issues) 15
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 16
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 17
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 18
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 19
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 20
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 21
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker 22
Needs differ Operations team need Machine physicists need Maximum availability of accelerator equipment State of the art beam diagnostics, testing new hardware Reliable and well established controls and procedures Flexible software and analysis tools (development tools) Safe operation Probing limits and extend the boundaries of operation Stable operations with optimized parameters Performing parameter scans outside of the comfort zone Maximize time for physics Optimize time for tests Calm and concentration in order to be able to respond efficiently to critical signals that could indicate serious problems Discussions that cross-fertilize new ideas. Calm to analyse and summarize the data. Special weapon: interlock Special weapon: tune kicker Quite contradicting needs! 23
Ideal conditions for operators Empty, quiet control room with plenty of space Beam test (MD) time Where are the operators? 24
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 25
Culture differs Operations team Machine physicists Role of the firefighter: - Preparation of the operation - Prevention, risk analysis - React to issues Role of the hunter: - Analyze patterns - Identify a potential victim (parameter) - Act on the beam Formal training and formal responsibility over hardware protection and safety of personnel No formal liability. Responsibility of the machine parameter choice and machine configuration Wide knowledge of all technical aspects of the machine Very deep but focused knowledge of beam effects Industrial environment with quality assurance and quality control Research environment without much constraints (apart from the author list) Heavy constraint of shift work Minor schedule constraint, very flexible Mainly technical degrees, BSc, MSc PhD, post-doc, professors Mainly local French-speaking workforce Mainly global English/Italian-speaking workforce Many reasons for communication issues and frustration on both sides Dedicated time reserved for tests to minimize the impact on stable operation 26
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of recent success stories of teamwork 27
Convergence? We share the same goal to optimize machine performance Beam physicists need operators as they need beam tests to check their models To assess the feasibility and required time for beam tests. To assess the feasibility of implementation of new tools and ideas, and get advice for reliable, safe implementation. To set up and contribute to beam tests to avoid spending an 8-hour beam test on changing settings of a device that is not on . Operators need beam physicists: To identify potential areas of improvement of performance in terms of machine parameters (beam tests) During commissioning and intensity ramp-up, to optimize machine parameters (optics correction, aperture checks, loss analysis, stability needs). For long-term follow-up and analysis of performance In case of unexpected issue that results in performance limitations: To put in place the required non standard diagnostics To observe and make models to reproduce what is happening Clear mutual need, more in next slides 28
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of success stories of teamwork Organized support for short and long term follow-up Beam induced heat loads (arcs and equipment) Luminosity, emittance Instabilities UFOs Availability and faults Common strategy for operational scenario and commissioning Organization of beam tests and performance gain Efficient common response to unexpected issues Requires good information flow: From operation to physicists and back 29
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of success stories of teamwork Organized support for short and long term follow-up Beam induced heat loads (arcs and equipment) Luminosity, emittance Instabilities UFOs Availability and faults Common strategy for operational scenario and commissioning Organization of beam tests and performance gain Efficient common response to unexpected issues 30
Common strategy for operational scenario and commissioning 2 important workshops at CERN with heavy participation from BE-OP, BE-ABP and equipment groups 2 important phases: Evian operational workshop to gather all the information from the past year of running (in December of year Y) Chamonix performance workshop (in January of year Y+1) to prepare the operational scenario for the next year based on all inputs Critical role of sharing: The issues and limitations encountered during the year (BE-OP and BE-ABP and equipment groups) The potential performance gain from changes that could be implemented following beam tests (BE-ABP) The difficulties and risks to implement and commission these changes (BE-OP, BE-ABP, equipment groups) Co-chairing of each session at Evian workshop by 1 operator and 1 physicist Demanding schedule with shorter winter shutdowns Allows providing the arguments to the management for optimizing the operational scenario 31
Output of Evian 2017 Lessons are learnt together Goals are set together Clear signal that one group can not succeed without the other 32
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of success stories of teamwork Organized support for short and long term follow-up Beam induced heat loads (arcs and equipment) Luminosity, emittance Instabilities UFOs Availability and faults Common strategy for operational scenario and commissioning Organization of beam tests and performance gain Efficient common response to unexpected issues 33
Examples of beam tests that have allowed performance gain Use of coupling and transverse damper in the PS intensity increase Novel beam production in PS (BCMS) emittance reduction Novel multi-turn extraction between PS and SPS (MTE) reduce beam losses Batch spacing reduction in SPS and LHC increase number of bunches Optics with new integer tune in SPS push stability limits Full detuning mode for LHC cavities RF power gain Luminosity levelling in LHC maximize luminosity along the fill Offset levelling Crossing angle anti-leveling Beta* leveling Bunch flattening Combined ramp and squeeze and faster ramp gain of cycle time Coupling and chromaticity correction restore stability margins Telescopic squeeze increase luminosity Novel ideas with significant impact on performance, Successfully implemented thanks to: Common discussions between beam physics, operations and equipment groups Impeccable organization of beam tests (in injectors and LHC) Countless sessions of testing with beam (so-called machine development: MD). 34
Common workshop to maximize the benefit of beam tests for operation, beam physics and equipment groups 35
Common workshop to maximize the benefit of beam tests for operation, beam physics and equipment groups https://indico.cern.ch/event/609486/ 36
Giovanni Rumolo and Hannes Bartosiks wrap-up after 42 talks Some of the ideas that emerged: - Feedback from new web-based beam test planning tool - More communication needed before, during and after the tests - Relax test program when there are other heavy activities for the OP crew - A lot of practical suggestions that ease the life of operators 37
Some of many examples of beam tests that allowed/will allow performance gain Jan Uythoven 38
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of success stories of teamwork Organized support for short and long term follow-up Beam induced heat loads (arcs and equipment) Luminosity, emittance Instabilities UFOs Availability and faults Common strategy for operational scenario and commissioning Organization of beam tests and performance gain Efficient common response to unexpected issues 39
Unexpected repeated beam dumps in LHC cell 16L2 Air inlet at cell 16L2 during technical stop with magnets at cryogenic temperature air species froze on the surfaces solid flakes fall into the beam when beam hits the flakes, generates a dense plasma of ions and electrons electrons excite the beam very fast transverse instability Fast beam abort (60 in 2017) and 1 quench No prior knowledge that there could have been such an air inlet Very complex mechanism had neither been predicted nor observed Very few observables (initially only losses) Many exotic theories (everyone at CERN became a beam physicist at that moment) Need to observe more and test more to understand more: Implementation of non-standard dedicated diagnostics and triggers Ad-hoc crisis meetings with operation, equipment experts and physicists Series of non-standard tests to identify correlations 40
16L2: lessons learnt In such non trivial cases, the ability to use non-standard diagnostics, triggers and filling schemes (already used for MDs) was key It is important for operation crews to be aware that these tools are there and could be used if needed Sometimes it pays off to be bold, but one should analyze carefully the risks. in such troubled times, the role of the operator is critical: make sure that these risks are well accounted for. Everyone at CERN rolled up their sleeves, and looked together for a solution The pre-existing network and teamwork between operation, physicists and equipment groups on other topics allowed for a very fast coordinated response. With time, what can go wrong, will go wrong 41
Agenda Who are machine operators and beam physicists at CERN? Operations and accelerator physicists Duties Needs Culture Convergence? Examples of success stories of teamwork Support for short and long term follow-up 16L2 issue mitigation Example of beam tests that allowed performance gain Outlook 42
Outlook: Proposed guideline for physicist driving a beam test Identify the shift that will be operating during the test Discuss and explain goals ahead (or at least at the beginning) Ask if there could be limitations or suggestions to avoid potential problems or to improve the situation (instrumentation, procedures) Do not leave unexperienced colleagues alone, do not come with too many people Assign a clear contact person from physicist team for the operation shifters to relate to for decisions and strategy And after the MD: Give feedback on how it went Ask for feedback on how it went Give credit to the operation shift in the related talks and presentations Keep the operation shift updated on the status of the studies Bringing back home to my group 43
Outlook: what could the operator gain by interacting more with physicists Better understanding of the importance of certain parameters and their follow-up More prompt reaction to unexpected issues thanks to this better understanding of the critical measurements and parameters Knowledge beyond routine operation which is very useful for troubleshooting, setting up new beams, commissioning new cycles. Use of non-standard diagnostics or analysis, which is not used for routine operation (e.g. bunch by bunch diagnostics). A preview of the new projects, new features, new ideas that can/will be implemented soon in operation and an opportunity to act and contribute more flexibility and more autonomy the same remarks apply to my opinion for the interaction of operators with equipment specialists 44
Concluding remarks Very efficient and exciting teamwork with operation at CERN Significant improvement of machine performance Structure put in place allows motivation, flexibility and resilience in case of issues Close teamwork can lead to frustration and conflict. Management needs to be very careful that the crosstalk does not perturb the core duties of the teams, both teams are properly respected and acknowledged for their contributions. The recipes are not universal: all this depends on the environment Dedicated professional operational team vs non professional Needs mechanisms to reward operator involvement beyond pure shiftwork, and to have physicists not to take for granted that it all works Needs the ability to maintain and transmit experience Management choice: do you choose your operation team to be following procedures or do you want to promote initiatives As for any relationship, successful teamwork does not come for free and requires additional time and effort on both sides. one needs time and one needs more time to be open to new ideas 45
Beam tests (MD) organization at CERN In LHC: Slow ramping machines Most beam tests require dedicated time (~20 days/year) Some tests/developments can be performed parasitically In injectors: Fast ramping machines Most beam tests performed in parallel cycles, that can be scheduled almost all the time Some beam tests that require special non-ppm hardware or settings are planned on dedicated time (once a week in 2017, twice a week in 2018) 48
Procedure for MD request: LHC Up to 6 weeks in advance of MD slot: MD request on asm.cern.ch 1 month in advance: 5 presentation at LHC study working group of the plans, where physicists and operators are invited 3 weeks in advance: Preparation of MD procedure, in view of decision on machine protection Assignment of an engineer in charge for liaison Discussion at the facilities operations meeting of new beam requests 1 week in advance: Presentation of planning at LHC machine committee 2 weeks after the MD: Presentation of results 1 to 2 months after the MD: Write up MD note 49
