.. _manualpreamble: ********************************************************** Preamble ********************************************************** See the :ref:`Useful links ` page for official resources, including the website and the GitHub repository. An introduction to the code and its capabilities is available in the talk: `Pencil Code Introduction (2007) `_. Contributors list and GitHub stats ================================== .. figure:: figs/cvsstat.png :align: center :alt: Check-in patterns as a function of time for different subroutines. The different users are marked by different symbols and different colors. Check-in patterns as a function of time for different subroutines. The different users are marked by different symbols and different colors. .. figure:: figs/GitHub-ci.png :align: center :alt: GitHub check-in pattern since 2002 GitHub check-in pattern since 2002 Contributors to the code: ========================= The up-to-date list of Pencil Code contributors can be found at `GitHub `_. (in inverse alphabetical order according to their user name) .. list-table:: **Contributors to the Pencil Code** :header-rows: 1 :widths: 20 30 50 * - **Username** - **Name** - **Affiliation** * - xiang-yu - `Xiang-Yu Li `__ - Pacific Northwest National Laboratory * - wladimir.lyra - `Wladimir Lyra `__ - New Mexico State Univ. * - weezy - `S. Louise Wilkin `__ - University of Newcastle * - wdobler - `Wolfgang Dobler `__ - Bruker, Potsdam * - vpariev - `Vladimir Pariev `__ - University of Rochester * - torkel - `Ulf Torkelsson `__ - Chalmers University * - tavo-buk - `Gustavo Guerrero `__ - Univ. Minas Gerais * - tgastine - `Thomas Gastine `__ - MPI for Solar System Research * - tobson, theine - `Tobias (Tobi) Heinemann `__ - NBIA, Copenhagen * - tarek - `Tarek A. Yousef `__ - University of Trondheim * - sven.bingert - `Sven Bingert `__ - Gesellschaft für wissenschaftliche Datenverarbeitung * - steveb - `Steve Berukoff `__ - UCLA * - asnodin - `Andrew Snodin `__ - University of Newcastle * - rplasson - `Raphael Plasson `__ - Avignon Université * - qiancg - Chengeng Qian - Beijing Inst. of Technology * - pkapyla - `Petri Käpylä `__ - University of Göttingen * - onlymee - `Antony (tOnY) Mee `__ - Bank of America Merrill Lynch, London * - nishkpph - Nishant K. Singh - IUCAA * - nils.e.haugen - `Nils Erland L. Haugen `__ - SINTEF, Trondheim * - NBabkovskaia - `Natalia Babkovskaia `__ - University of Helsinki * - mrheinhardt - `Matthias Rheinhardt `__ - Aalto University, Espoo * - mppiyali - `Piyali Chatterjee `__ - Bangalore * - mkorpi - `Maarit J. Korpi-Lagg (née Korpi, Mantere, Käpylä) `__ - Aalto University * - miikkavaisala - `Miikka Väisälä `__ - Academia Sinica, Inst. Astron. & Astro. * - michiellambrechts - `Michiel Lambrechts `__ - Lund Observatory * - Mewek - Ewa Karchniwy - Silesian University of Technology * - mcmillan - `David McMillan `__ - York University, Toronto * - mattias - `Mattias Christensson `__ - formerly at Nordita * - luizfelippesr - Luiz Felippe S. Rodrigues - Radboud University * - koenkemel - `Koen Kemel `__ - Nordita, Stockholm * - karlsson - `Torgny Karlsson `__ - Nordita * - jwarne - `Jörn Warnecke `__ - MPS, Göttingen * - jpekkila - Johannes Pekkilä - Aalto University * - jnskrueger - `Jonas Krueger `__ - Trondheim * - jaarnes - `Jørgen Aarnes `__ - Trondheim * - joishi - `Jeff S. Oishi `__ - Bates College * - JenSchober - `Jennifer Schober `__ - EPFL, Lausanne * - Illarl - Illa R. Losada - . * - Iomsn1 - `Simon Candelaresi `__ - University of Glasgow * - grsarson - `Graeme R. Sarson `__ - University of Newcastle * - fredgent - `Frederick Gent `__ - Aalto University, Espoo * - fadiesis - `Fabio Del Sordo `__ - Nordita, Stockholm * - dorch - `Bertil Dorch `__ - University of Copenhagen * - bdintrans - `Boris Dintrans `__ - Observatoire Midi-Pyrénées, Toulouse * - dhrubaditya - `Dhrubaditya Mitra `__ - Nordita, Stockholm * - colinmcnally - Colin McNally - NBIA, Copenhagen * - ChristerSandin - `Christer Sandin `__ - Nordita * - chaochinyang - `Chao-Chin Yang `__ - The University of Alabama * - Bourdin.KIS - `Philippe Bourdin `__ - Space Research Institute, Graz * - AxelBrandenburg - `Axel Brandenburg `__ - Nordita, Stockholm * - apichat - Apichat Neamvonk - University of Newcastle * - amjed - `Amjed Mohammed `__ - University of Oldenburg * - alihyder727 - Ali Hyder - New Mexico State Univ. * - alexanderhubbard - `Alex Hubbard `__ - American Museum of Natural History * - andreas-schreiber - `Andreas Schreiber `__ - MPI Heidelberg * - ajohan - `Anders Johansen `__ - GLOBE Institute, Copenhagen University Copyright and License ===================== .. rubric:: Copyright and License Copyright © 2001--2023 Wolfgang Dobler & Axel Brandenburg Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions under the same conditions. .. rubric:: License agreement and giving credit The content of all files under :file:`pserver:$USER@svn.nordita.org:/var/cvs/brandenb` are under the GNU General Public License (`GPL `_). We, the **Pencil Code** community, ask that in publications and presentations the use of the code (or parts of it) be acknowledged with reference to [JOSS]_ Pencil Code Collaboration, *J. Open Source Software*, **6**, 2807 (2021) The Pencil Code, a modular MPI code for partial differential equations and particles: multipurpose and multiuser-maintained. This automatically gives a reference to the web sites `http://www.nordita.org/software/pencil-code/ `_ and `https://github.com/pencil-code/pencil-code `_. As a courtesy to the people involved in developing particularly important parts of the program (use ``svn annotate src/*.f90`` or ``git blame -c src/filename.f90`` to find out who did what!) we suggest to give appropriate reference to one or several of the following (or other appropriate) papers (listed here in temporal order): - Dobler, W., Haugen, N. E. L., Yousef, T. A., & Brandenburg, A., |yprN|, 2003, E 68, 026304, 1-8, "Bottleneck effect in three-dimensional turbulence simulations", |astroph|{0303324} - Haugen, N. E. L., Brandenburg, A., & Dobler, W., |yapjlS|, 2003, 597, L141, L144, "Is nonhelical hydromagnetic turbulence peaked at small scales?", (`astroph/0303372 `__) - Brandenburg, A., K\"apyl\"a, P., & Mohammed, A., |ypf|, 2004, 16, 1020, 1027, "Non-Fickian diffusion and tau-approximation from numerical turbulence", |astroph| {0306521} - Johansen, A., Andersen, A. C., & Brandenburg, A., |yana|, 2004, 417, 361, 371, "Simulations of dust-trapping vortices in protoplanetary discs", |astroph| {0310059} - Haugen, N. E. L., Brandenburg, A., & Mee, A. J., |ymn|, 2004, 353, 947, 952, "Mach number dependence of the onset of dynamo action", |astroph|{0405453} - Brandenburg, A., & Multam\"aki, T., |yjourS|, 2004, Int. J. Astrobiol., 3, 209, 219, "How long can left and right handed life forms coexist?", |qbio| {0407008} - McMillan, D. G., & Sarson, G. R., |ypepi|, 2005, 153, 124, 135, "Dynamo simulations in a spherical shell of ideal gas using a high-order Cartesian magnetohydrodynamics code" - Heinemann, T., Dobler, W., Nordlund, Å., & Brandenburg, A., |yana|, 2006, 448, 731, 737, "Radiative transfer in decomposed domains", |astroph| {0503510} - Dobler, W., Stix, M., & Brandenburg, A., |yapj|, 2006, 638, 336, 347, "Convection and magnetic field generation in fully convective spheres", |astroph|{0410645} - Snodin, A. P., Brandenburg, A., Mee, A. J., & Shukurov, A., |ymn|, 2006, 373, 643, 652, "Simulating field-aligned diffusion of a cosmic ray gas", |astroph| {0507176} - Johansen, A., Klahr, H., & Henning, T., |yapj|, 2006, 636, 1121, 1134, "Dust sedimentation and self-sustained Kelvin-Helmholtz turbulence in protoplanetary disc mid-planes", |astroph| {0512272} - de Val-Borro, M. and 22 coauthors (incl. Lyra, W.), |ymn|, 2006, 370, 529, 558, "A comparative study of disc-planet interaction", |astroph|{0605237} - Johansen, A., Oishi, J. S., Mac Low, M. M., Klahr, H., Henning, T., & Youdin, A., |ynat|, 2007, 448, 1022, 1025, "Rapid planetesimal formation in turbulent circumstellar disks", |arXiv| {0708.3890} - Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., |yana|, 2008, 479, 883, 901, "Global magnetohydrodynamical models of turbulence in protoplanetary disks I. A cylindrical potential on a Cartesian grid and transport of solids", |arXiv|{0705.4090} - Brandenburg, A., Rädler, K.-H., Rheinhardt, M., & K\"apyl\"a, P. J., |yapj|, 2008, 676, 740, 751, "Magnetic diffusivity tensor and dynamo effects in rotating and shearing turbulence", |arXiv| {0710.4059} - Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., |yana|, 2008, 491, L41, L44, "Embryos grown in the dead zone. Assembling the first protoplanetary cores in low-mass selfgravitating circumstellar disks of gas and solids" - Lyra, W., Johansen, A., Klahr, H., & Piskunov, N., |yana|, 2009, 493, 1125, 1139, "Standing on the shoulders of giants. Trojan Earths and vortex trapping in low-mass selfgravitating protoplanetary disks of gas and solids" - Lyra, W., Johansen, A., Zsom, A., Klahr, H., & Piskunov, N., |yana|, 2009, 497, 869, 888, "Planet formation bursts at the borders of the dead zone in 2D numerical simulations of circumstellar disks", |arXiv| {0901.1638} - Mitra, D., Tavakol, R., Brandenburg, A., & Moss, D., |yapj|, 2009, 697, 923, 933, "Turbulent dynamos in spherical shell segments of varying geometrical extent", |arXiv| {0812.3106} - Haugen, N. E. L., & Kragset, S., |yjfm|, 2010, 661, 239, 261, "Particle impaction on a cylinder in a crossflow as function of Stokes and Reynolds numbers" - Rheinhardt, M., & Brandenburg, A., |yanaN|, 2010, 520, A28, "Test-field method for mean-field coefficients with MHD background", |arXiv| {1004.0689} - Babkovskaia, N., Haugen, N. E. L., Brandenburg, A., |yjour|, 2011, J. Comp. Phys., 230, 1, 12, "A high-order public domain code for direct numerical simulations of turbulent combustion", |arXiv| {1005.5301} - Johansen, A., Klahr, H., & Henning, Th., |yanaN|, 2011, 529, A62, "High-resolution simulations of planetesimal formation in turbulent protoplanetary discs" - Johansen, A., Youdin, A. N., & Lithwick, Y., |yanaN|, 2012, 537, A125, "Adding particle collisions to the formation of asteroids and Kuiper belt objects via streaming instabilities" - Lyra, W. & Kuchner, W., |ynat|, 2013, 499, 184, 187, "Formation of sharp eccentric rings in debris disks with gas but without planets" - Yang, C.-C., & Johansen, A., |yapjs|, 2016, 224, 39, "Integration of Particle-Gas Systems with Stiff Mutual Drag Interaction" - Roper Pol, A., Brandenburg, A., Kahniashvili, T., Kosowsky, A., & Mandal, S., |ygafd|, 2020, 114, 130, 161, "The timestep constraint in solving the gravitational wave equations sourced by hydromagnetic turbulence" This list is not always up-to-date. We therefore ask the developers to check in new relevant papers, avoiding however redundancies. We are aware of the fact that certain extensions to the code may still be under intense development and no paper can be quoted yet. Again, if your work directly profits from such code, as a courtesy to those developers, we suggest to contact them, if possible, and ask whether there is anything else that can be quoted instead. It is also sometimes nice to see that the **Pencil Code** is being acknowledged for having *inspired* certain other developments, so for example in the ``GALPROP`` program [Porter22]_. Foreword ======== This code was originally developed at the Turbulence Summer School of the Helmholtz Institute in Potsdam (2001). While some SPH and PPM codes for hydrodynamics and magnetohydrodynamics were publicly available, this did not seem to be generally the case for higher order finite-difference or spectral codes. This has changed since 2001; examples are the SpECTRE code, which is a discontinuous Galerkin code, and there are also the Snoopy and Dedalus codes, which are spectral. Having been approached by people interested in using our code, we decided to make it as flexible as possible and as user-friendly as seems reasonable, and to put it onto a public CVS repository. Since 21 September 2008 it is distributed via `https://github.com/pencil-code/pencil-code `_. The code can certainly not be treated as a black box (no code can), and in order to solve a new problem in an optimal way, users will need to find their own optimal set of parameters. In particular, you need to be careful in choosing the right values of viscosity, magnetic diffusivity, and radiative conductivity. The **Pencil Code** is primarily designed to deal with weakly compressible turbulent flows, which is why we use high-order first and second derivatives. To achieve good parallelization, we use explicit (as opposed to compact) finite differences. Typical scientific targets include driven MHD turbulence in a periodic box, convection in a slab with non-periodic upper and lower boundaries, a convective star embedded in a fully nonperiodic box, accretion disc turbulence in the shearing sheet approximation, etc. Furthermore, nonlocal radiation transport, inertial particles, dust coagulation, self-gravity, chemical reaction networks, and several other physical components are installed, but this number increases steadily. In addition to Cartesian coordinates, the code can also deal with spherical and cylindrical polar coordinates. Magnetic fields are implemented in terms of the magnetic vector potential to ensure that the field remains solenoidal (divergence-free). At the same time, having the magnetic vector potential readily available is a big advantage if one wants to monitor the magnetic helicity, for example. The code is therefore particularly well suited for all kinds of dynamo problems. The code is normally non-conservative; thus, conserved quantities should only be conserved up to the discretization error of the scheme (not to machine accuracy). There is no guarantee that a conservative code is more accurate with respect to quantities that are not explicitly conserved, such as entropy. Another important quantity that is (to our knowledge) not strictly conserved by ordinary flux conserving schemes is `magnetic helicity`. There are currently no plans to implement adaptive mesh refinement into the code, which would cause major technical complications. Given that turbulence is generically space-filling, local refinement to smaller scales would often not be very useful anyway. On the other hand, in some geometries turbulence may well be confined to certain regions in space, so one could indeed gain by solving the outer regions with fewer points. In order to be cache-efficient, we solve the equations along `pencils in the ``x`` direction. One very convenient side-effect is that auxiliary and derived variables use very little memory, as they are only ever defined on one pencil. The domain can be tiled in the ``y`` and ``z`` directions. On multiprocessor computers, the code can use `MPI` (Message Passing Interface) calls to communicate between processors. An easy switching mechanism allows the user to run the code on a machine without MPI libraries (e.g., a notebook computer). Ghost zones are used to implement boundary conditions on physical and processor boundaries. A high level of flexibility is achieved by encapsulating individual physical processes and variables in individual `modules`, which can be switched on or off in the file :file:`Makefile.local` in the local :file:`src/` directory. This approach avoids the use of difficult-to-read preprocessor directives, at the price of requiring one dummy module for each physics module. For nonmagnetic hydrodynamics, for example, one will use the module :file:`nomagnetic.f90` and specifies: .. code:: MAGNETIC = nomagnetic in :file:`Makefile.local`, while for MHD simulations, :file:`magnetic.f90` will be used: .. code:: MAGNETIC = magnetic Note that the term `module` as used here is only loosely related to Fortran modules: both :file:`magnetic.f90` and :file:`nomagnetic.f90` define an F90 module named *Magnetic* --- this is the basis of the switching mechanism we are using. Input parameters (which are set in the files :file:`start.in`, :file:`run.in` files) can be changed without recompilation. Furthermore, one can change the list of variables for monitoring (diagnostic) output on the fly, and there are mechanisms for making the code reload new parameters or exit gracefully at runtime. You may want to check for correctness of these files with the command :code:`pc_configtest`. The requirements for using the Pencil-MPI code are modest: you can use it on any Linux or Unix system with an F95 and C compiler suite, like GNU gcc and gfortran, together with the shell CSH, and the Perl interpreter are mandatory requirements. Although the Pencil Code is mainly designed to run on supercomputers, more than 50% of the users run their code also on Macs, and the other half uses either directly Linux on their laptops or they use VirtualBox on their Windows machine on which they install Ubuntu Linux. If you have IDL as well, you will be able to visualize the results (a number of sample procedures are provided), but other tools such as Python, DX (OpenDX, data explorer) can also be used and some relevant tools and routines come with the |PC|. If you want to make creative use of the code, this manual will contain far too little information. Its major aim is to give you an idea of the way the code is organized, so you can more efficiently *read the source code*, which contains a reasonable amount of comments. You might want to read through the various sample directories that are checked in. Choose one that is closest to your application and start modifying. For further enhancements that you may want to add to the code, you can take as an example the lines in the code that deal with related variables, functions, diagnostics, equations etc., which have already been implemented. Just remember: :code:`grep` is one of your best friends when you want to understand how certain variables or functions are used in the code. We will be happy to include user-supplied changes and updates to the code in future releases and welcome any feedback. Wolfgang Dobler Axel Brandenburg Acknowledgments =============== Many people have contributed in different ways to the development of this code. We thank first of all Åke Nordlund (Copenhagen Observatory) and Bob Stein (University of Michigan) who introduced us to the idea of using high-order schemes in compressible flows and who taught us a lot about simulations in general. The calculation of the power spectra, structure functions, the remeshing procedures, routines for changing the number of processors, as well as the shearing sheet approximation and the flux-limited diffusion approximation for radiative transfer were implemented by Nils Erland L. Haugen (University of Trondheim). Tobi Heinemann added the long characteristics method for radiative transfer as well as hydrogen ionization. He also added and/or improved shock diffusion for other variables and improved the resulting timestep control. Anders Johansen, Wladimir (Wlad) Lyra, and Jeff Oishi contributed to the implementation of the dust equations (which now comprises an array of different components). Antony (Tony) Mee (University of Newcastle) implemented shock viscosity and added the interstellar module together with Graeme R. Sarson (also University of Newcastle), who also implemented the geodynamo set-up together with David McMillan (currently also at the University of Newcastle). Tony also included a method for outputting auxiliary variables and enhanced the overall functionality of the code and related IDL and DX procedures. He also added, together with Andrew Snodin, the evolution equations for the cosmic ray energy density. Vladimir Pariev (University of Rochester) contributed to the development and testing of the potential field boundary condition at an early stage. The implementation of spherical and cylindrical coordinates is due to Dhrubaditya (Dhruba) Mitra and Wladimir Lyra. Wlad also implemented the global set-up for protoplanetary disks (as opposed to the local shearing sheet formalism). He also added a ``N``-body code (based on the particle module coded by Anders Johansen and Tony), and implemented the coupled evolution equations of neutrals and ions for two-fluid models of ambipolar diffusion. Boris Dintrans is in charge of implementing the anelastic and Boussinesq modules. Philippe-A. Bourdin implemented HDF5 support and wrote the optional IO-modules for high-performance computing featuring various communication strategies. He also contributed to the solar-corona module and worked on the IDL GUI, including the IDL routines for reading and working with large amounts of data. Again, this list contains other recent items that are not yet fully documented and acknowledged. Use of the PPARC supported supercomputers in St Andrews (Mhd) and Leicester (Ukaff) is acknowledged. We also acknowledge the Danish Center for Scientific Computing for granting time on Horseshoe, which is a 512+140 processor Beowulf cluster in Odense (Horseshoe). .. [JOSS] Pencil Code Collaboration, *J. Open Source Software*, **6**, 2807 (2021) The Pencil Code, a modular MPI code for partial differential equations and particles: multipurpose and multiuser-maintained. .. [Porter22] Porter, T. A., Jóhannesson, G., & Moskalenko, I. V. (2022). *Astrophys. J. Supp.*, 262, 30. “The GALPROP Cosmic-ray Propagation and Nonthermal Emissions Framework: Release v57.”