These are notes describing the TOUGH2.01-MP version. The current serial version of TOUGH is TOUGH2 V2.1. http://esd1.lbl.gov/research/projects/tough/software/tough2.html TOUGH2-MP is the parallel version of TOUGH2. http://esd1.lbl.gov/research/projects/tough/licensing/tough2mp.html This version is named TOUGH2.01-MP and is primarily a bugfix version of TOUGH2-MP. In this document, we refer to TOUGH2.01-MP as the "MP version". The simulation results of the MP version should be very similar to the results of the serial code. However, there are enough changes to the code base of the MP version that it can be difficult to directly compare the sources. The differences in the sources of the EOS modules between the serial and MP versions are minimal. While we do not present an exhaustive list of every change made to the new MP code, there are four basic categories of differences between the previous and current MP versions. 1) bug found in Aztec 2) bug fixes in MP code 3) new build system 4) formatting, cleanup (1) One bug found had given many users problems, but was not actually in TOUGH2.01-MP. Instead, it was a problem in the external Aztec library. A careless memory operation was causing spurious results for certain problems and would often result in the code stopping prematurely. Specifically, we replaced a memcpy() call with a memmove() call. Changing the code to be safer fixed all Aztec-related problems we encountered. The version of Aztec used by the MP version is no longer being maintained. To make the patch, we included the affected source file (az_sort.c), and the build system makes the source file replacement. (2) Bugs were found as we compiled and ran the code with different compilers, compiler flags, and on different machines. For example, the MP version will now build with at least the GNU and Intel compilers. Examples of bug fixes: a) variables not initialized before use b) dynamic arrays broadcasted to all processors, but only allocated on rank 0 c) access of arrays beyond the defined range d) variables declared with improper types e) actions only required on rank 0 not to be executed on other processors f) code simplifications g) always use implicit none h) prefer floating-point constants end with 'd0' i) always use real*8 instead of real*4 Some of these improvements could be considered major as they led to unexpected results. However, none of the modifications resulted in different simulation results for the tests that we ran. Some of the bugfixes allowed the simulation to avoid a segmentation fault. If your simulation was running, these bug fixes should not have an impact. If your simulation stopped early, this new version might help. (3) The build system now uses CMake. CMake is a free, cross-platform, open-source build system (http://www.cmake.org). CMake handles external packages and makes it easier to install TOUGH2-MP on clusters. (4) Examples of formatting changes: a) converting any Fortran77 styles to Fortran90 b) prefer comments using exclamation identifier c) remove extra spaces at ends of lines d) convert tabs to spaces There were also some changes made such that the MP version was more consistent with the serial version. There are some changes to the integer values to select both the relative permeability functions and the capillary pressure functions. To be consistent with the serial version, some of these selection values were modified. Here is a complete list of the new values: Appendix B Relative Permeabilities RELP() IRP=1 Linear functions IRP=2 Pickens IRP=3 Corey IRP=4 Grant IRP=5 all phases mobile IRP=6 Fatt and Klikoff IRP=7 van Genuchten-Mualem IRP=8 Verma IRP=9 X-curves IRP=10 X-curves IRP=11 X-curves IRP=12 Original ECM (isotropic) IRP=13 Original ECM (anisotropic) IRP=14 Generalized ECM Appendix C Capillary Pressure PCAP() ICP=1 Linear ICP=2 Pickens ICP=3 TRUST ICP=4 Milly ICP=5 Leverett ICP=6 Leverett ICP=7 van Genuchten ICP=8 PC=0 ICP=9 PC=0 ICP=10 PC=0 ICP=11 PC=0 ICP=12 (void) ICP=13 Original ECM ICP=14 Generalized ECM For example, previously to choose the generalized ECM treatment for relative permeabilities in the MP version, an IRP of 11 was used, but now the user should choose 14. Some EOS modules that were previously documented to work in parallel were not ported to TOUGH2.01-MP. The sources for some of the files required too many changes. The only EOS modules in the bugfix release are: EOS1, EOS2, EOS3, EOS4, EOS5, EOS7, EOS7r, EOS8, EWASG, and ECO2N. Any other EOS modules that were previously supported can be used with the previous MP version. The new CMake build system can also run the tests that are provided for each EOS module. A few things to note regarding these tests. The main purpose is to verify that the executable is viable. The tests are run with 2 processors using the command of the form "mpirun -np 2 executable". If the system does not allow using this command on the command line, the tests may fail. Also, the tests will report "Passed" simply if the run finishes without any errors. There is no attempt to verify that the results are correct. TOUGH2.01-MP Installation Directions ++++++++++++++++++++++++++++++++++++ Building TOUGH2.01-MP requires at least the following: a) Fortran and C compiler (we've tested with Intel, GNU, and PGI) b) MPI (all versions of MPI libraries we've tested have worked) c) CMake version 2.8.5 or higher (found at http://www.cmake.org/) d) GNU make e) Aztec library f) Metis library Typically, the MPI installation will provide wrappers for the compilers, such as mpicc and mpif90. Please ensure you can run a simple MPI test on your system before continuing. The two third party libraries required (Aztec and Metis) can be downloaded for free here: (Aztec requires a simple registration process.) http://www.cs.sandia.gov/CRF/aztec1.html http://www.cs.sandia.gov/CRF/Aztec_download.html http://glaros.dtc.umn.edu/gkhome/views/metis http://glaros.dtc.umn.edu/gkhome/metis/metis/download Note that Aztec also requires basic math libraries found in BLAS/LAPACK that are often included in other places on your system (such as Intel's MKL). The TOUGH2.01-MP build system tries to handle as much of this as possible for the user. The one action that *must* be done before building TOUGH2.01-MP is to copy the tarballs for Metis and Aztec in the tpls/ directory. At the time of writing, these tarballs would resemble the following: Aztec2.1.tar.gz metis-5.1.0.tar.gz or metis-5.0.2.tar.gz If you have a previous tough2-mp installed on your system, these *may* be the same files and can simply be copied. Note that if using GNU compilers, metis-5.1.0 requires at least version 4.6 or above due to their use of a new warning flag. (-Wno-unused-but-set-variable) Once the above requirements are met, you can build TOUGH2.01-MP with the following commands: make config debug=1 eos=EOS_NUM make debug=1 eos=EOS_NUM Example: make config debug=1 eos=eco2n make debug=1 eos=eco2n We recommend trying with debug=1 first. If the runs are OK, then re-compile without "debug=1" (or with debug=0). Note that a new executable will be created with a different name, for example: debug version: tough2-mp-eco2n.debug non-debug: tough2-mp-eco2n To run the included tests, try: make test debug=1 eos=eco2n This will only work on systems that support "mpirun -n 2 executable". The tests will run in parallel with 2 processors. Note that the testing mechanism is simply looking for a string in the output to verify the run completed. There is no attempt to ensure the results are correct.