Parallelization of UDFs in ANSYS FLUENT
Many of the advance ANSYS FLUENT users write UDF (User Defined Function) to ask ANSYS FLUENT to do some special jobs. These UDFs are very popular in ANSYS FLUENT user community. Most of the UDFs that we create are programmed for serial mode of ANSYS FLUENT since it only involves basic C program coding with required marcos. However, to utilize the power of parallel processing, we need to convert the serial UDF into parallel UDF. Read this blog for a step by step process to convert serial UDF to parallel UDF.
You may download the serial as well as parallel UDF (C Program) files mentioned in the test case below and their high resolution illustrative image comparing the two and this blog's PDF version for offline reading from the download section at the top of this page (you may require to login using your LearnCAx account for downloading these files).
In recent times I have been constantly writing about ANSYS FLUENT Customization for CFD Simulations. And in continuation to it, today I am sharing a new piece of knowledge which is short & simple but highly useful for ANSYS FLUENT users having basic knowledge of UDFs - Parallelization of UDFs
Today we will understand how to convert the serial UDFs to run in Parallel mode in few easy steps. To begin with, let us first understand what it means when we say parallel mode UDFs. When you run a UDF into a serial FLUENT process, it gets executed by that single process just once. Whereas, if you run a UDF under parallel mode of FLUENT, then FLUENT executes your UDF (C program) on the host process as well as on the ‘n’ node processes. It means that the same set of C program written in your UDF is read ‘n + 1’ times. A very common problem which can arise in such situation is that if in your UDF you are creating a file and writing some text into it, then all these processes will try to create this file at the same location and will try to insert the same text all at the same time. Similarly, if we try to run a UDF which calculates the average pressure using the formula “force/area” whose values are fetched from the mesh, then the host will perform “0/0” since mesh data is not present in the host process at all. Also in most of the cases you must have experienced that when a UDF meant for serial version of FLUENT is run on the parallel version of FLUENT then FLUENT crashes with some error message. So it’s obvious that you need to modify the UDF for supporting it in parallel version of FLUENT.
Let us now see how we can modify a UDF meant to be run on a single process to run on multiple processes i.e. parallel version of FLUENT. You will get the below information from ANSYS UDF guide but I thought of writing down the same in short with the help of the issue I faced.
Before we begin, let us first see what all are we going to cover –
- Compiler Directives
- Predicates
- Global Reduction Macros
Firstly, let’s understand the scenario. In one of my projects, one of the objectives was to find out the thermal information of a heat emitting device. So, at the end of the simulation I was using a DEFINE-ON-DEMAND macro based UDF to perform the following operations –
- Read a file containing IDs of the list of inlets & outlets of the heating device (all IDs are written in a single line and are seperated with spaces between them)
- Calculate the surface max temperature, surface min temperature and area weighted average of temperature for those IDs
- Write down the above calculated values into a file to be used for report generation
Now, since a need arised to speed up the simulation, I started shifting the FLUENT run into parallel mode, but all of a sudden my serial mode UDF stopped working as expected. Hence, I went thru ANSYS FLUENT's UDF guide and started learning about modify UDFs for parallel use. I now have the modified UDF which works both in parallel as well as in serial mode of FLUENT.
Let us watch side by side the serial UDF vs its parallel version and finally understand all the necessary modifications done to it.
Now let us understand the modifications one by one –
Modification 1: Type - Compiler Directives: These are used to execute commands only on certain types of processes, viz. Serial, Host and/or Parallel. In the above UDF we don’t need to use the host process to do any activity, hence we are excluding it by the #if !RP_HOST compiler directive. This will make the Serial (in serial mode) as well as the Node process (in parallel mode) execute the complete DEFINE_ON_DEMAND UDF when ever it is executed from FLUENT application on demand by the user. A compliler directive needs to be ended with #endif. If you look at the end of the UDF you will find that I have used #endif to close this compiler directive. You can wrap any part of your C program using a compiler directive. There are following types of compiler directives -
- #if RP_HOST - code written inside this will be performed only by the HOST process
- #if RP_NODE - code written inside this will be performed only by the NODE process
- #if PARALLEL - code written inside this will be performed only if parallel version of FLUENT is started
- #if !RP_HOST - code written inside this will be performed only by the SERIAL & NODE processes
- #if !RP_NODE - code written inside this will be performed only by the SERIAL & HOST processes
- #if !PARALLEL - code written inside this will be performed only if serial version of FLUENT is started
Modification 2: Type - Compiler Directives: Similar to “Modification 1” but this time we are using it for executing the wrapped if condition only if FLUENT is opened in parallel mode.
Modification 3: Type - Predicates: The use of predicates are similar to that of compiler directives. Predicates are pre-defined FLUENT parallel version's macros, which lets you run a set of codes for a specific process or on the process where the selected object's primary face is present. Below is the list of Predicates available in FLUENT's parallel version –
- MULTIPLE_COMPUTE_NODE_P
- ONE_COMPUTE_NODE_P
- ZERO_COMPUTE_NODE_P
- I_AM_NODE_HOST_P
- I_AM_NODE_ZERO_P
- I_AM_NODE_ONE_P
- I_AM_NODE_LAST_P
- I_AM_NODE_SAME_P(n)
- I_AM_NODE_LESS_P(n)
- I_AM_NODE_MORE_P(n)
- PRINCIPAL_FACE_P(f,t)
These above predicates can be used to conditionally perform a set of codes within a if condition, for e.g. if you wish to perform a set of codes on the first node process then you may use the following predicate –
if(I_AM_NODE_ZERO_P)
{
...expressions;
}
In this case, we are performing the file open to write operation only on the first node process to avoid multiple file writing operations on each of the nodes. As this if condition is wrapped inside the #if PARALLEL compiler directive, it wont’t be executed when FLUENT is opened in Serial mode and the file opening operation would be directly executed. Hence, this makes the file opening operation section of this UDF compatible in both Serial as well as Parallel mode of ANSYS FLUENT.
Modification 4: Type - Predicates: Similar to “Modification 3” but this time we are calculating all the required values from a surface thru that node process which has the surface's principal face present in it.
Modification 5: Type - Global Reduction Macros: We are performing node synochronisation operations here. What it means is that, now since we have multiple node processes, for every variable declared in all the node processes, there is a chance of this variable having different values in different node processes. For e.g. lets say, the mesh of a surface is distributed unevenly over two node processes and if we get the area of that surface from each of the nodes into a variable, then it is obvious that this variable will have different values for different nodes. Now since we need the summation of the values of this variable from all the nodes to get the actual area of the surface, these global reduction macros come into picture. Various available global reduction macros are as follows –
Global Summations -
Variables
- PRF_GISUM1(x)
- PRF_GRSUM1(x)
Arrays
- PRF_GISUM(x,N,iwork)
- PRF_GRSUM(x,N,iwork)
Global Maximums and Minimums -
Variables
- PRF_GIHIGH1(x)
- PRF_GRHIGH1(x)
- PRF_GILOW1(x)
- PRF_GRLOW1(x)
Arrays
- PRF_GIHIGH(x,N,iwork)
- PRF_GRHIGH(x,N,iwork)
- PRF_GILOW(x,N,iwork)
- PRF_GRLOW(x,N,iwork)
Global Logicals -
Variables
- PRF_GLOR1(x)
- PRF_GLAND1(x)
Arrays
- PRF_GLOR(x,N,work)
- PRF_GLAND(x,N,iwork)
Global Synchronization -
- PRF_GSYNC()
These global reduction macros present in FLUENT parallel mode come in various versions like most macros have two versions - one for integers & the other for real numbers. Then we have macros to operate on a variable as well as on an array.
In our case, we are traversing thru each of the surface using their IDs and are trying to get the lowest & highest temperature value from all the nodes and doing global summation of the values of temperature * area & area, so that these variables would get synchronized across all the node processes.
Modification 6: Type - Compiler Directives & Predicates: Same as “Modification 1 & 2”
Modification 7: Type - Compiler Directives & Predicates: Same as “Modification 1 & 2”
This is how you modify your serial UDF into parallel one. If you go thru the above serial vs parallel UDF comparison multiple times, you will not find it difficult to convert your own UDFs into parallel mode. I hope this knowledge share proves helpful to someone somewhere in need of it. If you feel it as a useful resource then kindly be in touch with us or even better let us know what you would like to hear from us in our upcoming blogs! Wish you good luck in your FLUENT customization work.
Feel free to contact me in case you have any query regarding any of your UDF work or if you need more information on any of the discussed topics from this blog, I will try to resolve your queries to the best of my knowledge.
Reference
- ANSYS FLUENT User Guide
The Author
{module [319]}