here are two places you can configure memory for your Apollo 2020 server.
TOMCAT JVM memory
Double click C:\Program Files\Hexagon\ERDAS APOLLO\tomcat\bin\APOLLOTOMCATw.exe, from following server property dialog, go to Java tab, and change “Maximum memory pool”. By default the Maximum memory pool is 4GB.
 Raster Data Server (RDS) JVM memory
RDS is an external process that is spawned by the ERDAS APOLLO ProcessManager service. The ProcessManager service is responsible for managing a pool of 64 bit RDS processes. It can spawn a new instance of RDS or utilize one from the pool of existing RDS processes in order to fulfill an imagery request
NOTE: Tomcat JVM memory and RDS JVM memory is not connected at all ( just like JBOSS JVM memory and RDS JVM memory in APOLLO 2015). So Tomcat JVM max memory limit will not be a limit for RDS processes... RDS processes directly consume available physical memory ..
By default, APOLLO server starts 3 java processes – one for the main server (which is now tomcat8.exe) then two child java.exe processes. The child processes are the RDS processes. The RDS processes are what does a lot of the imagery decoding/rendering, If the APOLLO server has started up correctly, you should see something like this in process explorer:
User can change the RDS JVM memory (e.g., Xmx memory for 64-bit RDS java tune up parameters) from Apollo data manager, also change how many RDS processes (Maximum Decoder processes)