Modded Minecraft
Modded Minecraft, and memory usage
A crappy graph, that doesn't really relate to the text, but hey, it's hand drawn so it's artisanal and stuffs.
Memory usage of Modded MC has changed over time. This fact should not be in dispute by anyone. It's grown, a lot. Most of this is outside modder's control - some of it is due to forge, but the majority is due to the changing vanilla landscape.
This leads to a common symptom on recent versions: the garbage collection tailspin of death. If your MC is taking 15-20 minutes to load, and is thrashing 100% CPU, you are very likely a victim of this issue. This is due to the growth of MC having hit an important threshold, that the crappy graph is trying to illustrate: the default sizing of Minecraft memory (1GB) is now too small to accommodate a normal size modded MC instance. This isn't because 1.8 suddenly ate all the memory - the growth has been incremental (though 1.8 was a particularly big bite) since 1.3.
A quick history of how versions changed the memory footprint
If you look at 1.2.5 you'll see a very different beast from even 1.3. The client is running the entire simulation in the same object space as the rendering, resulting in very very small amounts of memory being used. 1.2.5 could live in ~256MB perhaps.
1.3 split the server and the client logic. Suddenly, you have to double the footprint of the game - one copy of all the game objects for the server, one copy for the client. 1.3 needs 500MB before you're even running.
1.5 adds some more footprint - the client now has to track atlases of icons and textures, which takes up precious memory, adding a bunch of new utilization.
1.7 changes all the networking to a proper network library (netty) as well as changing how ID tracking works. This again takes up more memory. This is probably the last version where 1GB (the default) will get you a working game in a reasonable timeframe for a non-trivial mod set.
1.8 changes all the models to a proper formalism. This takes up a huge amount of memory, especially with forge's changes to allow for dynamic models and other nice features for modders. You could easily expect to need 1.5GB to run with a non-trivial mod set.
1.9 through 1.11 don't really change the memory footprint significantly, though there's optimizations we're trying to push into forge to help things.
Summary: memory has been growing for a lot longer than people think, and it crossed the "default values" threshold with 1.8, into "you need to fix it territory".
You ask: How do I fix it?
Easy:
-XX:+UseG1GC -Xmx4G -Xms4G -Dsun.rmi.dgc.server.gcInterval=2147483646 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=50 -XX:G1HeapRegionSize=32M Put this into the JVM Arguments box for your modded game. Ensure you're running the latest Java 8 from here: http://www.oracle.com/technetwork/java/javase/downloads/index.html (It's Java8 update112 at the writing of this post).
An explanation for those who are curious and want to experiment: -XX:+UseG1GC Turns on G1GC. This is a great garbage collector for interactive applications, such as Minecraft. It tries to keep garbage collection predictable, so it never takes a long time (big lag spikes) and doesn't repeatedly take lots of short times (microstuttering)
-Xmx4G -Xms4G Sets the heap size to 4G and keeps it pinned at 4G. Minecraft eats a lot of memory. If you have -Xms set to something smaller, the garbage collector may be convinced to try "harder" to garbage collect to that lower target. This can result in "big lag spikes" because those aggressive collections will be slow and painful.
-Dsun.rmi.dgc.server.gcInterval=2147483646 This tells the RMI layer not to do a full GC every minute. Yeah.
-XX:+UnlockExperimentalVMOptions Turns on experimental VM options. Duh.
-XX:G1NewSizePercent=20 Tells G1GC to put aside 20% of the heap as "new" space. This is space where new objects will be allocated, in general. You want a decent amount, cos MC makes a lot of objects (/me looks at BlockPos) and you don't want to have to run a collection whenever it gets full (this is a big source of microstutters).
-XX:MaxGCPauseMillis=50 This tells G1GC to try and not stop for more than 50 milliseconds when garbage collecting, if possible. This is a target, and G1GC will ignore you if you put a silly number in like 1 which is unattainable. 50 millis is the time for one server tick, and has given me buttery smooth performance on various setups since implementing it.
-XX:G1HeapRegionSize=32M This tells G1GC to allocate it's garbage collection blocks in units of 32megs. The reason for this is that chunk data is typically just over 8megs in size, and if you leave it default (16 megs), it'll treat all the chunk data as "humungous" and so it'll have to be garbage collected specially as a result. Some mods cause humongous allocations as well, such as journeymap, and this setting helps them too.
There's several other settings you can try, as well as using ConcurrentMarkSweep GC instead of G1GC, but I've found that since I switch to this setup about a year ago, I've had very few problems with memory bottlenecks. (Note: I play with 6GB because I have a 32GB beast computer, but 4GB should be ample for most existing packs. If you do notice it go slowly and use a lot of CPU during startup, however, your first thought should be to increase the memory a bit).
Multiple edits: because I'm terribad at reddit formatting
Edit: this can only really work in the vanilla launcher. Other launchers add other tuning, which will result in a big mess, sadly.