When to use this library
If you want to efficiently store large collections of data in memory. This library can dramatically reduce Full GC times and reduce memory consumption as well.When you have a data type which can be represented by an interface and you want a List for this type.
List<Iinterfacetype> list = new HugeArrayBuilder<Interfacetype>() {}.create();
The type needs to be described using an interface so its represented can be changed. (Using generated byte code)
The HugeArrayBuilder builds generated classes for the InterfaceType on demand. The builder can be configured to change how the HugeArrayList is created.
A more complex example is
HugeArrayList<Mutabletypes> hugeList = new HugeArrayBuilder<Mutabletypes>() {{
allocationSize = 1024*1024;
classLoader = myClassLoader;
}}.create();
How does the library differ
- Uses long for sizes and indecies.
- Uses column based data making the per element overhead minimal and speed up scans over a single or small set of attributes. This reduces memory usages by 2x or more.
- Stores attributes in direct memory as much as possible, reducing heap usage dramatically, 10x or more.
- Allocates blocks of data for consistent add() times, rather than exponentially growing an underlying store (with exponentially increasing delays on a grow of capacity)
Performance comparison
This test compares using HugeCollection vs ArrayList of JavaBeans. The class has 12 fields of various types which can be converted to primitives in different ways. By storing these primitives (or objects encoded as primitives), the time to perform GCs is almost eliminated.
Classes with fields which cannot be encoded as primitives are supported in version 0.1.1 and still show an improvement in GC times.
In both cases, the amount of memory used was halved.
The project web site
HugeCollections on Google CodeThe source
The best way to get the whole source, including tests is to use subversion.svn co http://vanilla-java.googlecode.com/svn/trunk/collections/ vanilla-java-collections cd vanilla-java-collections mvn test
Have you seen http://ehcache.org/documentation/user-guide/bigmemory ??
ReplyDeleteWhat is teh difference between this and that !!??? :(
@Ramya karivella, The difference is that BigMemory uses Serialization/Deserialization. This means you can use polymorphism and regular data value objects. The typical access time for 8 GB is 200 us.
ReplyDeleteMy approach avoids the need for any serialization which makes the amount of CPU and memory used much lower. Also no garbage is produced. Typical access time is around 0.5 us to perform a update (read+write) a random object!
You can iterate and update billions of entries without creating significant garbage. ;) (I have a unit test which does exactly that, it creates one Iterator) If you use indexed access, you can avoid any object creation.
Another difference is that objects in HugeCollections can occupy less memory, typically about half. The overhead of serialization means the objects in BigMemory can be 4x larger for a typical object. Memory is cheap, however cache is limited and more compact data structures can improve performance.
The missing thing is A Map implementation, eviction policies and disk storage. All non-trivial but planned.