MicroStream 4.x

2021 03 02 head

MicroStream approach empowers Java developers. The library provides a cool and compact approach to persist complex data models. You can persist your whole data model with less than a hundred lines of code. The library is efficient and works reliably.

Modern architectures based on bounded domains and Domain Driven Design are supported.

The company behind the product has promised to open source their product beginning of 2021.

I use the library to prototype bounded domain implementations requiring persistent data. The implementation speed and ease are awesome. I hope the current flaws of the product will be removed and allow productive use.

Constraints

Saving a Java Object

The library made some strange design decisions. If you want to persist an instance having fields based on collections, you must use the eager mode. Beware that you must explicitly commit the transaction if using the eager mode. The normal mode automatically commits the transaction. I hope that the API improves, and the behavior is coherent for similar functions.

Debugging problems

The source code is currently obfuscated. Debugging is difficult and cumbersome because variable names are synthetic and all comments are missing. The source code documentation is shallow and insufficient to understand the behavior of the library.

Saving a collection

If you have access to the collection, just call store on the collection. If the collection is hidden inside an object, you are forced to use an eager store. If you use standard object-oriented practices, you will have to use the less efficient approach of eager write operations.

Design

Our provider persists instances for all our bounded domains in need of persistence.

package net.tangly.core.providers;

import java.util.Collections;
import java.util.List;

import one.microstream.storage.types.EmbeddedStorageManager;
import org.jetbrains.annotations.NotNull;

/**
 * Provider with instances cached in memory and persisted onto the file system or a database.
 * <p>The update method uses an eager storage strategy to insure that all instance variables of a Java object are persisted. This approach is necessary due
 * to the implementation restrictions of MicroStream.</p>
 *
 * @param <T> type of the instances handled in the provider
 */
public record ProviderPersistence<T>(@NotNull EmbeddedStorageManager storageManager, @NotNull List<T> items) implements Provider<T> {
    @Override
    public List<T> items() {
        return Collections.unmodifiableList(items);
    }

    @Override
    public void update(@NotNull T entity) {
        if (!items.contains(entity)) {
            items.add(entity);
            storageManager.store(items);
        } else {
            var storage = storageManager.createEagerStorer();
            storage.store(entity);
            storage.commit();
        }
    }

    @Override
    public void updateAll(@NotNull Iterable<? extends T> entities) {
        entities.forEach(entity -> {
            if (!items.contains(entity)) {
                items.add(entity);
            } else {
                storageManager.store(entity);
            }
        });
        storageManager.store(items);
    }

    @Override
    public void delete(@NotNull T entity) {
        items.remove(entity);
    }
}

Current Flaws

In Memory File System

In-memory file systems are a must for unit and integration testing. A continuous integration pipeline does not always have access to a regular file system. Disk-based file systems are too slow for automatic testing [1].

We migrate with full success all our unit tests to use in-memory file systems in January 2021. We use the Jimfs library. When creating a Jimfs file system, you can specify whether it should imitate a UNIX- or Windows-style file system. They differ in the naming conventions and attribute usage, so choose the one you prefer. The Windows file system implements some Windows’ quirks, such as not permitting the creation of a file in the root directory.

The execution speed rocks. Thanks to the MicroStream developers. See their file systems example in their GitHub example repository

Java Records

A field cannot be manipulated anymore in record since Java JDK 1.5.1. The MicroStream 4.x releases crash when using record in JDK 15 and JDK 16. It only works with JDK 14. You are forced to use an obsolete JDK no more supported. For example, the IntelliJ IDEA stops to provide support for obsolete JDKs as soon as the new official JDK release is available. It hinders you to use the associated preview features [2] [3].

I have to verify if the library can cope with sealed declarations.

Next Steps

  • Wait for the release scheduled for April 2021 and pray they will support the record feature. Records are an official feature in JDK 16 released in March 2021 and no more a preview feature. See our Modern Java Development discussion how organizations cope with the Java release cycle or are just laggards.

  • Codify a performant approach to save whole instances using eager store operations.

  • Explore the migration features when the data schema is changing.

  • Wait and see if the promised licensing under an open source copyright will be published.


1. I have reported the bug how they used path instances. The developers were very supportive. Version 4.01 supports in-memory file systems such as Jimfs.
2. I have reported the bug in the microstream forum and informed them that their claim they support these JDK is plain wrong. The project sadly does not currently have a bug reporting system. I hope the version promised for April 2021 will finally solve these showstoppers.
3. Version 5.0 was released in July 2021 and finally provides support for records. You need to increase access to internal components with the option --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED to have a working approach.