ArmoniK Newsletter October 2023

ArmoniK is the only modern open-source orchestrator able to manage the distribution of huge graphs of computation tasks on premises and in the cloud for massive simulations in Finance, Healthcare and Industry.

Our aim with this communication is to regularly inform all the community of people interested in ArmoniK on the adoption of the product, the new features and some technical aspects. We hope actual users, prospects, cloud partners, HPC/HTC specialists and academics will find interesting news here.

Looking back at the decision to launch ArmoniK at the end of 2020, a long way has already been covered. The 18 years of ANEO R&D allowed us to build this open-source solution in 18 months through a CA-CIB & AWS partnership. First usage in production at CA-CIB now dates back to July 2022, with smooth production stability since then and excellent performances. The replacements of the two old-dated commercial solutions are in good shape. Our support team is now up and running. And we are awaiting the first community contribution before the end of the year.

Performance

We are proud to announce that after extensive benchmarking on AWS, the performance of ArmoniK has been remarkable.

To be more specific, we measured:

a peak throughput at 25k tasks/sec.
an orchestration cost per task between 10ms and 100ms.
a roundtrip latency at 120ms.

With the knowledge gained from these tests, we have already identified ways to improve the number of tasks per second. We are currently working on these improvements and will announce our findings in a future newsletter.

For more information see here.

Technical additions

We are continuously improving ArmoniK and are providing new features regularly. Here are 3 of the most recent changes. ArmoniK can manage tasks with dependencies according to an event-based approach.

Dependent tasks can have higher priorities, allowing for lower scheduling costs when scheduling heavily dependent tasks.

Another addition is the possibility to add external queue plugins. You can now develop your own plugins for the queue system of your choice and easily integrate them with ArmoniK.

Lastly, we added support for C++ Client and Worker in addition to the existing C#, Angular, and Python. You can now interact with ArmoniK from C++ and simplify the use of your native libraries from ArmoniK.

Roadmap

Deployment on ARM platforms has been tested and will soon be officially supported. This will allow you to take advantage of AWS Graviton Instances.

Along with the SQS integration, the performance on AWS should be much better than the one presented earlier.

As mentioned earlier, a POC is underway using GCP and you can expect ArmoniK to be integrated properly with GCP’s managed services. The final target is to use Pub/Sub for the queue and GKE with autopilot for the resource allocator. Another good platform to show amazing performances in the future.

ArmoniK deployment on Windows has been tested. Development on the CI/CD is still required to allow for official support. We are waiting for an opportunity to work on this.

There are discussions with actors coming from the HPC field, to support other resource allocators such as Slurm. Soon to be seen: ArmoniK on your preferred supercomputer.