In 2019 CN Group was approached by four steel trading companies which shared inventories with each other. Each company had its own data format, and mapping of similar products by hand was very time consuming. They wanted us to create a centralised system, Steel network, where they could each upload their inventory and mapping would be automated.
First, we created a shared data standard during a joint two-day workshop which went through many iterations of the standard’s design, in the end agreeing a version that was 90% OK for everyone. But this was not the final version.
There were many discussions about it over time helped by a great feature of F#: by showing our code to the product owner, F#’s clear syntax for defining domain types allowed us rapidly to agree a solution. Also, thanks to F#’s powerful type system, and algebraic data types, we were able to implement the Steel network standard so that only valid data are created and stored.
At the start, we made a single application and deployed it to Azure as an App service. It soon became clear that we didn’t need the full functionality running at all times, just the search API for the frontend was needed permanently, the rest can be kept offline and triggered when needed.
So, we introduced serverless technology to the project by way of Azure Functions combined with Table and Blob storage. This allowed us to deploy individual functions independently and test them one by one.
Every step of the parsing process has its own function and the whole thing is glued together by the Durable Functions framework. The change brought sizable reduction in cost. Azure functions are billed for the time and memory they consume when running. Azure provides a free tier so that, for example, the staging environment can be run almost at no charge.
Without the high overhead of a large and expensive app service plan the API now needs only enough computation power to process the search requests, and the most expensive part, the import parser, is kept offline until it is needed.
The original three tier application with one App service and SQL Server doing everything would have cost around 500 - 1000 EUR per month, but with this smarter approach, the cost of the whole environment is around 100 EUR per month.
Steel network was a great project to work on and we’ve learned many new things. We needed to come up with new ways of thinking about data and how to store them – and we did much to our customers’ satisfaction.
Before Steel network, one of the product owners told me, getting information involved many calls and faxes between companies. Now they just open their web browser and, voilà, they’ve got all they need in one place!
From our F# enthusiast Jiří Landsman, CN Group .NET department