Introduction

In today’s business, the faster you can get somewhere, the more beneficial it is. Whether it is speedy delivery or rapid decision making, having the right data at your disposal is a necessity.

When moving data from on-premise systems to the cloud, you are restricted to one of two outcomes: either invest in a very expensive CDC solution for fast, real-time processing or replicate your whole database several times a day which doesn’t give you the most up-to-date information, whilst also putting a lot of stress on the database itself. Most of the time, these solutions will present the data consumers on your platform with the raw data from several sources, and they will first have to process it before it can be used. This process is costly, time-consuming, and can be unsafe. Implementing Publisher with business logic at the source provides a more controlled real-time approach to managing data changes. It helps ensure data quality and consistency before the data is propagated to downstream systems, making it particularly valuable in scenarios where data integrity and business rule enforcement are critical.

High-level overview of Publishers flow of data

With Publisher, you will be running a constant flow of ready to digest data packages to your cloud at a fraction of the costs. The ingestion phase sets the foundation for downstream data processing, analysis, and reporting. A well-structured and reliable ingestion process is essential for ensuring that data is ingested, processed, and made available for decision-making in a timely and accurate manner.

You can compare Publisher to a modern-day meal delivery service; Instead of grocery shopping and having to prepare the meal yourself, you can pick the menu items from several restaurants and have them delivered to your door, exactly when you want to eat.

Our solution

Publisher is a component developed for running a constant flow of ready-to-digest data packets across your Cloud infrastructure – sourced directly from a database or from an exposed API. Whereas most Change Data Capture (CDC) solutions focus on capturing and transmitting changes on individual tables, Publisher focuses on forming well-defined, structured objects directly at the source, publishing them to a distributed message broker of your choice and allowing a multitude of consumers to independently consume and process the data in their own individual pipelines.

Collectively, this is known as a create-transform-publish pattern. The structured objects generated by the Publisher are what we refer to as business objects – you are no longer pushing technical data on changes in your database, but curated, key business information.

CDC vs. Create-Transform-Publish Pattern

As a user, you define the query and how the result of that query should be assembled into a business object with a minimal and easy to understand YAML configuration. Publisher takes care of the rest – assembling the data, serializing it, encrypting it and – finally – publishing it.

Publisher is the „middle man“ between on-premise systems and the cloud while also doing the required processing for you, and all that at a fraction of the costs. It provides a stable, fast, and efficient way to format your data and publish it to the cloud for further consumption.

Publisher is capable of processing and transporting historical data, as well as real-time data. It provides support for numerous systems and additionally offers a lightweight variant.

Use Cases

Moving data from on-premise systems to the cloud is a costly, time-consuming process. Expensive CDC solutions are the current go-to for this kind of operations. These solutions, however, don’t offer further data processing, e.g. creation of business objects. They are also, potentially, slow and unstable. In contrast to such systems, Publisher is creating a business object at the source end, enabling data customization for analytics, with minimal impact on the source database, and yielding datasets ready for ingestion and presentation.

Here are some of the scenarios in which Publisher can be used in practice:

• Migrating from on-premises to the cloud, enabling the construction of a data platform, enhanced analytics, and more. Migrating data from on-premises to the cloud offers benefits such as reducing costs, while also providing scalability, security, global accessibility, and an easier approach to data analytics. It also simplifies maintenance, provides automatic updates, and supports business continuity while enabling innovation and sustainability.

• Migrating data from one cloud to another. Although self-explanatory for many businesses who are currently in this situation, having the ability to ease that process can be of great benefit to them.

• Migrating data from one SQL database (or API) to a broker, which can then sync to another NoSQL database for analysis purposes, essentially bridging the gap on transferring data from SQL to NoSQL.

How can Publisher help you in your industry?

Publisher for Retail – Click & Collect

As e-commerce is booming, most retailers are looking at click-and-collect services for their customers. The issue that makes this feature hard to implement well is that stock levels at the store are sent to the e-commerce platform either at the end of the day or every few hours. The customer now has to rely on no one else purchasing the available stock in the time between batches, there is no guarantee that the stock is really available. Publisher enables the Retailer to capture each sale in real-time and update the stock available to promise, in addition, Publisher empowers the Retailer to deliver that event to the analytics platform of their choice at the same time.

Publisher for Finance – AI Fraud Detection

Event driven real-time detection of unusual account activity in retail banking gives customers extra security
that’s needed as criminal activity becomes more sophisticated. With phishing attacks on the rise globally it’s
not just about securing card transactions, it needs to cover all types of transfers as well. With Publisher and its wide range of available connectors, this type of protection would absolutely be possible as Publisher can enable event-driven ways of working, enabling the bank to react far quicker to change – giving a competitive advantage, as well as giving their consumers’ confidence on the banks systems.

Publisher for Telecoms – Campaign Effectiveness Assessment

In the telecoms industry one important area that data plays a critical role in, is the ability to assess the effectiveness of a campaign in real-time. This is becoming the standard for the industry, however it’s not so easy to see and analyse the right data at the right time to make the right calls. Publisher is the product that makes this possible. While most campaigns are analysed only after they have finished, now you could monitor their success as you go and pivot if needed mid campaign. Publisher enables this by capturing changes as they happen in a way that both saves time and money as well as enabling better final results.

Differentiation and Feature

Following the “Create-Transform-Publish at source” pattern, Publisher utilizes the Relational Database Management System to transform data into structured business objects right there at the source. Data undergoes formatting, serialization, compression, and encryption, resulting in a substantial reduction in network traffic. This approach also ensures robust security, making it suitable for use on both public internet and private networks.

This versatile product is designed to seamlessly connect and operate with a wide range of data sources, including JDBC-compliant databases and API sources, ensuring compatibility and flexibility for your data integration needs. It excels in swiftly transferring data from source to destination, achieving a blazing speed of up to 170k rows per second in certain use cases for seamless data movement.
Message formatting and serialization can indeed have a significant impact on both cost savings and efficiency by minimizing bandwidth usage. However, the extent of this impact varies based on factors specific to your application, such as data volume, processing capabilities, and system compatibility requirements.

Results and Benefits

Publisher comes packed with features that increase the amount and speed of data transactions, unlocking the full potential of your database, and all of that whilst keeping an eye on the costs. Here are just a few of the key features that set Publisher apart:

Cost-consciousness: When implementing Publisher, you will not have to use overpriced CDC-solutions or pay for any type of software licenses which perform similar work. This already decreases the costs of moving your data immensely. Publisher also gives you the possibility to access data faster and more frequently which makes for better-informed business decisions. Anything that can accelerate decision-making due to well-structured data, will if not immediately, eventually greatly cut the costs built up by the data analysis or the data science teams.

Time management improvement: As was already mentioned in the previous paragraph, any amount of work that does not fall onto your analytics or data science teams, will enable them to focus more of their time on the issues which fall under their jurisdiction. Simply transferring data should not be one of them.

Transformation at source: Instead of sending large batches of unstructured data to your cloud platform, Publisher runs queries on your database and puts them together in neat little packages for the consumers on your platform to digest immediately. These packages can even be compiled from several different on-premise sources and will be encrypted and enriched with metadata. This way your consumers will always be served the right data and no additional on-platform transformation is needed.

Usage of public internet: Publisher makes it possible to safely transfer the data over public internet by using data encryption, instead of having to use dedicated VPN connections or virtual private cloud solutions.

Take Action Now: Transform Your Business

Ready to experience the benefits of Dataphos Publisher for yourself? Head over to our website now if you want to learn more about Syntio and how we can help you in your data journey (Homepage) and if you want to learn more about Publisher and how to get started visit our documentation page (Publisher).

Both community and enterprise versions of Publisher are at your disposal, with the enterprise edition offering round-the-clock support, access to new feature requests, and valuable assistance for developing use cases to drive your business forward.

We’re excited about the potential of Publisher being able to help companies maintain a continuous flow of easily consumable data packets across their cloud infrastructure, and we hope you are too. Thanks for taking the time to read about our new product, and we look forward to hearing about how Publisher has made a difference in your life!