We have already dealt with the topic of data streaming in previous blog entries and found that data streaming is the new and modern way of batch processing.

Batch processing exists since the beginning of data processing. A form of processing in which input data is read in, stored and then processed with the master data. Everyone who grew up with data processing knows this form of processing. Even today, it is still carried out daily in nearly every computer center. The so-called "batch window" is strategically important and very often not sufficient for processing the data.

Nowadays, real-time reactions and analyses are important. The data is continuously processed and analyzed. The results are available almost in real time. Being able to react to these events in real time is essential and vital for many companies.

In contrast to this, the traditional method of making business analyses based on existing data, which serve as a basis for business decisions, is the so-called Business Intelligence (BI) method. The processing of data using the stream method is therefore different from BI, in which data is analyzed and processed as it is created.

Both methods are crucial for a modern company. The experiences of the present and the past are important for long-term future planning. For immediate reactions - especially in daily competition - the direct results of data streaming are important.

The main difference to batch processing is therefore the direct processing of the data after its creation or receipt. This processing takes place continuously and almost in real time. This depends, of course, on the application that processes and analyzes the data. In the Open Source area e.g. Apache Spark, Apache Flink, Apache Storm or Kafka Streams. The big players in the Cloud Computing and Big Data sector also offer streaming applications: AWS Kinesis, Google Dataflow or Microsoft Azure Stream Analytics.

The field of application of data streaming can be found in all areas and industries.

The "Internet of Things (IoT)" is also a classic area of application, as the received sensor data is processed in a time-critical manner. Thus, for example, maintenance work on machines can be recognized and carried out immediately.

The traditional mainframe IT, which used to process its data with classic batch or online processing, is now required to stream its data in real time to the systems that perform the analysis of the data.

tcVISION plays a central and important role in data streaming.

tcVISION is the supplier of the data that is created in an online processing environment on a mainframe system (CICS, IMS/DB, Software AG Adabas/Natural, CA IDMS) and is captured in real time by tcVISION and streamed into a Big Data environment. The same possibilities are offered by tcVISION in a batch environment. tcVISION includes several technology methods to detect batch change data and stream it into a Big Data environment for real-time analysis (log file processing, real-time capturing, batch compare).

B.O.S. Software has already introduced such a solution with Apache Kafka in 2017.

tcVISION as Confluent Source Connector.

Since the beginning of 2019, B.O.S. Software is a “Verified Standard” Partner of Confluent. The company with headquarters in Palo Alto, California, was founded by the developers of the open source solution Apache Kafka and offers a leading streaming platform that enables a company to maximize the value of its data.

Our partnership with Confluent enables users to easily and quickly integrate data from enterprise applications from mainframes and distributed systems into their streaming platform in real time and access these real-time streams with enterprise data.

Real-time analytics and BI solutions, as well as the modernization of applications that involve the relocation of data and applications, are greatly facilitated by the use of tcVISION. The effort for the implementation and realization of the Confluent streaming platform with integration of real-time data from mainframe and distributed systems is significantly reduced by tcVISION and can be implemented in shorter time.

You can find more detailed information here.

Practical application experiences at the customer's site are available and the acceptance and demand is high. tcVISION already supports all strategically important Big Data systems and applications. More will follow in the future.

An overview of all supported input and output targets can be found here.

Our tcVISION solution is best suited to connect the traditional mainframe (no matter if the operating system is called z/OS or z/VSE) to a Big Data environment or a cloud via data streaming.