data flow process
A data flow process is generally understood to be the structured and scheduled processing of data sets over several steps. This can take place within a software system or distributed over several systems. The best-known form of a data flow process is Extract, Transform, Load( ETL) for Data Warehouse( DWH).
A data flow process performs the processing of data automatically, i.e., autonomously and without interactive user input. For certain processes, however, the user or a configuration specification can influence the control of the process.
Often, but by no means exclusively, these data flow processes are operated as end-of-day processing.
A data flow process (as in process control in general) always requires a control unit. This includes a flow control engine, the engine, a metadata/configurationrepository, the repository, and a timing control, the scheduler. The latter component can possibly also be replaced by manually starting the process by an administrator.
As already mentioned at the beginning, data flow processes can either take place within a system or be distributed across several system boundaries. Examples:
Internal: Daily archiving processes can remain within one system, e.g. archiving of older e-mails in a mail server.
External export: Posting of pending payment orders takes place within the accounting system, but the postings are also automatically transferred to the bank's system.
Distributed: ETL processes always involve multiple systems, the input system as the original source system, the data warehouse, possibly the data mart and the business intelligence system.
Special software components are often used for larger data flow processes. Examples are software for business process modeling such as Microsoft's BizTalk server, software for instrument control such as Laboratory Virtual Instrument Engineering Workbench( LabVIEW), various ETL products or components of manufacturing control and accounting software or those for Enterprise Resource Planning( ERP) and Supply Chain Management( SCM).