Setting up Spring Cloud Data Flow Server

Spring cloud data flow server can be used to setup the pipeline of tasks or streams. Tasks and streams can be registered with data flow server and can be executed or monitored either using cloud data flow shell or data flow server UI console/ dashboard.

Setting up Data flow server

Spring cloud data flow server can be setup using two ways, either use spring provided spring boot application or develop your own spring cloud data flow server as spring boot application.

Using Spring provided Data flow server application

There are many ways to setup data flow server as given below.

• Local Machine
• Cloud Foundry
• Kubernetes

Please refer below documentation for more details for above installation types.
https://dataflow.spring.io/docs/installation/

Create Spring boot application as Data flow server

Here I will show you how to create your own spring boot application as spring cloud data flow server which you can run anywhere locally or in cloud. It gives you more freedom with application to setup the other dependencies like database or security features.

Maven Dependencies

We have used Spring Boot version 2.1.6.RELEASE which is supported by the current version of data flow server (2.2.1.RELEASE). Below are the dependencies required in pom.xml

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-dataflow-server</artifactId>
            <version>2.2.1.RELEASE</version>
        </dependency>

application.properties

In this properties file we are just creating an in-memory database using H2. However it is not required as it will automatically setup one if we don't configure it. We can use these properties to use any other database.

spring.datasource.url=jdbc:h2:tcp://localhost:19092/mem:dataflow
spring.datasource.jdbcUrl=jdbc:h2:tcp://localhost:19092/mem:dataflow
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

spring.jpa.database-platform=org.hibernate.dialect.H2Dialect

MainDataFlowServerApplication.java

This is our main class which need to be annotated  with @EnableDataFlowServer annotation. Since we are going to run this application in our local machine, we need to exclude some auto configurations like cloud foundry, kubernetes etc. until we intend to actually configure them. When required we can remove them from exclusions and configure necessary properties for auto configuration. See the below code.

@EnableDataFlowServer
@SpringBootApplication(
        exclude = {SessionAutoConfiguration.class,
                ManagementWebSecurityAutoConfiguration.class,
                SecurityAutoConfiguration.class,
                UserDetailsServiceAutoConfiguration.class,
                LocalDeployerAutoConfiguration.class,
                CloudFoundryDeployerAutoConfiguration.class,
                KubernetesAutoConfiguration.class}
)
public class MainDataFlowServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(MainDataFlowServerApplication.class, args);
    }
}

Git URL

Below GIT URL contains complete source code.
https://github.com/thetechnojournals/spring-tutorials/tree/master/SpringCloudDataFlowServer

Running data flow server application

Once you done with all required code and configuration setup, you can build the application and use below maven command to start your data flow server.

mvn spring-boot:run

Once application is started, hit the below URL in browser and you will see the given screen as dashboard of data flow server application where you can see all registered task and streams.\
URL: http://localhost:9393/dashboard/

Check my another post on registering and executing Spring Batch application with Spring cloud data flow server.
https://www.thetechnojournals.com/2019/12/spring-batch-job-with-spring-cloud-data.html