Validate Input data

Time needed to complete  ~60 minutes

Prerequisite for this tutorial is Configure Spring Batch Admin to use MySQL
and Developing first spring batch job

1.Introduction

What money is for bankers, data is for IT developers . Money can be dirty, data can be dirty too, for this reason developers need to check data using validation  before sending for processing. This article will show how validation can be used in spring batch jobs.

2.What is used in this tutorial:

1- Maven 3

2- Jdk 1.7

3- Tomcat 7.0.55

4- Eclipse Luna 4.4

5- Spring Core 3.2.9

6- Spring Batch 2.2.7

7- Spring Batch Admin Manager 1.3.0

8- MySQL 5 Database

3.Project Structure

Picture 1: Project structure

4. Configuration  files

- The database configuration where the data will be written is in file database.xml path src/main/resources/META-INF/spring/batch/dbConfig/database.xml, this is not the same as the database used by spring batch admin to store job repository data. First create the database named company_db and in the database.xml change db_username and db_password to suite company_db username and password.

Listing 1: database.xml

<beans xmlns="http://www.springframework.org/schema/beans"  
      xmlns:jdbc="http://www.springframework.org/schema/jdbc"   
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
      xsi:schemaLocation="http://www.springframework.org/schema/beans   
           http://www.springframework.org/schema/beans/spring-beans-3.2.xsd  
           http://www.springframework.org/schema/jdbc   
           http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd">  
   <!-- connect to database -->  
      <bean id="dataSource2"  
           class="org.springframework.jdbc.datasource.DriverManagerDataSource">  
           <property name="driverClassName" value="com.mysql.jdbc.Driver" />  
           <property name="url" value="jdbc:mysql://localhost:3306/company_db" />  
           <property name="username" value="db_username" /> 
           <property name="password" value="db_password" />
      </bean>  
      <bean id="transactionManager"  
           class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />  
 </beans>  

- The job definition that will read,process and write is in src/main/resources/META-INF/spring/batch/jobs/job-orderValidate.xml , Listing 2.

Listing 2: Job definition in job-orderValidate.xml

<beans xmlns="http://www.springframework.org/schema/beans"  
      xmlns:batch="http://www.springframework.org/schema/batch"   
      xmlns:task="http://www.springframework.org/schema/task"  
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
      xsi:schemaLocation="http://www.springframework.org/schema/batch  
           http://www.springframework.org/schema/batch/spring-batch-2.2.xsd  
           http://www.springframework.org/schema/beans   
           http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">  
      <!-- database configuration -->  
      <import resource="../dbConfig/database.xml" />  
      <!-- bean which data will be written to order table -->  
      <bean id="order" class="com.web.model.Order" scope="prototype" />  
      <bean id="validator"  
           class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" />  
      <!-- Batch job -->  
      <batch:job id="orderValidationJob">  
           <batch:step id="step1">  
                <batch:tasklet>  
                     <batch:chunk reader="cvsFileItemReader" processor="orderProcessor"  
                          writer="mysqlItemWriter" commit-interval="5">  
                     </batch:chunk>  
                </batch:tasklet>  
           </batch:step>  
      </batch:job>  
      <bean id="orderProcessor" class="com.web.processor.OrderProcessor">  
           <property name="validator" ref="validator" />  
      </bean>  
      <bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"  
           scope="step">  
           <property name="linesToSkip" value="1" />  
           <!-- Read order csv file -->  
           <property name="resource" value="#{jobParameters['inputDataFile']}" />  
           <property name="lineMapper">  
                <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">  
                     <!-- split fileds -->  
                     <property name="lineTokenizer">  
                          <bean  
                               class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer" />  
                     </property>  
                     <property name="fieldSetMapper">  
                          <bean class="com.web.mapper.OrderMapper" />  
                     </property>  
                </bean>  
           </property>  
      </bean>  
      <bean id="mysqlItemWriter"  
           class="org.springframework.batch.item.database.JdbcBatchItemWriter">  
           <property name="dataSource" ref="dataSource2" />  
           <property name="sql">  
                <value>  
       <![CDATA[      
            insert into tel_order(code,service_type,quantity,offer,bill_type,order_date)  
         values (:code, :serviceType, :quantity, :offer, :billType,:orderDate)  
       ]]>  
                </value>  
           </property>  
           <!-- mapping java object property to sql parameter -->  
           <property name="itemSqlParameterSourceProvider">  
                <bean  
                     class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />  
           </property>  
      </bean>  
 </beans>  

- The Order class using hibernate validator annotations, listing 3, and there is a custom @AccptedValues validator listing 4 and Listing 5.

Listing 3: Order class

 package com.web.model;  
   
 import java.io.Serializable;  
 import java.util.Date;  
   
 import javax.validation.constraints.Max;  
 import javax.validation.constraints.Min;  
 import javax.validation.constraints.NotNull;  
 import javax.validation.constraints.Past;  
 import javax.validation.constraints.Size;  
   
 import org.springframework.format.annotation.DateTimeFormat;  
   
 import com.web.validator.AcceptedValues;  
   
   
 public class Order implements Serializable {  
   
      private static final long serialVersionUID = 1853355651832301663L;  
   
      @Size(min = 5, max = 5, message = "Invalid code value")  
      private String code;  
        
      @AcceptedValues(acceptValues = { "fix", "mobile", "bundle", "internet" }, message = "Invalid service type")  
      private String serviceType;  
   
      @Min(value = 1, message = "Quantity less than 1")  
      @Max(value = 9, message = "Quantity more than 9")  
      private int quantity;  
        
      @AcceptedValues(acceptValues = { "small", "medium", "large", "fiber32" }, message = "Invalid offer")  
      private String offer;  
   
      @AcceptedValues(acceptValues = { "prepaid", "postpaid_mail","postpaid_email" }, message = "Invalid bill type")  
      private String billType;  
   
      @NotNull(message = "Date is required")  
      @Past(message = "Order date should be in the past")  
      @DateTimeFormat(pattern = "dd-MM-yyyy")  
      private Date orderDate; 

    // Setter, getters, toString, etc omitted    

Listing 4: Custom annotation type

 package com.web.validator;  
   
 import java.lang.annotation.Documented;  
 import java.lang.annotation.ElementType;  
 import java.lang.annotation.Retention;  
 import java.lang.annotation.RetentionPolicy;  
 import java.lang.annotation.Target;  
   
 import javax.validation.Constraint;  
 import javax.validation.Payload;  
   
 @Documented  
 @Constraint(validatedBy = AcceptedValidator.class)  
 @Target({ ElementType.METHOD, ElementType.FIELD })  
 @Retention(RetentionPolicy.RUNTIME)  
 public @interface AcceptedValues {  
   
      String[] acceptValues();  
   
      String message() default "Not allowded value";  
   
      Class<?>[] groups() default {};  
   
      Class<? extends Payload>[] payload() default {};  
 }  
   

Listing 5: Custom Validator implementation

 package com.web.validator;  
   
 import java.util.ArrayList;  
 import java.util.List;  
   
 import javax.validation.ConstraintValidator;  
 import javax.validation.ConstraintValidatorContext;  
   
 public class AcceptedValidator implements  
           ConstraintValidator<AcceptedValues, String> {  
   
      private List<String> valueList;  
   
      @Override  
      public void initialize(AcceptedValues constraintAnnotation) {  
           valueList = new ArrayList<String>();  
           for (String val : constraintAnnotation.acceptValues()) {  
                valueList.add(val.toLowerCase());  
           }  
      }  
   
      @Override  
      public boolean isValid(String value, ConstraintValidatorContext context) {  
           if (!valueList.contains(value.toLowerCase())) {  
                return false;  
           }  
           return true;  
      }  
   
 }  
   

- Because of the orderDate property in Order class, mapper is required for the reader, if not implemented
reader will not know the date format.

Listing 6: Order mapper

 package com.web.mapper;  
   
 import org.springframework.batch.item.file.mapping.FieldSetMapper;  
 import org.springframework.batch.item.file.transform.FieldSet;  
 import org.springframework.validation.BindException;  
   
 import com.web.model.Order;  
   
 public class OrderMapper implements FieldSetMapper<Order>{  
   
      @Override  
      public Order mapFieldSet(FieldSet fieldSet) throws BindException {  
           Order order = new Order();  
             
           order.setCode(fieldSet.readString(0));  
           order.setServiceType(fieldSet.readString(1));  
           order.setQuantity(fieldSet.readInt(2));  
           order.setOffer(fieldSet.readString(3));  
           order.setBillType(fieldSet.readString(4));  
           order.setOrderDate(fieldSet.readDate(5, "dd-MM-yyyy"));  
             
           return order;  
      }  
   
 }  
   

- The validation process takes place in OrderProcessor class, listing 7.

Listing 7: OrderProcessor class

 package com.web.processor;  
   
 import org.springframework.batch.item.ItemProcessor;  
 import org.springframework.batch.item.validator.ValidationException;  
 import org.springframework.validation.BindingResult;  
 import org.springframework.validation.DataBinder;  
 import org.springframework.validation.ObjectError;  
 import org.springframework.validation.Validator;  
   
 import com.web.model.Order;  
   
 public class OrderProcessor implements ItemProcessor<Order, Order> {  
   
      private Validator validator;  
   
      public void setValidator(Validator validator) {  
           this.validator = validator;  
      }  
   
      @Override  
      public Order process(Order item) throws Exception {  
   
           BindingResult results = BindAndValidate(item);  
           if (results.hasErrors())  
                buildValidationException(results);  
   
           item.setOffer(item.getOffer().toUpperCase());  
           return item;  
      }  
   
      private BindingResult BindAndValidate(Order item) {  
           DataBinder binder = new DataBinder(item);  
           binder.setValidator(validator);  
   
           binder.validate();  
   
           return binder.getBindingResult();  
   
      }  
   
      private void buildValidationException(BindingResult results) {  
           StringBuilder msg = new StringBuilder();  
           for (ObjectError error : results.getAllErrors()) {  
                msg.append("-*-*-*- \n" + error.toString() + "-*-*-*- \n");  
           }  
           throw new ValidationException(msg.toString());  
      }  
   
 }  
   

5. CSV files

- The input file with invalid data that will be used src/main/resources/cvs/invalidOrderData.csv line 7 mobilex and 12-08-2015.

- File with valid data can be found in src/main/resources/cvs/orders_12082014.csv

Listing 8: Invalid order record in data

 code,service_type,quantity,offer,bill_type,order_date  
 234ac,mobile,9,small,prepaid,12-08-2014  
 235ab,fix,1,medium,postpaid_mail,14-08-2014  
 237ar,bundle,1,small,postpaid_email,10-08-2014  
 236as,bundle,1,small,postpaid_email,12-08-2014  
 239ac,fix,1,large,postpaid_mail,12-08-2014  
 230mb,mobilex,2,small,prepaid,12-08-2015  
 244rx,mobile,1,large,postpaid_email,12-08-2014  
 274cb,mobile,5,medium,prepaid,12-08-2014  
 228wb,mobile,1,large,postpaid_email,12-07-2014  
 284mn,mobile,1,small,prepaid,12-08-2014  
 285rn,fix,1,large,prepaid,12-08-2014  
 292wr,bundle,1,medium,postpaid_email,12-08-2014  
 293un,mobile,1,medium,prepaid,12-08-2014  
 304po,bundle,1,large,postpaid_email,12-08-2014  
 305is,bundle,1,large,postpaid_email,12-08-2014  
 311jr,bundle,1,small,postpaid_mail,12-08-2014  
 312ps,internet,1,fiber32,postpaid_email,12-08-2014  
 317tr,mobile,3,small,prepaid,12-08-2014  
 329qv,mobile,1,medium,prepaid,12-08-2014  

Listing 9: Valid order data

 code,service_type,quantity,offer,bill_type,order_date  
 234ac,mobile,9,small,prepaid,12-08-2014  
 235ab,fix,1,medium,postpaid_mail,14-08-2014  
 237ar,bundle,1,small,postpaid_email,10-08-2014  
 236as,bundle,1,small,postpaid_email,12-08-2014  
 239ac,fix,1,large,postpaid_mail,12-08-2014  
 230mb,mobile,2,small,prepaid,12-08-2014  
 244rx,mobile,1,large,postpaid_email,12-08-2014  
 274cb,mobile,5,medium,prepaid,12-08-2014  
 228wb,mobile,1,large,postpaid_email,12-07-2014  
 284mn,mobile,1,small,prepaid,12-08-2014  
 285rn,fix,1,large,prepaid,12-08-2014  
 292wr,bundle,1,medium,postpaid_email,12-08-2014  
 293un,mobile,1,medium,prepaid,12-08-2014  
 304po,bundle,1,large,postpaid_email,12-08-2014  
 305is,bundle,1,large,postpaid_email,12-08-2014  
 311jr,bundle,1,small,postpaid_mail,12-08-2014  
 312ps,internet,1,fiber32,postpaid_email,12-08-2014  
 317tr,mobile,3,small,prepaid,12-08-2014  
 329qv,mobile,1,medium,prepaid,12-08-2014  

6. Running orderValidationJob in eclipse

- To run the application use code in class src/main/java/com/web/app/OrderValidationApp.java Listing 10.

Listing 10: OrderValidationApp Class

 package com.web.app;  
   
 import java.util.Date;  
   
 import org.apache.log4j.Logger;  
 import org.springframework.batch.core.Job;  
 import org.springframework.batch.core.JobExecution;  
 import org.springframework.batch.core.JobParameters;  
 import org.springframework.batch.core.JobParametersBuilder;  
 import org.springframework.batch.core.launch.JobLauncher;  
 import org.springframework.context.support.ClassPathXmlApplicationContext;  
   
 public class OrderValidationApp {  
   
      private static final Logger log = Logger.getLogger(OrderValidationApp.class);  
      private static ClassPathXmlApplicationContext context;  
   
      public static void main(String[] args) {  
   
           String exitStatus = "Exit Status: ";  
   
           String[] springConfig = { "launch-context.xml",  
                     "META-INF/spring/batch/jobs/job-orderValidate.xml" };  
   
           context = new ClassPathXmlApplicationContext(springConfig);  
   
           JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");  
           Job job = (Job) context.getBean("orderValidationJob");  
   
           try {  
                JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();  
   
                jobParametersBuilder.addString("dateTime", (new Date()).toString());  
                jobParametersBuilder.addString("inputDataFile",  
                          "classpath:cvs/invalidOrderData.csv");  
   
                JobParameters param = jobParametersBuilder.toJobParameters();  
                JobExecution execution = jobLauncher.run(job, param);  
                exitStatus += execution.getStatus().toString();  
   
           } catch (Exception e) {  
                log.info(e.getStackTrace());  
           } finally {  
                if (context != null)  
                     context.close();  
           }  
   
           log.info(exitStatus);  
           log.info("Job Finished");  
   
      }  
   
 }  
 

- The result of running is in Listing 11.

Listing 11: Output of orderValidationJob execution in eclipse

 .  
 .  
 .  
 org.springframework.batch.item.validator.ValidationException: -*-*-*-   
 Field error in object 'target' on field 'serviceType': rejected value [mobilex];   
 codes [AcceptedValues.target.serviceType,AcceptedValues.serviceType,AcceptedValues.java.lang.String,AcceptedValues];  
 arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [target.serviceType,serviceType];  
 arguments []; default message [serviceType],[Ljava.lang.String;@6d6474ec]; default message [Invalid service type]  
 -*-*-*-*-*-*-   
 Field error in object 'target' on field 'orderDate': rejected value [Wed Aug 12 00:00:00 CEST 2015];  
 codes [Past.target.orderDate,Past.orderDate,Past.java.util.Date,Past];  
 arguments [org.springframework.context.support.DefaultMessageSourceResolvable:   
 codes [target.orderDate,orderDate]; arguments []; default message [orderDate]];  
 default message [Order date should be in the past]-*-*-*-   
 .  
 .  
 .  
 INFO main app.OrderValidationApp:48 - Exit Status: FAILED  
 INFO main app.OrderValidationApp:49 - Job Finished  

7. Running orderJob From Spring Batch Admin

- In pom.xml change username_tc and password_tc to suite your local tomcat. 

- Start tomcat.

- Deploy application from run configuration in goal type clean install tomcat7:redeploy , click apply and 

Run buttons.

Note: If these steps are unfamiliar read steps 4 and 5 deploying spring batch admin Link How To

- open address localhost:8080 /springBatchAdminMysql/jobs  in web browser , picture 2

Picture 2: orderValidationJob in jobs list

- Click orderValidationJob, Change dateTime parameter, click launch button.  job will completed with FAILED status.

- Click on executions -> ID number of job execution -> FAILED under column Status to see validation messages picture 3.

Picture 3: Validation messages

8. Eclipse Project

 Download project source code