i'm trying use scalatest , spark-testing-base on maven integration testing spark. spark job reads in csv file, validates results, , inserts data database. i'm trying test validation putting in files of known format , seeing if , how fail. particular test makes sure validation passes. unfortunately, scalatest can't find tests.
relevant pom plugins:
<plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-surefire-plugin</artifactid> <configuration> <skiptests>true</skiptests> </configuration> </plugin> <!-- enable scalatest --> <plugin> <groupid>org.scalatest</groupid> <artifactid>scalatest-maven-plugin</artifactid> <version>1.0</version> <configuration> <reportsdirectory>${project.build.directory}/surefire-reports</reportsdirectory> <wildcardsuites>com.cainc.data.etl.schema.proficiency</wildcardsuites> </configuration> <executions> <execution> <id>test</id> <goals> <goal>test</goal> </goals> </execution> </executions> </plugin>
and here's test class:
class proficiencyschemaitest extends flatspec matchers sharedsparkcontext beforeandafter { private var schemastrategy: schemastrategy = _ private var datareader: dataframereader = _ before { val sqlcontext = new sqlcontext(sc) import sqlcontext._ import sqlcontext.implicits._ val datainreader = sqlcontext.read.format("com.databricks.spark.csv") .option("header", "true") .option("nullvalue", "") schemastrategy = schemastrategychooser("dim_state_test_proficiency") datareader = schemastrategy.applyschema(datainreader) } "proficiency validation" should "pass csv file proficiency-valid.csv" in { val datain = datareader.load("src/test/resources/proficiency-valid.csv") val valid: try[dataframe] = try(schemastrategy.validatecsv(datain)) valid match { case success(v) => () case failure(e) => fail("validation failed on should have been clean file: ", e) } } }
when run mvn test
, can't find tests , outputs message:
[info] --- scalatest-maven-plugin:1.0:test (test) @ load-csv-into-db --- [36mdiscovery starting.[0m [36mdiscovery completed in 54 milliseconds.[0m [36mrun starting. expected test count is: 0[0m [32mdiscoverysuite:[0m [36mrun completed in 133 milliseconds.[0m [36mtotal number of tests run: 0[0m [36msuites: completed 1, aborted 0[0m [36mtests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0[0m [33mno tests executed.[0m
update
using:
<suites>com.cainc.data.etl.schema.proficiency.proficiencyschemaitest</suites>
instead of:
<wildcardsuites>com.cainc.data.etl.schema.proficiency</wildcardsuites>
i can 1 test run. obviously, not ideal. it's possible wildcardsuites broken; i'm going open ticket on github , see happens.
this because there space characters in project path. remove space in project path , tests can discovered successfully. hope help.
Comments
Post a Comment