Skip to main content
Custom Spark jobs run Java Archive (JAR) files uploaded to the blob store. To configure a custom Spark job, sign in to Lucidworks Search and click Collections > Jobs. Then click Add+ and in the Custom and Other Jobs section, select Custom Spark Job. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.

​
Basic parameters

To enter advanced parameters in the UI, click Advanced. Those parameters are described in the advanced parameters section.
  • Spark job ID. The unique ID for the Spark job that references this job in the API. This is the id field in the configuration file. Required field.
  • Class name. The fully-qualified name of the Java/Scala class to invoke for this job. This is the klassName field in the configuration file. Required field.
  • Script ARGs. The arguments (ARGs), which are additional options to pass to the application when running this job. This is the submitArgs field in the configuration file. Optional field.

​
Advanced parameters

If you click the Advanced toggle, the following optional fields are displayed in the UI.
  • Spark Settings. This section lets you enter parameter name:parameter value options to configure Spark settings. This is the sparkConfig field in the configuration file.
  • Scala Script. The value in this text field overrides running className.main(args) default behavior. This is the script field in the configuration file.
Alternatives to custom Spark jobs are Custom Python job configuration specifications and Scala Script job configuration specifications.
Each custom Spark job run has its own driver pod. To get driver logs, see the “Get Logs for a Spark Job” section in Spark Jobs.

​
Learn more

​
Number of instances and cores allocated

To set the number of cores allocated for a job, add the following parameter keys and values in the Spark Settings field. This is done within the “advanced” job properties in the Lucidworks Search UI or the sparkConfig array of key/value objects, if defining a job via the Lucidworks Search API.If spark.kubernetes.executor.request.cores is unset, the default configuration, Spark sets the number of CPUs for the executor pod to be the same number as spark.executor.cores. For example, if spark.executor.cores is 3, Spark allocates 3 CPUs for the executor pod and runs 3 tasks in parallel. To under-allocate the CPU for the executor pod and still run multiple tasks in parallel, set spark.kubernetes.executor.request.cores to a lower value than spark.executor.cores.The ratio for spark.kubernetes.executor.request.cores to spark.executor.cores depends on the type of job: either CPU-bound or I/O-bound. Allocate more memory to the executor if more tasks are running in parallel on a single executor pod.If these settings not specified, the job launches with a driver using one core and 3GB of memory plus two executors, each using one core with 1GB of memory.

​
Memory allocation

The amount of memory allocated to the driver and executors is controlled on a per-job basis using the spark.executor.memory and spark.driver.memory parameters in the Spark Settings section of the job definition. This is found in the Lucidworks Search UI or within the sparkConfig array of key/value objects in the JSON definition of the job.

​
Configuration properties