Skip to main content
By default, Lucidworks Search stores signals in Solr. Alternatively, you can choose to store signals in Google Cloud Storage or Amazon S3. Storing signals in the cloud reduces the amount of data stored on a Solr cluster. Signals data files are periodically compacted into larger files to save storage space, improve performance, and make it easier to manage the files.
This feature is only available in Lucidworks Search 5.9.x for versions 5.9.6+.
Cloud signal storage is enabled at the time of deployment. To enable cloud signal storage for a new Lucidworks Search deployment, you must configure this feature in the deployment options. All signal types are supported, including custom signal types. The default signal schema can be customized.

Cloud Signal Storage

The course for Cloud Signal Storage focuses on how to use cloud signal storage in Fusion to enhance your data storage efficiency.
FAQCan I migrate existing signals to cloud signal storage?Currently, there is no migration path from default signal storage in Solr to cloud signal storage. This improvement is planned for a future release. To use cloud signal storage, start with a new deployment.

​
Data flow

Kafka Both internally and externally originated signals flow through the signals indexing pipeline. In contrast to default signals storage, when cloud signal storage is enabled, signals are not sent to Solr. Instead, signals are sent to the fusion.system.cloud-signals Kafka topic, the message system used by Lucidworks Search clouds signals to buffer signals before they are written to cloud storage. Signals consumer A signals consumer, which runs within a Kubernetes pod, polls the Kafka topic frequently and writes signals data files in Parquet format to a stable, temporary cloud storage location. Due to the write frequency, this results in a large number of small files. Signals compactor A signals compactor, which also runs within a Kubernetes pod, periodically compacts the signals data files sent by the signals consumer and writes them to a smaller number of large files. In contrast to aggregation, compaction does not change the data itself. The compacted signals data files are stored separately from the uncompacted signals data files. The compaction frequency is configurable, to meet the needs of your use case. Spark SQL aggregations Lucidworks Search Distributed Compute schedules and runs Spark jobs, which operate over the compacted Parquet partitions written by the signals compactor. The output is written to Solr for use at query time for boosting and recommendations.

​
Cloud signal schema

Signals are stored in Apache Parquet file format, which is designed to handle flat, column-oriented data.

​
Known issues

Some features in Lucidworks Search rely on signals data stored in Solr, not cloud signal storage. The following list contains features that are not supported: Other features not listed here may be impacted.