com.ibm.es.ama.zing.common.model.converter

Interface CustomConverterPlugin

All Superinterfaces:

ConverterBase

public interface CustomConverterPlugin
extends ConverterBase

Interface for all custom converter plugins. Developers can implement this interface to provide their own converter logic, and plug it to the converter pipeline.

A minimal custom converter needs to implement convert method, which applies the conversion to each Document object, and publishes possibly multiple result Document objects via given publisher of type Publisher<? super Document>.

Developers can also supply canProcess, canReadMetadataFrom, and postProcess, in order to control which crawl-data to process by this converter, which crawl-data to be used as metadata for conversion, and to set content-type to the resulting crawl-data. For example, we typically implement canProcess as follows.

 
 @Override
 public boolean canProcess(CrawlData data) {
     if (getGeneralSettings() != null) {
         return CustomConverterPlugin.super.canProcess(data);
     } else {
         return "text/csv".equals(data.getContentType());
     }
 }
 
 

If a converter configuration is available, this uses the default implementation of the method which checks typeIn and conditional fields of general settings section of the configuration. Otherwise it checks the content-type of the input crawl data manually.

Nested Class Summary

Nested classes/interfaces inherited from interface com.ibm.es.ama.zing.common.model.converter.ConverterBase

ConverterBase.ConverterStatus

Method Summary

Modifier and Type	Method and Description
default void	configure(java.util.Map<java.lang.String,java.lang.String> parameters) Interface method to configure this converter plugin by given parameters.
default ConverterBase.ConverterStatus	convert(Document document, AXMLBase.Publisher<? super Document> documentPublisher, java.lang.Iterable<Document> metaDocumentList) Interface method for implementing document-level conversion.
boolean	forcesDeletion() The implementation of this method defines a flag as to whether or not the deletion is issued for any old document with url that matches by prefix to url of crawl-data that is processed by this converter, before the results of this converter are added to the dataset.
default ConverterConfiguration	getConfiguration()
static void	main(java.lang.String[] args) Command line test runner's main method.
boolean	removesMetadata() The implementation of this method defines a flag as to whether or not after the conversion, this converter removes metadata, i.e., crawl-data for which canReadMetadataFrom is true, from the output.

Methods inherited from interface com.ibm.es.ama.zing.common.model.converter.ConverterBase

canProcess, canReadMetadataFrom, getGeneralSettings, logger, postProcess

Method Detail

removesMetadata

boolean removesMetadata()

Description copied from interface: ConverterBase

The implementation of this method defines a flag as to whether or not after the conversion, this converter removes metadata, i.e., crawl-data for which canReadMetadataFrom is true, from the output.

Specified by:

removesMetadata in interface ConverterBase

Returns:

true, if metadata needs to be removed.

forcesDeletion

boolean forcesDeletion()

Description copied from interface: ConverterBase

The implementation of this method defines a flag as to whether or not the deletion is issued for any old document with url that matches by prefix to url of crawl-data that is processed by this converter, before the results of this converter are added to the dataset. *

Specified by:

forcesDeletion in interface ConverterBase

Returns:

true, if the deletion of old documents with matching url should be issued.

configure

default void configure(java.util.Map<java.lang.String,java.lang.String> parameters)

Interface method to configure this converter plugin by given parameters.

Parameters:

parameters - key-value map

getConfiguration

default ConverterConfiguration getConfiguration()

Specified by:

getConfiguration in interface ConverterBase

Returns:

ConverterConfiguration of this converter, which is set by the converter pipeline framework.

convert

default ConverterBase.ConverterStatus convert(Document document,
                                              AXMLBase.Publisher<? super Document> documentPublisher,
                                              java.lang.Iterable<Document> metaDocumentList)
                                       throws AmaIngestionException

Interface method for implementing document-level conversion. This method is called on each AXML document, if enclosing crawl-data has appropriate content-type checked by canProcess().

Parameters:

document - input AXML document, enclosed in crawl-data for which canProcess is true.

documentPublisher - publisher for output AXML documents

metaDocumentList - metadata documents used for conversion, extracted from crawl-data for which canReadMetadataFrom is true.

Returns:

status code of the conversion

Throws:

AmaIngestionException

main

static void main(java.lang.String[] args)
          throws java.lang.Exception

Command line test runner's main method. Use this method to test and debug your converter plugin as a stand-alone application, e.g., from your Java IDE.

java com.ibm.es.ama.zing.common.model.converter.CustomConverterPlugin <class name> <input axml> [<key>=<value>]*

• <class name>: fully qualified name of plugin class
• <input axml>: input axml file fed to the converter
• <key>=<value>: parameters passed to configure method

Note that when running your plugin on the test runner, we don't supply converter configuration, so that values of getConfiguration and getGeneralSettings are both null.

Parameters:

args - command line args.

Throws:

java.lang.Exception