Pulsar
| Function | Description |
|---|---|
Schema Migration | If the specified Topic after mapping does not exist in the Target, BladePipe will automatically create the Topic, allowing setting the number of partitions. |
Incremental Data Sync | Allow subscribing to messages from the source Topic. |
Subscription Modification | Add, delete, or modify the subscribed topics. For more information, see Modify Subscription. |
Position Resetting | Reset positions by timestamp to consume the data in a past period again. |
Advanced Functions
| Function | Description |
|---|---|
Message Format | The following message formats are supported. See Message Format.
|
Limits
| Limit | Description |
|---|---|
Creating Tables in the Target in Advance | Only support automatic Topic creation for messages. |
Prerequisites
| Prerequisite | Description |
|---|---|
Port Preparation | Allow the migration and sync node (Worker) to connect to the nodes of Pulsar. |
Parameters
| Parameter | Description |
|---|---|
schemaFormat | Message format. For more information, see Message Format. |
subscriptionMode | Pulsar subscription mode, including Durable and NonDurable. |
consumeParallel | Degree of consuming Pulsar topics in parallel. |
receiverQueueSize | The size of a Pulsar receiver queue. |
receiverMaxNumBytes | The maximum bytes of a batch received from Pulsar (in bytes). |
receiverTimeout | The timeout period for receiving a batch from Pulsar (in seconds). |
connectionTimeoutMs | The timeout period for connection to Pulsar Client (in milliseconds). |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Port Preparation | Allow the migration and sync node (Worker) to connect to the nodes of Pulsar. |
Parameters
| Parameter | Description |
|---|---|
schemaFormat | Message format. For more information, see Message Format. |
batchWriteSize | The maximum data size of a single message. If the size exceeds the limit, the message will be split. |
enableBatching | Enable Pulsar to push messages in batches. |
batchingMaxBytes | The maximum bytes of messages in a batch pushed by Pulsar (in bytes). |
connectionTimeoutMs | The timeout period for connection to Pulsar Client (in milliseconds). |
compressionType | Set compression algorithm. Support LZ4, ZLIB, ZSTD, SNAPPY. |
envelopSchemaInclude | When schemaFormat is set to DEBEZIUM_ENVELOP_JSON_FOR_MQ, it means whether the message body contains schema information. |
Tips: To modify the general parameters, see General Parameters and Functions.