A Guide to Building Custom Data Pipelines with Datastreamer

Introduction

Organizations struggle to manage complex datasets. Datastreamer's custom data pipelines provide the infrastructure for efficient data handling, so teams can focus on business goals while converting raw data into actionable insights.

Three capabilities matter most:

• Classify large volumes of data: Organizing data into meaningful categories streamlines analysis, improves searchability, and speeds up informed decision-making.
• Move data efficiently: Transferring data across on-premises and cloud systems is critical for timely analysis.
• Scale operations effectively: As data volumes grow, scalable solutions with built-in auto-scaling become crucial, dynamically adjusting resources to meet workload demand.

Pipeline Steps Explained

Custom data pipelines follow three key steps:

1. Direct Data Upload: Brings data into the pipeline. It supports sources including S3, GCS, and PubSub, adapting to different data origins.
2. Product Sentiment Classifier: Analyzes each dataset record, extracting brand names from text and assigning sentiment scores with supporting reasoning, adding meaningful context to the data.
3. Document Inspector: Visualizes processed data in the UI. Output destinations include Datastreamer Searchable Storage, Webhook, Google Cloud Storage, Azure Blob, S3, Elasticsearch, and PubSub.

About the Data

The example uses two product reviews. One is positive, with a rating of 4.5, praising quality and design. The other is negative, with a rating of 2.0, criticizing price and build quality. Together they show how a pipeline processes and analyzes customer feedback for sentiment extraction and trend identification.

Creating the Pipeline

Open the Datastreamer Portal and create an account if you do not have one. Navigate to the Dynamic Pipeline section and select the New Pipeline button.

Direct Data Upload

This component is the data ingress point. Configuration is optional but recommended for defining where your data comes from. It lets you ingest data through the UI or over a secure HTTPS API.

Product Sentiment Classifier

Configure the classifier with these steps:

1. Click the plus (+) button below Direct Data Upload.
2. Select Product Sentiment Classifier from the menu.
3. Specify the text field to analyze (for example, review_text) in the Target Text field.
4. Results save to the destination path enrichment.product_sentiment, which you can adjust.
5. Set filter conditions for the records you want to process.

Data that does not match the criteria is not deleted. The pipeline simply skips processing it. This is useful when an ML model supports specific languages only.

Document Inspector

The final pipeline component displays content in the UI. No additional configuration is required, giving you easy access to enriched data, including the sentiment analysis results.

Deploy Your Pipeline

After saving the pipeline, click Deploy to spin up the required resources. Pipelines run in dedicated, isolated environments for performance and security.

Upload the Data

Once deployed, click Upload on the Direct Data Upload component. You can paste data as JSON, upload it as a file, or send it through the API using the generated curl commands.

Processing takes seconds, and results appear in the Document Inspector.

What's Next?

You can extend pipeline capabilities in several ways:

• Additional classifiers: Integrate more ML operations and pre-integrated NLP models for deeper analysis.
• Filter and route data: Manage data flow by creating multiple pipelines with complex routing.
• Use searchable storage and aggregations: Query and aggregate data directly within pipelines, using labels and classifiers for trend detection and sentiment analysis.

Conclusion

Datastreamer's pipelines offer a comprehensive solution for organizations looking to efficiently classify large datasets, enable smooth data movement, and scale operations with ease.

This approach simplifies data management while extracting meaningful insight from complex datasets.