In-Pipeline Data Aggregations for Smarter Document Analytics

What are aggregations?

Aggregations are analytical operations that condense large data volumes without returning individual documents. Rather than processing thousands or millions of entries separately, aggregations categorize data and perform calculations across those categories to reveal trends, patterns, and summaries.

Their capabilities include:

• Understand trends: monitor how sentiment regarding your brand evolves over time.
• Identify patterns: discover which topics frequently co-occur with product mentions.
• Summarize data: surface top categories, terms, or users within a dataset.

Common aggregation types:

• Terms aggregation: locates the most frequent values in a dataset, such as popular hashtags or active users.
• Date histogram aggregation: segments data by time periods to spotlight trends, such as daily post volumes or weekly sentiment changes.
• Significant terms aggregation: identifies unusually frequent terms in a data subset compared to the overall dataset, potentially detecting spikes related to emerging issues or viral events.

The power of aggregations inside your pipeline

Aggregations show their greatest value when deployed as a starting point within a Datastreamer pipeline. Instead of simply producing results, the condensed output, called "buckets," routes directly to the next pipeline stage for additional processing, transformation, or action.

Example scenario. When analyzing social media posts about BrandX, the pipeline first aggregates sentiment scores, then directs results in two directions:

• A custom function assesses whether negative sentiment has spiked.
• A Firehose Egress transmits daily topic summaries to your analytics platform.

When a sentiment spike is detected, a Webhook Egress activates real-time alerts.

Using aggregations within pipelines lets you:

• Automate complex reporting: generate daily summaries of critical metrics and route them to different teams or platforms.
• Trigger proactive actions: when sudden negative sentiment spikes or specific events occur, your pipeline automatically dispatches alerts to the appropriate teams.
• Enrich summarized data: feed aggregation results into custom functions or AI classifiers for deeper, more actionable insights.

These features turn your pipeline into a dynamic system that not only analyzes data but acts upon it.

Real-world use cases with social media data

The following examples show how to use the Datastreamer Searchable Storage Ingress. They assume you have:

1. Established a Datastreamer Searchable Storage using the Searchable Storage Egress component. This component loads your data into the configured Storage instance.
2. Loaded the storage with data using the unified schema, made available through our Unify Transformer.

Your actual storage content may vary based on specific data sources and transformation pipelines. Always verify available fields by checking your instance in the Datastreamer Portal before implementing any of the examples below.

All examples follow a comparable structure: the aggregation runs against your Datastreamer Searchable Storage, and the results route through the Webhook Egress component, which transmits the output to an external API for additional use.

1. Trending hashtag analysis

Scenario: surface the top five trending hashtags in Instagram posts about sustainability.

Lucene query: content.body:sustainability

Aggregation JSON:

{
  "top_hashtags": {
    "terms": {
      "field": "content.hashtags.keyword",
      "size": 5
    }
  }
}

This configuration uses an Elasticsearch terms aggregation on the content.hashtags field. It determines the top 5 most frequent hashtags by counting exact matches across documents. The results sort by frequency, delivering a clear view of the most prominent tags in the dataset.

The aggregation result might resemble:

This output shows which sustainability-related hashtags trend most. For instance, "sustainability" appeared in 2,541 posts, making it the most popular tag in the dataset. This aggregation type helps you rapidly identify which environmental topics resonate with your audience right now.

2. Sentiment over time tracking

Scenario: monitor weekly sentiment trends for a brand based on social media mentions.

Lucene query: content.body:sustainability

Aggregation JSON:

{
  "weekly_sentiment": {
    "date_histogram": {
      "field": "content.published",
      "calendar_interval": "week"
    },
    "aggs": {
      "avg_sentiment": {
        "avg": {
          "field": "enrichment.sentiment.confidence"
        }
      }
    }
  }
}

This aggregation uses the date_histogram function to partition content into weekly intervals based on the content.published field. It then determines the average sentiment confidence for each week using values from the enrichment.sentiment.confidence field.

In doing so, you generate a weekly sentiment trendline, one that monitors how public perception of a brand shifts over time.

Sample output:

This output reveals weekly sentiment shifts. For example, from May 1 to 7, 142 mentions showed an average sentiment score of 78% positive. In the following week, that declined to 62%, based on 89 mentions. This kind of decrease might suggest an issue worth investigating, such as a product concern or altered public messaging.

3. Influencer impact analysis

Scenario: surface the most influential authors taking part in tech-related discussions.

Lucene query: content.body:AI

Aggregation JSON:

{
  "influential_authors": {
    "significant_terms": {
      "field": "author.name.keyword",
      "background_filter": {
        "match_all": {}
      }
    }
  }
}

This aggregation uses the significant_terms function to highlight statistically significant authors. It examines the author.name.keyword field and compares term frequency within the query context (in this case, AI discussions) against the baseline frequency in the broader dataset, using a background_filter.

By identifying terms that appear disproportionately in filtered results compared to the entire dataset, this method surfaces authors who are unusually prominent or active within a specific topic, such as AI.

Sample output:

This output shows that @FutureOfTech was mentioned 85 times overall, with 67 of those mentions appearing in AI discussions, making their AI-related content roughly three times more frequent than average. Meanwhile, @TechAnalyst_AI shows an even stronger AI focus, based on both volume and statistical score.

This analysis helps you identify high-signal contributors whose commentary could shape narratives within a particular domain.

Turn aggregations into actionable intelligence

Aggregations within pipelines convert raw data into decision-ready insights. By condensing information at scale and passing it directly into automated workflows, you gain capabilities such as:

• Proactive monitoring: detect sentiment shifts, emerging hashtags, or sudden engagement spikes before they escalate into crises or missed opportunities.
• Resource optimization: direct your analysts' focus toward high-value anomalies, such as statistically significant influencers, rather than manually reviewing thousands of documents.

As the examples above show, whether you track sustainability trends, monitor brand sentiment, or identify niche voices, Datastreamer pipelines convert aggregations into automation-ready insights. Instead of merely processing data, you build an always-on analytics engine that adapts in real time to your audience.

Advanced use cases

Custom Functions extend beyond basic tasks. They also support advanced capabilities like external data enrichment, document normalization, and schema-aware processing.

Enriching documents with external data

Custom Functions can enrich your documents by connecting to third-party APIs or databases. For instance, if your document includes an address, you can contact an external geocoding service and append the corresponding coordinates.

Ready to get started?

1. Explore how it works by reviewing our Datastreamer Storage Egress and Datastreamer Storage Ingress components.
2. Experiment with chaining aggregations to notifications using the Webhook Egress component.
3. Connect with our team. We will help you design a pipeline that turns your data into a strategic advantage.