Top 8 NER APIs for Natural Language Processing

Given that natural language processing (NLP) is at the heart of online data extraction and named entity recognition (NER) is one of its key tools, let us explore which is the best Named Entity Recognition API at the core of any NLP application, across everything from text-based semantic search to video AI.

What Is NER?

Named entity recognition is a machine learning (ML) technique that breaks text up semantically, identifying parts of a sentence which fit into predefined categories. NLP is the AI-driven process that analyzes the language and draws out data and meaning from it. In short, it’s what we humans do every day when we read.

While NLP is an important development in the text analytics process, NER is the foundation on which an AI algorithm depends to analyse text for relative semantics and sentiment For instance, consider this comment:

“Richard Branson, founder of Virgin Galactic, is now offering manned space flights for as little as $200,000.”

This text contains information in the following categories (respectively) - named individual, job role, brand, topic, and monetary value. Because NER helps the ML model to understand the components of a text, it makes the sentiment analysis model increasingly sophisticated at understanding which parts of a sentence encode specific bits of information that are important to your business.

Why does NER matter?

NER is at the core of a text analytics process. To get an ML model to comprehend language it is first fed millions of examples in which the categories have already been specified. With iterations, the API becomes adept at identifying these elements in texts it is encountering for the first time. The more adept and robust the NER capability is, the more powerful is the text analytics engine.

NER is the catalyst behind many ML functions as shown below.

• Semantic Search

Google is now a semantic search engine. You can input a question, to which it will do its best to provide an answer. Alexa, Siri, chatbots and other digital assistants all use a form of semantic search to retrieve information based on identifying what data a user is seeking. This function can be a little hit and miss, but its applications are ever-expanding, and their efficiency is improving exponentially.

• Data Analytics

This is a catch-all term for the use of algorithms to derive analyses from raw data. It combines the act of identifying and extracting the relevant data with methodologies for displaying this data. This can range from a simple statistical presentation of what has been found, to graphic displays of its findings.

Data from YouTube views (including when viewers click off a particular video) can be used to analyze interest in and engagement with a particular topic. Data scraping of Ecommerce platforms can analyze product star ratings and give an aggregated score of how a product is being received.

• Text Analytics

Like data analytics, text analysis pulls information from raw text strings, using NER to focus in on the relevant information. It can be used to aggregate the mentions of a product, or the average price of a product, or which adjectives users are most commonly associating with a brand.

• Sentiment Analysis

Digging even further into the field of NER, sentiment analysis can tell the difference between a positive and negative review, even without data from star ratings. It knows that “overrated”, “fiddly” and “dumb” have negative connotations and that “helpful”, “fast” and “easy” have positive connotations.

Sophisticated systems adjust for context (“easy” might have a negative connotation in a computer game, for instance) and might understand the relation between entities. For example, “founder of Virgin Galactic” links a previously named individual to a company by means of their role.

• Video Content Analysis

Perhaps most complex of all, are systems that use facial recognition, speech analysis and image recognition to derive data from video content. Video content analysis will identify “unboxing” videos on YouTube, Twitch demos of your game, lip syncs of your audio content on TikTok, and more.

As the amount of online video content explodes, quicker and cleverer systems for video content analysis using NER are vital in order not to miss valuable information about how customers relate to your product or service.

So as you can see, NER is a vital component of NLP, and its potential is virtually limitless. Let’s now have a look at eight of the best NER APIs on the market, how well they performed in our benchmarking tests, and what opportunities each of them provides.

What Are The Top 8 NER APIs in 2021?

Repustate takes pride in its NER offering, which is the foundation of its deep semantic search capability. Our intuitive AI-driven NER API infers context within a query, providing higher accuracy results by balancing four crucial areas - accuracy, granularity, languages, and speed. There are other first-class NER APIs available such as Google Cloud NLP, Amazon Comprehend, Dandelion, etc. so we decided to evaluate them on all four criteria using open source code and input data in a bid to be fair and transparent for this Top 8 list.

1. Repustate’s NER API

In our comparative benchmarking test, our NER API came out on top in all four categories. Repustate offers Named entity recognition in 23 languages, was 95% accurate and offered a high degree of granularity across figures and text which our rivals couldn’t surpass.

We’re fast – with only a 60ms latency period, which is the least of all the HTTPS NLP tools we examined. It’s not a surprise that Repustate performed so well in our tests because we designed our NLP systems specifically to dig deeper and return more accurate data than our competitors.

2. Google Cloud NLP

Google’s NLP API offering performs quite well too, with up to 75% accuracy across a wide range of data categories and ten distinct languages. However, it’s not especially fast, with a latency period of over 1000ms, and its pricing structure is a little opaque.

Google Natural Language charges you per “unit”, a string of 1000 characters being equivalent to one unit. It also charges separately for entity, syntax, and sentiment analysis, and for content classification, meaning your expenditure can skyrocket, once you exceed the free tier of 5000 units. Google’s offering includes a healthcare specific NLP suite, which allows it to identify complex scientific terms.

3. Microsoft Azure Cognitive

Microsoft, offers access to its ML and NER technology via the Azure Cognitive Services suite. Aimed squarely at developers, it integrates with Microsoft’s Applied AI Services to help you customize off the shelf AI solutions.

However, in tests, Azure demonstrated an accuracy rating of only 50% (let’s face it, nobody prefers Bing to Google as a search engine) and a long latency period of over 3200ms. It is easy to use but does not score highly with users for customer support.

4. Dandelion

Dandelion performs well for semantic search and sentiment analysis functions, and as a specialist cloud-based service operating across seven European languages, it is impressive. It’s fast too, with a latency of just 250ms. That said, Dandelion’s NLP API manages to be only 63% accurate, better than Microsoft but a long way from comprehensive.

Their free tier operates up to 1000 units per day or 30,000 per month, which may seem generous, until you realize how even the best ML system devours text to improve, and just how vast your target HTTPS sites may be. In addition, its sentiment analysis function may not be hugely helpful in identifying the source of a sentiment, as this developer found.

5. Aylien

Aylien describes itself as a “News Intelligence Platform” that scans over 80,000 mainstream and long-tail news sources. If you are primarily focused on what’s trending, or what reviewers are saying about your product, this could be a usefully granular tool. However, it boasts the lowest overall accuracy of any of the systems we tested, with just 42% accuracy over all use cases. At six languages, it isn’t the most comprehensive API on the market, either. There’s no free tier, but Aylien does offer a free 14-day trial of the basic Pro version.

6. Amazon Comprehend

The last of the “big three” tech giants offering NLP APIs, Amazon’s Comprehend is 67% accurate in its entity identification. It’s reasonably fast too, with a 160ms latency lag. Like the Google NLP offering, it provides a medicine-specific version too.

It integrates with Amazon’s S3 and Redshift products as part of the overall Amazon Web Services suite, so if you have petabytes of text to scan, it’s a viable option. Like Google, their pricing structure is somewhat Byzantine, and they even provide a calculator to help you estimate your costs. You will benefit from economies of scale with Comprehend but it’s less valuable for smaller data pools.

7. SpaCy

Built on the Python programming platform, SpaCy describes itself as “industrial strength NLP’’ although its 45% accuracy performance in benchmarking might belie its confidence. It is exceptionally fast, with just 30ms of lag time but, crucially, it is not designed for use over HTTPS pages. Instead, it runs locally, making this a good solution for scanning your own servers or intranet, but not a tool for analyzing the wider web.

8. TextRazor

TextRazor is a comparatively sophisticated independent API, which scores well for speed (240ms latency), languages (12) and accuracy (61%). They offer five subscription tiers, and the free level allows for up to 500 requests per day, across their whole range of functions. Overall, it was one of the most impressive rivals to our own NLP API. It was the only NLP tool we assessed which could identify stock ticker symbols (i.e., $APPL = Apple). However, why settle for reasonably good, when you can have excellence instead?

Ending Notes

When CEOs and managers are attempting to gain real insight into customer satisfaction, they must move beyond the merely anecdotal. As a recent article in Forbes showed, more and more industry leaders are turning to Named entity recognition for greater confidence that their data analyses are accurate.

NER allows businesses to identify trends and sentiments whilst they are happening, meaning they can make better informed choices more quickly. Even with a global pandemic slowing economic growth, the NLP industry is anticipating a boom, with projected compound annual growth of 29.4% from 2021 to 2028, according to Fortune Business Insights.

With your competitors leveraging AI and NLP so assiduously, can you afford to turn a blind eye to its efficacy? With NER being such a vital part of any NLP tool, Repustate has been designed to provide the most detailed, accurate and useful data analysis on the market.