AI Strategy Dimensions & NLP Pipeline

Hi You!

In today’s edition, I will share some insights into the field, things I have been working on, AI resources, and the upcoming Live bootcamp.

Happy AI.

AI Leadership 👑

Data and AI Strategy Dimensions

I discussed the dimensions with a Data Professional from Europe.

There are 2 major dimensions of AI strategy →

1. Data Engineering/Architecture

2. AI Product Management with Business ROI

You need both to reduce your risk of building a failed AI.

Whether you are integrating AI into an existing product or building a separate tool.

Data Engineering/Architecture

You have data all over and around us and you still crave insights.

I will recommend you 1 book for it.

Modern Data Architecture on AWS by Behram Irani

You will find amazing content to build your knowledge around how data platforms work.

This is specifically for AWS but once you are comfortable with any cloud provider like GCP or Azure, you can easily switch.

I will recommend spending a few months on a single cloud provider of your choice after that you can switch it if you want.

Now the second part is understanding the technology as per the business requirement in your product.

AI Product Management with Business ROI

You need to make sure the product you are building is aligned with the customer's needs.

Otherwise, you will be next in the long list of AI graveyards.

You need to understand the technology at a level that you can easily communicate with the engineering team as well as stakeholders.

You probably have this issue of not understanding the AI and Data jargon used by the engineering team.

To solve this problem I am starting a Live Bootcamp 

Neurons to GenerativeAI → Beginner Friendly, 64 Chapters, 5 Weeks, 2 hours each session.

PS → 1 : 1 AI solutions brainstorm Let’s develop your AI idea.

Concept 🧑‍💻

NLP Data Cleaning Pipeline

As humans, if you read any text, you will only remember important words.

The key information you will keep in mind, not word to word.

Similarly, we want to process only important words, not the words that are not helping in prediction.

These words are called stop words.

Removal of stop words is one of the parts of text pre-processing.

Let me give you a pipeline that will help you understand the whole pre-processing of text, before feeding it into any language model.

Text Pre-Processing Pipeline

We will start with raw text data; at the end of the process, you will see what the data will look like.

You already know different types of LLMs like GPT were trained on internet data.

You cannot just feed that text internet data directly to the model.

Why?

Because the model will consider “India” and “INDIA” as 2 different entities.

We call this, the data is not normalized.

We want our text data to be normalized and should be on the same scale so that the model learns from it.

Following are the steps of cleaning the text data and making it ready to feed into the model.

Raw Text Data

When you pull data from the internet, it can be any form. The above line is just an example.

Encoding Removal

Encoding types are UTF-8, ASCII, Unicode, etc.

These encoding types can convert a character into a number and will be converted into binary for the computer to understand.

Lower casing

All the capital letters will be changed to small letters.

Digits to Words

Remove Special Characters - @!#$.%^&*,

Correction of Spelling

Remove Stop Words

Stemming

Lemmatization

Now the final text is ready to feed into a model.