AI progress with time

Concepts & ideas explored:

1. Development of AI and its use from language processing models to reasoning models.
2. Predicting on technology progress can go wrong most of the times, as we have learned from history as well.
3. Ambition of developing AI that supersedes human capabilities.
4. Accelerating automation process and possibilities available within its framework of capabilities.
5. Exploring heterogenous approach for smart AI computation.
6. Higher levels of automation are not equivalent of higher level of intelligence; in same way as higher level of productivity is not equivalent of higher level of intelligence.
7. Mapping progress of AI development by drawing parallels with development of human brain through different stages of development.
8. AI stages of development.
9. Machine learning and its ability of pattern recognition, solving problems with old knowledge but not solving novel problems.
10. Limited computation access and resources solving problem fast and efficiently than with a lot of resources and computation access in unified computing environment.
11. Concept of Fluid Intelligence: Not creating new knowledge or solving for new problems.
12. AI tech needs to be evaluated on different spectrum of capabilities than human intelligence or achievement. Using fluid intelligence as a measure. Ability to solve right and being efficient at the same time.
13. Achieving Artificial General Intelligence (AGI) by solving computation problem: using massive and costly computation. Perhaps with software-level (smart coordination among compute stack) breakthroughs such problems can be solved.
14. Creating powerful AI compute stack.
15. At the core, it is about understanding the AI intelligence, how it works and produce desired outcomes.

Initially, it was large language models (LLMs) which then evolved became better, turned into specific solution models. Image generation, video generation, ones trained on education related material, others on law, medicine etc.

But it appears, the ones with transformative potential were the '𝘙𝘦𝘢𝘴𝘰𝘯𝘪𝘯𝘨 𝘔𝘰𝘥𝘦𝘭𝘴.' Perhaps, they could be more precise, and action-oriented when given the questions or asked of solutions in the form of prompts.

With the span of time, time & again, it has been proven that with novel and transformative technology, founders/creators/technologists, often happen to make incorrect predictions.

Whether it is about:

1. Scaling-up tech
2. Tech implementation
3. Tech adoption.

Some predictions are made quite early about how tech will roll out or un-roll in future. Whilst other predictions never materialize for longer duration, even after making a lapse on prediction timeline.

Moving forward in tech, automation and digitalization, AI tech firms want their products to supersede the human capabilities. The way they (humans) are able to make contributions in economy, doing valuable tasks.

This will be the next stage in comparison to our present stage of catching up with tech.

At present, everything is mechanized. Our movement through 𝐜𝐚𝐫𝐬, doing work at 𝐜𝐨𝐦𝐩𝐮𝐭𝐞𝐫, or getting educated/entertained through 𝐭𝐯/𝐬𝐜𝐫𝐞𝐞𝐧𝐬.

So, it is augmentation from present stage to next stage, with integration of AI tech into these digitalized/mechanized equipment/machines.

Accelerating automation process

When it comes to accelerate the automation process, making systems capable to do tasks on its own, they would need relevant tools and hardware. Ones that would support models training across datasets, systems and domains.

Because they are designed to follow instructions.

To do the work, get feedback, which is made at the behest of objectives/instructions.

And then this process is repeated.

Making these LLMs transform into Intelligent Systems.

AI Backend Technicalities:

When autonomous systems are expected to work and become better at doing various diverse tasks. It becomes necessary that they take inputs of different kinds (texts, audio, video etc.) and then synthesize their response for different outputs accordingly.

For example: A voice-based audio book, Image that integrates with audio etc.

Heterogenous Computing

Here comes the role of heterogenous computing. Which chooses the right tool for right task. It combines different processors (relevant/suitable) with emerging architectures (hardwares, softwares, chips etc) and produces the intended result.

Each processor is used for different task. It can be common or specialized.

For example:

1. CPUs (Centralized Processing Units) -> Core tasks
2. GPUs (Graphic Processing Units) -> Great image processing
3. NPUs (Neural Processing Units) -> Specialized AI accelerators
4. TPUs (Tensor Processing Units) -> Specialized AI accelerators

So, when a certain task is commanded through AGI (Artificial General Intelligence), heterogenous computing chooses the right processor for computation so that particular task could be carried out.

Let's say, its image generation commanded through AGI.

With heterogenous computing in place, it will choose GPUs for computation to carry out the task of image generation.

This approach is considered balanced, practical and scalable when it comes to meeting the right demand at one place using relevant resources.

And to achieve complex level of automation, software tools and frameworks (that define limitations) are put in place. They coordinate and synthesize the output required from various processing chips, connected in a same environment (unified computing).

These software tools and frameworks are called orchestration tools, which also assists developers in creating AI apps seamlessly and managing their workloads (not letting them rewrite the codes to optimize).

And these AI applications are able to interact better with external: data, tools, system, with development of standards & protocols.

"Intelligence is the ability to recombine what you already know into new patterns, to solve novel problems efficiently,"

- François Chollet: co-founder of Ndea and co-creator of the ARC Prize, an intelligence benchmark that assesses the ability to adapt to reasoning tasks that have not been seen before.

Higher levels of automation = higher level of intelligence

Some tend to misinterpret, if higher levels of automation are equivalent to achieving higher level of intelligence.

As this concept is not true for humans, it is not true for machines as well. Because higher levels of automation only increase the rate of productivity. But the same cannot be said equivalent to having higher level of intelligence.

It is similar to thinking: Higher-level of Productivity = Higher-level of Intelligence.

Which is incorrect ❌

Drawing parallel between AI and human capabilities

The developers of AI often tend to see this tech in parallel with human capabilities. They would draw comparisons between both.

To measure probably, if how much successful they have become to create something (a machine) synonymous to human capabilities.

At present, they would compare the two by comparing what level of tasks can AI do efficiently and effectively, all while comparing it to humans' ability to complete and finish the same task.

And it's (AI) development is being monitored on same scale of measurement as humans tend to develop from:

1. Being infant: with new learning capabilities,
2. Moving to toddler stage: developing cognitive abilities.
3. Followed by reaching adult stage: where they find problems, solve them, do the planning and make decisions.

AI stages of development are:

1. First phase = Sensory systems of vision and sound
2. Second phase = Language
3. Cognition = Ability to engage in forward-looking planning

AI is still in the language phase, where it is learning to mimic and talk & respond like humans do. Now working towards next phase of development of cognitive learning where it learns to develop context, solve problems and predict better.

Machine Learning & Pattern Recognition

Something which machine learning tech is able to do better by memorizing patterns. Where it learns patterns and makes predictions that are quite accurate.

It is similar to human level intelligence of generating 'inference' from content and apply it to other contexts. Solving novel problems with innovative solutions -> This is something humans can do flawlessly but AI is still in learning stages.

Machine learning systems are being made better on same learning principle of inference level intelligence of humans. Where Machine Learning systems are trained to recognize patterns and apply that knowledge of pattern recognition to new data and make accurate predictions in different fields (still in works).

But at this stage, AI can be trained on specialized data and solve problems within that framework. It, however, still needs to learn to apply that learned knowledge across different fields and solve new problems.

Think of it like Machine Learning (which can be called one of the functions of AI as whole) model, which is trained on some sports, let's say football.

Machine Learning can learn and recognize all related patterns in football, make predictions & solve novel problems within that football related knowledge domain. But it has limitation to apply that knowledge on some boardgame and make predictions or solve new problems in a boardgame problem.

To achieve this level of intelligence in Machine Learning (AI) breakthroughs in knowledge (innovation) is required — new breakthroughs in algorithms, probably. To be able to make Machine Learning (ML) better at reasoning (logic application), memory (developing context), and decision making (planning capabilities).

Some of these breakthroughs might have found in language processing models in the form of text generation. Where learning on old patterns, that knowledge (old patterns) is applied on new knowledge, while keeping the medium same, i.e. text.

For example:

Think of training language model on business, literature and mathematics. And when problem related to data analytics is given, it can use its pattern recognition learnings from business, literature and mathematics, and apply that to creating a data analytics table (when it is given data to analyze and asked to respond by giving it a curated prompt).

'And this level of application might have been successful at small levels of language processing models, which demands limited computation to be efficient, when deployed on devices like smartphone and edge devices.'

This may mean, that due to limited access to computation, these small language models seem to outperform when deployed on small devices, in comparison to when deployed in a unified computing environment (where all sorts of computation processors, hardware's and software's are connected). Where results are better, but not efficient.

Concept of Fluid Intelligence in AI:

AI can work autonomously, generate images and videos of given data and produce great works of text & art, which is called generative AI.

But it cannot produce any new knowledge or do something for which it has no training.

It means, they do not have fluid intelligence.

Because they cannot:

1. Produce new knowledge.
2. Solve for new problems from already known data or learned patterns.

That is precisely the missing capability in AI as of now.

They are good at memorizing given patterns, but they are unable to develop new patterns on their own as and when needed.

Think of it as memorizing formulas and solving problems. And not being able to create new formulas on its own to solve new novel problems. Which human brain is sufficiently capable of.

What is Tech Level Intelligence?

Then comes the debate of what then can be called machine or computer intelligence.

Every being has a different scale of measurement based on their capabilities to do the work. And then they are evaluated against those capabilities to determine their level of intelligence.

Monkeys' intelligence cannot be assessed on passing medical admission exams (MCAT), for example.

Or a fish cannot be evaluated on its ability to climb a tree as fast as Monkey can.

Such assessments (passing MCAT) are part of human achievements. And relevant for evaluation, on human level spectrum of capabilities (think of spectrum as a band of colors from light to dark, with levels of achievements or capabilities).

Therefore, intelligence measurement scale for machines and tech such as AI can be based on a concept, such as 'Fluid Intelligence'.

This could possibly mean that the capability of AI to develop contexts and then being able to apply right contexts to right works/scenarios etc.

Reaching Artificial General Intelligence (AGI)

Reaching Artificial General Intelligence (AGI) while solving the massive computation requirement problem:

AGI has been in works since long. But it gained more highlight with generative AI getting famous in year 2022.

The results that AI generate are based on the computation. It is the foundation of underlying works, that produces the result that we use through AI.

So far AI has been successful in creating specialized models that requires specific computation (hardware, software's, chips etc). If AI model has been trained on generating or solving cases related to laws. It works well and uses specific computation.

But the real challenge was achieving the Artificial General Intelligence (AGI), where it is expected to solve all general problems.

And to do that, it would require all sorts of computation (hardwares, softwares, chips, data management softwares etc) in unified computing environment.

In such a way that when such a problem is put into query, it produces the right answer (solving it) using exactly the right computation stack (hardware, software, chips etc) without using others.

Idea is to make it efficient and accurate. Trying to find all solutions in one place. While resolving and designing backend structure, accordingly.

Think of it like encyclopedia. It provides answer of every information there is. It was made achievable by using targeted words, that would go and fetch the right information page. Fetching and using the only page required, while keeping all others intact and undisturbed.

This process of achieving AGI seemed too far before 2020. Because with all the available tech and computation abilities, the timeline of achieving AGI was indeed far. But then new technology breakthrough happened in 'Deep Learning' in year 2020.

The work in AI accelerated with this new tech capabilities. And achieving AGI seemed achievable in less time with more compatible computations: using specialized software's and hardware's.

However, even with this level of achievement, the costs remain high for deploying such computation.

Perhaps, even that could be reduced by some new technology breakthrough by developing relevant software's.

The underlying idea is to speed up the goal achievement by optimizing computation achievement using energy efficiently.

And it is being tried to achieve by Creating Powerful AI Compute Stack:

1. Infrastructure at base: Data centers (chips, servers, cables, cooling equipment etc.)

2. Hardware: Chips (GPU's etc)

3. Software:

a) Software's that enable use of specialized GPU's

b) Domain specific language optimized for machine learning and

c) Data management software

Focus is on such software development that connects and use the right hardware for compute, on need-to-use basis, and achieve the AGI.

And to do so, design compute architectures that are optimized for speed, latency, bandwidth and energy. Using heterogenous approach as the best option.

But simultaneously open to new possibilities of achieving the same, either through:

1. Developing a new architecture
2. Synthesis of philosophies & approaches: collaborating across industries
3. Using combination of specialized hardware & heterogenous approach
4. Breakthroughs in architectural design that may create new type of understanding or logic.

At the core of it all, is not about building smarter machines but actually trying to understand the type of intelligence the AI unfolds in future either through AGI or specialized learning models.

Source: https://ter.li/mittr_arm_ebrief_0825 by MIT Technology Review

Relevant words:

1. #ReportTheRoadToArtificialIntelligence
2. #TechSeries
3. #LLMs
4. #ReasoningModels
5. #Report
6. #NovelTech
7. #TransformativeTech
8. #IncorrectPredictions
9. #Automation
10. #Digitalization
11. #ValuableTasks
12. #Augmentation
13. #AIIntegration
14. #DesignedSystem
15. #Loops
16. #Objective
17. #IntelligentSystems
18. #AIbackend
19. #Technicalities
20. #AutonomousSystems
21. #ProcessingChips
22. #Intelligence
23. #NewPatterns
24. #SolveNovelProblems
25. #Productivity
26. #Comparing
27. #Measurement
28. #AIPhases
29. #ContextDevelopment
30. #PredictBetter
31. #Computation
32. #SmallLanguageModels
33. #Smalldevices
34. #Better Results
35. #FluidIntelligence
36. #AICapabilities
37. #GenerativeAI
38. #DevelopingContext
39. #ReachingAGI
40. #Computation
41. #AIComputeStack
42. #AGI
43. #DeepLearning
44. #Breakthrough
45. #Hardware
46. #SoftwareDevelopment

From author: I have drafted this as part of my learning about AI and how it works. This is not a well-researched or synthesized document, but something that I have built upon on existing knowledge to explore and understand better. Initially, it was in bits of posts, which I later compiled and turned into reading stack, as it appears now.

Happy to learn 🙌

Thank you for reading.