Guest post by Martin Stránský, Research Scientist @GoodAI

Figure 1. GoodAI architecture development roadmap comparison (full-size)

The evaluation of artificial intelligence is a very difficult problem for a number of reasons. For example, the lack of consensus on the basic desiderata necessary for intelligent machines is one of the primary barriers to the development of unified approaches towards comparing different agents. Despite a number of researchers specifically focusing on this topic (e.g. José Hernández-Orallo or Kristinn R. Thórisson to name a few), the area would benefit from more attention from the AI community.

Nevertheless, we believe that in order to steer towards promising areas of research and to identify potential dead-ends, we need to be able to meaningfully compare existing roadmaps. Such comparison requires the creation of a framework that defines processes on how to acquire important and comparable information from existing documents outlining their respective roadmaps. Without such a unified framework, each roadmap might not only differ in its target (e.g. general AI, human-level AI, conversational AI, etc…) but also in its approaches towards achieving that goal that might be impossible to compare and contrast.

This post offers a glimpse of how we, at GoodAI, are starting to look at this problem internally (comparing the progress of our three architecture teams), and how this might scale to comparisons across the wider community. This is still very much a work-in-progress, but we believe it might be beneficial to share these initial thoughts with the community, to start the discussion about, what we believe, is an important topic.

  • A Unit of a plan is called a milestone and describes some piece of work on a part of the architecture (e.g. a new module, a different structure, an improvement of a module by adding functionality, tuning parameters etc.)
  • Each milestone contains — Time Estimate, i.e. expected time spent on milestone assuming current team size, Characteristic of work or new features and Test of new features.
  • A plan can be interrupted by checkpoints which serve as common tests for two or more architectures.
  • We will see whether a particular team will achieve their self-designed tests and thereby can fulfill their original expectations on schedule.
  • Due to checkpoints it is possible to compare architectures in the middle of development.
  • We can see how far a team sees. Ideally after finishing the last milestone, the architecture should be prepared to pass through a curriculum (which will be developed in the meantime) and a final test afterwards.
  • Total time estimates. We can compare them as well.
  • We are still working on a unified set (among GoodAI architectures) of features which we will require from an architecture (desiderata for an architecture).
  1. A target is to produce a software (referred to as an architecture), which can be a part of some agent in some world.
  2. In the world there will be tasks that the agent should solve, or a reward based on world states that the agent should seek.
  3. An intelligent agent can adapt to an unknown/changing environment and solve previously unseen tasks.
  4. To check whether the ultimate goal was reached (no matter how defined), every approach needs some well defined final test, which shows how intelligent the agent is (preferably compared to humans).
Figure 2. Overview of a meta-roadmap (full-size)
  • The definition of an ultimate target,
  • A final test specification,
  • The proposed design of a curriculum, and
  • A roadmap for the development of an architecture.
Figure 3. Meta-roadmap with incorporated desiderata for different roadmaps (full-size)

Source link