AI-Driven Development

Artificial Intelligence (AI) has made remarkable progress in the last decade, revolutionizing fields such as robotics, computer vision, machine learning, and natural language processing. Researchers have developed more efficient and powerful models to tackle increasingly complex problems, leveraging large datasets and powerful computing resources.

Software development is one of the areas that is beginning to benefit from AI, with tools such as Copilot, Codex, and the recently released ChatGPT, among others, aiding developers in tasks such as code completion, bug prevention, code review, and even code generation, in response to prompts written in natural language. However, this is just the beginning of the boost that AI will bring to software development. In this article, we will review the current state of AI-driven software development and what the future may hold.

Underlying AI Technologies

Recent advances in areas of AI, such as machine learning, natural language processing, and deep learning, have sparked interest from big tech companies to develop AI-based products that power the software development industry. Here is a brief description of the most relevant AI technologies that underlie the AI-Driven Development.

Machine Learning (ML): Is a subset of AI that deals with the development of algorithms and models that enable machines to learn and improve from experience without being explicitly programmed. Machine learning models can be trained on data and then used to make predictions or decisions.

Natural Language Processing (NLP): Is a field of AI that deals with the interaction between computers and human languages. NLP technologies include text analysis, sentiment analysis, speech recognition, and machine translation.

Deep Learning (DL): Is a subset of machine learning that deals with the use of deep neural networks to analyze and understand data. Deep learning models are composed of multiple layers of interconnected nodes and are used for tasks such as image and speech recognition.

Supervised Learning (SL): Is a type of machine learning where the model is trained on a labeled dataset. The model learns to predict the output based on the input and the labeled output.

Unsupervised Learning (UL): Is a type of machine learning where the model is trained on an unlabeled dataset. The model learns to find patterns and structure in the data without any labeled output.

Reinforcement Learning and Human Feedback (RLHF): Is a type of machine learning where the model is trained using reinforcement learning and human feedback. The model learns to improve its performance based on the feedback it receives from humans.

Neural Network (NN): Is a type of machine learning algorithm that is modeled after the human brain. Neural networks consist of layers of interconnected nodes or neurons that are used to process and analyze data.

Convolutional Neural Network (CNN): Is a type of neural network that is commonly used for image and video recognition. CNNs use a technique called convolution to extract features from images and then classify them.

Recurrent Neural Network (RNN): Is a type of neural network that is used for sequential data such as time series or natural language. RNNs have a feedback loop that allows them to remember previous inputs and use them to make predictions.

Transformer Model (TM): Is a type of neural network architecture that is used for NLP tasks such as language translation and text summarization. The transformer model is based on the attention mechanism which allows it to focus on specific parts of the input while processing it. TM was introduced in the paper "Attention is All You Need" by Google researchers in 2017 and since then has been used to achieve state-of-the-art performance on a wide range of NLP tasks due to its ability to parallelize computations, which significantly reduces the amount of time needed for training.

Large Language Model (LLM): Is a type of neural network that is trained on a large dataset of text and is used for natural language processing tasks. Large language models such as GPT-3 have the ability to generate human-like text and have a wide range of applications in natural language generation, summarization, and question-answering.

What Is AI-Driven Development?

AI-Driven Development is a cutting-edge approach that utilizes the power of artificial intelligence to assist with various tasks in the software development process. One key difference between AI-Driven Development and other approaches is the use of machine learning algorithms and natural language processing to analyze and understand code, as well as to generate code suggestions and assistance. This can save developers time and effort by automating repetitive tasks and reducing the need for manual coding.

Additionally, the use of AI can potentially lead to higher-quality code by identifying and addressing errors and inconsistencies in real-time. Furthermore, AI-Driven Development can improve collaboration among team members by providing them with real-time feedback, suggestions and insights. It also enables developers to focus on more complex and creative tasks, rather than spending time on repetitive and time-consuming tasks. AI-Driven Development is rapidly gaining popularity in the software development industry and is expected to play a major role in the future of software development.

Prominent AI Tools

Next, we will briefly describe the most relevant AI tools that have emerged in the last two or three years and that currently already are, or will become in the near future, a great support for the software development sector.

1. Github Copilot

Copilot is a code completion tool developed by GitHub and OpenAI and released at the end of 2021. It uses OpenAI's Codex, a transformer model trained on billions of code lines from GitHub, to generate code based on the current file's contents and the user's cursor location. This makes coding faster and more efficient by reducing the need for manual coding and increasing the accuracy of generated code.

Copilot is compatible with popular code editors such as Visual Studio Code, Visual Studio, Neovim, and JetBrains IDEs, and supports a wide range of programming languages including Python, JavaScript, TypeScript, Ruby, and Go. This enables developers to use Copilot across a variety of projects and programming languages.

One of the most powerful features of Copilot is its ability to generate entire code lines, functions, tests, and documentation. It does this by analyzing the context of the code and leveraging the work of developers who have committed their code to GitHub, regardless of their software license. This means that developers can benefit from the knowledge and experience of other developers, even if they are working on different projects or using different languages.

Another important feature of Copilot is its ability to provide suggestions for code as you type. It will automatically suggest variables, functions, and other code elements based on your current context, making it easier to write code quickly and efficiently.

When the Copilot beta ended, GitHub announced the pricing for individual users. The subscription includes a 60-day free trial, with a monthly rate of $10 or a yearly rate of $100 per user. This makes it accessible for developers of all levels and ensures that they can continue to improve their coding skills with the help of GitHub Copilot.

2. OpenAI ChatGPT

ChatGPT is a state-of-the-art AI chatbot developed by OpenAI and launched as a prototype in November 2022. ChatGPT uses advanced NLP to engage in realistic conversations with humans. ChatGPT can generate articles, fictional stories, poems, and even computer code. ChatGPT also can answer questions, engage in conversations and, in some cases, deliver detailed responses to highly specific questions and queries.

ChatGPT is a LLM and is based on the GPT-3.5 (Generative Pretrained Transformer) architecture and uses a transformer neural network to generate human-like text. ChatGPT is fine-tuned on a dataset of human-generated text and can be used for a wide range of NLP tasks such as language translation, text summarization, question answering, and code generation. It has 175 billion parameters and allows the model to understand and generate text with a high level of complexity and accuracy.

ChatGPT uses a technique called unsupervised pre-training, where the model is trained on a large dataset of text without any specific task in mind. This allows the model to learn general patterns and features of the language, which can then be fine-tuned for specific tasks. The fine-tuning process involves training the model on a smaller dataset of text that is specific to the task at hand. This allows the model to adapt to the specific task and perform better on it.

One of the key strengths of ChatGPT is its ability to generate human-like text. The model is trained on a dataset of human-generated text, which allows it to learn the nuances and patterns of human language. Additionally, the RLHF (Reinforcement Learning with Human Feedback) technique allows the model to learn from the preferences and biases of the humans who provided the feedback, further improving the model's ability to generate human-like text.

In terms of applications, ChatGPT can be used for a wide range of natural language processing tasks such as language translation, text summarization, and question answering. It can also be used for coding tasks such as code generation and code completion. It can be fine-tuned on specific coding datasets and then used to generate code snippets or complete code blocks. The coding tasks are supported behind the scenes by the specialized OpenAI Codex model.

3. OpenAI Codex

OpenAI Codex is an AI model developed by OpenAI and released as an API in closed beta in mid-2021. Codex parses natural language and generates code in response. It is used to power GitHub Copilot and behind the scenes by ChatGPT. Codex is a descendant of OpenAI's GPT-3.5 model, fine-tuned for use in programming applications and has 12 billion parameters.

OpenAI Codex is a descendant of GPT-3.5. its training data contains both natural language and billions of source code lines from publicly available sources (159 gigabyte from 54 million repositories), including code in public GitHub repositories. OpenAI Codex is most capable in Python, but it is also proficient in over a dozen languages including JavaScript, Go, Java, Perl, PHP, Ruby, Swift, TypeScript, SQL, and even Shell.

OpenAI Codex aims to make coding more accessible by allowing developers to express their intent in natural language, rather than requiring them to know the syntax of a specific programming language. Additionally, it can also help to reduce the time and effort required to write code and it could be particularly useful for developers who are not familiar with a specific programming language or who need to quickly prototype an idea.

However, it's worth mentioning that OpenAI Codex is still in its early stages, and it might have limitations in its ability to understand more complex or specialized requirements, also it's not a replacement for the developer but more of an assistive tool.

Sample Case with OpenAI Codex

We present an example of code generated by OpenAI Codex in response to instructions (prompts) written in natural language. We have chosen OpenAI Codex for the greater facilities it provides for tasks related to software development. OpenAI has a playground page from which we can test the OpenAI Codex’s API: https://beta.openai.com/playground

1. Case Description

For our example we want to generate most of the boilerplate code for the backend part of a simplified Expenses Management REST API. We are going to ask OpenAI Codex to generate:

• A MySQL script to create tables and foreign keys
• A MySQL script to populate those tables with sample data
• The Model Java classes to map each table, using Springboot
• The Repository, Service, and Rest Controller Java classes to each entity, with Springboot

2. Tool Configuration

The first thing we need to do before using OpenAI Codex is to complete the three parameters on the right side of the OpenAI Playground page:

• Model: Select the option “code-davinci-002", which is the most updated version
• Temperature: Enter the value zero (0) in order to have deterministic results
• Maximum Length: Number of tokens in the result. Two thousand could be fine

3. Prompts Design and Tips

The quality of the responses you will receive from OpenAI Codex depends on how the prompts (indications) are specified; therefore, we must pay special attention to how we design the prompts. Currently, there are only a few good practices that have been identified as a result of experimentation with this tool, such as:

• The prompts are entered as a comment block in the target language. For example, for SQL or Java we use /* and */ to delimit comment blocks, but it’s different for other languages
• Indicate the target programming language at the beginning of the prompt
• Structure the prompt: Split out tasks into subtasks and enter each in a different line. You can use nested levels numbering for a better understanding from OpenAI Codex
• After the prompt, it’s a good practice to write the expected starting code, which is useful to lead the code generation
  
  

4. Prompts and Results

For the MySQL script part, we have entered our prompts in natural language and in our own free format, as a comment block, followed by an expected starting code, “CREATE TABLE” in this case:

/*
MySQL
 
Generate the script to create the tables and foreign keys:
Table: User, Fields and Types: id auto, firstName text(50), lastName text(50), email text(50), password text(20)
Table: Project, Fields and Types: id auto, name text(50)
Table: ExpenseType, Fields and Types: id auto, name text(50)
Table: Expense, Fields and Types: id auto, description text(100), idExpenseType numeric, idProject numeric, idUser numeric, date date
 
Generate the script to fill with sample data those tables: 5 Users, 3 Projects, 4 ExpenseTypes
Generate 30 Expenses with random combination for users, projects and expenseTypes, with dates between November 15 and December 15, 2022.
*/

CREATE TABLE 

As soon you click the Submit button, OpenAI Codex reads your prompt and generates the output code. Here is an excerpt of the results:

Next, in the same session and below the last generated code, we enter our prompts for the Java part as a comment block using natural language and in our own free form. The expected starting sentence in this case is a kind of “separator” before the “package” sentence, which could help us to identify the limits between the sequence of Java files that will be generated by Codex. Note as well how OpenAI Codex understands the context when we indicate “those tables” or “the respective”:

/*
/*
Java 11 and Springboot
 
1. Create the Java classes to map those tables, using the JPA annotations. Omit gets and sets and uses Lombok instead. Annotate relationships in both directions. Annotate with lazy the OneToMany relationships only.
2. Generate the respective repository Java classes.
3. Generate the respective service Java classes.
4. Generate the respective rest controller Java classes.
*/

==========
package com.encora.expensesapp.

After clicking the Submit button, OpenAI Codex will generate the code below that prompt. Example of Entity Java classes generated:

Example of Repository and Service Java classes generated:

Example of Rest Controller Java class generated:

As you would note, all the Java classes are generated on the same response page with the separator that we indicated (==========). We can take that long code and either manually or by a script split them in their respective files and subfolders, but why not use OpenAI Codex itself to code that shell script? This is the prompt for that task:

/*
# Create a bash script to process a text file and do the next actions, keeping the nesting level:
# 0. Add IFS='' to keep leading spaces.
# 1. Memorize the current folder.
# 2. Read lines until find one starting with "==========".
# 3. Repeat the next sub-steps until the end of file.
# 3.1. Read all the lines until the next "==========" and keep them.
# 3.2. In the lines of text identified in the step 3.1, extract, using awk, the text between the text "package " and the first occurrence of ";" and keep it.
# 3.3. Create an array by splitting the text identified in the step 3.2 using the delimiter ".".
# 3.4. For each item in the array identified in the step 3.3, create a subfolder.
# 3.5. In the lines of text identified in the step 3.1, extract the text after "public class " or after "public interface" and before " ", and keep it.
# 3.6. Create a new file in the innermost folder created in the step 3.4, name it using the text identified in the step 3.5, and use as a content the text identified in the step 3.1.
# 3.7. Change to the folder memorized in the step 1.
# 3.8. Return to step 3. 
 
#!/bin/bash

And this is part of the generated script shell:

Benefits and Drawbacks

Coming Future

• As we observe the rapid evolution of AI technologies in recent years, it is clear that innovation will continue to drive the development of more advanced techniques and algorithms in the fields of machine learning and natural language processing. Thus, we could expect significant advancements in the AI tools used for software development in the next years.
• In the realm of large language models, OpenAI has already announced the release of its next-generation model, GPT-4, in 2023. This model will feature a larger training dataset and more parameters than its predecessor. Additionally, major technology companies such as Google, Amazon, and Facebook are expected to make their own versions of LLM tools available to the general public in the very near future.
• Large corporations are also increasing their investments in the field of AI, with Microsoft recently committing $10 billion to research and development at OpenAI, solidifying its position as a leader in the AI industry.

Conclusions

• Despite still being in its infancy, AI-driven development has already demonstrated its potential to enhance the software industry, by automating tasks such as initial code creation, code completion, and error prediction, among others. As AI tools continue to evolve, we can expect even greater benefits for the software development sector.
• The current state of AI-driven development highlights the growing importance of effective communication with AI systems. This ability to convey our intentions to AI is referred to as Prompt Engineering and it will function as a meta-programming language that we will have to learn.
• It is unlikely that AI will entirely replace human developers in the near future. Instead, it will likely become an essential tool for developers, assisting them in acquiring knowledge and optimizing procedures. In this scenario, AI will handle the more tedious aspects of code development, allowing programmers to focus on more complex and creative tasks.

References

• Gartner’s Top 10 Tech Trends: AI-Driven Development: https://stefanini.com/en/insights/news/gartner-top-10-tech-trends-ai-driven-development
• About Github Copilot: https://docs.github.com/en/copilot/overview-of-github-copilot/about-github-copilot
• About ChatGPT: https://openai.com/blog/chatgpt/
• About OpenAI Codex: https://openai.com/blog/openai-codex/
• The Future is Now: How AI-Driven Development will Make Developers’ Life Easier in 2023: https://itnext.io/the-future-is-now-how-ai-driven-development-will-make-developers-life-easier-in-2023-f2a26a85cbbe
• The Future of AI-Driven Software Development: https://wearecapicua.com/ai-driven-software-development
• Prompt Engineering: https://microsoft.github.io/prompt-engineering/

About Encora

Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.