Project

Step 1: find a need#

Start from a population and its question, not from the data. Your README answers four questions:

1. Who is the target population?
2. What question do they ask today without finding a simple answer?
3. Which solutions already exist, and what are their limits?
4. Why does an agent answer better than a dashboard or a search engine?

Example of the expected level (do not reuse it):

• Population: urban cyclists, who want to know which roads on their route have the most accidents.
• Data: the yearly databases of road traffic injury accidents, published on data.gouv.fr.
• Limit of existing solutions: maps show points, but they do not answer "is my route dangerous in the evening?".

Step 2: choose and reserve your dataset#

Each student has their own dataset: the first one to reserve it on the course site keeps it.

The recommended source is data.gouv.fr, the French open public data platform. It has data on health, transport, housing, education, environment, employment and more.

The dataset must:

• let you answer your population's questions;
• be reusable: check the licence shown on its page;
• be readable in Python (CSV, JSON, Excel…);
• fit in the repository, or come with a script that downloads it.

To reserve it:

1. Choose your dataset.
2. On the Reserve your dataset page, enter its name and the link to its page.
3. Once reserved, no other student can choose it. First come, first served.

Step 3: build the agent#

Your agent answers in natural language by calling tools that you write yourself.

• Tools: at least three Python functions that query your data (filter, aggregate, search). The LLM reads their docstring to choose which one to call: write it as carefully as the code.
• Prompts: the system prompt describes the population, the expected tone and the main rule. The agent never states a number that does not come from a tool.
• Outside the data: when the answer is not in your data, the agent says so instead of making it up.
• Understand the trace: for a given question, you can say which tools were called, in what order and why. For a Deep Agent, this includes its task plan and the files it writes.

Step 4: evaluate with an LLM-as-a-judge#

You measure the quality of your agent with an LLM that grades its answers to 5 questions.

1. Write 5 questions your population would ask, each with its reference answer.
2. Compute the reference answers without the agent, in a notebook included in the repository. Otherwise, the agent is graded against itself.
3. Run the agent on each question.
4. Send the judge the question, the reference answer and the agent's answer. It returns a grade and its reasoning in a structured format.
5. Print a table of the 5 grades and their average.

Among your 5 questions, you need at least:

• one question that takes several steps;
• one question whose answer is not in the data.

End with a short analysis of the failures. Possible causes: a badly described tool, an ambiguous prompt, missing data, or a judge that is too strict or too lenient.

Submission#

You hand in a Git repository that the teacher must be able to run on the first try.

project/
├── README.md          # population, need, data, how to run
├── requirements.txt   # project dependencies
├── .env.example       # GOOGLE_API_KEY=
├── .gitignore         # lists .env and .venv/
├── data/              # data or download script
├── src/
│   ├── tools.py
│   ├── prompts.py
│   └── agent.py
└── eval/
    ├── questions.json     # 5 questions and their reference answers
    ├── references.ipynb   # reference answers computed without the agent
    └── judge.py

The teacher clones the repository, then:

1. python -m venv .venv
2. source .venv/bin/activate
3. pip install -r requirements.txt
4. copies .env.example to .env and adds their key;
5. python eval/judge.py

If any of these steps fails, the reproducibility part is graded 0.

Rules#

• No AI to write code. Generating or completing code with an AI (ChatGPT, Gemini, Copilot, Cursor, Claude…) is forbidden. The official documentation and the course materials are allowed. Gemini is only used to run your agent.
• A personal Gemini key. Each student gets their own key, credited with €50 of tokens. Do not share it. Keep an eye on your usage: a single question can trigger several LLM calls.
• Never publish the key. It goes in .env, which is listed in .gitignore. Only .env.example, without the key, is in the repository. If your key leaks, tell the teacher.

Defence#

The defence takes place the week after the February written exam. You present your agent, then change it live without AI: add a tool, change a behaviour of the prompt. Then come questions about your code.

If you cannot explain your code, the grade for the tools, agent and evaluation parts is capped.