Creating Learner Progress Monitoring Using Python, Pandas and Streamlit
- Habib Abdurrasyid
- 04 Apr, 2022
- · 4 min read
On this page
Why I Built This
As Academic Program Manager for the AWS re/Start program, I was managing several cohorts at the same time. Each cohort had its own Canvas LMS gradebook, and each gradebook could have more than 170 columns of mixed data. Knowledge Checks, Labs, scores and completion flags, all in one file.
Reviewing one cohort by hand was fine at first. Then the program grew to 10 or more cohorts per batch, and going through them by hand took too long and it was easy to make mistakes.
I needed something better, so I built a tool for it.
The Problem in Practice
Canvas LMS is a good system. The gradebook exports it gives you, though, are made to be complete, not to be read. Here is what came in every week:
- One CSV file per cohort
- 170+ columns per file, mixing Knowledge Checks, Labs, and other things
- No clear separation between KC scores, Lab completion, and general course data
- Weekly progress targets that needed to be checked for each learner
With 10 cohorts that is 10 files and hundreds of rows. When the India programs ran at the same time it went past 40 cohorts, and at that point reading them by hand stopped being realistic.
What I Built
I made a web application using Python, Pandas, and Streamlit. The idea was to keep it simple: upload the files, get the results.
The flow is:
graph LR
A[Upload CSVs] --> B[Parse Data]
B --> C[Aggregate KC and Lab]
C --> D[Check Weekly Targets]
D --> E[Download Summary]
No setup to speak of. The people using it are program coordinators, not engineers, so the interface had to stay out of their way.
What the app does:
- Accept multiple CSV uploads at once (one per cohort)
- Automatically find and parse Knowledge Check columns
- Automatically find and parse Lab completion columns
- Calculate each learner’s progress against the weekly targets
- Output a clean, downloadable summary per cohort
Why I picked boring tools
I picked tools I already knew well, so the build would be quick and the maintenance boring.
- Python for the core logic
- Pandas for handling 170+ column DataFrames without it getting messy
- Streamlit, because a working web UI in Python takes hours to build, not days
- PythonAnywhere for hosting, so anyone can open it without installing anything
It is not a fancy stack. It did what I needed: it ran reliably, and coordinators who do not write code could use it without being taught.
One Extra Thing: The Leaderboard
After the main tool was working, I added a learner-facing progress leaderboard using Google Sheets.
The idea was simple. Learners could see their own KC and Lab scores next to everyone else in the cohort. Two things came out of that:
- It reduced the number of reminder messages coordinators had to send
- It gave learners a way to monitor themselves without waiting for someone to tell them how they were doing
Small addition. Both sides got something out of it, which is not always the case.
Where it ended up
The tool ended up being used across multiple batches:
- 3 AWS re/Start batches total
- 10+ cohorts in Indonesia
- 40+ cohorts in India
The time to process each batch dropped a lot compared with doing it by hand, and coordinators could spend that time on learner support instead of spreadsheets.
What I Would Change Now
Looking back at this, there are things I would do differently.
Automated gradebook retrieval. Right now someone has to download the files from Canvas and upload them by hand. Pulling straight from the Canvas API would remove that step.
Built-in charts. The output is a clean CSV, but visualizing trends over time still takes extra work. Basic charts inside the app would help.
Timeline tracking. The tool shows the current state only. Tracking progress across weeks would make a struggling learner show up earlier.
Out of the three, the manual download is the one that still annoys me. It is the single step in the whole flow that needs a person.
