AIvories: A Piano Learning Platform with Gamification and Generative AI

School of Science and Technology 科技學院
Computing Programmes 電腦學系

AIvories: A Piano Learning Platform with Gamification and Generative AI

Cheung Ka Ho, Ng Ka Yu, Lam Chak Sum, Lau Kai Hing

Programme	Bachelor of Computing with Honours in Internet Technology Bachelor of Science with Honours in Computer Science
Supervisor	Dr. Keith Lee
Areas	Intelligent Applications
Year of Completion	2025

Objectives

Project Aim

This project aims to develop a web application for piano learner to learn music theory and piano techniques with practices in a more attractive way to increase student motivation to learn piano. The web application need user connect to a MIDI piano for real time interactions. The web application will exploit AI to transform music style for students to learn piano from diverse repertoire. Also, web applications will design more like a game. We want to create a solution that can make the music learning more engaging and personalize.

Project objective

To reach our aim, we decided few of project objectives for a clearer direction.

To design each stage of piano learning using gamification. Using a game level to separate every piano learning topic that helps students learn complex concepts in an easy and interesting way. Helping students understand the basic knowledge of music theory.
To design an interactive feedback mechanism that provides real-time corrections. Users connect their MIDI piano and engage with the website, which includes piano keys, sheet music, and piano performance tracking. This setup provides instant corrections, making the learning process more interactive.
To develop each game level by using reliable piano teaching materials. Providing suitable difficulties to each user base on their learning stage.
To develop an AI model that can transform one music style to another music style. Users can make use of the music for practicing. Providing personalized learning.
To develop adaptive learning goals (daily task, achievement, etc.) that adjust the difficulty based on the learner’s progress and performance. To prepare a questionnaire that elaborates user's feedback and experience. Helping improve and guide future development.

Videos

Demonstration Video

Presentation Video

Methodologies and Technologies used

Front-End Application

Built using React to provide a smooth and responsive user interface.
Allows users to select music, initiate practice sessions, and view progress in an engaging format.
Focused on maximizing usability and enjoyment, with interactive UI features.

Back-End Server

Developed with Python, responsible for:

1. Music generation logic based on user learning progress.
2. Communicating with the database to store practice records and deliver generated music efficiently.

Figure 1: The GPIO Layout of the Raspberry Pi 4

Database Layer

Utilizes MongoDB, a flexible NoSQL database to:
1. Store structured user data.
2. Manage music data, including generated clips and learning history.

Design Philosophy & Value Proposition

Prioritizes user engagement and ease of use, especially for learners.
Aims to stand out from other piano tuition apps through:

1. AI-enhanced music generation.
2. A personalized practice experience.

Backed by user behavior insights (Jackson, 2024), suggesting high interest in AI-driven features, which supports expectations of higher user satisfaction.

Figure 2: Software (UI function design)

AI Generation (ai_gen) Features

The AI Generation Interface consists of two key sections:

Genre Selection:
Users choose from a variety of musical styles (e.g., jazz, classical, pop), setting the tone for music transformation.
Music Generation:
After selecting a style, users input the music they want to convert.

“Start to Generate”: Initiates the conversion process.
“Re_gen”: Allows quick regeneration for alternative outputs.
Outputs include sound files and musical scores available for download.

Educational & Engagement Benefits

Creative Expression:
Empowers users to experiment with style conversion, boosting creativity and motivation.
Genre Exposure:
Encourages exploration across different musical structures and traditions, enhancing musical literacy.
Instant Feedback:
Users can review their output immediately, facilitating self-assessment and continuous improvement.

Sustained Engagement:
With endless stylistic variations, users face fresh challenges that keep learning dynamic and rewarding.

Figure 3: AI Generation 

Daily Task

Figure 4 shows that when the user presses the “Task” option, an interface containing the progress of all tasks will be displayed. The difference between this interface and the home page is that it not only displays the tasks that have been started but also lists all unstated tasks. This design allows users to clearly understand their learning progress and the tasks they can perform in the future.

Clear Learning Path: By listing all tasks, users can clearly see their learning progress and future learning plans, which helps them set specific learning goals.

Enhanced Self-Management Skills: Users can choose suitable tasks based on their time availability and learning needs, thereby improving their self-management skills.

Motivational Mechanism: When users see both completed and pending tasks, it fosters a sense of achievement, motivating them to continue their efforts and enhance their learning drive.

Figure 4: Task (full Information) 

Implementation

Home page:

Example:

Ch1-1 Learn how to identify notes on the keyboard, such as middle C

Ch1-2 Using middle C as a base point, learn the notes that make up the C chord (C, E, G).

Ch1-3 Consolidate middle c knowledge

Figure 5: Main Page

Have video before each lesson, user always interesting visual more than text. Also provide information to summary all what video tech.

Figure 6: Piano Lesson

Figure 7: Hand Position

Music Import / Select

User can upload self-prepared midi file to perform practice or select midi file uploaded before. User can practice with her favorite music, make piano practice more enjoyable.

Figure 8: Music List

Playing Note Interface:

MIDI & Visual Keyboard Integration
The system communicates with a MIDI keyboard to reflect real-time input on a visual piano keyboard, alongside interactive note tracking and feedback.

Flexible Progress Control
Users can pause and resume music playback freely, enabling learning during short or irregular practice sessions.
Dynamic Progress Indicators
Sheet music, note bars, and progress bars visually align with the music playback, guiding users through their performance.
Virtual Keyboard Support
For users without a MIDI keyboard, the system provides a mapped virtual keyboard to maintain functionality and accessibility.
Metronome Functionality
BPM (beats per minute) is auto detected, and a metronome sound is played to help users maintain rhythm.
Visual Feedback on Sheet Music
As users play, notes on the sheet music change color—green for correct, red for incorrect—providing instant performance feedback.
End-of-Session Performance Output
A summary or release of the user’s performance is generated once the session ends.

Figure 9: Music Note

When finish playing, result page has been shown. Overview of performance measure can be found here. If click again button, will play music again. If click Back button, will back to main page. If click Detail button, will show detail of this play.

Figure 10: Performance Score

Various statistics of user learning performance and preference:

Contain statistics to indicate preference of user. Include recent 7 day of number of lessons completed, number of music played, number of task finish and best note played accuracy per music.

Figure 11: Preference

System will record each record user music played. User can easily find her practice record with systematic design.

Wav to MIDI conversion

this feature currently focuses on converting WAV audio files into MIDI format to support other key functionalities of the platform.

User Upload & Conversion: Users upload WAV files, which are automatically converted to MIDI.
Interactive Outcomes:

View generated sheet music
Download the MIDI file
Upload to Self Study mode for personalized practice
Play back the converted piece

Figure 12: transform music input form

Customized Learning Experience:
This function empowers users to learn piano using music that resonates with them—familiar or personally meaningful—boosting motivation and aligning practice with individual progress and goals.

Figure 13: Generated music sheet (Twinkle Twinkle little star.wav)

Keyboard connection

To enhance the user experience, we’ve enabled keyboard controls for the virtual piano and display key notations to guide users on which keys correspond to which piano notes.

Figure 14: Keyboard connection UI

Conclusion

The project successfully met its objectives by creating a web platform that integrates gamification with real-time MIDI interaction to enhance beginner music education. This section highlights key achievements, limitations, and paths forward.

Key Achievements

Boosted Motivation & Engagement: Gamified MIDI interaction proved highly effective in increasing learner enthusiasm.
AI-Driven Personalization: Customized musical content improved user experience by addressing issues like limited variety and parental overreach.
Skill Development: Users showed clear progress in reading sheet music, rhythm, and fingering techniques.

Limitations

Small User Group: Initial testing lacked large-scale validation.
No Long-Term Data: Sustainability of improvements remains unproven over time.
Limited to Piano: Scalability to other instruments and topics is not yet established.
Uncontrolled Environments: Real-world variables like distractions and parental influence still pose challenges.

Future Development

Broader Sampling: Gradual expansion to more diverse user bases for better data.
Longitudinal Studies: Monitor progress and engagement over time.
System Scalability: Modular architecture to support additional instruments and content.
Adaptive Learning Tools: Include smart feedback systems and parental support features to ensure effectiveness in non-co