flutter_sequencer
This Flutter plugin lets you set up sampler instruments and create multi-track sequences of notes that play on those instruments. You can specify a loop range for a sequence and schedule volume automations.
It uses the sfizz SFZ player on both Android and iOS. The SFZ format can be used to create high-quality sample-based instruments and do subtractive synthesis. These instruments can have modulation parameters that can be controlled by your Flutter UI. The plugin also supports playing SF2 (SoundFont) files on both platforms, and on iOS, you can load any AudioUnit instrument.
The example app is a drum machine, but there are many things you could build. The plugin is not limited to "step" based sequencing. You could make a whole sample-based DAW. You could use it to make a cross-platform app to host your SFZ instrument. You could also use it for game sound effects, or even to generate a dynamic game soundtrack.
How to use
Create the sequence
final sequence = Sequence(tempo: 120.0, endBeat: 8.0);
You need to set the tempo and the end beat when you create the sequence.
Create instruments
final instruments = [
Sf2Instrument(path: "assets/sf2/TR-808.sf2", isAsset: true),
SfzInstrument(
path: "assets/sfz/GMPiano.sfz",
isAsset: true,
tuningPath: "assets/sfz/meanquar.scl",
),
RuntimeSfzInstrument(
id: "Sampled Synth",
sampleRoot: "assets/wav",
isAsset: true,
sfz: Sfz(
groups: [
SfzGroup(
regions: [
SfzRegion(sample: "D3.wav", noteNumber: 62),
SfzRegion(sample: "F3.wav", noteNumber: 65),
SfzRegion(sample: "Gsharp3.wav", noteNumber: 68),
],
),
],
),
),
RuntimeSfzInstrument(
id: "Generated Synth",
// This SFZ doesn't use any sample files, so just put "/" as a placeholder.
sampleRoot: "/",
isAsset: false,
// Based on the Unison Oscillator example here:
// https://sfz.tools/sfizz/quick_reference#unison-oscillator
sfz: Sfz(
groups: [
SfzGroup(
regions: [
SfzRegion(
sample: "*saw",
otherOpcodes: {
"oscillator_multi": "5",
"oscillator_detune": "50",
}
)
]
)
]
)
),
];
An instrument can be used to create one or more tracks. There are four instruments:
- SfzInstrument, to load a
.sfzfile and the samples it refers to.- On iOS and Android, it will be played by sfizz
- As far as I know, sfizz is the most complete and most frequently updated SFZ player library with a license that permits commercial use.
- Sfizz supports .wav and .flac sample files, among others. I recommend using .flac when possible, since it supports lossless compression. It's easy to convert audio files to FLAC format with ffmpeg.
- You can also create an SFZ that doesn't use any sample files by setting
sampleto a predefined waveform, such as*sine,*saw,*square,*triangle, or*noise. - Check which SFZ opcodes are supported by sfizz here: https://sfz.tools/sfizz/development/status/opcodes/
- Learn more about the SFZ format here: https://sfzformat.com
- RuntimeSfzInstrument, to build an SFZ at runtime.
- This will also be played by sfizz.
- Instead of a file, you pass an
Sfzobject to the constructor. See the example. - You might use this so that your app can build a synth using selected oscillators, or a sampler using user-provided samples.
- Sf2Instrument, to load a
.sf2SoundFont file.- On iOS, it will be played by the built-in Apple MIDI synth AudioUnit
- On Android, it will be played by tinysoundfont
- I recommend using SFZ format, since sfizz can stream samples from disk. This way you can load bigger sound fonts without running out of RAM.
- You can easily convert SF2 to SFZ with Polyphone. Just open the SF2, click the menu icon at the top right, and click "Export Soundfonts." Change the format to SFZ. The other options shouldn't matter.
- AudioUnitInstrument, to load an AudioUnit
- This will only work on iOS
- You might use this if you are making a DAW type of app.
For an SF2 or SFZ instrument, pass isAsset: true to load a path in the Flutter assets directory. You should use assets for "factory preset" sounds. To load user-provided or downloaded sounds from the filesystem, pass isAsset: false.
Important notes about isAsset: true
Note that on Android, SFZ files and samples that are loaded with isAsset: true will be extracted from the bundle into the application files directory (context.filesDir), since sfizz cannot read directly from Android assets. This means they will exist in two places on the device - in the APK in compressed (zipped) form and in the files directory in uncompressed form. So I only recommend using isAsset: true if your samples are small. If you want to include high-quality soundfonts in your app, your app should download them at runtime.
Note that on either platform, Flutter asset paths get URL-encoded. For example, if you put a file called "Piano G#5.wav" in your assets folder, it will end up being called "Piano%20G%235.wav" on the device. So if you are bundling an SFZ file as an asset, make sure that you either remove any special characters and spaces from the sample file names, or update the SFZ file to refer to the URL-encoded sample paths. I recommend just getting rid of any spaces and special characters.
Optional: keep engine running
GlobalState().setKeepEngineRunning(true);
This will keep the audio engine running even when all sequences are paused. Set this to true if you need to trigger sounds when the sequence is paused. Don't do it otherwise, since it will increase energy usage.
Create tracks
sequence.createTracks(instruments).then((tracks) {
setState(() {
this.tracks = tracks;
...
});
});
createTracks returns Future<List>. You probably want to store the value it completes with in your widget's state.
Schedule events on the tracks
track.addNote(noteNumber: 60, velocity: 0.7, startBeat: 0.0, durationBeats: 2.0);
This will add middle C (MIDI note number 60) to the sequence, starting from beat 0, and stopping after 2 beats.
track.addVolumeChange(volume: 0.75, beat: 2.0);
This will schedule a volume change. It can be used to do volume automation. Note that track volume is on a linear scale, not a logarithmic scale. You may want to use a logarithmic scale and convert to linear.
track.addMidiCC(ccNumber: 127, ccValue: 127, beat: 2.0);
This will schedule a MIDI CC event. These can be used to change parameters in a sound font. For example, in an SFZ, you can use the "cutoff_cc1" and "cutoff" opcodes to define a filter where the cutoff can be changed with MIDI CC events. There are many other parameters that can be controlled by CC as well, such as amp envelope, filter envelope, sample offset, and EQ parameters. For more information, see how to do modulations in SFZ and how to use filter cutoff in SFZ.
track.addMidiPitchBend(value: 1.0, beat: 2.0);
This will schedule a MIDI pitch bend event. The value can be from -1.0 to 1.0. Note that the value is NOT the number of semitones. The sound font defines how many semitones the bend range is. For example, in an SFZ, you can use the "bend_down" and "bend_up" opcodes to define how many cents the pitch will be changed when the bend value is set to -1.0 and 1.0, respectively.
Control playback
sequence.play();
sequence.pause();
sequence.stop();
Start, pause, or stop the sequence.
sequence.setBeat(double beat);
Set the playback position in the sequence, in beats.
sequence.setEndBeat(double beat);
Set the length of the sequence in beats.
sequence.setTempo(120.0);
Set the tempo in beats per minute.
sequence.setLoop(double loopStartBeat, double loopEndBeat);
Enable looping.
sequence.unsetLoop();
Disable looping.
Get real-time information about the sequence
You can use SingleTickerProviderStateMixin to make these calls on every frame.
sequence.getPosition(); // double
Gets the playback position of the sequence, in beats.
sequence.getIsPlaying(); // bool
Gets whether or not the sequence is playing. It may have stopped if it reached the end.
track.getVolume();
Gets the volume of a track. A VolumeEvent may have changed it during playback.
Real-time playback
track.startNoteNow(noteNumber: 60, velocity: 0.75);
Send a MIDI Note On message to the track immediately.
track.stopNoteNow(noteNumber: 60);
Send a MIDI Note Off message to the track immediately.
track.changeVolumeNow(volume: 0.5);
Change the track's volume immediately. Note that this is linear gain, not logarithmic.
How it works
The Android and iOS backends start their respective audio engines. The iOS one adds an AudioUnit for each track to an AVAudioEngine and connects it to a Mixer AudioUnit. The Android one has to do all of the rendering manually. Both of them share a "BaseScheduler", which can be found under the ios directory.
The backend has no concept of the sequence, the position in a sequence, the loop state, or even time as measured in seconds or beats. It just maintains a map of tracks, and it triggers events on those tracks when the appropriate number of frames have been rendered by the audio engine. The BaseScheduler has a Buffer for each track that holds its scheduled events. The Buffer is supposed to be thread-safe for one reader and one writer and real-time safe (i.e. it will not allocate memory, so it can be used on the audio render thread.)
The Sequence lives on the Dart front end. A Sequence has Tracks. Each Track is backed by a Buffer on the backend. When you add a note or a volume change to the track, it schedules an event on the Buffer at the appropriate frame, based on the tempo and sample rate.
The buffer might not be big enough to hold all the events. Also, when looping is enabled, events will occur indefinitely, so the buffer will never be big enough. To deal with this, the frontend will periodically "top off" each track's buffer.
Development instructions
Note that the Android build uses several third party libraries, including sfizz. The Gradle build will download them from GitHub into the android/third_party directory.
The iOS also uses the sfizz library. It will be downloaded by the prepare.sh script which CocoaPods will run.
To build the C++ tests on Mac OS, go into the cpp_test directory, and run
cmake .
make
Then, to run the tests,
./build/sequencer_test
I haven't tried it on Windows or Linux, but it should work without too many changes.
To Do
PRs are welcome! If you use this plugin in your project, please consider contributing by fixing a bug or by tackling one of these to-do items.
Difficulty: Easy
- Change position_frame_t to 64-bit integer to avoid overflow errors
- Make constants configurable (TOP_OFF_PERIOD_MS and LEAD_FRAMES)
- Support federated plugins
Difficulty: Medium
- Start using Dart null safety
- Support tempo automation
- MIDI Out instrument
- Create an instrument that doesn't make any sounds on its own and just sends MIDI events to a designated MIDI output port.
- Refactoring
- GlobalState, Sequence, and Track should have a clear separation of responsibilities
- Engine, Sequencer, Mixer, Scheduler do too
- "Track index" and "Track ID" are used interchangeably, "Track ID" should be used everywhere
- Make the names and organization of the different files (Plugin/Engine/Scheduler) consistent
- Support MacOS
- Most of the iOS code should be able to be reused
- Add more C++ tests
- For BaseScheduler, specifically.
- Add Dart tests
- Test loop edge cases
- Ensure there are no memory leaks or concurrency issues
Difficulty: Hard
- Support Windows
- Record audio output
- Support React Native?
- Could use dart2js
Difficulty: Very Hard
- (important) Audio graph
- Create a graph of tracks and effects where any output can be connected to any input.
- At that point, the most of the audio engine can be cross-platform. On iOS, it could be wrapped in one AudioUnit, and external AudioUnit instruments and effects could be connected to it via input and output buses.
- LabSound seems like the library to use
- It's based on WebAudio, so there could also be a web backend
- Can be used to add some key effects like reverb, delay, and compression
- Supports audio input, writing output to file, and other DAW stuff
- Support Web
![There is no sound in release mode but it works perfectly in debug mode [tested on iOS simulator and 2 iPhones]](https://avatars.githubusercontent.com/u/35217388?v=4)
![[iOS] Using the just_audio package to play a specific mp3 is breaking the flutter_sequencer but it doensn't throw any errors.](https://avatars.githubusercontent.com/u/9638827?v=4)
2.5k Dec 30, 2022
1.5k Jan 8, 2023
32 Dec 14, 2022
67 Sep 24, 2022
96 Oct 25, 2022
632 Jan 4, 2023
12 Jan 19, 2022
32 Jan 5, 2023
215 Dec 12, 2022
5 Dec 31, 2021
11 Oct 24, 2022
53 Dec 28, 2022
257 Dec 30, 2022
259 Jan 1, 2023
5 Jul 26, 2022
11 Dec 27, 2022
4 Dec 23, 2022
31 Nov 8, 2022