Data Migrator - provide a universal translator for data by being portable, diverse, and efficient in migrating and converting data across discrete schemas

Overview

Data Migrator

🚧 Alpha version - Use with care. 🚧

Features

  • πŸ’Ž The Rosetta Stone for Data. Convert data schemas with ease
  • πŸš€ Portable Data. Move data from Anywhere. starting with CSV & Appwrite
  • πŸ§‘β€πŸ’» Multiplatform. macOS, Linux, and Windows
  • ⏱ Asynchronous. A parallel data pipeline
  • 🍭 Feature rich. Type conversion adapters, create & edit schemas, upload data etc
  • 🀲 Open source. Everything is open source and free forever!

Data Migrator can do much more (and we are just getting started)

If you want to say thank you, star us on GitHub πŸ™Œ πŸ’™

Quickstart

The purpose of Data Migrator is to provide a universal translator for data by being portable, diverse, and efficient in migrating and converting data across discrete schemas.

Right now, it is only developed against macOS (Windows and Linux will be officially developed against soon). It shouldn't be too hard to get it running on those targets though.

To get it running on macOS, install Flutter and make sure to add the flutter tool to your path.

Then make sure to run flutter config --enable-macos-desktop to enable the macOS target.

Clone this repo, navigate to the directory in terminal, and execute flutter run. Choose macOS for the target if it prompts you.

Architecture

Data Migrator provides powerful and flexible tooling in the form of 3 categories of code:

  1. Schema Representation
  2. A Conversion Pipeline
  3. Diverse Toolchain of Data Origins

Schema Representation

Data Migrator represents schemas in a flexible manner that supports flat schema data stores like tabular data (csv & sql) and schema-less deep data stores like JSON.

A Conversion Pipeline

The conversion pipeline is meant to be asyncronous as it handles packets of data that are streamed through the conversion type adapters. Data Migrator is built to have a diverse and extendable toolchain of adapters to convert any data type to another.

It is even designed to handle source fields that can hold several data types being converted to a destination with several completely different data types. It handles this by generating, creating, and editing the conversion type adapters where you can define exactly what source data type gets converted to which destination data type for each individual field.

Diverse Toolchain of Data Origins

Data Migrator uses adapters called DataOrigins that handle all the responsibilities of the source and destination of the data. The provide configuration, data stream, data sink, and schema validation interfaces.

For DataOrigins that involve schema-less data stores, they are responsible for generating a schema for the data. In the case of CSV tabular data, the data types are not stored in the built-in schema, so the CSV DataOrigin will scan the data to generate the data types that fit in each field.

Right now, Data Migrator is focusing on developing the CSV and Appwrite DataOrigins. After the main features have been built out and tested, the next DataOrigin will be Firebase.

License

Copyright 2022 Tanner Meade

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Comments
  • Can't select CSV

    Can't select CSV

    Hi,

    I tried to select a blank CSV file. We basically want to write from Appwrite to the CSV file e.g. taking a backup.

    I failed to select CSV file on Web and MacOS as well

    For MacOS

    flutter: [backup_appwrite.csv]
    flutter: /Volumes/Macintosh HD/Users/x/Desktop/backup_appwrite.csv
    [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: Bad state: No element
    #0      Stream.firstWhere.<anonymous closure> (dart:async/stream.dart:1399:9)
    #1      _rootRun (dart:async/zone.dart:1418:47)
    #2      _CustomZone.run (dart:async/zone.dart:1328:19)
    #3      _CustomZone.runGuarded (dart:async/zone.dart:1236:7)
    #4      _BufferingStreamSubscription._sendDone.sendDone (dart:async/stream_impl.dart:394:13)
    #5      _rootRun (dart:async/zone.dart:1418:47)
    #6      _CustomZone.run (dart:async/zone.dart:1328:19)
    #7      _FutureListener.handleWhenComplete (dart:async/future_impl.dart:202:18)
    #8      Future._propagateToListeners.handleWhenCompleteCallback (dart:async/future_impl.dart:736:39)
    #9      Future._propagateToListeners (dart:async/future_impl.dart:792:11)
    #10     Future._completeWithValue (dart:async/future_impl.dart:601:5)
    #11     Future._asyncCompleteWithValue.<anonymous closure> (dart:async/future_impl.dart:639:7)
    #12     _rootRun (dart:async/zone.dart:1426:13)
    #13     _CustomZone.run (dart:async/zone.dart:1328:19)
    #14     _CustomZone.runGuarded (dart:async/zone.dart:1236:7)
    #15     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1276:23)
    #16     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
    #17     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)
    
    

    For Web:

    [backup_appwrite.csv]
    Error:       On web `path` is always `null`,
          You should access `bytes` property instead,
          Read more about it [here](https://github.com/miguelpruivo/flutter_file_picker/wiki/FAQ)
    
        at Object.throw_ [as throw] (http://localhost:63669/dart_sdk.js:5067:11)
        at platform_file.PlatformFile.new.get path [as path] (http://localhost:63669/packages/file_picker/src/platform_file.dart.lib.js:78:19)
        at _pickFiles (http://localhost:63669/packages/data_migrator/infastructure/data_origins/csv_origin/csv_origin.dart.lib.js:308:68)
        at _pickFiles.next (<anonymous>)
        at http://localhost:63669/dart_sdk.js:40571:33
        at _RootZone.runUnary (http://localhost:63669/dart_sdk.js:40441:59)
        at _FutureListener.thenAwait.handleValue (http://localhost:63669/dart_sdk.js:35363:29)
        at handleValueCallback (http://localhost:63669/dart_sdk.js:35931:49)
        at Function._propagateToListeners (http://localhost:63669/dart_sdk.js:35969:17)
        at _Future.new.[_completeWithValue] (http://localhost:63669/dart_sdk.js:35817:23)
        at async._AsyncCallbackEntry.new.callback (http://localhost:63669/dart_sdk.js:35838:35)
        at Object._microtaskLoop (http://localhost:63669/dart_sdk.js:40708:13)
        at _startMicrotaskLoop (http://localhost:63669/dart_sdk.js:40714:13)
        at http://localhost:63669/dart_sdk.js:36191:9
    

    Any help would be much appreciated.

    opened by rome2o 1
  • bug: UI no updating after select a collection

    bug: UI no updating after select a collection

    When you select a collection from appwrite you don't have any feedback after press the + icon, so you don't know if It was add or not.

    If you press the button multiples times and then closes the selection window, you will see multiples instances of the same collection, I think there is no reason why you should be allowing this, as you always will need one collection matching with the input data

    image

    bug 
    opened by gepd 3
  • Feature: Custom SchemaMap for DataOrigin

    Feature: Custom SchemaMap for DataOrigin

    DataOrigins have a use case where SchemaMaps have a special definition such as how Appwrite stores accounts. There needs to be the ability to define and connect the source data to a destination special SchemaMap that will be inserted not into the Apwrite database, but the accounts.

    enhancement 
    opened by tannermeade 0
  • Feature: Add support for CustomFields

    Feature: Add support for CustomFields

    DataOrigins have a use case where the schema fields have special representations that need to be identified by the DataOrigin to upload and save the data correctly.

    This will be defined by extending SchemaFields (and maybe overriding validation methods). DataOrigin uses the SchemaField runtime type to handle the schema creation and data upload. Use cases:

    • Appwrite’s $id can be defined and uploaded
    enhancement 
    opened by tannermeade 0
  • Feature: Add Firebase Support

    Feature: Add Firebase Support

    Add support for migrating data from Firebase.

    This includes accounts, which the passwords are scrypt hashed and encrypted/peppered with AES. The process for migrating passwords will include decrypting the AES hashes into regular scrypt+salt hashes.

    enhancement 
    opened by tannermeade 1
Owner
Tanner Meade
Tanner Meade
Multi Translator build with Flutter, It developed with DDD (Domain Driven Design) principles.

Multi Translator App An app utilizes to translate any text to multiple languages. Features Localization Multiple Translation Single Translation Deep L

Selim Üstel 4 Jul 18, 2022
State Persistence - Persist state across app launches. By default this library store state as a local JSON file called `data.json` in the applications data directory. Maintainer: @slightfoot

State Persistence Persist state across app launches. By default this library store state as a local JSON file called data.json in the applications dat

Flutter Community 71 Dec 23, 2021
AI Library to create efficient Artificial Neural Networks. Computation uses SIMD (Single Instruction Multiple Data) to improve performance.

eneural_net eNeural.net / Dart is an AI Library for efficient Artificial Neural Networks. The library is portable (native, JS/Web, Flutter) and the co

null 19 Mar 11, 2022
It's a universal app template to have a great animated splash screen and liquid slider. Just change the animation if you want (rive) and change the images or colours according to your app.

liquid A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started if this i

Zikyan Rasheed 27 Jul 4, 2022
Hassan uni links - A Flutter plugin project to help with App/Deep Links (Android) and Universal Links and Custom URL schemes

Uni Links A Flutter plugin project to help with App/Deep Links (Android) and Uni

Hassan Al-Sabti 0 Feb 12, 2022
Flutter remote control - The main use of LongPressDraggable and DragTarget to achieve the universal remote control interaction effect.

Flutter remote control - The main use of LongPressDraggable and DragTarget to achieve the universal remote control interaction effect.

ε”―ιΉΏ 160 Jul 22, 2022
An universal server management tool for Minecraft.

MinecraftCube Desktop A server management tool for Minecraft, that help players start any kind of minecraft server easier. Mainly support vanilla and

null 10 Jul 13, 2022
An efficient and easy CA-Client interaction flutter project

Cloud Accounting An efficient and easy CA-Client interaction flutter project This is an application for a CA to interact with its clients. Clients can

null 2 Dec 2, 2021
UIWidget is a Unity Package which helps developers to create, debug and deploy efficient, cross-platform Apps.

⚠️ The main repository of UIWidgets has been moved to https://github.com/Unity-Technologies/com.unity.uiwidgets. Please visit the new site if you have

Unity Technologies 1.9k Sep 19, 2022
A simple and efficient FFT implementation for Dart

fftea A simple and efficient Fast Fourier Transform (FFT) library. FFT converts a time domain signal to the frequency domain, and back again. This is

Liam Appelbe 21 Aug 31, 2022
Dartness is a progressive dart framework for building efficient and scalable server-side applications

Dartness is a framework for building efficient, scalable dart server-side applications. It provides an easy and quick way to develop modern standalone server.

Ricardo Romero 34 Sep 2, 2022
A Funtioning basic Clock UI APP with extra functionalities such as displaying thecurrent time location being used and checking time for other timezones simultaneosly.

clock_UI A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started if this

Anjola Favour Ayomikun 0 Dec 28, 2021
A flutter library for loading images from network, resizing as per container size and caching while being memory sensitive.

Optimized Cached Image A flutter library for loading images from network, resizing and caching them for memory sensitivity. This resizes and stores th

Anvith Bhat 70 Jun 30, 2022
A project made for the ECV Digital 2022. 🌐 The goal being to make a Flutter application to connect, register and display information of a given API.πŸ‘¨πŸΌπŸ”§

not_netflix A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started if t

QUENEL Laurent 2 May 7, 2022
A Dictionary application being developed for the Gamo Language.

gamo_dict A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started if thi

Wuletaw Wonte 7 Jul 1, 2022
Flutter project being built in the context of Smart India Hackathon { SIH }.

e_chardham_yatra A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started

Tanay-Dwivedi 4 Mar 23, 2022
This project uses transactions in Firebase(FirebaseAuth and FireStore) to send and receive virtual money across accounts

FinTech (WIP) This project uses transactions in Firebase(FirebaseAuth and FireStore) to send and receive virtual money across accounts. On account cre

Godson 3 Jul 19, 2022
Flutter widget to change `themeMode` during runtime and persist it across restarts.

theme_mode_handler Flutter widget to change themeMode during runtime and persist it across restarts. Motivation Flutter 1.9 introduced a new way to co

Arthur Denner 28 Nov 30, 2021