[FL-2263] Flasher service & RAM exec (#1006)

* WIP on stripping fw
* Compact FW build - use RAM_EXEC=1 COMPACT=1 DEBUG=0
* Fixed uninitialized storage struct; small fixes to compact fw
* Flasher srv w/mocked flash ops
* Fixed typos & accomodated FFF changes
* Alternative fw startup branch
* Working load & jmp to RAM fw
* +manifest processing for stage loader; + crc verification for stage payload
* Fixed questionable code & potential leaks
* Lowered screen update rate; added radio stack update stubs; working dfu write
* Console EP with manifest & stage validation
* Added microtar lib; minor ui fixes for updater
* Removed microtar
* Removed mtar #2
* Added a better version of microtar
* TAR archive api; LFS backup & restore core
* Recursive backup/restore
* LFS worker thread
* Added system apps to loader - not visible in UI; full update process with restarts
* Typo fix
* Dropped BL & f6; tooling for updater WIP
* Minor py fixes
* Minor fixes to make it build after merge
* Ported flash workaround from BL + fixed visuals
* Minor cleanup
* Chmod + loader app search fix
* Python linter fix
* Removed usb stuff & float read support for staged loader == -10% of binary size
* Added backup/restore & update pb requests
* Added stub impl to RPC for backup/restore/update commands
* Reworked TAR to use borrowed Storage api; slightly reduced build size by removing `static string`; hidden update-related RPC behind defines
* Moved backup&restore to storage
* Fixed new message types
* Backup/restore/update RPC impl
* Moved furi_hal_crc to LL; minor fixes
* CRC HAL rework to LL
* Purging STM HAL
* Brought back minimal DFU boot mode (no gui); additional crc state checks
* Added splash screen, BROKEN usb function
* Clock init rework WIP
* Stripped graphics from DFU mode
* Temp fix for unused static fun
* WIP update picker - broken!
* Fixed UI
* Bumping version
* Fixed RTC setup
* Backup to update folder instead of ext root
* Removed unused scenes & more usb remnants from staged loader
* CI updates
* Fixed update bundle name
* Temporary restored USB handler
* Attempt to prevent .text corruption
* Comments on how I spent this Saturday
* Added update file icon
* Documentation updates
* Moved common code to lib folder
* Storage: more unit tests
* Storage: blocking dir open, differentiate file and dir when freed.
* Major refactoring; added input processing to updater to allow retrying on failures (not very useful prob). Added API for extraction of thread return value
* Removed re-init check for manifest
* Changed low-level path manipulation to toolbox/path.h; makefile cleanup; tiny fix in lint.py
* Increased update worker stack size
* Text fixes in backup CLI
* Displaying number of update stages to run; removed timeout in handling errors
* Bumping version
* Added thread cleanup for spawner thread
* Updated build targets to exclude firmware bundle from 'ALL'
* Fixed makefile for update_package; skipping VCP init for update mode (ugly)
* Switched github build from ALL to update_package
* Added +x for dist_update.sh
* Cli: add total heap size to "free" command
* Moved (RAM) suffix to build version instead of git commit no.
* DFU comment
* Some fixes suggested by clang-tidy
* Fixed recursive PREFIX macro
* Makefile: gather all new rules in updater namespace. FuriHal: rename bootloader to boot, isr safe delays
* Github: correct build target name in firmware build
* FuriHal: move target switch to boot
* Makefile: fix firmware flash
* Furi, FuriHal: move kernel start to furi, early init
* Drop bootloader related stuff
* Drop cube. Drop bootloader linker script.
* Renamed update_hl, moved constants to #defines
* Moved update-related boot mode to separate bitfield
* Reworked updater cli to single entry point; fixed crash on tar cleanup
* Added Python replacement for dist shell scripts
* Linter fixes for dist.py +x
* Fixes for environment suffix
* Dropped bash scripts
* Added dirty build flag to version structure & interfaces
* Version string escapes
* Fixed flag logic in dist.py; added support for App instances being imported and not terminating the whole program
* Fixed fw address in ReadMe.md
* Rpc: fix crash on double screen start
* Return back original boot behavior and fix jump to system bootloader
* Cleanup code, add error sequence for RTC
* Update firmware readme
* FuriHal: drop boot, restructure RTC registers usage and add header register check
* Furi goes first
* Toolchain: add ccache support
* Renamed update bundle dir

Co-authored-by: DrZlo13 <who.just.the.doctor@gmail.com>
Co-authored-by: あく <alleteam@gmail.com>

applications/storage/storage.c

@@ -40,7 +40,9 @@ Storage* storage_app_alloc() {
         storage_data_init(&app->storage[i]);
     }
 
+#ifndef FURI_RAM_EXEC
     storage_int_init(&app->storage[ST_INT]);
+#endif
     storage_ext_init(&app->storage[ST_EXT]);
 
     // sd icon gui

applications/storage/storage.h

@@ -263,6 +263,22 @@ FS_Error storage_sd_info(Storage* api, SDInfo* info);
  */
 FS_Error storage_sd_status(Storage* api);
 
+/******************* Internal LFS Functions *******************/
+
+/** Backs up internal storage to a tar archive
+ * @param api pointer to the api
+ * @param dstmane destination archive path
+ * @return FS_Error operation result
+ */
+FS_Error storage_int_backup(Storage* api, const char* dstname);
+
+/** Restores internal storage from a tar archive
+ * @param api pointer to the api
+ * @param dstmane archive path
+ * @return FS_Error operation result
+ */
+FS_Error storage_int_restore(Storage* api, const char* dstname);
+
 /***************** Simplified Functions ******************/
 
 /**
@@ -309,4 +325,4 @@ void storage_get_next_filename(
 
 #ifdef __cplusplus
 }
-#endif
+#endif

applications/storage/storage_cli.c

@@ -521,7 +521,7 @@ void storage_cli(Cli* cli, string_t args, void* context) {
     string_clear(cmd);
 }
 
-void storage_cli_factory_reset(Cli* cli, string_t args, void* context) {
+static void storage_cli_factory_reset(Cli* cli, string_t args, void* context) {
     printf("All data will be lost. Are you sure (y/n)?\r\n");
     char c = cli_getc(cli);
     if(c == 'y' || c == 'Y') {
@@ -540,5 +540,7 @@ void storage_on_system_start() {
     cli_add_command(
         cli, "factory_reset", CliCommandFlagParallelSafe, storage_cli_factory_reset, NULL);
     furi_record_close("cli");
+#else
+    UNUSED(storage_cli_factory_reset);
 #endif
 }

applications/storage/storage_internal_api.c

@@ -0,0 +1,22 @@
+#include <furi/record.h>
+#include <m-string.h>
+#include "storage.h"
+#include <toolbox/tar/tar_archive.h>
+
+#define INT_PATH "/int"
+
+FS_Error storage_int_backup(Storage* api, const char* dstname) {
+    TarArchive* archive = tar_archive_alloc(api);
+    bool success = tar_archive_open(archive, dstname, TAR_OPEN_MODE_WRITE) &&
+                   tar_archive_add_dir(archive, INT_PATH, "") && tar_archive_finalize(archive);
+    tar_archive_free(archive);
+    return success ? FSE_OK : FSE_INTERNAL;
+}
+
+FS_Error storage_int_restore(Storage* api, const char* srcname) {
+    TarArchive* archive = tar_archive_alloc(api);
+    bool success = tar_archive_open(archive, srcname, TAR_OPEN_MODE_READ) &&
+                   tar_archive_unpack_to(archive, INT_PATH);
+    tar_archive_free(archive);
+    return success ? FSE_OK : FSE_INTERNAL;
+}

applications/storage/storage_processing.c

@@ -20,7 +20,11 @@ static StorageData* storage_get_storage_by_type(Storage* app, StorageType type)
 }
 
 static bool storage_type_is_not_valid(StorageType type) {
+#ifdef FURI_RAM_EXEC
+    return type != ST_EXT;
+#else
     return type >= ST_ERROR;
+#endif
 }
 
 static StorageData* get_storage_by_file(File* file, StorageData* storages) {

applications/storage/storages/storage_ext.c

@@ -54,10 +54,12 @@ static bool sd_mount_card(StorageData* storage, bool notify) {
             SDError status = f_mount(sd_data->fs, sd_data->path, 1);
 
             if(status == FR_OK || status == FR_NO_FILESYSTEM) {
+#ifndef FURI_RAM_EXEC
                 FATFS* fs;
                 uint32_t free_clusters;
 
                 status = f_getfree(sd_data->path, &free_clusters, &fs);
+#endif
 
                 if(status == FR_OK || status == FR_NO_FILESYSTEM) {
                     result = true;
@@ -110,6 +112,9 @@ FS_Error sd_unmount_card(StorageData* storage) {
 }
 
 FS_Error sd_format_card(StorageData* storage) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     uint8_t* work_area;
     SDData* sd_data = storage->data;
     SDError error;
@@ -135,11 +140,14 @@ FS_Error sd_format_card(StorageData* storage) {
     storage_data_unlock(storage);
 
     return storage_ext_parse_error(error);
+#endif
 }
 
 FS_Error sd_card_info(StorageData* storage, SDInfo* sd_info) {
+#ifndef FURI_RAM_EXEC
     uint32_t free_clusters, free_sectors, total_sectors;
     FATFS* fs;
+#endif
     SDData* sd_data = storage->data;
     SDError error;
 
@@ -150,20 +158,32 @@ FS_Error sd_card_info(StorageData* storage, SDInfo* sd_info) {
     storage_data_lock(storage);
     error = f_getlabel(sd_data->path, sd_info->label, NULL);
     if(error == FR_OK) {
+#ifndef FURI_RAM_EXEC
         error = f_getfree(sd_data->path, &free_clusters, &fs);
+#endif
     }
     storage_data_unlock(storage);
 
     if(error == FR_OK) {
         // calculate size
+#ifndef FURI_RAM_EXEC
         total_sectors = (fs->n_fatent - 2) * fs->csize;
         free_sectors = free_clusters * fs->csize;
+#endif
 
         uint16_t sector_size = _MAX_SS;
 #if _MAX_SS != _MIN_SS
         sector_size = fs->ssize;
 #endif
 
+#ifdef FURI_RAM_EXEC
+        sd_info->fs_type = 0;
+        sd_info->kb_total = 0;
+        sd_info->kb_free = 0;
+        sd_info->cluster_size = 512;
+        sd_info->sector_size = sector_size;
+#else
+        sd_info->fs_type = fs->fs_type;
         switch(fs->fs_type) {
         case FS_FAT12:
             sd_info->fs_type = FST_FAT12;
@@ -177,7 +197,6 @@ FS_Error sd_card_info(StorageData* storage, SDInfo* sd_info) {
         case FS_EXFAT:
             sd_info->fs_type = FST_EXFAT;
             break;
-
         default:
             sd_info->fs_type = FST_UNKNOWN;
             break;
@@ -187,6 +206,7 @@ FS_Error sd_card_info(StorageData* storage, SDInfo* sd_info) {
         sd_info->kb_free = free_sectors / 1024 * sector_size;
         sd_info->cluster_size = fs->csize;
         sd_info->sector_size = sector_size;
+#endif
     }
 
     return storage_ext_parse_error(error);
@@ -328,12 +348,16 @@ static uint16_t
 
 static uint16_t
     storage_ext_file_write(void* ctx, File* file, const void* buff, uint16_t const bytes_to_write) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     StorageData* storage = ctx;
     SDFile* file_data = storage_get_storage_file_data(file, storage);
     uint16_t bytes_written = 0;
     file->internal_error_id = f_write(file_data, buff, bytes_to_write, &bytes_written);
     file->error_id = storage_ext_parse_error(file->internal_error_id);
     return bytes_written;
+#endif
 }
 
 static bool
@@ -364,21 +388,29 @@ static uint64_t storage_ext_file_tell(void* ctx, File* file) {
 }
 
 static bool storage_ext_file_truncate(void* ctx, File* file) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     StorageData* storage = ctx;
     SDFile* file_data = storage_get_storage_file_data(file, storage);
 
     file->internal_error_id = f_truncate(file_data);
     file->error_id = storage_ext_parse_error(file->internal_error_id);
     return (file->error_id == FSE_OK);
+#endif
 }
 
 static bool storage_ext_file_sync(void* ctx, File* file) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     StorageData* storage = ctx;
     SDFile* file_data = storage_get_storage_file_data(file, storage);
 
     file->internal_error_id = f_sync(file_data);
     file->error_id = storage_ext_parse_error(file->internal_error_id);
     return (file->error_id == FSE_OK);
+#endif
 }
 
 static uint64_t storage_ext_file_size(void* ctx, File* file) {
@@ -479,13 +511,21 @@ static FS_Error storage_ext_common_stat(void* ctx, const char* path, FileInfo* f
 }
 
 static FS_Error storage_ext_common_remove(void* ctx, const char* path) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     SDError result = f_unlink(path);
     return storage_ext_parse_error(result);
+#endif
 }
 
 static FS_Error storage_ext_common_mkdir(void* ctx, const char* path) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     SDError result = f_mkdir(path);
     return storage_ext_parse_error(result);
+#endif
 }
 
 static FS_Error storage_ext_common_fs_info(
@@ -493,6 +533,9 @@ static FS_Error storage_ext_common_fs_info(
     const char* fs_path,
     uint64_t* total_space,
     uint64_t* free_space) {
+#ifdef FURI_RAM_EXEC
+    return FSE_NOT_READY;
+#else
     StorageData* storage = ctx;
     SDData* sd_data = storage->data;
 
@@ -519,6 +562,7 @@ static FS_Error storage_ext_common_fs_info(
     }
 
     return storage_ext_parse_error(fresult);
+#endif
 }
 
 /******************* Init Storage *******************/
@@ -566,4 +610,4 @@ void storage_ext_init(StorageData* storage) {
 
     // do not notify on first launch, notifications app is waiting for our thread to read settings
     storage_ext_tick_internal(storage, false);
-}
+}